2024-01-16T14:43:10.051 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T14:43:10.051 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196 --verbose --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --owner scheduled_rishabh@teuthology --name rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/orig.config.yaml 2024-01-16T14:43:10.092 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T14:43:10.178 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7518196' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3480 sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpF1KD1h76HBjH0XwN4zVha7la215WjgX4udBytaTPcNiEBzn4tDScWUgsRiVwXEaZSoZ6/lXdIkZtebJUZyYk= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPobHIqFf3RsbMpRH+fQ4bw7EVvPC1Vk9jbTiyHsYLDA1NXgfyymiah/c/7lPSi04qSVHSSA3UT0YkLpU3o3WsQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_14:38:05 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T14:43:10.178 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-16T14:43:10.179 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-16T14:43:10.179 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T14:43:10.180 INFO:teuthology.task.internal:Checking packages... 2024-01-16T14:43:10.202 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '73a4532fc4d97adee0b985d4d42874ea351a1c86' 2024-01-16T14:43:10.202 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T14:43:10.202 INFO:teuthology.packaging:ref: None 2024-01-16T14:43:10.203 INFO:teuthology.packaging:tag: None 2024-01-16T14:43:10.203 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-16T14:43:10.203 INFO:teuthology.packaging:sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:43:10.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-01-16T14:43:10.483 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g73a4532f 2024-01-16T14:43:10.484 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T14:43:10.491 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T14:43:10.491 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T14:43:10.513 INFO:teuthology.task.internal:Saving configuration 2024-01-16T14:43:10.528 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T14:43:10.534 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T14:43:10.558 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:13.206342', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpF1KD1h76HBjH0XwN4zVha7la215WjgX4udBytaTPcNiEBzn4tDScWUgsRiVwXEaZSoZ6/lXdIkZtebJUZyYk='} 2024-01-16T14:43:10.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:13.207698', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPobHIqFf3RsbMpRH+fQ4bw7EVvPC1Vk9jbTiyHsYLDA1NXgfyymiah/c/7lPSi04qSVHSSA3UT0YkLpU3o3WsQ='} 2024-01-16T14:43:10.583 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T14:43:10.588 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-16T14:43:10.588 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-16T14:43:10.588 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T14:43:10.659 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb865351160>, signals=[15]) 2024-01-16T14:43:10.660 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T14:43:10.666 INFO:teuthology.task.internal:Opening connections... 2024-01-16T14:43:10.666 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:43:10.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:10.773 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:43:10.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:10.859 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T14:43:10.865 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-01-16T14:43:10.882 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-01-16T14:43:10.883 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-01-16T14:43:10.939 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-01-16T14:43:10.939 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-01-16T14:43:10.939 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:10.940 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:10.941 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-01-16T14:43:10.995 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-01-16T14:43:11.012 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-01-16T14:43:11.012 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:43:11.069 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:43:11.070 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:43:11.070 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:43:11.070 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-01-16T14:43:11.093 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T14:43:11.102 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T14:43:11.107 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T14:43:11.107 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:11.110 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:43:11.125 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T14:43:11.131 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T14:43:11.131 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:11.166 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:43:11.192 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T14:43:11.246 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T14:43:11.260 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-16T14:43:11.260 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-16T14:43:11.260 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:11.261 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:43:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:11.277 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.277 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:11.277 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-01-16T14:43:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:43:11.278 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.278 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:43:11.278 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-01-16T14:43:11.333 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:11.334 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.334 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:11.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.338 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:43:11.338 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.338 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:43:11.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:43:11.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:11.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum upgrade -y linux-firmware 2024-01-16T14:43:14.031 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.4 MB 00:00 2024-01-16T14:43:14.152 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.4 MB 00:00 2024-01-16T14:43:15.806 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-01-16T14:43:15.910 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-01-16T14:43:18.175 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-01-16T14:43:18.721 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-01-16T14:43:22.510 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-01-16T14:43:23.114 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-01-16T14:43:27.822 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.9 MB 00:01 2024-01-16T14:43:28.321 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.9 MB 00:02 2024-01-16T14:43:29.398 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-01-16T14:43:30.135 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-01-16T14:43:30.335 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2024-01-16T14:43:32.112 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-01-16T14:43:35.166 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-16T14:43:37.095 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-16T14:43:37.459 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-16T14:43:37.470 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-16T14:43:37.470 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 1 Package 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 375 M 2024-01-16T14:43:37.471 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-16T14:43:39.521 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-01-16T14:43:39.530 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 1 Package 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 375 M 2024-01-16T14:43:39.531 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-01-16T14:44:01.897 INFO:teuthology.orchestra.run.smithi069.stdout:linux-firmware-20240109-142.el9.noarch.rpm 15 MB/s | 375 MB 00:24 2024-01-16T14:44:01.898 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:01.898 INFO:teuthology.orchestra.run.smithi069.stdout:Total 15 MB/s | 375 MB 00:24 2024-01-16T14:44:01.899 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-16T14:44:01.987 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-16T14:44:01.987 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-16T14:44:03.526 INFO:teuthology.orchestra.run.smithi043.stdout:linux-firmware-20240109-142.el9.noarch.rpm 16 MB/s | 375 MB 00:23 2024-01-16T14:44:03.527 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:44:03.527 INFO:teuthology.orchestra.run.smithi043.stdout:Total 16 MB/s | 375 MB 00:23 2024-01-16T14:44:03.527 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-01-16T14:44:03.622 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-01-16T14:44:03.623 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-01-16T14:44:04.403 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-16T14:44:04.404 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-16T14:44:06.109 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-01-16T14:44:06.110 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-01-16T14:44:10.699 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-16T14:44:10.956 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:11.028 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-16T14:44:12.438 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-01-16T14:44:12.666 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:44:12.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:12.687 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:12.785 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:12.817 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:12.817 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:12.820 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi069.front.sepia.ceph.com to complete... 2024-01-16T14:44:13.663 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:13.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:44:14.083 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:44:14.084 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:44:14.084 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-01-16T14:44:14.084 INFO:teuthology.orchestra.run.smithi043.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:44:14.084 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:44:14.084 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-01-16T14:44:14.398 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:44:14.398 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:14.797 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:44:14.797 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:44:14.799 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi043.front.sepia.ceph.com to complete... 2024-01-16T14:44:50.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16 14:44:50 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:44:50.763 INFO:teuthology.orchestra.run.smithi043.stderr:FINISHED --2024-01-16 14:44:50-- 2024-01-16T14:44:50.763 INFO:teuthology.orchestra.run.smithi043.stderr:Total wall clock time: 36s 2024-01-16T14:44:50.763 INFO:teuthology.orchestra.run.smithi043.stderr:Downloaded: 1 files, 315M in 36s (8.84 MB/s) 2024-01-16T14:44:50.781 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:44:50.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:44:52.782 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying... ######################################## 2024-01-16T14:44:52.854 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing... ######################################## 2024-01-16T14:44:52.939 INFO:teuthology.orchestra.run.smithi043.stdout:Updating / installing... 2024-01-16T14:45:00.703 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:45:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:45:37.995 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:45:37.995 DEBUG:teuthology.orchestra.run.smithi043:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:45:38.071 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:45:38.072 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:45:38.073 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:45:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:45:38.075 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:45:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:45:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:45:38.078 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:45:38.079 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:45:38.080 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:45:38.081 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:45:38.082 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:45:38.083 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:45:38.084 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:45:38.085 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:45:38.086 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:45:38.087 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:45:38.088 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:45:38.089 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:45:38.090 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:45:38.091 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:45:38.092 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:45:38.093 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:45:38.094 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:45:38.095 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:45:38.096 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:45:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:45:38.098 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:45:38.099 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:45:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:45:38.101 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:45:38.102 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:45:38.103 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:45:38.104 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:45:38.105 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:45:38.106 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:45:38.107 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:45:38.108 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:45:38.109 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:45:38.110 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:45:38.111 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:45:38.112 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:45:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:45:38.114 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:45:38.115 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:45:38.116 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:45:38.117 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:45:38.118 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:45:38.119 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:45:38.120 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:45:38.121 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:45:38.122 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:45:38.123 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:45:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:45:38.125 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:45:38.126 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:45:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:45:38.128 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:45:38.129 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:45:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:45:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:45:38.132 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:45:38.133 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:45:38.134 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:45:38.135 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:45:38.136 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:45:38.137 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:45:38.138 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:45:38.139 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:45:38.140 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:45:38.141 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:45:38.142 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:45:38.142 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:45:38.142 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:45:38.142 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:45:38.143 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:45:38.144 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:45:38.145 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:45:38.146 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:45:38.147 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:45:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:45:38.149 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:45:38.150 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:45:38.151 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:45:38.152 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:45:38.153 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:45:38.154 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:45:38.155 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:45:38.156 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:45:38.157 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:45:38.158 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:45:38.159 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:45:38.160 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:45:38.161 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:45:38.162 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:45:38.163 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:45:38.164 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:45:38.165 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:45:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:45:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:45:38.168 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:45:38.169 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:45:38.170 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:45:38.171 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:45:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:45:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:45:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:45:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:45:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:45:38.177 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:45:38.178 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:45:38.179 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:45:38.180 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:45:38.181 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:45:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:45:38.183 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:45:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:45:38.185 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:45:38.186 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:45:38.187 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:45:38.188 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:45:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:45:38.190 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:45:38.191 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:45:38.192 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:45:38.193 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:45:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:45:38.195 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:45:38.196 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:45:38.197 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:45:38.198 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:45:38.199 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:45:38.200 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:45:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:45:38.202 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:45:38.203 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:45:38.204 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:45:38.204 DEBUG:teuthology.orchestra.run.smithi043:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:45:38.227 INFO:teuthology.orchestra.run.smithi043.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-01-16T14:45:38.228 INFO:teuthology.orchestra.run.smithi043.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:45:38.229 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:45:38.229 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:45:38.229 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-01-16T14:45:38.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -qi grub2-tools 2024-01-16T14:45:38.318 INFO:teuthology.orchestra.run.smithi043.stdout:Name : grub2-tools 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Epoch : 1 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Version : 2.06 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Release : 68.el9 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: x86_64 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Group : Unspecified 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Size : 8264254 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:License : GPLv3+ 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Packager : builder@centos.org 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Vendor : CentOS 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:45:38.319 INFO:teuthology.orchestra.run.smithi043.stdout:Summary : Support tools for GRUB. 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:Description : 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:hardware devices. 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:45:38.320 INFO:teuthology.orchestra.run.smithi043.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:45:38.321 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:45:38.321 INFO:teuthology.task.kernel:Updating grub on smithi043 to boot 6.7.0-g2668949e0971 2024-01-16T14:45:38.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:45:38.779 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2024-01-16T14:45:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:45:40.202 INFO:teuthology.orchestra.run.smithi043.stderr:done 2024-01-16T14:45:40.204 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-01-16T14:45:40.219 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.3LHCVBkBOl 2024-01-16T14:45:40.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3LHCVBkBOl 2024-01-16T14:45:40.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.3LHCVBkBOl 2024-01-16T14:45:40.442 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.3LHCVBkBOl is 6KB 2024-01-16T14:45:40.462 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.3LHCVBkBOl 2024-01-16T14:45:40.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:45:40.544 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:45:40.544 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:45:40.544 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:45:40.544 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:45:40.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:45:40.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2024-01-16T14:46:10.680 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T14:46:10.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:46:10.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:46:29.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-01-16T14:46:38.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:46:38.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:31.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16 14:47:31 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:47:31.840 INFO:teuthology.orchestra.run.smithi069.stderr:FINISHED --2024-01-16 14:47:31-- 2024-01-16T14:47:31.841 INFO:teuthology.orchestra.run.smithi069.stderr:Total wall clock time: 3m 19s 2024-01-16T14:47:31.841 INFO:teuthology.orchestra.run.smithi069.stderr:Downloaded: 1 files, 315M in 3m 18s (1.59 MB/s) 2024-01-16T14:47:31.844 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:47:31.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:47:33.785 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying... ######################################## 2024-01-16T14:47:33.846 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing... ######################################## 2024-01-16T14:47:33.926 INFO:teuthology.orchestra.run.smithi069.stdout:Updating / installing... 2024-01-16T14:47:38.185 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T14:47:41.315 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:47:50.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:47:50.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:50.609 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-01-16T14:47:51.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:47:51.283 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-16T14:47:51.283 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-01-16T14:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout:6.7.0-g2668949e0971 2024-01-16T14:47:51.337 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:47:51.337 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:47:51.337 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:47:52.338 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T14:47:52.338 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-01-16T14:47:52.524 DEBUG:teuthology.parallel:result is None 2024-01-16T14:47:55.218 INFO:teuthology.orchestra.run.smithi069.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:48:17.493 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:48:17.493 DEBUG:teuthology.orchestra.run.smithi069:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:48:17.568 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:48:17.569 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:48:17.570 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:48:17.571 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:48:17.572 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:48:17.573 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:48:17.574 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:48:17.575 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:48:17.576 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:48:17.577 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:48:17.578 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:48:17.579 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:48:17.580 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:48:17.581 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:48:17.582 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:48:17.583 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:48:17.584 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:48:17.585 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:48:17.586 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:48:17.587 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:48:17.588 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:48:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:48:17.590 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:48:17.591 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:48:17.592 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:48:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:48:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:48:17.595 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:48:17.596 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:48:17.597 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:48:17.598 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:48:17.599 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:48:17.600 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:48:17.601 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:48:17.602 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:48:17.603 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:48:17.604 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:48:17.605 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:48:17.606 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:48:17.607 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:48:17.608 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:48:17.609 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:48:17.610 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:48:17.611 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:48:17.612 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:48:17.613 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:48:17.614 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:48:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:48:17.616 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:48:17.617 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:48:17.618 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:48:17.619 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:48:17.620 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:48:17.621 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:48:17.622 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:48:17.623 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:48:17.624 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:48:17.625 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:48:17.626 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:48:17.627 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:48:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:48:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:48:17.630 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:48:17.631 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:48:17.632 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:48:17.633 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:48:17.634 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:48:17.635 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:48:17.636 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:48:17.637 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:48:17.638 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:48:17.639 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:48:17.640 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:48:17.641 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:48:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:48:17.643 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:48:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:48:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:48:17.646 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:48:17.647 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:48:17.648 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:48:17.649 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:48:17.650 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:48:17.651 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:48:17.652 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:48:17.653 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:48:17.654 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:48:17.655 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:48:17.656 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:48:17.657 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:48:17.658 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:48:17.659 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:48:17.660 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:48:17.661 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:48:17.662 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:48:17.663 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:48:17.664 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:48:17.665 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:48:17.666 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:48:17.667 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:48:17.668 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:48:17.669 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:48:17.670 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:48:17.671 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:48:17.672 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:48:17.673 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:48:17.674 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:48:17.675 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:48:17.676 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:48:17.677 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:48:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:48:17.679 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:48:17.680 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:48:17.681 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:48:17.682 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:48:17.683 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:48:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:48:17.685 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:48:17.686 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:48:17.687 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:48:17.688 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:48:17.689 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:48:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:48:17.691 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:48:17.692 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:48:17.693 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:48:17.694 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:48:17.695 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:48:17.696 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:48:17.697 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:48:17.698 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:48:17.699 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:48:17.700 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:48:17.701 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:48:17.702 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:48:17.703 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:48:17.704 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:48:17.705 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:48:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:48:17.707 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:48:17.707 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:48:17.708 DEBUG:teuthology.orchestra.run.smithi069:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:48:17.729 INFO:teuthology.orchestra.run.smithi069.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:48:17.729 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:48:17.729 INFO:teuthology.orchestra.run.smithi069.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:48:17.729 INFO:teuthology.orchestra.run.smithi069.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:48:17.730 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-01-16T14:48:17.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-01-16T14:48:17.805 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:48:17.806 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:48:17.807 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:48:17.808 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:48:17.808 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 6.7.0-g2668949e0971 2024-01-16T14:48:17.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:48:18.408 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-01-16T14:48:19.790 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:48:19.821 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-01-16T14:48:19.833 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-16T14:48:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.w15ghM5mUl 2024-01-16T14:48:19.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w15ghM5mUl 2024-01-16T14:48:19.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.w15ghM5mUl 2024-01-16T14:48:20.051 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.w15ghM5mUl is 6KB 2024-01-16T14:48:20.099 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.w15ghM5mUl 2024-01-16T14:48:20.113 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:48:20.177 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:48:20.177 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:48:20.177 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:48:20.177 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:48:20.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:48:20.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-01-16T14:48:50.350 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-16T14:48:50.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:48:50.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:49:08.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-01-16T14:49:17.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:49:17.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:50:17.673 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T14:50:29.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:50:29.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:50:30.026 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-01-16T14:50:30.689 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:50:30.690 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-16T14:50:30.690 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-01-16T14:50:30.739 INFO:teuthology.orchestra.run.smithi069.stdout:6.7.0-g2668949e0971 2024-01-16T14:50:30.739 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:50:30.739 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:50:30.739 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:50:31.740 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-16T14:50:31.740 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:50:31.883 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-01-16T14:50:31.931 DEBUG:teuthology.parallel:result is None 2024-01-16T14:50:31.932 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T14:50:31.939 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T14:50:31.939 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:50:31.947 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:50:32.000 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T14:50:32.006 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T14:50:32.019 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T14:50:32.019 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:50:32.024 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:50:32.092 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T14:50:32.098 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T14:50:32.099 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:50:32.106 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:50:32.208 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:32.230 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:32.254 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:32.273 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:32.278 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T14:50:32.285 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T14:50:32.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:50:32.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:50:32.388 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T14:50:32.396 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T14:50:32.397 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:50:32.404 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:50:32.456 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:32.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:32.661 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:32.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:32.848 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:50:32.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:50:32.946 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:33.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:33.144 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:33.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:33.322 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:50:33.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:50:33.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-01-16T14:50:33.422 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-01-16T14:50:33.551 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:50:33.577 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:50:33.917 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T14:50:33.925 INFO:teuthology.task.internal:Starting timer... 2024-01-16T14:50:33.925 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T14:50:33.934 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T14:50:33.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-01-16T14:50:34.082 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-01-16T14:50:34.086 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-01-16T14:50:34.230 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-01-16T14:50:34.235 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T14:50:34.235 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-01-16T14:50:34.300 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-01-16T14:50:34.300 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-01-16T14:50:34.364 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-01-16T14:50:34.365 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive'} 2024-01-16T14:50:34.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:50:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:50:34.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:50:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:50:34.559 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T14:50:34.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:50:34.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:50:34.778 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T14:50:34.788 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-16T14:50:34.793 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-16T14:50:34.949 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T14:50:34.965 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T14:50:34.966 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T14:56:21.284 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2024-01-16T14:56:21.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:56:21.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:56:21.436 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-01-16T14:56:21.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-01-16T14:56:21.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:56:21.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:56:21.733 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-01-16T14:56:21.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-16T14:56:21.894 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T14:56:21.904 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T14:56:21.904 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:56:21.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:56:21.912 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:56:21.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:56:22.008 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:56:22.015 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:56:22.060 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:56:22.152 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-01-16T14:56:22.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-01-16T14:56:22.192 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-01-16T14:56:22.220 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-01-16T14:56:22.242 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:56:22.275 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:56:22.293 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:56:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:56:22.411 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.421 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.473 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:22.474 INFO:teuthology.run_tasks:Running task install... 2024-01-16T14:56:22.483 DEBUG:teuthology.task.install:project ceph 2024-01-16T14:56:22.483 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}} 2024-01-16T14:56:22.483 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'} 2024-01-16T14:56:22.483 INFO:teuthology.task.install:Using flavor: default 2024-01-16T14:56:22.488 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T14:56:22.488 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-16T14:56:22.489 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:56:22.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:56:22.491 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:56:22.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:56:22.724 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:56:22.725 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:56:22.728 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:56:22.728 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:56:22.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:56:22.840 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:56:22.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:56:22.861 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:56:22.862 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:56:22.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:56:22.952 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:56:22.952 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:56:22.963 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:56:22.963 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:56:23.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:56:23.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:56:23.238 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:56:23.242 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:56:23.327 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-01-16T14:56:23.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-01-16T14:56:23.339 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-01-16T14:56:23.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-01-16T14:56:23.695 INFO:teuthology.orchestra.run.smithi069.stdout:73 files removed 2024-01-16T14:56:23.719 INFO:teuthology.orchestra.run.smithi043.stdout:73 files removed 2024-01-16T14:56:23.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:56:23.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:56:24.641 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 193 kB/s | 78 kB 00:00 2024-01-16T14:56:24.704 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 190 kB/s | 78 kB 00:00 2024-01-16T14:56:25.220 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2024-01-16T14:56:25.221 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 23 kB/s | 9.5 kB 00:00 2024-01-16T14:56:25.747 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-01-16T14:56:25.762 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-01-16T14:56:26.673 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.4 MB 00:00 2024-01-16T14:56:26.725 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.4 MB 00:00 2024-01-16T14:56:28.650 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-01-16T14:56:28.735 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-01-16T14:56:30.587 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-01-16T14:56:32.898 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-01-16T14:56:33.713 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-01-16T14:56:36.164 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-01-16T14:56:38.055 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.9 MB 00:01 2024-01-16T14:56:40.106 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 15 kB 00:00 2024-01-16T14:56:40.782 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 5.9 MB 00:00 2024-01-16T14:56:42.524 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-01-16T14:56:45.824 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 20 MB 00:05 2024-01-16T14:56:51.432 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-16T14:56:51.681 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 20 MB 00:08 2024-01-16T14:56:54.122 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:56:54.124 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:56:54.142 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:56:54.143 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:56:54.145 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:56:54.146 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:56:54.148 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:56:54.152 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:56:54.153 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:56:54.234 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:56:54.254 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:56:54.255 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:56:54.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:56:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:56:54.258 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:56:54.259 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:56:54.260 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:56:54.261 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:56:54.262 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:56:54.263 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:56:54.264 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:56:54.265 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout:Install 135 Packages 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 16 Packages 2024-01-16T14:56:54.266 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:56:54.267 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 204 M 2024-01-16T14:56:54.267 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-16T14:56:55.008 INFO:teuthology.orchestra.run.smithi069.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-01-16T14:56:55.200 INFO:teuthology.orchestra.run.smithi069.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 4.4 MB/s | 861 kB 00:00 2024-01-16T14:56:55.275 INFO:teuthology.orchestra.run.smithi069.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.9 MB/s | 141 kB 00:00 2024-01-16T14:56:55.517 INFO:teuthology.orchestra.run.smithi069.stdout:(4/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 8.7 MB/s | 2.1 MB 00:00 2024-01-16T14:56:55.701 INFO:teuthology.orchestra.run.smithi069.stdout:(5/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 8.0 MB/s | 1.5 MB 00:00 2024-01-16T14:56:56.193 INFO:teuthology.orchestra.run.smithi069.stdout:(6/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 8.9 MB/s | 4.4 MB 00:00 2024-01-16T14:56:56.494 INFO:teuthology.orchestra.run.smithi069.stdout:(7/151): ceph-common-18.0.0-7629.g73a4532f.el9. 11 MB/s | 19 MB 00:01 2024-01-16T14:56:57.077 INFO:teuthology.orchestra.run.smithi069.stdout:(8/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 2.2 MB/s | 5.2 MB 00:02 2024-01-16T14:56:57.235 INFO:teuthology.orchestra.run.smithi069.stdout:(9/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 12 MB/s | 8.6 MB 00:00 2024-01-16T14:56:57.260 INFO:teuthology.orchestra.run.smithi069.stdout:(10/151): ceph-selinux-18.0.0-7629.g73a4532f.el 138 kB/s | 25 kB 00:00 2024-01-16T14:56:57.394 INFO:teuthology.orchestra.run.smithi069.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 1.6 MB/s | 217 kB 00:00 2024-01-16T14:56:57.483 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-16T14:56:57.511 INFO:teuthology.orchestra.run.smithi069.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 269 kB/s | 31 kB 00:00 2024-01-16T14:56:57.844 INFO:teuthology.orchestra.run.smithi069.stdout:(13/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 11 MB/s | 18 MB 00:01 2024-01-16T14:56:57.886 INFO:teuthology.orchestra.run.smithi069.stdout:(14/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 1.9 MB/s | 719 kB 00:00 2024-01-16T14:56:57.920 INFO:teuthology.orchestra.run.smithi069.stdout:(15/151): libcephsqlite-18.0.0-7629.g73a4532f.e 2.1 MB/s | 163 kB 00:00 2024-01-16T14:56:57.953 INFO:teuthology.orchestra.run.smithi069.stdout:(16/151): librados-devel-18.0.0-7629.g73a4532f. 1.9 MB/s | 127 kB 00:00 2024-01-16T14:56:58.012 INFO:teuthology.orchestra.run.smithi069.stdout:(17/151): libradosstriper1-18.0.0-7629.g73a4532 5.1 MB/s | 476 kB 00:00 2024-01-16T14:56:58.071 INFO:teuthology.orchestra.run.smithi069.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 786 kB/s | 45 kB 00:00 2024-01-16T14:56:58.146 INFO:teuthology.orchestra.run.smithi069.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 1.7 MB/s | 129 kB 00:00 2024-01-16T14:56:58.230 INFO:teuthology.orchestra.run.smithi069.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 1.9 MB/s | 161 kB 00:00 2024-01-16T14:56:58.330 INFO:teuthology.orchestra.run.smithi069.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 3.2 MB/s | 321 kB 00:00 2024-01-16T14:56:58.431 INFO:teuthology.orchestra.run.smithi069.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 2.9 MB/s | 298 kB 00:00 2024-01-16T14:56:58.522 INFO:teuthology.orchestra.run.smithi069.stdout:(23/151): python3-rgw-18.0.0-7629.g73a4532f.el9 1.1 MB/s | 99 kB 00:00 2024-01-16T14:56:58.606 INFO:teuthology.orchestra.run.smithi069.stdout:(24/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 1.0 MB/s | 85 kB 00:00 2024-01-16T14:56:58.806 INFO:teuthology.orchestra.run.smithi069.stdout:(25/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 5.6 MB/s | 4.8 MB 00:00 2024-01-16T14:56:58.973 INFO:teuthology.orchestra.run.smithi069.stdout:(26/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 8.2 MB/s | 3.0 MB 00:00 2024-01-16T14:56:59.074 INFO:teuthology.orchestra.run.smithi069.stdout:(27/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 641 kB/s | 171 kB 00:00 2024-01-16T14:56:59.099 INFO:teuthology.orchestra.run.smithi069.stdout:(28/151): ceph-grafana-dashboards-18.0.0-7629.g 225 kB/s | 28 kB 00:00 2024-01-16T14:56:59.149 INFO:teuthology.orchestra.run.smithi069.stdout:(29/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 1.7 MB/s | 133 kB 00:00 2024-01-16T14:56:59.358 INFO:teuthology.orchestra.run.smithi069.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 6.8 MB/s | 1.8 MB 00:00 2024-01-16T14:56:59.433 INFO:teuthology.orchestra.run.smithi069.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 3.3 MB/s | 246 kB 00:00 2024-01-16T14:56:59.500 INFO:teuthology.orchestra.run.smithi069.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 763 kB/s | 50 kB 00:00 2024-01-16T14:56:59.567 INFO:teuthology.orchestra.run.smithi069.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g7 230 kB/s | 15 kB 00:00 2024-01-16T14:57:00.076 INFO:teuthology.orchestra.run.smithi069.stdout:(34/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 15 MB/s | 41 MB 00:02 2024-01-16T14:57:00.109 INFO:teuthology.orchestra.run.smithi069.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 485 kB/s | 262 kB 00:00 2024-01-16T14:57:00.151 INFO:teuthology.orchestra.run.smithi069.stdout:(36/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 448 kB/s | 18 kB 00:00 2024-01-16T14:57:00.193 INFO:teuthology.orchestra.run.smithi069.stdout:(37/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 6.5 MB/s | 778 kB 00:00 2024-01-16T14:57:00.293 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:00.295 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:00.315 INFO:teuthology.orchestra.run.smithi043.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:57:00.316 INFO:teuthology.orchestra.run.smithi043.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:57:00.318 INFO:teuthology.orchestra.run.smithi043.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:57:00.319 INFO:teuthology.orchestra.run.smithi043.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:00.321 INFO:teuthology.orchestra.run.smithi043.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:00.325 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:57:00.326 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:57:00.416 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:57:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:57:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:57:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:57:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:57:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:57:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:57:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:57:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:57:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout:Install 135 Packages 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 16 Packages 2024-01-16T14:57:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi069.stdout:(38/151): ceph-mgr-diskprediction-local-18.0.0- 5.7 MB/s | 7.4 MB 00:01 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 204 M 2024-01-16T14:57:00.450 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-01-16T14:57:00.636 INFO:teuthology.orchestra.run.smithi069.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 920 kB 00:00 2024-01-16T14:57:00.727 INFO:teuthology.orchestra.run.smithi069.stdout:(40/151): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 662 kB 00:00 2024-01-16T14:57:00.752 INFO:teuthology.orchestra.run.smithi069.stdout:(41/151): boost-program-options-1.75.0-8.el9.x8 347 kB/s | 106 kB 00:00 2024-01-16T14:57:00.795 INFO:teuthology.orchestra.run.smithi069.stdout:(42/151): elfutils-libelf-devel-0.190-2.el9.x86 130 kB/s | 20 kB 00:00 2024-01-16T14:57:00.861 INFO:teuthology.orchestra.run.smithi069.stdout:(43/151): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 307 kB 00:00 2024-01-16T14:57:00.886 INFO:teuthology.orchestra.run.smithi069.stdout:(44/151): flexiblas-3.0.4-8.el9.x86_64.rpm 354 kB/s | 32 kB 00:00 2024-01-16T14:57:01.154 INFO:teuthology.orchestra.run.smithi069.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 59 kB/s | 17 kB 00:00 2024-01-16T14:57:01.179 INFO:teuthology.orchestra.run.smithi069.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 34 kB/s | 10 kB 00:00 2024-01-16T14:57:01.512 INFO:teuthology.orchestra.run.smithi069.stdout:(47/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 133 kB/s | 44 kB 00:00 2024-01-16T14:57:01.538 INFO:teuthology.orchestra.run.smithi069.stdout:(48/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 159 kB 00:00 2024-01-16T14:57:01.621 INFO:teuthology.orchestra.run.smithi069.stdout:(49/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.6 MB/s | 3.0 MB 00:00 2024-01-16T14:57:01.713 INFO:teuthology.orchestra.run.smithi069.stdout:(50/151): librdkafka-1.6.1-102.el9.x86_64.rpm 3.3 MB/s | 661 kB 00:00 2024-01-16T14:57:01.751 INFO:teuthology.orchestra.run.smithi043.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-01-16T14:57:01.822 INFO:teuthology.orchestra.run.smithi069.stdout:(51/151): libtool-2.4.6-45.el9.x86_64.rpm 2.0 MB/s | 578 kB 00:00 2024-01-16T14:57:01.839 INFO:teuthology.orchestra.run.smithi069.stdout:(52/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 114 kB/s | 24 kB 00:00 2024-01-16T14:57:01.855 INFO:teuthology.orchestra.run.smithi069.stdout:(53/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 332 kB/s | 46 kB 00:00 2024-01-16T14:57:01.923 INFO:teuthology.orchestra.run.smithi069.stdout:(54/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-01-16T14:57:01.939 INFO:teuthology.orchestra.run.smithi069.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 418 kB/s | 34 kB 00:00 2024-01-16T14:57:01.973 INFO:teuthology.orchestra.run.smithi069.stdout:(56/151): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-16T14:57:02.035 INFO:teuthology.orchestra.run.smithi043.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 3.0 MB/s | 861 kB 00:00 2024-01-16T14:57:02.118 INFO:teuthology.orchestra.run.smithi043.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.7 MB/s | 141 kB 00:00 2024-01-16T14:57:02.260 INFO:teuthology.orchestra.run.smithi043.stdout:(4/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 6.7 MB/s | 5.2 MB 00:00 2024-01-16T14:57:02.444 INFO:teuthology.orchestra.run.smithi043.stdout:(5/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 8.0 MB/s | 1.5 MB 00:00 2024-01-16T14:57:02.519 INFO:teuthology.orchestra.run.smithi043.stdout:(6/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 5.2 MB/s | 2.1 MB 00:00 2024-01-16T14:57:02.840 INFO:teuthology.orchestra.run.smithi069.stdout:(57/151): postgresql-13.11-1.el9.x86_64.rpm 1.8 MB/s | 1.5 MB 00:00 2024-01-16T14:57:02.869 INFO:teuthology.orchestra.run.smithi043.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 10 MB/s | 4.4 MB 00:00 2024-01-16T14:57:02.940 INFO:teuthology.orchestra.run.smithi069.stdout:(58/151): openblas-openmp-0.3.21-2.el9.x86_64.r 4.7 MB/s | 4.7 MB 00:01 2024-01-16T14:57:03.024 INFO:teuthology.orchestra.run.smithi069.stdout:(59/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-01-16T14:57:03.058 INFO:teuthology.orchestra.run.smithi069.stdout:(60/151): postgresql-contrib-13.11-1.el9.x86_64 3.8 MB/s | 811 kB 00:00 2024-01-16T14:57:03.082 INFO:teuthology.orchestra.run.smithi069.stdout:(61/151): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-01-16T14:57:03.391 INFO:teuthology.orchestra.run.smithi069.stdout:(62/151): postgresql-server-13.11-1.el9.x86_64. 16 MB/s | 5.7 MB 00:00 2024-01-16T14:57:03.425 INFO:teuthology.orchestra.run.smithi069.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 693 kB/s | 228 kB 00:00 2024-01-16T14:57:03.445 INFO:teuthology.orchestra.run.smithi043.stdout:(8/151): ceph-common-18.0.0-7629.g73a4532f.el9. 9.7 MB/s | 19 MB 00:01 2024-01-16T14:57:03.511 INFO:teuthology.orchestra.run.smithi043.stdout:(9/151): ceph-selinux-18.0.0-7629.g73a4532f.el9 383 kB/s | 25 kB 00:00 2024-01-16T14:57:03.642 INFO:teuthology.orchestra.run.smithi069.stdout:(64/151): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-01-16T14:57:03.667 INFO:teuthology.orchestra.run.smithi069.stdout:(65/151): python3-jsonpatch-1.21-16.el9.noarch. 90 kB/s | 24 kB 00:00 2024-01-16T14:57:03.678 INFO:teuthology.orchestra.run.smithi043.stdout:(10/151): ceph-radosgw-18.0.0-7629.g73a4532f.el 11 MB/s | 8.6 MB 00:00 2024-01-16T14:57:03.692 INFO:teuthology.orchestra.run.smithi069.stdout:(66/151): python3-jsonpointer-2.0-4.el9.noarch. 72 kB/s | 18 kB 00:00 2024-01-16T14:57:03.717 INFO:teuthology.orchestra.run.smithi069.stdout:(67/151): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-01-16T14:57:03.751 INFO:teuthology.orchestra.run.smithi069.stdout:(68/151): python3-markupsafe-1.1.1-12.el9.x86_6 560 kB/s | 32 kB 00:00 2024-01-16T14:57:03.754 INFO:teuthology.orchestra.run.smithi043.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 2.9 MB/s | 217 kB 00:00 2024-01-16T14:57:03.821 INFO:teuthology.orchestra.run.smithi043.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 473 kB/s | 31 kB 00:00 2024-01-16T14:57:03.930 INFO:teuthology.orchestra.run.smithi043.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 6.5 MB/s | 719 kB 00:00 2024-01-16T14:57:04.005 INFO:teuthology.orchestra.run.smithi043.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 2.2 MB/s | 163 kB 00:00 2024-01-16T14:57:04.018 INFO:teuthology.orchestra.run.smithi069.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-01-16T14:57:04.043 INFO:teuthology.orchestra.run.smithi069.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 669 kB/s | 217 kB 00:00 2024-01-16T14:57:04.068 INFO:teuthology.orchestra.run.smithi069.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 571 kB/s | 174 kB 00:00 2024-01-16T14:57:04.080 INFO:teuthology.orchestra.run.smithi043.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 1.7 MB/s | 127 kB 00:00 2024-01-16T14:57:04.172 INFO:teuthology.orchestra.run.smithi043.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 5.1 MB/s | 476 kB 00:00 2024-01-16T14:57:04.188 INFO:teuthology.orchestra.run.smithi069.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 347 kB/s | 41 kB 00:00 2024-01-16T14:57:04.252 INFO:teuthology.orchestra.run.smithi069.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 726 kB/s | 132 kB 00:00 2024-01-16T14:57:04.277 INFO:teuthology.orchestra.run.smithi069.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 1.0 MB/s | 211 kB 00:00 2024-01-16T14:57:04.303 INFO:teuthology.orchestra.run.smithi069.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 427 kB/s | 47 kB 00:00 2024-01-16T14:57:04.336 INFO:teuthology.orchestra.run.smithi069.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 749 kB/s | 43 kB 00:00 2024-01-16T14:57:04.554 INFO:teuthology.orchestra.run.smithi069.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 158 kB/s | 39 kB 00:00 2024-01-16T14:57:04.587 INFO:teuthology.orchestra.run.smithi069.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-01-16T14:57:04.681 INFO:teuthology.orchestra.run.smithi043.stdout:(17/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 9.4 MB/s | 4.8 MB 00:00 2024-01-16T14:57:04.756 INFO:teuthology.orchestra.run.smithi043.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 609 kB/s | 45 kB 00:00 2024-01-16T14:57:04.840 INFO:teuthology.orchestra.run.smithi043.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 1.5 MB/s | 129 kB 00:00 2024-01-16T14:57:04.879 INFO:teuthology.orchestra.run.smithi069.stdout:(79/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-01-16T14:57:04.896 INFO:teuthology.orchestra.run.smithi069.stdout:(80/151): uuid-1.6.2-55.el9.x86_64.rpm 166 kB/s | 56 kB 00:00 2024-01-16T14:57:04.915 INFO:teuthology.orchestra.run.smithi043.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 2.1 MB/s | 161 kB 00:00 2024-01-16T14:57:04.921 INFO:teuthology.orchestra.run.smithi069.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 272 kB/s | 87 kB 00:00 2024-01-16T14:57:04.938 INFO:teuthology.orchestra.run.smithi069.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-01-16T14:57:04.972 INFO:teuthology.orchestra.run.smithi069.stdout:(83/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 907 kB/s | 44 kB 00:00 2024-01-16T14:57:04.997 INFO:teuthology.orchestra.run.smithi069.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-01-16T14:57:05.007 INFO:teuthology.orchestra.run.smithi043.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 3.5 MB/s | 321 kB 00:00 2024-01-16T14:57:05.030 INFO:teuthology.orchestra.run.smithi069.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-01-16T14:57:05.064 INFO:teuthology.orchestra.run.smithi069.stdout:(86/151): libquadmath-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 194 kB 00:00 2024-01-16T14:57:05.107 INFO:teuthology.orchestra.run.smithi043.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 2.9 MB/s | 298 kB 00:00 2024-01-16T14:57:05.114 INFO:teuthology.orchestra.run.smithi069.stdout:(87/151): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 802 kB 00:00 2024-01-16T14:57:05.139 INFO:teuthology.orchestra.run.smithi069.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2024-01-16T14:57:05.173 INFO:teuthology.orchestra.run.smithi069.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-01-16T14:57:05.190 INFO:teuthology.orchestra.run.smithi043.stdout:(23/151): python3-rgw-18.0.0-7629.g73a4532f.el9 1.2 MB/s | 99 kB 00:00 2024-01-16T14:57:05.198 INFO:teuthology.orchestra.run.smithi069.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-01-16T14:57:05.266 INFO:teuthology.orchestra.run.smithi043.stdout:(24/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 1.1 MB/s | 85 kB 00:00 2024-01-16T14:57:05.282 INFO:teuthology.orchestra.run.smithi069.stdout:(91/151): python3-cryptography-36.0.1-4.el9.x86 8.2 MB/s | 1.1 MB 00:00 2024-01-16T14:57:05.307 INFO:teuthology.orchestra.run.smithi069.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 940 kB/s | 124 kB 00:00 2024-01-16T14:57:05.332 INFO:teuthology.orchestra.run.smithi069.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 917 kB/s | 113 kB 00:00 2024-01-16T14:57:05.366 INFO:teuthology.orchestra.run.smithi069.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-16T14:57:05.500 INFO:teuthology.orchestra.run.smithi043.stdout:(25/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 5.9 MB/s | 18 MB 00:02 2024-01-16T14:57:05.558 INFO:teuthology.orchestra.run.smithi043.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.9 MB/s | 171 kB 00:00 2024-01-16T14:57:05.609 INFO:teuthology.orchestra.run.smithi043.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 572 kB/s | 28 kB 00:00 2024-01-16T14:57:05.625 INFO:teuthology.orchestra.run.smithi069.stdout:(95/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.2 MB/s | 309 kB 00:00 2024-01-16T14:57:05.651 INFO:teuthology.orchestra.run.smithi069.stdout:(96/151): lua-devel-5.4.4-4.el9.x86_64.rpm 71 kB/s | 22 kB 00:00 2024-01-16T14:57:05.667 INFO:teuthology.orchestra.run.smithi043.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 2.3 MB/s | 133 kB 00:00 2024-01-16T14:57:05.743 INFO:teuthology.orchestra.run.smithi043.stdout:(29/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 6.3 MB/s | 3.0 MB 00:00 2024-01-16T14:57:05.775 INFO:teuthology.orchestra.run.smithi069.stdout:(97/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 209 kB/s | 26 kB 00:00 2024-01-16T14:57:05.917 INFO:teuthology.orchestra.run.smithi069.stdout:(98/151): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-01-16T14:57:05.943 INFO:teuthology.orchestra.run.smithi069.stdout:(99/151): liboath-2.6.7-2.el9.x86_64.rpm 292 kB/s | 49 kB 00:00 2024-01-16T14:57:05.976 INFO:teuthology.orchestra.run.smithi069.stdout:(100/151): python3-devel-3.9.17-2.el9.x86_64.rp 368 kB/s | 245 kB 00:00 2024-01-16T14:57:05.993 INFO:teuthology.orchestra.run.smithi043.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 5.4 MB/s | 1.8 MB 00:00 2024-01-16T14:57:06.001 INFO:teuthology.orchestra.run.smithi069.stdout:(101/151): libunwind-1.6.2-1.el9.x86_64.rpm 812 kB/s | 67 kB 00:00 2024-01-16T14:57:06.068 INFO:teuthology.orchestra.run.smithi069.stdout:(102/151): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-01-16T14:57:06.102 INFO:teuthology.orchestra.run.smithi069.stdout:(103/151): python3-asyncssh-2.13.2-3.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-01-16T14:57:06.124 INFO:teuthology.orchestra.run.smithi043.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 1.8 MB/s | 246 kB 00:00 2024-01-16T14:57:06.127 INFO:teuthology.orchestra.run.smithi069.stdout:(104/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 751 kB/s | 43 kB 00:00 2024-01-16T14:57:06.185 INFO:teuthology.orchestra.run.smithi069.stdout:(105/151): python3-cachetools-4.2.4-1.el9.noarc 394 kB/s | 32 kB 00:00 2024-01-16T14:57:06.211 INFO:teuthology.orchestra.run.smithi069.stdout:(106/151): python3-certifi-2021.10.8-2.el9.noar 181 kB/s | 15 kB 00:00 2024-01-16T14:57:06.245 INFO:teuthology.orchestra.run.smithi069.stdout:(107/151): python3-cheroot-8.6.0-4.el9.noarch.r 2.8 MB/s | 172 kB 00:00 2024-01-16T14:57:06.287 INFO:teuthology.orchestra.run.smithi069.stdout:(108/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-01-16T14:57:06.320 INFO:teuthology.orchestra.run.smithi069.stdout:(109/151): python3-google-auth-2.26.0-1.el9.noa 2.7 MB/s | 202 kB 00:00 2024-01-16T14:57:06.354 INFO:teuthology.orchestra.run.smithi069.stdout:(110/151): python3-jaraco-8.2.1-3.el9.noarch.rp 162 kB/s | 11 kB 00:00 2024-01-16T14:57:06.379 INFO:teuthology.orchestra.run.smithi069.stdout:(111/151): python3-jaraco-classes-3.2.1-5.el9.n 305 kB/s | 18 kB 00:00 2024-01-16T14:57:06.413 INFO:teuthology.orchestra.run.smithi069.stdout:(112/151): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-01-16T14:57:06.438 INFO:teuthology.orchestra.run.smithi069.stdout:(113/151): python3-jaraco-collections-3.0.0-8.e 279 kB/s | 23 kB 00:00 2024-01-16T14:57:06.463 INFO:teuthology.orchestra.run.smithi069.stdout:(114/151): python3-jaraco-functools-3.5.0-2.el9 234 kB/s | 19 kB 00:00 2024-01-16T14:57:06.488 INFO:teuthology.orchestra.run.smithi069.stdout:(115/151): python3-jaraco-text-3.2.0-6.el9.noar 263 kB/s | 20 kB 00:00 2024-01-16T14:57:06.522 INFO:teuthology.orchestra.run.smithi069.stdout:(116/151): python3-jwt+crypto-2.4.0-1.el9.noarc 157 kB/s | 9.0 kB 00:00 2024-01-16T14:57:06.539 INFO:teuthology.orchestra.run.smithi069.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 551 kB/s | 41 kB 00:00 2024-01-16T14:57:06.577 INFO:teuthology.orchestra.run.smithi043.stdout:(32/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 13 MB/s | 41 MB 00:03 2024-01-16T14:57:06.581 INFO:teuthology.orchestra.run.smithi069.stdout:(118/151): python3-logutils-0.3.5-21.el9.noarch 804 kB/s | 46 kB 00:00 2024-01-16T14:57:06.602 INFO:teuthology.orchestra.run.smithi043.stdout:(33/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 105 kB/s | 50 kB 00:00 2024-01-16T14:57:06.606 INFO:teuthology.orchestra.run.smithi069.stdout:(119/151): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-01-16T14:57:06.627 INFO:teuthology.orchestra.run.smithi043.stdout:(34/151): ceph-prometheus-alerts-18.0.0-7629.g7 307 kB/s | 15 kB 00:00 2024-01-16T14:57:06.648 INFO:teuthology.orchestra.run.smithi069.stdout:(120/151): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-01-16T14:57:06.665 INFO:teuthology.orchestra.run.smithi069.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 696 kB/s | 58 kB 00:00 2024-01-16T14:57:06.680 INFO:teuthology.orchestra.run.smithi043.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 3.4 MB/s | 262 kB 00:00 2024-01-16T14:57:06.699 INFO:teuthology.orchestra.run.smithi069.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-01-16T14:57:06.716 INFO:teuthology.orchestra.run.smithi069.stdout:(123/151): python3-portend-3.1.0-2.el9.noarch.r 248 kB/s | 16 kB 00:00 2024-01-16T14:57:06.730 INFO:teuthology.orchestra.run.smithi043.stdout:(36/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 371 kB/s | 18 kB 00:00 2024-01-16T14:57:06.741 INFO:teuthology.orchestra.run.smithi069.stdout:(124/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-01-16T14:57:06.766 INFO:teuthology.orchestra.run.smithi069.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 464 kB/s | 31 kB 00:00 2024-01-16T14:57:06.780 INFO:teuthology.orchestra.run.smithi043.stdout:(37/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 5.0 MB/s | 778 kB 00:00 2024-01-16T14:57:06.800 INFO:teuthology.orchestra.run.smithi069.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-01-16T14:57:06.825 INFO:teuthology.orchestra.run.smithi069.stdout:(127/151): python3-tempora-5.0.0-2.el9.noarch.r 618 kB/s | 36 kB 00:00 2024-01-16T14:57:06.850 INFO:teuthology.orchestra.run.smithi069.stdout:(128/151): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-01-16T14:57:06.876 INFO:teuthology.orchestra.run.smithi069.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 684 kB/s | 51 kB 00:00 2024-01-16T14:57:06.910 INFO:teuthology.orchestra.run.smithi069.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-01-16T14:57:06.943 INFO:teuthology.orchestra.run.smithi069.stdout:(131/151): python3-websocket-client-1.2.3-2.el9 981 kB/s | 90 kB 00:00 2024-01-16T14:57:06.968 INFO:teuthology.orchestra.run.smithi069.stdout:(132/151): python3-zc-lockfile-2.0-10.el9.noarc 349 kB/s | 20 kB 00:00 2024-01-16T14:57:06.972 INFO:teuthology.orchestra.run.smithi043.stdout:(38/151): ceph-mgr-diskprediction-local-18.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-01-16T14:57:07.001 INFO:teuthology.orchestra.run.smithi069.stdout:(133/151): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-01-16T14:57:07.035 INFO:teuthology.orchestra.run.smithi069.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-01-16T14:57:07.102 INFO:teuthology.orchestra.run.smithi069.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-01-16T14:57:07.161 INFO:teuthology.orchestra.run.smithi069.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 68 kB 00:00 2024-01-16T14:57:07.164 INFO:teuthology.orchestra.run.smithi043.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-01-16T14:57:07.198 INFO:teuthology.orchestra.run.smithi043.stdout:(40/151): automake-1.16.2-8.el9.noarch.rpm 1.4 MB/s | 662 kB 00:00 2024-01-16T14:57:07.223 INFO:teuthology.orchestra.run.smithi043.stdout:(41/151): boost-program-options-1.75.0-8.el9.x8 586 kB/s | 106 kB 00:00 2024-01-16T14:57:07.248 INFO:teuthology.orchestra.run.smithi043.stdout:(42/151): elfutils-libelf-devel-0.190-2.el9.x86 252 kB/s | 20 kB 00:00 2024-01-16T14:57:07.270 INFO:teuthology.orchestra.run.smithi069.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 21 MB/s | 2.2 MB 00:00 2024-01-16T14:57:07.282 INFO:teuthology.orchestra.run.smithi043.stdout:(43/151): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 307 kB 00:00 2024-01-16T14:57:07.307 INFO:teuthology.orchestra.run.smithi043.stdout:(44/151): flexiblas-3.0.4-8.el9.x86_64.rpm 577 kB/s | 32 kB 00:00 2024-01-16T14:57:07.379 INFO:teuthology.orchestra.run.smithi069.stdout:(138/151): librados2-18.0.0-7629.g73a4532f.el9. 8.8 MB/s | 3.3 MB 00:00 2024-01-16T14:57:07.437 INFO:teuthology.orchestra.run.smithi069.stdout:(139/151): NetworkManager-libnm-1.45.9-1.el9.x8 11 MB/s | 1.8 MB 00:00 2024-01-16T14:57:07.457 INFO:teuthology.orchestra.run.smithi043.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 99 kB/s | 17 kB 00:00 2024-01-16T14:57:07.462 INFO:teuthology.orchestra.run.smithi069.stdout:(140/151): NetworkManager-team-1.45.9-1.el9.x86 546 kB/s | 42 kB 00:00 2024-01-16T14:57:07.483 INFO:teuthology.orchestra.run.smithi043.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 58 kB/s | 10 kB 00:00 2024-01-16T14:57:07.496 INFO:teuthology.orchestra.run.smithi069.stdout:(141/151): NetworkManager-tui-1.45.9-1.el9.x86_ 4.2 MB/s | 246 kB 00:00 2024-01-16T14:57:07.521 INFO:teuthology.orchestra.run.smithi069.stdout:(142/151): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-01-16T14:57:07.546 INFO:teuthology.orchestra.run.smithi069.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 721 kB/s | 35 kB 00:00 2024-01-16T14:57:07.572 INFO:teuthology.orchestra.run.smithi069.stdout:(144/151): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 191 kB 00:00 2024-01-16T14:57:07.605 INFO:teuthology.orchestra.run.smithi069.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 4.4 MB/s | 254 kB 00:00 2024-01-16T14:57:07.631 INFO:teuthology.orchestra.run.smithi069.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 395 kB/s | 22 kB 00:00 2024-01-16T14:57:07.666 INFO:teuthology.orchestra.run.smithi043.stdout:(47/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.2 MB/s | 3.0 MB 00:00 2024-01-16T14:57:07.683 INFO:teuthology.orchestra.run.smithi043.stdout:(48/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 813 kB/s | 159 kB 00:00 2024-01-16T14:57:07.689 INFO:teuthology.orchestra.run.smithi069.stdout:(147/151): openssl-3.0.7-25.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-01-16T14:57:07.708 INFO:teuthology.orchestra.run.smithi043.stdout:(49/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 200 kB/s | 44 kB 00:00 2024-01-16T14:57:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:(50/151): librdkafka-1.6.1-102.el9.x86_64.rpm 7.9 MB/s | 661 kB 00:00 2024-01-16T14:57:07.781 INFO:teuthology.orchestra.run.smithi069.stdout:(148/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 16 MB/s | 2.1 MB 00:00 2024-01-16T14:57:07.792 INFO:teuthology.orchestra.run.smithi043.stdout:(51/151): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-01-16T14:57:07.815 INFO:teuthology.orchestra.run.smithi069.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 2.8 MB/s | 321 kB 00:00 2024-01-16T14:57:07.818 INFO:teuthology.orchestra.run.smithi043.stdout:(52/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 241 kB/s | 24 kB 00:00 2024-01-16T14:57:07.843 INFO:teuthology.orchestra.run.smithi043.stdout:(53/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 541 kB/s | 46 kB 00:00 2024-01-16T14:57:07.868 INFO:teuthology.orchestra.run.smithi043.stdout:(54/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 282 kB 00:00 2024-01-16T14:57:07.882 INFO:teuthology.orchestra.run.smithi069.stdout:(150/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-01-16T14:57:07.893 INFO:teuthology.orchestra.run.smithi043.stdout:(55/151): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-01-16T14:57:07.918 INFO:teuthology.orchestra.run.smithi043.stdout:(56/151): openblas-0.3.21-2.el9.x86_64.rpm 479 kB/s | 34 kB 00:00 2024-01-16T14:57:07.974 INFO:teuthology.orchestra.run.smithi069.stdout:(151/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 3.2 MB/s | 3.0 MB 00:00 2024-01-16T14:57:07.983 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:57:07.984 INFO:teuthology.orchestra.run.smithi069.stdout:Total 15 MB/s | 204 MB 00:13 2024-01-16T14:57:08.072 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-16T14:57:08.185 INFO:teuthology.orchestra.run.smithi043.stdout:(57/151): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-01-16T14:57:08.252 INFO:teuthology.orchestra.run.smithi043.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 4.7 MB/s | 1.5 MB 00:00 2024-01-16T14:57:08.361 INFO:teuthology.orchestra.run.smithi043.stdout:(59/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 6.9 MB/s | 3.0 MB 00:00 2024-01-16T14:57:08.394 INFO:teuthology.orchestra.run.smithi043.stdout:(60/151): postgresql-contrib-13.11-1.el9.x86_64 3.9 MB/s | 811 kB 00:00 2024-01-16T14:57:08.415 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-16T14:57:08.415 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-16T14:57:08.419 INFO:teuthology.orchestra.run.smithi043.stdout:(61/151): postgresql-private-libs-13.11-1.el9.x 878 kB/s | 134 kB 00:00 2024-01-16T14:57:08.670 INFO:teuthology.orchestra.run.smithi043.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-01-16T14:57:08.737 INFO:teuthology.orchestra.run.smithi043.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 737 kB/s | 228 kB 00:00 2024-01-16T14:57:08.846 INFO:teuthology.orchestra.run.smithi043.stdout:(64/151): postgresql-server-13.11-1.el9.x86_64. 13 MB/s | 5.7 MB 00:00 2024-01-16T14:57:08.871 INFO:teuthology.orchestra.run.smithi043.stdout:(65/151): python3-jsonpatch-1.21-16.el9.noarch. 132 kB/s | 24 kB 00:00 2024-01-16T14:57:08.888 INFO:teuthology.orchestra.run.smithi043.stdout:(66/151): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-01-16T14:57:08.913 INFO:teuthology.orchestra.run.smithi043.stdout:(67/151): python3-mako-1.1.4-6.el9.noarch.rpm 3.5 MB/s | 144 kB 00:00 2024-01-16T14:57:08.930 INFO:teuthology.orchestra.run.smithi043.stdout:(68/151): python3-markupsafe-1.1.1-12.el9.x86_6 551 kB/s | 32 kB 00:00 2024-01-16T14:57:09.097 INFO:teuthology.orchestra.run.smithi043.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-01-16T14:57:09.131 INFO:teuthology.orchestra.run.smithi043.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-01-16T14:57:09.173 INFO:teuthology.orchestra.run.smithi043.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 768 kB/s | 174 kB 00:00 2024-01-16T14:57:09.198 INFO:teuthology.orchestra.run.smithi043.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 428 kB/s | 41 kB 00:00 2024-01-16T14:57:09.223 INFO:teuthology.orchestra.run.smithi043.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 132 kB 00:00 2024-01-16T14:57:09.248 INFO:teuthology.orchestra.run.smithi043.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-01-16T14:57:09.273 INFO:teuthology.orchestra.run.smithi043.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 662 kB/s | 47 kB 00:00 2024-01-16T14:57:09.307 INFO:teuthology.orchestra.run.smithi043.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 751 kB/s | 43 kB 00:00 2024-01-16T14:57:09.541 INFO:teuthology.orchestra.run.smithi043.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 150 kB/s | 39 kB 00:00 2024-01-16T14:57:09.816 INFO:teuthology.orchestra.run.smithi043.stdout:(78/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-01-16T14:57:09.850 INFO:teuthology.orchestra.run.smithi043.stdout:(79/151): socat-1.7.4.1-5.el9.x86_64.rpm 569 kB/s | 300 kB 00:00 2024-01-16T14:57:09.875 INFO:teuthology.orchestra.run.smithi043.stdout:(80/151): uuid-1.6.2-55.el9.x86_64.rpm 177 kB/s | 56 kB 00:00 2024-01-16T14:57:09.900 INFO:teuthology.orchestra.run.smithi043.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-01-16T14:57:09.925 INFO:teuthology.orchestra.run.smithi043.stdout:(82/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-01-16T14:57:09.951 INFO:teuthology.orchestra.run.smithi043.stdout:(83/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 601 kB/s | 44 kB 00:00 2024-01-16T14:57:09.967 INFO:teuthology.orchestra.run.smithi043.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-01-16T14:57:10.009 INFO:teuthology.orchestra.run.smithi043.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-01-16T14:57:10.051 INFO:teuthology.orchestra.run.smithi043.stdout:(86/151): libgfortran-11.4.1-3.el9.x86_64.rpm 8.0 MB/s | 802 kB 00:00 2024-01-16T14:57:10.076 INFO:teuthology.orchestra.run.smithi043.stdout:(87/151): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 194 kB 00:00 2024-01-16T14:57:10.102 INFO:teuthology.orchestra.run.smithi043.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-01-16T14:57:10.135 INFO:teuthology.orchestra.run.smithi043.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2024-01-16T14:57:10.209 INFO:teuthology.orchestra.run.smithi043.stdout:(90/151): python3-cryptography-36.0.1-4.el9.x86 8.9 MB/s | 1.1 MB 00:00 2024-01-16T14:57:10.236 INFO:teuthology.orchestra.run.smithi043.stdout:(91/151): python3-ply-3.11-14.el9.noarch.rpm 817 kB/s | 103 kB 00:00 2024-01-16T14:57:10.261 INFO:teuthology.orchestra.run.smithi043.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.0 MB/s | 124 kB 00:00 2024-01-16T14:57:10.295 INFO:teuthology.orchestra.run.smithi043.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 2.0 MB/s | 113 kB 00:00 2024-01-16T14:57:10.320 INFO:teuthology.orchestra.run.smithi043.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-01-16T14:57:10.403 INFO:teuthology.orchestra.run.smithi043.stdout:(95/151): lua-devel-5.4.4-4.el9.x86_64.rpm 272 kB/s | 22 kB 00:00 2024-01-16T14:57:10.463 INFO:teuthology.orchestra.run.smithi043.stdout:(96/151): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-01-16T14:57:10.562 INFO:teuthology.orchestra.run.smithi043.stdout:(97/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.3 MB/s | 309 kB 00:00 2024-01-16T14:57:10.578 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-16T14:57:10.579 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-16T14:57:10.588 INFO:teuthology.orchestra.run.smithi043.stdout:(98/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-01-16T14:57:10.638 INFO:teuthology.orchestra.run.smithi043.stdout:(99/151): liboath-2.6.7-2.el9.x86_64.rpm 650 kB/s | 49 kB 00:00 2024-01-16T14:57:10.697 INFO:teuthology.orchestra.run.smithi043.stdout:(100/151): luarocks-3.9.2-1.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-01-16T14:57:10.722 INFO:teuthology.orchestra.run.smithi043.stdout:(101/151): libunwind-1.6.2-1.el9.x86_64.rpm 506 kB/s | 67 kB 00:00 2024-01-16T14:57:10.814 INFO:teuthology.orchestra.run.smithi043.stdout:(102/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 838 kB 00:00 2024-01-16T14:57:10.906 INFO:teuthology.orchestra.run.smithi043.stdout:(103/151): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-01-16T14:57:10.931 INFO:teuthology.orchestra.run.smithi043.stdout:(104/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 374 kB/s | 43 kB 00:00 2024-01-16T14:57:10.973 INFO:teuthology.orchestra.run.smithi043.stdout:(105/151): python3-asyncssh-2.13.2-3.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-01-16T14:57:10.999 INFO:teuthology.orchestra.run.smithi043.stdout:(106/151): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-01-16T14:57:11.024 INFO:teuthology.orchestra.run.smithi043.stdout:(107/151): python3-cachetools-4.2.4-1.el9.noarc 276 kB/s | 32 kB 00:00 2024-01-16T14:57:11.057 INFO:teuthology.orchestra.run.smithi043.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-01-16T14:57:11.091 INFO:teuthology.orchestra.run.smithi043.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-01-16T14:57:11.108 INFO:teuthology.orchestra.run.smithi043.stdout:(110/151): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-01-16T14:57:11.141 INFO:teuthology.orchestra.run.smithi043.stdout:(111/151): python3-google-auth-2.26.0-1.el9.noa 1.7 MB/s | 202 kB 00:00 2024-01-16T14:57:11.159 INFO:teuthology.orchestra.run.smithi043.stdout:(112/151): python3-jaraco-classes-3.2.1-5.el9.n 266 kB/s | 18 kB 00:00 2024-01-16T14:57:11.175 INFO:teuthology.orchestra.run.smithi043.stdout:(113/151): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-01-16T14:57:11.192 INFO:teuthology.orchestra.run.smithi043.stdout:(114/151): python3-jaraco-functools-3.5.0-2.el9 392 kB/s | 19 kB 00:00 2024-01-16T14:57:11.234 INFO:teuthology.orchestra.run.smithi043.stdout:(115/151): python3-jwt+crypto-2.4.0-1.el9.noarc 158 kB/s | 9.0 kB 00:00 2024-01-16T14:57:11.260 INFO:teuthology.orchestra.run.smithi043.stdout:(116/151): python3-jaraco-text-3.2.0-6.el9.noar 237 kB/s | 20 kB 00:00 2024-01-16T14:57:11.285 INFO:teuthology.orchestra.run.smithi043.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 446 kB/s | 41 kB 00:00 2024-01-16T14:57:11.352 INFO:teuthology.orchestra.run.smithi043.stdout:(118/151): python3-logutils-0.3.5-21.el9.noarch 507 kB/s | 46 kB 00:00 2024-01-16T14:57:11.394 INFO:teuthology.orchestra.run.smithi043.stdout:(119/151): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-01-16T14:57:11.419 INFO:teuthology.orchestra.run.smithi043.stdout:(120/151): python3-more-itertools-8.12.0-2.el9. 592 kB/s | 79 kB 00:00 2024-01-16T14:57:11.444 INFO:teuthology.orchestra.run.smithi043.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 631 kB/s | 58 kB 00:00 2024-01-16T14:57:11.470 INFO:teuthology.orchestra.run.smithi043.stdout:(122/151): python3-portend-3.1.0-2.el9.noarch.r 332 kB/s | 16 kB 00:00 2024-01-16T14:57:11.495 INFO:teuthology.orchestra.run.smithi043.stdout:(123/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-01-16T14:57:11.520 INFO:teuthology.orchestra.run.smithi043.stdout:(124/151): python3-repoze-lru-0.7-16.el9.noarch 623 kB/s | 31 kB 00:00 2024-01-16T14:57:11.554 INFO:teuthology.orchestra.run.smithi043.stdout:(125/151): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-01-16T14:57:11.579 INFO:teuthology.orchestra.run.smithi043.stdout:(126/151): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-01-16T14:57:11.604 INFO:teuthology.orchestra.run.smithi043.stdout:(127/151): python3-tempora-5.0.0-2.el9.noarch.r 724 kB/s | 36 kB 00:00 2024-01-16T14:57:11.638 INFO:teuthology.orchestra.run.smithi043.stdout:(128/151): python3-typing-extensions-4.4.0-2.el 875 kB/s | 51 kB 00:00 2024-01-16T14:57:11.663 INFO:teuthology.orchestra.run.smithi043.stdout:(129/151): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-01-16T14:57:11.730 INFO:teuthology.orchestra.run.smithi043.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-01-16T14:57:11.756 INFO:teuthology.orchestra.run.smithi043.stdout:(131/151): python3-websocket-client-1.2.3-2.el9 771 kB/s | 90 kB 00:00 2024-01-16T14:57:11.781 INFO:teuthology.orchestra.run.smithi043.stdout:(132/151): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2024-01-16T14:57:11.815 INFO:teuthology.orchestra.run.smithi043.stdout:(133/151): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-01-16T14:57:11.848 INFO:teuthology.orchestra.run.smithi043.stdout:(134/151): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2024-01-16T14:57:11.923 INFO:teuthology.orchestra.run.smithi043.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-01-16T14:57:11.965 INFO:teuthology.orchestra.run.smithi043.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-16T14:57:12.074 INFO:teuthology.orchestra.run.smithi043.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 21 MB/s | 2.2 MB 00:00 2024-01-16T14:57:12.158 INFO:teuthology.orchestra.run.smithi043.stdout:(138/151): NetworkManager-libnm-1.45.9-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-01-16T14:57:12.200 INFO:teuthology.orchestra.run.smithi043.stdout:(139/151): NetworkManager-team-1.45.9-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-01-16T14:57:12.291 INFO:teuthology.orchestra.run.smithi043.stdout:(140/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 6.8 MB/s | 3.0 MB 00:00 2024-01-16T14:57:12.325 INFO:teuthology.orchestra.run.smithi043.stdout:(141/151): NetworkManager-tui-1.45.9-1.el9.x86_ 1.9 MB/s | 246 kB 00:00 2024-01-16T14:57:12.350 INFO:teuthology.orchestra.run.smithi043.stdout:(142/151): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-01-16T14:57:12.376 INFO:teuthology.orchestra.run.smithi043.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 35 kB 00:00 2024-01-16T14:57:12.409 INFO:teuthology.orchestra.run.smithi043.stdout:(144/151): elfutils-libelf-0.190-2.el9.x86_64.r 3.3 MB/s | 191 kB 00:00 2024-01-16T14:57:12.443 INFO:teuthology.orchestra.run.smithi043.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-16T14:57:12.468 INFO:teuthology.orchestra.run.smithi043.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 396 kB/s | 22 kB 00:00 2024-01-16T14:57:12.518 INFO:teuthology.orchestra.run.smithi043.stdout:(147/151): openssl-3.0.7-25.el9.x86_64.rpm 16 MB/s | 1.2 MB 00:00 2024-01-16T14:57:12.610 INFO:teuthology.orchestra.run.smithi043.stdout:(148/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 17 MB/s | 2.1 MB 00:00 2024-01-16T14:57:12.636 INFO:teuthology.orchestra.run.smithi043.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 2.8 MB/s | 321 kB 00:00 2024-01-16T14:57:12.711 INFO:teuthology.orchestra.run.smithi043.stdout:(150/151): librados2-18.0.0-7629.g73a4532f.el9. 3.7 MB/s | 3.3 MB 00:00 2024-01-16T14:57:12.761 INFO:teuthology.orchestra.run.smithi043.stdout:(151/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.5 MB/s | 1.1 MB 00:00 2024-01-16T14:57:12.769 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:57:12.769 INFO:teuthology.orchestra.run.smithi043.stdout:Total 17 MB/s | 204 MB 00:12 2024-01-16T14:57:12.859 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-01-16T14:57:13.216 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-01-16T14:57:13.216 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-01-16T14:57:15.460 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-01-16T14:57:15.463 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-01-16T14:57:16.781 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-16T14:57:16.922 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:57:16.955 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:57:17.022 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:57:17.099 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:57:17.131 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:57:17.227 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:57:17.652 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:57:17.750 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:17.911 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:17.972 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:18.109 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:18.154 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:57:18.308 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:57:18.403 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:57:18.709 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:57:18.764 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:57:18.789 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:19.073 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:19.397 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:19.633 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:19.844 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:19.898 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:20.326 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:20.370 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:57:20.481 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:57:20.543 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:57:20.585 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:57:20.686 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:57:20.739 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:57:20.848 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:57:20.891 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:57:20.938 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:57:20.970 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:57:21.081 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:57:21.115 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:57:21.165 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:57:21.188 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:57:21.286 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:57:21.382 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:57:21.548 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:57:21.627 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:57:21.699 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-01-16T14:57:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:57:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:57:21.969 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:57:22.022 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:57:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:57:22.151 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:57:22.315 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:57:22.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:57:22.452 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:57:22.525 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:57:22.570 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:57:22.573 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:57:22.635 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:57:22.667 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:22.817 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:22.833 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:57:22.843 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:22.875 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:57:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:23.012 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:57:23.017 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:57:23.045 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:57:23.178 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:57:23.292 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:57:23.299 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:57:23.400 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:57:23.459 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:57:23.506 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:57:23.577 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:57:23.590 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:57:23.637 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:57:23.650 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:57:23.675 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:23.696 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:57:23.805 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:23.835 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:23.884 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:57:23.927 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:23.937 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:57:23.982 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:57:24.108 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:57:24.146 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:57:24.213 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:57:24.252 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:57:24.263 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:57:24.408 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:57:24.440 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:24.456 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:57:24.524 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:57:24.551 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:24.596 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:57:24.757 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:24.810 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:24.845 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:57:24.911 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:57:25.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:57:25.156 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:57:25.189 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:57:25.225 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:25.264 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:57:25.265 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:57:25.310 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:57:25.360 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:57:25.370 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:57:25.415 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:57:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:57:25.462 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:57:25.477 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:57:25.513 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:57:25.530 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:57:25.576 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:57:25.606 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:57:25.661 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:57:25.714 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:57:25.726 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:57:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:57:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:57:25.852 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:57:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:57:25.888 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:57:25.922 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:57:25.964 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:57:25.978 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:57:26.000 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:57:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:57:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:57:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:57:26.125 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:57:26.174 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:57:26.222 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:57:26.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:57:26.330 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:57:26.379 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:57:26.510 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:57:26.572 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:57:27.284 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:57:27.330 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:57:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:57:27.387 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:57:27.421 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:57:27.425 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:57:27.454 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:57:27.478 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:57:27.517 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:57:27.578 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:57:27.692 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:57:27.725 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:57:27.803 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:57:27.816 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:57:27.846 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:57:27.998 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:57:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:57:28.211 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:57:28.324 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:57:28.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:57:28.400 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:57:28.469 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:57:28.533 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:57:28.595 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:57:28.606 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:57:28.735 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:57:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:57:28.857 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:57:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:57:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:57:29.148 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:57:29.201 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:57:29.241 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:57:29.408 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:57:29.457 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:57:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:57:29.618 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:57:29.912 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:57:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:57:30.235 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:57:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:57:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:57:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:57:30.679 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:57:30.745 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:57:30.790 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:57:30.843 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:57:30.848 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:57:30.943 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:57:30.945 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:57:31.007 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:57:31.085 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:57:31.136 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:57:31.281 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:57:31.321 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:57:31.344 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:57:31.364 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:57:31.377 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:31.431 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:57:31.485 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:57:31.499 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:57:31.602 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:57:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:57:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:57:31.858 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:57:31.922 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:57:32.122 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:57:32.154 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:32.230 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:32.257 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:57:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:57:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:57:33.039 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:57:33.078 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:57:33.343 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:57:33.374 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:57:33.490 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:57:34.357 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:57:34.492 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:57:34.780 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:34.820 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:35.462 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:35.507 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:57:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:57:35.690 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:57:35.690 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:57:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:57:36.127 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:57:36.215 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:57:36.251 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:57:36.286 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:57:36.657 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:57:36.711 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:36.835 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:36.876 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:57:36.935 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:57:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:57:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:37.608 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:57:40.089 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:40.129 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:40.795 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:57:40.832 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:57:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:57:41.043 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:57:41.043 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:57:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:41.084 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:57:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-01-16T14:57:48.431 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:48.688 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:57:48.735 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:57:48.735 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:48.735 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:57:48.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:50.484 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:57:50.558 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:57:50.558 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:50.558 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:57:50.558 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:57:50.558 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:57:50.559 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:57:50.852 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:50.853 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:57:50.853 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:57:50.853 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:57:50.853 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:57:51.413 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:57:51.413 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:51.413 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:57:51.413 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:57:51.413 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:57:51.414 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:51.519 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:57:52.127 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:57:52.162 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:57:52.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:57:52.297 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:57:52.324 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:57:52.623 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:57:52.659 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:57:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:57:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:57:53.899 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:57:54.067 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:57:54.094 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-01-16T14:57:54.197 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:54.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:57:54.257 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:57:54.350 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:57:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:57:54.468 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:57:54.468 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:54.468 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:57:54.468 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:55.215 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:57:55.247 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:57:55.313 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:55.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:57:55.434 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:57:55.434 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:55.434 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:57:55.434 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:55.515 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:57:55.590 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:57:55.590 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:55.590 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:57:55.590 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:57:55.591 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:57:55.591 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:55.949 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:57:56.019 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:57:56.019 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:56.019 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:57:56.019 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:57:56.019 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:57:56.020 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:57:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:57:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:57:56.316 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:56.316 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:57:56.317 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:57:56.317 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:57:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:56.548 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:57:56.615 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:57:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:57:57.236 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:57:57.236 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:57:57.236 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:57:57.236 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:57:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:57:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:57:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:57:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:57:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:57:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:57:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:57:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:57:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:57:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:57:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:00.022 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:00.147 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:58:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:58:00.202 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:58:00.233 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:58:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:00.307 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:00.308 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:58:00.308 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:00.308 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:00.308 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:58:00.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:00.498 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:58:00.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:58:00.728 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:58:00.766 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:00.799 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:58:00.839 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:58:00.885 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:58:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:58:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:58:01.443 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:58:01.478 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:01.550 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:01.693 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:58:01.694 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:01.795 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:01.873 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:01.873 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:01.873 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:58:01.873 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:01.874 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:02.100 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:02.158 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:02.236 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:02.280 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:02.302 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:02.355 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:03.634 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:58:03.713 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:58:03.758 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:58:03.883 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:58:03.924 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:58:03.949 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:58:03.983 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:04.040 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:58:04.040 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:04.166 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:04.216 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:58:04.264 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:58:04.305 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:58:04.337 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:04.338 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:04.467 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:04.506 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:58:04.544 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:04.544 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:04.880 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:04.881 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:05.015 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:05.059 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:58:05.097 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:58:05.152 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:58:05.271 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi069.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:58:05.519 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:06.768 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:58:06.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:58:06.879 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:58:06.935 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:58:07.225 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:58:07.356 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:58:07.397 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:07.430 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:58:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:58:07.515 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:58:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:07.718 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:58:07.757 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:58:08.679 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:08.721 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:08.810 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:58:10.077 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:58:10.078 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:58:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:58:10.080 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:58:10.081 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:58:10.082 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:58:10.083 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:58:10.084 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:58:10.085 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:58:10.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:58:10.087 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:58:10.088 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:58:10.089 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:58:10.090 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:58:10.091 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:58:10.092 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:58:10.093 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:58:10.094 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:58:10.094 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:58:10.094 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:58:10.094 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:58:10.311 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:58:10.377 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:58:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:58:10.555 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:58:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:58:10.621 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:58:10.655 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:10.696 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:58:10.696 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:10.814 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:10.865 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:58:10.919 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:58:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:58:11.018 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:11.018 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:11.139 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:11.169 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:58:11.208 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:11.208 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:11.510 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:11.511 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:11.645 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:11.680 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:58:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:58:11.807 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:58:11.932 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:12.231 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:12.232 INFO:teuthology.orchestra.run.smithi043.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:58:12.232 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:58:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:58:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:58:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:58:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:58:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:58:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:58:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:58:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:58:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:58:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:58:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:58:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:58:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:12.310 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-16T14:58:12.952 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:58:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:58:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:58:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:58:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:58:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:58:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:58:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:58:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:58:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:58:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:58:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:58:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:58:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:58:18.997 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:58:18.997 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:18.997 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-01-16T14:58:18.997 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:18.997 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:18.998 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:18.999 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi043.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.005 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:58:19.006 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:58:19.007 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:58:19.008 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:58:19.009 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:58:19.010 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:58:19.011 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:58:19.012 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:19.013 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-01-16T14:58:19.583 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:19.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:58:19.729 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:58:19.788 INFO:teuthology.orchestra.run.smithi043.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:19.789 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:19.789 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:58:19.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:58:19.933 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:58:20.001 INFO:teuthology.orchestra.run.smithi069.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:20.002 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:20.002 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:58:20.004 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T14:58:20.004 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:20.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:58:20.083 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:20.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:58:20.183 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T14:58:20.183 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:20.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:58:20.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:58:20.373 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:20.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:58:20.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:58:20.592 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T14:58:20.592 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:20.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:58:20.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:58:20.781 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:20.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:58:20.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:58:20.986 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T14:58:20.987 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:20.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:58:21.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:58:21.177 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:21.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:58:21.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:58:21.381 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T14:58:22.206 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T14:58:22.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2024-01-16T14:58:22.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-01-16T14:58:22.307 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T14:58:22.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:58:22.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:58:22.404 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T14:58:22.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:58:22.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:58:22.605 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-16T14:58:22.605 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-16T14:58:22.605 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-16T14:58:22.606 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:22.606 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:58:22.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:22.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:22.932 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:22.932 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:58:22.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:23.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:23.346 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T14:58:23.346 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T14:58:23.346 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7518196', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3480, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpF1KD1h76HBjH0XwN4zVha7la215WjgX4udBytaTPcNiEBzn4tDScWUgsRiVwXEaZSoZ6/lXdIkZtebJUZyYk=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPobHIqFf3RsbMpRH+fQ4bw7EVvPC1Vk9jbTiyHsYLDA1NXgfyymiah/c/7lPSi04qSVHSSA3UT0YkLpU3o3WsQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-16_14:38:05', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T14:58:23.347 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:58:23.353 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:58:23.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:58:23.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:58:23.541 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:23.542 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:58:23.582 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:58:23.582 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-01-16T14:58:23.664 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:58:23.664 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:23.664 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-01-16 14:58:12.712653845 +0000 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-01-16 14:55:49.297102720 +0000 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-01-16 14:55:49.297102720 +0000 2024-01-16T14:58:23.665 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-01-16 14:55:49.297102720 +0000 2024-01-16T14:58:23.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:58:23.778 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-01-16T14:58:23.778 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-01-16T14:58:23.778 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000220299 s, 2.3 MB/s 2024-01-16T14:58:23.782 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:58:23.820 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-01-16T14:58:23.908 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:58:23.908 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:23.908 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-01-16T14:58:23.908 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:23.909 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:23.909 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-01-16 14:58:12.713653832 +0000 2024-01-16T14:58:23.909 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-01-16 14:55:49.885095445 +0000 2024-01-16T14:58:23.909 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-01-16 14:55:49.885095445 +0000 2024-01-16T14:58:23.909 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-01-16 14:55:49.885095445 +0000 2024-01-16T14:58:23.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:58:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-01-16T14:58:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-01-16T14:58:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000170057 s, 3.0 MB/s 2024-01-16T14:58:24.021 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:58:24.064 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-01-16T14:58:24.142 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:58:24.142 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:24.142 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-01-16T14:58:24.142 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:24.142 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:24.143 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-01-16 14:58:12.714653819 +0000 2024-01-16T14:58:24.143 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-01-16 14:55:50.470088208 +0000 2024-01-16T14:58:24.143 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-01-16 14:55:50.470088208 +0000 2024-01-16T14:58:24.143 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-01-16 14:55:50.470088208 +0000 2024-01-16T14:58:24.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:58:24.247 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-01-16T14:58:24.247 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-01-16T14:58:24.247 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.00025183 s, 2.0 MB/s 2024-01-16T14:58:24.249 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:58:24.288 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-01-16 14:58:12.714653819 +0000 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-01-16 14:55:51.052081007 +0000 2024-01-16T14:58:24.367 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-01-16 14:55:51.052081007 +0000 2024-01-16T14:58:24.368 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-01-16 14:55:51.052081007 +0000 2024-01-16T14:58:24.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:58:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-01-16T14:58:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-01-16T14:58:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000178186 s, 2.9 MB/s 2024-01-16T14:58:24.474 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:58:24.512 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:24.512 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:24.512 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:58:24.569 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:58:24.584 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-01-16T14:58:24.639 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:58:24.639 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:24.639 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-01-16T14:58:24.639 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:24.639 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:24.640 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-16 14:58:05.958454153 +0000 2024-01-16T14:58:24.640 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-16 14:55:39.114238788 +0000 2024-01-16T14:58:24.640 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-16 14:55:39.114238788 +0000 2024-01-16T14:58:24.640 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-16 14:55:39.114238788 +0000 2024-01-16T14:58:24.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:58:24.728 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-16T14:58:24.728 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-16T14:58:24.728 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000326864 s, 1.6 MB/s 2024-01-16T14:58:24.732 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:58:24.787 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-01-16T14:58:24.835 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-16 14:58:05.959454130 +0000 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-16 14:55:39.674225849 +0000 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-16 14:55:39.674225849 +0000 2024-01-16T14:58:24.836 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-16 14:55:39.674225849 +0000 2024-01-16T14:58:24.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:58:24.921 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-16T14:58:24.921 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-16T14:58:24.921 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000329945 s, 1.6 MB/s 2024-01-16T14:58:24.924 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:58:24.978 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-01-16T14:58:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-16 14:58:05.960454107 +0000 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-16 14:55:40.214213372 +0000 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-16 14:55:40.214213372 +0000 2024-01-16T14:58:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-16 14:55:40.214213372 +0000 2024-01-16T14:58:25.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:58:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-16T14:58:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-16T14:58:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000378506 s, 1.4 MB/s 2024-01-16T14:58:25.120 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:58:25.176 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-01-16T14:58:25.226 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:58:25.226 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-16 14:58:05.960454107 +0000 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-16 14:55:40.776200388 +0000 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-16 14:55:40.776200388 +0000 2024-01-16T14:58:25.227 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-16 14:55:40.776200388 +0000 2024-01-16T14:58:25.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:58:25.315 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-16T14:58:25.316 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-16T14:58:25.316 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000358787 s, 1.4 MB/s 2024-01-16T14:58:25.320 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:58:25.377 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:25.377 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:58:25.377 INFO:tasks.ceph:Generating config... 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] debug client = 20 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] debug ms = 1 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] pid file = 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-16T14:58:25.378 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T14:58:25.379 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T14:58:25.380 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T14:58:25.380 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T14:58:25.380 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T14:58:25.380 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T14:58:25.380 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-16T14:58:25.381 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-16T14:58:25.381 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T14:58:25.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T14:58:25.539 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T14:58:25.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T14:58:25.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:25.785 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.43'), ('mon.b', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]')] 2024-01-16T14:58:25.785 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.43,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-16T14:58:25.786 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:25.786 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:58:25.841 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.43 --add b 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:26.005 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:26.005 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: generated fsid eb89d060-5b4a-4a34-8ffb-6d8b15e6800a 2024-01-16T14:58:26.005 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = pacific 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 0 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:fsid eb89d060-5b4a-4a34-8ffb-6d8b15e6800a 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:last_changed 2024-01-16T14:58:26.024981+0000 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-01-16T14:58:26.024981+0000 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:min_mon_release 16 (pacific) 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:election_strategy: 1 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.b 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-16T14:58:26.006 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-16T14:58:26.009 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:58:26.061 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID eb89d060-5b4a-4a34-8ffb-6d8b15e6800a... 2024-01-16T14:58:26.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:58:26.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:58:26.239 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-01-16T14:58:26.239 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-01-16T14:58:26.239 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.240 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:58:26.241 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.43,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-01-16T14:58:26.242 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = eb89d060-5b4a-4a34-8ffb-6d8b15e6800a 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-01-16T14:58:26.243 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.244 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-01-16T14:58:26.245 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug finisher = 20 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:26.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-01-16T14:58:26.248 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-01-16T14:58:26.249 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-01-16T14:58:26.250 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.251 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.252 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:58:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = "172.21.15.43,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = eb89d060-5b4a-4a34-8ffb-6d8b15e6800a 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug trim objects = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: bdev async discard = True 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: bdev enable discard = True 2024-01-16T14:58:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore allocator = bitmap 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore block size = 96636764160 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore fsck on mount = True 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluefs = 1/20 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluestore = 1/20 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: debug rocksdb = 4/10 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd full ratio = 0.9 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: osd objectstore = bluestore 2024-01-16T14:58:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: osd op complaint time = 180 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug client = 20 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug finisher = 20 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-01-16T14:58:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:58:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: mon op complaint time = 120 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:58:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: client mount timeout = 600 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug client = 20 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = "" 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: rados mon op timeout = 900 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: rados osd op timeout = 900 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.b] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.c] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mds] 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug mds = 20 2024-01-16T14:58:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug mds balancer = 20 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: mds debug frag = True 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: mds debug scatterstat = True 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: mds op complaint time = 180 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: mds verify scatter = True 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: osd op complaint time = 180 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: rados mon op timeout = 900 2024-01-16T14:58:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: rados osd op timeout = 900 2024-01-16T14:58:26.290 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T14:58:26.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T14:58:26.441 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T14:58:26.442 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:26.442 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T14:58:26.494 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:26.495 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T14:58:26.544 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:58:26.544 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:26.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:58:26.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:26.685 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:58:26.685 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:26.785 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:58:26.785 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:26.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:58:26.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:26.934 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:58:26.934 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:27.030 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T14:58:27.031 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T14:58:27.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:58:27.233 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:58:27.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:58:27.439 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:58:27.450 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T14:58:27.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:58:27.644 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:58:27.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-16T14:58:27.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:58:27.932 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:58:27.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-16T14:58:28.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:58:28.227 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:58:28.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-16T14:58:28.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:58:28.507 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:58:28.516 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-16T14:58:28.608 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T14:58:28.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T14:58:28.747 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T14:58:28.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-16T14:58:28.948 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-16T14:58:29.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-16T14:58:29.132 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-16T14:58:29.185 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-16T14:58:29.305 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-16T14:58:29.366 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T14:58:29.366 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:58:29.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:29.474 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:29.474 INFO:tasks.ceph:role: osd.0 2024-01-16T14:58:29.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:58:29.474 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:29.574 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:29.575 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:29.575 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:29.575 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:29.608 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-01-16T14:58:29.614 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-01-16T14:58:29.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:29.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:29.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:29.902 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:29.902 INFO:tasks.ceph:role: osd.1 2024-01-16T14:58:29.902 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:58:29.902 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:29.996 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:29.997 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:29.997 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:29.997 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:29.997 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:30.031 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-01-16T14:58:30.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-01-16T14:58:30.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:30.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:30.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:30.341 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:30.341 INFO:tasks.ceph:role: osd.2 2024-01-16T14:58:30.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:58:30.341 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:30.438 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:30.472 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-01-16T14:58:30.477 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-01-16T14:58:30.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:30.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:30.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:30.789 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:30.789 INFO:tasks.ceph:role: osd.3 2024-01-16T14:58:30.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T14:58:30.789 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:30.885 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:30.886 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:30.918 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-01-16T14:58:30.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-01-16T14:58:30.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:31.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:31.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:31.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.322+0000 7f75e8610500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T14:58:31.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.323+0000 7f75e8610500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T14:58:31.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.323+0000 7f75e8610500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:31.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.323+0000 7f75e8610500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:31.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.323+0000 7f75e8610500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:31.326+0000 7f75e8610500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T14:58:34.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:34.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.760+0000 7fdf0e5a9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T14:58:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.761+0000 7fdf0e5a9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T14:58:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.761+0000 7fdf0e5a9500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:58:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.761+0000 7fdf0e5a9500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:58:34.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.761+0000 7fdf0e5a9500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:58:34.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:34.763+0000 7fdf0e5a9500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T14:58:37.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:38.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:38.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.162+0000 7fb94fc0c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T14:58:38.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.163+0000 7fb94fc0c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T14:58:38.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.163+0000 7fb94fc0c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:58:38.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.163+0000 7fb94fc0c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:58:38.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.163+0000 7fb94fc0c500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:58:38.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:38.165+0000 7fb94fc0c500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T14:58:41.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:41.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.580+0000 7fccf3e10500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-16T14:58:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.581+0000 7fccf3e10500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-16T14:58:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.581+0000 7fccf3e10500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:58:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.581+0000 7fccf3e10500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:58:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.581+0000 7fccf3e10500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:58:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:58:41.583+0000 7fccf3e10500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-16T14:58:44.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:44.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-16T14:58:44.944 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:44.945 INFO:tasks.ceph:role: osd.4 2024-01-16T14:58:44.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:58:44.945 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:58:45.045 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:45.045 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:45.046 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:45.551 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-16T14:58:45.557 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-16T14:58:45.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-16T14:58:45.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-16T14:58:45.856 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-16T14:58:45.937 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:45.938 INFO:tasks.ceph:role: osd.5 2024-01-16T14:58:45.938 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:58:45.938 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:58:46.024 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:46.025 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:46.507 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-16T14:58:46.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-16T14:58:46.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-16T14:58:46.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-16T14:58:46.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-16T14:58:46.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:46.807 INFO:tasks.ceph:role: osd.6 2024-01-16T14:58:46.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:58:46.808 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:58:46.887 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:46.887 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:46.887 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:46.887 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:46.888 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:47.394 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-16T14:58:47.401 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-16T14:58:47.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-16T14:58:47.591 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-16T14:58:47.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-16T14:58:47.781 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:47.781 INFO:tasks.ceph:role: osd.7 2024-01-16T14:58:47.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T14:58:47.781 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:58:47.882 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:47.883 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:48.327 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-16T14:58:48.332 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-16T14:58:48.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-16T14:58:48.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-16T14:58:48.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:48.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.729+0000 7f22ca412500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-16T14:58:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.732+0000 7f22ca412500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-16T14:58:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.732+0000 7f22ca412500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:58:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.732+0000 7f22ca412500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:58:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.732+0000 7f22ca412500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:58:48.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:48.738+0000 7f22ca412500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-16T14:58:51.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-16T14:58:51.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:52.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.148+0000 7f1638c1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-16T14:58:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.149+0000 7f1638c1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-16T14:58:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.149+0000 7f1638c1a500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:58:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.149+0000 7f1638c1a500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:58:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.149+0000 7f1638c1a500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:58:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:52.151+0000 7f1638c1a500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-16T14:58:53.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:53.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:55.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-16T14:58:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:55.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.555+0000 7f2e33612500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-16T14:58:55.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.556+0000 7f2e33612500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-16T14:58:55.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.556+0000 7f2e33612500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:58:55.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.556+0000 7f2e33612500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:58:55.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.556+0000 7f2e33612500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:58:55.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:55.558+0000 7f2e33612500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-16T14:58:58.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-16T14:58:58.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:58.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.966+0000 7fbb63b82500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-16T14:58:58.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.967+0000 7fbb63b82500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-16T14:58:58.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.968+0000 7fbb63b82500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:58:58.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.968+0000 7fbb63b82500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:58:58.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.968+0000 7fbb63b82500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:58:58.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:58:58.970+0000 7fbb63b82500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-16T14:59:01.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-16T14:59:01.963 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T14:59:01.964 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:01.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-16T14:59:02.066 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-16T14:59:02.165 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-16T14:59:02.264 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T14:59:02.376 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T14:59:02.479 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T14:59:02.580 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:02.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-16T14:59:02.681 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:02.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T14:59:02.777 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:02.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-16T14:59:02.871 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:02.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-16T14:59:02.966 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:02.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-16T14:59:03.061 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:03.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-16T14:59:03.198 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:03.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-16T14:59:03.290 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:03.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-16T14:59:03.384 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:03.384 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T14:59:03.443 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:03.443 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-16T14:59:03.499 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:03.499 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-16T14:59:03.555 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:03.556 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-16T14:59:03.607 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T14:59:03.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:59:03.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA== 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q== 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg== 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA== 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-01-16T14:59:03.706 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg== 2024-01-16T14:59:03.707 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA== 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA== 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:[client.2] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w== 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA== 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout:[client.3] 2024-01-16T14:59:03.708 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.y] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw== 2024-01-16T14:59:03.709 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.3] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.4] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.5] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.6] 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA== 2024-01-16T14:59:03.710 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.7] 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA== 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout:[client.2] 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w== 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout:[client.1] 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA== 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout:[client.3] 2024-01-16T14:59:03.711 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA== 2024-01-16T14:59:03.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:03.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:03.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:03.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:04.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:04.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:04.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.482 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.639 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:04.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:04.784 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:04.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:04.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:05.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:05.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:05.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.705 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:05.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:05.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:05.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:05.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:06.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:06.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:06.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:06.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:06.433 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T14:59:06.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T14:59:06.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:06.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T14:59:06.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-16T14:59:06.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:07.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-16T14:59:07.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-16T14:59:07.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:07.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-16T14:59:07.411 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:07.418 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:07.472 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T14:59:07.472 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T14:59:07.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T14:59:07.478 INFO:tasks.ceph.mon.a:Started 2024-01-16T14:59:07.479 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-16T14:59:07.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-16T14:59:07.486 INFO:tasks.ceph.mon.b:Started 2024-01-16T14:59:07.486 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-16T14:59:07.486 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-16T14:59:07.493 INFO:tasks.ceph.mon.c:Started 2024-01-16T14:59:07.493 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T14:59:07.493 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-16T14:59:07.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-16T14:59:07.501 INFO:tasks.ceph.mgr.y:Started 2024-01-16T14:59:07.501 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T14:59:07.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T14:59:07.506 INFO:tasks.ceph.mgr.x:Started 2024-01-16T14:59:07.507 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T14:59:07.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T14:59:07.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.717+0000 7f0dea000640 1 Processor -- start 2024-01-16T14:59:07.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.718+0000 7f0dea000640 1 -- start start 2024-01-16T14:59:07.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.719+0000 7f0dea000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0de40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.719+0000 7f0dea000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de40c6bf0 con 0x7f0de40d0fa0 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.719+0000 7f0dea000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de40d13d0 con 0x7f0de415d3b0 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.719+0000 7f0dea000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40cedc0 con 0x7f0de40d1d90 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.719+0000 7f0de8c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0de415d3b0 0x7f0de40d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:34628/0 (socket says 172.21.15.43:34628) 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de8c00640 1 -- 172.21.15.43:0/820647388 learned_addr learned my addr 172.21.15.43:0/820647388 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de3e00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0de40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de3e00640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 msgr2=0x7f0de40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-16T14:59:07.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de3e00640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 msgr2=0x7f0de40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:59:07.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de3e00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0de40ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:59:07.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.720+0000 7f0de3e00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0de40ce6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591576548 0 0) 0x7f0de40c6bf0 con 0x7f0de40d0fa0 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc4003520 con 0x7f0de40d0fa0 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456139737 0 0) 0x7f0dc4003520 con 0x7f0de40d0fa0 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 msgr2=0x7f0de40ce6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0de40ce6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 >> v1:172.21.15.69:6789/0 conn(0x7f0de415d3b0 legacy=0x7f0de40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de40c6e40 con 0x7f0de40d0fa0 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0ddc0019f0 con 0x7f0de40d0fa0 2024-01-16T14:59:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 >> v1:172.21.15.43:6789/0 conn(0x7f0de40d0fa0 legacy=0x7f0de40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0dc4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0dc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.724+0000 7f0de3400640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dc4004d70 0x7f0dc40051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40cedc0 con 0x7f0dc4003a10 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc400bc50 con 0x7f0dc4004d70 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc400be10 con 0x7f0de40d1d90 2024-01-16T14:59:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de9600640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dc4004d70 0x7f0dc40051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de8c00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0dc4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.725+0000 7f0de3e00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0dc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.726+0000 7f0de8c00640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 msgr2=0x7f0dc40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.726+0000 7f0de8c00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de40d1d90 0x7f0dc40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.726+0000 7f0de8c00640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dc4004d70 msgr2=0x7f0dc40051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.726+0000 7f0de8c00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dc4004d70 0x7f0dc40051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.726+0000 7f0de8c00640 1 -- 172.21.15.43:0/820647388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4003520 con 0x7f0dc4003a10 2024-01-16T14:59:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.727+0000 7f0de8c00640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0dc4003e30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001810 tx=0x7f0dc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e693ba46423f4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4004aa0 con 0x7f0dc4003a10 2024-01-16T14:59:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd402e070 con 0x7f0dc4003a10 2024-01-16T14:59:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0de3400640 1 -- 172.21.15.43:0/820647388 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4004e00 con 0x7f0dc4003a10 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0dea000640 1 -- 172.21.15.43:0/820647388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 msgr2=0x7f0dc4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0dea000640 1 --2- 172.21.15.43:0/820647388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0dc4003e30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001810 tx=0x7f0dc40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.728+0000 7f0de9600640 1 -- 172.21.15.43:0/820647388 reap_dead start 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.729+0000 7f0dea000640 1 -- 172.21.15.43:0/820647388 shutdown_connections 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.729+0000 7f0dea000640 1 -- 172.21.15.43:0/820647388 >> 172.21.15.43:0/820647388 conn(0x7f0de40d30a0 msgr2=0x7f0de40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.729+0000 7f0dea000640 1 -- 172.21.15.43:0/820647388 shutdown_connections 2024-01-16T14:59:07.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.729+0000 7f0dea000640 1 -- 172.21.15.43:0/820647388 wait complete. 2024-01-16T14:59:07.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.731+0000 7f0dea000640 1 Processor -- start 2024-01-16T14:59:07.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.731+0000 7f0dea000640 1 -- start start 2024-01-16T14:59:07.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0de409ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 0x7f0de41f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0de41f3810 0x7f0de41f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de409f7c0 con 0x7f0dc4003a10 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40cbb30 con 0x7f0de41f3810 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0dea000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de409dd90 con 0x7f0de409f340 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.732+0000 7f0de8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0de409ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 0x7f0de41f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 0x7f0de41f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:43332/0 (socket says 172.21.15.43:43332) 2024-01-16T14:59:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de3e00640 1 -- 172.21.15.43:0/3091899579 learned_addr learned my addr 172.21.15.43:0/3091899579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:07.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de9600640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0de41f3810 0x7f0de41f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de8c00640 1 -- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 msgr2=0x7f0de41f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de8c00640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 0x7f0de41f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de8c00640 1 -- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0de41f3810 msgr2=0x7f0de41f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de8c00640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0de41f3810 0x7f0de41f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.733+0000 7f0de8c00640 1 -- 172.21.15.43:0/3091899579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de409d180 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0de8c00640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0de409ee00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400bc20 tx=0x7f0dd4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd403b070 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd403c040 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40072d0 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de409d460 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de409da20 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.734+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0dd4030070 con 0x7f0dc4003a10 2024-01-16T14:59:07.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.735+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0dd4034bd0 con 0x7f0dc4003a10 2024-01-16T14:59:07.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.735+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de409dbb0 con 0x7f0dc4003a10 2024-01-16T14:59:07.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.739+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0dd4061ce0 con 0x7f0dc4003a10 2024-01-16T14:59:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.819+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0de404e840 con 0x7f0dc4003a10 2024-01-16T14:59:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.821+0000 7f0de0c00640 1 -- 172.21.15.43:0/3091899579 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0dd4050020 con 0x7f0dc4003a10 2024-01-16T14:59:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 msgr2=0x7f0de409ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0de409ee00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400bc20 tx=0x7f0dd4001120 comp rx=0 tx=0).stop 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 shutdown_connections 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0de41f3810 0x7f0de41f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0de409f340 0x7f0de41f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 --2- 172.21.15.43:0/3091899579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc4003a10 0x7f0de409ee00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 >> 172.21.15.43:0/3091899579 conn(0x7f0de40d30a0 msgr2=0x7f0de40c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 shutdown_connections 2024-01-16T14:59:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:07.823+0000 7f0dea000640 1 -- 172.21.15.43:0/3091899579 wait complete. 2024-01-16T14:59:07.822 INFO:tasks.ceph:check_enable_crimson: False 2024-01-16T14:59:07.823 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T14:59:07.823 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:07.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T14:59:07.900 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:07.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T14:59:08.032 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:08.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T14:59:08.116 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T14:59:08.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-16T14:59:08.250 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:08.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-16T14:59:08.285 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:08.304+0000 7fd56e63b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:59:08.355 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:08.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-16T14:59:08.355 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:08.367+0000 7fd56e63b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:59:08.446 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:08.460+0000 7fa23a63a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:59:08.447 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:08.447 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-16T14:59:08.502 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:08.521+0000 7fa23a63a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:59:08.584 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T14:59:08.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-16T14:59:08.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d 0 2024-01-16T14:59:08.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.969+0000 7f0ec9c00640 1 Processor -- start 2024-01-16T14:59:08.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.969+0000 7f0ec9c00640 1 -- start start 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 0x7f0ec40b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec9c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40b23e0 con 0x7f0ec4160bd0 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec9c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40d41b0 con 0x7f0ec40b48d0 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40cdc70 con 0x7f0ec40b3210 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec2000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0ec40b48d0 0x7f0ec4165280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45684/0 (socket says 172.21.15.69:45684) 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec2000640 1 -- 172.21.15.69:0/2049015756 learned_addr learned my addr 172.21.15.69:0/2049015756 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-16T14:59:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec3400640 1 --2- 172.21.15.69:0/2049015756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 0x7f0ec40b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.971+0000 7f0ec1600640 1 -- 172.21.15.69:0/2049015756 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126796934 0 0) 0x7f0ec40d41b0 con 0x7f0ec40b48d0 2024-01-16T14:59:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.972+0000 7f0ec1600640 1 -- 172.21.15.69:0/2049015756 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003520 con 0x7f0ec40b48d0 2024-01-16T14:59:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.972+0000 7f0ec3400640 1 -- 172.21.15.69:0/2049015756 >> v1:172.21.15.69:6789/0 conn(0x7f0ec40b48d0 legacy=0x7f0ec4165280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.972+0000 7f0ec3400640 1 -- 172.21.15.69:0/2049015756 >> v1:172.21.15.43:6789/0 conn(0x7f0ec4160bd0 legacy=0x7f0ec40b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.972+0000 7f0ec3400640 1 -- 172.21.15.69:0/2049015756 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4160fb0 con 0x7f0ec40b3210 2024-01-16T14:59:08.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.973+0000 7f0ec3400640 1 --2- 172.21.15.69:0/2049015756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 0x7f0ec40b1ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009dc0 tx=0x7f0ebc02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d55b631d702a032 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:08.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.973+0000 7f0ec1600640 1 -- 172.21.15.69:0/2049015756 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc02cc40 con 0x7f0ec40b3210 2024-01-16T14:59:08.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.973+0000 7f0ec1600640 1 -- 172.21.15.69:0/2049015756 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc02cda0 con 0x7f0ec40b3210 2024-01-16T14:59:08.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.973+0000 7f0ec1600640 1 -- 172.21.15.69:0/2049015756 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc034660 con 0x7f0ec40b3210 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2049015756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 msgr2=0x7f0ec40b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2049015756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 0x7f0ec40b1ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009dc0 tx=0x7f0ebc02be90 comp rx=0 tx=0).stop 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2049015756 shutdown_connections 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2049015756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b3210 0x7f0ec40b1ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2049015756 >> 172.21.15.69:0/2049015756 conn(0x7f0ec409c5a0 msgr2=0x7f0ec40d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2049015756 shutdown_connections 2024-01-16T14:59:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.974+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2049015756 wait complete. 2024-01-16T14:59:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.976+0000 7f0ec9c00640 1 Processor -- start 2024-01-16T14:59:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.976+0000 7f0ec9c00640 1 -- start start 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 0x7f0ec40b8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec4160bd0 0x7f0ec40c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40cb620 con 0x7f0ec4160bd0 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40c9a40 con 0x7f0ec40b3210 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40c9d20 con 0x7f0ec40b48d0 2024-01-16T14:59:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 0x7f0ec40b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57890/0 (socket says 172.21.15.69:57890) 2024-01-16T14:59:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 -- 172.21.15.69:0/2493227990 learned_addr learned my addr 172.21.15.69:0/2493227990 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 0x7f0ec40b8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36810/0 (socket says 172.21.15.69:36810) 2024-01-16T14:59:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec3400640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec4160bd0 0x7f0ec40c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 -- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 msgr2=0x7f0ec40b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 0x7f0ec40b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 -- 172.21.15.69:0/2493227990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec4160bd0 msgr2=0x7f0ec40c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec4160bd0 0x7f0ec40c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.977+0000 7f0ec2000640 1 -- 172.21.15.69:0/2493227990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc009a70 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ec2000640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c420 tx=0x7f0eac00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac005ce0 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac005e40 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec40b7b30 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0072d0 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec40b80f0 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0eac01c440 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.978+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0eac012810 con 0x7f0ec40b48d0 2024-01-16T14:59:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.979+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec4067560 con 0x7f0ec40b48d0 2024-01-16T14:59:08.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:08.982+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0eac021060 con 0x7f0ec40b48d0 2024-01-16T14:59:08.970 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:08.989+0000 7fd56e63b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:59:09.034 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.053+0000 7fd56e63b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:59:09.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.060+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d", "id": 0} v 0) v1 -- 0x7f0ec40a1210 con 0x7f0ec40b48d0 2024-01-16T14:59:09.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.066+0000 7f0ebaa00640 1 -- 172.21.15.69:0/2493227990 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0eac017070 con 0x7f0ec40b48d0 2024-01-16T14:59:09.046 INFO:teuthology.orchestra.run.smithi069.stdout:0 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 msgr2=0x7f0ec415d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c420 tx=0x7f0eac00c8e0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 shutdown_connections 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec4160bd0 0x7f0ec40c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec40b48d0 0x7f0ec415d070 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 --2- 172.21.15.69:0/2493227990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec40b3210 0x7f0ec40b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 >> 172.21.15.69:0/2493227990 conn(0x7f0ec409c5a0 msgr2=0x7f0ec406bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 shutdown_connections 2024-01-16T14:59:09.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.067+0000 7f0ec9c00640 1 -- 172.21.15.69:0/2493227990 wait complete. 2024-01-16T14:59:09.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new df57693a-062d-49fd-86dc-1bf0760e6ab2 1 2024-01-16T14:59:09.088 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.106+0000 7fd56e63b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:59:09.133 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.152+0000 7fa23a63a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:59:09.153 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.171+0000 7fd56e63b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:59:09.198 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.216+0000 7fa23a63a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:59:09.204 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.223+0000 7fd56e63b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:59:09.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.261+0000 7f58c2400640 1 Processor -- start 2024-01-16T14:59:09.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.262+0000 7f58c2400640 1 -- start start 2024-01-16T14:59:09.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58bc160ab0 0x7f58bc0d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0cff90 con 0x7f58bc160ab0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0cf190 con 0x7f58bc0cfae0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c2400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b2e60 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.263+0000 7f58c1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58c1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36816/0 (socket says 172.21.15.69:36816) 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57904/0 (socket says 172.21.15.69:57904) 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58c1a00640 1 -- 172.21.15.69:0/3032575141 learned_addr learned my addr 172.21.15.69:0/3032575141 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58c1000640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.43:3300/0 conn(0x7f58bc160ab0 0x7f58bc0d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 -- 172.21.15.69:0/3032575141 >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 msgr2=0x7f58bc0d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 -- 172.21.15.69:0/3032575141 >> v2:172.21.15.43:3300/0 conn(0x7f58bc160ab0 msgr2=0x7f58bc0d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.43:3300/0 conn(0x7f58bc160ab0 0x7f58bc0d3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58b3e00640 1 -- 172.21.15.69:0/3032575141 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0cebd0 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.264+0000 7f58c1a00640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:09.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.265+0000 7f58b3e00640 1 --2- 172.21.15.69:0/3032575141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b2dd0 tx=0x7f58a402b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ed2aebe988971e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.265+0000 7f58bbe00640 1 -- 172.21.15.69:0/3032575141 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4031520 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.265+0000 7f58bbe00640 1 -- 172.21.15.69:0/3032575141 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4031680 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.265+0000 7f58bbe00640 1 -- 172.21.15.69:0/3032575141 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4031a10 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 -- 172.21.15.69:0/3032575141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 msgr2=0x7f58bc0cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 --2- 172.21.15.69:0/3032575141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b2dd0 tx=0x7f58a402b7c0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 -- 172.21.15.69:0/3032575141 shutdown_connections 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.69:3300/0 conn(0x7f58bc0cfae0 0x7f58bc0d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 --2- 172.21.15.69:0/3032575141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0ce6f0 0x7f58bc0cf5a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 --2- 172.21.15.69:0/3032575141 >> v2:172.21.15.43:3300/0 conn(0x7f58bc160ab0 0x7f58bc0d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 -- 172.21.15.69:0/3032575141 >> 172.21.15.69:0/3032575141 conn(0x7f58bc0d3060 msgr2=0x7f58bc0d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 -- 172.21.15.69:0/3032575141 shutdown_connections 2024-01-16T14:59:09.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.266+0000 7f58c2400640 1 -- 172.21.15.69:0/3032575141 wait complete. 2024-01-16T14:59:09.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.269+0000 7f58c2400640 1 Processor -- start 2024-01-16T14:59:09.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.269+0000 7f58c2400640 1 -- start start 2024-01-16T14:59:09.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58bc0ce6f0 0x7f58bc15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58bc160ab0 0x7f58bc0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0ca130 con 0x7f58bc0ce6f0 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0ca3f0 con 0x7f58bc160ab0 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c2400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0ca6d0 con 0x7f58bc0cfae0 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58bc160ab0 0x7f58bc0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57908/0 (socket says 172.21.15.69:57908) 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 -- 172.21.15.69:0/347620668 learned_addr learned my addr 172.21.15.69:0/347620668 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:09.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58c1000640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58bc0ce6f0 0x7f58bc15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 -- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58bc160ab0 msgr2=0x7f58bc0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58bc160ab0 0x7f58bc0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 -- 172.21.15.69:0/347620668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58bc0ce6f0 msgr2=0x7f58bc15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58bc0ce6f0 0x7f58bc15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.271+0000 7f58b3e00640 1 -- 172.21.15.69:0/347620668 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4009b30 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b3e00640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58a4000c00 tx=0x7f58a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4031520 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4033c70 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a4009400 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4006e10 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc0cac90 con 0x7f58bc0cfae0 2024-01-16T14:59:09.255 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.272+0000 7fa23a63a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:59:09.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f58a403a040 con 0x7f58bc0cfae0 2024-01-16T14:59:09.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.272+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f58a4049a20 con 0x7f58bc0cfae0 2024-01-16T14:59:09.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.274+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0cae80 con 0x7f58bc0cfae0 2024-01-16T14:59:09.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.276+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f58a4068ce0 con 0x7f58bc0cfae0 2024-01-16T14:59:09.323 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.342+0000 7fd56e63b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:59:09.326 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.344+0000 7fa23a63a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:59:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.363+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "df57693a-062d-49fd-86dc-1bf0760e6ab2", "id": 1} v 0) v1 -- 0x7f58bc04e840 con 0x7f58bc0cfae0 2024-01-16T14:59:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.372+0000 7f58b9600640 1 -- 172.21.15.69:0/347620668 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df57693a-062d-49fd-86dc-1bf0760e6ab2", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f58a4057030 con 0x7f58bc0cfae0 2024-01-16T14:59:09.354 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 msgr2=0x7f58bc0b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58a4000c00 tx=0x7f58a40046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 shutdown_connections 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58bc160ab0 0x7f58bc0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58bc0cfae0 0x7f58bc0b5140 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 --2- 172.21.15.69:0/347620668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58bc0ce6f0 0x7f58bc15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 >> 172.21.15.69:0/347620668 conn(0x7f58bc0d3060 msgr2=0x7f58bc0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 shutdown_connections 2024-01-16T14:59:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.373+0000 7f58c2400640 1 -- 172.21.15.69:0/347620668 wait complete. 2024-01-16T14:59:09.370 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 8f01e100-16a0-45c4-878f-3541ba179b4e 2 2024-01-16T14:59:09.379 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.397+0000 7fa23a63a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:59:09.407 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.425+0000 7fd56e63b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:09.462 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.481+0000 7fd56e63b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:59:09.498 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.516+0000 7fa23a63a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:59:09.524 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.543+0000 7fd56e63b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:59:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.575+0000 7f2eeee00640 1 Processor -- start 2024-01-16T14:59:09.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.575+0000 7f2eeee00640 1 -- start start 2024-01-16T14:59:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.577+0000 7f2eeee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ee80cfcb0 0x7f2ee80d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.577+0000 7f2eeee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80cf190 con 0x7f2ee8160ab0 2024-01-16T14:59:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.577+0000 7f2eeee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80b2e60 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.577+0000 7f2eeee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee80cebd0 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eee400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2ee80ce6f0 0x7f2ee80b32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45698/0 (socket says 172.21.15.69:45698) 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eee400640 1 -- 172.21.15.69:0/232581668 learned_addr learned my addr 172.21.15.69:0/232581668 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eed000640 1 --2- 172.21.15.69:0/232581668 >> v2:172.21.15.69:3300/0 conn(0x7f2ee80cfcb0 0x7f2ee80d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eed000640 1 -- 172.21.15.69:0/232581668 >> v1:172.21.15.69:6789/0 conn(0x7f2ee80ce6f0 legacy=0x7f2ee80b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eed000640 1 -- 172.21.15.69:0/232581668 >> v1:172.21.15.43:6789/0 conn(0x7f2ee8160ab0 legacy=0x7f2ee80cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.578+0000 7f2eed000640 1 -- 172.21.15.69:0/232581668 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80ceeb0 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.579+0000 7f2eed000640 1 --2- 172.21.15.69:0/232581668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80cfcb0 0x7f2ee80d3b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00097a0 tx=0x7f2ed002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd99a94ff012b764 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.579+0000 7f2ee7e00640 1 -- 172.21.15.69:0/232581668 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0031830 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2ee7e00640 1 -- 172.21.15.69:0/232581668 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0031990 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2ee7e00640 1 -- 172.21.15.69:0/232581668 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0031d20 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2eeee00640 1 -- 172.21.15.69:0/232581668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80cfcb0 msgr2=0x7f2ee80d3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2eeee00640 1 --2- 172.21.15.69:0/232581668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80cfcb0 0x7f2ee80d3b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00097a0 tx=0x7f2ed002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2eeee00640 1 -- 172.21.15.69:0/232581668 shutdown_connections 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2eeee00640 1 --2- 172.21.15.69:0/232581668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80cfcb0 0x7f2ee80d3b80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.580+0000 7f2eeee00640 1 -- 172.21.15.69:0/232581668 >> 172.21.15.69:0/232581668 conn(0x7f2ee80d3060 msgr2=0x7f2ee80d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.581+0000 7f2eeee00640 1 -- 172.21.15.69:0/232581668 shutdown_connections 2024-01-16T14:59:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.581+0000 7f2eeee00640 1 -- 172.21.15.69:0/232581668 wait complete. 2024-01-16T14:59:09.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.582+0000 7f2eeee00640 1 Processor -- start 2024-01-16T14:59:09.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.582+0000 7f2eeee00640 1 -- start start 2024-01-16T14:59:09.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 0x7f2ee80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 0x7f2ee8158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80cad70 con 0x7f2ee8160ab0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8159f20 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8158c90 con 0x7f2ee80cfcb0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eeda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 0x7f2ee80c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.583+0000 7f2eed000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eed000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57910/0 (socket says 172.21.15.69:57910) 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eed000640 1 -- 172.21.15.69:0/3451701448 learned_addr learned my addr 172.21.15.69:0/3451701448 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eee400640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 0x7f2ee8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 -- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 msgr2=0x7f2ee81599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 -- 172.21.15.69:0/3451701448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 msgr2=0x7f2ee8158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 0x7f2ee8158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 -- 172.21.15.69:0/3451701448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0009400 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eee400640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 0x7f2ee8158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:09.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eed000640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:09.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.584+0000 7f2eeda00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 0x7f2ee80c75a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c3c0 tx=0x7f2ed800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed80052d0 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8005430 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8003830 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee80c7c30 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee80c81f0 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2ed8003c90 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.585+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f2ed80119b0 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.586+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee804e840 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.589+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2ed8017070 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.581 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:09.600+0000 7fd56e63b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:59:09.584 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.603+0000 7fa23a63a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:09.638 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.656+0000 7fa23a63a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:59:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.667+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f01e100-16a0-45c4-878f-3541ba179b4e", "id": 2} v 0) v1 -- 0x7f2ee815e0e0 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.676+0000 7f2ee5600640 1 -- 172.21.15.69:0/3451701448 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f01e100-16a0-45c4-878f-3541ba179b4e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2ed8011e00 con 0x7f2ee80ce6f0 2024-01-16T14:59:09.657 INFO:teuthology.orchestra.run.smithi069.stdout:2 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 msgr2=0x7f2ee80c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 0x7f2ee80c75a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c3c0 tx=0x7f2ed800c880 comp rx=0 tx=0).stop 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 shutdown_connections 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee8160ab0 0x7f2ee8158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ee80cfcb0 0x7f2ee81599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 --2- 172.21.15.69:0/3451701448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ee80ce6f0 0x7f2ee80c75a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 >> 172.21.15.69:0/3451701448 conn(0x7f2ee80d3060 msgr2=0x7f2ee80cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 shutdown_connections 2024-01-16T14:59:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.677+0000 7f2eeee00640 1 -- 172.21.15.69:0/3451701448 wait complete. 2024-01-16T14:59:09.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new d8ae43ad-6974-4048-a6dc-dcf76520ac23 3 2024-01-16T14:59:09.701 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.720+0000 7fa23a63a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:59:09.760 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:09.778+0000 7fa23a63a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:59:09.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.910+0000 7f03f3200640 1 Processor -- start 2024-01-16T14:59:09.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.910+0000 7f03f3200640 1 -- start start 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 0x7f03f40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f3200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40d2010 con 0x7f03f40d1bc0 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f40c6bf0 con 0x7f03f40d0fa0 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40d1400 con 0x7f03f415d3b0 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03f1e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 0x7f03f40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36854/0 (socket says 172.21.15.69:36854) 2024-01-16T14:59:09.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.912+0000 7f03ebe00640 1 -- 172.21.15.69:0/3448000748 learned_addr learned my addr 172.21.15.69:0/3448000748 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.913+0000 7f03f1e00640 1 -- 172.21.15.69:0/3448000748 >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 msgr2=0x7f03f40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.913+0000 7f03f1e00640 1 --2- 172.21.15.69:0/3448000748 >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.913+0000 7f03f1e00640 1 -- 172.21.15.69:0/3448000748 >> v1:172.21.15.43:6789/0 conn(0x7f03f40d1bc0 legacy=0x7f03f40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.913+0000 7f03f1e00640 1 -- 172.21.15.69:0/3448000748 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40cedc0 con 0x7f03f415d3b0 2024-01-16T14:59:09.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.913+0000 7f03ebe00640 1 --2- 172.21.15.69:0/3448000748 >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.914+0000 7f03f1e00640 1 --2- 172.21.15.69:0/3448000748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 0x7f03f40ce6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f03e40097a0 tx=0x7f03e402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d64719385405da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.914+0000 7f03f1400640 1 -- 172.21.15.69:0/3448000748 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e40316f0 con 0x7f03f415d3b0 2024-01-16T14:59:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.914+0000 7f03f1400640 1 -- 172.21.15.69:0/3448000748 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e4031850 con 0x7f03f415d3b0 2024-01-16T14:59:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.914+0000 7f03f1400640 1 -- 172.21.15.69:0/3448000748 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4031be0 con 0x7f03f415d3b0 2024-01-16T14:59:09.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 -- 172.21.15.69:0/3448000748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 msgr2=0x7f03f40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 --2- 172.21.15.69:0/3448000748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 0x7f03f40ce6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f03e40097a0 tx=0x7f03e402b790 comp rx=0 tx=0).stop 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 -- 172.21.15.69:0/3448000748 shutdown_connections 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 --2- 172.21.15.69:0/3448000748 >> v2:172.21.15.69:3300/0 conn(0x7f03f40d0fa0 0x7f03f40d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 --2- 172.21.15.69:0/3448000748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f415d3b0 0x7f03f40ce6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 -- 172.21.15.69:0/3448000748 >> 172.21.15.69:0/3448000748 conn(0x7f03f40d30a0 msgr2=0x7f03f40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 -- 172.21.15.69:0/3448000748 shutdown_connections 2024-01-16T14:59:09.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.915+0000 7f03f3200640 1 -- 172.21.15.69:0/3448000748 wait complete. 2024-01-16T14:59:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.917+0000 7f03f3200640 1 Processor -- start 2024-01-16T14:59:09.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.917+0000 7f03f3200640 1 -- start start 2024-01-16T14:59:09.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.917+0000 7f03f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f40d0fa0 0x7f03f414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 0x7f03f414bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b7e30 con 0x7f03f40d0fa0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f414ba00 con 0x7f03f415d3b0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41e5f10 con 0x7f03f40d1bc0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 0x7f03f414bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57922/0 (socket says 172.21.15.69:57922) 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 -- 172.21.15.69:0/529598342 learned_addr learned my addr 172.21.15.69:0/529598342 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f1e00640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f40d0fa0 0x7f03f414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 -- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 msgr2=0x7f03f414bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 0x7f03f414bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 -- 172.21.15.69:0/529598342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f40d0fa0 msgr2=0x7f03f414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f40d0fa0 0x7f03f414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03ebe00640 1 -- 172.21.15.69:0/529598342 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0094d0 con 0x7f03f40d1bc0 2024-01-16T14:59:09.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.918+0000 7f03f2800640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 0x7f03f414bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ebe00640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9b0 tx=0x7f03dc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00c9a0 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc00cb00 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e4009400 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00ce70 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f40b7500 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f03dc005c90 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.919+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f03dc0064b0 con 0x7f03f40d1bc0 2024-01-16T14:59:09.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.920+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f40b7fa0 con 0x7f03f40d1bc0 2024-01-16T14:59:09.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.923+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f03dc03bce0 con 0x7f03f40d1bc0 2024-01-16T14:59:09.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:09.999+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d8ae43ad-6974-4048-a6dc-dcf76520ac23", "id": 3} v 0) v1 -- 0x7f03f404e840 con 0x7f03f40d1bc0 2024-01-16T14:59:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.005+0000 7f03ea000640 1 -- 172.21.15.69:0/529598342 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8ae43ad-6974-4048-a6dc-dcf76520ac23", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f03dc02a070 con 0x7f03f40d1bc0 2024-01-16T14:59:09.987 INFO:teuthology.orchestra.run.smithi069.stdout:3 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 msgr2=0x7f03f41e59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9b0 tx=0x7f03dc00ee70 comp rx=0 tx=0).stop 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 shutdown_connections 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03f415d3b0 0x7f03f414bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03f40d1bc0 0x7f03f41e59d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 --2- 172.21.15.69:0/529598342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f40d0fa0 0x7f03f414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.007+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 >> 172.21.15.69:0/529598342 conn(0x7f03f40d30a0 msgr2=0x7f03f4160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:09.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.008+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 shutdown_connections 2024-01-16T14:59:09.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.008+0000 7f03f3200640 1 -- 172.21.15.69:0/529598342 wait complete. 2024-01-16T14:59:10.005 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new cd697498-049e-4431-9b54-53ef478ca92c 4 2024-01-16T14:59:10.053 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.071+0000 7fd56e63b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:59:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.208+0000 7fc0af400640 1 Processor -- start 2024-01-16T14:59:10.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.208+0000 7fc0af400640 1 -- start start 2024-01-16T14:59:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0a8160ab0 0x7fc0a80d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80ce6f0 0x7fc0a80cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 0x7fc0a80d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a80cff90 con 0x7fc0a80cfae0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a80cf190 con 0x7fc0a8160ab0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0af400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80b2e60 con 0x7fc0a80ce6f0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0a8160ab0 0x7fc0a80d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0a8160ab0 0x7fc0a80d3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36866/0 (socket says 172.21.15.69:36866) 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 -- 172.21.15.69:0/2659857293 learned_addr learned my addr 172.21.15.69:0/2659857293 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ad600640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80ce6f0 0x7fc0a80cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0aea00640 1 --2- 172.21.15.69:0/2659857293 >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 0x7fc0a80d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 -- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80ce6f0 msgr2=0x7fc0a80cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80ce6f0 0x7fc0a80cf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 -- 172.21.15.69:0/2659857293 >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 msgr2=0x7fc0a80d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 --2- 172.21.15.69:0/2659857293 >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 0x7fc0a80d0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0ae000640 1 -- 172.21.15.69:0/2659857293 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80cebd0 con 0x7fc0a8160ab0 2024-01-16T14:59:10.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.210+0000 7fc0aea00640 1 --2- 172.21.15.69:0/2659857293 >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 0x7fc0a80d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.211+0000 7fc0ae000640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80d3b80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8068030 tx=0x7fc09802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdda34dd8a24b8b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.212+0000 7fc0acc00640 1 -- 172.21.15.69:0/2659857293 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0980317e0 con 0x7fc0a8160ab0 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.212+0000 7fc0acc00640 1 -- 172.21.15.69:0/2659857293 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc098031940 con 0x7fc0a8160ab0 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.212+0000 7fc0acc00640 1 -- 172.21.15.69:0/2659857293 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098031cd0 con 0x7fc0a8160ab0 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.212+0000 7fc0af400640 1 -- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8160ab0 msgr2=0x7fc0a80d3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.212+0000 7fc0af400640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80d3b80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8068030 tx=0x7fc09802b880 comp rx=0 tx=0).stop 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 -- 172.21.15.69:0/2659857293 shutdown_connections 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 --2- 172.21.15.69:0/2659857293 >> v2:172.21.15.43:3300/0 conn(0x7fc0a80cfae0 0x7fc0a80d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80ce6f0 0x7fc0a80cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 --2- 172.21.15.69:0/2659857293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80d3b80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 -- 172.21.15.69:0/2659857293 >> 172.21.15.69:0/2659857293 conn(0x7fc0a80d3060 msgr2=0x7fc0a80d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 -- 172.21.15.69:0/2659857293 shutdown_connections 2024-01-16T14:59:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.213+0000 7fc0af400640 1 -- 172.21.15.69:0/2659857293 wait complete. 2024-01-16T14:59:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.214+0000 7fc0af400640 1 Processor -- start 2024-01-16T14:59:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.214+0000 7fc0af400640 1 -- start start 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80ce6f0 0x7fc0a8153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c7ca0 con 0x7fc0a8160ab0 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8153b40 con 0x7fc0a80ce6f0 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.215+0000 7fc0af400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c76e0 con 0x7fc0a80cfae0 2024-01-16T14:59:10.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80ce6f0 0x7fc0a8153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57930/0 (socket says 172.21.15.69:57930) 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 -- 172.21.15.69:0/1247337193 learned_addr learned my addr 172.21.15.69:0/1247337193 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0aea00640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 -- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80ce6f0 msgr2=0x7fc0a8153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80ce6f0 0x7fc0a8153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 -- 172.21.15.69:0/1247337193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a8160ab0 msgr2=0x7fc0a80c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 -- 172.21.15.69:0/1247337193 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980094d0 con 0x7fc0a80cfae0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.216+0000 7fc0ad600640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e890 tx=0x7fc09000ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000c940 con 0x7fc0a80cfae0 2024-01-16T14:59:10.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09000caa0 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000ce30 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a80c79c0 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a80cae80 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc090005c90 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.217+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fc090013470 con 0x7fc0a80cfae0 2024-01-16T14:59:10.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.218+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a804e840 con 0x7fc0a80cfae0 2024-01-16T14:59:10.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.220+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc090021060 con 0x7fc0a80cfae0 2024-01-16T14:59:10.220 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.239+0000 7fd56e63b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:59:10.233 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.252+0000 7fa23a63a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:59:10.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.298+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cd697498-049e-4431-9b54-53ef478ca92c", "id": 4} v 0) v1 -- 0x7fc0a80c64f0 con 0x7fc0a80cfae0 2024-01-16T14:59:10.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.304+0000 7fc0a6000640 1 -- 172.21.15.69:0/1247337193 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd697498-049e-4431-9b54-53ef478ca92c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc090029020 con 0x7fc0a80cfae0 2024-01-16T14:59:10.286 INFO:teuthology.orchestra.run.smithi069.stdout:4 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 msgr2=0x7fc0a80cb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e890 tx=0x7fc09000ed50 comp rx=0 tx=0).stop 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 shutdown_connections 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a8160ab0 0x7fc0a80c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80cfae0 0x7fc0a80cb3a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 --2- 172.21.15.69:0/1247337193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80ce6f0 0x7fc0a8153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 >> 172.21.15.69:0/1247337193 conn(0x7fc0a80d3060 msgr2=0x7fc0a815d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 shutdown_connections 2024-01-16T14:59:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.305+0000 7fc0af400640 1 -- 172.21.15.69:0/1247337193 wait complete. 2024-01-16T14:59:10.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 57205772-35f8-44d3-a420-bf41c3ac6ea5 5 2024-01-16T14:59:10.371 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.389+0000 7fd56e63b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:59:10.400 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.419+0000 7fa23a63a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:59:10.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.445+0000 7fd56e63b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:59:10.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.507+0000 7f771e600640 1 Processor -- start 2024-01-16T14:59:10.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.507+0000 7f771e600640 1 -- start start 2024-01-16T14:59:10.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771e600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 0x7f77180d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771e600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77180ce6f0 0x7f77180b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771e600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77180cf190 con 0x7f7718160ab0 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771e600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77180b2e60 con 0x7f77180cfcb0 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771e600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77180cebd0 con 0x7f77180ce6f0 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771d200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7718160ab0 0x7f77180cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45714/0 (socket says 172.21.15.69:45714) 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771d200640 1 -- 172.21.15.69:0/3417597649 learned_addr learned my addr 172.21.15.69:0/3417597649 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f771dc00640 1 --2- 172.21.15.69:0/3417597649 >> v2:172.21.15.69:3300/0 conn(0x7f77180ce6f0 0x7f77180b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.509+0000 7f770fc00640 1 --2- 172.21.15.69:0/3417597649 >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 0x7f77180d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.510+0000 7f771dc00640 1 -- 172.21.15.69:0/3417597649 >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 msgr2=0x7f77180d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.510+0000 7f771dc00640 1 --2- 172.21.15.69:0/3417597649 >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 0x7f77180d3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.510+0000 7f771dc00640 1 -- 172.21.15.69:0/3417597649 >> v1:172.21.15.69:6789/0 conn(0x7f7718160ab0 legacy=0x7f77180cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.510+0000 7f771dc00640 1 -- 172.21.15.69:0/3417597649 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180ceeb0 con 0x7f77180ce6f0 2024-01-16T14:59:10.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.510+0000 7f770fc00640 1 --2- 172.21.15.69:0/3417597649 >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 0x7f77180d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:10.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.511+0000 7f771dc00640 1 --2- 172.21.15.69:0/3417597649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77180ce6f0 0x7f77180b32f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f77180cda00 tx=0x7f771002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51a80ebb1a1f4d69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f7717e00640 1 -- 172.21.15.69:0/3417597649 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771002cbf0 con 0x7f77180ce6f0 2024-01-16T14:59:10.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f7717e00640 1 -- 172.21.15.69:0/3417597649 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771002cd50 con 0x7f77180ce6f0 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f7717e00640 1 -- 172.21.15.69:0/3417597649 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710034660 con 0x7f77180ce6f0 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 -- 172.21.15.69:0/3417597649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77180ce6f0 msgr2=0x7f77180b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 --2- 172.21.15.69:0/3417597649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77180ce6f0 0x7f77180b32f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f77180cda00 tx=0x7f771002be40 comp rx=0 tx=0).stop 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 -- 172.21.15.69:0/3417597649 shutdown_connections 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 --2- 172.21.15.69:0/3417597649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77180ce6f0 0x7f77180b32f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 --2- 172.21.15.69:0/3417597649 >> v2:172.21.15.43:3300/0 conn(0x7f77180cfcb0 0x7f77180d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 -- 172.21.15.69:0/3417597649 >> 172.21.15.69:0/3417597649 conn(0x7f77180d3060 msgr2=0x7f77180d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 -- 172.21.15.69:0/3417597649 shutdown_connections 2024-01-16T14:59:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.512+0000 7f771e600640 1 -- 172.21.15.69:0/3417597649 wait complete. 2024-01-16T14:59:10.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.514+0000 7f771e600640 1 Processor -- start 2024-01-16T14:59:10.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.514+0000 7f771e600640 1 -- start start 2024-01-16T14:59:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77180cfcb0 0x7f77180cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7718160ab0 0x7f771806b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180cab40 con 0x7f77180cfcb0 2024-01-16T14:59:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180aff80 con 0x7f7718160ab0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771e600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77180cad90 con 0x7f77180ce6f0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57936/0 (socket says 172.21.15.69:57936) 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771d200640 1 -- 172.21.15.69:0/369369646 learned_addr learned my addr 172.21.15.69:0/369369646 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.515+0000 7f771dc00640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7718160ab0 0x7f771806b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f770fc00640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77180cfcb0 0x7f77180cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 -- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7718160ab0 msgr2=0x7f771806b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7718160ab0 0x7f771806b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 -- 172.21.15.69:0/369369646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77180cfcb0 msgr2=0x7f77180cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77180cfcb0 0x7f77180cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 -- 172.21.15.69:0/369369646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710009a70 con 0x7f77180ce6f0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771d200640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f770800c480 tx=0x7f770800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708005ce0 con 0x7f77180ce6f0 2024-01-16T14:59:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708005e40 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77080072d0 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.516+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77180cb070 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.517+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77180cb350 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.517+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7708012400 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.517+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7708012c30 con 0x7f77180ce6f0 2024-01-16T14:59:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.518+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77180b0150 con 0x7f77180ce6f0 2024-01-16T14:59:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.520+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7708021060 con 0x7f77180ce6f0 2024-01-16T14:59:10.535 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.554+0000 7fa23a63a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:59:10.540 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.559+0000 7fd56e63b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:10.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.597+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "57205772-35f8-44d3-a420-bf41c3ac6ea5", "id": 5} v 0) v1 -- 0x7f771804e840 con 0x7f77180ce6f0 2024-01-16T14:59:10.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.604+0000 7f7715600640 1 -- 172.21.15.69:0/369369646 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57205772-35f8-44d3-a420-bf41c3ac6ea5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7708018070 con 0x7f77180ce6f0 2024-01-16T14:59:10.585 INFO:teuthology.orchestra.run.smithi069.stdout:5 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 msgr2=0x7f77181f71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f770800c480 tx=0x7f770800c940 comp rx=0 tx=0).stop 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 shutdown_connections 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7718160ab0 0x7f771806b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77180cfcb0 0x7f77180cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 --2- 172.21.15.69:0/369369646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77180ce6f0 0x7f77181f71a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 >> 172.21.15.69:0/369369646 conn(0x7f77180d3060 msgr2=0x7f77180d3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 shutdown_connections 2024-01-16T14:59:10.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.605+0000 7f771e600640 1 -- 172.21.15.69:0/369369646 wait complete. 2024-01-16T14:59:10.590 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.609+0000 7fa23a63a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:59:10.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 7b61ef01-2c5e-476c-95f0-665c3757bc39 6 2024-01-16T14:59:10.657 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.675+0000 7fd56e63b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:59:10.707 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.726+0000 7fa23a63a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:10.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.808+0000 7f9172c00640 1 Processor -- start 2024-01-16T14:59:10.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.808+0000 7f9172c00640 1 -- start start 2024-01-16T14:59:10.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9172c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9172c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f916c0d0fa0 0x7f916c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9172c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0c6bf0 con 0x7f916c15d3b0 2024-01-16T14:59:10.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9172c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0d13d0 con 0x7f916c0d0fa0 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9172c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0cedc0 con 0x7f916c0d1d90 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.809+0000 7f9170e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:57940/0 (socket says 172.21.15.69:57940) 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 -- 172.21.15.69:0/2443581672 learned_addr learned my addr 172.21.15.69:0/2443581672 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9172200640 1 --2- 172.21.15.69:0/2443581672 >> v2:172.21.15.43:3300/0 conn(0x7f916c0d0fa0 0x7f916c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 -- 172.21.15.69:0/2443581672 >> v2:172.21.15.43:3300/0 conn(0x7f916c0d0fa0 msgr2=0x7f916c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 --2- 172.21.15.69:0/2443581672 >> v2:172.21.15.43:3300/0 conn(0x7f916c0d0fa0 0x7f916c0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 -- 172.21.15.69:0/2443581672 >> v1:172.21.15.43:6789/0 conn(0x7f916c15d3b0 legacy=0x7f916c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.810+0000 7f9170e00640 1 -- 172.21.15.69:0/2443581672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0c6e40 con 0x7f916c0d1d90 2024-01-16T14:59:10.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.812+0000 7f9170e00640 1 --2- 172.21.15.69:0/2443581672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9154009870 tx=0x7f915402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d8cbbd2cf0056c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.812+0000 7f916be00640 1 -- 172.21.15.69:0/2443581672 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91540317c0 con 0x7f916c0d1d90 2024-01-16T14:59:10.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.812+0000 7f916be00640 1 -- 172.21.15.69:0/2443581672 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154031920 con 0x7f916c0d1d90 2024-01-16T14:59:10.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.812+0000 7f916be00640 1 -- 172.21.15.69:0/2443581672 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154031cb0 con 0x7f916c0d1d90 2024-01-16T14:59:10.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 -- 172.21.15.69:0/2443581672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 msgr2=0x7f916c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 --2- 172.21.15.69:0/2443581672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9154009870 tx=0x7f915402b860 comp rx=0 tx=0).stop 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 -- 172.21.15.69:0/2443581672 shutdown_connections 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 --2- 172.21.15.69:0/2443581672 >> v2:172.21.15.43:3300/0 conn(0x7f916c0d0fa0 0x7f916c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 --2- 172.21.15.69:0/2443581672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c0d1d90 0x7f916c0ce6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 -- 172.21.15.69:0/2443581672 >> 172.21.15.69:0/2443581672 conn(0x7f916c0d30a0 msgr2=0x7f916c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 -- 172.21.15.69:0/2443581672 shutdown_connections 2024-01-16T14:59:10.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.813+0000 7f9172c00640 1 -- 172.21.15.69:0/2443581672 wait complete. 2024-01-16T14:59:10.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.814+0000 7f9172c00640 1 Processor -- start 2024-01-16T14:59:10.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 -- start start 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f916c0d1d90 0x7f916c160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 0x7f916c15da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0cc440 con 0x7f916c0d1d90 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0cc690 con 0x7f916c0d0fa0 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.815+0000 7f9172c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0cc970 con 0x7f916c15d3b0 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9171800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9171800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36910/0 (socket says 172.21.15.69:36910) 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 0x7f916c15da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9171800640 1 -- 172.21.15.69:0/531197636 learned_addr learned my addr 172.21.15.69:0/531197636 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9170e00640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f916c0d1d90 0x7f916c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 -- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 msgr2=0x7f916c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 -- 172.21.15.69:0/531197636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f916c0d1d90 msgr2=0x7f916c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f916c0d1d90 0x7f916c160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9172200640 1 -- 172.21.15.69:0/531197636 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540094d0 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.816+0000 7f9171800640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9172200640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 0x7f916c15da10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9164007d90 tx=0x7f916400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164018070 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916400edd0 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164003bb0 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c0ccc50 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c0cd210 con 0x7f916c15d3b0 2024-01-16T14:59:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9164014720 con 0x7f916c15d3b0 2024-01-16T14:59:10.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.817+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f916401d410 con 0x7f916c15d3b0 2024-01-16T14:59:10.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.818+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c15d790 con 0x7f916c15d3b0 2024-01-16T14:59:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.820+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f91640149d0 con 0x7f916c15d3b0 2024-01-16T14:59:10.824 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.842+0000 7fd56e63b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:10.826 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:10.845+0000 7fa23a63a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:59:10.874 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:10.893+0000 7fd56e63b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:10.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.898+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7b61ef01-2c5e-476c-95f0-665c3757bc39", "id": 6} v 0) v1 -- 0x7f916c04e840 con 0x7f916c15d3b0 2024-01-16T14:59:10.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.905+0000 7f9169600640 1 -- 172.21.15.69:0/531197636 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b61ef01-2c5e-476c-95f0-665c3757bc39", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9164014c70 con 0x7f916c15d3b0 2024-01-16T14:59:10.887 INFO:teuthology.orchestra.run.smithi069.stdout:6 2024-01-16T14:59:10.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.906+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 msgr2=0x7f916c15da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 0x7f916c15da10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9164007d90 tx=0x7f916400e4b0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 shutdown_connections 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f916c15d3b0 0x7f916c15da10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f916c0d1d90 0x7f916c160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 --2- 172.21.15.69:0/531197636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f916c0d0fa0 0x7f916c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 >> 172.21.15.69:0/531197636 conn(0x7f916c0d30a0 msgr2=0x7f916c0c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 shutdown_connections 2024-01-16T14:59:10.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:10.907+0000 7f9172c00640 1 -- 172.21.15.69:0/531197636 wait complete. 2024-01-16T14:59:10.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 1a8a3071-bb4a-4fca-a170-8dad620d0dd9 7 2024-01-16T14:59:10.996 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:11.013+0000 7fa23a63a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:11.042 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:11.061+0000 7fd56e63b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:59:11.047 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:11.065+0000 7fa23a63a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.095+0000 7f0d9fe00640 1 Processor -- start 2024-01-16T14:59:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.096+0000 7f0d9fe00640 1 -- start start 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d9809cb30 0x7f0d980b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d980adb60 0x7f0d980adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9815b730 con 0x7f0d9809cb30 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d980ac740 con 0x7f0d980adb60 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.097+0000 7f0d9fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d980ac460 con 0x7f0d9815b350 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d980adb60 0x7f0d980adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34718/0 (socket says 172.21.15.69:34718) 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 -- 172.21.15.69:0/954266266 learned_addr learned my addr 172.21.15.69:0/954266266 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9ca00640 1 --2- 172.21.15.69:0/954266266 >> v2:172.21.15.43:3300/0 conn(0x7f0d9809cb30 0x7f0d980b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 -- 172.21.15.69:0/954266266 >> v2:172.21.15.69:3300/0 conn(0x7f0d980adb60 msgr2=0x7f0d980adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 --2- 172.21.15.69:0/954266266 >> v2:172.21.15.69:3300/0 conn(0x7f0d980adb60 0x7f0d980adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 -- 172.21.15.69:0/954266266 >> v2:172.21.15.43:3300/0 conn(0x7f0d9809cb30 msgr2=0x7f0d980b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 --2- 172.21.15.69:0/954266266 >> v2:172.21.15.43:3300/0 conn(0x7f0d9809cb30 0x7f0d980b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.098+0000 7f0d9d400640 1 -- 172.21.15.69:0/954266266 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d980b2a50 con 0x7f0d9815b350 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.099+0000 7f0d9d400640 1 --2- 172.21.15.69:0/954266266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0d98072730 tx=0x7f0d8802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8980cc8727e68c9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.099+0000 7f0d97e00640 1 -- 172.21.15.69:0/954266266 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88031870 con 0x7f0d9815b350 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d97e00640 1 -- 172.21.15.69:0/954266266 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d880319d0 con 0x7f0d9815b350 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d97e00640 1 -- 172.21.15.69:0/954266266 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88031d60 con 0x7f0d9815b350 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 -- 172.21.15.69:0/954266266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 msgr2=0x7f0d980abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/954266266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0d98072730 tx=0x7f0d8802b910 comp rx=0 tx=0).stop 2024-01-16T14:59:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 -- 172.21.15.69:0/954266266 shutdown_connections 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/954266266 >> v2:172.21.15.69:3300/0 conn(0x7f0d980adb60 0x7f0d980adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/954266266 >> v2:172.21.15.43:3300/0 conn(0x7f0d9809cb30 0x7f0d980b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/954266266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d980abe80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 -- 172.21.15.69:0/954266266 >> 172.21.15.69:0/954266266 conn(0x7f0d980b0ba0 msgr2=0x7f0d980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 -- 172.21.15.69:0/954266266 shutdown_connections 2024-01-16T14:59:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.100+0000 7f0d9fe00640 1 -- 172.21.15.69:0/954266266 wait complete. 2024-01-16T14:59:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.102+0000 7f0d9fe00640 1 Processor -- start 2024-01-16T14:59:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.102+0000 7f0d9fe00640 1 -- start start 2024-01-16T14:59:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9809cb30 0x7f0d981e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980adb60 0x7f0d981e3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981e8820 con 0x7f0d9809cb30 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981e3aa0 con 0x7f0d980adb60 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981e3cd0 con 0x7f0d9815b350 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980adb60 0x7f0d981e3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:34720/0 (socket says 172.21.15.69:34720) 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9de00640 1 -- 172.21.15.69:0/3345590746 learned_addr learned my addr 172.21.15.69:0/3345590746 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-16T14:59:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.103+0000 7f0d9d400640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9809cb30 0x7f0d981e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 -- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980adb60 msgr2=0x7f0d981e3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980adb60 0x7f0d981e3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 -- 172.21.15.69:0/3345590746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9809cb30 msgr2=0x7f0d981e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9809cb30 0x7f0d981e82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 -- 172.21.15.69:0/3345590746 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880094d0 con 0x7f0d9815b350 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9de00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000efc0 tx=0x7f0d9000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90014480 con 0x7f0d9815b350 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9000f040 con 0x7f0d9815b350 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90014820 con 0x7f0d9815b350 2024-01-16T14:59:11.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d981e3fb0 con 0x7f0d9815b350 2024-01-16T14:59:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.104+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d981e4570 con 0x7f0d9815b350 2024-01-16T14:59:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.105+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0d90014c80 con 0x7f0d9815b350 2024-01-16T14:59:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.105+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f0d900133e0 con 0x7f0d9815b350 2024-01-16T14:59:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.105+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9809d200 con 0x7f0d9815b350 2024-01-16T14:59:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.108+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0d90018020 con 0x7f0d9815b350 2024-01-16T14:59:11.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.186+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1a8a3071-bb4a-4fca-a170-8dad620d0dd9", "id": 7} v 0) v1 -- 0x7f0d981e4140 con 0x7f0d9815b350 2024-01-16T14:59:11.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.197+0000 7f0d95600640 1 -- 172.21.15.69:0/3345590746 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a8a3071-bb4a-4fca-a170-8dad620d0dd9", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0d9001d400 con 0x7f0d9815b350 2024-01-16T14:59:11.179 INFO:teuthology.orchestra.run.smithi069.stdout:7 2024-01-16T14:59:11.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 msgr2=0x7f0d9809acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000efc0 tx=0x7f0d9000c490 comp rx=0 tx=0).stop 2024-01-16T14:59:11.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 shutdown_connections 2024-01-16T14:59:11.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980adb60 0x7f0d981e3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d9815b350 0x7f0d9809acc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 --2- 172.21.15.69:0/3345590746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9809cb30 0x7f0d981e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.199+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 >> 172.21.15.69:0/3345590746 conn(0x7f0d980b0ba0 msgr2=0x7f0d980b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:11.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.200+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 shutdown_connections 2024-01-16T14:59:11.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:11.200+0000 7f0d9fe00640 1 -- 172.21.15.69:0/3345590746 wait complete. 2024-01-16T14:59:11.197 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T14:59:11.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T14:59:11.203 INFO:tasks.ceph.osd.0:Started 2024-01-16T14:59:11.203 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T14:59:11.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T14:59:11.209 INFO:tasks.ceph.osd.1:Started 2024-01-16T14:59:11.209 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T14:59:11.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T14:59:11.215 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:11.233+0000 7fa23a63a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:59:11.215 INFO:tasks.ceph.osd.2:Started 2024-01-16T14:59:11.215 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-16T14:59:11.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-16T14:59:11.220 INFO:tasks.ceph.osd.3:Started 2024-01-16T14:59:11.220 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-16T14:59:11.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-16T14:59:11.226 INFO:tasks.ceph.osd.4:Started 2024-01-16T14:59:11.226 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-16T14:59:11.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-16T14:59:11.232 INFO:tasks.ceph.osd.5:Started 2024-01-16T14:59:11.232 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-16T14:59:11.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-16T14:59:11.238 INFO:tasks.ceph.osd.6:Started 2024-01-16T14:59:11.239 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-16T14:59:11.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-16T14:59:11.243 INFO:tasks.ceph.osd.7:Started 2024-01-16T14:59:11.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.491+0000 7fcde2a00640 1 Processor -- start 2024-01-16T14:59:11.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.491+0000 7fcde2a00640 1 -- start start 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.492+0000 7fcde2a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcddc15a100 0x7fcddc0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.492+0000 7fcde2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcddc0ae7e0 0x7fcddc0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.492+0000 7fcde2a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc13da30 con 0x7fcddc0abff0 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.492+0000 7fcde2a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0ac840 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.492+0000 7fcde2a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0ac560 con 0x7fcddc15a100 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcdd3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcddc0abff0 0x7fcddc06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48432/0 (socket says 172.21.15.43:48432) 2024-01-16T14:59:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcdd3e00640 1 -- 172.21.15.43:0/635698446 learned_addr learned my addr 172.21.15.43:0/635698446 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:11.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcde0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcddc0ae7e0 0x7fcddc0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcddbe00640 1 --2- 172.21.15.43:0/635698446 >> v2:172.21.15.69:3300/0 conn(0x7fcddc15a100 0x7fcddc0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcde0a00640 1 -- 172.21.15.43:0/635698446 >> v2:172.21.15.69:3300/0 conn(0x7fcddc15a100 msgr2=0x7fcddc0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:11.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcde0a00640 1 --2- 172.21.15.43:0/635698446 >> v2:172.21.15.69:3300/0 conn(0x7fcddc15a100 0x7fcddc0b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcde0a00640 1 -- 172.21.15.43:0/635698446 >> v1:172.21.15.43:6789/0 conn(0x7fcddc0abff0 legacy=0x7fcddc06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcde0a00640 1 -- 172.21.15.43:0/635698446 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc070650 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.493+0000 7fcddb400640 1 -- 172.21.15.43:0/635698446 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242245831 0 0) 0x7fcddc13da30 con 0x7fcddc0abff0 2024-01-16T14:59:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.495+0000 7fcde0a00640 1 --2- 172.21.15.43:0/635698446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0ae7e0 0x7fcddc0b37d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcddc06bc90 tx=0x7fcdd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a868ffb46936019d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.495+0000 7fcddb400640 1 -- 172.21.15.43:0/635698446 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd402cc80 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.495+0000 7fcddb400640 1 -- 172.21.15.43:0/635698446 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd402cde0 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.495+0000 7fcddb400640 1 -- 172.21.15.43:0/635698446 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4034660 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 -- 172.21.15.43:0/635698446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0ae7e0 msgr2=0x7fcddc0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 --2- 172.21.15.43:0/635698446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0ae7e0 0x7fcddc0b37d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcddc06bc90 tx=0x7fcdd402bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 -- 172.21.15.43:0/635698446 shutdown_connections 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 --2- 172.21.15.43:0/635698446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0ae7e0 0x7fcddc0b37d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 --2- 172.21.15.43:0/635698446 >> v2:172.21.15.69:3300/0 conn(0x7fcddc15a100 0x7fcddc0b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 -- 172.21.15.43:0/635698446 >> 172.21.15.43:0/635698446 conn(0x7fcddc0b0d80 msgr2=0x7fcddc0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 -- 172.21.15.43:0/635698446 shutdown_connections 2024-01-16T14:59:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.496+0000 7fcde2a00640 1 -- 172.21.15.43:0/635698446 wait complete. 2024-01-16T14:59:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.498+0000 7fcde2a00640 1 Processor -- start 2024-01-16T14:59:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.498+0000 7fcde2a00640 1 -- start start 2024-01-16T14:59:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 0x7fcddc15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcddc15a100 0x7fcddc0afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc14f590 con 0x7fcddc0abff0 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc15a660 con 0x7fcddc15a100 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde2a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc15a8d0 con 0x7fcddc0ae7e0 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcddbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcddbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44520/0 (socket says 172.21.15.43:44520) 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcddbe00640 1 -- 172.21.15.43:0/4244834186 learned_addr learned my addr 172.21.15.43:0/4244834186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcde0a00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcddc15a100 0x7fcddc0afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.499+0000 7fcdd3e00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 0x7fcddc15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 -- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 msgr2=0x7fcddc15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 0x7fcddc15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 -- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcddc15a100 msgr2=0x7fcddc0afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcddc15a100 0x7fcddc0afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 -- 172.21.15.43:0/4244834186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4009a70 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcdd3e00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 0x7fcddc15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcddbe00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00e9b0 tx=0x7fcdcc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc009670 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc00c9a0 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00cd50 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.500+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc15abb0 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.501+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc15b170 con 0x7fcddc0abff0 2024-01-16T14:59:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.501+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcdcc005c90 con 0x7fcddc0abff0 2024-01-16T14:59:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.501+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fcdcc013470 con 0x7fcddc0abff0 2024-01-16T14:59:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.502+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc067960 con 0x7fcddc0abff0 2024-01-16T14:59:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.504+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcdcc021060 con 0x7fcddc0abff0 2024-01-16T14:59:11.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.568+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcddc0b1f30 con 0x7fcddc0abff0 2024-01-16T14:59:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.569+0000 7fcdd8c00640 1 -- 172.21.15.43:0/4244834186 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fcdcc029020 con 0x7fcddc0abff0 2024-01-16T14:59:11.551 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:11.551 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":10,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:11.188841+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 msgr2=0x7fcddc0986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00e9b0 tx=0x7fcdcc00ee70 comp rx=0 tx=0).stop 2024-01-16T14:59:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 shutdown_connections 2024-01-16T14:59:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcddc15a100 0x7fcddc0afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcddc0ae7e0 0x7fcddc15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 --2- 172.21.15.43:0/4244834186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcddc0abff0 0x7fcddc0986b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:11.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.570+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 >> 172.21.15.43:0/4244834186 conn(0x7fcddc0b0d80 msgr2=0x7fcddc0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:11.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.571+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 shutdown_connections 2024-01-16T14:59:11.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:11.571+0000 7fcde2a00640 1 -- 172.21.15.43:0/4244834186 wait complete. 2024-01-16T14:59:11.567 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T14:59:11.567 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T14:59:11.908 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-01-16T14:59:11.926+0000 7f87a880e500 -1 Falling back to public interface 2024-01-16T14:59:11.915 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-01-16T14:59:11.934+0000 7f950880e500 -1 Falling back to public interface 2024-01-16T14:59:11.917 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-01-16T14:59:11.936+0000 7f28bf61d500 -1 Falling back to public interface 2024-01-16T14:59:11.930 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-01-16T14:59:11.949+0000 7fec114fa500 -1 Falling back to public interface 2024-01-16T14:59:11.934 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-01-16T14:59:11.953+0000 7fdb83ecd500 -1 Falling back to public interface 2024-01-16T14:59:11.935 INFO:tasks.ceph.osd.3.smithi043.stderr:2024-01-16T14:59:11.954+0000 7f1f8549c500 -1 Falling back to public interface 2024-01-16T14:59:11.945 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-01-16T14:59:11.963+0000 7f180eef6500 -1 Falling back to public interface 2024-01-16T14:59:11.948 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-01-16T14:59:11.967+0000 7f7fcbe12500 -1 Falling back to public interface 2024-01-16T14:59:12.236 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:12.254+0000 7fd56e63b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:59:12.280 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:12.299+0000 7fa23a63a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:59:12.373 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:12.529 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.548+0000 7f469ba00640 1 Processor -- start 2024-01-16T14:59:12.529 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.548+0000 7f469ba00640 1 -- start start 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f469ba00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46940af410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f469ba00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4694071e50 con 0x7f469415d260 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f469ba00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4694161a00 con 0x7f469415d640 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f469ba00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46940b3b80 con 0x7f46940b22a0 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f4699000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f469415d260 0x7f46940b1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48434/0 (socket says 172.21.15.43:48434) 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.550+0000 7f4699000640 1 -- 172.21.15.43:0/2868872343 learned_addr learned my addr 172.21.15.43:0/2868872343 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:12.532 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4699a00640 1 --2- 172.21.15.43:0/2868872343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46940af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226319792 0 0) 0x7f4694071e50 con 0x7f469415d260 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4674003520 con 0x7f469415d260 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800184009 0 0) 0x7f4694161a00 con 0x7f469415d640 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694071e50 con 0x7f469415d640 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4699a00640 1 -- 172.21.15.43:0/2868872343 >> v1:172.21.15.69:6789/0 conn(0x7f469415d640 legacy=0x7f469406e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4699a00640 1 -- 172.21.15.43:0/2868872343 >> v1:172.21.15.43:6789/0 conn(0x7f469415d260 legacy=0x7f46940b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.533 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.551+0000 7f4699a00640 1 -- 172.21.15.43:0/2868872343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940af950 con 0x7f46940b22a0 2024-01-16T14:59:12.534 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.552+0000 7f4699a00640 1 --2- 172.21.15.43:0/2868872343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46940af410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f468c009dc0 tx=0x7f468c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fedbec1335640fb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.552+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c02ccd0 con 0x7f46940b22a0 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.552+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c02ce30 con 0x7f46940b22a0 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.552+0000 7f4693400640 1 -- 172.21.15.43:0/2868872343 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c034660 con 0x7f46940b22a0 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 -- 172.21.15.43:0/2868872343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 msgr2=0x7f46940af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 --2- 172.21.15.43:0/2868872343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46940af410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f468c009dc0 tx=0x7f468c02bf20 comp rx=0 tx=0).stop 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 -- 172.21.15.43:0/2868872343 shutdown_connections 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 --2- 172.21.15.43:0/2868872343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46940af410 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 -- 172.21.15.43:0/2868872343 >> 172.21.15.43:0/2868872343 conn(0x7f469406d560 msgr2=0x7f469406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 -- 172.21.15.43:0/2868872343 shutdown_connections 2024-01-16T14:59:12.535 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.553+0000 7f469ba00640 1 -- 172.21.15.43:0/2868872343 wait complete. 2024-01-16T14:59:12.536 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.555+0000 7f469ba00640 1 Processor -- start 2024-01-16T14:59:12.536 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.555+0000 7f469ba00640 1 -- start start 2024-01-16T14:59:12.537 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46941592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:12.537 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f469415d260 0x7f4694142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:12.537 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:12.537 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4694159a00 con 0x7f46941563c0 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469415eb20 con 0x7f469415d260 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f469ba00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469415ee00 con 0x7f46940b22a0 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f4699a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f4699000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46941592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f4699a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44536/0 (socket says 172.21.15.43:44536) 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f4699a00640 1 -- 172.21.15.43:0/3773722196 learned_addr learned my addr 172.21.15.43:0/3773722196 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:12.538 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.556+0000 7f4693e00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f469415d260 0x7f4694142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:12.539 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 -- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 msgr2=0x7f46941592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.539 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46941592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 -- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f469415d260 msgr2=0x7f4694142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f469415d260 0x7f4694142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 -- 172.21.15.43:0/3773722196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c009a70 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699000640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46941592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4699a00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f468c02dfd0 tx=0x7f468c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c03d600 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c03d760 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c03db10 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469415f0e0 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.557+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4694142a80 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.558+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f468c03b040 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.558+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f468c004790 con 0x7f46941563c0 2024-01-16T14:59:12.540 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.559+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469415f5a0 con 0x7f46941563c0 2024-01-16T14:59:12.542 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.561+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f468c044070 con 0x7f46941563c0 2024-01-16T14:59:12.560 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:12.579+0000 7fd56e63b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:59:12.584 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:12.603+0000 7fa23a63a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:59:12.609 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.628+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f469404e840 con 0x7f46941563c0 2024-01-16T14:59:12.610 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.628+0000 7f4690c00640 1 -- 172.21.15.43:0/3773722196 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f468c039070 con 0x7f46941563c0 2024-01-16T14:59:12.610 INFO:teuthology.misc.health.smithi043.stdout: 2024-01-16T14:59:12.610 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":10,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:11.188841+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 msgr2=0x7f469415f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f468c02dfd0 tx=0x7f468c004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 shutdown_connections 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f469415d260 0x7f4694142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46941563c0 0x7f469415f8e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 --2- 172.21.15.43:0/3773722196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46940b22a0 0x7f46941592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 >> 172.21.15.43:0/3773722196 conn(0x7f469406d560 msgr2=0x7f469406acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:12.611 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 shutdown_connections 2024-01-16T14:59:12.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:12.630+0000 7f469ba00640 1 -- 172.21.15.43:0/3773722196 wait complete. 2024-01-16T14:59:12.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T14:59:12.624 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:12.642+0000 7fd56e63b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:59:12.643 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:12.661+0000 7fa23a63a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:59:12.743 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:12.761+0000 7fd56e63b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:59:12.760 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-01-16T14:59:12.778+0000 7fa23a63a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:59:13.788 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:13.807+0000 7fd50f800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-16T14:59:13.790 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T14:59:13.809+0000 7fd50f800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-16T14:59:15.452 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-01-16T14:59:15.470+0000 7f950880e500 -1 osd.5 0 log_to_monitors true 2024-01-16T14:59:15.495 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-01-16T14:59:15.514+0000 7fdb83ecd500 -1 osd.4 0 log_to_monitors true 2024-01-16T14:59:15.508 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-01-16T14:59:15.527+0000 7fec114fa500 -1 osd.2 0 log_to_monitors true 2024-01-16T14:59:15.535 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-01-16T14:59:15.554+0000 7f180eef6500 -1 osd.1 0 log_to_monitors true 2024-01-16T14:59:15.672 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-01-16T14:59:15.691+0000 7f87a880e500 -1 osd.6 0 log_to_monitors true 2024-01-16T14:59:15.694 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-01-16T14:59:15.713+0000 7f28bf61d500 -1 osd.0 0 log_to_monitors true 2024-01-16T14:59:15.792 INFO:tasks.ceph.osd.3.smithi043.stderr:2024-01-16T14:59:15.811+0000 7f1f8549c500 -1 osd.3 0 log_to_monitors true 2024-01-16T14:59:15.799 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-01-16T14:59:15.818+0000 7f7fcbe12500 -1 osd.7 0 log_to_monitors true 2024-01-16T14:59:18.340 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-01-16T14:59:18.356+0000 7f28b9400640 -1 osd.0 0 waiting for initial osdmap 2024-01-16T14:59:18.342 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-01-16T14:59:18.361+0000 7f28afe00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:18.365 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-01-16T14:59:18.384+0000 7f1809400640 -1 osd.1 0 waiting for initial osdmap 2024-01-16T14:59:18.378 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-01-16T14:59:18.397+0000 7f17ffe00640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:18.378 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-01-16T14:59:18.397+0000 7fec0d800640 -1 osd.2 0 waiting for initial osdmap 2024-01-16T14:59:18.388 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-01-16T14:59:18.407+0000 7fec02400640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:18.751 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-01-16T14:59:18.769+0000 7fdb80200640 -1 osd.4 0 waiting for initial osdmap 2024-01-16T14:59:18.759 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-01-16T14:59:18.778+0000 7fdb74e00640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:18.842 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-01-16T14:59:18.861+0000 7f9502600640 -1 osd.5 0 waiting for initial osdmap 2024-01-16T14:59:18.849 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-01-16T14:59:18.868+0000 7f94f9000640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:18.875 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-01-16T14:59:18.893+0000 7f87a2c00640 -1 osd.6 0 waiting for initial osdmap 2024-01-16T14:59:18.880 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-01-16T14:59:18.899+0000 7f8799600640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:19.030 INFO:tasks.ceph.osd.3.smithi043.stderr:2024-01-16T14:59:19.049+0000 7f1f7fa00640 -1 osd.3 0 waiting for initial osdmap 2024-01-16T14:59:19.041 INFO:tasks.ceph.osd.3.smithi043.stderr:2024-01-16T14:59:19.060+0000 7f1f76400640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:19.202 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-01-16T14:59:19.221+0000 7f7fc5e00640 -1 osd.7 0 waiting for initial osdmap 2024-01-16T14:59:19.212 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-01-16T14:59:19.231+0000 7f7fbc800640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:19.430 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:19.610 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.629+0000 7f1122a00640 1 Processor -- start 2024-01-16T14:59:19.610 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.629+0000 7f1122a00640 1 -- start start 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.630+0000 7f1122a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f111c161770 0x7f111c071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.630+0000 7f1122a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f111c06af70 0x7f111c0abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1122a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c070090 con 0x7f111c071c70 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1122a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c070370 con 0x7f111c161770 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1122a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c070650 con 0x7f111c06af70 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1121600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f111c161770 0x7f111c071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1120c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f111c071c70 0x7f111c06a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48480/0 (socket says 172.21.15.43:48480) 2024-01-16T14:59:19.612 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1120c00640 1 -- 172.21.15.43:0/70990662 learned_addr learned my addr 172.21.15.43:0/70990662 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1122000640 1 --2- 172.21.15.43:0/70990662 >> v2:172.21.15.69:3300/0 conn(0x7f111c06af70 0x7f111c0abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1121600640 1 -- 172.21.15.43:0/70990662 >> v2:172.21.15.69:3300/0 conn(0x7f111c06af70 msgr2=0x7f111c0abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1121600640 1 --2- 172.21.15.43:0/70990662 >> v2:172.21.15.69:3300/0 conn(0x7f111c06af70 0x7f111c0abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f111be00640 1 -- 172.21.15.43:0/70990662 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773489718 0 0) 0x7f111c070090 con 0x7f111c071c70 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1121600640 1 -- 172.21.15.43:0/70990662 >> v1:172.21.15.43:6789/0 conn(0x7f111c071c70 legacy=0x7f111c06a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.613 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.631+0000 7f1121600640 1 -- 172.21.15.43:0/70990662 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c070930 con 0x7f111c161770 2024-01-16T14:59:19.614 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f1121600640 1 --2- 172.21.15.43:0/70990662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c161770 0x7f111c071410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f111c06e550 tx=0x7f110c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6288fc60979cea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:19.614 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f111be00640 1 -- 172.21.15.43:0/70990662 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c031830 con 0x7f111c161770 2024-01-16T14:59:19.614 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f111be00640 1 -- 172.21.15.43:0/70990662 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c031990 con 0x7f111c161770 2024-01-16T14:59:19.614 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f111be00640 1 -- 172.21.15.43:0/70990662 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c031d20 con 0x7f111c161770 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f1122a00640 1 -- 172.21.15.43:0/70990662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c161770 msgr2=0x7f111c071410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.633+0000 7f1122a00640 1 --2- 172.21.15.43:0/70990662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c161770 0x7f111c071410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f111c06e550 tx=0x7f110c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 -- 172.21.15.43:0/70990662 shutdown_connections 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 --2- 172.21.15.43:0/70990662 >> v2:172.21.15.69:3300/0 conn(0x7f111c06af70 0x7f111c0abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 --2- 172.21.15.43:0/70990662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c161770 0x7f111c071410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 -- 172.21.15.43:0/70990662 >> 172.21.15.43:0/70990662 conn(0x7f111c0b0d00 msgr2=0x7f111c0b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 -- 172.21.15.43:0/70990662 shutdown_connections 2024-01-16T14:59:19.615 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.634+0000 7f1122a00640 1 -- 172.21.15.43:0/70990662 wait complete. 2024-01-16T14:59:19.616 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.635+0000 7f1122a00640 1 Processor -- start 2024-01-16T14:59:19.617 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.636+0000 7f1122a00640 1 -- start start 2024-01-16T14:59:19.619 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111c06af70 0x7f111c154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.619 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.619 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111c1638f0 0x7f111c1f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.619 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c09c5a0 con 0x7f111c071c70 2024-01-16T14:59:19.619 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c1568f0 con 0x7f111c06af70 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1122a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c156bd0 con 0x7f111c1638f0 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40430/0 (socket says 172.21.15.43:40430) 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 -- 172.21.15.43:0/1121282955 learned_addr learned my addr 172.21.15.43:0/1121282955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 -- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111c1638f0 msgr2=0x7f111c1f89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111c1638f0 0x7f111c1f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1121600640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111c06af70 0x7f111c154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 -- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111c06af70 msgr2=0x7f111c154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111c06af70 0x7f111c154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.637+0000 7f1120c00640 1 -- 172.21.15.43:0/1121282955 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c009400 con 0x7f111c071c70 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1120c00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f110400c4d0 tx=0x7f110400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:19.620 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11040074e0 con 0x7f111c071c70 2024-01-16T14:59:19.621 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1104007660 con 0x7f111c071c70 2024-01-16T14:59:19.621 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104004a20 con 0x7f111c071c70 2024-01-16T14:59:19.621 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c156eb0 con 0x7f111c071c70 2024-01-16T14:59:19.621 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.638+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c09c880 con 0x7f111c071c70 2024-01-16T14:59:19.621 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.639+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c156750 con 0x7f111c071c70 2024-01-16T14:59:19.622 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.640+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f1104004b80 con 0x7f111c071c70 2024-01-16T14:59:19.622 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.641+0000 7f1119600640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 0x7f10f406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:19.622 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.641+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f1104088850 con 0x7f111c071c70 2024-01-16T14:59:19.622 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.641+0000 7f1121600640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 0x7f10f406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:19.623 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.642+0000 7f1121600640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 0x7f10f406d720 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f110c036000 tx=0x7f110c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:19.627 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.646+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f111c156750 con 0x7f111c071c70 2024-01-16T14:59:19.827 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.846+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f111c156750 con 0x7f111c071c70 2024-01-16T14:59:19.830 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.849+0000 7f1119600640 1 -- 172.21.15.43:0/1121282955 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10860 (secure 0 0 0) 0x7f111c156750 con 0x7f111c071c70 2024-01-16T14:59:19.831 INFO:teuthology.misc.health.smithi043.stdout: 2024-01-16T14:59:19.831 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":14,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:19.352588+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:19.834 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 msgr2=0x7f10f406d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.834 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 0x7f10f406d720 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f110c036000 tx=0x7f110c03c040 comp rx=0 tx=0).stop 2024-01-16T14:59:19.834 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 msgr2=0x7f111c09bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:19.834 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f110400c4d0 tx=0x7f110400c990 comp rx=0 tx=0).stop 2024-01-16T14:59:19.834 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 shutdown_connections 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f406b260 0x7f10f406d720 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111c1638f0 0x7f111c1f89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111c071c70 0x7f111c09bf10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 --2- 172.21.15.43:0/1121282955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111c06af70 0x7f111c154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 >> 172.21.15.43:0/1121282955 conn(0x7f111c0b0d00 msgr2=0x7f111c0afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 shutdown_connections 2024-01-16T14:59:19.835 INFO:teuthology.misc.health.smithi043.stderr:2024-01-16T14:59:19.853+0000 7f1122a00640 1 -- 172.21.15.43:0/1121282955 wait complete. 2024-01-16T14:59:19.848 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-16T14:59:19.848 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T14:59:19.849 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:59:19.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:59:19.856 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:59:19.856 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:59:19.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:59:19.864 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:59:19.864 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:59:19.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:59:19.872 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:59:19.872 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:59:19.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:59:19.879 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:59:19.879 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-16T14:59:19.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:20.012 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T14:59:20.014 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T14:59:20.034 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T14:59:20.035 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T14:59:20.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.084+0000 7fa33f600640 1 Processor -- start 2024-01-16T14:59:20.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.084+0000 7fa33f600640 1 -- start start 2024-01-16T14:59:20.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33f600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3380ad720 0x7fa3380b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33f600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33815c360 con 0x7fa33806d520 2024-01-16T14:59:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33f600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33815ba30 con 0x7fa3380abff0 2024-01-16T14:59:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33f600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3380b41a0 con 0x7fa3380ad720 2024-01-16T14:59:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33cc00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa33806d520 0x7fa33806d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48498/0 (socket says 172.21.15.43:48498) 2024-01-16T14:59:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.085+0000 7fa33cc00640 1 -- 172.21.15.43:0/1108032287 learned_addr learned my addr 172.21.15.43:0/1108032287 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290335659 0 0) 0x7fa33815c360 con 0x7fa33806d520 2024-01-16T14:59:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318003520 con 0x7fa33806d520 2024-01-16T14:59:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337e00640 1 --2- 172.21.15.43:0/1108032287 >> v2:172.21.15.69:3300/0 conn(0x7fa3380ad720 0x7fa3380b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023280260 0 0) 0x7fa318003520 con 0x7fa33806d520 2024-01-16T14:59:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 >> v2:172.21.15.69:3300/0 conn(0x7fa3380ad720 msgr2=0x7fa3380b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 --2- 172.21.15.43:0/1108032287 >> v2:172.21.15.69:3300/0 conn(0x7fa3380ad720 0x7fa3380b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 >> v1:172.21.15.69:6789/0 conn(0x7fa3380abff0 legacy=0x7fa3380b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33815bd10 con 0x7fa33806d520 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa328001670 con 0x7fa33806d520 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 >> v1:172.21.15.43:6789/0 conn(0x7fa33806d520 legacy=0x7fa33806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa318003970 0x7fa318003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa318004370 0x7fa318004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa31800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31800bf70 con 0x7fa318003970 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31800c0e0 con 0x7fa318005040 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.086+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31800c2a0 con 0x7fa318004370 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa31800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa337e00640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa318004370 0x7fa318004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33cc00640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa318003970 0x7fa318003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 -- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa318004370 msgr2=0x7fa318004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa318004370 0x7fa318004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 -- 172.21.15.43:0/1108032287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa318003970 msgr2=0x7fa318003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa318003970 0x7fa318003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.087+0000 7fa33d600640 1 -- 172.21.15.43:0/1108032287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33815bd10 con 0x7fa318005040 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.088+0000 7fa33d600640 1 -- 172.21.15.43:0/1108032287 reap_dead start 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.088+0000 7fa33d600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa31800b980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa330001f60 tx=0x7fa3180057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f5fa63b03f052d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.088+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330003b70 con 0x7fa318005040 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.088+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330003cf0 con 0x7fa318005040 2024-01-16T14:59:20.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.088+0000 7fa337400640 1 -- 172.21.15.43:0/1108032287 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3300052b0 con 0x7fa318005040 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 -- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 msgr2=0x7fa31800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa31800b980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa330001f60 tx=0x7fa3180057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 -- 172.21.15.43:0/1108032287 shutdown_connections 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 --2- 172.21.15.43:0/1108032287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa31800b980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 -- 172.21.15.43:0/1108032287 >> 172.21.15.43:0/1108032287 conn(0x7fa3380b0d80 msgr2=0x7fa3380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 -- 172.21.15.43:0/1108032287 shutdown_connections 2024-01-16T14:59:20.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.089+0000 7fa33f600640 1 -- 172.21.15.43:0/1108032287 wait complete. 2024-01-16T14:59:20.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.091+0000 7fa33f600640 1 Processor -- start 2024-01-16T14:59:20.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.091+0000 7fa33f600640 1 -- start start 2024-01-16T14:59:20.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa3380c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33815a870 0x7fa3381f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380c8400 con 0x7fa3380ca100 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33815acf0 con 0x7fa318005040 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33f600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3381f3460 con 0x7fa33815a870 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa337e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa337e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40468/0 (socket says 172.21.15.43:40468) 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa337e00640 1 -- 172.21.15.43:0/3398400910 learned_addr learned my addr 172.21.15.43:0/3398400910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33d600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33815a870 0x7fa3381f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.092+0000 7fa33cc00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa3380c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 -- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33815a870 msgr2=0x7fa3381f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33815a870 0x7fa3381f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 -- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 msgr2=0x7fa3380c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa3380c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 -- 172.21.15.43:0/3398400910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3381f3740 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa337e00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa320011930 tx=0x7fa320011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32000f9b0 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32000fb10 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32000fea0 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3381f3a20 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.093+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3381f2cb0 con 0x7fa3380ca100 2024-01-16T14:59:20.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.095+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3380afdb0 con 0x7fa3380ca100 2024-01-16T14:59:20.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.096+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fa320014440 con 0x7fa3380ca100 2024-01-16T14:59:20.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.096+0000 7fa334c00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 0x7fa31006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.097+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fa320089cb0 con 0x7fa3380ca100 2024-01-16T14:59:20.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.097+0000 7fa33cc00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 0x7fa31006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.097+0000 7fa33cc00640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 0x7fa31006d720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa328007e90 tx=0x7fa328001680 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.101+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3200550e0 con 0x7fa3380ca100 2024-01-16T14:59:20.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.299+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa338074770 con 0x7fa3380ca100 2024-01-16T14:59:20.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.301+0000 7fa334c00640 1 -- 172.21.15.43:0/3398400910 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10860 (secure 0 0 0) 0x7fa320058700 con 0x7fa3380ca100 2024-01-16T14:59:20.282 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:20.282 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":14,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:19.352588+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 msgr2=0x7fa31006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 0x7fa31006d720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa328007e90 tx=0x7fa328001680 comp rx=0 tx=0).stop 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 msgr2=0x7fa33815a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa320011930 tx=0x7fa320011df0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 shutdown_connections 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31006b260 0x7fa31006d720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33815a870 0x7fa3381f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3380ca100 0x7fa33815a330 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 --2- 172.21.15.43:0/3398400910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa318005040 0x7fa3380c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 >> 172.21.15.43:0/3398400910 conn(0x7fa3380b0d80 msgr2=0x7fa33806d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 shutdown_connections 2024-01-16T14:59:20.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.304+0000 7fa33f600640 1 -- 172.21.15.43:0/3398400910 wait complete. 2024-01-16T14:59:20.303 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-01-16T14:59:20.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:20.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.543+0000 7fa2b3e00640 1 Processor -- start 2024-01-16T14:59:20.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.543+0000 7fa2b3e00640 1 -- start start 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.544+0000 7fa2b3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2b4144680 0x7fa2b406f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.544+0000 7fa2b3e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b406fc90 con 0x7fa2b406a440 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2b3e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b406e340 con 0x7fa2b415c960 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2b3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40cd3f0 con 0x7fa2b4144680 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2abe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa2b406a440 0x7fa2b406d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48500/0 (socket says 172.21.15.43:48500) 2024-01-16T14:59:20.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2abe00640 1 -- 172.21.15.43:0/958093733 learned_addr learned my addr 172.21.15.43:0/958093733 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:20.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/958093733 >> v2:172.21.15.69:3300/0 conn(0x7fa2b4144680 0x7fa2b406f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.545+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753731251 0 0) 0x7fa29c001110 con 0x7fa2b406a440 2024-01-16T14:59:20.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.546+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa294003520 con 0x7fa2b406a440 2024-01-16T14:59:20.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.546+0000 7fa2b2a00640 1 -- 172.21.15.43:0/958093733 >> v1:172.21.15.69:6789/0 conn(0x7fa2b415c960 legacy=0x7fa2b406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.546+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551173850 0 0) 0x7fa294003520 con 0x7fa2b406a440 2024-01-16T14:59:20.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.546+0000 7fa2b2a00640 1 -- 172.21.15.43:0/958093733 >> v1:172.21.15.43:6789/0 conn(0x7fa2b406a440 legacy=0x7fa2b406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.546+0000 7fa2b2a00640 1 -- 172.21.15.43:0/958093733 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40cd6d0 con 0x7fa2b4144680 2024-01-16T14:59:20.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.547+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/958093733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b4144680 0x7fa2b406f750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40097a0 tx=0x7fa2a402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de1ca1c6db130cd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a402f050 con 0x7fa2b4144680 2024-01-16T14:59:20.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4031500 con 0x7fa2b4144680 2024-01-16T14:59:20.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b2000640 1 -- 172.21.15.43:0/958093733 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c001800 con 0x7fa2b4144680 2024-01-16T14:59:20.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b3e00640 1 -- 172.21.15.43:0/958093733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b4144680 msgr2=0x7fa2b406f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/958093733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b4144680 0x7fa2b406f750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40097a0 tx=0x7fa2a402b820 comp rx=0 tx=0).stop 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b3e00640 1 -- 172.21.15.43:0/958093733 shutdown_connections 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/958093733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b4144680 0x7fa2b406f750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.548+0000 7fa2b3e00640 1 -- 172.21.15.43:0/958093733 >> 172.21.15.43:0/958093733 conn(0x7fa2b40b1c70 msgr2=0x7fa2b40b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.549+0000 7fa2b3e00640 1 -- 172.21.15.43:0/958093733 shutdown_connections 2024-01-16T14:59:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.549+0000 7fa2b3e00640 1 -- 172.21.15.43:0/958093733 wait complete. 2024-01-16T14:59:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.551+0000 7fa2b3e00640 1 Processor -- start 2024-01-16T14:59:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.551+0000 7fa2b3e00640 1 -- start start 2024-01-16T14:59:20.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.552+0000 7fa2b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 0x7fa2b413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.552+0000 7fa2b3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.552+0000 7fa2b3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 0x7fa2b41f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.552+0000 7fa2b3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b413cbd0 con 0x7fa2b4144680 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b413ceb0 con 0x7fa2b415c960 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b413d190 con 0x7fa2b406a440 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40476/0 (socket says 172.21.15.43:40476) 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 -- 172.21.15.43:0/655619239 learned_addr learned my addr 172.21.15.43:0/655619239 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:20.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 0x7fa2b413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 -- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 msgr2=0x7fa2b413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 0x7fa2b413c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b3400640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 0x7fa2b41f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 -- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 msgr2=0x7fa2b41f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 0x7fa2b41f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2abe00640 1 -- 172.21.15.43:0/655619239 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4009400 con 0x7fa2b4144680 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 0x7fa2b413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.553+0000 7fa2b3400640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 0x7fa2b41f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2abe00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c730 tx=0x7fa29c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c005ad0 con 0x7fa2b4144680 2024-01-16T14:59:20.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c005c30 con 0x7fa2b4144680 2024-01-16T14:59:20.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c007070 con 0x7fa2b4144680 2024-01-16T14:59:20.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b413d470 con 0x7fa2b4144680 2024-01-16T14:59:20.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.554+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b40c50a0 con 0x7fa2b4144680 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.555+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b40c5230 con 0x7fa2b4144680 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.556+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fa29c0071d0 con 0x7fa2b4144680 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.557+0000 7fa2aaa00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 0x7fa28c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.557+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fa29c088c50 con 0x7fa2b4144680 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.557+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 0x7fa28c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:20.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.558+0000 7fa2b2a00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 0x7fa28c06d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4035000 tx=0x7fa2a4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:20.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.562+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa29c053fd0 con 0x7fa2b4144680 2024-01-16T14:59:20.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.800+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b413d990 con 0x7fa2b4144680 2024-01-16T14:59:20.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.801+0000 7fa2aaa00640 1 -- 172.21.15.43:0/655619239 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fa2b413d990 con 0x7fa2b4144680 2024-01-16T14:59:20.783 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:20.783 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-16T14:59:20.783 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 2 2024-01-16T14:59:20.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 msgr2=0x7fa28c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 0x7fa28c06d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4035000 tx=0x7fa2a4008030 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 msgr2=0x7fa2b41f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c730 tx=0x7fa29c00cbf0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 shutdown_connections 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28c06b230 0x7fa28c06d6f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2b415c960 0x7fa2b41f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2b4144680 0x7fa2b41f43b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 --2- 172.21.15.43:0/655619239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2b406a440 0x7fa2b413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 >> 172.21.15.43:0/655619239 conn(0x7fa2b40b1c70 msgr2=0x7fa2b4143860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 shutdown_connections 2024-01-16T14:59:20.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:20.805+0000 7fa2b3e00640 1 -- 172.21.15.43:0/655619239 wait complete. 2024-01-16T14:59:22.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:23.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.030+0000 7f0c39e00640 1 Processor -- start 2024-01-16T14:59:23.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.031+0000 7f0c39e00640 1 -- start start 2024-01-16T14:59:23.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c3415d5b0 0x7f0c340d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c340b1650 0x7f0c340d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340cf0c0 con 0x7f0c340d14a0 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c340b01b0 con 0x7f0c340b1650 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3415d990 con 0x7f0c3415d5b0 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c33e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0c340d14a0 0x7f0c340cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35948/0 (socket says 172.21.15.43:35948) 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c33e00640 1 -- 172.21.15.43:0/2109667873 learned_addr learned my addr 172.21.15.43:0/2109667873 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c39400640 1 --2- 172.21.15.43:0/2109667873 >> v2:172.21.15.69:3300/0 conn(0x7f0c340b1650 0x7f0c340d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.032+0000 7f0c38a00640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c3415d5b0 0x7f0c340d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c33400640 1 -- 172.21.15.43:0/2109667873 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419459517 0 0) 0x7f0c340cf0c0 con 0x7f0c340d14a0 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c33400640 1 -- 172.21.15.43:0/2109667873 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c14003520 con 0x7f0c340d14a0 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c39400640 1 -- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c3415d5b0 msgr2=0x7f0c340d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c39400640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c3415d5b0 0x7f0c340d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c39400640 1 -- 172.21.15.43:0/2109667873 >> v1:172.21.15.43:6789/0 conn(0x7f0c340d14a0 legacy=0x7f0c340cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.033+0000 7f0c39400640 1 -- 172.21.15.43:0/2109667873 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340d1e40 con 0x7f0c340b1650 2024-01-16T14:59:23.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.034+0000 7f0c39400640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c340b1650 0x7f0c340d1900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a945dec0af319ff7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.035+0000 7f0c33400640 1 -- 172.21.15.43:0/2109667873 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c02cc60 con 0x7f0c340b1650 2024-01-16T14:59:23.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.035+0000 7f0c33400640 1 -- 172.21.15.43:0/2109667873 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c02cdc0 con 0x7f0c340b1650 2024-01-16T14:59:23.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.035+0000 7f0c33400640 1 -- 172.21.15.43:0/2109667873 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c034660 con 0x7f0c340b1650 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 -- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c340b1650 msgr2=0x7f0c340d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c340b1650 0x7f0c340d1900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c02beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 -- 172.21.15.43:0/2109667873 shutdown_connections 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c340b1650 0x7f0c340d1900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2109667873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c3415d5b0 0x7f0c340d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 -- 172.21.15.43:0/2109667873 >> 172.21.15.43:0/2109667873 conn(0x7f0c3409c3b0 msgr2=0x7f0c340d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 -- 172.21.15.43:0/2109667873 shutdown_connections 2024-01-16T14:59:23.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.036+0000 7f0c39e00640 1 -- 172.21.15.43:0/2109667873 wait complete. 2024-01-16T14:59:23.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.038+0000 7f0c39e00640 1 Processor -- start 2024-01-16T14:59:23.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.038+0000 7f0c39e00640 1 -- start start 2024-01-16T14:59:23.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c340d14a0 0x7f0c34160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c3415d5b0 0x7f0c340c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340b6e80 con 0x7f0c340b1650 2024-01-16T14:59:23.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340c87e0 con 0x7f0c3415d5b0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340b7090 con 0x7f0c340d14a0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58122/0 (socket says 172.21.15.43:58122) 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 -- 172.21.15.43:0/2843699847 learned_addr learned my addr 172.21.15.43:0/2843699847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c39400640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c3415d5b0 0x7f0c340c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c33e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c340d14a0 0x7f0c34160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 -- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c340d14a0 msgr2=0x7f0c34160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c340d14a0 0x7f0c34160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 -- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c3415d5b0 msgr2=0x7f0c340c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c3415d5b0 0x7f0c340c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.039+0000 7f0c38a00640 1 -- 172.21.15.43:0/2843699847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c009a70 con 0x7f0c340b1650 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c38a00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400e9b0 tx=0x7f0c2400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24009670 con 0x7f0c340b1650 2024-01-16T14:59:23.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2400c9a0 con 0x7f0c340b1650 2024-01-16T14:59:23.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c340b7370 con 0x7f0c340b1650 2024-01-16T14:59:23.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2400cd50 con 0x7f0c340b1650 2024-01-16T14:59:23.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.040+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c340b7930 con 0x7f0c340b1650 2024-01-16T14:59:23.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.042+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3404e840 con 0x7f0c340b1650 2024-01-16T14:59:23.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.042+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f0c24013470 con 0x7f0c340b1650 2024-01-16T14:59:23.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.043+0000 7f0c30c00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 0x7f0c0c06d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.043+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c24089a10 con 0x7f0c340b1650 2024-01-16T14:59:23.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.044+0000 7f0c33e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 0x7f0c0c06d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.044+0000 7f0c33e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 0x7f0c0c06d720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c341562a0 tx=0x7f0c1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.048+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c24054c10 con 0x7f0c340b1650 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.308+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c340678d0 con 0x7f0c340b1650 2024-01-16T14:59:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.309+0000 7f0c30c00640 1 -- 172.21.15.43:0/2843699847 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7f0c24021e80 con 0x7f0c340b1650 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-16T14:59:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 3 2024-01-16T14:59:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 msgr2=0x7f0c0c06d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 0x7f0c0c06d720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c341562a0 tx=0x7f0c1c009040 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 msgr2=0x7f0c340b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400e9b0 tx=0x7f0c2400ee70 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 shutdown_connections 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0c0c06b260 0x7f0c0c06d720 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c3415d5b0 0x7f0c340c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c340d14a0 0x7f0c34160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 --2- 172.21.15.43:0/2843699847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c340b1650 0x7f0c340b7e50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 >> 172.21.15.43:0/2843699847 conn(0x7f0c3409c3b0 msgr2=0x7f0c3406bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 shutdown_connections 2024-01-16T14:59:23.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.312+0000 7f0c39e00640 1 -- 172.21.15.43:0/2843699847 wait complete. 2024-01-16T14:59:23.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.538+0000 7f3380400640 1 Processor -- start 2024-01-16T14:59:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.538+0000 7f3380400640 1 -- start start 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.539+0000 7f3380400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f337815d4d0 0x7f33780cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.539+0000 7f3380400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f33780ce260 0x7f33780d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.539+0000 7f3380400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337815d8b0 con 0x7f33780ce6c0 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.539+0000 7f3380400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33780b91b0 con 0x7f33780ce260 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.539+0000 7f3380400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33780cdfb0 con 0x7f337815d4d0 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f33780ce260 0x7f33780d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337e400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f33780ce6c0 0x7f33780b4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35954/0 (socket says 172.21.15.43:35954) 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337e400640 1 -- 172.21.15.43:0/2054446011 learned_addr learned my addr 172.21.15.43:0/2054446011 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337da00640 1 --2- 172.21.15.43:0/2054446011 >> v2:172.21.15.69:3300/0 conn(0x7f337815d4d0 0x7f33780cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337d000640 1 -- 172.21.15.43:0/2054446011 >> v2:172.21.15.69:3300/0 conn(0x7f337815d4d0 msgr2=0x7f33780cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337d000640 1 --2- 172.21.15.43:0/2054446011 >> v2:172.21.15.69:3300/0 conn(0x7f337815d4d0 0x7f33780cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337d000640 1 -- 172.21.15.43:0/2054446011 >> v1:172.21.15.43:6789/0 conn(0x7f33780ce6c0 legacy=0x7f33780b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f337d000640 1 -- 172.21.15.43:0/2054446011 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33780b2400 con 0x7f33780ce260 2024-01-16T14:59:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.540+0000 7f3377e00640 1 -- 172.21.15.43:0/2054446011 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891874278 0 0) 0x7f337815d8b0 con 0x7f33780ce6c0 2024-01-16T14:59:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.541+0000 7f337d000640 1 --2- 172.21.15.43:0/2054446011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33780ce260 0x7f33780d1850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f337806c470 tx=0x7f336002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e9f4b175da5bb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3377e00640 1 -- 172.21.15.43:0/2054446011 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360031830 con 0x7f33780ce260 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3377e00640 1 -- 172.21.15.43:0/2054446011 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3360031990 con 0x7f33780ce260 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3377e00640 1 -- 172.21.15.43:0/2054446011 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360031d20 con 0x7f33780ce260 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 -- 172.21.15.43:0/2054446011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33780ce260 msgr2=0x7f33780d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 --2- 172.21.15.43:0/2054446011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33780ce260 0x7f33780d1850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f337806c470 tx=0x7f336002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 -- 172.21.15.43:0/2054446011 shutdown_connections 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 --2- 172.21.15.43:0/2054446011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33780ce260 0x7f33780d1850 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 --2- 172.21.15.43:0/2054446011 >> v2:172.21.15.69:3300/0 conn(0x7f337815d4d0 0x7f33780cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.542+0000 7f3380400640 1 -- 172.21.15.43:0/2054446011 >> 172.21.15.43:0/2054446011 conn(0x7f3378071f90 msgr2=0x7f33780723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.543+0000 7f3380400640 1 -- 172.21.15.43:0/2054446011 shutdown_connections 2024-01-16T14:59:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.543+0000 7f3380400640 1 -- 172.21.15.43:0/2054446011 wait complete. 2024-01-16T14:59:23.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.544+0000 7f3380400640 1 Processor -- start 2024-01-16T14:59:23.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.544+0000 7f3380400640 1 -- start start 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33780ce260 0x7f337814e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33780ce6c0 0x7f33780c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337814ebc0 con 0x7f337815d4d0 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33780c70f0 con 0x7f33780ce6c0 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.545+0000 7f3380400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33780c73d0 con 0x7f33780ce260 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58136/0 (socket says 172.21.15.43:58136) 2024-01-16T14:59:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 -- 172.21.15.43:0/3880803207 learned_addr learned my addr 172.21.15.43:0/3880803207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337da00640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33780ce260 0x7f337814e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337d000640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33780ce6c0 0x7f33780c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 -- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33780ce260 msgr2=0x7f337814e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33780ce260 0x7f337814e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 -- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33780ce6c0 msgr2=0x7f33780c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33780ce6c0 0x7f33780c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.546+0000 7f337e400640 1 -- 172.21.15.43:0/3880803207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360009400 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f337e400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f337000c450 tx=0x7f337000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700074e0 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3370007660 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370004a20 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33780c76b0 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.547+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33780c7c70 con 0x7f337815d4d0 2024-01-16T14:59:23.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.548+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3378074770 con 0x7f337815d4d0 2024-01-16T14:59:23.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.549+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f3370004b80 con 0x7f337815d4d0 2024-01-16T14:59:23.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.550+0000 7f3375600640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 0x7f335006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.550+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3370088910 con 0x7f337815d4d0 2024-01-16T14:59:23.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.550+0000 7f337da00640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 0x7f335006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.551+0000 7f337da00640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 0x7f335006d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33680099c0 tx=0x7f3368003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.554+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3370053c60 con 0x7f337815d4d0 2024-01-16T14:59:23.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:23.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:23.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.769+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33780d1020 con 0x7f337815d4d0 2024-01-16T14:59:23.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.770+0000 7f3375600640 1 -- 172.21.15.43:0/3880803207 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12802 (secure 0 0 0) 0x7f3370018020 con 0x7f337815d4d0 2024-01-16T14:59:23.752 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:23.752 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":18,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:23.373908+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 msgr2=0x7f335006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 0x7f335006d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33680099c0 tx=0x7f3368003990 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 msgr2=0x7f33780c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f337000c450 tx=0x7f337000c910 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 shutdown_connections 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f335006b230 0x7f335006d6f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33780ce6c0 0x7f33780c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f337815d4d0 0x7f33780c86d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 --2- 172.21.15.43:0/3880803207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33780ce260 0x7f337814e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 >> 172.21.15.43:0/3880803207 conn(0x7f3378071f90 msgr2=0x7f33780b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 shutdown_connections 2024-01-16T14:59:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.773+0000 7f3380400640 1 -- 172.21.15.43:0/3880803207 wait complete. 2024-01-16T14:59:23.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:23.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:23.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.985+0000 7f2ff4600640 1 Processor -- start 2024-01-16T14:59:23.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.985+0000 7f2ff4600640 1 -- start start 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff4600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fec160b10 0x7f2fec06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff4600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fec072280 0x7f2fec06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff4600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec06cd50 con 0x7f2fec06c900 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff4600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0b2320 con 0x7f2fec072280 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff4600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec06df30 con 0x7f2fec160b10 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fec072280 0x7f2fec06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fec072280 0x7f2fec06c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58152/0 (socket says 172.21.15.43:58152) 2024-01-16T14:59:23.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff1200640 1 -- 172.21.15.43:0/2099907702 learned_addr learned my addr 172.21.15.43:0/2099907702 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.987+0000 7f2ff2600640 1 --1- 172.21.15.43:0/2099907702 >> v1:172.21.15.43:6789/0 conn(0x7f2fec06c900 0x7f2fec160fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35966/0 (socket says 172.21.15.43:35966) 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2ff1c00640 1 --2- 172.21.15.43:0/2099907702 >> v2:172.21.15.69:3300/0 conn(0x7f2fec160b10 0x7f2fec06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2ff1200640 1 -- 172.21.15.43:0/2099907702 >> v2:172.21.15.69:3300/0 conn(0x7f2fec160b10 msgr2=0x7f2fec06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2ff1200640 1 --2- 172.21.15.43:0/2099907702 >> v2:172.21.15.69:3300/0 conn(0x7f2fec160b10 0x7f2fec06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2ff1200640 1 -- 172.21.15.43:0/2099907702 >> v1:172.21.15.43:6789/0 conn(0x7f2fec06c900 legacy=0x7f2fec160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2ff1200640 1 -- 172.21.15.43:0/2099907702 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec071780 con 0x7f2fec072280 2024-01-16T14:59:23.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.988+0000 7f2febe00640 1 -- 172.21.15.43:0/2099907702 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500734502 0 0) 0x7f2fe4001490 con 0x7f2fec06c900 2024-01-16T14:59:23.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.989+0000 7f2ff1200640 1 --2- 172.21.15.43:0/2099907702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec072280 0x7f2fec06c3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0b4070 tx=0x7f2fd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af15187467a44f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.989+0000 7f2febe00640 1 -- 172.21.15.43:0/2099907702 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4031870 con 0x7f2fec072280 2024-01-16T14:59:23.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2febe00640 1 -- 172.21.15.43:0/2099907702 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd40319d0 con 0x7f2fec072280 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2febe00640 1 -- 172.21.15.43:0/2099907702 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4031d60 con 0x7f2fec072280 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 -- 172.21.15.43:0/2099907702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec072280 msgr2=0x7f2fec06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 --2- 172.21.15.43:0/2099907702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec072280 0x7f2fec06c3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0b4070 tx=0x7f2fd402b910 comp rx=0 tx=0).stop 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 -- 172.21.15.43:0/2099907702 shutdown_connections 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 --2- 172.21.15.43:0/2099907702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec072280 0x7f2fec06c3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 --2- 172.21.15.43:0/2099907702 >> v2:172.21.15.69:3300/0 conn(0x7f2fec160b10 0x7f2fec06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 -- 172.21.15.43:0/2099907702 >> 172.21.15.43:0/2099907702 conn(0x7f2fec0b0ba0 msgr2=0x7f2fec0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 -- 172.21.15.43:0/2099907702 shutdown_connections 2024-01-16T14:59:23.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.990+0000 7f2ff4600640 1 -- 172.21.15.43:0/2099907702 wait complete. 2024-01-16T14:59:23.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.992+0000 7f2ff4600640 1 Processor -- start 2024-01-16T14:59:23.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.992+0000 7f2ff4600640 1 -- start start 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fec06c900 0x7f2fec1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fec072280 0x7f2fec09bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec09a950 con 0x7f2fec160b10 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec09ab80 con 0x7f2fec072280 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff4600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec09ae60 con 0x7f2fec06c900 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.993+0000 7f2ff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58154/0 (socket says 172.21.15.43:58154) 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 -- 172.21.15.43:0/463662964 learned_addr learned my addr 172.21.15.43:0/463662964 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff1200640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fec072280 0x7f2fec09bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff1c00640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fec06c900 0x7f2fec1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 -- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fec06c900 msgr2=0x7f2fec1f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fec06c900 0x7f2fec1f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 -- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fec072280 msgr2=0x7f2fec09bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fec072280 0x7f2fec09bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 -- 172.21.15.43:0/463662964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40094d0 con 0x7f2fec160b10 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2ff2600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c780 tx=0x7f2fe400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.994+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400ce90 con 0x7f2fec160b10 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.995+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400a640 con 0x7f2fec160b10 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.995+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400a9d0 con 0x7f2fec160b10 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.995+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec09b140 con 0x7f2fec160b10 2024-01-16T14:59:23.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.995+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec0c8b00 con 0x7f2fec160b10 2024-01-16T14:59:23.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.996+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec09d200 con 0x7f2fec160b10 2024-01-16T14:59:23.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.997+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f2fe400ab30 con 0x7f2fec160b10 2024-01-16T14:59:23.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.998+0000 7f2fe9600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 0x7f2fc406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:23.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.998+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2fe4088a30 con 0x7f2fec160b10 2024-01-16T14:59:23.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.998+0000 7f2ff1c00640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 0x7f2fc406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:23.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:23.999+0000 7f2ff1c00640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 0x7f2fc406d6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009a30 tx=0x7f2fdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:23.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.002+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fe4053d60 con 0x7f2fec160b10 2024-01-16T14:59:24.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.194+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fec06cd50 con 0x7f2fec160b10 2024-01-16T14:59:24.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.195+0000 7f2fe9600640 1 -- 172.21.15.43:0/463662964 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fec06cd50 con 0x7f2fec160b10 2024-01-16T14:59:24.176 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 msgr2=0x7f2fc406d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 0x7f2fc406d6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009a30 tx=0x7f2fdc008040 comp rx=0 tx=0).stop 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 msgr2=0x7f2fec09a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c780 tx=0x7f2fe400cc40 comp rx=0 tx=0).stop 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 shutdown_connections 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fc406b230 0x7f2fc406d6f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fec072280 0x7f2fec09bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fec160b10 0x7f2fec09a410 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 --2- 172.21.15.43:0/463662964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fec06c900 0x7f2fec1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 >> 172.21.15.43:0/463662964 conn(0x7f2fec0b0ba0 msgr2=0x7f2fec0ad290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 shutdown_connections 2024-01-16T14:59:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.198+0000 7f2ff4600640 1 -- 172.21.15.43:0/463662964 wait complete. 2024-01-16T14:59:24.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:59:24.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.446+0000 7f09c1e00640 1 Processor -- start 2024-01-16T14:59:24.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.446+0000 7f09c1e00640 1 -- start start 2024-01-16T14:59:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09bc0cd7b0 0x7f09bc0cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc15fa70 con 0x7f09bc13dfd0 2024-01-16T14:59:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc15c580 con 0x7f09bc0cd7b0 2024-01-16T14:59:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0cfa10 con 0x7f09bc0cf050 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46776/0 (socket says 172.21.15.43:46776) 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09b3e00640 1 -- 172.21.15.43:0/479815783 learned_addr learned my addr 172.21.15.43:0/479815783 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.448+0000 7f09c1400640 1 --2- 172.21.15.43:0/479815783 >> v2:172.21.15.69:3300/0 conn(0x7f09bc0cd7b0 0x7f09bc0cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.449+0000 7f09b3e00640 1 -- 172.21.15.43:0/479815783 >> v2:172.21.15.69:3300/0 conn(0x7f09bc0cd7b0 msgr2=0x7f09bc0cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.449+0000 7f09b3e00640 1 --2- 172.21.15.43:0/479815783 >> v2:172.21.15.69:3300/0 conn(0x7f09bc0cd7b0 0x7f09bc0cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.449+0000 7f09b3e00640 1 -- 172.21.15.43:0/479815783 >> v1:172.21.15.69:6789/0 conn(0x7f09bc13dfd0 legacy=0x7f09bc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.449+0000 7f09b3e00640 1 -- 172.21.15.43:0/479815783 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc140900 con 0x7f09bc0cf050 2024-01-16T14:59:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.450+0000 7f09b3e00640 1 --2- 172.21.15.43:0/479815783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009870 tx=0x7f09a402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f7bbed1cca5a9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.450+0000 7f09bbe00640 1 -- 172.21.15.43:0/479815783 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4030840 con 0x7f09bc0cf050 2024-01-16T14:59:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.450+0000 7f09bbe00640 1 -- 172.21.15.43:0/479815783 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a40309a0 con 0x7f09bc0cf050 2024-01-16T14:59:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.450+0000 7f09bbe00640 1 -- 172.21.15.43:0/479815783 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4030d30 con 0x7f09bc0cf050 2024-01-16T14:59:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 -- 172.21.15.43:0/479815783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 msgr2=0x7f09bc0cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 --2- 172.21.15.43:0/479815783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009870 tx=0x7f09a402b980 comp rx=0 tx=0).stop 2024-01-16T14:59:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 -- 172.21.15.43:0/479815783 shutdown_connections 2024-01-16T14:59:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 --2- 172.21.15.43:0/479815783 >> v2:172.21.15.69:3300/0 conn(0x7f09bc0cd7b0 0x7f09bc0cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 --2- 172.21.15.43:0/479815783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc0cf4d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 -- 172.21.15.43:0/479815783 >> 172.21.15.43:0/479815783 conn(0x7f09bc06a860 msgr2=0x7f09bc06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 -- 172.21.15.43:0/479815783 shutdown_connections 2024-01-16T14:59:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.451+0000 7f09c1e00640 1 -- 172.21.15.43:0/479815783 wait complete. 2024-01-16T14:59:24.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.453+0000 7f09c1e00640 1 Processor -- start 2024-01-16T14:59:24.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.453+0000 7f09c1e00640 1 -- start start 2024-01-16T14:59:24.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09bc0cd7b0 0x7f09bc155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc13b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc15a760 con 0x7f09bc13dfd0 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc159b50 con 0x7f09bc0cd7b0 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.454+0000 7f09c1e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc159e30 con 0x7f09bc0cf050 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc13b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58158/0 (socket says 172.21.15.43:58158) 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc13b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46784/0 (socket says 172.21.15.43:46784) 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 -- 172.21.15.43:0/1041761421 learned_addr learned my addr 172.21.15.43:0/1041761421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c0a00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09bc0cd7b0 0x7f09bc155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 -- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 msgr2=0x7f09bc13b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc13b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 -- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09bc0cd7b0 msgr2=0x7f09bc155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09bc0cd7b0 0x7f09bc155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.455+0000 7f09c1400640 1 -- 172.21.15.43:0/1041761421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a40094d0 con 0x7f09bc13dfd0 2024-01-16T14:59:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09c1400640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09b400efc0 tx=0x7f09b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4014480 con 0x7f09bc13dfd0 2024-01-16T14:59:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b400f040 con 0x7f09bc13dfd0 2024-01-16T14:59:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4014820 con 0x7f09bc13dfd0 2024-01-16T14:59:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc15a110 con 0x7f09bc13dfd0 2024-01-16T14:59:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.456+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc1597e0 con 0x7f09bc13dfd0 2024-01-16T14:59:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.457+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc15a8d0 con 0x7f09bc13dfd0 2024-01-16T14:59:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.458+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f09b4014980 con 0x7f09bc13dfd0 2024-01-16T14:59:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.459+0000 7f09b9600640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 0x7f099406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.459+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09b408a210 con 0x7f09bc13dfd0 2024-01-16T14:59:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.459+0000 7f09c0a00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 0x7f099406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.460+0000 7f09c0a00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 0x7f099406d720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f09ac000f60 tx=0x7f09ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.464+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09b40554c0 con 0x7f09bc13dfd0 2024-01-16T14:59:24.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.667+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f09bc04e840 con 0x7f09bc13dfd0 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.668+0000 7f09b9600640 1 -- 172.21.15.43:0/1041761421 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f09b4058ae0 con 0x7f09bc13dfd0 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T14:59:24.649 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T14:59:24.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T14:59:24.650 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T14:59:24.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 msgr2=0x7f099406d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 0x7f099406d720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f09ac000f60 tx=0x7f09ac009040 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 msgr2=0x7f09bc15abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09b400efc0 tx=0x7f09b400c490 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 shutdown_connections 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099406b260 0x7f099406d720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09bc13dfd0 0x7f09bc15abf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09bc0cf050 0x7f09bc13b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 --2- 172.21.15.43:0/1041761421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09bc0cd7b0 0x7f09bc155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 >> 172.21.15.43:0/1041761421 conn(0x7f09bc06a860 msgr2=0x7f09bc15c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 shutdown_connections 2024-01-16T14:59:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.671+0000 7f09c1e00640 1 -- 172.21.15.43:0/1041761421 wait complete. 2024-01-16T14:59:24.671 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:59:24.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:59:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.914+0000 7f1ae4c00640 1 Processor -- start 2024-01-16T14:59:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.914+0000 7f1ae4c00640 1 -- start start 2024-01-16T14:59:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ae4c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 0x7f1ae0070760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ae4c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae0070d40 con 0x7f1ae00b4140 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ae4c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae0071020 con 0x7f1ae01435d0 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ae4c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae0071300 con 0x7f1ae01439b0 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ad6800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1ae00b4140 0x7f1ae0070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35968/0 (socket says 172.21.15.43:35968) 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.916+0000 7f1ad6800640 1 -- 172.21.15.43:0/2731830259 learned_addr learned my addr 172.21.15.43:0/2731830259 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:24.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adfe00640 1 --2- 172.21.15.43:0/2731830259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 0x7f1ae0070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692268815 0 0) 0x7f1ae0070d40 con 0x7f1ae00b4140 2024-01-16T14:59:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac0005730 con 0x7f1ae00b4140 2024-01-16T14:59:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332720948 0 0) 0x7f1ae0071020 con 0x7f1ae01435d0 2024-01-16T14:59:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae0070d40 con 0x7f1ae01435d0 2024-01-16T14:59:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adfe00640 1 -- 172.21.15.43:0/2731830259 >> v1:172.21.15.69:6789/0 conn(0x7f1ae01435d0 legacy=0x7f1ae00b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adfe00640 1 -- 172.21.15.43:0/2731830259 >> v1:172.21.15.43:6789/0 conn(0x7f1ae00b4140 legacy=0x7f1ae0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356208254 0 0) 0x7f1ac0005730 con 0x7f1ae00b4140 2024-01-16T14:59:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.917+0000 7f1adfe00640 1 -- 172.21.15.43:0/2731830259 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00715e0 con 0x7f1ae01439b0 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1adfe00640 1 --2- 172.21.15.43:0/2731830259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 0x7f1ae0070760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80097d0 tx=0x7f1ad802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad19dd6a45c9fd63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8001670 con 0x7f1ae01439b0 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad802e050 con 0x7f1ae01439b0 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1adea00640 1 -- 172.21.15.43:0/2731830259 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8030730 con 0x7f1ae01439b0 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1ae4c00640 1 -- 172.21.15.43:0/2731830259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 msgr2=0x7f1ae0070760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.919+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/2731830259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 0x7f1ae0070760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80097d0 tx=0x7f1ad802b800 comp rx=0 tx=0).stop 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.920+0000 7f1ae4c00640 1 -- 172.21.15.43:0/2731830259 shutdown_connections 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.920+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/2731830259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae01439b0 0x7f1ae0070760 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.920+0000 7f1ae4c00640 1 -- 172.21.15.43:0/2731830259 >> 172.21.15.43:0/2731830259 conn(0x7f1ae00cec10 msgr2=0x7f1ae00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.920+0000 7f1ae4c00640 1 -- 172.21.15.43:0/2731830259 shutdown_connections 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.920+0000 7f1ae4c00640 1 -- 172.21.15.43:0/2731830259 wait complete. 2024-01-16T14:59:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.922+0000 7f1ae4c00640 1 Processor -- start 2024-01-16T14:59:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.922+0000 7f1ae4c00640 1 -- start start 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 0x7f1ae00b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae01439b0 0x7f1ae013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b1410 con 0x7f1ae01435d0 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae013ccb0 con 0x7f1ae01439b0 2024-01-16T14:59:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ae4c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae013cf90 con 0x7f1ae00b4140 2024-01-16T14:59:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1adf400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1ad6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 0x7f1ae00b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1adf400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46808/0 (socket says 172.21.15.43:46808) 2024-01-16T14:59:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1adf400640 1 -- 172.21.15.43:0/375578089 learned_addr learned my addr 172.21.15.43:0/375578089 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.923+0000 7f1adfe00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae01439b0 0x7f1ae013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 -- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae01439b0 msgr2=0x7f1ae013ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae01439b0 0x7f1ae013ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 -- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 msgr2=0x7f1ae00b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 0x7f1ae00b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 -- 172.21.15.43:0/375578089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8009400 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1ad6800640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 0x7f1ae00b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1adf400640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c800 tx=0x7f1ad000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0003b70 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0003cd0 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000aa90 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.924+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae00ae8f0 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.925+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00aeeb0 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.925+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae004e840 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.927+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f1ad000abf0 con 0x7f1ae00b4140 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.928+0000 7f1ad7e00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 0x7f1ab806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.928+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ad0089ff0 con 0x7f1ae00b4140 2024-01-16T14:59:24.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.928+0000 7f1ad6800640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 0x7f1ab806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:24.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.929+0000 7f1ad6800640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 0x7f1ab806d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8001ab0 tx=0x7f1ac8001a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:24.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:24.942+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad0055330 con 0x7f1ae00b4140 2024-01-16T14:59:25.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.136+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1ae00b4520 con 0x7f1ae00b4140 2024-01-16T14:59:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.900+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1ad0055700 con 0x7f1ae00b4140 2024-01-16T14:59:25.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.901+0000 7f1ad7e00640 1 -- 172.21.15.43:0/375578089 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1ad003d400 con 0x7f1ae00b4140 2024-01-16T14:59:25.883 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T14:59:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 msgr2=0x7f1ab806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 0x7f1ab806d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8001ab0 tx=0x7f1ac8001a40 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 msgr2=0x7f1ae013c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c800 tx=0x7f1ad000ccc0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 shutdown_connections 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab806b230 0x7f1ab806d6f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae01439b0 0x7f1ae013ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ae01435d0 0x7f1ae00b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 --2- 172.21.15.43:0/375578089 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae00b4140 0x7f1ae013c6d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 >> 172.21.15.43:0/375578089 conn(0x7f1ae00cec10 msgr2=0x7f1ae013edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 shutdown_connections 2024-01-16T14:59:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:25.906+0000 7f1ae4c00640 1 -- 172.21.15.43:0/375578089 wait complete. 2024-01-16T14:59:25.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:59:26.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.161+0000 7fa1eca00640 1 Processor -- start 2024-01-16T14:59:26.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.161+0000 7fa1eca00640 1 -- start start 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1eca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa1e80c8930 0x7fa1e80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1eca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80c6bf0 con 0x7fa1e80d1d90 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1eca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80cedc0 con 0x7fa1e815d3b0 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1eca00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80c6e40 con 0x7fa1e80c8930 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1dfe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa1e80d1d90 0x7fa1e80b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35976/0 (socket says 172.21.15.43:35976) 2024-01-16T14:59:26.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1dfe00640 1 -- 172.21.15.43:0/738148472 learned_addr learned my addr 172.21.15.43:0/738148472 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.163+0000 7fa1e7200640 1 --2- 172.21.15.43:0/738148472 >> v2:172.21.15.43:3300/0 conn(0x7fa1e80c8930 0x7fa1e80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.164+0000 7fa1e5e00640 1 -- 172.21.15.43:0/738148472 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792114755 0 0) 0x7fa1e80c6bf0 con 0x7fa1e80d1d90 2024-01-16T14:59:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.164+0000 7fa1e5e00640 1 -- 172.21.15.43:0/738148472 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c8003520 con 0x7fa1e80d1d90 2024-01-16T14:59:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.164+0000 7fa1e7200640 1 -- 172.21.15.43:0/738148472 >> v1:172.21.15.69:6789/0 conn(0x7fa1e815d3b0 legacy=0x7fa1e80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.164+0000 7fa1e7200640 1 -- 172.21.15.43:0/738148472 >> v1:172.21.15.43:6789/0 conn(0x7fa1e80d1d90 legacy=0x7fa1e80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.164+0000 7fa1e7200640 1 -- 172.21.15.43:0/738148472 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80c7120 con 0x7fa1e80c8930 2024-01-16T14:59:26.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.165+0000 7fa1e7200640 1 --2- 172.21.15.43:0/738148472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e80c8930 0x7fa1e80ce6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be4caf663ff8f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.165+0000 7fa1e5e00640 1 -- 172.21.15.43:0/738148472 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e002cc60 con 0x7fa1e80c8930 2024-01-16T14:59:26.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1e5e00640 1 -- 172.21.15.43:0/738148472 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e002cdc0 con 0x7fa1e80c8930 2024-01-16T14:59:26.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1e5e00640 1 -- 172.21.15.43:0/738148472 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0034660 con 0x7fa1e80c8930 2024-01-16T14:59:26.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 -- 172.21.15.43:0/738148472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e80c8930 msgr2=0x7fa1e80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 --2- 172.21.15.43:0/738148472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e80c8930 0x7fa1e80ce6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e002beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 -- 172.21.15.43:0/738148472 shutdown_connections 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 --2- 172.21.15.43:0/738148472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e80c8930 0x7fa1e80ce6c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 -- 172.21.15.43:0/738148472 >> 172.21.15.43:0/738148472 conn(0x7fa1e80d30a0 msgr2=0x7fa1e80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 -- 172.21.15.43:0/738148472 shutdown_connections 2024-01-16T14:59:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.166+0000 7fa1eca00640 1 -- 172.21.15.43:0/738148472 wait complete. 2024-01-16T14:59:26.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.168+0000 7fa1eca00640 1 Processor -- start 2024-01-16T14:59:26.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.168+0000 7fa1eca00640 1 -- start start 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1e80c8930 0x7fa1e80ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 0x7fa1e80c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80caa10 con 0x7fa1e815d3b0 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c6ac0 con 0x7fa1e80d1d90 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1eca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81599c0 con 0x7fa1e80c8930 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1e7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1e7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58198/0 (socket says 172.21.15.43:58198) 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.169+0000 7fa1e7200640 1 -- 172.21.15.43:0/2794906727 learned_addr learned my addr 172.21.15.43:0/2794906727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1dfe00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 0x7fa1e80c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e6800640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1e80c8930 0x7fa1e80ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 -- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1e80c8930 msgr2=0x7fa1e80ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1e80c8930 0x7fa1e80ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 -- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 msgr2=0x7fa1e80c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 0x7fa1e80c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 -- 172.21.15.43:0/2794906727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0009a70 con 0x7fa1e815d3b0 2024-01-16T14:59:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1dfe00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 0x7fa1e80c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.170+0000 7fa1e7200640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.171+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0040650 con 0x7fa1e815d3b0 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.171+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e00407b0 con 0x7fa1e815d3b0 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.171+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0040b60 con 0x7fa1e815d3b0 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.171+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e8159c40 con 0x7fa1e815d3b0 2024-01-16T14:59:26.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.171+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e814af00 con 0x7fa1e815d3b0 2024-01-16T14:59:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.172+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e804e840 con 0x7fa1e815d3b0 2024-01-16T14:59:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.173+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa1e0040cc0 con 0x7fa1e815d3b0 2024-01-16T14:59:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.174+0000 7fa1dea00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 0x7fa1c006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.175+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa1e00ac7c0 con 0x7fa1e815d3b0 2024-01-16T14:59:26.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.175+0000 7fa1e6800640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 0x7fa1c006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.175+0000 7fa1e6800640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 0x7fa1c006d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8000f60 tx=0x7fa1d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.180+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1e0077800 con 0x7fa1e815d3b0 2024-01-16T14:59:26.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.376+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1e8159dd0 con 0x7fa1e815d3b0 2024-01-16T14:59:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.653+0000 7fa1dea00640 1 -- 172.21.15.43:0/2794906727 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa1e0009450 con 0x7fa1e815d3b0 2024-01-16T14:59:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T14:59:26.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 msgr2=0x7fa1c006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 0x7fa1c006d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8000f60 tx=0x7fa1d8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:26.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 msgr2=0x7fa1e814a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e0004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 shutdown_connections 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1c006b2c0 0x7fa1c006d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1e80d1d90 0x7fa1e80c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1e815d3b0 0x7fa1e814a9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 --2- 172.21.15.43:0/2794906727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1e80c8930 0x7fa1e80ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.657+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 >> 172.21.15.43:0/2794906727 conn(0x7fa1e80d30a0 msgr2=0x7fa1e80cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:26.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.658+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 shutdown_connections 2024-01-16T14:59:26.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.658+0000 7fa1eca00640 1 -- 172.21.15.43:0/2794906727 wait complete. 2024-01-16T14:59:26.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:59:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.878+0000 7f15b3600640 1 Processor -- start 2024-01-16T14:59:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.879+0000 7f15b3600640 1 -- start start 2024-01-16T14:59:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b3600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f15ac15d350 0x7f15ac0d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b3600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15ac0d1840 0x7f15ac0cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b3600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0cf240 con 0x7f15ac15d730 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b3600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac0d1fb0 con 0x7f15ac15d350 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b3600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac0ce050 con 0x7f15ac0d1840 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f15ac15d350 0x7f15ac0d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f15ac15d350 0x7f15ac0d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58204/0 (socket says 172.21.15.43:58204) 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.880+0000 7f15b0c00640 1 -- 172.21.15.43:0/986106308 learned_addr learned my addr 172.21.15.43:0/986106308 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.881+0000 7f15b0c00640 1 -- 172.21.15.43:0/986106308 >> v2:172.21.15.69:3300/0 conn(0x7f15ac0d1840 msgr2=0x7f15ac0cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.881+0000 7f15b0c00640 1 --2- 172.21.15.43:0/986106308 >> v2:172.21.15.69:3300/0 conn(0x7f15ac0d1840 0x7f15ac0cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.881+0000 7f15b0c00640 1 -- 172.21.15.43:0/986106308 >> v1:172.21.15.69:6789/0 conn(0x7f15ac15d730 legacy=0x7f15ac0ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.881+0000 7f15b0c00640 1 -- 172.21.15.43:0/986106308 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0ce330 con 0x7f15ac15d350 2024-01-16T14:59:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.882+0000 7f15b0c00640 1 --2- 172.21.15.43:0/986106308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac15d350 0x7f15ac0d3d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15ac072960 tx=0x7f159c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f84c1628b74342 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.882+0000 7f15ab400640 1 -- 172.21.15.43:0/986106308 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c031870 con 0x7f15ac15d350 2024-01-16T14:59:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.882+0000 7f15ab400640 1 -- 172.21.15.43:0/986106308 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c0319d0 con 0x7f15ac15d350 2024-01-16T14:59:26.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.882+0000 7f15ab400640 1 -- 172.21.15.43:0/986106308 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c031d60 con 0x7f15ac15d350 2024-01-16T14:59:26.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 -- 172.21.15.43:0/986106308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac15d350 msgr2=0x7f15ac0d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 --2- 172.21.15.43:0/986106308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac15d350 0x7f15ac0d3d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15ac072960 tx=0x7f159c02b910 comp rx=0 tx=0).stop 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 -- 172.21.15.43:0/986106308 shutdown_connections 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 --2- 172.21.15.43:0/986106308 >> v2:172.21.15.69:3300/0 conn(0x7f15ac0d1840 0x7f15ac0cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 --2- 172.21.15.43:0/986106308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac15d350 0x7f15ac0d3d30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 -- 172.21.15.43:0/986106308 >> 172.21.15.43:0/986106308 conn(0x7f15ac09c5c0 msgr2=0x7f15ac0d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 -- 172.21.15.43:0/986106308 shutdown_connections 2024-01-16T14:59:26.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.883+0000 7f15b3600640 1 -- 172.21.15.43:0/986106308 wait complete. 2024-01-16T14:59:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.885+0000 7f15b3600640 1 Processor -- start 2024-01-16T14:59:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.885+0000 7f15b3600640 1 -- start start 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 0x7f15ac0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 0x7f15ac160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0cbf20 con 0x7f15ac0d1840 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0cb310 con 0x7f15ac15d350 2024-01-16T14:59:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.886+0000 7f15b3600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0cb5f0 con 0x7f15ac15d730 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58206/0 (socket says 172.21.15.43:58206) 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 -- 172.21.15.43:0/2236094709 learned_addr learned my addr 172.21.15.43:0/2236094709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15abe00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 0x7f15ac0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 -- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 msgr2=0x7f15ac160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b1600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 0x7f15ac160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 0x7f15ac160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 -- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 msgr2=0x7f15ac0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 0x7f15ac0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 -- 172.21.15.43:0/2236094709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c0094d0 con 0x7f15ac0d1840 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15abe00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 0x7f15ac0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b1600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 0x7f15ac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.887+0000 7f15b0c00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f159c000c00 tx=0x7f159c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.888+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c03adf0 con 0x7f15ac0d1840 2024-01-16T14:59:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.888+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c0443f0 con 0x7f15ac0d1840 2024-01-16T14:59:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.888+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c0447a0 con 0x7f15ac0d1840 2024-01-16T14:59:26.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.888+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac0cb8d0 con 0x7f15ac0d1840 2024-01-16T14:59:26.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.888+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac0cb000 con 0x7f15ac0d1840 2024-01-16T14:59:26.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.889+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac0a1230 con 0x7f15ac0d1840 2024-01-16T14:59:26.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.890+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f159c04d450 con 0x7f15ac0d1840 2024-01-16T14:59:26.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.891+0000 7f15a8c00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 0x7f158406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:26.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.891+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f159c02d040 con 0x7f15ac0d1840 2024-01-16T14:59:26.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.892+0000 7f15abe00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 0x7f158406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:26.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.892+0000 7f15abe00640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 0x7f158406d7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f15ac1f6de0 tx=0x7f15940073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:26.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:26.896+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f159c077130 con 0x7f15ac0d1840 2024-01-16T14:59:27.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.127+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f15ac04e920 con 0x7f15ac0d1840 2024-01-16T14:59:27.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.697+0000 7f15a8c00640 1 -- 172.21.15.43:0/2236094709 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f159c07a750 con 0x7f15ac0d1840 2024-01-16T14:59:27.682 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 2 and data pool 3 2024-01-16T14:59:27.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.704+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 msgr2=0x7f158406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.704+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 0x7f158406d7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f15ac1f6de0 tx=0x7f15940073c0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.704+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 msgr2=0x7f15ac148120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.704+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f159c000c00 tx=0x7f159c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 shutdown_connections 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f158406b310 0x7f158406d7d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15ac15d350 0x7f15ac0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15ac15d730 0x7f15ac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 --2- 172.21.15.43:0/2236094709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15ac0d1840 0x7f15ac148120 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 >> 172.21.15.43:0/2236094709 conn(0x7f15ac09c5c0 msgr2=0x7f15ac0b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 shutdown_connections 2024-01-16T14:59:27.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.706+0000 7f15b3600640 1 -- 172.21.15.43:0/2236094709 wait complete. 2024-01-16T14:59:27.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.922+0000 7f5184a00640 1 Processor -- start 2024-01-16T14:59:27.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.923+0000 7f5184a00640 1 -- start start 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5184a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f51800d14a0 0x7f51800d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5184a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800cf0c0 con 0x7f518015d5b0 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5184a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800d1e40 con 0x7f51800b0640 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5184a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51800b01b0 con 0x7f51800d14a0 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5177e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f51800d14a0 0x7f51800d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5177e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f51800d14a0 0x7f51800d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58212/0 (socket says 172.21.15.43:58212) 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f5177e00640 1 -- 172.21.15.43:0/1155324554 learned_addr learned my addr 172.21.15.43:0/1155324554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.924+0000 7f517f400640 1 --1- 172.21.15.43:0/1155324554 >> v1:172.21.15.43:6789/0 conn(0x7f518015d5b0 0x7f51800d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35982/0 (socket says 172.21.15.43:35982) 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.925+0000 7f5177e00640 1 -- 172.21.15.43:0/1155324554 >> v1:172.21.15.69:6789/0 conn(0x7f51800b0640 legacy=0x7f51800b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.925+0000 7f5177e00640 1 -- 172.21.15.43:0/1155324554 >> v1:172.21.15.43:6789/0 conn(0x7f518015d5b0 legacy=0x7f51800d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.925+0000 7f5177e00640 1 -- 172.21.15.43:0/1155324554 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518015d990 con 0x7f51800d14a0 2024-01-16T14:59:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.925+0000 7f517ea00640 1 -- 172.21.15.43:0/1155324554 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197370656 0 0) 0x7f51800cf0c0 con 0x7f518015d5b0 2024-01-16T14:59:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.926+0000 7f5177e00640 1 --2- 172.21.15.43:0/1155324554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800d14a0 0x7f51800d1900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5168009870 tx=0x7f516802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be72f8ede2c57cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.926+0000 7f517ea00640 1 -- 172.21.15.43:0/1155324554 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168030860 con 0x7f51800d14a0 2024-01-16T14:59:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.926+0000 7f517ea00640 1 -- 172.21.15.43:0/1155324554 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51680309c0 con 0x7f51800d14a0 2024-01-16T14:59:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.926+0000 7f517ea00640 1 -- 172.21.15.43:0/1155324554 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168030d50 con 0x7f51800d14a0 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 -- 172.21.15.43:0/1155324554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800d14a0 msgr2=0x7f51800d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 --2- 172.21.15.43:0/1155324554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800d14a0 0x7f51800d1900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5168009870 tx=0x7f516802b9a0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 -- 172.21.15.43:0/1155324554 shutdown_connections 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 --2- 172.21.15.43:0/1155324554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800d14a0 0x7f51800d1900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 -- 172.21.15.43:0/1155324554 >> 172.21.15.43:0/1155324554 conn(0x7f518009c3b0 msgr2=0x7f51800d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 -- 172.21.15.43:0/1155324554 shutdown_connections 2024-01-16T14:59:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.927+0000 7f5184a00640 1 -- 172.21.15.43:0/1155324554 wait complete. 2024-01-16T14:59:27.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.929+0000 7f5184a00640 1 Processor -- start 2024-01-16T14:59:27.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.929+0000 7f5184a00640 1 -- start start 2024-01-16T14:59:27.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51800d14a0 0x7f5180152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 0x7f51800b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51800cb3f0 con 0x7f51800b0640 2024-01-16T14:59:27.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518015f5a0 con 0x7f51800d14a0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f5184a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518015e760 con 0x7f518015d5b0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f517f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f517f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58220/0 (socket says 172.21.15.43:58220) 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.930+0000 7f517f400640 1 -- 172.21.15.43:0/234929876 learned_addr learned my addr 172.21.15.43:0/234929876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f5177e00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51800d14a0 0x7f5180152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517fe00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 0x7f51800b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 -- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 msgr2=0x7f51800b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 0x7f51800b8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 -- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51800d14a0 msgr2=0x7f5180152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51800d14a0 0x7f5180152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:27.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 -- 172.21.15.43:0/234929876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680094d0 con 0x7f51800b0640 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517fe00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 0x7f51800b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f517f400640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f517000c430 tx=0x7f517000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51700074e0 con 0x7f51800b0640 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170007660 con 0x7f51800b0640 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518015ea40 con 0x7f51800b0640 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.931+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5170004a20 con 0x7f51800b0640 2024-01-16T14:59:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.932+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5180152c60 con 0x7f51800b0640 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.933+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518004e840 con 0x7f51800b0640 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.934+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5170004b80 con 0x7f51800b0640 2024-01-16T14:59:27.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.935+0000 7f5177400640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 0x7f515806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.935+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5170089aa0 con 0x7f51800b0640 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.936+0000 7f5177e00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 0x7f515806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:27.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.936+0000 7f5177e00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 0x7f515806d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5168036000 tx=0x7f516803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:27.940+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5170054870 con 0x7f51800b0640 2024-01-16T14:59:28.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.135+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51800678d0 con 0x7f51800b0640 2024-01-16T14:59:28.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.138+0000 7f5177400640 1 -- 172.21.15.43:0/234929876 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16470 (secure 0 0 0) 0x7f5170057e90 con 0x7f51800b0640 2024-01-16T14:59:28.119 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:28.119 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":21,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:27.129631+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:28.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.140+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 msgr2=0x7f515806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.140+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 0x7f515806d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5168036000 tx=0x7f516803b040 comp rx=0 tx=0).stop 2024-01-16T14:59:28.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.140+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 msgr2=0x7f51800b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.140+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f517000c430 tx=0x7f517000c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 shutdown_connections 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f515806b2c0 0x7f515806d780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51800d14a0 0x7f5180152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f518015d5b0 0x7f51800b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 --2- 172.21.15.43:0/234929876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51800b0640 0x7f51800b5c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 >> 172.21.15.43:0/234929876 conn(0x7f518009c3b0 msgr2=0x7f518006bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 shutdown_connections 2024-01-16T14:59:28.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.141+0000 7f5184a00640 1 -- 172.21.15.43:0/234929876 wait complete. 2024-01-16T14:59:28.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.374+0000 7faca3400640 1 Processor -- start 2024-01-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.374+0000 7faca3400640 1 -- start start 2024-01-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c14b1c0 0x7fac9c162670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fac9c0cf760 0x7fac9c1655e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca3400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9c162bb0 con 0x7fac9c1651a0 2024-01-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c165ca0 con 0x7fac9c0cf760 2024-01-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c165f80 con 0x7fac9c14b1c0 2024-01-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fac9c0cf760 0x7fac9c1655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca1600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fac9c1651a0 0x7fac9c0cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35998/0 (socket says 172.21.15.43:35998) 2024-01-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.376+0000 7faca1600640 1 -- 172.21.15.43:0/3099517597 learned_addr learned my addr 172.21.15.43:0/3099517597 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.377+0000 7faca2000640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c14b1c0 0x7fac9c162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.377+0000 7faca2a00640 1 -- 172.21.15.43:0/3099517597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c14b1c0 msgr2=0x7fac9c162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.377+0000 7faca2a00640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c14b1c0 0x7fac9c162670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.377+0000 7faca2a00640 1 -- 172.21.15.43:0/3099517597 >> v1:172.21.15.43:6789/0 conn(0x7fac9c1651a0 legacy=0x7fac9c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.377+0000 7faca2a00640 1 -- 172.21.15.43:0/3099517597 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c161e70 con 0x7fac9c0cf760 2024-01-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.378+0000 7faca2a00640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c0cf760 0x7fac9c1655e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac94009d90 tx=0x7fac9402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fa78616603ca9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.378+0000 7faca0c00640 1 -- 172.21.15.43:0/3099517597 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9402cbc0 con 0x7fac9c0cf760 2024-01-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.378+0000 7faca0c00640 1 -- 172.21.15.43:0/3099517597 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9402cd20 con 0x7fac9c0cf760 2024-01-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.378+0000 7faca0c00640 1 -- 172.21.15.43:0/3099517597 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94034660 con 0x7fac9c0cf760 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 -- 172.21.15.43:0/3099517597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c0cf760 msgr2=0x7fac9c1655e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c0cf760 0x7fac9c1655e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac94009d90 tx=0x7fac9402be10 comp rx=0 tx=0).stop 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 -- 172.21.15.43:0/3099517597 shutdown_connections 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c0cf760 0x7fac9c1655e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 --2- 172.21.15.43:0/3099517597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c14b1c0 0x7fac9c162670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 -- 172.21.15.43:0/3099517597 >> 172.21.15.43:0/3099517597 conn(0x7fac9c06a860 msgr2=0x7fac9c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 -- 172.21.15.43:0/3099517597 shutdown_connections 2024-01-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.379+0000 7faca3400640 1 -- 172.21.15.43:0/3099517597 wait complete. 2024-01-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.381+0000 7faca3400640 1 Processor -- start 2024-01-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.381+0000 7faca3400640 1 -- start start 2024-01-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c0cf760 0x7fac9c0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac9c1651a0 0x7fac9c14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0cc4e0 con 0x7fac9c14b1c0 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0cc7c0 con 0x7fac9c1651a0 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0ccaa0 con 0x7fac9c0cf760 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58236/0 (socket says 172.21.15.43:58236) 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca1600640 1 -- 172.21.15.43:0/469658140 learned_addr learned my addr 172.21.15.43:0/469658140 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.382+0000 7faca2000640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c0cf760 0x7fac9c0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca2a00640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac9c1651a0 0x7fac9c14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 -- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c0cf760 msgr2=0x7fac9c0ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c0cf760 0x7fac9c0ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 -- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac9c1651a0 msgr2=0x7fac9c14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac9c1651a0 0x7fac9c14fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 -- 172.21.15.43:0/469658140 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94009a70 con 0x7fac9c14b1c0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca1600640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c480 tx=0x7fac8400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84005ce0 con 0x7fac9c14b1c0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005e40 con 0x7fac9c14b1c0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c0ccd80 con 0x7fac9c14b1c0 2024-01-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840072f0 con 0x7fac9c14b1c0 2024-01-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.383+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c0cc050 con 0x7fac9c14b1c0 2024-01-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.385+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c06c5c0 con 0x7fac9c14b1c0 2024-01-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.386+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac84007510 con 0x7fac9c14b1c0 2024-01-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.387+0000 7fac9a000640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 0x7fac7406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.387+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac84089100 con 0x7fac9c14b1c0 2024-01-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.387+0000 7faca2000640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 0x7fac7406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.388+0000 7faca2000640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 0x7fac7406d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.392+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac84054010 con 0x7fac9c14b1c0 2024-01-16T14:59:28.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.584+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac9c04e840 con 0x7fac9c14b1c0 2024-01-16T14:59:28.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.586+0000 7fac9a000640 1 -- 172.21.15.43:0/469658140 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16470 (secure 0 0 0) 0x7fac84057630 con 0x7fac9c14b1c0 2024-01-16T14:59:28.568 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:28.568 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":21,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:27.129631+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:28.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 msgr2=0x7fac7406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 0x7fac7406d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac8c006fd0 tx=0x7fac8c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 msgr2=0x7fac9c14f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c480 tx=0x7fac8400c940 comp rx=0 tx=0).stop 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 shutdown_connections 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fac7406b2c0 0x7fac7406d780 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac9c1651a0 0x7fac9c14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac9c14b1c0 0x7fac9c14f8a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 --2- 172.21.15.43:0/469658140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9c0cf760 0x7fac9c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.589+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 >> 172.21.15.43:0/469658140 conn(0x7fac9c06a860 msgr2=0x7fac9c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.590+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 shutdown_connections 2024-01-16T14:59:28.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.590+0000 7faca3400640 1 -- 172.21.15.43:0/469658140 wait complete. 2024-01-16T14:59:28.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:59:28.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.833+0000 7f3145600640 1 Processor -- start 2024-01-16T14:59:28.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.833+0000 7f3145600640 1 -- start start 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.835+0000 7f3145600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f31400abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.835+0000 7f3145600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31400adb60 0x7f31400adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.835+0000 7f3145600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314015b730 con 0x7f314015b350 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.835+0000 7f3145600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31400ac740 con 0x7f31400adb60 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.835+0000 7f3145600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31400ac460 con 0x7f314009cb30 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31400adb60 0x7f31400adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31400adb60 0x7f31400adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58238/0 (socket says 172.21.15.43:58238) 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 -- 172.21.15.43:0/3939455719 learned_addr learned my addr 172.21.15.43:0/3939455719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313ea00640 1 --1- 172.21.15.43:0/3939455719 >> v1:172.21.15.43:6789/0 conn(0x7f314015b350 0x7f31400b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36008/0 (socket says 172.21.15.43:36008) 2024-01-16T14:59:28.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f3137e00640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f31400abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 -- 172.21.15.43:0/3939455719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 msgr2=0x7f31400abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f31400abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 -- 172.21.15.43:0/3939455719 >> v1:172.21.15.43:6789/0 conn(0x7f314015b350 legacy=0x7f31400b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.836+0000 7f313f400640 1 -- 172.21.15.43:0/3939455719 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400b2a50 con 0x7f31400adb60 2024-01-16T14:59:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.837+0000 7f313f400640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f31400adfb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3138009e50 tx=0x7f313802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77903998dd2bb41d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f313e000640 1 -- 172.21.15.43:0/3939455719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313802cb00 con 0x7f31400adb60 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f313e000640 1 -- 172.21.15.43:0/3939455719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313802cc60 con 0x7f31400adb60 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f313e000640 1 -- 172.21.15.43:0/3939455719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31380343f0 con 0x7f31400adb60 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f3145600640 1 -- 172.21.15.43:0/3939455719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 msgr2=0x7f31400adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f3145600640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f31400adfb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3138009e50 tx=0x7f313802bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f3145600640 1 -- 172.21.15.43:0/3939455719 shutdown_connections 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.838+0000 7f3145600640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f31400adfb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.839+0000 7f3145600640 1 --2- 172.21.15.43:0/3939455719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f31400abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.839+0000 7f3145600640 1 -- 172.21.15.43:0/3939455719 >> 172.21.15.43:0/3939455719 conn(0x7f31400b0ba0 msgr2=0x7f31400b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.839+0000 7f3145600640 1 -- 172.21.15.43:0/3939455719 shutdown_connections 2024-01-16T14:59:28.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.839+0000 7f3145600640 1 -- 172.21.15.43:0/3939455719 wait complete. 2024-01-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.841+0000 7f3145600640 1 Processor -- start 2024-01-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.841+0000 7f3145600640 1 -- start start 2024-01-16T14:59:28.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f3140159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 0x7f3140156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3140152e50 con 0x7f31400adb60 2024-01-16T14:59:28.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314009bb50 con 0x7f314015b350 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3145600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314009be30 con 0x7f314009cb30 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.842+0000 7f3137e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58254/0 (socket says 172.21.15.43:58254) 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 -- 172.21.15.43:0/3256450093 learned_addr learned my addr 172.21.15.43:0/3256450093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f313ea00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f3140159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f313f400640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 0x7f3140156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 -- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 msgr2=0x7f3140159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f3140159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 -- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 msgr2=0x7f3140156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 0x7f3140156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 -- 172.21.15.43:0/3256450093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138009b30 con 0x7f31400adb60 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f313f400640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 0x7f3140156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.843+0000 7f3137e00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f312800e870 tx=0x7f312800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.844+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280094f0 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.844+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312800c940 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.844+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800ccf0 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.844+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314009c110 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.844+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314009a450 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.845+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314009d200 con 0x7f31400adb60 2024-01-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.846+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3128013470 con 0x7f31400adb60 2024-01-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.847+0000 7f3136a00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 0x7f311806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.847+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3128089c60 con 0x7f31400adb60 2024-01-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.847+0000 7f313ea00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 0x7f311806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.847+0000 7f313ea00640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 0x7f311806d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3130000f60 tx=0x7f3130007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:28.851+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3128054b60 con 0x7f31400adb60 2024-01-16T14:59:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.091+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f314004e920 con 0x7f31400adb60 2024-01-16T14:59:29.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.733+0000 7f3136a00640 1 -- 172.21.15.43:0/3256450093 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f31280170b0 con 0x7f31400adb60 2024-01-16T14:59:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 msgr2=0x7f311806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 0x7f311806d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3130000f60 tx=0x7f3130007c80 comp rx=0 tx=0).stop 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 msgr2=0x7f3140099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f312800e870 tx=0x7f312800ed30 comp rx=0 tx=0).stop 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 shutdown_connections 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f311806b2c0 0x7f311806d780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314015b350 0x7f3140156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31400adb60 0x7f3140099a00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 --2- 172.21.15.43:0/3256450093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f314009cb30 0x7f3140159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.737+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 >> 172.21.15.43:0/3256450093 conn(0x7f31400b0ba0 msgr2=0x7f314014bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.738+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 shutdown_connections 2024-01-16T14:59:29.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.738+0000 7f3145600640 1 -- 172.21.15.43:0/3256450093 wait complete. 2024-01-16T14:59:29.741 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:59:29.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T14:59:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.994+0000 7effda800640 1 Processor -- start 2024-01-16T14:59:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.994+0000 7effda800640 1 -- start start 2024-01-16T14:59:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effda800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effd4162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effda800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4162fa0 con 0x7effd406c3c0 2024-01-16T14:59:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effda800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd40706a0 con 0x7effd4162220 2024-01-16T14:59:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effda800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd4070c60 con 0x7effd4162600 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effd3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7effd406c3c0 0x7effd40710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36018/0 (socket says 172.21.15.43:36018) 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effd3e00640 1 -- 172.21.15.43:0/3678600648 learned_addr learned my addr 172.21.15.43:0/3678600648 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effcbe00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effd4162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.996+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361057869 0 0) 0x7effd4162fa0 con 0x7effd406c3c0 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb4003520 con 0x7effd406c3c0 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3213616682 0 0) 0x7effb4003520 con 0x7effd406c3c0 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 msgr2=0x7effd4162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effd4162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 >> v1:172.21.15.69:6789/0 conn(0x7effd4162220 legacy=0x7effd40b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40722b0 con 0x7effd406c3c0 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7effcc0019f0 con 0x7effd406c3c0 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 >> v1:172.21.15.43:6789/0 conn(0x7effd406c3c0 legacy=0x7effd40710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effb40039a0 0x7effb4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40047c0 0x7effb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4070c60 con 0x7effb40039a0 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb400ba30 con 0x7effb40047c0 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb400bbf0 con 0x7effd4162600 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.997+0000 7effd3400640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effb40039a0 0x7effb4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3e00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effcbe00640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40047c0 0x7effb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3400640 1 -- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 msgr2=0x7effb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3400640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd4162600 0x7effb4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3400640 1 -- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40047c0 msgr2=0x7effb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3400640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40047c0 0x7effb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.998+0000 7effd3400640 1 -- 172.21.15.43:0/3678600648 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4003520 con 0x7effb40039a0 2024-01-16T14:59:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.999+0000 7effd3400640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effb40039a0 0x7effb4003dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effc4001720 tx=0x7effb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fcaf6086c93d6ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.999+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc40052d0 con 0x7effb40039a0 2024-01-16T14:59:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.999+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc402e050 con 0x7effb40039a0 2024-01-16T14:59:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:29.999+0000 7effd2a00640 1 -- 172.21.15.43:0/3678600648 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc40035f0 con 0x7effb40039a0 2024-01-16T14:59:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 -- 172.21.15.43:0/3678600648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effb40039a0 msgr2=0x7effb4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 --2- 172.21.15.43:0/3678600648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effb40039a0 0x7effb4003dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effc4001720 tx=0x7effb40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effd3e00640 1 -- 172.21.15.43:0/3678600648 reap_dead start 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 -- 172.21.15.43:0/3678600648 shutdown_connections 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 -- 172.21.15.43:0/3678600648 >> 172.21.15.43:0/3678600648 conn(0x7effd40b0ba0 msgr2=0x7effd40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 -- 172.21.15.43:0/3678600648 shutdown_connections 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.000+0000 7effda800640 1 -- 172.21.15.43:0/3678600648 wait complete. 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.002+0000 7effda800640 1 Processor -- start 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.002+0000 7effda800640 1 -- start start 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 0x7effd40cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd40b2ac0 0x7effd40b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40b3490 con 0x7effd4153630 2024-01-16T14:59:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40ad6d0 con 0x7effb40039a0 2024-01-16T14:59:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effda800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40ac540 con 0x7effd40b2ac0 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effcbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effcbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58262/0 (socket says 172.21.15.43:58262) 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.003+0000 7effcbe00640 1 -- 172.21.15.43:0/1700021396 learned_addr learned my addr 172.21.15.43:0/1700021396 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effd3e00640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd40b2ac0 0x7effd40b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effd3400640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 0x7effd40cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 -- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd40b2ac0 msgr2=0x7effd40b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd40b2ac0 0x7effd40b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 -- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 msgr2=0x7effd40cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 0x7effd40cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 -- 172.21.15.43:0/1700021396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40ac820 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effd3400640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 0x7effd40cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcbe00640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effbc011630 tx=0x7effbc011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc00f920 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc00fa80 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.004+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc00fdf0 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.005+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd40acb00 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.005+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd40acde0 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.006+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd409d200 con 0x7effd4153630 2024-01-16T14:59:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.007+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7effbc015460 con 0x7effd4153630 2024-01-16T14:59:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.008+0000 7effcb400640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 0x7effac06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.008+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effbc089ff0 con 0x7effd4153630 2024-01-16T14:59:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.008+0000 7effd3400640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 0x7effac06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.009+0000 7effd3400640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 0x7effac06d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7effc4033000 tx=0x7effc4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.012+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effbc054e50 con 0x7effd4153630 2024-01-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.249+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7effd404e920 con 0x7effd4153630 2024-01-16T14:59:30.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.736+0000 7effcb400640 1 -- 172.21.15.43:0/1700021396 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7effbc0220c0 con 0x7effd4153630 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 msgr2=0x7effac06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 0x7effac06d7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7effc4033000 tx=0x7effc4009030 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 msgr2=0x7effd40abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effbc011630 tx=0x7effbc011af0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 shutdown_connections 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7effac06b2f0 0x7effac06d7b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effd40b2ac0 0x7effd40b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effd4153630 0x7effd40abe80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 --2- 172.21.15.43:0/1700021396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb40039a0 0x7effd40cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 >> 172.21.15.43:0/1700021396 conn(0x7effd40b0ba0 msgr2=0x7effd4070ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:30.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.740+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 shutdown_connections 2024-01-16T14:59:30.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.741+0000 7effda800640 1 -- 172.21.15.43:0/1700021396 wait complete. 2024-01-16T14:59:30.744 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T14:59:30.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.977+0000 7ff21f800640 1 Processor -- start 2024-01-16T14:59:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.978+0000 7ff21f800640 1 -- start start 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff218146c40 0x7ff21806ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff21806efb0 0x7ff21806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff21806bf60 0x7ff2180b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21806f410 con 0x7ff218146c40 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21815da60 con 0x7ff21806efb0 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21f800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff218147020 con 0x7ff21806bf60 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.979+0000 7ff21ce00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff218146c40 0x7ff21806ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff218146c40 0x7ff21806ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58264/0 (socket says 172.21.15.43:58264) 2024-01-16T14:59:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 -- 172.21.15.43:0/1434081209 learned_addr learned my addr 172.21.15.43:0/1434081209 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 -- 172.21.15.43:0/1434081209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff21806bf60 msgr2=0x7ff2180b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 --2- 172.21.15.43:0/1434081209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff21806bf60 0x7ff2180b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 -- 172.21.15.43:0/1434081209 >> v2:172.21.15.69:3300/0 conn(0x7ff21806efb0 msgr2=0x7ff21806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 --2- 172.21.15.43:0/1434081209 >> v2:172.21.15.69:3300/0 conn(0x7ff21806efb0 0x7ff21806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.980+0000 7ff21ce00640 1 -- 172.21.15.43:0/1434081209 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21806c730 con 0x7ff218146c40 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.981+0000 7ff21ce00640 1 --2- 172.21.15.43:0/1434081209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff218146c40 0x7ff21806ea70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2180d2060 tx=0x7ff20802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22984a63ee5562a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.981+0000 7ff217400640 1 -- 172.21.15.43:0/1434081209 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20802ca70 con 0x7ff218146c40 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.981+0000 7ff217400640 1 -- 172.21.15.43:0/1434081209 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20802cbd0 con 0x7ff218146c40 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.981+0000 7ff217400640 1 -- 172.21.15.43:0/1434081209 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080343f0 con 0x7ff218146c40 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 -- 172.21.15.43:0/1434081209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff218146c40 msgr2=0x7ff21806ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 --2- 172.21.15.43:0/1434081209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff218146c40 0x7ff21806ea70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2180d2060 tx=0x7ff20802bed0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 -- 172.21.15.43:0/1434081209 shutdown_connections 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 --2- 172.21.15.43:0/1434081209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff21806bf60 0x7ff2180b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 --2- 172.21.15.43:0/1434081209 >> v2:172.21.15.69:3300/0 conn(0x7ff21806efb0 0x7ff21806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 --2- 172.21.15.43:0/1434081209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff218146c40 0x7ff21806ea70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 -- 172.21.15.43:0/1434081209 >> 172.21.15.43:0/1434081209 conn(0x7ff2180663d0 msgr2=0x7ff2180667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 -- 172.21.15.43:0/1434081209 shutdown_connections 2024-01-16T14:59:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.982+0000 7ff21f800640 1 -- 172.21.15.43:0/1434081209 wait complete. 2024-01-16T14:59:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.984+0000 7ff21f800640 1 Processor -- start 2024-01-16T14:59:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.984+0000 7ff21f800640 1 -- start start 2024-01-16T14:59:30.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 0x7ff21813c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 0x7ff21813cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21813f820 con 0x7ff21806bf60 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21813ec10 con 0x7ff21806efb0 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.985+0000 7ff21f800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21813eef0 con 0x7ff218146c40 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58268/0 (socket says 172.21.15.43:58268) 2024-01-16T14:59:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 -- 172.21.15.43:0/3679652723 learned_addr learned my addr 172.21.15.43:0/3679652723 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21d800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 0x7ff21813cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff217e00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 0x7ff21813c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 -- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 msgr2=0x7ff21813cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 0x7ff21813cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 -- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 msgr2=0x7ff21813c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 0x7ff21813c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 -- 172.21.15.43:0/3679652723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208009b30 con 0x7ff21806bf60 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff217e00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 0x7ff21813c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21d800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 0x7ff21813cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.986+0000 7ff21ce00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2080093e0 tx=0x7ff208004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.987+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20803e450 con 0x7ff21806bf60 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.987+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20803e5b0 con 0x7ff21806bf60 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.987+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20803e960 con 0x7ff21806bf60 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.987+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21813f1d0 con 0x7ff21806bf60 2024-01-16T14:59:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.987+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21813fcb0 con 0x7ff21806bf60 2024-01-16T14:59:30.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.988+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218074770 con 0x7ff21806bf60 2024-01-16T14:59:30.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.989+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff20803eac0 con 0x7ff21806bf60 2024-01-16T14:59:30.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.990+0000 7ff214c00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:30.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.990+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2080acd30 con 0x7ff21806bf60 2024-01-16T14:59:30.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.990+0000 7ff217e00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:30.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.990+0000 7ff217e00640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 0x7ff1f006d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff200006fd0 tx=0x7ff200008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:30.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:30.994+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff208077c20 con 0x7ff21806bf60 2024-01-16T14:59:31.040 INFO:tasks.ceph.mds.b.smithi069.stderr:2024-01-16T14:59:31.058+0000 7f844d600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:59:31.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.226+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff218160c50 con 0x7ff21806bf60 2024-01-16T14:59:31.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.227+0000 7ff214c00640 1 -- 172.21.15.43:0/3679652723 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7ff20807b240 con 0x7ff21806bf60 2024-01-16T14:59:31.208 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:31.208 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:59:31.209 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T14:59:31.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 msgr2=0x7ff1f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 0x7ff1f006d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff200006fd0 tx=0x7ff200008040 comp rx=0 tx=0).stop 2024-01-16T14:59:31.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 msgr2=0x7ff218140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2080093e0 tx=0x7ff208004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:31.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 shutdown_connections 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff1f006b2c0 0x7ff1f006d780 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff21806efb0 0x7ff21813c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff218146c40 0x7ff21813cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 --2- 172.21.15.43:0/3679652723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff21806bf60 0x7ff218140860 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 >> 172.21.15.43:0/3679652723 conn(0x7ff2180663d0 msgr2=0x7ff21815fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 shutdown_connections 2024-01-16T14:59:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.230+0000 7ff21f800640 1 -- 172.21.15.43:0/3679652723 wait complete. 2024-01-16T14:59:31.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.464+0000 7f99d2c00640 1 Processor -- start 2024-01-16T14:59:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.465+0000 7f99d2c00640 1 -- start start 2024-01-16T14:59:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.466+0000 7f99d2c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99cc0d1900 0x7f99cc0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.466+0000 7f99d2c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc0cf0c0 con 0x7f99cc0d14a0 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.466+0000 7f99d2c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc0b01b0 con 0x7f99cc15d5b0 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.466+0000 7f99d2c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99cc15d990 con 0x7f99cc0d1900 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.466+0000 7f99d2200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99cc0d1900 0x7f99cc0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99d0e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f99cc0d14a0 0x7f99cc0b0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36032/0 (socket says 172.21.15.43:36032) 2024-01-16T14:59:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99d0e00640 1 -- 172.21.15.43:0/1273674716 learned_addr learned my addr 172.21.15.43:0/1273674716 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99d2200640 1 -- 172.21.15.43:0/1273674716 >> v1:172.21.15.69:6789/0 conn(0x7f99cc15d5b0 legacy=0x7f99cc0d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99d2200640 1 -- 172.21.15.43:0/1273674716 >> v1:172.21.15.43:6789/0 conn(0x7f99cc0d14a0 legacy=0x7f99cc0b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99d2200640 1 -- 172.21.15.43:0/1273674716 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc0ceb00 con 0x7f99cc0d1900 2024-01-16T14:59:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.467+0000 7f99cbe00640 1 -- 172.21.15.43:0/1273674716 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983899841 0 0) 0x7f99cc0cf0c0 con 0x7f99cc0d14a0 2024-01-16T14:59:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.468+0000 7f99d2200640 1 --2- 172.21.15.43:0/1273674716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc0d1900 0x7f99cc0b0d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009d90 tx=0x7f99c402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0342c7901d9777f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.468+0000 7f99cbe00640 1 -- 172.21.15.43:0/1273674716 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c402cd00 con 0x7f99cc0d1900 2024-01-16T14:59:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.468+0000 7f99cbe00640 1 -- 172.21.15.43:0/1273674716 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c402ce60 con 0x7f99cc0d1900 2024-01-16T14:59:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.468+0000 7f99cbe00640 1 -- 172.21.15.43:0/1273674716 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4034660 con 0x7f99cc0d1900 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 -- 172.21.15.43:0/1273674716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc0d1900 msgr2=0x7f99cc0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 --2- 172.21.15.43:0/1273674716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc0d1900 0x7f99cc0b0d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009d90 tx=0x7f99c402bf50 comp rx=0 tx=0).stop 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 -- 172.21.15.43:0/1273674716 shutdown_connections 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 --2- 172.21.15.43:0/1273674716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc0d1900 0x7f99cc0b0d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 -- 172.21.15.43:0/1273674716 >> 172.21.15.43:0/1273674716 conn(0x7f99cc09c3b0 msgr2=0x7f99cc0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 -- 172.21.15.43:0/1273674716 shutdown_connections 2024-01-16T14:59:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.469+0000 7f99d2c00640 1 -- 172.21.15.43:0/1273674716 wait complete. 2024-01-16T14:59:31.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.471+0000 7f99d2c00640 1 Processor -- start 2024-01-16T14:59:31.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.471+0000 7f99d2c00640 1 -- start start 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc0d14a0 0x7f99cc0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc0d1900 0x7f99cc14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1f70d0 con 0x7f99cc15d5b0 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1f6490 con 0x7f99cc0d14a0 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1f6730 con 0x7f99cc0d1900 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58288/0 (socket says 172.21.15.43:58288) 2024-01-16T14:59:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.472+0000 7f99d2200640 1 -- 172.21.15.43:0/4006367375 learned_addr learned my addr 172.21.15.43:0/4006367375 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d1800640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc0d14a0 0x7f99cc0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 -- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc0d1900 msgr2=0x7f99cc14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc0d1900 0x7f99cc14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 -- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc0d14a0 msgr2=0x7f99cc0c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc0d14a0 0x7f99cc0c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 -- 172.21.15.43:0/4006367375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4009a70 con 0x7f99cc15d5b0 2024-01-16T14:59:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2200640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009eb0 tx=0x7f99c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c403d600 con 0x7f99cc15d5b0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c403d780 con 0x7f99cc15d5b0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c403db30 con 0x7f99cc15d5b0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc1f6a10 con 0x7f99cc15d5b0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.473+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc15eb30 con 0x7f99cc15d5b0 2024-01-16T14:59:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.475+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc1f7210 con 0x7f99cc15d5b0 2024-01-16T14:59:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.476+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f99c403b040 con 0x7f99cc15d5b0 2024-01-16T14:59:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.476+0000 7f99c9600640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 0x7f99a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.477+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99c40ac280 con 0x7f99cc15d5b0 2024-01-16T14:59:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.477+0000 7f99d1800640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 0x7f99a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.477+0000 7f99d1800640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 0x7f99a406d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000f60 tx=0x7f99bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.481+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99cc1f7210 con 0x7f99cc15d5b0 2024-01-16T14:59:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.675+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99cc1f7210 con 0x7f99cc15d5b0 2024-01-16T14:59:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.678+0000 7f99c9600640 1 -- 172.21.15.43:0/4006367375 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f99cc1f7210 con 0x7f99cc15d5b0 2024-01-16T14:59:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:31.660 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 msgr2=0x7f99a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 0x7f99a406d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000f60 tx=0x7f99bc009040 comp rx=0 tx=0).stop 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 msgr2=0x7f99cc15e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009eb0 tx=0x7f99c4004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 shutdown_connections 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a406b2c0 0x7f99a406d780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99cc15d5b0 0x7f99cc15e5f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc0d1900 0x7f99cc14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 --2- 172.21.15.43:0/4006367375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc0d14a0 0x7f99cc0c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 >> 172.21.15.43:0/4006367375 conn(0x7f99cc09c3b0 msgr2=0x7f99cc0d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 shutdown_connections 2024-01-16T14:59:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.681+0000 7f99d2c00640 1 -- 172.21.15.43:0/4006367375 wait complete. 2024-01-16T14:59:31.681 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T14:59:31.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:31.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.909+0000 7f1b23e00640 1 Processor -- start 2024-01-16T14:59:31.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.909+0000 7f1b23e00640 1 -- start start 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b240cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24140900 con 0x7f1b240cd7b0 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2415fa70 con 0x7f1b2413dfd0 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2415c580 con 0x7f1b240cf050 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1b240cd7b0 0x7f1b240cdc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36038/0 (socket says 172.21.15.43:36038) 2024-01-16T14:59:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.911+0000 7f1b23400640 1 -- 172.21.15.43:0/4036202755 learned_addr learned my addr 172.21.15.43:0/4036202755 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b19e00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b240cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239070082 0 0) 0x7f1b24140900 con 0x7f1b240cd7b0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003520 con 0x7f1b240cd7b0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1637528325 0 0) 0x7f1b04003520 con 0x7f1b240cd7b0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 msgr2=0x7f1b240cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b240cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 >> v1:172.21.15.69:6789/0 conn(0x7f1b2413dfd0 legacy=0x7f1b2415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2415ee60 con 0x7f1b240cd7b0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f1b1c001670 con 0x7f1b240cd7b0 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 >> v1:172.21.15.43:6789/0 conn(0x7f1b240cd7b0 legacy=0x7f1b240cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b04003a10 0x7f1b04003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2415c580 con 0x7f1b04003a10 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0400bb10 con 0x7f1b04004d00 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.912+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0400bca0 con 0x7f1b240cf050 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b04003a10 0x7f1b04003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 -- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 msgr2=0x7f1b040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b19e00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 -- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04004d00 msgr2=0x7f1b04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b23400640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04004d00 0x7f1b04005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b22a00640 1 -- 172.21.15.43:0/4036202755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04003520 con 0x7f1b04003a10 2024-01-16T14:59:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.913+0000 7f1b19e00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b240cf050 0x7f1b040047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:31.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.914+0000 7f1b22a00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b04003a10 0x7f1b04003e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b14001e50 tx=0x7f1b040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317f060d4ed15e79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003b70 con 0x7f1b04003a10 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b14003cd0 con 0x7f1b04003a10 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b22000640 1 -- 172.21.15.43:0/4036202755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14005310 con 0x7f1b04003a10 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 -- 172.21.15.43:0/4036202755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b04003a10 msgr2=0x7f1b04003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 --2- 172.21.15.43:0/4036202755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b04003a10 0x7f1b04003e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b14001e50 tx=0x7f1b040057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23400640 1 -- 172.21.15.43:0/4036202755 reap_dead start 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 -- 172.21.15.43:0/4036202755 shutdown_connections 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 -- 172.21.15.43:0/4036202755 >> 172.21.15.43:0/4036202755 conn(0x7f1b2406a860 msgr2=0x7f1b2406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 -- 172.21.15.43:0/4036202755 shutdown_connections 2024-01-16T14:59:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.915+0000 7f1b23e00640 1 -- 172.21.15.43:0/4036202755 wait complete. 2024-01-16T14:59:31.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.917+0000 7f1b23e00640 1 Processor -- start 2024-01-16T14:59:31.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.917+0000 7f1b23e00640 1 -- start start 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 0x7f1b240c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b2413b750 0x7f1b241565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24159820 con 0x7f1b24158780 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24159b00 con 0x7f1b04003a10 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.918+0000 7f1b23e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24159de0 con 0x7f1b2413b750 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58300/0 (socket says 172.21.15.43:58300) 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 -- 172.21.15.43:0/3918292697 learned_addr learned my addr 172.21.15.43:0/3918292697 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b23400640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b2413b750 0x7f1b241565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b22a00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 0x7f1b240c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 -- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b2413b750 msgr2=0x7f1b241565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b2413b750 0x7f1b241565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 -- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 msgr2=0x7f1b240c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 0x7f1b240c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 -- 172.21.15.43:0/3918292697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2415a0c0 con 0x7f1b24158780 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b22a00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 0x7f1b240c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.919+0000 7f1b19e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0108b0 tx=0x7f1b0c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.920+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c00eb60 con 0x7f1b24158780 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.920+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c00ecc0 con 0x7f1b24158780 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.920+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c01c400 con 0x7f1b24158780 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.920+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2415a3a0 con 0x7f1b24158780 2024-01-16T14:59:31.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.920+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2413be50 con 0x7f1b24158780 2024-01-16T14:59:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.921+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b24155a00 con 0x7f1b24158780 2024-01-16T14:59:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.922+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1b0c01c560 con 0x7f1b24158780 2024-01-16T14:59:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.923+0000 7f1b1b400640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 0x7f1afc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.923+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b0c08a180 con 0x7f1b24158780 2024-01-16T14:59:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.923+0000 7f1b22a00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 0x7f1afc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:31.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.924+0000 7f1b22a00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 0x7f1afc06d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:31.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:31.927+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b24155a00 con 0x7f1b24158780 2024-01-16T14:59:32.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.116+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b2415f700 con 0x7f1afc06b2c0 2024-01-16T14:59:32.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.141+0000 7f1b1b400640 1 -- 172.21.15.43:0/3918292697 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264517 (secure 0 0 0) 0x7f1b14007db0 con 0x7f1afc06b2c0 2024-01-16T14:59:32.126 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:32.127 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T14:59:32.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 msgr2=0x7f1afc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 0x7f1afc06d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14009030 comp rx=0 tx=0).stop 2024-01-16T14:59:32.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 msgr2=0x7f1b2412e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0108b0 tx=0x7f1b0c010d70 comp rx=0 tx=0).stop 2024-01-16T14:59:32.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 shutdown_connections 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1afc06b2c0 0x7f1afc06d780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b2413b750 0x7f1b241565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b24158780 0x7f1b2412e430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 --2- 172.21.15.43:0/3918292697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b04003a10 0x7f1b240c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.147+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 >> 172.21.15.43:0/3918292697 conn(0x7f1b2406a860 msgr2=0x7f1b24071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.148+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 shutdown_connections 2024-01-16T14:59:32.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.148+0000 7f1b23e00640 1 -- 172.21.15.43:0/3918292697 wait complete. 2024-01-16T14:59:32.149 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:59:30.788128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240352,"kb_used_data":2960,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734368,"statfs":{"total":773094113280,"available":772847992832,"internally_reserved":0,"allocated":3031040,"data_stored":1803372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137704"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908219+0000","last_change":"2024-01-16T14:59:27.733425+0000","last_active":"2024-01-16T14:59:28.908219+0000","last_peered":"2024-01-16T14:59:28.908219+0000","last_clean":"2024-01-16T14:59:28.908219+0000","last_became_active":"2024-01-16T14:59:27.733108+0000","last_became_peered":"2024-01-16T14:59:27.733108+0000","last_unstale":"2024-01-16T14:59:28.908219+0000","last_undegraded":"2024-01-16T14:59:28.908219+0000","last_fullsized":"2024-01-16T14:59:28.908219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:28:12.476929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908249+0000","last_change":"2024-01-16T14:59:27.715575+0000","last_active":"2024-01-16T14:59:28.908249+0000","last_peered":"2024-01-16T14:59:28.908249+0000","last_clean":"2024-01-16T14:59:28.908249+0000","last_became_active":"2024-01-16T14:59:27.714841+0000","last_became_peered":"2024-01-16T14:59:27.714841+0000","last_unstale":"2024-01-16T14:59:28.908249+0000","last_undegraded":"2024-01-16T14:59:28.908249+0000","last_fullsized":"2024-01-16T14:59:28.908249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908551+0000","last_change":"2024-01-16T14:59:27.718807+0000","last_active":"2024-01-16T14:59:28.908551+0000","last_peered":"2024-01-16T14:59:28.908551+0000","last_clean":"2024-01-16T14:59:28.908551+0000","last_became_active":"2024-01-16T14:59:27.717301+0000","last_became_peered":"2024-01-16T14:59:27.717301+0000","last_unstale":"2024-01-16T14:59:28.908551+0000","last_undegraded":"2024-01-16T14:59:28.908551+0000","last_fullsized":"2024-01-16T14:59:28.908551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906129+0000","last_change":"2024-01-16T14:59:27.720914+0000","last_active":"2024-01-16T14:59:28.906129+0000","last_peered":"2024-01-16T14:59:28.906129+0000","last_clean":"2024-01-16T14:59:28.906129+0000","last_became_active":"2024-01-16T14:59:27.720561+0000","last_became_peered":"2024-01-16T14:59:27.720561+0000","last_unstale":"2024-01-16T14:59:28.906129+0000","last_undegraded":"2024-01-16T14:59:28.906129+0000","last_fullsized":"2024-01-16T14:59:28.906129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:16.656352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906008+0000","last_change":"2024-01-16T14:59:27.721194+0000","last_active":"2024-01-16T14:59:28.906008+0000","last_peered":"2024-01-16T14:59:28.906008+0000","last_clean":"2024-01-16T14:59:28.906008+0000","last_became_active":"2024-01-16T14:59:27.719980+0000","last_became_peered":"2024-01-16T14:59:27.719980+0000","last_unstale":"2024-01-16T14:59:28.906008+0000","last_undegraded":"2024-01-16T14:59:28.906008+0000","last_fullsized":"2024-01-16T14:59:28.906008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:54.580479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905723+0000","last_change":"2024-01-16T14:59:27.732776+0000","last_active":"2024-01-16T14:59:28.905723+0000","last_peered":"2024-01-16T14:59:28.905723+0000","last_clean":"2024-01-16T14:59:28.905723+0000","last_became_active":"2024-01-16T14:59:27.732373+0000","last_became_peered":"2024-01-16T14:59:27.732373+0000","last_unstale":"2024-01-16T14:59:28.905723+0000","last_undegraded":"2024-01-16T14:59:28.905723+0000","last_fullsized":"2024-01-16T14:59:28.905723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904627+0000","last_change":"2024-01-16T14:59:27.722159+0000","last_active":"2024-01-16T14:59:28.904627+0000","last_peered":"2024-01-16T14:59:28.904627+0000","last_clean":"2024-01-16T14:59:28.904627+0000","last_became_active":"2024-01-16T14:59:27.721928+0000","last_became_peered":"2024-01-16T14:59:27.721928+0000","last_unstale":"2024-01-16T14:59:28.904627+0000","last_undegraded":"2024-01-16T14:59:28.904627+0000","last_fullsized":"2024-01-16T14:59:28.904627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:17.023676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904587+0000","last_change":"2024-01-16T14:59:27.721172+0000","last_active":"2024-01-16T14:59:28.904587+0000","last_peered":"2024-01-16T14:59:28.904587+0000","last_clean":"2024-01-16T14:59:28.904587+0000","last_became_active":"2024-01-16T14:59:27.720829+0000","last_became_peered":"2024-01-16T14:59:27.720829+0000","last_unstale":"2024-01-16T14:59:28.904587+0000","last_undegraded":"2024-01-16T14:59:28.904587+0000","last_fullsized":"2024-01-16T14:59:28.904587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:39.594220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908493+0000","last_change":"2024-01-16T14:59:27.721916+0000","last_active":"2024-01-16T14:59:28.908493+0000","last_peered":"2024-01-16T14:59:28.908493+0000","last_clean":"2024-01-16T14:59:28.908493+0000","last_became_active":"2024-01-16T14:59:27.721604+0000","last_became_peered":"2024-01-16T14:59:27.721604+0000","last_unstale":"2024-01-16T14:59:28.908493+0000","last_undegraded":"2024-01-16T14:59:28.908493+0000","last_fullsized":"2024-01-16T14:59:28.908493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:05.858739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711127+0000","last_change":"2024-01-16T14:59:27.729823+0000","last_active":"2024-01-16T14:59:28.711127+0000","last_peered":"2024-01-16T14:59:28.711127+0000","last_clean":"2024-01-16T14:59:28.711127+0000","last_became_active":"2024-01-16T14:59:27.724755+0000","last_became_peered":"2024-01-16T14:59:27.724755+0000","last_unstale":"2024-01-16T14:59:28.711127+0000","last_undegraded":"2024-01-16T14:59:28.711127+0000","last_fullsized":"2024-01-16T14:59:28.711127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710385+0000","last_change":"2024-01-16T14:59:27.725448+0000","last_active":"2024-01-16T14:59:28.710385+0000","last_peered":"2024-01-16T14:59:28.710385+0000","last_clean":"2024-01-16T14:59:28.710385+0000","last_became_active":"2024-01-16T14:59:27.724294+0000","last_became_peered":"2024-01-16T14:59:27.724294+0000","last_unstale":"2024-01-16T14:59:28.710385+0000","last_undegraded":"2024-01-16T14:59:28.710385+0000","last_fullsized":"2024-01-16T14:59:28.710385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709647+0000","last_change":"2024-01-16T14:59:27.725056+0000","last_active":"2024-01-16T14:59:28.709647+0000","last_peered":"2024-01-16T14:59:28.709647+0000","last_clean":"2024-01-16T14:59:28.709647+0000","last_became_active":"2024-01-16T14:59:27.724771+0000","last_became_peered":"2024-01-16T14:59:27.724771+0000","last_unstale":"2024-01-16T14:59:28.709647+0000","last_undegraded":"2024-01-16T14:59:28.709647+0000","last_fullsized":"2024-01-16T14:59:28.709647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905952+0000","last_change":"2024-01-16T14:59:27.726131+0000","last_active":"2024-01-16T14:59:28.905952+0000","last_peered":"2024-01-16T14:59:28.905952+0000","last_clean":"2024-01-16T14:59:28.905952+0000","last_became_active":"2024-01-16T14:59:27.725830+0000","last_became_peered":"2024-01-16T14:59:27.725830+0000","last_unstale":"2024-01-16T14:59:28.905952+0000","last_undegraded":"2024-01-16T14:59:28.905952+0000","last_fullsized":"2024-01-16T14:59:28.905952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:45.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906705+0000","last_change":"2024-01-16T14:59:27.713525+0000","last_active":"2024-01-16T14:59:28.906705+0000","last_peered":"2024-01-16T14:59:28.906705+0000","last_clean":"2024-01-16T14:59:28.906705+0000","last_became_active":"2024-01-16T14:59:27.712762+0000","last_became_peered":"2024-01-16T14:59:27.712762+0000","last_unstale":"2024-01-16T14:59:28.906705+0000","last_undegraded":"2024-01-16T14:59:28.906705+0000","last_fullsized":"2024-01-16T14:59:28.906705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:41.373234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906179+0000","last_change":"2024-01-16T14:59:27.700808+0000","last_active":"2024-01-16T14:59:28.906179+0000","last_peered":"2024-01-16T14:59:28.906179+0000","last_clean":"2024-01-16T14:59:28.906179+0000","last_became_active":"2024-01-16T14:59:27.700477+0000","last_became_peered":"2024-01-16T14:59:27.700477+0000","last_unstale":"2024-01-16T14:59:28.906179+0000","last_undegraded":"2024-01-16T14:59:28.906179+0000","last_fullsized":"2024-01-16T14:59:28.906179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905239+0000","last_change":"2024-01-16T14:59:27.720864+0000","last_active":"2024-01-16T14:59:28.905239+0000","last_peered":"2024-01-16T14:59:28.905239+0000","last_clean":"2024-01-16T14:59:28.905239+0000","last_became_active":"2024-01-16T14:59:27.719599+0000","last_became_peered":"2024-01-16T14:59:27.719599+0000","last_unstale":"2024-01-16T14:59:28.905239+0000","last_undegraded":"2024-01-16T14:59:28.905239+0000","last_fullsized":"2024-01-16T14:59:28.905239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907987+0000","last_change":"2024-01-16T14:59:27.717946+0000","last_active":"2024-01-16T14:59:28.907987+0000","last_peered":"2024-01-16T14:59:28.907987+0000","last_clean":"2024-01-16T14:59:28.907987+0000","last_became_active":"2024-01-16T14:59:27.717711+0000","last_became_peered":"2024-01-16T14:59:27.717711+0000","last_unstale":"2024-01-16T14:59:28.907987+0000","last_undegraded":"2024-01-16T14:59:28.907987+0000","last_fullsized":"2024-01-16T14:59:28.907987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:03.931690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703527+0000","last_change":"2024-01-16T14:59:27.726121+0000","last_active":"2024-01-16T14:59:28.703527+0000","last_peered":"2024-01-16T14:59:28.703527+0000","last_clean":"2024-01-16T14:59:28.703527+0000","last_became_active":"2024-01-16T14:59:27.725643+0000","last_became_peered":"2024-01-16T14:59:27.725643+0000","last_unstale":"2024-01-16T14:59:28.703527+0000","last_undegraded":"2024-01-16T14:59:28.703527+0000","last_fullsized":"2024-01-16T14:59:28.703527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907957+0000","last_change":"2024-01-16T14:59:27.720619+0000","last_active":"2024-01-16T14:59:28.907957+0000","last_peered":"2024-01-16T14:59:28.907957+0000","last_clean":"2024-01-16T14:59:28.907957+0000","last_became_active":"2024-01-16T14:59:27.720348+0000","last_became_peered":"2024-01-16T14:59:27.720348+0000","last_unstale":"2024-01-16T14:59:28.907957+0000","last_undegraded":"2024-01-16T14:59:28.907957+0000","last_fullsized":"2024-01-16T14:59:28.907957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:58.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710390+0000","last_change":"2024-01-16T14:59:27.727783+0000","last_active":"2024-01-16T14:59:28.710390+0000","last_peered":"2024-01-16T14:59:28.710390+0000","last_clean":"2024-01-16T14:59:28.710390+0000","last_became_active":"2024-01-16T14:59:27.727562+0000","last_became_peered":"2024-01-16T14:59:27.727562+0000","last_unstale":"2024-01-16T14:59:28.710390+0000","last_undegraded":"2024-01-16T14:59:28.710390+0000","last_fullsized":"2024-01-16T14:59:28.710390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907372+0000","last_change":"2024-01-16T14:59:27.724755+0000","last_active":"2024-01-16T14:59:28.907372+0000","last_peered":"2024-01-16T14:59:28.907372+0000","last_clean":"2024-01-16T14:59:28.907372+0000","last_became_active":"2024-01-16T14:59:27.724443+0000","last_became_peered":"2024-01-16T14:59:27.724443+0000","last_unstale":"2024-01-16T14:59:28.907372+0000","last_undegraded":"2024-01-16T14:59:28.907372+0000","last_fullsized":"2024-01-16T14:59:28.907372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:06.824346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906082+0000","last_change":"2024-01-16T14:59:27.732620+0000","last_active":"2024-01-16T14:59:28.906082+0000","last_peered":"2024-01-16T14:59:28.906082+0000","last_clean":"2024-01-16T14:59:28.906082+0000","last_became_active":"2024-01-16T14:59:27.732037+0000","last_became_peered":"2024-01-16T14:59:27.732037+0000","last_unstale":"2024-01-16T14:59:28.906082+0000","last_undegraded":"2024-01-16T14:59:28.906082+0000","last_fullsized":"2024-01-16T14:59:28.906082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:13.794976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905598+0000","last_change":"2024-01-16T14:59:27.719539+0000","last_active":"2024-01-16T14:59:28.905598+0000","last_peered":"2024-01-16T14:59:28.905598+0000","last_clean":"2024-01-16T14:59:28.905598+0000","last_became_active":"2024-01-16T14:59:27.719253+0000","last_became_peered":"2024-01-16T14:59:27.719253+0000","last_unstale":"2024-01-16T14:59:28.905598+0000","last_undegraded":"2024-01-16T14:59:28.905598+0000","last_fullsized":"2024-01-16T14:59:28.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:42:03.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905607+0000","last_change":"2024-01-16T14:59:27.715773+0000","last_active":"2024-01-16T14:59:28.905607+0000","last_peered":"2024-01-16T14:59:28.905607+0000","last_clean":"2024-01-16T14:59:28.905607+0000","last_became_active":"2024-01-16T14:59:27.715482+0000","last_became_peered":"2024-01-16T14:59:27.715482+0000","last_unstale":"2024-01-16T14:59:28.905607+0000","last_undegraded":"2024-01-16T14:59:28.905607+0000","last_fullsized":"2024-01-16T14:59:28.905607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906301+0000","last_change":"2024-01-16T14:59:27.726710+0000","last_active":"2024-01-16T14:59:28.906301+0000","last_peered":"2024-01-16T14:59:28.906301+0000","last_clean":"2024-01-16T14:59:28.906301+0000","last_became_active":"2024-01-16T14:59:27.723820+0000","last_became_peered":"2024-01-16T14:59:27.723820+0000","last_unstale":"2024-01-16T14:59:28.906301+0000","last_undegraded":"2024-01-16T14:59:28.906301+0000","last_fullsized":"2024-01-16T14:59:28.906301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:10.938227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711055+0000","last_change":"2024-01-16T14:59:27.726914+0000","last_active":"2024-01-16T14:59:28.711055+0000","last_peered":"2024-01-16T14:59:28.711055+0000","last_clean":"2024-01-16T14:59:28.711055+0000","last_became_active":"2024-01-16T14:59:27.725638+0000","last_became_peered":"2024-01-16T14:59:27.725638+0000","last_unstale":"2024-01-16T14:59:28.711055+0000","last_undegraded":"2024-01-16T14:59:28.711055+0000","last_fullsized":"2024-01-16T14:59:28.711055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907822+0000","last_change":"2024-01-16T14:59:27.703634+0000","last_active":"2024-01-16T14:59:28.907822+0000","last_peered":"2024-01-16T14:59:28.907822+0000","last_clean":"2024-01-16T14:59:28.907822+0000","last_became_active":"2024-01-16T14:59:27.703343+0000","last_became_peered":"2024-01-16T14:59:27.703343+0000","last_unstale":"2024-01-16T14:59:28.907822+0000","last_undegraded":"2024-01-16T14:59:28.907822+0000","last_fullsized":"2024-01-16T14:59:28.907822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906778+0000","last_change":"2024-01-16T14:59:27.724563+0000","last_active":"2024-01-16T14:59:28.906778+0000","last_peered":"2024-01-16T14:59:28.906778+0000","last_clean":"2024-01-16T14:59:28.906778+0000","last_became_active":"2024-01-16T14:59:27.723074+0000","last_became_peered":"2024-01-16T14:59:27.723074+0000","last_unstale":"2024-01-16T14:59:28.906778+0000","last_undegraded":"2024-01-16T14:59:28.906778+0000","last_fullsized":"2024-01-16T14:59:28.906778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:05.783445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907568+0000","last_change":"2024-01-16T14:59:27.724629+0000","last_active":"2024-01-16T14:59:28.907568+0000","last_peered":"2024-01-16T14:59:28.907568+0000","last_clean":"2024-01-16T14:59:28.907568+0000","last_became_active":"2024-01-16T14:59:27.724363+0000","last_became_peered":"2024-01-16T14:59:27.724363+0000","last_unstale":"2024-01-16T14:59:28.907568+0000","last_undegraded":"2024-01-16T14:59:28.907568+0000","last_fullsized":"2024-01-16T14:59:28.907568+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:48.253356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906430+0000","last_change":"2024-01-16T14:59:27.722940+0000","last_active":"2024-01-16T14:59:28.906430+0000","last_peered":"2024-01-16T14:59:28.906430+0000","last_clean":"2024-01-16T14:59:28.906430+0000","last_became_active":"2024-01-16T14:59:27.722567+0000","last_became_peered":"2024-01-16T14:59:27.722567+0000","last_unstale":"2024-01-16T14:59:28.906430+0000","last_undegraded":"2024-01-16T14:59:28.906430+0000","last_fullsized":"2024-01-16T14:59:28.906430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:13.739812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703087+0000","last_change":"2024-01-16T14:59:27.722429+0000","last_active":"2024-01-16T14:59:28.703087+0000","last_peered":"2024-01-16T14:59:28.703087+0000","last_clean":"2024-01-16T14:59:28.703087+0000","last_became_active":"2024-01-16T14:59:27.722092+0000","last_became_peered":"2024-01-16T14:59:27.722092+0000","last_unstale":"2024-01-16T14:59:28.703087+0000","last_undegraded":"2024-01-16T14:59:28.703087+0000","last_fullsized":"2024-01-16T14:59:28.703087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709008+0000","last_change":"2024-01-16T14:59:27.719513+0000","last_active":"2024-01-16T14:59:28.709008+0000","last_peered":"2024-01-16T14:59:28.709008+0000","last_clean":"2024-01-16T14:59:28.709008+0000","last_became_active":"2024-01-16T14:59:27.718757+0000","last_became_peered":"2024-01-16T14:59:27.718757+0000","last_unstale":"2024-01-16T14:59:28.709008+0000","last_undegraded":"2024-01-16T14:59:28.709008+0000","last_fullsized":"2024-01-16T14:59:28.709008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.707752+0000","last_change":"2024-01-16T14:59:27.706529+0000","last_active":"2024-01-16T14:59:28.707752+0000","last_peered":"2024-01-16T14:59:28.707752+0000","last_clean":"2024-01-16T14:59:28.707752+0000","last_became_active":"2024-01-16T14:59:27.706241+0000","last_became_peered":"2024-01-16T14:59:27.706241+0000","last_unstale":"2024-01-16T14:59:28.707752+0000","last_undegraded":"2024-01-16T14:59:28.707752+0000","last_fullsized":"2024-01-16T14:59:28.707752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:40.765365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906601+0000","last_change":"2024-01-16T14:59:26.697929+0000","last_active":"2024-01-16T14:59:28.906601+0000","last_peered":"2024-01-16T14:59:28.906601+0000","last_clean":"2024-01-16T14:59:28.906601+0000","last_became_active":"2024-01-16T14:59:26.697578+0000","last_became_peered":"2024-01-16T14:59:26.697578+0000","last_unstale":"2024-01-16T14:59:28.906601+0000","last_undegraded":"2024-01-16T14:59:28.906601+0000","last_fullsized":"2024-01-16T14:59:28.906601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:26.816846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907687+0000","last_change":"2024-01-16T14:59:27.723311+0000","last_active":"2024-01-16T14:59:28.907687+0000","last_peered":"2024-01-16T14:59:28.907687+0000","last_clean":"2024-01-16T14:59:28.907687+0000","last_became_active":"2024-01-16T14:59:27.722975+0000","last_became_peered":"2024-01-16T14:59:27.722975+0000","last_unstale":"2024-01-16T14:59:28.907687+0000","last_undegraded":"2024-01-16T14:59:28.907687+0000","last_fullsized":"2024-01-16T14:59:28.907687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710186+0000","last_change":"2024-01-16T14:59:26.683247+0000","last_active":"2024-01-16T14:59:28.710186+0000","last_peered":"2024-01-16T14:59:28.710186+0000","last_clean":"2024-01-16T14:59:28.710186+0000","last_became_active":"2024-01-16T14:59:26.682727+0000","last_became_peered":"2024-01-16T14:59:26.682727+0000","last_unstale":"2024-01-16T14:59:28.710186+0000","last_undegraded":"2024-01-16T14:59:28.710186+0000","last_fullsized":"2024-01-16T14:59:28.710186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905763+0000","last_change":"2024-01-16T14:59:26.687905+0000","last_active":"2024-01-16T14:59:28.905763+0000","last_peered":"2024-01-16T14:59:28.905763+0000","last_clean":"2024-01-16T14:59:28.905763+0000","last_became_active":"2024-01-16T14:59:26.687570+0000","last_became_peered":"2024-01-16T14:59:26.687570+0000","last_unstale":"2024-01-16T14:59:28.905763+0000","last_undegraded":"2024-01-16T14:59:28.905763+0000","last_fullsized":"2024-01-16T14:59:28.905763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904948+0000","last_change":"2024-01-16T14:59:27.715786+0000","last_active":"2024-01-16T14:59:28.904948+0000","last_peered":"2024-01-16T14:59:28.904948+0000","last_clean":"2024-01-16T14:59:28.904948+0000","last_became_active":"2024-01-16T14:59:27.715516+0000","last_became_peered":"2024-01-16T14:59:27.715516+0000","last_unstale":"2024-01-16T14:59:28.904948+0000","last_undegraded":"2024-01-16T14:59:28.904948+0000","last_fullsized":"2024-01-16T14:59:28.904948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:14.095920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905832+0000","last_change":"2024-01-16T14:59:26.698294+0000","last_active":"2024-01-16T14:59:28.905832+0000","last_peered":"2024-01-16T14:59:28.905832+0000","last_clean":"2024-01-16T14:59:28.905832+0000","last_became_active":"2024-01-16T14:59:26.697917+0000","last_became_peered":"2024-01-16T14:59:26.697917+0000","last_unstale":"2024-01-16T14:59:28.905832+0000","last_undegraded":"2024-01-16T14:59:28.905832+0000","last_fullsized":"2024-01-16T14:59:28.905832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:15.675945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905800+0000","last_change":"2024-01-16T14:59:27.704715+0000","last_active":"2024-01-16T14:59:28.905800+0000","last_peered":"2024-01-16T14:59:28.905800+0000","last_clean":"2024-01-16T14:59:28.905800+0000","last_became_active":"2024-01-16T14:59:27.704439+0000","last_became_peered":"2024-01-16T14:59:27.704439+0000","last_unstale":"2024-01-16T14:59:28.905800+0000","last_undegraded":"2024-01-16T14:59:28.905800+0000","last_fullsized":"2024-01-16T14:59:28.905800+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710792+0000","last_change":"2024-01-16T14:59:26.689740+0000","last_active":"2024-01-16T14:59:28.710792+0000","last_peered":"2024-01-16T14:59:28.710792+0000","last_clean":"2024-01-16T14:59:28.710792+0000","last_became_active":"2024-01-16T14:59:26.689434+0000","last_became_peered":"2024-01-16T14:59:26.689434+0000","last_unstale":"2024-01-16T14:59:28.710792+0000","last_undegraded":"2024-01-16T14:59:28.710792+0000","last_fullsized":"2024-01-16T14:59:28.710792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:39.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907111+0000","last_change":"2024-01-16T14:59:27.733517+0000","last_active":"2024-01-16T14:59:28.907111+0000","last_peered":"2024-01-16T14:59:28.907111+0000","last_clean":"2024-01-16T14:59:28.907111+0000","last_became_active":"2024-01-16T14:59:27.732223+0000","last_became_peered":"2024-01-16T14:59:27.732223+0000","last_unstale":"2024-01-16T14:59:28.907111+0000","last_undegraded":"2024-01-16T14:59:28.907111+0000","last_fullsized":"2024-01-16T14:59:28.907111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:55.788928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907375+0000","last_change":"2024-01-16T14:59:26.693230+0000","last_active":"2024-01-16T14:59:28.907375+0000","last_peered":"2024-01-16T14:59:28.907375+0000","last_clean":"2024-01-16T14:59:28.907375+0000","last_became_active":"2024-01-16T14:59:26.693007+0000","last_became_peered":"2024-01-16T14:59:26.693007+0000","last_unstale":"2024-01-16T14:59:28.907375+0000","last_undegraded":"2024-01-16T14:59:28.907375+0000","last_fullsized":"2024-01-16T14:59:28.907375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909099+0000","last_change":"2024-01-16T14:59:27.723932+0000","last_active":"2024-01-16T14:59:28.909099+0000","last_peered":"2024-01-16T14:59:28.909099+0000","last_clean":"2024-01-16T14:59:28.909099+0000","last_became_active":"2024-01-16T14:59:27.723629+0000","last_became_peered":"2024-01-16T14:59:27.723629+0000","last_unstale":"2024-01-16T14:59:28.909099+0000","last_undegraded":"2024-01-16T14:59:28.909099+0000","last_fullsized":"2024-01-16T14:59:28.909099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908417+0000","last_change":"2024-01-16T14:59:26.698500+0000","last_active":"2024-01-16T14:59:28.908417+0000","last_peered":"2024-01-16T14:59:28.908417+0000","last_clean":"2024-01-16T14:59:28.908417+0000","last_became_active":"2024-01-16T14:59:26.698193+0000","last_became_peered":"2024-01-16T14:59:26.698193+0000","last_unstale":"2024-01-16T14:59:28.908417+0000","last_undegraded":"2024-01-16T14:59:28.908417+0000","last_fullsized":"2024-01-16T14:59:28.908417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:32.268892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709922+0000","last_change":"2024-01-16T14:59:27.724449+0000","last_active":"2024-01-16T14:59:28.709922+0000","last_peered":"2024-01-16T14:59:28.709922+0000","last_clean":"2024-01-16T14:59:28.709922+0000","last_became_active":"2024-01-16T14:59:27.723990+0000","last_became_peered":"2024-01-16T14:59:27.723990+0000","last_unstale":"2024-01-16T14:59:28.709922+0000","last_undegraded":"2024-01-16T14:59:28.709922+0000","last_fullsized":"2024-01-16T14:59:28.709922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710357+0000","last_change":"2024-01-16T14:59:26.688962+0000","last_active":"2024-01-16T14:59:28.710357+0000","last_peered":"2024-01-16T14:59:28.710357+0000","last_clean":"2024-01-16T14:59:28.710357+0000","last_became_active":"2024-01-16T14:59:26.687694+0000","last_became_peered":"2024-01-16T14:59:26.687694+0000","last_unstale":"2024-01-16T14:59:28.710357+0000","last_undegraded":"2024-01-16T14:59:28.710357+0000","last_fullsized":"2024-01-16T14:59:28.710357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907384+0000","last_change":"2024-01-16T14:59:27.718245+0000","last_active":"2024-01-16T14:59:28.907384+0000","last_peered":"2024-01-16T14:59:28.907384+0000","last_clean":"2024-01-16T14:59:28.907384+0000","last_became_active":"2024-01-16T14:59:27.717994+0000","last_became_peered":"2024-01-16T14:59:27.717994+0000","last_unstale":"2024-01-16T14:59:28.907384+0000","last_undegraded":"2024-01-16T14:59:28.907384+0000","last_fullsized":"2024-01-16T14:59:28.907384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:37.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907794+0000","last_change":"2024-01-16T14:59:26.682235+0000","last_active":"2024-01-16T14:59:28.907794+0000","last_peered":"2024-01-16T14:59:28.907794+0000","last_clean":"2024-01-16T14:59:28.907794+0000","last_became_active":"2024-01-16T14:59:26.681912+0000","last_became_peered":"2024-01-16T14:59:26.681912+0000","last_unstale":"2024-01-16T14:59:28.907794+0000","last_undegraded":"2024-01-16T14:59:28.907794+0000","last_fullsized":"2024-01-16T14:59:28.907794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906880+0000","last_change":"2024-01-16T14:59:27.718808+0000","last_active":"2024-01-16T14:59:28.906880+0000","last_peered":"2024-01-16T14:59:28.906880+0000","last_clean":"2024-01-16T14:59:28.906880+0000","last_became_active":"2024-01-16T14:59:27.717611+0000","last_became_peered":"2024-01-16T14:59:27.717611+0000","last_unstale":"2024-01-16T14:59:28.906880+0000","last_undegraded":"2024-01-16T14:59:28.906880+0000","last_fullsized":"2024-01-16T14:59:28.906880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:37.694723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906417+0000","last_change":"2024-01-16T14:59:26.701196+0000","last_active":"2024-01-16T14:59:28.906417+0000","last_peered":"2024-01-16T14:59:28.906417+0000","last_clean":"2024-01-16T14:59:28.906417+0000","last_became_active":"2024-01-16T14:59:26.700600+0000","last_became_peered":"2024-01-16T14:59:26.700600+0000","last_unstale":"2024-01-16T14:59:28.906417+0000","last_undegraded":"2024-01-16T14:59:28.906417+0000","last_fullsized":"2024-01-16T14:59:28.906417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:20.742041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906388+0000","last_change":"2024-01-16T14:59:27.729587+0000","last_active":"2024-01-16T14:59:28.906388+0000","last_peered":"2024-01-16T14:59:28.906388+0000","last_clean":"2024-01-16T14:59:28.906388+0000","last_became_active":"2024-01-16T14:59:27.729374+0000","last_became_peered":"2024-01-16T14:59:27.729374+0000","last_unstale":"2024-01-16T14:59:28.906388+0000","last_undegraded":"2024-01-16T14:59:28.906388+0000","last_fullsized":"2024-01-16T14:59:28.906388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:07.977388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709779+0000","last_change":"2024-01-16T14:59:26.682595+0000","last_active":"2024-01-16T14:59:28.709779+0000","last_peered":"2024-01-16T14:59:28.709779+0000","last_clean":"2024-01-16T14:59:28.709779+0000","last_became_active":"2024-01-16T14:59:26.682263+0000","last_became_peered":"2024-01-16T14:59:26.682263+0000","last_unstale":"2024-01-16T14:59:28.709779+0000","last_undegraded":"2024-01-16T14:59:28.709779+0000","last_fullsized":"2024-01-16T14:59:28.709779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:05.652841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709357+0000","last_change":"2024-01-16T14:59:27.721378+0000","last_active":"2024-01-16T14:59:28.709357+0000","last_peered":"2024-01-16T14:59:28.709357+0000","last_clean":"2024-01-16T14:59:28.709357+0000","last_became_active":"2024-01-16T14:59:27.721133+0000","last_became_peered":"2024-01-16T14:59:27.721133+0000","last_unstale":"2024-01-16T14:59:28.709357+0000","last_undegraded":"2024-01-16T14:59:28.709357+0000","last_fullsized":"2024-01-16T14:59:28.709357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703039+0000","last_change":"2024-01-16T14:59:26.688711+0000","last_active":"2024-01-16T14:59:28.703039+0000","last_peered":"2024-01-16T14:59:28.703039+0000","last_clean":"2024-01-16T14:59:28.703039+0000","last_became_active":"2024-01-16T14:59:26.688404+0000","last_became_peered":"2024-01-16T14:59:26.688404+0000","last_unstale":"2024-01-16T14:59:28.703039+0000","last_undegraded":"2024-01-16T14:59:28.703039+0000","last_fullsized":"2024-01-16T14:59:28.703039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904624+0000","last_change":"2024-01-16T14:59:27.724086+0000","last_active":"2024-01-16T14:59:28.904624+0000","last_peered":"2024-01-16T14:59:28.904624+0000","last_clean":"2024-01-16T14:59:28.904624+0000","last_became_active":"2024-01-16T14:59:27.723846+0000","last_became_peered":"2024-01-16T14:59:27.723846+0000","last_unstale":"2024-01-16T14:59:28.904624+0000","last_undegraded":"2024-01-16T14:59:28.904624+0000","last_fullsized":"2024-01-16T14:59:28.904624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.530195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709625+0000","last_change":"2024-01-16T14:59:26.687473+0000","last_active":"2024-01-16T14:59:28.709625+0000","last_peered":"2024-01-16T14:59:28.709625+0000","last_clean":"2024-01-16T14:59:28.709625+0000","last_became_active":"2024-01-16T14:59:26.687161+0000","last_became_peered":"2024-01-16T14:59:26.687161+0000","last_unstale":"2024-01-16T14:59:28.709625+0000","last_undegraded":"2024-01-16T14:59:28.709625+0000","last_fullsized":"2024-01-16T14:59:28.709625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908536+0000","last_change":"2024-01-16T14:59:27.726514+0000","last_active":"2024-01-16T14:59:28.908536+0000","last_peered":"2024-01-16T14:59:28.908536+0000","last_clean":"2024-01-16T14:59:28.908536+0000","last_became_active":"2024-01-16T14:59:27.726196+0000","last_became_peered":"2024-01-16T14:59:27.726196+0000","last_unstale":"2024-01-16T14:59:28.908536+0000","last_undegraded":"2024-01-16T14:59:28.908536+0000","last_fullsized":"2024-01-16T14:59:28.908536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:35.130412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703648+0000","last_change":"2024-01-16T14:59:26.693748+0000","last_active":"2024-01-16T14:59:28.703648+0000","last_peered":"2024-01-16T14:59:28.703648+0000","last_clean":"2024-01-16T14:59:28.703648+0000","last_became_active":"2024-01-16T14:59:26.693411+0000","last_became_peered":"2024-01-16T14:59:26.693411+0000","last_unstale":"2024-01-16T14:59:28.703648+0000","last_undegraded":"2024-01-16T14:59:28.703648+0000","last_fullsized":"2024-01-16T14:59:28.703648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908456+0000","last_change":"2024-01-16T14:59:27.724021+0000","last_active":"2024-01-16T14:59:28.908456+0000","last_peered":"2024-01-16T14:59:28.908456+0000","last_clean":"2024-01-16T14:59:28.908456+0000","last_became_active":"2024-01-16T14:59:27.723760+0000","last_became_peered":"2024-01-16T14:59:27.723760+0000","last_unstale":"2024-01-16T14:59:28.908456+0000","last_undegraded":"2024-01-16T14:59:28.908456+0000","last_fullsized":"2024-01-16T14:59:28.908456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:56.547755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908196+0000","last_change":"2024-01-16T14:59:26.687768+0000","last_active":"2024-01-16T14:59:28.908196+0000","last_peered":"2024-01-16T14:59:28.908196+0000","last_clean":"2024-01-16T14:59:28.908196+0000","last_became_active":"2024-01-16T14:59:26.687439+0000","last_became_peered":"2024-01-16T14:59:26.687439+0000","last_unstale":"2024-01-16T14:59:28.908196+0000","last_undegraded":"2024-01-16T14:59:28.908196+0000","last_fullsized":"2024-01-16T14:59:28.908196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905945+0000","last_change":"2024-01-16T14:59:27.716631+0000","last_active":"2024-01-16T14:59:28.905945+0000","last_peered":"2024-01-16T14:59:28.905945+0000","last_clean":"2024-01-16T14:59:28.905945+0000","last_became_active":"2024-01-16T14:59:27.716370+0000","last_became_peered":"2024-01-16T14:59:27.716370+0000","last_unstale":"2024-01-16T14:59:28.905945+0000","last_undegraded":"2024-01-16T14:59:28.905945+0000","last_fullsized":"2024-01-16T14:59:28.905945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.471222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703521+0000","last_change":"2024-01-16T14:59:26.695961+0000","last_active":"2024-01-16T14:59:28.703521+0000","last_peered":"2024-01-16T14:59:28.703521+0000","last_clean":"2024-01-16T14:59:28.703521+0000","last_became_active":"2024-01-16T14:59:26.695602+0000","last_became_peered":"2024-01-16T14:59:26.695602+0000","last_unstale":"2024-01-16T14:59:28.703521+0000","last_undegraded":"2024-01-16T14:59:28.703521+0000","last_fullsized":"2024-01-16T14:59:28.703521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:07.651834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907463+0000","last_change":"2024-01-16T14:59:27.736089+0000","last_active":"2024-01-16T14:59:28.907463+0000","last_peered":"2024-01-16T14:59:28.907463+0000","last_clean":"2024-01-16T14:59:28.907463+0000","last_became_active":"2024-01-16T14:59:27.731504+0000","last_became_peered":"2024-01-16T14:59:27.731504+0000","last_unstale":"2024-01-16T14:59:28.907463+0000","last_undegraded":"2024-01-16T14:59:28.907463+0000","last_fullsized":"2024-01-16T14:59:28.907463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:20.319731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:27.705910+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T14:59:27.705910+0000","last_peered":"2024-01-16T14:59:27.705910+0000","last_clean":"2024-01-16T14:59:27.705910+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T14:59:27.705910+0000","last_undegraded":"2024-01-16T14:59:27.705910+0000","last_fullsized":"2024-01-16T14:59:27.705910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905174+0000","last_change":"2024-01-16T14:59:26.693490+0000","last_active":"2024-01-16T14:59:28.905174+0000","last_peered":"2024-01-16T14:59:28.905174+0000","last_clean":"2024-01-16T14:59:28.905174+0000","last_became_active":"2024-01-16T14:59:26.692487+0000","last_became_peered":"2024-01-16T14:59:26.692487+0000","last_unstale":"2024-01-16T14:59:28.905174+0000","last_undegraded":"2024-01-16T14:59:28.905174+0000","last_fullsized":"2024-01-16T14:59:28.905174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906140+0000","last_change":"2024-01-16T14:59:27.721477+0000","last_active":"2024-01-16T14:59:28.906140+0000","last_peered":"2024-01-16T14:59:28.906140+0000","last_clean":"2024-01-16T14:59:28.906140+0000","last_became_active":"2024-01-16T14:59:27.720924+0000","last_became_peered":"2024-01-16T14:59:27.720924+0000","last_unstale":"2024-01-16T14:59:28.906140+0000","last_undegraded":"2024-01-16T14:59:28.906140+0000","last_fullsized":"2024-01-16T14:59:28.906140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.326753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904387+0000","last_change":"2024-01-16T14:59:26.701503+0000","last_active":"2024-01-16T14:59:28.904387+0000","last_peered":"2024-01-16T14:59:28.904387+0000","last_clean":"2024-01-16T14:59:28.904387+0000","last_became_active":"2024-01-16T14:59:26.701190+0000","last_became_peered":"2024-01-16T14:59:26.701190+0000","last_unstale":"2024-01-16T14:59:28.904387+0000","last_undegraded":"2024-01-16T14:59:28.904387+0000","last_fullsized":"2024-01-16T14:59:28.904387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906011+0000","last_change":"2024-01-16T14:59:27.728625+0000","last_active":"2024-01-16T14:59:28.906011+0000","last_peered":"2024-01-16T14:59:28.906011+0000","last_clean":"2024-01-16T14:59:28.906011+0000","last_became_active":"2024-01-16T14:59:27.728177+0000","last_became_peered":"2024-01-16T14:59:27.728177+0000","last_unstale":"2024-01-16T14:59:28.906011+0000","last_undegraded":"2024-01-16T14:59:28.906011+0000","last_fullsized":"2024-01-16T14:59:28.906011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:38.465332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703101+0000","last_change":"2024-01-16T14:59:26.695022+0000","last_active":"2024-01-16T14:59:28.703101+0000","last_peered":"2024-01-16T14:59:28.703101+0000","last_clean":"2024-01-16T14:59:28.703101+0000","last_became_active":"2024-01-16T14:59:26.694756+0000","last_became_peered":"2024-01-16T14:59:26.694756+0000","last_unstale":"2024-01-16T14:59:28.703101+0000","last_undegraded":"2024-01-16T14:59:28.703101+0000","last_fullsized":"2024-01-16T14:59:28.703101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:41.781228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905205+0000","last_change":"2024-01-16T14:59:27.706635+0000","last_active":"2024-01-16T14:59:28.905205+0000","last_peered":"2024-01-16T14:59:28.905205+0000","last_clean":"2024-01-16T14:59:28.905205+0000","last_became_active":"2024-01-16T14:59:27.706296+0000","last_became_peered":"2024-01-16T14:59:27.706296+0000","last_unstale":"2024-01-16T14:59:28.905205+0000","last_undegraded":"2024-01-16T14:59:28.905205+0000","last_fullsized":"2024-01-16T14:59:28.905205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906768+0000","last_change":"2024-01-16T14:59:26.700908+0000","last_active":"2024-01-16T14:59:28.906768+0000","last_peered":"2024-01-16T14:59:28.906768+0000","last_clean":"2024-01-16T14:59:28.906768+0000","last_became_active":"2024-01-16T14:59:26.699693+0000","last_became_peered":"2024-01-16T14:59:26.699693+0000","last_unstale":"2024-01-16T14:59:28.906768+0000","last_undegraded":"2024-01-16T14:59:28.906768+0000","last_fullsized":"2024-01-16T14:59:28.906768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:10.078798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708429+0000","last_change":"2024-01-16T14:59:27.717434+0000","last_active":"2024-01-16T14:59:28.708429+0000","last_peered":"2024-01-16T14:59:28.708429+0000","last_clean":"2024-01-16T14:59:28.708429+0000","last_became_active":"2024-01-16T14:59:27.717061+0000","last_became_peered":"2024-01-16T14:59:27.717061+0000","last_unstale":"2024-01-16T14:59:28.708429+0000","last_undegraded":"2024-01-16T14:59:28.708429+0000","last_fullsized":"2024-01-16T14:59:28.708429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:14.259904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907983+0000","last_change":"2024-01-16T14:59:26.689585+0000","last_active":"2024-01-16T14:59:28.907983+0000","last_peered":"2024-01-16T14:59:28.907983+0000","last_clean":"2024-01-16T14:59:28.907983+0000","last_became_active":"2024-01-16T14:59:26.689193+0000","last_became_peered":"2024-01-16T14:59:26.689193+0000","last_unstale":"2024-01-16T14:59:28.907983+0000","last_undegraded":"2024-01-16T14:59:28.907983+0000","last_fullsized":"2024-01-16T14:59:28.907983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:40.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907283+0000","last_change":"2024-01-16T14:59:27.723605+0000","last_active":"2024-01-16T14:59:28.907283+0000","last_peered":"2024-01-16T14:59:28.907283+0000","last_clean":"2024-01-16T14:59:28.907283+0000","last_became_active":"2024-01-16T14:59:27.723322+0000","last_became_peered":"2024-01-16T14:59:27.723322+0000","last_unstale":"2024-01-16T14:59:28.907283+0000","last_undegraded":"2024-01-16T14:59:28.907283+0000","last_fullsized":"2024-01-16T14:59:28.907283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.187834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908060+0000","last_change":"2024-01-16T14:59:26.701967+0000","last_active":"2024-01-16T14:59:28.908060+0000","last_peered":"2024-01-16T14:59:28.908060+0000","last_clean":"2024-01-16T14:59:28.908060+0000","last_became_active":"2024-01-16T14:59:26.701686+0000","last_became_peered":"2024-01-16T14:59:26.701686+0000","last_unstale":"2024-01-16T14:59:28.908060+0000","last_undegraded":"2024-01-16T14:59:28.908060+0000","last_fullsized":"2024-01-16T14:59:28.908060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:09.766721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907757+0000","last_change":"2024-01-16T14:59:27.709576+0000","last_active":"2024-01-16T14:59:28.907757+0000","last_peered":"2024-01-16T14:59:28.907757+0000","last_clean":"2024-01-16T14:59:28.907757+0000","last_became_active":"2024-01-16T14:59:27.706317+0000","last_became_peered":"2024-01-16T14:59:27.706317+0000","last_unstale":"2024-01-16T14:59:28.907757+0000","last_undegraded":"2024-01-16T14:59:28.907757+0000","last_fullsized":"2024-01-16T14:59:28.907757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908338+0000","last_change":"2024-01-16T14:59:26.693739+0000","last_active":"2024-01-16T14:59:28.908338+0000","last_peered":"2024-01-16T14:59:28.908338+0000","last_clean":"2024-01-16T14:59:28.908338+0000","last_became_active":"2024-01-16T14:59:26.693473+0000","last_became_peered":"2024-01-16T14:59:26.693473+0000","last_unstale":"2024-01-16T14:59:28.908338+0000","last_undegraded":"2024-01-16T14:59:28.908338+0000","last_fullsized":"2024-01-16T14:59:28.908338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906445+0000","last_change":"2024-01-16T14:59:27.717414+0000","last_active":"2024-01-16T14:59:28.906445+0000","last_peered":"2024-01-16T14:59:28.906445+0000","last_clean":"2024-01-16T14:59:28.906445+0000","last_became_active":"2024-01-16T14:59:27.717078+0000","last_became_peered":"2024-01-16T14:59:27.717078+0000","last_unstale":"2024-01-16T14:59:28.906445+0000","last_undegraded":"2024-01-16T14:59:28.906445+0000","last_fullsized":"2024-01-16T14:59:28.906445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:30.634017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711015+0000","last_change":"2024-01-16T14:59:26.696098+0000","last_active":"2024-01-16T14:59:28.711015+0000","last_peered":"2024-01-16T14:59:28.711015+0000","last_clean":"2024-01-16T14:59:28.711015+0000","last_became_active":"2024-01-16T14:59:26.695792+0000","last_became_peered":"2024-01-16T14:59:26.695792+0000","last_unstale":"2024-01-16T14:59:28.711015+0000","last_undegraded":"2024-01-16T14:59:28.711015+0000","last_fullsized":"2024-01-16T14:59:28.711015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908934+0000","last_change":"2024-01-16T14:59:27.712757+0000","last_active":"2024-01-16T14:59:28.908934+0000","last_peered":"2024-01-16T14:59:28.908934+0000","last_clean":"2024-01-16T14:59:28.908934+0000","last_became_active":"2024-01-16T14:59:27.712417+0000","last_became_peered":"2024-01-16T14:59:27.712417+0000","last_unstale":"2024-01-16T14:59:28.908934+0000","last_undegraded":"2024-01-16T14:59:28.908934+0000","last_fullsized":"2024-01-16T14:59:28.908934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904378+0000","last_change":"2024-01-16T14:59:26.699479+0000","last_active":"2024-01-16T14:59:28.904378+0000","last_peered":"2024-01-16T14:59:28.904378+0000","last_clean":"2024-01-16T14:59:28.904378+0000","last_became_active":"2024-01-16T14:59:26.699195+0000","last_became_peered":"2024-01-16T14:59:26.699195+0000","last_unstale":"2024-01-16T14:59:28.904378+0000","last_undegraded":"2024-01-16T14:59:28.904378+0000","last_fullsized":"2024-01-16T14:59:28.904378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.521651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708137+0000","last_change":"2024-01-16T14:59:27.715745+0000","last_active":"2024-01-16T14:59:28.708137+0000","last_peered":"2024-01-16T14:59:28.708137+0000","last_clean":"2024-01-16T14:59:28.708137+0000","last_became_active":"2024-01-16T14:59:27.710835+0000","last_became_peered":"2024-01-16T14:59:27.710835+0000","last_unstale":"2024-01-16T14:59:28.708137+0000","last_undegraded":"2024-01-16T14:59:28.708137+0000","last_fullsized":"2024-01-16T14:59:28.708137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:53.870850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906409+0000","last_change":"2024-01-16T14:59:26.697228+0000","last_active":"2024-01-16T14:59:28.906409+0000","last_peered":"2024-01-16T14:59:28.906409+0000","last_clean":"2024-01-16T14:59:28.906409+0000","last_became_active":"2024-01-16T14:59:26.696136+0000","last_became_peered":"2024-01-16T14:59:26.696136+0000","last_unstale":"2024-01-16T14:59:28.906409+0000","last_undegraded":"2024-01-16T14:59:28.906409+0000","last_fullsized":"2024-01-16T14:59:28.906409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:37.293826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905459+0000","last_change":"2024-01-16T14:59:27.711533+0000","last_active":"2024-01-16T14:59:28.905459+0000","last_peered":"2024-01-16T14:59:28.905459+0000","last_clean":"2024-01-16T14:59:28.905459+0000","last_became_active":"2024-01-16T14:59:27.711228+0000","last_became_peered":"2024-01-16T14:59:27.711228+0000","last_unstale":"2024-01-16T14:59:28.905459+0000","last_undegraded":"2024-01-16T14:59:28.905459+0000","last_fullsized":"2024-01-16T14:59:28.905459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:28.132533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907464+0000","last_change":"2024-01-16T14:59:26.695937+0000","last_active":"2024-01-16T14:59:28.907464+0000","last_peered":"2024-01-16T14:59:28.907464+0000","last_clean":"2024-01-16T14:59:28.907464+0000","last_became_active":"2024-01-16T14:59:26.695628+0000","last_became_peered":"2024-01-16T14:59:26.695628+0000","last_unstale":"2024-01-16T14:59:28.907464+0000","last_undegraded":"2024-01-16T14:59:28.907464+0000","last_fullsized":"2024-01-16T14:59:28.907464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:36.523217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906350+0000","last_change":"2024-01-16T14:59:27.720565+0000","last_active":"2024-01-16T14:59:28.906350+0000","last_peered":"2024-01-16T14:59:28.906350+0000","last_clean":"2024-01-16T14:59:28.906350+0000","last_became_active":"2024-01-16T14:59:27.716762+0000","last_became_peered":"2024-01-16T14:59:27.716762+0000","last_unstale":"2024-01-16T14:59:28.906350+0000","last_undegraded":"2024-01-16T14:59:28.906350+0000","last_fullsized":"2024-01-16T14:59:28.906350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:26.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906643+0000","last_change":"2024-01-16T14:59:26.700414+0000","last_active":"2024-01-16T14:59:28.906643+0000","last_peered":"2024-01-16T14:59:28.906643+0000","last_clean":"2024-01-16T14:59:28.906643+0000","last_became_active":"2024-01-16T14:59:26.700111+0000","last_became_peered":"2024-01-16T14:59:26.700111+0000","last_unstale":"2024-01-16T14:59:28.906643+0000","last_undegraded":"2024-01-16T14:59:28.906643+0000","last_fullsized":"2024-01-16T14:59:28.906643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:29.462511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709936+0000","last_change":"2024-01-16T14:59:27.718486+0000","last_active":"2024-01-16T14:59:28.709936+0000","last_peered":"2024-01-16T14:59:28.709936+0000","last_clean":"2024-01-16T14:59:28.709936+0000","last_became_active":"2024-01-16T14:59:27.712753+0000","last_became_peered":"2024-01-16T14:59:27.712753+0000","last_unstale":"2024-01-16T14:59:28.709936+0000","last_undegraded":"2024-01-16T14:59:28.709936+0000","last_fullsized":"2024-01-16T14:59:28.709936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907057+0000","last_change":"2024-01-16T14:59:27.736702+0000","last_active":"2024-01-16T14:59:28.907057+0000","last_peered":"2024-01-16T14:59:28.907057+0000","last_clean":"2024-01-16T14:59:28.907057+0000","last_became_active":"2024-01-16T14:59:27.736433+0000","last_became_peered":"2024-01-16T14:59:27.736433+0000","last_unstale":"2024-01-16T14:59:28.907057+0000","last_undegraded":"2024-01-16T14:59:28.907057+0000","last_fullsized":"2024-01-16T14:59:28.907057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:13.830199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907201+0000","last_change":"2024-01-16T14:59:26.703355+0000","last_active":"2024-01-16T14:59:28.907201+0000","last_peered":"2024-01-16T14:59:28.907201+0000","last_clean":"2024-01-16T14:59:28.907201+0000","last_became_active":"2024-01-16T14:59:26.703059+0000","last_became_peered":"2024-01-16T14:59:26.703059+0000","last_unstale":"2024-01-16T14:59:28.907201+0000","last_undegraded":"2024-01-16T14:59:28.907201+0000","last_fullsized":"2024-01-16T14:59:28.907201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.422714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905630+0000","last_change":"2024-01-16T14:59:27.726539+0000","last_active":"2024-01-16T14:59:28.905630+0000","last_peered":"2024-01-16T14:59:28.905630+0000","last_clean":"2024-01-16T14:59:28.905630+0000","last_became_active":"2024-01-16T14:59:27.726269+0000","last_became_peered":"2024-01-16T14:59:27.726269+0000","last_unstale":"2024-01-16T14:59:28.905630+0000","last_undegraded":"2024-01-16T14:59:28.905630+0000","last_fullsized":"2024-01-16T14:59:28.905630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907840+0000","last_change":"2024-01-16T14:59:26.702143+0000","last_active":"2024-01-16T14:59:28.907840+0000","last_peered":"2024-01-16T14:59:28.907840+0000","last_clean":"2024-01-16T14:59:28.907840+0000","last_became_active":"2024-01-16T14:59:26.701732+0000","last_became_peered":"2024-01-16T14:59:26.701732+0000","last_unstale":"2024-01-16T14:59:28.907840+0000","last_undegraded":"2024-01-16T14:59:28.907840+0000","last_fullsized":"2024-01-16T14:59:28.907840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:44.454533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908894+0000","last_change":"2024-01-16T14:59:27.719650+0000","last_active":"2024-01-16T14:59:28.908894+0000","last_peered":"2024-01-16T14:59:28.908894+0000","last_clean":"2024-01-16T14:59:28.908894+0000","last_became_active":"2024-01-16T14:59:27.719345+0000","last_became_peered":"2024-01-16T14:59:27.719345+0000","last_unstale":"2024-01-16T14:59:28.908894+0000","last_undegraded":"2024-01-16T14:59:28.908894+0000","last_fullsized":"2024-01-16T14:59:28.908894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906403+0000","last_change":"2024-01-16T14:59:26.692569+0000","last_active":"2024-01-16T14:59:28.906403+0000","last_peered":"2024-01-16T14:59:28.906403+0000","last_clean":"2024-01-16T14:59:28.906403+0000","last_became_active":"2024-01-16T14:59:26.689432+0000","last_became_peered":"2024-01-16T14:59:26.689432+0000","last_unstale":"2024-01-16T14:59:28.906403+0000","last_undegraded":"2024-01-16T14:59:28.906403+0000","last_fullsized":"2024-01-16T14:59:28.906403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:58.385575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905910+0000","last_change":"2024-01-16T14:59:27.728184+0000","last_active":"2024-01-16T14:59:28.905910+0000","last_peered":"2024-01-16T14:59:28.905910+0000","last_clean":"2024-01-16T14:59:28.905910+0000","last_became_active":"2024-01-16T14:59:27.727889+0000","last_became_peered":"2024-01-16T14:59:27.727889+0000","last_unstale":"2024-01-16T14:59:28.905910+0000","last_undegraded":"2024-01-16T14:59:28.905910+0000","last_fullsized":"2024-01-16T14:59:28.905910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.211584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909226+0000","last_change":"2024-01-16T14:59:26.697456+0000","last_active":"2024-01-16T14:59:28.909226+0000","last_peered":"2024-01-16T14:59:28.909226+0000","last_clean":"2024-01-16T14:59:28.909226+0000","last_became_active":"2024-01-16T14:59:26.697129+0000","last_became_peered":"2024-01-16T14:59:26.697129+0000","last_unstale":"2024-01-16T14:59:28.909226+0000","last_undegraded":"2024-01-16T14:59:28.909226+0000","last_fullsized":"2024-01-16T14:59:28.909226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9119999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5459999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2309999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.542}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":74114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4140000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8799999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":819200,"data_stored":663508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:32.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.440+0000 7f07b7800640 1 Processor -- start 2024-01-16T14:59:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.440+0000 7f07b7800640 1 -- start start 2024-01-16T14:59:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.441+0000 7f07b7800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f07b00d17d0 0x7f07b015c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.441+0000 7f07b7800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.441+0000 7f07b7800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b00cf410 con 0x7f07b00d1bb0 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.441+0000 7f07b7800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b00ce7f0 con 0x7f07b00d17d0 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.441+0000 7f07b7800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b015c6c0 con 0x7f07b00cdf70 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.442+0000 7f07b5800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.442+0000 7f07b5800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46980/0 (socket says 172.21.15.43:46980) 2024-01-16T14:59:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.442+0000 7f07b5800640 1 -- 172.21.15.43:0/2179709975 learned_addr learned my addr 172.21.15.43:0/2179709975 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.442+0000 7f07b4e00640 1 --2- 172.21.15.43:0/2179709975 >> v2:172.21.15.69:3300/0 conn(0x7f07b00d17d0 0x7f07b015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.443+0000 7f07b4e00640 1 -- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 msgr2=0x7f07b00b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.443+0000 7f07b4e00640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.443+0000 7f07b4e00640 1 -- 172.21.15.43:0/2179709975 >> v1:172.21.15.69:6789/0 conn(0x7f07b00d1bb0 legacy=0x7f07b00cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.443+0000 7f07b4e00640 1 -- 172.21.15.43:0/2179709975 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b00ce510 con 0x7f07b00d17d0 2024-01-16T14:59:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.443+0000 7f07b5800640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.444+0000 7f07b4e00640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b015c180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07a00097a0 tx=0x7f07a002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58198a4b8927fd4a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.445+0000 7f07af400640 1 -- 172.21.15.43:0/2179709975 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a002d040 con 0x7f07b00d17d0 2024-01-16T14:59:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.445+0000 7f07af400640 1 -- 172.21.15.43:0/2179709975 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0031810 con 0x7f07b00d17d0 2024-01-16T14:59:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.445+0000 7f07af400640 1 -- 172.21.15.43:0/2179709975 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0031ba0 con 0x7f07b00d17d0 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 -- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 msgr2=0x7f07b015c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b015c180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07a00097a0 tx=0x7f07a002b8b0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 -- 172.21.15.43:0/2179709975 shutdown_connections 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00cdf70 0x7f07b00b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 --2- 172.21.15.43:0/2179709975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b015c180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 -- 172.21.15.43:0/2179709975 >> 172.21.15.43:0/2179709975 conn(0x7f07b00661f0 msgr2=0x7f07b0066600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 -- 172.21.15.43:0/2179709975 shutdown_connections 2024-01-16T14:59:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.446+0000 7f07b7800640 1 -- 172.21.15.43:0/2179709975 wait complete. 2024-01-16T14:59:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.448+0000 7f07b7800640 1 Processor -- start 2024-01-16T14:59:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.449+0000 7f07b7800640 1 -- start start 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b0140940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00d1bb0 0x7f07b014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0153840 con 0x7f07b00cdf70 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0154210 con 0x7f07b00d17d0 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b7800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0152c30 con 0x7f07b00d1bb0 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58310/0 (socket says 172.21.15.43:58310) 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b4e00640 1 -- 172.21.15.43:0/1536033585 learned_addr learned my addr 172.21.15.43:0/1536033585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07b5800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00d1bb0 0x7f07b014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.450+0000 7f07afe00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b0140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 -- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00d1bb0 msgr2=0x7f07b014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00d1bb0 0x7f07b014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 -- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 msgr2=0x7f07b0140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b0140940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 -- 172.21.15.43:0/1536033585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8009b30 con 0x7f07b00cdf70 2024-01-16T14:59:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b4e00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f07a00098c0 tx=0x7f07a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a003ace0 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a003ae60 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a0009400 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.451+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0044640 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.452+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b01531f0 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.453+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0074770 con 0x7f07b00cdf70 2024-01-16T14:59:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.454+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f07a004d440 con 0x7f07b00cdf70 2024-01-16T14:59:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.455+0000 7f07acc00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 0x7f078806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.455+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07a00abde0 con 0x7f07b00cdf70 2024-01-16T14:59:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.455+0000 7f07afe00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 0x7f078806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.456+0000 7f07afe00640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 0x7f078806d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0798007fd0 tx=0x7f0798009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.460+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07a0076e00 con 0x7f07b00cdf70 2024-01-16T14:59:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.650+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07b00cf0a0 con 0x7f078806b2c0 2024-01-16T14:59:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.670+0000 7f07acc00640 1 -- 172.21.15.43:0/1536033585 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264517 (secure 0 0 0) 0x7f07980050d0 con 0x7f078806b2c0 2024-01-16T14:59:32.656 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 msgr2=0x7f078806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 0x7f078806d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0798007fd0 tx=0x7f0798009040 comp rx=0 tx=0).stop 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 msgr2=0x7f07b0153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f07a00098c0 tx=0x7f07a00046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 shutdown_connections 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078806b2c0 0x7f078806d780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07b00d17d0 0x7f07b0140940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07b00d1bb0 0x7f07b014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 --2- 172.21.15.43:0/1536033585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07b00cdf70 0x7f07b0153cd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.676+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 >> 172.21.15.43:0/1536033585 conn(0x7f07b00661f0 msgr2=0x7f07b015ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.677+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 shutdown_connections 2024-01-16T14:59:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.677+0000 7f07b7800640 1 -- 172.21.15.43:0/1536033585 wait complete. 2024-01-16T14:59:32.677 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-16T14:59:30.788128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240352,"kb_used_data":2960,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734368,"statfs":{"total":773094113280,"available":772847992832,"internally_reserved":0,"allocated":3031040,"data_stored":1803372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137704"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908219+0000","last_change":"2024-01-16T14:59:27.733425+0000","last_active":"2024-01-16T14:59:28.908219+0000","last_peered":"2024-01-16T14:59:28.908219+0000","last_clean":"2024-01-16T14:59:28.908219+0000","last_became_active":"2024-01-16T14:59:27.733108+0000","last_became_peered":"2024-01-16T14:59:27.733108+0000","last_unstale":"2024-01-16T14:59:28.908219+0000","last_undegraded":"2024-01-16T14:59:28.908219+0000","last_fullsized":"2024-01-16T14:59:28.908219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:28:12.476929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908249+0000","last_change":"2024-01-16T14:59:27.715575+0000","last_active":"2024-01-16T14:59:28.908249+0000","last_peered":"2024-01-16T14:59:28.908249+0000","last_clean":"2024-01-16T14:59:28.908249+0000","last_became_active":"2024-01-16T14:59:27.714841+0000","last_became_peered":"2024-01-16T14:59:27.714841+0000","last_unstale":"2024-01-16T14:59:28.908249+0000","last_undegraded":"2024-01-16T14:59:28.908249+0000","last_fullsized":"2024-01-16T14:59:28.908249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908551+0000","last_change":"2024-01-16T14:59:27.718807+0000","last_active":"2024-01-16T14:59:28.908551+0000","last_peered":"2024-01-16T14:59:28.908551+0000","last_clean":"2024-01-16T14:59:28.908551+0000","last_became_active":"2024-01-16T14:59:27.717301+0000","last_became_peered":"2024-01-16T14:59:27.717301+0000","last_unstale":"2024-01-16T14:59:28.908551+0000","last_undegraded":"2024-01-16T14:59:28.908551+0000","last_fullsized":"2024-01-16T14:59:28.908551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906129+0000","last_change":"2024-01-16T14:59:27.720914+0000","last_active":"2024-01-16T14:59:28.906129+0000","last_peered":"2024-01-16T14:59:28.906129+0000","last_clean":"2024-01-16T14:59:28.906129+0000","last_became_active":"2024-01-16T14:59:27.720561+0000","last_became_peered":"2024-01-16T14:59:27.720561+0000","last_unstale":"2024-01-16T14:59:28.906129+0000","last_undegraded":"2024-01-16T14:59:28.906129+0000","last_fullsized":"2024-01-16T14:59:28.906129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:16.656352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906008+0000","last_change":"2024-01-16T14:59:27.721194+0000","last_active":"2024-01-16T14:59:28.906008+0000","last_peered":"2024-01-16T14:59:28.906008+0000","last_clean":"2024-01-16T14:59:28.906008+0000","last_became_active":"2024-01-16T14:59:27.719980+0000","last_became_peered":"2024-01-16T14:59:27.719980+0000","last_unstale":"2024-01-16T14:59:28.906008+0000","last_undegraded":"2024-01-16T14:59:28.906008+0000","last_fullsized":"2024-01-16T14:59:28.906008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:54.580479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905723+0000","last_change":"2024-01-16T14:59:27.732776+0000","last_active":"2024-01-16T14:59:28.905723+0000","last_peered":"2024-01-16T14:59:28.905723+0000","last_clean":"2024-01-16T14:59:28.905723+0000","last_became_active":"2024-01-16T14:59:27.732373+0000","last_became_peered":"2024-01-16T14:59:27.732373+0000","last_unstale":"2024-01-16T14:59:28.905723+0000","last_undegraded":"2024-01-16T14:59:28.905723+0000","last_fullsized":"2024-01-16T14:59:28.905723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904627+0000","last_change":"2024-01-16T14:59:27.722159+0000","last_active":"2024-01-16T14:59:28.904627+0000","last_peered":"2024-01-16T14:59:28.904627+0000","last_clean":"2024-01-16T14:59:28.904627+0000","last_became_active":"2024-01-16T14:59:27.721928+0000","last_became_peered":"2024-01-16T14:59:27.721928+0000","last_unstale":"2024-01-16T14:59:28.904627+0000","last_undegraded":"2024-01-16T14:59:28.904627+0000","last_fullsized":"2024-01-16T14:59:28.904627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:17.023676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904587+0000","last_change":"2024-01-16T14:59:27.721172+0000","last_active":"2024-01-16T14:59:28.904587+0000","last_peered":"2024-01-16T14:59:28.904587+0000","last_clean":"2024-01-16T14:59:28.904587+0000","last_became_active":"2024-01-16T14:59:27.720829+0000","last_became_peered":"2024-01-16T14:59:27.720829+0000","last_unstale":"2024-01-16T14:59:28.904587+0000","last_undegraded":"2024-01-16T14:59:28.904587+0000","last_fullsized":"2024-01-16T14:59:28.904587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:39.594220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908493+0000","last_change":"2024-01-16T14:59:27.721916+0000","last_active":"2024-01-16T14:59:28.908493+0000","last_peered":"2024-01-16T14:59:28.908493+0000","last_clean":"2024-01-16T14:59:28.908493+0000","last_became_active":"2024-01-16T14:59:27.721604+0000","last_became_peered":"2024-01-16T14:59:27.721604+0000","last_unstale":"2024-01-16T14:59:28.908493+0000","last_undegraded":"2024-01-16T14:59:28.908493+0000","last_fullsized":"2024-01-16T14:59:28.908493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:05.858739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711127+0000","last_change":"2024-01-16T14:59:27.729823+0000","last_active":"2024-01-16T14:59:28.711127+0000","last_peered":"2024-01-16T14:59:28.711127+0000","last_clean":"2024-01-16T14:59:28.711127+0000","last_became_active":"2024-01-16T14:59:27.724755+0000","last_became_peered":"2024-01-16T14:59:27.724755+0000","last_unstale":"2024-01-16T14:59:28.711127+0000","last_undegraded":"2024-01-16T14:59:28.711127+0000","last_fullsized":"2024-01-16T14:59:28.711127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710385+0000","last_change":"2024-01-16T14:59:27.725448+0000","last_active":"2024-01-16T14:59:28.710385+0000","last_peered":"2024-01-16T14:59:28.710385+0000","last_clean":"2024-01-16T14:59:28.710385+0000","last_became_active":"2024-01-16T14:59:27.724294+0000","last_became_peered":"2024-01-16T14:59:27.724294+0000","last_unstale":"2024-01-16T14:59:28.710385+0000","last_undegraded":"2024-01-16T14:59:28.710385+0000","last_fullsized":"2024-01-16T14:59:28.710385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709647+0000","last_change":"2024-01-16T14:59:27.725056+0000","last_active":"2024-01-16T14:59:28.709647+0000","last_peered":"2024-01-16T14:59:28.709647+0000","last_clean":"2024-01-16T14:59:28.709647+0000","last_became_active":"2024-01-16T14:59:27.724771+0000","last_became_peered":"2024-01-16T14:59:27.724771+0000","last_unstale":"2024-01-16T14:59:28.709647+0000","last_undegraded":"2024-01-16T14:59:28.709647+0000","last_fullsized":"2024-01-16T14:59:28.709647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905952+0000","last_change":"2024-01-16T14:59:27.726131+0000","last_active":"2024-01-16T14:59:28.905952+0000","last_peered":"2024-01-16T14:59:28.905952+0000","last_clean":"2024-01-16T14:59:28.905952+0000","last_became_active":"2024-01-16T14:59:27.725830+0000","last_became_peered":"2024-01-16T14:59:27.725830+0000","last_unstale":"2024-01-16T14:59:28.905952+0000","last_undegraded":"2024-01-16T14:59:28.905952+0000","last_fullsized":"2024-01-16T14:59:28.905952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:45.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906705+0000","last_change":"2024-01-16T14:59:27.713525+0000","last_active":"2024-01-16T14:59:28.906705+0000","last_peered":"2024-01-16T14:59:28.906705+0000","last_clean":"2024-01-16T14:59:28.906705+0000","last_became_active":"2024-01-16T14:59:27.712762+0000","last_became_peered":"2024-01-16T14:59:27.712762+0000","last_unstale":"2024-01-16T14:59:28.906705+0000","last_undegraded":"2024-01-16T14:59:28.906705+0000","last_fullsized":"2024-01-16T14:59:28.906705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:41.373234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906179+0000","last_change":"2024-01-16T14:59:27.700808+0000","last_active":"2024-01-16T14:59:28.906179+0000","last_peered":"2024-01-16T14:59:28.906179+0000","last_clean":"2024-01-16T14:59:28.906179+0000","last_became_active":"2024-01-16T14:59:27.700477+0000","last_became_peered":"2024-01-16T14:59:27.700477+0000","last_unstale":"2024-01-16T14:59:28.906179+0000","last_undegraded":"2024-01-16T14:59:28.906179+0000","last_fullsized":"2024-01-16T14:59:28.906179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905239+0000","last_change":"2024-01-16T14:59:27.720864+0000","last_active":"2024-01-16T14:59:28.905239+0000","last_peered":"2024-01-16T14:59:28.905239+0000","last_clean":"2024-01-16T14:59:28.905239+0000","last_became_active":"2024-01-16T14:59:27.719599+0000","last_became_peered":"2024-01-16T14:59:27.719599+0000","last_unstale":"2024-01-16T14:59:28.905239+0000","last_undegraded":"2024-01-16T14:59:28.905239+0000","last_fullsized":"2024-01-16T14:59:28.905239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907987+0000","last_change":"2024-01-16T14:59:27.717946+0000","last_active":"2024-01-16T14:59:28.907987+0000","last_peered":"2024-01-16T14:59:28.907987+0000","last_clean":"2024-01-16T14:59:28.907987+0000","last_became_active":"2024-01-16T14:59:27.717711+0000","last_became_peered":"2024-01-16T14:59:27.717711+0000","last_unstale":"2024-01-16T14:59:28.907987+0000","last_undegraded":"2024-01-16T14:59:28.907987+0000","last_fullsized":"2024-01-16T14:59:28.907987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:03.931690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703527+0000","last_change":"2024-01-16T14:59:27.726121+0000","last_active":"2024-01-16T14:59:28.703527+0000","last_peered":"2024-01-16T14:59:28.703527+0000","last_clean":"2024-01-16T14:59:28.703527+0000","last_became_active":"2024-01-16T14:59:27.725643+0000","last_became_peered":"2024-01-16T14:59:27.725643+0000","last_unstale":"2024-01-16T14:59:28.703527+0000","last_undegraded":"2024-01-16T14:59:28.703527+0000","last_fullsized":"2024-01-16T14:59:28.703527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907957+0000","last_change":"2024-01-16T14:59:27.720619+0000","last_active":"2024-01-16T14:59:28.907957+0000","last_peered":"2024-01-16T14:59:28.907957+0000","last_clean":"2024-01-16T14:59:28.907957+0000","last_became_active":"2024-01-16T14:59:27.720348+0000","last_became_peered":"2024-01-16T14:59:27.720348+0000","last_unstale":"2024-01-16T14:59:28.907957+0000","last_undegraded":"2024-01-16T14:59:28.907957+0000","last_fullsized":"2024-01-16T14:59:28.907957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:58.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710390+0000","last_change":"2024-01-16T14:59:27.727783+0000","last_active":"2024-01-16T14:59:28.710390+0000","last_peered":"2024-01-16T14:59:28.710390+0000","last_clean":"2024-01-16T14:59:28.710390+0000","last_became_active":"2024-01-16T14:59:27.727562+0000","last_became_peered":"2024-01-16T14:59:27.727562+0000","last_unstale":"2024-01-16T14:59:28.710390+0000","last_undegraded":"2024-01-16T14:59:28.710390+0000","last_fullsized":"2024-01-16T14:59:28.710390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907372+0000","last_change":"2024-01-16T14:59:27.724755+0000","last_active":"2024-01-16T14:59:28.907372+0000","last_peered":"2024-01-16T14:59:28.907372+0000","last_clean":"2024-01-16T14:59:28.907372+0000","last_became_active":"2024-01-16T14:59:27.724443+0000","last_became_peered":"2024-01-16T14:59:27.724443+0000","last_unstale":"2024-01-16T14:59:28.907372+0000","last_undegraded":"2024-01-16T14:59:28.907372+0000","last_fullsized":"2024-01-16T14:59:28.907372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:06.824346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906082+0000","last_change":"2024-01-16T14:59:27.732620+0000","last_active":"2024-01-16T14:59:28.906082+0000","last_peered":"2024-01-16T14:59:28.906082+0000","last_clean":"2024-01-16T14:59:28.906082+0000","last_became_active":"2024-01-16T14:59:27.732037+0000","last_became_peered":"2024-01-16T14:59:27.732037+0000","last_unstale":"2024-01-16T14:59:28.906082+0000","last_undegraded":"2024-01-16T14:59:28.906082+0000","last_fullsized":"2024-01-16T14:59:28.906082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:13.794976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905598+0000","last_change":"2024-01-16T14:59:27.719539+0000","last_active":"2024-01-16T14:59:28.905598+0000","last_peered":"2024-01-16T14:59:28.905598+0000","last_clean":"2024-01-16T14:59:28.905598+0000","last_became_active":"2024-01-16T14:59:27.719253+0000","last_became_peered":"2024-01-16T14:59:27.719253+0000","last_unstale":"2024-01-16T14:59:28.905598+0000","last_undegraded":"2024-01-16T14:59:28.905598+0000","last_fullsized":"2024-01-16T14:59:28.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:42:03.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905607+0000","last_change":"2024-01-16T14:59:27.715773+0000","last_active":"2024-01-16T14:59:28.905607+0000","last_peered":"2024-01-16T14:59:28.905607+0000","last_clean":"2024-01-16T14:59:28.905607+0000","last_became_active":"2024-01-16T14:59:27.715482+0000","last_became_peered":"2024-01-16T14:59:27.715482+0000","last_unstale":"2024-01-16T14:59:28.905607+0000","last_undegraded":"2024-01-16T14:59:28.905607+0000","last_fullsized":"2024-01-16T14:59:28.905607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906301+0000","last_change":"2024-01-16T14:59:27.726710+0000","last_active":"2024-01-16T14:59:28.906301+0000","last_peered":"2024-01-16T14:59:28.906301+0000","last_clean":"2024-01-16T14:59:28.906301+0000","last_became_active":"2024-01-16T14:59:27.723820+0000","last_became_peered":"2024-01-16T14:59:27.723820+0000","last_unstale":"2024-01-16T14:59:28.906301+0000","last_undegraded":"2024-01-16T14:59:28.906301+0000","last_fullsized":"2024-01-16T14:59:28.906301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:10.938227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711055+0000","last_change":"2024-01-16T14:59:27.726914+0000","last_active":"2024-01-16T14:59:28.711055+0000","last_peered":"2024-01-16T14:59:28.711055+0000","last_clean":"2024-01-16T14:59:28.711055+0000","last_became_active":"2024-01-16T14:59:27.725638+0000","last_became_peered":"2024-01-16T14:59:27.725638+0000","last_unstale":"2024-01-16T14:59:28.711055+0000","last_undegraded":"2024-01-16T14:59:28.711055+0000","last_fullsized":"2024-01-16T14:59:28.711055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907822+0000","last_change":"2024-01-16T14:59:27.703634+0000","last_active":"2024-01-16T14:59:28.907822+0000","last_peered":"2024-01-16T14:59:28.907822+0000","last_clean":"2024-01-16T14:59:28.907822+0000","last_became_active":"2024-01-16T14:59:27.703343+0000","last_became_peered":"2024-01-16T14:59:27.703343+0000","last_unstale":"2024-01-16T14:59:28.907822+0000","last_undegraded":"2024-01-16T14:59:28.907822+0000","last_fullsized":"2024-01-16T14:59:28.907822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906778+0000","last_change":"2024-01-16T14:59:27.724563+0000","last_active":"2024-01-16T14:59:28.906778+0000","last_peered":"2024-01-16T14:59:28.906778+0000","last_clean":"2024-01-16T14:59:28.906778+0000","last_became_active":"2024-01-16T14:59:27.723074+0000","last_became_peered":"2024-01-16T14:59:27.723074+0000","last_unstale":"2024-01-16T14:59:28.906778+0000","last_undegraded":"2024-01-16T14:59:28.906778+0000","last_fullsized":"2024-01-16T14:59:28.906778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:05.783445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907568+0000","last_change":"2024-01-16T14:59:27.724629+0000","last_active":"2024-01-16T14:59:28.907568+0000","last_peered":"2024-01-16T14:59:28.907568+0000","last_clean":"2024-01-16T14:59:28.907568+0000","last_became_active":"2024-01-16T14:59:27.724363+0000","last_became_peered":"2024-01-16T14:59:27.724363+0000","last_unstale":"2024-01-16T14:59:28.907568+0000","last_undegraded":"2024-01-16T14:59:28.907568+0000","last_fullsized":"2024-01-16T14:59:28.907568+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:48.253356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906430+0000","last_change":"2024-01-16T14:59:27.722940+0000","last_active":"2024-01-16T14:59:28.906430+0000","last_peered":"2024-01-16T14:59:28.906430+0000","last_clean":"2024-01-16T14:59:28.906430+0000","last_became_active":"2024-01-16T14:59:27.722567+0000","last_became_peered":"2024-01-16T14:59:27.722567+0000","last_unstale":"2024-01-16T14:59:28.906430+0000","last_undegraded":"2024-01-16T14:59:28.906430+0000","last_fullsized":"2024-01-16T14:59:28.906430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:13.739812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703087+0000","last_change":"2024-01-16T14:59:27.722429+0000","last_active":"2024-01-16T14:59:28.703087+0000","last_peered":"2024-01-16T14:59:28.703087+0000","last_clean":"2024-01-16T14:59:28.703087+0000","last_became_active":"2024-01-16T14:59:27.722092+0000","last_became_peered":"2024-01-16T14:59:27.722092+0000","last_unstale":"2024-01-16T14:59:28.703087+0000","last_undegraded":"2024-01-16T14:59:28.703087+0000","last_fullsized":"2024-01-16T14:59:28.703087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709008+0000","last_change":"2024-01-16T14:59:27.719513+0000","last_active":"2024-01-16T14:59:28.709008+0000","last_peered":"2024-01-16T14:59:28.709008+0000","last_clean":"2024-01-16T14:59:28.709008+0000","last_became_active":"2024-01-16T14:59:27.718757+0000","last_became_peered":"2024-01-16T14:59:27.718757+0000","last_unstale":"2024-01-16T14:59:28.709008+0000","last_undegraded":"2024-01-16T14:59:28.709008+0000","last_fullsized":"2024-01-16T14:59:28.709008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.707752+0000","last_change":"2024-01-16T14:59:27.706529+0000","last_active":"2024-01-16T14:59:28.707752+0000","last_peered":"2024-01-16T14:59:28.707752+0000","last_clean":"2024-01-16T14:59:28.707752+0000","last_became_active":"2024-01-16T14:59:27.706241+0000","last_became_peered":"2024-01-16T14:59:27.706241+0000","last_unstale":"2024-01-16T14:59:28.707752+0000","last_undegraded":"2024-01-16T14:59:28.707752+0000","last_fullsized":"2024-01-16T14:59:28.707752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:40.765365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906601+0000","last_change":"2024-01-16T14:59:26.697929+0000","last_active":"2024-01-16T14:59:28.906601+0000","last_peered":"2024-01-16T14:59:28.906601+0000","last_clean":"2024-01-16T14:59:28.906601+0000","last_became_active":"2024-01-16T14:59:26.697578+0000","last_became_peered":"2024-01-16T14:59:26.697578+0000","last_unstale":"2024-01-16T14:59:28.906601+0000","last_undegraded":"2024-01-16T14:59:28.906601+0000","last_fullsized":"2024-01-16T14:59:28.906601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:26.816846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907687+0000","last_change":"2024-01-16T14:59:27.723311+0000","last_active":"2024-01-16T14:59:28.907687+0000","last_peered":"2024-01-16T14:59:28.907687+0000","last_clean":"2024-01-16T14:59:28.907687+0000","last_became_active":"2024-01-16T14:59:27.722975+0000","last_became_peered":"2024-01-16T14:59:27.722975+0000","last_unstale":"2024-01-16T14:59:28.907687+0000","last_undegraded":"2024-01-16T14:59:28.907687+0000","last_fullsized":"2024-01-16T14:59:28.907687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710186+0000","last_change":"2024-01-16T14:59:26.683247+0000","last_active":"2024-01-16T14:59:28.710186+0000","last_peered":"2024-01-16T14:59:28.710186+0000","last_clean":"2024-01-16T14:59:28.710186+0000","last_became_active":"2024-01-16T14:59:26.682727+0000","last_became_peered":"2024-01-16T14:59:26.682727+0000","last_unstale":"2024-01-16T14:59:28.710186+0000","last_undegraded":"2024-01-16T14:59:28.710186+0000","last_fullsized":"2024-01-16T14:59:28.710186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905763+0000","last_change":"2024-01-16T14:59:26.687905+0000","last_active":"2024-01-16T14:59:28.905763+0000","last_peered":"2024-01-16T14:59:28.905763+0000","last_clean":"2024-01-16T14:59:28.905763+0000","last_became_active":"2024-01-16T14:59:26.687570+0000","last_became_peered":"2024-01-16T14:59:26.687570+0000","last_unstale":"2024-01-16T14:59:28.905763+0000","last_undegraded":"2024-01-16T14:59:28.905763+0000","last_fullsized":"2024-01-16T14:59:28.905763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904948+0000","last_change":"2024-01-16T14:59:27.715786+0000","last_active":"2024-01-16T14:59:28.904948+0000","last_peered":"2024-01-16T14:59:28.904948+0000","last_clean":"2024-01-16T14:59:28.904948+0000","last_became_active":"2024-01-16T14:59:27.715516+0000","last_became_peered":"2024-01-16T14:59:27.715516+0000","last_unstale":"2024-01-16T14:59:28.904948+0000","last_undegraded":"2024-01-16T14:59:28.904948+0000","last_fullsized":"2024-01-16T14:59:28.904948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:14.095920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905832+0000","last_change":"2024-01-16T14:59:26.698294+0000","last_active":"2024-01-16T14:59:28.905832+0000","last_peered":"2024-01-16T14:59:28.905832+0000","last_clean":"2024-01-16T14:59:28.905832+0000","last_became_active":"2024-01-16T14:59:26.697917+0000","last_became_peered":"2024-01-16T14:59:26.697917+0000","last_unstale":"2024-01-16T14:59:28.905832+0000","last_undegraded":"2024-01-16T14:59:28.905832+0000","last_fullsized":"2024-01-16T14:59:28.905832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:15.675945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905800+0000","last_change":"2024-01-16T14:59:27.704715+0000","last_active":"2024-01-16T14:59:28.905800+0000","last_peered":"2024-01-16T14:59:28.905800+0000","last_clean":"2024-01-16T14:59:28.905800+0000","last_became_active":"2024-01-16T14:59:27.704439+0000","last_became_peered":"2024-01-16T14:59:27.704439+0000","last_unstale":"2024-01-16T14:59:28.905800+0000","last_undegraded":"2024-01-16T14:59:28.905800+0000","last_fullsized":"2024-01-16T14:59:28.905800+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710792+0000","last_change":"2024-01-16T14:59:26.689740+0000","last_active":"2024-01-16T14:59:28.710792+0000","last_peered":"2024-01-16T14:59:28.710792+0000","last_clean":"2024-01-16T14:59:28.710792+0000","last_became_active":"2024-01-16T14:59:26.689434+0000","last_became_peered":"2024-01-16T14:59:26.689434+0000","last_unstale":"2024-01-16T14:59:28.710792+0000","last_undegraded":"2024-01-16T14:59:28.710792+0000","last_fullsized":"2024-01-16T14:59:28.710792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:39.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907111+0000","last_change":"2024-01-16T14:59:27.733517+0000","last_active":"2024-01-16T14:59:28.907111+0000","last_peered":"2024-01-16T14:59:28.907111+0000","last_clean":"2024-01-16T14:59:28.907111+0000","last_became_active":"2024-01-16T14:59:27.732223+0000","last_became_peered":"2024-01-16T14:59:27.732223+0000","last_unstale":"2024-01-16T14:59:28.907111+0000","last_undegraded":"2024-01-16T14:59:28.907111+0000","last_fullsized":"2024-01-16T14:59:28.907111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:55.788928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907375+0000","last_change":"2024-01-16T14:59:26.693230+0000","last_active":"2024-01-16T14:59:28.907375+0000","last_peered":"2024-01-16T14:59:28.907375+0000","last_clean":"2024-01-16T14:59:28.907375+0000","last_became_active":"2024-01-16T14:59:26.693007+0000","last_became_peered":"2024-01-16T14:59:26.693007+0000","last_unstale":"2024-01-16T14:59:28.907375+0000","last_undegraded":"2024-01-16T14:59:28.907375+0000","last_fullsized":"2024-01-16T14:59:28.907375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909099+0000","last_change":"2024-01-16T14:59:27.723932+0000","last_active":"2024-01-16T14:59:28.909099+0000","last_peered":"2024-01-16T14:59:28.909099+0000","last_clean":"2024-01-16T14:59:28.909099+0000","last_became_active":"2024-01-16T14:59:27.723629+0000","last_became_peered":"2024-01-16T14:59:27.723629+0000","last_unstale":"2024-01-16T14:59:28.909099+0000","last_undegraded":"2024-01-16T14:59:28.909099+0000","last_fullsized":"2024-01-16T14:59:28.909099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908417+0000","last_change":"2024-01-16T14:59:26.698500+0000","last_active":"2024-01-16T14:59:28.908417+0000","last_peered":"2024-01-16T14:59:28.908417+0000","last_clean":"2024-01-16T14:59:28.908417+0000","last_became_active":"2024-01-16T14:59:26.698193+0000","last_became_peered":"2024-01-16T14:59:26.698193+0000","last_unstale":"2024-01-16T14:59:28.908417+0000","last_undegraded":"2024-01-16T14:59:28.908417+0000","last_fullsized":"2024-01-16T14:59:28.908417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:32.268892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709922+0000","last_change":"2024-01-16T14:59:27.724449+0000","last_active":"2024-01-16T14:59:28.709922+0000","last_peered":"2024-01-16T14:59:28.709922+0000","last_clean":"2024-01-16T14:59:28.709922+0000","last_became_active":"2024-01-16T14:59:27.723990+0000","last_became_peered":"2024-01-16T14:59:27.723990+0000","last_unstale":"2024-01-16T14:59:28.709922+0000","last_undegraded":"2024-01-16T14:59:28.709922+0000","last_fullsized":"2024-01-16T14:59:28.709922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710357+0000","last_change":"2024-01-16T14:59:26.688962+0000","last_active":"2024-01-16T14:59:28.710357+0000","last_peered":"2024-01-16T14:59:28.710357+0000","last_clean":"2024-01-16T14:59:28.710357+0000","last_became_active":"2024-01-16T14:59:26.687694+0000","last_became_peered":"2024-01-16T14:59:26.687694+0000","last_unstale":"2024-01-16T14:59:28.710357+0000","last_undegraded":"2024-01-16T14:59:28.710357+0000","last_fullsized":"2024-01-16T14:59:28.710357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907384+0000","last_change":"2024-01-16T14:59:27.718245+0000","last_active":"2024-01-16T14:59:28.907384+0000","last_peered":"2024-01-16T14:59:28.907384+0000","last_clean":"2024-01-16T14:59:28.907384+0000","last_became_active":"2024-01-16T14:59:27.717994+0000","last_became_peered":"2024-01-16T14:59:27.717994+0000","last_unstale":"2024-01-16T14:59:28.907384+0000","last_undegraded":"2024-01-16T14:59:28.907384+0000","last_fullsized":"2024-01-16T14:59:28.907384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:37.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907794+0000","last_change":"2024-01-16T14:59:26.682235+0000","last_active":"2024-01-16T14:59:28.907794+0000","last_peered":"2024-01-16T14:59:28.907794+0000","last_clean":"2024-01-16T14:59:28.907794+0000","last_became_active":"2024-01-16T14:59:26.681912+0000","last_became_peered":"2024-01-16T14:59:26.681912+0000","last_unstale":"2024-01-16T14:59:28.907794+0000","last_undegraded":"2024-01-16T14:59:28.907794+0000","last_fullsized":"2024-01-16T14:59:28.907794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906880+0000","last_change":"2024-01-16T14:59:27.718808+0000","last_active":"2024-01-16T14:59:28.906880+0000","last_peered":"2024-01-16T14:59:28.906880+0000","last_clean":"2024-01-16T14:59:28.906880+0000","last_became_active":"2024-01-16T14:59:27.717611+0000","last_became_peered":"2024-01-16T14:59:27.717611+0000","last_unstale":"2024-01-16T14:59:28.906880+0000","last_undegraded":"2024-01-16T14:59:28.906880+0000","last_fullsized":"2024-01-16T14:59:28.906880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:37.694723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906417+0000","last_change":"2024-01-16T14:59:26.701196+0000","last_active":"2024-01-16T14:59:28.906417+0000","last_peered":"2024-01-16T14:59:28.906417+0000","last_clean":"2024-01-16T14:59:28.906417+0000","last_became_active":"2024-01-16T14:59:26.700600+0000","last_became_peered":"2024-01-16T14:59:26.700600+0000","last_unstale":"2024-01-16T14:59:28.906417+0000","last_undegraded":"2024-01-16T14:59:28.906417+0000","last_fullsized":"2024-01-16T14:59:28.906417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:20.742041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906388+0000","last_change":"2024-01-16T14:59:27.729587+0000","last_active":"2024-01-16T14:59:28.906388+0000","last_peered":"2024-01-16T14:59:28.906388+0000","last_clean":"2024-01-16T14:59:28.906388+0000","last_became_active":"2024-01-16T14:59:27.729374+0000","last_became_peered":"2024-01-16T14:59:27.729374+0000","last_unstale":"2024-01-16T14:59:28.906388+0000","last_undegraded":"2024-01-16T14:59:28.906388+0000","last_fullsized":"2024-01-16T14:59:28.906388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:07.977388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709779+0000","last_change":"2024-01-16T14:59:26.682595+0000","last_active":"2024-01-16T14:59:28.709779+0000","last_peered":"2024-01-16T14:59:28.709779+0000","last_clean":"2024-01-16T14:59:28.709779+0000","last_became_active":"2024-01-16T14:59:26.682263+0000","last_became_peered":"2024-01-16T14:59:26.682263+0000","last_unstale":"2024-01-16T14:59:28.709779+0000","last_undegraded":"2024-01-16T14:59:28.709779+0000","last_fullsized":"2024-01-16T14:59:28.709779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:05.652841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709357+0000","last_change":"2024-01-16T14:59:27.721378+0000","last_active":"2024-01-16T14:59:28.709357+0000","last_peered":"2024-01-16T14:59:28.709357+0000","last_clean":"2024-01-16T14:59:28.709357+0000","last_became_active":"2024-01-16T14:59:27.721133+0000","last_became_peered":"2024-01-16T14:59:27.721133+0000","last_unstale":"2024-01-16T14:59:28.709357+0000","last_undegraded":"2024-01-16T14:59:28.709357+0000","last_fullsized":"2024-01-16T14:59:28.709357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703039+0000","last_change":"2024-01-16T14:59:26.688711+0000","last_active":"2024-01-16T14:59:28.703039+0000","last_peered":"2024-01-16T14:59:28.703039+0000","last_clean":"2024-01-16T14:59:28.703039+0000","last_became_active":"2024-01-16T14:59:26.688404+0000","last_became_peered":"2024-01-16T14:59:26.688404+0000","last_unstale":"2024-01-16T14:59:28.703039+0000","last_undegraded":"2024-01-16T14:59:28.703039+0000","last_fullsized":"2024-01-16T14:59:28.703039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904624+0000","last_change":"2024-01-16T14:59:27.724086+0000","last_active":"2024-01-16T14:59:28.904624+0000","last_peered":"2024-01-16T14:59:28.904624+0000","last_clean":"2024-01-16T14:59:28.904624+0000","last_became_active":"2024-01-16T14:59:27.723846+0000","last_became_peered":"2024-01-16T14:59:27.723846+0000","last_unstale":"2024-01-16T14:59:28.904624+0000","last_undegraded":"2024-01-16T14:59:28.904624+0000","last_fullsized":"2024-01-16T14:59:28.904624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.530195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709625+0000","last_change":"2024-01-16T14:59:26.687473+0000","last_active":"2024-01-16T14:59:28.709625+0000","last_peered":"2024-01-16T14:59:28.709625+0000","last_clean":"2024-01-16T14:59:28.709625+0000","last_became_active":"2024-01-16T14:59:26.687161+0000","last_became_peered":"2024-01-16T14:59:26.687161+0000","last_unstale":"2024-01-16T14:59:28.709625+0000","last_undegraded":"2024-01-16T14:59:28.709625+0000","last_fullsized":"2024-01-16T14:59:28.709625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908536+0000","last_change":"2024-01-16T14:59:27.726514+0000","last_active":"2024-01-16T14:59:28.908536+0000","last_peered":"2024-01-16T14:59:28.908536+0000","last_clean":"2024-01-16T14:59:28.908536+0000","last_became_active":"2024-01-16T14:59:27.726196+0000","last_became_peered":"2024-01-16T14:59:27.726196+0000","last_unstale":"2024-01-16T14:59:28.908536+0000","last_undegraded":"2024-01-16T14:59:28.908536+0000","last_fullsized":"2024-01-16T14:59:28.908536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:35.130412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703648+0000","last_change":"2024-01-16T14:59:26.693748+0000","last_active":"2024-01-16T14:59:28.703648+0000","last_peered":"2024-01-16T14:59:28.703648+0000","last_clean":"2024-01-16T14:59:28.703648+0000","last_became_active":"2024-01-16T14:59:26.693411+0000","last_became_peered":"2024-01-16T14:59:26.693411+0000","last_unstale":"2024-01-16T14:59:28.703648+0000","last_undegraded":"2024-01-16T14:59:28.703648+0000","last_fullsized":"2024-01-16T14:59:28.703648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908456+0000","last_change":"2024-01-16T14:59:27.724021+0000","last_active":"2024-01-16T14:59:28.908456+0000","last_peered":"2024-01-16T14:59:28.908456+0000","last_clean":"2024-01-16T14:59:28.908456+0000","last_became_active":"2024-01-16T14:59:27.723760+0000","last_became_peered":"2024-01-16T14:59:27.723760+0000","last_unstale":"2024-01-16T14:59:28.908456+0000","last_undegraded":"2024-01-16T14:59:28.908456+0000","last_fullsized":"2024-01-16T14:59:28.908456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:56.547755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908196+0000","last_change":"2024-01-16T14:59:26.687768+0000","last_active":"2024-01-16T14:59:28.908196+0000","last_peered":"2024-01-16T14:59:28.908196+0000","last_clean":"2024-01-16T14:59:28.908196+0000","last_became_active":"2024-01-16T14:59:26.687439+0000","last_became_peered":"2024-01-16T14:59:26.687439+0000","last_unstale":"2024-01-16T14:59:28.908196+0000","last_undegraded":"2024-01-16T14:59:28.908196+0000","last_fullsized":"2024-01-16T14:59:28.908196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905945+0000","last_change":"2024-01-16T14:59:27.716631+0000","last_active":"2024-01-16T14:59:28.905945+0000","last_peered":"2024-01-16T14:59:28.905945+0000","last_clean":"2024-01-16T14:59:28.905945+0000","last_became_active":"2024-01-16T14:59:27.716370+0000","last_became_peered":"2024-01-16T14:59:27.716370+0000","last_unstale":"2024-01-16T14:59:28.905945+0000","last_undegraded":"2024-01-16T14:59:28.905945+0000","last_fullsized":"2024-01-16T14:59:28.905945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.471222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703521+0000","last_change":"2024-01-16T14:59:26.695961+0000","last_active":"2024-01-16T14:59:28.703521+0000","last_peered":"2024-01-16T14:59:28.703521+0000","last_clean":"2024-01-16T14:59:28.703521+0000","last_became_active":"2024-01-16T14:59:26.695602+0000","last_became_peered":"2024-01-16T14:59:26.695602+0000","last_unstale":"2024-01-16T14:59:28.703521+0000","last_undegraded":"2024-01-16T14:59:28.703521+0000","last_fullsized":"2024-01-16T14:59:28.703521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:07.651834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907463+0000","last_change":"2024-01-16T14:59:27.736089+0000","last_active":"2024-01-16T14:59:28.907463+0000","last_peered":"2024-01-16T14:59:28.907463+0000","last_clean":"2024-01-16T14:59:28.907463+0000","last_became_active":"2024-01-16T14:59:27.731504+0000","last_became_peered":"2024-01-16T14:59:27.731504+0000","last_unstale":"2024-01-16T14:59:28.907463+0000","last_undegraded":"2024-01-16T14:59:28.907463+0000","last_fullsized":"2024-01-16T14:59:28.907463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:20.319731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:27.705910+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T14:59:27.705910+0000","last_peered":"2024-01-16T14:59:27.705910+0000","last_clean":"2024-01-16T14:59:27.705910+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T14:59:27.705910+0000","last_undegraded":"2024-01-16T14:59:27.705910+0000","last_fullsized":"2024-01-16T14:59:27.705910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905174+0000","last_change":"2024-01-16T14:59:26.693490+0000","last_active":"2024-01-16T14:59:28.905174+0000","last_peered":"2024-01-16T14:59:28.905174+0000","last_clean":"2024-01-16T14:59:28.905174+0000","last_became_active":"2024-01-16T14:59:26.692487+0000","last_became_peered":"2024-01-16T14:59:26.692487+0000","last_unstale":"2024-01-16T14:59:28.905174+0000","last_undegraded":"2024-01-16T14:59:28.905174+0000","last_fullsized":"2024-01-16T14:59:28.905174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906140+0000","last_change":"2024-01-16T14:59:27.721477+0000","last_active":"2024-01-16T14:59:28.906140+0000","last_peered":"2024-01-16T14:59:28.906140+0000","last_clean":"2024-01-16T14:59:28.906140+0000","last_became_active":"2024-01-16T14:59:27.720924+0000","last_became_peered":"2024-01-16T14:59:27.720924+0000","last_unstale":"2024-01-16T14:59:28.906140+0000","last_undegraded":"2024-01-16T14:59:28.906140+0000","last_fullsized":"2024-01-16T14:59:28.906140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.326753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904387+0000","last_change":"2024-01-16T14:59:26.701503+0000","last_active":"2024-01-16T14:59:28.904387+0000","last_peered":"2024-01-16T14:59:28.904387+0000","last_clean":"2024-01-16T14:59:28.904387+0000","last_became_active":"2024-01-16T14:59:26.701190+0000","last_became_peered":"2024-01-16T14:59:26.701190+0000","last_unstale":"2024-01-16T14:59:28.904387+0000","last_undegraded":"2024-01-16T14:59:28.904387+0000","last_fullsized":"2024-01-16T14:59:28.904387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906011+0000","last_change":"2024-01-16T14:59:27.728625+0000","last_active":"2024-01-16T14:59:28.906011+0000","last_peered":"2024-01-16T14:59:28.906011+0000","last_clean":"2024-01-16T14:59:28.906011+0000","last_became_active":"2024-01-16T14:59:27.728177+0000","last_became_peered":"2024-01-16T14:59:27.728177+0000","last_unstale":"2024-01-16T14:59:28.906011+0000","last_undegraded":"2024-01-16T14:59:28.906011+0000","last_fullsized":"2024-01-16T14:59:28.906011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:38.465332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703101+0000","last_change":"2024-01-16T14:59:26.695022+0000","last_active":"2024-01-16T14:59:28.703101+0000","last_peered":"2024-01-16T14:59:28.703101+0000","last_clean":"2024-01-16T14:59:28.703101+0000","last_became_active":"2024-01-16T14:59:26.694756+0000","last_became_peered":"2024-01-16T14:59:26.694756+0000","last_unstale":"2024-01-16T14:59:28.703101+0000","last_undegraded":"2024-01-16T14:59:28.703101+0000","last_fullsized":"2024-01-16T14:59:28.703101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:41.781228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905205+0000","last_change":"2024-01-16T14:59:27.706635+0000","last_active":"2024-01-16T14:59:28.905205+0000","last_peered":"2024-01-16T14:59:28.905205+0000","last_clean":"2024-01-16T14:59:28.905205+0000","last_became_active":"2024-01-16T14:59:27.706296+0000","last_became_peered":"2024-01-16T14:59:27.706296+0000","last_unstale":"2024-01-16T14:59:28.905205+0000","last_undegraded":"2024-01-16T14:59:28.905205+0000","last_fullsized":"2024-01-16T14:59:28.905205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906768+0000","last_change":"2024-01-16T14:59:26.700908+0000","last_active":"2024-01-16T14:59:28.906768+0000","last_peered":"2024-01-16T14:59:28.906768+0000","last_clean":"2024-01-16T14:59:28.906768+0000","last_became_active":"2024-01-16T14:59:26.699693+0000","last_became_peered":"2024-01-16T14:59:26.699693+0000","last_unstale":"2024-01-16T14:59:28.906768+0000","last_undegraded":"2024-01-16T14:59:28.906768+0000","last_fullsized":"2024-01-16T14:59:28.906768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:10.078798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708429+0000","last_change":"2024-01-16T14:59:27.717434+0000","last_active":"2024-01-16T14:59:28.708429+0000","last_peered":"2024-01-16T14:59:28.708429+0000","last_clean":"2024-01-16T14:59:28.708429+0000","last_became_active":"2024-01-16T14:59:27.717061+0000","last_became_peered":"2024-01-16T14:59:27.717061+0000","last_unstale":"2024-01-16T14:59:28.708429+0000","last_undegraded":"2024-01-16T14:59:28.708429+0000","last_fullsized":"2024-01-16T14:59:28.708429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:14.259904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907983+0000","last_change":"2024-01-16T14:59:26.689585+0000","last_active":"2024-01-16T14:59:28.907983+0000","last_peered":"2024-01-16T14:59:28.907983+0000","last_clean":"2024-01-16T14:59:28.907983+0000","last_became_active":"2024-01-16T14:59:26.689193+0000","last_became_peered":"2024-01-16T14:59:26.689193+0000","last_unstale":"2024-01-16T14:59:28.907983+0000","last_undegraded":"2024-01-16T14:59:28.907983+0000","last_fullsized":"2024-01-16T14:59:28.907983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:40.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907283+0000","last_change":"2024-01-16T14:59:27.723605+0000","last_active":"2024-01-16T14:59:28.907283+0000","last_peered":"2024-01-16T14:59:28.907283+0000","last_clean":"2024-01-16T14:59:28.907283+0000","last_became_active":"2024-01-16T14:59:27.723322+0000","last_became_peered":"2024-01-16T14:59:27.723322+0000","last_unstale":"2024-01-16T14:59:28.907283+0000","last_undegraded":"2024-01-16T14:59:28.907283+0000","last_fullsized":"2024-01-16T14:59:28.907283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.187834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908060+0000","last_change":"2024-01-16T14:59:26.701967+0000","last_active":"2024-01-16T14:59:28.908060+0000","last_peered":"2024-01-16T14:59:28.908060+0000","last_clean":"2024-01-16T14:59:28.908060+0000","last_became_active":"2024-01-16T14:59:26.701686+0000","last_became_peered":"2024-01-16T14:59:26.701686+0000","last_unstale":"2024-01-16T14:59:28.908060+0000","last_undegraded":"2024-01-16T14:59:28.908060+0000","last_fullsized":"2024-01-16T14:59:28.908060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:09.766721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907757+0000","last_change":"2024-01-16T14:59:27.709576+0000","last_active":"2024-01-16T14:59:28.907757+0000","last_peered":"2024-01-16T14:59:28.907757+0000","last_clean":"2024-01-16T14:59:28.907757+0000","last_became_active":"2024-01-16T14:59:27.706317+0000","last_became_peered":"2024-01-16T14:59:27.706317+0000","last_unstale":"2024-01-16T14:59:28.907757+0000","last_undegraded":"2024-01-16T14:59:28.907757+0000","last_fullsized":"2024-01-16T14:59:28.907757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908338+0000","last_change":"2024-01-16T14:59:26.693739+0000","last_active":"2024-01-16T14:59:28.908338+0000","last_peered":"2024-01-16T14:59:28.908338+0000","last_clean":"2024-01-16T14:59:28.908338+0000","last_became_active":"2024-01-16T14:59:26.693473+0000","last_became_peered":"2024-01-16T14:59:26.693473+0000","last_unstale":"2024-01-16T14:59:28.908338+0000","last_undegraded":"2024-01-16T14:59:28.908338+0000","last_fullsized":"2024-01-16T14:59:28.908338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906445+0000","last_change":"2024-01-16T14:59:27.717414+0000","last_active":"2024-01-16T14:59:28.906445+0000","last_peered":"2024-01-16T14:59:28.906445+0000","last_clean":"2024-01-16T14:59:28.906445+0000","last_became_active":"2024-01-16T14:59:27.717078+0000","last_became_peered":"2024-01-16T14:59:27.717078+0000","last_unstale":"2024-01-16T14:59:28.906445+0000","last_undegraded":"2024-01-16T14:59:28.906445+0000","last_fullsized":"2024-01-16T14:59:28.906445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:30.634017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711015+0000","last_change":"2024-01-16T14:59:26.696098+0000","last_active":"2024-01-16T14:59:28.711015+0000","last_peered":"2024-01-16T14:59:28.711015+0000","last_clean":"2024-01-16T14:59:28.711015+0000","last_became_active":"2024-01-16T14:59:26.695792+0000","last_became_peered":"2024-01-16T14:59:26.695792+0000","last_unstale":"2024-01-16T14:59:28.711015+0000","last_undegraded":"2024-01-16T14:59:28.711015+0000","last_fullsized":"2024-01-16T14:59:28.711015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908934+0000","last_change":"2024-01-16T14:59:27.712757+0000","last_active":"2024-01-16T14:59:28.908934+0000","last_peered":"2024-01-16T14:59:28.908934+0000","last_clean":"2024-01-16T14:59:28.908934+0000","last_became_active":"2024-01-16T14:59:27.712417+0000","last_became_peered":"2024-01-16T14:59:27.712417+0000","last_unstale":"2024-01-16T14:59:28.908934+0000","last_undegraded":"2024-01-16T14:59:28.908934+0000","last_fullsized":"2024-01-16T14:59:28.908934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904378+0000","last_change":"2024-01-16T14:59:26.699479+0000","last_active":"2024-01-16T14:59:28.904378+0000","last_peered":"2024-01-16T14:59:28.904378+0000","last_clean":"2024-01-16T14:59:28.904378+0000","last_became_active":"2024-01-16T14:59:26.699195+0000","last_became_peered":"2024-01-16T14:59:26.699195+0000","last_unstale":"2024-01-16T14:59:28.904378+0000","last_undegraded":"2024-01-16T14:59:28.904378+0000","last_fullsized":"2024-01-16T14:59:28.904378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.521651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708137+0000","last_change":"2024-01-16T14:59:27.715745+0000","last_active":"2024-01-16T14:59:28.708137+0000","last_peered":"2024-01-16T14:59:28.708137+0000","last_clean":"2024-01-16T14:59:28.708137+0000","last_became_active":"2024-01-16T14:59:27.710835+0000","last_became_peered":"2024-01-16T14:59:27.710835+0000","last_unstale":"2024-01-16T14:59:28.708137+0000","last_undegraded":"2024-01-16T14:59:28.708137+0000","last_fullsized":"2024-01-16T14:59:28.708137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:53.870850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906409+0000","last_change":"2024-01-16T14:59:26.697228+0000","last_active":"2024-01-16T14:59:28.906409+0000","last_peered":"2024-01-16T14:59:28.906409+0000","last_clean":"2024-01-16T14:59:28.906409+0000","last_became_active":"2024-01-16T14:59:26.696136+0000","last_became_peered":"2024-01-16T14:59:26.696136+0000","last_unstale":"2024-01-16T14:59:28.906409+0000","last_undegraded":"2024-01-16T14:59:28.906409+0000","last_fullsized":"2024-01-16T14:59:28.906409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:37.293826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905459+0000","last_change":"2024-01-16T14:59:27.711533+0000","last_active":"2024-01-16T14:59:28.905459+0000","last_peered":"2024-01-16T14:59:28.905459+0000","last_clean":"2024-01-16T14:59:28.905459+0000","last_became_active":"2024-01-16T14:59:27.711228+0000","last_became_peered":"2024-01-16T14:59:27.711228+0000","last_unstale":"2024-01-16T14:59:28.905459+0000","last_undegraded":"2024-01-16T14:59:28.905459+0000","last_fullsized":"2024-01-16T14:59:28.905459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:28.132533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907464+0000","last_change":"2024-01-16T14:59:26.695937+0000","last_active":"2024-01-16T14:59:28.907464+0000","last_peered":"2024-01-16T14:59:28.907464+0000","last_clean":"2024-01-16T14:59:28.907464+0000","last_became_active":"2024-01-16T14:59:26.695628+0000","last_became_peered":"2024-01-16T14:59:26.695628+0000","last_unstale":"2024-01-16T14:59:28.907464+0000","last_undegraded":"2024-01-16T14:59:28.907464+0000","last_fullsized":"2024-01-16T14:59:28.907464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:36.523217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906350+0000","last_change":"2024-01-16T14:59:27.720565+0000","last_active":"2024-01-16T14:59:28.906350+0000","last_peered":"2024-01-16T14:59:28.906350+0000","last_clean":"2024-01-16T14:59:28.906350+0000","last_became_active":"2024-01-16T14:59:27.716762+0000","last_became_peered":"2024-01-16T14:59:27.716762+0000","last_unstale":"2024-01-16T14:59:28.906350+0000","last_undegraded":"2024-01-16T14:59:28.906350+0000","last_fullsized":"2024-01-16T14:59:28.906350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:26.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906643+0000","last_change":"2024-01-16T14:59:26.700414+0000","last_active":"2024-01-16T14:59:28.906643+0000","last_peered":"2024-01-16T14:59:28.906643+0000","last_clean":"2024-01-16T14:59:28.906643+0000","last_became_active":"2024-01-16T14:59:26.700111+0000","last_became_peered":"2024-01-16T14:59:26.700111+0000","last_unstale":"2024-01-16T14:59:28.906643+0000","last_undegraded":"2024-01-16T14:59:28.906643+0000","last_fullsized":"2024-01-16T14:59:28.906643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:29.462511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709936+0000","last_change":"2024-01-16T14:59:27.718486+0000","last_active":"2024-01-16T14:59:28.709936+0000","last_peered":"2024-01-16T14:59:28.709936+0000","last_clean":"2024-01-16T14:59:28.709936+0000","last_became_active":"2024-01-16T14:59:27.712753+0000","last_became_peered":"2024-01-16T14:59:27.712753+0000","last_unstale":"2024-01-16T14:59:28.709936+0000","last_undegraded":"2024-01-16T14:59:28.709936+0000","last_fullsized":"2024-01-16T14:59:28.709936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907057+0000","last_change":"2024-01-16T14:59:27.736702+0000","last_active":"2024-01-16T14:59:28.907057+0000","last_peered":"2024-01-16T14:59:28.907057+0000","last_clean":"2024-01-16T14:59:28.907057+0000","last_became_active":"2024-01-16T14:59:27.736433+0000","last_became_peered":"2024-01-16T14:59:27.736433+0000","last_unstale":"2024-01-16T14:59:28.907057+0000","last_undegraded":"2024-01-16T14:59:28.907057+0000","last_fullsized":"2024-01-16T14:59:28.907057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:13.830199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907201+0000","last_change":"2024-01-16T14:59:26.703355+0000","last_active":"2024-01-16T14:59:28.907201+0000","last_peered":"2024-01-16T14:59:28.907201+0000","last_clean":"2024-01-16T14:59:28.907201+0000","last_became_active":"2024-01-16T14:59:26.703059+0000","last_became_peered":"2024-01-16T14:59:26.703059+0000","last_unstale":"2024-01-16T14:59:28.907201+0000","last_undegraded":"2024-01-16T14:59:28.907201+0000","last_fullsized":"2024-01-16T14:59:28.907201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.422714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905630+0000","last_change":"2024-01-16T14:59:27.726539+0000","last_active":"2024-01-16T14:59:28.905630+0000","last_peered":"2024-01-16T14:59:28.905630+0000","last_clean":"2024-01-16T14:59:28.905630+0000","last_became_active":"2024-01-16T14:59:27.726269+0000","last_became_peered":"2024-01-16T14:59:27.726269+0000","last_unstale":"2024-01-16T14:59:28.905630+0000","last_undegraded":"2024-01-16T14:59:28.905630+0000","last_fullsized":"2024-01-16T14:59:28.905630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907840+0000","last_change":"2024-01-16T14:59:26.702143+0000","last_active":"2024-01-16T14:59:28.907840+0000","last_peered":"2024-01-16T14:59:28.907840+0000","last_clean":"2024-01-16T14:59:28.907840+0000","last_became_active":"2024-01-16T14:59:26.701732+0000","last_became_peered":"2024-01-16T14:59:26.701732+0000","last_unstale":"2024-01-16T14:59:28.907840+0000","last_undegraded":"2024-01-16T14:59:28.907840+0000","last_fullsized":"2024-01-16T14:59:28.907840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:44.454533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908894+0000","last_change":"2024-01-16T14:59:27.719650+0000","last_active":"2024-01-16T14:59:28.908894+0000","last_peered":"2024-01-16T14:59:28.908894+0000","last_clean":"2024-01-16T14:59:28.908894+0000","last_became_active":"2024-01-16T14:59:27.719345+0000","last_became_peered":"2024-01-16T14:59:27.719345+0000","last_unstale":"2024-01-16T14:59:28.908894+0000","last_undegraded":"2024-01-16T14:59:28.908894+0000","last_fullsized":"2024-01-16T14:59:28.908894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906403+0000","last_change":"2024-01-16T14:59:26.692569+0000","last_active":"2024-01-16T14:59:28.906403+0000","last_peered":"2024-01-16T14:59:28.906403+0000","last_clean":"2024-01-16T14:59:28.906403+0000","last_became_active":"2024-01-16T14:59:26.689432+0000","last_became_peered":"2024-01-16T14:59:26.689432+0000","last_unstale":"2024-01-16T14:59:28.906403+0000","last_undegraded":"2024-01-16T14:59:28.906403+0000","last_fullsized":"2024-01-16T14:59:28.906403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:58.385575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905910+0000","last_change":"2024-01-16T14:59:27.728184+0000","last_active":"2024-01-16T14:59:28.905910+0000","last_peered":"2024-01-16T14:59:28.905910+0000","last_clean":"2024-01-16T14:59:28.905910+0000","last_became_active":"2024-01-16T14:59:27.727889+0000","last_became_peered":"2024-01-16T14:59:27.727889+0000","last_unstale":"2024-01-16T14:59:28.905910+0000","last_undegraded":"2024-01-16T14:59:28.905910+0000","last_fullsized":"2024-01-16T14:59:28.905910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.211584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909226+0000","last_change":"2024-01-16T14:59:26.697456+0000","last_active":"2024-01-16T14:59:28.909226+0000","last_peered":"2024-01-16T14:59:28.909226+0000","last_clean":"2024-01-16T14:59:28.909226+0000","last_became_active":"2024-01-16T14:59:26.697129+0000","last_became_peered":"2024-01-16T14:59:26.697129+0000","last_unstale":"2024-01-16T14:59:28.909226+0000","last_undegraded":"2024-01-16T14:59:28.909226+0000","last_fullsized":"2024-01-16T14:59:28.909226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9119999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5459999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2309999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.542}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":74114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4140000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8799999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":819200,"data_stored":663508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:32.682 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T14:59:32.683 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T14:59:32.683 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T14:59:32.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T14:59:32.684 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T14:59:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.957+0000 7f1cc7e00640 1 Processor -- start 2024-01-16T14:59:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.957+0000 7f1cc7e00640 1 -- start start 2024-01-16T14:59:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 0x7f1cc815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cc7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1cc80cf050 0x7f1cc80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cc7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80ce7c0 con 0x7f1cc806c130 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cc7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80cfa10 con 0x7f1cc80cf050 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8140900 con 0x7f1cc813dfd0 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cbde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1cc80cf050 0x7f1cc80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cbde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1cc80cf050 0x7f1cc80cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34028/0 (socket says 172.21.15.43:34028) 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.959+0000 7f1cbde00640 1 -- 172.21.15.43:0/1493186918 learned_addr learned my addr 172.21.15.43:0/1493186918 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cbde00640 1 -- 172.21.15.43:0/1493186918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 msgr2=0x7f1cc815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cc6a00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 0x7f1cc815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cbde00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 0x7f1cc815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cbde00640 1 -- 172.21.15.43:0/1493186918 >> v1:172.21.15.69:6789/0 conn(0x7f1cc806c130 legacy=0x7f1cc806c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cbde00640 1 -- 172.21.15.43:0/1493186918 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc815fa70 con 0x7f1cc80cf050 2024-01-16T14:59:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.960+0000 7f1cc6a00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 0x7f1cc815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.961+0000 7f1cbde00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc80cf050 0x7f1cc80cf4d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc152e0d34d72f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.961+0000 7f1cc6000640 1 -- 172.21.15.43:0/1493186918 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0031640 con 0x7f1cc80cf050 2024-01-16T14:59:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.961+0000 7f1cc6000640 1 -- 172.21.15.43:0/1493186918 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb00317a0 con 0x7f1cc80cf050 2024-01-16T14:59:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.961+0000 7f1cc6000640 1 -- 172.21.15.43:0/1493186918 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0031b30 con 0x7f1cc80cf050 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 -- 172.21.15.43:0/1493186918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc80cf050 msgr2=0x7f1cc80cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc80cf050 0x7f1cc80cf4d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 -- 172.21.15.43:0/1493186918 shutdown_connections 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc80cf050 0x7f1cc80cf4d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/1493186918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc813dfd0 0x7f1cc815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 -- 172.21.15.43:0/1493186918 >> 172.21.15.43:0/1493186918 conn(0x7f1cc806a860 msgr2=0x7f1cc806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 -- 172.21.15.43:0/1493186918 shutdown_connections 2024-01-16T14:59:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.962+0000 7f1cc7e00640 1 -- 172.21.15.43:0/1493186918 wait complete. 2024-01-16T14:59:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.964+0000 7f1cc7e00640 1 Processor -- start 2024-01-16T14:59:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.964+0000 7f1cc7e00640 1 -- start start 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cc806c130 0x7f1cc8154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc80cf050 0x7f1cc813ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8070b80 con 0x7f1cc813dfd0 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8070e60 con 0x7f1cc806c130 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8071140 con 0x7f1cc80cf050 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34032/0 (socket says 172.21.15.43:34032) 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc7400640 1 -- 172.21.15.43:0/3638533406 learned_addr learned my addr 172.21.15.43:0/3638533406 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cbde00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc80cf050 0x7f1cc813ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.965+0000 7f1cc6a00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cc806c130 0x7f1cc8154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 -- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc80cf050 msgr2=0x7f1cc813ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc80cf050 0x7f1cc813ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 -- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cc806c130 msgr2=0x7f1cc8154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cc806c130 0x7f1cc8154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 -- 172.21.15.43:0/3638533406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00094d0 con 0x7f1cc813dfd0 2024-01-16T14:59:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7400640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000c780 tx=0x7f1cc000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0004b60 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc000ce90 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc8071420 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc000a850 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.966+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc8071dc0 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.967+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc8072010 con 0x7f1cc813dfd0 2024-01-16T14:59:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.968+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1cc000a9b0 con 0x7f1cc813dfd0 2024-01-16T14:59:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.969+0000 7f1cbf400640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 0x7f1ca006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.969+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cc0089e30 con 0x7f1cc813dfd0 2024-01-16T14:59:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.970+0000 7f1cc6a00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 0x7f1ca006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.970+0000 7f1cc6a00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 0x7f1ca006d7d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81f6660 tx=0x7f1cb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:32.974+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cc0054c80 con 0x7f1cc813dfd0 2024-01-16T14:59:33.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.216+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1cc8071940 con 0x7f1cc813dfd0 2024-01-16T14:59:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.222+0000 7f1cbf400640 1 -- 172.21.15.43:0/3638533406 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171553 (secure 0 0 0) 0x7f1cc8071940 con 0x7f1cc813dfd0 2024-01-16T14:59:33.204 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 msgr2=0x7f1ca006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 0x7f1ca006d7d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81f6660 tx=0x7f1cb8008040 comp rx=0 tx=0).stop 2024-01-16T14:59:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 msgr2=0x7f1cc80b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000c780 tx=0x7f1cc000cc40 comp rx=0 tx=0).stop 2024-01-16T14:59:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 shutdown_connections 2024-01-16T14:59:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ca006b310 0x7f1ca006d7d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1cc813dfd0 0x7f1cc80b2220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cc80cf050 0x7f1cc813ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.227+0000 7f1cc7e00640 1 --2- 172.21.15.43:0/3638533406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cc806c130 0x7f1cc8154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.228+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 >> 172.21.15.43:0/3638533406 conn(0x7f1cc806a860 msgr2=0x7f1cc8159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.228+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 shutdown_connections 2024-01-16T14:59:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.228+0000 7f1cc7e00640 1 -- 172.21.15.43:0/3638533406 wait complete. 2024-01-16T14:59:33.228 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":504016423},{"type":"v1","addr":"172.21.15.43:6833","nonce":504016423}]},"active_addr":"172.21.15.43:6833/504016423","active_change":"2024-01-16T14:59:12.765771+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3651484719}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3355055558}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":593090099}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":125344714}]}]} 2024-01-16T14:59:33.232 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T14:59:33.232 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T14:59:33.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.472+0000 7faad8a00640 1 Processor -- start 2024-01-16T14:59:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.472+0000 7faad8a00640 1 -- start start 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.473+0000 7faad8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faad40cd6e0 0x7faad40cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.473+0000 7faad8a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40cbf10 con 0x7faad40ce0d0 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.473+0000 7faad8a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40ce500 con 0x7faad415d550 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.473+0000 7faad8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40d05e0 con 0x7faad40cd6e0 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7faad40ce0d0 0x7faad40cc3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52892/0 (socket says 172.21.15.43:52892) 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad3e00640 1 -- 172.21.15.43:0/1280224873 learned_addr learned my addr 172.21.15.43:0/1280224873 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2a00640 1 --2- 172.21.15.43:0/1280224873 >> v2:172.21.15.69:3300/0 conn(0x7faad40cd6e0 0x7faad40cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952368782 0 0) 0x7faad40cbf10 con 0x7faad40ce0d0 2024-01-16T14:59:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab4003520 con 0x7faad40ce0d0 2024-01-16T14:59:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2148065617 0 0) 0x7faab4003520 con 0x7faad40ce0d0 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 >> v2:172.21.15.69:3300/0 conn(0x7faad40cd6e0 msgr2=0x7faad40cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 --2- 172.21.15.43:0/1280224873 >> v2:172.21.15.69:3300/0 conn(0x7faad40cd6e0 0x7faad40cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 >> v1:172.21.15.69:6789/0 conn(0x7faad415d550 legacy=0x7faad40cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40cb5e0 con 0x7faad40ce0d0 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7faacc0019f0 con 0x7faad40ce0d0 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.474+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 >> v1:172.21.15.43:6789/0 conn(0x7faad40ce0d0 legacy=0x7faad40cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 0x7faab4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab40047c0 0x7faab4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab40052a0 0x7faab400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40d05e0 con 0x7faab4003a30 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab400c170 con 0x7faab40052a0 2024-01-16T14:59:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab400c330 con 0x7faab40047c0 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 0x7faab4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad2a00640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab40047c0 0x7faab4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 -- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab40047c0 msgr2=0x7faab4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab40047c0 0x7faab4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 -- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab40052a0 msgr2=0x7faab400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab40052a0 0x7faab400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3e00640 1 -- 172.21.15.43:0/1280224873 reap_dead start 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.475+0000 7faad3400640 1 -- 172.21.15.43:0/1280224873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4003520 con 0x7faab4003a30 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.476+0000 7faad3400640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 0x7faab4003e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac4001750 tx=0x7faab40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5ac01f368c0865 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac40052d0 con 0x7faab4003a30 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac402e050 con 0x7faab4003a30 2024-01-16T14:59:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad2000640 1 -- 172.21.15.43:0/1280224873 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4003830 con 0x7faab4003a30 2024-01-16T14:59:33.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad8a00640 1 -- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 msgr2=0x7faab4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad8a00640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 0x7faab4003e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac4001750 tx=0x7faab40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad8a00640 1 -- 172.21.15.43:0/1280224873 shutdown_connections 2024-01-16T14:59:33.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad8a00640 1 --2- 172.21.15.43:0/1280224873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faab4003a30 0x7faab4003e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.477+0000 7faad8a00640 1 -- 172.21.15.43:0/1280224873 >> 172.21.15.43:0/1280224873 conn(0x7faad40d0e60 msgr2=0x7faad40d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.478+0000 7faad8a00640 1 -- 172.21.15.43:0/1280224873 shutdown_connections 2024-01-16T14:59:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.478+0000 7faad8a00640 1 -- 172.21.15.43:0/1280224873 wait complete. 2024-01-16T14:59:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.479+0000 7faad8a00640 1 Processor -- start 2024-01-16T14:59:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.480+0000 7faad8a00640 1 -- start start 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab4003a30 0x7faad4157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad406c090 0x7faad406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad409e800 con 0x7faad40ca4d0 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad406ca30 con 0x7faad406c090 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad4157d60 con 0x7faab4003a30 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34044/0 (socket says 172.21.15.43:34044) 2024-01-16T14:59:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 -- 172.21.15.43:0/747491998 learned_addr learned my addr 172.21.15.43:0/747491998 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 -- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab4003a30 msgr2=0x7faad4157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab4003a30 0x7faad4157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 -- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad406c090 msgr2=0x7faad406c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad2a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad406c090 0x7faad406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad406c090 0x7faad406c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.481+0000 7faad3e00640 1 -- 172.21.15.43:0/747491998 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad415ea60 con 0x7faad40ca4d0 2024-01-16T14:59:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faad3e00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faacc001370 tx=0x7faacc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc00cea0 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc00a610 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad40cb1b0 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc00a9c0 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.482+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad415f000 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.483+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad404e840 con 0x7faad40ca4d0 2024-01-16T14:59:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.484+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faacc00ab20 con 0x7faad40ca4d0 2024-01-16T14:59:33.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.485+0000 7faacb400640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 0x7faaac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.485+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faacc089850 con 0x7faad40ca4d0 2024-01-16T14:59:33.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.485+0000 7faad3400640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 0x7faaac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.486+0000 7faad3400640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 0x7faaac06d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faac4033000 tx=0x7faac402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.489+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faacc054750 con 0x7faad40ca4d0 2024-01-16T14:59:33.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.685+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad4159a10 con 0x7faad40ca4d0 2024-01-16T14:59:33.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.688+0000 7faacb400640 1 -- 172.21.15.43:0/747491998 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7faacc057d70 con 0x7faad40ca4d0 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:33.670 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:33.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 msgr2=0x7faaac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 0x7faaac06d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faac4033000 tx=0x7faac402f000 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 msgr2=0x7faad40ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faacc001370 tx=0x7faacc00cc50 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 shutdown_connections 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faaac06b2c0 0x7faaac06d780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad406c090 0x7faad406c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faad40ca4d0 0x7faad40ca950 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 --2- 172.21.15.43:0/747491998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faab4003a30 0x7faad4157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 >> 172.21.15.43:0/747491998 conn(0x7faad40d0e60 msgr2=0x7faad40b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 shutdown_connections 2024-01-16T14:59:33.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.691+0000 7faad8a00640 1 -- 172.21.15.43:0/747491998 wait complete. 2024-01-16T14:59:33.694 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T14:59:33.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.943+0000 7f8c04a00640 1 Processor -- start 2024-01-16T14:59:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.943+0000 7f8c04a00640 1 -- start start 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.944+0000 7f8c04a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8c0006c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8c04a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0015e080 con 0x7f8c00150920 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8c04a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c000716f0 con 0x7f8c00150540 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8c04a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0006cac0 con 0x7f8c0006c130 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8bfde00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8c00150920 0x7f8c00071b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52894/0 (socket says 172.21.15.43:52894) 2024-01-16T14:59:33.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8bfde00640 1 -- 172.21.15.43:0/657103872 learned_addr learned my addr 172.21.15.43:0/657103872 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8bff200640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8c0006c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881195548 0 0) 0x7f8c0015e080 con 0x7f8c00150920 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.945+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be0003520 con 0x7f8c00150920 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645705716 0 0) 0x7f8be0003520 con 0x7f8c00150920 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 msgr2=0x7f8c0006c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8c0006c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 >> v1:172.21.15.69:6789/0 conn(0x7f8c00150540 legacy=0x7f8c000cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00070dc0 con 0x7f8c00150920 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8be8001690 con 0x7f8c00150920 2024-01-16T14:59:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 >> v1:172.21.15.43:6789/0 conn(0x7f8c00150920 legacy=0x7f8c00071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8be0003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8be00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be0004d00 0x7f8be0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0006cac0 con 0x7f8be0003fe0 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be000bbc0 con 0x7f8be0004d00 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be000bd80 con 0x7f8c0006c130 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.946+0000 7f8bfde00640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8be00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfe800640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8be0003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bff200640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be0004d00 0x7f8be0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfde00640 1 -- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 msgr2=0x7f8be0003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfde00640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0006c130 0x7f8be0003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfde00640 1 -- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be0004d00 msgr2=0x7f8be0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfde00640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be0004d00 0x7f8be0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.947+0000 7f8bfde00640 1 -- 172.21.15.43:0/657103872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be0003520 con 0x7f8be0003fe0 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.948+0000 7f8bfde00640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8be00047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8c000cec00 tx=0x7f8be00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e651b63c63e31fba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.948+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8005ad0 con 0x7f8be0003fe0 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.948+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be8005c30 con 0x7f8be0003fe0 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.948+0000 7f8bfd400640 1 -- 172.21.15.43:0/657103872 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8007070 con 0x7f8be0003fe0 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 -- 172.21.15.43:0/657103872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 msgr2=0x7f8be00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 --2- 172.21.15.43:0/657103872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8be00047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8c000cec00 tx=0x7f8be00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8bff200640 1 -- 172.21.15.43:0/657103872 reap_dead start 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 -- 172.21.15.43:0/657103872 shutdown_connections 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 -- 172.21.15.43:0/657103872 >> 172.21.15.43:0/657103872 conn(0x7f8c0006a860 msgr2=0x7f8c0006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 -- 172.21.15.43:0/657103872 shutdown_connections 2024-01-16T14:59:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.949+0000 7f8c04a00640 1 -- 172.21.15.43:0/657103872 wait complete. 2024-01-16T14:59:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.951+0000 7f8c04a00640 1 Processor -- start 2024-01-16T14:59:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.951+0000 7f8c04a00640 1 -- start start 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0013bd40 0x7f8c0013c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 0x7f8c000aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c001426a0 con 0x7f8be0003fe0 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c00143ca0 con 0x7f8c000afb10 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8c04a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000b04d0 con 0x7f8c0013bd40 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8bfe800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8bfe800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34056/0 (socket says 172.21.15.43:34056) 2024-01-16T14:59:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8bfe800640 1 -- 172.21.15.43:0/2496575490 learned_addr learned my addr 172.21.15.43:0/2496575490 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8bfde00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0013bd40 0x7f8c0013c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.952+0000 7f8bff200640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 0x7f8c000aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 -- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0013bd40 msgr2=0x7f8c0013c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0013bd40 0x7f8c0013c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 -- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 msgr2=0x7f8c000aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 0x7f8c000aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 -- 172.21.15.43:0/2496575490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c001f3920 con 0x7f8be0003fe0 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bff200640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 0x7f8c000aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bfe800640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000e4d0 tx=0x7f8bf000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf00053a0 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf00035f0 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0003980 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0015b1d0 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.953+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c00143eb0 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.955+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0004e840 con 0x7f8be0003fe0 2024-01-16T14:59:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.955+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8bf0003ae0 con 0x7f8be0003fe0 2024-01-16T14:59:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.956+0000 7f8bf6a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 0x7f8bd806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.956+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bf0089a40 con 0x7f8be0003fe0 2024-01-16T14:59:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.957+0000 7f8bfde00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 0x7f8bd806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.957+0000 7f8bfde00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 0x7f8bd806d7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8be8034000 tx=0x7f8be8037040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:33.961+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bf0054810 con 0x7f8be0003fe0 2024-01-16T14:59:34.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.157+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c00070720 con 0x7f8be0003fe0 2024-01-16T14:59:34.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.158+0000 7f8bf6a00640 1 -- 172.21.15.43:0/2496575490 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f8bf0057e30 con 0x7f8be0003fe0 2024-01-16T14:59:34.140 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:34.140 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 msgr2=0x7f8bd806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 0x7f8bd806d7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8be8034000 tx=0x7f8be8037040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 msgr2=0x7f8c0015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000e4d0 tx=0x7f8bf000e990 comp rx=0 tx=0).stop 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 shutdown_connections 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8bd806b2f0 0x7f8bd806d7b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c000afb10 0x7f8c000aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c0013bd40 0x7f8c0013c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 --2- 172.21.15.43:0/2496575490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8be0003fe0 0x7f8c0015ac90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 >> 172.21.15.43:0/2496575490 conn(0x7f8c0006a860 msgr2=0x7f8c000cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 shutdown_connections 2024-01-16T14:59:34.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.161+0000 7f8c04a00640 1 -- 172.21.15.43:0/2496575490 wait complete. 2024-01-16T14:59:34.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T14:59:34.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T14:59:34.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T14:59:34.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-16T14:59:34.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-16T14:59:34.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-16T14:59:34.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-16T14:59:34.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-16T14:59:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.493+0000 7f1013800640 1 Processor -- start 2024-01-16T14:59:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.493+0000 7f1013800640 1 -- start start 2024-01-16T14:59:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.495+0000 7f1013800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f100404ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.495+0000 7f1013800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1004038070 0x7f1004053560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.495+0000 7f1013800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10040061d0 con 0x7f10040384f0 2024-01-16T14:59:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.495+0000 7f1013800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100404cfa0 con 0x7f1004038070 2024-01-16T14:59:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.495+0000 7f1013800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1004056ea0 con 0x7f10040e77b0 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.496+0000 7f1010e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f100404ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.496+0000 7f1010e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f100404ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54874/0 (socket says 172.21.15.43:54874) 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.496+0000 7f1010e00640 1 -- 172.21.15.43:0/3599982384 learned_addr learned my addr 172.21.15.43:0/3599982384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.496+0000 7f100be00640 1 --2- 172.21.15.43:0/3599982384 >> v2:172.21.15.43:3300/0 conn(0x7f1004038070 0x7f1004053560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7f100be00640 1 -- 172.21.15.43:0/3599982384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 msgr2=0x7f100404ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7f100be00640 1 --2- 172.21.15.43:0/3599982384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f100404ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7f100be00640 1 -- 172.21.15.43:0/3599982384 >> v1:172.21.15.43:6789/0 conn(0x7f10040384f0 legacy=0x7f1004006f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7f100be00640 1 -- 172.21.15.43:0/3599982384 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004006420 con 0x7f1004038070 2024-01-16T14:59:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7eff76e00640 1 Processor -- start 2024-01-16T14:59:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.497+0000 7eff76e00640 1 -- start start 2024-01-16T14:59:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.499+0000 7f100be00640 1 --2- 172.21.15.43:0/3599982384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f1004053560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40097a0 tx=0x7f0ff402b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9046537e310ffbf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.499+0000 7eff76e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7eff680530c0 0x7eff68039b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7f100b400640 1 -- 172.21.15.43:0/3599982384 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4030710 con 0x7f1004038070 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7f100b400640 1 -- 172.21.15.43:0/3599982384 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff402e050 con 0x7f1004038070 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7f100b400640 1 -- 172.21.15.43:0/3599982384 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4030ad0 con 0x7f1004038070 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7eff76e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6803b300 con 0x7eff6804a2e0 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7eff76e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6803b5a0 con 0x7eff68036170 2024-01-16T14:59:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7eff76e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6803b880 con 0x7eff680530c0 2024-01-16T14:59:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.500+0000 7f8a4b000640 1 Processor -- start 2024-01-16T14:59:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f8a4b000640 1 -- start start 2024-01-16T14:59:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 -- 172.21.15.43:0/3599982384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 msgr2=0x7f1004053560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 --2- 172.21.15.43:0/3599982384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f1004053560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40097a0 tx=0x7f0ff402b730 comp rx=0 tx=0).stop 2024-01-16T14:59:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7eff74e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7eff680530c0 0x7eff68039b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7eff6f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7eff68036170 0x7eff680529b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:56536/0 (socket says 172.21.15.43:56536) 2024-01-16T14:59:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7eff6f400640 1 -- 172.21.15.43:0/3321638333 learned_addr learned my addr 172.21.15.43:0/3321638333 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 -- 172.21.15.43:0/3599982384 shutdown_connections 2024-01-16T14:59:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 --2- 172.21.15.43:0/3599982384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f1004053560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 --2- 172.21.15.43:0/3599982384 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f100404ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.501+0000 7f1013800640 1 -- 172.21.15.43:0/3599982384 >> 172.21.15.43:0/3599982384 conn(0x7f100403aa00 msgr2=0x7f100403ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7f1013800640 1 -- 172.21.15.43:0/3599982384 shutdown_connections 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7eff74e00640 1 -- 172.21.15.43:0/3321638333 >> v1:172.21.15.69:6789/0 conn(0x7eff68036170 legacy=0x7eff680529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7eff6ea00640 1 -- 172.21.15.43:0/3321638333 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006997199 0 0) 0x7eff6803b5a0 con 0x7eff68036170 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7eff74e00640 1 -- 172.21.15.43:0/3321638333 >> v1:172.21.15.43:6789/0 conn(0x7eff6804a2e0 legacy=0x7eff6803c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7eff74e00640 1 -- 172.21.15.43:0/3321638333 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6803bb60 con 0x7eff680530c0 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.502+0000 7f8a4b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 0x7f8a440ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f8a4b000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a44070ae0 0x7f8a440b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f8a4b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a440b3c20 con 0x7f8a4415a590 2024-01-16T14:59:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f8a4b000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a440b3f00 con 0x7f8a44070ae0 2024-01-16T14:59:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f8a4b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a44070650 con 0x7f8a440b1360 2024-01-16T14:59:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f7f6b000640 1 Processor -- start 2024-01-16T14:59:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.503+0000 7f1013800640 1 -- 172.21.15.43:0/3599982384 wait complete. 2024-01-16T14:59:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f7f6b000640 1 -- start start 2024-01-16T14:59:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7eff74e00640 1 --2- 172.21.15.43:0/3321638333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff680530c0 0x7eff68039b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff70063e60 tx=0x7eff700a5ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b330c1d1bf60d96 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f8a49c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 0x7f8a440ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f8a49c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 0x7f8a440ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33504/0 (socket says 172.21.15.43:33504) 2024-01-16T14:59:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f8a49c00640 1 -- 172.21.15.43:0/2118954129 learned_addr learned my addr 172.21.15.43:0/2118954129 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f8a49200640 1 --1- 172.21.15.43:0/2118954129 >> v1:172.21.15.69:6789/0 conn(0x7f8a4415a590 0x7f8a44156f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:56540/0 (socket says 172.21.15.43:56540) 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.504+0000 7f8a4a600640 1 --2- 172.21.15.43:0/2118954129 >> v2:172.21.15.43:3300/0 conn(0x7f8a44070ae0 0x7f8a440b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7f8a43e00640 1 -- 172.21.15.43:0/2118954129 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399570643 0 0) 0x7f8a440b3c20 con 0x7f8a4415a590 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7f8a4a600640 1 -- 172.21.15.43:0/2118954129 >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 msgr2=0x7f8a440ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7f8a4a600640 1 --2- 172.21.15.43:0/2118954129 >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 0x7f8a440ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7f8a4a600640 1 -- 172.21.15.43:0/2118954129 >> v1:172.21.15.69:6789/0 conn(0x7f8a4415a590 legacy=0x7f8a44156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7f8a4a600640 1 -- 172.21.15.43:0/2118954129 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a44070270 con 0x7f8a44070ae0 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7eff6ea00640 1 -- 172.21.15.43:0/3321638333 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff700a6d30 con 0x7eff680530c0 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7eff6ea00640 1 -- 172.21.15.43:0/3321638333 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff700a6e90 con 0x7eff680530c0 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.505+0000 7eff6ea00640 1 -- 172.21.15.43:0/3321638333 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff700a6d30 con 0x7eff680530c0 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f6b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 0x7f7f6415c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f6b000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f64071150 con 0x7f7f64071360 2024-01-16T14:59:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f6b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6415ba30 con 0x7f7f64157f20 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f6b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64070b90 con 0x7f7f640717c0 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f69c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7f64157f20 0x7f7f640700d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:56548/0 (socket says 172.21.15.43:56548) 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7eff76e00640 1 -- 172.21.15.43:0/3321638333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff680530c0 msgr2=0x7eff68039b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7eff76e00640 1 --2- 172.21.15.43:0/3321638333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff680530c0 0x7eff68039b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff70063e60 tx=0x7eff700a5ec0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7eff76e00640 1 -- 172.21.15.43:0/3321638333 shutdown_connections 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7eff76e00640 1 --2- 172.21.15.43:0/3321638333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff680530c0 0x7eff68039b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7eff76e00640 1 -- 172.21.15.43:0/3321638333 >> 172.21.15.43:0/3321638333 conn(0x7eff68053770 msgr2=0x7eff68053b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f8a4a600640 1 --2- 172.21.15.43:0/2118954129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a44070ae0 0x7f8a440b25b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a44068140 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998249f80b85b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f8a43e00640 1 -- 172.21.15.43:0/2118954129 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c031830 con 0x7f8a44070ae0 2024-01-16T14:59:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.506+0000 7f7f69c00640 1 -- 172.21.15.43:0/2214538591 learned_addr learned my addr 172.21.15.43:0/2214538591 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f8a43e00640 1 -- 172.21.15.43:0/2118954129 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c031990 con 0x7f8a44070ae0 2024-01-16T14:59:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f8a43e00640 1 -- 172.21.15.43:0/2118954129 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c031d20 con 0x7f8a44070ae0 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f7f6a600640 1 --2- 172.21.15.43:0/2214538591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 0x7f7f6415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f7f6a600640 1 -- 172.21.15.43:0/2214538591 >> v1:172.21.15.69:6789/0 conn(0x7f7f64157f20 legacy=0x7f7f640700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f7f6a600640 1 -- 172.21.15.43:0/2214538591 >> v1:172.21.15.43:6789/0 conn(0x7f7f64071360 legacy=0x7f7f6415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f7f6a600640 1 -- 172.21.15.43:0/2214538591 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64070e70 con 0x7f7f640717c0 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f49cfe00640 1 Processor -- start 2024-01-16T14:59:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f49cfe00640 1 -- start start 2024-01-16T14:59:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.508+0000 7eff76e00640 1 -- 172.21.15.43:0/3321638333 shutdown_connections 2024-01-16T14:59:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f1013800640 1 Processor -- start 2024-01-16T14:59:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.507+0000 7f1013800640 1 -- start start 2024-01-16T14:59:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.508+0000 7eff76e00640 1 -- 172.21.15.43:0/3321638333 wait complete. 2024-01-16T14:59:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f8a4b000640 1 -- 172.21.15.43:0/2118954129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a44070ae0 msgr2=0x7f8a440b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f100403c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10040384f0 0x7f100417a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f10041807a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1004050210 con 0x7f1004038070 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100403d610 con 0x7f10040384f0 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1013800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100403cd70 con 0x7f10040e77b0 2024-01-16T14:59:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f8a4b000640 1 --2- 172.21.15.43:0/2118954129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a44070ae0 0x7f8a440b25b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a44068140 tx=0x7f8a3c02b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f1011800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f10041807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f1011800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f10041807a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54890/0 (socket says 172.21.15.43:54890) 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f1011800640 1 -- 172.21.15.43:0/148345331 learned_addr learned my addr 172.21.15.43:0/148345331 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f49cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f49c800c070 0x7f49c8038050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f49cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49c8053c40 0x7f49c80def80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f49cfe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80541b0 con 0x7f49c80df4c0 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f49cfe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8005f50 con 0x7f49c800c070 2024-01-16T14:59:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f49cfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8006230 con 0x7f49c8053c40 2024-01-16T14:59:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.509+0000 7f8a4b000640 1 -- 172.21.15.43:0/2118954129 shutdown_connections 2024-01-16T14:59:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f8a4b000640 1 --2- 172.21.15.43:0/2118954129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a44070ae0 0x7f8a440b25b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f8a4b000640 1 --2- 172.21.15.43:0/2118954129 >> v2:172.21.15.69:3300/0 conn(0x7f8a440b1360 0x7f8a440ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f8a4b000640 1 -- 172.21.15.43:0/2118954129 >> 172.21.15.43:0/2118954129 conn(0x7f8a440b0840 msgr2=0x7f8a440b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f100be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10040384f0 0x7f100417a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f1010e00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f100403c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f49cea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f49c800c070 0x7f49c8038050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f49cea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f49c800c070 0x7f49c8038050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34094/0 (socket says 172.21.15.43:34094) 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.510+0000 7f49cea00640 1 -- 172.21.15.43:0/3261510498 learned_addr learned my addr 172.21.15.43:0/3261510498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f8a4b000640 1 -- 172.21.15.43:0/2118954129 shutdown_connections 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f8a4b000640 1 -- 172.21.15.43:0/2118954129 wait complete. 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 -- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10040384f0 msgr2=0x7f100417a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10040384f0 0x7f100417a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 -- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 msgr2=0x7f100403c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1004038070 0x7f100403c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4009400 con 0x7f10040e77b0 2024-01-16T14:59:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f100be00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10040384f0 0x7f100417a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f49ce000640 1 --2- 172.21.15.43:0/3261510498 >> v2:172.21.15.69:3300/0 conn(0x7f49c8053c40 0x7f49c80def80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7eff76e00640 1 Processor -- start 2024-01-16T14:59:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f7f6a600640 1 --2- 172.21.15.43:0/2214538591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 0x7f7f6415c900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00b620 tx=0x7f7f5c0302b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e348dd387c47971b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f49cea00640 1 -- 172.21.15.43:0/3261510498 >> v2:172.21.15.69:3300/0 conn(0x7f49c8053c40 msgr2=0x7f49c80def80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f49cea00640 1 --2- 172.21.15.43:0/3261510498 >> v2:172.21.15.69:3300/0 conn(0x7f49c8053c40 0x7f49c80def80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f49cea00640 1 -- 172.21.15.43:0/3261510498 >> v1:172.21.15.43:6789/0 conn(0x7f49c80df4c0 legacy=0x7f49c804fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f49cea00640 1 -- 172.21.15.43:0/3261510498 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8006510 con 0x7f49c800c070 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7f1011800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f10041807a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f100c02cee0 tx=0x7f100c067f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c06be10 con 0x7f10040e77b0 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c0721c0 con 0x7f10040e77b0 2024-01-16T14:59:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c06be10 con 0x7f10040e77b0 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.511+0000 7eff76e00640 1 -- start start 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff68036170 0x7eff6817ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff6804a2e0 0x7eff68008ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 0x7eff680090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6804eae0 con 0x7eff68036170 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6804de60 con 0x7eff6804a2e0 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7eff76e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6804e140 con 0x7eff680530c0 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7eff6fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff68036170 0x7eff6817ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7eff74e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 0x7eff680090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100403d050 con 0x7f10040e77b0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f7f63e00640 1 -- 172.21.15.43:0/2214538591 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00b2d0 con 0x7f7f640717c0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f7f63e00640 1 -- 172.21.15.43:0/2214538591 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c030e60 con 0x7f7f640717c0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f7f63e00640 1 -- 172.21.15.43:0/2214538591 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00b2d0 con 0x7f7f640717c0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7f49cea00640 1 --2- 172.21.15.43:0/3261510498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c800c070 0x7f49c8038050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f49c800df30 tx=0x7f49c002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf17e767b89085f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7eff74e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 0x7eff680090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54894/0 (socket says 172.21.15.43:54894) 2024-01-16T14:59:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7eff74e00640 1 -- 172.21.15.43:0/1095602194 learned_addr learned my addr 172.21.15.43:0/1095602194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7f49cd600640 1 -- 172.21.15.43:0/3261510498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0030870 con 0x7f49c800c070 2024-01-16T14:59:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7f9530600640 1 Processor -- start 2024-01-16T14:59:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7f49cd600640 1 -- 172.21.15.43:0/3261510498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00309d0 con 0x7f49c800c070 2024-01-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.512+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100403ca80 con 0x7f10040e77b0 2024-01-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.513+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10040556c0 con 0x7f10040e77b0 2024-01-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7f9530600640 1 -- start start 2024-01-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7fd25ca00640 1 Processor -- start 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 -- 172.21.15.43:0/1095602194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 msgr2=0x7eff680090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 0x7eff680090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 -- 172.21.15.43:0/1095602194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff6804a2e0 msgr2=0x7eff68008ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff6804a2e0 0x7eff68008ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70068590 con 0x7eff68036170 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff74e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff680530c0 0x7eff680090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.514+0000 7fd25ca00640 1 -- start start 2024-01-16T14:59:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7eff6fe00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff68036170 0x7eff6817ca50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c390 tx=0x7eff5800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7f8a4b000640 1 Processor -- start 2024-01-16T14:59:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7f7f6b000640 1 -- 172.21.15.43:0/2214538591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 msgr2=0x7f7f6415c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7f7f6b000640 1 --2- 172.21.15.43:0/2214538591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 0x7f7f6415c900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00b620 tx=0x7f7f5c0302b0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7f8a4b000640 1 -- start start 2024-01-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f7f6b000640 1 -- 172.21.15.43:0/2214538591 shutdown_connections 2024-01-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f7f6b000640 1 --2- 172.21.15.43:0/2214538591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f640717c0 0x7f7f6415c900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f7f6b000640 1 -- 172.21.15.43:0/2214538591 >> 172.21.15.43:0/2214538591 conn(0x7f7f640b0840 msgr2=0x7f7f640b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.515+0000 7f49cd600640 1 -- 172.21.15.43:0/3261510498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0030d60 con 0x7f49c800c070 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f49cfe00640 1 -- 172.21.15.43:0/3261510498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c800c070 msgr2=0x7f49c8038050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f49cfe00640 1 --2- 172.21.15.43:0/3261510498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c800c070 0x7f49c8038050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f49c800df30 tx=0x7f49c002b9b0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f100c0723b0 con 0x7f10040e77b0 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580052d0 con 0x7eff68036170 2024-01-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58005430 con 0x7eff68036170 2024-01-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7f9530600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9528071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7fd25ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2540acb50 0x7fd2540719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7fd25ca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd254070dc0 con 0x7fd25406d520 2024-01-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.516+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58003830 con 0x7eff68036170 2024-01-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f49cfe00640 1 -- 172.21.15.43:0/3261510498 shutdown_connections 2024-01-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f49cfe00640 1 --2- 172.21.15.43:0/3261510498 >> v2:172.21.15.69:3300/0 conn(0x7f49c8053c40 0x7f49c80def80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f49cfe00640 1 --2- 172.21.15.43:0/3261510498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c800c070 0x7f49c8038050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f49cfe00640 1 -- 172.21.15.43:0/3261510498 >> 172.21.15.43:0/3261510498 conn(0x7f49c8009f80 msgr2=0x7f49c800a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6804e420 con 0x7eff68036170 2024-01-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff680e89b0 con 0x7eff68036170 2024-01-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f952e600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f95280715d0 0x7f95280ad890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52940/0 (socket says 172.21.15.43:52940) 2024-01-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7f952e600640 1 -- 172.21.15.43:0/2588395445 learned_addr learned my addr 172.21.15.43:0/2588395445 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7fd25ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd254161d00 con 0x7fd2540b2080 2024-01-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7fd25ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2540cedb0 con 0x7fd2540acb50 2024-01-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f49cfe00640 1 -- 172.21.15.43:0/3261510498 shutdown_connections 2024-01-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.517+0000 7fd25a000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd25406d520 0x7fd25406d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52942/0 (socket says 172.21.15.43:52942) 2024-01-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd25a000640 1 -- 172.21.15.43:0/1431302585 learned_addr learned my addr 172.21.15.43:0/1431302585 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd259600640 1 --2- 172.21.15.43:0/1431302585 >> v2:172.21.15.69:3300/0 conn(0x7fd2540acb50 0x7fd2540719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f952d200640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9528071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f1008c00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fe406b320 0x7f0fe406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95280ad400 con 0x7f95280715d0 2024-01-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95280acb50 con 0x7f95280b3b60 2024-01-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7eff68054830 con 0x7eff68036170 2024-01-16T14:59:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f8a4b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 0x7f8a44145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd259600640 1 -- 172.21.15.43:0/1431302585 >> v1:172.21.15.69:6789/0 conn(0x7fd2540b2080 legacy=0x7fd2540b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd259600640 1 -- 172.21.15.43:0/1431302585 >> v1:172.21.15.43:6789/0 conn(0x7fd25406d520 legacy=0x7fd25406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95280acdf0 con 0x7f952806dc30 2024-01-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f8a4b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a440b1360 0x7f8a440adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd259600640 1 -- 172.21.15.43:0/1431302585 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2540ad030 con 0x7fd2540acb50 2024-01-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7fd258c00640 1 -- 172.21.15.43:0/1431302585 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99671072 0 0) 0x7fd254070dc0 con 0x7fd25406d520 2024-01-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f1010e00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fe406b320 0x7f0fe406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.518+0000 7f8a4b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a4415a590 0x7f8a44145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a4b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44144b80 con 0x7f8a4415a590 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a4b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441461f0 con 0x7f8a44070ae0 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a4b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441464d0 con 0x7f8a440b1360 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a49c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 0x7f8a44145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a49c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 0x7f8a44145c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33540/0 (socket says 172.21.15.43:33540) 2024-01-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a49c00640 1 -- 172.21.15.43:0/4106284098 learned_addr learned my addr 172.21.15.43:0/4106284098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f49cfe00640 1 -- 172.21.15.43:0/3261510498 wait complete. 2024-01-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f7f6b000640 1 -- 172.21.15.43:0/2214538591 shutdown_connections 2024-01-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f100c0d20a0 con 0x7f10040e77b0 2024-01-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618945840 0 0) 0x7f95280acb50 con 0x7f95280b3b60 2024-01-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f1010e00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fe406b320 0x7f0fe406d7e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009b30 tx=0x7f0ffc009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a49200640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a440b1360 0x7f8a440adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f8a4a600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a4415a590 0x7f8a44145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49c00640 1 -- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a440b1360 msgr2=0x7f8a440adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49c00640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a440b1360 0x7f8a440adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49c00640 1 -- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a4415a590 msgr2=0x7f8a44145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f1008c00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f0fe40741a0 0x7f0fe4076640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49c00640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a4415a590 0x7f8a44145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49c00640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c009400 con 0x7f8a44070ae0 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a49200640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a440b1360 0x7f8a440adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f8a4a600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a4415a590 0x7f8a44145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9508003520 con 0x7f95280b3b60 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334941316 0 0) 0x7f95280ad400 con 0x7f95280715d0 2024-01-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95280acb50 con 0x7f95280715d0 2024-01-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fe4076d10 con 0x7f0fe40741a0 2024-01-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f100c0679b0 con 0x7f10040e77b0 2024-01-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241685103 0 0) 0x7f95280acb50 con 0x7f95280715d0 2024-01-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 msgr2=0x7f9528071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9528071090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 >> v1:172.21.15.69:6789/0 conn(0x7f95280b3b60 legacy=0x7f952806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.520+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280720a0 con 0x7f95280715d0 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f95180015f0 con 0x7f95280715d0 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f8a49c00640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 0x7f8a44145c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a340070b0 tx=0x7f8a3400d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 >> v1:172.21.15.43:6789/0 conn(0x7f95280715d0 legacy=0x7f95280ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f100be00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f0fe40741a0 0x7f0fe4076640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.519+0000 7f7f6b000640 1 -- 172.21.15.43:0/2214538591 wait complete. 2024-01-16T14:59:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508004b90 0x7f9508004f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9508003760 0x7f9508003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9508004120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34009e40 con 0x7f8a44070ae0 2024-01-16T14:59:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3400f040 con 0x7f8a44070ae0 2024-01-16T14:59:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34015620 con 0x7f8a44070ae0 2024-01-16T14:59:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7fd259600640 1 --2- 172.21.15.43:0/1431302585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540acb50 0x7fd2540719d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0097a0 tx=0x7fd23c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f89894adf036245 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95280acdf0 con 0x7f9508003760 2024-01-16T14:59:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950800ba30 con 0x7f9508004b90 2024-01-16T14:59:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.521+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950800bbf0 con 0x7f952806dc30 2024-01-16T14:59:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f100be00640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f0fe40741a0 0x7f0fe4076640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f952d200640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9508003760 0x7f9508003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f7f6b000640 1 Processor -- start 2024-01-16T14:59:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f952e600640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9508004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a44144df0 con 0x7f8a44070ae0 2024-01-16T14:59:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff58003990 con 0x7eff68036170 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f952dc00640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508004b90 0x7f9508004f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f952d200640 1 -- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 msgr2=0x7f9508004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f952d200640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f952806dc30 0x7f9508004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f952d200640 1 -- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508004b90 msgr2=0x7f9508004f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f952d200640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508004b90 0x7f9508004f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f952d200640 1 -- 172.21.15.43:0/2588395445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280acb50 con 0x7f9508003760 2024-01-16T14:59:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f49cfe00640 1 Processor -- start 2024-01-16T14:59:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7eff67e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff4806b310 0x7eff4806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f25ba200640 1 Processor -- start 2024-01-16T14:59:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f25ba200640 1 -- start start 2024-01-16T14:59:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a441467b0 con 0x7f8a44070ae0 2024-01-16T14:59:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a4406cbb0 con 0x7f8a44070ae0 2024-01-16T14:59:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7f7f6b000640 1 -- start start 2024-01-16T14:59:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.523+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff580897f0 con 0x7eff68036170 2024-01-16T14:59:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.522+0000 7fd258c00640 1 -- 172.21.15.43:0/1431302585 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c030610 con 0x7fd2540acb50 2024-01-16T14:59:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7fd258c00640 1 -- 172.21.15.43:0/1431302585 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c030770 con 0x7fd2540acb50 2024-01-16T14:59:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7fd258c00640 1 -- 172.21.15.43:0/1431302585 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c030b00 con 0x7fd2540acb50 2024-01-16T14:59:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7eff6f400640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff4806b310 0x7eff4806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7eff6f400640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff4806b310 0x7eff4806d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff60007fd0 tx=0x7eff60007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7eff67e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7eff48073530 0x7eff48075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff48076000 con 0x7eff48073530 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff58089b60 con 0x7eff68036170 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7eff74e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7eff48073530 0x7eff48075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== osd.1 v2:172.21.15.43:6823/2025179551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0fe4076d10 con 0x7f0fe40741a0 2024-01-16T14:59:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f49cfe00640 1 -- start start 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c800c070 0x7f49c804dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c8053c40 0x7f49c8180df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 0x7f49c8181330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80d1250 con 0x7f49c8053c40 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8036100 con 0x7f49c800c070 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f49cfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8035b40 con 0x7f49c80df4c0 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f49cf400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 0x7f49c8181330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f49cf400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 0x7f49c8181330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54944/0 (socket says 172.21.15.43:54944) 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f49cf400640 1 -- 172.21.15.43:0/895409009 learned_addr learned my addr 172.21.15.43:0/895409009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f49cea00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c800c070 0x7f49c804dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f49ce000640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c8053c40 0x7f49c8180df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7fd25ca00640 1 -- 172.21.15.43:0/1431302585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540acb50 msgr2=0x7fd2540719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7fd25ca00640 1 --2- 172.21.15.43:0/1431302585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540acb50 0x7fd2540719d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0097a0 tx=0x7fd23c02b960 comp rx=0 tx=0).stop 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7fd25ca00640 1 -- 172.21.15.43:0/1431302585 shutdown_connections 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7fd25ca00640 1 --2- 172.21.15.43:0/1431302585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540acb50 0x7fd2540719d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7fd25ca00640 1 -- 172.21.15.43:0/1431302585 >> 172.21.15.43:0/1431302585 conn(0x7fd2540b0d80 msgr2=0x7fd2540b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7fd25ca00640 1 -- 172.21.15.43:0/1431302585 shutdown_connections 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f952d200640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9508003760 0x7f9508003be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f95080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa4f5b85d704411 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951002c6b0 con 0x7f9508003760 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951002c810 con 0x7f9508003760 2024-01-16T14:59:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f9527e00640 1 -- 172.21.15.43:0/2588395445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951002cba0 con 0x7f9508003760 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f25ba200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25b4140910 0x7f25b4140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f25ba200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25b4141520 0x7f25b406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25ba200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b413ec90 con 0x7f25b415ed20 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25ba200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b4141230 con 0x7f25b4141520 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25ba200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b4070890 con 0x7f25b4140910 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25b2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25b4141520 0x7f25b406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25b2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25b4141520 0x7f25b406ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34136/0 (socket says 172.21.15.43:34136) 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f25b2a00640 1 -- 172.21.15.43:0/49466148 learned_addr learned my addr 172.21.15.43:0/49466148 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f25b3400640 1 --2- 172.21.15.43:0/49466148 >> v2:172.21.15.69:3300/0 conn(0x7f25b4140910 0x7f25b4140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7fd25ca00640 1 -- 172.21.15.43:0/1431302585 wait complete. 2024-01-16T14:59:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f64071360 0x7f7f6414aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 0x7f7f641f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64157f20 0x7f7f640b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6414af70 con 0x7f7f640717c0 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640b2670 con 0x7f7f64157f20 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.524+0000 7f7f6b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640b1a60 con 0x7f7f64071360 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f7f69200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 0x7f7f641f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f7f69200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 0x7f7f641f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34120/0 (socket says 172.21.15.43:34120) 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f7f69200640 1 -- 172.21.15.43:0/2741590706 learned_addr learned my addr 172.21.15.43:0/2741590706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.525+0000 7f7f6a600640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64157f20 0x7f7f640b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69c00640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f64071360 0x7f7f6414aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69200640 1 -- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f64071360 msgr2=0x7f7f6414aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69200640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f64071360 0x7f7f6414aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69200640 1 -- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64157f20 msgr2=0x7f7f640b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69200640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64157f20 0x7f7f640b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.526+0000 7f7f69200640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c00bc80 con 0x7f7f640717c0 2024-01-16T14:59:34.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f7f69200640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 0x7f7f641f83e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00c430 tx=0x7f7f4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7eff74e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7eff48073530 0x7eff48075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9508003760 msgr2=0x7f9508003be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f9530600640 1 --2- 172.21.15.43:0/2588395445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9508003760 0x7f9508003be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f95080057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f952e600640 1 -- 172.21.15.43:0/2588395445 reap_dead start 2024-01-16T14:59:34.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a34015780 con 0x7f8a44070ae0 2024-01-16T14:59:34.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c011790 con 0x7f7f640717c0 2024-01-16T14:59:34.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 shutdown_connections 2024-01-16T14:59:34.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 >> 172.21.15.43:0/2588395445 conn(0x7f95280b0d80 msgr2=0x7f95280b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f640b1d40 con 0x7f7f640717c0 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 shutdown_connections 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f25b2a00640 1 -- 172.21.15.43:0/49466148 >> v2:172.21.15.69:3300/0 conn(0x7f25b4140910 msgr2=0x7f25b4140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f25b2a00640 1 --2- 172.21.15.43:0/49466148 >> v2:172.21.15.69:3300/0 conn(0x7f25b4140910 0x7f25b4140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f25b2a00640 1 -- 172.21.15.43:0/49466148 >> v1:172.21.15.69:6789/0 conn(0x7f25b415ed20 legacy=0x7f25b415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f25b2a00640 1 -- 172.21.15.43:0/49466148 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4070b70 con 0x7f25b4141520 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f640b2300 con 0x7f7f640717c0 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c0118f0 con 0x7f7f640717c0 2024-01-16T14:59:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c011790 con 0x7f7f640717c0 2024-01-16T14:59:34.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f9530600640 1 -- 172.21.15.43:0/2588395445 wait complete. 2024-01-16T14:59:34.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f25b2a00640 1 --2- 172.21.15.43:0/49466148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4141520 0x7f25b406ff40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25b4072370 tx=0x7f25ac02bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded262ce1165ff2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f49cf400640 1 -- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c800c070 msgr2=0x7f49c804dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f49cf400640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c800c070 0x7f49c804dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f49cf400640 1 -- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c8053c40 msgr2=0x7f49c8180df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f49cf400640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c8053c40 0x7f49c8180df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.527+0000 7f49cf400640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00094d0 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.528+0000 7f49cf400640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 0x7f49c8181330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000b60 tx=0x7f49d00689e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d006bc00 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d006bd80 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0072400 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c8035e20 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.529+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c804e4e0 con 0x7f49c80df4c0 2024-01-16T14:59:34.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f49ce000640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49c8053c40 0x7f49c8180df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7fd25ca00640 1 Processor -- start 2024-01-16T14:59:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7fd25ca00640 1 -- start start 2024-01-16T14:59:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f34000fc0 con 0x7f7f640717c0 2024-01-16T14:59:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== osd.0 v2:172.21.15.43:6800/1320786927 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7eff48076000 con 0x7eff48073530 2024-01-16T14:59:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f8a41600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1c06b390 0x7f8a1c06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a3408a5e0 con 0x7f8a44070ae0 2024-01-16T14:59:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f8a41600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f8a1c0735b0 0x7f8a1c0759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49a0000fc0 con 0x7f49c80df4c0 2024-01-16T14:59:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a1c076080 con 0x7f8a1c0735b0 2024-01-16T14:59:34.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.530+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a3408a990 con 0x7f8a44070ae0 2024-01-16T14:59:34.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.531+0000 7f25b2000640 1 -- 172.21.15.43:0/49466148 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac02caa0 con 0x7f25b4141520 2024-01-16T14:59:34.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.531+0000 7f9530600640 1 Processor -- start 2024-01-16T14:59:34.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.531+0000 7f8a49200640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1c06b390 0x7f8a1c06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7f25b2000640 1 -- 172.21.15.43:0/49466148 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac02cc00 con 0x7f25b4141520 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd25406d520 0x7fd25409a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 0x7fd2540c5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540b2080 0x7fd2542038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25409a260 con 0x7fd2540acb50 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25409abc0 con 0x7fd2540b2080 2024-01-16T14:59:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7fd25ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2540995e0 con 0x7fd25406d520 2024-01-16T14:59:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7f25b2000640 1 -- 172.21.15.43:0/49466148 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac0343f0 con 0x7f25b4141520 2024-01-16T14:59:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7f8a49200640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1c06b390 0x7f8a1c06d850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8a44147030 tx=0x7f8a2c007c50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.531+0000 7f8a4a600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f8a1c0735b0 0x7f8a1c0759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.532+0000 7f9530600640 1 -- start start 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7f8a4a600640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f8a1c0735b0 0x7f8a1c0759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7f25ba200640 1 -- 172.21.15.43:0/49466148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4141520 msgr2=0x7f25b406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7f25ba200640 1 --2- 172.21.15.43:0/49466148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4141520 0x7f25b406ff40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25b4072370 tx=0x7f25ac02bf00 comp rx=0 tx=0).stop 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7fd259600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 0x7fd2540c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7fd259600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 0x7fd2540c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34144/0 (socket says 172.21.15.43:34144) 2024-01-16T14:59:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.533+0000 7fd259600640 1 -- 172.21.15.43:0/3249238988 learned_addr learned my addr 172.21.15.43:0/3249238988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508003760 0x7f952815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f25ba200640 1 -- 172.21.15.43:0/49466148 shutdown_connections 2024-01-16T14:59:34.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9528154620 0x7f9528152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f25ba200640 1 --2- 172.21.15.43:0/49466148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4141520 0x7f25b406ff40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f25ba200640 1 --2- 172.21.15.43:0/49466148 >> v2:172.21.15.69:3300/0 conn(0x7f25b4140910 0x7f25b4140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f25ba200640 1 -- 172.21.15.43:0/49466148 >> 172.21.15.43:0/49466148 conn(0x7f25b40b2e00 msgr2=0x7f25b40b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd25aa00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540b2080 0x7fd2542038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f25ba200640 1 -- 172.21.15.43:0/49466148 shutdown_connections 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd259600640 1 -- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd25406d520 msgr2=0x7fd25409a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd259600640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd25406d520 0x7fd25409a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd259600640 1 -- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540b2080 msgr2=0x7fd2542038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd259600640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540b2080 0x7fd2542038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd259600640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c009400 con 0x7fd2540acb50 2024-01-16T14:59:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7fd25aa00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2540b2080 0x7fd2542038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== osd.2 v2:172.21.15.43:6808/2671083580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8a1c076080 con 0x7f8a1c0735b0 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 0x7f9528156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9528156eb0 con 0x7f9528154620 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9528154190 con 0x7f9508003760 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.534+0000 7f9530600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9528158890 con 0x7f95281564f0 2024-01-16T14:59:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7fd259600640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 0x7fd2540c5260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd23c02bf70 tx=0x7fd23c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f952e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 0x7f9528156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f952e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 0x7f9528156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54962/0 (socket says 172.21.15.43:54962) 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f952e600640 1 -- 172.21.15.43:0/3215551664 learned_addr learned my addr 172.21.15.43:0/3215551664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f952d200640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9528154620 0x7f9528152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952e600640 1 -- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508003760 msgr2=0x7f952815a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952e600640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9508003760 0x7f952815a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952e600640 1 -- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9528154620 msgr2=0x7f9528152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c03e040 con 0x7fd2540acb50 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c039cf0 con 0x7fd2540acb50 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c044630 con 0x7fd2540acb50 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952e600640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9528154620 0x7f9528152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952e600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528153580 con 0x7f95281564f0 2024-01-16T14:59:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2540998c0 con 0x7fd2540acb50 2024-01-16T14:59:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f4c011a60 con 0x7f7f640717c0 2024-01-16T14:59:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd254099e80 con 0x7fd2540acb50 2024-01-16T14:59:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f952d200640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9528154620 0x7f9528152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.535+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f49d0072560 con 0x7f49c80df4c0 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f49c6a00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a806b310 0x7f49a806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.536+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49d0103c20 con 0x7f49c80df4c0 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.537+0000 7f49c6a00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f49a8073530 0x7f49a8075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.537+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49a8076000 con 0x7f49a8073530 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.537+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49d01040c0 con 0x7f49c80df4c0 2024-01-16T14:59:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f49cea00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a806b310 0x7f49a806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2540b1f10 con 0x7fd2540acb50 2024-01-16T14:59:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f952e600640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 0x7f9528156970 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95280cba10 tx=0x7f951800eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f49ce000640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f49a8073530 0x7f49a8075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f7f61600640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f3c06b320 0x7f7f3c06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f4c089d80 con 0x7f7f640717c0 2024-01-16T14:59:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951800c610 con 0x7f95281564f0 2024-01-16T14:59:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951800c770 con 0x7f95281564f0 2024-01-16T14:59:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f49cea00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a806b310 0x7f49a806d7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49c0036000 tx=0x7f49c0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951800cb00 con 0x7f95281564f0 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f9530600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9528153860 con 0x7f95281564f0 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f9530600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9528153e20 con 0x7f95281564f0 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f7f61600640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f7f3c073540 0x7f7f3c075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f3c076010 con 0x7f7f3c073540 2024-01-16T14:59:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f4c08a130 con 0x7f7f640717c0 2024-01-16T14:59:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f7f69c00640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f3c06b320 0x7f7f3c06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.539+0000 7f49ce000640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f49a8073530 0x7f49a8075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.540+0000 7f7f6a600640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f7f3c073540 0x7f7f3c075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.540+0000 7f7f69c00640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f3c06b320 0x7f7f3c06d7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005880 tx=0x7f7f54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.538+0000 7f25ba200640 1 -- 172.21.15.43:0/49466148 wait complete. 2024-01-16T14:59:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.541+0000 7f25ba200640 1 Processor -- start 2024-01-16T14:59:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.541+0000 7f7f6a600640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f7f3c073540 0x7f7f3c075960 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.541+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== osd.4 v2:172.21.15.69:6816/2273091943 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f49a8076000 con 0x7f49a8073530 2024-01-16T14:59:34.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.542+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== osd.3 v2:172.21.15.43:6813/889849964 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7f3c076010 con 0x7f7f3c073540 2024-01-16T14:59:34.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.542+0000 7f25ba200640 1 -- start start 2024-01-16T14:59:34.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.541+0000 7f9530600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9528071a60 con 0x7f95281564f0 2024-01-16T14:59:34.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.543+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd23c043390 con 0x7fd2540acb50 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4140910 0x7f25b4144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4141520 0x7f25b41695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 0x7f25b416a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4142230 con 0x7f25b4140910 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4141920 con 0x7f25b4141520 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25ba200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b416ad80 con 0x7f25b415ed20 2024-01-16T14:59:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25b3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4140910 0x7f25b4144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7fd252000640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd22c06b2c0 0x7fd22c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd23c0ac630 con 0x7fd2540acb50 2024-01-16T14:59:34.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 0x7f25b416a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 0x7f25b416a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:54976/0 (socket says 172.21.15.43:54976) 2024-01-16T14:59:34.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.545+0000 7f25b3e00640 1 -- 172.21.15.43:0/4073431391 learned_addr learned my addr 172.21.15.43:0/4073431391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b2a00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4141520 0x7f25b41695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7fd25a000640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd22c06b2c0 0x7fd22c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3e00640 1 -- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4141520 msgr2=0x7f25b41695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7fd252000640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7fd22c073460 0x7fd22c075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.547+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c075f30 con 0x7fd22c073460 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3e00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4141520 0x7f25b41695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3e00640 1 -- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4140910 msgr2=0x7f25b4144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3e00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4140910 0x7f25b4144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3e00640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac009b30 con 0x7f25b415ed20 2024-01-16T14:59:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.546+0000 7f25b3400640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b4140910 0x7f25b4144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.547+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd23c0ac9a0 con 0x7fd2540acb50 2024-01-16T14:59:34.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.547+0000 7f25b3e00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 0x7f25b416a550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25a400b410 tx=0x7f25a400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.548+0000 7fd25aa00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7fd22c073460 0x7fd22c075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.548+0000 7fd25a000640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd22c06b2c0 0x7fd22c06d780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd244007fd0 tx=0x7fd244007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.548+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4004020 con 0x7f25b415ed20 2024-01-16T14:59:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.549+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4004180 con 0x7f25b415ed20 2024-01-16T14:59:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.549+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4011630 con 0x7f25b415ed20 2024-01-16T14:59:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.549+0000 7fd25aa00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7fd22c073460 0x7fd22c075880 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.549+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f951800cc60 con 0x7f95281564f0 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.550+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b4141be0 con 0x7f25b415ed20 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.550+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b416aa90 con 0x7f25b415ed20 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.550+0000 7f9525600640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f950006b390 0x7f950006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.550+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f951808a260 con 0x7f95281564f0 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.550+0000 7f9525600640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f95000735b0 0x7f95000759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.551+0000 7f952dc00640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f950006b390 0x7f950006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.552+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25b40b0780 con 0x7f25b415ed20 2024-01-16T14:59:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.551+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9500076080 con 0x7f95000735b0 2024-01-16T14:59:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.551+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f951808a610 con 0x7f95281564f0 2024-01-16T14:59:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.552+0000 7f952dc00640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f950006b390 0x7f950006d850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9520001da0 tx=0x7f9520009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.552+0000 7f952d200640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f95000735b0 0x7f95000759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.553+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== osd.5 v2:172.21.15.69:6808/1266529003 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd22c075f30 con 0x7fd22c073460 2024-01-16T14:59:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.553+0000 7f952d200640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f95000735b0 0x7f95000759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.554+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25a4011790 con 0x7f25b415ed20 2024-01-16T14:59:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== osd.6 v2:172.21.15.69:6800/3561904235 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9500076080 con 0x7f95000735b0 2024-01-16T14:59:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.555+0000 7f25ab400640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f258c06b390 0x7f258c06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25a408a0b0 con 0x7f25b415ed20 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f25ab400640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f258c0735b0 0x7f258c0759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f258c076080 con 0x7f258c0735b0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f25b3400640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f258c06b390 0x7f258c06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.556+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25a408a460 con 0x7f25b415ed20 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.557+0000 7f25b3400640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f258c06b390 0x7f258c06d850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f259c006fd0 tx=0x7f259c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.557+0000 7f25b2a00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f258c0735b0 0x7f258c0759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.559+0000 7f25b2a00640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f258c0735b0 0x7f258c0759d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.561+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== osd.7 v2:172.21.15.69:6824/1736397519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f258c076080 con 0x7f258c0735b0 2024-01-16T14:59:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.562+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 --> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f34002cb0 con 0x7f7f3c073540 2024-01-16T14:59:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.562+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 --> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10040069e0 con 0x7f0fe40741a0 2024-01-16T14:59:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.563+0000 7f7f61600640 1 -- 172.21.15.43:0/2741590706 <== osd.3 v2:172.21.15.43:6813/889849964 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7f34002cb0 con 0x7f7f3c073540 2024-01-16T14:59:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.564+0000 7f1008c00640 1 -- 172.21.15.43:0/148345331 <== osd.1 v2:172.21.15.43:6823/2025179551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0ff4030cc0 con 0x7f0fe40741a0 2024-01-16T14:59:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.564+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f7f3c073540 msgr2=0x7f7f3c075960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.564+0000 7f7f6b000640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f7f3c073540 0x7f7f3c075960 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.565+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f3c06b320 msgr2=0x7f7f3c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.565+0000 7f7f6b000640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f3c06b320 0x7f7f3c06d7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005880 tx=0x7f7f54009040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.565+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 msgr2=0x7f7f641f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.565+0000 7f7f6b000640 1 --2- 172.21.15.43:0/2741590706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f640717c0 0x7f7f641f83e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00c430 tx=0x7f7f4c00c8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.565+0000 7f7f6a600640 1 -- 172.21.15.43:0/2741590706 reap_dead start 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 shutdown_connections 2024-01-16T14:59:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 >> 172.21.15.43:0/2741590706 conn(0x7f7f640b0840 msgr2=0x7f7f64158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f0fe40741a0 msgr2=0x7f0fe4076640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f0fe40741a0 0x7f0fe4076640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fe406b320 msgr2=0x7f0fe406d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fe406b320 0x7f0fe406d7e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009b30 tx=0x7f0ffc009240 comp rx=0 tx=0).stop 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 msgr2=0x7f10041807a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1013800640 1 --2- 172.21.15.43:0/148345331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10040e77b0 0x7f10041807a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f100c02cee0 tx=0x7f100c067f20 comp rx=0 tx=0).stop 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f1011800640 1 -- 172.21.15.43:0/148345331 reap_dead start 2024-01-16T14:59:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.566+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 shutdown_connections 2024-01-16T14:59:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.567+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 shutdown_connections 2024-01-16T14:59:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.567+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 >> 172.21.15.43:0/148345331 conn(0x7f100403aa00 msgr2=0x7f100403a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.567+0000 7f7f6b000640 1 -- 172.21.15.43:0/2741590706 wait complete. 2024-01-16T14:59:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.567+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 shutdown_connections 2024-01-16T14:59:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.567+0000 7f1013800640 1 -- 172.21.15.43:0/148345331 wait complete. 2024-01-16T14:59:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.569+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 --> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a440cb100 con 0x7f8a1c0735b0 2024-01-16T14:59:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.570+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 --> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7eff6804ed50 con 0x7eff48073530 2024-01-16T14:59:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.570+0000 7f8a41600640 1 -- 172.21.15.43:0/4106284098 <== osd.2 v2:172.21.15.43:6808/2671083580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a440cb100 con 0x7f8a1c0735b0 2024-01-16T14:59:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.571+0000 7eff67e00640 1 -- 172.21.15.43:0/1095602194 <== osd.0 v2:172.21.15.43:6800/1320786927 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7eff6804ed50 con 0x7eff48073530 2024-01-16T14:59:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.571+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f8a1c0735b0 msgr2=0x7f8a1c0759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.571+0000 7f8a4b000640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f8a1c0735b0 0x7f8a1c0759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7eff48073530 msgr2=0x7eff48075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7eff48073530 0x7eff48075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff4806b310 msgr2=0x7eff4806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff4806b310 0x7eff4806d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff60007fd0 tx=0x7eff60007900 comp rx=0 tx=0).stop 2024-01-16T14:59:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff68036170 msgr2=0x7eff6817ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 --2- 172.21.15.43:0/1095602194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7eff68036170 0x7eff6817ca50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c390 tx=0x7eff5800c850 comp rx=0 tx=0).stop 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff74e00640 1 -- 172.21.15.43:0/1095602194 reap_dead start 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 shutdown_connections 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.572+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 >> 172.21.15.43:0/1095602194 conn(0x7eff68053770 msgr2=0x7eff68051c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.574+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 shutdown_connections 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 --> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f49a0002cb0 con 0x7f49a8073530 2024-01-16T14:59:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7eff76e00640 1 -- 172.21.15.43:0/1095602194 wait complete. 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1c06b390 msgr2=0x7f8a1c06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f8a4b000640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1c06b390 0x7f8a1c06d850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8a44147030 tx=0x7f8a2c007c50 comp rx=0 tx=0).stop 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 msgr2=0x7f8a44145c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f8a4b000640 1 --2- 172.21.15.43:0/4106284098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a44070ae0 0x7f8a44145c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a340070b0 tx=0x7f8a3400d4a0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.575+0000 7f8a4a600640 1 -- 172.21.15.43:0/4106284098 reap_dead start 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.576+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 shutdown_connections 2024-01-16T14:59:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.576+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 >> 172.21.15.43:0/4106284098 conn(0x7f8a440b0840 msgr2=0x7f8a4406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.577+0000 7f49c6a00640 1 -- 172.21.15.43:0/895409009 <== osd.4 v2:172.21.15.69:6816/2273091943 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f49a0002cb0 con 0x7f49a8073530 2024-01-16T14:59:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.577+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 shutdown_connections 2024-01-16T14:59:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.577+0000 7f8a4b000640 1 -- 172.21.15.43:0/4106284098 wait complete. 2024-01-16T14:59:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.577+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f49a8073530 msgr2=0x7f49a8075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.577+0000 7f49cfe00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f49a8073530 0x7f49a8075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a806b310 msgr2=0x7f49a806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a806b310 0x7f49a806d7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49c0036000 tx=0x7f49c0008030 comp rx=0 tx=0).stop 2024-01-16T14:59:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 msgr2=0x7f49c8181330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 --2- 172.21.15.43:0/895409009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c80df4c0 0x7f49c8181330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000b60 tx=0x7f49d00689e0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cf400640 1 -- 172.21.15.43:0/895409009 reap_dead start 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 shutdown_connections 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.578+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 >> 172.21.15.43:0/895409009 conn(0x7f49c8009f80 msgr2=0x7f49c8036560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.579+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 --> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd25406c000 con 0x7fd22c073460 2024-01-16T14:59:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.581+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 shutdown_connections 2024-01-16T14:59:34.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.581+0000 7f49cfe00640 1 -- 172.21.15.43:0/895409009 wait complete. 2024-01-16T14:59:34.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.581+0000 7fd252000640 1 -- 172.21.15.43:0/3249238988 <== osd.5 v2:172.21.15.69:6808/1266529003 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd25406c000 con 0x7fd22c073460 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.581+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7fd22c073460 msgr2=0x7fd22c075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.581+0000 7fd25ca00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7fd22c073460 0x7fd22c075880 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.582+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd22c06b2c0 msgr2=0x7fd22c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.582+0000 7fd25ca00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd22c06b2c0 0x7fd22c06d780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd244007fd0 tx=0x7fd244007900 comp rx=0 tx=0).stop 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.582+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 msgr2=0x7fd2540c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.582+0000 7fd25ca00640 1 --2- 172.21.15.43:0/3249238988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2540acb50 0x7fd2540c5260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd23c02bf70 tx=0x7fd23c004730 comp rx=0 tx=0).stop 2024-01-16T14:59:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.583+0000 7fd25aa00640 1 -- 172.21.15.43:0/3249238988 reap_dead start 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.583+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 shutdown_connections 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.583+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 >> 172.21.15.43:0/3249238988 conn(0x7fd2540b0d80 msgr2=0x7fd2540aed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.585+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 shutdown_connections 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.585+0000 7fd25ca00640 1 -- 172.21.15.43:0/3249238988 wait complete. 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.587+0000 7f9530600640 1 -- 172.21.15.43:0/3215551664 --> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95280ad090 con 0x7f95000735b0 2024-01-16T14:59:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.589+0000 7f9525600640 1 -- 172.21.15.43:0/3215551664 <== osd.6 v2:172.21.15.69:6800/3561904235 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9510004470 con 0x7f95000735b0 2024-01-16T14:59:34.591 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:34.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:59:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.591+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f95000735b0 msgr2=0x7f95000759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.591+0000 7f951ea00640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f95000735b0 0x7f95000759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f950006b390 msgr2=0x7f950006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f950006b390 0x7f950006d850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9520001da0 tx=0x7f9520009040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 msgr2=0x7f9528156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 --2- 172.21.15.43:0/3215551664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95281564f0 0x7f9528156970 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95280cba10 tx=0x7f951800eb90 comp rx=0 tx=0).stop 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f952e600640 1 -- 172.21.15.43:0/3215551664 reap_dead start 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 shutdown_connections 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.592+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 >> 172.21.15.43:0/3215551664 conn(0x7f95280b0d80 msgr2=0x7f952806ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.593+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 shutdown_connections 2024-01-16T14:59:34.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.593+0000 7f951ea00640 1 -- 172.21.15.43:0/3215551664 wait complete. 2024-01-16T14:59:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.594+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 --> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25b40702e0 con 0x7f258c0735b0 2024-01-16T14:59:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.595+0000 7f25ab400640 1 -- 172.21.15.43:0/4073431391 <== osd.7 v2:172.21.15.69:6824/1736397519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25b40702e0 con 0x7f258c0735b0 2024-01-16T14:59:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f258c0735b0 msgr2=0x7f258c0759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.595 INFO:teuthology.orchestra.run.smithi043.stdout:60129542150 2024-01-16T14:59:34.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f258c0735b0 0x7f258c0759d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f258c06b390 msgr2=0x7f258c06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f258c06b390 0x7f258c06d850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f259c006fd0 tx=0x7f259c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 msgr2=0x7f25b416a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25ba200640 1 --2- 172.21.15.43:0/4073431391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b415ed20 0x7f25b416a550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25a400b410 tx=0x7f25a400b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.596+0000 7f25b3e00640 1 -- 172.21.15.43:0/4073431391 reap_dead start 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.597+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 shutdown_connections 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.597+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 >> 172.21.15.43:0/4073431391 conn(0x7f25b40b2e00 msgr2=0x7f25b413f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.597+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 shutdown_connections 2024-01-16T14:59:34.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.597+0000 7f25ba200640 1 -- 172.21.15.43:0/4073431391 wait complete. 2024-01-16T14:59:34.597 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:34.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:59:34.598 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:34.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:59:34.599 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:34.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:59:34.600 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:34.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:59:34.601 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:34.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:59:34.603 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:34.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:59:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.946+0000 7f701b000640 1 Processor -- start 2024-01-16T14:59:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.946+0000 7f701b000640 1 -- start start 2024-01-16T14:59:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.948+0000 7f701b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70140cedb0 0x7f70141681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.949+0000 7f701b000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140d0be0 con 0x7f701415d420 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.949+0000 7f701b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140d1a40 con 0x7f70140d00f0 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.949+0000 7f701b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701415d800 con 0x7f70140cedb0 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.949+0000 7f7013e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f701415d420 0x7f70140cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52958/0 (socket says 172.21.15.43:52958) 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.949+0000 7f7013e00640 1 -- 172.21.15.43:0/3915284044 learned_addr learned my addr 172.21.15.43:0/3915284044 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.950+0000 7f7019000640 1 --2- 172.21.15.43:0/3915284044 >> v2:172.21.15.69:3300/0 conn(0x7f70140cedb0 0x7f70141681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.951+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638350391 0 0) 0x7f70140d0be0 con 0x7f701415d420 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.951+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4003520 con 0x7f701415d420 2024-01-16T14:59:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.951+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920954253 0 0) 0x7f70140d1a40 con 0x7f70140d00f0 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.951+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70140d0be0 con 0x7f70140d00f0 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943190762 0 0) 0x7f6ff4003520 con 0x7f701415d420 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 >> v2:172.21.15.69:3300/0 conn(0x7f70140cedb0 msgr2=0x7f70141681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 --2- 172.21.15.43:0/3915284044 >> v2:172.21.15.69:3300/0 conn(0x7f70140cedb0 0x7f70141681b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 >> v1:172.21.15.69:6789/0 conn(0x7f70140d00f0 legacy=0x7f70140ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140b3910 con 0x7f701415d420 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6ffc001670 con 0x7f701415d420 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 >> v1:172.21.15.43:6789/0 conn(0x7f701415d420 legacy=0x7f70140cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ff4004bd0 0x7f6ff4004fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 0x7f6ff4003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4004120 0x7f6ff400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701415d800 con 0x7f6ff4003760 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400c080 con 0x7f6ff4004bd0 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.952+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400c240 con 0x7f6ff4004120 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.953+0000 7f7013e00640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ff4004bd0 0x7f6ff4004fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.953+0000 7f7013400640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 0x7f6ff4003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.954+0000 7f7013400640 1 -- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4004120 msgr2=0x7f6ff400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.954+0000 7f7013400640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4004120 0x7f6ff400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.954+0000 7f7013400640 1 -- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ff4004bd0 msgr2=0x7f6ff4004fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.954+0000 7f7013400640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ff4004bd0 0x7f6ff4004fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.954+0000 7f7013400640 1 -- 172.21.15.43:0/3915284044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140b3910 con 0x7f6ff4003760 2024-01-16T14:59:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.955+0000 7f7019000640 1 -- 172.21.15.43:0/3915284044 reap_dead start 2024-01-16T14:59:34.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.957+0000 7f7013400640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 0x7f6ff4003be0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f70040018c0 tx=0x7f6ff40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bd4687c3d2bfc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.958+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004005330 con 0x7f6ff4003760 2024-01-16T14:59:34.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.958+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7004005490 con 0x7f6ff4003760 2024-01-16T14:59:34.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.959+0000 7f7012a00640 1 -- 172.21.15.43:0/3915284044 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004003810 con 0x7f6ff4003760 2024-01-16T14:59:34.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.960+0000 7f701b000640 1 -- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 msgr2=0x7f6ff4003be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.960+0000 7f701b000640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 0x7f6ff4003be0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f70040018c0 tx=0x7f6ff40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.960+0000 7f701b000640 1 -- 172.21.15.43:0/3915284044 shutdown_connections 2024-01-16T14:59:34.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.960+0000 7f701b000640 1 --2- 172.21.15.43:0/3915284044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ff4003760 0x7f6ff4003be0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.960+0000 7f701b000640 1 -- 172.21.15.43:0/3915284044 >> 172.21.15.43:0/3915284044 conn(0x7f701409c3e0 msgr2=0x7f70140d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.961+0000 7f701b000640 1 -- 172.21.15.43:0/3915284044 shutdown_connections 2024-01-16T14:59:34.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.961+0000 7f701b000640 1 -- 172.21.15.43:0/3915284044 wait complete. 2024-01-16T14:59:34.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.961+0000 7fbcc7200640 1 Processor -- start 2024-01-16T14:59:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.961+0000 7fbcc7200640 1 -- start start 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcc0006fc0 0x7fbcc00df0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc00e3a60 con 0x7fbcc000aaa0 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc00535c0 con 0x7fbcc0053070 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc004f810 con 0x7fbcc0006fc0 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc5e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbcc000aaa0 0x7fbcc00e2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52972/0 (socket says 172.21.15.43:52972) 2024-01-16T14:59:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.964+0000 7fbcc5e00640 1 -- 172.21.15.43:0/3332123997 learned_addr learned my addr 172.21.15.43:0/3332123997 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.965+0000 7fbcc6800640 1 --2- 172.21.15.43:0/3332123997 >> v2:172.21.15.69:3300/0 conn(0x7fbcc0006fc0 0x7fbcc00df0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.965+0000 7f701b000640 1 Processor -- start 2024-01-16T14:59:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.965+0000 7f701b000640 1 -- start start 2024-01-16T14:59:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.966+0000 7fbcc4a00640 1 -- 172.21.15.43:0/3332123997 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954837215 0 0) 0x7fbcc00535c0 con 0x7fbcc0053070 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.966+0000 7fbcc4a00640 1 -- 172.21.15.43:0/3332123997 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca8003520 con 0x7fbcc0053070 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.966+0000 7fbcc6800640 1 -- 172.21.15.43:0/3332123997 >> v1:172.21.15.69:6789/0 conn(0x7fbcc0053070 legacy=0x7fbcc00e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.966+0000 7fbcc6800640 1 -- 172.21.15.43:0/3332123997 >> v1:172.21.15.43:6789/0 conn(0x7fbcc000aaa0 legacy=0x7fbcc00e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.966+0000 7fbcc6800640 1 -- 172.21.15.43:0/3332123997 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00538a0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4003760 0x7f70140ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70140b5c00 0x7f70140b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70140b7d10 con 0x7f70140b65c0 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70140b6a40 con 0x7f70140b5c00 2024-01-16T14:59:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.967+0000 7f701b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701415dd20 con 0x7f6ff4003760 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7fbcc6800640 1 --2- 172.21.15.43:0/3332123997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00df0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00097a0 tx=0x7fbcb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b333012c665ae15e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7f7019000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7f7019000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34190/0 (socket says 172.21.15.43:34190) 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7f7019000640 1 -- 172.21.15.43:0/992153548 learned_addr learned my addr 172.21.15.43:0/992153548 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7f7013e00640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4003760 0x7f70140ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.968+0000 7f7013400640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70140b5c00 0x7f70140b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7fbcc4a00640 1 -- 172.21.15.43:0/3332123997 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0030840 con 0x7fbcc0006fc0 2024-01-16T14:59:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7fbcc4a00640 1 -- 172.21.15.43:0/3332123997 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb00309a0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7fbcc4a00640 1 -- 172.21.15.43:0/3332123997 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0030d70 con 0x7fbcc0006fc0 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7f7019000640 1 -- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4003760 msgr2=0x7f70140ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7f7019000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4003760 0x7f70140ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7f7019000640 1 -- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70140b5c00 msgr2=0x7f70140b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7f7019000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70140b5c00 0x7f70140b6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 -- 172.21.15.43:0/3332123997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc0006fc0 msgr2=0x7fbcc00df0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 --2- 172.21.15.43:0/3332123997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00df0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00097a0 tx=0x7fbcb002b980 comp rx=0 tx=0).stop 2024-01-16T14:59:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.969+0000 7f7019000640 1 -- 172.21.15.43:0/992153548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701415e000 con 0x7f70140b65c0 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 -- 172.21.15.43:0/3332123997 shutdown_connections 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 --2- 172.21.15.43:0/3332123997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00df0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 -- 172.21.15.43:0/3332123997 >> 172.21.15.43:0/3332123997 conn(0x7fbcc0009f80 msgr2=0x7fbcc000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.970+0000 7fbcc7200640 1 -- 172.21.15.43:0/3332123997 shutdown_connections 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.971+0000 7f7019000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f701406ac10 tx=0x7f700c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.971+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c018040 con 0x7f70140b65c0 2024-01-16T14:59:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.971+0000 7fbcc7200640 1 -- 172.21.15.43:0/3332123997 wait complete. 2024-01-16T14:59:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.971+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c01b790 con 0x7f70140b65c0 2024-01-16T14:59:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.971+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c009c60 con 0x7f70140b65c0 2024-01-16T14:59:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.972+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701415e2e0 con 0x7f70140b65c0 2024-01-16T14:59:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.972+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701415e8a0 con 0x7f70140b65c0 2024-01-16T14:59:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.974+0000 7fbcc7200640 1 Processor -- start 2024-01-16T14:59:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.975+0000 7fbcc7200640 1 -- start start 2024-01-16T14:59:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.975+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4005250 con 0x7f70140b65c0 2024-01-16T14:59:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.976+0000 7f562e400640 1 Processor -- start 2024-01-16T14:59:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.976+0000 7f562e400640 1 -- start start 2024-01-16T14:59:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.977+0000 7fbcc7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.977+0000 7fbcc7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc000aaa0 0x7fbcc00e1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.977+0000 7fbcc7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 0x7fbcc00d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc000aaa0 0x7fbcc00e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc004a030 con 0x7fbcc0006fc0 2024-01-16T14:59:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00d4f90 con 0x7fbcc000aaa0 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00d5270 con 0x7fbcc0053070 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34202/0 (socket says 172.21.15.43:34202) 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc5e00640 1 -- 172.21.15.43:0/27717790 learned_addr learned my addr 172.21.15.43:0/27717790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7fbcc6800640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 0x7fbcc00d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f562e400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f562806f1a0 0x7f56280b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f562e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 0x7f562806bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f562e400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562815d970 con 0x7f5628145f80 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f562e400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56280b4f80 con 0x7f562806f1a0 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f562e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56280d14e0 con 0x7f56280d0fe0 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f5627400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5628145f80 0x7f562806ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52980/0 (socket says 172.21.15.43:52980) 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.978+0000 7f5627400640 1 -- 172.21.15.43:0/957707067 learned_addr learned my addr 172.21.15.43:0/957707067 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7f5626a00640 1 --2- 172.21.15.43:0/957707067 >> v2:172.21.15.69:3300/0 conn(0x7f562806f1a0 0x7f56280b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7f5627e00640 1 --2- 172.21.15.43:0/957707067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 0x7f562806bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7fbcc5e00640 1 -- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 msgr2=0x7fbcc00d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7fbcc5e00640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 0x7fbcc00d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7fbcc5e00640 1 -- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc000aaa0 msgr2=0x7fbcc00e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7fbcc5e00640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc000aaa0 0x7fbcc00e1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7fbcc5e00640 1 -- 172.21.15.43:0/27717790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb0009400 con 0x7fbcc0006fc0 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f5627e00640 1 -- 172.21.15.43:0/957707067 >> v2:172.21.15.69:3300/0 conn(0x7f562806f1a0 msgr2=0x7f56280b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f5627e00640 1 --2- 172.21.15.43:0/957707067 >> v2:172.21.15.69:3300/0 conn(0x7f562806f1a0 0x7f56280b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.976+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f700c01bb20 con 0x7f70140b65c0 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7f700be00640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 0x7f6fec06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.979+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f700c092640 con 0x7f70140b65c0 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f7013e00640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 0x7f6fec06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f5627e00640 1 -- 172.21.15.43:0/957707067 >> v1:172.21.15.43:6789/0 conn(0x7f5628145f80 legacy=0x7f562806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f5627e00640 1 -- 172.21.15.43:0/957707067 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280d17c0 con 0x7f56280d0fe0 2024-01-16T14:59:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f755e000640 1 Processor -- start 2024-01-16T14:59:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7fbcc6800640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 0x7fbcc00d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.980+0000 7f755e000640 1 -- start start 2024-01-16T14:59:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7f5627e00640 1 --2- 172.21.15.43:0/957707067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 0x7f562806bc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5628067190 tx=0x7f562002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2541fb3188d8d72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7f7013e00640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 0x7f6fec06d800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7014050020 tx=0x7f6ffc001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7fc2f5400640 1 Processor -- start 2024-01-16T14:59:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7f5626000640 1 -- 172.21.15.43:0/957707067 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562002cb90 con 0x7f56280d0fe0 2024-01-16T14:59:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7f5626000640 1 -- 172.21.15.43:0/957707067 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562002ccf0 con 0x7f56280d0fe0 2024-01-16T14:59:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7f5626000640 1 -- 172.21.15.43:0/957707067 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620034660 con 0x7f56280d0fe0 2024-01-16T14:59:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7fbcc5e00640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80691e0 tx=0x7fbcc8067fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.982+0000 7fc2f5400640 1 -- start start 2024-01-16T14:59:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7f755e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75580ce260 0x7f75580d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7f755e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75580ce6a0 0x7f75580d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7f755e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755815d8b0 con 0x7f755815d4d0 2024-01-16T14:59:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f562e400640 1 -- 172.21.15.43:0/957707067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 msgr2=0x7f562806bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f562e400640 1 --2- 172.21.15.43:0/957707067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 0x7f562806bc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5628067190 tx=0x7f562002bde0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7f755e000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75580b91b0 con 0x7f75580ce260 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.983+0000 7f755e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75580b2400 con 0x7f75580ce6a0 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f7556a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75580ce260 0x7f75580d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f7556a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75580ce260 0x7f75580d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34208/0 (socket says 172.21.15.43:34208) 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f7556a00640 1 -- 172.21.15.43:0/3595710180 learned_addr learned my addr 172.21.15.43:0/3595710180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.984+0000 7f7557e00640 1 --2- 172.21.15.43:0/3595710180 >> v2:172.21.15.69:3300/0 conn(0x7f75580ce6a0 0x7f75580d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f7556a00640 1 -- 172.21.15.43:0/3595710180 >> v2:172.21.15.69:3300/0 conn(0x7f75580ce6a0 msgr2=0x7f75580d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f7556a00640 1 --2- 172.21.15.43:0/3595710180 >> v2:172.21.15.69:3300/0 conn(0x7f75580ce6a0 0x7f75580d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f7556a00640 1 -- 172.21.15.43:0/3595710180 >> v1:172.21.15.43:6789/0 conn(0x7f755815d4d0 legacy=0x7f75580cda70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f0161930 0x7fc2f00ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f9463800640 1 Processor -- start 2024-01-16T14:59:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f562e400640 1 -- 172.21.15.43:0/957707067 shutdown_connections 2024-01-16T14:59:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f562e400640 1 --2- 172.21.15.43:0/957707067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56280d0fe0 0x7f562806bc80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f562e400640 1 --2- 172.21.15.43:0/957707067 >> v2:172.21.15.69:3300/0 conn(0x7f562806f1a0 0x7f56280b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f562e400640 1 -- 172.21.15.43:0/957707067 >> 172.21.15.43:0/957707067 conn(0x7f56280663d0 msgr2=0x7f56280667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f562e400640 1 -- 172.21.15.43:0/957707067 shutdown_connections 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f9463800640 1 -- start start 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7f7556a00640 1 -- 172.21.15.43:0/3595710180 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75580b26e0 con 0x7f75580ce260 2024-01-16T14:59:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7f700fc00640 1 Processor -- start 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2f00719c0 0x7fc2f00ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2f006d520 0x7fc2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f006ded0 con 0x7fc2f00719c0 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f00ad510 con 0x7fc2f006d520 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.985+0000 7fc2f5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00acc50 con 0x7fc2f0161930 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2f00719c0 0x7fc2f00ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2f00719c0 0x7fc2f00ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34216/0 (socket says 172.21.15.43:34216) 2024-01-16T14:59:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 -- 172.21.15.43:0/2259172037 learned_addr learned my addr 172.21.15.43:0/2259172037 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.987+0000 7f562e400640 1 -- 172.21.15.43:0/957707067 wait complete. 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 -- 172.21.15.43:0/2259172037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f0161930 msgr2=0x7fc2f00ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 --2- 172.21.15.43:0/2259172037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f0161930 0x7fc2f00ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 -- 172.21.15.43:0/2259172037 >> v2:172.21.15.69:3300/0 conn(0x7fc2f006d520 msgr2=0x7fc2f006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ef200640 1 --2- 172.21.15.43:0/2259172037 >> v2:172.21.15.69:3300/0 conn(0x7fc2f006d520 0x7fc2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 --2- 172.21.15.43:0/2259172037 >> v2:172.21.15.69:3300/0 conn(0x7fc2f006d520 0x7fc2f006d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7fc2ede00640 1 -- 172.21.15.43:0/2259172037 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0071e60 con 0x7fc2f00719c0 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.986+0000 7f700fc00640 1 -- start start 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.987+0000 7f7556a00640 1 --2- 172.21.15.43:0/3595710180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75580ce260 0x7f75580d1850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75580d3cd0 tx=0x7f755002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fdb4ec7aeeab64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 0x7f9454039b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 0x7f945403c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f94540529f0 0x7f9454052e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94540533c0 con 0x7f94540361b0 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945403b340 con 0x7f94540529f0 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f9463800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945403b5e0 con 0x7f945404a320 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700fc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700815d4d0 0x7f70080cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700fc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70080ce260 0x7f70080d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700fc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700815d8b0 con 0x7f70080ce6c0 2024-01-16T14:59:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc806bd20 con 0x7fbcc0006fc0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f700c05d590 con 0x7f70140b65c0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc80721c0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc806bea0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7fbcc7200640 1 -- 172.21.15.43:0/27717790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc004a280 con 0x7fbcc0006fc0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7fbcc7200640 1 -- 172.21.15.43:0/27717790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc00d0ae0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f945be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 0x7f945403c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700fc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70080b91b0 con 0x7f70080ce260 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f700fc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080cdfb0 con 0x7f700815d4d0 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.988+0000 7f6fffc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70080ce260 0x7f70080d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f9460e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 0x7f9454039b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70080ce260 0x7f70080d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33704/0 (socket says 172.21.15.43:33704) 2024-01-16T14:59:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 -- 172.21.15.43:0/3800898165 learned_addr learned my addr 172.21.15.43:0/3800898165 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f700d200640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700815d4d0 0x7f70080cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f562e400640 1 Processor -- start 2024-01-16T14:59:34.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f9460e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 0x7f9454039b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55018/0 (socket says 172.21.15.43:55018) 2024-01-16T14:59:34.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f9460e00640 1 -- 172.21.15.43:0/1061717784 learned_addr learned my addr 172.21.15.43:0/1061717784 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f9461800640 1 --2- 172.21.15.43:0/1061717784 >> v2:172.21.15.69:3300/0 conn(0x7f94540529f0 0x7f9454052e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f7556000640 1 -- 172.21.15.43:0/3595710180 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755002cb10 con 0x7f75580ce260 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f7556000640 1 -- 172.21.15.43:0/3595710180 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755002cc70 con 0x7f75580ce260 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f7556000640 1 -- 172.21.15.43:0/3595710180 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75500343f0 con 0x7f75580ce260 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 -- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700815d4d0 msgr2=0x7f70080cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700815d4d0 0x7f70080cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 -- 172.21.15.43:0/3800898165 >> v1:172.21.15.69:6789/0 conn(0x7f70080ce6c0 legacy=0x7f70080b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7f6fffc00640 1 -- 172.21.15.43:0/3800898165 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080b2400 con 0x7f70080ce260 2024-01-16T14:59:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f9461800640 1 -- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 msgr2=0x7f9454039b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.989+0000 7fc2ede00640 1 --2- 172.21.15.43:0/2259172037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f00719c0 0x7fc2f00ad9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0074240 tx=0x7fc2d802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8171a4f492451c7f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f562e400640 1 -- start start 2024-01-16T14:59:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 -- 172.21.15.43:0/3595710180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75580ce260 msgr2=0x7f75580d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 --2- 172.21.15.43:0/3595710180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75580ce260 0x7f75580d1850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75580d3cd0 tx=0x7f755002bf70 comp rx=0 tx=0).stop 2024-01-16T14:59:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 -- 172.21.15.43:0/3595710180 shutdown_connections 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 --2- 172.21.15.43:0/3595710180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75580ce260 0x7f75580d1850 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 --2- 172.21.15.43:0/3595710180 >> v2:172.21.15.69:3300/0 conn(0x7f75580ce6a0 0x7f75580d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 -- 172.21.15.43:0/3595710180 >> 172.21.15.43:0/3595710180 conn(0x7f7558071f90 msgr2=0x7f75580723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7fc2ed400640 1 -- 172.21.15.43:0/2259172037 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8030850 con 0x7fc2f00719c0 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7fc2ed400640 1 -- 172.21.15.43:0/2259172037 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d80309b0 con 0x7fc2f00719c0 2024-01-16T14:59:34.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 -- 172.21.15.43:0/3595710180 shutdown_connections 2024-01-16T14:59:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f755e000640 1 -- 172.21.15.43:0/3595710180 wait complete. 2024-01-16T14:59:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f9461800640 1 --2- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 0x7f9454039b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f9461800640 1 -- 172.21.15.43:0/1061717784 >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 msgr2=0x7f945403c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f9461800640 1 --2- 172.21.15.43:0/1061717784 >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 0x7f945403c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f9461800640 1 -- 172.21.15.43:0/1061717784 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945403b8c0 con 0x7f94540529f0 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.990+0000 7f945be00640 1 --2- 172.21.15.43:0/1061717784 >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 0x7f945403c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 0x7f5628154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5628145f80 0x7f56280c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56280d02e0 con 0x7f56280d0fe0 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56280c7430 con 0x7f5628145f80 2024-01-16T14:59:34.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7f562e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56280c6820 con 0x7f562806f1a0 2024-01-16T14:59:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.991+0000 7fbcc7200640 1 -- 172.21.15.43:0/27717790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc98005250 con 0x7fbcc0006fc0 2024-01-16T14:59:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f9461800640 1 --2- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540529f0 0x7f9454052e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94540543c0 tx=0x7f944c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519fb603524fc1d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f5627400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f5626a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 0x7f5628154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f5626a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 0x7f5628154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34238/0 (socket says 172.21.15.43:34238) 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f5627400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55044/0 (socket says 172.21.15.43:55044) 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7f5627400640 1 -- 172.21.15.43:0/324202737 learned_addr learned my addr 172.21.15.43:0/324202737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbcc80725b0 con 0x7fbcc0006fc0 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7fc2ed400640 1 -- 172.21.15.43:0/2259172037 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8030d40 con 0x7fc2f00719c0 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7fc2f5400640 1 -- 172.21.15.43:0/2259172037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f00719c0 msgr2=0x7fc2f00ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.992+0000 7fc2f5400640 1 --2- 172.21.15.43:0/2259172037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f00719c0 0x7fc2f00ad9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0074240 tx=0x7fc2d802b990 comp rx=0 tx=0).stop 2024-01-16T14:59:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f945b400640 1 -- 172.21.15.43:0/1061717784 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c02e070 con 0x7f94540529f0 2024-01-16T14:59:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f6fffc00640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce260 0x7f70080d1850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009870 tx=0x7f6ff802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c36da9a99c0c32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 -- 172.21.15.43:0/2259172037 shutdown_connections 2024-01-16T14:59:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 --2- 172.21.15.43:0/2259172037 >> v2:172.21.15.69:3300/0 conn(0x7fc2f006d520 0x7fc2f006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f945b400640 1 -- 172.21.15.43:0/1061717784 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c0315a0 con 0x7f94540529f0 2024-01-16T14:59:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 --2- 172.21.15.43:0/2259172037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f00719c0 0x7fc2f00ad9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f945b400640 1 -- 172.21.15.43:0/1061717784 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c031930 con 0x7f94540529f0 2024-01-16T14:59:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f7b38600640 1 Processor -- start 2024-01-16T14:59:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 --2- 172.21.15.43:0/2259172037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f0161930 0x7fc2f00ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 -- 172.21.15.43:0/2259172037 >> 172.21.15.43:0/2259172037 conn(0x7fc2f00b0d80 msgr2=0x7fc2f00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7fc2f5400640 1 -- 172.21.15.43:0/2259172037 shutdown_connections 2024-01-16T14:59:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7fbcbe000640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 0x7fbca006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcc8103790 con 0x7fbcc0006fc0 2024-01-16T14:59:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.993+0000 7f7b38600640 1 -- start start 2024-01-16T14:59:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7fbcc5400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 0x7fbca006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f755e000640 1 Processor -- start 2024-01-16T14:59:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7fc2f5400640 1 -- 172.21.15.43:0/2259172037 wait complete. 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f9463800640 1 -- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540529f0 msgr2=0x7f9454052e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f9463800640 1 --2- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540529f0 0x7f9454052e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94540543c0 tx=0x7f944c02b840 comp rx=0 tx=0).stop 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f7007e00640 1 -- 172.21.15.43:0/3800898165 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8030840 con 0x7f70080ce260 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f7007e00640 1 -- 172.21.15.43:0/3800898165 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff80309a0 con 0x7f70080ce260 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f7007e00640 1 -- 172.21.15.43:0/3800898165 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8030d30 con 0x7f70080ce260 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 -- 172.21.15.43:0/1061717784 shutdown_connections 2024-01-16T14:59:34.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 --2- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540529f0 0x7f9454052e80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 --2- 172.21.15.43:0/1061717784 >> v2:172.21.15.43:3300/0 conn(0x7f94540361b0 0x7f945403c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 --2- 172.21.15.43:0/1061717784 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945404a320 0x7f9454039b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 -- 172.21.15.43:0/1061717784 >> 172.21.15.43:0/1061717784 conn(0x7f94540537b0 msgr2=0x7f9454053b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7fbcc5400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 0x7fbca006d7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00de120 tx=0x7fbcb8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 -- 172.21.15.43:0/1061717784 shutdown_connections 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f7b38600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b300d1f60 0x7f7b300b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f7b38600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3015d8b0 con 0x7f7b300ce260 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 -- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce260 msgr2=0x7f70080d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce260 0x7f70080d1850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009870 tx=0x7f6ff802b980 comp rx=0 tx=0).stop 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f7b38600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b2400 con 0x7f7b3015d4d0 2024-01-16T14:59:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f7b38600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300b91b0 con 0x7f7b300d1f60 2024-01-16T14:59:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 -- 172.21.15.43:0/3800898165 shutdown_connections 2024-01-16T14:59:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce260 0x7f70080d1850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 --2- 172.21.15.43:0/3800898165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700815d4d0 0x7f70080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f700fc00640 1 -- 172.21.15.43:0/3800898165 >> 172.21.15.43:0/3800898165 conn(0x7f7008071f90 msgr2=0x7f70080723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f755e000640 1 -- start start 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627e00640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5628145f80 0x7f56280c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627400640 1 -- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5628145f80 msgr2=0x7f56280c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5628145f80 0x7f56280c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627400640 1 -- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 msgr2=0x7f5628154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 0x7f5628154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.994+0000 7f5627400640 1 -- 172.21.15.43:0/324202737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5620009a70 con 0x7f562806f1a0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f5627400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f561000b430 tx=0x7f561000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011830 con 0x7f562806f1a0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56100119b0 con 0x7f562806f1a0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011d60 con 0x7f562806f1a0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56280c6b00 con 0x7f562806f1a0 2024-01-16T14:59:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56280c70c0 con 0x7f562806f1a0 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f7b35200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7b300ce260 0x7f7b300d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52994/0 (socket says 172.21.15.43:52994) 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.996+0000 7f7b35200640 1 -- 172.21.15.43:0/2484878574 learned_addr learned my addr 172.21.15.43:0/2484878574 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f7b36600640 1 --2- 172.21.15.43:0/2484878574 >> v2:172.21.15.43:3300/0 conn(0x7f7b300d1f60 0x7f7b300b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f7b36600640 1 -- 172.21.15.43:0/2484878574 >> v1:172.21.15.69:6789/0 conn(0x7f7b3015d4d0 legacy=0x7f7b300cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f7b36600640 1 -- 172.21.15.43:0/2484878574 >> v1:172.21.15.43:6789/0 conn(0x7f7b300ce260 legacy=0x7f7b300d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f7b36600640 1 -- 172.21.15.43:0/2484878574 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300b26e0 con 0x7f7b300d1f60 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f755e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75580ce260 0x7f75580c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f755e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75580ce6a0 0x7f755815e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f700fc00640 1 -- 172.21.15.43:0/3800898165 shutdown_connections 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.997+0000 7f755e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f755e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75580c72c0 con 0x7f755815d4d0 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f755e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7558160880 con 0x7f75580ce6a0 2024-01-16T14:59:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f755e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75580cba80 con 0x7f75580ce260 2024-01-16T14:59:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.995+0000 7f9463800640 1 -- 172.21.15.43:0/1061717784 wait complete. 2024-01-16T14:59:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f9463800640 1 Processor -- start 2024-01-16T14:59:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f7557e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f7557e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34262/0 (socket says 172.21.15.43:34262) 2024-01-16T14:59:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f9463800640 1 -- start start 2024-01-16T14:59:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f7557e00640 1 -- 172.21.15.43:0/1695703917 learned_addr learned my addr 172.21.15.43:0/1695703917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7b36600640 1 --2- 172.21.15.43:0/2484878574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300b4cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009d90 tx=0x7f7b2802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abc205afef31a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7557e00640 1 -- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75580ce260 msgr2=0x7f75580c6d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7557e00640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75580ce260 0x7f75580c6d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7557e00640 1 -- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75580ce6a0 msgr2=0x7f755815e430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7557e00640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75580ce6a0 0x7f755815e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7f7557e00640 1 -- 172.21.15.43:0/1695703917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550009b30 con 0x7f755815d4d0 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.999+0000 7fc2f5400640 1 Processor -- start 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.000+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f8005250 con 0x7f562806f1a0 2024-01-16T14:59:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.000+0000 7f7557e00640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f754800e930 tx=0x7f754800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:34.998+0000 7f700fc00640 1 -- 172.21.15.43:0/3800898165 wait complete. 2024-01-16T14:59:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.000+0000 7fc2f5400640 1 -- start start 2024-01-16T14:59:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f9463800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f9463800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945404a320 0x7f9454173110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f700fc00640 1 Processor -- start 2024-01-16T14:59:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f9463800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 0x7f945404b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f9460e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f9460e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33734/0 (socket says 172.21.15.43:33734) 2024-01-16T14:59:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.001+0000 7f700fc00640 1 -- start start 2024-01-16T14:59:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f9460e00640 1 -- 172.21.15.43:0/2083659237 learned_addr learned my addr 172.21.15.43:0/2083659237 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f9461800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 0x7f945404b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f006d520 0x7fc2f0160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 0x7fc2f015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0160510 con 0x7fc2f0161930 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0160e70 con 0x7fc2f006d520 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2f5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f014bfd0 con 0x7fc2f00719c0 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f7b2fe00640 1 -- 172.21.15.43:0/2484878574 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2802cc60 con 0x7f7b300d1f60 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f7b2fe00640 1 -- 172.21.15.43:0/2484878574 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2802cdc0 con 0x7f7b300d1f60 2024-01-16T14:59:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f7b2fe00640 1 -- 172.21.15.43:0/2484878574 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28034660 con 0x7f7b300d1f60 2024-01-16T14:59:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945404a700 con 0x7f945404a320 2024-01-16T14:59:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f945be00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945404a320 0x7f9454173110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94540080d0 con 0x7f94540361b0 2024-01-16T14:59:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9454173b60 con 0x7f94540529f0 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2ede00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2ee800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f006d520 0x7fc2f0160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2ede00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55052/0 (socket says 172.21.15.43:55052) 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7fc2ede00640 1 -- 172.21.15.43:0/3435406866 learned_addr learned my addr 172.21.15.43:0/3435406866 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ef200640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 0x7fc2f015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.002+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548009690 con 0x7f755815d4d0 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f7b38600640 1 -- 172.21.15.43:0/2484878574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 msgr2=0x7f7b300b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f7b38600640 1 --2- 172.21.15.43:0/2484878574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300b4cf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009d90 tx=0x7f7b2802beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ede00640 1 -- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f006d520 msgr2=0x7fc2f0160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ede00640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f006d520 0x7fc2f0160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ede00640 1 -- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 msgr2=0x7fc2f015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75581605a0 con 0x7f755815d4d0 2024-01-16T14:59:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75580cc000 con 0x7f755815d4d0 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800c980 con 0x7f755815d4d0 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800cd30 con 0x7f755815d4d0 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ede00640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 0x7fc2f015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.003+0000 7fc2ede00640 1 -- 172.21.15.43:0/3435406866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d80094d0 con 0x7fc2f00719c0 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 -- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 msgr2=0x7f945404b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 0x7f945404b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 -- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945404a320 msgr2=0x7f9454173110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945404a320 0x7f9454173110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c009400 con 0x7f94540361b0 2024-01-16T14:59:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f7b38600640 1 -- 172.21.15.43:0/2484878574 shutdown_connections 2024-01-16T14:59:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9461800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 0x7f945404b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f7b38600640 1 --2- 172.21.15.43:0/2484878574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300b4cf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f7b38600640 1 -- 172.21.15.43:0/2484878574 >> 172.21.15.43:0/2484878574 conn(0x7f7b30071f90 msgr2=0x7f7b300723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcc80ce6b0 con 0x7fbcc0006fc0 2024-01-16T14:59:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f7b38600640 1 -- 172.21.15.43:0/2484878574 shutdown_connections 2024-01-16T14:59:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f700fc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 0x7f70080cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7f9460e00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f945c0693c0 tx=0x7f945c074a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.004+0000 7fc2ef200640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 0x7fc2f015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce6c0 0x7f70082033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f7b38600640 1 -- 172.21.15.43:0/2484878574 wait complete. 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700fc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 0x7f70080b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c072b20 con 0x7f94540361b0 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c06a440 con 0x7f94540361b0 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c06a7d0 con 0x7f94540361b0 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945404a950 con 0x7f94540361b0 2024-01-16T14:59:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945404af10 con 0x7f94540361b0 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945400fd40 con 0x7f94540361b0 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700fc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008203bf0 con 0x7f700815d4d0 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 0x7f70080cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700fc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008203ed0 con 0x7f70080ce6c0 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f700fc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70082041b0 con 0x7f70080ce260 2024-01-16T14:59:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.006+0000 7f700d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 0x7f70080cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55058/0 (socket says 172.21.15.43:55058) 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.006+0000 7f700d200640 1 -- 172.21.15.43:0/1134025352 learned_addr learned my addr 172.21.15.43:0/1134025352 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.006+0000 7f6fffc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce6c0 0x7f70082033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.006+0000 7f700dc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 0x7f70080b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f700dc00640 1 -- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 msgr2=0x7f70080cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f700dc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 0x7f70080cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f700dc00640 1 -- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce6c0 msgr2=0x7f70082033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f700dc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce6c0 0x7f70082033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f700dc00640 1 -- 172.21.15.43:0/1134025352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80094d0 con 0x7f700815d4d0 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7fc2ede00640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c00 tx=0x7fc2d8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7548013470 con 0x7f755815d4d0 2024-01-16T14:59:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f700dc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 0x7f70080b7580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f70080d0d20 tx=0x7f700000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.005+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f561001a400 con 0x7f562806f1a0 2024-01-16T14:59:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000006d50 con 0x7f700815d4d0 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f561f400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 0x7f560006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.007+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5610089e20 con 0x7f562806f1a0 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f7b38600640 1 Processor -- start 2024-01-16T14:59:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700815ecb0 con 0x7f700815d4d0 2024-01-16T14:59:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7008203900 con 0x7f700815d4d0 2024-01-16T14:59:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.009+0000 7f7b38600640 1 -- start start 2024-01-16T14:59:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.008+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000003970 con 0x7f700815d4d0 2024-01-16T14:59:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.009+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003d60 con 0x7f700815d4d0 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.009+0000 7f5626a00640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 0x7f560006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.009+0000 7f754f400640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 0x7f7530076060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.009+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7548089da0 con 0x7f755815d4d0 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.010+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f945c06a930 con 0x7f94540361b0 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.010+0000 7f7557400640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 0x7f7530076060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.010+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd8005250 con 0x7f700815d4d0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.010+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7528005250 con 0x7f755815d4d0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.011+0000 7f7b38600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b300ce260 0x7f7b30154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.011+0000 7f9458c00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 0x7f943406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.011+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f945c103bd0 con 0x7f94540361b0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.012+0000 7f7b38600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.012+0000 7f7557400640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 0x7f7530076060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7540001440 tx=0x7f75400079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.012+0000 7f945be00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 0x7f943406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.012+0000 7f7b35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b300ce260 0x7f7b30154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b38600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3015d4d0 0x7f7b3015da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b38600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300c9280 con 0x7f7b300d1f60 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f5626a00640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 0x7f560006d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5628157620 tx=0x7f5618008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b35200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b35200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34286/0 (socket says 172.21.15.43:34286) 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b35200640 1 -- 172.21.15.43:0/3677127060 learned_addr learned my addr 172.21.15.43:0/3677127060 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f945be00640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 0x7f943406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f94540e1550 tx=0x7f9444008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301605d0 con 0x7f7b300ce260 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301548a0 con 0x7f7b3015d4d0 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.013+0000 7f7b36600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3015d4d0 0x7f7b3015da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7b35200640 1 -- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3015d4d0 msgr2=0x7f7b3015da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7b35200640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3015d4d0 0x7f7b3015da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7b35200640 1 -- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b300ce260 msgr2=0x7f7b30154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7b35200640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b300ce260 0x7f7b30154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7b35200640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28009a70 con 0x7f7b300d1f60 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.014+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f70000050d0 con 0x7f700815d4d0 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.015+0000 7f7b35200640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c400 tx=0x7f7b1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.016+0000 7f7005600640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 0x7f6fe006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.016+0000 7f700d200640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 0x7f6fe006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.017+0000 7f700d200640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 0x7f6fe006d7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009b60 tx=0x7f6ff0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.018+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7000058f50 con 0x7f700815d4d0 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.019+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7548054d40 con 0x7f755815d4d0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.019+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18005ce0 con 0x7f7b300d1f60 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.019+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b300c96a0 con 0x7f7b300d1f60 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.019+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b300c8bc0 con 0x7f7b300d1f60 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.021+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b30074770 con 0x7f7b300d1f60 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.026+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b18005e40 con 0x7f7b300d1f60 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.026+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18007270 con 0x7f7b300d1f60 2024-01-16T14:59:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.027+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7000058d20 con 0x7f700815d4d0 2024-01-16T14:59:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.028+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b180073d0 con 0x7f7b300d1f60 2024-01-16T14:59:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.029+0000 7f7b2d600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 0x7f7b0806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.029+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b180895b0 con 0x7f7b300d1f60 2024-01-16T14:59:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.030+0000 7f7b35c00640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 0x7f7b0806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.031+0000 7f7b35c00640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 0x7f7b0806d7b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b200017f0 tx=0x7f7b20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.036+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b18054490 con 0x7f7b300d1f60 2024-01-16T14:59:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.037+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8039e20 con 0x7fc2f00719c0 2024-01-16T14:59:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.037+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8043400 con 0x7fc2f00719c0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.037+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80437b0 con 0x7fc2f00719c0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.037+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f015fb70 con 0x7fc2f00719c0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.037+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f0160130 con 0x7fc2f00719c0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.038+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5610058670 con 0x7f562806f1a0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.038+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f006e0e0 con 0x7fc2f00719c0 2024-01-16T14:59:35.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.041+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f945c0cec60 con 0x7f94540361b0 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.041+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2d802e070 con 0x7fc2f00719c0 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.042+0000 7fc2e6a00640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 0x7fc2c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.043+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2d803e070 con 0x7fc2f00719c0 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.044+0000 7fc2ee800640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 0x7fc2c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.045+0000 7fc2ee800640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 0x7fc2c806d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2f015d520 tx=0x7fc2e0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.053+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2d80783c0 con 0x7fc2f00719c0 2024-01-16T14:59:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.401+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6fe40057d0 con 0x7f70140b65c0 2024-01-16T14:59:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.402+0000 7fbcc7200640 1 -- 172.21.15.43:0/27717790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbc98005500 con 0x7fbcc0006fc0 2024-01-16T14:59:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.402+0000 7f700be00640 1 -- 172.21.15.43:0/992153548 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f700c060bb0 con 0x7f70140b65c0 2024-01-16T14:59:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:60129542147 2024-01-16T14:59:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.403+0000 7fbcbe000640 1 -- 172.21.15.43:0/27717790 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcc80d1cd0 con 0x7fbcc0006fc0 2024-01-16T14:59:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.406+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 msgr2=0x7f6fec06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.406+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 0x7f6fec06d800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7014050020 tx=0x7f6ffc001630 comp rx=0 tx=0).stop 2024-01-16T14:59:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.406+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 msgr2=0x7f701415edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.406+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f701406ac10 tx=0x7f700c004410 comp rx=0 tx=0).stop 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 shutdown_connections 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fec06b340 0x7f6fec06d800 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70140b5c00 0x7f70140b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70140b65c0 0x7f701415edc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 --2- 172.21.15.43:0/992153548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ff4003760 0x7f70140ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.407+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 >> 172.21.15.43:0/992153548 conn(0x7f701409c3e0 msgr2=0x7f70140b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.408+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 shutdown_connections 2024-01-16T14:59:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.408+0000 7f701b000640 1 -- 172.21.15.43:0/992153548 wait complete. 2024-01-16T14:59:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 msgr2=0x7fbca006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 0x7fbca006d7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00de120 tx=0x7fbcb8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 msgr2=0x7fbcc00e0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80691e0 tx=0x7fbcc8067fe0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 shutdown_connections 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca006b310 0x7fbca006d7d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc000aaa0 0x7fbcc00e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc0053070 0x7fbcc00d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 --2- 172.21.15.43:0/27717790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc0006fc0 0x7fbcc00e0f10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 >> 172.21.15.43:0/27717790 conn(0x7fbcc0009f80 msgr2=0x7fbcc0050af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.409+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 shutdown_connections 2024-01-16T14:59:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.410+0000 7fbcb7400640 1 -- 172.21.15.43:0/27717790 wait complete. 2024-01-16T14:59:35.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.3 2024-01-16T14:59:35.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.1 2024-01-16T14:59:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.443+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6fd80057d0 con 0x7f700815d4d0 2024-01-16T14:59:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.443+0000 7f7005600640 1 -- 172.21.15.43:0/1134025352 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7000001710 con 0x7f700815d4d0 2024-01-16T14:59:35.424 INFO:teuthology.orchestra.run.smithi043.stdout:60129542147 2024-01-16T14:59:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.445+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7528005500 con 0x7f755815d4d0 2024-01-16T14:59:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.445+0000 7f754f400640 1 -- 172.21.15.43:0/1695703917 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7548021070 con 0x7f755815d4d0 2024-01-16T14:59:35.426 INFO:teuthology.orchestra.run.smithi043.stdout:60129542147 2024-01-16T14:59:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 msgr2=0x7f6fe006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 0x7f6fe006d7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009b60 tx=0x7f6ff0009190 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 msgr2=0x7f70080b7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 0x7f70080b7580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f70080d0d20 tx=0x7f700000cb90 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 shutdown_connections 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fe006b310 0x7f6fe006d7d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70080ce6c0 0x7f70082033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700815d4d0 0x7f70080b7580 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 --2- 172.21.15.43:0/1134025352 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70080ce260 0x7f70080cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.446+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 >> 172.21.15.43:0/1134025352 conn(0x7f7008071f90 msgr2=0x7f700815bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.447+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 shutdown_connections 2024-01-16T14:59:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.447+0000 7f700fc00640 1 -- 172.21.15.43:0/1134025352 wait complete. 2024-01-16T14:59:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.449+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f94540079c0 con 0x7f94540361b0 2024-01-16T14:59:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.449+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 msgr2=0x7f7530076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.449+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 0x7f7530076060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7540001440 tx=0x7f75400079d0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.449+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 msgr2=0x7f7558146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.449+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f754800e930 tx=0x7f754800edf0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.450+0000 7f9458c00640 1 -- 172.21.15.43:0/2083659237 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f945c098020 con 0x7f94540361b0 2024-01-16T14:59:35.431 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 shutdown_connections 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7530073ba0 0x7f7530076060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75580ce6a0 0x7f755815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f755815d4d0 0x7f7558146280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 --2- 172.21.15.43:0/1695703917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75580ce260 0x7f75580c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 >> 172.21.15.43:0/1695703917 conn(0x7f7558071f90 msgr2=0x7f75580b67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 shutdown_connections 2024-01-16T14:59:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.452+0000 7f755e000640 1 -- 172.21.15.43:0/1695703917 wait complete. 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 msgr2=0x7f943406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 0x7f943406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f94540e1550 tx=0x7f9444008040 comp rx=0 tx=0).stop 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 msgr2=0x7f9454007b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f945c0693c0 tx=0x7f945c074a20 comp rx=0 tx=0).stop 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 shutdown_connections 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406b2c0 0x7f943406d780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94540529f0 0x7f945404b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945404a320 0x7f9454173110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 --2- 172.21.15.43:0/2083659237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94540361b0 0x7f9454007b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 >> 172.21.15.43:0/2083659237 conn(0x7f94540537b0 msgr2=0x7f94540378d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 shutdown_connections 2024-01-16T14:59:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.453+0000 7f9463800640 1 -- 172.21.15.43:0/2083659237 wait complete. 2024-01-16T14:59:35.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.456+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7b3015b840 con 0x7f7b300d1f60 2024-01-16T14:59:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.456+0000 7f7b2d600640 1 -- 172.21.15.43:0/3677127060 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7b18057ab0 con 0x7f7b300d1f60 2024-01-16T14:59:35.437 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 msgr2=0x7f7b0806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 0x7f7b0806d7b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b200017f0 tx=0x7f7b20009040 comp rx=0 tx=0).stop 2024-01-16T14:59:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 msgr2=0x7f7b300c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c400 tx=0x7f7b1800c8c0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 shutdown_connections 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b0806b2f0 0x7f7b0806d7b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3015d4d0 0x7f7b3015da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b300d1f60 0x7f7b300c7cb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 --2- 172.21.15.43:0/3677127060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b300ce260 0x7f7b30154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.460+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 >> 172.21.15.43:0/3677127060 conn(0x7f7b30071f90 msgr2=0x7f7b3015b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.461+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 shutdown_connections 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.461+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f55f8005500 con 0x7f562806f1a0 2024-01-16T14:59:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.461+0000 7f7b38600640 1 -- 172.21.15.43:0/3677127060 wait complete. 2024-01-16T14:59:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.462+0000 7f561f400640 1 -- 172.21.15.43:0/324202737 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5610015020 con 0x7f562806f1a0 2024-01-16T14:59:35.443 INFO:teuthology.orchestra.run.smithi043.stdout:60129542148 2024-01-16T14:59:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.465+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 msgr2=0x7f560006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.465+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 0x7f560006d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5628157620 tx=0x7f5618008040 comp rx=0 tx=0).stop 2024-01-16T14:59:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.465+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 msgr2=0x7f562813f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.465+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f561000b430 tx=0x7f561000b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 shutdown_connections 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f560006b390 0x7f560006d850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5628145f80 0x7f56280c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56280d0fe0 0x7f5628154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 --2- 172.21.15.43:0/324202737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f562806f1a0 0x7f562813f610 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 >> 172.21.15.43:0/324202737 conn(0x7f56280663d0 msgr2=0x7f5628146360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 shutdown_connections 2024-01-16T14:59:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.466+0000 7f562e400640 1 -- 172.21.15.43:0/324202737 wait complete. 2024-01-16T14:59:35.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.6 2024-01-16T14:59:35.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.5 2024-01-16T14:59:35.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.2 2024-01-16T14:59:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.479+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc2f00cedd0 con 0x7fc2f00719c0 2024-01-16T14:59:35.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.480+0000 7fc2e6a00640 1 -- 172.21.15.43:0/3435406866 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc2d8080e60 con 0x7fc2f00719c0 2024-01-16T14:59:35.461 INFO:teuthology.orchestra.run.smithi043.stdout:60129542147 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.483+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 msgr2=0x7fc2c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.483+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 0x7fc2c806d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2f015d520 tx=0x7fc2e0009060 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.483+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 msgr2=0x7fc2f00b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.483+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c00 tx=0x7fc2d8004730 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 shutdown_connections 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc2c806b2c0 0x7fc2c806d780 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f0161930 0x7fc2f015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f00719c0 0x7fc2f00b30b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 --2- 172.21.15.43:0/3435406866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f006d520 0x7fc2f0160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 >> 172.21.15.43:0/3435406866 conn(0x7fc2f00b0d80 msgr2=0x7fc2f00aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 shutdown_connections 2024-01-16T14:59:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:35.484+0000 7fc2f5400640 1 -- 172.21.15.43:0/3435406866 wait complete. 2024-01-16T14:59:35.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.0 2024-01-16T14:59:35.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-01-16T14:59:35.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.4 2024-01-16T14:59:36.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T14:59:36.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T14:59:36.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T14:59:36.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T14:59:36.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T14:59:36.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T14:59:36.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T14:59:36.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T14:59:36.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.721+0000 7f7e18600640 1 Processor -- start 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f4339c00640 1 Processor -- start 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.721+0000 7f7e18600640 1 -- start start 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.723+0000 7f7e18600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e100d2920 0x7f7e100d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.723+0000 7f7e18600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b1d20 0x7f7e100b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e18600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100d1d50 con 0x7f7e100b4c80 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e18600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100d2030 con 0x7f7e100d2920 2024-01-16T14:59:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e18600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100d2310 con 0x7f7e100b1d20 2024-01-16T14:59:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e15200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e100d2920 0x7f7e100d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e15200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e100d2920 0x7f7e100d13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34288/0 (socket says 172.21.15.43:34288) 2024-01-16T14:59:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f7e15200640 1 -- 172.21.15.43:0/2853070645 learned_addr learned my addr 172.21.15.43:0/2853070645 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.725+0000 7f7e16600640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b1d20 0x7f7e100b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.725+0000 7f7e15200640 1 -- 172.21.15.43:0/2853070645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b1d20 msgr2=0x7f7e100b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.725+0000 7f7e15200640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b1d20 0x7f7e100b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.725+0000 7f7e15200640 1 -- 172.21.15.43:0/2853070645 >> v1:172.21.15.43:6789/0 conn(0x7f7e100b4c80 legacy=0x7f7e100b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.725+0000 7f7e15200640 1 -- 172.21.15.43:0/2853070645 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100b65e0 con 0x7f7e100d2920 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.724+0000 7f4339c00640 1 -- start start 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.726+0000 7f4339c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f433415d4d0 0x7f43340cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.726+0000 7f4339c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f43340d1f60 0x7f43340b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.726+0000 7f4339c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433415d8b0 con 0x7f43340ce260 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.726+0000 7f4339c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43340b91b0 con 0x7f43340d1f60 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.726+0000 7f4339c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43340cdfb0 con 0x7f433415d4d0 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.727+0000 7f4332000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f43340ce260 0x7f43340d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53008/0 (socket says 172.21.15.43:53008) 2024-01-16T14:59:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.727+0000 7f4332000640 1 -- 172.21.15.43:0/3893489964 learned_addr learned my addr 172.21.15.43:0/3893489964 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.727+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> v2:172.21.15.43:3300/0 conn(0x7f43340d1f60 0x7f43340b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895181175 0 0) 0x7f433415d8b0 con 0x7f43340ce260 2024-01-16T14:59:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4332a00640 1 --2- 172.21.15.43:0/3893489964 >> v2:172.21.15.69:3300/0 conn(0x7f433415d4d0 0x7f43340cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4314003520 con 0x7f43340ce260 2024-01-16T14:59:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211087218 0 0) 0x7f4314003520 con 0x7f43340ce260 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 >> v2:172.21.15.69:3300/0 conn(0x7f433415d4d0 msgr2=0x7f43340cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 --2- 172.21.15.43:0/3893489964 >> v2:172.21.15.69:3300/0 conn(0x7f433415d4d0 0x7f43340cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.728+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 >> v2:172.21.15.43:3300/0 conn(0x7f43340d1f60 msgr2=0x7f43340b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 --2- 172.21.15.43:0/3893489964 >> v2:172.21.15.43:3300/0 conn(0x7f43340d1f60 0x7f43340b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340b2400 con 0x7f43340ce260 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f432c001a10 con 0x7f43340ce260 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 >> v1:172.21.15.43:6789/0 conn(0x7f43340ce260 legacy=0x7f43340d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4314003970 0x7f4314003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43140047c0 0x7f4314004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f431400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340b91b0 con 0x7f4314005180 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340cdfb0 con 0x7f4314003970 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431400c0a0 con 0x7f43140047c0 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> v2:172.21.15.43:3300/0 conn(0x7f43340d1f60 0x7f43340b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f4332000640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43140047c0 0x7f4314004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.727+0000 7f7e15200640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100d2920 0x7f7e100d13e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a717b98a70d03a1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f7e0fe00640 1 -- 172.21.15.43:0/2853070645 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00031640 con 0x7f7e100d2920 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.729+0000 7f7e0fe00640 1 -- 172.21.15.43:0/2853070645 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e000317a0 con 0x7f7e100d2920 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f431400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 -- 172.21.15.43:0/2853070645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100d2920 msgr2=0x7f7e100d13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100d2920 0x7f7e100d13e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 -- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43140047c0 msgr2=0x7f4314004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43140047c0 0x7f4314004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 -- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4314003970 msgr2=0x7f4314003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4314003970 0x7f4314003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f4333400640 1 -- 172.21.15.43:0/3893489964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4314003520 con 0x7f4314005180 2024-01-16T14:59:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.731+0000 7f4332000640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43140047c0 0x7f4314004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 -- 172.21.15.43:0/2853070645 shutdown_connections 2024-01-16T14:59:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b1d20 0x7f7e100b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 --2- 172.21.15.43:0/2853070645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100d2920 0x7f7e100d13e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.730+0000 7f7e18600640 1 -- 172.21.15.43:0/2853070645 >> 172.21.15.43:0/2853070645 conn(0x7f7e100721a0 msgr2=0x7f7e100cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.732+0000 7f7e18600640 1 -- 172.21.15.43:0/2853070645 shutdown_connections 2024-01-16T14:59:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.732+0000 7f4333400640 1 -- 172.21.15.43:0/3893489964 reap_dead start 2024-01-16T14:59:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.732+0000 7f4333400640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f431400b980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f431c009c60 tx=0x7f43140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18f96a82d9d3e11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.733+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c02c940 con 0x7f4314005180 2024-01-16T14:59:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.733+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c02caa0 con 0x7f4314005180 2024-01-16T14:59:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.734+0000 7f4339c00640 1 -- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 msgr2=0x7f431400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.734+0000 7f4331600640 1 -- 172.21.15.43:0/3893489964 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c02c940 con 0x7f4314005180 2024-01-16T14:59:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.734+0000 7f4339c00640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f431400b980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f431c009c60 tx=0x7f43140057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.732+0000 7f7e18600640 1 -- 172.21.15.43:0/2853070645 wait complete. 2024-01-16T14:59:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.735+0000 7f7e18600640 1 Processor -- start 2024-01-16T14:59:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.735+0000 7f4339c00640 1 -- 172.21.15.43:0/3893489964 shutdown_connections 2024-01-16T14:59:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.735+0000 7f4339c00640 1 --2- 172.21.15.43:0/3893489964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f431400b980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.735+0000 7f4339c00640 1 -- 172.21.15.43:0/3893489964 >> 172.21.15.43:0/3893489964 conn(0x7f4334071f90 msgr2=0x7f43340723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.735+0000 7f7e18600640 1 -- start start 2024-01-16T14:59:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.736+0000 7f4339c00640 1 -- 172.21.15.43:0/3893489964 shutdown_connections 2024-01-16T14:59:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.737+0000 7f7e18600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.737+0000 7f7e18600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 0x7f7e101596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.737+0000 7f7e18600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 0x7f7e100cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.738+0000 7f7e15c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.738+0000 7f7e15c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34314/0 (socket says 172.21.15.43:34314) 2024-01-16T14:59:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.736+0000 7f4339c00640 1 -- 172.21.15.43:0/3893489964 wait complete. 2024-01-16T14:59:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e15200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 0x7f7e101596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f4339c00640 1 Processor -- start 2024-01-16T14:59:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e15200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 0x7f7e101596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55102/0 (socket says 172.21.15.43:55102) 2024-01-16T14:59:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f4339c00640 1 -- start start 2024-01-16T14:59:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e15c00640 1 -- 172.21.15.43:0/2967697322 learned_addr learned my addr 172.21.15.43:0/2967697322 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100cce60 con 0x7f7e100b1d20 2024-01-16T14:59:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100c5530 con 0x7f7e100cc4a0 2024-01-16T14:59:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.739+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100c3fc0 con 0x7f7e100b4c80 2024-01-16T14:59:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.740+0000 7f7e16600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 0x7f7e100cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e16600640 1 -- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 msgr2=0x7f7e101596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e16600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 0x7f7e101596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e16600640 1 -- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 msgr2=0x7f7e100c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f4339c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f43340ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e16600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e16600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000094d0 con 0x7f7e100cc4a0 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.741+0000 7f7e15c00640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4339c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4339c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43340c89c0 0x7f43340c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4339c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334160860 con 0x7f4314005180 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4339c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c9360 con 0x7f43341603e0 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4339c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340caa00 con 0x7f43340c89c0 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f7e16600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 0x7f7e100cc920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007d70 tx=0x7f7e0800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4332000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.742+0000 7f4332a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f43340ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4333400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43340c89c0 0x7f43340c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4332000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33780/0 (socket says 172.21.15.43:33780) 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4332000640 1 -- 172.21.15.43:0/2192103280 learned_addr learned my addr 172.21.15.43:0/2192103280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08003990 con 0x7f7e100cc4a0 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08003b10 con 0x7f7e100cc4a0 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08004030 con 0x7f7e100cc4a0 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e100c5c10 con 0x7f7e100cc4a0 2024-01-16T14:59:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e100c4c00 con 0x7f7e100cc4a0 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4332000640 1 -- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43340c89c0 msgr2=0x7f43340c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4332000640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43340c89c0 0x7f43340c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.743+0000 7f4332000640 1 -- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 msgr2=0x7f43340ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.744+0000 7f4332000640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f43340ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.744+0000 7f4332000640 1 -- 172.21.15.43:0/2192103280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340cace0 con 0x7f43341603e0 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.744+0000 7f4332a00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f43340ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.745+0000 7f4332000640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f432c001350 tx=0x7f432c007430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.745+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c0133f0 con 0x7f43341603e0 2024-01-16T14:59:36.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.745+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c013550 con 0x7f43341603e0 2024-01-16T14:59:36.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.745+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c013900 con 0x7f43341603e0 2024-01-16T14:59:36.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.746+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340cafc0 con 0x7f43341603e0 2024-01-16T14:59:36.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.746+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43340cb580 con 0x7f43341603e0 2024-01-16T14:59:36.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.748+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4334074770 con 0x7f43341603e0 2024-01-16T14:59:36.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.747+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10074770 con 0x7f7e100cc4a0 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.750+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7e08004190 con 0x7f7e100cc4a0 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.751+0000 7f7e0d600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 0x7f7de806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.752+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e08089b10 con 0x7f7e100cc4a0 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.751+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f432c013a60 con 0x7f43341603e0 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.752+0000 7f432aa00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 0x7f430c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.752+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f432c08ad50 con 0x7f43341603e0 2024-01-16T14:59:36.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.753+0000 7f7e15c00640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 0x7f7de806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.753+0000 7f4332a00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 0x7f430c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.754+0000 7f7e15c00640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 0x7f7de806d850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e100ce3a0 tx=0x7f7df8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.754+0000 7f4332a00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 0x7f430c06d880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4324006b10 tx=0x7f4324008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.763+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f432c055ce0 con 0x7f43341603e0 2024-01-16T14:59:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.775+0000 7f2550800640 1 Processor -- start 2024-01-16T14:59:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.776+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e08054a10 con 0x7f7e100cc4a0 2024-01-16T14:59:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.777+0000 7f2550800640 1 -- start start 2024-01-16T14:59:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.780+0000 7fde23e00640 1 Processor -- start 2024-01-16T14:59:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f2550800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f254806eb00 0x7f254806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.780+0000 7fde23e00640 1 -- start start 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7fde23e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde1c0ce260 0x7fde1c0d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7fde23e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c15d8b0 con 0x7fde1c15d4d0 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7fde23e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c0b2400 con 0x7fde1c0ce6a0 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7fde23e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0b91b0 con 0x7fde1c0ce260 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.783+0000 7fde20a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde1c0ce260 0x7fde1c0d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.783+0000 7fde20a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde1c0ce260 0x7fde1c0d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34330/0 (socket says 172.21.15.43:34330) 2024-01-16T14:59:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.783+0000 7fde20a00640 1 -- 172.21.15.43:0/66786308 learned_addr learned my addr 172.21.15.43:0/66786308 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.783+0000 7fde20a00640 1 -- 172.21.15.43:0/66786308 >> v1:172.21.15.69:6789/0 conn(0x7fde1c0ce6a0 legacy=0x7fde1c0b4cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.783+0000 7fde20a00640 1 -- 172.21.15.43:0/66786308 >> v1:172.21.15.43:6789/0 conn(0x7fde1c15d4d0 legacy=0x7fde1c0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f2550800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254815c3e0 con 0x7f254815ed20 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f2550800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254815b840 con 0x7f25480cfa00 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f2550800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254806f420 con 0x7f254806eb00 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f254de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f254806eb00 0x7f254806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f254de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f254806eb00 0x7f254806eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34320/0 (socket says 172.21.15.43:34320) 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.781+0000 7f254de00640 1 -- 172.21.15.43:0/1525429746 learned_addr learned my addr 172.21.15.43:0/1525429746 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7f254de00640 1 -- 172.21.15.43:0/1525429746 >> v1:172.21.15.69:6789/0 conn(0x7f25480cfa00 legacy=0x7f25480cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.782+0000 7f254de00640 1 -- 172.21.15.43:0/1525429746 >> v1:172.21.15.43:6789/0 conn(0x7f254815ed20 legacy=0x7f254815f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:36.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.784+0000 7fde20a00640 1 -- 172.21.15.43:0/66786308 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0b26e0 con 0x7fde1c0ce260 2024-01-16T14:59:36.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.784+0000 7f254de00640 1 -- 172.21.15.43:0/1525429746 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254815bab0 con 0x7f254806eb00 2024-01-16T14:59:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.786+0000 7f254de00640 1 --2- 172.21.15.43:0/1525429746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f254806eb00 0x7f254806eee0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f253800dd10 tx=0x7f253800d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bff8fa47e793cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.786+0000 7fde20a00640 1 --2- 172.21.15.43:0/66786308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce260 0x7fde1c0d1850 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009870 tx=0x7fde0c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0431f434c95b33 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.786+0000 7f6b6f000640 1 Processor -- start 2024-01-16T14:59:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.786+0000 7f6b6f000640 1 -- start start 2024-01-16T14:59:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.787+0000 7fde1be00640 1 -- 172.21.15.43:0/66786308 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c030860 con 0x7fde1c0ce260 2024-01-16T14:59:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.787+0000 7fde1be00640 1 -- 172.21.15.43:0/66786308 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0309c0 con 0x7fde1c0ce260 2024-01-16T14:59:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.787+0000 7fde1be00640 1 -- 172.21.15.43:0/66786308 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c030d50 con 0x7fde1c0ce260 2024-01-16T14:59:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.787+0000 7f254ca00640 1 -- 172.21.15.43:0/1525429746 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538034d50 con 0x7f254806eb00 2024-01-16T14:59:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.788+0000 7f254ca00640 1 -- 172.21.15.43:0/1525429746 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253800d2e0 con 0x7f254806eb00 2024-01-16T14:59:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.788+0000 7f254ca00640 1 -- 172.21.15.43:0/1525429746 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800b650 con 0x7f254806eb00 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.788+0000 7fde23e00640 1 -- 172.21.15.43:0/66786308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce260 msgr2=0x7fde1c0d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.788+0000 7fde23e00640 1 --2- 172.21.15.43:0/66786308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce260 0x7fde1c0d1850 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009870 tx=0x7fde0c02b9a0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b6f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b681619a0 0x7f6b680ae210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b6f000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6b680ae750 0x7f6b680b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7fde23e00640 1 -- 172.21.15.43:0/66786308 shutdown_connections 2024-01-16T14:59:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7fde23e00640 1 --2- 172.21.15.43:0/66786308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce260 0x7fde1c0d1850 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b6f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b68161d80 con 0x7f6b680acc20 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b6f000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680ad170 con 0x7f6b680ae750 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b6f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680ad450 con 0x7f6b681619a0 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b67400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6b680ae750 0x7f6b680b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b67400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6b680ae750 0x7f6b680b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34342/0 (socket says 172.21.15.43:34342) 2024-01-16T14:59:36.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7f6b67400640 1 -- 172.21.15.43:0/2442128313 learned_addr learned my addr 172.21.15.43:0/2442128313 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f2550800640 1 -- 172.21.15.43:0/1525429746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f254806eb00 msgr2=0x7f254806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f2550800640 1 --2- 172.21.15.43:0/1525429746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f254806eb00 0x7f254806eee0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f253800dd10 tx=0x7f253800d650 comp rx=0 tx=0).stop 2024-01-16T14:59:36.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f6b67400640 1 -- 172.21.15.43:0/2442128313 >> v2:172.21.15.69:3300/0 conn(0x7f6b681619a0 msgr2=0x7f6b680ae210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T14:59:36.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.789+0000 7fde23e00640 1 -- 172.21.15.43:0/66786308 >> 172.21.15.43:0/66786308 conn(0x7fde1c071f90 msgr2=0x7fde1c0723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f2550800640 1 -- 172.21.15.43:0/1525429746 shutdown_connections 2024-01-16T14:59:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f6b67400640 1 --2- 172.21.15.43:0/2442128313 >> v2:172.21.15.69:3300/0 conn(0x7f6b681619a0 0x7f6b680ae210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f6b67400640 1 -- 172.21.15.43:0/2442128313 >> v1:172.21.15.69:6789/0 conn(0x7f6b680acc20 legacy=0x7f6b6806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f6b67400640 1 -- 172.21.15.43:0/2442128313 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b680ad730 con 0x7f6b680ae750 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f2550800640 1 --2- 172.21.15.43:0/1525429746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f254806eb00 0x7f254806eee0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.790+0000 7f2550800640 1 -- 172.21.15.43:0/1525429746 >> 172.21.15.43:0/1525429746 conn(0x7f25480663d0 msgr2=0x7f25480667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.791+0000 7f2550800640 1 -- 172.21.15.43:0/1525429746 shutdown_connections 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.791+0000 7fde23e00640 1 -- 172.21.15.43:0/66786308 shutdown_connections 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.791+0000 7fde23e00640 1 -- 172.21.15.43:0/66786308 wait complete. 2024-01-16T14:59:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.791+0000 7f2550800640 1 -- 172.21.15.43:0/1525429746 wait complete. 2024-01-16T14:59:36.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.792+0000 7f6b67400640 1 --2- 172.21.15.43:0/2442128313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b680b32a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b680b3270 tx=0x7f6b5802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddc26b549758507 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.793+0000 7f6b66a00640 1 -- 172.21.15.43:0/2442128313 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030800 con 0x7f6b680ae750 2024-01-16T14:59:36.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.793+0000 7f6b66a00640 1 -- 172.21.15.43:0/2442128313 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b58030960 con 0x7f6b680ae750 2024-01-16T14:59:36.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.794+0000 7fde23e00640 1 Processor -- start 2024-01-16T14:59:36.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.794+0000 7fde23e00640 1 -- start start 2024-01-16T14:59:36.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f2550800640 1 Processor -- start 2024-01-16T14:59:36.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 -- 172.21.15.43:0/2442128313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 msgr2=0x7f6b680b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2442128313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b680b32a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b680b3270 tx=0x7f6b5802b940 comp rx=0 tx=0).stop 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.794+0000 7f6b66a00640 1 -- 172.21.15.43:0/2442128313 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030cf0 con 0x7f6b680ae750 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 -- 172.21.15.43:0/2442128313 shutdown_connections 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2442128313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b680b32a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2442128313 >> v2:172.21.15.69:3300/0 conn(0x7f6b681619a0 0x7f6b680ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f6b6f000640 1 -- 172.21.15.43:0/2442128313 >> 172.21.15.43:0/2442128313 conn(0x7f6b680709f0 msgr2=0x7f6b680718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 0x7fde1c1f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c15f7e0 0x7fde1c1f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1f0760 con 0x7fde1c0ce6a0 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1f3120 con 0x7fde1c15f7e0 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde23e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1f6b70 con 0x7fde1c15d4d0 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde21400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde21400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34356/0 (socket says 172.21.15.43:34356) 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.796+0000 7fde21400640 1 -- 172.21.15.43:0/3093400585 learned_addr learned my addr 172.21.15.43:0/3093400585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7fde20a00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 0x7fde1c1f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.795+0000 7f2550800640 1 -- start start 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25480cfa00 0x7f2548157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f254815ed20 0x7f25480d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254813c1f0 con 0x7f25480cfa00 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25480b4380 con 0x7f254806eb00 2024-01-16T14:59:36.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7f2550800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25480b45d0 con 0x7f254815ed20 2024-01-16T14:59:36.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.797+0000 7fde21e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c15f7e0 0x7fde1c1f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f6b6f000640 1 -- 172.21.15.43:0/2442128313 shutdown_connections 2024-01-16T14:59:36.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f254d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25480cfa00 0x7f2548157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f254de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f254de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33806/0 (socket says 172.21.15.43:33806) 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f254de00640 1 -- 172.21.15.43:0/2434694966 learned_addr learned my addr 172.21.15.43:0/2434694966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f6b6f000640 1 -- 172.21.15.43:0/2442128313 wait complete. 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde20a00640 1 -- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c15f7e0 msgr2=0x7fde1c1f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde20a00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c15f7e0 0x7fde1c1f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde20a00640 1 -- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 msgr2=0x7fde1c1f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde20a00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde20a00640 1 -- 172.21.15.43:0/3093400585 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0094d0 con 0x7fde1c15d4d0 2024-01-16T14:59:36.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7f254e800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f254815ed20 0x7f25480d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.798+0000 7fde21400640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f254de00640 1 -- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f254815ed20 msgr2=0x7f25480d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f254de00640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f254815ed20 0x7f25480d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f254de00640 1 -- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25480cfa00 msgr2=0x7f2548157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f254de00640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25480cfa00 0x7f2548157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f254de00640 1 -- 172.21.15.43:0/2434694966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253800d970 con 0x7f254806eb00 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7fde20a00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 0x7fde1c1f6290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde0c033000 tx=0x7fde0c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.799+0000 7f52a1800640 1 Processor -- start 2024-01-16T14:59:36.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7f52a1800640 1 -- start start 2024-01-16T14:59:36.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c039d60 con 0x7fde1c15d4d0 2024-01-16T14:59:36.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7f254de00640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f253800dd10 tx=0x7f25380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c004020 con 0x7fde1c15d4d0 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c039d60 con 0x7fde1c15d4d0 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800be80 con 0x7f254806eb00 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538046660 con 0x7f254806eb00 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.800+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538046a10 con 0x7f254806eb00 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.801+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25480b48b0 con 0x7f254806eb00 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.801+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25480b4e70 con 0x7f254806eb00 2024-01-16T14:59:36.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.801+0000 7f6b6f000640 1 Processor -- start 2024-01-16T14:59:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.801+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c1f6e50 con 0x7fde1c15d4d0 2024-01-16T14:59:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.801+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c1f2db0 con 0x7fde1c15d4d0 2024-01-16T14:59:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f6b6f000640 1 -- start start 2024-01-16T14:59:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f4398200640 1 Processor -- start 2024-01-16T14:59:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f4398200640 1 -- start start 2024-01-16T14:59:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7f6b6f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b680acc20 0x7f6b680b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c1f3290 con 0x7fde1c15d4d0 2024-01-16T14:59:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2548074770 con 0x7f254806eb00 2024-01-16T14:59:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7f6b6f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b681e9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681ec8a0 con 0x7f6b680ae750 2024-01-16T14:59:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f4398200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43900cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f4398200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439015d8b0 con 0x7f43900ce260 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f4398200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900cdfb0 con 0x7f43900d1f60 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f4398200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43900b91b0 con 0x7f439015d4d0 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681ecb80 con 0x7f6b681619a0 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681ece60 con 0x7f6b680acc20 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33810/0 (socket says 172.21.15.43:33810) 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f6b6d000640 1 -- 172.21.15.43:0/2593093896 learned_addr learned my addr 172.21.15.43:0/2593093896 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b67400640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b681e9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f52a1800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f529c153f10 0x7f529c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f52a1800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0ad6b0 con 0x7f529c06ddb0 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f52a1800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0b2860 con 0x7f529c15b050 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.802+0000 7f52a1800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c0b2b40 con 0x7f529c153f10 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7f529b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f529c06ddb0 0x7f529c0adb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53034/0 (socket says 172.21.15.43:53034) 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7f529b400640 1 -- 172.21.15.43:0/1763591068 learned_addr learned my addr 172.21.15.43:0/1763591068 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.803+0000 7f529aa00640 1 --2- 172.21.15.43:0/1763591068 >> v2:172.21.15.43:3300/0 conn(0x7f529c153f10 0x7f529c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f5299600640 1 -- 172.21.15.43:0/1763591068 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761083591 0 0) 0x7f529c0ad6b0 con 0x7f529c06ddb0 2024-01-16T14:59:36.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f5299600640 1 -- 172.21.15.43:0/1763591068 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527c003520 con 0x7f529c06ddb0 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f529aa00640 1 -- 172.21.15.43:0/1763591068 >> v1:172.21.15.69:6789/0 conn(0x7f529c15b050 legacy=0x7f529c0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f529aa00640 1 -- 172.21.15.43:0/1763591068 >> v1:172.21.15.43:6789/0 conn(0x7f529c06ddb0 legacy=0x7f529c0adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.804+0000 7f529aa00640 1 -- 172.21.15.43:0/1763591068 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c0b2e20 con 0x7f529c153f10 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f4394e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f43900ce260 0x7f43900d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53036/0 (socket says 172.21.15.43:53036) 2024-01-16T14:59:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f4394e00640 1 -- 172.21.15.43:0/2312682119 learned_addr learned my addr 172.21.15.43:0/2312682119 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b67e00640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b680acc20 0x7f6b680b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b6d000640 1 -- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b680acc20 msgr2=0x7f6b680b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b6d000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b680acc20 0x7f6b680b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b6d000640 1 -- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 msgr2=0x7f6b681e9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7f6b6d000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b681e9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.806+0000 7f6b6d000640 1 -- 172.21.15.43:0/2593093896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580094d0 con 0x7f6b681619a0 2024-01-16T14:59:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.806+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649105823 0 0) 0x7f439015d8b0 con 0x7f43900ce260 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.806+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4370003520 con 0x7f43900ce260 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.806+0000 7f4395800640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43900cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.806+0000 7f529aa00640 1 --2- 172.21.15.43:0/1763591068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c153f10 0x7f529c15aaa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5294009d90 tx=0x7f529402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd6f6102c1114ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f6b6d000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000c7a0 tx=0x7f6b6000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104093890 0 0) 0x7f4370003520 con 0x7f43900ce260 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 msgr2=0x7f43900cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43900cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 >> v1:172.21.15.69:6789/0 conn(0x7f43900d1f60 legacy=0x7f43900b4cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900b2400 con 0x7f43900ce260 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.805+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fde0c004190 con 0x7fde1c15d4d0 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7fde19600640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 0x7fddf406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde0c03e070 con 0x7fde1c15d4d0 2024-01-16T14:59:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f5299600640 1 -- 172.21.15.43:0/1763591068 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529402cbd0 con 0x7f529c153f10 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4380001650 con 0x7f43900ce260 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 >> v1:172.21.15.43:6789/0 conn(0x7f43900ce260 legacy=0x7f43900d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4370003f60 0x7f43700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4370004d00 0x7f4370005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900b91b0 con 0x7f4370003f60 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437000baf0 con 0x7f4370004d00 2024-01-16T14:59:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437000bcb0 con 0x7f439015d4d0 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2538046b70 con 0x7f254806eb00 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f5299600640 1 -- 172.21.15.43:0/1763591068 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529402cd30 con 0x7f529c153f10 2024-01-16T14:59:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.807+0000 7f5299600640 1 -- 172.21.15.43:0/1763591068 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529402cea0 con 0x7f529c153f10 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60003b70 con 0x7f6b681619a0 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60003cd0 con 0x7f6b681619a0 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000aa90 con 0x7f6b681619a0 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b681ed140 con 0x7f6b681619a0 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7fde21400640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 0x7fddf406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b680b18e0 con 0x7f6b681619a0 2024-01-16T14:59:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f4396200640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4370004d00 0x7f4370005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.808+0000 7f4395800640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f52a1800640 1 -- 172.21.15.43:0/1763591068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c153f10 msgr2=0x7f529c15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f52a1800640 1 --2- 172.21.15.43:0/1763591068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c153f10 0x7f529c15aaa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5294009d90 tx=0x7f529402be20 comp rx=0 tx=0).stop 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4396200640 1 -- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 msgr2=0x7f43700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4396200640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4396200640 1 -- 172.21.15.43:0/2312682119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4370003f60 msgr2=0x7f43700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f2546000640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 0x7f252006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f52a1800640 1 -- 172.21.15.43:0/1763591068 shutdown_connections 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f52a1800640 1 --2- 172.21.15.43:0/1763591068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c153f10 0x7f529c15aaa0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f52a1800640 1 -- 172.21.15.43:0/1763591068 >> 172.21.15.43:0/1763591068 conn(0x7f529c0b0d80 msgr2=0x7f529c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4394e00640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4370003f60 0x7f43700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25380b4ce0 con 0x7f254806eb00 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f254d400640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 0x7f252006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.810+0000 7f52a1800640 1 -- 172.21.15.43:0/1763591068 shutdown_connections 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.810+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b38005250 con 0x7f6b681619a0 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.810+0000 7f52a1800640 1 -- 172.21.15.43:0/1763591068 wait complete. 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4396200640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4370003f60 0x7f43700047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4396200640 1 -- 172.21.15.43:0/2312682119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900b2400 con 0x7f4370004d00 2024-01-16T14:59:36.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.809+0000 7f4395800640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439015d4d0 0x7f43700039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.811+0000 7fde21400640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 0x7fddf406d7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde04007fd0 tx=0x7fde04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.811+0000 7f4396200640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4370004d00 0x7f4370005180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43900b6150 tx=0x7f43700057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e3934fbc2da101 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.812+0000 7f254d400640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 0x7f252006d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2548152190 tx=0x7f2530007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.812+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6b6000abf0 con 0x7f6b681619a0 2024-01-16T14:59:36.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.813+0000 7f52a1800640 1 Processor -- start 2024-01-16T14:59:36.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.813+0000 7f52a1800640 1 -- start start 2024-01-16T14:59:36.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f6b5fe00640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 0x7f6b4006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b60089900 con 0x7f6b681619a0 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f6b67e00640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 0x7f6b4006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f529c06ddb0 0x7f529c0c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529c070ae0 0x7f529c070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0714b0 con 0x7f529c15b050 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0c8400 con 0x7f529c06ddb0 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.814+0000 7f52a1800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0c86e0 con 0x7f529c070ae0 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34414/0 (socket says 172.21.15.43:34414) 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 -- 172.21.15.43:0/3961600984 learned_addr learned my addr 172.21.15.43:0/3961600984 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529aa00640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f529c06ddb0 0x7f529c0c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f6b67e00640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 0x7f6b4006d7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000a730 tx=0x7f6b50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529b400640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529c070ae0 0x7f529c070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 -- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529c070ae0 msgr2=0x7f529c070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529c070ae0 0x7f529c070f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 -- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f529c06ddb0 msgr2=0x7f529c0c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f529c06ddb0 0x7f529c0c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.815+0000 7f529a000640 1 -- 172.21.15.43:0/3961600984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294009a70 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.816+0000 7f529a000640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f528400b420 tx=0x7f528400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.816+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284011830 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.816+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c1592d0 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.816+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284011990 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.816+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284011d40 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.817+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c159890 con 0x7f529c15b050 2024-01-16T14:59:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.818+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c067960 con 0x7f529c15b050 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.819+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f528401b460 con 0x7f529c15b050 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.821+0000 7f5292a00640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 0x7f527406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.821+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52840896b0 con 0x7f529c15b050 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.821+0000 7f529aa00640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 0x7f527406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.822+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde0c076ce0 con 0x7fde1c15d4d0 2024-01-16T14:59:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.822+0000 7f529aa00640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 0x7f527406d800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5294036000 tx=0x7f5294030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.823+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378032600 con 0x7f4370004d00 2024-01-16T14:59:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.824+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f253807fc50 con 0x7f254806eb00 2024-01-16T14:59:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.825+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5284054500 con 0x7f529c15b050 2024-01-16T14:59:36.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.837+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437802e050 con 0x7f4370004d00 2024-01-16T14:59:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.837+0000 7f438fe00640 1 -- 172.21.15.43:0/2312682119 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378032600 con 0x7f4370004d00 2024-01-16T14:59:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.837+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b600547e0 con 0x7f6b681619a0 2024-01-16T14:59:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.838+0000 7f4398200640 1 -- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4370004d00 msgr2=0x7f4370005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.838+0000 7f4398200640 1 --2- 172.21.15.43:0/2312682119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4370004d00 0x7f4370005180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43900b6150 tx=0x7f43700057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.839+0000 7f4396200640 1 -- 172.21.15.43:0/2312682119 reap_dead start 2024-01-16T14:59:36.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.839+0000 7f4398200640 1 -- 172.21.15.43:0/2312682119 shutdown_connections 2024-01-16T14:59:36.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.839+0000 7f4398200640 1 -- 172.21.15.43:0/2312682119 >> 172.21.15.43:0/2312682119 conn(0x7f4390071f90 msgr2=0x7f43900723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.839+0000 7f4398200640 1 -- 172.21.15.43:0/2312682119 shutdown_connections 2024-01-16T14:59:36.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.840+0000 7f4398200640 1 -- 172.21.15.43:0/2312682119 wait complete. 2024-01-16T14:59:36.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.843+0000 7f4398200640 1 Processor -- start 2024-01-16T14:59:36.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.843+0000 7f4398200640 1 -- start start 2024-01-16T14:59:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.846+0000 7f4398200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.846+0000 7f4398200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43900c73f0 0x7f43900cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.846+0000 7f4398200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43901fa760 0x7f43901fac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.846+0000 7f4395800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4398200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439015e010 con 0x7f43900c7010 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4398200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900cc320 con 0x7f43900c73f0 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4398200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439015e1f0 con 0x7f43901fa760 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4395800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34420/0 (socket says 172.21.15.43:34420) 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4395800640 1 -- 172.21.15.43:0/2470629542 learned_addr learned my addr 172.21.15.43:0/2470629542 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.847+0000 7f4394e00640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43900c73f0 0x7f43900cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.848+0000 7f4395800640 1 -- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43901fa760 msgr2=0x7f43901fac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.848+0000 7f4395800640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43901fa760 0x7f43901fac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.848+0000 7f4395800640 1 -- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43900c73f0 msgr2=0x7f43900cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.848+0000 7f4395800640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43900c73f0 0x7f43900cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.848+0000 7f4395800640 1 -- 172.21.15.43:0/2470629542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439015e4d0 con 0x7f43900c7010 2024-01-16T14:59:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.849+0000 7f4395800640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f438800f460 tx=0x7f438800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.851+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438800e040 con 0x7f43900c7010 2024-01-16T14:59:36.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.851+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438800f040 con 0x7f43900c7010 2024-01-16T14:59:36.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.851+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388003d90 con 0x7f43900c7010 2024-01-16T14:59:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.851+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439015e7b0 con 0x7f43900c7010 2024-01-16T14:59:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.852+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43901fb150 con 0x7f43900c7010 2024-01-16T14:59:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.854+0000 7f99da800640 1 Processor -- start 2024-01-16T14:59:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.856+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4388015640 con 0x7f43900c7010 2024-01-16T14:59:36.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.858+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4360005250 con 0x7f43900c7010 2024-01-16T14:59:36.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.858+0000 7f99da800640 1 -- start start 2024-01-16T14:59:36.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.858+0000 7f438d600640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 0x7f436806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.858+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f438808a700 con 0x7f43900c7010 2024-01-16T14:59:36.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.861+0000 7f4394e00640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 0x7f436806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.861+0000 7f99da800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 0x7f99d406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.862+0000 7f99da800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 0x7f99d4070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.862+0000 7f99da800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40b21a0 con 0x7f99d4071040 2024-01-16T14:59:36.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.863+0000 7f99da800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d415ddf0 con 0x7f99d4150520 2024-01-16T14:59:36.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.863+0000 7f99da800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40cb7a0 con 0x7f99d406c650 2024-01-16T14:59:36.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.863+0000 7f4394e00640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 0x7f436806d7d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f43900cc8b0 tx=0x7f4380009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.863+0000 7f99d8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 0x7f99d4070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.863+0000 7f99d9e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f99d4071040 0x7f99d415cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53046/0 (socket says 172.21.15.43:53046) 2024-01-16T14:59:36.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.864+0000 7f99d9e00640 1 -- 172.21.15.43:0/3747966564 learned_addr learned my addr 172.21.15.43:0/3747966564 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:36.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.864+0000 7f99d9400640 1 --2- 172.21.15.43:0/3747966564 >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 0x7f99d406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.865+0000 7f99d8a00640 1 -- 172.21.15.43:0/3747966564 >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 msgr2=0x7f99d406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.866+0000 7f99d8a00640 1 --2- 172.21.15.43:0/3747966564 >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 0x7f99d406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.866+0000 7f99d8a00640 1 -- 172.21.15.43:0/3747966564 >> v1:172.21.15.43:6789/0 conn(0x7f99d4071040 legacy=0x7f99d415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.866+0000 7f99d8a00640 1 -- 172.21.15.43:0/3747966564 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4150900 con 0x7f99d406c650 2024-01-16T14:59:36.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.866+0000 7f99d9400640 1 --2- 172.21.15.43:0/3747966564 >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 0x7f99d406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:36.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.868+0000 7f99d8a00640 1 --2- 172.21.15.43:0/3747966564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 0x7f99d4070b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f99c40097a0 tx=0x7f99c402dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e3b55beda2e7f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.868+0000 7f99d3e00640 1 -- 172.21.15.43:0/3747966564 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c402f8a0 con 0x7f99d406c650 2024-01-16T14:59:36.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.868+0000 7f99d3e00640 1 -- 172.21.15.43:0/3747966564 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c402fa00 con 0x7f99d406c650 2024-01-16T14:59:36.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.868+0000 7f99d3e00640 1 -- 172.21.15.43:0/3747966564 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c402fdd0 con 0x7f99d406c650 2024-01-16T14:59:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.869+0000 7f99da800640 1 -- 172.21.15.43:0/3747966564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 msgr2=0x7f99d4070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.869+0000 7f99da800640 1 --2- 172.21.15.43:0/3747966564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 0x7f99d4070b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f99c40097a0 tx=0x7f99c402dae0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 -- 172.21.15.43:0/3747966564 shutdown_connections 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 --2- 172.21.15.43:0/3747966564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d406c650 0x7f99d4070b00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 --2- 172.21.15.43:0/3747966564 >> v2:172.21.15.43:3300/0 conn(0x7f99d4150520 0x7f99d406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 -- 172.21.15.43:0/3747966564 >> 172.21.15.43:0/3747966564 conn(0x7f99d40b0840 msgr2=0x7f99d40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 -- 172.21.15.43:0/3747966564 shutdown_connections 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f99da800640 1 -- 172.21.15.43:0/3747966564 wait complete. 2024-01-16T14:59:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.870+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4388055670 con 0x7f43900c7010 2024-01-16T14:59:36.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.873+0000 7f99da800640 1 Processor -- start 2024-01-16T14:59:36.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.873+0000 7f99da800640 1 -- start start 2024-01-16T14:59:36.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.875+0000 7f99da800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.876+0000 7f99da800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.876+0000 7f99da800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.876+0000 7f99d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.876+0000 7f99d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33846/0 (socket says 172.21.15.43:33846) 2024-01-16T14:59:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.877+0000 7f99d8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.877+0000 7f99d8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:55184/0 (socket says 172.21.15.43:55184) 2024-01-16T14:59:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.877+0000 7f99d9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.878+0000 7f99d9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34430/0 (socket says 172.21.15.43:34430) 2024-01-16T14:59:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.878+0000 7f99d9400640 1 -- 172.21.15.43:0/263257970 learned_addr learned my addr 172.21.15.43:0/263257970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.878+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40ac3f0 con 0x7f99d414f780 2024-01-16T14:59:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.878+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40a94c0 con 0x7f99d4071040 2024-01-16T14:59:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.878+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40a8940 con 0x7f99d4150520 2024-01-16T14:59:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.879+0000 7f99d9e00640 1 -- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 msgr2=0x7f99d40cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.879+0000 7f99d9e00640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.879+0000 7f99d9e00640 1 -- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 msgr2=0x7f99d414e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.879+0000 7f99d9e00640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.879+0000 7f99d9e00640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4009400 con 0x7f99d414f780 2024-01-16T14:59:36.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.880+0000 7f99d9e00640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00caa0 tx=0x7f99cc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.880+0000 7f99d8a00640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:36.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.880+0000 7f99d9400640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:36.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.881+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00ac00 con 0x7f99d414f780 2024-01-16T14:59:36.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.881+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d40a8c20 con 0x7f99d414f780 2024-01-16T14:59:36.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.882+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00ad60 con 0x7f99d414f780 2024-01-16T14:59:36.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.882+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d40a91e0 con 0x7f99d414f780 2024-01-16T14:59:36.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.882+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00ac00 con 0x7f99d414f780 2024-01-16T14:59:36.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.884+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a4005250 con 0x7f99d414f780 2024-01-16T14:59:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.897+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f99cc007070 con 0x7f99d414f780 2024-01-16T14:59:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.898+0000 7f99d1600640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 0x7f99ac06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.899+0000 7f99d9400640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 0x7f99ac06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.900+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99cc058d90 con 0x7f99d414f780 2024-01-16T14:59:36.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.900+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99cc055160 con 0x7f99d414f780 2024-01-16T14:59:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:36.905+0000 7f99d9400640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 0x7f99ac06d7e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006e80 tx=0x7f99bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.158+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f43340d0ea0 con 0x7f43341603e0 2024-01-16T14:59:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.159+0000 7f432aa00640 1 -- 172.21.15.43:0/2192103280 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f432c059300 con 0x7f43341603e0 2024-01-16T14:59:37.143 INFO:teuthology.orchestra.run.smithi043.stdout:60129542151 2024-01-16T14:59:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.163+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 msgr2=0x7f430c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.163+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 0x7f430c06d880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4324006b10 tx=0x7f4324008040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.163+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 msgr2=0x7f43340b7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.163+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f432c001350 tx=0x7f432c007430 comp rx=0 tx=0).stop 2024-01-16T14:59:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 shutdown_connections 2024-01-16T14:59:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f430c06b3c0 0x7f430c06d880 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43341603e0 0x7f43340b7580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43340c89c0 0x7f43340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 --2- 172.21.15.43:0/2192103280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4314005180 0x7f43340ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.164+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 >> 172.21.15.43:0/2192103280 conn(0x7f4334071f90 msgr2=0x7f4334159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.165+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 shutdown_connections 2024-01-16T14:59:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.165+0000 7f4339c00640 1 -- 172.21.15.43:0/2192103280 wait complete. 2024-01-16T14:59:37.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.3 2024-01-16T14:59:37.172 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.210+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7e1015db70 con 0x7f7e100cc4a0 2024-01-16T14:59:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.211+0000 7f7e0d600640 1 -- 172.21.15.43:0/2967697322 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e08019020 con 0x7f7e100cc4a0 2024-01-16T14:59:37.192 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.214+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 msgr2=0x7f7de806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.214+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 0x7f7de806d850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e100ce3a0 tx=0x7f7df8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.214+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 msgr2=0x7f7e100cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.214+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 0x7f7e100cc920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007d70 tx=0x7f7e0800e4b0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 shutdown_connections 2024-01-16T14:59:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b390 0x7f7de806d850 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100cc4a0 0x7f7e100cc920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e100b4c80 0x7f7e101596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 --2- 172.21.15.43:0/2967697322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e100b1d20 0x7f7e100c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 >> 172.21.15.43:0/2967697322 conn(0x7f7e100721a0 msgr2=0x7f7e1015ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 shutdown_connections 2024-01-16T14:59:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.215+0000 7f7e18600640 1 -- 172.21.15.43:0/2967697322 wait complete. 2024-01-16T14:59:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.225+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6b38005500 con 0x7f6b681619a0 2024-01-16T14:59:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.225+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2593093896 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b60057e00 con 0x7f6b681619a0 2024-01-16T14:59:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:60129542150 2024-01-16T14:59:37.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 msgr2=0x7f6b4006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 0x7f6b4006d7d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000a730 tx=0x7f6b50008040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 msgr2=0x7f6b681eba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000c7a0 tx=0x7f6b6000cc60 comp rx=0 tx=0).stop 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 shutdown_connections 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6b4006b310 0x7f6b4006d7d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b681619a0 0x7f6b681eba20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b680ae750 0x7f6b681e9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 --2- 172.21.15.43:0/2593093896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b680acc20 0x7f6b680b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 >> 172.21.15.43:0/2593093896 conn(0x7f6b680709f0 msgr2=0x7f6b68160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.228+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 shutdown_connections 2024-01-16T14:59:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.229+0000 7f6b6f000640 1 -- 172.21.15.43:0/2593093896 wait complete. 2024-01-16T14:59:37.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.1 2024-01-16T14:59:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.241+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fde1c074770 con 0x7fde1c15d4d0 2024-01-16T14:59:37.222 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.241+0000 7fde19600640 1 -- 172.21.15.43:0/3093400585 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde0c034090 con 0x7fde1c15d4d0 2024-01-16T14:59:37.223 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 msgr2=0x7fddf406d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 0x7fddf406d7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde04007fd0 tx=0x7fde04009040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 msgr2=0x7fde1c1f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 0x7fde1c1f6290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde0c033000 tx=0x7fde0c004730 comp rx=0 tx=0).stop 2024-01-16T14:59:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 shutdown_connections 2024-01-16T14:59:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddf406b320 0x7fddf406d7e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f529c159ab0 con 0x7f529c15b050 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c15f7e0 0x7fde1c1f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde1c15d4d0 0x7fde1c1f6290 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 --2- 172.21.15.43:0/3093400585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0ce6a0 0x7fde1c1f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 >> 172.21.15.43:0/3093400585 conn(0x7fde1c071f90 msgr2=0x7fde1c158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.245+0000 7f5292a00640 1 -- 172.21.15.43:0/3961600984 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5284057b20 con 0x7f529c15b050 2024-01-16T14:59:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 shutdown_connections 2024-01-16T14:59:37.228 INFO:teuthology.orchestra.run.smithi043.stdout:60129542150 2024-01-16T14:59:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.244+0000 7fde23e00640 1 -- 172.21.15.43:0/3093400585 wait complete. 2024-01-16T14:59:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 msgr2=0x7f527406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 0x7f527406d800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5294036000 tx=0x7f5294030000 comp rx=0 tx=0).stop 2024-01-16T14:59:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 msgr2=0x7f529c144af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f528400b420 tx=0x7f528400b8e0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f25480ce3d0 con 0x7f254806eb00 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 shutdown_connections 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f527406b340 0x7f527406d800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f529c070ae0 0x7f529c070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f529c15b050 0x7f529c144af0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f528400b420 tx=0x7f528400b8e0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 --2- 172.21.15.43:0/3961600984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f529c06ddb0 0x7f529c0c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 >> 172.21.15.43:0/3961600984 conn(0x7f529c0b0d80 msgr2=0x7f529c06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 shutdown_connections 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.248+0000 7f52a1800640 1 -- 172.21.15.43:0/3961600984 wait complete. 2024-01-16T14:59:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.249+0000 7f2546000640 1 -- 172.21.15.43:0/2434694966 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f253804b020 con 0x7f254806eb00 2024-01-16T14:59:37.231 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:37.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.5 2024-01-16T14:59:37.231 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 msgr2=0x7f252006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 0x7f252006d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2548152190 tx=0x7f2530007e30 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 msgr2=0x7f254813bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f253800dd10 tx=0x7f25380046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 shutdown_connections 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252006b310 0x7f252006d7d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f254815ed20 0x7f25480d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25480cfa00 0x7f2548157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 --2- 172.21.15.43:0/2434694966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f254806eb00 0x7f254813bcb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 >> 172.21.15.43:0/2434694966 conn(0x7f25480663d0 msgr2=0x7f25480667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 shutdown_connections 2024-01-16T14:59:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.252+0000 7f2550800640 1 -- 172.21.15.43:0/2434694966 wait complete. 2024-01-16T14:59:37.244 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.2 2024-01-16T14:59:37.244 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.0 2024-01-16T14:59:37.253 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.6 2024-01-16T14:59:37.256 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.294+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4360005500 con 0x7f43900c7010 2024-01-16T14:59:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.294+0000 7f438d600640 1 -- 172.21.15.43:0/2470629542 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4388058c90 con 0x7f43900c7010 2024-01-16T14:59:37.275 INFO:teuthology.orchestra.run.smithi043.stdout:60129542150 2024-01-16T14:59:37.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 msgr2=0x7f436806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 0x7f436806d7d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f43900cc8b0 tx=0x7f4380009040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 msgr2=0x7f43900cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f438800f460 tx=0x7f438800f920 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 shutdown_connections 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f436806b310 0x7f436806d7d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43900c73f0 0x7f43900cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43901fa760 0x7f43901fac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 --2- 172.21.15.43:0/2470629542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43900c7010 0x7f43900cb8a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 >> 172.21.15.43:0/2470629542 conn(0x7f4390071f90 msgr2=0x7f43900b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.296+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 shutdown_connections 2024-01-16T14:59:37.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.297+0000 7f4398200640 1 -- 172.21.15.43:0/2470629542 wait complete. 2024-01-16T14:59:37.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.308+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f99a40057d0 con 0x7f99d414f780 2024-01-16T14:59:37.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.308+0000 7f99d1600640 1 -- 172.21.15.43:0/263257970 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f99cc017090 con 0x7f99d414f780 2024-01-16T14:59:37.289 INFO:teuthology.orchestra.run.smithi043.stdout:60129542149 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 msgr2=0x7f99ac06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 0x7f99ac06d7e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006e80 tx=0x7f99bc008040 comp rx=0 tx=0).stop 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 msgr2=0x7f99d40cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00caa0 tx=0x7f99cc00cf60 comp rx=0 tx=0).stop 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 shutdown_connections 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99ac06b320 0x7f99ac06d7e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d414f780 0x7f99d40cf120 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4150520 0x7f99d40cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 --2- 172.21.15.43:0/263257970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4071040 0x7f99d414e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 >> 172.21.15.43:0/263257970 conn(0x7f99d40b0840 msgr2=0x7f99d40b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 shutdown_connections 2024-01-16T14:59:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.311+0000 7f99da800640 1 -- 172.21.15.43:0/263257970 wait complete. 2024-01-16T14:59:37.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.7 2024-01-16T14:59:37.295 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.4 2024-01-16T14:59:37.310 DEBUG:teuthology.parallel:result is None 2024-01-16T14:59:37.310 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T14:59:37.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:37.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.589+0000 7f2558e00640 1 Processor -- start 2024-01-16T14:59:37.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.590+0000 7f2558e00640 1 -- start start 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.591+0000 7f2558e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f255415d5b0 0x7f25540d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.591+0000 7f2558e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.591+0000 7f2558e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25540cf0c0 con 0x7f25540cf7a0 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.591+0000 7f2558e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25540b01b0 con 0x7f255415d5b0 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.591+0000 7f2558e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255415d990 con 0x7f25540d14a0 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f255415d5b0 0x7f25540d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f255415d5b0 0x7f25540d3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34432/0 (socket says 172.21.15.43:34432) 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 -- 172.21.15.43:0/567243875 learned_addr learned my addr 172.21.15.43:0/567243875 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2552a00640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 -- 172.21.15.43:0/567243875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 msgr2=0x7f25540d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 -- 172.21.15.43:0/567243875 >> v1:172.21.15.43:6789/0 conn(0x7f25540cf7a0 legacy=0x7f25540cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2553400640 1 -- 172.21.15.43:0/567243875 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540d1e40 con 0x7f255415d5b0 2024-01-16T14:59:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.592+0000 7f2552000640 1 -- 172.21.15.43:0/567243875 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417533792 0 0) 0x7f25540cf0c0 con 0x7f25540cf7a0 2024-01-16T14:59:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.594+0000 7f2553400640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540d3990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e50 tx=0x7f254402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd08d068c7351a82 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.594+0000 7f2552000640 1 -- 172.21.15.43:0/567243875 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254402cb00 con 0x7f255415d5b0 2024-01-16T14:59:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.594+0000 7f2552000640 1 -- 172.21.15.43:0/567243875 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254402cc60 con 0x7f255415d5b0 2024-01-16T14:59:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.594+0000 7f2552000640 1 -- 172.21.15.43:0/567243875 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440343f0 con 0x7f255415d5b0 2024-01-16T14:59:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 -- 172.21.15.43:0/567243875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 msgr2=0x7f25540d3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540d3990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2544009e50 tx=0x7f254402bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 -- 172.21.15.43:0/567243875 shutdown_connections 2024-01-16T14:59:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 --2- 172.21.15.43:0/567243875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540d3990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 -- 172.21.15.43:0/567243875 >> 172.21.15.43:0/567243875 conn(0x7f255409c3b0 msgr2=0x7f25540d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 -- 172.21.15.43:0/567243875 shutdown_connections 2024-01-16T14:59:37.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.595+0000 7f2558e00640 1 -- 172.21.15.43:0/567243875 wait complete. 2024-01-16T14:59:37.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.597+0000 7f2558e00640 1 Processor -- start 2024-01-16T14:59:37.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.597+0000 7f2558e00640 1 -- start start 2024-01-16T14:59:37.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25540cf7a0 0x7f2554157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255415ae00 con 0x7f255415d5b0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25541f2770 con 0x7f25540cf7a0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2558e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25541598d0 con 0x7f25540d14a0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2553e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2553e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34436/0 (socket says 172.21.15.43:34436) 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.598+0000 7f2553e00640 1 -- 172.21.15.43:0/760337299 learned_addr learned my addr 172.21.15.43:0/760337299 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2552a00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553400640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25540cf7a0 0x7f2554157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 -- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 msgr2=0x7f25540c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 -- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25540cf7a0 msgr2=0x7f2554157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25540cf7a0 0x7f2554157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 -- 172.21.15.43:0/760337299 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009b30 con 0x7f255415d5b0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2553e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f254c00c390 tx=0x7f254c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0035f0 con 0x7f255415d5b0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c003750 con 0x7f255415d5b0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c003b00 con 0x7f255415d5b0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.599+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2554159bb0 con 0x7f255415d5b0 2024-01-16T14:59:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.600+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255415a170 con 0x7f255415d5b0 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.601+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255404e840 con 0x7f255415d5b0 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.601+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f254c003c60 con 0x7f255415d5b0 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.602+0000 7f254b400640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 0x7f252c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.603+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f254c0891a0 con 0x7f255415d5b0 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.603+0000 7f2553400640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 0x7f252c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.603+0000 7f2553400640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 0x7f252c06d780 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2544035000 tx=0x7f2544030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.607+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f254c054010 con 0x7f255415d5b0 2024-01-16T14:59:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.790+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25540afe40 con 0x7f252c06b2c0 2024-01-16T14:59:37.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.802+0000 7f254b400640 1 -- 172.21.15.43:0/760337299 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264404 (secure 0 0 0) 0x7f2544044090 con 0x7f252c06b2c0 2024-01-16T14:59:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:37.788 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 msgr2=0x7f252c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 0x7f252c06d780 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2544035000 tx=0x7f2544030000 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 msgr2=0x7f25540cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f254c00c390 tx=0x7f254c00c850 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 shutdown_connections 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f252c06b2c0 0x7f252c06d780 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f255415d5b0 0x7f25540cb730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25540d14a0 0x7f25540c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 --2- 172.21.15.43:0/760337299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25540cf7a0 0x7f2554157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 >> 172.21.15.43:0/760337299 conn(0x7f255409c3b0 msgr2=0x7f255415db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 shutdown_connections 2024-01-16T14:59:37.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:37.808+0000 7f2558e00640 1 -- 172.21.15.43:0/760337299 wait complete. 2024-01-16T14:59:37.808 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T14:59:36.789779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240372,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734348,"statfs":{"total":773094113280,"available":772847972352,"internally_reserved":0,"allocated":3055616,"data_stored":1815972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.107296"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908219+0000","last_change":"2024-01-16T14:59:27.733425+0000","last_active":"2024-01-16T14:59:28.908219+0000","last_peered":"2024-01-16T14:59:28.908219+0000","last_clean":"2024-01-16T14:59:28.908219+0000","last_became_active":"2024-01-16T14:59:27.733108+0000","last_became_peered":"2024-01-16T14:59:27.733108+0000","last_unstale":"2024-01-16T14:59:28.908219+0000","last_undegraded":"2024-01-16T14:59:28.908219+0000","last_fullsized":"2024-01-16T14:59:28.908219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:28:12.476929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908249+0000","last_change":"2024-01-16T14:59:27.715575+0000","last_active":"2024-01-16T14:59:28.908249+0000","last_peered":"2024-01-16T14:59:28.908249+0000","last_clean":"2024-01-16T14:59:28.908249+0000","last_became_active":"2024-01-16T14:59:27.714841+0000","last_became_peered":"2024-01-16T14:59:27.714841+0000","last_unstale":"2024-01-16T14:59:28.908249+0000","last_undegraded":"2024-01-16T14:59:28.908249+0000","last_fullsized":"2024-01-16T14:59:28.908249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908551+0000","last_change":"2024-01-16T14:59:27.718807+0000","last_active":"2024-01-16T14:59:28.908551+0000","last_peered":"2024-01-16T14:59:28.908551+0000","last_clean":"2024-01-16T14:59:28.908551+0000","last_became_active":"2024-01-16T14:59:27.717301+0000","last_became_peered":"2024-01-16T14:59:27.717301+0000","last_unstale":"2024-01-16T14:59:28.908551+0000","last_undegraded":"2024-01-16T14:59:28.908551+0000","last_fullsized":"2024-01-16T14:59:28.908551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906129+0000","last_change":"2024-01-16T14:59:27.720914+0000","last_active":"2024-01-16T14:59:28.906129+0000","last_peered":"2024-01-16T14:59:28.906129+0000","last_clean":"2024-01-16T14:59:28.906129+0000","last_became_active":"2024-01-16T14:59:27.720561+0000","last_became_peered":"2024-01-16T14:59:27.720561+0000","last_unstale":"2024-01-16T14:59:28.906129+0000","last_undegraded":"2024-01-16T14:59:28.906129+0000","last_fullsized":"2024-01-16T14:59:28.906129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:16.656352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906008+0000","last_change":"2024-01-16T14:59:27.721194+0000","last_active":"2024-01-16T14:59:28.906008+0000","last_peered":"2024-01-16T14:59:28.906008+0000","last_clean":"2024-01-16T14:59:28.906008+0000","last_became_active":"2024-01-16T14:59:27.719980+0000","last_became_peered":"2024-01-16T14:59:27.719980+0000","last_unstale":"2024-01-16T14:59:28.906008+0000","last_undegraded":"2024-01-16T14:59:28.906008+0000","last_fullsized":"2024-01-16T14:59:28.906008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:54.580479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905723+0000","last_change":"2024-01-16T14:59:27.732776+0000","last_active":"2024-01-16T14:59:28.905723+0000","last_peered":"2024-01-16T14:59:28.905723+0000","last_clean":"2024-01-16T14:59:28.905723+0000","last_became_active":"2024-01-16T14:59:27.732373+0000","last_became_peered":"2024-01-16T14:59:27.732373+0000","last_unstale":"2024-01-16T14:59:28.905723+0000","last_undegraded":"2024-01-16T14:59:28.905723+0000","last_fullsized":"2024-01-16T14:59:28.905723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904627+0000","last_change":"2024-01-16T14:59:27.722159+0000","last_active":"2024-01-16T14:59:28.904627+0000","last_peered":"2024-01-16T14:59:28.904627+0000","last_clean":"2024-01-16T14:59:28.904627+0000","last_became_active":"2024-01-16T14:59:27.721928+0000","last_became_peered":"2024-01-16T14:59:27.721928+0000","last_unstale":"2024-01-16T14:59:28.904627+0000","last_undegraded":"2024-01-16T14:59:28.904627+0000","last_fullsized":"2024-01-16T14:59:28.904627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:17.023676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904587+0000","last_change":"2024-01-16T14:59:27.721172+0000","last_active":"2024-01-16T14:59:28.904587+0000","last_peered":"2024-01-16T14:59:28.904587+0000","last_clean":"2024-01-16T14:59:28.904587+0000","last_became_active":"2024-01-16T14:59:27.720829+0000","last_became_peered":"2024-01-16T14:59:27.720829+0000","last_unstale":"2024-01-16T14:59:28.904587+0000","last_undegraded":"2024-01-16T14:59:28.904587+0000","last_fullsized":"2024-01-16T14:59:28.904587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:39.594220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908493+0000","last_change":"2024-01-16T14:59:27.721916+0000","last_active":"2024-01-16T14:59:28.908493+0000","last_peered":"2024-01-16T14:59:28.908493+0000","last_clean":"2024-01-16T14:59:28.908493+0000","last_became_active":"2024-01-16T14:59:27.721604+0000","last_became_peered":"2024-01-16T14:59:27.721604+0000","last_unstale":"2024-01-16T14:59:28.908493+0000","last_undegraded":"2024-01-16T14:59:28.908493+0000","last_fullsized":"2024-01-16T14:59:28.908493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:05.858739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711127+0000","last_change":"2024-01-16T14:59:27.729823+0000","last_active":"2024-01-16T14:59:28.711127+0000","last_peered":"2024-01-16T14:59:28.711127+0000","last_clean":"2024-01-16T14:59:28.711127+0000","last_became_active":"2024-01-16T14:59:27.724755+0000","last_became_peered":"2024-01-16T14:59:27.724755+0000","last_unstale":"2024-01-16T14:59:28.711127+0000","last_undegraded":"2024-01-16T14:59:28.711127+0000","last_fullsized":"2024-01-16T14:59:28.711127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710385+0000","last_change":"2024-01-16T14:59:27.725448+0000","last_active":"2024-01-16T14:59:28.710385+0000","last_peered":"2024-01-16T14:59:28.710385+0000","last_clean":"2024-01-16T14:59:28.710385+0000","last_became_active":"2024-01-16T14:59:27.724294+0000","last_became_peered":"2024-01-16T14:59:27.724294+0000","last_unstale":"2024-01-16T14:59:28.710385+0000","last_undegraded":"2024-01-16T14:59:28.710385+0000","last_fullsized":"2024-01-16T14:59:28.710385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709647+0000","last_change":"2024-01-16T14:59:27.725056+0000","last_active":"2024-01-16T14:59:28.709647+0000","last_peered":"2024-01-16T14:59:28.709647+0000","last_clean":"2024-01-16T14:59:28.709647+0000","last_became_active":"2024-01-16T14:59:27.724771+0000","last_became_peered":"2024-01-16T14:59:27.724771+0000","last_unstale":"2024-01-16T14:59:28.709647+0000","last_undegraded":"2024-01-16T14:59:28.709647+0000","last_fullsized":"2024-01-16T14:59:28.709647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905952+0000","last_change":"2024-01-16T14:59:27.726131+0000","last_active":"2024-01-16T14:59:28.905952+0000","last_peered":"2024-01-16T14:59:28.905952+0000","last_clean":"2024-01-16T14:59:28.905952+0000","last_became_active":"2024-01-16T14:59:27.725830+0000","last_became_peered":"2024-01-16T14:59:27.725830+0000","last_unstale":"2024-01-16T14:59:28.905952+0000","last_undegraded":"2024-01-16T14:59:28.905952+0000","last_fullsized":"2024-01-16T14:59:28.905952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:45.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906705+0000","last_change":"2024-01-16T14:59:27.713525+0000","last_active":"2024-01-16T14:59:28.906705+0000","last_peered":"2024-01-16T14:59:28.906705+0000","last_clean":"2024-01-16T14:59:28.906705+0000","last_became_active":"2024-01-16T14:59:27.712762+0000","last_became_peered":"2024-01-16T14:59:27.712762+0000","last_unstale":"2024-01-16T14:59:28.906705+0000","last_undegraded":"2024-01-16T14:59:28.906705+0000","last_fullsized":"2024-01-16T14:59:28.906705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:41.373234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906179+0000","last_change":"2024-01-16T14:59:27.700808+0000","last_active":"2024-01-16T14:59:28.906179+0000","last_peered":"2024-01-16T14:59:28.906179+0000","last_clean":"2024-01-16T14:59:28.906179+0000","last_became_active":"2024-01-16T14:59:27.700477+0000","last_became_peered":"2024-01-16T14:59:27.700477+0000","last_unstale":"2024-01-16T14:59:28.906179+0000","last_undegraded":"2024-01-16T14:59:28.906179+0000","last_fullsized":"2024-01-16T14:59:28.906179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905239+0000","last_change":"2024-01-16T14:59:27.720864+0000","last_active":"2024-01-16T14:59:28.905239+0000","last_peered":"2024-01-16T14:59:28.905239+0000","last_clean":"2024-01-16T14:59:28.905239+0000","last_became_active":"2024-01-16T14:59:27.719599+0000","last_became_peered":"2024-01-16T14:59:27.719599+0000","last_unstale":"2024-01-16T14:59:28.905239+0000","last_undegraded":"2024-01-16T14:59:28.905239+0000","last_fullsized":"2024-01-16T14:59:28.905239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907987+0000","last_change":"2024-01-16T14:59:27.717946+0000","last_active":"2024-01-16T14:59:28.907987+0000","last_peered":"2024-01-16T14:59:28.907987+0000","last_clean":"2024-01-16T14:59:28.907987+0000","last_became_active":"2024-01-16T14:59:27.717711+0000","last_became_peered":"2024-01-16T14:59:27.717711+0000","last_unstale":"2024-01-16T14:59:28.907987+0000","last_undegraded":"2024-01-16T14:59:28.907987+0000","last_fullsized":"2024-01-16T14:59:28.907987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:03.931690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703527+0000","last_change":"2024-01-16T14:59:27.726121+0000","last_active":"2024-01-16T14:59:28.703527+0000","last_peered":"2024-01-16T14:59:28.703527+0000","last_clean":"2024-01-16T14:59:28.703527+0000","last_became_active":"2024-01-16T14:59:27.725643+0000","last_became_peered":"2024-01-16T14:59:27.725643+0000","last_unstale":"2024-01-16T14:59:28.703527+0000","last_undegraded":"2024-01-16T14:59:28.703527+0000","last_fullsized":"2024-01-16T14:59:28.703527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907957+0000","last_change":"2024-01-16T14:59:27.720619+0000","last_active":"2024-01-16T14:59:28.907957+0000","last_peered":"2024-01-16T14:59:28.907957+0000","last_clean":"2024-01-16T14:59:28.907957+0000","last_became_active":"2024-01-16T14:59:27.720348+0000","last_became_peered":"2024-01-16T14:59:27.720348+0000","last_unstale":"2024-01-16T14:59:28.907957+0000","last_undegraded":"2024-01-16T14:59:28.907957+0000","last_fullsized":"2024-01-16T14:59:28.907957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:58.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710390+0000","last_change":"2024-01-16T14:59:27.727783+0000","last_active":"2024-01-16T14:59:28.710390+0000","last_peered":"2024-01-16T14:59:28.710390+0000","last_clean":"2024-01-16T14:59:28.710390+0000","last_became_active":"2024-01-16T14:59:27.727562+0000","last_became_peered":"2024-01-16T14:59:27.727562+0000","last_unstale":"2024-01-16T14:59:28.710390+0000","last_undegraded":"2024-01-16T14:59:28.710390+0000","last_fullsized":"2024-01-16T14:59:28.710390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907372+0000","last_change":"2024-01-16T14:59:27.724755+0000","last_active":"2024-01-16T14:59:28.907372+0000","last_peered":"2024-01-16T14:59:28.907372+0000","last_clean":"2024-01-16T14:59:28.907372+0000","last_became_active":"2024-01-16T14:59:27.724443+0000","last_became_peered":"2024-01-16T14:59:27.724443+0000","last_unstale":"2024-01-16T14:59:28.907372+0000","last_undegraded":"2024-01-16T14:59:28.907372+0000","last_fullsized":"2024-01-16T14:59:28.907372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:06.824346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906082+0000","last_change":"2024-01-16T14:59:27.732620+0000","last_active":"2024-01-16T14:59:28.906082+0000","last_peered":"2024-01-16T14:59:28.906082+0000","last_clean":"2024-01-16T14:59:28.906082+0000","last_became_active":"2024-01-16T14:59:27.732037+0000","last_became_peered":"2024-01-16T14:59:27.732037+0000","last_unstale":"2024-01-16T14:59:28.906082+0000","last_undegraded":"2024-01-16T14:59:28.906082+0000","last_fullsized":"2024-01-16T14:59:28.906082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:13.794976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905598+0000","last_change":"2024-01-16T14:59:27.719539+0000","last_active":"2024-01-16T14:59:28.905598+0000","last_peered":"2024-01-16T14:59:28.905598+0000","last_clean":"2024-01-16T14:59:28.905598+0000","last_became_active":"2024-01-16T14:59:27.719253+0000","last_became_peered":"2024-01-16T14:59:27.719253+0000","last_unstale":"2024-01-16T14:59:28.905598+0000","last_undegraded":"2024-01-16T14:59:28.905598+0000","last_fullsized":"2024-01-16T14:59:28.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:42:03.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905607+0000","last_change":"2024-01-16T14:59:27.715773+0000","last_active":"2024-01-16T14:59:28.905607+0000","last_peered":"2024-01-16T14:59:28.905607+0000","last_clean":"2024-01-16T14:59:28.905607+0000","last_became_active":"2024-01-16T14:59:27.715482+0000","last_became_peered":"2024-01-16T14:59:27.715482+0000","last_unstale":"2024-01-16T14:59:28.905607+0000","last_undegraded":"2024-01-16T14:59:28.905607+0000","last_fullsized":"2024-01-16T14:59:28.905607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906301+0000","last_change":"2024-01-16T14:59:27.726710+0000","last_active":"2024-01-16T14:59:28.906301+0000","last_peered":"2024-01-16T14:59:28.906301+0000","last_clean":"2024-01-16T14:59:28.906301+0000","last_became_active":"2024-01-16T14:59:27.723820+0000","last_became_peered":"2024-01-16T14:59:27.723820+0000","last_unstale":"2024-01-16T14:59:28.906301+0000","last_undegraded":"2024-01-16T14:59:28.906301+0000","last_fullsized":"2024-01-16T14:59:28.906301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:10.938227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711055+0000","last_change":"2024-01-16T14:59:27.726914+0000","last_active":"2024-01-16T14:59:28.711055+0000","last_peered":"2024-01-16T14:59:28.711055+0000","last_clean":"2024-01-16T14:59:28.711055+0000","last_became_active":"2024-01-16T14:59:27.725638+0000","last_became_peered":"2024-01-16T14:59:27.725638+0000","last_unstale":"2024-01-16T14:59:28.711055+0000","last_undegraded":"2024-01-16T14:59:28.711055+0000","last_fullsized":"2024-01-16T14:59:28.711055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907822+0000","last_change":"2024-01-16T14:59:27.703634+0000","last_active":"2024-01-16T14:59:28.907822+0000","last_peered":"2024-01-16T14:59:28.907822+0000","last_clean":"2024-01-16T14:59:28.907822+0000","last_became_active":"2024-01-16T14:59:27.703343+0000","last_became_peered":"2024-01-16T14:59:27.703343+0000","last_unstale":"2024-01-16T14:59:28.907822+0000","last_undegraded":"2024-01-16T14:59:28.907822+0000","last_fullsized":"2024-01-16T14:59:28.907822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906778+0000","last_change":"2024-01-16T14:59:27.724563+0000","last_active":"2024-01-16T14:59:28.906778+0000","last_peered":"2024-01-16T14:59:28.906778+0000","last_clean":"2024-01-16T14:59:28.906778+0000","last_became_active":"2024-01-16T14:59:27.723074+0000","last_became_peered":"2024-01-16T14:59:27.723074+0000","last_unstale":"2024-01-16T14:59:28.906778+0000","last_undegraded":"2024-01-16T14:59:28.906778+0000","last_fullsized":"2024-01-16T14:59:28.906778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:05.783445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907568+0000","last_change":"2024-01-16T14:59:27.724629+0000","last_active":"2024-01-16T14:59:28.907568+0000","last_peered":"2024-01-16T14:59:28.907568+0000","last_clean":"2024-01-16T14:59:28.907568+0000","last_became_active":"2024-01-16T14:59:27.724363+0000","last_became_peered":"2024-01-16T14:59:27.724363+0000","last_unstale":"2024-01-16T14:59:28.907568+0000","last_undegraded":"2024-01-16T14:59:28.907568+0000","last_fullsized":"2024-01-16T14:59:28.907568+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:48.253356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906430+0000","last_change":"2024-01-16T14:59:27.722940+0000","last_active":"2024-01-16T14:59:28.906430+0000","last_peered":"2024-01-16T14:59:28.906430+0000","last_clean":"2024-01-16T14:59:28.906430+0000","last_became_active":"2024-01-16T14:59:27.722567+0000","last_became_peered":"2024-01-16T14:59:27.722567+0000","last_unstale":"2024-01-16T14:59:28.906430+0000","last_undegraded":"2024-01-16T14:59:28.906430+0000","last_fullsized":"2024-01-16T14:59:28.906430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:13.739812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703087+0000","last_change":"2024-01-16T14:59:27.722429+0000","last_active":"2024-01-16T14:59:28.703087+0000","last_peered":"2024-01-16T14:59:28.703087+0000","last_clean":"2024-01-16T14:59:28.703087+0000","last_became_active":"2024-01-16T14:59:27.722092+0000","last_became_peered":"2024-01-16T14:59:27.722092+0000","last_unstale":"2024-01-16T14:59:28.703087+0000","last_undegraded":"2024-01-16T14:59:28.703087+0000","last_fullsized":"2024-01-16T14:59:28.703087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709008+0000","last_change":"2024-01-16T14:59:27.719513+0000","last_active":"2024-01-16T14:59:28.709008+0000","last_peered":"2024-01-16T14:59:28.709008+0000","last_clean":"2024-01-16T14:59:28.709008+0000","last_became_active":"2024-01-16T14:59:27.718757+0000","last_became_peered":"2024-01-16T14:59:27.718757+0000","last_unstale":"2024-01-16T14:59:28.709008+0000","last_undegraded":"2024-01-16T14:59:28.709008+0000","last_fullsized":"2024-01-16T14:59:28.709008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.707752+0000","last_change":"2024-01-16T14:59:27.706529+0000","last_active":"2024-01-16T14:59:28.707752+0000","last_peered":"2024-01-16T14:59:28.707752+0000","last_clean":"2024-01-16T14:59:28.707752+0000","last_became_active":"2024-01-16T14:59:27.706241+0000","last_became_peered":"2024-01-16T14:59:27.706241+0000","last_unstale":"2024-01-16T14:59:28.707752+0000","last_undegraded":"2024-01-16T14:59:28.707752+0000","last_fullsized":"2024-01-16T14:59:28.707752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:40.765365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906601+0000","last_change":"2024-01-16T14:59:26.697929+0000","last_active":"2024-01-16T14:59:28.906601+0000","last_peered":"2024-01-16T14:59:28.906601+0000","last_clean":"2024-01-16T14:59:28.906601+0000","last_became_active":"2024-01-16T14:59:26.697578+0000","last_became_peered":"2024-01-16T14:59:26.697578+0000","last_unstale":"2024-01-16T14:59:28.906601+0000","last_undegraded":"2024-01-16T14:59:28.906601+0000","last_fullsized":"2024-01-16T14:59:28.906601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:26.816846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907687+0000","last_change":"2024-01-16T14:59:27.723311+0000","last_active":"2024-01-16T14:59:28.907687+0000","last_peered":"2024-01-16T14:59:28.907687+0000","last_clean":"2024-01-16T14:59:28.907687+0000","last_became_active":"2024-01-16T14:59:27.722975+0000","last_became_peered":"2024-01-16T14:59:27.722975+0000","last_unstale":"2024-01-16T14:59:28.907687+0000","last_undegraded":"2024-01-16T14:59:28.907687+0000","last_fullsized":"2024-01-16T14:59:28.907687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710186+0000","last_change":"2024-01-16T14:59:26.683247+0000","last_active":"2024-01-16T14:59:28.710186+0000","last_peered":"2024-01-16T14:59:28.710186+0000","last_clean":"2024-01-16T14:59:28.710186+0000","last_became_active":"2024-01-16T14:59:26.682727+0000","last_became_peered":"2024-01-16T14:59:26.682727+0000","last_unstale":"2024-01-16T14:59:28.710186+0000","last_undegraded":"2024-01-16T14:59:28.710186+0000","last_fullsized":"2024-01-16T14:59:28.710186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905763+0000","last_change":"2024-01-16T14:59:26.687905+0000","last_active":"2024-01-16T14:59:28.905763+0000","last_peered":"2024-01-16T14:59:28.905763+0000","last_clean":"2024-01-16T14:59:28.905763+0000","last_became_active":"2024-01-16T14:59:26.687570+0000","last_became_peered":"2024-01-16T14:59:26.687570+0000","last_unstale":"2024-01-16T14:59:28.905763+0000","last_undegraded":"2024-01-16T14:59:28.905763+0000","last_fullsized":"2024-01-16T14:59:28.905763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904948+0000","last_change":"2024-01-16T14:59:27.715786+0000","last_active":"2024-01-16T14:59:28.904948+0000","last_peered":"2024-01-16T14:59:28.904948+0000","last_clean":"2024-01-16T14:59:28.904948+0000","last_became_active":"2024-01-16T14:59:27.715516+0000","last_became_peered":"2024-01-16T14:59:27.715516+0000","last_unstale":"2024-01-16T14:59:28.904948+0000","last_undegraded":"2024-01-16T14:59:28.904948+0000","last_fullsized":"2024-01-16T14:59:28.904948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:14.095920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905832+0000","last_change":"2024-01-16T14:59:26.698294+0000","last_active":"2024-01-16T14:59:28.905832+0000","last_peered":"2024-01-16T14:59:28.905832+0000","last_clean":"2024-01-16T14:59:28.905832+0000","last_became_active":"2024-01-16T14:59:26.697917+0000","last_became_peered":"2024-01-16T14:59:26.697917+0000","last_unstale":"2024-01-16T14:59:28.905832+0000","last_undegraded":"2024-01-16T14:59:28.905832+0000","last_fullsized":"2024-01-16T14:59:28.905832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:15.675945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905800+0000","last_change":"2024-01-16T14:59:27.704715+0000","last_active":"2024-01-16T14:59:28.905800+0000","last_peered":"2024-01-16T14:59:28.905800+0000","last_clean":"2024-01-16T14:59:28.905800+0000","last_became_active":"2024-01-16T14:59:27.704439+0000","last_became_peered":"2024-01-16T14:59:27.704439+0000","last_unstale":"2024-01-16T14:59:28.905800+0000","last_undegraded":"2024-01-16T14:59:28.905800+0000","last_fullsized":"2024-01-16T14:59:28.905800+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710792+0000","last_change":"2024-01-16T14:59:26.689740+0000","last_active":"2024-01-16T14:59:28.710792+0000","last_peered":"2024-01-16T14:59:28.710792+0000","last_clean":"2024-01-16T14:59:28.710792+0000","last_became_active":"2024-01-16T14:59:26.689434+0000","last_became_peered":"2024-01-16T14:59:26.689434+0000","last_unstale":"2024-01-16T14:59:28.710792+0000","last_undegraded":"2024-01-16T14:59:28.710792+0000","last_fullsized":"2024-01-16T14:59:28.710792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:39.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907111+0000","last_change":"2024-01-16T14:59:27.733517+0000","last_active":"2024-01-16T14:59:28.907111+0000","last_peered":"2024-01-16T14:59:28.907111+0000","last_clean":"2024-01-16T14:59:28.907111+0000","last_became_active":"2024-01-16T14:59:27.732223+0000","last_became_peered":"2024-01-16T14:59:27.732223+0000","last_unstale":"2024-01-16T14:59:28.907111+0000","last_undegraded":"2024-01-16T14:59:28.907111+0000","last_fullsized":"2024-01-16T14:59:28.907111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:55.788928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907375+0000","last_change":"2024-01-16T14:59:26.693230+0000","last_active":"2024-01-16T14:59:28.907375+0000","last_peered":"2024-01-16T14:59:28.907375+0000","last_clean":"2024-01-16T14:59:28.907375+0000","last_became_active":"2024-01-16T14:59:26.693007+0000","last_became_peered":"2024-01-16T14:59:26.693007+0000","last_unstale":"2024-01-16T14:59:28.907375+0000","last_undegraded":"2024-01-16T14:59:28.907375+0000","last_fullsized":"2024-01-16T14:59:28.907375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909099+0000","last_change":"2024-01-16T14:59:27.723932+0000","last_active":"2024-01-16T14:59:28.909099+0000","last_peered":"2024-01-16T14:59:28.909099+0000","last_clean":"2024-01-16T14:59:28.909099+0000","last_became_active":"2024-01-16T14:59:27.723629+0000","last_became_peered":"2024-01-16T14:59:27.723629+0000","last_unstale":"2024-01-16T14:59:28.909099+0000","last_undegraded":"2024-01-16T14:59:28.909099+0000","last_fullsized":"2024-01-16T14:59:28.909099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908417+0000","last_change":"2024-01-16T14:59:26.698500+0000","last_active":"2024-01-16T14:59:28.908417+0000","last_peered":"2024-01-16T14:59:28.908417+0000","last_clean":"2024-01-16T14:59:28.908417+0000","last_became_active":"2024-01-16T14:59:26.698193+0000","last_became_peered":"2024-01-16T14:59:26.698193+0000","last_unstale":"2024-01-16T14:59:28.908417+0000","last_undegraded":"2024-01-16T14:59:28.908417+0000","last_fullsized":"2024-01-16T14:59:28.908417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:32.268892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709922+0000","last_change":"2024-01-16T14:59:27.724449+0000","last_active":"2024-01-16T14:59:28.709922+0000","last_peered":"2024-01-16T14:59:28.709922+0000","last_clean":"2024-01-16T14:59:28.709922+0000","last_became_active":"2024-01-16T14:59:27.723990+0000","last_became_peered":"2024-01-16T14:59:27.723990+0000","last_unstale":"2024-01-16T14:59:28.709922+0000","last_undegraded":"2024-01-16T14:59:28.709922+0000","last_fullsized":"2024-01-16T14:59:28.709922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710357+0000","last_change":"2024-01-16T14:59:26.688962+0000","last_active":"2024-01-16T14:59:28.710357+0000","last_peered":"2024-01-16T14:59:28.710357+0000","last_clean":"2024-01-16T14:59:28.710357+0000","last_became_active":"2024-01-16T14:59:26.687694+0000","last_became_peered":"2024-01-16T14:59:26.687694+0000","last_unstale":"2024-01-16T14:59:28.710357+0000","last_undegraded":"2024-01-16T14:59:28.710357+0000","last_fullsized":"2024-01-16T14:59:28.710357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907384+0000","last_change":"2024-01-16T14:59:27.718245+0000","last_active":"2024-01-16T14:59:28.907384+0000","last_peered":"2024-01-16T14:59:28.907384+0000","last_clean":"2024-01-16T14:59:28.907384+0000","last_became_active":"2024-01-16T14:59:27.717994+0000","last_became_peered":"2024-01-16T14:59:27.717994+0000","last_unstale":"2024-01-16T14:59:28.907384+0000","last_undegraded":"2024-01-16T14:59:28.907384+0000","last_fullsized":"2024-01-16T14:59:28.907384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:37.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907794+0000","last_change":"2024-01-16T14:59:26.682235+0000","last_active":"2024-01-16T14:59:28.907794+0000","last_peered":"2024-01-16T14:59:28.907794+0000","last_clean":"2024-01-16T14:59:28.907794+0000","last_became_active":"2024-01-16T14:59:26.681912+0000","last_became_peered":"2024-01-16T14:59:26.681912+0000","last_unstale":"2024-01-16T14:59:28.907794+0000","last_undegraded":"2024-01-16T14:59:28.907794+0000","last_fullsized":"2024-01-16T14:59:28.907794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906880+0000","last_change":"2024-01-16T14:59:27.718808+0000","last_active":"2024-01-16T14:59:28.906880+0000","last_peered":"2024-01-16T14:59:28.906880+0000","last_clean":"2024-01-16T14:59:28.906880+0000","last_became_active":"2024-01-16T14:59:27.717611+0000","last_became_peered":"2024-01-16T14:59:27.717611+0000","last_unstale":"2024-01-16T14:59:28.906880+0000","last_undegraded":"2024-01-16T14:59:28.906880+0000","last_fullsized":"2024-01-16T14:59:28.906880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:37.694723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906417+0000","last_change":"2024-01-16T14:59:26.701196+0000","last_active":"2024-01-16T14:59:28.906417+0000","last_peered":"2024-01-16T14:59:28.906417+0000","last_clean":"2024-01-16T14:59:28.906417+0000","last_became_active":"2024-01-16T14:59:26.700600+0000","last_became_peered":"2024-01-16T14:59:26.700600+0000","last_unstale":"2024-01-16T14:59:28.906417+0000","last_undegraded":"2024-01-16T14:59:28.906417+0000","last_fullsized":"2024-01-16T14:59:28.906417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:20.742041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906388+0000","last_change":"2024-01-16T14:59:27.729587+0000","last_active":"2024-01-16T14:59:28.906388+0000","last_peered":"2024-01-16T14:59:28.906388+0000","last_clean":"2024-01-16T14:59:28.906388+0000","last_became_active":"2024-01-16T14:59:27.729374+0000","last_became_peered":"2024-01-16T14:59:27.729374+0000","last_unstale":"2024-01-16T14:59:28.906388+0000","last_undegraded":"2024-01-16T14:59:28.906388+0000","last_fullsized":"2024-01-16T14:59:28.906388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:07.977388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709779+0000","last_change":"2024-01-16T14:59:26.682595+0000","last_active":"2024-01-16T14:59:28.709779+0000","last_peered":"2024-01-16T14:59:28.709779+0000","last_clean":"2024-01-16T14:59:28.709779+0000","last_became_active":"2024-01-16T14:59:26.682263+0000","last_became_peered":"2024-01-16T14:59:26.682263+0000","last_unstale":"2024-01-16T14:59:28.709779+0000","last_undegraded":"2024-01-16T14:59:28.709779+0000","last_fullsized":"2024-01-16T14:59:28.709779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:05.652841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709357+0000","last_change":"2024-01-16T14:59:27.721378+0000","last_active":"2024-01-16T14:59:28.709357+0000","last_peered":"2024-01-16T14:59:28.709357+0000","last_clean":"2024-01-16T14:59:28.709357+0000","last_became_active":"2024-01-16T14:59:27.721133+0000","last_became_peered":"2024-01-16T14:59:27.721133+0000","last_unstale":"2024-01-16T14:59:28.709357+0000","last_undegraded":"2024-01-16T14:59:28.709357+0000","last_fullsized":"2024-01-16T14:59:28.709357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703039+0000","last_change":"2024-01-16T14:59:26.688711+0000","last_active":"2024-01-16T14:59:28.703039+0000","last_peered":"2024-01-16T14:59:28.703039+0000","last_clean":"2024-01-16T14:59:28.703039+0000","last_became_active":"2024-01-16T14:59:26.688404+0000","last_became_peered":"2024-01-16T14:59:26.688404+0000","last_unstale":"2024-01-16T14:59:28.703039+0000","last_undegraded":"2024-01-16T14:59:28.703039+0000","last_fullsized":"2024-01-16T14:59:28.703039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904624+0000","last_change":"2024-01-16T14:59:27.724086+0000","last_active":"2024-01-16T14:59:28.904624+0000","last_peered":"2024-01-16T14:59:28.904624+0000","last_clean":"2024-01-16T14:59:28.904624+0000","last_became_active":"2024-01-16T14:59:27.723846+0000","last_became_peered":"2024-01-16T14:59:27.723846+0000","last_unstale":"2024-01-16T14:59:28.904624+0000","last_undegraded":"2024-01-16T14:59:28.904624+0000","last_fullsized":"2024-01-16T14:59:28.904624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.530195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709625+0000","last_change":"2024-01-16T14:59:26.687473+0000","last_active":"2024-01-16T14:59:28.709625+0000","last_peered":"2024-01-16T14:59:28.709625+0000","last_clean":"2024-01-16T14:59:28.709625+0000","last_became_active":"2024-01-16T14:59:26.687161+0000","last_became_peered":"2024-01-16T14:59:26.687161+0000","last_unstale":"2024-01-16T14:59:28.709625+0000","last_undegraded":"2024-01-16T14:59:28.709625+0000","last_fullsized":"2024-01-16T14:59:28.709625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908536+0000","last_change":"2024-01-16T14:59:27.726514+0000","last_active":"2024-01-16T14:59:28.908536+0000","last_peered":"2024-01-16T14:59:28.908536+0000","last_clean":"2024-01-16T14:59:28.908536+0000","last_became_active":"2024-01-16T14:59:27.726196+0000","last_became_peered":"2024-01-16T14:59:27.726196+0000","last_unstale":"2024-01-16T14:59:28.908536+0000","last_undegraded":"2024-01-16T14:59:28.908536+0000","last_fullsized":"2024-01-16T14:59:28.908536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:35.130412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703648+0000","last_change":"2024-01-16T14:59:26.693748+0000","last_active":"2024-01-16T14:59:28.703648+0000","last_peered":"2024-01-16T14:59:28.703648+0000","last_clean":"2024-01-16T14:59:28.703648+0000","last_became_active":"2024-01-16T14:59:26.693411+0000","last_became_peered":"2024-01-16T14:59:26.693411+0000","last_unstale":"2024-01-16T14:59:28.703648+0000","last_undegraded":"2024-01-16T14:59:28.703648+0000","last_fullsized":"2024-01-16T14:59:28.703648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908456+0000","last_change":"2024-01-16T14:59:27.724021+0000","last_active":"2024-01-16T14:59:28.908456+0000","last_peered":"2024-01-16T14:59:28.908456+0000","last_clean":"2024-01-16T14:59:28.908456+0000","last_became_active":"2024-01-16T14:59:27.723760+0000","last_became_peered":"2024-01-16T14:59:27.723760+0000","last_unstale":"2024-01-16T14:59:28.908456+0000","last_undegraded":"2024-01-16T14:59:28.908456+0000","last_fullsized":"2024-01-16T14:59:28.908456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:56.547755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908196+0000","last_change":"2024-01-16T14:59:26.687768+0000","last_active":"2024-01-16T14:59:28.908196+0000","last_peered":"2024-01-16T14:59:28.908196+0000","last_clean":"2024-01-16T14:59:28.908196+0000","last_became_active":"2024-01-16T14:59:26.687439+0000","last_became_peered":"2024-01-16T14:59:26.687439+0000","last_unstale":"2024-01-16T14:59:28.908196+0000","last_undegraded":"2024-01-16T14:59:28.908196+0000","last_fullsized":"2024-01-16T14:59:28.908196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905945+0000","last_change":"2024-01-16T14:59:27.716631+0000","last_active":"2024-01-16T14:59:28.905945+0000","last_peered":"2024-01-16T14:59:28.905945+0000","last_clean":"2024-01-16T14:59:28.905945+0000","last_became_active":"2024-01-16T14:59:27.716370+0000","last_became_peered":"2024-01-16T14:59:27.716370+0000","last_unstale":"2024-01-16T14:59:28.905945+0000","last_undegraded":"2024-01-16T14:59:28.905945+0000","last_fullsized":"2024-01-16T14:59:28.905945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.471222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703521+0000","last_change":"2024-01-16T14:59:26.695961+0000","last_active":"2024-01-16T14:59:28.703521+0000","last_peered":"2024-01-16T14:59:28.703521+0000","last_clean":"2024-01-16T14:59:28.703521+0000","last_became_active":"2024-01-16T14:59:26.695602+0000","last_became_peered":"2024-01-16T14:59:26.695602+0000","last_unstale":"2024-01-16T14:59:28.703521+0000","last_undegraded":"2024-01-16T14:59:28.703521+0000","last_fullsized":"2024-01-16T14:59:28.703521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:07.651834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907463+0000","last_change":"2024-01-16T14:59:27.736089+0000","last_active":"2024-01-16T14:59:28.907463+0000","last_peered":"2024-01-16T14:59:28.907463+0000","last_clean":"2024-01-16T14:59:28.907463+0000","last_became_active":"2024-01-16T14:59:27.731504+0000","last_became_peered":"2024-01-16T14:59:27.731504+0000","last_unstale":"2024-01-16T14:59:28.907463+0000","last_undegraded":"2024-01-16T14:59:28.907463+0000","last_fullsized":"2024-01-16T14:59:28.907463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:20.319731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:27.705910+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T14:59:27.705910+0000","last_peered":"2024-01-16T14:59:27.705910+0000","last_clean":"2024-01-16T14:59:27.705910+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T14:59:27.705910+0000","last_undegraded":"2024-01-16T14:59:27.705910+0000","last_fullsized":"2024-01-16T14:59:27.705910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905174+0000","last_change":"2024-01-16T14:59:26.693490+0000","last_active":"2024-01-16T14:59:28.905174+0000","last_peered":"2024-01-16T14:59:28.905174+0000","last_clean":"2024-01-16T14:59:28.905174+0000","last_became_active":"2024-01-16T14:59:26.692487+0000","last_became_peered":"2024-01-16T14:59:26.692487+0000","last_unstale":"2024-01-16T14:59:28.905174+0000","last_undegraded":"2024-01-16T14:59:28.905174+0000","last_fullsized":"2024-01-16T14:59:28.905174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906140+0000","last_change":"2024-01-16T14:59:27.721477+0000","last_active":"2024-01-16T14:59:28.906140+0000","last_peered":"2024-01-16T14:59:28.906140+0000","last_clean":"2024-01-16T14:59:28.906140+0000","last_became_active":"2024-01-16T14:59:27.720924+0000","last_became_peered":"2024-01-16T14:59:27.720924+0000","last_unstale":"2024-01-16T14:59:28.906140+0000","last_undegraded":"2024-01-16T14:59:28.906140+0000","last_fullsized":"2024-01-16T14:59:28.906140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.326753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904387+0000","last_change":"2024-01-16T14:59:26.701503+0000","last_active":"2024-01-16T14:59:28.904387+0000","last_peered":"2024-01-16T14:59:28.904387+0000","last_clean":"2024-01-16T14:59:28.904387+0000","last_became_active":"2024-01-16T14:59:26.701190+0000","last_became_peered":"2024-01-16T14:59:26.701190+0000","last_unstale":"2024-01-16T14:59:28.904387+0000","last_undegraded":"2024-01-16T14:59:28.904387+0000","last_fullsized":"2024-01-16T14:59:28.904387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906011+0000","last_change":"2024-01-16T14:59:27.728625+0000","last_active":"2024-01-16T14:59:28.906011+0000","last_peered":"2024-01-16T14:59:28.906011+0000","last_clean":"2024-01-16T14:59:28.906011+0000","last_became_active":"2024-01-16T14:59:27.728177+0000","last_became_peered":"2024-01-16T14:59:27.728177+0000","last_unstale":"2024-01-16T14:59:28.906011+0000","last_undegraded":"2024-01-16T14:59:28.906011+0000","last_fullsized":"2024-01-16T14:59:28.906011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:38.465332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703101+0000","last_change":"2024-01-16T14:59:26.695022+0000","last_active":"2024-01-16T14:59:28.703101+0000","last_peered":"2024-01-16T14:59:28.703101+0000","last_clean":"2024-01-16T14:59:28.703101+0000","last_became_active":"2024-01-16T14:59:26.694756+0000","last_became_peered":"2024-01-16T14:59:26.694756+0000","last_unstale":"2024-01-16T14:59:28.703101+0000","last_undegraded":"2024-01-16T14:59:28.703101+0000","last_fullsized":"2024-01-16T14:59:28.703101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:41.781228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905205+0000","last_change":"2024-01-16T14:59:27.706635+0000","last_active":"2024-01-16T14:59:28.905205+0000","last_peered":"2024-01-16T14:59:28.905205+0000","last_clean":"2024-01-16T14:59:28.905205+0000","last_became_active":"2024-01-16T14:59:27.706296+0000","last_became_peered":"2024-01-16T14:59:27.706296+0000","last_unstale":"2024-01-16T14:59:28.905205+0000","last_undegraded":"2024-01-16T14:59:28.905205+0000","last_fullsized":"2024-01-16T14:59:28.905205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906768+0000","last_change":"2024-01-16T14:59:26.700908+0000","last_active":"2024-01-16T14:59:28.906768+0000","last_peered":"2024-01-16T14:59:28.906768+0000","last_clean":"2024-01-16T14:59:28.906768+0000","last_became_active":"2024-01-16T14:59:26.699693+0000","last_became_peered":"2024-01-16T14:59:26.699693+0000","last_unstale":"2024-01-16T14:59:28.906768+0000","last_undegraded":"2024-01-16T14:59:28.906768+0000","last_fullsized":"2024-01-16T14:59:28.906768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:10.078798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708429+0000","last_change":"2024-01-16T14:59:27.717434+0000","last_active":"2024-01-16T14:59:28.708429+0000","last_peered":"2024-01-16T14:59:28.708429+0000","last_clean":"2024-01-16T14:59:28.708429+0000","last_became_active":"2024-01-16T14:59:27.717061+0000","last_became_peered":"2024-01-16T14:59:27.717061+0000","last_unstale":"2024-01-16T14:59:28.708429+0000","last_undegraded":"2024-01-16T14:59:28.708429+0000","last_fullsized":"2024-01-16T14:59:28.708429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:14.259904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907983+0000","last_change":"2024-01-16T14:59:26.689585+0000","last_active":"2024-01-16T14:59:28.907983+0000","last_peered":"2024-01-16T14:59:28.907983+0000","last_clean":"2024-01-16T14:59:28.907983+0000","last_became_active":"2024-01-16T14:59:26.689193+0000","last_became_peered":"2024-01-16T14:59:26.689193+0000","last_unstale":"2024-01-16T14:59:28.907983+0000","last_undegraded":"2024-01-16T14:59:28.907983+0000","last_fullsized":"2024-01-16T14:59:28.907983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:40.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907283+0000","last_change":"2024-01-16T14:59:27.723605+0000","last_active":"2024-01-16T14:59:28.907283+0000","last_peered":"2024-01-16T14:59:28.907283+0000","last_clean":"2024-01-16T14:59:28.907283+0000","last_became_active":"2024-01-16T14:59:27.723322+0000","last_became_peered":"2024-01-16T14:59:27.723322+0000","last_unstale":"2024-01-16T14:59:28.907283+0000","last_undegraded":"2024-01-16T14:59:28.907283+0000","last_fullsized":"2024-01-16T14:59:28.907283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.187834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908060+0000","last_change":"2024-01-16T14:59:26.701967+0000","last_active":"2024-01-16T14:59:28.908060+0000","last_peered":"2024-01-16T14:59:28.908060+0000","last_clean":"2024-01-16T14:59:28.908060+0000","last_became_active":"2024-01-16T14:59:26.701686+0000","last_became_peered":"2024-01-16T14:59:26.701686+0000","last_unstale":"2024-01-16T14:59:28.908060+0000","last_undegraded":"2024-01-16T14:59:28.908060+0000","last_fullsized":"2024-01-16T14:59:28.908060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:09.766721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907757+0000","last_change":"2024-01-16T14:59:27.709576+0000","last_active":"2024-01-16T14:59:28.907757+0000","last_peered":"2024-01-16T14:59:28.907757+0000","last_clean":"2024-01-16T14:59:28.907757+0000","last_became_active":"2024-01-16T14:59:27.706317+0000","last_became_peered":"2024-01-16T14:59:27.706317+0000","last_unstale":"2024-01-16T14:59:28.907757+0000","last_undegraded":"2024-01-16T14:59:28.907757+0000","last_fullsized":"2024-01-16T14:59:28.907757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:29.838805+0000","last_change":"2024-01-16T14:59:26.693739+0000","last_active":"2024-01-16T14:59:29.838805+0000","last_peered":"2024-01-16T14:59:29.838805+0000","last_clean":"2024-01-16T14:59:29.838805+0000","last_became_active":"2024-01-16T14:59:26.693473+0000","last_became_peered":"2024-01-16T14:59:26.693473+0000","last_unstale":"2024-01-16T14:59:29.838805+0000","last_undegraded":"2024-01-16T14:59:29.838805+0000","last_fullsized":"2024-01-16T14:59:29.838805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906445+0000","last_change":"2024-01-16T14:59:27.717414+0000","last_active":"2024-01-16T14:59:28.906445+0000","last_peered":"2024-01-16T14:59:28.906445+0000","last_clean":"2024-01-16T14:59:28.906445+0000","last_became_active":"2024-01-16T14:59:27.717078+0000","last_became_peered":"2024-01-16T14:59:27.717078+0000","last_unstale":"2024-01-16T14:59:28.906445+0000","last_undegraded":"2024-01-16T14:59:28.906445+0000","last_fullsized":"2024-01-16T14:59:28.906445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:30.634017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711015+0000","last_change":"2024-01-16T14:59:26.696098+0000","last_active":"2024-01-16T14:59:28.711015+0000","last_peered":"2024-01-16T14:59:28.711015+0000","last_clean":"2024-01-16T14:59:28.711015+0000","last_became_active":"2024-01-16T14:59:26.695792+0000","last_became_peered":"2024-01-16T14:59:26.695792+0000","last_unstale":"2024-01-16T14:59:28.711015+0000","last_undegraded":"2024-01-16T14:59:28.711015+0000","last_fullsized":"2024-01-16T14:59:28.711015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908934+0000","last_change":"2024-01-16T14:59:27.712757+0000","last_active":"2024-01-16T14:59:28.908934+0000","last_peered":"2024-01-16T14:59:28.908934+0000","last_clean":"2024-01-16T14:59:28.908934+0000","last_became_active":"2024-01-16T14:59:27.712417+0000","last_became_peered":"2024-01-16T14:59:27.712417+0000","last_unstale":"2024-01-16T14:59:28.908934+0000","last_undegraded":"2024-01-16T14:59:28.908934+0000","last_fullsized":"2024-01-16T14:59:28.908934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904378+0000","last_change":"2024-01-16T14:59:26.699479+0000","last_active":"2024-01-16T14:59:28.904378+0000","last_peered":"2024-01-16T14:59:28.904378+0000","last_clean":"2024-01-16T14:59:28.904378+0000","last_became_active":"2024-01-16T14:59:26.699195+0000","last_became_peered":"2024-01-16T14:59:26.699195+0000","last_unstale":"2024-01-16T14:59:28.904378+0000","last_undegraded":"2024-01-16T14:59:28.904378+0000","last_fullsized":"2024-01-16T14:59:28.904378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.521651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708137+0000","last_change":"2024-01-16T14:59:27.715745+0000","last_active":"2024-01-16T14:59:28.708137+0000","last_peered":"2024-01-16T14:59:28.708137+0000","last_clean":"2024-01-16T14:59:28.708137+0000","last_became_active":"2024-01-16T14:59:27.710835+0000","last_became_peered":"2024-01-16T14:59:27.710835+0000","last_unstale":"2024-01-16T14:59:28.708137+0000","last_undegraded":"2024-01-16T14:59:28.708137+0000","last_fullsized":"2024-01-16T14:59:28.708137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:53.870850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906409+0000","last_change":"2024-01-16T14:59:26.697228+0000","last_active":"2024-01-16T14:59:28.906409+0000","last_peered":"2024-01-16T14:59:28.906409+0000","last_clean":"2024-01-16T14:59:28.906409+0000","last_became_active":"2024-01-16T14:59:26.696136+0000","last_became_peered":"2024-01-16T14:59:26.696136+0000","last_unstale":"2024-01-16T14:59:28.906409+0000","last_undegraded":"2024-01-16T14:59:28.906409+0000","last_fullsized":"2024-01-16T14:59:28.906409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:37.293826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905459+0000","last_change":"2024-01-16T14:59:27.711533+0000","last_active":"2024-01-16T14:59:28.905459+0000","last_peered":"2024-01-16T14:59:28.905459+0000","last_clean":"2024-01-16T14:59:28.905459+0000","last_became_active":"2024-01-16T14:59:27.711228+0000","last_became_peered":"2024-01-16T14:59:27.711228+0000","last_unstale":"2024-01-16T14:59:28.905459+0000","last_undegraded":"2024-01-16T14:59:28.905459+0000","last_fullsized":"2024-01-16T14:59:28.905459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:28.132533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907464+0000","last_change":"2024-01-16T14:59:26.695937+0000","last_active":"2024-01-16T14:59:28.907464+0000","last_peered":"2024-01-16T14:59:28.907464+0000","last_clean":"2024-01-16T14:59:28.907464+0000","last_became_active":"2024-01-16T14:59:26.695628+0000","last_became_peered":"2024-01-16T14:59:26.695628+0000","last_unstale":"2024-01-16T14:59:28.907464+0000","last_undegraded":"2024-01-16T14:59:28.907464+0000","last_fullsized":"2024-01-16T14:59:28.907464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:36.523217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906350+0000","last_change":"2024-01-16T14:59:27.720565+0000","last_active":"2024-01-16T14:59:28.906350+0000","last_peered":"2024-01-16T14:59:28.906350+0000","last_clean":"2024-01-16T14:59:28.906350+0000","last_became_active":"2024-01-16T14:59:27.716762+0000","last_became_peered":"2024-01-16T14:59:27.716762+0000","last_unstale":"2024-01-16T14:59:28.906350+0000","last_undegraded":"2024-01-16T14:59:28.906350+0000","last_fullsized":"2024-01-16T14:59:28.906350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:26.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906643+0000","last_change":"2024-01-16T14:59:26.700414+0000","last_active":"2024-01-16T14:59:28.906643+0000","last_peered":"2024-01-16T14:59:28.906643+0000","last_clean":"2024-01-16T14:59:28.906643+0000","last_became_active":"2024-01-16T14:59:26.700111+0000","last_became_peered":"2024-01-16T14:59:26.700111+0000","last_unstale":"2024-01-16T14:59:28.906643+0000","last_undegraded":"2024-01-16T14:59:28.906643+0000","last_fullsized":"2024-01-16T14:59:28.906643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:29.462511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709936+0000","last_change":"2024-01-16T14:59:27.718486+0000","last_active":"2024-01-16T14:59:28.709936+0000","last_peered":"2024-01-16T14:59:28.709936+0000","last_clean":"2024-01-16T14:59:28.709936+0000","last_became_active":"2024-01-16T14:59:27.712753+0000","last_became_peered":"2024-01-16T14:59:27.712753+0000","last_unstale":"2024-01-16T14:59:28.709936+0000","last_undegraded":"2024-01-16T14:59:28.709936+0000","last_fullsized":"2024-01-16T14:59:28.709936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907057+0000","last_change":"2024-01-16T14:59:27.736702+0000","last_active":"2024-01-16T14:59:28.907057+0000","last_peered":"2024-01-16T14:59:28.907057+0000","last_clean":"2024-01-16T14:59:28.907057+0000","last_became_active":"2024-01-16T14:59:27.736433+0000","last_became_peered":"2024-01-16T14:59:27.736433+0000","last_unstale":"2024-01-16T14:59:28.907057+0000","last_undegraded":"2024-01-16T14:59:28.907057+0000","last_fullsized":"2024-01-16T14:59:28.907057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:13.830199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907201+0000","last_change":"2024-01-16T14:59:26.703355+0000","last_active":"2024-01-16T14:59:28.907201+0000","last_peered":"2024-01-16T14:59:28.907201+0000","last_clean":"2024-01-16T14:59:28.907201+0000","last_became_active":"2024-01-16T14:59:26.703059+0000","last_became_peered":"2024-01-16T14:59:26.703059+0000","last_unstale":"2024-01-16T14:59:28.907201+0000","last_undegraded":"2024-01-16T14:59:28.907201+0000","last_fullsized":"2024-01-16T14:59:28.907201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.422714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905630+0000","last_change":"2024-01-16T14:59:27.726539+0000","last_active":"2024-01-16T14:59:28.905630+0000","last_peered":"2024-01-16T14:59:28.905630+0000","last_clean":"2024-01-16T14:59:28.905630+0000","last_became_active":"2024-01-16T14:59:27.726269+0000","last_became_peered":"2024-01-16T14:59:27.726269+0000","last_unstale":"2024-01-16T14:59:28.905630+0000","last_undegraded":"2024-01-16T14:59:28.905630+0000","last_fullsized":"2024-01-16T14:59:28.905630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907840+0000","last_change":"2024-01-16T14:59:26.702143+0000","last_active":"2024-01-16T14:59:28.907840+0000","last_peered":"2024-01-16T14:59:28.907840+0000","last_clean":"2024-01-16T14:59:28.907840+0000","last_became_active":"2024-01-16T14:59:26.701732+0000","last_became_peered":"2024-01-16T14:59:26.701732+0000","last_unstale":"2024-01-16T14:59:28.907840+0000","last_undegraded":"2024-01-16T14:59:28.907840+0000","last_fullsized":"2024-01-16T14:59:28.907840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:44.454533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908894+0000","last_change":"2024-01-16T14:59:27.719650+0000","last_active":"2024-01-16T14:59:28.908894+0000","last_peered":"2024-01-16T14:59:28.908894+0000","last_clean":"2024-01-16T14:59:28.908894+0000","last_became_active":"2024-01-16T14:59:27.719345+0000","last_became_peered":"2024-01-16T14:59:27.719345+0000","last_unstale":"2024-01-16T14:59:28.908894+0000","last_undegraded":"2024-01-16T14:59:28.908894+0000","last_fullsized":"2024-01-16T14:59:28.908894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906403+0000","last_change":"2024-01-16T14:59:26.692569+0000","last_active":"2024-01-16T14:59:28.906403+0000","last_peered":"2024-01-16T14:59:28.906403+0000","last_clean":"2024-01-16T14:59:28.906403+0000","last_became_active":"2024-01-16T14:59:26.689432+0000","last_became_peered":"2024-01-16T14:59:26.689432+0000","last_unstale":"2024-01-16T14:59:28.906403+0000","last_undegraded":"2024-01-16T14:59:28.906403+0000","last_fullsized":"2024-01-16T14:59:28.906403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:58.385575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905910+0000","last_change":"2024-01-16T14:59:27.728184+0000","last_active":"2024-01-16T14:59:28.905910+0000","last_peered":"2024-01-16T14:59:28.905910+0000","last_clean":"2024-01-16T14:59:28.905910+0000","last_became_active":"2024-01-16T14:59:27.727889+0000","last_became_peered":"2024-01-16T14:59:27.727889+0000","last_unstale":"2024-01-16T14:59:28.905910+0000","last_undegraded":"2024-01-16T14:59:28.905910+0000","last_fullsized":"2024-01-16T14:59:28.905910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.211584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909226+0000","last_change":"2024-01-16T14:59:26.697456+0000","last_active":"2024-01-16T14:59:28.909226+0000","last_peered":"2024-01-16T14:59:28.909226+0000","last_clean":"2024-01-16T14:59:28.909226+0000","last_became_active":"2024-01-16T14:59:26.697129+0000","last_became_peered":"2024-01-16T14:59:26.697129+0000","last_unstale":"2024-01-16T14:59:28.909226+0000","last_undegraded":"2024-01-16T14:59:28.909226+0000","last_fullsized":"2024-01-16T14:59:28.909226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7849999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0739999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:37.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T14:59:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.083+0000 7f8481200640 1 Processor -- start 2024-01-16T14:59:38.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.083+0000 7f8481200640 1 -- start start 2024-01-16T14:59:38.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.085+0000 7f8481200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f847c06b730 0x7f847c165d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.085+0000 7f8481200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c066680 con 0x7f847c1412a0 2024-01-16T14:59:38.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.085+0000 7f8481200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0668b0 con 0x7f847c0661f0 2024-01-16T14:59:38.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.085+0000 7f8481200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c0cc4b0 con 0x7f847c06b730 2024-01-16T14:59:38.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.085+0000 7f847b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f847c1412a0 0x7f847c0cb690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53068/0 (socket says 172.21.15.43:53068) 2024-01-16T14:59:38.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847b400640 1 -- 172.21.15.43:0/2446087832 learned_addr learned my addr 172.21.15.43:0/2446087832 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847be00640 1 --2- 172.21.15.43:0/2446087832 >> v2:172.21.15.69:3300/0 conn(0x7f847c06b730 0x7f847c165d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683363340 0 0) 0x7f847c066680 con 0x7f847c1412a0 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c003520 con 0x7f847c1412a0 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815419911 0 0) 0x7f845c003520 con 0x7f847c1412a0 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 >> v2:172.21.15.69:3300/0 conn(0x7f847c06b730 msgr2=0x7f847c165d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 --2- 172.21.15.43:0/2446087832 >> v2:172.21.15.69:3300/0 conn(0x7f847c06b730 0x7f847c165d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 >> v1:172.21.15.69:6789/0 conn(0x7f847c0661f0 legacy=0x7f847c0cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0ce950 con 0x7f847c1412a0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8464001670 con 0x7f847c1412a0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 >> v1:172.21.15.43:6789/0 conn(0x7f847c1412a0 legacy=0x7f847c0cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f845c003a30 0x7f845c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.086+0000 7f847a000640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847a000640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f845c0052a0 0x7f845c00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0cc4b0 con 0x7f845c0047c0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0105b0 con 0x7f845c003a30 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c010770 con 0x7f845c0052a0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847be00640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f845c0052a0 0x7f845c00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847b400640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f845c003a30 0x7f845c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 -- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f845c0052a0 msgr2=0x7f845c00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f845c0052a0 0x7f845c00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 -- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f845c003a30 msgr2=0x7f845c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f845c003a30 0x7f845c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847aa00640 1 -- 172.21.15.43:0/2446087832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c003520 con 0x7f845c0047c0 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847be00640 1 -- 172.21.15.43:0/2446087832 reap_dead start 2024-01-16T14:59:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.087+0000 7f847b400640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f845c003a30 0x7f845c003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.088+0000 7f847aa00640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f845c004c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f846c001720 tx=0x7f845c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19968b98eef40120 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0052d0 con 0x7f845c0047c0 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c02e050 con 0x7f845c0047c0 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f847a000640 1 -- 172.21.15.43:0/2446087832 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c003830 con 0x7f845c0047c0 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 -- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 msgr2=0x7f845c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f845c004c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f846c001720 tx=0x7f845c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 -- 172.21.15.43:0/2446087832 shutdown_connections 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 --2- 172.21.15.43:0/2446087832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 -- 172.21.15.43:0/2446087832 >> 172.21.15.43:0/2446087832 conn(0x7f847c0cf4b0 msgr2=0x7f847c0cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 -- 172.21.15.43:0/2446087832 shutdown_connections 2024-01-16T14:59:38.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.089+0000 7f8481200640 1 -- 172.21.15.43:0/2446087832 wait complete. 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.091+0000 7f8481200640 1 Processor -- start 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.091+0000 7f8481200640 1 -- start start 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c140440 0x7f847c13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c13ef00 0x7f847c13f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1429e0 con 0x7f845c0047c0 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c13fcd0 con 0x7f847c13ef00 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f8481200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1421d0 con 0x7f847c140440 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.092+0000 7f847b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34446/0 (socket says 172.21.15.43:34446) 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 -- 172.21.15.43:0/3993583193 learned_addr learned my addr 172.21.15.43:0/3993583193 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:38.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 -- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c140440 msgr2=0x7f847c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c140440 0x7f847c13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 -- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c13ef00 msgr2=0x7f847c13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c13ef00 0x7f847c13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 -- 172.21.15.43:0/3993583193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c13e450 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f847b400640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f846400d570 tx=0x7f846400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84640074e0 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8464007640 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.093+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464004a00 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.094+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c142420 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.094+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c1408c0 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.095+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c04e840 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.096+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8464004b60 con 0x7f845c0047c0 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.097+0000 7f8473400640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 0x7f845406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.097+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84640892c0 con 0x7f845c0047c0 2024-01-16T14:59:38.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.097+0000 7f847aa00640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 0x7f845406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.098+0000 7f847aa00640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 0x7f845406d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f846c033000 tx=0x7f846c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.102+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8464054120 con 0x7f845c0047c0 2024-01-16T14:59:38.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.285+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f847c0cf1f0 con 0x7f845406b2c0 2024-01-16T14:59:38.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.305+0000 7f8473400640 1 -- 172.21.15.43:0/3993583193 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264404 (secure 0 0 0) 0x7f846c005390 con 0x7f845406b2c0 2024-01-16T14:59:38.290 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:38.291 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T14:59:38.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 msgr2=0x7f845406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 0x7f845406d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f846c033000 tx=0x7f846c02f000 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 msgr2=0x7f847c141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f846400d570 tx=0x7f846400da30 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 shutdown_connections 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406b2c0 0x7f845406d780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c13ef00 0x7f847c13f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c140440 0x7f847c13df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 --2- 172.21.15.43:0/3993583193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f845c0047c0 0x7f847c141a80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 >> 172.21.15.43:0/3993583193 conn(0x7f847c0cf4b0 msgr2=0x7f847c15a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 shutdown_connections 2024-01-16T14:59:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.310+0000 7f8481200640 1 -- 172.21.15.43:0/3993583193 wait complete. 2024-01-16T14:59:38.309 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T14:59:36.789779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240372,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734348,"statfs":{"total":773094113280,"available":772847972352,"internally_reserved":0,"allocated":3055616,"data_stored":1815972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.107296"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908219+0000","last_change":"2024-01-16T14:59:27.733425+0000","last_active":"2024-01-16T14:59:28.908219+0000","last_peered":"2024-01-16T14:59:28.908219+0000","last_clean":"2024-01-16T14:59:28.908219+0000","last_became_active":"2024-01-16T14:59:27.733108+0000","last_became_peered":"2024-01-16T14:59:27.733108+0000","last_unstale":"2024-01-16T14:59:28.908219+0000","last_undegraded":"2024-01-16T14:59:28.908219+0000","last_fullsized":"2024-01-16T14:59:28.908219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:28:12.476929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908249+0000","last_change":"2024-01-16T14:59:27.715575+0000","last_active":"2024-01-16T14:59:28.908249+0000","last_peered":"2024-01-16T14:59:28.908249+0000","last_clean":"2024-01-16T14:59:28.908249+0000","last_became_active":"2024-01-16T14:59:27.714841+0000","last_became_peered":"2024-01-16T14:59:27.714841+0000","last_unstale":"2024-01-16T14:59:28.908249+0000","last_undegraded":"2024-01-16T14:59:28.908249+0000","last_fullsized":"2024-01-16T14:59:28.908249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908551+0000","last_change":"2024-01-16T14:59:27.718807+0000","last_active":"2024-01-16T14:59:28.908551+0000","last_peered":"2024-01-16T14:59:28.908551+0000","last_clean":"2024-01-16T14:59:28.908551+0000","last_became_active":"2024-01-16T14:59:27.717301+0000","last_became_peered":"2024-01-16T14:59:27.717301+0000","last_unstale":"2024-01-16T14:59:28.908551+0000","last_undegraded":"2024-01-16T14:59:28.908551+0000","last_fullsized":"2024-01-16T14:59:28.908551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906129+0000","last_change":"2024-01-16T14:59:27.720914+0000","last_active":"2024-01-16T14:59:28.906129+0000","last_peered":"2024-01-16T14:59:28.906129+0000","last_clean":"2024-01-16T14:59:28.906129+0000","last_became_active":"2024-01-16T14:59:27.720561+0000","last_became_peered":"2024-01-16T14:59:27.720561+0000","last_unstale":"2024-01-16T14:59:28.906129+0000","last_undegraded":"2024-01-16T14:59:28.906129+0000","last_fullsized":"2024-01-16T14:59:28.906129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:16.656352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906008+0000","last_change":"2024-01-16T14:59:27.721194+0000","last_active":"2024-01-16T14:59:28.906008+0000","last_peered":"2024-01-16T14:59:28.906008+0000","last_clean":"2024-01-16T14:59:28.906008+0000","last_became_active":"2024-01-16T14:59:27.719980+0000","last_became_peered":"2024-01-16T14:59:27.719980+0000","last_unstale":"2024-01-16T14:59:28.906008+0000","last_undegraded":"2024-01-16T14:59:28.906008+0000","last_fullsized":"2024-01-16T14:59:28.906008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:54.580479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905723+0000","last_change":"2024-01-16T14:59:27.732776+0000","last_active":"2024-01-16T14:59:28.905723+0000","last_peered":"2024-01-16T14:59:28.905723+0000","last_clean":"2024-01-16T14:59:28.905723+0000","last_became_active":"2024-01-16T14:59:27.732373+0000","last_became_peered":"2024-01-16T14:59:27.732373+0000","last_unstale":"2024-01-16T14:59:28.905723+0000","last_undegraded":"2024-01-16T14:59:28.905723+0000","last_fullsized":"2024-01-16T14:59:28.905723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904627+0000","last_change":"2024-01-16T14:59:27.722159+0000","last_active":"2024-01-16T14:59:28.904627+0000","last_peered":"2024-01-16T14:59:28.904627+0000","last_clean":"2024-01-16T14:59:28.904627+0000","last_became_active":"2024-01-16T14:59:27.721928+0000","last_became_peered":"2024-01-16T14:59:27.721928+0000","last_unstale":"2024-01-16T14:59:28.904627+0000","last_undegraded":"2024-01-16T14:59:28.904627+0000","last_fullsized":"2024-01-16T14:59:28.904627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:17.023676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904587+0000","last_change":"2024-01-16T14:59:27.721172+0000","last_active":"2024-01-16T14:59:28.904587+0000","last_peered":"2024-01-16T14:59:28.904587+0000","last_clean":"2024-01-16T14:59:28.904587+0000","last_became_active":"2024-01-16T14:59:27.720829+0000","last_became_peered":"2024-01-16T14:59:27.720829+0000","last_unstale":"2024-01-16T14:59:28.904587+0000","last_undegraded":"2024-01-16T14:59:28.904587+0000","last_fullsized":"2024-01-16T14:59:28.904587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:39.594220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908493+0000","last_change":"2024-01-16T14:59:27.721916+0000","last_active":"2024-01-16T14:59:28.908493+0000","last_peered":"2024-01-16T14:59:28.908493+0000","last_clean":"2024-01-16T14:59:28.908493+0000","last_became_active":"2024-01-16T14:59:27.721604+0000","last_became_peered":"2024-01-16T14:59:27.721604+0000","last_unstale":"2024-01-16T14:59:28.908493+0000","last_undegraded":"2024-01-16T14:59:28.908493+0000","last_fullsized":"2024-01-16T14:59:28.908493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:05.858739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711127+0000","last_change":"2024-01-16T14:59:27.729823+0000","last_active":"2024-01-16T14:59:28.711127+0000","last_peered":"2024-01-16T14:59:28.711127+0000","last_clean":"2024-01-16T14:59:28.711127+0000","last_became_active":"2024-01-16T14:59:27.724755+0000","last_became_peered":"2024-01-16T14:59:27.724755+0000","last_unstale":"2024-01-16T14:59:28.711127+0000","last_undegraded":"2024-01-16T14:59:28.711127+0000","last_fullsized":"2024-01-16T14:59:28.711127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:33.582538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710385+0000","last_change":"2024-01-16T14:59:27.725448+0000","last_active":"2024-01-16T14:59:28.710385+0000","last_peered":"2024-01-16T14:59:28.710385+0000","last_clean":"2024-01-16T14:59:28.710385+0000","last_became_active":"2024-01-16T14:59:27.724294+0000","last_became_peered":"2024-01-16T14:59:27.724294+0000","last_unstale":"2024-01-16T14:59:28.710385+0000","last_undegraded":"2024-01-16T14:59:28.710385+0000","last_fullsized":"2024-01-16T14:59:28.710385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709647+0000","last_change":"2024-01-16T14:59:27.725056+0000","last_active":"2024-01-16T14:59:28.709647+0000","last_peered":"2024-01-16T14:59:28.709647+0000","last_clean":"2024-01-16T14:59:28.709647+0000","last_became_active":"2024-01-16T14:59:27.724771+0000","last_became_peered":"2024-01-16T14:59:27.724771+0000","last_unstale":"2024-01-16T14:59:28.709647+0000","last_undegraded":"2024-01-16T14:59:28.709647+0000","last_fullsized":"2024-01-16T14:59:28.709647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905952+0000","last_change":"2024-01-16T14:59:27.726131+0000","last_active":"2024-01-16T14:59:28.905952+0000","last_peered":"2024-01-16T14:59:28.905952+0000","last_clean":"2024-01-16T14:59:28.905952+0000","last_became_active":"2024-01-16T14:59:27.725830+0000","last_became_peered":"2024-01-16T14:59:27.725830+0000","last_unstale":"2024-01-16T14:59:28.905952+0000","last_undegraded":"2024-01-16T14:59:28.905952+0000","last_fullsized":"2024-01-16T14:59:28.905952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:45.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906705+0000","last_change":"2024-01-16T14:59:27.713525+0000","last_active":"2024-01-16T14:59:28.906705+0000","last_peered":"2024-01-16T14:59:28.906705+0000","last_clean":"2024-01-16T14:59:28.906705+0000","last_became_active":"2024-01-16T14:59:27.712762+0000","last_became_peered":"2024-01-16T14:59:27.712762+0000","last_unstale":"2024-01-16T14:59:28.906705+0000","last_undegraded":"2024-01-16T14:59:28.906705+0000","last_fullsized":"2024-01-16T14:59:28.906705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:41.373234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906179+0000","last_change":"2024-01-16T14:59:27.700808+0000","last_active":"2024-01-16T14:59:28.906179+0000","last_peered":"2024-01-16T14:59:28.906179+0000","last_clean":"2024-01-16T14:59:28.906179+0000","last_became_active":"2024-01-16T14:59:27.700477+0000","last_became_peered":"2024-01-16T14:59:27.700477+0000","last_unstale":"2024-01-16T14:59:28.906179+0000","last_undegraded":"2024-01-16T14:59:28.906179+0000","last_fullsized":"2024-01-16T14:59:28.906179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905239+0000","last_change":"2024-01-16T14:59:27.720864+0000","last_active":"2024-01-16T14:59:28.905239+0000","last_peered":"2024-01-16T14:59:28.905239+0000","last_clean":"2024-01-16T14:59:28.905239+0000","last_became_active":"2024-01-16T14:59:27.719599+0000","last_became_peered":"2024-01-16T14:59:27.719599+0000","last_unstale":"2024-01-16T14:59:28.905239+0000","last_undegraded":"2024-01-16T14:59:28.905239+0000","last_fullsized":"2024-01-16T14:59:28.905239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907987+0000","last_change":"2024-01-16T14:59:27.717946+0000","last_active":"2024-01-16T14:59:28.907987+0000","last_peered":"2024-01-16T14:59:28.907987+0000","last_clean":"2024-01-16T14:59:28.907987+0000","last_became_active":"2024-01-16T14:59:27.717711+0000","last_became_peered":"2024-01-16T14:59:27.717711+0000","last_unstale":"2024-01-16T14:59:28.907987+0000","last_undegraded":"2024-01-16T14:59:28.907987+0000","last_fullsized":"2024-01-16T14:59:28.907987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:03.931690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703527+0000","last_change":"2024-01-16T14:59:27.726121+0000","last_active":"2024-01-16T14:59:28.703527+0000","last_peered":"2024-01-16T14:59:28.703527+0000","last_clean":"2024-01-16T14:59:28.703527+0000","last_became_active":"2024-01-16T14:59:27.725643+0000","last_became_peered":"2024-01-16T14:59:27.725643+0000","last_unstale":"2024-01-16T14:59:28.703527+0000","last_undegraded":"2024-01-16T14:59:28.703527+0000","last_fullsized":"2024-01-16T14:59:28.703527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907957+0000","last_change":"2024-01-16T14:59:27.720619+0000","last_active":"2024-01-16T14:59:28.907957+0000","last_peered":"2024-01-16T14:59:28.907957+0000","last_clean":"2024-01-16T14:59:28.907957+0000","last_became_active":"2024-01-16T14:59:27.720348+0000","last_became_peered":"2024-01-16T14:59:27.720348+0000","last_unstale":"2024-01-16T14:59:28.907957+0000","last_undegraded":"2024-01-16T14:59:28.907957+0000","last_fullsized":"2024-01-16T14:59:28.907957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:58.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710390+0000","last_change":"2024-01-16T14:59:27.727783+0000","last_active":"2024-01-16T14:59:28.710390+0000","last_peered":"2024-01-16T14:59:28.710390+0000","last_clean":"2024-01-16T14:59:28.710390+0000","last_became_active":"2024-01-16T14:59:27.727562+0000","last_became_peered":"2024-01-16T14:59:27.727562+0000","last_unstale":"2024-01-16T14:59:28.710390+0000","last_undegraded":"2024-01-16T14:59:28.710390+0000","last_fullsized":"2024-01-16T14:59:28.710390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907372+0000","last_change":"2024-01-16T14:59:27.724755+0000","last_active":"2024-01-16T14:59:28.907372+0000","last_peered":"2024-01-16T14:59:28.907372+0000","last_clean":"2024-01-16T14:59:28.907372+0000","last_became_active":"2024-01-16T14:59:27.724443+0000","last_became_peered":"2024-01-16T14:59:27.724443+0000","last_unstale":"2024-01-16T14:59:28.907372+0000","last_undegraded":"2024-01-16T14:59:28.907372+0000","last_fullsized":"2024-01-16T14:59:28.907372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:06.824346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906082+0000","last_change":"2024-01-16T14:59:27.732620+0000","last_active":"2024-01-16T14:59:28.906082+0000","last_peered":"2024-01-16T14:59:28.906082+0000","last_clean":"2024-01-16T14:59:28.906082+0000","last_became_active":"2024-01-16T14:59:27.732037+0000","last_became_peered":"2024-01-16T14:59:27.732037+0000","last_unstale":"2024-01-16T14:59:28.906082+0000","last_undegraded":"2024-01-16T14:59:28.906082+0000","last_fullsized":"2024-01-16T14:59:28.906082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:13.794976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905598+0000","last_change":"2024-01-16T14:59:27.719539+0000","last_active":"2024-01-16T14:59:28.905598+0000","last_peered":"2024-01-16T14:59:28.905598+0000","last_clean":"2024-01-16T14:59:28.905598+0000","last_became_active":"2024-01-16T14:59:27.719253+0000","last_became_peered":"2024-01-16T14:59:27.719253+0000","last_unstale":"2024-01-16T14:59:28.905598+0000","last_undegraded":"2024-01-16T14:59:28.905598+0000","last_fullsized":"2024-01-16T14:59:28.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:42:03.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905607+0000","last_change":"2024-01-16T14:59:27.715773+0000","last_active":"2024-01-16T14:59:28.905607+0000","last_peered":"2024-01-16T14:59:28.905607+0000","last_clean":"2024-01-16T14:59:28.905607+0000","last_became_active":"2024-01-16T14:59:27.715482+0000","last_became_peered":"2024-01-16T14:59:27.715482+0000","last_unstale":"2024-01-16T14:59:28.905607+0000","last_undegraded":"2024-01-16T14:59:28.905607+0000","last_fullsized":"2024-01-16T14:59:28.905607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906301+0000","last_change":"2024-01-16T14:59:27.726710+0000","last_active":"2024-01-16T14:59:28.906301+0000","last_peered":"2024-01-16T14:59:28.906301+0000","last_clean":"2024-01-16T14:59:28.906301+0000","last_became_active":"2024-01-16T14:59:27.723820+0000","last_became_peered":"2024-01-16T14:59:27.723820+0000","last_unstale":"2024-01-16T14:59:28.906301+0000","last_undegraded":"2024-01-16T14:59:28.906301+0000","last_fullsized":"2024-01-16T14:59:28.906301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:10.938227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711055+0000","last_change":"2024-01-16T14:59:27.726914+0000","last_active":"2024-01-16T14:59:28.711055+0000","last_peered":"2024-01-16T14:59:28.711055+0000","last_clean":"2024-01-16T14:59:28.711055+0000","last_became_active":"2024-01-16T14:59:27.725638+0000","last_became_peered":"2024-01-16T14:59:27.725638+0000","last_unstale":"2024-01-16T14:59:28.711055+0000","last_undegraded":"2024-01-16T14:59:28.711055+0000","last_fullsized":"2024-01-16T14:59:28.711055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907822+0000","last_change":"2024-01-16T14:59:27.703634+0000","last_active":"2024-01-16T14:59:28.907822+0000","last_peered":"2024-01-16T14:59:28.907822+0000","last_clean":"2024-01-16T14:59:28.907822+0000","last_became_active":"2024-01-16T14:59:27.703343+0000","last_became_peered":"2024-01-16T14:59:27.703343+0000","last_unstale":"2024-01-16T14:59:28.907822+0000","last_undegraded":"2024-01-16T14:59:28.907822+0000","last_fullsized":"2024-01-16T14:59:28.907822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906778+0000","last_change":"2024-01-16T14:59:27.724563+0000","last_active":"2024-01-16T14:59:28.906778+0000","last_peered":"2024-01-16T14:59:28.906778+0000","last_clean":"2024-01-16T14:59:28.906778+0000","last_became_active":"2024-01-16T14:59:27.723074+0000","last_became_peered":"2024-01-16T14:59:27.723074+0000","last_unstale":"2024-01-16T14:59:28.906778+0000","last_undegraded":"2024-01-16T14:59:28.906778+0000","last_fullsized":"2024-01-16T14:59:28.906778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:05.783445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907568+0000","last_change":"2024-01-16T14:59:27.724629+0000","last_active":"2024-01-16T14:59:28.907568+0000","last_peered":"2024-01-16T14:59:28.907568+0000","last_clean":"2024-01-16T14:59:28.907568+0000","last_became_active":"2024-01-16T14:59:27.724363+0000","last_became_peered":"2024-01-16T14:59:27.724363+0000","last_unstale":"2024-01-16T14:59:28.907568+0000","last_undegraded":"2024-01-16T14:59:28.907568+0000","last_fullsized":"2024-01-16T14:59:28.907568+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:48.253356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906430+0000","last_change":"2024-01-16T14:59:27.722940+0000","last_active":"2024-01-16T14:59:28.906430+0000","last_peered":"2024-01-16T14:59:28.906430+0000","last_clean":"2024-01-16T14:59:28.906430+0000","last_became_active":"2024-01-16T14:59:27.722567+0000","last_became_peered":"2024-01-16T14:59:27.722567+0000","last_unstale":"2024-01-16T14:59:28.906430+0000","last_undegraded":"2024-01-16T14:59:28.906430+0000","last_fullsized":"2024-01-16T14:59:28.906430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:13.739812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703087+0000","last_change":"2024-01-16T14:59:27.722429+0000","last_active":"2024-01-16T14:59:28.703087+0000","last_peered":"2024-01-16T14:59:28.703087+0000","last_clean":"2024-01-16T14:59:28.703087+0000","last_became_active":"2024-01-16T14:59:27.722092+0000","last_became_peered":"2024-01-16T14:59:27.722092+0000","last_unstale":"2024-01-16T14:59:28.703087+0000","last_undegraded":"2024-01-16T14:59:28.703087+0000","last_fullsized":"2024-01-16T14:59:28.703087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:33.440398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709008+0000","last_change":"2024-01-16T14:59:27.719513+0000","last_active":"2024-01-16T14:59:28.709008+0000","last_peered":"2024-01-16T14:59:28.709008+0000","last_clean":"2024-01-16T14:59:28.709008+0000","last_became_active":"2024-01-16T14:59:27.718757+0000","last_became_peered":"2024-01-16T14:59:27.718757+0000","last_unstale":"2024-01-16T14:59:28.709008+0000","last_undegraded":"2024-01-16T14:59:28.709008+0000","last_fullsized":"2024-01-16T14:59:28.709008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.707752+0000","last_change":"2024-01-16T14:59:27.706529+0000","last_active":"2024-01-16T14:59:28.707752+0000","last_peered":"2024-01-16T14:59:28.707752+0000","last_clean":"2024-01-16T14:59:28.707752+0000","last_became_active":"2024-01-16T14:59:27.706241+0000","last_became_peered":"2024-01-16T14:59:27.706241+0000","last_unstale":"2024-01-16T14:59:28.707752+0000","last_undegraded":"2024-01-16T14:59:28.707752+0000","last_fullsized":"2024-01-16T14:59:28.707752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:40.765365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906601+0000","last_change":"2024-01-16T14:59:26.697929+0000","last_active":"2024-01-16T14:59:28.906601+0000","last_peered":"2024-01-16T14:59:28.906601+0000","last_clean":"2024-01-16T14:59:28.906601+0000","last_became_active":"2024-01-16T14:59:26.697578+0000","last_became_peered":"2024-01-16T14:59:26.697578+0000","last_unstale":"2024-01-16T14:59:28.906601+0000","last_undegraded":"2024-01-16T14:59:28.906601+0000","last_fullsized":"2024-01-16T14:59:28.906601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:26.816846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907687+0000","last_change":"2024-01-16T14:59:27.723311+0000","last_active":"2024-01-16T14:59:28.907687+0000","last_peered":"2024-01-16T14:59:28.907687+0000","last_clean":"2024-01-16T14:59:28.907687+0000","last_became_active":"2024-01-16T14:59:27.722975+0000","last_became_peered":"2024-01-16T14:59:27.722975+0000","last_unstale":"2024-01-16T14:59:28.907687+0000","last_undegraded":"2024-01-16T14:59:28.907687+0000","last_fullsized":"2024-01-16T14:59:28.907687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710186+0000","last_change":"2024-01-16T14:59:26.683247+0000","last_active":"2024-01-16T14:59:28.710186+0000","last_peered":"2024-01-16T14:59:28.710186+0000","last_clean":"2024-01-16T14:59:28.710186+0000","last_became_active":"2024-01-16T14:59:26.682727+0000","last_became_peered":"2024-01-16T14:59:26.682727+0000","last_unstale":"2024-01-16T14:59:28.710186+0000","last_undegraded":"2024-01-16T14:59:28.710186+0000","last_fullsized":"2024-01-16T14:59:28.710186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905763+0000","last_change":"2024-01-16T14:59:26.687905+0000","last_active":"2024-01-16T14:59:28.905763+0000","last_peered":"2024-01-16T14:59:28.905763+0000","last_clean":"2024-01-16T14:59:28.905763+0000","last_became_active":"2024-01-16T14:59:26.687570+0000","last_became_peered":"2024-01-16T14:59:26.687570+0000","last_unstale":"2024-01-16T14:59:28.905763+0000","last_undegraded":"2024-01-16T14:59:28.905763+0000","last_fullsized":"2024-01-16T14:59:28.905763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904948+0000","last_change":"2024-01-16T14:59:27.715786+0000","last_active":"2024-01-16T14:59:28.904948+0000","last_peered":"2024-01-16T14:59:28.904948+0000","last_clean":"2024-01-16T14:59:28.904948+0000","last_became_active":"2024-01-16T14:59:27.715516+0000","last_became_peered":"2024-01-16T14:59:27.715516+0000","last_unstale":"2024-01-16T14:59:28.904948+0000","last_undegraded":"2024-01-16T14:59:28.904948+0000","last_fullsized":"2024-01-16T14:59:28.904948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:14.095920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905832+0000","last_change":"2024-01-16T14:59:26.698294+0000","last_active":"2024-01-16T14:59:28.905832+0000","last_peered":"2024-01-16T14:59:28.905832+0000","last_clean":"2024-01-16T14:59:28.905832+0000","last_became_active":"2024-01-16T14:59:26.697917+0000","last_became_peered":"2024-01-16T14:59:26.697917+0000","last_unstale":"2024-01-16T14:59:28.905832+0000","last_undegraded":"2024-01-16T14:59:28.905832+0000","last_fullsized":"2024-01-16T14:59:28.905832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:57:15.675945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905800+0000","last_change":"2024-01-16T14:59:27.704715+0000","last_active":"2024-01-16T14:59:28.905800+0000","last_peered":"2024-01-16T14:59:28.905800+0000","last_clean":"2024-01-16T14:59:28.905800+0000","last_became_active":"2024-01-16T14:59:27.704439+0000","last_became_peered":"2024-01-16T14:59:27.704439+0000","last_unstale":"2024-01-16T14:59:28.905800+0000","last_undegraded":"2024-01-16T14:59:28.905800+0000","last_fullsized":"2024-01-16T14:59:28.905800+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710792+0000","last_change":"2024-01-16T14:59:26.689740+0000","last_active":"2024-01-16T14:59:28.710792+0000","last_peered":"2024-01-16T14:59:28.710792+0000","last_clean":"2024-01-16T14:59:28.710792+0000","last_became_active":"2024-01-16T14:59:26.689434+0000","last_became_peered":"2024-01-16T14:59:26.689434+0000","last_unstale":"2024-01-16T14:59:28.710792+0000","last_undegraded":"2024-01-16T14:59:28.710792+0000","last_fullsized":"2024-01-16T14:59:28.710792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:39.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907111+0000","last_change":"2024-01-16T14:59:27.733517+0000","last_active":"2024-01-16T14:59:28.907111+0000","last_peered":"2024-01-16T14:59:28.907111+0000","last_clean":"2024-01-16T14:59:28.907111+0000","last_became_active":"2024-01-16T14:59:27.732223+0000","last_became_peered":"2024-01-16T14:59:27.732223+0000","last_unstale":"2024-01-16T14:59:28.907111+0000","last_undegraded":"2024-01-16T14:59:28.907111+0000","last_fullsized":"2024-01-16T14:59:28.907111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:55.788928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907375+0000","last_change":"2024-01-16T14:59:26.693230+0000","last_active":"2024-01-16T14:59:28.907375+0000","last_peered":"2024-01-16T14:59:28.907375+0000","last_clean":"2024-01-16T14:59:28.907375+0000","last_became_active":"2024-01-16T14:59:26.693007+0000","last_became_peered":"2024-01-16T14:59:26.693007+0000","last_unstale":"2024-01-16T14:59:28.907375+0000","last_undegraded":"2024-01-16T14:59:28.907375+0000","last_fullsized":"2024-01-16T14:59:28.907375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909099+0000","last_change":"2024-01-16T14:59:27.723932+0000","last_active":"2024-01-16T14:59:28.909099+0000","last_peered":"2024-01-16T14:59:28.909099+0000","last_clean":"2024-01-16T14:59:28.909099+0000","last_became_active":"2024-01-16T14:59:27.723629+0000","last_became_peered":"2024-01-16T14:59:27.723629+0000","last_unstale":"2024-01-16T14:59:28.909099+0000","last_undegraded":"2024-01-16T14:59:28.909099+0000","last_fullsized":"2024-01-16T14:59:28.909099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:39:05.886211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908417+0000","last_change":"2024-01-16T14:59:26.698500+0000","last_active":"2024-01-16T14:59:28.908417+0000","last_peered":"2024-01-16T14:59:28.908417+0000","last_clean":"2024-01-16T14:59:28.908417+0000","last_became_active":"2024-01-16T14:59:26.698193+0000","last_became_peered":"2024-01-16T14:59:26.698193+0000","last_unstale":"2024-01-16T14:59:28.908417+0000","last_undegraded":"2024-01-16T14:59:28.908417+0000","last_fullsized":"2024-01-16T14:59:28.908417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:32.268892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709922+0000","last_change":"2024-01-16T14:59:27.724449+0000","last_active":"2024-01-16T14:59:28.709922+0000","last_peered":"2024-01-16T14:59:28.709922+0000","last_clean":"2024-01-16T14:59:28.709922+0000","last_became_active":"2024-01-16T14:59:27.723990+0000","last_became_peered":"2024-01-16T14:59:27.723990+0000","last_unstale":"2024-01-16T14:59:28.709922+0000","last_undegraded":"2024-01-16T14:59:28.709922+0000","last_fullsized":"2024-01-16T14:59:28.709922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:15.745270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.710357+0000","last_change":"2024-01-16T14:59:26.688962+0000","last_active":"2024-01-16T14:59:28.710357+0000","last_peered":"2024-01-16T14:59:28.710357+0000","last_clean":"2024-01-16T14:59:28.710357+0000","last_became_active":"2024-01-16T14:59:26.687694+0000","last_became_peered":"2024-01-16T14:59:26.687694+0000","last_unstale":"2024-01-16T14:59:28.710357+0000","last_undegraded":"2024-01-16T14:59:28.710357+0000","last_fullsized":"2024-01-16T14:59:28.710357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907384+0000","last_change":"2024-01-16T14:59:27.718245+0000","last_active":"2024-01-16T14:59:28.907384+0000","last_peered":"2024-01-16T14:59:28.907384+0000","last_clean":"2024-01-16T14:59:28.907384+0000","last_became_active":"2024-01-16T14:59:27.717994+0000","last_became_peered":"2024-01-16T14:59:27.717994+0000","last_unstale":"2024-01-16T14:59:28.907384+0000","last_undegraded":"2024-01-16T14:59:28.907384+0000","last_fullsized":"2024-01-16T14:59:28.907384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:37.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907794+0000","last_change":"2024-01-16T14:59:26.682235+0000","last_active":"2024-01-16T14:59:28.907794+0000","last_peered":"2024-01-16T14:59:28.907794+0000","last_clean":"2024-01-16T14:59:28.907794+0000","last_became_active":"2024-01-16T14:59:26.681912+0000","last_became_peered":"2024-01-16T14:59:26.681912+0000","last_unstale":"2024-01-16T14:59:28.907794+0000","last_undegraded":"2024-01-16T14:59:28.907794+0000","last_fullsized":"2024-01-16T14:59:28.907794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906880+0000","last_change":"2024-01-16T14:59:27.718808+0000","last_active":"2024-01-16T14:59:28.906880+0000","last_peered":"2024-01-16T14:59:28.906880+0000","last_clean":"2024-01-16T14:59:28.906880+0000","last_became_active":"2024-01-16T14:59:27.717611+0000","last_became_peered":"2024-01-16T14:59:27.717611+0000","last_unstale":"2024-01-16T14:59:28.906880+0000","last_undegraded":"2024-01-16T14:59:28.906880+0000","last_fullsized":"2024-01-16T14:59:28.906880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:37.694723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906417+0000","last_change":"2024-01-16T14:59:26.701196+0000","last_active":"2024-01-16T14:59:28.906417+0000","last_peered":"2024-01-16T14:59:28.906417+0000","last_clean":"2024-01-16T14:59:28.906417+0000","last_became_active":"2024-01-16T14:59:26.700600+0000","last_became_peered":"2024-01-16T14:59:26.700600+0000","last_unstale":"2024-01-16T14:59:28.906417+0000","last_undegraded":"2024-01-16T14:59:28.906417+0000","last_fullsized":"2024-01-16T14:59:28.906417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:20.742041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906388+0000","last_change":"2024-01-16T14:59:27.729587+0000","last_active":"2024-01-16T14:59:28.906388+0000","last_peered":"2024-01-16T14:59:28.906388+0000","last_clean":"2024-01-16T14:59:28.906388+0000","last_became_active":"2024-01-16T14:59:27.729374+0000","last_became_peered":"2024-01-16T14:59:27.729374+0000","last_unstale":"2024-01-16T14:59:28.906388+0000","last_undegraded":"2024-01-16T14:59:28.906388+0000","last_fullsized":"2024-01-16T14:59:28.906388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:07.977388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709779+0000","last_change":"2024-01-16T14:59:26.682595+0000","last_active":"2024-01-16T14:59:28.709779+0000","last_peered":"2024-01-16T14:59:28.709779+0000","last_clean":"2024-01-16T14:59:28.709779+0000","last_became_active":"2024-01-16T14:59:26.682263+0000","last_became_peered":"2024-01-16T14:59:26.682263+0000","last_unstale":"2024-01-16T14:59:28.709779+0000","last_undegraded":"2024-01-16T14:59:28.709779+0000","last_fullsized":"2024-01-16T14:59:28.709779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:27:05.652841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709357+0000","last_change":"2024-01-16T14:59:27.721378+0000","last_active":"2024-01-16T14:59:28.709357+0000","last_peered":"2024-01-16T14:59:28.709357+0000","last_clean":"2024-01-16T14:59:28.709357+0000","last_became_active":"2024-01-16T14:59:27.721133+0000","last_became_peered":"2024-01-16T14:59:27.721133+0000","last_unstale":"2024-01-16T14:59:28.709357+0000","last_undegraded":"2024-01-16T14:59:28.709357+0000","last_fullsized":"2024-01-16T14:59:28.709357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:21.913547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703039+0000","last_change":"2024-01-16T14:59:26.688711+0000","last_active":"2024-01-16T14:59:28.703039+0000","last_peered":"2024-01-16T14:59:28.703039+0000","last_clean":"2024-01-16T14:59:28.703039+0000","last_became_active":"2024-01-16T14:59:26.688404+0000","last_became_peered":"2024-01-16T14:59:26.688404+0000","last_unstale":"2024-01-16T14:59:28.703039+0000","last_undegraded":"2024-01-16T14:59:28.703039+0000","last_fullsized":"2024-01-16T14:59:28.703039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904624+0000","last_change":"2024-01-16T14:59:27.724086+0000","last_active":"2024-01-16T14:59:28.904624+0000","last_peered":"2024-01-16T14:59:28.904624+0000","last_clean":"2024-01-16T14:59:28.904624+0000","last_became_active":"2024-01-16T14:59:27.723846+0000","last_became_peered":"2024-01-16T14:59:27.723846+0000","last_unstale":"2024-01-16T14:59:28.904624+0000","last_undegraded":"2024-01-16T14:59:28.904624+0000","last_fullsized":"2024-01-16T14:59:28.904624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.530195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709625+0000","last_change":"2024-01-16T14:59:26.687473+0000","last_active":"2024-01-16T14:59:28.709625+0000","last_peered":"2024-01-16T14:59:28.709625+0000","last_clean":"2024-01-16T14:59:28.709625+0000","last_became_active":"2024-01-16T14:59:26.687161+0000","last_became_peered":"2024-01-16T14:59:26.687161+0000","last_unstale":"2024-01-16T14:59:28.709625+0000","last_undegraded":"2024-01-16T14:59:28.709625+0000","last_fullsized":"2024-01-16T14:59:28.709625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:12.299901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908536+0000","last_change":"2024-01-16T14:59:27.726514+0000","last_active":"2024-01-16T14:59:28.908536+0000","last_peered":"2024-01-16T14:59:28.908536+0000","last_clean":"2024-01-16T14:59:28.908536+0000","last_became_active":"2024-01-16T14:59:27.726196+0000","last_became_peered":"2024-01-16T14:59:27.726196+0000","last_unstale":"2024-01-16T14:59:28.908536+0000","last_undegraded":"2024-01-16T14:59:28.908536+0000","last_fullsized":"2024-01-16T14:59:28.908536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:35.130412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703648+0000","last_change":"2024-01-16T14:59:26.693748+0000","last_active":"2024-01-16T14:59:28.703648+0000","last_peered":"2024-01-16T14:59:28.703648+0000","last_clean":"2024-01-16T14:59:28.703648+0000","last_became_active":"2024-01-16T14:59:26.693411+0000","last_became_peered":"2024-01-16T14:59:26.693411+0000","last_unstale":"2024-01-16T14:59:28.703648+0000","last_undegraded":"2024-01-16T14:59:28.703648+0000","last_fullsized":"2024-01-16T14:59:28.703648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908456+0000","last_change":"2024-01-16T14:59:27.724021+0000","last_active":"2024-01-16T14:59:28.908456+0000","last_peered":"2024-01-16T14:59:28.908456+0000","last_clean":"2024-01-16T14:59:28.908456+0000","last_became_active":"2024-01-16T14:59:27.723760+0000","last_became_peered":"2024-01-16T14:59:27.723760+0000","last_unstale":"2024-01-16T14:59:28.908456+0000","last_undegraded":"2024-01-16T14:59:28.908456+0000","last_fullsized":"2024-01-16T14:59:28.908456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:56.547755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908196+0000","last_change":"2024-01-16T14:59:26.687768+0000","last_active":"2024-01-16T14:59:28.908196+0000","last_peered":"2024-01-16T14:59:28.908196+0000","last_clean":"2024-01-16T14:59:28.908196+0000","last_became_active":"2024-01-16T14:59:26.687439+0000","last_became_peered":"2024-01-16T14:59:26.687439+0000","last_unstale":"2024-01-16T14:59:28.908196+0000","last_undegraded":"2024-01-16T14:59:28.908196+0000","last_fullsized":"2024-01-16T14:59:28.908196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:14.573764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905945+0000","last_change":"2024-01-16T14:59:27.716631+0000","last_active":"2024-01-16T14:59:28.905945+0000","last_peered":"2024-01-16T14:59:28.905945+0000","last_clean":"2024-01-16T14:59:28.905945+0000","last_became_active":"2024-01-16T14:59:27.716370+0000","last_became_peered":"2024-01-16T14:59:27.716370+0000","last_unstale":"2024-01-16T14:59:28.905945+0000","last_undegraded":"2024-01-16T14:59:28.905945+0000","last_fullsized":"2024-01-16T14:59:28.905945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:36:28.471222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703521+0000","last_change":"2024-01-16T14:59:26.695961+0000","last_active":"2024-01-16T14:59:28.703521+0000","last_peered":"2024-01-16T14:59:28.703521+0000","last_clean":"2024-01-16T14:59:28.703521+0000","last_became_active":"2024-01-16T14:59:26.695602+0000","last_became_peered":"2024-01-16T14:59:26.695602+0000","last_unstale":"2024-01-16T14:59:28.703521+0000","last_undegraded":"2024-01-16T14:59:28.703521+0000","last_fullsized":"2024-01-16T14:59:28.703521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:37:07.651834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907463+0000","last_change":"2024-01-16T14:59:27.736089+0000","last_active":"2024-01-16T14:59:28.907463+0000","last_peered":"2024-01-16T14:59:28.907463+0000","last_clean":"2024-01-16T14:59:28.907463+0000","last_became_active":"2024-01-16T14:59:27.731504+0000","last_became_peered":"2024-01-16T14:59:27.731504+0000","last_unstale":"2024-01-16T14:59:28.907463+0000","last_undegraded":"2024-01-16T14:59:28.907463+0000","last_fullsized":"2024-01-16T14:59:28.907463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:20.319731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:27.705910+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T14:59:27.705910+0000","last_peered":"2024-01-16T14:59:27.705910+0000","last_clean":"2024-01-16T14:59:27.705910+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T14:59:27.705910+0000","last_undegraded":"2024-01-16T14:59:27.705910+0000","last_fullsized":"2024-01-16T14:59:27.705910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905174+0000","last_change":"2024-01-16T14:59:26.693490+0000","last_active":"2024-01-16T14:59:28.905174+0000","last_peered":"2024-01-16T14:59:28.905174+0000","last_clean":"2024-01-16T14:59:28.905174+0000","last_became_active":"2024-01-16T14:59:26.692487+0000","last_became_peered":"2024-01-16T14:59:26.692487+0000","last_unstale":"2024-01-16T14:59:28.905174+0000","last_undegraded":"2024-01-16T14:59:28.905174+0000","last_fullsized":"2024-01-16T14:59:28.905174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:52.699344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906140+0000","last_change":"2024-01-16T14:59:27.721477+0000","last_active":"2024-01-16T14:59:28.906140+0000","last_peered":"2024-01-16T14:59:28.906140+0000","last_clean":"2024-01-16T14:59:28.906140+0000","last_became_active":"2024-01-16T14:59:27.720924+0000","last_became_peered":"2024-01-16T14:59:27.720924+0000","last_unstale":"2024-01-16T14:59:28.906140+0000","last_undegraded":"2024-01-16T14:59:28.906140+0000","last_fullsized":"2024-01-16T14:59:28.906140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.326753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904387+0000","last_change":"2024-01-16T14:59:26.701503+0000","last_active":"2024-01-16T14:59:28.904387+0000","last_peered":"2024-01-16T14:59:28.904387+0000","last_clean":"2024-01-16T14:59:28.904387+0000","last_became_active":"2024-01-16T14:59:26.701190+0000","last_became_peered":"2024-01-16T14:59:26.701190+0000","last_unstale":"2024-01-16T14:59:28.904387+0000","last_undegraded":"2024-01-16T14:59:28.904387+0000","last_fullsized":"2024-01-16T14:59:28.904387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:17.349580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906011+0000","last_change":"2024-01-16T14:59:27.728625+0000","last_active":"2024-01-16T14:59:28.906011+0000","last_peered":"2024-01-16T14:59:28.906011+0000","last_clean":"2024-01-16T14:59:28.906011+0000","last_became_active":"2024-01-16T14:59:27.728177+0000","last_became_peered":"2024-01-16T14:59:27.728177+0000","last_unstale":"2024-01-16T14:59:28.906011+0000","last_undegraded":"2024-01-16T14:59:28.906011+0000","last_fullsized":"2024-01-16T14:59:28.906011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:38.465332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.703101+0000","last_change":"2024-01-16T14:59:26.695022+0000","last_active":"2024-01-16T14:59:28.703101+0000","last_peered":"2024-01-16T14:59:28.703101+0000","last_clean":"2024-01-16T14:59:28.703101+0000","last_became_active":"2024-01-16T14:59:26.694756+0000","last_became_peered":"2024-01-16T14:59:26.694756+0000","last_unstale":"2024-01-16T14:59:28.703101+0000","last_undegraded":"2024-01-16T14:59:28.703101+0000","last_fullsized":"2024-01-16T14:59:28.703101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:41.781228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905205+0000","last_change":"2024-01-16T14:59:27.706635+0000","last_active":"2024-01-16T14:59:28.905205+0000","last_peered":"2024-01-16T14:59:28.905205+0000","last_clean":"2024-01-16T14:59:28.905205+0000","last_became_active":"2024-01-16T14:59:27.706296+0000","last_became_peered":"2024-01-16T14:59:27.706296+0000","last_unstale":"2024-01-16T14:59:28.905205+0000","last_undegraded":"2024-01-16T14:59:28.905205+0000","last_fullsized":"2024-01-16T14:59:28.905205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:01.333391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906768+0000","last_change":"2024-01-16T14:59:26.700908+0000","last_active":"2024-01-16T14:59:28.906768+0000","last_peered":"2024-01-16T14:59:28.906768+0000","last_clean":"2024-01-16T14:59:28.906768+0000","last_became_active":"2024-01-16T14:59:26.699693+0000","last_became_peered":"2024-01-16T14:59:26.699693+0000","last_unstale":"2024-01-16T14:59:28.906768+0000","last_undegraded":"2024-01-16T14:59:28.906768+0000","last_fullsized":"2024-01-16T14:59:28.906768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:10.078798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708429+0000","last_change":"2024-01-16T14:59:27.717434+0000","last_active":"2024-01-16T14:59:28.708429+0000","last_peered":"2024-01-16T14:59:28.708429+0000","last_clean":"2024-01-16T14:59:28.708429+0000","last_became_active":"2024-01-16T14:59:27.717061+0000","last_became_peered":"2024-01-16T14:59:27.717061+0000","last_unstale":"2024-01-16T14:59:28.708429+0000","last_undegraded":"2024-01-16T14:59:28.708429+0000","last_fullsized":"2024-01-16T14:59:28.708429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:14.259904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907983+0000","last_change":"2024-01-16T14:59:26.689585+0000","last_active":"2024-01-16T14:59:28.907983+0000","last_peered":"2024-01-16T14:59:28.907983+0000","last_clean":"2024-01-16T14:59:28.907983+0000","last_became_active":"2024-01-16T14:59:26.689193+0000","last_became_peered":"2024-01-16T14:59:26.689193+0000","last_unstale":"2024-01-16T14:59:28.907983+0000","last_undegraded":"2024-01-16T14:59:28.907983+0000","last_fullsized":"2024-01-16T14:59:28.907983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:40.201729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907283+0000","last_change":"2024-01-16T14:59:27.723605+0000","last_active":"2024-01-16T14:59:28.907283+0000","last_peered":"2024-01-16T14:59:28.907283+0000","last_clean":"2024-01-16T14:59:28.907283+0000","last_became_active":"2024-01-16T14:59:27.723322+0000","last_became_peered":"2024-01-16T14:59:27.723322+0000","last_unstale":"2024-01-16T14:59:28.907283+0000","last_undegraded":"2024-01-16T14:59:28.907283+0000","last_fullsized":"2024-01-16T14:59:28.907283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.187834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908060+0000","last_change":"2024-01-16T14:59:26.701967+0000","last_active":"2024-01-16T14:59:28.908060+0000","last_peered":"2024-01-16T14:59:28.908060+0000","last_clean":"2024-01-16T14:59:28.908060+0000","last_became_active":"2024-01-16T14:59:26.701686+0000","last_became_peered":"2024-01-16T14:59:26.701686+0000","last_unstale":"2024-01-16T14:59:28.908060+0000","last_undegraded":"2024-01-16T14:59:28.908060+0000","last_fullsized":"2024-01-16T14:59:28.908060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:09.766721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907757+0000","last_change":"2024-01-16T14:59:27.709576+0000","last_active":"2024-01-16T14:59:28.907757+0000","last_peered":"2024-01-16T14:59:28.907757+0000","last_clean":"2024-01-16T14:59:28.907757+0000","last_became_active":"2024-01-16T14:59:27.706317+0000","last_became_peered":"2024-01-16T14:59:27.706317+0000","last_unstale":"2024-01-16T14:59:28.907757+0000","last_undegraded":"2024-01-16T14:59:28.907757+0000","last_fullsized":"2024-01-16T14:59:28.907757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:29.838805+0000","last_change":"2024-01-16T14:59:26.693739+0000","last_active":"2024-01-16T14:59:29.838805+0000","last_peered":"2024-01-16T14:59:29.838805+0000","last_clean":"2024-01-16T14:59:29.838805+0000","last_became_active":"2024-01-16T14:59:26.693473+0000","last_became_peered":"2024-01-16T14:59:26.693473+0000","last_unstale":"2024-01-16T14:59:29.838805+0000","last_undegraded":"2024-01-16T14:59:29.838805+0000","last_fullsized":"2024-01-16T14:59:29.838805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:47.081850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906445+0000","last_change":"2024-01-16T14:59:27.717414+0000","last_active":"2024-01-16T14:59:28.906445+0000","last_peered":"2024-01-16T14:59:28.906445+0000","last_clean":"2024-01-16T14:59:28.906445+0000","last_became_active":"2024-01-16T14:59:27.717078+0000","last_became_peered":"2024-01-16T14:59:27.717078+0000","last_unstale":"2024-01-16T14:59:28.906445+0000","last_undegraded":"2024-01-16T14:59:28.906445+0000","last_fullsized":"2024-01-16T14:59:28.906445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:30.634017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.711015+0000","last_change":"2024-01-16T14:59:26.696098+0000","last_active":"2024-01-16T14:59:28.711015+0000","last_peered":"2024-01-16T14:59:28.711015+0000","last_clean":"2024-01-16T14:59:28.711015+0000","last_became_active":"2024-01-16T14:59:26.695792+0000","last_became_peered":"2024-01-16T14:59:26.695792+0000","last_unstale":"2024-01-16T14:59:28.711015+0000","last_undegraded":"2024-01-16T14:59:28.711015+0000","last_fullsized":"2024-01-16T14:59:28.711015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:13.088398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908934+0000","last_change":"2024-01-16T14:59:27.712757+0000","last_active":"2024-01-16T14:59:28.908934+0000","last_peered":"2024-01-16T14:59:28.908934+0000","last_clean":"2024-01-16T14:59:28.908934+0000","last_became_active":"2024-01-16T14:59:27.712417+0000","last_became_peered":"2024-01-16T14:59:27.712417+0000","last_unstale":"2024-01-16T14:59:28.908934+0000","last_undegraded":"2024-01-16T14:59:28.908934+0000","last_fullsized":"2024-01-16T14:59:28.908934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:11.346297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.904378+0000","last_change":"2024-01-16T14:59:26.699479+0000","last_active":"2024-01-16T14:59:28.904378+0000","last_peered":"2024-01-16T14:59:28.904378+0000","last_clean":"2024-01-16T14:59:28.904378+0000","last_became_active":"2024-01-16T14:59:26.699195+0000","last_became_peered":"2024-01-16T14:59:26.699195+0000","last_unstale":"2024-01-16T14:59:28.904378+0000","last_undegraded":"2024-01-16T14:59:28.904378+0000","last_fullsized":"2024-01-16T14:59:28.904378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:52.521651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.708137+0000","last_change":"2024-01-16T14:59:27.715745+0000","last_active":"2024-01-16T14:59:28.708137+0000","last_peered":"2024-01-16T14:59:28.708137+0000","last_clean":"2024-01-16T14:59:28.708137+0000","last_became_active":"2024-01-16T14:59:27.710835+0000","last_became_peered":"2024-01-16T14:59:27.710835+0000","last_unstale":"2024-01-16T14:59:28.708137+0000","last_undegraded":"2024-01-16T14:59:28.708137+0000","last_fullsized":"2024-01-16T14:59:28.708137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:53.870850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906409+0000","last_change":"2024-01-16T14:59:26.697228+0000","last_active":"2024-01-16T14:59:28.906409+0000","last_peered":"2024-01-16T14:59:28.906409+0000","last_clean":"2024-01-16T14:59:28.906409+0000","last_became_active":"2024-01-16T14:59:26.696136+0000","last_became_peered":"2024-01-16T14:59:26.696136+0000","last_unstale":"2024-01-16T14:59:28.906409+0000","last_undegraded":"2024-01-16T14:59:28.906409+0000","last_fullsized":"2024-01-16T14:59:28.906409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:37.293826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905459+0000","last_change":"2024-01-16T14:59:27.711533+0000","last_active":"2024-01-16T14:59:28.905459+0000","last_peered":"2024-01-16T14:59:28.905459+0000","last_clean":"2024-01-16T14:59:28.905459+0000","last_became_active":"2024-01-16T14:59:27.711228+0000","last_became_peered":"2024-01-16T14:59:27.711228+0000","last_unstale":"2024-01-16T14:59:28.905459+0000","last_undegraded":"2024-01-16T14:59:28.905459+0000","last_fullsized":"2024-01-16T14:59:28.905459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:28.132533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907464+0000","last_change":"2024-01-16T14:59:26.695937+0000","last_active":"2024-01-16T14:59:28.907464+0000","last_peered":"2024-01-16T14:59:28.907464+0000","last_clean":"2024-01-16T14:59:28.907464+0000","last_became_active":"2024-01-16T14:59:26.695628+0000","last_became_peered":"2024-01-16T14:59:26.695628+0000","last_unstale":"2024-01-16T14:59:28.907464+0000","last_undegraded":"2024-01-16T14:59:28.907464+0000","last_fullsized":"2024-01-16T14:59:28.907464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:36.523217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906350+0000","last_change":"2024-01-16T14:59:27.720565+0000","last_active":"2024-01-16T14:59:28.906350+0000","last_peered":"2024-01-16T14:59:28.906350+0000","last_clean":"2024-01-16T14:59:28.906350+0000","last_became_active":"2024-01-16T14:59:27.716762+0000","last_became_peered":"2024-01-16T14:59:27.716762+0000","last_unstale":"2024-01-16T14:59:28.906350+0000","last_undegraded":"2024-01-16T14:59:28.906350+0000","last_fullsized":"2024-01-16T14:59:28.906350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:26.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906643+0000","last_change":"2024-01-16T14:59:26.700414+0000","last_active":"2024-01-16T14:59:28.906643+0000","last_peered":"2024-01-16T14:59:28.906643+0000","last_clean":"2024-01-16T14:59:28.906643+0000","last_became_active":"2024-01-16T14:59:26.700111+0000","last_became_peered":"2024-01-16T14:59:26.700111+0000","last_unstale":"2024-01-16T14:59:28.906643+0000","last_undegraded":"2024-01-16T14:59:28.906643+0000","last_fullsized":"2024-01-16T14:59:28.906643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:29.462511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.709936+0000","last_change":"2024-01-16T14:59:27.718486+0000","last_active":"2024-01-16T14:59:28.709936+0000","last_peered":"2024-01-16T14:59:28.709936+0000","last_clean":"2024-01-16T14:59:28.709936+0000","last_became_active":"2024-01-16T14:59:27.712753+0000","last_became_peered":"2024-01-16T14:59:27.712753+0000","last_unstale":"2024-01-16T14:59:28.709936+0000","last_undegraded":"2024-01-16T14:59:28.709936+0000","last_fullsized":"2024-01-16T14:59:28.709936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907057+0000","last_change":"2024-01-16T14:59:27.736702+0000","last_active":"2024-01-16T14:59:28.907057+0000","last_peered":"2024-01-16T14:59:28.907057+0000","last_clean":"2024-01-16T14:59:28.907057+0000","last_became_active":"2024-01-16T14:59:27.736433+0000","last_became_peered":"2024-01-16T14:59:27.736433+0000","last_unstale":"2024-01-16T14:59:28.907057+0000","last_undegraded":"2024-01-16T14:59:28.907057+0000","last_fullsized":"2024-01-16T14:59:28.907057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:13.830199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907201+0000","last_change":"2024-01-16T14:59:26.703355+0000","last_active":"2024-01-16T14:59:28.907201+0000","last_peered":"2024-01-16T14:59:28.907201+0000","last_clean":"2024-01-16T14:59:28.907201+0000","last_became_active":"2024-01-16T14:59:26.703059+0000","last_became_peered":"2024-01-16T14:59:26.703059+0000","last_unstale":"2024-01-16T14:59:28.907201+0000","last_undegraded":"2024-01-16T14:59:28.907201+0000","last_fullsized":"2024-01-16T14:59:28.907201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.422714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905630+0000","last_change":"2024-01-16T14:59:27.726539+0000","last_active":"2024-01-16T14:59:28.905630+0000","last_peered":"2024-01-16T14:59:28.905630+0000","last_clean":"2024-01-16T14:59:28.905630+0000","last_became_active":"2024-01-16T14:59:27.726269+0000","last_became_peered":"2024-01-16T14:59:27.726269+0000","last_unstale":"2024-01-16T14:59:28.905630+0000","last_undegraded":"2024-01-16T14:59:28.905630+0000","last_fullsized":"2024-01-16T14:59:28.905630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:02.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.907840+0000","last_change":"2024-01-16T14:59:26.702143+0000","last_active":"2024-01-16T14:59:28.907840+0000","last_peered":"2024-01-16T14:59:28.907840+0000","last_clean":"2024-01-16T14:59:28.907840+0000","last_became_active":"2024-01-16T14:59:26.701732+0000","last_became_peered":"2024-01-16T14:59:26.701732+0000","last_unstale":"2024-01-16T14:59:28.907840+0000","last_undegraded":"2024-01-16T14:59:28.907840+0000","last_fullsized":"2024-01-16T14:59:28.907840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:44.454533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.908894+0000","last_change":"2024-01-16T14:59:27.719650+0000","last_active":"2024-01-16T14:59:28.908894+0000","last_peered":"2024-01-16T14:59:28.908894+0000","last_clean":"2024-01-16T14:59:28.908894+0000","last_became_active":"2024-01-16T14:59:27.719345+0000","last_became_peered":"2024-01-16T14:59:27.719345+0000","last_unstale":"2024-01-16T14:59:28.908894+0000","last_undegraded":"2024-01-16T14:59:28.908894+0000","last_fullsized":"2024-01-16T14:59:28.908894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:11.250304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.906403+0000","last_change":"2024-01-16T14:59:26.692569+0000","last_active":"2024-01-16T14:59:28.906403+0000","last_peered":"2024-01-16T14:59:28.906403+0000","last_clean":"2024-01-16T14:59:28.906403+0000","last_became_active":"2024-01-16T14:59:26.689432+0000","last_became_peered":"2024-01-16T14:59:26.689432+0000","last_unstale":"2024-01-16T14:59:28.906403+0000","last_undegraded":"2024-01-16T14:59:28.906403+0000","last_fullsized":"2024-01-16T14:59:28.906403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:58.385575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.905910+0000","last_change":"2024-01-16T14:59:27.728184+0000","last_active":"2024-01-16T14:59:28.905910+0000","last_peered":"2024-01-16T14:59:28.905910+0000","last_clean":"2024-01-16T14:59:28.905910+0000","last_became_active":"2024-01-16T14:59:27.727889+0000","last_became_peered":"2024-01-16T14:59:27.727889+0000","last_unstale":"2024-01-16T14:59:28.905910+0000","last_undegraded":"2024-01-16T14:59:28.905910+0000","last_fullsized":"2024-01-16T14:59:28.905910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:26.646296+0000","last_clean_scrub_stamp":"2024-01-16T14:59:26.646296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:16.211584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:28.909226+0000","last_change":"2024-01-16T14:59:26.697456+0000","last_active":"2024-01-16T14:59:28.909226+0000","last_peered":"2024-01-16T14:59:28.909226+0000","last_clean":"2024-01-16T14:59:28.909226+0000","last_became_active":"2024-01-16T14:59:26.697129+0000","last_became_peered":"2024-01-16T14:59:26.697129+0000","last_unstale":"2024-01-16T14:59:28.909226+0000","last_undegraded":"2024-01-16T14:59:28.909226+0000","last_fullsized":"2024-01-16T14:59:28.909226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:25.474790+0000","last_clean_scrub_stamp":"2024-01-16T14:59:25.474790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:07.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7849999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0739999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T14:59:38.314 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T14:59:38.314 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T14:59:38.315 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T14:59:38.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T14:59:38.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.586+0000 7f1b20800640 1 Processor -- start 2024-01-16T14:59:38.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.587+0000 7f1b20800640 1 -- start start 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b20800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b18160970 0x7f1b180b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b20800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1b1806dc30 0x7f1b180acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b20800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b18071cd0 con 0x7f1b180b3140 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b20800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b180b2120 con 0x7f1b1806dc30 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b20800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b180b2400 con 0x7f1b18160970 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.588+0000 7f1b1e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1b1806dc30 0x7f1b180acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1b1806dc30 0x7f1b180acbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34448/0 (socket says 172.21.15.43:34448) 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 -- 172.21.15.43:0/971830897 learned_addr learned my addr 172.21.15.43:0/971830897 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:38.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1de00640 1 --2- 172.21.15.43:0/971830897 >> v2:172.21.15.69:3300/0 conn(0x7f1b18160970 0x7f1b180b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 -- 172.21.15.43:0/971830897 >> v2:172.21.15.69:3300/0 conn(0x7f1b18160970 msgr2=0x7f1b180b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 --2- 172.21.15.43:0/971830897 >> v2:172.21.15.69:3300/0 conn(0x7f1b18160970 0x7f1b180b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 -- 172.21.15.43:0/971830897 >> v1:172.21.15.69:6789/0 conn(0x7f1b180b3140 legacy=0x7f1b1806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:38.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.589+0000 7f1b1e800640 1 -- 172.21.15.43:0/971830897 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b180b26e0 con 0x7f1b1806dc30 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.590+0000 7f1b1e800640 1 --2- 172.21.15.43:0/971830897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b1806dc30 0x7f1b180acbe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1b180b5210 tx=0x7f1b1002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d07eeab736e2619 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.590+0000 7f1b1ca00640 1 -- 172.21.15.43:0/971830897 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1002c9f0 con 0x7f1b1806dc30 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.590+0000 7f1b1ca00640 1 -- 172.21.15.43:0/971830897 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1002cb50 con 0x7f1b1806dc30 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.590+0000 7f1b1ca00640 1 -- 172.21.15.43:0/971830897 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b100343f0 con 0x7f1b1806dc30 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 -- 172.21.15.43:0/971830897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b1806dc30 msgr2=0x7f1b180acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 --2- 172.21.15.43:0/971830897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b1806dc30 0x7f1b180acbe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1b180b5210 tx=0x7f1b1002be50 comp rx=0 tx=0).stop 2024-01-16T14:59:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 -- 172.21.15.43:0/971830897 shutdown_connections 2024-01-16T14:59:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 --2- 172.21.15.43:0/971830897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b1806dc30 0x7f1b180acbe0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 --2- 172.21.15.43:0/971830897 >> v2:172.21.15.69:3300/0 conn(0x7f1b18160970 0x7f1b180b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 -- 172.21.15.43:0/971830897 >> 172.21.15.43:0/971830897 conn(0x7f1b180b0d80 msgr2=0x7f1b180b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 -- 172.21.15.43:0/971830897 shutdown_connections 2024-01-16T14:59:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.591+0000 7f1b20800640 1 -- 172.21.15.43:0/971830897 wait complete. 2024-01-16T14:59:38.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.593+0000 7f1b20800640 1 Processor -- start 2024-01-16T14:59:38.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.593+0000 7f1b20800640 1 -- start start 2024-01-16T14:59:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b1806dc30 0x7f1b1809cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b180b3140 0x7f1b18153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18154420 con 0x7f1b18160970 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18158d20 con 0x7f1b1806dc30 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b20800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18159000 con 0x7f1b180b3140 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b1e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b1e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34454/0 (socket says 172.21.15.43:34454) 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b1e800640 1 -- 172.21.15.43:0/1930175480 learned_addr learned my addr 172.21.15.43:0/1930175480 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b1d400640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b180b3140 0x7f1b18153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.595+0000 7f1b1de00640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b1806dc30 0x7f1b1809cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 -- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b180b3140 msgr2=0x7f1b18153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b180b3140 0x7f1b18153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 -- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b1806dc30 msgr2=0x7f1b1809cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b1806dc30 0x7f1b1809cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 -- 172.21.15.43:0/1930175480 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10009b30 con 0x7f1b18160970 2024-01-16T14:59:38.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b1e800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b10036040 tx=0x7f1b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1003e450 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1003e5d0 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b181592e0 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1003e980 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.596+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b181598a0 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.598+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b18159ac0 con 0x7f1b18160970 2024-01-16T14:59:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.599+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1b1003eae0 con 0x7f1b18160970 2024-01-16T14:59:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.600+0000 7f1b16000640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 0x7f1af006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.600+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b100ac9d0 con 0x7f1b18160970 2024-01-16T14:59:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.600+0000 7f1b1de00640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 0x7f1af006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.601+0000 7f1b1de00640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 0x7f1af006d780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.605+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b10077980 con 0x7f1b18160970 2024-01-16T14:59:38.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.827+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1b18074770 con 0x7f1b18160970 2024-01-16T14:59:38.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.828+0000 7f1b16000640 1 -- 172.21.15.43:0/1930175480 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1b1007afa0 con 0x7f1b18160970 2024-01-16T14:59:38.809 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:38.809 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.830+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 msgr2=0x7f1af006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.830+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 0x7f1af006d780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08008040 comp rx=0 tx=0).stop 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.830+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 msgr2=0x7f1b18159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.830+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b10036040 tx=0x7f1b10004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 shutdown_connections 2024-01-16T14:59:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1af006b2c0 0x7f1af006d780 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b18160970 0x7f1b18159dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b180b3140 0x7f1b18153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 --2- 172.21.15.43:0/1930175480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b1806dc30 0x7f1b1809cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 >> 172.21.15.43:0/1930175480 conn(0x7f1b180b0d80 msgr2=0x7f1b1806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 shutdown_connections 2024-01-16T14:59:38.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:38.831+0000 7f1b20800640 1 -- 172.21.15.43:0/1930175480 wait complete. 2024-01-16T14:59:38.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T14:59:38.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:39.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.066+0000 7f2095e00640 1 Processor -- start 2024-01-16T14:59:39.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.066+0000 7f2095e00640 1 -- start start 2024-01-16T14:59:39.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2095e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f209013dfd0 0x7f209015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2095e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 0x7f209006c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2095e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20900ce7c0 con 0x7f20900cf050 2024-01-16T14:59:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2095e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2090140900 con 0x7f209013dfd0 2024-01-16T14:59:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2095e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209015fa70 con 0x7f20900cf500 2024-01-16T14:59:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f2094a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f209013dfd0 0x7f209015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f208fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f20900cf050 0x7f209006c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53074/0 (socket says 172.21.15.43:53074) 2024-01-16T14:59:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.068+0000 7f208fe00640 1 -- 172.21.15.43:0/2351348548 learned_addr learned my addr 172.21.15.43:0/2351348548 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:39.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2094a00640 1 -- 172.21.15.43:0/2351348548 >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 msgr2=0x7f209006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2095400640 1 --2- 172.21.15.43:0/2351348548 >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 0x7f209006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2094a00640 1 --2- 172.21.15.43:0/2351348548 >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 0x7f209006c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f208f400640 1 -- 172.21.15.43:0/2351348548 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383829987 0 0) 0x7f20900ce7c0 con 0x7f20900cf050 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2094a00640 1 -- 172.21.15.43:0/2351348548 >> v1:172.21.15.43:6789/0 conn(0x7f20900cf050 legacy=0x7f209006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2094a00640 1 -- 172.21.15.43:0/2351348548 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209015ee60 con 0x7f209013dfd0 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.069+0000 7f2095400640 1 --2- 172.21.15.43:0/2351348548 >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 0x7f209006c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.070+0000 7f2094a00640 1 --2- 172.21.15.43:0/2351348548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f209015be70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f20901416f0 tx=0x7f208002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84379d002d6cb52a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.070+0000 7f208f400640 1 -- 172.21.15.43:0/2351348548 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080031830 con 0x7f209013dfd0 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.070+0000 7f208f400640 1 -- 172.21.15.43:0/2351348548 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2080031990 con 0x7f209013dfd0 2024-01-16T14:59:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.070+0000 7f208f400640 1 -- 172.21.15.43:0/2351348548 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080031d20 con 0x7f209013dfd0 2024-01-16T14:59:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 -- 172.21.15.43:0/2351348548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 msgr2=0x7f209015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 --2- 172.21.15.43:0/2351348548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f209015be70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f20901416f0 tx=0x7f208002b8d0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 -- 172.21.15.43:0/2351348548 shutdown_connections 2024-01-16T14:59:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 --2- 172.21.15.43:0/2351348548 >> v2:172.21.15.69:3300/0 conn(0x7f20900cf500 0x7f209006c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 --2- 172.21.15.43:0/2351348548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f209015be70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 -- 172.21.15.43:0/2351348548 >> 172.21.15.43:0/2351348548 conn(0x7f209006a860 msgr2=0x7f209006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 -- 172.21.15.43:0/2351348548 shutdown_connections 2024-01-16T14:59:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.071+0000 7f2095e00640 1 -- 172.21.15.43:0/2351348548 wait complete. 2024-01-16T14:59:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.073+0000 7f2095e00640 1 Processor -- start 2024-01-16T14:59:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.073+0000 7f2095e00640 1 -- start start 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900cf050 0x7f209013b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 0x7f20901f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20900b2390 con 0x7f209013dfd0 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20900b2670 con 0x7f20900cf500 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20900b2950 con 0x7f20900cf050 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34480/0 (socket says 172.21.15.43:34480) 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2095400640 1 -- 172.21.15.43:0/1344086494 learned_addr learned my addr 172.21.15.43:0/1344086494 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f208fe00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 0x7f20901f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.074+0000 7f2094a00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900cf050 0x7f209013b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 -- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900cf050 msgr2=0x7f209013b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900cf050 0x7f209013b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 -- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 msgr2=0x7f20901f09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 0x7f20901f09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 -- 172.21.15.43:0/1344086494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2080009400 con 0x7f209013dfd0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f208fe00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 0x7f20901f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095400640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f208800cef0 tx=0x7f208800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208800d840 con 0x7f209013dfd0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208800d9c0 con 0x7f209013dfd0 2024-01-16T14:59:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208800dd70 con 0x7f209013dfd0 2024-01-16T14:59:39.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20900b2c30 con 0x7f209013dfd0 2024-01-16T14:59:39.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.075+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209013bab0 con 0x7f209013dfd0 2024-01-16T14:59:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.077+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209004e840 con 0x7f209013dfd0 2024-01-16T14:59:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.077+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f208801e3b0 con 0x7f209013dfd0 2024-01-16T14:59:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.078+0000 7f208cc00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 0x7f206806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.078+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f208808a5e0 con 0x7f209013dfd0 2024-01-16T14:59:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.079+0000 7f2094a00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 0x7f206806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.079+0000 7f2094a00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 0x7f206806d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2080036000 tx=0x7f208003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.083+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20880555a0 con 0x7f209013dfd0 2024-01-16T14:59:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.277+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20900ce9a0 con 0x7f209013dfd0 2024-01-16T14:59:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.280+0000 7f208cc00640 1 -- 172.21.15.43:0/1344086494 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f2088058bc0 con 0x7f209013dfd0 2024-01-16T14:59:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:39.261 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.282+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 msgr2=0x7f206806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.282+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 0x7f206806d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2080036000 tx=0x7f208003c040 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.282+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 msgr2=0x7f2090154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.282+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f208800cef0 tx=0x7f208800a710 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 shutdown_connections 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f206806b2c0 0x7f206806d780 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900cf500 0x7f20901f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f209013dfd0 0x7f2090154050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 --2- 172.21.15.43:0/1344086494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900cf050 0x7f209013b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 >> 172.21.15.43:0/1344086494 conn(0x7f209006a860 msgr2=0x7f209015a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 shutdown_connections 2024-01-16T14:59:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.283+0000 7f2095e00640 1 -- 172.21.15.43:0/1344086494 wait complete. 2024-01-16T14:59:39.284 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:39.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:39.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.529+0000 7efc4b400640 1 Processor -- start 2024-01-16T14:59:39.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.529+0000 7efc4b400640 1 -- start start 2024-01-16T14:59:39.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc4b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc440abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc4b400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4415b730 con 0x7efc4415b350 2024-01-16T14:59:39.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc4b400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc440ac740 con 0x7efc4409cb30 2024-01-16T14:59:39.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc4b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc440ac460 con 0x7efc440ae270 2024-01-16T14:59:39.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc48a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efc4415b350 0x7efc440b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53078/0 (socket says 172.21.15.43:53078) 2024-01-16T14:59:39.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc48a00640 1 -- 172.21.15.43:0/184670343 learned_addr learned my addr 172.21.15.43:0/184670343 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:39.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc49400640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc440abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485263753 0 0) 0x7efc34001110 con 0x7efc4415b350 2024-01-16T14:59:39.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.531+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc24003520 con 0x7efc4415b350 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591657114 0 0) 0x7efc24003520 con 0x7efc4415b350 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 msgr2=0x7efc440abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc440abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 >> v1:172.21.15.69:6789/0 conn(0x7efc4409cb30 legacy=0x7efc440adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440b2a50 con 0x7efc4415b350 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7efc34001800 con 0x7efc4415b350 2024-01-16T14:59:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 >> v1:172.21.15.43:6789/0 conn(0x7efc4415b350 legacy=0x7efc440b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc24003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc24004d00 0x7efc24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc440ac460 con 0x7efc240039a0 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2400b980 con 0x7efc24004d00 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2400bb40 con 0x7efc440ae270 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc48a00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc24003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.532+0000 7efc49400640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc24004d00 0x7efc24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc3bc00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc48a00640 1 -- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 msgr2=0x7efc240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc48a00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc48a00640 1 -- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc24004d00 msgr2=0x7efc24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc48a00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc24004d00 0x7efc24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc48a00640 1 -- 172.21.15.43:0/184670343 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc24003520 con 0x7efc240039a0 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.533+0000 7efc3bc00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc440ae270 0x7efc240047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.534+0000 7efc48a00640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc24003d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c240 tx=0x7efc240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e075fe5746c70b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.534+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34005ad0 con 0x7efc240039a0 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.534+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc34005c50 con 0x7efc240039a0 2024-01-16T14:59:39.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.534+0000 7efc43e00640 1 -- 172.21.15.43:0/184670343 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3402c810 con 0x7efc240039a0 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 -- 172.21.15.43:0/184670343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 msgr2=0x7efc24003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 --2- 172.21.15.43:0/184670343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc24003d80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efc3400c240 tx=0x7efc240057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc49400640 1 -- 172.21.15.43:0/184670343 reap_dead start 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 -- 172.21.15.43:0/184670343 shutdown_connections 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 -- 172.21.15.43:0/184670343 >> 172.21.15.43:0/184670343 conn(0x7efc440b0ba0 msgr2=0x7efc440b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 -- 172.21.15.43:0/184670343 shutdown_connections 2024-01-16T14:59:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.535+0000 7efc4b400640 1 -- 172.21.15.43:0/184670343 wait complete. 2024-01-16T14:59:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.537+0000 7efc4b400640 1 Processor -- start 2024-01-16T14:59:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.537+0000 7efc4b400640 1 -- start start 2024-01-16T14:59:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4414bfc0 0x7efc44151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc4414c440 0x7efc441518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc441491b0 con 0x7efc240039a0 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4414b8b0 con 0x7efc4414c440 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc4b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4414a810 con 0x7efc4414bfc0 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc48a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc48a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34500/0 (socket says 172.21.15.43:34500) 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.538+0000 7efc48a00640 1 -- 172.21.15.43:0/1049583423 learned_addr learned my addr 172.21.15.43:0/1049583423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc49400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc4414c440 0x7efc441518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc3bc00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4414bfc0 0x7efc44151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 -- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4414bfc0 msgr2=0x7efc44151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4414bfc0 0x7efc44151390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 -- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc4414c440 msgr2=0x7efc441518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc4414c440 0x7efc441518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 -- 172.21.15.43:0/1049583423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4414aaf0 con 0x7efc240039a0 2024-01-16T14:59:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc48a00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc3402c3b0 tx=0x7efc34001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34039070 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3402cc20 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3403c040 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.539+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4414add0 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.540+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4414b390 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.541+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4414b520 con 0x7efc240039a0 2024-01-16T14:59:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.542+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efc34032070 con 0x7efc240039a0 2024-01-16T14:59:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.543+0000 7efc41600640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 0x7efc1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.543+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc340aace0 con 0x7efc240039a0 2024-01-16T14:59:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.543+0000 7efc3bc00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 0x7efc1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.544+0000 7efc3bc00640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 0x7efc1c06d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc2c001270 tx=0x7efc2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.548+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc34075d00 con 0x7efc240039a0 2024-01-16T14:59:39.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.742+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc440acbf0 con 0x7efc240039a0 2024-01-16T14:59:39.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.742+0000 7efc41600640 1 -- 172.21.15.43:0/1049583423 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc34079320 con 0x7efc240039a0 2024-01-16T14:59:39.724 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T14:59:39.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 msgr2=0x7efc1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 0x7efc1c06d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc2c001270 tx=0x7efc2c009040 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 msgr2=0x7efc44148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc3402c3b0 tx=0x7efc34001b90 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 shutdown_connections 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc1c06b2c0 0x7efc1c06d780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc4414c440 0x7efc441518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4414bfc0 0x7efc44151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 --2- 172.21.15.43:0/1049583423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc240039a0 0x7efc44148c70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 >> 172.21.15.43:0/1049583423 conn(0x7efc440b0ba0 msgr2=0x7efc44159840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 shutdown_connections 2024-01-16T14:59:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.745+0000 7efc4b400640 1 -- 172.21.15.43:0/1049583423 wait complete. 2024-01-16T14:59:39.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:59:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.983+0000 7f466f200640 1 Processor -- start 2024-01-16T14:59:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.983+0000 7f466f200640 1 -- start start 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f466815f270 0x7f46680cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466806d9e0 0x7f46680cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466815f930 con 0x7f4668144770 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46680b43b0 con 0x7f466815f270 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680b3b10 con 0x7f466806d9e0 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f466815f270 0x7f46680cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f466815f270 0x7f46680cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34510/0 (socket says 172.21.15.43:34510) 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.985+0000 7f466d400640 1 -- 172.21.15.43:0/1990707090 learned_addr learned my addr 172.21.15.43:0/1990707090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.986+0000 7f466e800640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466806d9e0 0x7f46680cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.986+0000 7f466d400640 1 -- 172.21.15.43:0/1990707090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466806d9e0 msgr2=0x7f46680cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.986+0000 7f466d400640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466806d9e0 0x7f46680cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.986+0000 7f466d400640 1 -- 172.21.15.43:0/1990707090 >> v1:172.21.15.69:6789/0 conn(0x7f4668144770 legacy=0x7f466815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.986+0000 7f466d400640 1 -- 172.21.15.43:0/1990707090 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680b3df0 con 0x7f466815f270 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.987+0000 7f466d400640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466815f270 0x7f46680cd4a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4650009870 tx=0x7f465002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d7c9a6fc9386fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.987+0000 7f466ca00640 1 -- 172.21.15.43:0/1990707090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650030840 con 0x7f466815f270 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.987+0000 7f466ca00640 1 -- 172.21.15.43:0/1990707090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46500309a0 con 0x7f466815f270 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.987+0000 7f466ca00640 1 -- 172.21.15.43:0/1990707090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650030d30 con 0x7f466815f270 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 -- 172.21.15.43:0/1990707090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466815f270 msgr2=0x7f46680cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466815f270 0x7f46680cd4a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4650009870 tx=0x7f465002b980 comp rx=0 tx=0).stop 2024-01-16T14:59:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 -- 172.21.15.43:0/1990707090 shutdown_connections 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466815f270 0x7f46680cd4a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 --2- 172.21.15.43:0/1990707090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466806d9e0 0x7f46680cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 -- 172.21.15.43:0/1990707090 >> 172.21.15.43:0/1990707090 conn(0x7f46680cec10 msgr2=0x7f46680cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 -- 172.21.15.43:0/1990707090 shutdown_connections 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.988+0000 7f466f200640 1 -- 172.21.15.43:0/1990707090 wait complete. 2024-01-16T14:59:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.990+0000 7f466f200640 1 Processor -- start 2024-01-16T14:59:39.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.990+0000 7f466f200640 1 -- start start 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4668144770 0x7f466813b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466815f270 0x7f466813b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814d280 con 0x7f466806d9e0 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466815c310 con 0x7f4668144770 2024-01-16T14:59:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466815c560 con 0x7f466815f270 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34512/0 (socket says 172.21.15.43:34512) 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.991+0000 7f466de00640 1 -- 172.21.15.43:0/229239281 learned_addr learned my addr 172.21.15.43:0/229239281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466e800640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466815f270 0x7f466813b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466d400640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4668144770 0x7f466813b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 -- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466815f270 msgr2=0x7f466813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466815f270 0x7f466813b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 -- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4668144770 msgr2=0x7f466813b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4668144770 0x7f466813b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 -- 172.21.15.43:0/229239281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500094d0 con 0x7f466806d9e0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466de00640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f465800c450 tx=0x7f465800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580052d0 con 0x7f466806d9e0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4658005430 con 0x7f466806d9e0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.992+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466815c840 con 0x7f466806d9e0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.993+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658012870 con 0x7f466806d9e0 2024-01-16T14:59:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.993+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466815ce00 con 0x7f466806d9e0 2024-01-16T14:59:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.994+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466804e840 con 0x7f466806d9e0 2024-01-16T14:59:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.995+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f46580129d0 con 0x7f466806d9e0 2024-01-16T14:59:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.996+0000 7f4666000640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 0x7f464006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.996+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46580896d0 con 0x7f466806d9e0 2024-01-16T14:59:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.996+0000 7f466d400640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 0x7f464006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:39.997+0000 7f466d400640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 0x7f464006d7b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4650036000 tx=0x7f465003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:39.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.000+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4658054540 con 0x7f466806d9e0 2024-01-16T14:59:40.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.199+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46680b2460 con 0x7f466806d9e0 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.200+0000 7f4666000640 1 -- 172.21.15.43:0/229239281 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4658057b60 con 0x7f466806d9e0 2024-01-16T14:59:40.181 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 msgr2=0x7f464006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 0x7f464006d7b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4650036000 tx=0x7f465003b040 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 msgr2=0x7f466813a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f465800c450 tx=0x7f465800c910 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 shutdown_connections 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f464006b2f0 0x7f464006d7b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4668144770 0x7f466813b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f466815f270 0x7f466813b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 --2- 172.21.15.43:0/229239281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f466806d9e0 0x7f466813a8a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 >> 172.21.15.43:0/229239281 conn(0x7f46680cec10 msgr2=0x7f4668143260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.203+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 shutdown_connections 2024-01-16T14:59:40.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.204+0000 7f466f200640 1 -- 172.21.15.43:0/229239281 wait complete. 2024-01-16T14:59:40.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:59:40.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.444+0000 7fc3a3600640 1 Processor -- start 2024-01-16T14:59:40.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.445+0000 7fc3a3600640 1 -- start start 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.446+0000 7fc3a3600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c15b460 0x7fc39c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.446+0000 7fc3a3600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc39c06ab50 0x7fc39c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.446+0000 7fc3a3600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c15fe20 con 0x7fc39c072530 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.446+0000 7fc3a3600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39c15b840 con 0x7fc39c06ab50 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.446+0000 7fc3a3600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c15f060 con 0x7fc39c15b460 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc39c06ab50 0x7fc39c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc39c06ab50 0x7fc39c0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34520/0 (socket says 172.21.15.43:34520) 2024-01-16T14:59:40.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 -- 172.21.15.43:0/120243457 learned_addr learned my addr 172.21.15.43:0/120243457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a0c00640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c15b460 0x7fc39c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 -- 172.21.15.43:0/120243457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c15b460 msgr2=0x7fc39c06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c15b460 0x7fc39c06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 -- 172.21.15.43:0/120243457 >> v1:172.21.15.69:6789/0 conn(0x7fc39c072530 legacy=0x7fc39c06a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.447+0000 7fc3a1600640 1 -- 172.21.15.43:0/120243457 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c15f340 con 0x7fc39c06ab50 2024-01-16T14:59:40.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.448+0000 7fc3a1600640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c06ab50 0x7fc39c0ce040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc3940098a0 tx=0x7fc39402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524cfb8d2efbbc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc39be00640 1 -- 172.21.15.43:0/120243457 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394030840 con 0x7fc39c06ab50 2024-01-16T14:59:40.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc39be00640 1 -- 172.21.15.43:0/120243457 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3940309a0 con 0x7fc39c06ab50 2024-01-16T14:59:40.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc39be00640 1 -- 172.21.15.43:0/120243457 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394030d30 con 0x7fc39c06ab50 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 -- 172.21.15.43:0/120243457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c06ab50 msgr2=0x7fc39c0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c06ab50 0x7fc39c0ce040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc3940098a0 tx=0x7fc39402b980 comp rx=0 tx=0).stop 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 -- 172.21.15.43:0/120243457 shutdown_connections 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c06ab50 0x7fc39c0ce040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 --2- 172.21.15.43:0/120243457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c15b460 0x7fc39c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.449+0000 7fc3a3600640 1 -- 172.21.15.43:0/120243457 >> 172.21.15.43:0/120243457 conn(0x7fc39c0b1c90 msgr2=0x7fc39c0b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.450+0000 7fc3a3600640 1 -- 172.21.15.43:0/120243457 shutdown_connections 2024-01-16T14:59:40.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.450+0000 7fc3a3600640 1 -- 172.21.15.43:0/120243457 wait complete. 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.451+0000 7fc3a3600640 1 Processor -- start 2024-01-16T14:59:40.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.452+0000 7fc3a3600640 1 -- start start 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.452+0000 7fc3a3600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c06ab50 0x7fc39c15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a3600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a3600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39c15b460 0x7fc39c0cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a3600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c0c63b0 con 0x7fc39c072530 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a3600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c1f3e50 con 0x7fc39c15b460 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a3600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c15ac30 con 0x7fc39c06ab50 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34536/0 (socket says 172.21.15.43:34536) 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 -- 172.21.15.43:0/3171414321 learned_addr learned my addr 172.21.15.43:0/3171414321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a1600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39c15b460 0x7fc39c0cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc3a0c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c06ab50 0x7fc39c15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 -- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c06ab50 msgr2=0x7fc39c15a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c06ab50 0x7fc39c15a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 -- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39c15b460 msgr2=0x7fc39c0cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39c15b460 0x7fc39c0cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.453+0000 7fc393c00640 1 -- 172.21.15.43:0/3171414321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3940094d0 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc393c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c8a0 tx=0x7fc38400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384003b70 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384003cf0 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38400aa30 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c1f3240 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.454+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c1f3800 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.455+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39c04e920 con 0x7fc39c072530 2024-01-16T14:59:40.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.456+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc38400ab90 con 0x7fc39c072530 2024-01-16T14:59:40.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.457+0000 7fc399600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 0x7fc37406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.457+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc384089a20 con 0x7fc39c072530 2024-01-16T14:59:40.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.457+0000 7fc3a0c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 0x7fc37406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.458+0000 7fc3a0c00640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 0x7fc37406d7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc38c006fd0 tx=0x7fc38c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.461+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc384054890 con 0x7fc39c072530 2024-01-16T14:59:40.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.662+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc39c0718b0 con 0x7fc39c072530 2024-01-16T14:59:40.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.663+0000 7fc399600640 1 -- 172.21.15.43:0/3171414321 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc384003e60 con 0x7fc39c072530 2024-01-16T14:59:40.644 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T14:59:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 msgr2=0x7fc37406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 0x7fc37406d7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc38c006fd0 tx=0x7fc38c008040 comp rx=0 tx=0).stop 2024-01-16T14:59:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 msgr2=0x7fc39c0c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c8a0 tx=0x7fc38400cd60 comp rx=0 tx=0).stop 2024-01-16T14:59:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 shutdown_connections 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc37406b2f0 0x7fc37406d7b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39c15b460 0x7fc39c0cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc39c072530 0x7fc39c0c68a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 --2- 172.21.15.43:0/3171414321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39c06ab50 0x7fc39c15a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.665+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 >> 172.21.15.43:0/3171414321 conn(0x7fc39c0b1c90 msgr2=0x7fc39c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.666+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 shutdown_connections 2024-01-16T14:59:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.666+0000 7fc3a3600640 1 -- 172.21.15.43:0/3171414321 wait complete. 2024-01-16T14:59:40.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.908+0000 7f4c66e00640 1 Processor -- start 2024-01-16T14:59:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.908+0000 7f4c66e00640 1 -- start start 2024-01-16T14:59:40.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c66e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c6006d520 0x7f4c6006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c66e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c600ad720 0x7f4c6015ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c66e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6015c360 con 0x7f4c600abff0 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c66e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600b41a0 con 0x7f4c6006d520 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c66e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6006de40 con 0x7f4c600ad720 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c5fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c6006d520 0x7f4c6006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c5fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c6006d520 0x7f4c6006d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34540/0 (socket says 172.21.15.43:34540) 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.910+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3248430729 learned_addr learned my addr 172.21.15.43:0/3248430729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5f400640 1 --2- 172.21.15.43:0/3248430729 >> v2:172.21.15.69:3300/0 conn(0x7f4c600ad720 0x7f4c6015ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3248430729 >> v2:172.21.15.69:3300/0 conn(0x7f4c600ad720 msgr2=0x7f4c6015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/3248430729 >> v2:172.21.15.69:3300/0 conn(0x7f4c600ad720 0x7f4c6015ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3248430729 >> v1:172.21.15.43:6789/0 conn(0x7f4c600abff0 legacy=0x7f4c6015f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3248430729 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6015ba30 con 0x7f4c6006d520 2024-01-16T14:59:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.911+0000 7f4c5ea00640 1 -- 172.21.15.43:0/3248430729 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431525649 0 0) 0x7f4c6015c360 con 0x7f4c600abff0 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.912+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/3248430729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6006d520 0x7f4c6006d900 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4c600adb00 tx=0x7f4c5002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d743d1cbf0da78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.912+0000 7f4c5ea00640 1 -- 172.21.15.43:0/3248430729 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50030860 con 0x7f4c6006d520 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.912+0000 7f4c5ea00640 1 -- 172.21.15.43:0/3248430729 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c500309c0 con 0x7f4c6006d520 2024-01-16T14:59:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.912+0000 7f4c5ea00640 1 -- 172.21.15.43:0/3248430729 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50030d50 con 0x7f4c6006d520 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 -- 172.21.15.43:0/3248430729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6006d520 msgr2=0x7f4c6006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 --2- 172.21.15.43:0/3248430729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6006d520 0x7f4c6006d900 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4c600adb00 tx=0x7f4c5002b9a0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 -- 172.21.15.43:0/3248430729 shutdown_connections 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 --2- 172.21.15.43:0/3248430729 >> v2:172.21.15.69:3300/0 conn(0x7f4c600ad720 0x7f4c6015ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 --2- 172.21.15.43:0/3248430729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6006d520 0x7f4c6006d900 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 -- 172.21.15.43:0/3248430729 >> 172.21.15.43:0/3248430729 conn(0x7f4c600b0d80 msgr2=0x7f4c600b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 -- 172.21.15.43:0/3248430729 shutdown_connections 2024-01-16T14:59:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.913+0000 7f4c66e00640 1 -- 172.21.15.43:0/3248430729 wait complete. 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.915+0000 7f4c66e00640 1 Processor -- start 2024-01-16T14:59:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.915+0000 7f4c66e00640 1 -- start start 2024-01-16T14:59:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6006d520 0x7f4c601493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 0x7f4c600ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600c9440 con 0x7f4c600ad720 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600c9720 con 0x7f4c600abff0 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c66e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600c9a00 con 0x7f4c6006d520 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c64e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c64e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34546/0 (socket says 172.21.15.43:34546) 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c64e00640 1 -- 172.21.15.43:0/698049034 learned_addr learned my addr 172.21.15.43:0/698049034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.916+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6006d520 0x7f4c601493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c5f400640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 0x7f4c600ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 -- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6006d520 msgr2=0x7f4c601493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6006d520 0x7f4c601493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 -- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 msgr2=0x7f4c600ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 0x7f4c600ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 -- 172.21.15.43:0/698049034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500094d0 con 0x7f4c600ad720 2024-01-16T14:59:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c5f400640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 0x7f4c600ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c64e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c7c0 tx=0x7f4c5800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58003da0 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58005070 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58005420 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c600c9ce0 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.917+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c600c88a0 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.919+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c60074770 con 0x7f4c600ad720 2024-01-16T14:59:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.920+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4c58005580 con 0x7f4c600ad720 2024-01-16T14:59:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.920+0000 7f4c57e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 0x7f4c3806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.921+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c58089500 con 0x7f4c600ad720 2024-01-16T14:59:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.921+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 0x7f4c3806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.921+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 0x7f4c3806d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c50036000 tx=0x7f4c50008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:40.925+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c580542d0 con 0x7f4c600ad720 2024-01-16T14:59:41.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.157+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c6015fbe0 con 0x7f4c600ad720 2024-01-16T14:59:41.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.158+0000 7f4c57e00640 1 -- 172.21.15.43:0/698049034 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f4c580578f0 con 0x7f4c600ad720 2024-01-16T14:59:41.139 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:41.139 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:59:41.139 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.160+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 msgr2=0x7f4c3806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.160+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 0x7f4c3806d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c50036000 tx=0x7f4c50008030 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.160+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 msgr2=0x7f4c600caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.160+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c7c0 tx=0x7f4c5800cc80 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 shutdown_connections 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806b2c0 0x7f4c3806d780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600abff0 0x7f4c600ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c600ad720 0x7f4c600caae0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 --2- 172.21.15.43:0/698049034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6006d520 0x7f4c601493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 >> 172.21.15.43:0/698049034 conn(0x7f4c600b0d80 msgr2=0x7f4c6015ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 shutdown_connections 2024-01-16T14:59:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.161+0000 7f4c66e00640 1 -- 172.21.15.43:0/698049034 wait complete. 2024-01-16T14:59:41.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.389+0000 7f6fd7200640 1 Processor -- start 2024-01-16T14:59:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.390+0000 7f6fd7200640 1 -- start start 2024-01-16T14:59:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.391+0000 7f6fd7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fd006d520 0x7f6fd006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.391+0000 7f6fd7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fd015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.391+0000 7f6fd7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd015c360 con 0x7f6fd00ad720 2024-01-16T14:59:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.391+0000 7f6fd7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00b41a0 con 0x7f6fd006d520 2024-01-16T14:59:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.391+0000 7f6fd7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd006de40 con 0x7f6fd00abff0 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fc7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6fd00ad720 0x7f6fd015ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53104/0 (socket says 172.21.15.43:53104) 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fc7e00640 1 -- 172.21.15.43:0/1012692624 learned_addr learned my addr 172.21.15.43:0/1012692624 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fd5200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fd015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/1012692624 >> v2:172.21.15.69:3300/0 conn(0x7f6fd006d520 0x7f6fd006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107071811 0 0) 0x7f6fd015c360 con 0x7f6fd00ad720 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.392+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb0003520 con 0x7f6fd00ad720 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899703188 0 0) 0x7f6fb0003520 con 0x7f6fd00ad720 2024-01-16T14:59:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 msgr2=0x7f6fd015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fd015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 >> v2:172.21.15.69:3300/0 conn(0x7f6fd006d520 msgr2=0x7f6fd006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 --2- 172.21.15.43:0/1012692624 >> v2:172.21.15.69:3300/0 conn(0x7f6fd006d520 0x7f6fd006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd015ba30 con 0x7f6fd00ad720 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6fb8001670 con 0x7f6fd00ad720 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 >> v1:172.21.15.43:6789/0 conn(0x7f6fd00ad720 legacy=0x7f6fd015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0003970 0x7f6fb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fb0004d00 0x7f6fb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd006de40 con 0x7f6fb0004d00 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb000b980 con 0x7f6fb0003970 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb000bac0 con 0x7f6fd00abff0 2024-01-16T14:59:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fd5200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fb0004d00 0x7f6fb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.393+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0003970 0x7f6fb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fd5200640 1 -- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 msgr2=0x7f6fb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fd5200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00abff0 0x7f6fb00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fd5200640 1 -- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0003970 msgr2=0x7f6fb0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fd5200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0003970 0x7f6fb0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fd5200640 1 -- 172.21.15.43:0/1012692624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0003520 con 0x7f6fb0004d00 2024-01-16T14:59:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.394+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0003970 0x7f6fb0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.395+0000 7f6fd5200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fb0004d00 0x7f6fb0005180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009f90 tx=0x7f6fb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5011ca033f854698 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.395+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc802cbd0 con 0x7f6fb0004d00 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.395+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc800f040 con 0x7f6fb0004d00 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.395+0000 7f6fcf400640 1 -- 172.21.15.43:0/1012692624 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8034440 con 0x7f6fb0004d00 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 -- 172.21.15.43:0/1012692624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fb0004d00 msgr2=0x7f6fb0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 --2- 172.21.15.43:0/1012692624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fb0004d00 0x7f6fb0005180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009f90 tx=0x7f6fb00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd5200640 1 -- 172.21.15.43:0/1012692624 reap_dead start 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 -- 172.21.15.43:0/1012692624 shutdown_connections 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 -- 172.21.15.43:0/1012692624 >> 172.21.15.43:0/1012692624 conn(0x7f6fd00b0d80 msgr2=0x7f6fd00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 -- 172.21.15.43:0/1012692624 shutdown_connections 2024-01-16T14:59:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.396+0000 7f6fd7200640 1 -- 172.21.15.43:0/1012692624 wait complete. 2024-01-16T14:59:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.398+0000 7f6fd7200640 1 Processor -- start 2024-01-16T14:59:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.398+0000 7f6fd7200640 1 -- start start 2024-01-16T14:59:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 0x7f6fd0146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00cd240 0x7f6fd009a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00cd6c0 con 0x7f6fd009c070 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01469c0 con 0x7f6fb0004d00 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fd7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd009afd0 con 0x7f6fd00cd240 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34560/0 (socket says 172.21.15.43:34560) 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.399+0000 7f6fc7e00640 1 -- 172.21.15.43:0/2809957115 learned_addr learned my addr 172.21.15.43:0/2809957115 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fd5200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00cd240 0x7f6fd009a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 0x7f6fd0146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 -- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00cd240 msgr2=0x7f6fd009a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00cd240 0x7f6fd009a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 -- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 msgr2=0x7f6fd0146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 0x7f6fd0146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 -- 172.21.15.43:0/2809957115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd009b2b0 con 0x7f6fd009c070 2024-01-16T14:59:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 0x7f6fd0146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800e780 tx=0x7f6fb800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb800ee90 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb800c870 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb800cc00 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.400+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd009b590 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.401+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd009bb50 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.402+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0074770 con 0x7f6fd009c070 2024-01-16T14:59:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.403+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6fb801d460 con 0x7f6fd009c070 2024-01-16T14:59:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.404+0000 7f6fccc00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 0x7f6fa806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.404+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fb8089cc0 con 0x7f6fd009c070 2024-01-16T14:59:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.404+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 0x7f6fa806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.405+0000 7f6fcfe00640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 0x7f6fa806d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009b20 tx=0x7f6fc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.408+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fb8090080 con 0x7f6fd009c070 2024-01-16T14:59:41.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.602+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd009bd10 con 0x7f6fd009c070 2024-01-16T14:59:41.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.604+0000 7f6fccc00640 1 -- 172.21.15.43:0/2809957115 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f6fb8054a90 con 0x7f6fd009c070 2024-01-16T14:59:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:41.586 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 msgr2=0x7f6fa806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 0x7f6fa806d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009b20 tx=0x7f6fc0008040 comp rx=0 tx=0).stop 2024-01-16T14:59:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 msgr2=0x7f6fd00ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800e780 tx=0x7f6fb800ec40 comp rx=0 tx=0).stop 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 shutdown_connections 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa806b2c0 0x7f6fa806d780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fd00cd240 0x7f6fd009a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fd009c070 0x7f6fd00ccd00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 --2- 172.21.15.43:0/2809957115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fb0004d00 0x7f6fd0146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 >> 172.21.15.43:0/2809957115 conn(0x7f6fd00b0d80 msgr2=0x7f6fd00aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 shutdown_connections 2024-01-16T14:59:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.607+0000 7f6fd7200640 1 -- 172.21.15.43:0/2809957115 wait complete. 2024-01-16T14:59:41.606 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:41.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:41.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.837+0000 7f536ae00640 1 Processor -- start 2024-01-16T14:59:41.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.837+0000 7f536ae00640 1 -- start start 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53640cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536ae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536415f930 con 0x7f536406d9e0 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536ae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53640b3b10 con 0x7f5364144770 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53640b43b0 con 0x7f536415f270 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536a400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f536406d9e0 0x7f53640cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53112/0 (socket says 172.21.15.43:53112) 2024-01-16T14:59:41.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f536a400640 1 -- 172.21.15.43:0/1545164288 learned_addr learned my addr 172.21.15.43:0/1545164288 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:41.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.839+0000 7f5369000640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53640cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486829417 0 0) 0x7f536415f930 con 0x7f536406d9e0 2024-01-16T14:59:41.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344003520 con 0x7f536406d9e0 2024-01-16T14:59:41.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523111762 0 0) 0x7f5344003520 con 0x7f536406d9e0 2024-01-16T14:59:41.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 msgr2=0x7f53640cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53640cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 >> v1:172.21.15.69:6789/0 conn(0x7f5364144770 legacy=0x7f536415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640b3df0 con 0x7f536406d9e0 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f535c0019f0 con 0x7f536406d9e0 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 >> v1:172.21.15.43:6789/0 conn(0x7f536406d9e0 legacy=0x7f53640cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5344003a10 0x7f5344003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344004d00 0x7f5344005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53640b43b0 con 0x7f5344003a10 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534400ba90 con 0x7f5344004d00 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534400bc50 con 0x7f536415f270 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.840+0000 7f5369a00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5344003a10 0x7f5344003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369000640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f536a400640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344004d00 0x7f5344005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369a00640 1 -- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 msgr2=0x7f53440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369a00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f536415f270 0x7f53440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369a00640 1 -- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344004d00 msgr2=0x7f5344005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369a00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344004d00 0x7f5344005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.841+0000 7f5369a00640 1 -- 172.21.15.43:0/1545164288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5344003520 con 0x7f5344003a10 2024-01-16T14:59:41.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.842+0000 7f5369a00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5344003a10 0x7f5344003e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5354001720 tx=0x7f53440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de05da0a2c47dac4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.842+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53540052d0 con 0x7f5344003a10 2024-01-16T14:59:41.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.842+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535402e050 con 0x7f5344003a10 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.842+0000 7f5363e00640 1 -- 172.21.15.43:0/1545164288 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53540035f0 con 0x7f5344003a10 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 -- 172.21.15.43:0/1545164288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5344003a10 msgr2=0x7f5344003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 --2- 172.21.15.43:0/1545164288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5344003a10 0x7f5344003e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5354001720 tx=0x7f53440057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536a400640 1 -- 172.21.15.43:0/1545164288 reap_dead start 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 -- 172.21.15.43:0/1545164288 shutdown_connections 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 -- 172.21.15.43:0/1545164288 >> 172.21.15.43:0/1545164288 conn(0x7f53640cec10 msgr2=0x7f53640cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 -- 172.21.15.43:0/1545164288 shutdown_connections 2024-01-16T14:59:41.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.843+0000 7f536ae00640 1 -- 172.21.15.43:0/1545164288 wait complete. 2024-01-16T14:59:41.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.845+0000 7f536ae00640 1 Processor -- start 2024-01-16T14:59:41.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.845+0000 7f536ae00640 1 -- start start 2024-01-16T14:59:41.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344003a10 0x7f536412fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 0x7f53640acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5364130310 con 0x7f53641f4870 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5364130da0 con 0x7f5344003a10 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f536ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53641429b0 con 0x7f5364143380 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f5369000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f5369000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34580/0 (socket says 172.21.15.43:34580) 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.846+0000 7f5369000640 1 -- 172.21.15.43:0/2983417126 learned_addr learned my addr 172.21.15.43:0/2983417126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f536a400640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 0x7f53640acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369a00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344003a10 0x7f536412fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 -- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 msgr2=0x7f53640acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 0x7f53640acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 -- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344003a10 msgr2=0x7f536412fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344003a10 0x7f536412fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 -- 172.21.15.43:0/2983417126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53641f4cd0 con 0x7f53641f4870 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f536a400640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 0x7f53640acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5369000640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f534c011660 tx=0x7f534c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00f9c0 con 0x7f53641f4870 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c00fb20 con 0x7f53641f4870 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00fe90 con 0x7f53641f4870 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.847+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5364130500 con 0x7f53641f4870 2024-01-16T14:59:41.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.848+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5364130ac0 con 0x7f53641f4870 2024-01-16T14:59:41.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.849+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5364143760 con 0x7f53641f4870 2024-01-16T14:59:41.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.849+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f534c015460 con 0x7f53641f4870 2024-01-16T14:59:41.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.850+0000 7f5361600640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 0x7f533c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:41.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.850+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f534c08a0a0 con 0x7f53641f4870 2024-01-16T14:59:41.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.851+0000 7f5369a00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 0x7f533c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:41.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.851+0000 7f5369a00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 0x7f533c06d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5354033000 tx=0x7f5354009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:41.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:41.855+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5364143760 con 0x7f53641f4870 2024-01-16T14:59:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.049+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5364143760 con 0x7f53641f4870 2024-01-16T14:59:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.050+0000 7f5361600640 1 -- 172.21.15.43:0/2983417126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5364143760 con 0x7f53641f4870 2024-01-16T14:59:42.031 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.052+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 msgr2=0x7f533c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.052+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 0x7f533c06d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5354033000 tx=0x7f5354009030 comp rx=0 tx=0).stop 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.052+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 msgr2=0x7f5364142e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.052+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f534c011660 tx=0x7f534c011b20 comp rx=0 tx=0).stop 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 shutdown_connections 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533c06b2c0 0x7f533c06d780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5364143380 0x7f53640acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53641f4870 0x7f5364142e40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 --2- 172.21.15.43:0/2983417126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5344003a10 0x7f536412fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 >> 172.21.15.43:0/2983417126 conn(0x7f53640cec10 msgr2=0x7f536413f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 shutdown_connections 2024-01-16T14:59:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.053+0000 7f536ae00640 1 -- 172.21.15.43:0/2983417126 wait complete. 2024-01-16T14:59:42.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T14:59:42.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.293+0000 7f69c8a00640 1 Processor -- start 2024-01-16T14:59:42.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.293+0000 7f69c8a00640 1 -- start start 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69c40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c8a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40c6bf0 con 0x7f69c40d0fa0 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c8a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40d13d0 con 0x7f69c415d3b0 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cedc0 con 0x7f69c40d1d90 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c3200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f69c40d0fa0 0x7f69c40b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53122/0 (socket says 172.21.15.43:53122) 2024-01-16T14:59:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.294+0000 7f69c3200640 1 -- 172.21.15.43:0/171229317 learned_addr learned my addr 172.21.15.43:0/171229317 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1e00640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69c40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253429589 0 0) 0x7f69c40c6bf0 con 0x7f69c40d0fa0 2024-01-16T14:59:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a4003520 con 0x7f69c40d0fa0 2024-01-16T14:59:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1083744932 0 0) 0x7f69a4003520 con 0x7f69c40d0fa0 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 msgr2=0x7f69c40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69c40ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 >> v1:172.21.15.69:6789/0 conn(0x7f69c415d3b0 legacy=0x7f69c40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40c6e40 con 0x7f69c40d0fa0 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f69bc001670 con 0x7f69c40d0fa0 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 >> v1:172.21.15.43:6789/0 conn(0x7f69c40d0fa0 legacy=0x7f69c40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69a4003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69a4003f50 0x7f69a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69a4004d00 0x7f69a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cedc0 con 0x7f69a4004d00 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a400bb40 con 0x7f69a4003f50 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.295+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a400bd00 con 0x7f69c40d1d90 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69a4004d00 0x7f69a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c1e00640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69a4003f50 0x7f69a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c2800640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69a4003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 -- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 msgr2=0x7f69a4003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69c40d1d90 0x7f69a4003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 -- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69a4003f50 msgr2=0x7f69a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69a4003f50 0x7f69a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.296+0000 7f69c3200640 1 -- 172.21.15.43:0/171229317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4003520 con 0x7f69a4004d00 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.297+0000 7f69c3200640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69a4004d00 0x7f69a4005180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c190 tx=0x7f69a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa55d42248819f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc005ad0 con 0x7f69a4004d00 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc005c30 con 0x7f69a4004d00 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c1400640 1 -- 172.21.15.43:0/171229317 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc02c810 con 0x7f69a4004d00 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 -- 172.21.15.43:0/171229317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69a4004d00 msgr2=0x7f69a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 --2- 172.21.15.43:0/171229317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69a4004d00 0x7f69a4005180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c190 tx=0x7f69a40057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c3200640 1 -- 172.21.15.43:0/171229317 reap_dead start 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 -- 172.21.15.43:0/171229317 shutdown_connections 2024-01-16T14:59:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 -- 172.21.15.43:0/171229317 >> 172.21.15.43:0/171229317 conn(0x7f69c40d30a0 msgr2=0x7f69c40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 -- 172.21.15.43:0/171229317 shutdown_connections 2024-01-16T14:59:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.298+0000 7f69c8a00640 1 -- 172.21.15.43:0/171229317 wait complete. 2024-01-16T14:59:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.300+0000 7f69c8a00640 1 Processor -- start 2024-01-16T14:59:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.300+0000 7f69c8a00640 1 -- start start 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69a4004d00 0x7f69c40ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69c409d4d0 0x7f69c4151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41f3880 con 0x7f69c409d050 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41f2f70 con 0x7f69c409d4d0 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40a0300 con 0x7f69a4004d00 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.301+0000 7f69c1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34600/0 (socket says 172.21.15.43:34600) 2024-01-16T14:59:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 -- 172.21.15.43:0/1947640212 learned_addr learned my addr 172.21.15.43:0/1947640212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c2800640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69a4004d00 0x7f69c40ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c3200640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69c409d4d0 0x7f69c4151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 -- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69a4004d00 msgr2=0x7f69c40ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69a4004d00 0x7f69c40ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 -- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69c409d4d0 msgr2=0x7f69c4151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69c409d4d0 0x7f69c4151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 -- 172.21.15.43:0/1947640212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c41f3230 con 0x7f69c409d050 2024-01-16T14:59:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.302+0000 7f69c1e00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011e30 tx=0x7f69ac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.303+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00e040 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.303+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00f040 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.303+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac009c00 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.303+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c41f3510 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.303+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c41f3d10 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.304+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c406c310 con 0x7f69c409d050 2024-01-16T14:59:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.305+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f69ac015850 con 0x7f69c409d050 2024-01-16T14:59:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.306+0000 7f69baa00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 0x7f699c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.306+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69ac08a970 con 0x7f69c409d050 2024-01-16T14:59:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.307+0000 7f69c2800640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 0x7f699c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.307+0000 7f69c2800640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 0x7f699c06d7b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f69b4003700 tx=0x7f69b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.311+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69ac0557d0 con 0x7f69c409d050 2024-01-16T14:59:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.503+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69c404e840 con 0x7f69c409d050 2024-01-16T14:59:42.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.504+0000 7f69baa00640 1 -- 172.21.15.43:0/1947640212 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69ac058df0 con 0x7f69c409d050 2024-01-16T14:59:42.485 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 msgr2=0x7f699c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 0x7f699c06d7b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f69b4003700 tx=0x7f69b4009040 comp rx=0 tx=0).stop 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 msgr2=0x7f69c4151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011e30 tx=0x7f69ac00f580 comp rx=0 tx=0).stop 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 shutdown_connections 2024-01-16T14:59:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f699c06b2f0 0x7f699c06d7b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69c409d4d0 0x7f69c4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c409d050 0x7f69c4151110 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 --2- 172.21.15.43:0/1947640212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f69a4004d00 0x7f69c40ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 >> 172.21.15.43:0/1947640212 conn(0x7f69c40d30a0 msgr2=0x7f69c406bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 shutdown_connections 2024-01-16T14:59:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.507+0000 7f69c8a00640 1 -- 172.21.15.43:0/1947640212 wait complete. 2024-01-16T14:59:42.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T14:59:42.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.754+0000 7f2ff7e00640 1 Processor -- start 2024-01-16T14:59:42.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.754+0000 7f2ff7e00640 1 -- start start 2024-01-16T14:59:42.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ff815f270 0x7f2ff80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff815f930 con 0x7f2ff806d9e0 2024-01-16T14:59:42.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff80b3b10 con 0x7f2ff8144770 2024-01-16T14:59:42.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff80b43b0 con 0x7f2ff815f270 2024-01-16T14:59:42.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2ff806d9e0 0x7f2ff80cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53138/0 (socket says 172.21.15.43:53138) 2024-01-16T14:59:42.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2ff7400640 1 -- 172.21.15.43:0/3923160670 learned_addr learned my addr 172.21.15.43:0/3923160670 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:42.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.756+0000 7f2fefe00640 1 --2- 172.21.15.43:0/3923160670 >> v2:172.21.15.69:3300/0 conn(0x7f2ff815f270 0x7f2ff80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645152998 0 0) 0x7f2ff815f930 con 0x7f2ff806d9e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8003520 con 0x7f2ff806d9e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356250631 0 0) 0x7f2fd8003520 con 0x7f2ff806d9e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 >> v2:172.21.15.69:3300/0 conn(0x7f2ff815f270 msgr2=0x7f2ff80cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 --2- 172.21.15.43:0/3923160670 >> v2:172.21.15.69:3300/0 conn(0x7f2ff815f270 0x7f2ff80cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 >> v1:172.21.15.69:6789/0 conn(0x7f2ff8144770 legacy=0x7f2ff815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff80b3df0 con 0x7f2ff806d9e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f2ff00019f0 con 0x7f2ff806d9e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 >> v1:172.21.15.43:6789/0 conn(0x7f2ff806d9e0 legacy=0x7f2ff80cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 0x7f2fd8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80047c0 0x7f2fd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fd8005180 0x7f2fd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff80b43b0 con 0x7f2fd80039e0 2024-01-16T14:59:42.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd800c0b0 con 0x7f2fd80047c0 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.757+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd800c270 con 0x7f2fd8005180 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 0x7f2fd8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2fefe00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80047c0 0x7f2fd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff7400640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fd8005180 0x7f2fd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 -- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fd8005180 msgr2=0x7f2fd800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fd8005180 0x7f2fd800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 -- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80047c0 msgr2=0x7f2fd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80047c0 0x7f2fd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff6a00640 1 -- 172.21.15.43:0/3923160670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8003520 con 0x7f2fd80039e0 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff7400640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fd8005180 0x7f2fd800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:42.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.758+0000 7f2ff7400640 1 -- 172.21.15.43:0/3923160670 reap_dead start 2024-01-16T14:59:42.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.759+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 0x7f2fd8003e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8001750 tx=0x7f2fd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca80ba326891b58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80052d0 con 0x7f2fd80039e0 2024-01-16T14:59:42.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe802e050 con 0x7f2fd80039e0 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff6000640 1 -- 172.21.15.43:0/3923160670 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8003830 con 0x7f2fd80039e0 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff7e00640 1 -- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 msgr2=0x7f2fd8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 0x7f2fd8003e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8001750 tx=0x7f2fd80057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff7e00640 1 -- 172.21.15.43:0/3923160670 shutdown_connections 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/3923160670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fd80039e0 0x7f2fd8003e00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.760+0000 7f2ff7e00640 1 -- 172.21.15.43:0/3923160670 >> 172.21.15.43:0/3923160670 conn(0x7f2ff80cec10 msgr2=0x7f2ff80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.761+0000 7f2ff7e00640 1 -- 172.21.15.43:0/3923160670 shutdown_connections 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.761+0000 7f2ff7e00640 1 -- 172.21.15.43:0/3923160670 wait complete. 2024-01-16T14:59:42.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.762+0000 7f2ff7e00640 1 Processor -- start 2024-01-16T14:59:42.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.763+0000 7f2ff7e00640 1 -- start start 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.763+0000 7f2ff7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80039e0 0x7f2ff81e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ff812bfe0 0x7f2ff812c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff815dbe0 con 0x7f2ff81f7080 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff815d050 con 0x7f2fd80039e0 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81e51d0 con 0x7f2ff812bfe0 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2fefe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2fefe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34620/0 (socket says 172.21.15.43:34620) 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2fefe00640 1 -- 172.21.15.43:0/71704164 learned_addr learned my addr 172.21.15.43:0/71704164 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:42.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80039e0 0x7f2ff81e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.764+0000 7f2ff7400640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ff812bfe0 0x7f2ff812c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 -- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ff812bfe0 msgr2=0x7f2ff812c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ff812bfe0 0x7f2ff812c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 -- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80039e0 msgr2=0x7f2ff81e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80039e0 0x7f2ff81e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 -- 172.21.15.43:0/71704164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff815e000 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2fefe00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0011700 tx=0x7f2fe0011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000f9c0 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe000fb20 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff81e4f00 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000fe90 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.765+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff815d520 con 0x7f2ff81f7080 2024-01-16T14:59:42.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.766+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff804e840 con 0x7f2ff81f7080 2024-01-16T14:59:42.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.767+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2fe0015460 con 0x7f2ff81f7080 2024-01-16T14:59:42.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.768+0000 7f2feea00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 0x7f2fd006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:42.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.768+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fe0089ff0 con 0x7f2ff81f7080 2024-01-16T14:59:42.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.768+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 0x7f2fd006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:42.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.769+0000 7f2ff6a00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 0x7f2fd006d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8033000 tx=0x7f2fe8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:42.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.773+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fe00bce90 con 0x7f2ff81f7080 2024-01-16T14:59:42.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.968+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ff8143fa0 con 0x7f2ff81f7080 2024-01-16T14:59:42.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.969+0000 7f2feea00640 1 -- 172.21.15.43:0/71704164 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2fe00220a0 con 0x7f2ff81f7080 2024-01-16T14:59:42.950 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 msgr2=0x7f2fd006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 0x7f2fd006d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8033000 tx=0x7f2fe8009030 comp rx=0 tx=0).stop 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 msgr2=0x7f2ff81f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0011700 tx=0x7f2fe0011bc0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 shutdown_connections 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2fd006b2c0 0x7f2fd006d780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ff812bfe0 0x7f2ff812c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ff81f7080 0x7f2ff81f66e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 --2- 172.21.15.43:0/71704164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd80039e0 0x7f2ff81e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:42.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 >> 172.21.15.43:0/71704164 conn(0x7f2ff80cec10 msgr2=0x7f2ff80ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:42.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 shutdown_connections 2024-01-16T14:59:42.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:42.972+0000 7f2ff7e00640 1 -- 172.21.15.43:0/71704164 wait complete. 2024-01-16T14:59:42.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:43.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.210+0000 7fe7b7200640 1 Processor -- start 2024-01-16T14:59:43.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.210+0000 7fe7b7200640 1 -- start start 2024-01-16T14:59:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7b8157770 0x7fe7b806cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7b806d080 0x7fe7b815d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 0x7fe7b80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b806d500 con 0x7fe7b806d080 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b8161670 con 0x7fe7b8157770 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7b7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815e610 con 0x7fe7b815ddb0 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7afe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7b806d080 0x7fe7b815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7afe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7b806d080 0x7fe7b815d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44642/0 (socket says 172.21.15.43:44642) 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.212+0000 7fe7afe00640 1 -- 172.21.15.43:0/1446348489 learned_addr learned my addr 172.21.15.43:0/1446348489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7b5e00640 1 --2- 172.21.15.43:0/1446348489 >> v2:172.21.15.69:3300/0 conn(0x7fe7b8157770 0x7fe7b806cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7b6800640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 0x7fe7b80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7afe00640 1 -- 172.21.15.43:0/1446348489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 msgr2=0x7fe7b80ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7afe00640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 0x7fe7b80ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7afe00640 1 -- 172.21.15.43:0/1446348489 >> v2:172.21.15.69:3300/0 conn(0x7fe7b8157770 msgr2=0x7fe7b806cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7afe00640 1 --2- 172.21.15.43:0/1446348489 >> v2:172.21.15.69:3300/0 conn(0x7fe7b8157770 0x7fe7b806cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7afe00640 1 -- 172.21.15.43:0/1446348489 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b8160d40 con 0x7fe7b806d080 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.213+0000 7fe7b6800640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 0x7fe7b80ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.214+0000 7fe7afe00640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b806d080 0x7fe7b815d870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80adb70 tx=0x7fe7a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78062b51d9f53f10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.214+0000 7fe7b5400640 1 -- 172.21.15.43:0/1446348489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0031870 con 0x7fe7b806d080 2024-01-16T14:59:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.214+0000 7fe7b5400640 1 -- 172.21.15.43:0/1446348489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a00319d0 con 0x7fe7b806d080 2024-01-16T14:59:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.214+0000 7fe7b5400640 1 -- 172.21.15.43:0/1446348489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0031d60 con 0x7fe7b806d080 2024-01-16T14:59:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 -- 172.21.15.43:0/1446348489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b806d080 msgr2=0x7fe7b815d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b806d080 0x7fe7b815d870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80adb70 tx=0x7fe7a002b910 comp rx=0 tx=0).stop 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 -- 172.21.15.43:0/1446348489 shutdown_connections 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b815ddb0 0x7fe7b80ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 --2- 172.21.15.43:0/1446348489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b806d080 0x7fe7b815d870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 --2- 172.21.15.43:0/1446348489 >> v2:172.21.15.69:3300/0 conn(0x7fe7b8157770 0x7fe7b806cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 -- 172.21.15.43:0/1446348489 >> 172.21.15.43:0/1446348489 conn(0x7fe7b806a860 msgr2=0x7fe7b806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 -- 172.21.15.43:0/1446348489 shutdown_connections 2024-01-16T14:59:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.215+0000 7fe7b7200640 1 -- 172.21.15.43:0/1446348489 wait complete. 2024-01-16T14:59:43.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.217+0000 7fe7b7200640 1 Processor -- start 2024-01-16T14:59:43.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.217+0000 7fe7b7200640 1 -- start start 2024-01-16T14:59:43.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 0x7fe7b80ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7b8157770 0x7fe7b81f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815eb20 con 0x7fe7b815ddb0 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815e820 con 0x7fe7b8157770 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.218+0000 7fe7b7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b81f35a0 con 0x7fe7b806d080 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44646/0 (socket says 172.21.15.43:44646) 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 -- 172.21.15.43:0/3204309423 learned_addr learned my addr 172.21.15.43:0/3204309423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b5e00640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 0x7fe7b80ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7afe00640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7b8157770 0x7fe7b81f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 -- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 msgr2=0x7fe7b80ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 0x7fe7b80ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 -- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7b8157770 msgr2=0x7fe7b81f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7b8157770 0x7fe7b81f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b6800640 1 -- 172.21.15.43:0/3204309423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00094d0 con 0x7fe7b815ddb0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.219+0000 7fe7b5e00640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 0x7fe7b80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7b6800640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000cd50 tx=0x7fe7b000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000f680 con 0x7fe7b815ddb0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b000f800 con 0x7fe7b815ddb0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000fbb0 con 0x7fe7b815ddb0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b81f3880 con 0x7fe7b815ddb0 2024-01-16T14:59:43.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.220+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b80cd1a0 con 0x7fe7b815ddb0 2024-01-16T14:59:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.221+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b804e840 con 0x7fe7b815ddb0 2024-01-16T14:59:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.222+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe7b000b040 con 0x7fe7b815ddb0 2024-01-16T14:59:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.223+0000 7fe7ae000640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 0x7fe79006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.223+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7b008a550 con 0x7fe7b815ddb0 2024-01-16T14:59:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.223+0000 7fe7b5e00640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 0x7fe79006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.224+0000 7fe7b5e00640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 0x7fe79006d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8006fd0 tx=0x7fe7a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.227+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7b00553b0 con 0x7fe7b815ddb0 2024-01-16T14:59:43.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.463+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7b815ec60 con 0x7fe7b815ddb0 2024-01-16T14:59:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.464+0000 7fe7ae000640 1 -- 172.21.15.43:0/3204309423 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fe7b00589d0 con 0x7fe7b815ddb0 2024-01-16T14:59:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:43.445 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:59:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.466+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 msgr2=0x7fe79006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 0x7fe79006d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8006fd0 tx=0x7fe7a8008040 comp rx=0 tx=0).stop 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 msgr2=0x7fe7b815efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000cd50 tx=0x7fe7b000a650 comp rx=0 tx=0).stop 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 shutdown_connections 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79006b2c0 0x7fe79006d780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7b8157770 0x7fe7b81f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7b815ddb0 0x7fe7b815efb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 --2- 172.21.15.43:0/3204309423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7b806d080 0x7fe7b80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 >> 172.21.15.43:0/3204309423 conn(0x7fe7b806a860 msgr2=0x7fe7b80aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 shutdown_connections 2024-01-16T14:59:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.467+0000 7fe7b7200640 1 -- 172.21.15.43:0/3204309423 wait complete. 2024-01-16T14:59:43.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:43.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.709+0000 7f7dc9800640 1 Processor -- start 2024-01-16T14:59:43.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.710+0000 7f7dc9800640 1 -- start start 2024-01-16T14:59:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc9800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7dc40cf050 0x7f7dc40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc9800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40ce7c0 con 0x7f7dc406c130 2024-01-16T14:59:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc9800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40cfa10 con 0x7f7dc40cf050 2024-01-16T14:59:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc9800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4140900 con 0x7f7dc413dfd0 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:52460/0 (socket says 172.21.15.43:52460) 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.712+0000 7f7dc3e00640 1 -- 172.21.15.43:0/3178384463 learned_addr learned my addr 172.21.15.43:0/3178384463 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.713+0000 7f7dbb200640 1 --2- 172.21.15.43:0/3178384463 >> v2:172.21.15.69:3300/0 conn(0x7f7dc40cf050 0x7f7dc40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.713+0000 7f7dc3e00640 1 -- 172.21.15.43:0/3178384463 >> v2:172.21.15.69:3300/0 conn(0x7f7dc40cf050 msgr2=0x7f7dc40cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.713+0000 7f7dc3e00640 1 --2- 172.21.15.43:0/3178384463 >> v2:172.21.15.69:3300/0 conn(0x7f7dc40cf050 0x7f7dc40cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.713+0000 7f7dc3e00640 1 -- 172.21.15.43:0/3178384463 >> v1:172.21.15.69:6789/0 conn(0x7f7dc406c130 legacy=0x7f7dc406c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.713+0000 7f7dc3e00640 1 -- 172.21.15.43:0/3178384463 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc415fa70 con 0x7f7dc413dfd0 2024-01-16T14:59:43.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.715+0000 7f7dc3e00640 1 --2- 172.21.15.43:0/3178384463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b184ffcedf5ba21e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.715+0000 7f7dc3400640 1 -- 172.21.15.43:0/3178384463 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db402cb00 con 0x7f7dc413dfd0 2024-01-16T14:59:43.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.715+0000 7f7dc3400640 1 -- 172.21.15.43:0/3178384463 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db402cc60 con 0x7f7dc413dfd0 2024-01-16T14:59:43.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.715+0000 7f7dc3400640 1 -- 172.21.15.43:0/3178384463 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40343f0 con 0x7f7dc413dfd0 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 -- 172.21.15.43:0/3178384463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 msgr2=0x7f7dc415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3178384463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db402bf60 comp rx=0 tx=0).stop 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 -- 172.21.15.43:0/3178384463 shutdown_connections 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3178384463 >> v2:172.21.15.69:3300/0 conn(0x7f7dc40cf050 0x7f7dc40cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3178384463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc413dfd0 0x7f7dc415be70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 -- 172.21.15.43:0/3178384463 >> 172.21.15.43:0/3178384463 conn(0x7f7dc406a860 msgr2=0x7f7dc406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.716+0000 7f7dc9800640 1 -- 172.21.15.43:0/3178384463 shutdown_connections 2024-01-16T14:59:43.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.717+0000 7f7dc9800640 1 -- 172.21.15.43:0/3178384463 wait complete. 2024-01-16T14:59:43.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.719+0000 7f7dc9800640 1 Processor -- start 2024-01-16T14:59:43.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.719+0000 7f7dc9800640 1 -- start start 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc406c130 0x7f7dc41392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dc413dfd0 0x7f7dc40b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4138fd0 con 0x7f7dc40cf050 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4138cf0 con 0x7f7dc413dfd0 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dc9800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc41383c0 con 0x7f7dc406c130 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dbb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dbb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44658/0 (socket says 172.21.15.43:44658) 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.720+0000 7f7dbb200640 1 -- 172.21.15.43:0/3854222944 learned_addr learned my addr 172.21.15.43:0/3854222944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dc8e00640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dc413dfd0 0x7f7dc40b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dc3e00640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc406c130 0x7f7dc41392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 -- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc406c130 msgr2=0x7f7dc41392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc406c130 0x7f7dc41392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 -- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dc413dfd0 msgr2=0x7f7dc40b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dc413dfd0 0x7f7dc40b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 -- 172.21.15.43:0/3854222944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009b30 con 0x7f7dc40cf050 2024-01-16T14:59:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dbb200640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00d800 tx=0x7f7dac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.721+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac00b6c0 con 0x7f7dc40cf050 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.722+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac00b820 con 0x7f7dc40cf050 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.722+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac00bbb0 con 0x7f7dc40cf050 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.722+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc41386a0 con 0x7f7dc40cf050 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.722+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40cb8d0 con 0x7f7dc40cf050 2024-01-16T14:59:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.723+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc404e840 con 0x7f7dc40cf050 2024-01-16T14:59:43.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.724+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7dac00bd10 con 0x7f7dc40cf050 2024-01-16T14:59:43.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.725+0000 7f7dc0c00640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 0x7f7d9c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:43.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.725+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dac08a210 con 0x7f7dc40cf050 2024-01-16T14:59:43.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.726+0000 7f7dc3e00640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 0x7f7d9c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:43.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.726+0000 7f7dc3e00640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 0x7f7d9c06d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7db4035000 tx=0x7f7db4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:43.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.730+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7dac054fe0 con 0x7f7dc40cf050 2024-01-16T14:59:43.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.925+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dc41401a0 con 0x7f7dc40cf050 2024-01-16T14:59:43.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.928+0000 7f7dc0c00640 1 -- 172.21.15.43:0/3854222944 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f7dac058600 con 0x7f7dc40cf050 2024-01-16T14:59:43.909 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:43.909 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:43.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.930+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 msgr2=0x7f7d9c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.930+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 0x7f7d9c06d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7db4035000 tx=0x7f7db4030000 comp rx=0 tx=0).stop 2024-01-16T14:59:43.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.930+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 msgr2=0x7f7dc40cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:43.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.930+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00d800 tx=0x7f7dac00dcc0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 shutdown_connections 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d9c06b2c0 0x7f7d9c06d780 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7dc413dfd0 0x7f7dc40b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dc40cf050 0x7f7dc40cb390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 --2- 172.21.15.43:0/3854222944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7dc406c130 0x7f7dc41392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 >> 172.21.15.43:0/3854222944 conn(0x7f7dc406a860 msgr2=0x7f7dc413c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:43.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 shutdown_connections 2024-01-16T14:59:43.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:43.931+0000 7f7dc9800640 1 -- 172.21.15.43:0/3854222944 wait complete. 2024-01-16T14:59:43.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.166+0000 7f877ea00640 1 Processor -- start 2024-01-16T14:59:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.167+0000 7f877ea00640 1 -- start start 2024-01-16T14:59:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.168+0000 7f877ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.168+0000 7f877ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87780ce750 0x7f87780b6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.168+0000 7f877ea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87780d1dd0 con 0x7f87780d18c0 2024-01-16T14:59:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.168+0000 7f877ea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87780d20b0 con 0x7f87780ce750 2024-01-16T14:59:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.168+0000 7f877ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87780d2390 con 0x7f877815d580 2024-01-16T14:59:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.169+0000 7f8777e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.169+0000 7f8777e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:52478/0 (socket says 172.21.15.43:52478) 2024-01-16T14:59:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.169+0000 7f8777400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f87780d18c0 0x7f87780ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:33630/0 (socket says 172.21.15.43:33630) 2024-01-16T14:59:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.169+0000 7f8777e00640 1 -- 172.21.15.43:0/3640867759 learned_addr learned my addr 172.21.15.43:0/3640867759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.169+0000 7f877ca00640 1 --2- 172.21.15.43:0/3640867759 >> v2:172.21.15.69:3300/0 conn(0x7f87780ce750 0x7f87780b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.170+0000 7f8777e00640 1 -- 172.21.15.43:0/3640867759 >> v2:172.21.15.69:3300/0 conn(0x7f87780ce750 msgr2=0x7f87780b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.170+0000 7f8777e00640 1 --2- 172.21.15.43:0/3640867759 >> v2:172.21.15.69:3300/0 conn(0x7f87780ce750 0x7f87780b6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.170+0000 7f8776a00640 1 -- 172.21.15.43:0/3640867759 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676858399 0 0) 0x7f87780d1dd0 con 0x7f87780d18c0 2024-01-16T14:59:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.170+0000 7f8777e00640 1 -- 172.21.15.43:0/3640867759 >> v1:172.21.15.69:6789/0 conn(0x7f87780d18c0 legacy=0x7f87780ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.170+0000 7f8777e00640 1 -- 172.21.15.43:0/3640867759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87780d2670 con 0x7f877815d580 2024-01-16T14:59:44.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.171+0000 7f8777e00640 1 --2- 172.21.15.43:0/3640867759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8768009870 tx=0x7f876802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce2d366ebbf300e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f8776a00640 1 -- 172.21.15.43:0/3640867759 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876802f840 con 0x7f877815d580 2024-01-16T14:59:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f8776a00640 1 -- 172.21.15.43:0/3640867759 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876802f9a0 con 0x7f877815d580 2024-01-16T14:59:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f8776a00640 1 -- 172.21.15.43:0/3640867759 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876802fd30 con 0x7f877815d580 2024-01-16T14:59:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 -- 172.21.15.43:0/3640867759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 msgr2=0x7f87780d1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 --2- 172.21.15.43:0/3640867759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8768009870 tx=0x7f876802ba20 comp rx=0 tx=0).stop 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 -- 172.21.15.43:0/3640867759 shutdown_connections 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 --2- 172.21.15.43:0/3640867759 >> v2:172.21.15.69:3300/0 conn(0x7f87780ce750 0x7f87780b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 --2- 172.21.15.43:0/3640867759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f877815d580 0x7f87780d1380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 -- 172.21.15.43:0/3640867759 >> 172.21.15.43:0/3640867759 conn(0x7f8778071f90 msgr2=0x7f87780723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 -- 172.21.15.43:0/3640867759 shutdown_connections 2024-01-16T14:59:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.172+0000 7f877ea00640 1 -- 172.21.15.43:0/3640867759 wait complete. 2024-01-16T14:59:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.174+0000 7f877ea00640 1 Processor -- start 2024-01-16T14:59:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.174+0000 7f877ea00640 1 -- start start 2024-01-16T14:59:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 0x7f8778156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 0x7f8778157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780b5970 con 0x7f87780d18c0 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780b5310 con 0x7f877815d580 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.175+0000 7f877ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87780b4700 con 0x7f87780ce750 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44668/0 (socket says 172.21.15.43:44668) 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 -- 172.21.15.43:0/185289826 learned_addr learned my addr 172.21.15.43:0/185289826 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777e00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 0x7f8778156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f877ca00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 0x7f8778157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 -- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 msgr2=0x7f8778156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 0x7f8778156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 -- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 msgr2=0x7f8778157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 0x7f8778157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 -- 172.21.15.43:0/185289826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680094d0 con 0x7f87780d18c0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777e00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 0x7f8778156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f877ca00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 0x7f8778157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.176+0000 7f8777400640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f876000c820 tx=0x7f876000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.177+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876000a640 con 0x7f87780d18c0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.177+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876000a7c0 con 0x7f87780d18c0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.177+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876000ab70 con 0x7f87780d18c0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.177+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87780d2ff0 con 0x7f87780d18c0 2024-01-16T14:59:44.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.177+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87780b4cc0 con 0x7f87780d18c0 2024-01-16T14:59:44.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.178+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8778074770 con 0x7f87780d18c0 2024-01-16T14:59:44.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.179+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f876000acd0 con 0x7f87780d18c0 2024-01-16T14:59:44.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.180+0000 7f876fe00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 0x7f875006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:44.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.180+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8760089070 con 0x7f87780d18c0 2024-01-16T14:59:44.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.180+0000 7f8777e00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 0x7f875006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:44.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.181+0000 7f8777e00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 0x7f875006d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8768036000 tx=0x7f8768008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:44.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.184+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87600bbe90 con 0x7f87780d18c0 2024-01-16T14:59:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.415+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8778072510 con 0x7f87780d18c0 2024-01-16T14:59:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.417+0000 7f876fe00640 1 -- 172.21.15.43:0/185289826 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f8760018020 con 0x7f87780d18c0 2024-01-16T14:59:44.398 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T14:59:44.398 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T14:59:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T14:59:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 msgr2=0x7f875006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 0x7f875006d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8768036000 tx=0x7f8768008030 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 msgr2=0x7f8778157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f876000c820 tx=0x7f876000cce0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 shutdown_connections 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f875006b2f0 0x7f875006d7b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f877815d580 0x7f8778157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87780d18c0 0x7f8778157750 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 --2- 172.21.15.43:0/185289826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87780ce750 0x7f8778156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.419+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 >> 172.21.15.43:0/185289826 conn(0x7f8778071f90 msgr2=0x7f877815e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.420+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 shutdown_connections 2024-01-16T14:59:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:44.420+0000 7f877ea00640 1 -- 172.21.15.43:0/185289826 wait complete. 2024-01-16T14:59:44.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:59:27.130477+0000', 'modified': '2024-01-16T14:59:30.251270+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6834/971655709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 971655709}, {'type': 'v1', 'addr': '172.21.15.69:6834', 'nonce': 971655709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T14:59:44.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T14:59:44.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-16T14:59:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.074+0000 7f10be800640 1 Processor -- start 2024-01-16T14:59:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.074+0000 7f10be800640 1 -- start start 2024-01-16T14:59:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f10b806d520 0x7f10b806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b80acb50 0x7f10b80719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10b8165430 0x7f10b81658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b8070dc0 con 0x7f10b806d520 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b8161d00 con 0x7f10b8165430 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10be800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b806de40 con 0x7f10b80acb50 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10b7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f10b806d520 0x7f10b806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10b7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f10b806d520 0x7f10b806d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44670/0 (socket says 172.21.15.43:44670) 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.076+0000 7f10b7400640 1 -- 172.21.15.43:0/337791577 learned_addr learned my addr 172.21.15.43:0/337791577 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10afe00640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b80acb50 0x7f10b80719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7e00640 1 --2- 172.21.15.43:0/337791577 >> v2:172.21.15.69:3300/0 conn(0x7f10b8165430 0x7f10b81658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7400640 1 -- 172.21.15.43:0/337791577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b80acb50 msgr2=0x7f10b80719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7400640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b80acb50 0x7f10b80719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7400640 1 -- 172.21.15.43:0/337791577 >> v2:172.21.15.69:3300/0 conn(0x7f10b8165430 msgr2=0x7f10b81658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7400640 1 --2- 172.21.15.43:0/337791577 >> v2:172.21.15.69:3300/0 conn(0x7f10b8165430 0x7f10b81658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.077+0000 7f10b7400640 1 -- 172.21.15.43:0/337791577 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80cedb0 con 0x7f10b806d520 2024-01-16T14:59:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.078+0000 7f10b7400640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b806d520 0x7f10b806d900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10b80ae0d0 tx=0x7f10a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda4981a0b6d5d79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10b6a00640 1 -- 172.21.15.43:0/337791577 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8031870 con 0x7f10b806d520 2024-01-16T14:59:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10b6a00640 1 -- 172.21.15.43:0/337791577 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a80319d0 con 0x7f10b806d520 2024-01-16T14:59:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10b6a00640 1 -- 172.21.15.43:0/337791577 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8031d60 con 0x7f10b806d520 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 -- 172.21.15.43:0/337791577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b806d520 msgr2=0x7f10b806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b806d520 0x7f10b806d900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f10b80ae0d0 tx=0x7f10a802b910 comp rx=0 tx=0).stop 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 -- 172.21.15.43:0/337791577 shutdown_connections 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 --2- 172.21.15.43:0/337791577 >> v2:172.21.15.69:3300/0 conn(0x7f10b8165430 0x7f10b81658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b80acb50 0x7f10b80719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 --2- 172.21.15.43:0/337791577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b806d520 0x7f10b806d900 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 -- 172.21.15.43:0/337791577 >> 172.21.15.43:0/337791577 conn(0x7f10b80b0d80 msgr2=0x7f10b80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.079+0000 7f10be800640 1 -- 172.21.15.43:0/337791577 shutdown_connections 2024-01-16T14:59:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.080+0000 7f10be800640 1 -- 172.21.15.43:0/337791577 wait complete. 2024-01-16T14:59:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.081+0000 7f10be800640 1 Processor -- start 2024-01-16T14:59:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.081+0000 7f10be800640 1 -- start start 2024-01-16T14:59:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b806d520 0x7f10b809cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b8165430 0x7f10b8163a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b809a870 con 0x7f10b80acb50 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b809ab50 con 0x7f10b8165430 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.082+0000 7f10be800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b809ae30 con 0x7f10b806d520 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44684/0 (socket says 172.21.15.43:44684) 2024-01-16T14:59:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 -- 172.21.15.43:0/1852511171 learned_addr learned my addr 172.21.15.43:0/1852511171 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 -- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b806d520 msgr2=0x7f10b809cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10b7400640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b806d520 0x7f10b809cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b806d520 0x7f10b809cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 -- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b8165430 msgr2=0x7f10b8163a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10b7e00640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b8165430 0x7f10b8163a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b8165430 0x7f10b8163a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 -- 172.21.15.43:0/1852511171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80094d0 con 0x7f10b80acb50 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.083+0000 7f10afe00640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10a000d800 tx=0x7f10a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.084+0000 7f10af400640 1 -- 172.21.15.43:0/1852511171 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a00096a0 con 0x7f10b80acb50 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.084+0000 7f10af400640 1 -- 172.21.15.43:0/1852511171 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a0009800 con 0x7f10b80acb50 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.084+0000 7f10af400640 1 -- 172.21.15.43:0/1852511171 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0009990 con 0x7f10b80acb50 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.084+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b809c860 con 0x7f10b80acb50 2024-01-16T14:59:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.084+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b815a950 con 0x7f10b80acb50 2024-01-16T14:59:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.086+0000 7f10af400640 1 -- 172.21.15.43:0/1852511171 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f10a0009af0 con 0x7f10b80acb50 2024-01-16T14:59:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.087+0000 7f10af400640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 0x7f109006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.087+0000 7f10af400640 1 -- 172.21.15.43:0/1852511171 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a008a0e0 con 0x7f10b80acb50 2024-01-16T14:59:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.087+0000 7f10b7400640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 0x7f109006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.087+0000 7f10c1626740 1 Processor -- start 2024-01-16T14:59:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.088+0000 7f10c1626740 1 -- start start 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.088+0000 7f10b7400640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 0x7f109006d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10a8036000 tx=0x7f10a803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.088+0000 7f10c1626740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.088+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f06ba0 0x559316f0b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.088+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f4f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10c1626740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316e6a880 con 0x559316f0bea0 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316e79ae0 con 0x559316f06ba0 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x559316e771d0 con 0x559316f0b5e0 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44698/0 (socket says 172.21.15.43:44698) 2024-01-16T14:59:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 -- 172.21.15.43:0/2081886763 learned_addr learned my addr 172.21.15.43:0/2081886763 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7e00640 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f4f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 -- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 msgr2=0x559316f4f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f4f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10afe00640 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f06ba0 0x559316f0b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 -- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f06ba0 msgr2=0x559316f0b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f06ba0 0x559316f0b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.089+0000 7f10b7400640 1 -- 172.21.15.43:0/2081886763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559316e3a860 con 0x559316f0bea0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.090+0000 7f10b7400640 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009930 tx=0x7f10a8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6baf823df2ed2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.090+0000 7f10a7e00640 1 -- 172.21.15.43:0/2081886763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80454c0 con 0x559316f0bea0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.090+0000 7f10a7e00640 1 -- 172.21.15.43:0/2081886763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8049070 con 0x559316f0bea0 2024-01-16T14:59:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.090+0000 7f10a7e00640 1 -- 172.21.15.43:0/2081886763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8045820 con 0x559316f0bea0 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 -- 172.21.15.43:0/2081886763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 msgr2=0x559316f0c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009930 tx=0x7f10a8030000 comp rx=0 tx=0).stop 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 -- 172.21.15.43:0/2081886763 shutdown_connections 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f06ba0 0x559316f0b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f4f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 --2- 172.21.15.43:0/2081886763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f0bea0 0x559316f0c280 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 -- 172.21.15.43:0/2081886763 >> 172.21.15.43:0/2081886763 conn(0x559316ecdc70 msgr2=0x559316f51990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 -- 172.21.15.43:0/2081886763 shutdown_connections 2024-01-16T14:59:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.091+0000 7f10c1626740 1 -- 172.21.15.43:0/2081886763 wait complete. 2024-01-16T14:59:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.092+0000 7f10c1626740 1 -- start start 2024-01-16T14:59:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.092+0000 7f10c1626740 1 Processor -- start 2024-01-16T14:59:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f61840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f57a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316e72ac0 con 0x559316f06ba0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316e0f2f0 con 0x559316f0b5e0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x559316e2cc70 con 0x559316f0bea0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10c1626740 10 client.0 _ll_register_callbacks cb 0x559316e313b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44710/0 (socket says 172.21.15.43:44710) 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.093+0000 7f10b7400640 1 -- 172.21.15.43:0/2235428535 learned_addr learned my addr 172.21.15.43:0/2235428535 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10afe00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f61840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 -- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 msgr2=0x559316f57a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7e00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f57a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f57a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 -- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 msgr2=0x559316f61840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f61840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 -- 172.21.15.43:0/2235428535 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b80cedb0 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10afe00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f61840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7e00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f57a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10b7400640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f10a80090b0 tx=0x7f10a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb68793b2266c90 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8050480 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a804b070 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.094+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559316e6baa0 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10a80505e0 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 10 client.4665 fetch_fsmap learned FSMap version 8 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x559316e71140 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559316e6baa0 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f10a8050740 con 0x559316f06ba0 2024-01-16T14:59:45.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 10 client.4665 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:59:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 10 client.4665 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T14:59:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 20 client.4665 populate_metadata read hostname 'smithi043' 2024-01-16T14:59:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f54e30 0x559316f48a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 4 client.4665 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T14:59:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.095+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 --> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559316e6b7c0 con 0x559316f54e30 2024-01-16T14:59:45.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.096+0000 7f10afe00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f54e30 0x559316f48a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.097+0000 7f10afe00640 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f54e30 0x559316f48a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d346d3cc50f30d6 server_cookie=b30c3e93859255fc in_seq=0 out_seq=0 2024-01-16T14:59:45.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.097+0000 7f10a4c00640 10 client.4665 ms_handle_connect on v2:172.21.15.69:6832/971655709 2024-01-16T14:59:45.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10a4c00640 1 -- 172.21.15.43:0/2235428535 <== mds.0 v2:172.21.15.69:6832/971655709 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559316e6b7c0 con 0x559316f54e30 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10a4c00640 10 client.4665 handle_command_reply: tid=0 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 client.4665 shutdown 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 msgr2=0x559316f30da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f10a80090b0 tx=0x7f10a8004730 comp rx=0 tx=0).stop 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 shutdown_connections 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f54e30 0x559316f48a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f61840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f57a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 --2- 172.21.15.43:0/2235428535 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f30da0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10a4c00640 0 client.4665 ms_handle_reset on v2:172.21.15.69:6832/971655709 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 >> 172.21.15.43:0/2235428535 conn(0x559316ecdc70 msgr2=0x559316f51990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 shutdown_connections 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 1 -- 172.21.15.43:0/2235428535 wait complete. 2024-01-16T14:59:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.099+0000 7f10c1626740 20 client.4665 trim_cache size 0 max 16384 2024-01-16T14:59:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.117+0000 7f10c1626740 1 Processor -- start 2024-01-16T14:59:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.117+0000 7f10c1626740 1 -- start start 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.117+0000 7f10c1626740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f022a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f05d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10c1626740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316bd5f20 con 0x559316f06ba0 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316e7b2f0 con 0x559316f0b5e0 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x559316f524d0 con 0x559316f0bea0 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44720/0 (socket says 172.21.15.43:44720) 2024-01-16T14:59:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 -- 172.21.15.43:0/4265965938 learned_addr learned my addr 172.21.15.43:0/4265965938 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 -- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 msgr2=0x559316f05d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f05d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10afe00640 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 -- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 msgr2=0x559316f022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.118+0000 7f10b7400640 1 -- 172.21.15.43:0/4265965938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559316e0eb80 con 0x559316f06ba0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10b7400640 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009960 tx=0x7f10a8052040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1747e88887212c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10acc00640 1 -- 172.21.15.43:0/4265965938 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80457d0 con 0x559316f06ba0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10acc00640 1 -- 172.21.15.43:0/4265965938 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8045b60 con 0x559316f06ba0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10acc00640 1 -- 172.21.15.43:0/4265965938 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8050680 con 0x559316f06ba0 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 -- 172.21.15.43:0/4265965938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 msgr2=0x559316f51140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009960 tx=0x7f10a8052040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 -- 172.21.15.43:0/4265965938 shutdown_connections 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0b5e0 0x559316f022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0bea0 0x559316f05d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 --2- 172.21.15.43:0/4265965938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316f51140 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.119+0000 7f10c1626740 1 -- 172.21.15.43:0/4265965938 >> 172.21.15.43:0/4265965938 conn(0x559316ecdc70 msgr2=0x559316f0a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.120+0000 7f10c1626740 1 -- 172.21.15.43:0/4265965938 shutdown_connections 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.120+0000 7f10c1626740 1 -- 172.21.15.43:0/4265965938 wait complete. 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.120+0000 7f10c1626740 1 -- start start 2024-01-16T14:59:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.120+0000 7f10c1626740 1 Processor -- start 2024-01-16T14:59:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f3a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0bea0 0x559316f3a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316fbcca0 con 0x559316f06ba0 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x559316f55580 con 0x559316f0bea0 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x559316f6ae20 con 0x559316f0b5e0 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.121+0000 7f10c1626740 10 client.0 _ll_register_callbacks cb 0x7f10a803a420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44722/0 (socket says 172.21.15.43:44722) 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 -- 172.21.15.43:0/3793078658 learned_addr learned my addr 172.21.15.43:0/3793078658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 -- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 msgr2=0x559316f3a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f3a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 -- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0bea0 msgr2=0x559316f3a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0bea0 0x559316f3a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 -- 172.21.15.43:0/3793078658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559316e71140 con 0x559316f06ba0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10b7400640 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009930 tx=0x7f10a8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70930d89ca9b105 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.122+0000 7f10a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80508a0 con 0x559316f06ba0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a804b040 con 0x559316f06ba0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10a8049070 con 0x559316f06ba0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10a8050d00 con 0x559316f06ba0 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 10 client.4671 fetch_fsmap learned FSMap version 8 2024-01-16T14:59:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x559316e34170 con 0x559316f06ba0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8049070 con 0x559316f06ba0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f10a80688a0 con 0x559316f06ba0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 10 client.4671 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 10 client.4671 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 20 client.4671 populate_metadata read hostname 'smithi043' 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f485f0 0x559316f489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 4 client.4671 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.123+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 --> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559316b736c0 con 0x559316f485f0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.124+0000 7f10afe00640 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f485f0 0x559316f489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.125+0000 7f10afe00640 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f485f0 0x559316f489d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f94cff1ff6d2f11b server_cookie=4559a20f69d7d311 in_seq=0 out_seq=0 2024-01-16T14:59:45.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.125+0000 7f10a4c00640 10 client.4671 ms_handle_connect on v2:172.21.15.69:6832/971655709 2024-01-16T14:59:45.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.125+0000 7f10a4c00640 1 -- 172.21.15.43:0/3793078658 <== mds.0 v2:172.21.15.69:6832/971655709 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x559316b736c0 con 0x559316f485f0 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.125+0000 7f10a4c00640 10 client.4671 handle_command_reply: tid=0 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.125+0000 7f10c1626740 1 client.4671 shutdown 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 msgr2=0x559316fbb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009930 tx=0x7f10a8008030 comp rx=0 tx=0).stop 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 shutdown_connections 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:6832/971655709,v1:172.21.15.69:6834/971655709] conn(0x559316f485f0 0x559316f489d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559316f0bea0 0x559316f3a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x559316f0b5e0 0x559316f3a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 --2- 172.21.15.43:0/3793078658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559316f06ba0 0x559316fbb150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10a4c00640 0 client.4671 ms_handle_reset on v2:172.21.15.69:6832/971655709 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 >> 172.21.15.43:0/3793078658 conn(0x559316ecdc70 msgr2=0x559316f30e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 shutdown_connections 2024-01-16T14:59:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 1 -- 172.21.15.43:0/3793078658 wait complete. 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.126+0000 7f10c1626740 20 client.4671 trim_cache size 0 max 16384 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4270, 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 17.418734413999999, 2024-01-16T14:59:45.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 8, 2024-01-16T14:59:45.112 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 22, 2024-01-16T14:59:45.112 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 22, 2024-01-16T14:59:45.112 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25.068686203999999 2024-01-16T14:59:45.112 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 msgr2=0x7f109006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 0x7f109006d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10a8036000 tx=0x7f10a803c040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 msgr2=0x7f10b815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10a000d800 tx=0x7f10a000dcc0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 shutdown_connections 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f109006b2c0 0x7f109006d780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10b8165430 0x7f10b8163a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10b80acb50 0x7f10b815a020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 --2- 172.21.15.43:0/1852511171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10b806d520 0x7f10b809cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 >> 172.21.15.43:0/1852511171 conn(0x7f10b80b0d80 msgr2=0x7f10b8160110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 shutdown_connections 2024-01-16T14:59:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.127+0000 7f10be800640 1 -- 172.21.15.43:0/1852511171 wait complete. 2024-01-16T14:59:45.125 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-16T14:59:45.158 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-16T14:59:45.158 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-16T14:59:45.158 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-16T14:59:45.158 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:59:45.158 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:45.159 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:45.159 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:45.159 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T14:59:45.159 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:59:45.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:45.160 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T14:59:45.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:45.219 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-01-16T14:59:45.320 INFO:teuthology.orchestra.run.smithi069.stderr:Cannot find device "ceph-brx" 2024-01-16T14:59:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:45.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:45.325 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T14:59:45.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:45.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-01-16T14:59:45.482 INFO:teuthology.orchestra.run.smithi043.stderr:Cannot find device "ceph-brx" 2024-01-16T14:59:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:59:45.486 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-16T14:59:45.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:59:45.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.731+0000 7f863d400640 1 Processor -- start 2024-01-16T14:59:45.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.732+0000 7f863d400640 1 -- start start 2024-01-16T14:59:45.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f863815d5b0 0x7f86380d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863d400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86380cf0c0 con 0x7f86380b1650 2024-01-16T14:59:45.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863d400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86380b01b0 con 0x7f86380d14a0 2024-01-16T14:59:45.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863d400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86380d05c0 con 0x7f863815d5b0 2024-01-16T14:59:45.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863ca00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f86380b1650 0x7f86380cfeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39296/0 (socket says 172.21.15.43:39296) 2024-01-16T14:59:45.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f863ca00640 1 -- 172.21.15.43:0/3378692183 learned_addr learned my addr 172.21.15.43:0/3378692183 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:45.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.733+0000 7f8637e00640 1 --2- 172.21.15.43:0/3378692183 >> v2:172.21.15.69:3300/0 conn(0x7f863815d5b0 0x7f86380d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109309332 0 0) 0x7f86380cf0c0 con 0x7f86380b1650 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8618003520 con 0x7f86380b1650 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084998910 0 0) 0x7f8618003520 con 0x7f86380b1650 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 >> v2:172.21.15.69:3300/0 conn(0x7f863815d5b0 msgr2=0x7f86380d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 --2- 172.21.15.43:0/3378692183 >> v2:172.21.15.69:3300/0 conn(0x7f863815d5b0 0x7f86380d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 >> v1:172.21.15.69:6789/0 conn(0x7f86380d14a0 legacy=0x7f86380cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863815d990 con 0x7f86380b1650 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f86300019f0 con 0x7f86380b1650 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 >> v1:172.21.15.43:6789/0 conn(0x7f86380b1650 legacy=0x7f86380cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f8618003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86180047c0 0x7f8618004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8618005180 0x7f861800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86380d05c0 con 0x7f86180039e0 2024-01-16T14:59:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861800c0b0 con 0x7f86180047c0 2024-01-16T14:59:45.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.734+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861800c270 con 0x7f8618005180 2024-01-16T14:59:45.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f8618003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 -- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8618005180 msgr2=0x7f861800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8618005180 0x7f861800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 -- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86180047c0 msgr2=0x7f8618004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86180047c0 0x7f8618004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f8637e00640 1 -- 172.21.15.43:0/3378692183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618003520 con 0x7f86180039e0 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.735+0000 7f863ca00640 1 -- 172.21.15.43:0/3378692183 reap_dead start 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.736+0000 7f8637e00640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f8618003e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8628009ac0 tx=0x7f86180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfd01d403156710 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.736+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862802c6d0 con 0x7f86180039e0 2024-01-16T14:59:45.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.736+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862802c830 con 0x7f86180039e0 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.736+0000 7f8637400640 1 -- 172.21.15.43:0/3378692183 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862802cbc0 con 0x7f86180039e0 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 -- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 msgr2=0x7f8618003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f8618003e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8628009ac0 tx=0x7f86180057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 -- 172.21.15.43:0/3378692183 shutdown_connections 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 --2- 172.21.15.43:0/3378692183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f8618003e00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 -- 172.21.15.43:0/3378692183 >> 172.21.15.43:0/3378692183 conn(0x7f863809c3b0 msgr2=0x7f86380d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 -- 172.21.15.43:0/3378692183 shutdown_connections 2024-01-16T14:59:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.737+0000 7f863d400640 1 -- 172.21.15.43:0/3378692183 wait complete. 2024-01-16T14:59:45.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.739+0000 7f863d400640 1 Processor -- start 2024-01-16T14:59:45.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.739+0000 7f863d400640 1 -- start start 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86380cc9c0 0x7f863815a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 0x7f863806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863806c120 con 0x7f86180039e0 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86380cce40 con 0x7f86380cc9c0 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f863d400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86380b2c30 con 0x7f863806b760 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f8637e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f8637e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44734/0 (socket says 172.21.15.43:44734) 2024-01-16T14:59:45.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.740+0000 7f8637e00640 1 -- 172.21.15.43:0/2624003021 learned_addr learned my addr 172.21.15.43:0/2624003021 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f862f200640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86380cc9c0 0x7f863815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f863ca00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 0x7f863806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 -- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 msgr2=0x7f863806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 0x7f863806bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 -- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86380cc9c0 msgr2=0x7f863815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86380cc9c0 0x7f863815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 -- 172.21.15.43:0/2624003021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86380b2f10 con 0x7f86180039e0 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f863ca00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 0x7f863806bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:45.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8637e00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f862802c270 tx=0x7f8628004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.741+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628003810 con 0x7f86180039e0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.742+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8628003970 con 0x7f86180039e0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.742+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628003ce0 con 0x7f86180039e0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.742+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86380b31f0 con 0x7f86180039e0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.742+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86380b37b0 con 0x7f86180039e0 2024-01-16T14:59:45.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.743+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863804e840 con 0x7f86180039e0 2024-01-16T14:59:45.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.744+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8628040450 con 0x7f86180039e0 2024-01-16T14:59:45.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.745+0000 7f8634c00640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 0x7f861006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.745+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86280abdb0 con 0x7f86180039e0 2024-01-16T14:59:45.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.746+0000 7f862f200640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 0x7f861006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.746+0000 7f862f200640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 0x7f861006d7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f86380cd1d0 tx=0x7f8620009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.750+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8628076b80 con 0x7f86180039e0 2024-01-16T14:59:45.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.989+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f86380678d0 con 0x7f86180039e0 2024-01-16T14:59:45.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.990+0000 7f8634c00640 1 -- 172.21.15.43:0/2624003021 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8628044060 con 0x7f86180039e0 2024-01-16T14:59:45.970 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:59:45.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.992+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 msgr2=0x7f861006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.992+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 0x7f861006d7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f86380cd1d0 tx=0x7f8620009040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.992+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 msgr2=0x7f86380b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.992+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f862802c270 tx=0x7f8628004730 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 shutdown_connections 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f861006b2f0 0x7f861006d7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86380cc9c0 0x7f863815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f863806b760 0x7f863806bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 --2- 172.21.15.43:0/2624003021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86180039e0 0x7f86380b25a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 >> 172.21.15.43:0/2624003021 conn(0x7f863809c3b0 msgr2=0x7f863815db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 shutdown_connections 2024-01-16T14:59:45.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:45.993+0000 7f863d400640 1 -- 172.21.15.43:0/2624003021 wait complete. 2024-01-16T14:59:45.994 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:59:45.994 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:59:45.994 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T14:59:45.994 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T14:59:45.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:45.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:59:45.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:59:45.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:59:45.995 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:59:45.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:45.995 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:46.051 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42452sec preferred_lft 42452sec 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:46.052 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T14:59:46.053 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:59:46.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:46.053 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:46.053 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:46.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:59:46.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr flush dev ceph-brx 2024-01-16T14:59:46.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set ceph-brx up 2024-01-16T14:59:46.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:59:46.054 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:46.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:46.511 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:59:46.592 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T14:59:46.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:46.596 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T14:59:46.651 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T14:59:46.651 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:59:46.651 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T14:59:46.651 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T14:59:46.651 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:59:46.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T14:59:46.652 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:47.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:47.403 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T14:59:47.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:47.462 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-01-16T14:59:47.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:47.519 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:47.520 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:47.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:47.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-16T14:59:47.520 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:47.670 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-16T14:59:47.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-16T14:59:47.671 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:47.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:47.963 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:47.963 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:47.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T14:59:47.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:59:47.963 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:48.134 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T14:59:48.134 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:48.135 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:48.183 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T14:59:48.183 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:48.183 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:48.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T14:59:48.332 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T14:59:48.388 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:48.389 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:48.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:48.391 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T14:59:48.435 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:59:48.436 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T14:59:48.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T14:59:48.564 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T14:59:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T14:59:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T14:59:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T14:59:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:48.613 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:48.614 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:48.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:48.615 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T14:59:48.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T14:59:48.730+0000 7fc9176b0480 -1 init, newargv = 0x562b58aaf760 newargc=15 2024-01-16T14:59:48.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[43124]: starting ceph client 2024-01-16T14:59:48.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[43124]: starting fuse 2024-01-16T14:59:49.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T14:59:49.882 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T14:59:49.941 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T14:59:49.942 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:49.943 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:49.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:49.944 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T14:59:50.000 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T14:59:50.000 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:50.000 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T14:59:50.001 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T14:59:50.002 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:59:50.002 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:50.002 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T14:59:50.002 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T14:59:50.188 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.43124.asok 2024-01-16T14:59:50.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:59:50.206 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43124.asok 2024-01-16T14:59:50.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:50.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43124.asok status 2024-01-16T14:59:50.452 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T14:59:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T14:59:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:59:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:59:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "43124", 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4686, 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4686 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T14:59:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3614633645 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3614633645 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.4686 192.168.0.1:0/3614633645", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/3614633645", 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 8, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 22, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:59:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T14:59:50.455 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T14:59:50.455 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T14:59:50.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:59:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.703+0000 7f0490e00640 1 Processor -- start 2024-01-16T14:59:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.704+0000 7f0490e00640 1 -- start start 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f0490e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f048c0ad720 0x7f048c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f0490e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c15c360 con 0x7f048c0abff0 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f0490e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c15ba30 con 0x7f048c06d520 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f0490e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048c0b41a0 con 0x7f048c0ad720 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f048a800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f048c0abff0 0x7f048c0b25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39302/0 (socket says 172.21.15.43:39302) 2024-01-16T14:59:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.705+0000 7f048a800640 1 -- 172.21.15.43:0/1934214734 learned_addr learned my addr 172.21.15.43:0/1934214734 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0483e00640 1 --2- 172.21.15.43:0/1934214734 >> v2:172.21.15.69:3300/0 conn(0x7f048c0ad720 0x7f048c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316738730 0 0) 0x7f048c15c360 con 0x7f048c0abff0 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003520 con 0x7f048c0abff0 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181418257 0 0) 0x7f046c003520 con 0x7f048c0abff0 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 >> v2:172.21.15.69:3300/0 conn(0x7f048c0ad720 msgr2=0x7f048c0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 --2- 172.21.15.43:0/1934214734 >> v2:172.21.15.69:3300/0 conn(0x7f048c0ad720 0x7f048c0b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 >> v1:172.21.15.69:6789/0 conn(0x7f048c06d520 legacy=0x7f048c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c15bd10 con 0x7f048c0abff0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f04840019f0 con 0x7f048c0abff0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 >> v1:172.21.15.43:6789/0 conn(0x7f048c0abff0 legacy=0x7f048c0b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c003970 0x7f046c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c0047c0 0x7f046c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.706+0000 7f0489400640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 0x7f046c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c00c0a0 con 0x7f046c005180 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c00c210 con 0x7f046c003970 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c00c3d0 con 0x7f046c0047c0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0483e00640 1 --2- 172.21.15.43:0/1934214734 >> v2:172.21.15.69:3300/0 conn(0x7f048c0ad720 0x7f048c0b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 0x7f046c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0489e00640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c003970 0x7f046c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f0483e00640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c0047c0 0x7f046c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 -- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c0047c0 msgr2=0x7f046c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c0047c0 0x7f046c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 -- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c003970 msgr2=0x7f046c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f046c003970 0x7f046c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.707+0000 7f048a800640 1 -- 172.21.15.43:0/1934214734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c003520 con 0x7f046c005180 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.709+0000 7f048a800640 1 -- 172.21.15.43:0/1934214734 reap_dead start 2024-01-16T14:59:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.709+0000 7f048a800640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 0x7f046c00b980 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f048400a830 tx=0x7f046c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b664c5fa4a94ca3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.712+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840072d0 con 0x7f046c005180 2024-01-16T14:59:50.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.712+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048402f070 con 0x7f046c005180 2024-01-16T14:59:50.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.712+0000 7f0489400640 1 -- 172.21.15.43:0/1934214734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840076b0 con 0x7f046c005180 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 -- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 msgr2=0x7f046c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 0x7f046c00b980 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f048400a830 tx=0x7f046c0057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 -- 172.21.15.43:0/1934214734 shutdown_connections 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 --2- 172.21.15.43:0/1934214734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c005180 0x7f046c00b980 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 -- 172.21.15.43:0/1934214734 >> 172.21.15.43:0/1934214734 conn(0x7f048c0b0d80 msgr2=0x7f048c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 -- 172.21.15.43:0/1934214734 shutdown_connections 2024-01-16T14:59:50.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.713+0000 7f0490e00640 1 -- 172.21.15.43:0/1934214734 wait complete. 2024-01-16T14:59:50.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.715+0000 7f0490e00640 1 Processor -- start 2024-01-16T14:59:50.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.716+0000 7f0490e00640 1 -- start start 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c005180 0x7f048c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c15e1e0 0x7f048c0987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c155100 con 0x7f048c154730 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0ccd60 con 0x7f048c15e1e0 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0490e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0cd040 con 0x7f046c005180 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f048a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f048a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44754/0 (socket says 172.21.15.43:44754) 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f048a800640 1 -- 172.21.15.43:0/3295644654 learned_addr learned my addr 172.21.15.43:0/3295644654 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.717+0000 7f0489e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c005180 0x7f048c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f0483e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c15e1e0 0x7f048c0987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 -- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c005180 msgr2=0x7f048c0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c005180 0x7f048c0cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 -- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c15e1e0 msgr2=0x7f048c0987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c15e1e0 0x7f048c0987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 -- 172.21.15.43:0/3295644654 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0cd320 con 0x7f048c154730 2024-01-16T14:59:50.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f048a800640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0484000f80 tx=0x7f0484001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840019b0 con 0x7f048c154730 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484004b60 con 0x7f048c154730 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484007400 con 0x7f048c154730 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.718+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c0cd600 con 0x7f048c154730 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.719+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c15e660 con 0x7f048c154730 2024-01-16T14:59:50.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.720+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c074770 con 0x7f048c154730 2024-01-16T14:59:50.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.721+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0484031030 con 0x7f048c154730 2024-01-16T14:59:50.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.722+0000 7f0482000640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:50.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.722+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04840ab930 con 0x7f048c154730 2024-01-16T14:59:50.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.722+0000 7f0489e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:50.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.723+0000 7f0489e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 0x7f046406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f047c005490 tx=0x7f047c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:50.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.726+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0484076700 con 0x7f048c154730 2024-01-16T14:59:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.958+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f048c074770 con 0x7f048c154730 2024-01-16T14:59:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.959+0000 7f0482000640 1 -- 172.21.15.43:0/3295644654 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0484079d20 con 0x7f048c154730 2024-01-16T14:59:50.940 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:59:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.961+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 msgr2=0x7f046406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.961+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 0x7f046406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f047c005490 tx=0x7f047c009040 comp rx=0 tx=0).stop 2024-01-16T14:59:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 msgr2=0x7f048c154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0484000f80 tx=0x7f0484001d80 comp rx=0 tx=0).stop 2024-01-16T14:59:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 shutdown_connections 2024-01-16T14:59:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f046406b2c0 0x7f046406d780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c15e1e0 0x7f048c0987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f048c154730 0x7f048c154bc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 --2- 172.21.15.43:0/3295644654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f046c005180 0x7f048c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 >> 172.21.15.43:0/3295644654 conn(0x7f048c0b0d80 msgr2=0x7f048c0af120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 shutdown_connections 2024-01-16T14:59:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:50.962+0000 7f0490e00640 1 -- 172.21.15.43:0/3295644654 wait complete. 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:59:50.961 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:59:50.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:50.962 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T14:59:51.015 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T14:59:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.017 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:59:51.110 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T14:59:51.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.115 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T14:59:51.173 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T14:59:51.173 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:59:51.173 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T14:59:51.173 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T14:59:51.173 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:59:51.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T14:59:51.175 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:51.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.376 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T14:59:51.428 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T14:59:51.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.429 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-01-16T14:59:51.479 INFO:teuthology.orchestra.run.smithi043.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-16T14:59:51.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.480 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:51.480 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:51.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:59:51.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-16T14:59:51.480 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:51.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether ea:9d:ee:0a:d4:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:59:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::e89d:eeff:fe0a:d4a0/64 scope link 2024-01-16T14:59:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether ea:9d:ee:0a:d4:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:59:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:59:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::e89d:eeff:fe0a:d4a0/64 scope link 2024-01-16T14:59:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:51.833 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-16T14:59:51.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-16T14:59:51.833 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:52.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:52.108 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T14:59:52.109 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T14:59:52.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.1 up 2024-01-16T14:59:52.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:59:52.109 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T14:59:52.280 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-16T14:59:52.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:52.281 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:52.331 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-16T14:59:52.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:52.332 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:52.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T14:59:52.477 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T14:59:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T14:59:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T14:59:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T14:59:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T14:59:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:52.537 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T14:59:52.538 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:52.539 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:52.540 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:52.540 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:59:52.540 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:59:52.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:52.541 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T14:59:52.592 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T14:59:52.592 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-16T14:59:52.592 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-16T14:59:52.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T14:59:52.684 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:52.729 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T14:59:52.729 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T14:59:52.730 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T14:59:52.731 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:52.732 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T14:59:52.732 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:52.732 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:59:52.732 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T14:59:52.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:52.733 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T14:59:52.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi043.stderr:2024-01-16T14:59:52.824+0000 7fbb5c03a480 -1 init, newargv = 0x563f53334a70 newargc=15 2024-01-16T14:59:52.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi043.stderr:ceph-fuse[43670]: starting ceph client 2024-01-16T14:59:52.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi043.stderr:ceph-fuse[43670]: starting fuse 2024-01-16T14:59:52.825 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T14:59:52.826 INFO:teuthology.orchestra.run.smithi043.stdout:42 2024-01-16T14:59:52.826 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-16T14:59:52.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:59:52.827 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T14:59:52.827 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T14:59:52.828 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T14:59:52.828 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T14:59:52.828 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:52.828 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T14:59:52.829 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-16T14:59:52.829 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T14:59:52.829 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-16T14:59:52.829 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:52.830 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T14:59:52.830 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T14:59:52.830 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T14:59:52.830 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T14:59:52.831 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:52.831 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T14:59:52.831 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:59:52.831 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:59:52.832 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:59:52.832 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T14:59:52.832 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T14:59:52.832 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T14:59:52.833 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:59:52.833 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T14:59:52.833 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.2")) 2024-01-16T14:59:52.833 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T14:59:53.029 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.2.43670.asok 2024-01-16T14:59:53.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:59:53.048 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.43670.asok 2024-01-16T14:59:53.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:53.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.43670.asok status 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "2", 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "43670", 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4701, 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4701 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1434147149 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1434147149 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.4701 192.168.0.2:0/1434147149", 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.2:0/1434147149", 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 8, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 22, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T14:59:53.288 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T14:59:53.288 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T14:59:53.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:59:53.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.520+0000 7f6f15200640 1 Processor -- start 2024-01-16T14:59:53.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.520+0000 7f6f15200640 1 -- start start 2024-01-16T14:59:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f15200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6f1013dfd0 0x7f6f1015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f15200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f100ce7c0 con 0x7f6f100cf500 2024-01-16T14:59:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f15200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f10140900 con 0x7f6f100cf050 2024-01-16T14:59:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f15200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1006cf50 con 0x7f6f1013dfd0 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f0f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6f1013dfd0 0x7f6f1015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f0fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6f100cf500 0x7f6f1006c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40440/0 (socket says 172.21.15.43:40440) 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.522+0000 7f6f0fe00640 1 -- 172.21.15.43:0/3821550042 learned_addr learned my addr 172.21.15.43:0/3821550042 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.523+0000 7f6f0f400640 1 -- 172.21.15.43:0/3821550042 >> v1:172.21.15.69:6789/0 conn(0x7f6f100cf050 legacy=0x7f6f1006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.523+0000 7f6f0f400640 1 -- 172.21.15.43:0/3821550042 >> v1:172.21.15.43:6789/0 conn(0x7f6f100cf500 legacy=0x7f6f1006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.523+0000 7f6f0f400640 1 -- 172.21.15.43:0/3821550042 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1015fa70 con 0x7f6f1013dfd0 2024-01-16T14:59:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.523+0000 7f6f0e000640 1 -- 172.21.15.43:0/3821550042 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641935121 0 0) 0x7f6f100ce7c0 con 0x7f6f100cf500 2024-01-16T14:59:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.524+0000 7f6f0f400640 1 --2- 172.21.15.43:0/3821550042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f1015be70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71dc8e97c6babcfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f0e000640 1 -- 172.21.15.43:0/3821550042 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00030610 con 0x7f6f1013dfd0 2024-01-16T14:59:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f0e000640 1 -- 172.21.15.43:0/3821550042 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00030770 con 0x7f6f1013dfd0 2024-01-16T14:59:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f0e000640 1 -- 172.21.15.43:0/3821550042 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00030b00 con 0x7f6f1013dfd0 2024-01-16T14:59:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f15200640 1 -- 172.21.15.43:0/3821550042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 msgr2=0x7f6f1015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f15200640 1 --2- 172.21.15.43:0/3821550042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f1015be70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6f000097a0 tx=0x7f6f0002b960 comp rx=0 tx=0).stop 2024-01-16T14:59:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f15200640 1 -- 172.21.15.43:0/3821550042 shutdown_connections 2024-01-16T14:59:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f15200640 1 --2- 172.21.15.43:0/3821550042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f1015be70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.525+0000 7f6f15200640 1 -- 172.21.15.43:0/3821550042 >> 172.21.15.43:0/3821550042 conn(0x7f6f1006a860 msgr2=0x7f6f1006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.526+0000 7f6f15200640 1 -- 172.21.15.43:0/3821550042 shutdown_connections 2024-01-16T14:59:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.526+0000 7f6f15200640 1 -- 172.21.15.43:0/3821550042 wait complete. 2024-01-16T14:59:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.527+0000 7f6f15200640 1 Processor -- start 2024-01-16T14:59:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.527+0000 7f6f15200640 1 -- start start 2024-01-16T14:59:53.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f100cf050 0x7f6f100b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f100cf500 0x7f6f10152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100700d0 con 0x7f6f1013dfd0 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f10070c50 con 0x7f6f100cf050 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.528+0000 7f6f15200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f10070320 con 0x7f6f100cf500 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35942/0 (socket says 172.21.15.43:35942) 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 -- 172.21.15.43:0/1032113949 learned_addr learned my addr 172.21.15.43:0/1032113949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0f400640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f100cf050 0x7f6f100b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0ea00640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f100cf500 0x7f6f10152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 -- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f100cf500 msgr2=0x7f6f10152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f100cf500 0x7f6f10152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 -- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f100cf050 msgr2=0x7f6f100b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f100cf050 0x7f6f100b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.529+0000 7f6f0fe00640 1 -- 172.21.15.43:0/1032113949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009400 con 0x7f6f1013dfd0 2024-01-16T14:59:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f0fe00640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c7d0 tx=0x7f6f0800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0800a640 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0800a7a0 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f10070600 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0800ab90 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.530+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f10130410 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.531+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1004e840 con 0x7f6f1013dfd0 2024-01-16T14:59:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.532+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6f0800acf0 con 0x7f6f1013dfd0 2024-01-16T14:59:53.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.533+0000 7f6f07400640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 0x7f6ee806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.533+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f08089200 con 0x7f6f1013dfd0 2024-01-16T14:59:53.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.534+0000 7f6f0f400640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 0x7f6ee806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.534+0000 7f6f0f400640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 0x7f6ee806d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.539+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f08053fd0 con 0x7f6f1013dfd0 2024-01-16T14:59:53.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:53.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:53.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.788+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6f10070d90 con 0x7f6f1013dfd0 2024-01-16T14:59:53.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.788+0000 7f6f07400640 1 -- 172.21.15.43:0/1032113949 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6f08018060 con 0x7f6f1013dfd0 2024-01-16T14:59:53.770 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:59:53.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 msgr2=0x7f6ee806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 0x7f6ee806d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f00008030 comp rx=0 tx=0).stop 2024-01-16T14:59:53.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 msgr2=0x7f6f10154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c7d0 tx=0x7f6f0800cc90 comp rx=0 tx=0).stop 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 shutdown_connections 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ee806b2c0 0x7f6ee806d780 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f1013dfd0 0x7f6f10154aa0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f100cf500 0x7f6f10152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 --2- 172.21.15.43:0/1032113949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f100cf050 0x7f6f100b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 >> 172.21.15.43:0/1032113949 conn(0x7f6f1006a860 msgr2=0x7f6f1013f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 shutdown_connections 2024-01-16T14:59:53.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:53.791+0000 7f6f15200640 1 -- 172.21.15.43:0/1032113949 wait complete. 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:59:53.790 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:59:53.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:53.791 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T14:59:53.841 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:53.841 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:53.841 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:53.841 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:53.842 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:59:53.843 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42606sec preferred_lft 42606sec 2024-01-16T14:59:53.843 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T14:59:53.843 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:53.844 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T14:59:53.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T14:59:53.844 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:54.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:54.333 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:59:54.410 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T14:59:54.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:54.414 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T14:59:54.468 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T14:59:54.468 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:59:54.468 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:59:54.468 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:59:54.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:54.469 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:54.469 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:54.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:59:54.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:59:54.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:59:54.470 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:55.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:55.154 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T14:59:55.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:55.208 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-01-16T14:59:55.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:55.258 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:55.258 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:55.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:55.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-16T14:59:55.259 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:55.396 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-16T14:59:55.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-16T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:55.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:55.675 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:55.675 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:55.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T14:59:55.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T14:59:55.675 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:55.839 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T14:59:55.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:55.839 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:55.886 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T14:59:55.886 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:55.887 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:55.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T14:59:56.080 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T14:59:56.128 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T14:59:56.129 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:59:56.130 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:59:56.131 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T14:59:56.131 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:56.131 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:56.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.132 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T14:59:56.230 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T14:59:56.230 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T14:59:56.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T14:59:56.312 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T14:59:56.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T14:59:56.362 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T14:59:56.363 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:56.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.365 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T14:59:56.428 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T14:59:56.447+0000 7fb423a37480 -1 init, newargv = 0x558f0be02c10 newargc=15 2024-01-16T14:59:56.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[41518]: starting ceph client 2024-01-16T14:59:56.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[41518]: starting fuse 2024-01-16T14:59:56.448 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T14:59:56.449 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T14:59:56.449 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T14:59:56.450 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T14:59:56.451 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T14:59:56.451 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:59:56.451 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:56.451 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T14:59:56.451 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T14:59:56.648 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.41518.asok 2024-01-16T14:59:56.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:59:56.665 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41518.asok 2024-01-16T14:59:56.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:56.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41518.asok status 2024-01-16T14:59:56.899 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "41518", 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:56.900 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4603, 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4603 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T14:59:56.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2292032231 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2292032231 2024-01-16T14:59:56.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4603 192.168.0.1:0/2292032231", 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2292032231", 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 8, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 22, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T14:59:56.903 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T14:59:56.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T14:59:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.166+0000 7f855ce00640 1 Processor -- start 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.166+0000 7f855ce00640 1 -- start start 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.167+0000 7f855ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f855813dfd0 0x7f855815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.167+0000 7f855ce00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580ce7c0 con 0x7f85580cf500 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f855ce00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558140900 con 0x7f85580cf050 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f855ce00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855806cf50 con 0x7f855813dfd0 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f8557e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f85580cf500 0x7f855806c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40452/0 (socket says 172.21.15.43:40452) 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f8557e00640 1 -- 172.21.15.43:0/3066440786 learned_addr learned my addr 172.21.15.43:0/3066440786 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f8557400640 1 --2- 172.21.15.43:0/3066440786 >> v2:172.21.15.69:3300/0 conn(0x7f855813dfd0 0x7f855815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034735499 0 0) 0x7f85580ce7c0 con 0x7f85580cf500 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.168+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538003520 con 0x7f85580cf500 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589967419 0 0) 0x7f8538003520 con 0x7f85580cf500 2024-01-16T14:59:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 >> v2:172.21.15.69:3300/0 conn(0x7f855813dfd0 msgr2=0x7f855815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 --2- 172.21.15.43:0/3066440786 >> v2:172.21.15.69:3300/0 conn(0x7f855813dfd0 0x7f855815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 >> v1:172.21.15.69:6789/0 conn(0x7f85580cf050 legacy=0x7f855806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855815fa70 con 0x7f85580cf500 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f85500019f0 con 0x7f85580cf500 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 >> v1:172.21.15.43:6789/0 conn(0x7f85580cf500 legacy=0x7f855806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8538003a30 0x7f8538003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 0x7f8538004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85380051d0 0x7f853800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855806cf50 con 0x7f85380047c0 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853800c100 con 0x7f85380051d0 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853800c2c0 con 0x7f8538003a30 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.169+0000 7f854fe00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 0x7f8538004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f8557e00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85380051d0 0x7f853800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f854fe00640 1 -- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8538003a30 msgr2=0x7f8538003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f854fe00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8538003a30 0x7f8538003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f854fe00640 1 -- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85380051d0 msgr2=0x7f853800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f854fe00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85380051d0 0x7f853800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f854fe00640 1 -- 172.21.15.43:0/3066440786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538003520 con 0x7f85380047c0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f8557e00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85380051d0 0x7f853800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.170+0000 7f8557e00640 1 -- 172.21.15.43:0/3066440786 reap_dead start 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.171+0000 7f854fe00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 0x7f8538004c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8540001750 tx=0x7f85380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f57f538f1ea6576 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.171+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400052d0 con 0x7f85380047c0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.171+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854002e050 con 0x7f85380047c0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.171+0000 7f8556a00640 1 -- 172.21.15.43:0/3066440786 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540003830 con 0x7f85380047c0 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 -- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 msgr2=0x7f8538004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 0x7f8538004c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8540001750 tx=0x7f85380057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 -- 172.21.15.43:0/3066440786 shutdown_connections 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 --2- 172.21.15.43:0/3066440786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85380047c0 0x7f8538004c40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 -- 172.21.15.43:0/3066440786 >> 172.21.15.43:0/3066440786 conn(0x7f855806a860 msgr2=0x7f855806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 -- 172.21.15.43:0/3066440786 shutdown_connections 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.172+0000 7f855ce00640 1 -- 172.21.15.43:0/3066440786 wait complete. 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.174+0000 7f855ce00640 1 Processor -- start 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.174+0000 7f855ce00640 1 -- start start 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85380047c0 0x7f8558156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 0x7f855815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580700d0 con 0x7f85581f2a70 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855814d9e0 con 0x7f85581f2ef0 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f855ce00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8558070930 con 0x7f85380047c0 2024-01-16T14:59:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f854fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f854fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35964/0 (socket says 172.21.15.43:35964) 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.175+0000 7f854fe00640 1 -- 172.21.15.43:0/3255263378 learned_addr learned my addr 172.21.15.43:0/3255263378 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f8557400640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85380047c0 0x7f8558156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f8557e00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 0x7f855815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 -- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85380047c0 msgr2=0x7f8558156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85380047c0 0x7f8558156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 -- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 msgr2=0x7f855815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 0x7f855815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 -- 172.21.15.43:0/3255263378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85581568a0 con 0x7f85581f2a70 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f8557e00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 0x7f855815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854fe00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8540001870 tx=0x7f8540001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.176+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540007640 con 0x7f85581f2a70 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.177+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400069e0 con 0x7f85581f2a70 2024-01-16T14:59:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.177+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540006d70 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.177+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855815ae30 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.177+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85580705c0 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.178+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8558070ac0 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.179+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f85400035f0 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.180+0000 7f854f400640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 0x7f853006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.180+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85400ab840 con 0x7f85581f2a70 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.180+0000 7f8557400640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 0x7f853006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.181+0000 7f8557400640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 0x7f853006d7b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8548009ad0 tx=0x7f8548008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.184+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8540076580 con 0x7f85581f2a70 2024-01-16T14:59:57.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.417+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f855804e840 con 0x7f85581f2a70 2024-01-16T14:59:57.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.418+0000 7f854f400640 1 -- 172.21.15.43:0/3255263378 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8540041070 con 0x7f85581f2a70 2024-01-16T14:59:57.399 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T14:59:57.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 msgr2=0x7f853006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 0x7f853006d7b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8548009ad0 tx=0x7f8548008040 comp rx=0 tx=0).stop 2024-01-16T14:59:57.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 msgr2=0x7f855815a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8540001870 tx=0x7f8540001120 comp rx=0 tx=0).stop 2024-01-16T14:59:57.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 shutdown_connections 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f853006b2f0 0x7f853006d7b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85581f2ef0 0x7f855815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85581f2a70 0x7f855815a200 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 --2- 172.21.15.43:0/3255263378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85380047c0 0x7f8558156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 >> 172.21.15.43:0/3255263378 conn(0x7f855806a860 msgr2=0x7f85581401a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 shutdown_connections 2024-01-16T14:59:57.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T14:59:57.421+0000 7f855ce00640 1 -- 172.21.15.43:0/3255263378 wait complete. 2024-01-16T14:59:57.420 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T14:59:57.420 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T14:59:57.420 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T14:59:57.420 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-16T14:59:57.420 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T14:59:57.421 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T14:59:57.421 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T14:59:57.421 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T14:59:57.421 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:59:57.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.421 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T14:59:57.478 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:57.478 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T14:59:57.479 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T14:59:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:57.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.482 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T14:59:57.573 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T14:59:57.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.577 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T14:59:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T14:59:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T14:59:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T14:59:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T14:59:57.627 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T14:59:57.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T14:59:57.628 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:57.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.811 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T14:59:57.860 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T14:59:57.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.861 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-01-16T14:59:57.906 INFO:teuthology.orchestra.run.smithi069.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-16T14:59:57.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:57.908 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:57.908 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:57.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:59:57.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-16T14:59:57.908 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:58.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:58.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.180 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:58.181 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 5e:40:82:7b:3b:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:59:58.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:59:58.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c40:82ff:fe7b:3b1d/64 scope link 2024-01-16T14:59:58.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:58.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T14:59:58.279 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T14:59:58.279 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 5e:40:82:7b:3b:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c40:82ff:fe7b:3b1d/64 scope link 2024-01-16T14:59:58.280 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T14:59:58.284 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-16T14:59:58.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-16T14:59:58.285 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:58.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:58.547 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T14:59:58.547 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T14:59:58.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.1 up 2024-01-16T14:59:58.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T14:59:58.547 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T14:59:58.710 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-16T14:59:58.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:58.711 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-16T14:59:58.758 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-16T14:59:58.758 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T14:59:58.758 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-16T14:59:58.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T14:59:58.895 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T14:59:58.949 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:58.950 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:59:58.951 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:59:58.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:58.953 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T14:59:59.003 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T14:59:59.003 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-16T14:59:59.004 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-16T14:59:59.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T14:59:59.084 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T14:59:59.124 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T14:59:59.124 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T14:59:59.124 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T14:59:59.124 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T14:59:59.125 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:59:59.126 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T14:59:59.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.127 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T14:59:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:2024-01-16T14:59:59.209+0000 7fe65663f480 -1 init, newargv = 0x561a3d683b70 newargc=15 2024-01-16T14:59:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[41970]: starting ceph client 2024-01-16T14:59:59.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[41970]: starting fuse 2024-01-16T14:59:59.211 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T14:59:59.211 INFO:teuthology.orchestra.run.smithi069.stdout:42 2024-01-16T14:59:59.212 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:59.212 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.3")) 2024-01-16T14:59:59.213 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T14:59:59.413 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.3.41970.asok 2024-01-16T14:59:59.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T14:59:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T14:59:59.429 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41970.asok 2024-01-16T14:59:59.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41970.asok status 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "3", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "41970", 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T14:59:59.665 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4469, 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4469 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 216788476 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T14:59:59.666 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 216788476 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4469 192.168.0.2:0/216788476", 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.2:0/216788476", 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 8, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 22, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T14:59:59.667 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T14:59:59.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.680 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T14:59:59.680 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:59.734 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T14:59:59.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:59.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T14:59:59.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.834 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T14:59:59.834 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:59.891 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T14:59:59.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:59.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-16T14:59:59.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T14:59:59.994 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T14:59:59.994 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:00.052 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:00:00.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:00.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:00.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:00.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:00.147 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:00:00.147 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:00.200 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:00:00.200 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:00.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:00.201 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:00.293 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-16T15:00:00.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.591+0000 7ff014800640 1 Processor -- start 2024-01-16T15:00:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.591+0000 7ff014800640 1 -- start start 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff014800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff00c15a100 0x7ff00c0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff014800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0abff0 0x7ff00c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff014800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c13da30 con 0x7ff00c0ae7e0 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff014800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00c0ac840 con 0x7ff00c15a100 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff014800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0ac560 con 0x7ff00c0abff0 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff011e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff00c15a100 0x7ff00c0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff011e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff00c15a100 0x7ff00c0b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35968/0 (socket says 172.21.15.43:35968) 2024-01-16T15:00:00.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.593+0000 7ff011e00640 1 -- 172.21.15.43:0/2296607875 learned_addr learned my addr 172.21.15.43:0/2296607875 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:00.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.594+0000 7ff011400640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0abff0 0x7ff00c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.594+0000 7ff011e00640 1 -- 172.21.15.43:0/2296607875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0abff0 msgr2=0x7ff00c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:00.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.594+0000 7ff011e00640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0abff0 0x7ff00c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.594+0000 7ff011e00640 1 -- 172.21.15.43:0/2296607875 >> v1:172.21.15.69:6789/0 conn(0x7ff00c0ae7e0 legacy=0x7ff00c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.594+0000 7ff011e00640 1 -- 172.21.15.43:0/2296607875 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c070650 con 0x7ff00c15a100 2024-01-16T15:00:00.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.596+0000 7ff011e00640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c0b3290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feffc009870 tx=0x7feffc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d106160a892e3a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.596+0000 7ff010a00640 1 -- 172.21.15.43:0/2296607875 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc031640 con 0x7ff00c15a100 2024-01-16T15:00:00.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.596+0000 7ff010a00640 1 -- 172.21.15.43:0/2296607875 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc0317a0 con 0x7ff00c15a100 2024-01-16T15:00:00.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.596+0000 7ff010a00640 1 -- 172.21.15.43:0/2296607875 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc031b30 con 0x7ff00c15a100 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 -- 172.21.15.43:0/2296607875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 msgr2=0x7ff00c0b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c0b3290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7feffc009870 tx=0x7feffc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 -- 172.21.15.43:0/2296607875 shutdown_connections 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0abff0 0x7ff00c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 --2- 172.21.15.43:0/2296607875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c0b3290 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 -- 172.21.15.43:0/2296607875 >> 172.21.15.43:0/2296607875 conn(0x7ff00c0b0d80 msgr2=0x7ff00c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 -- 172.21.15.43:0/2296607875 shutdown_connections 2024-01-16T15:00:00.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.597+0000 7ff014800640 1 -- 172.21.15.43:0/2296607875 wait complete. 2024-01-16T15:00:00.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.599+0000 7ff014800640 1 Processor -- start 2024-01-16T15:00:00.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.599+0000 7ff014800640 1 -- start start 2024-01-16T15:00:00.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff00c0abff0 0x7ff00c0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0ae7e0 0x7ff00c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c15aed0 con 0x7ff00c15a100 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c15b3e0 con 0x7ff00c0abff0 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff014800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0c9fe0 con 0x7ff00c0ae7e0 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff012800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff012800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35980/0 (socket says 172.21.15.43:35980) 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff012800640 1 -- 172.21.15.43:0/167953909 learned_addr learned my addr 172.21.15.43:0/167953909 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff011400640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0ae7e0 0x7ff00c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.600+0000 7ff011e00640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff00c0abff0 0x7ff00c0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 -- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0ae7e0 msgr2=0x7ff00c15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0ae7e0 0x7ff00c15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 -- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff00c0abff0 msgr2=0x7ff00c0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff00c0abff0 0x7ff00c0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 -- 172.21.15.43:0/167953909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0094d0 con 0x7ff00c15a100 2024-01-16T15:00:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff012800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff00400c6b0 tx=0x7ff00400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004003b70 con 0x7ff00c15a100 2024-01-16T15:00:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004003cf0 con 0x7ff00c15a100 2024-01-16T15:00:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004005310 con 0x7ff00c15a100 2024-01-16T15:00:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.601+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c0ca2c0 con 0x7ff00c15a100 2024-01-16T15:00:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.602+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c1f2ef0 con 0x7ff00c15a100 2024-01-16T15:00:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.603+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00c067960 con 0x7ff00c15a100 2024-01-16T15:00:00.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.604+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff004005470 con 0x7ff00c15a100 2024-01-16T15:00:00.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.605+0000 7ff00a000640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 0x7fefe406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.605+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff004089aa0 con 0x7ff00c15a100 2024-01-16T15:00:00.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.606+0000 7ff011e00640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 0x7fefe406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.606+0000 7ff011e00640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 0x7fefe406d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feffc036000 tx=0x7feffc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.610+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0040547e0 con 0x7ff00c15a100 2024-01-16T15:00:00.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.808+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff00c0ca7e0 con 0x7ff00c15a100 2024-01-16T15:00:00.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.810+0000 7ff00a000640 1 -- 172.21.15.43:0/167953909 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7ff00c0ca7e0 con 0x7ff00c15a100 2024-01-16T15:00:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:00.791 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 msgr2=0x7fefe406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 0x7fefe406d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feffc036000 tx=0x7feffc03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 msgr2=0x7ff00c1f29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff00400c6b0 tx=0x7ff00400cb70 comp rx=0 tx=0).stop 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 shutdown_connections 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe406b2c0 0x7fefe406d780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00c15a100 0x7ff00c1f29b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff00c0ae7e0 0x7ff00c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 --2- 172.21.15.43:0/167953909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff00c0abff0 0x7ff00c0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 >> 172.21.15.43:0/167953909 conn(0x7ff00c0b0d80 msgr2=0x7ff00c06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 shutdown_connections 2024-01-16T15:00:00.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:00.813+0000 7ff014800640 1 -- 172.21.15.43:0/167953909 wait complete. 2024-01-16T15:00:00.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:00.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.044+0000 7f09c3e00640 1 Processor -- start 2024-01-16T15:00:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.044+0000 7f09c3e00640 1 -- start start 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c406d9e0 0x7f09c4165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c3e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c415f930 con 0x7f09c415f270 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c3e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40b3b10 con 0x7f09c4144770 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c40b43b0 con 0x7f09c406d9e0 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c2000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f09c415f270 0x7f09c40cd4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40462/0 (socket says 172.21.15.43:40462) 2024-01-16T15:00:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.046+0000 7f09c2000640 1 -- 172.21.15.43:0/3071473690 learned_addr learned my addr 172.21.15.43:0/3071473690 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:01.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c3400640 1 --2- 172.21.15.43:0/3071473690 >> v2:172.21.15.69:3300/0 conn(0x7f09c406d9e0 0x7f09c4165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636738398 0 0) 0x7f09c415f930 con 0x7f09c415f270 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a4003520 con 0x7f09c415f270 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968494534 0 0) 0x7f09a4003520 con 0x7f09c415f270 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 >> v2:172.21.15.69:3300/0 conn(0x7f09c406d9e0 msgr2=0x7f09c4165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 --2- 172.21.15.43:0/3071473690 >> v2:172.21.15.69:3300/0 conn(0x7f09c406d9e0 0x7f09c4165ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 >> v1:172.21.15.69:6789/0 conn(0x7f09c4144770 legacy=0x7f09c415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40b3df0 con 0x7f09c415f270 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f09ac001690 con 0x7f09c415f270 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.047+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 >> v1:172.21.15.43:6789/0 conn(0x7f09c415f270 legacy=0x7f09c40cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 0x7f09a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09a40047c0 0x7f09a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a4005180 0x7f09a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c40b43b0 con 0x7f09a40039e0 2024-01-16T15:00:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a400c0b0 con 0x7f09a4005180 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a400c270 con 0x7f09a40047c0 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 0x7f09a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2000640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09a40047c0 0x7f09a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c3400640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a4005180 0x7f09a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 -- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09a40047c0 msgr2=0x7f09a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09a40047c0 0x7f09a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 -- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a4005180 msgr2=0x7f09a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a4005180 0x7f09a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c2a00640 1 -- 172.21.15.43:0/3071473690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4003520 con 0x7f09a40039e0 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c3400640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a4005180 0x7f09a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.048+0000 7f09c3400640 1 -- 172.21.15.43:0/3071473690 reap_dead start 2024-01-16T15:00:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c2a00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 0x7f09a4003e00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09b4001e50 tx=0x7f09a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f4b11b795efce5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4003b70 con 0x7f09a40039e0 2024-01-16T15:00:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b4003cd0 con 0x7f09a40039e0 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c1600640 1 -- 172.21.15.43:0/3071473690 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4005310 con 0x7f09a40039e0 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c3e00640 1 -- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 msgr2=0x7f09a4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c3e00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 0x7f09a4003e00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f09b4001e50 tx=0x7f09a40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c3e00640 1 -- 172.21.15.43:0/3071473690 shutdown_connections 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c3e00640 1 --2- 172.21.15.43:0/3071473690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a40039e0 0x7f09a4003e00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.050+0000 7f09c3e00640 1 -- 172.21.15.43:0/3071473690 >> 172.21.15.43:0/3071473690 conn(0x7f09c40cec10 msgr2=0x7f09c40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.051+0000 7f09c3e00640 1 -- 172.21.15.43:0/3071473690 shutdown_connections 2024-01-16T15:00:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.051+0000 7f09c3e00640 1 -- 172.21.15.43:0/3071473690 wait complete. 2024-01-16T15:00:01.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.052+0000 7f09c3e00640 1 Processor -- start 2024-01-16T15:00:01.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.052+0000 7f09c3e00640 1 -- start start 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a40039e0 0x7f09c415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c40c9cf0 0x7f09c40ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4133a10 con 0x7f09c4133050 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c413dfc0 con 0x7f09a40039e0 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.053+0000 7f09c3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c415ae80 con 0x7f09c40c9cf0 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36010/0 (socket says 172.21.15.43:36010) 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 -- 172.21.15.43:0/1888071145 learned_addr learned my addr 172.21.15.43:0/1888071145 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c2a00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a40039e0 0x7f09c415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c2000640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c40c9cf0 0x7f09c40ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 -- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c40c9cf0 msgr2=0x7f09c40ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c40c9cf0 0x7f09c40ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 -- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a40039e0 msgr2=0x7f09c415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a40039e0 0x7f09c415a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 -- 172.21.15.43:0/1888071145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40c8c50 con 0x7f09c4133050 2024-01-16T15:00:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.054+0000 7f09c3400640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f09c406de40 tx=0x7f09bc010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.055+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc004230 con 0x7f09c4133050 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.055+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc004390 con 0x7f09c4133050 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.055+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00e870 con 0x7f09c4133050 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.055+0000 7f09c3e00640 1 -- 172.21.15.43:0/1888071145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c40c8f30 con 0x7f09c4133050 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.055+0000 7f09c3e00640 1 -- 172.21.15.43:0/1888071145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c40c94f0 con 0x7f09c4133050 2024-01-16T15:00:01.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.056+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c404e840 con 0x7f09c4133050 2024-01-16T15:00:01.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.057+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09bc00e9d0 con 0x7f09c4133050 2024-01-16T15:00:01.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.058+0000 7f09baa00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 0x7f099c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.058+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09bc089b10 con 0x7f09c4133050 2024-01-16T15:00:01.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.058+0000 7f09c2a00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 0x7f099c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.059+0000 7f09c2a00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 0x7f099c06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f09b4034000 tx=0x7f09b4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.063+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09bc0548e0 con 0x7f09c4133050 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.254+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09c40b2460 con 0x7f09c4133050 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.255+0000 7f09baa00640 1 -- 172.21.15.43:0/1888071145 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09bc057f00 con 0x7f09c4133050 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:01.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.257+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 msgr2=0x7f099c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.257+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 0x7f099c06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f09b4034000 tx=0x7f09b4009030 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.257+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 msgr2=0x7f09c41334d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.257+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f09c406de40 tx=0x7f09bc010c20 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 shutdown_connections 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f099c06b2c0 0x7f099c06d780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09c4133050 0x7f09c41334d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c40c9cf0 0x7f09c40ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 --2- 172.21.15.43:0/1888071145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a40039e0 0x7f09c415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 >> 172.21.15.43:0/1888071145 conn(0x7f09c40cec10 msgr2=0x7f09c4143a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 shutdown_connections 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.258+0000 7f09b3e00640 1 -- 172.21.15.43:0/1888071145 wait complete. 2024-01-16T15:00:01.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.486+0000 7f84b1600640 1 Processor -- start 2024-01-16T15:00:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.486+0000 7f84b1600640 1 -- start start 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.487+0000 7f84b1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f84ac1435d0 0x7f84ac071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.487+0000 7f84b1600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ac070d40 con 0x7f84ac1439b0 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.487+0000 7f84b1600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ac071020 con 0x7f84ac0b4140 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.487+0000 7f84b1600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ac071300 con 0x7f84ac1435d0 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84abe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f84ac1435d0 0x7f84ac071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84a3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f84ac1439b0 0x7f84ac0b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40472/0 (socket says 172.21.15.43:40472) 2024-01-16T15:00:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84a3e00640 1 -- 172.21.15.43:0/510408523 learned_addr learned my addr 172.21.15.43:0/510408523 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84ab400640 1 -- 172.21.15.43:0/510408523 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731290322 0 0) 0x7f84ac070d40 con 0x7f84ac1439b0 2024-01-16T15:00:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84abe00640 1 -- 172.21.15.43:0/510408523 >> v1:172.21.15.69:6789/0 conn(0x7f84ac0b4140 legacy=0x7f84ac070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84abe00640 1 -- 172.21.15.43:0/510408523 >> v1:172.21.15.43:6789/0 conn(0x7f84ac1439b0 legacy=0x7f84ac0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.488+0000 7f84abe00640 1 -- 172.21.15.43:0/510408523 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac0715e0 con 0x7f84ac1435d0 2024-01-16T15:00:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84abe00640 1 --2- 172.21.15.43:0/510408523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1435d0 0x7f84ac071de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f849c0097a0 tx=0x7f849c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b7b3ca3e1e6531 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84ab400640 1 -- 172.21.15.43:0/510408523 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c02e070 con 0x7f84ac1435d0 2024-01-16T15:00:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84ab400640 1 -- 172.21.15.43:0/510408523 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849c0315a0 con 0x7f84ac1435d0 2024-01-16T15:00:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84ab400640 1 -- 172.21.15.43:0/510408523 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c031930 con 0x7f84ac1435d0 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84b1600640 1 -- 172.21.15.43:0/510408523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1435d0 msgr2=0x7f84ac071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.490+0000 7f84b1600640 1 --2- 172.21.15.43:0/510408523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1435d0 0x7f84ac071de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f849c0097a0 tx=0x7f849c02b840 comp rx=0 tx=0).stop 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.491+0000 7f84b1600640 1 -- 172.21.15.43:0/510408523 shutdown_connections 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.491+0000 7f84b1600640 1 --2- 172.21.15.43:0/510408523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1435d0 0x7f84ac071de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.491+0000 7f84b1600640 1 -- 172.21.15.43:0/510408523 >> 172.21.15.43:0/510408523 conn(0x7f84ac0cec10 msgr2=0x7f84ac0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.491+0000 7f84b1600640 1 -- 172.21.15.43:0/510408523 shutdown_connections 2024-01-16T15:00:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.491+0000 7f84b1600640 1 -- 172.21.15.43:0/510408523 wait complete. 2024-01-16T15:00:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.492+0000 7f84b1600640 1 Processor -- start 2024-01-16T15:00:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 -- start start 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84ac0b4140 0x7f84ac15a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 0x7f84ac1f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac1f2030 con 0x7f84ac1439b0 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac1f13b0 con 0x7f84ac0b4140 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.493+0000 7f84b1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac1f1690 con 0x7f84ac1435d0 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36020/0 (socket says 172.21.15.43:36020) 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 -- 172.21.15.43:0/1968067416 learned_addr learned my addr 172.21.15.43:0/1968067416 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84a3e00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 0x7f84ac1f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84abe00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84ac0b4140 0x7f84ac15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 -- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 msgr2=0x7f84ac1f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 0x7f84ac1f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 -- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84ac0b4140 msgr2=0x7f84ac15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84ac0b4140 0x7f84ac15a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84b0c00640 1 -- 172.21.15.43:0/1968067416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c009400 con 0x7f84ac1439b0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.494+0000 7f84a3e00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 0x7f84ac1f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84b0c00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c780 tx=0x7f84a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4004b60 con 0x7f84ac1439b0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a400ce90 con 0x7f84ac1439b0 2024-01-16T15:00:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a400a850 con 0x7f84ac1439b0 2024-01-16T15:00:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ac1f1cc0 con 0x7f84ac1439b0 2024-01-16T15:00:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.495+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ac1f1970 con 0x7f84ac1439b0 2024-01-16T15:00:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.497+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ac04e840 con 0x7f84ac1439b0 2024-01-16T15:00:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.497+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f84a400a9b0 con 0x7f84ac1439b0 2024-01-16T15:00:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.498+0000 7f84a8c00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 0x7f848406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.498+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a4089db0 con 0x7f84ac1439b0 2024-01-16T15:00:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.498+0000 7f84abe00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 0x7f848406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.499+0000 7f84abe00640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 0x7f848406d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f849c036000 tx=0x7f849c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.503+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84a4054c20 con 0x7f84ac1439b0 2024-01-16T15:00:01.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.694+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84ac071e80 con 0x7f84ac1439b0 2024-01-16T15:00:01.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.695+0000 7f84a8c00640 1 -- 172.21.15.43:0/1968067416 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84a4058240 con 0x7f84ac1439b0 2024-01-16T15:00:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 msgr2=0x7f848406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 0x7f848406d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f849c036000 tx=0x7f849c03d040 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 msgr2=0x7f84ac0a8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c780 tx=0x7f84a400cc40 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 shutdown_connections 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f848406b2c0 0x7f848406d780 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84ac1439b0 0x7f84ac0a8620 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84ac1435d0 0x7f84ac1f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 --2- 172.21.15.43:0/1968067416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84ac0b4140 0x7f84ac15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 >> 172.21.15.43:0/1968067416 conn(0x7f84ac0cec10 msgr2=0x7f84ac0b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 shutdown_connections 2024-01-16T15:00:01.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.698+0000 7f84b1600640 1 -- 172.21.15.43:0/1968067416 wait complete. 2024-01-16T15:00:01.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:01.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.943+0000 7f41a5400640 1 Processor -- start 2024-01-16T15:00:01.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.943+0000 7f41a5400640 1 -- start start 2024-01-16T15:00:01.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.944+0000 7f41a5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f41a00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.944+0000 7f41a5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41a00b3210 0x7f41a0167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f41a5400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00b23e0 con 0x7f41a00b48d0 2024-01-16T15:00:01.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f41a5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a00cdc70 con 0x7f41a00b3210 2024-01-16T15:00:01.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f41a5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00b2870 con 0x7f41a0160bd0 2024-01-16T15:00:01.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419de00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f41a00b48d0 0x7f41a00b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40474/0 (socket says 172.21.15.43:40474) 2024-01-16T15:00:01.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419de00640 1 -- 172.21.15.43:0/2026276206 learned_addr learned my addr 172.21.15.43:0/2026276206 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:01.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419e800640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f41a00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419f200640 1 --2- 172.21.15.43:0/2026276206 >> v2:172.21.15.69:3300/0 conn(0x7f41a00b3210 0x7f41a0167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245098443 0 0) 0x7f41a00b23e0 con 0x7f41a00b48d0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.945+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4180003520 con 0x7f41a00b48d0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833911809 0 0) 0x7f4180003520 con 0x7f41a00b48d0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 msgr2=0x7f41a00b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f41a00b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 >> v2:172.21.15.69:3300/0 conn(0x7f41a00b3210 msgr2=0x7f41a0167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 --2- 172.21.15.43:0/2026276206 >> v2:172.21.15.69:3300/0 conn(0x7f41a00b3210 0x7f41a0167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00d41b0 con 0x7f41a00b48d0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4188001670 con 0x7f41a00b48d0 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 >> v1:172.21.15.43:6789/0 conn(0x7f41a00b48d0 legacy=0x7f41a00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4180003970 0x7f4180003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41800047c0 0x7f4180004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f4180005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00cdc70 con 0x7f4180003970 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00b2870 con 0x7f41800047c0 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418000b980 con 0x7f41a0160bd0 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.946+0000 7f419e800640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4180003970 0x7f4180003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419de00640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41800047c0 0x7f4180004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419f200640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f4180005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419e800640 1 -- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 msgr2=0x7f4180005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419e800640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a0160bd0 0x7f4180005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419e800640 1 -- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41800047c0 msgr2=0x7f4180004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419e800640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41800047c0 0x7f4180004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419e800640 1 -- 172.21.15.43:0/2026276206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4180003520 con 0x7f4180003970 2024-01-16T15:00:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.947+0000 7f419de00640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41800047c0 0x7f4180004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.948+0000 7f419e800640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4180003970 0x7f4180003d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4190009af0 tx=0x7f41800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb4180b745e3b83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.948+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419002c6b0 con 0x7f4180003970 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419002c810 con 0x7f4180003970 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f419d400640 1 -- 172.21.15.43:0/2026276206 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419002cba0 con 0x7f4180003970 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 -- 172.21.15.43:0/2026276206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4180003970 msgr2=0x7f4180003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 --2- 172.21.15.43:0/2026276206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4180003970 0x7f4180003d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4190009af0 tx=0x7f41800057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f419f200640 1 -- 172.21.15.43:0/2026276206 reap_dead start 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 -- 172.21.15.43:0/2026276206 shutdown_connections 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 -- 172.21.15.43:0/2026276206 >> 172.21.15.43:0/2026276206 conn(0x7f41a009c5a0 msgr2=0x7f41a00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 -- 172.21.15.43:0/2026276206 shutdown_connections 2024-01-16T15:00:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.949+0000 7f41a5400640 1 -- 172.21.15.43:0/2026276206 wait complete. 2024-01-16T15:00:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.951+0000 7f41a5400640 1 Processor -- start 2024-01-16T15:00:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.951+0000 7f41a5400640 1 -- start start 2024-01-16T15:00:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4180003970 0x7f41a00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 0x7f41a00b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00c8170 con 0x7f41a00cacb0 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00cb530 con 0x7f4180003970 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.952+0000 7f41a5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00c78a0 con 0x7f41a00b0d80 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36038/0 (socket says 172.21.15.43:36038) 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 -- 172.21.15.43:0/2628398772 learned_addr learned my addr 172.21.15.43:0/2628398772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419f200640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 0x7f41a00b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419e800640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4180003970 0x7f41a00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 -- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 msgr2=0x7f41a00b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 0x7f41a00b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 -- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4180003970 msgr2=0x7f41a00ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4180003970 0x7f41a00ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419de00640 1 -- 172.21.15.43:0/2628398772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00caa10 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.953+0000 7f419f200640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 0x7f41a00b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f419de00640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4188000f60 tx=0x7f418800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880074e0 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4188007640 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880049e0 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a00cb250 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.954+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a00c7d90 con 0x7f41a00cacb0 2024-01-16T15:00:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.955+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a00a1210 con 0x7f41a00cacb0 2024-01-16T15:00:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.956+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4188004b40 con 0x7f41a00cacb0 2024-01-16T15:00:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.957+0000 7f4196a00640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 0x7f417806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.958+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4188089690 con 0x7f41a00cacb0 2024-01-16T15:00:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.958+0000 7f419e800640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 0x7f417806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.958+0000 7f419e800640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 0x7f417806d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4190036000 tx=0x7f4190030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:01.962+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4188054460 con 0x7f41a00cacb0 2024-01-16T15:00:02.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.146+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41a004e920 con 0x7f41a00cacb0 2024-01-16T15:00:02.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.147+0000 7f4196a00640 1 -- 172.21.15.43:0/2628398772 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4188057a80 con 0x7f41a00cacb0 2024-01-16T15:00:02.128 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:02.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 msgr2=0x7f417806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 0x7f417806d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4190036000 tx=0x7f4190030000 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 msgr2=0x7f41a015a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4188000f60 tx=0x7f418800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 shutdown_connections 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f417806b2c0 0x7f417806d780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41a00b0d80 0x7f41a00b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41a00cacb0 0x7f41a015a0a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 --2- 172.21.15.43:0/2628398772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4180003970 0x7f41a00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.149+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 >> 172.21.15.43:0/2628398772 conn(0x7f41a009c5a0 msgr2=0x7f41a00b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.150+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 shutdown_connections 2024-01-16T15:00:02.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.150+0000 7f41a5400640 1 -- 172.21.15.43:0/2628398772 wait complete. 2024-01-16T15:00:02.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.375+0000 7fb0ab200640 1 Processor -- start 2024-01-16T15:00:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.375+0000 7fb0ab200640 1 -- start start 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0ab200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 0x7fb0a40cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0ab200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0a40cf590 0x7fb0a406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0ab200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40cf9d0 con 0x7fb0a414b1c0 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0ab200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a4165ca0 con 0x7fb0a40cf590 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0ab200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a4165f80 con 0x7fb0a4162670 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0aa800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0a40cf590 0x7fb0a406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0aa800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0a40cf590 0x7fb0a406c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36052/0 (socket says 172.21.15.43:36052) 2024-01-16T15:00:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.377+0000 7fb0aa800640 1 -- 172.21.15.43:0/3756396096 learned_addr learned my addr 172.21.15.43:0/3756396096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:02.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0a9400640 1 --2- 172.21.15.43:0/3756396096 >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 0x7fb0a40cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0aa800640 1 -- 172.21.15.43:0/3756396096 >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 msgr2=0x7fb0a40cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0aa800640 1 --2- 172.21.15.43:0/3756396096 >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 0x7fb0a40cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0aa800640 1 -- 172.21.15.43:0/3756396096 >> v1:172.21.15.43:6789/0 conn(0x7fb0a414b1c0 legacy=0x7fb0a41651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0aa800640 1 -- 172.21.15.43:0/3756396096 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4161e70 con 0x7fb0a40cf590 2024-01-16T15:00:02.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.378+0000 7fb0a9400640 1 --2- 172.21.15.43:0/3756396096 >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 0x7fb0a40cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:02.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.379+0000 7fb0aa800640 1 --2- 172.21.15.43:0/3756396096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a40cf590 0x7fb0a406c640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40b3b20 tx=0x7fb09c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afff6fe8963677a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.379+0000 7fb0a8a00640 1 -- 172.21.15.43:0/3756396096 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c031870 con 0x7fb0a40cf590 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.379+0000 7fb0a8a00640 1 -- 172.21.15.43:0/3756396096 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c0319d0 con 0x7fb0a40cf590 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.379+0000 7fb0a8a00640 1 -- 172.21.15.43:0/3756396096 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c031d60 con 0x7fb0a40cf590 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 -- 172.21.15.43:0/3756396096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a40cf590 msgr2=0x7fb0a406c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3756396096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a40cf590 0x7fb0a406c640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40b3b20 tx=0x7fb09c02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 -- 172.21.15.43:0/3756396096 shutdown_connections 2024-01-16T15:00:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3756396096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a40cf590 0x7fb0a406c640 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3756396096 >> v2:172.21.15.69:3300/0 conn(0x7fb0a4162670 0x7fb0a40cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 -- 172.21.15.43:0/3756396096 >> 172.21.15.43:0/3756396096 conn(0x7fb0a406a860 msgr2=0x7fb0a406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 -- 172.21.15.43:0/3756396096 shutdown_connections 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.380+0000 7fb0ab200640 1 -- 172.21.15.43:0/3756396096 wait complete. 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.382+0000 7fb0ab200640 1 Processor -- start 2024-01-16T15:00:02.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.382+0000 7fb0ab200640 1 -- start start 2024-01-16T15:00:02.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0a40cf590 0x7fb0a4148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 0x7fb0a406ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4161600 con 0x7fb0a414b1c0 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a41480a0 con 0x7fb0a4162670 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0ab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a41474b0 con 0x7fb0a40cf590 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36060/0 (socket says 172.21.15.43:36060) 2024-01-16T15:00:02.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 -- 172.21.15.43:0/3172070318 learned_addr learned my addr 172.21.15.43:0/3172070318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:02.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0aa800640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 0x7fb0a406ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9e00640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0a40cf590 0x7fb0a4148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 -- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0a40cf590 msgr2=0x7fb0a4148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0a40cf590 0x7fb0a4148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 -- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 msgr2=0x7fb0a406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 0x7fb0a406ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0a9400640 1 -- 172.21.15.43:0/3172070318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c0094d0 con 0x7fb0a414b1c0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.383+0000 7fb0aa800640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 0x7fb0a406ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0a9400640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c7f0 tx=0x7fb08c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c007a30 con 0x7fb0a414b1c0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c007b90 con 0x7fb0a414b1c0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a4147770 con 0x7fb0a414b1c0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00ce40 con 0x7fb0a414b1c0 2024-01-16T15:00:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.384+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a4147d30 con 0x7fb0a414b1c0 2024-01-16T15:00:02.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.385+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a404e840 con 0x7fb0a414b1c0 2024-01-16T15:00:02.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.386+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb08c0113f0 con 0x7fb0a414b1c0 2024-01-16T15:00:02.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.387+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 0x7fb07c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.387+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb08c089e40 con 0x7fb0a414b1c0 2024-01-16T15:00:02.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.388+0000 7fb0a9e00640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 0x7fb07c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.388+0000 7fb0a9e00640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 0x7fb07c06d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb0940012e0 tx=0x7fb094009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.392+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb08c054c10 con 0x7fb0a414b1c0 2024-01-16T15:00:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.586+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0a40ce9a0 con 0x7fb0a414b1c0 2024-01-16T15:00:02.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.588+0000 7fb0a2000640 1 -- 172.21.15.43:0/3172070318 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7fb08c01f090 con 0x7fb0a414b1c0 2024-01-16T15:00:02.569 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:02.569 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:02.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 msgr2=0x7fb07c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 0x7fb07c06d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb0940012e0 tx=0x7fb094009040 comp rx=0 tx=0).stop 2024-01-16T15:00:02.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 msgr2=0x7fb0a41610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c7f0 tx=0x7fb08c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 shutdown_connections 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07c06b2c0 0x7fb07c06d780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0a4162670 0x7fb0a406ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0a414b1c0 0x7fb0a41610c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 --2- 172.21.15.43:0/3172070318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0a40cf590 0x7fb0a4148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.591+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 >> 172.21.15.43:0/3172070318 conn(0x7fb0a406a860 msgr2=0x7fb0a4165170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.592+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 shutdown_connections 2024-01-16T15:00:02.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.592+0000 7fb0ab200640 1 -- 172.21.15.43:0/3172070318 wait complete. 2024-01-16T15:00:02.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:02.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:02.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.835+0000 7fa041600640 1 Processor -- start 2024-01-16T15:00:02.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.835+0000 7fa041600640 1 -- start start 2024-01-16T15:00:02.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa041600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa03c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa041600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c15b730 con 0x7fa03c09cb30 2024-01-16T15:00:02.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa041600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0ac740 con 0x7fa03c15b350 2024-01-16T15:00:02.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa041600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c0ac460 con 0x7fa03c0ae270 2024-01-16T15:00:02.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa03a000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa03c09cb30 0x7fa03c0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40494/0 (socket says 172.21.15.43:40494) 2024-01-16T15:00:02.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.837+0000 7fa03a000640 1 -- 172.21.15.43:0/3393923962 learned_addr learned my addr 172.21.15.43:0/3393923962 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:02.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa03b400640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa03c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858454897 0 0) 0x7fa024001110 con 0x7fa03c09cb30 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa01c003520 con 0x7fa03c09cb30 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894039776 0 0) 0x7fa01c003520 con 0x7fa03c09cb30 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 msgr2=0x7fa03c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa03c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 >> v1:172.21.15.69:6789/0 conn(0x7fa03c15b350 legacy=0x7fa03c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c0b2a50 con 0x7fa03c09cb30 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.838+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa024001800 con 0x7fa03c09cb30 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 >> v1:172.21.15.43:6789/0 conn(0x7fa03c09cb30 legacy=0x7fa03c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa01c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa01c0047c0 0x7fa01c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa01c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c0ac460 con 0x7fa01c0039a0 2024-01-16T15:00:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c00ba30 con 0x7fa01c0047c0 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c00bbf0 con 0x7fa03c0ae270 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa01c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03a000640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa01c0047c0 0x7fa01c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03b400640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa01c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 -- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 msgr2=0x7fa01c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c0ae270 0x7fa01c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 -- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa01c0047c0 msgr2=0x7fa01c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa01c0047c0 0x7fa01c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.839+0000 7fa03aa00640 1 -- 172.21.15.43:0/3393923962 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c003520 con 0x7fa01c0039a0 2024-01-16T15:00:02.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.840+0000 7fa03aa00640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa01c003dc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa02c001c80 tx=0x7fa01c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f778299906092134 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c003b70 con 0x7fa01c0039a0 2024-01-16T15:00:02.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c003cd0 con 0x7fa01c0039a0 2024-01-16T15:00:02.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa039600640 1 -- 172.21.15.43:0/3393923962 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c005310 con 0x7fa01c0039a0 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa041600640 1 -- 172.21.15.43:0/3393923962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 msgr2=0x7fa01c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa041600640 1 --2- 172.21.15.43:0/3393923962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa01c003dc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa02c001c80 tx=0x7fa01c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa03b400640 1 -- 172.21.15.43:0/3393923962 reap_dead start 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa041600640 1 -- 172.21.15.43:0/3393923962 shutdown_connections 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa041600640 1 -- 172.21.15.43:0/3393923962 >> 172.21.15.43:0/3393923962 conn(0x7fa03c0b0ba0 msgr2=0x7fa03c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.841+0000 7fa041600640 1 -- 172.21.15.43:0/3393923962 shutdown_connections 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.842+0000 7fa041600640 1 -- 172.21.15.43:0/3393923962 wait complete. 2024-01-16T15:00:02.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.843+0000 7fa041600640 1 Processor -- start 2024-01-16T15:00:02.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.843+0000 7fa041600640 1 -- start start 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c14ee80 0x7fa03c14f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa03c09b480 0x7fa03c09b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c15c210 con 0x7fa01c0039a0 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c14f840 con 0x7fa03c09b480 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.844+0000 7fa041600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c09c200 con 0x7fa03c14ee80 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36084/0 (socket says 172.21.15.43:36084) 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 -- 172.21.15.43:0/2746248855 learned_addr learned my addr 172.21.15.43:0/2746248855 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03a000640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c14ee80 0x7fa03c14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 -- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c14ee80 msgr2=0x7fa03c14f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c14ee80 0x7fa03c14f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03b400640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa03c09b480 0x7fa03c09b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 -- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa03c09b480 msgr2=0x7fa03c09b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa03c09b480 0x7fa03c09b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 -- 172.21.15.43:0/2746248855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c15c420 con 0x7fa01c0039a0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.845+0000 7fa03aa00640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa02c02bdb0 tx=0x7fa02c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.846+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c003e40 con 0x7fa01c0039a0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.846+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c005690 con 0x7fa01c0039a0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.846+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c007260 con 0x7fa01c0039a0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.846+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c15c700 con 0x7fa01c0039a0 2024-01-16T15:00:02.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.846+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03c0cfcb0 con 0x7fa01c0039a0 2024-01-16T15:00:02.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.847+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03c09d200 con 0x7fa01c0039a0 2024-01-16T15:00:02.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.848+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa02c030070 con 0x7fa01c0039a0 2024-01-16T15:00:02.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.849+0000 7fa032a00640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 0x7fa01406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.850+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa02c0ab400 con 0x7fa01c0039a0 2024-01-16T15:00:02.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.850+0000 7fa03a000640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 0x7fa01406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.850+0000 7fa03a000640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 0x7fa01406d7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa024000f60 tx=0x7fa024000ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:02.855+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa02c076390 con 0x7fa01c0039a0 2024-01-16T15:00:03.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.049+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa03c0ac740 con 0x7fa01c0039a0 2024-01-16T15:00:03.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.050+0000 7fa032a00640 1 -- 172.21.15.43:0/2746248855 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa03c0ac740 con 0x7fa01c0039a0 2024-01-16T15:00:03.031 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 msgr2=0x7fa01406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 0x7fa01406d7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa024000f60 tx=0x7fa024000ef0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 msgr2=0x7fa03c0cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa02c02bdb0 tx=0x7fa02c0014a0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 shutdown_connections 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa01406b2f0 0x7fa01406d7b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa03c09b480 0x7fa03c09b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa03c14ee80 0x7fa03c14f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 --2- 172.21.15.43:0/2746248855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa01c0039a0 0x7fa03c0cf770 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.053+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 >> 172.21.15.43:0/2746248855 conn(0x7fa03c0b0ba0 msgr2=0x7fa03c159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.054+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 shutdown_connections 2024-01-16T15:00:03.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.054+0000 7fa041600640 1 -- 172.21.15.43:0/2746248855 wait complete. 2024-01-16T15:00:03.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:03.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.292+0000 7f4aa6400640 1 Processor -- start 2024-01-16T15:00:03.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.293+0000 7f4aa6400640 1 -- start start 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa6400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4aa01435d0 0x7f4aa0071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa6400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4aa01439b0 0x7f4aa00b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa6400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa0070d40 con 0x7f4aa00b3f70 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa6400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa00b43e0 con 0x7f4aa01435d0 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa6400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa0071020 con 0x7f4aa01439b0 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.294+0000 7f4aa5000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4aa01435d0 0x7f4aa0071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4aa00b3f70 0x7f4aa0070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38696/0 (socket says 172.21.15.43:38696) 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5a00640 1 -- 172.21.15.43:0/2405552372 learned_addr learned my addr 172.21.15.43:0/2405552372 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4a9fe00640 1 --2- 172.21.15.43:0/2405552372 >> v2:172.21.15.69:3300/0 conn(0x7f4aa01439b0 0x7f4aa00b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5000640 1 -- 172.21.15.43:0/2405552372 >> v2:172.21.15.69:3300/0 conn(0x7f4aa01439b0 msgr2=0x7f4aa00b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5000640 1 --2- 172.21.15.43:0/2405552372 >> v2:172.21.15.69:3300/0 conn(0x7f4aa01439b0 0x7f4aa00b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5000640 1 -- 172.21.15.43:0/2405552372 >> v1:172.21.15.43:6789/0 conn(0x7f4aa00b3f70 legacy=0x7f4aa0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4aa5000640 1 -- 172.21.15.43:0/2405552372 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0071300 con 0x7f4aa01435d0 2024-01-16T15:00:03.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.295+0000 7f4a9f400640 1 -- 172.21.15.43:0/2405552372 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228539698 0 0) 0x7f4aa0070d40 con 0x7f4aa00b3f70 2024-01-16T15:00:03.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.296+0000 7f4aa5000640 1 --2- 172.21.15.43:0/2405552372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0071de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cbe80 tx=0x7f4a9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34a2f87e9c76c79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4a9f400640 1 -- 172.21.15.43:0/2405552372 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9002cbf0 con 0x7f4aa01435d0 2024-01-16T15:00:03.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4a9f400640 1 -- 172.21.15.43:0/2405552372 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9002cd50 con 0x7f4aa01435d0 2024-01-16T15:00:03.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4a9f400640 1 -- 172.21.15.43:0/2405552372 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90034660 con 0x7f4aa01435d0 2024-01-16T15:00:03.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 -- 172.21.15.43:0/2405552372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 msgr2=0x7f4aa0071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 --2- 172.21.15.43:0/2405552372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0071de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cbe80 tx=0x7f4a9002be40 comp rx=0 tx=0).stop 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 -- 172.21.15.43:0/2405552372 shutdown_connections 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 --2- 172.21.15.43:0/2405552372 >> v2:172.21.15.69:3300/0 conn(0x7f4aa01439b0 0x7f4aa00b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 --2- 172.21.15.43:0/2405552372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0071de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 -- 172.21.15.43:0/2405552372 >> 172.21.15.43:0/2405552372 conn(0x7f4aa00cec10 msgr2=0x7f4aa00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 -- 172.21.15.43:0/2405552372 shutdown_connections 2024-01-16T15:00:03.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.297+0000 7f4aa6400640 1 -- 172.21.15.43:0/2405552372 wait complete. 2024-01-16T15:00:03.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.299+0000 7f4aa6400640 1 Processor -- start 2024-01-16T15:00:03.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.299+0000 7f4aa6400640 1 -- start start 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 0x7f4aa00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa01439b0 0x7f4aa0139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c4c40 con 0x7f4aa01435d0 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c4f20 con 0x7f4aa00b3f70 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4aa6400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c5200 con 0x7f4aa01439b0 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.300+0000 7f4a9fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55642/0 (socket says 172.21.15.43:55642) 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 -- 172.21.15.43:0/3796765250 learned_addr learned my addr 172.21.15.43:0/3796765250 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:03.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4aa5000640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 0x7f4aa00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4aa5a00640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa01439b0 0x7f4aa0139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 -- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa01439b0 msgr2=0x7f4aa0139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa01439b0 0x7f4aa0139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 -- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 msgr2=0x7f4aa00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 0x7f4aa00c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 -- 172.21.15.43:0/3796765250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90009a70 con 0x7f4aa01435d0 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4aa5000640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 0x7f4aa00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:03.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.301+0000 7f4a9fe00640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800d5c0 tx=0x7f4a8800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.302+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880125f0 con 0x7f4aa01435d0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.302+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88012750 con 0x7f4aa01435d0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.302+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88012b00 con 0x7f4aa01435d0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.302+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa00c54e0 con 0x7f4aa01435d0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.302+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa00c9690 con 0x7f4aa01435d0 2024-01-16T15:00:03.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.303+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa00c9850 con 0x7f4aa01435d0 2024-01-16T15:00:03.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.304+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4a88012c60 con 0x7f4aa01435d0 2024-01-16T15:00:03.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.305+0000 7f4a9cc00640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 0x7f4a7806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.305+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a8808a030 con 0x7f4aa01435d0 2024-01-16T15:00:03.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.305+0000 7f4aa5000640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 0x7f4a7806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.306+0000 7f4aa5000640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 0x7f4a7806d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a90036000 tx=0x7f4a90030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.310+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a88054ea0 con 0x7f4aa01435d0 2024-01-16T15:00:03.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.499+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4aa004e840 con 0x7f4aa01435d0 2024-01-16T15:00:03.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.500+0000 7f4a9cc00640 1 -- 172.21.15.43:0/3796765250 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a880584c0 con 0x7f4aa01435d0 2024-01-16T15:00:03.481 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 msgr2=0x7f4a7806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 0x7f4a7806d780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a90036000 tx=0x7f4a90030000 comp rx=0 tx=0).stop 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 msgr2=0x7f4aa0139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800d5c0 tx=0x7f4a8800da80 comp rx=0 tx=0).stop 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 shutdown_connections 2024-01-16T15:00:03.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a7806b2c0 0x7f4a7806d780 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aa01439b0 0x7f4aa0139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4aa01435d0 0x7f4aa0139220 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 --2- 172.21.15.43:0/3796765250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aa00b3f70 0x7f4aa00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 >> 172.21.15.43:0/3796765250 conn(0x7f4aa00cec10 msgr2=0x7f4aa015c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 shutdown_connections 2024-01-16T15:00:03.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.502+0000 7f4aa6400640 1 -- 172.21.15.43:0/3796765250 wait complete. 2024-01-16T15:00:03.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:03.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.746+0000 7f6286800640 1 Processor -- start 2024-01-16T15:00:03.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.747+0000 7f6286800640 1 -- start start 2024-01-16T15:00:03.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.748+0000 7f6286800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62800d1fd0 0x7f62800ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.748+0000 7f6286800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f62800b33d0 0x7f62800b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.748+0000 7f6286800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62800b2980 con 0x7f62800cfa30 2024-01-16T15:00:03.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.748+0000 7f6286800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62800cf220 con 0x7f62800b33d0 2024-01-16T15:00:03.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.748+0000 7f6286800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62800cf500 con 0x7f62800d1fd0 2024-01-16T15:00:03.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f62800cfa30 0x7f62800cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38698/0 (socket says 172.21.15.43:38698) 2024-01-16T15:00:03.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285400640 1 -- 172.21.15.43:0/1570599618 learned_addr learned my addr 172.21.15.43:0/1570599618 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:03.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285e00640 1 --2- 172.21.15.43:0/1570599618 >> v2:172.21.15.43:3300/0 conn(0x7f62800b33d0 0x7f62800b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6284a00640 1 --2- 172.21.15.43:0/1570599618 >> v2:172.21.15.69:3300/0 conn(0x7f62800d1fd0 0x7f62800ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285e00640 1 -- 172.21.15.43:0/1570599618 >> v2:172.21.15.69:3300/0 conn(0x7f62800d1fd0 msgr2=0x7f62800ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285e00640 1 --2- 172.21.15.43:0/1570599618 >> v2:172.21.15.69:3300/0 conn(0x7f62800d1fd0 0x7f62800ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.749+0000 7f6285e00640 1 -- 172.21.15.43:0/1570599618 >> v1:172.21.15.43:6789/0 conn(0x7f62800cfa30 legacy=0x7f62800cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.750+0000 7f6285e00640 1 -- 172.21.15.43:0/1570599618 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800b2bd0 con 0x7f62800b33d0 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.750+0000 7f627fe00640 1 -- 172.21.15.43:0/1570599618 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513115386 0 0) 0x7f62800b2980 con 0x7f62800cfa30 2024-01-16T15:00:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.751+0000 7f6285e00640 1 --2- 172.21.15.43:0/1570599618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800b33d0 0x7f62800b3860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f62800c9db0 tx=0x7f627802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c1f91fd48fdcec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.751+0000 7f627fe00640 1 -- 172.21.15.43:0/1570599618 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802cc80 con 0x7f62800b33d0 2024-01-16T15:00:03.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.751+0000 7f627fe00640 1 -- 172.21.15.43:0/1570599618 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627802cde0 con 0x7f62800b33d0 2024-01-16T15:00:03.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.751+0000 7f627fe00640 1 -- 172.21.15.43:0/1570599618 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278034660 con 0x7f62800b33d0 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 -- 172.21.15.43:0/1570599618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800b33d0 msgr2=0x7f62800b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 --2- 172.21.15.43:0/1570599618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800b33d0 0x7f62800b3860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f62800c9db0 tx=0x7f627802bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 -- 172.21.15.43:0/1570599618 shutdown_connections 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 --2- 172.21.15.43:0/1570599618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800b33d0 0x7f62800b3860 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 --2- 172.21.15.43:0/1570599618 >> v2:172.21.15.69:3300/0 conn(0x7f62800d1fd0 0x7f62800ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 -- 172.21.15.43:0/1570599618 >> 172.21.15.43:0/1570599618 conn(0x7f62800d3020 msgr2=0x7f62800d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 -- 172.21.15.43:0/1570599618 shutdown_connections 2024-01-16T15:00:03.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.752+0000 7f6286800640 1 -- 172.21.15.43:0/1570599618 wait complete. 2024-01-16T15:00:03.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.754+0000 7f6286800640 1 Processor -- start 2024-01-16T15:00:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.754+0000 7f6286800640 1 -- start start 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62800b33d0 0x7f62800cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 0x7f62801f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800b6170 con 0x7f62800cfa30 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801477a0 con 0x7f62800d1fd0 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6286800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800ccf60 con 0x7f62800b33d0 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6284a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6284a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55660/0 (socket says 172.21.15.43:55660) 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6284a00640 1 -- 172.21.15.43:0/3373610327 learned_addr learned my addr 172.21.15.43:0/3373610327 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.755+0000 7f6285400640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62800b33d0 0x7f62800cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6285e00640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 0x7f62801f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 -- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62800b33d0 msgr2=0x7f62800cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62800b33d0 0x7f62800cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 -- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 msgr2=0x7f62801f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 0x7f62801f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 -- 172.21.15.43:0/3373610327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009a70 con 0x7f62800cfa30 2024-01-16T15:00:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6285e00640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 0x7f62801f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6284a00640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f626800e9b0 tx=0x7f626800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268009670 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626800c9a0 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626800cd50 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.756+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62800b11e0 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.757+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62800b5e00 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.758+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62800d0a40 con 0x7f62800cfa30 2024-01-16T15:00:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.759+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6268013470 con 0x7f62800cfa30 2024-01-16T15:00:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.760+0000 7f627d600640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 0x7f625806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.760+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6268089fb0 con 0x7f62800cfa30 2024-01-16T15:00:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.760+0000 7f6285400640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 0x7f625806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.761+0000 7f6285400640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 0x7f625806d7b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6270000f60 tx=0x7f6270007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.765+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6268054cf0 con 0x7f62800cfa30 2024-01-16T15:00:03.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.961+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62800b1700 con 0x7f62800cfa30 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.962+0000 7f627d600640 1 -- 172.21.15.43:0/3373610327 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62680210a0 con 0x7f62800cfa30 2024-01-16T15:00:03.943 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 msgr2=0x7f625806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 0x7f625806d7b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6270000f60 tx=0x7f6270007b50 comp rx=0 tx=0).stop 2024-01-16T15:00:03.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 msgr2=0x7f62800cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f626800e9b0 tx=0x7f626800ee70 comp rx=0 tx=0).stop 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 shutdown_connections 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f625806b2f0 0x7f625806d7b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62800d1fd0 0x7f62801f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62800cfa30 0x7f62800cca20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 --2- 172.21.15.43:0/3373610327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62800b33d0 0x7f62800cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.964+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 >> 172.21.15.43:0/3373610327 conn(0x7f62800d3020 msgr2=0x7f62800cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.965+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 shutdown_connections 2024-01-16T15:00:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:03.965+0000 7f6286800640 1 -- 172.21.15.43:0/3373610327 wait complete. 2024-01-16T15:00:03.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.198+0000 7fa527e00640 1 Processor -- start 2024-01-16T15:00:04.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.198+0000 7fa527e00640 1 -- start start 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.199+0000 7fa527e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.199+0000 7fa527e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5280cd7b0 0x7fa5280cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.199+0000 7fa527e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52815fa70 con 0x7fa52813dfd0 2024-01-16T15:00:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.199+0000 7fa527e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52815c580 con 0x7fa5280cd7b0 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.199+0000 7fa527e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280cfa10 con 0x7fa5280cf050 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.200+0000 7fa527400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5280cd7b0 0x7fa5280cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.200+0000 7fa526000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.200+0000 7fa526000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:47346/0 (socket says 172.21.15.43:47346) 2024-01-16T15:00:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.200+0000 7fa526000640 1 -- 172.21.15.43:0/1058914724 learned_addr learned my addr 172.21.15.43:0/1058914724 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.201+0000 7fa526000640 1 -- 172.21.15.43:0/1058914724 >> v2:172.21.15.69:3300/0 conn(0x7fa5280cd7b0 msgr2=0x7fa5280cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.201+0000 7fa526000640 1 --2- 172.21.15.43:0/1058914724 >> v2:172.21.15.69:3300/0 conn(0x7fa5280cd7b0 0x7fa5280cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.201+0000 7fa526000640 1 -- 172.21.15.43:0/1058914724 >> v1:172.21.15.69:6789/0 conn(0x7fa52813dfd0 legacy=0x7fa52815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.201+0000 7fa526000640 1 -- 172.21.15.43:0/1058914724 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528140900 con 0x7fa5280cf050 2024-01-16T15:00:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.202+0000 7fa526000640 1 --2- 172.21.15.43:0/1058914724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa510009870 tx=0x7fa51002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22a72c090fdbfc72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.203+0000 7fa525600640 1 -- 172.21.15.43:0/1058914724 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510030840 con 0x7fa5280cf050 2024-01-16T15:00:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.203+0000 7fa525600640 1 -- 172.21.15.43:0/1058914724 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100309a0 con 0x7fa5280cf050 2024-01-16T15:00:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.203+0000 7fa525600640 1 -- 172.21.15.43:0/1058914724 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510030d30 con 0x7fa5280cf050 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.203+0000 7fa527e00640 1 -- 172.21.15.43:0/1058914724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 msgr2=0x7fa5280cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.203+0000 7fa527e00640 1 --2- 172.21.15.43:0/1058914724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa510009870 tx=0x7fa51002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 -- 172.21.15.43:0/1058914724 shutdown_connections 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 --2- 172.21.15.43:0/1058914724 >> v2:172.21.15.69:3300/0 conn(0x7fa5280cd7b0 0x7fa5280cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 --2- 172.21.15.43:0/1058914724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5280cf050 0x7fa5280cf4d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 -- 172.21.15.43:0/1058914724 >> 172.21.15.43:0/1058914724 conn(0x7fa52806a860 msgr2=0x7fa52806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 -- 172.21.15.43:0/1058914724 shutdown_connections 2024-01-16T15:00:04.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.204+0000 7fa527e00640 1 -- 172.21.15.43:0/1058914724 wait complete. 2024-01-16T15:00:04.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.205+0000 7fa527e00640 1 Processor -- start 2024-01-16T15:00:04.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 -- start start 2024-01-16T15:00:04.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5280cf050 0x7fa52815a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 0x7fa52815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa528155e10 con 0x7fa5280cd7b0 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280c7fd0 con 0x7fa5280cf050 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.206+0000 7fa527e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52813fb40 con 0x7fa52813dfd0 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55666/0 (socket says 172.21.15.43:55666) 2024-01-16T15:00:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 -- 172.21.15.43:0/4066101687 learned_addr learned my addr 172.21.15.43:0/4066101687 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:04.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526000640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5280cf050 0x7fa52815a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa527400640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 0x7fa52815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 -- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 msgr2=0x7fa52815a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 0x7fa52815a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 -- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5280cf050 msgr2=0x7fa52815a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5280cf050 0x7fa52815a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa526a00640 1 -- 172.21.15.43:0/4066101687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5100094d0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.207+0000 7fa527400640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 0x7fa52815a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa526a00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c3c0 tx=0x7fa51800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5180052d0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa518005450 con 0x7fa5280cd7b0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518007260 con 0x7fa5280cd7b0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52813f2a0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.208+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52813f860 con 0x7fa5280cd7b0 2024-01-16T15:00:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.209+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52804e840 con 0x7fa5280cd7b0 2024-01-16T15:00:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.210+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa5180073c0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.211+0000 7fa51ea00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 0x7fa50006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.211+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa518088fb0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.211+0000 7fa526000640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 0x7fa50006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.212+0000 7fa526000640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 0x7fa50006d7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa510036000 tx=0x7fa510008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.215+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5180578a0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.411+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5280ce9a0 con 0x7fa5280cd7b0 2024-01-16T15:00:04.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.414+0000 7fa51ea00640 1 -- 172.21.15.43:0/4066101687 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7fa518057430 con 0x7fa5280cd7b0 2024-01-16T15:00:04.395 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:04.395 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:04.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.416+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 msgr2=0x7fa50006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.416+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 0x7fa50006d7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa510036000 tx=0x7fa510008030 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.416+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 msgr2=0x7fa52813b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.416+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c3c0 tx=0x7fa51800c880 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 shutdown_connections 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa50006b2f0 0x7fa50006d7b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5280cf050 0x7fa52815a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52813dfd0 0x7fa52815a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 --2- 172.21.15.43:0/4066101687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5280cd7b0 0x7fa52813b940 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 >> 172.21.15.43:0/4066101687 conn(0x7fa52806a860 msgr2=0x7fa528140b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 shutdown_connections 2024-01-16T15:00:04.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.417+0000 7fa527e00640 1 -- 172.21.15.43:0/4066101687 wait complete. 2024-01-16T15:00:04.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:04.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.665+0000 7fd58b400640 1 Processor -- start 2024-01-16T15:00:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.665+0000 7fd58b400640 1 -- start start 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd58b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd58406cad0 0x7fd58406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd58b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd584072280 0x7fd5840b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd58b400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd584071780 con 0x7fd584160b10 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd58b400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840b2320 con 0x7fd584072280 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd58b400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58406df30 con 0x7fd58406cad0 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd589400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd584072280 0x7fd5840b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd588a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd584160b10 0x7fd58406c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38704/0 (socket says 172.21.15.43:38704) 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd589400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd584072280 0x7fd5840b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55678/0 (socket says 172.21.15.43:55678) 2024-01-16T15:00:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.667+0000 7fd588a00640 1 -- 172.21.15.43:0/3830184764 learned_addr learned my addr 172.21.15.43:0/3830184764 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.668+0000 7fd583e00640 1 --2- 172.21.15.43:0/3830184764 >> v2:172.21.15.69:3300/0 conn(0x7fd58406cad0 0x7fd58406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.668+0000 7fd589400640 1 -- 172.21.15.43:0/3830184764 >> v2:172.21.15.69:3300/0 conn(0x7fd58406cad0 msgr2=0x7fd58406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.668+0000 7fd589400640 1 --2- 172.21.15.43:0/3830184764 >> v2:172.21.15.69:3300/0 conn(0x7fd58406cad0 0x7fd58406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.668+0000 7fd589400640 1 -- 172.21.15.43:0/3830184764 >> v1:172.21.15.43:6789/0 conn(0x7fd584160b10 legacy=0x7fd58406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.668+0000 7fd589400640 1 -- 172.21.15.43:0/3830184764 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840711c0 con 0x7fd584072280 2024-01-16T15:00:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.669+0000 7fd589400640 1 --2- 172.21.15.43:0/3830184764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5840b2740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5840d0100 tx=0x7fd57c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9455a081b1e54f47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd583400640 1 -- 172.21.15.43:0/3830184764 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c02cbd0 con 0x7fd584072280 2024-01-16T15:00:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd583400640 1 -- 172.21.15.43:0/3830184764 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c02cd30 con 0x7fd584072280 2024-01-16T15:00:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd583400640 1 -- 172.21.15.43:0/3830184764 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c02cea0 con 0x7fd584072280 2024-01-16T15:00:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 -- 172.21.15.43:0/3830184764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 msgr2=0x7fd5840b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 --2- 172.21.15.43:0/3830184764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5840b2740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5840d0100 tx=0x7fd57c02be20 comp rx=0 tx=0).stop 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 -- 172.21.15.43:0/3830184764 shutdown_connections 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 --2- 172.21.15.43:0/3830184764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5840b2740 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 --2- 172.21.15.43:0/3830184764 >> v2:172.21.15.69:3300/0 conn(0x7fd58406cad0 0x7fd58406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.670+0000 7fd58b400640 1 -- 172.21.15.43:0/3830184764 >> 172.21.15.43:0/3830184764 conn(0x7fd5840b0ba0 msgr2=0x7fd5840b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.671+0000 7fd58b400640 1 -- 172.21.15.43:0/3830184764 shutdown_connections 2024-01-16T15:00:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.671+0000 7fd58b400640 1 -- 172.21.15.43:0/3830184764 wait complete. 2024-01-16T15:00:04.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.672+0000 7fd58b400640 1 Processor -- start 2024-01-16T15:00:04.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.672+0000 7fd58b400640 1 -- start start 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd58406cad0 0x7fd58414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd584160b10 0x7fd584070490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840ce930 con 0x7fd584072280 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840cddb0 con 0x7fd584160b10 2024-01-16T15:00:04.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.673+0000 7fd58b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840ce090 con 0x7fd58406cad0 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55688/0 (socket says 172.21.15.43:55688) 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 -- 172.21.15.43:0/1643019715 learned_addr learned my addr 172.21.15.43:0/1643019715 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd588a00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd58406cad0 0x7fd58414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd589400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd584160b10 0x7fd584070490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 -- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd58406cad0 msgr2=0x7fd58414c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd58406cad0 0x7fd58414c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 -- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd584160b10 msgr2=0x7fd584070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd584160b10 0x7fd584070490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.674+0000 7fd583e00640 1 -- 172.21.15.43:0/1643019715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009a70 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd583e00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00d800 tx=0x7fd56c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00b6c0 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c00b820 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00bbb0 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5840ce370 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.675+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5841f2ff0 con 0x7fd584072280 2024-01-16T15:00:04.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.676+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58409d200 con 0x7fd584072280 2024-01-16T15:00:04.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.678+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd56c00bd10 con 0x7fd584072280 2024-01-16T15:00:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.679+0000 7fd580c00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 0x7fd55c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.679+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd56c08a140 con 0x7fd584072280 2024-01-16T15:00:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.679+0000 7fd588a00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 0x7fd55c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.680+0000 7fd588a00640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 0x7fd55c06d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd574005880 tx=0x7fd574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.684+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd56c054fa0 con 0x7fd584072280 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.884+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd58404e920 con 0x7fd584072280 2024-01-16T15:00:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.885+0000 7fd580c00640 1 -- 172.21.15.43:0/1643019715 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd56c0585c0 con 0x7fd584072280 2024-01-16T15:00:04.866 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 msgr2=0x7fd55c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 0x7fd55c06d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd574005880 tx=0x7fd574009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 msgr2=0x7fd5841f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00d800 tx=0x7fd56c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 shutdown_connections 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd55c06b2c0 0x7fd55c06d780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd584160b10 0x7fd584070490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd584072280 0x7fd5841f2ab0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 --2- 172.21.15.43:0/1643019715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd58406cad0 0x7fd58414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 >> 172.21.15.43:0/1643019715 conn(0x7fd5840b0ba0 msgr2=0x7fd58406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 shutdown_connections 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:04.888+0000 7fd58b400640 1 -- 172.21.15.43:0/1643019715 wait complete. 2024-01-16T15:00:04.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.132+0000 7fcc33000640 1 Processor -- start 2024-01-16T15:00:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.132+0000 7fcc33000640 1 -- start start 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc33000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc33000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc33000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0c6bf0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc33000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c0d1430 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc33000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0cedc0 con 0x7fcc2c0d1d90 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc31c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc31c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55694/0 (socket says 172.21.15.43:55694) 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc31c00640 1 -- 172.21.15.43:0/2285242209 learned_addr learned my addr 172.21.15.43:0/2285242209 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.134+0000 7fcc32600640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.135+0000 7fcc31c00640 1 -- 172.21.15.43:0/2285242209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 msgr2=0x7fcc2c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.135+0000 7fcc31c00640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.135+0000 7fcc31c00640 1 -- 172.21.15.43:0/2285242209 >> v1:172.21.15.69:6789/0 conn(0x7fcc2c0d0fa0 legacy=0x7fcc2c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.135+0000 7fcc31c00640 1 -- 172.21.15.43:0/2285242209 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0c6e40 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.136+0000 7fcc31c00640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009870 tx=0x7fcc1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f9a38d3adc0b1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.136+0000 7fcc2be00640 1 -- 172.21.15.43:0/2285242209 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c031640 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.136+0000 7fcc2be00640 1 -- 172.21.15.43:0/2285242209 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c0317a0 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.136+0000 7fcc2be00640 1 -- 172.21.15.43:0/2285242209 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c031b30 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 -- 172.21.15.43:0/2285242209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c15d3b0 msgr2=0x7fcc2c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009870 tx=0x7fcc1c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 -- 172.21.15.43:0/2285242209 shutdown_connections 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 --2- 172.21.15.43:0/2285242209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0ce6c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 -- 172.21.15.43:0/2285242209 >> 172.21.15.43:0/2285242209 conn(0x7fcc2c0d30a0 msgr2=0x7fcc2c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 -- 172.21.15.43:0/2285242209 shutdown_connections 2024-01-16T15:00:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.137+0000 7fcc33000640 1 -- 172.21.15.43:0/2285242209 wait complete. 2024-01-16T15:00:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.139+0000 7fcc33000640 1 Processor -- start 2024-01-16T15:00:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.139+0000 7fcc33000640 1 -- start start 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c1f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0b8f60 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c15e7c0 con 0x7fcc2c15d3b0 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc33000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c15ef20 con 0x7fcc2c0d1d90 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc31c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc31c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55700/0 (socket says 172.21.15.43:55700) 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.140+0000 7fcc31c00640 1 -- 172.21.15.43:0/348018444 learned_addr learned my addr 172.21.15.43:0/348018444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc32600640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 -- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 msgr2=0x7fcc2c1f2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31200640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c1f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c1f2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 -- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 msgr2=0x7fcc2c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 -- 172.21.15.43:0/348018444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0094d0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc32600640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:05.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.141+0000 7fcc31c00640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c000c00 tx=0x7fcc1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.142+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c03ac60 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.142+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c03adc0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.142+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c044630 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.142+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c15f200 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.142+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c15f7c0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.143+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c15f9b0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.144+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcc1c02f050 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.145+0000 7fcc29600640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 0x7fcc0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.145+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc1c0abbb0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.145+0000 7fcc31200640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 0x7fcc0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.146+0000 7fcc31200640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 0x7fcc0406d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0c6b60 tx=0x7fcc14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.149+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc1c0768f0 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.345+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcc2c04e840 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.346+0000 7fcc29600640 1 -- 172.21.15.43:0/348018444 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcc1c035020 con 0x7fcc2c0d0fa0 2024-01-16T15:00:05.327 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:05.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 msgr2=0x7fcc0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 0x7fcc0406d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0c6b60 tx=0x7fcc14009040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 msgr2=0x7fcc2c0b8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c000c00 tx=0x7fcc1c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 shutdown_connections 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc0406b2c0 0x7fcc0406d780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c15d3b0 0x7fcc2c0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0d1d90 0x7fcc2c1f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 --2- 172.21.15.43:0/348018444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc2c0d0fa0 0x7fcc2c0b8a20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 >> 172.21.15.43:0/348018444 conn(0x7fcc2c0d30a0 msgr2=0x7fcc2c0c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 shutdown_connections 2024-01-16T15:00:05.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.349+0000 7fcc33000640 1 -- 172.21.15.43:0/348018444 wait complete. 2024-01-16T15:00:05.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:05.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.593+0000 7f7adde00640 1 Processor -- start 2024-01-16T15:00:05.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.593+0000 7f7adde00640 1 -- start start 2024-01-16T15:00:05.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.594+0000 7f7adde00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7ad80c8930 0x7f7ad80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.594+0000 7f7adde00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80c6bf0 con 0x7f7ad815d3b0 2024-01-16T15:00:05.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7adde00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80cedc0 con 0x7f7ad80d1d90 2024-01-16T15:00:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7adde00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad80c6e40 con 0x7f7ad80c8930 2024-01-16T15:00:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7adca00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7ad815d3b0 0x7f7ad80d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38720/0 (socket says 172.21.15.43:38720) 2024-01-16T15:00:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7adca00640 1 -- 172.21.15.43:0/3127988933 learned_addr learned my addr 172.21.15.43:0/3127988933 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7add400640 1 --2- 172.21.15.43:0/3127988933 >> v2:172.21.15.69:3300/0 conn(0x7f7ad80c8930 0x7f7ad80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225422667 0 0) 0x7f7ad80c6bf0 con 0x7f7ad815d3b0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.595+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab8003520 con 0x7f7ad815d3b0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489453311 0 0) 0x7f7ab8003520 con 0x7f7ad815d3b0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 >> v2:172.21.15.69:3300/0 conn(0x7f7ad80c8930 msgr2=0x7f7ad80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 --2- 172.21.15.43:0/3127988933 >> v2:172.21.15.69:3300/0 conn(0x7f7ad80c8930 0x7f7ad80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 >> v1:172.21.15.69:6789/0 conn(0x7f7ad80d1d90 legacy=0x7f7ad80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad80c7120 con 0x7f7ad815d3b0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7ac0001670 con 0x7f7ad815d3b0 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 >> v1:172.21.15.43:6789/0 conn(0x7f7ad815d3b0 legacy=0x7f7ad80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 0x7f7ab8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80047c0 0x7f7ab8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80052a0 0x7f7ab800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad80c6e40 con 0x7f7ab8003a30 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab800c170 con 0x7f7ab80047c0 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab800c330 con 0x7f7ab80052a0 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.596+0000 7f7adca00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 0x7f7ab8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7ad7e00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80047c0 0x7f7ab8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7adca00640 1 -- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80052a0 msgr2=0x7f7ab800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7add400640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80052a0 0x7f7ab800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7adca00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80052a0 0x7f7ab800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7adca00640 1 -- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80047c0 msgr2=0x7f7ab8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7adca00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80047c0 0x7f7ab8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7adca00640 1 -- 172.21.15.43:0/3127988933 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8003520 con 0x7f7ab8003a30 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7ad7e00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80047c0 0x7f7ab8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:05.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.597+0000 7f7add400640 1 -- 172.21.15.43:0/3127988933 reap_dead start 2024-01-16T15:00:05.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.598+0000 7f7adca00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 0x7f7ab8003e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c0e0 tx=0x7f7ab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6476db8bdcb5881e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.598+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0005ad0 con 0x7f7ab8003a30 2024-01-16T15:00:05.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.598+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac0005c30 con 0x7f7ab8003a30 2024-01-16T15:00:05.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.598+0000 7f7ad7400640 1 -- 172.21.15.43:0/3127988933 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac002c9d0 con 0x7f7ab8003a30 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 -- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 msgr2=0x7f7ab8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 0x7f7ab8003e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c0e0 tx=0x7f7ab80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 -- 172.21.15.43:0/3127988933 shutdown_connections 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 --2- 172.21.15.43:0/3127988933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ab8003a30 0x7f7ab8003e50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 -- 172.21.15.43:0/3127988933 >> 172.21.15.43:0/3127988933 conn(0x7f7ad80d30a0 msgr2=0x7f7ad80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 -- 172.21.15.43:0/3127988933 shutdown_connections 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.599+0000 7f7adde00640 1 -- 172.21.15.43:0/3127988933 wait complete. 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.601+0000 7f7adde00640 1 Processor -- start 2024-01-16T15:00:05.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.601+0000 7f7adde00640 1 -- start start 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab8003a30 0x7f7ad80cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad809cde0 0x7f7ad814ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad80cfd20 con 0x7f7ad80cf360 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad814b380 con 0x7f7ad809cde0 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7adde00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad814b620 con 0x7f7ab8003a30 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7add400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7add400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55712/0 (socket says 172.21.15.43:55712) 2024-01-16T15:00:05.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7add400640 1 -- 172.21.15.43:0/2069076223 learned_addr learned my addr 172.21.15.43:0/2069076223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:05.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.602+0000 7f7ad7e00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad809cde0 0x7f7ad814ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7adca00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab8003a30 0x7f7ad80cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 -- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab8003a30 msgr2=0x7f7ad80cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab8003a30 0x7f7ad80cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 -- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad809cde0 msgr2=0x7f7ad814ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad809cde0 0x7f7ad814ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 -- 172.21.15.43:0/2069076223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad809d260 con 0x7f7ad80cf360 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7add400640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80c9bf0 tx=0x7f7ad0010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad000e870 con 0x7f7ad80cf360 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad000e9f0 con 0x7f7ad80cf360 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad000eda0 con 0x7f7ad80cf360 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.603+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad814e160 con 0x7f7ad80cf360 2024-01-16T15:00:05.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.604+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad814b870 con 0x7f7ad80cf360 2024-01-16T15:00:05.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.605+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad804e840 con 0x7f7ad80cf360 2024-01-16T15:00:05.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.606+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7ad0021460 con 0x7f7ad80cf360 2024-01-16T15:00:05.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.607+0000 7f7ad4c00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 0x7f7ab006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.607+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ad0089960 con 0x7f7ad80cf360 2024-01-16T15:00:05.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.607+0000 7f7adca00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 0x7f7ab006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.608+0000 7f7adca00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 0x7f7ab006d7b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0034000 tx=0x7f7ac0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.611+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ad0054790 con 0x7f7ad80cf360 2024-01-16T15:00:05.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.809+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ad80d00a0 con 0x7f7ad80cf360 2024-01-16T15:00:05.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.810+0000 7f7ad4c00640 1 -- 172.21.15.43:0/2069076223 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ad0057db0 con 0x7f7ad80cf360 2024-01-16T15:00:05.791 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 msgr2=0x7f7ab006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 0x7f7ab006d7b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0034000 tx=0x7f7ac0009030 comp rx=0 tx=0).stop 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 msgr2=0x7f7ad80cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80c9bf0 tx=0x7f7ad0010c20 comp rx=0 tx=0).stop 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 shutdown_connections 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ab006b2f0 0x7f7ab006d7b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad809cde0 0x7f7ad814ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad80cf360 0x7f7ad80cf7e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 --2- 172.21.15.43:0/2069076223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab8003a30 0x7f7ad80cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 >> 172.21.15.43:0/2069076223 conn(0x7f7ad80d30a0 msgr2=0x7f7ad80b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 shutdown_connections 2024-01-16T15:00:05.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:05.813+0000 7f7adde00640 1 -- 172.21.15.43:0/2069076223 wait complete. 2024-01-16T15:00:05.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.047+0000 7f6f6d200640 1 Processor -- start 2024-01-16T15:00:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.047+0000 7f6f6d200640 1 -- start start 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f6d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f680cf760 0x7f6f6806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f6d200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f68165ca0 con 0x7f6f68162670 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f6d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f68165f80 con 0x7f6f6814b1c0 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f6d200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f68161e70 con 0x7f6f680cf760 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f5fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6f68162670 0x7f6f680cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38724/0 (socket says 172.21.15.43:38724) 2024-01-16T15:00:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f5fe00640 1 -- 172.21.15.43:0/3736454631 learned_addr learned my addr 172.21.15.43:0/3736454631 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.049+0000 7f6f67e00640 1 --2- 172.21.15.43:0/3736454631 >> v2:172.21.15.69:3300/0 conn(0x7f6f680cf760 0x7f6f6806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415146708 0 0) 0x7f6f68165ca0 con 0x7f6f68162670 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f48003520 con 0x7f6f68162670 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813783241 0 0) 0x7f6f48003520 con 0x7f6f68162670 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 >> v2:172.21.15.69:3300/0 conn(0x7f6f680cf760 msgr2=0x7f6f6806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 --2- 172.21.15.43:0/3736454631 >> v2:172.21.15.69:3300/0 conn(0x7f6f680cf760 0x7f6f6806c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 >> v1:172.21.15.69:6789/0 conn(0x7f6f6814b1c0 legacy=0x7f6f681651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f68162150 con 0x7f6f68162670 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6f50001670 con 0x7f6f68162670 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 >> v1:172.21.15.43:6789/0 conn(0x7f6f68162670 legacy=0x7f6f680cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480047c0 0x7f6f48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f480051f0 0x7f6f4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68161e70 con 0x7f6f480039e0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4800c150 con 0x7f6f480047c0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.050+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4800c310 con 0x7f6f480051f0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f5fe00640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480047c0 0x7f6f48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 -- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f480051f0 msgr2=0x7f6f4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f480051f0 0x7f6f4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 -- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480047c0 msgr2=0x7f6f48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480047c0 0x7f6f48004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67400640 1 -- 172.21.15.43:0/3736454631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48003520 con 0x7f6f480039e0 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f67e00640 1 -- 172.21.15.43:0/3736454631 reap_dead start 2024-01-16T15:00:06.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.051+0000 7f6f5fe00640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f480047c0 0x7f6f48004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.052+0000 7f6f67400640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f48003e00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f58001750 tx=0x7f6f480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae823c0b16aa0e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.052+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f580052d0 con 0x7f6f480039e0 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.052+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5802e050 con 0x7f6f480039e0 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.052+0000 7f6f66a00640 1 -- 172.21.15.43:0/3736454631 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f58003830 con 0x7f6f480039e0 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 -- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 msgr2=0x7f6f48003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f48003e00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f58001750 tx=0x7f6f480057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 -- 172.21.15.43:0/3736454631 shutdown_connections 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 --2- 172.21.15.43:0/3736454631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f48003e00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 -- 172.21.15.43:0/3736454631 >> 172.21.15.43:0/3736454631 conn(0x7f6f6806a860 msgr2=0x7f6f6806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 -- 172.21.15.43:0/3736454631 shutdown_connections 2024-01-16T15:00:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.053+0000 7f6f6d200640 1 -- 172.21.15.43:0/3736454631 wait complete. 2024-01-16T15:00:06.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.055+0000 7f6f6d200640 1 Processor -- start 2024-01-16T15:00:06.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.055+0000 7f6f6d200640 1 -- start start 2024-01-16T15:00:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68142d90 0x7f6f680cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f680cd690 0x7f6f680cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f681490b0 con 0x7f6f480039e0 2024-01-16T15:00:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68160d60 con 0x7f6f68142d90 2024-01-16T15:00:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f6d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6814a9d0 con 0x7f6f680cd690 2024-01-16T15:00:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.056+0000 7f6f67400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55734/0 (socket says 172.21.15.43:55734) 2024-01-16T15:00:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 -- 172.21.15.43:0/2794859126 learned_addr learned my addr 172.21.15.43:0/2794859126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:06.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f5fe00640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68142d90 0x7f6f680cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67e00640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f680cd690 0x7f6f680cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 -- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f680cd690 msgr2=0x7f6f680cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f680cd690 0x7f6f680cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 -- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68142d90 msgr2=0x7f6f680cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68142d90 0x7f6f680cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 -- 172.21.15.43:0/2794859126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680cccc0 con 0x7f6f480039e0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.057+0000 7f6f67400640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6f58000c00 tx=0x7f6f58001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.058+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f58007640 con 0x7f6f480039e0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.058+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f580069e0 con 0x7f6f480039e0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.058+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f58006d90 con 0x7f6f480039e0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.058+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f680cc0f0 con 0x7f6f480039e0 2024-01-16T15:00:06.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.058+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f680cc670 con 0x7f6f480039e0 2024-01-16T15:00:06.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.059+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6804e840 con 0x7f6f480039e0 2024-01-16T15:00:06.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.060+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6f580035f0 con 0x7f6f480039e0 2024-01-16T15:00:06.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.061+0000 7f6f5f400640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 0x7f6f4006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.061+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f580ab9b0 con 0x7f6f480039e0 2024-01-16T15:00:06.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.061+0000 7f6f5fe00640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 0x7f6f4006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.062+0000 7f6f5fe00640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 0x7f6f4006d7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f6814a280 tx=0x7f6f50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.066+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f580766f0 con 0x7f6f480039e0 2024-01-16T15:00:06.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.262+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f680b16b0 con 0x7f6f480039e0 2024-01-16T15:00:06.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.265+0000 7f6f5f400640 1 -- 172.21.15.43:0/2794859126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7f6f58041020 con 0x7f6f480039e0 2024-01-16T15:00:06.246 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:06.246 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 msgr2=0x7f6f4006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 0x7f6f4006d7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f6814a280 tx=0x7f6f50009040 comp rx=0 tx=0).stop 2024-01-16T15:00:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 msgr2=0x7f6f6814a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6f58000c00 tx=0x7f6f58001120 comp rx=0 tx=0).stop 2024-01-16T15:00:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 shutdown_connections 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f4006b2f0 0x7f6f4006d7b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68142d90 0x7f6f680cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f680cd690 0x7f6f680cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 --2- 172.21.15.43:0/2794859126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f480039e0 0x7f6f6814a490 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 >> 172.21.15.43:0/2794859126 conn(0x7f6f6806a860 msgr2=0x7f6f6806ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 shutdown_connections 2024-01-16T15:00:06.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.268+0000 7f6f6d200640 1 -- 172.21.15.43:0/2794859126 wait complete. 2024-01-16T15:00:06.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:06.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.513+0000 7efd17e00640 1 Processor -- start 2024-01-16T15:00:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.514+0000 7efd17e00640 1 -- start start 2024-01-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.515+0000 7efd17e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efd1815f270 0x7efd180cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.515+0000 7efd17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 0x7efd180cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.515+0000 7efd17e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1815f930 con 0x7efd18144770 2024-01-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.515+0000 7efd17e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180b43b0 con 0x7efd1815f270 2024-01-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.515+0000 7efd17e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180b3b10 con 0x7efd1806d9e0 2024-01-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efd1815f270 0x7efd180cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efd1815f270 0x7efd180cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55746/0 (socket says 172.21.15.43:55746) 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 -- 172.21.15.43:0/1397495136 learned_addr learned my addr 172.21.15.43:0/1397495136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd17400640 1 --2- 172.21.15.43:0/1397495136 >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 0x7efd180cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 -- 172.21.15.43:0/1397495136 >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 msgr2=0x7efd180cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 --2- 172.21.15.43:0/1397495136 >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 0x7efd180cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.516+0000 7efd16000640 1 -- 172.21.15.43:0/1397495136 >> v1:172.21.15.69:6789/0 conn(0x7efd18144770 legacy=0x7efd1815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.517+0000 7efd16000640 1 -- 172.21.15.43:0/1397495136 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd180b3df0 con 0x7efd1815f270 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.517+0000 7efd17400640 1 --2- 172.21.15.43:0/1397495136 >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 0x7efd180cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.518+0000 7efd16000640 1 --2- 172.21.15.43:0/1397495136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1815f270 0x7efd180cd4a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd18144eb0 tx=0x7efd0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba2b5a61ffdc82a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.518+0000 7efd15600640 1 -- 172.21.15.43:0/1397495136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0802cb10 con 0x7efd1815f270 2024-01-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.518+0000 7efd15600640 1 -- 172.21.15.43:0/1397495136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0802cc70 con 0x7efd1815f270 2024-01-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.518+0000 7efd15600640 1 -- 172.21.15.43:0/1397495136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd080343f0 con 0x7efd1815f270 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 -- 172.21.15.43:0/1397495136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1815f270 msgr2=0x7efd180cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 --2- 172.21.15.43:0/1397495136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1815f270 0x7efd180cd4a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd18144eb0 tx=0x7efd0802bf70 comp rx=0 tx=0).stop 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 -- 172.21.15.43:0/1397495136 shutdown_connections 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 --2- 172.21.15.43:0/1397495136 >> v2:172.21.15.69:3300/0 conn(0x7efd1806d9e0 0x7efd180cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 --2- 172.21.15.43:0/1397495136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1815f270 0x7efd180cd4a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 -- 172.21.15.43:0/1397495136 >> 172.21.15.43:0/1397495136 conn(0x7efd180cec10 msgr2=0x7efd180cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 -- 172.21.15.43:0/1397495136 shutdown_connections 2024-01-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.519+0000 7efd17e00640 1 -- 172.21.15.43:0/1397495136 wait complete. 2024-01-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.521+0000 7efd17e00640 1 Processor -- start 2024-01-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.521+0000 7efd17e00640 1 -- start start 2024-01-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 0x7efd1813f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 0x7efd18070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1815c7e0 con 0x7efd18144770 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1815ca80 con 0x7efd1815f270 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1815cd60 con 0x7efd1806d9e0 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55750/0 (socket says 172.21.15.43:55750) 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 -- 172.21.15.43:0/4229769365 learned_addr learned my addr 172.21.15.43:0/4229769365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd17400640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 0x7efd18070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 -- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 msgr2=0x7efd1813f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16a00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 0x7efd1813f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 0x7efd1813f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 -- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 msgr2=0x7efd18070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 0x7efd18070a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.522+0000 7efd16000640 1 -- 172.21.15.43:0/4229769365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd08009b30 con 0x7efd18144770 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd16a00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 0x7efd1813f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd17400640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 0x7efd18070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd16000640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efd08000f80 tx=0x7efd08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08009510 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0803e450 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0803e800 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1815d040 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.523+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd18070f90 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.525+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1815d500 con 0x7efd18144770 2024-01-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.526+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efd0803e960 con 0x7efd18144770 2024-01-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.526+0000 7efd0ea00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 0x7efcf006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.527+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd080ac920 con 0x7efd18144770 2024-01-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.527+0000 7efd16a00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 0x7efcf006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.527+0000 7efd16a00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 0x7efcf006d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd00000f60 tx=0x7efd00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.531+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd08077820 con 0x7efd18144770 2024-01-16T15:00:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.726+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd1804e840 con 0x7efd18144770 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.727+0000 7efd0ea00640 1 -- 172.21.15.43:0/4229769365 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd0807ae40 con 0x7efd18144770 2024-01-16T15:00:06.708 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.729+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 msgr2=0x7efcf006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 0x7efcf006d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd00000f60 tx=0x7efd00009040 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 msgr2=0x7efd1813fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efd08000f80 tx=0x7efd08004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 shutdown_connections 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcf006b2c0 0x7efcf006d780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1815f270 0x7efd18070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd18144770 0x7efd1813fea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 --2- 172.21.15.43:0/4229769365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd1806d9e0 0x7efd1813f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 >> 172.21.15.43:0/4229769365 conn(0x7efd180cec10 msgr2=0x7efd1815eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 shutdown_connections 2024-01-16T15:00:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.730+0000 7efd17e00640 1 -- 172.21.15.43:0/4229769365 wait complete. 2024-01-16T15:00:06.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.962+0000 7fbe3b600640 1 Processor -- start 2024-01-16T15:00:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.962+0000 7fbe3b600640 1 -- start start 2024-01-16T15:00:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe3b600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 0x7fbe340abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 0x7fbe340adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe3b600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3415b730 con 0x7fbe3409cb30 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe3b600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe340ac740 con 0x7fbe3415b350 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe3b600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe340ac460 con 0x7fbe340adb60 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe38c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 0x7fbe340abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe38c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 0x7fbe340abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:35124/0 (socket says 172.21.15.43:35124) 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe38c00640 1 -- 172.21.15.43:0/544755740 learned_addr learned my addr 172.21.15.43:0/544755740 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.964+0000 7fbe39600640 1 --2- 172.21.15.43:0/544755740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 0x7fbe340adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.965+0000 7fbe39600640 1 -- 172.21.15.43:0/544755740 >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 msgr2=0x7fbe340abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.965+0000 7fbe39600640 1 --2- 172.21.15.43:0/544755740 >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 0x7fbe340abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.965+0000 7fbe39600640 1 -- 172.21.15.43:0/544755740 >> v1:172.21.15.69:6789/0 conn(0x7fbe3409cb30 legacy=0x7fbe340b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.965+0000 7fbe39600640 1 -- 172.21.15.43:0/544755740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe340b2a50 con 0x7fbe340adb60 2024-01-16T15:00:06.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.967+0000 7fbe39600640 1 --2- 172.21.15.43:0/544755740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 0x7fbe340adfb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009d90 tx=0x7fbe2c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8220c5d8e2b94b1a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.967+0000 7fbe33e00640 1 -- 172.21.15.43:0/544755740 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c02cc50 con 0x7fbe340adb60 2024-01-16T15:00:06.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.967+0000 7fbe33e00640 1 -- 172.21.15.43:0/544755740 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c02cdb0 con 0x7fbe340adb60 2024-01-16T15:00:06.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.967+0000 7fbe33e00640 1 -- 172.21.15.43:0/544755740 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c034660 con 0x7fbe340adb60 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 -- 172.21.15.43:0/544755740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 msgr2=0x7fbe340adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 --2- 172.21.15.43:0/544755740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 0x7fbe340adfb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009d90 tx=0x7fbe2c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 -- 172.21.15.43:0/544755740 shutdown_connections 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 --2- 172.21.15.43:0/544755740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe340adb60 0x7fbe340adfb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 --2- 172.21.15.43:0/544755740 >> v2:172.21.15.69:3300/0 conn(0x7fbe3415b350 0x7fbe340abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 -- 172.21.15.43:0/544755740 >> 172.21.15.43:0/544755740 conn(0x7fbe340b0ba0 msgr2=0x7fbe340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 -- 172.21.15.43:0/544755740 shutdown_connections 2024-01-16T15:00:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.968+0000 7fbe3b600640 1 -- 172.21.15.43:0/544755740 wait complete. 2024-01-16T15:00:06.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.970+0000 7fbe3b600640 1 Processor -- start 2024-01-16T15:00:06.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.970+0000 7fbe3b600640 1 -- start start 2024-01-16T15:00:06.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe3409cb30 0x7fbe34157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe340adb60 0x7fbe341e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3409a2b0 con 0x7fbe3415b350 2024-01-16T15:00:06.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3409a550 con 0x7fbe340adb60 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe3b600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3409a830 con 0x7fbe3409cb30 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.971+0000 7fbe39600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe38c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe3409cb30 0x7fbe34157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55754/0 (socket says 172.21.15.43:55754) 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 -- 172.21.15.43:0/4046375736 learned_addr learned my addr 172.21.15.43:0/4046375736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe2be00640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe340adb60 0x7fbe341e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 -- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe3409cb30 msgr2=0x7fbe34157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe3409cb30 0x7fbe34157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 -- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe340adb60 msgr2=0x7fbe341e0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe340adb60 0x7fbe341e0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 -- 172.21.15.43:0/4046375736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c009a70 con 0x7fbe3415b350 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.972+0000 7fbe39600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009eb0 tx=0x7fbe2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.973+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c03e610 con 0x7fbe3415b350 2024-01-16T15:00:06.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.973+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c03e770 con 0x7fbe3415b350 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.973+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c03eb20 con 0x7fbe3415b350 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.973+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3409ab10 con 0x7fbe3415b350 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.973+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe341576a0 con 0x7fbe3415b350 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.974+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3409b030 con 0x7fbe3415b350 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.975+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbe2c03b050 con 0x7fbe3415b350 2024-01-16T15:00:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.976+0000 7fbe31600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 0x7fbe0c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.976+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe2c0ac440 con 0x7fbe3415b350 2024-01-16T15:00:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.977+0000 7fbe38c00640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 0x7fbe0c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.977+0000 7fbe38c00640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 0x7fbe0c06d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbe34155e20 tx=0x7fbe24009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:06.981+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe3409b030 con 0x7fbe3415b350 2024-01-16T15:00:07.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.172+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe3409b030 con 0x7fbe3415b350 2024-01-16T15:00:07.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.173+0000 7fbe31600640 1 -- 172.21.15.43:0/4046375736 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe3409b030 con 0x7fbe3415b350 2024-01-16T15:00:07.154 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:07.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 msgr2=0x7fbe0c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 0x7fbe0c06d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbe34155e20 tx=0x7fbe24009240 comp rx=0 tx=0).stop 2024-01-16T15:00:07.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 msgr2=0x7fbe341e1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009eb0 tx=0x7fbe2c004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:07.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 shutdown_connections 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbe0c06b2c0 0x7fbe0c06d780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe340adb60 0x7fbe341e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe3415b350 0x7fbe341e1310 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 --2- 172.21.15.43:0/4046375736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe3409cb30 0x7fbe34157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 >> 172.21.15.43:0/4046375736 conn(0x7fbe340b0ba0 msgr2=0x7fbe3406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 shutdown_connections 2024-01-16T15:00:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.176+0000 7fbe3b600640 1 -- 172.21.15.43:0/4046375736 wait complete. 2024-01-16T15:00:07.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.419+0000 7f1d47000640 1 Processor -- start 2024-01-16T15:00:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.419+0000 7f1d47000640 1 -- start start 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d47000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 0x7f1d4015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d47000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4015f930 con 0x7f1d400b35a0 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d47000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d400b3b10 con 0x7f1d4015f0a0 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d47000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400b43b0 con 0x7f1d40144770 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d45200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1d400b35a0 0x7f1d4006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38740/0 (socket says 172.21.15.43:38740) 2024-01-16T15:00:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d45200640 1 -- 172.21.15.43:0/97472762 learned_addr learned my addr 172.21.15.43:0/97472762 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.421+0000 7f1d45c00640 1 --2- 172.21.15.43:0/97472762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 0x7f1d4015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.422+0000 7f1d3fe00640 1 -- 172.21.15.43:0/97472762 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154975026 0 0) 0x7f1d4015f930 con 0x7f1d400b35a0 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.422+0000 7f1d3fe00640 1 -- 172.21.15.43:0/97472762 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20003520 con 0x7f1d400b35a0 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.422+0000 7f1d45c00640 1 -- 172.21.15.43:0/97472762 >> v1:172.21.15.69:6789/0 conn(0x7f1d4015f0a0 legacy=0x7f1d400ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.422+0000 7f1d45c00640 1 -- 172.21.15.43:0/97472762 >> v1:172.21.15.43:6789/0 conn(0x7f1d400b35a0 legacy=0x7f1d4006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.422+0000 7f1d45c00640 1 -- 172.21.15.43:0/97472762 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400b3df0 con 0x7f1d40144770 2024-01-16T15:00:07.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.423+0000 7f1d45c00640 1 --2- 172.21.15.43:0/97472762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 0x7f1d4015eb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d300097d0 tx=0x7f1d3002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e1025b1d3f758b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d3fe00640 1 -- 172.21.15.43:0/97472762 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30031890 con 0x7f1d40144770 2024-01-16T15:00:07.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d3fe00640 1 -- 172.21.15.43:0/97472762 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d300319f0 con 0x7f1d40144770 2024-01-16T15:00:07.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d3fe00640 1 -- 172.21.15.43:0/97472762 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30031d80 con 0x7f1d40144770 2024-01-16T15:00:07.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d47000640 1 -- 172.21.15.43:0/97472762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 msgr2=0x7f1d4015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d47000640 1 --2- 172.21.15.43:0/97472762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 0x7f1d4015eb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d300097d0 tx=0x7f1d3002b930 comp rx=0 tx=0).stop 2024-01-16T15:00:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d47000640 1 -- 172.21.15.43:0/97472762 shutdown_connections 2024-01-16T15:00:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d47000640 1 --2- 172.21.15.43:0/97472762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d40144770 0x7f1d4015eb60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.424+0000 7f1d47000640 1 -- 172.21.15.43:0/97472762 >> 172.21.15.43:0/97472762 conn(0x7f1d400cec10 msgr2=0x7f1d400cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.425+0000 7f1d47000640 1 -- 172.21.15.43:0/97472762 shutdown_connections 2024-01-16T15:00:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.425+0000 7f1d47000640 1 -- 172.21.15.43:0/97472762 wait complete. 2024-01-16T15:00:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.426+0000 7f1d47000640 1 Processor -- start 2024-01-16T15:00:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.426+0000 7f1d47000640 1 -- start start 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d400b35a0 0x7f1d4014a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d4015f0a0 0x7f1d400afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d401435b0 con 0x7f1d40144770 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4014b8c0 con 0x7f1d4015f0a0 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.427+0000 7f1d47000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4014acb0 con 0x7f1d400b35a0 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55768/0 (socket says 172.21.15.43:55768) 2024-01-16T15:00:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 -- 172.21.15.43:0/2009619262 learned_addr learned my addr 172.21.15.43:0/2009619262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 -- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d400b35a0 msgr2=0x7f1d4014a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d400b35a0 0x7f1d4014a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d46600640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d4015f0a0 0x7f1d400afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 -- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d4015f0a0 msgr2=0x7f1d400afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d4015f0a0 0x7f1d400afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 -- 172.21.15.43:0/2009619262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d30009400 con 0x7f1d40144770 2024-01-16T15:00:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.428+0000 7f1d45200640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c500 tx=0x7f1d2800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.429+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28005ce0 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.429+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28005e40 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.429+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28007330 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.429+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4014af90 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.429+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4014b550 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.430+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4004e840 con 0x7f1d40144770 2024-01-16T15:00:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.431+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1d28007490 con 0x7f1d40144770 2024-01-16T15:00:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.432+0000 7f1d3d600640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 0x7f1d1806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.432+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d28089170 con 0x7f1d40144770 2024-01-16T15:00:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.432+0000 7f1d45c00640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 0x7f1d1806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.433+0000 7f1d45c00640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 0x7f1d1806d7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d30036000 tx=0x7f1d30030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.436+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d28054130 con 0x7f1d40144770 2024-01-16T15:00:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.628+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d4014ba00 con 0x7f1d40144770 2024-01-16T15:00:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.628+0000 7f1d3d600640 1 -- 172.21.15.43:0/2009619262 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d28057750 con 0x7f1d40144770 2024-01-16T15:00:07.609 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 msgr2=0x7f1d1806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 0x7f1d1806d7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d30036000 tx=0x7f1d30030000 comp rx=0 tx=0).stop 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 msgr2=0x7f1d40143070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800c500 tx=0x7f1d2800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 shutdown_connections 2024-01-16T15:00:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d1806b2f0 0x7f1d1806d7b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d4015f0a0 0x7f1d400afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d40144770 0x7f1d40143070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 --2- 172.21.15.43:0/2009619262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d400b35a0 0x7f1d4014a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 >> 172.21.15.43:0/2009619262 conn(0x7f1d400cec10 msgr2=0x7f1d400cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 shutdown_connections 2024-01-16T15:00:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.631+0000 7f1d47000640 1 -- 172.21.15.43:0/2009619262 wait complete. 2024-01-16T15:00:07.700 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-16T15:00:07.700 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:00:07.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:07.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.942+0000 7fd8afe00640 1 Processor -- start 2024-01-16T15:00:07.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.942+0000 7fd8afe00640 1 -- start start 2024-01-16T15:00:07.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.943+0000 7fd8afe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd8b00d00d0 0x7fd8b00d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8afe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00afd30 con 0x7fd8b015d370 2024-01-16T15:00:07.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8afe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00d0c30 con 0x7fd8b00b1f00 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8afe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00cef60 con 0x7fd8b00d00d0 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8aea00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd8b015d370 0x7fd8b00cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38746/0 (socket says 172.21.15.43:38746) 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8aea00640 1 -- 172.21.15.43:0/3392410328 learned_addr learned my addr 172.21.15.43:0/3392410328 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8ae000640 1 --2- 172.21.15.43:0/3392410328 >> v2:172.21.15.43:3300/0 conn(0x7fd8b00d00d0 0x7fd8b00d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8ad600640 1 -- 172.21.15.43:0/3392410328 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313743139 0 0) 0x7fd8b00afd30 con 0x7fd8b015d370 2024-01-16T15:00:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8ae000640 1 -- 172.21.15.43:0/3392410328 >> v1:172.21.15.69:6789/0 conn(0x7fd8b00b1f00 legacy=0x7fd8b00b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8ae000640 1 -- 172.21.15.43:0/3392410328 >> v1:172.21.15.43:6789/0 conn(0x7fd8b015d370 legacy=0x7fd8b00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.944+0000 7fd8ae000640 1 -- 172.21.15.43:0/3392410328 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00d0ef0 con 0x7fd8b00d00d0 2024-01-16T15:00:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8ae000640 1 --2- 172.21.15.43:0/3392410328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b00d00d0 0x7fd8b00d1c70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009d90 tx=0x7fd8a002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb9355c09db54ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8ad600640 1 -- 172.21.15.43:0/3392410328 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a002cbd0 con 0x7fd8b00d00d0 2024-01-16T15:00:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8ad600640 1 -- 172.21.15.43:0/3392410328 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a002cd30 con 0x7fd8b00d00d0 2024-01-16T15:00:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8ad600640 1 -- 172.21.15.43:0/3392410328 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a002cea0 con 0x7fd8b00d00d0 2024-01-16T15:00:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8afe00640 1 -- 172.21.15.43:0/3392410328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b00d00d0 msgr2=0x7fd8b00d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.946+0000 7fd8afe00640 1 --2- 172.21.15.43:0/3392410328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b00d00d0 0x7fd8b00d1c70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009d90 tx=0x7fd8a002be20 comp rx=0 tx=0).stop 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.947+0000 7fd8afe00640 1 -- 172.21.15.43:0/3392410328 shutdown_connections 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.947+0000 7fd8afe00640 1 --2- 172.21.15.43:0/3392410328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b00d00d0 0x7fd8b00d1c70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.947+0000 7fd8afe00640 1 -- 172.21.15.43:0/3392410328 >> 172.21.15.43:0/3392410328 conn(0x7fd8b009c3d0 msgr2=0x7fd8b00d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.947+0000 7fd8afe00640 1 -- 172.21.15.43:0/3392410328 shutdown_connections 2024-01-16T15:00:07.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.947+0000 7fd8afe00640 1 -- 172.21.15.43:0/3392410328 wait complete. 2024-01-16T15:00:07.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.948+0000 7fd8afe00640 1 Processor -- start 2024-01-16T15:00:07.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.949+0000 7fd8afe00640 1 -- start start 2024-01-16T15:00:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.949+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8b00b1f00 0x7fd8b00c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 0x7fd8b015b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8afe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015b9a0 con 0x7fd8b015d370 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8afe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c99c0 con 0x7fd8b00d00d0 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8afe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c7c30 con 0x7fd8b00b1f00 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55788/0 (socket says 172.21.15.43:55788) 2024-01-16T15:00:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 -- 172.21.15.43:0/1373063223 learned_addr learned my addr 172.21.15.43:0/1373063223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8ae000640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 0x7fd8b015b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8aea00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8b00b1f00 0x7fd8b00c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 -- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8b00b1f00 msgr2=0x7fd8b00c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8b00b1f00 0x7fd8b00c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 -- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 msgr2=0x7fd8b015b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 0x7fd8b015b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8af400640 1 -- 172.21.15.43:0/1373063223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0009a70 con 0x7fd8b015d370 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.950+0000 7fd8ae000640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 0x7fd8b015b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8af400640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800b410 tx=0x7fd8a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8004020 con 0x7fd8b015d370 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a8004180 con 0x7fd8b015d370 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8011630 con 0x7fd8b015d370 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00c7f10 con 0x7fd8b015d370 2024-01-16T15:00:07.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.951+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00c84d0 con 0x7fd8b015d370 2024-01-16T15:00:07.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.952+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004e840 con 0x7fd8b015d370 2024-01-16T15:00:07.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.953+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd8a8011790 con 0x7fd8b015d370 2024-01-16T15:00:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.954+0000 7fd8a7400640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 0x7fd88806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.954+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8a8089850 con 0x7fd8b015d370 2024-01-16T15:00:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.954+0000 7fd8aea00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 0x7fd88806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.955+0000 7fd8aea00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 0x7fd88806d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd898000f60 tx=0x7fd898009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:07.959+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8a8054750 con 0x7fd8b015d370 2024-01-16T15:00:08.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.156+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8b00b49e0 con 0x7fd8b015d370 2024-01-16T15:00:08.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.158+0000 7fd8a7400640 1 -- 172.21.15.43:0/1373063223 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7fd8a8015020 con 0x7fd8b015d370 2024-01-16T15:00:08.140 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:08.140 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:08.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 msgr2=0x7fd88806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 0x7fd88806d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd898000f60 tx=0x7fd898009040 comp rx=0 tx=0).stop 2024-01-16T15:00:08.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 msgr2=0x7fd8b00c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800b410 tx=0x7fd8a800b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 shutdown_connections 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd88806b2c0 0x7fd88806d780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8b00d00d0 0x7fd8b015b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8b015d370 0x7fd8b00c9480 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 --2- 172.21.15.43:0/1373063223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8b00b1f00 0x7fd8b00c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.161+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 >> 172.21.15.43:0/1373063223 conn(0x7fd8b009c3d0 msgr2=0x7fd8b00cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.162+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 shutdown_connections 2024-01-16T15:00:08.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.162+0000 7fd8afe00640 1 -- 172.21.15.43:0/1373063223 wait complete. 2024-01-16T15:00:08.164 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:08.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.411+0000 7f2c0be00640 1 Processor -- start 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.412+0000 7f2c0be00640 1 -- start start 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.413+0000 7f2c0be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c0c0c8930 0x7f2c0c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.413+0000 7f2c0be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0c6bf0 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.413+0000 7f2c0be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0cedc0 con 0x7f2c0c0d1d90 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.413+0000 7f2c0be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c6e40 con 0x7f2c0c0c8930 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.414+0000 7f2c0aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2c0c15d3b0 0x7f2c0c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38762/0 (socket says 172.21.15.43:38762) 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.414+0000 7f2c0aa00640 1 -- 172.21.15.43:0/4269449772 learned_addr learned my addr 172.21.15.43:0/4269449772 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.414+0000 7f2c0b400640 1 --2- 172.21.15.43:0/4269449772 >> v2:172.21.15.69:3300/0 conn(0x7f2c0c0c8930 0x7f2c0c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.414+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048114031 0 0) 0x7f2c0c0c6bf0 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.414+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec003520 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653499589 0 0) 0x7f2bec003520 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 >> v2:172.21.15.69:3300/0 conn(0x7f2c0c0c8930 msgr2=0x7f2c0c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 --2- 172.21.15.43:0/4269449772 >> v2:172.21.15.69:3300/0 conn(0x7f2c0c0c8930 0x7f2c0c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 >> v1:172.21.15.69:6789/0 conn(0x7f2c0c0d1d90 legacy=0x7f2c0c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0c7120 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f2bfc001670 con 0x7f2c0c15d3b0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 >> v1:172.21.15.43:6789/0 conn(0x7f2c0c15d3b0 legacy=0x7f2c0c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0039e0 0x7f2bec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2bec0047c0 0x7f2bec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2bec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c6e40 con 0x7f2bec0051f0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec00c150 con 0x7f2bec0039e0 2024-01-16T15:00:08.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec00c310 con 0x7f2bec0047c0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c0b400640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2bec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c0aa00640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0039e0 0x7f2bec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c0a000640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2bec0047c0 0x7f2bec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c0b400640 1 -- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2bec0047c0 msgr2=0x7f2bec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.415+0000 7f2c0b400640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2bec0047c0 0x7f2bec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0b400640 1 -- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0039e0 msgr2=0x7f2bec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0b400640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0039e0 0x7f2bec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0b400640 1 -- 172.21.15.43:0/4269449772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec003520 con 0x7f2bec0051f0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0aa00640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0039e0 0x7f2bec003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0a000640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2bec0047c0 0x7f2bec004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.416+0000 7f2c0b400640 1 -- 172.21.15.43:0/4269449772 reap_dead start 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c0b400640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2bec00b980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009ac0 tx=0x7f2bec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2dcfcd3c94455f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c04004020 con 0x7f2bec0051f0 2024-01-16T15:00:08.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c04004180 con 0x7f2bec0051f0 2024-01-16T15:00:08.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c09600640 1 -- 172.21.15.43:0/4269449772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0402c630 con 0x7f2bec0051f0 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c0be00640 1 -- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 msgr2=0x7f2bec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.417+0000 7f2c0be00640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2bec00b980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009ac0 tx=0x7f2bec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.418+0000 7f2c0be00640 1 -- 172.21.15.43:0/4269449772 shutdown_connections 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.418+0000 7f2c0be00640 1 --2- 172.21.15.43:0/4269449772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2bec00b980 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.418+0000 7f2c0be00640 1 -- 172.21.15.43:0/4269449772 >> 172.21.15.43:0/4269449772 conn(0x7f2c0c0d30a0 msgr2=0x7f2c0c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.418+0000 7f2c0be00640 1 -- 172.21.15.43:0/4269449772 shutdown_connections 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.418+0000 7f2c0be00640 1 -- 172.21.15.43:0/4269449772 wait complete. 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.419+0000 7f2c0be00640 1 Processor -- start 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.420+0000 7f2c0be00640 1 -- start start 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c0c09cde0 0x7f2c0c09d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 0x7f2c0c0b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c09d780 con 0x7f2bec0051f0 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c15a9e0 con 0x7f2c0c0b6340 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c06baf0 con 0x7f2c0c09cde0 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55818/0 (socket says 172.21.15.43:55818) 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0aa00640 1 -- 172.21.15.43:0/1781619964 learned_addr learned my addr 172.21.15.43:0/1781619964 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0a000640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c0c09cde0 0x7f2c0c09d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.421+0000 7f2c0b400640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 0x7f2c0c0b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 -- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c0c09cde0 msgr2=0x7f2c0c09d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c0c09cde0 0x7f2c0c09d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 -- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 msgr2=0x7f2c0c0b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 0x7f2c0c0b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 -- 172.21.15.43:0/1781619964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c06bdd0 con 0x7f2bec0051f0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0b400640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 0x7f2c0c0b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0aa00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc001200 tx=0x7f2bfc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc0074e0 con 0x7f2bec0051f0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc007640 con 0x7f2bec0051f0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc0049e0 con 0x7f2bec0051f0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c06c0b0 con 0x7f2bec0051f0 2024-01-16T15:00:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.422+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c06c670 con 0x7f2bec0051f0 2024-01-16T15:00:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.423+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c072900 con 0x7f2bec0051f0 2024-01-16T15:00:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.424+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2bfc004b40 con 0x7f2bec0051f0 2024-01-16T15:00:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.426+0000 7f2c03400640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 0x7f2be406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.426+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bfc089620 con 0x7f2bec0051f0 2024-01-16T15:00:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.426+0000 7f2c0a000640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 0x7f2be406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.427+0000 7f2c0a000640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 0x7f2be406d7b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4007fd0 tx=0x7f2bf4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.430+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bfc054360 con 0x7f2bec0051f0 2024-01-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.623+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c0c04e840 con 0x7f2bec0051f0 2024-01-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.624+0000 7f2c03400640 1 -- 172.21.15.43:0/1781619964 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bfc0190a0 con 0x7f2bec0051f0 2024-01-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 msgr2=0x7f2be406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 0x7f2be406d7b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4007fd0 tx=0x7f2bf4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 msgr2=0x7f2c0c1514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc001200 tx=0x7f2bfc00dd40 comp rx=0 tx=0).stop 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 shutdown_connections 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2be406b2f0 0x7f2be406d7b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c0c0b6340 0x7f2c0c0b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c0c09cde0 0x7f2c0c09d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 --2- 172.21.15.43:0/1781619964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bec0051f0 0x7f2c0c1514b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.627+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 >> 172.21.15.43:0/1781619964 conn(0x7f2c0c0d30a0 msgr2=0x7f2c0c0b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.628+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 shutdown_connections 2024-01-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.628+0000 7f2c0be00640 1 -- 172.21.15.43:0/1781619964 wait complete. 2024-01-16T15:00:08.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:08.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.870+0000 7fa912400640 1 Processor -- start 2024-01-16T15:00:08.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.870+0000 7fa912400640 1 -- start start 2024-01-16T15:00:08.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa912400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa90c0b48d0 0x7fa90c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa912400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa90c0b3210 0x7fa90c167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa912400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0b23e0 con 0x7fa90c160bd0 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa912400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0cdc70 con 0x7fa90c0b48d0 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa912400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0b2870 con 0x7fa90c0b3210 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa90aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa90c0b48d0 0x7fa90c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa90b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa90c160bd0 0x7fa90c0b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38778/0 (socket says 172.21.15.43:38778) 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.871+0000 7fa90b400640 1 -- 172.21.15.43:0/1502247106 learned_addr learned my addr 172.21.15.43:0/1502247106 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90be00640 1 --2- 172.21.15.43:0/1502247106 >> v2:172.21.15.69:3300/0 conn(0x7fa90c0b3210 0x7fa90c167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90aa00640 1 -- 172.21.15.43:0/1502247106 >> v2:172.21.15.69:3300/0 conn(0x7fa90c0b3210 msgr2=0x7fa90c167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90aa00640 1 --2- 172.21.15.43:0/1502247106 >> v2:172.21.15.69:3300/0 conn(0x7fa90c0b3210 0x7fa90c167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90a000640 1 -- 172.21.15.43:0/1502247106 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267574332 0 0) 0x7fa90c0b23e0 con 0x7fa90c160bd0 2024-01-16T15:00:08.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90aa00640 1 -- 172.21.15.43:0/1502247106 >> v1:172.21.15.43:6789/0 conn(0x7fa90c160bd0 legacy=0x7fa90c0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.872+0000 7fa90aa00640 1 -- 172.21.15.43:0/1502247106 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0d41b0 con 0x7fa90c0b48d0 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.873+0000 7fa90aa00640 1 --2- 172.21.15.43:0/1502247106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0b1ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0d2e10 tx=0x7fa8f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a81e7b94325b17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.873+0000 7fa90a000640 1 -- 172.21.15.43:0/1502247106 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f402e070 con 0x7fa90c0b48d0 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.873+0000 7fa90a000640 1 -- 172.21.15.43:0/1502247106 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f40315a0 con 0x7fa90c0b48d0 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.873+0000 7fa90a000640 1 -- 172.21.15.43:0/1502247106 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4031930 con 0x7fa90c0b48d0 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 -- 172.21.15.43:0/1502247106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b48d0 msgr2=0x7fa90c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 --2- 172.21.15.43:0/1502247106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0b1ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0d2e10 tx=0x7fa8f402b840 comp rx=0 tx=0).stop 2024-01-16T15:00:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 -- 172.21.15.43:0/1502247106 shutdown_connections 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 --2- 172.21.15.43:0/1502247106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0b1ea0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 --2- 172.21.15.43:0/1502247106 >> v2:172.21.15.69:3300/0 conn(0x7fa90c0b3210 0x7fa90c167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 -- 172.21.15.43:0/1502247106 >> 172.21.15.43:0/1502247106 conn(0x7fa90c09c5a0 msgr2=0x7fa90c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 -- 172.21.15.43:0/1502247106 shutdown_connections 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.874+0000 7fa912400640 1 -- 172.21.15.43:0/1502247106 wait complete. 2024-01-16T15:00:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.876+0000 7fa912400640 1 Processor -- start 2024-01-16T15:00:08.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.876+0000 7fa912400640 1 -- start start 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 0x7fa90c15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0cd8f0 con 0x7fa90c0b3210 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c15ccb0 con 0x7fa90c0b48d0 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa912400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0cd050 con 0x7fa90c160bd0 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa90b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa90b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55838/0 (socket says 172.21.15.43:55838) 2024-01-16T15:00:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.877+0000 7fa90b400640 1 -- 172.21.15.43:0/4272704363 learned_addr learned my addr 172.21.15.43:0/4272704363 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90aa00640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90be00640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 0x7fa90c15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 -- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 msgr2=0x7fa90c15c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 0x7fa90c15c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 -- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa90c0b48d0 msgr2=0x7fa90c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 -- 172.21.15.43:0/4272704363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4009400 con 0x7fa90c0b3210 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90be00640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 0x7fa90c15c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa90b400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c7c0 tx=0x7fa8fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00a640 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc00a7c0 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c0cd330 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.878+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00ab70 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.879+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c1f6b10 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.880+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c1f6970 con 0x7fa90c0b3210 2024-01-16T15:00:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.881+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa8fc00acd0 con 0x7fa90c0b3210 2024-01-16T15:00:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.881+0000 7fa903400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 0x7fa8e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.882+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8fc0893d0 con 0x7fa90c0b3210 2024-01-16T15:00:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.882+0000 7fa90aa00640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 0x7fa8e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.883+0000 7fa90aa00640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 0x7fa8e406d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4036000 tx=0x7fa8f403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:08.887+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa90c1f6970 con 0x7fa90c0b3210 2024-01-16T15:00:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.080+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa90c1f6970 con 0x7fa90c0b3210 2024-01-16T15:00:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.081+0000 7fa903400640 1 -- 172.21.15.43:0/4272704363 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa90c1f6970 con 0x7fa90c0b3210 2024-01-16T15:00:09.062 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 msgr2=0x7fa8e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 0x7fa8e406d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4036000 tx=0x7fa8f403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 msgr2=0x7fa90c15cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c7c0 tx=0x7fa8fc00cc80 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 shutdown_connections 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa8e406b2c0 0x7fa8e406d780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa90c0b48d0 0x7fa90c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa90c160bd0 0x7fa90c15c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 --2- 172.21.15.43:0/4272704363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa90c0b3210 0x7fa90c15cf00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 >> 172.21.15.43:0/4272704363 conn(0x7fa90c09c5a0 msgr2=0x7fa90c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 shutdown_connections 2024-01-16T15:00:09.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.084+0000 7fa912400640 1 -- 172.21.15.43:0/4272704363 wait complete. 2024-01-16T15:00:09.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.325+0000 7fd67a200640 1 Processor -- start 2024-01-16T15:00:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.326+0000 7fd67a200640 1 -- start start 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.327+0000 7fd67a200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd67415efa0 0x7fd674161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.327+0000 7fd67a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd67406da40 0x7fd67406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.327+0000 7fd67a200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67406c9a0 con 0x7fd6740b2970 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.327+0000 7fd67a200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67406cc80 con 0x7fd67415efa0 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.327+0000 7fd67a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67406cf60 con 0x7fd67406da40 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd67415efa0 0x7fd674161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd67415efa0 0x7fd674161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55854/0 (socket says 172.21.15.43:55854) 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 -- 172.21.15.43:0/956688670 learned_addr learned my addr 172.21.15.43:0/956688670 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd66be00640 1 --2- 172.21.15.43:0/956688670 >> v2:172.21.15.69:3300/0 conn(0x7fd67406da40 0x7fd67406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 -- 172.21.15.43:0/956688670 >> v2:172.21.15.69:3300/0 conn(0x7fd67406da40 msgr2=0x7fd67406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 --2- 172.21.15.43:0/956688670 >> v2:172.21.15.69:3300/0 conn(0x7fd67406da40 0x7fd67406c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 -- 172.21.15.43:0/956688670 >> v1:172.21.15.69:6789/0 conn(0x7fd6740b2970 legacy=0x7fd6741650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.328+0000 7fd678e00640 1 -- 172.21.15.43:0/956688670 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67406d240 con 0x7fd67415efa0 2024-01-16T15:00:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.329+0000 7fd678e00640 1 --2- 172.21.15.43:0/956688670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd674161ac0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd6740b4670 tx=0x7fd66402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3649554f3b552d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.330+0000 7fd673e00640 1 -- 172.21.15.43:0/956688670 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6640317e0 con 0x7fd67415efa0 2024-01-16T15:00:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.330+0000 7fd673e00640 1 -- 172.21.15.43:0/956688670 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664031940 con 0x7fd67415efa0 2024-01-16T15:00:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.330+0000 7fd673e00640 1 -- 172.21.15.43:0/956688670 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664031cd0 con 0x7fd67415efa0 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 -- 172.21.15.43:0/956688670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 msgr2=0x7fd674161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 --2- 172.21.15.43:0/956688670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd674161ac0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd6740b4670 tx=0x7fd66402b880 comp rx=0 tx=0).stop 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 -- 172.21.15.43:0/956688670 shutdown_connections 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 --2- 172.21.15.43:0/956688670 >> v2:172.21.15.69:3300/0 conn(0x7fd67406da40 0x7fd67406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 --2- 172.21.15.43:0/956688670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd674161ac0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 -- 172.21.15.43:0/956688670 >> 172.21.15.43:0/956688670 conn(0x7fd6740cec10 msgr2=0x7fd6740cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 -- 172.21.15.43:0/956688670 shutdown_connections 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.331+0000 7fd67a200640 1 -- 172.21.15.43:0/956688670 wait complete. 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.333+0000 7fd67a200640 1 Processor -- start 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.333+0000 7fd67a200640 1 -- start start 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd67406da40 0x7fd674153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 0x7fd6740af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674154d20 con 0x7fd67415efa0 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674154fc0 con 0x7fd6740b2970 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd67a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6741552a0 con 0x7fd67406da40 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd679800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd679800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55864/0 (socket says 172.21.15.43:55864) 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.334+0000 7fd679800640 1 -- 172.21.15.43:0/248219533 learned_addr learned my addr 172.21.15.43:0/248219533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd66be00640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 0x7fd6740af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd678e00640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd67406da40 0x7fd674153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 -- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd67406da40 msgr2=0x7fd674153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd67406da40 0x7fd674153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 -- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 msgr2=0x7fd6740af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 0x7fd6740af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 -- 172.21.15.43:0/248219533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640094d0 con 0x7fd67415efa0 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd66be00640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 0x7fd6740af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd679800640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c780 tx=0x7fd66c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c003d80 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c00ce90 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0052b0 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.335+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd674155580 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.336+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd674154320 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.337+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd674155aa0 con 0x7fd67415efa0 2024-01-16T15:00:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.338+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd66c005410 con 0x7fd67415efa0 2024-01-16T15:00:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.339+0000 7fd671600640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.339+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd66c089010 con 0x7fd67415efa0 2024-01-16T15:00:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.339+0000 7fd678e00640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.340+0000 7fd678e00640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 0x7fd64c06d780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd664036000 tx=0x7fd66403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.343+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd674155aa0 con 0x7fd67415efa0 2024-01-16T15:00:09.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.538+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd674155aa0 con 0x7fd67415efa0 2024-01-16T15:00:09.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.539+0000 7fd671600640 1 -- 172.21.15.43:0/248219533 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd674155aa0 con 0x7fd67415efa0 2024-01-16T15:00:09.520 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 msgr2=0x7fd64c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 0x7fd64c06d780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd664036000 tx=0x7fd66403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 msgr2=0x7fd6740b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c780 tx=0x7fd66c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 shutdown_connections 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6740b2970 0x7fd6740af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd67415efa0 0x7fd6740b0e50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 --2- 172.21.15.43:0/248219533 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd67406da40 0x7fd674153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 >> 172.21.15.43:0/248219533 conn(0x7fd6740cec10 msgr2=0x7fd67415f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 shutdown_connections 2024-01-16T15:00:09.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.542+0000 7fd67a200640 1 -- 172.21.15.43:0/248219533 wait complete. 2024-01-16T15:00:09.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T15:00:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.784+0000 7fd45c400640 1 Processor -- start 2024-01-16T15:00:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.784+0000 7fd45c400640 1 -- start start 2024-01-16T15:00:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.785+0000 7fd45c400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd4540b39a0 0x7fd454071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd45c400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd454071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd45c400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45413da30 con 0x7fd45415a100 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd45c400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4540ac840 con 0x7fd4540b39a0 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd45c400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ac560 con 0x7fd4540abff0 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd459a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd45415a100 0x7fd4540b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38790/0 (socket says 172.21.15.43:38790) 2024-01-16T15:00:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd459a00640 1 -- 172.21.15.43:0/411519778 learned_addr learned my addr 172.21.15.43:0/411519778 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd459000640 1 --2- 172.21.15.43:0/411519778 >> v2:172.21.15.69:3300/0 conn(0x7fd4540b39a0 0x7fd454071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd45a400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd454071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.786+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368593132 0 0) 0x7fd45413da30 con 0x7fd45415a100 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd434003520 con 0x7fd45415a100 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031960144 0 0) 0x7fd434003520 con 0x7fd45415a100 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 msgr2=0x7fd454071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd454071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 >> v2:172.21.15.69:3300/0 conn(0x7fd4540b39a0 msgr2=0x7fd454071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 --2- 172.21.15.43:0/411519778 >> v2:172.21.15.69:3300/0 conn(0x7fd4540b39a0 0x7fd454071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454070650 con 0x7fd45415a100 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd444001670 con 0x7fd45415a100 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 >> v1:172.21.15.43:6789/0 conn(0x7fd45415a100 legacy=0x7fd4540b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd434003970 0x7fd434003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd4340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd434005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ac840 con 0x7fd434004d00 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ac560 con 0x7fd434003970 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43400b980 con 0x7fd4540abff0 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.787+0000 7fd45a400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd434005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd459a00640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd434003970 0x7fd434003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd459000640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd4340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd45a400640 1 -- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 msgr2=0x7fd4340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd45a400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540abff0 0x7fd4340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd45a400640 1 -- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd434003970 msgr2=0x7fd434003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd45a400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd434003970 0x7fd434003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.788+0000 7fd45a400640 1 -- 172.21.15.43:0/411519778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434003520 con 0x7fd434004d00 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.789+0000 7fd45a400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd434005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fc0 tx=0x7fd4340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3161294b006c32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.789+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c02cc20 con 0x7fd434004d00 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.789+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c00f040 con 0x7fd434004d00 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.789+0000 7fd453e00640 1 -- 172.21.15.43:0/411519778 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c034440 con 0x7fd434004d00 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 -- 172.21.15.43:0/411519778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 msgr2=0x7fd434005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 --2- 172.21.15.43:0/411519778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd434005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fc0 tx=0x7fd4340057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45a400640 1 -- 172.21.15.43:0/411519778 reap_dead start 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 -- 172.21.15.43:0/411519778 shutdown_connections 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 -- 172.21.15.43:0/411519778 >> 172.21.15.43:0/411519778 conn(0x7fd4540b0d80 msgr2=0x7fd4540b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 -- 172.21.15.43:0/411519778 shutdown_connections 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.790+0000 7fd45c400640 1 -- 172.21.15.43:0/411519778 wait complete. 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.792+0000 7fd45c400640 1 Processor -- start 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.792+0000 7fd45c400640 1 -- start start 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4540715f0 0x7fd4540cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540980b0 0x7fd454098560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c8b60 con 0x7fd434004d00 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540cc4c0 con 0x7fd4540715f0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd45c400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45406d7e0 con 0x7fd4540980b0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd459a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd459a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55894/0 (socket says 172.21.15.43:55894) 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.793+0000 7fd459a00640 1 -- 172.21.15.43:0/3158141082 learned_addr learned my addr 172.21.15.43:0/3158141082 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459000640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4540715f0 0x7fd4540cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 -- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540980b0 msgr2=0x7fd454098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540980b0 0x7fd454098560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 -- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4540715f0 msgr2=0x7fd4540cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4540715f0 0x7fd4540cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 -- 172.21.15.43:0/3158141082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45406dac0 con 0x7fd434004d00 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd459a00640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd44400e840 tx=0x7fd44400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44400c850 con 0x7fd434004d00 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44400c9b0 con 0x7fd434004d00 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44400cd60 con 0x7fd434004d00 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4540c84a0 con 0x7fd434004d00 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.794+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd454098aa0 con 0x7fd434004d00 2024-01-16T15:00:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.796+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4540c8ca0 con 0x7fd434004d00 2024-01-16T15:00:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.796+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd44401d460 con 0x7fd434004d00 2024-01-16T15:00:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.797+0000 7fd451600640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 0x7fd42c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.797+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd444089b00 con 0x7fd434004d00 2024-01-16T15:00:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.798+0000 7fd459000640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 0x7fd42c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.798+0000 7fd459000640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 0x7fd42c06d7b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd454167e80 tx=0x7fd43c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:09.802+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4540c8ca0 con 0x7fd434004d00 2024-01-16T15:00:10.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.069+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fd4540c8ca0 con 0x7fd434004d00 2024-01-16T15:00:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.826+0000 7fd451600640 1 -- 172.21.15.43:0/3158141082 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v58) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd4540c8ca0 con 0x7fd434004d00 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.829+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 msgr2=0x7fd42c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.829+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 0x7fd42c06d7b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd454167e80 tx=0x7fd43c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.829+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 msgr2=0x7fd4540cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.829+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd44400e840 tx=0x7fd44400ed00 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.829+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 shutdown_connections 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd42c06b2f0 0x7fd42c06d7b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4540715f0 0x7fd4540cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4540980b0 0x7fd454098560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 --2- 172.21.15.43:0/3158141082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd434004d00 0x7fd4540cba40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 >> 172.21.15.43:0/3158141082 conn(0x7fd4540b0d80 msgr2=0x7fd4540b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 shutdown_connections 2024-01-16T15:00:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:10.830+0000 7fd45c400640 1 -- 172.21.15.43:0/3158141082 wait complete. 2024-01-16T15:00:10.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:00:11.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.070+0000 7ff6dd800640 1 Processor -- start 2024-01-16T15:00:11.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.070+0000 7ff6dd800640 1 -- start start 2024-01-16T15:00:11.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.071+0000 7ff6dd800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.071+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d80ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6dd800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d8160cc0 con 0x7ff6d81608e0 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6dd800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d80d2ab0 con 0x7ff6d80d36d0 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6dd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80b3a10 con 0x7ff6d80b3220 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:35258/0 (socket says 172.21.15.43:35258) 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6cfe00640 1 -- 172.21.15.43:0/2010456269 learned_addr learned my addr 172.21.15.43:0/2010456269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.072+0000 7ff6dce00640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d80ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.073+0000 7ff6cfe00640 1 -- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 msgr2=0x7ff6d80ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.073+0000 7ff6cfe00640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d80ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.073+0000 7ff6cfe00640 1 -- 172.21.15.43:0/2010456269 >> v1:172.21.15.69:6789/0 conn(0x7ff6d81608e0 legacy=0x7ff6d80d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.073+0000 7ff6cfe00640 1 -- 172.21.15.43:0/2010456269 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80b3cf0 con 0x7ff6d80d36d0 2024-01-16T15:00:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.074+0000 7ff6cfe00640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009870 tx=0x7ff6c002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cdbfcdcc9fd42ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6d7400640 1 -- 172.21.15.43:0/2010456269 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0030840 con 0x7ff6d80d36d0 2024-01-16T15:00:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6d7400640 1 -- 172.21.15.43:0/2010456269 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c00309a0 con 0x7ff6d80d36d0 2024-01-16T15:00:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6d7400640 1 -- 172.21.15.43:0/2010456269 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0030d30 con 0x7ff6d80d36d0 2024-01-16T15:00:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 -- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 msgr2=0x7ff6d80b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009870 tx=0x7ff6c002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 -- 172.21.15.43:0/2010456269 shutdown_connections 2024-01-16T15:00:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80b2ce0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 --2- 172.21.15.43:0/2010456269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d80ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.075+0000 7ff6dd800640 1 -- 172.21.15.43:0/2010456269 >> 172.21.15.43:0/2010456269 conn(0x7ff6d8066b70 msgr2=0x7ff6d809c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.076+0000 7ff6dd800640 1 -- 172.21.15.43:0/2010456269 shutdown_connections 2024-01-16T15:00:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.076+0000 7ff6dd800640 1 -- 172.21.15.43:0/2010456269 wait complete. 2024-01-16T15:00:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.077+0000 7ff6dd800640 1 Processor -- start 2024-01-16T15:00:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 -- start start 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8159120 con 0x7ff6d81608e0 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80b1c90 con 0x7ff6d80d36d0 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.078+0000 7ff6dd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8159350 con 0x7ff6d80b3220 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55904/0 (socket says 172.21.15.43:55904) 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 -- 172.21.15.43:0/3403999628 learned_addr learned my addr 172.21.15.43:0/3403999628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6cfe00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6d7e00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 -- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 msgr2=0x7ff6d8152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d8152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 -- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 msgr2=0x7ff6d80c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6dce00640 1 -- 172.21.15.43:0/3403999628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00094d0 con 0x7ff6d81608e0 2024-01-16T15:00:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.079+0000 7ff6d7e00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d8152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6dce00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000efc0 tx=0x7ff6d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0009ba0 con 0x7ff6d81608e0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000f040 con 0x7ff6d81608e0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d8159630 con 0x7ff6d81608e0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00146c0 con 0x7ff6d81608e0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.080+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d8159bf0 con 0x7ff6d81608e0 2024-01-16T15:00:11.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.081+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d804e840 con 0x7ff6d81608e0 2024-01-16T15:00:11.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.082+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff6d0014820 con 0x7ff6d81608e0 2024-01-16T15:00:11.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.083+0000 7ff6d4c00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 0x7ff6b006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.083+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6d008a7d0 con 0x7ff6d81608e0 2024-01-16T15:00:11.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.084+0000 7ff6d7e00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 0x7ff6b006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.084+0000 7ff6d7e00640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 0x7ff6b006d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8000f60 tx=0x7ff6c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.088+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6d00555a0 con 0x7ff6d81608e0 2024-01-16T15:00:11.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.282+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff6d80c6600 con 0x7ff6d81608e0 2024-01-16T15:00:11.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.287+0000 7ff6d4c00640 1 -- 172.21.15.43:0/3403999628 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff6d0058bc0 con 0x7ff6d81608e0 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 msgr2=0x7ff6b006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 0x7ff6b006d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8000f60 tx=0x7ff6c8009040 comp rx=0 tx=0).stop 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 msgr2=0x7ff6d815a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000efc0 tx=0x7ff6d000c490 comp rx=0 tx=0).stop 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 shutdown_connections 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6b006b2c0 0x7ff6b006d780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6d80d36d0 0x7ff6d80c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6d81608e0 0x7ff6d815a900 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 --2- 172.21.15.43:0/3403999628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6d80b3220 0x7ff6d8152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 >> 172.21.15.43:0/3403999628 conn(0x7ff6d8066b70 msgr2=0x7ff6d806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 shutdown_connections 2024-01-16T15:00:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.291+0000 7ff6dd800640 1 -- 172.21.15.43:0/3403999628 wait complete. 2024-01-16T15:00:11.292 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:00:11.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:11.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:11.526 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:00:11.526 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:11.526 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:11.586 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:11.586 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:11.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:11.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.886+0000 7fb717200640 1 Processor -- start 2024-01-16T15:00:11.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.886+0000 7fb717200640 1 -- start start 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb717200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb71813dfd0 0x7fb71815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb717200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb7180cf500 0x7fb71806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb717200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180ce7c0 con 0x7fb7180cf050 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb717200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb718140900 con 0x7fb71813dfd0 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb717200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71815fa70 con 0x7fb7180cf500 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb715e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb71813dfd0 0x7fb71815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb715e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb71813dfd0 0x7fb71815be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55916/0 (socket says 172.21.15.43:55916) 2024-01-16T15:00:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.888+0000 7fb715e00640 1 -- 172.21.15.43:0/3790746760 learned_addr learned my addr 172.21.15.43:0/3790746760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.889+0000 7fb716800640 1 --2- 172.21.15.43:0/3790746760 >> v2:172.21.15.69:3300/0 conn(0x7fb7180cf500 0x7fb71806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.889+0000 7fb715e00640 1 -- 172.21.15.43:0/3790746760 >> v2:172.21.15.69:3300/0 conn(0x7fb7180cf500 msgr2=0x7fb71806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.889+0000 7fb715e00640 1 --2- 172.21.15.43:0/3790746760 >> v2:172.21.15.69:3300/0 conn(0x7fb7180cf500 0x7fb71806c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.889+0000 7fb715e00640 1 -- 172.21.15.43:0/3790746760 >> v1:172.21.15.43:6789/0 conn(0x7fb7180cf050 legacy=0x7fb71806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.889+0000 7fb715e00640 1 -- 172.21.15.43:0/3790746760 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71815ee60 con 0x7fb71813dfd0 2024-01-16T15:00:11.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.890+0000 7fb715e00640 1 --2- 172.21.15.43:0/3790746760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815be70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb7181416f0 tx=0x7fb70802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fcdae25319873b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.890+0000 7fb714a00640 1 -- 172.21.15.43:0/3790746760 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708030860 con 0x7fb71813dfd0 2024-01-16T15:00:11.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.890+0000 7fb714a00640 1 -- 172.21.15.43:0/3790746760 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7080309c0 con 0x7fb71813dfd0 2024-01-16T15:00:11.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.890+0000 7fb714a00640 1 -- 172.21.15.43:0/3790746760 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708030d50 con 0x7fb71813dfd0 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 -- 172.21.15.43:0/3790746760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 msgr2=0x7fb71815be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 --2- 172.21.15.43:0/3790746760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815be70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb7181416f0 tx=0x7fb70802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 -- 172.21.15.43:0/3790746760 shutdown_connections 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 --2- 172.21.15.43:0/3790746760 >> v2:172.21.15.69:3300/0 conn(0x7fb7180cf500 0x7fb71806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 --2- 172.21.15.43:0/3790746760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815be70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 -- 172.21.15.43:0/3790746760 >> 172.21.15.43:0/3790746760 conn(0x7fb71806a860 msgr2=0x7fb71806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 -- 172.21.15.43:0/3790746760 shutdown_connections 2024-01-16T15:00:11.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.891+0000 7fb717200640 1 -- 172.21.15.43:0/3790746760 wait complete. 2024-01-16T15:00:11.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.893+0000 7fb717200640 1 Processor -- start 2024-01-16T15:00:11.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.894+0000 7fb717200640 1 -- start start 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 0x7fb718152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 0x7fb71815ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb718152e60 con 0x7fb71813dfd0 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71815cb00 con 0x7fb7180cf050 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb717200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71815cde0 con 0x7fb7180cf500 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55922/0 (socket says 172.21.15.43:55922) 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 -- 172.21.15.43:0/1991054513 learned_addr learned my addr 172.21.15.43:0/1991054513 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb715e00640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 0x7fb718152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 -- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 msgr2=0x7fb71815ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb715400640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 0x7fb71815ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 0x7fb71815ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 -- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 msgr2=0x7fb718152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 0x7fb718152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb716800640 1 -- 172.21.15.43:0/1991054513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080094d0 con 0x7fb71813dfd0 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb715400640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 0x7fb71815ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.895+0000 7fb715e00640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 0x7fb718152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb716800640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb71000cd70 tx=0x7fb71000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710003b90 con 0x7fb71813dfd0 2024-01-16T15:00:11.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710003cf0 con 0x7fb71813dfd0 2024-01-16T15:00:11.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710004240 con 0x7fb71813dfd0 2024-01-16T15:00:11.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71815d0c0 con 0x7fb71813dfd0 2024-01-16T15:00:11.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.896+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71815d680 con 0x7fb71813dfd0 2024-01-16T15:00:11.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.898+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71815d8a0 con 0x7fb71813dfd0 2024-01-16T15:00:11.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.898+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb7100043a0 con 0x7fb71813dfd0 2024-01-16T15:00:11.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.899+0000 7fb70e000640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 0x7fb6f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.899+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb710012070 con 0x7fb71813dfd0 2024-01-16T15:00:11.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.900+0000 7fb715e00640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 0x7fb6f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.900+0000 7fb715e00640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 0x7fb6f006d780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb708036000 tx=0x7fb70803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:11.904+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb710055430 con 0x7fb71813dfd0 2024-01-16T15:00:12.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.101+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb71813caa0 con 0x7fb71813dfd0 2024-01-16T15:00:12.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.104+0000 7fb70e000640 1 -- 172.21.15.43:0/1991054513 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16457 (secure 0 0 0) 0x7fb710058a50 con 0x7fb71813dfd0 2024-01-16T15:00:12.085 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:12.085 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":22,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T14:59:28.696194+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.106+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 msgr2=0x7fb6f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.106+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 0x7fb6f006d780 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb708036000 tx=0x7fb70803b040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.106+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 msgr2=0x7fb71815e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.106+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb71000cd70 tx=0x7fb71000a660 comp rx=0 tx=0).stop 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 shutdown_connections 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb6f006b2c0 0x7fb6f006d780 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb71813dfd0 0x7fb71815e410 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7180cf500 0x7fb71815ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 --2- 172.21.15.43:0/1991054513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7180cf050 0x7fb718152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 >> 172.21.15.43:0/1991054513 conn(0x7fb71806a860 msgr2=0x7fb71806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 shutdown_connections 2024-01-16T15:00:12.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.107+0000 7fb717200640 1 -- 172.21.15.43:0/1991054513 wait complete. 2024-01-16T15:00:12.108 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:12.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:12.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.355+0000 7feae8800640 1 Processor -- start 2024-01-16T15:00:12.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.355+0000 7feae8800640 1 -- start start 2024-01-16T15:00:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.356+0000 7feae8800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 0x7feae00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae8800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 0x7feae0167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae8800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00b23e0 con 0x7feae00b48d0 2024-01-16T15:00:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae8800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae00cdc70 con 0x7feae0160bd0 2024-01-16T15:00:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae8800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00b2870 con 0x7feae00b3210 2024-01-16T15:00:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 0x7feae00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae6800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 0x7feae0167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 0x7feae00b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:35286/0 (socket says 172.21.15.43:35286) 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.357+0000 7feae5e00640 1 -- 172.21.15.43:0/1600509637 learned_addr learned my addr 172.21.15.43:0/1600509637 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.358+0000 7feae6800640 1 -- 172.21.15.43:0/1600509637 >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 msgr2=0x7feae00b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.358+0000 7feae6800640 1 --2- 172.21.15.43:0/1600509637 >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 0x7feae00b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.358+0000 7feae6800640 1 -- 172.21.15.43:0/1600509637 >> v1:172.21.15.69:6789/0 conn(0x7feae00b48d0 legacy=0x7feae00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.358+0000 7feae6800640 1 -- 172.21.15.43:0/1600509637 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae00d41b0 con 0x7feae00b3210 2024-01-16T15:00:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.360+0000 7feae6800640 1 --2- 172.21.15.43:0/1600509637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 0x7feae0167290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9360ccc32601eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.360+0000 7feae4a00640 1 -- 172.21.15.43:0/1600509637 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead802e070 con 0x7feae00b3210 2024-01-16T15:00:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.360+0000 7feae4a00640 1 -- 172.21.15.43:0/1600509637 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80315a0 con 0x7feae00b3210 2024-01-16T15:00:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.360+0000 7feae4a00640 1 -- 172.21.15.43:0/1600509637 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8031930 con 0x7feae00b3210 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 -- 172.21.15.43:0/1600509637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 msgr2=0x7feae0167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 --2- 172.21.15.43:0/1600509637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 0x7feae0167290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802b840 comp rx=0 tx=0).stop 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 -- 172.21.15.43:0/1600509637 shutdown_connections 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 --2- 172.21.15.43:0/1600509637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae00b3210 0x7feae0167290 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 --2- 172.21.15.43:0/1600509637 >> v2:172.21.15.69:3300/0 conn(0x7feae0160bd0 0x7feae00b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 -- 172.21.15.43:0/1600509637 >> 172.21.15.43:0/1600509637 conn(0x7feae009c5a0 msgr2=0x7feae00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 -- 172.21.15.43:0/1600509637 shutdown_connections 2024-01-16T15:00:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.361+0000 7feae8800640 1 -- 172.21.15.43:0/1600509637 wait complete. 2024-01-16T15:00:12.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.364+0000 7feae8800640 1 Processor -- start 2024-01-16T15:00:12.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.364+0000 7feae8800640 1 -- start start 2024-01-16T15:00:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 0x7feae01552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 0x7feae00caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00b8970 con 0x7feae00b48d0 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00cd140 con 0x7feae00b3210 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.365+0000 7feae8800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00cc590 con 0x7feae0160bd0 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55934/0 (socket says 172.21.15.43:55934) 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 -- 172.21.15.43:0/3500359980 learned_addr learned my addr 172.21.15.43:0/3500359980 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5e00640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 0x7feae01552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae6800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 0x7feae00caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 -- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 msgr2=0x7feae00caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 0x7feae00caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 -- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 msgr2=0x7feae01552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 0x7feae01552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5400640 1 -- 172.21.15.43:0/3500359980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8009400 con 0x7feae00b48d0 2024-01-16T15:00:12.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae6800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 0x7feae00caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.366+0000 7feae5e00640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 0x7feae01552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feae5400640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feac800c7d0 tx=0x7feac800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8003b70 con 0x7feae00b48d0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8003cd0 con 0x7feae00b48d0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae00cc810 con 0x7feae00b48d0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac800aa30 con 0x7feae00b48d0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.367+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae00ccdd0 con 0x7feae00b48d0 2024-01-16T15:00:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.368+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae00a1210 con 0x7feae00b48d0 2024-01-16T15:00:12.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.369+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feac800ab90 con 0x7feae00b48d0 2024-01-16T15:00:12.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.370+0000 7feade000640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 0x7feab806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.370+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feac8089b30 con 0x7feae00b48d0 2024-01-16T15:00:12.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.370+0000 7feae5e00640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 0x7feab806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.371+0000 7feae5e00640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 0x7feab806d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fead0009a20 tx=0x7fead0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.375+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feac8054870 con 0x7feae00b48d0 2024-01-16T15:00:12.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.567+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feae004e920 con 0x7feae00b48d0 2024-01-16T15:00:12.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.569+0000 7feade000640 1 -- 172.21.15.43:0/3500359980 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7feac8057e90 con 0x7feae00b48d0 2024-01-16T15:00:12.549 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.571+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 msgr2=0x7feab806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.571+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 0x7feab806d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fead0009a20 tx=0x7fead0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.571+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 msgr2=0x7feae00ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.571+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feac800c7d0 tx=0x7feac800cc90 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 shutdown_connections 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feab806b2c0 0x7feab806d780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae0160bd0 0x7feae00caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae00b48d0 0x7feae00ca980 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 --2- 172.21.15.43:0/3500359980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae00b3210 0x7feae01552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 >> 172.21.15.43:0/3500359980 conn(0x7feae009c5a0 msgr2=0x7feae00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 shutdown_connections 2024-01-16T15:00:12.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.572+0000 7feae8800640 1 -- 172.21.15.43:0/3500359980 wait complete. 2024-01-16T15:00:12.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.811+0000 7f3ca8200640 1 Processor -- start 2024-01-16T15:00:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.811+0000 7f3ca8200640 1 -- start start 2024-01-16T15:00:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3ca00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca8200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca0157b30 con 0x7f3ca015aae0 2024-01-16T15:00:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca8200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca00b34f0 con 0x7f3ca00b2d70 2024-01-16T15:00:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca015aec0 con 0x7f3ca006d470 2024-01-16T15:00:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca5800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f3ca015aae0 0x7f3ca006cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38802/0 (socket says 172.21.15.43:38802) 2024-01-16T15:00:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca5800640 1 -- 172.21.15.43:0/2247917422 learned_addr learned my addr 172.21.15.43:0/2247917422 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.813+0000 7f3ca4e00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3ca00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296362513 0 0) 0x7f3ca0157b30 con 0x7f3ca015aae0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80003520 con 0x7f3ca015aae0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2207449088 0 0) 0x7f3c80003520 con 0x7f3ca015aae0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 msgr2=0x7f3ca00b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3ca00b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 >> v1:172.21.15.69:6789/0 conn(0x7f3ca00b2d70 legacy=0x7f3ca0158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00b37d0 con 0x7f3ca015aae0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f3c90001670 con 0x7f3ca015aae0 2024-01-16T15:00:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 >> v1:172.21.15.43:6789/0 conn(0x7f3ca015aae0 legacy=0x7f3ca006cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c800039a0 0x7f3c80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3c800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c80004d00 0x7f3c80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca015aec0 con 0x7f3c800039a0 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8000ba60 con 0x7f3c80004d00 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.814+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8000de30 con 0x7f3ca006d470 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c800039a0 0x7f3c80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca6200640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c80004d00 0x7f3c80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca4e00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3c800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 -- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 msgr2=0x7f3c800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3c800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 -- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c80004d00 msgr2=0x7f3c80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c80004d00 0x7f3c80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca5800640 1 -- 172.21.15.43:0/2247917422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80003520 con 0x7f3c800039a0 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca4e00640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca006d470 0x7f3c800047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.815+0000 7f3ca6200640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c80004d00 0x7f3c80005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.816+0000 7f3ca5800640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c800039a0 0x7f3c80003dc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c210 tx=0x7f3c800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d7ca98b30fa1eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.816+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90005ad0 con 0x7f3c800039a0 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.816+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90005c50 con 0x7f3c800039a0 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.816+0000 7f3c9fe00640 1 -- 172.21.15.43:0/2247917422 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9002c890 con 0x7f3c800039a0 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 -- 172.21.15.43:0/2247917422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c800039a0 msgr2=0x7f3c80003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 --2- 172.21.15.43:0/2247917422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c800039a0 0x7f3c80003dc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c210 tx=0x7f3c800057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca6200640 1 -- 172.21.15.43:0/2247917422 reap_dead start 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 -- 172.21.15.43:0/2247917422 shutdown_connections 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 -- 172.21.15.43:0/2247917422 >> 172.21.15.43:0/2247917422 conn(0x7f3ca00b0ba0 msgr2=0x7f3ca00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 -- 172.21.15.43:0/2247917422 shutdown_connections 2024-01-16T15:00:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.817+0000 7f3ca8200640 1 -- 172.21.15.43:0/2247917422 wait complete. 2024-01-16T15:00:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.819+0000 7f3ca8200640 1 Processor -- start 2024-01-16T15:00:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.819+0000 7f3ca8200640 1 -- start start 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c800039a0 0x7f3ca0151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 0x7f3ca00ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca01523b0 con 0x7f3ca009bd10 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00ae390 con 0x7f3c800039a0 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.820+0000 7f3ca8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca01f2810 con 0x7f3ca00ad9d0 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55964/0 (socket says 172.21.15.43:55964) 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 -- 172.21.15.43:0/1254256394 learned_addr learned my addr 172.21.15.43:0/1254256394 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca4e00640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 0x7f3ca00ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca5800640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c800039a0 0x7f3ca0151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 -- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 msgr2=0x7f3ca00ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 0x7f3ca00ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 -- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c800039a0 msgr2=0x7f3ca0151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c800039a0 0x7f3ca0151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 -- 172.21.15.43:0/1254256394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca01f2af0 con 0x7f3ca009bd10 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca4e00640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 0x7f3ca00ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.821+0000 7f3ca6200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00ceab0 tx=0x7f3c9800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.822+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98006d50 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.822+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98003970 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.822+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98003d20 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.822+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca01f2dd0 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.822+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca01f35d0 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.823+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca01f32f0 con 0x7f3ca009bd10 2024-01-16T15:00:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.824+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3c98005070 con 0x7f3ca009bd10 2024-01-16T15:00:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.825+0000 7f3c9d600640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 0x7f3c7806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.825+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c98089900 con 0x7f3ca009bd10 2024-01-16T15:00:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.825+0000 7f3ca5800640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 0x7f3c7806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.826+0000 7f3ca5800640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 0x7f3c7806d7b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c90034000 tx=0x7f3c90009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:12.830+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ca01f32f0 con 0x7f3ca009bd10 2024-01-16T15:00:13.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.028+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ca01f32f0 con 0x7f3ca009bd10 2024-01-16T15:00:13.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.029+0000 7f3c9d600640 1 -- 172.21.15.43:0/1254256394 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ca01f32f0 con 0x7f3ca009bd10 2024-01-16T15:00:13.010 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.031+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 msgr2=0x7f3c7806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.031+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 0x7f3c7806d7b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c90034000 tx=0x7f3c90009030 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.031+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 msgr2=0x7f3ca009c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.031+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00ceab0 tx=0x7f3c9800dc00 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 shutdown_connections 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c7806b2f0 0x7f3c7806d7b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ca009bd10 0x7f3ca009c1c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ca00ad9d0 0x7f3ca00ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 --2- 172.21.15.43:0/1254256394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c800039a0 0x7f3ca0151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 >> 172.21.15.43:0/1254256394 conn(0x7f3ca00b0ba0 msgr2=0x7f3ca0153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 shutdown_connections 2024-01-16T15:00:13.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.032+0000 7f3ca8200640 1 -- 172.21.15.43:0/1254256394 wait complete. 2024-01-16T15:00:13.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.267+0000 7f5120200640 1 Processor -- start 2024-01-16T15:00:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.267+0000 7f5120200640 1 -- start start 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5118162220 0x7f51180b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 0x7f5118162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f511806c3c0 0x7f511806c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5118162fa0 con 0x7f5118162220 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511806cd60 con 0x7f5118162600 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.269+0000 7f5120200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180706a0 con 0x7f511806c3c0 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5118162220 0x7f51180b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5118162220 0x7f51180b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38324/0 (socket says 172.21.15.43:38324) 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 -- 172.21.15.43:0/215941652 learned_addr learned my addr 172.21.15.43:0/215941652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511ce00640 1 --2- 172.21.15.43:0/215941652 >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 0x7f5118162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 -- 172.21.15.43:0/215941652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f511806c3c0 msgr2=0x7f511806c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 --2- 172.21.15.43:0/215941652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f511806c3c0 0x7f511806c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 -- 172.21.15.43:0/215941652 >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 msgr2=0x7f5118162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 --2- 172.21.15.43:0/215941652 >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 0x7f5118162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511d800640 1 -- 172.21.15.43:0/215941652 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5118070c60 con 0x7f5118162220 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.270+0000 7f511ce00640 1 --2- 172.21.15.43:0/215941652 >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 0x7f5118162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.271+0000 7f511d800640 1 --2- 172.21.15.43:0/215941652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118162220 0x7f51180b1ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f511806d8f0 tx=0x7f510802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04ff21ec005508a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.272+0000 7f5117e00640 1 -- 172.21.15.43:0/215941652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108031640 con 0x7f5118162220 2024-01-16T15:00:13.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.272+0000 7f5117e00640 1 -- 172.21.15.43:0/215941652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51080317a0 con 0x7f5118162220 2024-01-16T15:00:13.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.272+0000 7f5117e00640 1 -- 172.21.15.43:0/215941652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108031b30 con 0x7f5118162220 2024-01-16T15:00:13.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.272+0000 7f5120200640 1 -- 172.21.15.43:0/215941652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118162220 msgr2=0x7f51180b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.272+0000 7f5120200640 1 --2- 172.21.15.43:0/215941652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118162220 0x7f51180b1ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f511806d8f0 tx=0x7f510802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 -- 172.21.15.43:0/215941652 shutdown_connections 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 --2- 172.21.15.43:0/215941652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f511806c3c0 0x7f511806c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 --2- 172.21.15.43:0/215941652 >> v2:172.21.15.69:3300/0 conn(0x7f5118162600 0x7f5118162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 --2- 172.21.15.43:0/215941652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118162220 0x7f51180b1ea0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 -- 172.21.15.43:0/215941652 >> 172.21.15.43:0/215941652 conn(0x7f51180b0ba0 msgr2=0x7f51180b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 -- 172.21.15.43:0/215941652 shutdown_connections 2024-01-16T15:00:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.273+0000 7f5120200640 1 -- 172.21.15.43:0/215941652 wait complete. 2024-01-16T15:00:13.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.274+0000 7f5120200640 1 Processor -- start 2024-01-16T15:00:13.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.275+0000 7f5120200640 1 -- start start 2024-01-16T15:00:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 0x7f5118099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 0x7f51180ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118099f40 con 0x7f511806c3c0 2024-01-16T15:00:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511809ad80 con 0x7f5118162220 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f5120200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511809a170 con 0x7f5118162600 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38328/0 (socket says 172.21.15.43:38328) 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 -- 172.21.15.43:0/4201150677 learned_addr learned my addr 172.21.15.43:0/4201150677 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511ce00640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 0x7f5118099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511e200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 0x7f51180ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 -- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 msgr2=0x7f51180ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 0x7f51180ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 -- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 msgr2=0x7f5118099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 0x7f5118099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511d800640 1 -- 172.21.15.43:0/4201150677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080094d0 con 0x7f511806c3c0 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511ce00640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 0x7f5118099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.276+0000 7f511e200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 0x7f51180ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f511d800640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f51080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510803ac60 con 0x7f511806c3c0 2024-01-16T15:00:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510803ade0 con 0x7f511806c3c0 2024-01-16T15:00:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108044630 con 0x7f511806c3c0 2024-01-16T15:00:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511809a450 con 0x7f511806c3c0 2024-01-16T15:00:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.277+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511809aa10 con 0x7f511806c3c0 2024-01-16T15:00:13.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.278+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511806d1a0 con 0x7f511806c3c0 2024-01-16T15:00:13.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.279+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f510802f050 con 0x7f511806c3c0 2024-01-16T15:00:13.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.280+0000 7f5115600640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 0x7f50f006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.280+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51080abd20 con 0x7f511806c3c0 2024-01-16T15:00:13.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.281+0000 7f511ce00640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 0x7f50f006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.281+0000 7f511ce00640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 0x7f50f006d7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5100006fd0 tx=0x7f5100008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.285+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5108076af0 con 0x7f511806c3c0 2024-01-16T15:00:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.478+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f511809d200 con 0x7f511806c3c0 2024-01-16T15:00:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.479+0000 7f5115600640 1 -- 172.21.15.43:0/4201150677 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f51080350a0 con 0x7f511806c3c0 2024-01-16T15:00:13.460 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 msgr2=0x7f50f006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 0x7f50f006d7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5100006fd0 tx=0x7f5100008040 comp rx=0 tx=0).stop 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 msgr2=0x7f511809c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f51080046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 shutdown_connections 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50f006b2f0 0x7f50f006d7b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5118162220 0x7f5118099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5118162600 0x7f51180ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 --2- 172.21.15.43:0/4201150677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511806c3c0 0x7f511809c420 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 >> 172.21.15.43:0/4201150677 conn(0x7f51180b0ba0 msgr2=0x7f51180b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 shutdown_connections 2024-01-16T15:00:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.482+0000 7f5120200640 1 -- 172.21.15.43:0/4201150677 wait complete. 2024-01-16T15:00:13.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:13.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.720+0000 7f8140400640 1 Processor -- start 2024-01-16T15:00:13.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.720+0000 7f8140400640 1 -- start start 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f8140400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813815b340 0x7f81380abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f8140400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f813806dc30 0x7f81380b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f8140400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380ac530 con 0x7f81380cbcb0 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f8140400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81380b2080 con 0x7f813806dc30 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f8140400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380b2360 con 0x7f813815b340 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f813e400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f813806dc30 0x7f81380b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f813e400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f813806dc30 0x7f81380b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38332/0 (socket says 172.21.15.43:38332) 2024-01-16T15:00:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f813e400640 1 -- 172.21.15.43:0/2207589180 learned_addr learned my addr 172.21.15.43:0/2207589180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:13.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.722+0000 7f813da00640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813815b340 0x7f81380abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.723+0000 7f813e400640 1 -- 172.21.15.43:0/2207589180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813815b340 msgr2=0x7f81380abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.723+0000 7f813e400640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813815b340 0x7f81380abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.723+0000 7f813e400640 1 -- 172.21.15.43:0/2207589180 >> v1:172.21.15.69:6789/0 conn(0x7f81380cbcb0 legacy=0x7f813806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.723+0000 7f813e400640 1 -- 172.21.15.43:0/2207589180 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380b2640 con 0x7f813806dc30 2024-01-16T15:00:13.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.724+0000 7f813e400640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813806dc30 0x7f81380b3120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8130009e50 tx=0x7f813002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ed94458b8dc840 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.725+0000 7f8137e00640 1 -- 172.21.15.43:0/2207589180 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813002cb00 con 0x7f813806dc30 2024-01-16T15:00:13.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.725+0000 7f8137e00640 1 -- 172.21.15.43:0/2207589180 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813002cc60 con 0x7f813806dc30 2024-01-16T15:00:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.725+0000 7f8137e00640 1 -- 172.21.15.43:0/2207589180 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300343f0 con 0x7f813806dc30 2024-01-16T15:00:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.725+0000 7f8140400640 1 -- 172.21.15.43:0/2207589180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813806dc30 msgr2=0x7f81380b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.725+0000 7f8140400640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813806dc30 0x7f81380b3120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8130009e50 tx=0x7f813002bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 -- 172.21.15.43:0/2207589180 shutdown_connections 2024-01-16T15:00:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813806dc30 0x7f81380b3120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 --2- 172.21.15.43:0/2207589180 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813815b340 0x7f81380abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 -- 172.21.15.43:0/2207589180 >> 172.21.15.43:0/2207589180 conn(0x7f81380b0d80 msgr2=0x7f81380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 -- 172.21.15.43:0/2207589180 shutdown_connections 2024-01-16T15:00:13.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.726+0000 7f8140400640 1 -- 172.21.15.43:0/2207589180 wait complete. 2024-01-16T15:00:13.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.728+0000 7f8140400640 1 Processor -- start 2024-01-16T15:00:13.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.728+0000 7f8140400640 1 -- start start 2024-01-16T15:00:13.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.729+0000 7f8140400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813806dc30 0x7f81381f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.729+0000 7f8140400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81380cbcb0 0x7f81381f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f8140400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f8140400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813814be60 con 0x7f813815b340 2024-01-16T15:00:13.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f8140400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381f6830 con 0x7f813806dc30 2024-01-16T15:00:13.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f8140400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8138152ec0 con 0x7f81380cbcb0 2024-01-16T15:00:13.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f813e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f813e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38340/0 (socket says 172.21.15.43:38340) 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f813e400640 1 -- 172.21.15.43:0/2827185115 learned_addr learned my addr 172.21.15.43:0/2827185115 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f813d000640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81380cbcb0 0x7f81381f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.730+0000 7f813da00640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813806dc30 0x7f81381f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 -- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81380cbcb0 msgr2=0x7f81381f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81380cbcb0 0x7f81381f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 -- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813806dc30 msgr2=0x7f81381f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813806dc30 0x7f81381f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 -- 172.21.15.43:0/2827185115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130009b30 con 0x7f813815b340 2024-01-16T15:00:13.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f813e400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8130009e50 tx=0x7f8130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130009510 con 0x7f813815b340 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813003e450 con 0x7f813815b340 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813003e800 con 0x7f813815b340 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.731+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81381531a0 con 0x7f813815b340 2024-01-16T15:00:13.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.732+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8138153760 con 0x7f813815b340 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.733+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8138074770 con 0x7f813815b340 2024-01-16T15:00:13.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.734+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f813003e960 con 0x7f813815b340 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.735+0000 7f8135600640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 0x7f811006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.735+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f81300ac970 con 0x7f813815b340 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.735+0000 7f813da00640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 0x7f811006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.736+0000 7f813da00640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 0x7f811006d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8128009ad0 tx=0x7f8128008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.739+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81300776c0 con 0x7f813815b340 2024-01-16T15:00:13.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.955+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f813806e050 con 0x7f813815b340 2024-01-16T15:00:13.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.956+0000 7f8135600640 1 -- 172.21.15.43:0/2827185115 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f813007ace0 con 0x7f813815b340 2024-01-16T15:00:13.937 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 msgr2=0x7f811006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 0x7f811006d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8128009ad0 tx=0x7f8128008040 comp rx=0 tx=0).stop 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 msgr2=0x7f813815c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8130009e50 tx=0x7f8130004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 shutdown_connections 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f811006b2c0 0x7f811006d780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813815b340 0x7f813815c1a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81380cbcb0 0x7f81381f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 --2- 172.21.15.43:0/2827185115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813806dc30 0x7f81381f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 >> 172.21.15.43:0/2827185115 conn(0x7f81380b0d80 msgr2=0x7f81380af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 shutdown_connections 2024-01-16T15:00:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:13.959+0000 7f8140400640 1 -- 172.21.15.43:0/2827185115 wait complete. 2024-01-16T15:00:13.959 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:13.959 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:14.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.250+0000 7fbcdd000640 1 Processor -- start 2024-01-16T15:00:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.250+0000 7fbcdd000640 1 -- start start 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.251+0000 7fbcdd000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcd815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.251+0000 7fbcdd000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcd80cf050 0x7fbcd80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.251+0000 7fbcdd000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80ce7c0 con 0x7fbcd806c130 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.251+0000 7fbcdd000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd80cfa10 con 0x7fbcd80cf050 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.251+0000 7fbcdd000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8140900 con 0x7fbcd813dfd0 2024-01-16T15:00:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbcd7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbcd806c130 0x7fbcd806c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49644/0 (socket says 172.21.15.43:49644) 2024-01-16T15:00:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbcd7e00640 1 -- 172.21.15.43:0/3526672242 learned_addr learned my addr 172.21.15.43:0/3526672242 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:14.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbccfe00640 1 --2- 172.21.15.43:0/3526672242 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80cf050 0x7fbcd80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbcd7400640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcd815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741147278 0 0) 0x7fbcd80ce7c0 con 0x7fbcd806c130 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.252+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb8003520 con 0x7fbcd806c130 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286825554 0 0) 0x7fbcb8003520 con 0x7fbcd806c130 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 msgr2=0x7fbcd815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcd815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80cf050 msgr2=0x7fbcd80cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 --2- 172.21.15.43:0/3526672242 >> v2:172.21.15.69:3300/0 conn(0x7fbcd80cf050 0x7fbcd80cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd815fa70 con 0x7fbcd806c130 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fbcd00019f0 con 0x7fbcd806c130 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 >> v1:172.21.15.43:6789/0 conn(0x7fbcd806c130 legacy=0x7fbcd806c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcb80039e0 0x7fbcb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd80cfa10 con 0x7fbcb8004d00 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8140900 con 0x7fbcb80039e0 2024-01-16T15:00:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb800b980 con 0x7fbcd813dfd0 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.253+0000 7fbcd7e00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbccfe00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7400640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcb80039e0 0x7fbcb8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7e00640 1 -- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 msgr2=0x7fbcb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7e00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813dfd0 0x7fbcb80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7e00640 1 -- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcb80039e0 msgr2=0x7fbcb8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7e00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcb80039e0 0x7fbcb8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.254+0000 7fbcd7e00640 1 -- 172.21.15.43:0/3526672242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb8003520 con 0x7fbcb8004d00 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.255+0000 7fbcd7e00640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcb8005180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000a840 tx=0x7fbcb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786418d085a3a181 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.255+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0034630 con 0x7fbcb8004d00 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.255+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd002f070 con 0x7fbcb8004d00 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.255+0000 7fbcd6a00640 1 -- 172.21.15.43:0/3526672242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0034a70 con 0x7fbcb8004d00 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 -- 172.21.15.43:0/3526672242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 msgr2=0x7fbcb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 --2- 172.21.15.43:0/3526672242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcb8005180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000a840 tx=0x7fbcb80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcd7e00640 1 -- 172.21.15.43:0/3526672242 reap_dead start 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 -- 172.21.15.43:0/3526672242 shutdown_connections 2024-01-16T15:00:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 -- 172.21.15.43:0/3526672242 >> 172.21.15.43:0/3526672242 conn(0x7fbcd806a860 msgr2=0x7fbcd806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 -- 172.21.15.43:0/3526672242 shutdown_connections 2024-01-16T15:00:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.256+0000 7fbcdd000640 1 -- 172.21.15.43:0/3526672242 wait complete. 2024-01-16T15:00:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.258+0000 7fbcdd000640 1 Processor -- start 2024-01-16T15:00:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.258+0000 7fbcdd000640 1 -- start start 2024-01-16T15:00:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 0x7fbcd813f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd813fee0 0x7fbcd81e4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8145210 con 0x7fbcb8004d00 2024-01-16T15:00:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8140360 con 0x7fbcd813fee0 2024-01-16T15:00:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.259+0000 7fbcdd000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8158a00 con 0x7fbcd813b590 2024-01-16T15:00:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38360/0 (socket says 172.21.15.43:38360) 2024-01-16T15:00:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 -- 172.21.15.43:0/786801946 learned_addr learned my addr 172.21.15.43:0/786801946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 -- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 msgr2=0x7fbcd813f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbccfe00640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 0x7fbcd813f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7e00640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd813fee0 0x7fbcd81e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 0x7fbcd813f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 -- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd813fee0 msgr2=0x7fbcd81e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd813fee0 0x7fbcd81e4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 -- 172.21.15.43:0/786801946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8158720 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbccfe00640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 0x7fbcd813f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.260+0000 7fbcd7400640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011570 tx=0x7fbcc8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.261+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8015930 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.261+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc8015a90 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.261+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8015e60 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.261+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8144e30 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.261+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd81583d0 con 0x7fbcb8004d00 2024-01-16T15:00:14.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.262+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd804e840 con 0x7fbcb8004d00 2024-01-16T15:00:14.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.263+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbcc801d400 con 0x7fbcb8004d00 2024-01-16T15:00:14.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.264+0000 7fbccf400640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 0x7fbcb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.264+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbcc808a400 con 0x7fbcb8004d00 2024-01-16T15:00:14.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.264+0000 7fbccfe00640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 0x7fbcb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.265+0000 7fbccfe00640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 0x7fbcb006d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8158c00 tx=0x7fbcc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.268+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcc8055280 con 0x7fbcb8004d00 2024-01-16T15:00:14.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.463+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcd8144fe0 con 0x7fbcb8004d00 2024-01-16T15:00:14.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.465+0000 7fbccf400640 1 -- 172.21.15.43:0/786801946 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16651 (secure 0 0 0) 0x7fbcd8144fe0 con 0x7fbcb8004d00 2024-01-16T15:00:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:14.446 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":23,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:13.714101+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.468+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 msgr2=0x7fbcb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.468+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 0x7fbcb006d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8158c00 tx=0x7fbcc0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.468+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 msgr2=0x7fbcd8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.468+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011570 tx=0x7fbcc8011a30 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 shutdown_connections 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbcb006b2c0 0x7fbcb006d780 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd813fee0 0x7fbcd81e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd813b590 0x7fbcd813f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 --2- 172.21.15.43:0/786801946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcb8004d00 0x7fbcd8155be0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 >> 172.21.15.43:0/786801946 conn(0x7fbcd806a860 msgr2=0x7fbcd80cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 shutdown_connections 2024-01-16T15:00:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.469+0000 7fbcdd000640 1 -- 172.21.15.43:0/786801946 wait complete. 2024-01-16T15:00:14.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:14.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:14.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.721+0000 7fa31b000640 1 Processor -- start 2024-01-16T15:00:14.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.721+0000 7fa31b000640 1 -- start start 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa31b000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31415b350 0x7fa3140abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa31b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa31409cb30 0x7fa3140b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa31b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31415b730 con 0x7fa3140adb60 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa31b000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3140ac740 con 0x7fa31415b350 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa31b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3140ac460 con 0x7fa31409cb30 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31415b350 0x7fa3140abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31415b350 0x7fa3140abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38364/0 (socket says 172.21.15.43:38364) 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 -- 172.21.15.43:0/3197487500 learned_addr learned my addr 172.21.15.43:0/3197487500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 -- 172.21.15.43:0/3197487500 >> v2:172.21.15.69:3300/0 conn(0x7fa31409cb30 msgr2=0x7fa3140b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 --2- 172.21.15.43:0/3197487500 >> v2:172.21.15.69:3300/0 conn(0x7fa31409cb30 0x7fa3140b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 -- 172.21.15.43:0/3197487500 >> v1:172.21.15.69:6789/0 conn(0x7fa3140adb60 legacy=0x7fa3140adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.723+0000 7fa313e00640 1 -- 172.21.15.43:0/3197487500 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140b2a50 con 0x7fa31415b350 2024-01-16T15:00:14.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa313e00640 1 --2- 172.21.15.43:0/3197487500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31415b350 0x7fa3140abe80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa3140d0060 tx=0x7fa30402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b6f0a32405b283 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa312a00640 1 -- 172.21.15.43:0/3197487500 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304031870 con 0x7fa31415b350 2024-01-16T15:00:14.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa312a00640 1 -- 172.21.15.43:0/3197487500 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3040319d0 con 0x7fa31415b350 2024-01-16T15:00:14.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa312a00640 1 -- 172.21.15.43:0/3197487500 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304031d60 con 0x7fa31415b350 2024-01-16T15:00:14.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa31b000640 1 -- 172.21.15.43:0/3197487500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31415b350 msgr2=0x7fa3140abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.725+0000 7fa31b000640 1 --2- 172.21.15.43:0/3197487500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31415b350 0x7fa3140abe80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa3140d0060 tx=0x7fa30402b910 comp rx=0 tx=0).stop 2024-01-16T15:00:14.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 -- 172.21.15.43:0/3197487500 shutdown_connections 2024-01-16T15:00:14.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 --2- 172.21.15.43:0/3197487500 >> v2:172.21.15.69:3300/0 conn(0x7fa31409cb30 0x7fa3140b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 --2- 172.21.15.43:0/3197487500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31415b350 0x7fa3140abe80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 -- 172.21.15.43:0/3197487500 >> 172.21.15.43:0/3197487500 conn(0x7fa3140b0ba0 msgr2=0x7fa3140b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 -- 172.21.15.43:0/3197487500 shutdown_connections 2024-01-16T15:00:14.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.726+0000 7fa31b000640 1 -- 172.21.15.43:0/3197487500 wait complete. 2024-01-16T15:00:14.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.728+0000 7fa31b000640 1 Processor -- start 2024-01-16T15:00:14.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.728+0000 7fa31b000640 1 -- start start 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3140adb60 0x7fa3140c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 0x7fa3140c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3141f61d0 con 0x7fa31409cb30 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa314146050 con 0x7fa31415b350 2024-01-16T15:00:14.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.729+0000 7fa31b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa314145ba0 con 0x7fa3140adb60 2024-01-16T15:00:14.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38374/0 (socket says 172.21.15.43:38374) 2024-01-16T15:00:14.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313e00640 1 -- 172.21.15.43:0/2830408100 learned_addr learned my addr 172.21.15.43:0/2830408100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:14.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313400640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3140adb60 0x7fa3140c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313e00640 1 -- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3140adb60 msgr2=0x7fa3140c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa313e00640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3140adb60 0x7fa3140c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.731+0000 7fa313e00640 1 -- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 msgr2=0x7fa3140c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.730+0000 7fa319000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 0x7fa3140c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.731+0000 7fa313e00640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 0x7fa3140c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.731+0000 7fa313e00640 1 -- 172.21.15.43:0/2830408100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040094d0 con 0x7fa31409cb30 2024-01-16T15:00:14.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.731+0000 7fa319000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 0x7fa3140c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.731+0000 7fa313e00640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa304000c00 tx=0x7fa3040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.732+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30403ad10 con 0x7fa31409cb30 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.732+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30403ae70 con 0x7fa31409cb30 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.732+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304044610 con 0x7fa31409cb30 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.732+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31409b140 con 0x7fa31409cb30 2024-01-16T15:00:14.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.732+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141465f0 con 0x7fa31409cb30 2024-01-16T15:00:14.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.734+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3141467e0 con 0x7fa31409cb30 2024-01-16T15:00:14.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.735+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa304044770 con 0x7fa31409cb30 2024-01-16T15:00:14.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.736+0000 7fa30be00640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 0x7fa2ec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.736+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa30402d040 con 0x7fa31409cb30 2024-01-16T15:00:14.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.737+0000 7fa313400640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 0x7fa2ec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.737+0000 7fa313400640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 0x7fa2ec06d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa31414c5e0 tx=0x7fa2fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.741+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3040770a0 con 0x7fa31409cb30 2024-01-16T15:00:14.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.937+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa31415b730 con 0x7fa31409cb30 2024-01-16T15:00:14.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.938+0000 7fa30be00640 1 -- 172.21.15.43:0/2830408100 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa31415b730 con 0x7fa31409cb30 2024-01-16T15:00:14.919 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 msgr2=0x7fa2ec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 0x7fa2ec06d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa31414c5e0 tx=0x7fa2fc008040 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 msgr2=0x7fa314145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa304000c00 tx=0x7fa3040046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 shutdown_connections 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa2ec06b2c0 0x7fa2ec06d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31415b350 0x7fa3140c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3140adb60 0x7fa3140c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 --2- 172.21.15.43:0/2830408100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31409cb30 0x7fa314145660 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 >> 172.21.15.43:0/2830408100 conn(0x7fa3140b0ba0 msgr2=0x7fa314151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 shutdown_connections 2024-01-16T15:00:14.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:14.941+0000 7fa31b000640 1 -- 172.21.15.43:0/2830408100 wait complete. 2024-01-16T15:00:14.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.184+0000 7f733be00640 1 Processor -- start 2024-01-16T15:00:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.184+0000 7f733be00640 1 -- start start 2024-01-16T15:00:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.185+0000 7f733be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 0x7f733406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f733be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733406d7a0 con 0x7f73340ae490 2024-01-16T15:00:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f733be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733406ce70 con 0x7f7334070490 2024-01-16T15:00:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f733be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733415a5b0 con 0x7f73340acd70 2024-01-16T15:00:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f7338a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f73340ae490 0x7f73340ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49648/0 (socket says 172.21.15.43:49648) 2024-01-16T15:00:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f7338a00640 1 -- 172.21.15.43:0/3343668951 learned_addr learned my addr 172.21.15.43:0/3343668951 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f7339400640 1 --2- 172.21.15.43:0/3343668951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 0x7f733406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.186+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009561262 0 0) 0x7f733406d7a0 con 0x7f73340ae490 2024-01-16T15:00:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.187+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7314003520 con 0x7f73340ae490 2024-01-16T15:00:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.187+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679144248 0 0) 0x7f7314003520 con 0x7f73340ae490 2024-01-16T15:00:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.187+0000 7f7339400640 1 -- 172.21.15.43:0/3343668951 >> v1:172.21.15.69:6789/0 conn(0x7f7334070490 legacy=0x7f7334070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.187+0000 7f7339400640 1 -- 172.21.15.43:0/3343668951 >> v1:172.21.15.43:6789/0 conn(0x7f73340ae490 legacy=0x7f73340ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.187+0000 7f7339400640 1 -- 172.21.15.43:0/3343668951 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733406d150 con 0x7f73340acd70 2024-01-16T15:00:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.188+0000 7f7339400640 1 --2- 172.21.15.43:0/3343668951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 0x7f733406dc30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f73240097d0 tx=0x7f732402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7a7ac57c3bf519 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.189+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c001670 con 0x7f73340acd70 2024-01-16T15:00:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.189+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732402e050 con 0x7f73340acd70 2024-01-16T15:00:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.189+0000 7f7333e00640 1 -- 172.21.15.43:0/3343668951 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324030730 con 0x7f73340acd70 2024-01-16T15:00:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.189+0000 7f733be00640 1 -- 172.21.15.43:0/3343668951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 msgr2=0x7f733406dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.189+0000 7f733be00640 1 --2- 172.21.15.43:0/3343668951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 0x7f733406dc30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f73240097d0 tx=0x7f732402b800 comp rx=0 tx=0).stop 2024-01-16T15:00:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.190+0000 7f733be00640 1 -- 172.21.15.43:0/3343668951 shutdown_connections 2024-01-16T15:00:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.190+0000 7f733be00640 1 --2- 172.21.15.43:0/3343668951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340acd70 0x7f733406dc30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.190+0000 7f733be00640 1 -- 172.21.15.43:0/3343668951 >> 172.21.15.43:0/3343668951 conn(0x7f73340b0ba0 msgr2=0x7f73340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.190+0000 7f733be00640 1 -- 172.21.15.43:0/3343668951 shutdown_connections 2024-01-16T15:00:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.190+0000 7f733be00640 1 -- 172.21.15.43:0/3343668951 wait complete. 2024-01-16T15:00:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.191+0000 7f733be00640 1 Processor -- start 2024-01-16T15:00:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.192+0000 7f733be00640 1 -- start start 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.192+0000 7f733be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f733be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 0x7f733414edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f733be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340ae490 0x7f733414b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f733be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733414e4a0 con 0x7f7334070490 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f733be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733414b990 con 0x7f73340acd70 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f733be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733414b740 con 0x7f73340ae490 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f7339400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f7339400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38376/0 (socket says 172.21.15.43:38376) 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f7339400640 1 -- 172.21.15.43:0/3221766290 learned_addr learned my addr 172.21.15.43:0/3221766290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f7338a00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 0x7f733414edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.193+0000 7f7339e00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340ae490 0x7f733414b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 -- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340ae490 msgr2=0x7f733414b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340ae490 0x7f733414b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 -- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 msgr2=0x7f733414edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 0x7f733414edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 -- 172.21.15.43:0/3221766290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324009400 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7338a00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 0x7f733414edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7339400640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7324000c00 tx=0x7f73240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324004230 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324004390 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324003850 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.194+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733414e6f0 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.195+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733414bc30 con 0x7f7334070490 2024-01-16T15:00:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.196+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73340ae890 con 0x7f7334070490 2024-01-16T15:00:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.197+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7324043030 con 0x7f7334070490 2024-01-16T15:00:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.198+0000 7f7331600640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 0x7f730c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.198+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7324041070 con 0x7f7334070490 2024-01-16T15:00:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.199+0000 7f7338a00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 0x7f730c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.199+0000 7f7338a00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 0x7f730c06d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f731c0017a0 tx=0x7f731c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.203+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73240763a0 con 0x7f7334070490 2024-01-16T15:00:15.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.398+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f733409d200 con 0x7f7334070490 2024-01-16T15:00:15.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.399+0000 7f7331600640 1 -- 172.21.15.43:0/3221766290 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73240799c0 con 0x7f7334070490 2024-01-16T15:00:15.380 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:15.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.401+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 msgr2=0x7f730c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.401+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 0x7f730c06d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f731c0017a0 tx=0x7f731c009040 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.401+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 msgr2=0x7f7334142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.401+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7324000c00 tx=0x7f73240046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 shutdown_connections 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f730c06b2c0 0x7f730c06d780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73340acd70 0x7f733414edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73340ae490 0x7f733414b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 --2- 172.21.15.43:0/3221766290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7334070490 0x7f7334142da0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 >> 172.21.15.43:0/3221766290 conn(0x7f73340b0ba0 msgr2=0x7f73340b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 shutdown_connections 2024-01-16T15:00:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.402+0000 7f733be00640 1 -- 172.21.15.43:0/3221766290 wait complete. 2024-01-16T15:00:15.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:15.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.642+0000 7f8004a00640 1 Processor -- start 2024-01-16T15:00:15.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.643+0000 7f8004a00640 1 -- start start 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.644+0000 7f8004a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f80000ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.644+0000 7f8004a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80000d1d90 0x7f80000b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.644+0000 7f8004a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80000c6bf0 con 0x7f80000d0fa0 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.644+0000 7f8004a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000d1400 con 0x7f80000d1d90 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.644+0000 7f8004a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80000cedc0 con 0x7f800015d3b0 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffde00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f80000d0fa0 0x7f80000d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49654/0 (socket says 172.21.15.43:49654) 2024-01-16T15:00:15.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffde00640 1 -- 172.21.15.43:0/167352131 learned_addr learned my addr 172.21.15.43:0/167352131 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7fff200640 1 --2- 172.21.15.43:0/167352131 >> v2:172.21.15.69:3300/0 conn(0x7f80000d1d90 0x7f80000b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffe800640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f80000ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705376154 0 0) 0x7f80000c6bf0 con 0x7f80000d0fa0 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f80000d0fa0 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885057526 0 0) 0x7f7fe0003520 con 0x7f80000d0fa0 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 msgr2=0x7f80000ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f80000ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 >> v2:172.21.15.69:3300/0 conn(0x7f80000d1d90 msgr2=0x7f80000b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.645+0000 7f7ffd400640 1 --2- 172.21.15.43:0/167352131 >> v2:172.21.15.69:3300/0 conn(0x7f80000d1d90 0x7f80000b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000c6e40 con 0x7f80000d0fa0 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7fe80019f0 con 0x7f80000d0fa0 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 >> v1:172.21.15.43:6789/0 conn(0x7f80000d0fa0 legacy=0x7f80000d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f7fe00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fe0003fb0 0x7f7fe00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0004d00 0x7f7fe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80000d1400 con 0x7f7fe0003fb0 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80000cedc0 con 0x7f7fe0004d00 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe000ba80 con 0x7f800015d3b0 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffde00640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fe0003fb0 0x7f7fe00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffe800640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f7fe00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7fff200640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0004d00 0x7f7fe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffde00640 1 -- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 msgr2=0x7f7fe00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffde00640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f800015d3b0 0x7f7fe00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffde00640 1 -- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0004d00 msgr2=0x7f7fe0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.646+0000 7f7ffde00640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0004d00 0x7f7fe0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.647+0000 7f7ffde00640 1 -- 172.21.15.43:0/167352131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0003520 con 0x7f7fe0003fb0 2024-01-16T15:00:15.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.647+0000 7f7fff200640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0004d00 0x7f7fe0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f7ffde00640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fe0003fb0 0x7f7fe00047c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800a780 tx=0x7f7fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641fff2ed0d63de3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8034630 con 0x7f7fe0003fb0 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe802f070 con 0x7f7fe0003fb0 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f7ffd400640 1 -- 172.21.15.43:0/167352131 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8034a10 con 0x7f7fe0003fb0 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f8004a00640 1 -- 172.21.15.43:0/167352131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fe0003fb0 msgr2=0x7f7fe00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f8004a00640 1 --2- 172.21.15.43:0/167352131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fe0003fb0 0x7f7fe00047c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800a780 tx=0x7f7fe00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.648+0000 7f7fff200640 1 -- 172.21.15.43:0/167352131 reap_dead start 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.649+0000 7f8004a00640 1 -- 172.21.15.43:0/167352131 shutdown_connections 2024-01-16T15:00:15.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.649+0000 7f8004a00640 1 -- 172.21.15.43:0/167352131 >> 172.21.15.43:0/167352131 conn(0x7f80000d30a0 msgr2=0x7f80000d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.649+0000 7f8004a00640 1 -- 172.21.15.43:0/167352131 shutdown_connections 2024-01-16T15:00:15.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.649+0000 7f8004a00640 1 -- 172.21.15.43:0/167352131 wait complete. 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.651+0000 7f8004a00640 1 Processor -- start 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.651+0000 7f8004a00640 1 -- start start 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0003fb0 0x7f800015aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80000cb760 0x7f800009cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80000a0700 con 0x7f80000cb2e0 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800015afd0 con 0x7f7fe0003fb0 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f8004a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80000b6e50 con 0x7f80000cb760 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7ffde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7ffde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38402/0 (socket says 172.21.15.43:38402) 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7ffde00640 1 -- 172.21.15.43:0/2236413571 learned_addr learned my addr 172.21.15.43:0/2236413571 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7fff200640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80000cb760 0x7f800009cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7ffe800640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0003fb0 0x7f800015aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.652+0000 7f7ffde00640 1 -- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80000cb760 msgr2=0x7f800009cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ffde00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80000cb760 0x7f800009cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ffde00640 1 -- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0003fb0 msgr2=0x7f800015aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ffde00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0003fb0 0x7f800015aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ffde00640 1 -- 172.21.15.43:0/2236413571 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000b7130 con 0x7f80000cb2e0 2024-01-16T15:00:15.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ffde00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7fe802cad0 tx=0x7f7fe8001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8034840 con 0x7f80000cb2e0 2024-01-16T15:00:15.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe80349a0 con 0x7f80000cb2e0 2024-01-16T15:00:15.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8034d30 con 0x7f80000cb2e0 2024-01-16T15:00:15.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80000b7410 con 0x7f80000cb2e0 2024-01-16T15:00:15.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.653+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80000cae90 con 0x7f80000cb2e0 2024-01-16T15:00:15.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.655+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800006cd50 con 0x7f80000cb2e0 2024-01-16T15:00:15.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.656+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7fe8031030 con 0x7f80000cb2e0 2024-01-16T15:00:15.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.657+0000 7f7ff6a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 0x7f7fd806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.657+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7fe80ac340 con 0x7f80000cb2e0 2024-01-16T15:00:15.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.657+0000 7f7ffe800640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 0x7f7fd806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.658+0000 7f7ffe800640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 0x7f7fd806d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009870 tx=0x7f7ff0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.661+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fe80771b0 con 0x7f80000cb2e0 2024-01-16T15:00:15.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.865+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f800006cd50 con 0x7f80000cb2e0 2024-01-16T15:00:15.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.866+0000 7f7ff6a00640 1 -- 172.21.15.43:0/2236413571 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7fe807a7d0 con 0x7f80000cb2e0 2024-01-16T15:00:15.847 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:15.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 msgr2=0x7f7fd806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 0x7f7fd806d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009870 tx=0x7f7ff0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 msgr2=0x7f800009fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7fe802cad0 tx=0x7f7fe8001360 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 shutdown_connections 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fd806b2c0 0x7f7fd806d780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80000cb760 0x7f800009cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80000cb2e0 0x7f800009fc80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 --2- 172.21.15.43:0/2236413571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fe0003fb0 0x7f800015aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.868+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 >> 172.21.15.43:0/2236413571 conn(0x7f80000d30a0 msgr2=0x7f800006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.869+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 shutdown_connections 2024-01-16T15:00:15.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:15.869+0000 7f8004a00640 1 -- 172.21.15.43:0/2236413571 wait complete. 2024-01-16T15:00:15.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:16.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.099+0000 7f1361c00640 1 Processor -- start 2024-01-16T15:00:16.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.099+0000 7f1361c00640 1 -- start start 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f1361c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f135c1416f0 0x7f135c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f1361c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 0x7f135c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f1361c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c070bf0 con 0x7f135c06cb20 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f1361c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c06db30 con 0x7f135c1416f0 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f1361c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c06d850 con 0x7f135c141e80 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.100+0000 7f135be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f135c1416f0 0x7f135c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f135c1416f0 0x7f135c15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38410/0 (socket says 172.21.15.43:38410) 2024-01-16T15:00:16.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 -- 172.21.15.43:0/4291382296 learned_addr learned my addr 172.21.15.43:0/4291382296 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:16.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f1353e00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 0x7f135c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 -- 172.21.15.43:0/4291382296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 msgr2=0x7f135c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 0x7f135c15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 -- 172.21.15.43:0/4291382296 >> v1:172.21.15.69:6789/0 conn(0x7f135c06cb20 legacy=0x7f135c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f135be00640 1 -- 172.21.15.43:0/4291382296 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c0705a0 con 0x7f135c1416f0 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.101+0000 7f1353e00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 0x7f135c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.102+0000 7f135be00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c15ed50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f134c009870 tx=0x7f134c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b38b0e32100a8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.102+0000 7f135b400640 1 -- 172.21.15.43:0/4291382296 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031640 con 0x7f135c1416f0 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.102+0000 7f135b400640 1 -- 172.21.15.43:0/4291382296 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c0317a0 con 0x7f135c1416f0 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.102+0000 7f135b400640 1 -- 172.21.15.43:0/4291382296 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031b30 con 0x7f135c1416f0 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 -- 172.21.15.43:0/4291382296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 msgr2=0x7f135c15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c15ed50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f134c009870 tx=0x7f134c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 -- 172.21.15.43:0/4291382296 shutdown_connections 2024-01-16T15:00:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c141e80 0x7f135c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 --2- 172.21.15.43:0/4291382296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c15ed50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 -- 172.21.15.43:0/4291382296 >> 172.21.15.43:0/4291382296 conn(0x7f135c0b0840 msgr2=0x7f135c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 -- 172.21.15.43:0/4291382296 shutdown_connections 2024-01-16T15:00:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.103+0000 7f1361c00640 1 -- 172.21.15.43:0/4291382296 wait complete. 2024-01-16T15:00:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.105+0000 7f1361c00640 1 Processor -- start 2024-01-16T15:00:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.105+0000 7f1361c00640 1 -- start start 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c06cb20 0x7f135c0ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f135c141e80 0x7f135c1f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c15b940 con 0x7f135c1416f0 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b3c60 con 0x7f135c141e80 2024-01-16T15:00:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.106+0000 7f1361c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0ae1b0 con 0x7f135c06cb20 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38414/0 (socket says 172.21.15.43:38414) 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 -- 172.21.15.43:0/256587589 learned_addr learned my addr 172.21.15.43:0/256587589 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f135be00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c06cb20 0x7f135c0ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1361200640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f135c141e80 0x7f135c1f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 -- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c06cb20 msgr2=0x7f135c0ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c06cb20 0x7f135c0ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 -- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f135c141e80 msgr2=0x7f135c1f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f135c141e80 0x7f135c1f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 -- 172.21.15.43:0/256587589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0094d0 con 0x7f135c1416f0 2024-01-16T15:00:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.107+0000 7f1353e00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f134400c820 tx=0x7f134400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.108+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440079d0 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.108+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1344007b30 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.108+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344007ea0 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.108+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c0ad910 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.108+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c0aded0 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.109+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c0cca10 con 0x7f135c1416f0 2024-01-16T15:00:16.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.110+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f13440113f0 con 0x7f135c1416f0 2024-01-16T15:00:16.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.111+0000 7f1358c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 0x7f133406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.111+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1344089c30 con 0x7f135c1416f0 2024-01-16T15:00:16.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.111+0000 7f135be00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 0x7f133406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.112+0000 7f135be00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 0x7f133406d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f134c036000 tx=0x7f134c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.116+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1344054ab0 con 0x7f135c1416f0 2024-01-16T15:00:16.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.323+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f135c04e840 con 0x7f135c1416f0 2024-01-16T15:00:16.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.324+0000 7f1358c00640 1 -- 172.21.15.43:0/256587589 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f13440580d0 con 0x7f135c1416f0 2024-01-16T15:00:16.305 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 msgr2=0x7f133406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 0x7f133406d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f134c036000 tx=0x7f134c03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 msgr2=0x7f135c0b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f134400c820 tx=0x7f134400cce0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 shutdown_connections 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133406b2c0 0x7f133406d780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f135c141e80 0x7f135c1f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f135c1416f0 0x7f135c0b3720 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 --2- 172.21.15.43:0/256587589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f135c06cb20 0x7f135c0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 >> 172.21.15.43:0/256587589 conn(0x7f135c0b0840 msgr2=0x7f135c1444e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:16.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 shutdown_connections 2024-01-16T15:00:16.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.327+0000 7f1361c00640 1 -- 172.21.15.43:0/256587589 wait complete. 2024-01-16T15:00:16.326 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:00:16.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:16.423 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:16.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:00:16.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.424 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:00:16.484 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:16.484 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:16.542 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:16.542 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:16.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:16.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.844+0000 7f5b96400640 1 Processor -- start 2024-01-16T15:00:16.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.844+0000 7f5b96400640 1 -- start start 2024-01-16T15:00:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.846+0000 7f5b96400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b9015d5b0 0x7f5b900d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.846+0000 7f5b96400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.846+0000 7f5b96400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900cf0c0 con 0x7f5b900d14a0 2024-01-16T15:00:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.846+0000 7f5b96400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b900b01b0 con 0x7f5b9015d5b0 2024-01-16T15:00:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.846+0000 7f5b96400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9015d990 con 0x7f5b900b1650 2024-01-16T15:00:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.847+0000 7f5b95a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.847+0000 7f5b95a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:60936/0 (socket says 172.21.15.43:60936) 2024-01-16T15:00:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.847+0000 7f5b95a00640 1 -- 172.21.15.43:0/2099141293 learned_addr learned my addr 172.21.15.43:0/2099141293 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.847+0000 7f5b95000640 1 --2- 172.21.15.43:0/2099141293 >> v2:172.21.15.69:3300/0 conn(0x7f5b9015d5b0 0x7f5b900d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.848+0000 7f5b95a00640 1 -- 172.21.15.43:0/2099141293 >> v2:172.21.15.69:3300/0 conn(0x7f5b9015d5b0 msgr2=0x7f5b900d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.848+0000 7f5b95a00640 1 --2- 172.21.15.43:0/2099141293 >> v2:172.21.15.69:3300/0 conn(0x7f5b9015d5b0 0x7f5b900d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.848+0000 7f5b95a00640 1 -- 172.21.15.43:0/2099141293 >> v1:172.21.15.69:6789/0 conn(0x7f5b900d14a0 legacy=0x7f5b900cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.848+0000 7f5b95a00640 1 -- 172.21.15.43:0/2099141293 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900d1e40 con 0x7f5b900b1650 2024-01-16T15:00:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.849+0000 7f5b95a00640 1 --2- 172.21.15.43:0/2099141293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009e50 tx=0x7f5b8802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d35504db9a6b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b8f400640 1 -- 172.21.15.43:0/2099141293 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8802cb00 con 0x7f5b900b1650 2024-01-16T15:00:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b8f400640 1 -- 172.21.15.43:0/2099141293 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8802cc60 con 0x7f5b900b1650 2024-01-16T15:00:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b8f400640 1 -- 172.21.15.43:0/2099141293 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880343f0 con 0x7f5b900b1650 2024-01-16T15:00:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 -- 172.21.15.43:0/2099141293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 msgr2=0x7f5b900d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 --2- 172.21.15.43:0/2099141293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009e50 tx=0x7f5b8802bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 -- 172.21.15.43:0/2099141293 shutdown_connections 2024-01-16T15:00:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 --2- 172.21.15.43:0/2099141293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b900b1650 0x7f5b900d1900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 --2- 172.21.15.43:0/2099141293 >> v2:172.21.15.69:3300/0 conn(0x7f5b9015d5b0 0x7f5b900d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.850+0000 7f5b96400640 1 -- 172.21.15.43:0/2099141293 >> 172.21.15.43:0/2099141293 conn(0x7f5b9009c3b0 msgr2=0x7f5b900d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.851+0000 7f5b96400640 1 -- 172.21.15.43:0/2099141293 shutdown_connections 2024-01-16T15:00:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.851+0000 7f5b96400640 1 -- 172.21.15.43:0/2099141293 wait complete. 2024-01-16T15:00:16.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.852+0000 7f5b96400640 1 Processor -- start 2024-01-16T15:00:16.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.853+0000 7f5b96400640 1 -- start start 2024-01-16T15:00:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.853+0000 7f5b96400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 0x7f5b901f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.853+0000 7f5b96400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b96400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9015d5b0 0x7f5b901f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b96400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b901f3f80 con 0x7f5b900d14a0 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b96400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b901f0950 con 0x7f5b900b1650 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b96400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b901f39c0 con 0x7f5b9015d5b0 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38428/0 (socket says 172.21.15.43:38428) 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 -- 172.21.15.43:0/3213777389 learned_addr learned my addr 172.21.15.43:0/3213777389 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b95a00640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9015d5b0 0x7f5b901f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b95000640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 0x7f5b901f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 -- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9015d5b0 msgr2=0x7f5b901f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9015d5b0 0x7f5b901f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 -- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 msgr2=0x7f5b901f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 0x7f5b901f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b8fe00640 1 -- 172.21.15.43:0/3213777389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88009b30 con 0x7f5b900d14a0 2024-01-16T15:00:16.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.854+0000 7f5b95000640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 0x7f5b901f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:16.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b8fe00640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c3c0 tx=0x7f5b7800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780052d0 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78005450 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78007260 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b901f3ca0 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.855+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b900b7160 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.857+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b900b6ce0 con 0x7f5b900d14a0 2024-01-16T15:00:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.857+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5b780073c0 con 0x7f5b900d14a0 2024-01-16T15:00:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.858+0000 7f5b8cc00640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 0x7f5b6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.858+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5b78089090 con 0x7f5b900d14a0 2024-01-16T15:00:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.858+0000 7f5b95000640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 0x7f5b6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.859+0000 7f5b95000640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 0x7f5b6806d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009ac0 tx=0x7f5b80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:16.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:16.862+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b900b6ce0 con 0x7f5b900d14a0 2024-01-16T15:00:17.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.056+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b900b6ce0 con 0x7f5b900d14a0 2024-01-16T15:00:17.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.059+0000 7f5b8cc00640 1 -- 172.21.15.43:0/3213777389 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f5b900b6ce0 con 0x7f5b900d14a0 2024-01-16T15:00:17.040 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:17.040 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 msgr2=0x7f5b6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 0x7f5b6806d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009ac0 tx=0x7f5b80008040 comp rx=0 tx=0).stop 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 msgr2=0x7f5b900b7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c3c0 tx=0x7f5b7800c880 comp rx=0 tx=0).stop 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 shutdown_connections 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5b6806b2c0 0x7f5b6806d780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b9015d5b0 0x7f5b901f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b900d14a0 0x7f5b900b7680 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 --2- 172.21.15.43:0/3213777389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b900b1650 0x7f5b901f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 >> 172.21.15.43:0/3213777389 conn(0x7f5b9009c3b0 msgr2=0x7f5b9006bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.061+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 shutdown_connections 2024-01-16T15:00:17.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.062+0000 7f5b96400640 1 -- 172.21.15.43:0/3213777389 wait complete. 2024-01-16T15:00:17.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:17.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:17.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.366+0000 7ff463200640 1 Processor -- start 2024-01-16T15:00:17.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.366+0000 7ff463200640 1 -- start start 2024-01-16T15:00:17.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.368+0000 7ff463200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.368+0000 7ff463200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff4640b35a0 0x7ff46415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.368+0000 7ff463200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46415f930 con 0x7ff46406c9d0 2024-01-16T15:00:17.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.368+0000 7ff463200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4640b43b0 con 0x7ff4640b35a0 2024-01-16T15:00:17.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.368+0000 7ff463200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640b3b10 con 0x7ff464144770 2024-01-16T15:00:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.369+0000 7ff461e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.369+0000 7ff461e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:60952/0 (socket says 172.21.15.43:60952) 2024-01-16T15:00:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.369+0000 7ff461e00640 1 -- 172.21.15.43:0/3402483054 learned_addr learned my addr 172.21.15.43:0/3402483054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.369+0000 7ff462800640 1 --1- 172.21.15.43:0/3402483054 >> v1:172.21.15.69:6789/0 conn(0x7ff46406c9d0 0x7ff46406ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:37234/0 (socket says 172.21.15.43:37234) 2024-01-16T15:00:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.369+0000 7ff45be00640 1 --2- 172.21.15.43:0/3402483054 >> v2:172.21.15.69:3300/0 conn(0x7ff4640b35a0 0x7ff46415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.370+0000 7ff461e00640 1 -- 172.21.15.43:0/3402483054 >> v2:172.21.15.69:3300/0 conn(0x7ff4640b35a0 msgr2=0x7ff46415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.370+0000 7ff461e00640 1 --2- 172.21.15.43:0/3402483054 >> v2:172.21.15.69:3300/0 conn(0x7ff4640b35a0 0x7ff46415f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.370+0000 7ff461e00640 1 -- 172.21.15.43:0/3402483054 >> v1:172.21.15.69:6789/0 conn(0x7ff46406c9d0 legacy=0x7ff46406ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.370+0000 7ff461e00640 1 -- 172.21.15.43:0/3402483054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640b3df0 con 0x7ff464144770 2024-01-16T15:00:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.371+0000 7ff461e00640 1 --2- 172.21.15.43:0/3402483054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4540098a0 tx=0x7ff45402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3fbcc75e06f55c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff461400640 1 -- 172.21.15.43:0/3402483054 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454030840 con 0x7ff464144770 2024-01-16T15:00:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff461400640 1 -- 172.21.15.43:0/3402483054 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4540309a0 con 0x7ff464144770 2024-01-16T15:00:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff461400640 1 -- 172.21.15.43:0/3402483054 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454030d30 con 0x7ff464144770 2024-01-16T15:00:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 -- 172.21.15.43:0/3402483054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 msgr2=0x7ff46415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 --2- 172.21.15.43:0/3402483054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4540098a0 tx=0x7ff45402b980 comp rx=0 tx=0).stop 2024-01-16T15:00:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 -- 172.21.15.43:0/3402483054 shutdown_connections 2024-01-16T15:00:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 --2- 172.21.15.43:0/3402483054 >> v2:172.21.15.69:3300/0 conn(0x7ff4640b35a0 0x7ff46415f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 --2- 172.21.15.43:0/3402483054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff46415eb60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.372+0000 7ff463200640 1 -- 172.21.15.43:0/3402483054 >> 172.21.15.43:0/3402483054 conn(0x7ff4640cec10 msgr2=0x7ff4640cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.373+0000 7ff463200640 1 -- 172.21.15.43:0/3402483054 shutdown_connections 2024-01-16T15:00:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.373+0000 7ff463200640 1 -- 172.21.15.43:0/3402483054 wait complete. 2024-01-16T15:00:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.374+0000 7ff463200640 1 Processor -- start 2024-01-16T15:00:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.374+0000 7ff463200640 1 -- start start 2024-01-16T15:00:17.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 0x7ff46413f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff4641f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464142bf0 con 0x7ff46406c9d0 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4641420d0 con 0x7ff4640b35a0 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.375+0000 7ff463200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff464142350 con 0x7ff464144770 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38442/0 (socket says 172.21.15.43:38442) 2024-01-16T15:00:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 -- 172.21.15.43:0/345446318 learned_addr learned my addr 172.21.15.43:0/345446318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff462800640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff4641f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff45be00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 0x7ff46413f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 -- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 msgr2=0x7ff4641f2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff4641f2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 -- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 msgr2=0x7ff46413f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 0x7ff46413f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff461e00640 1 -- 172.21.15.43:0/345446318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4540094d0 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.376+0000 7ff45be00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 0x7ff46413f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff461e00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff4540099c0 tx=0x7ff4540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45403e070 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454039a10 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454039da0 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff464142630 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.377+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4640702d0 con 0x7ff46406c9d0 2024-01-16T15:00:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.378+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff464070130 con 0x7ff46406c9d0 2024-01-16T15:00:17.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.381+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff454055430 con 0x7ff46406c9d0 2024-01-16T15:00:17.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.382+0000 7ff45a000640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 0x7ff43c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.382+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff4540acb10 con 0x7ff46406c9d0 2024-01-16T15:00:17.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.382+0000 7ff45be00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 0x7ff43c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.383+0000 7ff45be00640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 0x7ff43c06d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff4641f7420 tx=0x7ff44c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.386+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4540778e0 con 0x7ff46406c9d0 2024-01-16T15:00:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.580+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff464070130 con 0x7ff46406c9d0 2024-01-16T15:00:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.581+0000 7ff45a000640 1 -- 172.21.15.43:0/345446318 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff464070130 con 0x7ff46406c9d0 2024-01-16T15:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:17.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 msgr2=0x7ff43c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 0x7ff43c06d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff4641f7420 tx=0x7ff44c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 msgr2=0x7ff464141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff4540099c0 tx=0x7ff4540046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 shutdown_connections 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.583+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff43c06b2c0 0x7ff43c06d780 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4640b35a0 0x7ff46413f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff464144770 0x7ff4641f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 --2- 172.21.15.43:0/345446318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46406c9d0 0x7ff464141b90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 >> 172.21.15.43:0/345446318 conn(0x7ff4640cec10 msgr2=0x7ff4640b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 shutdown_connections 2024-01-16T15:00:17.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.584+0000 7ff463200640 1 -- 172.21.15.43:0/345446318 wait complete. 2024-01-16T15:00:17.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:17.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.827+0000 7fe627e00640 1 Processor -- start 2024-01-16T15:00:17.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.828+0000 7fe627e00640 1 -- start start 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.829+0000 7fe627e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe628153ea0 0x7fe6280ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.829+0000 7fe627e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe628070850 0x7fe628071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.829+0000 7fe627e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6280ce650 con 0x7fe628154280 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe628154710 con 0x7fe628070850 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62815e0a0 con 0x7fe628153ea0 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe628070850 0x7fe628071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe628070850 0x7fe628071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38446/0 (socket says 172.21.15.43:38446) 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 -- 172.21.15.43:0/344136098 learned_addr learned my addr 172.21.15.43:0/344136098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe626000640 1 --1- 172.21.15.43:0/344136098 >> v1:172.21.15.43:6789/0 conn(0x7fe628154280 0x7fe62815d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49664/0 (socket says 172.21.15.43:49664) 2024-01-16T15:00:17.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe626a00640 1 --2- 172.21.15.43:0/344136098 >> v2:172.21.15.69:3300/0 conn(0x7fe628153ea0 0x7fe6280ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 -- 172.21.15.43:0/344136098 >> v2:172.21.15.69:3300/0 conn(0x7fe628153ea0 msgr2=0x7fe6280ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:17.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 --2- 172.21.15.43:0/344136098 >> v2:172.21.15.69:3300/0 conn(0x7fe628153ea0 0x7fe6280ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 -- 172.21.15.43:0/344136098 >> v1:172.21.15.43:6789/0 conn(0x7fe628154280 legacy=0x7fe62815d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.830+0000 7fe627400640 1 -- 172.21.15.43:0/344136098 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62815e380 con 0x7fe628070850 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe627400640 1 --2- 172.21.15.43:0/344136098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628070850 0x7fe628071350 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe6280cfc10 tx=0x7fe62002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cbed04293cda45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe625600640 1 -- 172.21.15.43:0/344136098 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620030860 con 0x7fe628070850 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe625600640 1 -- 172.21.15.43:0/344136098 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6200309c0 con 0x7fe628070850 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe625600640 1 -- 172.21.15.43:0/344136098 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620030d50 con 0x7fe628070850 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe627e00640 1 -- 172.21.15.43:0/344136098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628070850 msgr2=0x7fe628071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.832+0000 7fe627e00640 1 --2- 172.21.15.43:0/344136098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628070850 0x7fe628071350 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe6280cfc10 tx=0x7fe62002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 -- 172.21.15.43:0/344136098 shutdown_connections 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 --2- 172.21.15.43:0/344136098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628070850 0x7fe628071350 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 --2- 172.21.15.43:0/344136098 >> v2:172.21.15.69:3300/0 conn(0x7fe628153ea0 0x7fe6280ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 -- 172.21.15.43:0/344136098 >> 172.21.15.43:0/344136098 conn(0x7fe6280cec10 msgr2=0x7fe6280cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 -- 172.21.15.43:0/344136098 shutdown_connections 2024-01-16T15:00:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.833+0000 7fe627e00640 1 -- 172.21.15.43:0/344136098 wait complete. 2024-01-16T15:00:17.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.835+0000 7fe627e00640 1 Processor -- start 2024-01-16T15:00:17.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.835+0000 7fe627e00640 1 -- start start 2024-01-16T15:00:17.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 0x7fe6281e89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe628153ea0 0x7fe628150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280aeb80 con 0x7fe628154280 2024-01-16T15:00:17.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280ae830 con 0x7fe628070850 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6281e8fe0 con 0x7fe628153ea0 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38456/0 (socket says 172.21.15.43:38456) 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe627400640 1 -- 172.21.15.43:0/3119491028 learned_addr learned my addr 172.21.15.43:0/3119491028 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe626a00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 0x7fe6281e89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.836+0000 7fe626000640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe628153ea0 0x7fe628150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 -- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe628153ea0 msgr2=0x7fe628150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe628153ea0 0x7fe628150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 -- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 msgr2=0x7fe6281e89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 0x7fe6281e89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 -- 172.21.15.43:0/3119491028 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6200094d0 con 0x7fe628154280 2024-01-16T15:00:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe626a00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 0x7fe6281e89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:17.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627400640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6200090b0 tx=0x7fe620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620039c90 con 0x7fe628154280 2024-01-16T15:00:17.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620039df0 con 0x7fe628154280 2024-01-16T15:00:17.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6281e92c0 con 0x7fe628154280 2024-01-16T15:00:17.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620043640 con 0x7fe628154280 2024-01-16T15:00:17.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.837+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe628150560 con 0x7fe628154280 2024-01-16T15:00:17.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.838+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6280aecc0 con 0x7fe628154280 2024-01-16T15:00:17.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.840+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe6200437a0 con 0x7fe628154280 2024-01-16T15:00:17.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.840+0000 7fe61ea00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 0x7fe60006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:17.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.841+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe62003e070 con 0x7fe628154280 2024-01-16T15:00:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.841+0000 7fe626a00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 0x7fe60006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.842+0000 7fe626a00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 0x7fe60006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe618006fd0 tx=0x7fe618008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:17.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:17.845+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6280aecc0 con 0x7fe628154280 2024-01-16T15:00:18.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.040+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6280aecc0 con 0x7fe628154280 2024-01-16T15:00:18.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.041+0000 7fe61ea00640 1 -- 172.21.15.43:0/3119491028 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe6280aecc0 con 0x7fe628154280 2024-01-16T15:00:18.022 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 msgr2=0x7fe60006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 0x7fe60006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe618006fd0 tx=0x7fe618008040 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 msgr2=0x7fe6281e6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6200090b0 tx=0x7fe620004730 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 shutdown_connections 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe60006b2c0 0x7fe60006d780 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe628154280 0x7fe6281e6de0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe628153ea0 0x7fe628150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 --2- 172.21.15.43:0/3119491028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe628070850 0x7fe6281e89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 >> 172.21.15.43:0/3119491028 conn(0x7fe6280cec10 msgr2=0x7fe62815d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:18.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 shutdown_connections 2024-01-16T15:00:18.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.044+0000 7fe627e00640 1 -- 172.21.15.43:0/3119491028 wait complete. 2024-01-16T15:00:18.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:18.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.269+0000 7fa4a7e00640 1 Processor -- start 2024-01-16T15:00:18.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.270+0000 7fa4a7e00640 1 -- start start 2024-01-16T15:00:18.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.271+0000 7fa4a7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa4a815f270 0x7fa4a80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa4a7e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a815f930 con 0x7fa4a8144770 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa4a7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a80b3b10 con 0x7fa4a806d9e0 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa4a7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a80b43b0 con 0x7fa4a815f270 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa4a6a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa4a8144770 0x7fa4a815eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49676/0 (socket says 172.21.15.43:49676) 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa4a6a00640 1 -- 172.21.15.43:0/4278901448 learned_addr learned my addr 172.21.15.43:0/4278901448 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa49fe00640 1 --2- 172.21.15.43:0/4278901448 >> v2:172.21.15.43:3300/0 conn(0x7fa4a815f270 0x7fa4a80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa49fe00640 1 -- 172.21.15.43:0/4278901448 >> v1:172.21.15.69:6789/0 conn(0x7fa4a806d9e0 legacy=0x7fa4a80cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:18.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa49fe00640 1 -- 172.21.15.43:0/4278901448 >> v1:172.21.15.43:6789/0 conn(0x7fa4a8144770 legacy=0x7fa4a815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.272+0000 7fa49fe00640 1 -- 172.21.15.43:0/4278901448 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80b3df0 con 0x7fa4a815f270 2024-01-16T15:00:18.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.273+0000 7fa49fe00640 1 --2- 172.21.15.43:0/4278901448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a815f270 0x7fa4a80cd4a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa4900097a0 tx=0x7fa49002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7af3df3c55e8e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.274+0000 7fa4a6000640 1 -- 172.21.15.43:0/4278901448 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490031830 con 0x7fa4a815f270 2024-01-16T15:00:18.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.274+0000 7fa4a6000640 1 -- 172.21.15.43:0/4278901448 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa490031990 con 0x7fa4a815f270 2024-01-16T15:00:18.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.274+0000 7fa4a6000640 1 -- 172.21.15.43:0/4278901448 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490031d20 con 0x7fa4a815f270 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.274+0000 7fa4a7e00640 1 -- 172.21.15.43:0/4278901448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a815f270 msgr2=0x7fa4a80cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/4278901448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a815f270 0x7fa4a80cd4a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa4900097a0 tx=0x7fa49002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 -- 172.21.15.43:0/4278901448 shutdown_connections 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/4278901448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a815f270 0x7fa4a80cd4a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 -- 172.21.15.43:0/4278901448 >> 172.21.15.43:0/4278901448 conn(0x7fa4a80cec10 msgr2=0x7fa4a80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 -- 172.21.15.43:0/4278901448 shutdown_connections 2024-01-16T15:00:18.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.275+0000 7fa4a7e00640 1 -- 172.21.15.43:0/4278901448 wait complete. 2024-01-16T15:00:18.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.277+0000 7fa4a7e00640 1 Processor -- start 2024-01-16T15:00:18.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.277+0000 7fa4a7e00640 1 -- start start 2024-01-16T15:00:18.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4a806d9e0 0x7fa4a8156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4a815f270 0x7fa4a80706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a813f150 con 0x7fa4a8144770 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a813f430 con 0x7fa4a815f270 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a813f710 con 0x7fa4a806d9e0 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa49fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa49fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38470/0 (socket says 172.21.15.43:38470) 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa49fe00640 1 -- 172.21.15.43:0/503397934 learned_addr learned my addr 172.21.15.43:0/503397934 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a7400640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4a815f270 0x7fa4a80706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.278+0000 7fa4a6a00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4a806d9e0 0x7fa4a8156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 -- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4a806d9e0 msgr2=0x7fa4a8156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4a806d9e0 0x7fa4a8156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 -- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4a815f270 msgr2=0x7fa4a80706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4a815f270 0x7fa4a80706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 -- 172.21.15.43:0/503397934 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa490009400 con 0x7fa4a8144770 2024-01-16T15:00:18.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49fe00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa490000c00 tx=0x7fa4900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49003aa60 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49003abc0 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4900443f0 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.279+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a813f9f0 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.280+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a8156dd0 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.281+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a804e840 con 0x7fa4a8144770 2024-01-16T15:00:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.282+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa49004d450 con 0x7fa4a8144770 2024-01-16T15:00:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.282+0000 7fa49ea00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 0x7fa48006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.283+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa4900abe60 con 0x7fa4a8144770 2024-01-16T15:00:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.283+0000 7fa4a6a00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 0x7fa48006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.283+0000 7fa4a6a00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 0x7fa48006d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa498005880 tx=0x7fa498009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.287+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa490076c40 con 0x7fa4a8144770 2024-01-16T15:00:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.488+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4a80b2460 con 0x7fa4a8144770 2024-01-16T15:00:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.489+0000 7fa49ea00640 1 -- 172.21.15.43:0/503397934 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa490035070 con 0x7fa4a8144770 2024-01-16T15:00:18.469 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 msgr2=0x7fa48006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 0x7fa48006d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa498005880 tx=0x7fa498009040 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 msgr2=0x7fa4a813e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa490000c00 tx=0x7fa4900046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 shutdown_connections 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa48006b2c0 0x7fa48006d780 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4a815f270 0x7fa4a80706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4a8144770 0x7fa4a813e7e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 --2- 172.21.15.43:0/503397934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4a806d9e0 0x7fa4a8156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.491+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 >> 172.21.15.43:0/503397934 conn(0x7fa4a80cec10 msgr2=0x7fa4a80717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.492+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 shutdown_connections 2024-01-16T15:00:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.492+0000 7fa4a7e00640 1 -- 172.21.15.43:0/503397934 wait complete. 2024-01-16T15:00:18.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.729+0000 7fa6c3200640 1 Processor -- start 2024-01-16T15:00:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.729+0000 7fa6c3200640 1 -- start start 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6c4071620 0x7fa6c415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c3200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6c4070e90 0x7fa6c40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40ce670 con 0x7fa6c4071a00 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c3200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40ce8d0 con 0x7fa6c4070e90 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40700d0 con 0x7fa6c4071620 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c2800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6c4070e90 0x7fa6c40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c2800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6c4070e90 0x7fa6c40ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38472/0 (socket says 172.21.15.43:38472) 2024-01-16T15:00:18.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.731+0000 7fa6c2800640 1 -- 172.21.15.43:0/3496895735 learned_addr learned my addr 172.21.15.43:0/3496895735 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:18.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.732+0000 7fa6c2800640 1 -- 172.21.15.43:0/3496895735 >> v2:172.21.15.69:3300/0 conn(0x7fa6c4071620 msgr2=0x7fa6c415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:18.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.732+0000 7fa6c2800640 1 --2- 172.21.15.43:0/3496895735 >> v2:172.21.15.69:3300/0 conn(0x7fa6c4071620 0x7fa6c415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.732+0000 7fa6c2800640 1 -- 172.21.15.43:0/3496895735 >> v1:172.21.15.69:6789/0 conn(0x7fa6c4071a00 legacy=0x7fa6c40cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:18.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.732+0000 7fa6c2800640 1 -- 172.21.15.43:0/3496895735 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40703b0 con 0x7fa6c4070e90 2024-01-16T15:00:18.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c2800640 1 --2- 172.21.15.43:0/3496895735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4070e90 0x7fa6c40ce130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40b17d0 tx=0x7fa6bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ebd42e85c172c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c1400640 1 -- 172.21.15.43:0/3496895735 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc02ca80 con 0x7fa6c4070e90 2024-01-16T15:00:18.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c1400640 1 -- 172.21.15.43:0/3496895735 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc02cbe0 con 0x7fa6c4070e90 2024-01-16T15:00:18.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c1400640 1 -- 172.21.15.43:0/3496895735 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0343f0 con 0x7fa6c4070e90 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c3200640 1 -- 172.21.15.43:0/3496895735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4070e90 msgr2=0x7fa6c40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.733+0000 7fa6c3200640 1 --2- 172.21.15.43:0/3496895735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4070e90 0x7fa6c40ce130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40b17d0 tx=0x7fa6bc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 -- 172.21.15.43:0/3496895735 shutdown_connections 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 --2- 172.21.15.43:0/3496895735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4070e90 0x7fa6c40ce130 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 --2- 172.21.15.43:0/3496895735 >> v2:172.21.15.69:3300/0 conn(0x7fa6c4071620 0x7fa6c415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 -- 172.21.15.43:0/3496895735 >> 172.21.15.43:0/3496895735 conn(0x7fa6c40b0840 msgr2=0x7fa6c40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 -- 172.21.15.43:0/3496895735 shutdown_connections 2024-01-16T15:00:18.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.734+0000 7fa6c3200640 1 -- 172.21.15.43:0/3496895735 wait complete. 2024-01-16T15:00:18.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.735+0000 7fa6c3200640 1 Processor -- start 2024-01-16T15:00:18.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 -- start start 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 0x7fa6c40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c4071620 0x7fa6c413fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c414c9b0 con 0x7fa6c4071a00 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b1e40 con 0x7fa6c4071620 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.736+0000 7fa6c3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b2120 con 0x7fa6c4070e90 2024-01-16T15:00:18.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38488/0 (socket says 172.21.15.43:38488) 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 -- 172.21.15.43:0/558982620 learned_addr learned my addr 172.21.15.43:0/558982620 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6bbe00640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c4071620 0x7fa6c413fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c1e00640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 0x7fa6c40b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 -- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 msgr2=0x7fa6c40b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 0x7fa6c40b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 -- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c4071620 msgr2=0x7fa6c413fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c4071620 0x7fa6c413fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c2800640 1 -- 172.21.15.43:0/558982620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc009b30 con 0x7fa6c4071a00 2024-01-16T15:00:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.737+0000 7fa6c1e00640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 0x7fa6c40b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6c2800640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc036040 tx=0x7fa6bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc03e450 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc03e5d0 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc03e980 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c40b2400 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.738+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c40b29c0 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.739+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c404e840 con 0x7fa6c4071a00 2024-01-16T15:00:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.740+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa6bc02cd50 con 0x7fa6c4071a00 2024-01-16T15:00:18.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.741+0000 7fa6ba000640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 0x7fa69c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:18.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.741+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa6bc0ac9b0 con 0x7fa6c4071a00 2024-01-16T15:00:18.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.741+0000 7fa6c1e00640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 0x7fa69c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:18.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.742+0000 7fa6c1e00640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 0x7fa69c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:18.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.745+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6bc077670 con 0x7fa6c4071a00 2024-01-16T15:00:18.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.949+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa6c4157f60 con 0x7fa6c4071a00 2024-01-16T15:00:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.950+0000 7fa6ba000640 1 -- 172.21.15.43:0/558982620 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa6bc043020 con 0x7fa6c4071a00 2024-01-16T15:00:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 msgr2=0x7fa69c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 0x7fa69c06d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4008040 comp rx=0 tx=0).stop 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 msgr2=0x7fa6c4142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc036040 tx=0x7fa6bc004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 shutdown_connections 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa69c06b2c0 0x7fa69c06d780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c4071620 0x7fa6c413fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6c4071a00 0x7fa6c4142690 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 --2- 172.21.15.43:0/558982620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c4070e90 0x7fa6c40b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 >> 172.21.15.43:0/558982620 conn(0x7fa6c40b0840 msgr2=0x7fa6c406d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 shutdown_connections 2024-01-16T15:00:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:18.953+0000 7fa6c3200640 1 -- 172.21.15.43:0/558982620 wait complete. 2024-01-16T15:00:18.951 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:18.951 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:19.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.251+0000 7f397da00640 1 Processor -- start 2024-01-16T15:00:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.252+0000 7f397da00640 1 -- start start 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f397da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3978146010 0x7f39781463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f397da00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39780d0fe0 0x7f397806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f397da00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397815d970 con 0x7f397806efd0 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f397da00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397806f460 con 0x7f3978146010 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f397da00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397806b890 con 0x7f39780d0fe0 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f3976a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3978146010 0x7f39781463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f3977400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f397806efd0 0x7f3978160f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49682/0 (socket says 172.21.15.43:49682) 2024-01-16T15:00:19.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.253+0000 7f3977400640 1 -- 172.21.15.43:0/3500707927 learned_addr learned my addr 172.21.15.43:0/3500707927 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.254+0000 7f3976a00640 1 -- 172.21.15.43:0/3500707927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39780d0fe0 msgr2=0x7f397806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.254+0000 7f3976a00640 1 --2- 172.21.15.43:0/3500707927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39780d0fe0 0x7f397806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.254+0000 7f3976a00640 1 -- 172.21.15.43:0/3500707927 >> v1:172.21.15.43:6789/0 conn(0x7f397806efd0 legacy=0x7f3978160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.254+0000 7f3976a00640 1 -- 172.21.15.43:0/3500707927 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780d1550 con 0x7f3978146010 2024-01-16T15:00:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.255+0000 7f3976a00640 1 --2- 172.21.15.43:0/3500707927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3978146010 0x7f39781463f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f39680097a0 tx=0x7f396802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c187e14ff996267b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.255+0000 7f3976000640 1 -- 172.21.15.43:0/3500707927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396802d040 con 0x7f3978146010 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.255+0000 7f3976000640 1 -- 172.21.15.43:0/3500707927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968031810 con 0x7f3978146010 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.255+0000 7f3976000640 1 -- 172.21.15.43:0/3500707927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968031ba0 con 0x7f3978146010 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 -- 172.21.15.43:0/3500707927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3978146010 msgr2=0x7f39781463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 --2- 172.21.15.43:0/3500707927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3978146010 0x7f39781463f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f39680097a0 tx=0x7f396802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 -- 172.21.15.43:0/3500707927 shutdown_connections 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 --2- 172.21.15.43:0/3500707927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39780d0fe0 0x7f397806ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 --2- 172.21.15.43:0/3500707927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3978146010 0x7f39781463f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 -- 172.21.15.43:0/3500707927 >> 172.21.15.43:0/3500707927 conn(0x7f39780663d0 msgr2=0x7f39780667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 -- 172.21.15.43:0/3500707927 shutdown_connections 2024-01-16T15:00:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.256+0000 7f397da00640 1 -- 172.21.15.43:0/3500707927 wait complete. 2024-01-16T15:00:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.258+0000 7f397da00640 1 Processor -- start 2024-01-16T15:00:19.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.258+0000 7f397da00640 1 -- start start 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 0x7f39780ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3978146010 0x7f3978143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397815d440 con 0x7f397806efd0 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39781e8870 con 0x7f39780d0fe0 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f397da00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39781e8b50 con 0x7f3978146010 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f3976a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f3976a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38502/0 (socket says 172.21.15.43:38502) 2024-01-16T15:00:19.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.259+0000 7f3976a00640 1 -- 172.21.15.43:0/3275559161 learned_addr learned my addr 172.21.15.43:0/3275559161 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f396de00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 0x7f39780ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3977400640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3978146010 0x7f3978143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 -- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3978146010 msgr2=0x7f3978143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3978146010 0x7f3978143d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 -- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 msgr2=0x7f39780ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 0x7f39780ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 -- 172.21.15.43:0/3275559161 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968009400 con 0x7f397806efd0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f396de00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 0x7f39780ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f3976a00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3968000c00 tx=0x7f39680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396803ad60 con 0x7f397806efd0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.260+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680443f0 con 0x7f397806efd0 2024-01-16T15:00:19.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.261+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39781e8e30 con 0x7f397806efd0 2024-01-16T15:00:19.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.261+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680447a0 con 0x7f397806efd0 2024-01-16T15:00:19.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.261+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39781e93f0 con 0x7f397806efd0 2024-01-16T15:00:19.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.262+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39780d0e50 con 0x7f397806efd0 2024-01-16T15:00:19.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.263+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f396804d450 con 0x7f397806efd0 2024-01-16T15:00:19.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.264+0000 7f396f400640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 0x7f395006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.264+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f39680abfa0 con 0x7f397806efd0 2024-01-16T15:00:19.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.264+0000 7f396de00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 0x7f395006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.265+0000 7f396de00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 0x7f395006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3960006fd0 tx=0x7f3960008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.268+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f39680dee90 con 0x7f397806efd0 2024-01-16T15:00:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.462+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3978145870 con 0x7f397806efd0 2024-01-16T15:00:19.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.465+0000 7f396f400640 1 -- 172.21.15.43:0/3275559161 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f3968076d90 con 0x7f397806efd0 2024-01-16T15:00:19.447 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:19.447 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 msgr2=0x7f395006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 0x7f395006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3960006fd0 tx=0x7f3960008040 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 msgr2=0x7f39781e9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3968000c00 tx=0x7f39680046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 shutdown_connections 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f395006b2c0 0x7f395006d780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39780d0fe0 0x7f39780ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3978146010 0x7f3978143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 --2- 172.21.15.43:0/3275559161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397806efd0 0x7f39781e9910 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 >> 172.21.15.43:0/3275559161 conn(0x7f39780663d0 msgr2=0x7f3978140990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 shutdown_connections 2024-01-16T15:00:19.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.468+0000 7f397da00640 1 -- 172.21.15.43:0/3275559161 wait complete. 2024-01-16T15:00:19.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:19.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.715+0000 7fb8bf200640 1 Processor -- start 2024-01-16T15:00:19.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.715+0000 7fb8bf200640 1 -- start start 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bf200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb8c0144770 0x7fb8c015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 0x7fb8c015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c015f930 con 0x7fb8c006c9d0 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bf200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c00b43b0 con 0x7fb8c0144770 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00b3b10 con 0x7fb8c00b35a0 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb8c0144770 0x7fb8c015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb8c0144770 0x7fb8c015eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38510/0 (socket says 172.21.15.43:38510) 2024-01-16T15:00:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.717+0000 7fb8bde00640 1 -- 172.21.15.43:0/361693343 learned_addr learned my addr 172.21.15.43:0/361693343 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 0x7fb8c015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8bde00640 1 -- 172.21.15.43:0/361693343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 msgr2=0x7fb8c015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8bde00640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 0x7fb8c015f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8bde00640 1 -- 172.21.15.43:0/361693343 >> v1:172.21.15.69:6789/0 conn(0x7fb8c006c9d0 legacy=0x7fb8c006ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8bde00640 1 -- 172.21.15.43:0/361693343 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c00b3df0 con 0x7fb8c0144770 2024-01-16T15:00:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.718+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 0x7fb8c015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.719+0000 7fb8bde00640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c0144770 0x7fb8c015eb60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00098a0 tx=0x7fb8b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46268a780000bd3e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.719+0000 7fb8bd400640 1 -- 172.21.15.43:0/361693343 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0030840 con 0x7fb8c0144770 2024-01-16T15:00:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.719+0000 7fb8bd400640 1 -- 172.21.15.43:0/361693343 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b00309a0 con 0x7fb8c0144770 2024-01-16T15:00:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.719+0000 7fb8bd400640 1 -- 172.21.15.43:0/361693343 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0030d30 con 0x7fb8c0144770 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 -- 172.21.15.43:0/361693343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c0144770 msgr2=0x7fb8c015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c0144770 0x7fb8c015eb60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00098a0 tx=0x7fb8b002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 -- 172.21.15.43:0/361693343 shutdown_connections 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c00b35a0 0x7fb8c015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 --2- 172.21.15.43:0/361693343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c0144770 0x7fb8c015eb60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 -- 172.21.15.43:0/361693343 >> 172.21.15.43:0/361693343 conn(0x7fb8c00cec10 msgr2=0x7fb8c00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 -- 172.21.15.43:0/361693343 shutdown_connections 2024-01-16T15:00:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.720+0000 7fb8bf200640 1 -- 172.21.15.43:0/361693343 wait complete. 2024-01-16T15:00:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.722+0000 7fb8bf200640 1 Processor -- start 2024-01-16T15:00:19.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.722+0000 7fb8bf200640 1 -- start start 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 0x7fb8c00c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c0144770 0x7fb8c00c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00c4be0 con 0x7fb8c006c9d0 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00c3fd0 con 0x7fb8c00b35a0 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00c42b0 con 0x7fb8c0144770 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38516/0 (socket says 172.21.15.43:38516) 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.723+0000 7fb8bde00640 1 -- 172.21.15.43:0/3043543642 learned_addr learned my addr 172.21.15.43:0/3043543642 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 0x7fb8c00c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8be800640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c0144770 0x7fb8c00c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 -- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c0144770 msgr2=0x7fb8c00c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c0144770 0x7fb8c00c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 -- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 msgr2=0x7fb8c00c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 0x7fb8c00c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 -- 172.21.15.43:0/3043543642 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00094d0 con 0x7fb8c006c9d0 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 0x7fb8c00c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.724+0000 7fb8bde00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0000c00 tx=0x7fb8b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.725+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0030810 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.725+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b0039d90 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.725+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0043670 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.725+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c00c4590 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.725+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c00c3c30 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.726+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c004e840 con 0x7fb8c006c9d0 2024-01-16T15:00:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.727+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb8b002f390 con 0x7fb8c006c9d0 2024-01-16T15:00:19.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.728+0000 7fb8b6000640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 0x7fb89806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.728+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8b00abfd0 con 0x7fb8c006c9d0 2024-01-16T15:00:19.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.729+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 0x7fb89806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.729+0000 7fb8b7e00640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 0x7fb89806d7b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.733+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8b0076ed0 con 0x7fb8c006c9d0 2024-01-16T15:00:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.927+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8c00b2460 con 0x7fb8c006c9d0 2024-01-16T15:00:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.927+0000 7fb8b6000640 1 -- 172.21.15.43:0/3043543642 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8b0048020 con 0x7fb8c006c9d0 2024-01-16T15:00:19.908 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 msgr2=0x7fb89806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 0x7fb89806d7b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 msgr2=0x7fb8c00c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0000c00 tx=0x7fb8b0004730 comp rx=0 tx=0).stop 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 shutdown_connections 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb89806b2f0 0x7fb89806d7b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8c00b35a0 0x7fb8c00c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8c0144770 0x7fb8c00c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 --2- 172.21.15.43:0/3043543642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c006c9d0 0x7fb8c00c6cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.930+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 >> 172.21.15.43:0/3043543642 conn(0x7fb8c00cec10 msgr2=0x7fb8c015ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.931+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 shutdown_connections 2024-01-16T15:00:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:19.931+0000 7fb8bf200640 1 -- 172.21.15.43:0/3043543642 wait complete. 2024-01-16T15:00:19.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:20.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.167+0000 7f6930a00640 1 Processor -- start 2024-01-16T15:00:20.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.167+0000 7f6930a00640 1 -- start start 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f6930a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f692c1439b0 0x7f692c070760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f6930a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c070d40 con 0x7f692c1435d0 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f6930a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c071020 con 0x7f692c0b4140 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f6930a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692c071300 con 0x7f692c1439b0 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f692b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f692c1435d0 0x7f692c0b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49686/0 (socket says 172.21.15.43:49686) 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f692b400640 1 -- 172.21.15.43:0/2017344582 learned_addr learned my addr 172.21.15.43:0/2017344582 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:20.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.169+0000 7f692be00640 1 --2- 172.21.15.43:0/2017344582 >> v2:172.21.15.43:3300/0 conn(0x7f692c1439b0 0x7f692c070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.170+0000 7f692aa00640 1 -- 172.21.15.43:0/2017344582 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216084028 0 0) 0x7f692c070d40 con 0x7f692c1435d0 2024-01-16T15:00:20.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.170+0000 7f692be00640 1 -- 172.21.15.43:0/2017344582 >> v1:172.21.15.69:6789/0 conn(0x7f692c0b4140 legacy=0x7f692c070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.170+0000 7f692be00640 1 -- 172.21.15.43:0/2017344582 >> v1:172.21.15.43:6789/0 conn(0x7f692c1435d0 legacy=0x7f692c0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.170+0000 7f692be00640 1 -- 172.21.15.43:0/2017344582 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0715e0 con 0x7f692c1439b0 2024-01-16T15:00:20.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.171+0000 7f692be00640 1 --2- 172.21.15.43:0/2017344582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1439b0 0x7f692c070760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6924009d90 tx=0x7f692402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e743fb2ca3570d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.171+0000 7f692aa00640 1 -- 172.21.15.43:0/2017344582 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692402cbd0 con 0x7f692c1439b0 2024-01-16T15:00:20.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.171+0000 7f692aa00640 1 -- 172.21.15.43:0/2017344582 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692402cd30 con 0x7f692c1439b0 2024-01-16T15:00:20.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.171+0000 7f692aa00640 1 -- 172.21.15.43:0/2017344582 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692402cea0 con 0x7f692c1439b0 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 -- 172.21.15.43:0/2017344582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1439b0 msgr2=0x7f692c070760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 --2- 172.21.15.43:0/2017344582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1439b0 0x7f692c070760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6924009d90 tx=0x7f692402be20 comp rx=0 tx=0).stop 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 -- 172.21.15.43:0/2017344582 shutdown_connections 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 --2- 172.21.15.43:0/2017344582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1439b0 0x7f692c070760 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 -- 172.21.15.43:0/2017344582 >> 172.21.15.43:0/2017344582 conn(0x7f692c0cec10 msgr2=0x7f692c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 -- 172.21.15.43:0/2017344582 shutdown_connections 2024-01-16T15:00:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.172+0000 7f6930a00640 1 -- 172.21.15.43:0/2017344582 wait complete. 2024-01-16T15:00:20.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.174+0000 7f6930a00640 1 Processor -- start 2024-01-16T15:00:20.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.174+0000 7f6930a00640 1 -- start start 2024-01-16T15:00:20.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f692c0b4140 0x7f692c1f5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f692c1439b0 0x7f692c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c15e270 con 0x7f692c1435d0 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1f5bc0 con 0x7f692c0b4140 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6930a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692c15d8d0 con 0x7f692c1439b0 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6923e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6923e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38530/0 (socket says 172.21.15.43:38530) 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.175+0000 7f6923e00640 1 -- 172.21.15.43:0/701975256 learned_addr learned my addr 172.21.15.43:0/701975256 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f692b400640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f692c0b4140 0x7f692c1f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f692be00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f692c1439b0 0x7f692c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 -- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f692c1439b0 msgr2=0x7f692c141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f692c1439b0 0x7f692c141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 -- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f692c0b4140 msgr2=0x7f692c1f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f692c0b4140 0x7f692c1f5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 -- 172.21.15.43:0/701975256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924009a70 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923e00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f691400c430 tx=0x7f691400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914011790 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69140118f0 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914011ce0 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.176+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c15dbb0 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.177+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c155280 con 0x7f692c1435d0 2024-01-16T15:00:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.178+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c04e840 con 0x7f692c1435d0 2024-01-16T15:00:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.179+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f691401c460 con 0x7f692c1435d0 2024-01-16T15:00:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.180+0000 7f6923400640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 0x7f690406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.180+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6914089c20 con 0x7f692c1435d0 2024-01-16T15:00:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.180+0000 7f692b400640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 0x7f690406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.181+0000 7f692b400640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 0x7f690406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f691c000f60 tx=0x7f691c007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.185+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6914054a00 con 0x7f692c1435d0 2024-01-16T15:00:20.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.372+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f692c0cd4e0 con 0x7f692c1435d0 2024-01-16T15:00:20.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.372+0000 7f6923400640 1 -- 172.21.15.43:0/701975256 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6914058020 con 0x7f692c1435d0 2024-01-16T15:00:20.353 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 msgr2=0x7f690406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 0x7f690406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f691c000f60 tx=0x7f691c007b50 comp rx=0 tx=0).stop 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 msgr2=0x7f692c0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f691400c430 tx=0x7f691400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 shutdown_connections 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f690406b2c0 0x7f690406d780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f692c1439b0 0x7f692c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c1435d0 0x7f692c0cd700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 --2- 172.21.15.43:0/701975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f692c0b4140 0x7f692c1f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 >> 172.21.15.43:0/701975256 conn(0x7f692c0cec10 msgr2=0x7f692c0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 shutdown_connections 2024-01-16T15:00:20.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.375+0000 7f6930a00640 1 -- 172.21.15.43:0/701975256 wait complete. 2024-01-16T15:00:20.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.609+0000 7f49d0000640 1 Processor -- start 2024-01-16T15:00:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.609+0000 7f49d0000640 1 -- start start 2024-01-16T15:00:20.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.610+0000 7f49d0000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49c80cf380 0x7f49c80ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.610+0000 7f49d0000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80cd2b0 con 0x7f49c8158f40 2024-01-16T15:00:20.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.610+0000 7f49d0000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80d02e0 con 0x7f49c80cb9d0 2024-01-16T15:00:20.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.610+0000 7f49d0000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80cbe60 con 0x7f49c80cf380 2024-01-16T15:00:20.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.611+0000 7f49cd600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f49c8158f40 0x7f49c80cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49700/0 (socket says 172.21.15.43:49700) 2024-01-16T15:00:20.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.611+0000 7f49cd600640 1 -- 172.21.15.43:0/3809754747 learned_addr learned my addr 172.21.15.43:0/3809754747 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:20.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.611+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3809754747 >> v2:172.21.15.69:3300/0 conn(0x7f49c80cf380 0x7f49c80ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.611+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093477524 0 0) 0x7f49c80cd2b0 con 0x7f49c8158f40 2024-01-16T15:00:20.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.611+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8003520 con 0x7f49c8158f40 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2288677987 0 0) 0x7f49a8003520 con 0x7f49c8158f40 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 >> v2:172.21.15.69:3300/0 conn(0x7f49c80cf380 msgr2=0x7f49c80ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 --2- 172.21.15.43:0/3809754747 >> v2:172.21.15.69:3300/0 conn(0x7f49c80cf380 0x7f49c80ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 >> v1:172.21.15.69:6789/0 conn(0x7f49c80cb9d0 legacy=0x7f49c815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80cd4e0 con 0x7f49c8158f40 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f49b0001670 con 0x7f49c8158f40 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 >> v1:172.21.15.43:6789/0 conn(0x7f49c8158f40 legacy=0x7f49c80cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a80047c0 0x7f49a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49a8005180 0x7f49a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80cbe60 con 0x7f49a80039c0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a800c080 con 0x7f49a80047c0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a800c240 con 0x7f49a8005180 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.612+0000 7f49cd600640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a80047c0 0x7f49a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49ce000640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49a8005180 0x7f49a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49cd600640 1 -- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49a8005180 msgr2=0x7f49a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49cd600640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49a8005180 0x7f49a800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49cd600640 1 -- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a80047c0 msgr2=0x7f49a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49cd600640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a80047c0 0x7f49a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49cd600640 1 -- 172.21.15.43:0/3809754747 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8003520 con 0x7f49a80039c0 2024-01-16T15:00:20.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.613+0000 7f49ce000640 1 -- 172.21.15.43:0/3809754747 reap_dead start 2024-01-16T15:00:20.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.614+0000 7f49cd600640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49a8003de0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f49b000c0e0 tx=0x7f49a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e8994d9dafa002 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.614+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0005ad0 con 0x7f49a80039c0 2024-01-16T15:00:20.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.614+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b0005c30 con 0x7f49a80039c0 2024-01-16T15:00:20.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.614+0000 7f49c7e00640 1 -- 172.21.15.43:0/3809754747 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b002ca50 con 0x7f49a80039c0 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 -- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 msgr2=0x7f49a8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49a8003de0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f49b000c0e0 tx=0x7f49a80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 -- 172.21.15.43:0/3809754747 shutdown_connections 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 --2- 172.21.15.43:0/3809754747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49a8003de0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 -- 172.21.15.43:0/3809754747 >> 172.21.15.43:0/3809754747 conn(0x7f49c80cdb20 msgr2=0x7f49c80cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 -- 172.21.15.43:0/3809754747 shutdown_connections 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.615+0000 7f49d0000640 1 -- 172.21.15.43:0/3809754747 wait complete. 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.617+0000 7f49d0000640 1 Processor -- start 2024-01-16T15:00:20.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.617+0000 7f49d0000640 1 -- start start 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 0x7f49c80c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 0x7f49c80afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80ca050 con 0x7f49a80039c0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80ca330 con 0x7f49c80c9ab0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49d0000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80ca610 con 0x7f49c806c6a0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38550/0 (socket says 172.21.15.43:38550) 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 -- 172.21.15.43:0/3467368719 learned_addr learned my addr 172.21.15.43:0/3467368719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 0x7f49c80c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49ce000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 0x7f49c80afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 -- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 msgr2=0x7f49c80c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 0x7f49c80c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 -- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 msgr2=0x7f49c80afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 0x7f49c80afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49cd600640 1 -- 172.21.15.43:0/3467368719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80ca8f0 con 0x7f49a80039c0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 0x7f49c80c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.618+0000 7f49ce000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 0x7f49c80afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49cd600640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f49b0000c00 tx=0x7f49b002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b003c040 con 0x7f49a80039c0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b002cc40 con 0x7f49a80039c0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0007070 con 0x7f49a80039c0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c8157b00 con 0x7f49a80039c0 2024-01-16T15:00:20.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.619+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c80b0390 con 0x7f49a80039c0 2024-01-16T15:00:20.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.620+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c80ca750 con 0x7f49a80039c0 2024-01-16T15:00:20.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.621+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f49b0032070 con 0x7f49a80039c0 2024-01-16T15:00:20.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.622+0000 7f49c5600640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 0x7f49a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:20.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.622+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f49b00ab170 con 0x7f49a80039c0 2024-01-16T15:00:20.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.622+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 0x7f49a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:20.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.623+0000 7f49ccc00640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 0x7f49a006d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f49c815af00 tx=0x7f49b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:20.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.628+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49c80ca750 con 0x7f49a80039c0 2024-01-16T15:00:20.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.821+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49c80ca750 con 0x7f49a80039c0 2024-01-16T15:00:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.822+0000 7f49c5600640 1 -- 172.21.15.43:0/3467368719 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49c80ca750 con 0x7f49a80039c0 2024-01-16T15:00:20.803 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:20.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.824+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 msgr2=0x7f49a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 0x7f49a006d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f49c815af00 tx=0x7f49b8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 msgr2=0x7f49c81575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f49b0000c00 tx=0x7f49b002e070 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 shutdown_connections 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49a006b2c0 0x7f49a006d780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49c80c9ab0 0x7f49c80afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49c806c6a0 0x7f49c80c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 --2- 172.21.15.43:0/3467368719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49a80039c0 0x7f49c81575c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 >> 172.21.15.43:0/3467368719 conn(0x7f49c80cdb20 msgr2=0x7f49c815c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 shutdown_connections 2024-01-16T15:00:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:20.825+0000 7f49d0000640 1 -- 172.21.15.43:0/3467368719 wait complete. 2024-01-16T15:00:20.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:21.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.059+0000 7fb859600640 1 Processor -- start 2024-01-16T15:00:21.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.059+0000 7fb859600640 1 -- start start 2024-01-16T15:00:21.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.060+0000 7fb859600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb8540abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.060+0000 7fb859600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb8540cbcb0 0x7fb85406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.060+0000 7fb859600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8540b4130 con 0x7fb8540ce880 2024-01-16T15:00:21.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.060+0000 7fb859600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8540ac530 con 0x7fb8540cbcb0 2024-01-16T15:00:21.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.060+0000 7fb859600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540b2080 con 0x7fb85415b340 2024-01-16T15:00:21.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb84be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb8540cbcb0 0x7fb85406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb853400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb8540ce880 0x7fb85415ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49708/0 (socket says 172.21.15.43:49708) 2024-01-16T15:00:21.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb853400640 1 -- 172.21.15.43:0/3319900848 learned_addr learned my addr 172.21.15.43:0/3319900848 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:21.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb852a00640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb8540abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb84be00640 1 -- 172.21.15.43:0/3319900848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 msgr2=0x7fb8540abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.062+0000 7fb84be00640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb8540abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.061+0000 7fb852000640 1 -- 172.21.15.43:0/3319900848 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782142803 0 0) 0x7fb8540b4130 con 0x7fb8540ce880 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.062+0000 7fb84be00640 1 -- 172.21.15.43:0/3319900848 >> v1:172.21.15.43:6789/0 conn(0x7fb8540ce880 legacy=0x7fb85415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.062+0000 7fb84be00640 1 -- 172.21.15.43:0/3319900848 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8540b2360 con 0x7fb8540cbcb0 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.062+0000 7fb852a00640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb8540abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.063+0000 7fb84be00640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb85406d520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097a0 tx=0x7fb83c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c7ff03c1518df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.063+0000 7fb852000640 1 -- 172.21.15.43:0/3319900848 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c02d040 con 0x7fb8540cbcb0 2024-01-16T15:00:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.063+0000 7fb852000640 1 -- 172.21.15.43:0/3319900848 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c031810 con 0x7fb8540cbcb0 2024-01-16T15:00:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.063+0000 7fb852000640 1 -- 172.21.15.43:0/3319900848 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c031ba0 con 0x7fb8540cbcb0 2024-01-16T15:00:21.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 -- 172.21.15.43:0/3319900848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 msgr2=0x7fb85406d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb85406d520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097a0 tx=0x7fb83c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 -- 172.21.15.43:0/3319900848 shutdown_connections 2024-01-16T15:00:21.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb85406d520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 --2- 172.21.15.43:0/3319900848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb8540abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 -- 172.21.15.43:0/3319900848 >> 172.21.15.43:0/3319900848 conn(0x7fb8540b0d80 msgr2=0x7fb8540b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 -- 172.21.15.43:0/3319900848 shutdown_connections 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.064+0000 7fb859600640 1 -- 172.21.15.43:0/3319900848 wait complete. 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.066+0000 7fb859600640 1 Processor -- start 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.066+0000 7fb859600640 1 -- start start 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8540ce880 0x7fb854098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb854150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85406d600 con 0x7fb8540cbcb0 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854142d60 con 0x7fb8540ce880 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb859600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85406d040 con 0x7fb85415b340 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38570/0 (socket says 172.21.15.43:38570) 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 -- 172.21.15.43:0/3525713033 learned_addr learned my addr 172.21.15.43:0/3525713033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 -- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 msgr2=0x7fb854150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb854150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 -- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8540ce880 msgr2=0x7fb854098a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8540ce880 0x7fb854098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.067+0000 7fb852a00640 1 -- 172.21.15.43:0/3525713033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c009400 con 0x7fb8540cbcb0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb852a00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb84400d800 tx=0x7fb84400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844004030 con 0x7fb8540cbcb0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844004190 con 0x7fb8540cbcb0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84400b660 con 0x7fb8540cbcb0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85406d320 con 0x7fb8540cbcb0 2024-01-16T15:00:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.068+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb854155700 con 0x7fb8540cbcb0 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.069+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb854074770 con 0x7fb8540cbcb0 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.070+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb84400b7c0 con 0x7fb8540cbcb0 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.071+0000 7fb84aa00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 0x7fb82c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.071+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb844089d40 con 0x7fb8540cbcb0 2024-01-16T15:00:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.071+0000 7fb84be00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 0x7fb82c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.072+0000 7fb84be00640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 0x7fb82c06d7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb83c036000 tx=0x7fb83c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.075+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8440585b0 con 0x7fb8540cbcb0 2024-01-16T15:00:21.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.284+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb85415f040 con 0x7fb8540cbcb0 2024-01-16T15:00:21.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.285+0000 7fb84aa00640 1 -- 172.21.15.43:0/3525713033 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb844058140 con 0x7fb8540cbcb0 2024-01-16T15:00:21.265 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 msgr2=0x7fb82c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 0x7fb82c06d7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb83c036000 tx=0x7fb83c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 msgr2=0x7fb8540704b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb84400d800 tx=0x7fb84400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 shutdown_connections 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb82c06b2f0 0x7fb82c06d7b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8540ce880 0x7fb854098a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85415b340 0x7fb854150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 --2- 172.21.15.43:0/3525713033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8540cbcb0 0x7fb8540704b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.287+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 >> 172.21.15.43:0/3525713033 conn(0x7fb8540b0d80 msgr2=0x7fb8540b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.288+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 shutdown_connections 2024-01-16T15:00:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.288+0000 7fb859600640 1 -- 172.21.15.43:0/3525713033 wait complete. 2024-01-16T15:00:21.287 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:00:21.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:21.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:21.389 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:21.389 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:00:21.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:21.389 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:00:21.451 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:21.451 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:21.512 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:21.512 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:21.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:21.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.794+0000 7fb470e00640 1 Processor -- start 2024-01-16T15:00:21.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.794+0000 7fb470e00640 1 -- start start 2024-01-16T15:00:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.795+0000 7fb470e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb46c0cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.795+0000 7fb470e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb46c06c410 0x7fb46c0cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.795+0000 7fb470e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46c0cbf10 con 0x7fb46c06b880 2024-01-16T15:00:21.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.795+0000 7fb470e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46c0d05e0 con 0x7fb46c06c410 2024-01-16T15:00:21.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.795+0000 7fb470e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0cb5e0 con 0x7fb46c15d550 2024-01-16T15:00:21.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb46c06b880 0x7fb46c06bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49710/0 (socket says 172.21.15.43:49710) 2024-01-16T15:00:21.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46aa00640 1 -- 172.21.15.43:0/440575400 learned_addr learned my addr 172.21.15.43:0/440575400 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46b400640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb46c0cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46be00640 1 --2- 172.21.15.43:0/440575400 >> v2:172.21.15.69:3300/0 conn(0x7fb46c06c410 0x7fb46c0cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377474820 0 0) 0x7fb46c0cbf10 con 0x7fb46c06b880 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.796+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c003520 con 0x7fb46c06b880 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938816497 0 0) 0x7fb44c003520 con 0x7fb46c06b880 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 msgr2=0x7fb46c0cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb46c0cfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 >> v2:172.21.15.69:3300/0 conn(0x7fb46c06c410 msgr2=0x7fb46c0cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 --2- 172.21.15.43:0/440575400 >> v2:172.21.15.69:3300/0 conn(0x7fb46c06c410 0x7fb46c0cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0cb8c0 con 0x7fb46c06b880 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb454001670 con 0x7fb46c06b880 2024-01-16T15:00:21.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 >> v1:172.21.15.43:6789/0 conn(0x7fb46c06b880 legacy=0x7fb46c06bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb44c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c003fb0 0x7fb44c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb44c004d00 0x7fb44c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0d05e0 con 0x7fb44c004d00 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0cb5e0 con 0x7fb44c003fb0 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c00ba80 con 0x7fb46c15d550 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46be00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb44c004d00 0x7fb44c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.797+0000 7fb46b400640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb44c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46aa00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c003fb0 0x7fb44c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46be00640 1 -- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 msgr2=0x7fb44c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46be00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c15d550 0x7fb44c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46be00640 1 -- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c003fb0 msgr2=0x7fb44c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46be00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c003fb0 0x7fb44c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.798+0000 7fb46be00640 1 -- 172.21.15.43:0/440575400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c003520 con 0x7fb44c004d00 2024-01-16T15:00:21.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.799+0000 7fb46be00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb44c004d00 0x7fb44c005180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb464009f90 tx=0x7fb44c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110d65de4900a0f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.799+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46402cc20 con 0x7fb44c004d00 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.799+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46400f040 con 0x7fb44c004d00 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.799+0000 7fb46a000640 1 -- 172.21.15.43:0/440575400 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464034440 con 0x7fb44c004d00 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 -- 172.21.15.43:0/440575400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb44c004d00 msgr2=0x7fb44c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 --2- 172.21.15.43:0/440575400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb44c004d00 0x7fb44c005180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb464009f90 tx=0x7fb44c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb46be00640 1 -- 172.21.15.43:0/440575400 reap_dead start 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 -- 172.21.15.43:0/440575400 shutdown_connections 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 -- 172.21.15.43:0/440575400 >> 172.21.15.43:0/440575400 conn(0x7fb46c0d0e60 msgr2=0x7fb46c0d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 -- 172.21.15.43:0/440575400 shutdown_connections 2024-01-16T15:00:21.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.800+0000 7fb470e00640 1 -- 172.21.15.43:0/440575400 wait complete. 2024-01-16T15:00:21.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.802+0000 7fb470e00640 1 Processor -- start 2024-01-16T15:00:21.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.802+0000 7fb470e00640 1 -- start start 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c004d00 0x7fb46c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c0c9400 0x7fb46c0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0cae70 con 0x7fb46c15d930 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0ca1f0 con 0x7fb44c004d00 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb470e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0ca4d0 con 0x7fb46c0c9400 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38594/0 (socket says 172.21.15.43:38594) 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 -- 172.21.15.43:0/2944888629 learned_addr learned my addr 172.21.15.43:0/2944888629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46aa00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c0c9400 0x7fb46c0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46b400640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c004d00 0x7fb46c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 -- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c0c9400 msgr2=0x7fb46c0c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c0c9400 0x7fb46c0c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 -- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c004d00 msgr2=0x7fb46c0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c004d00 0x7fb46c0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.803+0000 7fb46be00640 1 -- 172.21.15.43:0/2944888629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0ca7b0 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb46be00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb464037040 tx=0x7fb464009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464034d50 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb464003990 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c0caa90 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464003d20 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.804+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c0ce580 con 0x7fb46c15d930 2024-01-16T15:00:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.805+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c0cafb0 con 0x7fb46c15d930 2024-01-16T15:00:21.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.806+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb464004030 con 0x7fb46c15d930 2024-01-16T15:00:21.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.807+0000 7fb463400640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 0x7fb44406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:21.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.807+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb4640ab260 con 0x7fb46c15d930 2024-01-16T15:00:21.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.808+0000 7fb46b400640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 0x7fb44406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:21.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.808+0000 7fb46b400640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 0x7fb44406d7b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009a10 tx=0x7fb45c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:21.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:21.812+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb46c0cafb0 con 0x7fb46c15d930 2024-01-16T15:00:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.011+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb46c0ce580 con 0x7fb46c15d930 2024-01-16T15:00:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.013+0000 7fb463400640 1 -- 172.21.15.43:0/2944888629 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7fb46c0ce580 con 0x7fb46c15d930 2024-01-16T15:00:21.995 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:21.995 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 msgr2=0x7fb44406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 0x7fb44406d7b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009a10 tx=0x7fb45c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 msgr2=0x7fb46c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb464037040 tx=0x7fb464009750 comp rx=0 tx=0).stop 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 shutdown_connections 2024-01-16T15:00:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb44406b2f0 0x7fb44406d7b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb46c15d930 0x7fb46c15ddb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb46c0c9400 0x7fb46c0c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 --2- 172.21.15.43:0/2944888629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb44c004d00 0x7fb46c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.016+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 >> 172.21.15.43:0/2944888629 conn(0x7fb46c0d0e60 msgr2=0x7fb46c0ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.017+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 shutdown_connections 2024-01-16T15:00:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.017+0000 7fb470e00640 1 -- 172.21.15.43:0/2944888629 wait complete. 2024-01-16T15:00:22.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:22.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.254+0000 7fe79cc00640 1 Processor -- start 2024-01-16T15:00:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.254+0000 7fe79cc00640 1 -- start start 2024-01-16T15:00:22.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe79cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe79815d3b0 0x7fe7980ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe79cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 0x7fe7980d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe79cc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980c6bf0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe79cc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7980d1430 con 0x7fe79815d3b0 2024-01-16T15:00:22.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe79cc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7980d1fa0 con 0x7fe7980d0fa0 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.256+0000 7fe797400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe79815d3b0 0x7fe7980ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe79815d3b0 0x7fe7980ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38596/0 (socket says 172.21.15.43:38596) 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 -- 172.21.15.43:0/284006888 learned_addr learned my addr 172.21.15.43:0/284006888 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe78fe00640 1 --2- 172.21.15.43:0/284006888 >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 0x7fe7980d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 -- 172.21.15.43:0/284006888 >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 msgr2=0x7fe7980d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 --2- 172.21.15.43:0/284006888 >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 0x7fe7980d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 -- 172.21.15.43:0/284006888 >> v1:172.21.15.43:6789/0 conn(0x7fe7980d1bc0 legacy=0x7fe7980b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe796a00640 1 -- 172.21.15.43:0/284006888 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976602842 0 0) 0x7fe7980c6bf0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe797400640 1 -- 172.21.15.43:0/284006888 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980cedc0 con 0x7fe79815d3b0 2024-01-16T15:00:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.257+0000 7fe78fe00640 1 --2- 172.21.15.43:0/284006888 >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 0x7fe7980d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.259+0000 7fe797400640 1 --2- 172.21.15.43:0/284006888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe79815d3b0 0x7fe7980ce6c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe79808c4a0 tx=0x7fe78802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e72d5381051c58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.259+0000 7fe796a00640 1 -- 172.21.15.43:0/284006888 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788031870 con 0x7fe79815d3b0 2024-01-16T15:00:22.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe796a00640 1 -- 172.21.15.43:0/284006888 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7880319d0 con 0x7fe79815d3b0 2024-01-16T15:00:22.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe796a00640 1 -- 172.21.15.43:0/284006888 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788031d60 con 0x7fe79815d3b0 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 -- 172.21.15.43:0/284006888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe79815d3b0 msgr2=0x7fe7980ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 --2- 172.21.15.43:0/284006888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe79815d3b0 0x7fe7980ce6c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe79808c4a0 tx=0x7fe78802b910 comp rx=0 tx=0).stop 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 -- 172.21.15.43:0/284006888 shutdown_connections 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 --2- 172.21.15.43:0/284006888 >> v2:172.21.15.69:3300/0 conn(0x7fe7980d0fa0 0x7fe7980d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 --2- 172.21.15.43:0/284006888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe79815d3b0 0x7fe7980ce6c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 -- 172.21.15.43:0/284006888 >> 172.21.15.43:0/284006888 conn(0x7fe7980d30a0 msgr2=0x7fe7980d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 -- 172.21.15.43:0/284006888 shutdown_connections 2024-01-16T15:00:22.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.260+0000 7fe79cc00640 1 -- 172.21.15.43:0/284006888 wait complete. 2024-01-16T15:00:22.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.262+0000 7fe79cc00640 1 Processor -- start 2024-01-16T15:00:22.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.262+0000 7fe79cc00640 1 -- start start 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 0x7fe79814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 0x7fe7980b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79815a4a0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79815a780 con 0x7fe79815d3b0 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe79cc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79815aa60 con 0x7fe7980d0fa0 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe78fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe78fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38610/0 (socket says 172.21.15.43:38610) 2024-01-16T15:00:22.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.263+0000 7fe78fe00640 1 -- 172.21.15.43:0/2914657997 learned_addr learned my addr 172.21.15.43:0/2914657997 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe797e00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 0x7fe7980b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 -- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 msgr2=0x7fe79814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe797400640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 0x7fe79814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 0x7fe79814adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 -- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 msgr2=0x7fe7980b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 0x7fe7980b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 -- 172.21.15.43:0/2914657997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7880094d0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe797e00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 0x7fe7980b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe797400640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 0x7fe79814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.264+0000 7fe78fe00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe78000d7d0 tx=0x7fe78000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.265+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780004030 con 0x7fe7980d1bc0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.265+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780004190 con 0x7fe7980d1bc0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.265+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780004520 con 0x7fe7980d1bc0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.265+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79815ad40 con 0x7fe7980d1bc0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.265+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79815a100 con 0x7fe7980d1bc0 2024-01-16T15:00:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.266+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79804e840 con 0x7fe7980d1bc0 2024-01-16T15:00:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.267+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe7800096a0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.268+0000 7fe78f400640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 0x7fe77006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.268+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe780089d90 con 0x7fe7980d1bc0 2024-01-16T15:00:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.268+0000 7fe797400640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 0x7fe77006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.269+0000 7fe797400640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 0x7fe77006d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe788036000 tx=0x7fe78803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.273+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe780054b80 con 0x7fe7980d1bc0 2024-01-16T15:00:22.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.465+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe79806b750 con 0x7fe7980d1bc0 2024-01-16T15:00:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.466+0000 7fe78f400640 1 -- 172.21.15.43:0/2914657997 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7800581a0 con 0x7fe7980d1bc0 2024-01-16T15:00:22.447 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 msgr2=0x7fe77006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 0x7fe77006d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe788036000 tx=0x7fe78803c040 comp rx=0 tx=0).stop 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 msgr2=0x7fe798159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe78000d7d0 tx=0x7fe78000dc90 comp rx=0 tx=0).stop 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 shutdown_connections 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe77006b2f0 0x7fe77006d7b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe79815d3b0 0x7fe7980b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7980d1bc0 0x7fe798159bc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 --2- 172.21.15.43:0/2914657997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7980d0fa0 0x7fe79814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 >> 172.21.15.43:0/2914657997 conn(0x7fe7980d30a0 msgr2=0x7fe798160240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 shutdown_connections 2024-01-16T15:00:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.469+0000 7fe79cc00640 1 -- 172.21.15.43:0/2914657997 wait complete. 2024-01-16T15:00:22.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:22.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.719+0000 7fcfc8200640 1 Processor -- start 2024-01-16T15:00:22.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.720+0000 7fcfc8200640 1 -- start start 2024-01-16T15:00:22.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.721+0000 7fcfc8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 0x7fcfc00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.721+0000 7fcfc8200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcfc00adb60 0x7fcfc00adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.721+0000 7fcfc8200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc015b730 con 0x7fcfc015b350 2024-01-16T15:00:22.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc8200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00ac740 con 0x7fcfc00adb60 2024-01-16T15:00:22.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00ac460 con 0x7fcfc009cb30 2024-01-16T15:00:22.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc5800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcfc015b350 0x7fcfc00b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:37304/0 (socket says 172.21.15.43:37304) 2024-01-16T15:00:22.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc5800640 1 -- 172.21.15.43:0/1552013647 learned_addr learned my addr 172.21.15.43:0/1552013647 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:22.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/1552013647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 0x7fcfc00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.722+0000 7fcfc6200640 1 --2- 172.21.15.43:0/1552013647 >> v2:172.21.15.69:3300/0 conn(0x7fcfc00adb60 0x7fcfc00adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.723+0000 7fcfbfe00640 1 -- 172.21.15.43:0/1552013647 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191483167 0 0) 0x7fcfb0001110 con 0x7fcfc015b350 2024-01-16T15:00:22.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.723+0000 7fcfc4e00640 1 -- 172.21.15.43:0/1552013647 >> v2:172.21.15.69:3300/0 conn(0x7fcfc00adb60 msgr2=0x7fcfc00adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:22.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.723+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/1552013647 >> v2:172.21.15.69:3300/0 conn(0x7fcfc00adb60 0x7fcfc00adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.723+0000 7fcfc4e00640 1 -- 172.21.15.43:0/1552013647 >> v1:172.21.15.69:6789/0 conn(0x7fcfc015b350 legacy=0x7fcfc00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.723+0000 7fcfc4e00640 1 -- 172.21.15.43:0/1552013647 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00b2a50 con 0x7fcfc009cb30 2024-01-16T15:00:22.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.725+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/1552013647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 0x7fcfc00abe80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80097a0 tx=0x7fcfa802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=120469d6776fafa3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.725+0000 7fcfbfe00640 1 -- 172.21.15.43:0/1552013647 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8031830 con 0x7fcfc009cb30 2024-01-16T15:00:22.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.725+0000 7fcfbfe00640 1 -- 172.21.15.43:0/1552013647 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa8031990 con 0x7fcfc009cb30 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.725+0000 7fcfbfe00640 1 -- 172.21.15.43:0/1552013647 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8031d20 con 0x7fcfc009cb30 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 -- 172.21.15.43:0/1552013647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 msgr2=0x7fcfc00abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 --2- 172.21.15.43:0/1552013647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 0x7fcfc00abe80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80097a0 tx=0x7fcfa802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 -- 172.21.15.43:0/1552013647 shutdown_connections 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 --2- 172.21.15.43:0/1552013647 >> v2:172.21.15.69:3300/0 conn(0x7fcfc00adb60 0x7fcfc00adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 --2- 172.21.15.43:0/1552013647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc009cb30 0x7fcfc00abe80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 -- 172.21.15.43:0/1552013647 >> 172.21.15.43:0/1552013647 conn(0x7fcfc00b0ba0 msgr2=0x7fcfc00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 -- 172.21.15.43:0/1552013647 shutdown_connections 2024-01-16T15:00:22.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.726+0000 7fcfc8200640 1 -- 172.21.15.43:0/1552013647 wait complete. 2024-01-16T15:00:22.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.728+0000 7fcfc8200640 1 Processor -- start 2024-01-16T15:00:22.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.728+0000 7fcfc8200640 1 -- start start 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfc009cb30 0x7fcfc0150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc015b350 0x7fcfc0147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01514b0 con 0x7fcfc00adb60 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01511d0 con 0x7fcfc009cb30 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0070f30 con 0x7fcfc015b350 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.729+0000 7fcfc4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38614/0 (socket says 172.21.15.43:38614) 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 -- 172.21.15.43:0/555965546 learned_addr learned my addr 172.21.15.43:0/555965546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc5800640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfc009cb30 0x7fcfc0150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc6200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc015b350 0x7fcfc0147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 -- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc015b350 msgr2=0x7fcfc0147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc015b350 0x7fcfc0147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 -- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfc009cb30 msgr2=0x7fcfc0150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfc009cb30 0x7fcfc0150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 -- 172.21.15.43:0/555965546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8009400 con 0x7fcfc00adb60 2024-01-16T15:00:22.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.730+0000 7fcfc4e00640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8034040 tx=0x7fcfa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.731+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa803a8f0 con 0x7fcfc00adb60 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.731+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa803aa70 con 0x7fcfc00adb60 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.731+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc0071210 con 0x7fcfc00adb60 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.731+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa803ae20 con 0x7fcfc00adb60 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.731+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc0150ec0 con 0x7fcfc00adb60 2024-01-16T15:00:22.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.732+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc0070d90 con 0x7fcfc00adb60 2024-01-16T15:00:22.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.733+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcfa804d440 con 0x7fcfc00adb60 2024-01-16T15:00:22.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.734+0000 7fcfbd600640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 0x7fcf9806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.734+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcfa80abd80 con 0x7fcfc00adb60 2024-01-16T15:00:22.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.734+0000 7fcfc5800640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 0x7fcf9806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.735+0000 7fcfc5800640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 0x7fcf9806d7d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0000f60 tx=0x7fcfb00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.739+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfc0070d90 con 0x7fcfc00adb60 2024-01-16T15:00:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.938+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfc0071730 con 0x7fcfc00adb60 2024-01-16T15:00:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.939+0000 7fcfbd600640 1 -- 172.21.15.43:0/555965546 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcfc0071730 con 0x7fcfc00adb60 2024-01-16T15:00:22.920 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 msgr2=0x7fcf9806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 0x7fcf9806d7d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0000f60 tx=0x7fcfb00079b0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 msgr2=0x7fcfc0150980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8034040 tx=0x7fcfa80046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 shutdown_connections 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf9806b310 0x7fcf9806d7d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfc015b350 0x7fcfc0147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfc00adb60 0x7fcfc0150980 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 --2- 172.21.15.43:0/555965546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfc009cb30 0x7fcfc0150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 >> 172.21.15.43:0/555965546 conn(0x7fcfc00b0ba0 msgr2=0x7fcfc00af4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 shutdown_connections 2024-01-16T15:00:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:22.942+0000 7fcfc8200640 1 -- 172.21.15.43:0/555965546 wait complete. 2024-01-16T15:00:22.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:23.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.194+0000 7fbd16800640 1 Processor -- start 2024-01-16T15:00:23.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.194+0000 7fbd16800640 1 -- start start 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.195+0000 7fbd16800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd100cefb0 0x7fbd100acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.195+0000 7fbd16800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd100cf440 con 0x7fbd100cf600 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.195+0000 7fbd16800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1006ee00 con 0x7fbd1006a440 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.195+0000 7fbd16800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd1006fa40 con 0x7fbd100cefb0 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd100cefb0 0x7fbd100acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd100cefb0 0x7fbd100acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37628/0 (socket says 172.21.15.43:37628) 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 -- 172.21.15.43:0/2562818499 learned_addr learned my addr 172.21.15.43:0/2562818499 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:23.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd15e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbd100cf600 0x7fbd100ad310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58414/0 (socket says 172.21.15.43:58414) 2024-01-16T15:00:23.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 -- 172.21.15.43:0/2562818499 >> v1:172.21.15.69:6789/0 conn(0x7fbd1006a440 legacy=0x7fbd1006a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 -- 172.21.15.43:0/2562818499 >> v1:172.21.15.43:6789/0 conn(0x7fbd100cf600 legacy=0x7fbd100ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd14a00640 1 -- 172.21.15.43:0/2562818499 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1006f0a0 con 0x7fbd100cefb0 2024-01-16T15:00:23.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.196+0000 7fbd0fe00640 1 -- 172.21.15.43:0/2562818499 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034406652 0 0) 0x7fbd08001110 con 0x7fbd100cf600 2024-01-16T15:00:23.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.197+0000 7fbd14a00640 1 --2- 172.21.15.43:0/2562818499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd100cefb0 0x7fbd100acdd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009870 tx=0x7fbcf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b8d4d59e576d21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd0fe00640 1 -- 172.21.15.43:0/2562818499 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8030860 con 0x7fbd100cefb0 2024-01-16T15:00:23.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd0fe00640 1 -- 172.21.15.43:0/2562818499 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf80309c0 con 0x7fbd100cefb0 2024-01-16T15:00:23.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd0fe00640 1 -- 172.21.15.43:0/2562818499 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8030d50 con 0x7fbd100cefb0 2024-01-16T15:00:23.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd16800640 1 -- 172.21.15.43:0/2562818499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd100cefb0 msgr2=0x7fbd100acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd16800640 1 --2- 172.21.15.43:0/2562818499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd100cefb0 0x7fbd100acdd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009870 tx=0x7fbcf802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd16800640 1 -- 172.21.15.43:0/2562818499 shutdown_connections 2024-01-16T15:00:23.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd16800640 1 --2- 172.21.15.43:0/2562818499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd100cefb0 0x7fbd100acdd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.198+0000 7fbd16800640 1 -- 172.21.15.43:0/2562818499 >> 172.21.15.43:0/2562818499 conn(0x7fbd100cfd80 msgr2=0x7fbd100d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:23.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.199+0000 7fbd16800640 1 -- 172.21.15.43:0/2562818499 shutdown_connections 2024-01-16T15:00:23.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.199+0000 7fbd16800640 1 -- 172.21.15.43:0/2562818499 wait complete. 2024-01-16T15:00:23.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.200+0000 7fbd16800640 1 Processor -- start 2024-01-16T15:00:23.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.200+0000 7fbd16800640 1 -- start start 2024-01-16T15:00:23.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 0x7fbd101499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 0x7fbd10155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10153630 con 0x7fbd1006a440 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10149f00 con 0x7fbd101f14f0 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.201+0000 7fbd16800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd101f1a40 con 0x7fbd100cf600 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37632/0 (socket says 172.21.15.43:37632) 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 -- 172.21.15.43:0/220611984 learned_addr learned my addr 172.21.15.43:0/220611984 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15e00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 0x7fbd10155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd14a00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 0x7fbd101499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 -- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 msgr2=0x7fbd101499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 0x7fbd101499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 -- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 msgr2=0x7fbd10155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 0x7fbd10155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 -- 172.21.15.43:0/220611984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf80094d0 con 0x7fbd1006a440 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15e00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 0x7fbd10155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd14a00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 0x7fbd101499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.202+0000 7fbd15400640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c870 tx=0x7fbd0000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.203+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00003b70 con 0x7fbd1006a440 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.203+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00003cf0 con 0x7fbd1006a440 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.203+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0000aa90 con 0x7fbd1006a440 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.203+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd101f1d40 con 0x7fbd1006a440 2024-01-16T15:00:23.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.203+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd101f2300 con 0x7fbd1006a440 2024-01-16T15:00:23.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.204+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10066f80 con 0x7fbd1006a440 2024-01-16T15:00:23.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.205+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbd0000abf0 con 0x7fbd1006a440 2024-01-16T15:00:23.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.206+0000 7fbd0d600640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 0x7fbce806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.206+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbd00089b10 con 0x7fbd1006a440 2024-01-16T15:00:23.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.206+0000 7fbd14a00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 0x7fbce806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.207+0000 7fbd14a00640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 0x7fbce806d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8036000 tx=0x7fbcf803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.210+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd00054900 con 0x7fbd1006a440 2024-01-16T15:00:23.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.411+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd1004e840 con 0x7fbd1006a440 2024-01-16T15:00:23.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.411+0000 7fbd0d600640 1 -- 172.21.15.43:0/220611984 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd00003e60 con 0x7fbd1006a440 2024-01-16T15:00:23.393 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 msgr2=0x7fbce806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 0x7fbce806d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8036000 tx=0x7fbcf803b040 comp rx=0 tx=0).stop 2024-01-16T15:00:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 msgr2=0x7fbd101530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c870 tx=0x7fbd0000cd30 comp rx=0 tx=0).stop 2024-01-16T15:00:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 shutdown_connections 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce806b2c0 0x7fbce806d780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd101f14f0 0x7fbd10155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd100cf600 0x7fbd101499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 --2- 172.21.15.43:0/220611984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd1006a440 0x7fbd101530f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.414+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 >> 172.21.15.43:0/220611984 conn(0x7fbd100cfd80 msgr2=0x7fbd100ad5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.415+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 shutdown_connections 2024-01-16T15:00:23.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.415+0000 7fbd16800640 1 -- 172.21.15.43:0/220611984 wait complete. 2024-01-16T15:00:23.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.662+0000 7fbd2e000640 1 Processor -- start 2024-01-16T15:00:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.662+0000 7fbd2e000640 1 -- start start 2024-01-16T15:00:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.663+0000 7fbd2e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd2815d5d0 0x7fbd280ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 0x7fbd280cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd280d04b0 con 0x7fbd280d1670 2024-01-16T15:00:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2e000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd280ce4e0 con 0x7fbd2815d5d0 2024-01-16T15:00:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280cdb80 con 0x7fbd280cf710 2024-01-16T15:00:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd2815d5d0 0x7fbd280ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd2815d5d0 0x7fbd280ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37634/0 (socket says 172.21.15.43:37634) 2024-01-16T15:00:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 -- 172.21.15.43:0/861269648 learned_addr learned my addr 172.21.15.43:0/861269648 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd27e00640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 0x7fbd280cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 -- 172.21.15.43:0/861269648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 msgr2=0x7fbd280cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 0x7fbd280cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 -- 172.21.15.43:0/861269648 >> v1:172.21.15.69:6789/0 conn(0x7fbd280d1670 legacy=0x7fbd280d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd2cc00640 1 -- 172.21.15.43:0/861269648 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd280d01d0 con 0x7fbd2815d5d0 2024-01-16T15:00:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.664+0000 7fbd27e00640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 0x7fbd280cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:23.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.666+0000 7fbd2cc00640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd2815d5d0 0x7fbd280ce900 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009870 tx=0x7fbd1802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa006e655cdc558 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.666+0000 7fbd27400640 1 -- 172.21.15.43:0/861269648 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd180317c0 con 0x7fbd2815d5d0 2024-01-16T15:00:23.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.666+0000 7fbd27400640 1 -- 172.21.15.43:0/861269648 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18031920 con 0x7fbd2815d5d0 2024-01-16T15:00:23.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.666+0000 7fbd27400640 1 -- 172.21.15.43:0/861269648 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18031cb0 con 0x7fbd2815d5d0 2024-01-16T15:00:23.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 -- 172.21.15.43:0/861269648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd2815d5d0 msgr2=0x7fbd280ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd2815d5d0 0x7fbd280ce900 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009870 tx=0x7fbd1802b860 comp rx=0 tx=0).stop 2024-01-16T15:00:23.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 -- 172.21.15.43:0/861269648 shutdown_connections 2024-01-16T15:00:23.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd280cf710 0x7fbd280cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 --2- 172.21.15.43:0/861269648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd2815d5d0 0x7fbd280ce900 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 -- 172.21.15.43:0/861269648 >> 172.21.15.43:0/861269648 conn(0x7fbd2809c3d0 msgr2=0x7fbd280d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 -- 172.21.15.43:0/861269648 shutdown_connections 2024-01-16T15:00:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.667+0000 7fbd2e000640 1 -- 172.21.15.43:0/861269648 wait complete. 2024-01-16T15:00:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.669+0000 7fbd2e000640 1 Processor -- start 2024-01-16T15:00:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.669+0000 7fbd2e000640 1 -- start start 2024-01-16T15:00:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd280cf710 0x7fbd280ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 0x7fbd2806c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280cd110 con 0x7fbd280d1670 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280c99b0 con 0x7fbd280cf710 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280cc590 con 0x7fbd2815d5d0 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37642/0 (socket says 172.21.15.43:37642) 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd27e00640 1 -- 172.21.15.43:0/2401396821 learned_addr learned my addr 172.21.15.43:0/2401396821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.671+0000 7fbd2cc00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd280cf710 0x7fbd280ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd2d600640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 0x7fbd2806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 -- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 msgr2=0x7fbd2806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 0x7fbd2806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 -- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd280cf710 msgr2=0x7fbd280ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd280cf710 0x7fbd280ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 -- 172.21.15.43:0/2401396821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180094d0 con 0x7fbd280d1670 2024-01-16T15:00:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd2d600640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 0x7fbd2806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd27e00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c7f0 tx=0x7fbd1000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10007820 con 0x7fbd280d1670 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd10007980 con 0x7fbd280d1670 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10007cf0 con 0x7fbd280d1670 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.672+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd280cc870 con 0x7fbd280d1670 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.673+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280cce30 con 0x7fbd280d1670 2024-01-16T15:00:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.674+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2804e840 con 0x7fbd280d1670 2024-01-16T15:00:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.675+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbd100113f0 con 0x7fbd280d1670 2024-01-16T15:00:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.676+0000 7fbd24c00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 0x7fbd0006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.676+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbd10089f20 con 0x7fbd280d1670 2024-01-16T15:00:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.676+0000 7fbd2cc00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 0x7fbd0006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.677+0000 7fbd2cc00640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 0x7fbd0006d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.681+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd10054d80 con 0x7fbd280d1670 2024-01-16T15:00:23.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:23.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.898+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd280678d0 con 0x7fbd280d1670 2024-01-16T15:00:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.898+0000 7fbd24c00640 1 -- 172.21.15.43:0/2401396821 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbd1001f080 con 0x7fbd280d1670 2024-01-16T15:00:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:23.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.900+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 msgr2=0x7fbd0006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.900+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 0x7fbd0006d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803d040 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.900+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 msgr2=0x7fbd280c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.900+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c7f0 tx=0x7fbd1000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 shutdown_connections 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd0006b2c0 0x7fbd0006d780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd2815d5d0 0x7fbd2806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd280d1670 0x7fbd280c9470 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 --2- 172.21.15.43:0/2401396821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd280cf710 0x7fbd280ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 >> 172.21.15.43:0/2401396821 conn(0x7fbd2809c3d0 msgr2=0x7fbd280b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 shutdown_connections 2024-01-16T15:00:23.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:23.901+0000 7fbd2e000640 1 -- 172.21.15.43:0/2401396821 wait complete. 2024-01-16T15:00:23.899 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:00:23.899 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:23.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.277+0000 7f2a2be00640 1 Processor -- start 2024-01-16T15:00:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.278+0000 7f2a2be00640 1 -- start start 2024-01-16T15:00:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.279+0000 7f2a2be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a2c06d250 0x7f2a2c15d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.279+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c15ddb0 0x7f2a2c070720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.279+0000 7f2a2be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2c161670 con 0x7f2a2c157770 2024-01-16T15:00:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.279+0000 7f2a2be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2c15e610 con 0x7f2a2c06d250 2024-01-16T15:00:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.279+0000 7f2a2be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c160d40 con 0x7f2a2c15ddb0 2024-01-16T15:00:24.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c15ddb0 0x7f2a2c070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2a2c157770 0x7f2a2c06cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:49074/0 (socket says 172.21.15.43:49074) 2024-01-16T15:00:24.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2aa00640 1 -- 172.21.15.43:0/3701462397 learned_addr learned my addr 172.21.15.43:0/3701462397 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2a000640 1 --2- 172.21.15.43:0/3701462397 >> v2:172.21.15.69:3300/0 conn(0x7f2a2c06d250 0x7f2a2c15d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2a000640 1 -- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c15ddb0 msgr2=0x7f2a2c070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2a000640 1 --2- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c15ddb0 0x7f2a2c070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2a000640 1 -- 172.21.15.43:0/3701462397 >> v1:172.21.15.69:6789/0 conn(0x7f2a2c157770 legacy=0x7f2a2c06cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.282+0000 7f2a2a000640 1 -- 172.21.15.43:0/3701462397 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c161020 con 0x7f2a2c06d250 2024-01-16T15:00:24.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.283+0000 7f2a2a000640 1 --2- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c06d250 0x7f2a2c15d870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a140097a0 tx=0x7f2a1402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e5556550fda2cfd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.283+0000 7f2a29600640 1 -- 172.21.15.43:0/3701462397 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14031830 con 0x7f2a2c06d250 2024-01-16T15:00:24.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.283+0000 7f2a29600640 1 -- 172.21.15.43:0/3701462397 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a14031990 con 0x7f2a2c06d250 2024-01-16T15:00:24.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.283+0000 7f2a29600640 1 -- 172.21.15.43:0/3701462397 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14031d20 con 0x7f2a2c06d250 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 -- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c06d250 msgr2=0x7f2a2c15d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c06d250 0x7f2a2c15d870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a140097a0 tx=0x7f2a1402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 -- 172.21.15.43:0/3701462397 shutdown_connections 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c06d250 0x7f2a2c15d870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3701462397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c15ddb0 0x7f2a2c070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 -- 172.21.15.43:0/3701462397 >> 172.21.15.43:0/3701462397 conn(0x7f2a2c06a860 msgr2=0x7f2a2c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 -- 172.21.15.43:0/3701462397 shutdown_connections 2024-01-16T15:00:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.284+0000 7f2a2be00640 1 -- 172.21.15.43:0/3701462397 wait complete. 2024-01-16T15:00:24.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.286+0000 7f2a2be00640 1 Processor -- start 2024-01-16T15:00:24.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.286+0000 7f2a2be00640 1 -- start start 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c157770 0x7f2a2c0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c15ddb0 0x7f2a2c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0b2190 con 0x7f2a2c06d250 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0b2470 con 0x7f2a2c15ddb0 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0b2750 con 0x7f2a2c157770 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.287+0000 7f2a2aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37646/0 (socket says 172.21.15.43:37646) 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 -- 172.21.15.43:0/2170507088 learned_addr learned my addr 172.21.15.43:0/2170507088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2a000640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c157770 0x7f2a2c0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2b400640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c15ddb0 0x7f2a2c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 -- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c157770 msgr2=0x7f2a2c0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c157770 0x7f2a2c0b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 -- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c15ddb0 msgr2=0x7f2a2c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c15ddb0 0x7f2a2c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 -- 172.21.15.43:0/2170507088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a14009400 con 0x7f2a2c06d250 2024-01-16T15:00:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.288+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c4a0 tx=0x7f2a1c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.289+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c005ce0 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.289+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c005e40 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.289+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c161c80 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.289+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c007270 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.289+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c0b2d10 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.290+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c0b2f00 con 0x7f2a2c06d250 2024-01-16T15:00:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.291+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2a1c007490 con 0x7f2a2c06d250 2024-01-16T15:00:24.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.292+0000 7f2a22a00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 0x7f2a0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.292+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2a1c089000 con 0x7f2a2c06d250 2024-01-16T15:00:24.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.292+0000 7f2a2a000640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 0x7f2a0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.293+0000 7f2a2a000640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 0x7f2a0406d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a14036000 tx=0x7f2a1403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.296+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a1c053f30 con 0x7f2a2c06d250 2024-01-16T15:00:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.490+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a2c04e840 con 0x7f2a2c06d250 2024-01-16T15:00:24.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.493+0000 7f2a22a00640 1 -- 172.21.15.43:0/2170507088 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f2a1c017090 con 0x7f2a2c06d250 2024-01-16T15:00:24.474 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:24.474 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.495+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 msgr2=0x7f2a0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.495+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 0x7f2a0406d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a14036000 tx=0x7f2a1403c040 comp rx=0 tx=0).stop 2024-01-16T15:00:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.495+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 msgr2=0x7f2a2c141550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.495+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c4a0 tx=0x7f2a1c00c960 comp rx=0 tx=0).stop 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 shutdown_connections 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a0406b2c0 0x7f2a0406d780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c15ddb0 0x7f2a2c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a2c157770 0x7f2a2c0b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 --2- 172.21.15.43:0/2170507088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c06d250 0x7f2a2c141550 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 >> 172.21.15.43:0/2170507088 conn(0x7f2a2c06a860 msgr2=0x7f2a2c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 shutdown_connections 2024-01-16T15:00:24.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.496+0000 7f2a2be00640 1 -- 172.21.15.43:0/2170507088 wait complete. 2024-01-16T15:00:24.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:24.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:24.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.742+0000 7f4913e00640 1 Processor -- start 2024-01-16T15:00:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.742+0000 7f4913e00640 1 -- start start 2024-01-16T15:00:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.743+0000 7f4913e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f491415d3b0 0x7f49140ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.743+0000 7f4913e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49140d1d90 0x7f49140b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.743+0000 7f4913e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49140c6bf0 con 0x7f49140d0fa0 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.743+0000 7f4913e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49140d1430 con 0x7f491415d3b0 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.743+0000 7f4913e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49140cedc0 con 0x7f49140d1d90 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f491415d3b0 0x7f49140ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f491415d3b0 0x7f49140ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37650/0 (socket says 172.21.15.43:37650) 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 -- 172.21.15.43:0/2261123720 learned_addr learned my addr 172.21.15.43:0/2261123720 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4913400640 1 --2- 172.21.15.43:0/2261123720 >> v2:172.21.15.69:3300/0 conn(0x7f49140d1d90 0x7f49140b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 -- 172.21.15.43:0/2261123720 >> v2:172.21.15.69:3300/0 conn(0x7f49140d1d90 msgr2=0x7f49140b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 --2- 172.21.15.43:0/2261123720 >> v2:172.21.15.69:3300/0 conn(0x7f49140d1d90 0x7f49140b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 -- 172.21.15.43:0/2261123720 >> v1:172.21.15.69:6789/0 conn(0x7f49140d0fa0 legacy=0x7f49140d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.744+0000 7f4912a00640 1 -- 172.21.15.43:0/2261123720 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140c6e40 con 0x7f491415d3b0 2024-01-16T15:00:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.746+0000 7f4912a00640 1 --2- 172.21.15.43:0/2261123720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415d3b0 0x7f49140ce6c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4914072ae0 tx=0x7f490402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5835e15091e0b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.746+0000 7f4911600640 1 -- 172.21.15.43:0/2261123720 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904031870 con 0x7f491415d3b0 2024-01-16T15:00:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.746+0000 7f4911600640 1 -- 172.21.15.43:0/2261123720 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49040319d0 con 0x7f491415d3b0 2024-01-16T15:00:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.746+0000 7f4911600640 1 -- 172.21.15.43:0/2261123720 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904031d60 con 0x7f491415d3b0 2024-01-16T15:00:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 -- 172.21.15.43:0/2261123720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415d3b0 msgr2=0x7f49140ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 --2- 172.21.15.43:0/2261123720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415d3b0 0x7f49140ce6c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4914072ae0 tx=0x7f490402b910 comp rx=0 tx=0).stop 2024-01-16T15:00:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 -- 172.21.15.43:0/2261123720 shutdown_connections 2024-01-16T15:00:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 --2- 172.21.15.43:0/2261123720 >> v2:172.21.15.69:3300/0 conn(0x7f49140d1d90 0x7f49140b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 --2- 172.21.15.43:0/2261123720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415d3b0 0x7f49140ce6c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 -- 172.21.15.43:0/2261123720 >> 172.21.15.43:0/2261123720 conn(0x7f49140d30a0 msgr2=0x7f49140d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 -- 172.21.15.43:0/2261123720 shutdown_connections 2024-01-16T15:00:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.747+0000 7f4913e00640 1 -- 172.21.15.43:0/2261123720 wait complete. 2024-01-16T15:00:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.749+0000 7f4913e00640 1 Processor -- start 2024-01-16T15:00:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.749+0000 7f4913e00640 1 -- start start 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140d1d90 0x7f491415b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491415d3b0 0x7f491406bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491415bc40 con 0x7f49140d0fa0 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140b6350 con 0x7f491415d3b0 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49140cbe80 con 0x7f49140d1d90 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4912a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4912a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37652/0 (socket says 172.21.15.43:37652) 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4912a00640 1 -- 172.21.15.43:0/944463660 learned_addr learned my addr 172.21.15.43:0/944463660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4912000640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140d1d90 0x7f491415b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.750+0000 7f4913400640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491415d3b0 0x7f491406bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 -- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140d1d90 msgr2=0x7f491415b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140d1d90 0x7f491415b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 -- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491415d3b0 msgr2=0x7f491406bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491415d3b0 0x7f491406bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 -- 172.21.15.43:0/944463660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040094d0 con 0x7f49140d0fa0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4912a00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f490402b950 tx=0x7f49040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490403abe0 con 0x7f49140d0fa0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490403ad60 con 0x7f49140d0fa0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904044630 con 0x7f49140d0fa0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.751+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49140cc160 con 0x7f49140d0fa0 2024-01-16T15:00:24.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.752+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49140b5fe0 con 0x7f49140d0fa0 2024-01-16T15:00:24.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.753+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491404e840 con 0x7f49140d0fa0 2024-01-16T15:00:24.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.754+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f490404d450 con 0x7f49140d0fa0 2024-01-16T15:00:24.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.754+0000 7f490aa00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 0x7f48ec06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:24.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.755+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f490402d040 con 0x7f49140d0fa0 2024-01-16T15:00:24.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.755+0000 7f4912000640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 0x7f48ec06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:24.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.756+0000 7f4912000640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 0x7f48ec06d7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f49140cc610 tx=0x7f48fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:24.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.759+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4904076e40 con 0x7f49140d0fa0 2024-01-16T15:00:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.963+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4914160680 con 0x7f49140d0fa0 2024-01-16T15:00:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.964+0000 7f490aa00640 1 -- 172.21.15.43:0/944463660 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4904035070 con 0x7f49140d0fa0 2024-01-16T15:00:24.945 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 msgr2=0x7f48ec06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 0x7f48ec06d7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f49140cc610 tx=0x7f48fc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 msgr2=0x7f4914159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f490402b950 tx=0x7f49040046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 shutdown_connections 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06b2f0 0x7f48ec06d7b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491415d3b0 0x7f491406bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140d1d90 0x7f491415b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 --2- 172.21.15.43:0/944463660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140d0fa0 0x7f4914159960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 >> 172.21.15.43:0/944463660 conn(0x7f49140d30a0 msgr2=0x7f49140d3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 shutdown_connections 2024-01-16T15:00:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:24.967+0000 7f4913e00640 1 -- 172.21.15.43:0/944463660 wait complete. 2024-01-16T15:00:24.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:25.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.217+0000 7fd7d3e00640 1 Processor -- start 2024-01-16T15:00:25.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.217+0000 7fd7d3e00640 1 -- start start 2024-01-16T15:00:25.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.219+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 0x7fd7d40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.219+0000 7fd7d3e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d415f930 con 0x7fd7d406d9e0 2024-01-16T15:00:25.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.219+0000 7fd7d3e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d40b3b10 con 0x7fd7d4144770 2024-01-16T15:00:25.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.219+0000 7fd7d3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d40b43b0 con 0x7fd7d415f270 2024-01-16T15:00:25.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.220+0000 7fd7d2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 0x7fd7d40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.220+0000 7fd7d3400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd7d406d9e0 0x7fd7d40cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58418/0 (socket says 172.21.15.43:58418) 2024-01-16T15:00:25.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.220+0000 7fd7d3400640 1 -- 172.21.15.43:0/2596789860 learned_addr learned my addr 172.21.15.43:0/2596789860 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:25.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.221+0000 7fd7d2000640 1 -- 172.21.15.43:0/2596789860 >> v1:172.21.15.69:6789/0 conn(0x7fd7d4144770 legacy=0x7fd7d415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.221+0000 7fd7d2000640 1 -- 172.21.15.43:0/2596789860 >> v1:172.21.15.43:6789/0 conn(0x7fd7d406d9e0 legacy=0x7fd7d40cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.221+0000 7fd7d2000640 1 -- 172.21.15.43:0/2596789860 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d40b3df0 con 0x7fd7d415f270 2024-01-16T15:00:25.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.221+0000 7fd7d1600640 1 -- 172.21.15.43:0/2596789860 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379268291 0 0) 0x7fd7d415f930 con 0x7fd7d406d9e0 2024-01-16T15:00:25.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.222+0000 7fd7d2000640 1 --2- 172.21.15.43:0/2596789860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 0x7fd7d40cd4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0097d0 tx=0x7fd7bc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f1ad8127a55b778 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.222+0000 7fd7d1600640 1 -- 172.21.15.43:0/2596789860 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc031890 con 0x7fd7d415f270 2024-01-16T15:00:25.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.222+0000 7fd7d1600640 1 -- 172.21.15.43:0/2596789860 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc0319f0 con 0x7fd7d415f270 2024-01-16T15:00:25.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.222+0000 7fd7d1600640 1 -- 172.21.15.43:0/2596789860 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc031d80 con 0x7fd7d415f270 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 -- 172.21.15.43:0/2596789860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 msgr2=0x7fd7d40cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/2596789860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 0x7fd7d40cd4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0097d0 tx=0x7fd7bc02b930 comp rx=0 tx=0).stop 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 -- 172.21.15.43:0/2596789860 shutdown_connections 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/2596789860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d415f270 0x7fd7d40cd4a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 -- 172.21.15.43:0/2596789860 >> 172.21.15.43:0/2596789860 conn(0x7fd7d40cec10 msgr2=0x7fd7d40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 -- 172.21.15.43:0/2596789860 shutdown_connections 2024-01-16T15:00:25.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.223+0000 7fd7d3e00640 1 -- 172.21.15.43:0/2596789860 wait complete. 2024-01-16T15:00:25.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.225+0000 7fd7d3e00640 1 Processor -- start 2024-01-16T15:00:25.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.226+0000 7fd7d3e00640 1 -- start start 2024-01-16T15:00:25.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d406d9e0 0x7fd7d414dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d415f270 0x7fd7d413b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d413f370 con 0x7fd7d4144770 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d413f610 con 0x7fd7d415f270 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d413f8f0 con 0x7fd7d406d9e0 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37660/0 (socket says 172.21.15.43:37660) 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d2000640 1 -- 172.21.15.43:0/3995662007 learned_addr learned my addr 172.21.15.43:0/3995662007 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.227+0000 7fd7d2a00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d406d9e0 0x7fd7d414dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d3400640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d415f270 0x7fd7d413b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 -- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d406d9e0 msgr2=0x7fd7d414dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d406d9e0 0x7fd7d414dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 -- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d415f270 msgr2=0x7fd7d413b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d415f270 0x7fd7d413b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 -- 172.21.15.43:0/3995662007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc009400 con 0x7fd7d4144770 2024-01-16T15:00:25.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.228+0000 7fd7d2000640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0097d0 tx=0x7fd7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.229+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc031830 con 0x7fd7d4144770 2024-01-16T15:00:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.229+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc0443f0 con 0x7fd7d4144770 2024-01-16T15:00:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.229+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0447a0 con 0x7fd7d4144770 2024-01-16T15:00:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.229+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d413fbd0 con 0x7fd7d4144770 2024-01-16T15:00:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.229+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d414b290 con 0x7fd7d4144770 2024-01-16T15:00:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.230+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d41400f0 con 0x7fd7d4144770 2024-01-16T15:00:25.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.234+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd7bc04d450 con 0x7fd7d4144770 2024-01-16T15:00:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.235+0000 7fd7caa00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 0x7fd7ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.236+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd7bc0ac370 con 0x7fd7d4144770 2024-01-16T15:00:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.236+0000 7fd7d2a00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 0x7fd7ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.236+0000 7fd7d2a00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 0x7fd7ac06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7d41418e0 tx=0x7fd7c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.242+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7d41400f0 con 0x7fd7d4144770 2024-01-16T15:00:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.437+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7d41400f0 con 0x7fd7d4144770 2024-01-16T15:00:25.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.438+0000 7fd7caa00640 1 -- 172.21.15.43:0/3995662007 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd7d41400f0 con 0x7fd7d4144770 2024-01-16T15:00:25.419 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 msgr2=0x7fd7ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 0x7fd7ac06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7d41418e0 tx=0x7fd7c4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 msgr2=0x7fd7d413b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0097d0 tx=0x7fd7bc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 shutdown_connections 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd7ac06b2c0 0x7fd7ac06d780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d415f270 0x7fd7d413b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7d4144770 0x7fd7d413b260 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 --2- 172.21.15.43:0/3995662007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7d406d9e0 0x7fd7d414dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 >> 172.21.15.43:0/3995662007 conn(0x7fd7d40cec10 msgr2=0x7fd7d406c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 shutdown_connections 2024-01-16T15:00:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.441+0000 7fd7d3e00640 1 -- 172.21.15.43:0/3995662007 wait complete. 2024-01-16T15:00:25.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:25.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.677+0000 7f06f0e00640 1 Processor -- start 2024-01-16T15:00:25.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.677+0000 7f06f0e00640 1 -- start start 2024-01-16T15:00:25.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.678+0000 7f06f0e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06ec144770 0x7f06ec15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06f0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f06ec0b35a0 0x7f06ec15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06f0e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec06ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06f0e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec0b43b0 con 0x7f06ec144770 2024-01-16T15:00:25.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06f0e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec15f930 con 0x7f06ec0b35a0 2024-01-16T15:00:25.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06f0e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0b3b10 con 0x7f06ec06c9d0 2024-01-16T15:00:25.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06eb400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06ec144770 0x7f06ec15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06eb400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06ec144770 0x7f06ec15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37674/0 (socket says 172.21.15.43:37674) 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06eb400640 1 -- 172.21.15.43:0/1134957921 learned_addr learned my addr 172.21.15.43:0/1134957921 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06ebe00640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec06ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.679+0000 7f06e3e00640 1 --2- 172.21.15.43:0/1134957921 >> v2:172.21.15.69:3300/0 conn(0x7f06ec0b35a0 0x7f06ec15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.680+0000 7f06eb400640 1 -- 172.21.15.43:0/1134957921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 msgr2=0x7f06ec06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.680+0000 7f06eb400640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec06ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.680+0000 7f06eb400640 1 -- 172.21.15.43:0/1134957921 >> v2:172.21.15.69:3300/0 conn(0x7f06ec0b35a0 msgr2=0x7f06ec15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.680+0000 7f06eb400640 1 --2- 172.21.15.43:0/1134957921 >> v2:172.21.15.69:3300/0 conn(0x7f06ec0b35a0 0x7f06ec15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.680+0000 7f06eb400640 1 -- 172.21.15.43:0/1134957921 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0b3df0 con 0x7f06ec144770 2024-01-16T15:00:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.681+0000 7f06eb400640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec144770 0x7f06ec15eb60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0ae900 tx=0x7f06dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cfbb94290a4e45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.681+0000 7f06eaa00640 1 -- 172.21.15.43:0/1134957921 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc031870 con 0x7f06ec144770 2024-01-16T15:00:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.681+0000 7f06eaa00640 1 -- 172.21.15.43:0/1134957921 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc0319d0 con 0x7f06ec144770 2024-01-16T15:00:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.681+0000 7f06eaa00640 1 -- 172.21.15.43:0/1134957921 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc031d60 con 0x7f06ec144770 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 -- 172.21.15.43:0/1134957921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec144770 msgr2=0x7f06ec15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec144770 0x7f06ec15eb60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0ae900 tx=0x7f06dc02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 -- 172.21.15.43:0/1134957921 shutdown_connections 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec06ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 --2- 172.21.15.43:0/1134957921 >> v2:172.21.15.69:3300/0 conn(0x7f06ec0b35a0 0x7f06ec15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 --2- 172.21.15.43:0/1134957921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec144770 0x7f06ec15eb60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 -- 172.21.15.43:0/1134957921 >> 172.21.15.43:0/1134957921 conn(0x7f06ec0cec10 msgr2=0x7f06ec0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 -- 172.21.15.43:0/1134957921 shutdown_connections 2024-01-16T15:00:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.682+0000 7f06f0e00640 1 -- 172.21.15.43:0/1134957921 wait complete. 2024-01-16T15:00:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.684+0000 7f06f0e00640 1 Processor -- start 2024-01-16T15:00:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.684+0000 7f06f0e00640 1 -- start start 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 0x7f06ec1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1f66b0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1f6990 con 0x7f06ec144770 2024-01-16T15:00:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.685+0000 7f06f0e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1f6c70 con 0x7f06ec06c9d0 2024-01-16T15:00:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37680/0 (socket says 172.21.15.43:37680) 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 -- 172.21.15.43:0/2761176095 learned_addr learned my addr 172.21.15.43:0/2761176095 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06ebe00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 0x7f06ec1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06eb400640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 -- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 msgr2=0x7f06ec1f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec1f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 -- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 msgr2=0x7f06ec1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 0x7f06ec1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 -- 172.21.15.43:0/2761176095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0094d0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06ebe00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 0x7f06ec1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.686+0000 7f06e3e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06d400e9b0 tx=0x7f06d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.687+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d4009670 con 0x7f06ec0b35a0 2024-01-16T15:00:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.687+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d400c9a0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.687+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d400cd50 con 0x7f06ec0b35a0 2024-01-16T15:00:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.687+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec1f6f50 con 0x7f06ec0b35a0 2024-01-16T15:00:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.687+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec1f62d0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.688+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec04e840 con 0x7f06ec0b35a0 2024-01-16T15:00:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.689+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f06d4013470 con 0x7f06ec0b35a0 2024-01-16T15:00:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.690+0000 7f06e3400640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 0x7f06c406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.690+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f06d408a050 con 0x7f06ec0b35a0 2024-01-16T15:00:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.690+0000 7f06eb400640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 0x7f06c406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.691+0000 7f06eb400640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 0x7f06c406d7b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f06dc036000 tx=0x7f06dc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.695+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06d4054f80 con 0x7f06ec0b35a0 2024-01-16T15:00:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.889+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06ec1f70e0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.890+0000 7f06e3400640 1 -- 172.21.15.43:0/2761176095 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f06d40585a0 con 0x7f06ec0b35a0 2024-01-16T15:00:25.871 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 msgr2=0x7f06c406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 0x7f06c406d7b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f06dc036000 tx=0x7f06dc03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 msgr2=0x7f06ec153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06d400e9b0 tx=0x7f06d400ee70 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 shutdown_connections 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06c406b2f0 0x7f06c406d7b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06ec144770 0x7f06ec1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06ec0b35a0 0x7f06ec153970 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 --2- 172.21.15.43:0/2761176095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06ec06c9d0 0x7f06ec1f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 >> 172.21.15.43:0/2761176095 conn(0x7f06ec0cec10 msgr2=0x7f06ec0cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 shutdown_connections 2024-01-16T15:00:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:25.893+0000 7f06f0e00640 1 -- 172.21.15.43:0/2761176095 wait complete. 2024-01-16T15:00:25.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:26.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.140+0000 7f1294600640 1 Processor -- start 2024-01-16T15:00:26.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.140+0000 7f1294600640 1 -- start start 2024-01-16T15:00:26.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1294600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 0x7f128c0abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1294600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f128c06dc30 0x7f128c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1294600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0ac530 con 0x7f128c0cbcb0 2024-01-16T15:00:26.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1294600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c0b2080 con 0x7f128c06dc30 2024-01-16T15:00:26.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1294600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c0b2360 con 0x7f128c15b340 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1292600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f128c06dc30 0x7f128c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1292600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f128c06dc30 0x7f128c0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37684/0 (socket says 172.21.15.43:37684) 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1291200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f128c0cbcb0 0x7f128c06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58422/0 (socket says 172.21.15.43:58422) 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1292600640 1 -- 172.21.15.43:0/1609121148 learned_addr learned my addr 172.21.15.43:0/1609121148 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.142+0000 7f1291c00640 1 --2- 172.21.15.43:0/1609121148 >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 0x7f128c0abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.143+0000 7f1292600640 1 -- 172.21.15.43:0/1609121148 >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 msgr2=0x7f128c0abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.143+0000 7f1292600640 1 --2- 172.21.15.43:0/1609121148 >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 0x7f128c0abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.143+0000 7f1292600640 1 -- 172.21.15.43:0/1609121148 >> v1:172.21.15.43:6789/0 conn(0x7f128c0cbcb0 legacy=0x7f128c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.143+0000 7f1292600640 1 -- 172.21.15.43:0/1609121148 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c0b2640 con 0x7f128c06dc30 2024-01-16T15:00:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.143+0000 7f1291c00640 1 --2- 172.21.15.43:0/1609121148 >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 0x7f128c0abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:26.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.144+0000 7f1292600640 1 --2- 172.21.15.43:0/1609121148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c0b3120 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f128c0b1e40 tx=0x7f128402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e839fabecc1ef24 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.144+0000 7f128be00640 1 -- 172.21.15.43:0/1609121148 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284030870 con 0x7f128c06dc30 2024-01-16T15:00:26.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f128be00640 1 -- 172.21.15.43:0/1609121148 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12840309d0 con 0x7f128c06dc30 2024-01-16T15:00:26.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f128be00640 1 -- 172.21.15.43:0/1609121148 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284030d60 con 0x7f128c06dc30 2024-01-16T15:00:26.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 -- 172.21.15.43:0/1609121148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 msgr2=0x7f128c0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 --2- 172.21.15.43:0/1609121148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c0b3120 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f128c0b1e40 tx=0x7f128402b9b0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 -- 172.21.15.43:0/1609121148 shutdown_connections 2024-01-16T15:00:26.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 --2- 172.21.15.43:0/1609121148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c0b3120 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 --2- 172.21.15.43:0/1609121148 >> v2:172.21.15.69:3300/0 conn(0x7f128c15b340 0x7f128c0abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 -- 172.21.15.43:0/1609121148 >> 172.21.15.43:0/1609121148 conn(0x7f128c0b0d80 msgr2=0x7f128c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:26.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.145+0000 7f1294600640 1 -- 172.21.15.43:0/1609121148 shutdown_connections 2024-01-16T15:00:26.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.146+0000 7f1294600640 1 -- 172.21.15.43:0/1609121148 wait complete. 2024-01-16T15:00:26.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.147+0000 7f1294600640 1 Processor -- start 2024-01-16T15:00:26.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.147+0000 7f1294600640 1 -- start start 2024-01-16T15:00:26.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f128c0cbcb0 0x7f128c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f128c15b340 0x7f128c09c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c157a00 con 0x7f128c06dc30 2024-01-16T15:00:26.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c1549c0 con 0x7f128c15b340 2024-01-16T15:00:26.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.148+0000 7f1294600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c156aa0 con 0x7f128c0cbcb0 2024-01-16T15:00:26.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37700/0 (socket says 172.21.15.43:37700) 2024-01-16T15:00:26.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 -- 172.21.15.43:0/4191350822 learned_addr learned my addr 172.21.15.43:0/4191350822 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 -- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f128c0cbcb0 msgr2=0x7f128c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1292600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f128c15b340 0x7f128c09c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f128c0cbcb0 0x7f128c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 -- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f128c15b340 msgr2=0x7f128c09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f128c15b340 0x7f128c09c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.149+0000 7f1291c00640 1 -- 172.21.15.43:0/4191350822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840094d0 con 0x7f128c06dc30 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1291c00640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f127400e730 tx=0x7f127400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274004230 con 0x7f128c06dc30 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274004390 con 0x7f128c06dc30 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c156d80 con 0x7f128c06dc30 2024-01-16T15:00:26.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127400c8f0 con 0x7f128c06dc30 2024-01-16T15:00:26.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.150+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c157340 con 0x7f128c06dc30 2024-01-16T15:00:26.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.151+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c157c70 con 0x7f128c06dc30 2024-01-16T15:00:26.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.152+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f127400ca50 con 0x7f128c06dc30 2024-01-16T15:00:26.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.155+0000 7f1289600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 0x7f126406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.155+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f12740897f0 con 0x7f128c06dc30 2024-01-16T15:00:26.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.155+0000 7f1291200640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 0x7f126406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.155+0000 7f1291200640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 0x7f126406d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f127c007fd0 tx=0x7f127c007d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.159+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1274054670 con 0x7f128c06dc30 2024-01-16T15:00:26.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.365+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f128c074770 con 0x7f128c06dc30 2024-01-16T15:00:26.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.365+0000 7f1289600640 1 -- 172.21.15.43:0/4191350822 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1274020080 con 0x7f128c06dc30 2024-01-16T15:00:26.347 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:26.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.368+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 msgr2=0x7f126406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.368+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 0x7f126406d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f127c007fd0 tx=0x7f127c007d80 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.368+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 msgr2=0x7f128c154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.368+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f127400e730 tx=0x7f127400ebf0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 shutdown_connections 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f126406b2f0 0x7f126406d7b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f128c15b340 0x7f128c09c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f128c0cbcb0 0x7f128c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 --2- 172.21.15.43:0/4191350822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f128c06dc30 0x7f128c154480 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 >> 172.21.15.43:0/4191350822 conn(0x7f128c0b0d80 msgr2=0x7f128c0b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 shutdown_connections 2024-01-16T15:00:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.369+0000 7f1294600640 1 -- 172.21.15.43:0/4191350822 wait complete. 2024-01-16T15:00:26.369 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:00:26.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:26.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:26.465 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:26.465 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:00:26.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:26.465 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:00:26.522 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:26.522 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:26.578 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:26.578 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:26.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:26.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.876+0000 7fa6da400640 1 Processor -- start 2024-01-16T15:00:26.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.877+0000 7fa6da400640 1 -- start start 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6da400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6d415b050 0x7fa6d415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6da400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6d406ddb0 0x7fa6d406e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6da400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40ad6b0 con 0x7fa6d4153f10 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6da400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d406e7a0 con 0x7fa6d406ddb0 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6da400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d40b2860 con 0x7fa6d415b050 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6d3400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa6d4153f10 0x7fa6d40b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58428/0 (socket says 172.21.15.43:58428) 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6d3400640 1 -- 172.21.15.43:0/2141247527 learned_addr learned my addr 172.21.15.43:0/2141247527 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:26.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.878+0000 7fa6d3e00640 1 --2- 172.21.15.43:0/2141247527 >> v2:172.21.15.43:3300/0 conn(0x7fa6d406ddb0 0x7fa6d406e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d2a00640 1 --2- 172.21.15.43:0/2141247527 >> v2:172.21.15.69:3300/0 conn(0x7fa6d415b050 0x7fa6d415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d2000640 1 -- 172.21.15.43:0/2141247527 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691683561 0 0) 0x7fa6d40ad6b0 con 0x7fa6d4153f10 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d2000640 1 -- 172.21.15.43:0/2141247527 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4003520 con 0x7fa6d4153f10 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d3e00640 1 -- 172.21.15.43:0/2141247527 >> v2:172.21.15.69:3300/0 conn(0x7fa6d415b050 msgr2=0x7fa6d415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d3e00640 1 --2- 172.21.15.43:0/2141247527 >> v2:172.21.15.69:3300/0 conn(0x7fa6d415b050 0x7fa6d415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d3e00640 1 -- 172.21.15.43:0/2141247527 >> v1:172.21.15.43:6789/0 conn(0x7fa6d4153f10 legacy=0x7fa6d40b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.879+0000 7fa6d3e00640 1 -- 172.21.15.43:0/2141247527 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40b2b40 con 0x7fa6d406ddb0 2024-01-16T15:00:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.880+0000 7fa6d3e00640 1 --2- 172.21.15.43:0/2141247527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d406ddb0 0x7fa6d406e260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4157a80 tx=0x7fa6cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac23cd1c1457837 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.880+0000 7fa6d2000640 1 -- 172.21.15.43:0/2141247527 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc031830 con 0x7fa6d406ddb0 2024-01-16T15:00:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.880+0000 7fa6d2000640 1 -- 172.21.15.43:0/2141247527 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc031990 con 0x7fa6d406ddb0 2024-01-16T15:00:26.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.880+0000 7fa6d2000640 1 -- 172.21.15.43:0/2141247527 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc031d00 con 0x7fa6d406ddb0 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 -- 172.21.15.43:0/2141247527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d406ddb0 msgr2=0x7fa6d406e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 --2- 172.21.15.43:0/2141247527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d406ddb0 0x7fa6d406e260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4157a80 tx=0x7fa6cc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 -- 172.21.15.43:0/2141247527 shutdown_connections 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 --2- 172.21.15.43:0/2141247527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d406ddb0 0x7fa6d406e260 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 --2- 172.21.15.43:0/2141247527 >> v2:172.21.15.69:3300/0 conn(0x7fa6d415b050 0x7fa6d415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 -- 172.21.15.43:0/2141247527 >> 172.21.15.43:0/2141247527 conn(0x7fa6d40b0d80 msgr2=0x7fa6d40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 -- 172.21.15.43:0/2141247527 shutdown_connections 2024-01-16T15:00:26.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.881+0000 7fa6da400640 1 -- 172.21.15.43:0/2141247527 wait complete. 2024-01-16T15:00:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.883+0000 7fa6da400640 1 Processor -- start 2024-01-16T15:00:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.883+0000 7fa6da400640 1 -- start start 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d406ddb0 0x7fa6d4145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d415b050 0x7fa6d4158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d4149f90 con 0x7fa6d4153f10 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d40ae2e0 con 0x7fa6d406ddb0 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6da400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d4167140 con 0x7fa6d415b050 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37724/0 (socket says 172.21.15.43:37724) 2024-01-16T15:00:26.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.884+0000 7fa6d2a00640 1 -- 172.21.15.43:0/57858512 learned_addr learned my addr 172.21.15.43:0/57858512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d3400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d406ddb0 0x7fa6d4145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d3e00640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d415b050 0x7fa6d4158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 -- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d415b050 msgr2=0x7fa6d4158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d415b050 0x7fa6d4158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 -- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d406ddb0 msgr2=0x7fa6d4145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d406ddb0 0x7fa6d4145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 -- 172.21.15.43:0/57858512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc009400 con 0x7fa6d4153f10 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6d2a00640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc007e40 tx=0x7fa6bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc014620 con 0x7fa6d4153f10 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00ee90 con 0x7fa6d4153f10 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0149c0 con 0x7fa6d4153f10 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.885+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d4167420 con 0x7fa6d4153f10 2024-01-16T15:00:26.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.886+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d414a510 con 0x7fa6d4153f10 2024-01-16T15:00:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.887+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d4074770 con 0x7fa6d4153f10 2024-01-16T15:00:26.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.888+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa6bc014b20 con 0x7fa6d4153f10 2024-01-16T15:00:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.889+0000 7fa6cb400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 0x7fa6ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.889+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa6bc08a670 con 0x7fa6d4153f10 2024-01-16T15:00:26.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.889+0000 7fa6d3400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 0x7fa6ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.889+0000 7fa6d3400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 0x7fa6ac06d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4005880 tx=0x7fa6c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:26.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:26.893+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6bc0553c0 con 0x7fa6d4153f10 2024-01-16T15:00:27.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.087+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6d40b3340 con 0x7fa6d4153f10 2024-01-16T15:00:27.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.089+0000 7fa6cb400640 1 -- 172.21.15.43:0/57858512 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7fa6bc0589e0 con 0x7fa6d4153f10 2024-01-16T15:00:27.070 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:27.070 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 msgr2=0x7fa6ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 0x7fa6ac06d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4005880 tx=0x7fa6c4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 msgr2=0x7fa6d4158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc007e40 tx=0x7fa6bc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 shutdown_connections 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ac06b2c0 0x7fa6ac06d780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d415b050 0x7fa6d4158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d4153f10 0x7fa6d4158880 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 --2- 172.21.15.43:0/57858512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d406ddb0 0x7fa6d4145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 >> 172.21.15.43:0/57858512 conn(0x7fa6d40b0d80 msgr2=0x7fa6d415d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 shutdown_connections 2024-01-16T15:00:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.092+0000 7fa6da400640 1 -- 172.21.15.43:0/57858512 wait complete. 2024-01-16T15:00:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:27.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:27.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.340+0000 7fd14ae00640 1 Processor -- start 2024-01-16T15:00:27.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.341+0000 7fd14ae00640 1 -- start start 2024-01-16T15:00:27.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.342+0000 7fd14ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1440ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.342+0000 7fd14ae00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1440cf380 0x7fd14406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.342+0000 7fd14ae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1440cd2b0 con 0x7fd1440cb9d0 2024-01-16T15:00:27.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.342+0000 7fd14ae00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1440d02e0 con 0x7fd1440cf380 2024-01-16T15:00:27.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.342+0000 7fd14ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14406f670 con 0x7fd144158f40 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1440cf380 0x7fd14406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1440cf380 0x7fd14406f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37734/0 (socket says 172.21.15.43:37734) 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 -- 172.21.15.43:0/2174262118 learned_addr learned my addr 172.21.15.43:0/2174262118 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd143400640 1 --1- 172.21.15.43:0/2174262118 >> v1:172.21.15.43:6789/0 conn(0x7fd1440cb9d0 0x7fd1440cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58432/0 (socket says 172.21.15.43:58432) 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd143e00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1440ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 -- 172.21.15.43:0/2174262118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 msgr2=0x7fd1440ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1440ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 -- 172.21.15.43:0/2174262118 >> v1:172.21.15.43:6789/0 conn(0x7fd1440cb9d0 legacy=0x7fd1440cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd142a00640 1 -- 172.21.15.43:0/2174262118 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651013520 0 0) 0x7fd1440cd2b0 con 0x7fd1440cb9d0 2024-01-16T15:00:27.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.343+0000 7fd148e00640 1 -- 172.21.15.43:0/2174262118 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440cbe30 con 0x7fd1440cf380 2024-01-16T15:00:27.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.345+0000 7fd148e00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd14406f130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e23515808d6901f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.345+0000 7fd142a00640 1 -- 172.21.15.43:0/2174262118 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c030840 con 0x7fd1440cf380 2024-01-16T15:00:27.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.345+0000 7fd142a00640 1 -- 172.21.15.43:0/2174262118 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c0309a0 con 0x7fd1440cf380 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.345+0000 7fd142a00640 1 -- 172.21.15.43:0/2174262118 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c030d30 con 0x7fd1440cf380 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 -- 172.21.15.43:0/2174262118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 msgr2=0x7fd14406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd14406f130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 -- 172.21.15.43:0/2174262118 shutdown_connections 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd14406f130 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 --2- 172.21.15.43:0/2174262118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1440ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 -- 172.21.15.43:0/2174262118 >> 172.21.15.43:0/2174262118 conn(0x7fd1440cdb20 msgr2=0x7fd1440cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 -- 172.21.15.43:0/2174262118 shutdown_connections 2024-01-16T15:00:27.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.346+0000 7fd14ae00640 1 -- 172.21.15.43:0/2174262118 wait complete. 2024-01-16T15:00:27.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.348+0000 7fd14ae00640 1 Processor -- start 2024-01-16T15:00:27.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.348+0000 7fd14ae00640 1 -- start start 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1440cb9d0 0x7fd1440cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1441f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440b0890 con 0x7fd1440cf380 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441575e0 con 0x7fd1440cb9d0 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd14ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441f6780 con 0x7fd144158f40 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd143400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd143400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37750/0 (socket says 172.21.15.43:37750) 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd143400640 1 -- 172.21.15.43:0/1198404463 learned_addr learned my addr 172.21.15.43:0/1198404463 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:27.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd148e00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1441f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.350+0000 7fd143e00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1440cb9d0 0x7fd1440cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 -- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 msgr2=0x7fd1441f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1441f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 -- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1440cb9d0 msgr2=0x7fd1440cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1440cb9d0 0x7fd1440cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 -- 172.21.15.43:0/1198404463 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0094d0 con 0x7fd1440cf380 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd148e00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1441f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd143400640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00c890 tx=0x7fd12c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00a640 con 0x7fd1440cf380 2024-01-16T15:00:27.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c00a7c0 con 0x7fd1440cf380 2024-01-16T15:00:27.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00ab70 con 0x7fd1440cf380 2024-01-16T15:00:27.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.351+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1441f6a20 con 0x7fd1440cf380 2024-01-16T15:00:27.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.352+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1441f6fe0 con 0x7fd1440cf380 2024-01-16T15:00:27.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.353+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14406fb50 con 0x7fd1440cf380 2024-01-16T15:00:27.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.354+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd12c00acd0 con 0x7fd1440cf380 2024-01-16T15:00:27.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.355+0000 7fd13be00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 0x7fd11c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.355+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd12c0890e0 con 0x7fd1440cf380 2024-01-16T15:00:27.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.355+0000 7fd143e00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 0x7fd11c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.356+0000 7fd143e00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 0x7fd11c06d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd134009a30 tx=0x7fd134008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.359+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd12c053f60 con 0x7fd1440cf380 2024-01-16T15:00:27.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.553+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd14404e920 con 0x7fd1440cf380 2024-01-16T15:00:27.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.554+0000 7fd13be00640 1 -- 172.21.15.43:0/1198404463 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd12c057580 con 0x7fd1440cf380 2024-01-16T15:00:27.535 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 msgr2=0x7fd11c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 0x7fd11c06d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd134009a30 tx=0x7fd134008040 comp rx=0 tx=0).stop 2024-01-16T15:00:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 msgr2=0x7fd1441570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00c890 tx=0x7fd12c00cd50 comp rx=0 tx=0).stop 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 shutdown_connections 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd11c06b2c0 0x7fd11c06d780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd144158f40 0x7fd1441f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1440cf380 0x7fd1441570a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 --2- 172.21.15.43:0/1198404463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1440cb9d0 0x7fd1440cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 >> 172.21.15.43:0/1198404463 conn(0x7fd1440cdb20 msgr2=0x7fd1440d16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 shutdown_connections 2024-01-16T15:00:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.557+0000 7fd14ae00640 1 -- 172.21.15.43:0/1198404463 wait complete. 2024-01-16T15:00:27.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.806+0000 7f82ccc00640 1 Processor -- start 2024-01-16T15:00:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.807+0000 7f82ccc00640 1 -- start start 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82ccc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f82c80cfa30 0x7f82c80cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82ccc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80b2980 con 0x7f82c80d1fd0 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82ccc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80d0350 con 0x7f82c80b33d0 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82ccc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c80cf030 con 0x7f82c80cfa30 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82c7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f82c80cfa30 0x7f82c80cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.808+0000 7f82c6a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f82c80d1fd0 0x7f82c80ce920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58446/0 (socket says 172.21.15.43:58446) 2024-01-16T15:00:27.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.809+0000 7f82c6a00640 1 -- 172.21.15.43:0/1010444352 learned_addr learned my addr 172.21.15.43:0/1010444352 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:27.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.809+0000 7f82c6000640 1 -- 172.21.15.43:0/1010444352 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915072275 0 0) 0x7f82c80b2980 con 0x7f82c80d1fd0 2024-01-16T15:00:27.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.809+0000 7f82c7400640 1 -- 172.21.15.43:0/1010444352 >> v1:172.21.15.69:6789/0 conn(0x7f82c80b33d0 legacy=0x7f82c80b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:27.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.809+0000 7f82c7400640 1 -- 172.21.15.43:0/1010444352 >> v1:172.21.15.43:6789/0 conn(0x7f82c80d1fd0 legacy=0x7f82c80ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.809+0000 7f82c7400640 1 -- 172.21.15.43:0/1010444352 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80cf220 con 0x7f82c80cfa30 2024-01-16T15:00:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.810+0000 7f82c7400640 1 --2- 172.21.15.43:0/1010444352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80cfa30 0x7f82c80cfe10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f82b80097a0 tx=0x7f82b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7178ee71aefccb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82c6000640 1 -- 172.21.15.43:0/1010444352 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8031830 con 0x7f82c80cfa30 2024-01-16T15:00:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82c6000640 1 -- 172.21.15.43:0/1010444352 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b8031990 con 0x7f82c80cfa30 2024-01-16T15:00:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82c6000640 1 -- 172.21.15.43:0/1010444352 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8031d20 con 0x7f82c80cfa30 2024-01-16T15:00:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82ccc00640 1 -- 172.21.15.43:0/1010444352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80cfa30 msgr2=0x7f82c80cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82ccc00640 1 --2- 172.21.15.43:0/1010444352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80cfa30 0x7f82c80cfe10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f82b80097a0 tx=0x7f82b802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82ccc00640 1 -- 172.21.15.43:0/1010444352 shutdown_connections 2024-01-16T15:00:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82ccc00640 1 --2- 172.21.15.43:0/1010444352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80cfa30 0x7f82c80cfe10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.811+0000 7f82ccc00640 1 -- 172.21.15.43:0/1010444352 >> 172.21.15.43:0/1010444352 conn(0x7f82c80d3020 msgr2=0x7f82c80d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.812+0000 7f82ccc00640 1 -- 172.21.15.43:0/1010444352 shutdown_connections 2024-01-16T15:00:27.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.812+0000 7f82ccc00640 1 -- 172.21.15.43:0/1010444352 wait complete. 2024-01-16T15:00:27.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.813+0000 7f82ccc00640 1 Processor -- start 2024-01-16T15:00:27.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.813+0000 7f82ccc00640 1 -- start start 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82c80cfa30 0x7f82c80c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82c80d1fd0 0x7f82c8151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80cbba0 con 0x7f82c80b33d0 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80caf90 con 0x7f82c80cfa30 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.814+0000 7f82ccc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80cb270 con 0x7f82c80d1fd0 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37776/0 (socket says 172.21.15.43:37776) 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 -- 172.21.15.43:0/2840119247 learned_addr learned my addr 172.21.15.43:0/2840119247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c6a00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82c80cfa30 0x7f82c80c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 -- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82c80d1fd0 msgr2=0x7f82c8151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82c80d1fd0 0x7f82c8151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 -- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82c80cfa30 msgr2=0x7f82c80c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82c80cfa30 0x7f82c80c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.815+0000 7f82c7400640 1 -- 172.21.15.43:0/2840119247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8009400 con 0x7f82c80b33d0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82c7400640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098c0 tx=0x7f82b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b803ba70 con 0x7f82c80b33d0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b803bbf0 con 0x7f82c80b33d0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80453f0 con 0x7f82c80b33d0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80cb550 con 0x7f82c80b33d0 2024-01-16T15:00:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.816+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c80ccdf0 con 0x7f82c80b33d0 2024-01-16T15:00:27.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.817+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c80d0a40 con 0x7f82c80b33d0 2024-01-16T15:00:27.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.818+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f82b804e450 con 0x7f82c80b33d0 2024-01-16T15:00:27.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.819+0000 7f82bf400640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 0x7f82a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.820+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f82b80abf70 con 0x7f82c80b33d0 2024-01-16T15:00:27.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.820+0000 7f82c6a00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 0x7f82a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.820+0000 7f82c6a00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 0x7f82a006d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007fd0 tx=0x7f82b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:27.824+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82b8076cc0 con 0x7f82c80b33d0 2024-01-16T15:00:28.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.022+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82c804e840 con 0x7f82c80b33d0 2024-01-16T15:00:28.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.023+0000 7f82bf400640 1 -- 172.21.15.43:0/2840119247 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82b807a2e0 con 0x7f82c80b33d0 2024-01-16T15:00:28.003 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:28.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.025+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 msgr2=0x7f82a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.025+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 0x7f82a006d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007fd0 tx=0x7f82b0009040 comp rx=0 tx=0).stop 2024-01-16T15:00:28.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.025+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 msgr2=0x7f82c80cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.025+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098c0 tx=0x7f82b80046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.025+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 shutdown_connections 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f82a006b2c0 0x7f82a006d780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82c80cfa30 0x7f82c80c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82c80d1fd0 0x7f82c8151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 --2- 172.21.15.43:0/2840119247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c80b33d0 0x7f82c80cc8b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 >> 172.21.15.43:0/2840119247 conn(0x7f82c80d3020 msgr2=0x7f82c80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 shutdown_connections 2024-01-16T15:00:28.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.026+0000 7f82ccc00640 1 -- 172.21.15.43:0/2840119247 wait complete. 2024-01-16T15:00:28.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.276+0000 7f818b200640 1 Processor -- start 2024-01-16T15:00:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.277+0000 7f818b200640 1 -- start start 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.278+0000 7f818b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f818c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.278+0000 7f818b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818c0c6bf0 con 0x7f818c15d3b0 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.278+0000 7f818b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818c0d13d0 con 0x7f818c0d0fa0 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.278+0000 7f818b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0cedc0 con 0x7f818c0d1d90 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8189e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f818c15d3b0 0x7f818c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58456/0 (socket says 172.21.15.43:58456) 2024-01-16T15:00:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8189e00640 1 -- 172.21.15.43:0/187089804 learned_addr learned my addr 172.21.15.43:0/187089804 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:28.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8189400640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f818c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044362892 0 0) 0x7f818c0c6bf0 con 0x7f818c15d3b0 2024-01-16T15:00:28.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c003520 con 0x7f818c15d3b0 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511208838 0 0) 0x7f816c003520 con 0x7f818c15d3b0 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 msgr2=0x7f818c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f818c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 >> v1:172.21.15.69:6789/0 conn(0x7f818c0d0fa0 legacy=0x7f818c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c0c6e40 con 0x7f818c15d3b0 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.279+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f817c001670 con 0x7f818c15d3b0 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 >> v1:172.21.15.43:6789/0 conn(0x7f818c15d3b0 legacy=0x7f818c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f816c003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c003fe0 0x7f816c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f816c004d00 0x7f816c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0cedc0 con 0x7f816c004d00 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c00bbc0 con 0x7f816c003fe0 2024-01-16T15:00:28.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c00bd80 con 0x7f818c0d1d90 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f818a800640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f816c004d00 0x7f816c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8189400640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c003fe0 0x7f816c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f8189e00640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f816c003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f818a800640 1 -- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 msgr2=0x7f816c003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f818a800640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c0d1d90 0x7f816c003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.280+0000 7f818a800640 1 -- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c003fe0 msgr2=0x7f816c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.281+0000 7f818a800640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c003fe0 0x7f816c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.281+0000 7f818a800640 1 -- 172.21.15.43:0/187089804 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c003520 con 0x7f816c004d00 2024-01-16T15:00:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f818a800640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f816c004d00 0x7f816c005180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8184001750 tx=0x7f816c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65af994dc036a973 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840052d0 con 0x7f816c004d00 2024-01-16T15:00:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818402e050 con 0x7f816c004d00 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f8188a00640 1 -- 172.21.15.43:0/187089804 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184003830 con 0x7f816c004d00 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f818b200640 1 -- 172.21.15.43:0/187089804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f816c004d00 msgr2=0x7f816c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f818b200640 1 --2- 172.21.15.43:0/187089804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f816c004d00 0x7f816c005180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8184001750 tx=0x7f816c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.282+0000 7f818a800640 1 -- 172.21.15.43:0/187089804 reap_dead start 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.283+0000 7f818b200640 1 -- 172.21.15.43:0/187089804 shutdown_connections 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.283+0000 7f818b200640 1 -- 172.21.15.43:0/187089804 >> 172.21.15.43:0/187089804 conn(0x7f818c0d30a0 msgr2=0x7f818c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.283+0000 7f818b200640 1 -- 172.21.15.43:0/187089804 shutdown_connections 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.283+0000 7f818b200640 1 -- 172.21.15.43:0/187089804 wait complete. 2024-01-16T15:00:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.285+0000 7f818b200640 1 Processor -- start 2024-01-16T15:00:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.285+0000 7f818b200640 1 -- start start 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c004d00 0x7f818c0b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c15bec0 0x7f818c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0a0350 con 0x7f818c09dbd0 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0a0d20 con 0x7f816c004d00 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818c09fd20 con 0x7f818c15bec0 2024-01-16T15:00:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f8189400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f8189400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37790/0 (socket says 172.21.15.43:37790) 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f8189400640 1 -- 172.21.15.43:0/3882556874 learned_addr learned my addr 172.21.15.43:0/3882556874 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f818a800640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c15bec0 0x7f818c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.286+0000 7f8189e00640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c004d00 0x7f818c0b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 -- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c15bec0 msgr2=0x7f818c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c15bec0 0x7f818c0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 -- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c004d00 msgr2=0x7f818c0b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c004d00 0x7f818c0b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 -- 172.21.15.43:0/3882556874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c09e050 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8189400640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8174007f30 tx=0x7f8174011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400e030 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8174016470 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174016820 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818c09e320 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.287+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818c09ffe0 con 0x7f818c09dbd0 2024-01-16T15:00:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.289+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818c0a0500 con 0x7f818c09dbd0 2024-01-16T15:00:28.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.290+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8174016980 con 0x7f818c09dbd0 2024-01-16T15:00:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.291+0000 7f8182000640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 0x7f816406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.291+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f817400f070 con 0x7f818c09dbd0 2024-01-16T15:00:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.291+0000 7f8189e00640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 0x7f816406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.292+0000 7f8189e00640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 0x7f816406d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f817c001ae0 tx=0x7f817c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.296+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f818c0a0500 con 0x7f818c09dbd0 2024-01-16T15:00:28.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.489+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f818c04e840 con 0x7f818c09dbd0 2024-01-16T15:00:28.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.490+0000 7f8182000640 1 -- 172.21.15.43:0/3882556874 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f81740559d0 con 0x7f818c09dbd0 2024-01-16T15:00:28.471 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 msgr2=0x7f816406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 0x7f816406d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f817c001ae0 tx=0x7f817c009040 comp rx=0 tx=0).stop 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 msgr2=0x7f818c0a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8174007f30 tx=0x7f8174011660 comp rx=0 tx=0).stop 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 shutdown_connections 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f816406b2f0 0x7f816406d7b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f818c15bec0 0x7f818c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f818c09dbd0 0x7f818c0a07e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 --2- 172.21.15.43:0/3882556874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f816c004d00 0x7f818c0b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 >> 172.21.15.43:0/3882556874 conn(0x7f818c0d30a0 msgr2=0x7f818c0cfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:28.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 shutdown_connections 2024-01-16T15:00:28.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.493+0000 7f818b200640 1 -- 172.21.15.43:0/3882556874 wait complete. 2024-01-16T15:00:28.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:28.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.738+0000 7f0e3fa00640 1 Processor -- start 2024-01-16T15:00:28.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.738+0000 7f0e3fa00640 1 -- start start 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3fa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e380717a0 0x7f0e38168390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3fa00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380ae8a0 con 0x7f0e3806dc30 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3fa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380ad400 con 0x7f0e380b3b60 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3fa00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e38071bd0 con 0x7f0e380717a0 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e380717a0 0x7f0e38168390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e380717a0 0x7f0e38168390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37804/0 (socket says 172.21.15.43:37804) 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e3da00640 1 -- 172.21.15.43:0/758060615 learned_addr learned my addr 172.21.15.43:0/758060615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:28.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.740+0000 7f0e2fe00640 1 --1- 172.21.15.43:0/758060615 >> v1:172.21.15.43:6789/0 conn(0x7f0e3806dc30 0x7f0e38071090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58468/0 (socket says 172.21.15.43:58468) 2024-01-16T15:00:28.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.741+0000 7f0e3da00640 1 -- 172.21.15.43:0/758060615 >> v1:172.21.15.69:6789/0 conn(0x7f0e380b3b60 legacy=0x7f0e3806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:28.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.741+0000 7f0e3da00640 1 -- 172.21.15.43:0/758060615 >> v1:172.21.15.43:6789/0 conn(0x7f0e3806dc30 legacy=0x7f0e38071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.741+0000 7f0e3da00640 1 -- 172.21.15.43:0/758060615 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e380acb50 con 0x7f0e380717a0 2024-01-16T15:00:28.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.742+0000 7f0e3da00640 1 --2- 172.21.15.43:0/758060615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380717a0 0x7f0e38168390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009e50 tx=0x7f0e3002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b8aca63dbde232 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.742+0000 7f0e37e00640 1 -- 172.21.15.43:0/758060615 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3002cb20 con 0x7f0e380717a0 2024-01-16T15:00:28.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.742+0000 7f0e37e00640 1 -- 172.21.15.43:0/758060615 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3002cc80 con 0x7f0e380717a0 2024-01-16T15:00:28.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.742+0000 7f0e37e00640 1 -- 172.21.15.43:0/758060615 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e300343f0 con 0x7f0e380717a0 2024-01-16T15:00:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 -- 172.21.15.43:0/758060615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380717a0 msgr2=0x7f0e38168390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/758060615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380717a0 0x7f0e38168390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009e50 tx=0x7f0e3002bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 -- 172.21.15.43:0/758060615 shutdown_connections 2024-01-16T15:00:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/758060615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380717a0 0x7f0e38168390 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 -- 172.21.15.43:0/758060615 >> 172.21.15.43:0/758060615 conn(0x7f0e380b0d80 msgr2=0x7f0e380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:28.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 -- 172.21.15.43:0/758060615 shutdown_connections 2024-01-16T15:00:28.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.743+0000 7f0e3fa00640 1 -- 172.21.15.43:0/758060615 wait complete. 2024-01-16T15:00:28.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.745+0000 7f0e3fa00640 1 Processor -- start 2024-01-16T15:00:28.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.745+0000 7f0e3fa00640 1 -- start start 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 0x7f0e380af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e380b3b60 0x7f0e380afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381e6e40 con 0x7f0e3806dc30 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380c85e0 con 0x7f0e380b3b60 2024-01-16T15:00:28.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3815cad0 con 0x7f0e380717a0 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37810/0 (socket says 172.21.15.43:37810) 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.746+0000 7f0e3d000640 1 -- 172.21.15.43:0/3390845622 learned_addr learned my addr 172.21.15.43:0/3390845622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e2fe00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 0x7f0e380af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 -- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 msgr2=0x7f0e380af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 0x7f0e380af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 -- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e380b3b60 msgr2=0x7f0e380afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e380b3b60 0x7f0e380afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 -- 172.21.15.43:0/3390845622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e30009b30 con 0x7f0e3806dc30 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e2fe00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 0x7f0e380af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:28.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3d000640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800c430 tx=0x7f0e2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011790 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280118f0 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011ce0 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.747+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3815cdb0 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.748+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381e7330 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.749+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3815cfa0 con 0x7f0e3806dc30 2024-01-16T15:00:28.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.750+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0e2801c460 con 0x7f0e3806dc30 2024-01-16T15:00:28.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.750+0000 7f0e35600640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 0x7f0e1006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:28.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.750+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0e28089cb0 con 0x7f0e3806dc30 2024-01-16T15:00:28.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.751+0000 7f0e2fe00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 0x7f0e1006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:28.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.751+0000 7f0e2fe00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 0x7f0e1006d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0e38150230 tx=0x7f0e20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:28.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.755+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e3815cfa0 con 0x7f0e3806dc30 2024-01-16T15:00:28.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.968+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0e3815cfa0 con 0x7f0e3806dc30 2024-01-16T15:00:28.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.968+0000 7f0e35600640 1 -- 172.21.15.43:0/3390845622 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0e3815cfa0 con 0x7f0e3806dc30 2024-01-16T15:00:28.949 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.971+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 msgr2=0x7f0e1006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.971+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 0x7f0e1006d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0e38150230 tx=0x7f0e20009040 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.971+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 msgr2=0x7f0e3815c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.971+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800c430 tx=0x7f0e2800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 shutdown_connections 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1006b2c0 0x7f0e1006d780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e380b3b60 0x7f0e380afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e380717a0 0x7f0e380af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 --2- 172.21.15.43:0/3390845622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e3806dc30 0x7f0e3815c010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 >> 172.21.15.43:0/3390845622 conn(0x7f0e380b0d80 msgr2=0x7f0e3806cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 shutdown_connections 2024-01-16T15:00:28.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:28.972+0000 7f0e3fa00640 1 -- 172.21.15.43:0/3390845622 wait complete. 2024-01-16T15:00:28.973 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:00:28.973 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:29.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:29.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.279+0000 7f3555e00640 1 Processor -- start 2024-01-16T15:00:29.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.279+0000 7f3555e00640 1 -- start start 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.280+0000 7f3555e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3550153f10 0x7f355015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.280+0000 7f3555e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f355015b050 0x7f35500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.280+0000 7f3555e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500ad6b0 con 0x7f355006ddb0 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.280+0000 7f3555e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500b25c0 con 0x7f3550153f10 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.280+0000 7f3555e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500b2860 con 0x7f355015b050 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3550153f10 0x7f355015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3550153f10 0x7f355015aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37818/0 (socket says 172.21.15.43:37818) 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 -- 172.21.15.43:0/3451040712 learned_addr learned my addr 172.21.15.43:0/3451040712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354e000640 1 --2- 172.21.15.43:0/3451040712 >> v2:172.21.15.69:3300/0 conn(0x7f355015b050 0x7f35500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 -- 172.21.15.43:0/3451040712 >> v2:172.21.15.69:3300/0 conn(0x7f355015b050 msgr2=0x7f35500b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 --2- 172.21.15.43:0/3451040712 >> v2:172.21.15.69:3300/0 conn(0x7f355015b050 0x7f35500b2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 -- 172.21.15.43:0/3451040712 >> v1:172.21.15.69:6789/0 conn(0x7f355006ddb0 legacy=0x7f355015d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.281+0000 7f354ea00640 1 -- 172.21.15.43:0/3451040712 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500b2b40 con 0x7f3550153f10 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.282+0000 7f354ea00640 1 --2- 172.21.15.43:0/3451040712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3550153f10 0x7f355015aaa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f355006e9a0 tx=0x7f354002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cab4725cf1352d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.283+0000 7f354d600640 1 -- 172.21.15.43:0/3451040712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540031870 con 0x7f3550153f10 2024-01-16T15:00:29.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.283+0000 7f354d600640 1 -- 172.21.15.43:0/3451040712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35400319d0 con 0x7f3550153f10 2024-01-16T15:00:29.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.283+0000 7f354d600640 1 -- 172.21.15.43:0/3451040712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540031d60 con 0x7f3550153f10 2024-01-16T15:00:29.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 -- 172.21.15.43:0/3451040712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3550153f10 msgr2=0x7f355015aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 --2- 172.21.15.43:0/3451040712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3550153f10 0x7f355015aaa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f355006e9a0 tx=0x7f354002b910 comp rx=0 tx=0).stop 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 -- 172.21.15.43:0/3451040712 shutdown_connections 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 --2- 172.21.15.43:0/3451040712 >> v2:172.21.15.69:3300/0 conn(0x7f355015b050 0x7f35500b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 --2- 172.21.15.43:0/3451040712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3550153f10 0x7f355015aaa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 -- 172.21.15.43:0/3451040712 >> 172.21.15.43:0/3451040712 conn(0x7f35500b0d80 msgr2=0x7f35500b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 -- 172.21.15.43:0/3451040712 shutdown_connections 2024-01-16T15:00:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.284+0000 7f3555e00640 1 -- 172.21.15.43:0/3451040712 wait complete. 2024-01-16T15:00:29.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.285+0000 7f3555e00640 1 Processor -- start 2024-01-16T15:00:29.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.286+0000 7f3555e00640 1 -- start start 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.286+0000 7f3555e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f355006ddb0 0x7f355014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.286+0000 7f3555e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3550153f10 0x7f3550158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f3555e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f3555e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355009c5f0 con 0x7f355015b050 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f3555e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355009c8d0 con 0x7f355006ddb0 2024-01-16T15:00:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f3555e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355009cbb0 con 0x7f3550153f10 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37820/0 (socket says 172.21.15.43:37820) 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 -- 172.21.15.43:0/3875971135 learned_addr learned my addr 172.21.15.43:0/3875971135 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354e000640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3550153f10 0x7f3550158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354ea00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f355006ddb0 0x7f355014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 -- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3550153f10 msgr2=0x7f3550158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3550153f10 0x7f3550158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 -- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f355006ddb0 msgr2=0x7f355014cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f355006ddb0 0x7f355014cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.287+0000 7f354f400640 1 -- 172.21.15.43:0/3875971135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35400094d0 con 0x7f355015b050 2024-01-16T15:00:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f354f400640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f354800c780 tx=0x7f354800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548004b60 con 0x7f355015b050 2024-01-16T15:00:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354800ce90 con 0x7f355015b050 2024-01-16T15:00:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354800a850 con 0x7f355015b050 2024-01-16T15:00:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355009ce90 con 0x7f355015b050 2024-01-16T15:00:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.288+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501f3ae0 con 0x7f355015b050 2024-01-16T15:00:29.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.290+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35500abff0 con 0x7f355015b050 2024-01-16T15:00:29.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.291+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f354800a9b0 con 0x7f355015b050 2024-01-16T15:00:29.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.291+0000 7f3547400640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 0x7f352806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.292+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3548089e10 con 0x7f355015b050 2024-01-16T15:00:29.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.292+0000 7f354ea00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 0x7f352806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.293+0000 7f354ea00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 0x7f352806d7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3540036000 tx=0x7f354003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.296+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f354808d030 con 0x7f355015b050 2024-01-16T15:00:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.490+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35500741d0 con 0x7f355015b050 2024-01-16T15:00:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.492+0000 7f3547400640 1 -- 172.21.15.43:0/3875971135 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f3548054be0 con 0x7f355015b050 2024-01-16T15:00:29.473 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:29.473 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.494+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 msgr2=0x7f352806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.494+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 0x7f352806d7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3540036000 tx=0x7f354003c040 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.494+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 msgr2=0x7f3550158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.494+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f354800c780 tx=0x7f354800cc40 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 shutdown_connections 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f352806b310 0x7f352806d7d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f355015b050 0x7f3550158dc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3550153f10 0x7f3550158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 --2- 172.21.15.43:0/3875971135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f355006ddb0 0x7f355014cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 >> 172.21.15.43:0/3875971135 conn(0x7f35500b0d80 msgr2=0x7f355006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 shutdown_connections 2024-01-16T15:00:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.495+0000 7f3555e00640 1 -- 172.21.15.43:0/3875971135 wait complete. 2024-01-16T15:00:29.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:29.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:29.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.737+0000 7fbb16200640 1 Processor -- start 2024-01-16T15:00:29.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.737+0000 7fbb16200640 1 -- start start 2024-01-16T15:00:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1015b350 0x7fbb100abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbb1009cb30 0x7fbb100b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbb100adb60 0x7fbb100adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1015b730 con 0x7fbb1009cb30 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb100ac740 con 0x7fbb100adb60 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb16200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100ac460 con 0x7fbb1015b350 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb07e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbb1009cb30 0x7fbb100b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb07e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbb1009cb30 0x7fbb100b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37830/0 (socket says 172.21.15.43:37830) 2024-01-16T15:00:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb07e00640 1 -- 172.21.15.43:0/2489384189 learned_addr learned my addr 172.21.15.43:0/2489384189 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:29.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb0f400640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1015b350 0x7fbb100abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.739+0000 7fbb0fe00640 1 --2- 172.21.15.43:0/2489384189 >> v2:172.21.15.69:3300/0 conn(0x7fbb100adb60 0x7fbb100adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.740+0000 7fbb07e00640 1 -- 172.21.15.43:0/2489384189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1015b350 msgr2=0x7fbb100abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.740+0000 7fbb07e00640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1015b350 0x7fbb100abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.740+0000 7fbb07e00640 1 -- 172.21.15.43:0/2489384189 >> v2:172.21.15.69:3300/0 conn(0x7fbb100adb60 msgr2=0x7fbb100adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.740+0000 7fbb07e00640 1 --2- 172.21.15.43:0/2489384189 >> v2:172.21.15.69:3300/0 conn(0x7fbb100adb60 0x7fbb100adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.740+0000 7fbb07e00640 1 -- 172.21.15.43:0/2489384189 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb100b2a50 con 0x7fbb1009cb30 2024-01-16T15:00:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.741+0000 7fbb07e00640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb1009cb30 0x7fbb100b3530 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbb10072730 tx=0x7fbaf802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfe927f068fb1e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.741+0000 7fbb0ea00640 1 -- 172.21.15.43:0/2489384189 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8031870 con 0x7fbb1009cb30 2024-01-16T15:00:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.741+0000 7fbb0ea00640 1 -- 172.21.15.43:0/2489384189 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf80319d0 con 0x7fbb1009cb30 2024-01-16T15:00:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.741+0000 7fbb0ea00640 1 -- 172.21.15.43:0/2489384189 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8031d60 con 0x7fbb1009cb30 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 -- 172.21.15.43:0/2489384189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb1009cb30 msgr2=0x7fbb100b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb1009cb30 0x7fbb100b3530 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbb10072730 tx=0x7fbaf802b910 comp rx=0 tx=0).stop 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 -- 172.21.15.43:0/2489384189 shutdown_connections 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 --2- 172.21.15.43:0/2489384189 >> v2:172.21.15.69:3300/0 conn(0x7fbb100adb60 0x7fbb100adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb1009cb30 0x7fbb100b3530 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 --2- 172.21.15.43:0/2489384189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1015b350 0x7fbb100abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 -- 172.21.15.43:0/2489384189 >> 172.21.15.43:0/2489384189 conn(0x7fbb100b0ba0 msgr2=0x7fbb100b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 -- 172.21.15.43:0/2489384189 shutdown_connections 2024-01-16T15:00:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.742+0000 7fbb16200640 1 -- 172.21.15.43:0/2489384189 wait complete. 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.744+0000 7fbb16200640 1 Processor -- start 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.744+0000 7fbb16200640 1 -- start start 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1009cb30 0x7fbb1013d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 0x7fbb1014dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1009bd40 con 0x7fbb100adb60 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb10153f10 con 0x7fbb1015b350 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb16200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1009b4a0 con 0x7fbb1009cb30 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37832/0 (socket says 172.21.15.43:37832) 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.745+0000 7fbb07e00640 1 -- 172.21.15.43:0/373973974 learned_addr learned my addr 172.21.15.43:0/373973974 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb0f400640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1009cb30 0x7fbb1013d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 -- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1009cb30 msgr2=0x7fbb1013d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1009cb30 0x7fbb1013d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 -- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 msgr2=0x7fbb1014dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb0fe00640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 0x7fbb1014dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 0x7fbb1014dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 -- 172.21.15.43:0/373973974 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf80094d0 con 0x7fbb100adb60 2024-01-16T15:00:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb0fe00640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 0x7fbb1014dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07e00640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf803ad30 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf80443f0 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.746+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80447a0 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.747+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1009b780 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.747+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1013dee0 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.748+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb101540a0 con 0x7fbb100adb60 2024-01-16T15:00:29.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.749+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbaf8044900 con 0x7fbb100adb60 2024-01-16T15:00:29.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.750+0000 7fbb07400640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 0x7fbae806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:29.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.750+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbaf802d040 con 0x7fbb100adb60 2024-01-16T15:00:29.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.750+0000 7fbb0f400640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 0x7fbae806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:29.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.751+0000 7fbb0f400640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 0x7fbae806d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb00006fd0 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:29.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.754+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbaf8077040 con 0x7fbb100adb60 2024-01-16T15:00:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.950+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb100ae4c0 con 0x7fbb100adb60 2024-01-16T15:00:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.951+0000 7fbb07400640 1 -- 172.21.15.43:0/373973974 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbaf807a660 con 0x7fbb100adb60 2024-01-16T15:00:29.932 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 msgr2=0x7fbae806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 0x7fbae806d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb00006fd0 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 msgr2=0x7fbb1009aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf80046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 shutdown_connections 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbae806b2c0 0x7fbae806d780 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb1015b350 0x7fbb1014dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb100adb60 0x7fbb1009aa50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 --2- 172.21.15.43:0/373973974 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb1009cb30 0x7fbb1013d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 >> 172.21.15.43:0/373973974 conn(0x7fbb100b0ba0 msgr2=0x7fbb100b01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 shutdown_connections 2024-01-16T15:00:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:29.954+0000 7fbb16200640 1 -- 172.21.15.43:0/373973974 wait complete. 2024-01-16T15:00:29.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:30.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.195+0000 7faf3b200640 1 Processor -- start 2024-01-16T15:00:30.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.195+0000 7faf3b200640 1 -- start start 2024-01-16T15:00:30.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.196+0000 7faf3b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 0x7faf3c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.196+0000 7faf3b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf3c0d0fa0 0x7faf3c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.196+0000 7faf3b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3c0d2010 con 0x7faf3c0d1bc0 2024-01-16T15:00:30.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.196+0000 7faf3b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0c6bf0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.196+0000 7faf3b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0d1400 con 0x7faf3c15d3b0 2024-01-16T15:00:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.197+0000 7faf39400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf3c0d0fa0 0x7faf3c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.197+0000 7faf39400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf3c0d0fa0 0x7faf3c0d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37844/0 (socket says 172.21.15.43:37844) 2024-01-16T15:00:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.197+0000 7faf39400640 1 -- 172.21.15.43:0/435355500 learned_addr learned my addr 172.21.15.43:0/435355500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.197+0000 7faf39e00640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 0x7faf3c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.198+0000 7faf39400640 1 -- 172.21.15.43:0/435355500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 msgr2=0x7faf3c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.198+0000 7faf39400640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 0x7faf3c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.198+0000 7faf39400640 1 -- 172.21.15.43:0/435355500 >> v1:172.21.15.69:6789/0 conn(0x7faf3c0d1bc0 legacy=0x7faf3c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.198+0000 7faf39400640 1 -- 172.21.15.43:0/435355500 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0cedc0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.198+0000 7faf39e00640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 0x7faf3c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:30.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.199+0000 7faf39400640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c0d1680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faf24009870 tx=0x7faf2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70caf094f3c78603 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.199+0000 7faf38a00640 1 -- 172.21.15.43:0/435355500 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24031640 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.199+0000 7faf38a00640 1 -- 172.21.15.43:0/435355500 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf240317a0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.199+0000 7faf38a00640 1 -- 172.21.15.43:0/435355500 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24031b30 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 -- 172.21.15.43:0/435355500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 msgr2=0x7faf3c0d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c0d1680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faf24009870 tx=0x7faf2402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 -- 172.21.15.43:0/435355500 shutdown_connections 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c0d1680 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 --2- 172.21.15.43:0/435355500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c15d3b0 0x7faf3c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 -- 172.21.15.43:0/435355500 >> 172.21.15.43:0/435355500 conn(0x7faf3c0d30a0 msgr2=0x7faf3c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 -- 172.21.15.43:0/435355500 shutdown_connections 2024-01-16T15:00:30.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.200+0000 7faf3b200640 1 -- 172.21.15.43:0/435355500 wait complete. 2024-01-16T15:00:30.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.202+0000 7faf3b200640 1 Processor -- start 2024-01-16T15:00:30.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.202+0000 7faf3b200640 1 -- start start 2024-01-16T15:00:30.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c0d1bc0 0x7faf3c09e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c15d3b0 0x7faf3c09ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c15cba0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0cab50 con 0x7faf3c15d3b0 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0cad60 con 0x7faf3c0d1bc0 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37858/0 (socket says 172.21.15.43:37858) 2024-01-16T15:00:30.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf39e00640 1 -- 172.21.15.43:0/1502210484 learned_addr learned my addr 172.21.15.43:0/1502210484 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:30.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf39400640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c0d1bc0 0x7faf3c09e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.203+0000 7faf3a800640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c15d3b0 0x7faf3c09ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 -- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c0d1bc0 msgr2=0x7faf3c09e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c0d1bc0 0x7faf3c09e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 -- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c15d3b0 msgr2=0x7faf3c09ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c15d3b0 0x7faf3c09ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 -- 172.21.15.43:0/1502210484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240094d0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf39e00640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e7e0 tx=0x7faf2c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c004030 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c004190 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c004520 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.204+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c0cb040 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.205+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c0cb600 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.206+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c04e840 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.207+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faf2c00c6d0 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.208+0000 7faf32000640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 0x7faf1406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.208+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faf2c089940 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.208+0000 7faf39400640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 0x7faf1406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.209+0000 7faf39400640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 0x7faf1406d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf24036000 tx=0x7faf2403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.212+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf2c054730 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.408+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf3c073e10 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.409+0000 7faf32000640 1 -- 172.21.15.43:0/1502210484 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf2c020020 con 0x7faf3c0d0fa0 2024-01-16T15:00:30.389 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.411+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 msgr2=0x7faf1406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 0x7faf1406d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf24036000 tx=0x7faf2403c040 comp rx=0 tx=0).stop 2024-01-16T15:00:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 msgr2=0x7faf3c15c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e7e0 tx=0x7faf2c00eca0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 shutdown_connections 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1406b2c0 0x7faf1406d780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c15d3b0 0x7faf3c09ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf3c0d1bc0 0x7faf3c09e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 --2- 172.21.15.43:0/1502210484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf3c0d0fa0 0x7faf3c15c660 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 >> 172.21.15.43:0/1502210484 conn(0x7faf3c0d30a0 msgr2=0x7faf3c06c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 shutdown_connections 2024-01-16T15:00:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.412+0000 7faf3b200640 1 -- 172.21.15.43:0/1502210484 wait complete. 2024-01-16T15:00:30.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:30.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.653+0000 7f8f59200640 1 Processor -- start 2024-01-16T15:00:30.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.653+0000 7f8f59200640 1 -- start start 2024-01-16T15:00:30.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f59200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f541435d0 0x7f8f54071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f59200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f540b4140 0x7f8f54070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f59200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f54070d40 con 0x7f8f541439b0 2024-01-16T15:00:30.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f59200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f54071020 con 0x7f8f541435d0 2024-01-16T15:00:30.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f59200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f54071300 con 0x7f8f540b4140 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f53400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f541435d0 0x7f8f54071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f53400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f541435d0 0x7f8f54071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37860/0 (socket says 172.21.15.43:37860) 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.655+0000 7f8f53400640 1 -- 172.21.15.43:0/2608475522 learned_addr learned my addr 172.21.15.43:0/2608475522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.656+0000 7f8f53e00640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f540b4140 0x7f8f54070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.656+0000 7f8f53400640 1 -- 172.21.15.43:0/2608475522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f540b4140 msgr2=0x7f8f54070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.656+0000 7f8f53400640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f540b4140 0x7f8f54070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.656+0000 7f8f53400640 1 -- 172.21.15.43:0/2608475522 >> v1:172.21.15.69:6789/0 conn(0x7f8f541439b0 legacy=0x7f8f540b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.656+0000 7f8f53400640 1 -- 172.21.15.43:0/2608475522 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540715e0 con 0x7f8f541435d0 2024-01-16T15:00:30.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.657+0000 7f8f53400640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f541435d0 0x7f8f54071de0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009e50 tx=0x7f8f4402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b779e7ce6ae876e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.657+0000 7f8f52a00640 1 -- 172.21.15.43:0/2608475522 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4402cb00 con 0x7f8f541435d0 2024-01-16T15:00:30.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f52a00640 1 -- 172.21.15.43:0/2608475522 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4402cc60 con 0x7f8f541435d0 2024-01-16T15:00:30.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f52a00640 1 -- 172.21.15.43:0/2608475522 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440343f0 con 0x7f8f541435d0 2024-01-16T15:00:30.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 -- 172.21.15.43:0/2608475522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f541435d0 msgr2=0x7f8f54071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f541435d0 0x7f8f54071de0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009e50 tx=0x7f8f4402bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 -- 172.21.15.43:0/2608475522 shutdown_connections 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f540b4140 0x7f8f54070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 --2- 172.21.15.43:0/2608475522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f541435d0 0x7f8f54071de0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 -- 172.21.15.43:0/2608475522 >> 172.21.15.43:0/2608475522 conn(0x7f8f540cec10 msgr2=0x7f8f540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.658+0000 7f8f59200640 1 -- 172.21.15.43:0/2608475522 shutdown_connections 2024-01-16T15:00:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.659+0000 7f8f59200640 1 -- 172.21.15.43:0/2608475522 wait complete. 2024-01-16T15:00:30.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.660+0000 7f8f59200640 1 Processor -- start 2024-01-16T15:00:30.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.660+0000 7f8f59200640 1 -- start start 2024-01-16T15:00:30.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 0x7f8f5415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 0x7f8f541570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5415b6d0 con 0x7f8f540b4140 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f541553a0 con 0x7f8f541439b0 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.661+0000 7f8f59200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f54155680 con 0x7f8f541435d0 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37862/0 (socket says 172.21.15.43:37862) 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 -- 172.21.15.43:0/1465902970 learned_addr learned my addr 172.21.15.43:0/1465902970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f4a800640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 0x7f8f5415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53e00640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 0x7f8f541570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 -- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 msgr2=0x7f8f5415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 0x7f8f5415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 -- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 msgr2=0x7f8f541570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 0x7f8f541570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53400640 1 -- 172.21.15.43:0/1465902970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44009b30 con 0x7f8f540b4140 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f53e00640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 0x7f8f541570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.662+0000 7f8f4a800640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 0x7f8f5415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:30.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f53400640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009e50 tx=0x7f8f44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44009510 con 0x7f8f540b4140 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4403e450 con 0x7f8f540b4140 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4403e800 con 0x7f8f540b4140 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54155960 con 0x7f8f540b4140 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.663+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54155f20 con 0x7f8f540b4140 2024-01-16T15:00:30.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.664+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5404e840 con 0x7f8f540b4140 2024-01-16T15:00:30.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.665+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8f4403e960 con 0x7f8f540b4140 2024-01-16T15:00:30.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.666+0000 7f8f4be00640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 0x7f8f2c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.666+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8f440aca10 con 0x7f8f540b4140 2024-01-16T15:00:30.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.666+0000 7f8f4a800640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 0x7f8f2c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.667+0000 7f8f4a800640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 0x7f8f2c06d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8f5415d3d0 tx=0x7f8f3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.670+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f44077890 con 0x7f8f540b4140 2024-01-16T15:00:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.859+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f540b2d20 con 0x7f8f540b4140 2024-01-16T15:00:30.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.860+0000 7f8f4be00640 1 -- 172.21.15.43:0/1465902970 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f4407aeb0 con 0x7f8f540b4140 2024-01-16T15:00:30.841 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 msgr2=0x7f8f2c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 0x7f8f2c06d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8f5415d3d0 tx=0x7f8f3c009040 comp rx=0 tx=0).stop 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 msgr2=0x7f8f54154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009e50 tx=0x7f8f44004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 shutdown_connections 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f2c06b2c0 0x7f8f2c06d780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f541439b0 0x7f8f541570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f541435d0 0x7f8f5415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 --2- 172.21.15.43:0/1465902970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f540b4140 0x7f8f54154d30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 >> 172.21.15.43:0/1465902970 conn(0x7f8f540cec10 msgr2=0x7f8f540cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:30.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 shutdown_connections 2024-01-16T15:00:30.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:30.863+0000 7f8f59200640 1 -- 172.21.15.43:0/1465902970 wait complete. 2024-01-16T15:00:30.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.080+0000 7f0970800640 1 Processor -- start 2024-01-16T15:00:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.081+0000 7f0970800640 1 -- start start 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.082+0000 7f0970800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f096806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.082+0000 7f0970800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b2080 con 0x7f096815b340 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.082+0000 7f0970800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b2360 con 0x7f09680ce880 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.082+0000 7f0970800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b2640 con 0x7f09680cbcb0 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.083+0000 7f096de00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f096815b340 0x7f09680abff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58480/0 (socket says 172.21.15.43:58480) 2024-01-16T15:00:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.083+0000 7f096de00640 1 -- 172.21.15.43:0/355440681 learned_addr learned my addr 172.21.15.43:0/355440681 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:31.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.083+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f096806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.083+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37236359 0 0) 0x7f09680b2080 con 0x7f096815b340 2024-01-16T15:00:31.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.083+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0948003520 con 0x7f096815b340 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088790205 0 0) 0x7f0948003520 con 0x7f096815b340 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 msgr2=0x7f096806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f096806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 >> v1:172.21.15.69:6789/0 conn(0x7f09680ce880 legacy=0x7f09680b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680b2920 con 0x7f096815b340 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0958001670 con 0x7f096815b340 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f096806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 >> v1:172.21.15.43:6789/0 conn(0x7f096815b340 legacy=0x7f09680abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f09480039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0948003ee0 0x7f09480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0948004d00 0x7f0948005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b2640 con 0x7f0948003ee0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094800ba30 con 0x7f0948004d00 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094800bbf0 con 0x7f09680cbcb0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.084+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0948003ee0 0x7f09480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096de00640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f09480039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096e800640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0948004d00 0x7f0948005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096d400640 1 -- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 msgr2=0x7f09480039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09680cbcb0 0x7f09480039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096d400640 1 -- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0948004d00 msgr2=0x7f0948005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0948004d00 0x7f0948005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096d400640 1 -- 172.21.15.43:0/355440681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950009400 con 0x7f0948003ee0 2024-01-16T15:00:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.085+0000 7f096e800640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0948004d00 0x7f0948005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:31.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.086+0000 7f096d400640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0948003ee0 0x7f09480047c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0950009c20 tx=0x7f09480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdaf55460d55903c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.086+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095002c810 con 0x7f0948003ee0 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.086+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095002c970 con 0x7f0948003ee0 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.086+0000 7f096ca00640 1 -- 172.21.15.43:0/355440681 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095002cce0 con 0x7f0948003ee0 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 -- 172.21.15.43:0/355440681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0948003ee0 msgr2=0x7f09480047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 --2- 172.21.15.43:0/355440681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0948003ee0 0x7f09480047c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0950009c20 tx=0x7f09480057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f096e800640 1 -- 172.21.15.43:0/355440681 reap_dead start 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 -- 172.21.15.43:0/355440681 shutdown_connections 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 -- 172.21.15.43:0/355440681 >> 172.21.15.43:0/355440681 conn(0x7f09680b0d80 msgr2=0x7f09680b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 -- 172.21.15.43:0/355440681 shutdown_connections 2024-01-16T15:00:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.087+0000 7f0970800640 1 -- 172.21.15.43:0/355440681 wait complete. 2024-01-16T15:00:31.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.089+0000 7f0970800640 1 Processor -- start 2024-01-16T15:00:31.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.089+0000 7f0970800640 1 -- start start 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 0x7f096814b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 0x7f096815c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096814f7f0 con 0x7f096814d2d0 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096814bc00 con 0x7f096815b720 2024-01-16T15:00:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f0970800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096814eec0 con 0x7f0948003ee0 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37890/0 (socket says 172.21.15.43:37890) 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 -- 172.21.15.43:0/810924136 learned_addr learned my addr 172.21.15.43:0/810924136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096de00640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 0x7f096814b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096d400640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 0x7f096815c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 -- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 msgr2=0x7f096814b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 0x7f096814b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 -- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 msgr2=0x7f096815c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.090+0000 7f096e800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 0x7f096815c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f096e800640 1 -- 172.21.15.43:0/810924136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096814f1a0 con 0x7f096814d2d0 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f096d400640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 0x7f096815c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f096de00640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 0x7f096814b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f096e800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09680aca20 tx=0x7f096000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960003d80 con 0x7f096814d2d0 2024-01-16T15:00:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096000de50 con 0x7f096814d2d0 2024-01-16T15:00:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096814f480 con 0x7f096814d2d0 2024-01-16T15:00:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09600052b0 con 0x7f096814d2d0 2024-01-16T15:00:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.091+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096815ce90 con 0x7f096814d2d0 2024-01-16T15:00:31.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.092+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096814f610 con 0x7f096814d2d0 2024-01-16T15:00:31.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.093+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0960005410 con 0x7f096814d2d0 2024-01-16T15:00:31.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.095+0000 7f0966000640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 0x7f094006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.095+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0960089780 con 0x7f096814d2d0 2024-01-16T15:00:31.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.095+0000 7f096de00640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 0x7f094006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.095+0000 7f096de00640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 0x7f094006d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0958001eb0 tx=0x7f0958000f70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.099+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09600544d0 con 0x7f096814d2d0 2024-01-16T15:00:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.305+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0968074770 con 0x7f096814d2d0 2024-01-16T15:00:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.306+0000 7f0966000640 1 -- 172.21.15.43:0/810924136 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0960018080 con 0x7f096814d2d0 2024-01-16T15:00:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.308+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 msgr2=0x7f094006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.308+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 0x7f094006d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0958001eb0 tx=0x7f0958000f70 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.308+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 msgr2=0x7f096815d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.308+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09680aca20 tx=0x7f096000dc00 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 shutdown_connections 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f094006b2f0 0x7f094006d7b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f096815b720 0x7f096815c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f096814d2d0 0x7f096815d8c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 --2- 172.21.15.43:0/810924136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0948003ee0 0x7f096814b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 >> 172.21.15.43:0/810924136 conn(0x7f09680b0d80 msgr2=0x7f096815a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 shutdown_connections 2024-01-16T15:00:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.309+0000 7f0970800640 1 -- 172.21.15.43:0/810924136 wait complete. 2024-01-16T15:00:31.309 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:00:31.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:31.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:31.412 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:31.412 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:00:31.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:31.413 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:00:31.460 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:00:31.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.708+0000 7f47ff200640 1 Processor -- start 2024-01-16T15:00:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.708+0000 7f47ff200640 1 -- start start 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47ff200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f47f8143570 0x7f47f80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47ff200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 0x7f47f80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47ff200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8071700 con 0x7f47f815e6b0 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47ff200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f815b410 con 0x7f47f8143570 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47ff200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f815b130 con 0x7f47f8143950 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47fde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f47f8143570 0x7f47f80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47fe800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f47f815e6b0 0x7f47f815a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58486/0 (socket says 172.21.15.43:58486) 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.710+0000 7f47fe800640 1 -- 172.21.15.43:0/81399955 learned_addr learned my addr 172.21.15.43:0/81399955 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fd400640 1 --2- 172.21.15.43:0/81399955 >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 0x7f47f80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fde00640 1 -- 172.21.15.43:0/81399955 >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 msgr2=0x7f47f80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fde00640 1 --2- 172.21.15.43:0/81399955 >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 0x7f47f80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fde00640 1 -- 172.21.15.43:0/81399955 >> v1:172.21.15.43:6789/0 conn(0x7f47f815e6b0 legacy=0x7f47f815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fde00640 1 -- 172.21.15.43:0/81399955 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f815dbd0 con 0x7f47f8143570 2024-01-16T15:00:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.711+0000 7f47fd400640 1 --2- 172.21.15.43:0/81399955 >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 0x7f47f80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.712+0000 7f47fde00640 1 --2- 172.21.15.43:0/81399955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f80cf050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f47f8070f10 tx=0x7f47e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c682e1d95b6aafd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47fca00640 1 -- 172.21.15.43:0/81399955 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e802e070 con 0x7f47f8143570 2024-01-16T15:00:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47fca00640 1 -- 172.21.15.43:0/81399955 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e80315a0 con 0x7f47f8143570 2024-01-16T15:00:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47fca00640 1 -- 172.21.15.43:0/81399955 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8031930 con 0x7f47f8143570 2024-01-16T15:00:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 -- 172.21.15.43:0/81399955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 msgr2=0x7f47f80cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 --2- 172.21.15.43:0/81399955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f80cf050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f47f8070f10 tx=0x7f47e802b840 comp rx=0 tx=0).stop 2024-01-16T15:00:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 -- 172.21.15.43:0/81399955 shutdown_connections 2024-01-16T15:00:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 --2- 172.21.15.43:0/81399955 >> v2:172.21.15.69:3300/0 conn(0x7f47f8143950 0x7f47f80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 --2- 172.21.15.43:0/81399955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f80cf050 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.713+0000 7f47ff200640 1 -- 172.21.15.43:0/81399955 >> 172.21.15.43:0/81399955 conn(0x7f47f806a860 msgr2=0x7f47f806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.714+0000 7f47ff200640 1 -- 172.21.15.43:0/81399955 shutdown_connections 2024-01-16T15:00:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.714+0000 7f47ff200640 1 -- 172.21.15.43:0/81399955 wait complete. 2024-01-16T15:00:31.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.715+0000 7f47ff200640 1 Processor -- start 2024-01-16T15:00:31.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.716+0000 7f47ff200640 1 -- start start 2024-01-16T15:00:31.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47f8143950 0x7f47f81e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47f815e6b0 0x7f47f813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80c8cf0 con 0x7f47f8143570 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f81e5000 con 0x7f47f815e6b0 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47ff200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f81f2e40 con 0x7f47f8143950 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37920/0 (socket says 172.21.15.43:37920) 2024-01-16T15:00:31.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 -- 172.21.15.43:0/1045800281 learned_addr learned my addr 172.21.15.43:0/1045800281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:31.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 -- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47f8143950 msgr2=0x7f47f81e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47f8143950 0x7f47f81e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 -- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47f815e6b0 msgr2=0x7f47f813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fe800640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47f815e6b0 0x7f47f813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47f815e6b0 0x7f47f813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.717+0000 7f47fde00640 1 -- 172.21.15.43:0/1045800281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009400 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.718+0000 7f47fde00640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009890 tx=0x7f47e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.718+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8032900 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.718+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e8032a60 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.718+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f81e4a40 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.718+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8032e10 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.719+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f81e4dd0 con 0x7f47f8143570 2024-01-16T15:00:31.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.720+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f80c8e30 con 0x7f47f8143570 2024-01-16T15:00:31.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.721+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f47e803b3f0 con 0x7f47f8143570 2024-01-16T15:00:31.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.722+0000 7f47f6000640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 0x7f47d006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.722+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f47e80abd40 con 0x7f47f8143570 2024-01-16T15:00:31.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.723+0000 7f47fd400640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 0x7f47d006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.723+0000 7f47fd400640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 0x7f47d006d780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f47e0006fd0 tx=0x7f47e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.727+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47e8076bc0 con 0x7f47f8143570 2024-01-16T15:00:31.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.965+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47f804e840 con 0x7f47f8143570 2024-01-16T15:00:31.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.966+0000 7f47f6000640 1 -- 172.21.15.43:0/1045800281 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f47e807a1e0 con 0x7f47f8143570 2024-01-16T15:00:31.948 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:31.948 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:31.948 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 msgr2=0x7f47d006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 0x7f47d006d780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f47e0006fd0 tx=0x7f47e0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 msgr2=0x7f47f81f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009890 tx=0x7f47e80046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 shutdown_connections 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47d006b2c0 0x7f47d006d780 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47f815e6b0 0x7f47f813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47f8143950 0x7f47f81e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 --2- 172.21.15.43:0/1045800281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47f8143570 0x7f47f81f6790 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 >> 172.21.15.43:0/1045800281 conn(0x7f47f806a860 msgr2=0x7f47f80b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 shutdown_connections 2024-01-16T15:00:31.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:31.969+0000 7f47ff200640 1 -- 172.21.15.43:0/1045800281 wait complete. 2024-01-16T15:00:31.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:32.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.204+0000 7fa193200640 1 Processor -- start 2024-01-16T15:00:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.204+0000 7fa193200640 1 -- start start 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa193200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa19413dfd0 0x7fa19415be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa193200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa1940cf050 0x7fa1940cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa193200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1940ce7c0 con 0x7fa19406c130 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa193200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1940cfa10 con 0x7fa1940cf050 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa193200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa194140900 con 0x7fa19413dfd0 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa191400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa1940cf050 0x7fa1940cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa191400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa1940cf050 0x7fa1940cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37924/0 (socket says 172.21.15.43:37924) 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa191400640 1 -- 172.21.15.43:0/3500026225 learned_addr learned my addr 172.21.15.43:0/3500026225 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.206+0000 7fa191e00640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa19413dfd0 0x7fa19415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.207+0000 7fa191400640 1 -- 172.21.15.43:0/3500026225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa19413dfd0 msgr2=0x7fa19415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.207+0000 7fa191400640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa19413dfd0 0x7fa19415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.207+0000 7fa191400640 1 -- 172.21.15.43:0/3500026225 >> v1:172.21.15.43:6789/0 conn(0x7fa19406c130 legacy=0x7fa19406c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.207+0000 7fa191400640 1 -- 172.21.15.43:0/3500026225 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19415fa70 con 0x7fa1940cf050 2024-01-16T15:00:32.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.208+0000 7fa191400640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1940cf050 0x7fa1940cf4d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009870 tx=0x7fa17c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957aba6f82e3579a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.208+0000 7fa190a00640 1 -- 172.21.15.43:0/3500026225 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c030840 con 0x7fa1940cf050 2024-01-16T15:00:32.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.208+0000 7fa190a00640 1 -- 172.21.15.43:0/3500026225 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c0309a0 con 0x7fa1940cf050 2024-01-16T15:00:32.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.208+0000 7fa190a00640 1 -- 172.21.15.43:0/3500026225 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c030d30 con 0x7fa1940cf050 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 -- 172.21.15.43:0/3500026225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1940cf050 msgr2=0x7fa1940cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1940cf050 0x7fa1940cf4d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009870 tx=0x7fa17c02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 -- 172.21.15.43:0/3500026225 shutdown_connections 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1940cf050 0x7fa1940cf4d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 --2- 172.21.15.43:0/3500026225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa19413dfd0 0x7fa19415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 -- 172.21.15.43:0/3500026225 >> 172.21.15.43:0/3500026225 conn(0x7fa19406a860 msgr2=0x7fa19406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 -- 172.21.15.43:0/3500026225 shutdown_connections 2024-01-16T15:00:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.209+0000 7fa193200640 1 -- 172.21.15.43:0/3500026225 wait complete. 2024-01-16T15:00:32.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.211+0000 7fa193200640 1 Processor -- start 2024-01-16T15:00:32.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.212+0000 7fa193200640 1 -- start start 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa19406c130 0x7fa1940700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1940cf050 0x7fa1940715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194070f70 con 0x7fa19413dfd0 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194071f20 con 0x7fa19406c130 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa193200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940706d0 con 0x7fa1940cf050 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa192800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa192800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37926/0 (socket says 172.21.15.43:37926) 2024-01-16T15:00:32.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa192800640 1 -- 172.21.15.43:0/879973458 learned_addr learned my addr 172.21.15.43:0/879973458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:32.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa191e00640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa19406c130 0x7fa1940700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.213+0000 7fa191400640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1940cf050 0x7fa1940715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 -- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1940cf050 msgr2=0x7fa1940715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1940cf050 0x7fa1940715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 -- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa19406c130 msgr2=0x7fa1940700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa19406c130 0x7fa1940700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 -- 172.21.15.43:0/879973458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0094d0 con 0x7fa19413dfd0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.214+0000 7fa192800640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00c7a0 tx=0x7fa18c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.215+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c003b70 con 0x7fa19413dfd0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.215+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c003cf0 con 0x7fa19413dfd0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.215+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c003ea0 con 0x7fa19413dfd0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.215+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1940709b0 con 0x7fa19413dfd0 2024-01-16T15:00:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.215+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa194156850 con 0x7fa19413dfd0 2024-01-16T15:00:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.216+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19404e840 con 0x7fa19413dfd0 2024-01-16T15:00:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.217+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa18c00aa90 con 0x7fa19413dfd0 2024-01-16T15:00:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.218+0000 7fa18a000640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 0x7fa16c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.218+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa18c0899c0 con 0x7fa19413dfd0 2024-01-16T15:00:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.218+0000 7fa191e00640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 0x7fa16c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.219+0000 7fa191e00640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 0x7fa16c06d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa1840079a0 tx=0x7fa184008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.222+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa18c0547c0 con 0x7fa19413dfd0 2024-01-16T15:00:32.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.417+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1940ce9a0 con 0x7fa19413dfd0 2024-01-16T15:00:32.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.419+0000 7fa18a000640 1 -- 172.21.15.43:0/879973458 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7fa18c057de0 con 0x7fa19413dfd0 2024-01-16T15:00:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:32.400 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 msgr2=0x7fa16c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 0x7fa16c06d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa1840079a0 tx=0x7fa184008040 comp rx=0 tx=0).stop 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 msgr2=0x7fa1940b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00c7a0 tx=0x7fa18c00cc60 comp rx=0 tx=0).stop 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 shutdown_connections 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa16c06b2c0 0x7fa16c06d780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa19413dfd0 0x7fa1940b2b50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1940cf050 0x7fa1940715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 --2- 172.21.15.43:0/879973458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa19406c130 0x7fa1940700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 >> 172.21.15.43:0/879973458 conn(0x7fa19406a860 msgr2=0x7fa19415a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.422+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 shutdown_connections 2024-01-16T15:00:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.423+0000 7fa193200640 1 -- 172.21.15.43:0/879973458 wait complete. 2024-01-16T15:00:32.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:25.424534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:26.432367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:32.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.667+0000 7f4bc7200640 1 Processor -- start 2024-01-16T15:00:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.668+0000 7f4bc7200640 1 -- start start 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc80d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80d2010 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80c6bf0 con 0x7f4bc815d3b0 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80d1400 con 0x7f4bc80d0fa0 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.669+0000 7f4bc5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37938/0 (socket says 172.21.15.43:37938) 2024-01-16T15:00:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 -- 172.21.15.43:0/3878780185 learned_addr learned my addr 172.21.15.43:0/3878780185 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5400640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc80d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 -- 172.21.15.43:0/3878780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 msgr2=0x7f4bc80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc80d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 -- 172.21.15.43:0/3878780185 >> v1:172.21.15.69:6789/0 conn(0x7f4bc80d1bc0 legacy=0x7f4bc80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.670+0000 7f4bc5e00640 1 -- 172.21.15.43:0/3878780185 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80cedc0 con 0x7f4bc815d3b0 2024-01-16T15:00:32.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.671+0000 7f4bc5e00640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009e50 tx=0x7f4bb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494b8b5fd82979e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.671+0000 7f4bc4a00640 1 -- 172.21.15.43:0/3878780185 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb802c9e0 con 0x7f4bc815d3b0 2024-01-16T15:00:32.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc4a00640 1 -- 172.21.15.43:0/3878780185 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb802cb40 con 0x7f4bc815d3b0 2024-01-16T15:00:32.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc4a00640 1 -- 172.21.15.43:0/3878780185 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80343f0 con 0x7f4bc815d3b0 2024-01-16T15:00:32.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 -- 172.21.15.43:0/3878780185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc815d3b0 msgr2=0x7f4bc80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009e50 tx=0x7f4bb802be40 comp rx=0 tx=0).stop 2024-01-16T15:00:32.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 -- 172.21.15.43:0/3878780185 shutdown_connections 2024-01-16T15:00:32.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc80d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 --2- 172.21.15.43:0/3878780185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc815d3b0 0x7f4bc80ce6c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.672+0000 7f4bc7200640 1 -- 172.21.15.43:0/3878780185 >> 172.21.15.43:0/3878780185 conn(0x7f4bc80d30a0 msgr2=0x7f4bc80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.673+0000 7f4bc7200640 1 -- 172.21.15.43:0/3878780185 shutdown_connections 2024-01-16T15:00:32.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.673+0000 7f4bc7200640 1 -- 172.21.15.43:0/3878780185 wait complete. 2024-01-16T15:00:32.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.675+0000 7f4bc7200640 1 Processor -- start 2024-01-16T15:00:32.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.675+0000 7f4bc7200640 1 -- start start 2024-01-16T15:00:32.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc8158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc815d3b0 0x7f4bc81f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80a0940 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80cbc70 con 0x7f4bc815d3b0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80cbf50 con 0x7f4bc80d0fa0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37952/0 (socket says 172.21.15.43:37952) 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.676+0000 7f4bc5400640 1 -- 172.21.15.43:0/2981167518 learned_addr learned my addr 172.21.15.43:0/2981167518 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5e00640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc8158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc6800640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc815d3b0 0x7f4bc81f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 -- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 msgr2=0x7f4bc8158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc8158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 -- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc815d3b0 msgr2=0x7f4bc81f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc815d3b0 0x7f4bc81f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 -- 172.21.15.43:0/2981167518 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009b30 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.677+0000 7f4bc5400640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c7f0 tx=0x7f4bb000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.678+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0004220 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.678+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0004380 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.678+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0007840 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.678+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80cc230 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.678+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc8157b30 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.679+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc80cc750 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.680+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4bb0007bb0 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.681+0000 7f4bbe000640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 0x7f4ba006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.681+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4bb0089b50 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.681+0000 7f4bc5e00640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 0x7f4ba006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.682+0000 7f4bc5e00640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 0x7f4ba006d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8035000 tx=0x7f4bb8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.686+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bc80cc750 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.879+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4bc80cc750 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.880+0000 7f4bbe000640 1 -- 172.21.15.43:0/2981167518 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4bc80cc750 con 0x7f4bc80d1bc0 2024-01-16T15:00:32.861 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.882+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 msgr2=0x7f4ba006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.882+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 0x7f4ba006d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8035000 tx=0x7f4bb8030000 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.882+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 msgr2=0x7f4bc81575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.882+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c7f0 tx=0x7f4bb000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 shutdown_connections 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ba006b2c0 0x7f4ba006d780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc815d3b0 0x7f4bc81f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bc80d1bc0 0x7f4bc81575f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 --2- 172.21.15.43:0/2981167518 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4bc80d0fa0 0x7f4bc8158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 >> 172.21.15.43:0/2981167518 conn(0x7f4bc80d30a0 msgr2=0x7f4bc80c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 shutdown_connections 2024-01-16T15:00:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:32.883+0000 7f4bc7200640 1 -- 172.21.15.43:0/2981167518 wait complete. 2024-01-16T15:00:32.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:33.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.114+0000 7fdebf200640 1 Processor -- start 2024-01-16T15:00:33.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.115+0000 7fdebf200640 1 -- start start 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.116+0000 7fdebf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdec006a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.116+0000 7fdebf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdec006a980 0x7fdec0161040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.116+0000 7fdebf200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec0140580 con 0x7fdec01401a0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.116+0000 7fdebf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec0143850 con 0x7fdec006a980 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.116+0000 7fdebf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec006ae10 con 0x7fdec006f0a0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebde00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdec01401a0 0x7fdec015d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46500/0 (socket says 172.21.15.43:46500) 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebde00640 1 -- 172.21.15.43:0/678858641 learned_addr learned my addr 172.21.15.43:0/678858641 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebd400640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdec006a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125079167 0 0) 0x7fdec0140580 con 0x7fdec01401a0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea0003520 con 0x7fdec01401a0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.117+0000 7fdebe800640 1 --2- 172.21.15.43:0/678858641 >> v2:172.21.15.69:3300/0 conn(0x7fdec006a980 0x7fdec0161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176335046 0 0) 0x7fdea0003520 con 0x7fdec01401a0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 msgr2=0x7fdec006a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdec006a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 >> v2:172.21.15.69:3300/0 conn(0x7fdec006a980 msgr2=0x7fdec0161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 --2- 172.21.15.43:0/678858641 >> v2:172.21.15.69:3300/0 conn(0x7fdec006a980 0x7fdec0161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec01428f0 con 0x7fdec01401a0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fdeb0001670 con 0x7fdec01401a0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 >> v1:172.21.15.43:6789/0 conn(0x7fdec01401a0 legacy=0x7fdec015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdea00039e0 0x7fdea0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdea00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdea0004d00 0x7fdea0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0143850 con 0x7fdea00039e0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec006ae10 con 0x7fdea0004d00 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea000ba30 con 0x7fdec006f0a0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.118+0000 7fdebde00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdea00039e0 0x7fdea0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebe800640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdea0004d00 0x7fdea0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebd400640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdea00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebde00640 1 -- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 msgr2=0x7fdea00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebde00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdec006f0a0 0x7fdea00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebde00640 1 -- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdea0004d00 msgr2=0x7fdea0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebde00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdea0004d00 0x7fdea0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.119+0000 7fdebde00640 1 -- 172.21.15.43:0/678858641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0003520 con 0x7fdea00039e0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.120+0000 7fdebde00640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdea00039e0 0x7fdea0003e00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c0e0 tx=0x7fdea00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4553ec1a7c151cdf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.120+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0005ad0 con 0x7fdea00039e0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.120+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0005c50 con 0x7fdea00039e0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.120+0000 7fdebca00640 1 -- 172.21.15.43:0/678858641 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb002ca50 con 0x7fdea00039e0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 -- 172.21.15.43:0/678858641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdea00039e0 msgr2=0x7fdea0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 --2- 172.21.15.43:0/678858641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdea00039e0 0x7fdea0003e00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c0e0 tx=0x7fdea00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebe800640 1 -- 172.21.15.43:0/678858641 reap_dead start 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 -- 172.21.15.43:0/678858641 shutdown_connections 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 -- 172.21.15.43:0/678858641 >> 172.21.15.43:0/678858641 conn(0x7fdec006bbb0 msgr2=0x7fdec00b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 -- 172.21.15.43:0/678858641 shutdown_connections 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.121+0000 7fdebf200640 1 -- 172.21.15.43:0/678858641 wait complete. 2024-01-16T15:00:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.123+0000 7fdebf200640 1 Processor -- start 2024-01-16T15:00:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.123+0000 7fdebf200640 1 -- start start 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdea00039e0 0x7fdec012f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0153d00 0x7fdec0131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec012f940 con 0x7fdec0130c10 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec012fb70 con 0x7fdec0153d00 2024-01-16T15:00:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec012fe50 con 0x7fdea00039e0 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebe800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebe800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53092/0 (socket says 172.21.15.43:53092) 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.124+0000 7fdebe800640 1 -- 172.21.15.43:0/1388298261 learned_addr learned my addr 172.21.15.43:0/1388298261 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebde00640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdea00039e0 0x7fdec012f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebd400640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0153d00 0x7fdec0131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 -- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdea00039e0 msgr2=0x7fdec012f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdea00039e0 0x7fdec012f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 -- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0153d00 msgr2=0x7fdec0131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0153d00 0x7fdec0131e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 -- 172.21.15.43:0/1388298261 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0130130 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebe800640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdec0154860 tx=0x7fdeb80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb800e070 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb8011e10 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8003990 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.125+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec0130410 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.126+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec0132370 con 0x7fdec0130c10 2024-01-16T15:00:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.127+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec004e840 con 0x7fdec0130c10 2024-01-16T15:00:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.128+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdeb8003af0 con 0x7fdec0130c10 2024-01-16T15:00:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.128+0000 7fdeb6000640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 0x7fde9806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.129+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdeb808a2c0 con 0x7fdec0130c10 2024-01-16T15:00:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.129+0000 7fdebde00640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 0x7fde9806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.130+0000 7fdebde00640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 0x7fde9806d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0034000 tx=0x7fdeb0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.134+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdeb8055010 con 0x7fdec0130c10 2024-01-16T15:00:33.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.328+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdec00b4230 con 0x7fdec0130c10 2024-01-16T15:00:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.329+0000 7fdeb6000640 1 -- 172.21.15.43:0/1388298261 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdeb8058630 con 0x7fdec0130c10 2024-01-16T15:00:33.310 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:00:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.331+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 msgr2=0x7fde9806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.331+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 0x7fde9806d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0034000 tx=0x7fdeb0007000 comp rx=0 tx=0).stop 2024-01-16T15:00:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 msgr2=0x7fdec01410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdec0154860 tx=0x7fdeb80115a0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 shutdown_connections 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde9806b2c0 0x7fde9806d780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0153d00 0x7fdec0131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdec0130c10 0x7fdec01410c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 --2- 172.21.15.43:0/1388298261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdea00039e0 0x7fdec012f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 >> 172.21.15.43:0/1388298261 conn(0x7fdec006bbb0 msgr2=0x7fdec00b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 shutdown_connections 2024-01-16T15:00:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.332+0000 7fdebf200640 1 -- 172.21.15.43:0/1388298261 wait complete. 2024-01-16T15:00:33.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.568+0000 7fd729e00640 1 Processor -- start 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.568+0000 7fd729e00640 1 -- start start 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd729e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7240ad450 0x7fd724154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd729e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd724070850 0x7fd724071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd729e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7240ce650 con 0x7fd724153ea0 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd729e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7240ad8e0 con 0x7fd7240ad450 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd729e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7241547c0 con 0x7fd724070850 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd71be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7240ad450 0x7fd724154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd71be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7240ad450 0x7fd724154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53094/0 (socket says 172.21.15.43:53094) 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.570+0000 7fd71be00640 1 -- 172.21.15.43:0/2611187382 learned_addr learned my addr 172.21.15.43:0/2611187382 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.571+0000 7fd729400640 1 --2- 172.21.15.43:0/2611187382 >> v2:172.21.15.69:3300/0 conn(0x7fd724070850 0x7fd724071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.571+0000 7fd71be00640 1 -- 172.21.15.43:0/2611187382 >> v2:172.21.15.69:3300/0 conn(0x7fd724070850 msgr2=0x7fd724071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.571+0000 7fd71be00640 1 --2- 172.21.15.43:0/2611187382 >> v2:172.21.15.69:3300/0 conn(0x7fd724070850 0x7fd724071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.571+0000 7fd71be00640 1 -- 172.21.15.43:0/2611187382 >> v1:172.21.15.69:6789/0 conn(0x7fd724153ea0 legacy=0x7fd72415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.571+0000 7fd71be00640 1 -- 172.21.15.43:0/2611187382 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72415e0a0 con 0x7fd7240ad450 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.572+0000 7fd71be00640 1 --2- 172.21.15.43:0/2611187382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd724154280 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd724157e50 tx=0x7fd70c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41bd6d54efe2027 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.572+0000 7fd723e00640 1 -- 172.21.15.43:0/2611187382 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c031870 con 0x7fd7240ad450 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.572+0000 7fd723e00640 1 -- 172.21.15.43:0/2611187382 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c0319d0 con 0x7fd7240ad450 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.572+0000 7fd723e00640 1 -- 172.21.15.43:0/2611187382 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c031d60 con 0x7fd7240ad450 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 -- 172.21.15.43:0/2611187382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 msgr2=0x7fd724154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 --2- 172.21.15.43:0/2611187382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd724154280 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd724157e50 tx=0x7fd70c02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 -- 172.21.15.43:0/2611187382 shutdown_connections 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 --2- 172.21.15.43:0/2611187382 >> v2:172.21.15.69:3300/0 conn(0x7fd724070850 0x7fd724071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 --2- 172.21.15.43:0/2611187382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd724154280 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 -- 172.21.15.43:0/2611187382 >> 172.21.15.43:0/2611187382 conn(0x7fd7240cec10 msgr2=0x7fd7240cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 -- 172.21.15.43:0/2611187382 shutdown_connections 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.573+0000 7fd729e00640 1 -- 172.21.15.43:0/2611187382 wait complete. 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.575+0000 7fd729e00640 1 Processor -- start 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.575+0000 7fd729e00640 1 -- start start 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd724070850 0x7fd724152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd724153ea0 0x7fd72414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72414f8a0 con 0x7fd7240ad450 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72414ec20 con 0x7fd724070850 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd729e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72414ef00 con 0x7fd724153ea0 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd71be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd71be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53110/0 (socket says 172.21.15.43:53110) 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd71be00640 1 -- 172.21.15.43:0/883248841 learned_addr learned my addr 172.21.15.43:0/883248841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.576+0000 7fd728a00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd724070850 0x7fd724152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd729400640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd724153ea0 0x7fd72414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 -- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd724153ea0 msgr2=0x7fd72414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd724153ea0 0x7fd72414d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 -- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd724070850 msgr2=0x7fd724152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd724070850 0x7fd724152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 -- 172.21.15.43:0/883248841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c0094d0 con 0x7fd7240ad450 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd71be00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0097f0 tx=0x7fd70c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c031820 con 0x7fd7240ad450 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c0443f0 con 0x7fd7240ad450 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0447a0 con 0x7fd7240ad450 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.577+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72414f1e0 con 0x7fd7240ad450 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.578+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd724153510 con 0x7fd7240ad450 2024-01-16T15:00:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.579+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd724153700 con 0x7fd7240ad450 2024-01-16T15:00:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.580+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd70c04d450 con 0x7fd7240ad450 2024-01-16T15:00:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.580+0000 7fd721600640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 0x7fd6fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.580+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd70c02d040 con 0x7fd7240ad450 2024-01-16T15:00:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.581+0000 7fd728a00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 0x7fd6fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.581+0000 7fd728a00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 0x7fd6fc06d780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd714006fd0 tx=0x7fd714008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.585+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd70c0771f0 con 0x7fd7240ad450 2024-01-16T15:00:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.777+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd72414f9e0 con 0x7fd7240ad450 2024-01-16T15:00:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.778+0000 7fd721600640 1 -- 172.21.15.43:0/883248841 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd72414f9e0 con 0x7fd7240ad450 2024-01-16T15:00:33.759 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:00:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 msgr2=0x7fd6fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 0x7fd6fc06d780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd714006fd0 tx=0x7fd714008040 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 msgr2=0x7fd72414d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0097f0 tx=0x7fd70c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 shutdown_connections 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd6fc06b2c0 0x7fd6fc06d780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd724153ea0 0x7fd72414d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7240ad450 0x7fd72414d340 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.780+0000 7fd729e00640 1 --2- 172.21.15.43:0/883248841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd724070850 0x7fd724152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.781+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 >> 172.21.15.43:0/883248841 conn(0x7fd7240cec10 msgr2=0x7fd7240ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.781+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 shutdown_connections 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:33.781+0000 7fd729e00640 1 -- 172.21.15.43:0/883248841 wait complete. 2024-01-16T15:00:33.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:33.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.000+0000 7f13c6600640 1 Processor -- start 2024-01-16T15:00:33.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.000+0000 7f13c6600640 1 -- start start 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13c6600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006d520 0x7f13c006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13c6600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c015c360 con 0x7f13c00ad720 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13c6600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c015faf0 con 0x7f13c00abff0 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13c6600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00b41a0 con 0x7f13c006d520 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13b7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f13c00ad720 0x7f13c015ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46506/0 (socket says 172.21.15.43:46506) 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13b7e00640 1 -- 172.21.15.43:0/260440335 learned_addr learned my addr 172.21.15.43:0/260440335 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.002+0000 7f13bf400640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006d520 0x7f13c006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842638115 0 0) 0x7f13c015c360 con 0x7f13c00ad720 2024-01-16T15:00:33.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0003520 con 0x7f13c00ad720 2024-01-16T15:00:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598257697 0 0) 0x7f13a0003520 con 0x7f13c00ad720 2024-01-16T15:00:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006d520 msgr2=0x7f13c006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006d520 0x7f13c006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 >> v1:172.21.15.69:6789/0 conn(0x7f13c00abff0 legacy=0x7f13c015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c006de40 con 0x7f13c00ad720 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f13a8001670 con 0x7f13c00ad720 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 >> v1:172.21.15.43:6789/0 conn(0x7f13c00ad720 legacy=0x7f13c015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a00039a0 0x7f13a0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0004300 0x7f13a00047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004fe0 0x7f13a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000bec0 con 0x7f13a0004300 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000c030 con 0x7f13a0004fe0 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000c1f0 con 0x7f13a00039a0 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.003+0000 7f13bf400640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006d520 0x7f13c006d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0004300 0x7f13a00047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13bfe00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004fe0 0x7f13a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 -- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a00039a0 msgr2=0x7f13a0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a00039a0 0x7f13a0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 -- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004fe0 msgr2=0x7f13a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004fe0 0x7f13a000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13b7e00640 1 -- 172.21.15.43:0/260440335 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0003520 con 0x7f13a0004300 2024-01-16T15:00:33.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.004+0000 7f13bfe00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004fe0 0x7f13a000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.005+0000 7f13b7e00640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0004300 0x7f13a00047e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c210 tx=0x7f13a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976745eb1ca3ddc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.005+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8005ad0 con 0x7f13a0004300 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.005+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8005c30 con 0x7f13a0004300 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.005+0000 7f13bea00640 1 -- 172.21.15.43:0/260440335 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a802c870 con 0x7f13a0004300 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 -- 172.21.15.43:0/260440335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0004300 msgr2=0x7f13a00047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 --2- 172.21.15.43:0/260440335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0004300 0x7f13a00047e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c210 tx=0x7f13a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13bfe00640 1 -- 172.21.15.43:0/260440335 reap_dead start 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 -- 172.21.15.43:0/260440335 shutdown_connections 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 -- 172.21.15.43:0/260440335 >> 172.21.15.43:0/260440335 conn(0x7f13c00b0d80 msgr2=0x7f13c00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 -- 172.21.15.43:0/260440335 shutdown_connections 2024-01-16T15:00:33.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.006+0000 7f13c6600640 1 -- 172.21.15.43:0/260440335 wait complete. 2024-01-16T15:00:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.008+0000 7f13c6600640 1 Processor -- start 2024-01-16T15:00:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.008+0000 7f13c6600640 1 -- start start 2024-01-16T15:00:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 0x7f13c015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c0158f80 0x7f13c009cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015aa20 con 0x7f13c01474a0 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015a050 con 0x7f13c0158f80 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13c6600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c009cff0 con 0x7f13a0004300 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13bfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13bfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53142/0 (socket says 172.21.15.43:53142) 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.009+0000 7f13bfe00640 1 -- 172.21.15.43:0/1040782394 learned_addr learned my addr 172.21.15.43:0/1040782394 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13b7e00640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c0158f80 0x7f13c009cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bf400640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 0x7f13c015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 -- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 msgr2=0x7f13c015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 0x7f13c015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 -- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c0158f80 msgr2=0x7f13c009cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c0158f80 0x7f13c009cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 -- 172.21.15.43:0/1040782394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c0159440 con 0x7f13c01474a0 2024-01-16T15:00:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bf400640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 0x7f13c015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13bfe00640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f13c0072930 tx=0x7f13b800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.010+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8003d80 con 0x7f13c01474a0 2024-01-16T15:00:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.011+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b800de60 con 0x7f13c01474a0 2024-01-16T15:00:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.011+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b80052b0 con 0x7f13c01474a0 2024-01-16T15:00:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.011+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c0159720 con 0x7f13c01474a0 2024-01-16T15:00:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.011+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c0159ce0 con 0x7f13c01474a0 2024-01-16T15:00:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.012+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c0074770 con 0x7f13c01474a0 2024-01-16T15:00:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.013+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f13b8005410 con 0x7f13c01474a0 2024-01-16T15:00:33.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.014+0000 7f13b7400640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 0x7f139806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.014+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f13b8089160 con 0x7f13c01474a0 2024-01-16T15:00:33.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.014+0000 7f13bf400640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 0x7f139806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.015+0000 7f13bf400640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 0x7f139806d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13b0004790 tx=0x7f13b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.018+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13b8053e20 con 0x7f13c01474a0 2024-01-16T15:00:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.248+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13c015a190 con 0x7f13c01474a0 2024-01-16T15:00:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.249+0000 7f13b7400640 1 -- 172.21.15.43:0/1040782394 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f13b8057440 con 0x7f13c01474a0 2024-01-16T15:00:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:34.230 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T15:00:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.251+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 msgr2=0x7f139806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.251+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 0x7f139806d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13b0004790 tx=0x7f13b0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 msgr2=0x7f13c0147930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f13c0072930 tx=0x7f13b800dc70 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 shutdown_connections 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b2f0 0x7f139806d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13c0158f80 0x7f13c009cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c01474a0 0x7f13c0147930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 --2- 172.21.15.43:0/1040782394 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13a0004300 0x7f13c015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 >> 172.21.15.43:0/1040782394 conn(0x7f13c00b0d80 msgr2=0x7f13c00b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 shutdown_connections 2024-01-16T15:00:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.252+0000 7f13c6600640 1 -- 172.21.15.43:0/1040782394 wait complete. 2024-01-16T15:00:34.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.489+0000 7f410a600640 1 Processor -- start 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.489+0000 7f410a600640 1 -- start start 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.490+0000 7f410a600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.490+0000 7f410a600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41040cf050 0x7f41040cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.490+0000 7f410a600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41040ce7c0 con 0x7f410406c130 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.491+0000 7f410a600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41040cfa10 con 0x7f41040cf050 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.491+0000 7f410a600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104140900 con 0x7f410413dfd0 2024-01-16T15:00:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.491+0000 7f4109200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.491+0000 7f4109200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:59910/0 (socket says 172.21.15.43:59910) 2024-01-16T15:00:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.491+0000 7f4109200640 1 -- 172.21.15.43:0/2390131759 learned_addr learned my addr 172.21.15.43:0/2390131759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.492+0000 7f4103e00640 1 --2- 172.21.15.43:0/2390131759 >> v2:172.21.15.69:3300/0 conn(0x7f41040cf050 0x7f41040cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.492+0000 7f4109200640 1 -- 172.21.15.43:0/2390131759 >> v2:172.21.15.69:3300/0 conn(0x7f41040cf050 msgr2=0x7f41040cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.492+0000 7f4109200640 1 --2- 172.21.15.43:0/2390131759 >> v2:172.21.15.69:3300/0 conn(0x7f41040cf050 0x7f41040cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.492+0000 7f4109200640 1 -- 172.21.15.43:0/2390131759 >> v1:172.21.15.69:6789/0 conn(0x7f410406c130 legacy=0x7f410406c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.492+0000 7f4109200640 1 -- 172.21.15.43:0/2390131759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410415fa70 con 0x7f410413dfd0 2024-01-16T15:00:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.494+0000 7f4109200640 1 --2- 172.21.15.43:0/2390131759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009870 tx=0x7f40f402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e4a5029e8815cec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.494+0000 7f4103400640 1 -- 172.21.15.43:0/2390131759 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4030840 con 0x7f410413dfd0 2024-01-16T15:00:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f4103400640 1 -- 172.21.15.43:0/2390131759 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f40309a0 con 0x7f410413dfd0 2024-01-16T15:00:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f4103400640 1 -- 172.21.15.43:0/2390131759 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4030d30 con 0x7f410413dfd0 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 -- 172.21.15.43:0/2390131759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 msgr2=0x7f410415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 --2- 172.21.15.43:0/2390131759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009870 tx=0x7f40f402b980 comp rx=0 tx=0).stop 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 -- 172.21.15.43:0/2390131759 shutdown_connections 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 --2- 172.21.15.43:0/2390131759 >> v2:172.21.15.69:3300/0 conn(0x7f41040cf050 0x7f41040cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 --2- 172.21.15.43:0/2390131759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f410413dfd0 0x7f410415be70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 -- 172.21.15.43:0/2390131759 >> 172.21.15.43:0/2390131759 conn(0x7f410406a860 msgr2=0x7f410406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 -- 172.21.15.43:0/2390131759 shutdown_connections 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.495+0000 7f410a600640 1 -- 172.21.15.43:0/2390131759 wait complete. 2024-01-16T15:00:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.497+0000 7f410a600640 1 Processor -- start 2024-01-16T15:00:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.497+0000 7f410a600640 1 -- start start 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 0x7f41041e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41040cf050 0x7f4104156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41041576c0 con 0x7f410413dfd0 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41041579a0 con 0x7f410406c130 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.498+0000 7f410a600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104157c80 con 0x7f41040cf050 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53154/0 (socket says 172.21.15.43:53154) 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 -- 172.21.15.43:0/1772436840 learned_addr learned my addr 172.21.15.43:0/1772436840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109200640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 0x7f41041e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4103e00640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41040cf050 0x7f4104156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 -- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41040cf050 msgr2=0x7f4104156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41040cf050 0x7f4104156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 -- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 msgr2=0x7f41041e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 0x7f41041e5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109c00640 1 -- 172.21.15.43:0/1772436840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40094d0 con 0x7f410413dfd0 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.499+0000 7f4109200640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 0x7f41041e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f4109c00640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c710 tx=0x7f40fc00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc006d50 con 0x7f410413dfd0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc00ce20 con 0x7f410413dfd0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc003bb0 con 0x7f410413dfd0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104157f60 con 0x7f410413dfd0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.500+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41041573d0 con 0x7f410413dfd0 2024-01-16T15:00:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.501+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410404e840 con 0x7f410413dfd0 2024-01-16T15:00:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.502+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f40fc003d10 con 0x7f410413dfd0 2024-01-16T15:00:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.503+0000 7f4100c00640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 0x7f40dc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.503+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f40fc089790 con 0x7f410413dfd0 2024-01-16T15:00:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.504+0000 7f4109200640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 0x7f40dc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.504+0000 7f4109200640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 0x7f40dc06d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036000 tx=0x7f40f403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.507+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40fc0580a0 con 0x7f410413dfd0 2024-01-16T15:00:34.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.699+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41040ce9a0 con 0x7f410413dfd0 2024-01-16T15:00:34.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.701+0000 7f4100c00640 1 -- 172.21.15.43:0/1772436840 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f40fc057c30 con 0x7f410413dfd0 2024-01-16T15:00:34.682 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:34.682 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:34.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 msgr2=0x7f40dc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 0x7f40dc06d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036000 tx=0x7f40f403b040 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 msgr2=0x7f410415d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c710 tx=0x7f40fc00cbd0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 shutdown_connections 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40dc06b2c0 0x7f40dc06d780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f410413dfd0 0x7f410415d0b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41040cf050 0x7f4104156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 --2- 172.21.15.43:0/1772436840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f410406c130 0x7f41041e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 >> 172.21.15.43:0/1772436840 conn(0x7f410406a860 msgr2=0x7f4104139c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 shutdown_connections 2024-01-16T15:00:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.704+0000 7f410a600640 1 -- 172.21.15.43:0/1772436840 wait complete. 2024-01-16T15:00:34.704 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:00:34.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:34.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.940+0000 7f4f75200640 1 Processor -- start 2024-01-16T15:00:34.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.941+0000 7f4f75200640 1 -- start start 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 0x7f4f700d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 0x7f4f700d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f700cf7a0 0x7f4f700cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f700cf0c0 con 0x7f4f700cf7a0 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f700d1e40 con 0x7f4f700d14a0 2024-01-16T15:00:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f75200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700b01b0 con 0x7f4f7015d5b0 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.942+0000 7f4f6fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f700cf7a0 0x7f4f700cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f700cf7a0 0x7f4f700cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53166/0 (socket says 172.21.15.43:53166) 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 -- 172.21.15.43:0/1240507972 learned_addr learned my addr 172.21.15.43:0/1240507972 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 0x7f4f700d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f67e00640 1 --2- 172.21.15.43:0/1240507972 >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 0x7f4f700d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 -- 172.21.15.43:0/1240507972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 msgr2=0x7f4f700d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 0x7f4f700d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 -- 172.21.15.43:0/1240507972 >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 msgr2=0x7f4f700d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 --2- 172.21.15.43:0/1240507972 >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 0x7f4f700d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6fe00640 1 -- 172.21.15.43:0/1240507972 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7015d990 con 0x7f4f700cf7a0 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f67e00640 1 --2- 172.21.15.43:0/1240507972 >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 0x7f4f700d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.943+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 0x7f4f700d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.944+0000 7f4f6fe00640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f700cfc00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f7008bf60 tx=0x7f4f6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e4ce5697ba8d8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.945+0000 7f4f6ea00640 1 -- 172.21.15.43:0/1240507972 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6802ca80 con 0x7f4f700cf7a0 2024-01-16T15:00:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.945+0000 7f4f6ea00640 1 -- 172.21.15.43:0/1240507972 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6802cbe0 con 0x7f4f700cf7a0 2024-01-16T15:00:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.945+0000 7f4f6ea00640 1 -- 172.21.15.43:0/1240507972 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680343f0 con 0x7f4f700cf7a0 2024-01-16T15:00:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 -- 172.21.15.43:0/1240507972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 msgr2=0x7f4f700cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f700cfc00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f7008bf60 tx=0x7f4f6802bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 -- 172.21.15.43:0/1240507972 shutdown_connections 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f700cfc00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 --2- 172.21.15.43:0/1240507972 >> v2:172.21.15.69:3300/0 conn(0x7f4f700d14a0 0x7f4f700d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 --2- 172.21.15.43:0/1240507972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f7015d5b0 0x7f4f700d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 -- 172.21.15.43:0/1240507972 >> 172.21.15.43:0/1240507972 conn(0x7f4f7009c3b0 msgr2=0x7f4f700d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 -- 172.21.15.43:0/1240507972 shutdown_connections 2024-01-16T15:00:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.946+0000 7f4f75200640 1 -- 172.21.15.43:0/1240507972 wait complete. 2024-01-16T15:00:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.948+0000 7f4f75200640 1 Processor -- start 2024-01-16T15:00:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.948+0000 7f4f75200640 1 -- start start 2024-01-16T15:00:34.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f700d14a0 0x7f4f700cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 0x7f4f700b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f70159210 con 0x7f4f700cf7a0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700cc530 con 0x7f4f7015d5b0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f75200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700b6c60 con 0x7f4f700d14a0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f6f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f6f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53176/0 (socket says 172.21.15.43:53176) 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f6f400640 1 -- 172.21.15.43:0/2770453086 learned_addr learned my addr 172.21.15.43:0/2770453086 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f6fe00640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 0x7f4f700b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.949+0000 7f4f67e00640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f700d14a0 0x7f4f700cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 -- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f700d14a0 msgr2=0x7f4f700cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f700d14a0 0x7f4f700cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 -- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 msgr2=0x7f4f700b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 0x7f4f700b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 -- 172.21.15.43:0/2770453086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68009b30 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6fe00640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 0x7f4f700b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f6f400640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000c7f0 tx=0x7f4f6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60007a30 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60007b90 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000ce40 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f700b6f40 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.950+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f700b7500 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.952+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7004e840 con 0x7f4f700cf7a0 2024-01-16T15:00:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.952+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4f600113f0 con 0x7f4f700cf7a0 2024-01-16T15:00:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.953+0000 7f4f67400640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 0x7f4f4806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.954+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4f60089e40 con 0x7f4f700cf7a0 2024-01-16T15:00:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.954+0000 7f4f67e00640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 0x7f4f4806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.954+0000 7f4f67e00640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 0x7f4f4806d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4f701f5c40 tx=0x7f4f58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:34.958+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f60054b90 con 0x7f4f700cf7a0 2024-01-16T15:00:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.188+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f700678d0 con 0x7f4f700cf7a0 2024-01-16T15:00:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.189+0000 7f4f67400640 1 -- 172.21.15.43:0/2770453086 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f4f6001f090 con 0x7f4f700cf7a0 2024-01-16T15:00:35.170 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:35.170 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:35.171 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 msgr2=0x7f4f4806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 0x7f4f4806d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4f701f5c40 tx=0x7f4f58008040 comp rx=0 tx=0).stop 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 msgr2=0x7f4f70158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000c7f0 tx=0x7f4f6000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 shutdown_connections 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4806b2c0 0x7f4f4806d780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f7015d5b0 0x7f4f700b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f700d14a0 0x7f4f700cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 --2- 172.21.15.43:0/2770453086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f700cf7a0 0x7f4f70158cd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 >> 172.21.15.43:0/2770453086 conn(0x7f4f7009c3b0 msgr2=0x7f4f7015c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 shutdown_connections 2024-01-16T15:00:35.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.192+0000 7f4f75200640 1 -- 172.21.15.43:0/2770453086 wait complete. 2024-01-16T15:00:35.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.434+0000 7fdd6f800640 1 Processor -- start 2024-01-16T15:00:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.434+0000 7fdd6f800640 1 -- start start 2024-01-16T15:00:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6f800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd68160970 0x7fdd680b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6f800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd68071cd0 con 0x7fdd6806dc30 2024-01-16T15:00:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6f800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd680b2120 con 0x7fdd680b3140 2024-01-16T15:00:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6f800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd680b2400 con 0x7fdd68160970 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6d800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdd6806dc30 0x7fdd680acbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46516/0 (socket says 172.21.15.43:46516) 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.436+0000 7fdd6d800640 1 -- 172.21.15.43:0/4187074789 learned_addr learned my addr 172.21.15.43:0/4187074789 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797693353 0 0) 0x7fdd68071cd0 con 0x7fdd6806dc30 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/4187074789 >> v2:172.21.15.69:3300/0 conn(0x7fdd68160970 0x7fdd680b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd48003520 con 0x7fdd6806dc30 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776521349 0 0) 0x7fdd48003520 con 0x7fdd6806dc30 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 >> v2:172.21.15.69:3300/0 conn(0x7fdd68160970 msgr2=0x7fdd680b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 --2- 172.21.15.43:0/4187074789 >> v2:172.21.15.69:3300/0 conn(0x7fdd68160970 0x7fdd680b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 >> v1:172.21.15.69:6789/0 conn(0x7fdd680b3140 legacy=0x7fdd6806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680b26e0 con 0x7fdd6806dc30 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fdd600019f0 con 0x7fdd6806dc30 2024-01-16T15:00:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 >> v1:172.21.15.43:6789/0 conn(0x7fdd6806dc30 legacy=0x7fdd680acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.437+0000 7fdd67400640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd480052a0 0x7fdd4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680b2400 con 0x7fdd48003970 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4800c170 con 0x7fdd480047c0 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4800c330 con 0x7fdd480052a0 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd67e00640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 -- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd480052a0 msgr2=0x7fdd4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd480052a0 0x7fdd4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 -- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd480047c0 msgr2=0x7fdd48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd480047c0 0x7fdd48004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6ce00640 1 -- 172.21.15.43:0/4187074789 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd48003520 con 0x7fdd48003970 2024-01-16T15:00:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.438+0000 7fdd6d800640 1 -- 172.21.15.43:0/4187074789 reap_dead start 2024-01-16T15:00:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.440+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800ba00 tx=0x7fdd480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e419d03925e04c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.440+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58032620 con 0x7fdd48003970 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.440+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd58032780 con 0x7fdd48003970 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.440+0000 7fdd67400640 1 -- 172.21.15.43:0/4187074789 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58032b10 con 0x7fdd48003970 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 -- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 msgr2=0x7fdd48003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800ba00 tx=0x7fdd480057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 -- 172.21.15.43:0/4187074789 shutdown_connections 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 --2- 172.21.15.43:0/4187074789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd48003970 0x7fdd48003d90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 -- 172.21.15.43:0/4187074789 >> 172.21.15.43:0/4187074789 conn(0x7fdd680b0d80 msgr2=0x7fdd680b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 -- 172.21.15.43:0/4187074789 shutdown_connections 2024-01-16T15:00:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.441+0000 7fdd6f800640 1 -- 172.21.15.43:0/4187074789 wait complete. 2024-01-16T15:00:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.443+0000 7fdd6f800640 1 Processor -- start 2024-01-16T15:00:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.443+0000 7fdd6f800640 1 -- start start 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd48003970 0x7fdd68147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd6815ef90 0x7fdd681469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68149db0 con 0x7fdd681493f0 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68148580 con 0x7fdd48003970 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6f800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68148860 con 0x7fdd6815ef90 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53202/0 (socket says 172.21.15.43:53202) 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.444+0000 7fdd6d800640 1 -- 172.21.15.43:0/2092990135 learned_addr learned my addr 172.21.15.43:0/2092990135 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd48003970 0x7fdd68147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd67e00640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd6815ef90 0x7fdd681469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 -- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd6815ef90 msgr2=0x7fdd681469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd6815ef90 0x7fdd681469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 -- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd48003970 msgr2=0x7fdd68147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd48003970 0x7fdd68147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 -- 172.21.15.43:0/2092990135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd68148b40 con 0x7fdd681493f0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6d800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdd680aecc0 tx=0x7fdd6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000cea0 con 0x7fdd681493f0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6000a5b0 con 0x7fdd681493f0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000a940 con 0x7fdd681493f0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.445+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd68148e20 con 0x7fdd681493f0 2024-01-16T15:00:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.446+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd68149100 con 0x7fdd681493f0 2024-01-16T15:00:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.447+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd68074770 con 0x7fdd681493f0 2024-01-16T15:00:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.448+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdd6000aaa0 con 0x7fdd681493f0 2024-01-16T15:00:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.448+0000 7fdd64c00640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 0x7fdd4006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.449+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdd60089480 con 0x7fdd681493f0 2024-01-16T15:00:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.449+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 0x7fdd4006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.450+0000 7fdd6ce00640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 0x7fdd4006d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd48004740 tx=0x7fdd5803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.453+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd60054140 con 0x7fdd681493f0 2024-01-16T15:00:35.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.683+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd6814a850 con 0x7fdd681493f0 2024-01-16T15:00:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.684+0000 7fdd64c00640 1 -- 172.21.15.43:0/2092990135 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fdd6001a020 con 0x7fdd681493f0 2024-01-16T15:00:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2634054568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2634054568},{"type":"v1","addr":"172.21.15.43:6837","nonce":2634054568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1229050345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1229050345},{"type":"v1","addr":"172.21.15.69:6835","nonce":1229050345}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2508559858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2508559858},{"type":"v1","addr":"172.21.15.43:6835","nonce":2508559858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:27.130477+0000","modified":"2024-01-16T14:59:30.251270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/971655709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":971655709},{"type":"v1","addr":"172.21.15.69:6834","nonce":971655709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.686+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 msgr2=0x7fdd4006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.686+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 0x7fdd4006d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd48004740 tx=0x7fdd5803d040 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.686+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 msgr2=0x7fdd68149870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.686+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdd680aecc0 tx=0x7fdd6000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 shutdown_connections 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd4006b2c0 0x7fdd4006d780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd681493f0 0x7fdd68149870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd6815ef90 0x7fdd681469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 --2- 172.21.15.43:0/2092990135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd48003970 0x7fdd68147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 >> 172.21.15.43:0/2092990135 conn(0x7fdd680b0d80 msgr2=0x7fdd6806d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 shutdown_connections 2024-01-16T15:00:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.687+0000 7fdd6f800640 1 -- 172.21.15.43:0/2092990135 wait complete. 2024-01-16T15:00:35.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.929+0000 7f7c82600640 1 Processor -- start 2024-01-16T15:00:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.930+0000 7f7c82600640 1 -- start start 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c82600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c7c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c82600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0afd30 con 0x7f7c7c15d370 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c82600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0d0c30 con 0x7f7c7c0d00d0 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c82600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0cef60 con 0x7f7c7c0b1f00 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c81200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7c7c15d370 0x7f7c7c0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46520/0 (socket says 172.21.15.43:46520) 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.931+0000 7f7c81200640 1 -- 172.21.15.43:0/1653130308 learned_addr learned my addr 172.21.15.43:0/1653130308 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c7c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405434961 0 0) 0x7f7c7c0afd30 con 0x7f7c7c15d370 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c003520 con 0x7f7c7c15d370 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643675508 0 0) 0x7f7c5c003520 con 0x7f7c7c15d370 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 msgr2=0x7f7c7c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c7c0d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 >> v1:172.21.15.69:6789/0 conn(0x7f7c7c0d00d0 legacy=0x7f7c7c0b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0d0ef0 con 0x7f7c7c15d370 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7c6c001670 con 0x7f7c7c15d370 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 >> v1:172.21.15.43:6789/0 conn(0x7f7c7c15d370 legacy=0x7f7c7c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c5c003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c003f50 0x7f7c5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0cef60 con 0x7f7c5c004d00 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c00bb40 con 0x7f7c5c003f50 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.932+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c00bd00 con 0x7f7c7c0b1f00 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c73e00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c003f50 0x7f7c5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81200640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c5c003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 -- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 msgr2=0x7f7c5c003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c7c0b1f00 0x7f7c5c003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 -- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c003f50 msgr2=0x7f7c5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c003f50 0x7f7c5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.933+0000 7f7c81c00640 1 -- 172.21.15.43:0/1653130308 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c003520 con 0x7f7c5c004d00 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.934+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009fc0 tx=0x7f7c5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504d89b3e3e24300 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7402cc20 con 0x7f7c5c004d00 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7400f040 con 0x7f7c5c004d00 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c7be00640 1 -- 172.21.15.43:0/1653130308 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74034440 con 0x7f7c5c004d00 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c82600640 1 -- 172.21.15.43:0/1653130308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c5c004d00 msgr2=0x7f7c5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c82600640 1 --2- 172.21.15.43:0/1653130308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009fc0 tx=0x7f7c5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c81c00640 1 -- 172.21.15.43:0/1653130308 reap_dead start 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c82600640 1 -- 172.21.15.43:0/1653130308 shutdown_connections 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.935+0000 7f7c82600640 1 -- 172.21.15.43:0/1653130308 >> 172.21.15.43:0/1653130308 conn(0x7f7c7c09c3d0 msgr2=0x7f7c7c0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.936+0000 7f7c82600640 1 -- 172.21.15.43:0/1653130308 shutdown_connections 2024-01-16T15:00:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.936+0000 7f7c82600640 1 -- 172.21.15.43:0/1653130308 wait complete. 2024-01-16T15:00:35.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.937+0000 7f7c82600640 1 Processor -- start 2024-01-16T15:00:35.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.937+0000 7f7c82600640 1 -- start start 2024-01-16T15:00:35.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c004d00 0x7f7c7c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 0x7f7c7c0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f2820 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c15a980 con 0x7f7c7c0c8e40 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.938+0000 7f7c82600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f5ec0 con 0x7f7c5c004d00 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53216/0 (socket says 172.21.15.43:53216) 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 -- 172.21.15.43:0/1492621337 learned_addr learned my addr 172.21.15.43:0/1492621337 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 0x7f7c7c0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 -- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c004d00 msgr2=0x7f7c7c15a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c004d00 0x7f7c7c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 -- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 msgr2=0x7f7c7c0b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 0x7f7c7c0b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c73e00640 1 -- 172.21.15.43:0/1492621337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c1f61a0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.939+0000 7f7c81c00640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 0x7f7c7c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c73e00640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400f7a0 tx=0x7f7c6400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400feb0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c640052e0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400d990 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c1f6480 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.940+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c0c97c0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.941+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c04e840 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.942+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7c6400daf0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.943+0000 7f7c79600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 0x7f7c5406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.943+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7c64089cb0 con 0x7f7c7c0c89c0 2024-01-16T15:00:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.943+0000 7f7c81200640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 0x7f7c5406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.944+0000 7f7c81200640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 0x7f7c5406d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0058b0 tx=0x7f7c6c001d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:35.947+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c64054a90 con 0x7f7c7c0c89c0 2024-01-16T15:00:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.147+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c7c0b4230 con 0x7f7c7c0c89c0 2024-01-16T15:00:36.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.150+0000 7f7c79600640 1 -- 172.21.15.43:0/1492621337 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f7c640580b0 con 0x7f7c7c0c89c0 2024-01-16T15:00:36.130 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:36.131 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":24,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:14.726979+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:25.424534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:26.432367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1e","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 msgr2=0x7f7c5406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 0x7f7c5406d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0058b0 tx=0x7f7c6c001d10 comp rx=0 tx=0).stop 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 msgr2=0x7f7c7c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400f7a0 tx=0x7f7c6400fc60 comp rx=0 tx=0).stop 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 shutdown_connections 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c5406b2c0 0x7f7c5406d780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c7c0c8e40 0x7f7c7c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7c0c89c0 0x7f7c7c0b3420 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 --2- 172.21.15.43:0/1492621337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c004d00 0x7f7c7c15a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.152+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 >> 172.21.15.43:0/1492621337 conn(0x7f7c7c09c3d0 msgr2=0x7f7c7c0b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.153+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 shutdown_connections 2024-01-16T15:00:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.153+0000 7f7c82600640 1 -- 172.21.15.43:0/1492621337 wait complete. 2024-01-16T15:00:36.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-16T15:00:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.400+0000 7f0651000640 1 Processor -- start 2024-01-16T15:00:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.400+0000 7f0651000640 1 -- start start 2024-01-16T15:00:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f0651000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f064c0d3270 0x7f064c0d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f0651000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f064c0b6760 0x7f064c0b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f0651000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064c0d0b40 con 0x7f064c15d770 2024-01-16T15:00:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f0651000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c0d3c30 con 0x7f064c0b6760 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f0651000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c15db50 con 0x7f064c0d3270 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f064a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f064c0b6760 0x7f064c0b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f064a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f064c0b6760 0x7f064c0b6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53224/0 (socket says 172.21.15.43:53224) 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.402+0000 7f064a800640 1 -- 172.21.15.43:0/1275907437 learned_addr learned my addr 172.21.15.43:0/1275907437 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.403+0000 7f0649400640 1 --2- 172.21.15.43:0/1275907437 >> v2:172.21.15.69:3300/0 conn(0x7f064c0d3270 0x7f064c0d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.403+0000 7f064a800640 1 -- 172.21.15.43:0/1275907437 >> v2:172.21.15.69:3300/0 conn(0x7f064c0d3270 msgr2=0x7f064c0d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.403+0000 7f064a800640 1 --2- 172.21.15.43:0/1275907437 >> v2:172.21.15.69:3300/0 conn(0x7f064c0d3270 0x7f064c0d36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.403+0000 7f064a800640 1 -- 172.21.15.43:0/1275907437 >> v1:172.21.15.69:6789/0 conn(0x7f064c15d770 legacy=0x7f064c0d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.403+0000 7f064a800640 1 -- 172.21.15.43:0/1275907437 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c0b7150 con 0x7f064c0b6760 2024-01-16T15:00:36.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.404+0000 7f064a800640 1 --2- 172.21.15.43:0/1275907437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0b6760 0x7f064c0b6c10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f064c0b93f0 tx=0x7f064402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71eb4520526359fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.404+0000 7f0648a00640 1 -- 172.21.15.43:0/1275907437 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064402ca80 con 0x7f064c0b6760 2024-01-16T15:00:36.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.404+0000 7f0648a00640 1 -- 172.21.15.43:0/1275907437 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064402cbe0 con 0x7f064c0b6760 2024-01-16T15:00:36.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.404+0000 7f0648a00640 1 -- 172.21.15.43:0/1275907437 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440343f0 con 0x7f064c0b6760 2024-01-16T15:00:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.405+0000 7f0651000640 1 -- 172.21.15.43:0/1275907437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0b6760 msgr2=0x7f064c0b6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.405+0000 7f0651000640 1 --2- 172.21.15.43:0/1275907437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0b6760 0x7f064c0b6c10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f064c0b93f0 tx=0x7f064402bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 -- 172.21.15.43:0/1275907437 shutdown_connections 2024-01-16T15:00:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 --2- 172.21.15.43:0/1275907437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0b6760 0x7f064c0b6c10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 --2- 172.21.15.43:0/1275907437 >> v2:172.21.15.69:3300/0 conn(0x7f064c0d3270 0x7f064c0d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 -- 172.21.15.43:0/1275907437 >> 172.21.15.43:0/1275907437 conn(0x7f064c0721a0 msgr2=0x7f064c06b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 -- 172.21.15.43:0/1275907437 shutdown_connections 2024-01-16T15:00:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.406+0000 7f0651000640 1 -- 172.21.15.43:0/1275907437 wait complete. 2024-01-16T15:00:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.408+0000 7f0651000640 1 Processor -- start 2024-01-16T15:00:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.408+0000 7f0651000640 1 -- start start 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 0x7f064c1f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f064c15d770 0x7f064c0cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c153290 con 0x7f064c0d3270 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ca3b0 con 0x7f064c0b6760 2024-01-16T15:00:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0651000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ca600 con 0x7f064c15d770 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53238/0 (socket says 172.21.15.43:53238) 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 -- 172.21.15.43:0/1056115553 learned_addr learned my addr 172.21.15.43:0/1056115553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649e00640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 0x7f064c1f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 -- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f064c15d770 msgr2=0x7f064c0cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f064c15d770 0x7f064c0cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 -- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 msgr2=0x7f064c1f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 0x7f064c1f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649400640 1 -- 172.21.15.43:0/1056115553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0644009b30 con 0x7f064c0d3270 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.410+0000 7f0649e00640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 0x7f064c1f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0649400640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f063400d7d0 tx=0x7f063400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634018070 con 0x7f064c0d3270 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634004020 con 0x7f064c0d3270 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06340041b0 con 0x7f064c0d3270 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c0ca8e0 con 0x7f064c0d3270 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.411+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c156880 con 0x7f064c0d3270 2024-01-16T15:00:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.413+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c154060 con 0x7f064c0d3270 2024-01-16T15:00:36.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.413+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0634004310 con 0x7f064c0d3270 2024-01-16T15:00:36.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.414+0000 7f0642000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 0x7f062406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.414+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0634089b60 con 0x7f064c0d3270 2024-01-16T15:00:36.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.415+0000 7f0649e00640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 0x7f062406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.415+0000 7f0649e00640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 0x7f062406d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f063c003eb0 tx=0x7f063c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.419+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0634054950 con 0x7f064c0d3270 2024-01-16T15:00:36.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.654+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f064c074770 con 0x7f064c0d3270 2024-01-16T15:00:36.635 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:00:36.654+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:00:36.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.658+0000 7f0642000640 1 -- 172.21.15.43:0/1056115553 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0634057f70 con 0x7f064c0d3270 2024-01-16T15:00:36.638 INFO:teuthology.orchestra.run.smithi043.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-16T15:00:36.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.660+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 msgr2=0x7f062406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.660+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 0x7f062406d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f063c003eb0 tx=0x7f063c009040 comp rx=0 tx=0).stop 2024-01-16T15:00:36.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 msgr2=0x7f064c0cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f063400d7d0 tx=0x7f063400dc90 comp rx=0 tx=0).stop 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 shutdown_connections 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f062406b2c0 0x7f062406d780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f064c15d770 0x7f064c0cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f064c0d3270 0x7f064c0cac20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 --2- 172.21.15.43:0/1056115553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f064c0b6760 0x7f064c1f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 >> 172.21.15.43:0/1056115553 conn(0x7f064c0721a0 msgr2=0x7f064c0b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 shutdown_connections 2024-01-16T15:00:36.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.661+0000 7f0651000640 1 -- 172.21.15.43:0/1056115553 wait complete. 2024-01-16T15:00:36.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-16T15:00:36.670 INFO:tasks.ceph.mds.b.smithi069.stderr: -3854> 2024-01-16T14:59:31.058+0000 7f844d600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:00:36.744 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:00:36.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.903+0000 7f6f1fe00640 1 Processor -- start 2024-01-16T15:00:36.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.903+0000 7f6f1fe00640 1 -- start start 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.905+0000 7f6f1fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 0x7f6f2015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.905+0000 7f6f1fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 0x7f6f200ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.905+0000 7f6f1fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2015f930 con 0x7f6f200b35a0 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.905+0000 7f6f1fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f200b43b0 con 0x7f6f20144770 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.905+0000 7f6f1fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2015f530 con 0x7f6f2015f0a0 2024-01-16T15:00:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.906+0000 7f6f1f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 0x7f6f200ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.906+0000 7f6f1ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 0x7f6f2015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.906+0000 7f6f17e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6f200b35a0 0x7f6f2006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:49858/0 (socket says 172.21.15.43:49858) 2024-01-16T15:00:36.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.906+0000 7f6f17e00640 1 -- 172.21.15.43:0/1269180969 learned_addr learned my addr 172.21.15.43:0/1269180969 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:36.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1f400640 1 -- 172.21.15.43:0/1269180969 >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 msgr2=0x7f6f2015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1f400640 1 --2- 172.21.15.43:0/1269180969 >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 0x7f6f2015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1f400640 1 -- 172.21.15.43:0/1269180969 >> v1:172.21.15.69:6789/0 conn(0x7f6f200b35a0 legacy=0x7f6f2006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1f400640 1 -- 172.21.15.43:0/1269180969 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f200ce9f0 con 0x7f6f2015f0a0 2024-01-16T15:00:36.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1e000640 1 -- 172.21.15.43:0/1269180969 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387523017 0 0) 0x7f6f2015f930 con 0x7f6f200b35a0 2024-01-16T15:00:36.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.907+0000 7f6f1ea00640 1 --2- 172.21.15.43:0/1269180969 >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 0x7f6f2015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:36.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.909+0000 7f6f1f400640 1 --2- 172.21.15.43:0/1269180969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 0x7f6f200ce4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009d90 tx=0x7f6f1802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac924b78befd6da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.909+0000 7f6f1e000640 1 -- 172.21.15.43:0/1269180969 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1802cc50 con 0x7f6f2015f0a0 2024-01-16T15:00:36.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.909+0000 7f6f1e000640 1 -- 172.21.15.43:0/1269180969 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1802cdb0 con 0x7f6f2015f0a0 2024-01-16T15:00:36.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.909+0000 7f6f1e000640 1 -- 172.21.15.43:0/1269180969 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18034660 con 0x7f6f2015f0a0 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 -- 172.21.15.43:0/1269180969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 msgr2=0x7f6f200ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/1269180969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 0x7f6f200ce4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009d90 tx=0x7f6f1802bea0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 -- 172.21.15.43:0/1269180969 shutdown_connections 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/1269180969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f2015f0a0 0x7f6f200ce4b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/1269180969 >> v2:172.21.15.69:3300/0 conn(0x7f6f20144770 0x7f6f2015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 -- 172.21.15.43:0/1269180969 >> 172.21.15.43:0/1269180969 conn(0x7f6f200cec10 msgr2=0x7f6f200cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 -- 172.21.15.43:0/1269180969 shutdown_connections 2024-01-16T15:00:36.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.910+0000 7f6f1fe00640 1 -- 172.21.15.43:0/1269180969 wait complete. 2024-01-16T15:00:36.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.912+0000 7f6f1fe00640 1 Processor -- start 2024-01-16T15:00:36.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.912+0000 7f6f1fe00640 1 -- start start 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f200b35a0 0x7f6f201f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f2015f0a0 0x7f6f20071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2015c870 con 0x7f6f20144770 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2015a860 con 0x7f6f2015f0a0 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2015c2b0 con 0x7f6f200b35a0 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53252/0 (socket says 172.21.15.43:53252) 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f17e00640 1 -- 172.21.15.43:0/535677253 learned_addr learned my addr 172.21.15.43:0/535677253 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.913+0000 7f6f1ea00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f200b35a0 0x7f6f201f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f1f400640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f2015f0a0 0x7f6f20071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 -- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f200b35a0 msgr2=0x7f6f201f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f200b35a0 0x7f6f201f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 -- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f2015f0a0 msgr2=0x7f6f20071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f2015f0a0 0x7f6f20071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 -- 172.21.15.43:0/535677253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f10009400 con 0x7f6f20144770 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f17e00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c4d0 tx=0x7f6f0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f080074e0 con 0x7f6f20144770 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f08007660 con 0x7f6f20144770 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08004a20 con 0x7f6f20144770 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.914+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f18009a70 con 0x7f6f20144770 2024-01-16T15:00:36.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.915+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2015cf40 con 0x7f6f20144770 2024-01-16T15:00:36.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.916+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2004e840 con 0x7f6f20144770 2024-01-16T15:00:36.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.917+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6f08004b80 con 0x7f6f20144770 2024-01-16T15:00:36.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.918+0000 7f6f16a00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 0x7f6ef806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.918+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6f08088e50 con 0x7f6f20144770 2024-01-16T15:00:36.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.918+0000 7f6f1ea00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 0x7f6ef806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.919+0000 7f6f1ea00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 0x7f6ef806d7b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f2013b160 tx=0x7f6f10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:36.922+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f08053bf0 con 0x7f6f20144770 2024-01-16T15:00:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.158+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6f200b2460 con 0x7f6f20144770 2024-01-16T15:00:37.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.668+0000 7f6f16a00640 1 -- 172.21.15.43:0/535677253 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6f080180a0 con 0x7f6f20144770 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 msgr2=0x7f6ef806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 0x7f6ef806d7b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f2013b160 tx=0x7f6f10008040 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 msgr2=0x7f6f2015a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c4d0 tx=0x7f6f0800c990 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 shutdown_connections 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ef806b2f0 0x7f6ef806d7b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f2015f0a0 0x7f6f20071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f20144770 0x7f6f2015a320 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 --2- 172.21.15.43:0/535677253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f200b35a0 0x7f6f201f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 >> 172.21.15.43:0/535677253 conn(0x7f6f200cec10 msgr2=0x7f6f200aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 shutdown_connections 2024-01-16T15:00:37.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.672+0000 7f6f1fe00640 1 -- 172.21.15.43:0/535677253 wait complete. 2024-01-16T15:00:37.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-16T15:00:37.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.914+0000 7f29e0a00640 1 Processor -- start 2024-01-16T15:00:37.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.914+0000 7f29e0a00640 1 -- start start 2024-01-16T15:00:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.916+0000 7f29e0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29dc06dae0 0x7f29dc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.916+0000 7f29e0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc15bbd0 0x7f29dc070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.916+0000 7f29e0a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc144ca0 con 0x7f29dc1448c0 2024-01-16T15:00:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.916+0000 7f29e0a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc15c450 con 0x7f29dc06dae0 2024-01-16T15:00:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.916+0000 7f29e0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc0715b0 con 0x7f29dc15bbd0 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29dc06dae0 0x7f29dc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29dc06dae0 0x7f29dc15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53254/0 (socket says 172.21.15.43:53254) 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 -- 172.21.15.43:0/2192482367 learned_addr learned my addr 172.21.15.43:0/2192482367 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29dbe00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc15bbd0 0x7f29dc070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 -- 172.21.15.43:0/2192482367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc15bbd0 msgr2=0x7f29dc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc15bbd0 0x7f29dc070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 -- 172.21.15.43:0/2192482367 >> v1:172.21.15.69:6789/0 conn(0x7f29dc1448c0 legacy=0x7f29dc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.917+0000 7f29d3e00640 1 -- 172.21.15.43:0/2192482367 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc071890 con 0x7f29dc06dae0 2024-01-16T15:00:37.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.919+0000 7f29d3e00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc15b690 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009850 tx=0x7f29c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5131801bccdfbca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.919+0000 7f29daa00640 1 -- 172.21.15.43:0/2192482367 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4004230 con 0x7f29dc06dae0 2024-01-16T15:00:37.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.919+0000 7f29daa00640 1 -- 172.21.15.43:0/2192482367 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4004390 con 0x7f29dc06dae0 2024-01-16T15:00:37.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.919+0000 7f29daa00640 1 -- 172.21.15.43:0/2192482367 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4003810 con 0x7f29dc06dae0 2024-01-16T15:00:37.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 -- 172.21.15.43:0/2192482367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 msgr2=0x7f29dc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc15b690 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009850 tx=0x7f29c402b960 comp rx=0 tx=0).stop 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 -- 172.21.15.43:0/2192482367 shutdown_connections 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc15b690 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 --2- 172.21.15.43:0/2192482367 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc15bbd0 0x7f29dc070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 -- 172.21.15.43:0/2192482367 >> 172.21.15.43:0/2192482367 conn(0x7f29dc0b0aa0 msgr2=0x7f29dc0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 -- 172.21.15.43:0/2192482367 shutdown_connections 2024-01-16T15:00:37.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.920+0000 7f29e0a00640 1 -- 172.21.15.43:0/2192482367 wait complete. 2024-01-16T15:00:37.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.922+0000 7f29e0a00640 1 Processor -- start 2024-01-16T15:00:37.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.922+0000 7f29e0a00640 1 -- start start 2024-01-16T15:00:37.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 0x7f29dc1f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 0x7f29dc0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc15d0a0 con 0x7f29dc06dae0 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1f3fb0 con 0x7f29dc15bbd0 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29e0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc13efb0 con 0x7f29dc1448c0 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29db400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29db400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53260/0 (socket says 172.21.15.43:53260) 2024-01-16T15:00:37.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29db400640 1 -- 172.21.15.43:0/1649076033 learned_addr learned my addr 172.21.15.43:0/1649076033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:37.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.923+0000 7f29d3e00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 0x7f29dc1f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29dbe00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 0x7f29dc0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 -- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 msgr2=0x7f29dc1f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 0x7f29dc1f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 -- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 msgr2=0x7f29dc0ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 0x7f29dc0ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 -- 172.21.15.43:0/1649076033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4009460 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29dbe00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 0x7f29dc0ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29d3e00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 0x7f29dc1f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29db400640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c450 tx=0x7f29cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0052d0 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc005430 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc012870 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.924+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc0724f0 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.925+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc1f34a0 con 0x7f29dc06dae0 2024-01-16T15:00:37.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.925+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc15c060 con 0x7f29dc06dae0 2024-01-16T15:00:37.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.927+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f29cc0129d0 con 0x7f29dc06dae0 2024-01-16T15:00:37.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.927+0000 7f29d3400640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 0x7f29b406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.927+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f29cc089740 con 0x7f29dc06dae0 2024-01-16T15:00:37.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.928+0000 7f29d3e00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 0x7f29b406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.928+0000 7f29d3e00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 0x7f29b406d7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f29c4035000 tx=0x7f29c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:37.932+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29cc0545c0 con 0x7f29dc06dae0 2024-01-16T15:00:38.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.121+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f29dc141cb0 con 0x7f29dc06dae0 2024-01-16T15:00:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.966+0000 7f29d3400640 1 -- 172.21.15.43:0/1649076033 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f29cc057be0 con 0x7f29dc06dae0 2024-01-16T15:00:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' removed 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 msgr2=0x7f29b406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 0x7f29b406d7d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f29c4035000 tx=0x7f29c4008030 comp rx=0 tx=0).stop 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 msgr2=0x7f29dc13ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c450 tx=0x7f29cc00c910 comp rx=0 tx=0).stop 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 shutdown_connections 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29b406b310 0x7f29b406d7d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29dc15bbd0 0x7f29dc0ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29dc1448c0 0x7f29dc1f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 --2- 172.21.15.43:0/1649076033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29dc06dae0 0x7f29dc13ea70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 >> 172.21.15.43:0/1649076033 conn(0x7f29dc0b0aa0 msgr2=0x7f29dc0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 shutdown_connections 2024-01-16T15:00:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:38.970+0000 7f29e0a00640 1 -- 172.21.15.43:0/1649076033 wait complete. 2024-01-16T15:00:38.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-16T15:00:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.209+0000 7fb51bc00640 1 Processor -- start 2024-01-16T15:00:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.209+0000 7fb51bc00640 1 -- start start 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb51bc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb51409cb30 0x7fb5140abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb51bc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5140adb60 0x7fb5140adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb51bc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51415b730 con 0x7fb51415b350 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb51bc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5140ac740 con 0x7fb5140adb60 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb51bc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5140ac460 con 0x7fb51409cb30 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb519c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5140adb60 0x7fb5140adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb519200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb51415b350 0x7fb5140b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46526/0 (socket says 172.21.15.43:46526) 2024-01-16T15:00:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.210+0000 7fb519200640 1 -- 172.21.15.43:0/2215925647 learned_addr learned my addr 172.21.15.43:0/2215925647 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.211+0000 7fb513e00640 1 --2- 172.21.15.43:0/2215925647 >> v2:172.21.15.69:3300/0 conn(0x7fb51409cb30 0x7fb5140abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.211+0000 7fb519c00640 1 -- 172.21.15.43:0/2215925647 >> v2:172.21.15.69:3300/0 conn(0x7fb51409cb30 msgr2=0x7fb5140abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.211+0000 7fb519c00640 1 --2- 172.21.15.43:0/2215925647 >> v2:172.21.15.69:3300/0 conn(0x7fb51409cb30 0x7fb5140abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.211+0000 7fb519c00640 1 -- 172.21.15.43:0/2215925647 >> v1:172.21.15.43:6789/0 conn(0x7fb51415b350 legacy=0x7fb5140b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.211+0000 7fb519c00640 1 -- 172.21.15.43:0/2215925647 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140b2a50 con 0x7fb5140adb60 2024-01-16T15:00:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.212+0000 7fb519c00640 1 --2- 172.21.15.43:0/2215925647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5140adf70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5140d0060 tx=0x7fb50c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bd06d4312420c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.212+0000 7fb513400640 1 -- 172.21.15.43:0/2215925647 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c02cc80 con 0x7fb5140adb60 2024-01-16T15:00:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.212+0000 7fb513400640 1 -- 172.21.15.43:0/2215925647 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c02cde0 con 0x7fb5140adb60 2024-01-16T15:00:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.212+0000 7fb513400640 1 -- 172.21.15.43:0/2215925647 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c034660 con 0x7fb5140adb60 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 -- 172.21.15.43:0/2215925647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 msgr2=0x7fb5140adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 --2- 172.21.15.43:0/2215925647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5140adf70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5140d0060 tx=0x7fb50c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 -- 172.21.15.43:0/2215925647 shutdown_connections 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 --2- 172.21.15.43:0/2215925647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5140adf70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 --2- 172.21.15.43:0/2215925647 >> v2:172.21.15.69:3300/0 conn(0x7fb51409cb30 0x7fb5140abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 -- 172.21.15.43:0/2215925647 >> 172.21.15.43:0/2215925647 conn(0x7fb5140b0ba0 msgr2=0x7fb5140b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 -- 172.21.15.43:0/2215925647 shutdown_connections 2024-01-16T15:00:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.213+0000 7fb51bc00640 1 -- 172.21.15.43:0/2215925647 wait complete. 2024-01-16T15:00:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.215+0000 7fb51bc00640 1 Processor -- start 2024-01-16T15:00:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.215+0000 7fb51bc00640 1 -- start start 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb51409cb30 0x7fb514158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 0x7fb51409ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141576c0 con 0x7fb5140adb60 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141579a0 con 0x7fb51415b350 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb51bc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb514157c80 con 0x7fb51409cb30 2024-01-16T15:00:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb513e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb513e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53288/0 (socket says 172.21.15.43:53288) 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.216+0000 7fb513e00640 1 -- 172.21.15.43:0/1896653168 learned_addr learned my addr 172.21.15.43:0/1896653168 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb519c00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 0x7fb51409ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 -- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb51409cb30 msgr2=0x7fb514158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb519200640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb51409cb30 0x7fb514158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb51409cb30 0x7fb514158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 -- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 msgr2=0x7fb51409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 0x7fb51409ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 -- 172.21.15.43:0/1896653168 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c009a70 con 0x7fb5140adb60 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb519c00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 0x7fb51409ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb513e00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00c7f0 tx=0x7fb4fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.217+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc004220 con 0x7fb5140adb60 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.218+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc0043a0 con 0x7fb5140adb60 2024-01-16T15:00:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.218+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc007860 con 0x7fb5140adb60 2024-01-16T15:00:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.218+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb514157f60 con 0x7fb5140adb60 2024-01-16T15:00:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.218+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5141e6260 con 0x7fb5140adb60 2024-01-16T15:00:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.219+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5141581a0 con 0x7fb5140adb60 2024-01-16T15:00:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.220+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb4fc007bd0 con 0x7fb5140adb60 2024-01-16T15:00:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.221+0000 7fb510c00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 0x7fb4ec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.221+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fb4fc089ae0 con 0x7fb5140adb60 2024-01-16T15:00:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.221+0000 7fb519200640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 0x7fb4ec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.222+0000 7fb519200640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 0x7fb4ec06d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb504005880 tx=0x7fb504009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.225+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5141581a0 con 0x7fb5140adb60 2024-01-16T15:00:39.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.421+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5141581a0 con 0x7fb5140adb60 2024-01-16T15:00:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.977+0000 7fb510c00640 1 -- 172.21.15.43:0/1896653168 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb5141581a0 con 0x7fb5140adb60 2024-01-16T15:00:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' removed 2024-01-16T15:00:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.981+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 msgr2=0x7fb4ec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.981+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 0x7fb4ec06d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb504005880 tx=0x7fb504009040 comp rx=0 tx=0).stop 2024-01-16T15:00:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.981+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 msgr2=0x7fb5141e5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.981+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00c7f0 tx=0x7fb4fc00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 shutdown_connections 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4ec06b2c0 0x7fb4ec06d780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb51415b350 0x7fb51409ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5140adb60 0x7fb5141e5d20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 --2- 172.21.15.43:0/1896653168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb51409cb30 0x7fb514158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 >> 172.21.15.43:0/1896653168 conn(0x7fb5140b0ba0 msgr2=0x7fb5140aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 shutdown_connections 2024-01-16T15:00:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:39.982+0000 7fb51bc00640 1 -- 172.21.15.43:0/1896653168 wait complete. 2024-01-16T15:00:39.989 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:00:39.989 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:00:39.989 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:00:39.990 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:39.990 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:00:39.990 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:00:39.990 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:00:39.990 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:39.990 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:00:39.990 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:00:39.991 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:00:39.991 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:39.991 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:00:39.991 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:00:39.991 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:00:39.991 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:40.007 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:00:40.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:00:40.012 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:00:40.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:00:40.013 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:00:40.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:00:40.018 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:00:40.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:00:40.060 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:00:40.060 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:40.065 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:00:40.066 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:40.066 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:00:40.066 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:40.072 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:00:40.072 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:40.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:00:40.213 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:00:40.223 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:00:40.229 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:00:40.233 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:00:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.287+0000 7f946ee00640 1 Processor -- start 2024-01-16T15:00:40.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.287+0000 7f946ee00640 1 -- start start 2024-01-16T15:00:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.288+0000 7f946ee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f946815d5d0 0x7f94680ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.288+0000 7f946ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680d1d80 0x7f94680cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.288+0000 7f946ee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680d04b0 con 0x7f94680cf710 2024-01-16T15:00:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.288+0000 7f946ee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680ce4e0 con 0x7f946815d5d0 2024-01-16T15:00:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.288+0000 7f946ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680cdb80 con 0x7f94680d1d80 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f946815d5d0 0x7f94680ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680d1d80 0x7f94680cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f946815d5d0 0x7f94680ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:53298/0 (socket says 172.21.15.43:53298) 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 -- 172.21.15.43:0/2691003253 learned_addr learned my addr 172.21.15.43:0/2691003253 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 -- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680d1d80 msgr2=0x7f94680cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 --2- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680d1d80 0x7f94680cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 -- 172.21.15.43:0/2691003253 >> v1:172.21.15.69:6789/0 conn(0x7f94680cf710 legacy=0x7f94680d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.289+0000 7f946da00640 1 -- 172.21.15.43:0/2691003253 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680d01d0 con 0x7f946815d5d0 2024-01-16T15:00:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.290+0000 7f946da00640 1 --2- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f946815d5d0 0x7f94680ce900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9458009e50 tx=0x7f945802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53cd6053d1aa30d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f9467e00640 1 -- 172.21.15.43:0/2691003253 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945802ca70 con 0x7f946815d5d0 2024-01-16T15:00:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f9467e00640 1 -- 172.21.15.43:0/2691003253 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945802cbd0 con 0x7f946815d5d0 2024-01-16T15:00:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f9467e00640 1 -- 172.21.15.43:0/2691003253 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94580343f0 con 0x7f946815d5d0 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 -- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f946815d5d0 msgr2=0x7f94680ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 --2- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f946815d5d0 0x7f94680ce900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9458009e50 tx=0x7f945802bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 -- 172.21.15.43:0/2691003253 shutdown_connections 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 --2- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680d1d80 0x7f94680cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 --2- 172.21.15.43:0/2691003253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f946815d5d0 0x7f94680ce900 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.291+0000 7f946ee00640 1 -- 172.21.15.43:0/2691003253 >> 172.21.15.43:0/2691003253 conn(0x7f946809c3d0 msgr2=0x7f94680d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.292+0000 7f946ee00640 1 -- 172.21.15.43:0/2691003253 shutdown_connections 2024-01-16T15:00:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.292+0000 7f946ee00640 1 -- 172.21.15.43:0/2691003253 wait complete. 2024-01-16T15:00:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.293+0000 7f946ee00640 1 Processor -- start 2024-01-16T15:00:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.293+0000 7f946ee00640 1 -- start start 2024-01-16T15:00:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680cf710 0x7f94680b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94680d1d80 0x7f946815b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680cc7f0 con 0x7f946815d5d0 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680cca90 con 0x7f94680d1d80 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.294+0000 7f946ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680ccd70 con 0x7f94680cf710 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53316/0 (socket says 172.21.15.43:53316) 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 -- 172.21.15.43:0/3697344631 learned_addr learned my addr 172.21.15.43:0/3697344631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946d000640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94680d1d80 0x7f946815b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946da00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680cf710 0x7f94680b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 -- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680cf710 msgr2=0x7f94680b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680cf710 0x7f94680b8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 -- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94680d1d80 msgr2=0x7f946815b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94680d1d80 0x7f946815b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.295+0000 7f946e400640 1 -- 172.21.15.43:0/3697344631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9458009b30 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f946e400640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f946000e7f0 tx=0x7f946000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460009510 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946000ee40 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946000cb80 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94680ca200 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.296+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94680ca7c0 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.297+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680ca9b0 con 0x7f946815d5d0 2024-01-16T15:00:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.298+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f946000cce0 con 0x7f946815d5d0 2024-01-16T15:00:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.299+0000 7f9465600640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 0x7f944006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.299+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f94600898e0 con 0x7f946815d5d0 2024-01-16T15:00:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.299+0000 7f946da00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 0x7f944006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.300+0000 7f946da00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 0x7f944006d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9458035000 tx=0x7f9458030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.304+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9460054b00 con 0x7f946815d5d0 2024-01-16T15:00:40.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:40.512+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f946804e840 con 0x7f946815d5d0 2024-01-16T15:00:40.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.000+0000 7f9465600640 1 -- 172.21.15.43:0/3697344631 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9460017080 con 0x7f946815d5d0 2024-01-16T15:00:40.981 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:00:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.003+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 msgr2=0x7f944006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.003+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 0x7f944006d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9458035000 tx=0x7f9458030000 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.003+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 msgr2=0x7f94680cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.003+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f946000e7f0 tx=0x7f946000ebf0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 shutdown_connections 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f944006b2c0 0x7f944006d780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94680d1d80 0x7f946815b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f946815d5d0 0x7f94680cace0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 --2- 172.21.15.43:0/3697344631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94680cf710 0x7f94680b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 >> 172.21.15.43:0/3697344631 conn(0x7f946809c3d0 msgr2=0x7f94680b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 shutdown_connections 2024-01-16T15:00:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.004+0000 7f946ee00640 1 -- 172.21.15.43:0/3697344631 wait complete. 2024-01-16T15:00:41.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:00:41.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.233+0000 7fe8bc000640 1 Processor -- start 2024-01-16T15:00:41.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.233+0000 7fe8bc000640 1 -- start start 2024-01-16T15:00:41.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8bc000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8b40ad720 0x7fe8b40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8bc000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8b40abff0 0x7fe8b40b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8bc000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b415c360 con 0x7fe8b406d520 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8bc000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40b41a0 con 0x7fe8b40ad720 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8bc000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b415ba30 con 0x7fe8b40abff0 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.235+0000 7fe8b8c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8b40ad720 0x7fe8b40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8b40ad720 0x7fe8b40b2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53318/0 (socket says 172.21.15.43:53318) 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 -- 172.21.15.43:0/3391969969 learned_addr learned my addr 172.21.15.43:0/3391969969 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b9600640 1 --1- 172.21.15.43:0/3391969969 >> v1:172.21.15.43:6789/0 conn(0x7fe8b406d520 0x7fe8b406d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46546/0 (socket says 172.21.15.43:46546) 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 -- 172.21.15.43:0/3391969969 >> v2:172.21.15.69:3300/0 conn(0x7fe8b40abff0 msgr2=0x7fe8b40b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 --2- 172.21.15.43:0/3391969969 >> v2:172.21.15.69:3300/0 conn(0x7fe8b40abff0 0x7fe8b40b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 -- 172.21.15.43:0/3391969969 >> v1:172.21.15.43:6789/0 conn(0x7fe8b406d520 legacy=0x7fe8b406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.236+0000 7fe8b8c00640 1 -- 172.21.15.43:0/3391969969 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b415bd10 con 0x7fe8b40ad720 2024-01-16T15:00:41.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.237+0000 7fe8b8c00640 1 --2- 172.21.15.43:0/3391969969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40b2080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4072930 tx=0x7fe89c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a023133cbcfe3670 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.238+0000 7fe8b3e00640 1 -- 172.21.15.43:0/3391969969 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c030870 con 0x7fe8b40ad720 2024-01-16T15:00:41.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.238+0000 7fe8b3e00640 1 -- 172.21.15.43:0/3391969969 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c0309d0 con 0x7fe8b40ad720 2024-01-16T15:00:41.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.238+0000 7fe8b3e00640 1 -- 172.21.15.43:0/3391969969 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c030d60 con 0x7fe8b40ad720 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.238+0000 7fe8bc000640 1 -- 172.21.15.43:0/3391969969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 msgr2=0x7fe8b40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.238+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3391969969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40b2080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4072930 tx=0x7fe89c02b9b0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 -- 172.21.15.43:0/3391969969 shutdown_connections 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3391969969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40b2080 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3391969969 >> v2:172.21.15.69:3300/0 conn(0x7fe8b40abff0 0x7fe8b40b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 -- 172.21.15.43:0/3391969969 >> 172.21.15.43:0/3391969969 conn(0x7fe8b40b0d80 msgr2=0x7fe8b40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 -- 172.21.15.43:0/3391969969 shutdown_connections 2024-01-16T15:00:41.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.239+0000 7fe8bc000640 1 -- 172.21.15.43:0/3391969969 wait complete. 2024-01-16T15:00:41.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.240+0000 7fe8bc000640 1 Processor -- start 2024-01-16T15:00:41.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.241+0000 7fe8bc000640 1 -- start start 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8b406d520 0x7fe8b40afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8b40abff0 0x7fe8b415c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b41528e0 con 0x7fe8b40ad720 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b41540c0 con 0x7fe8b406d520 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8bc000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b4151f40 con 0x7fe8b40abff0 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8ba000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8ba000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53330/0 (socket says 172.21.15.43:53330) 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8ba000640 1 -- 172.21.15.43:0/3817501604 learned_addr learned my addr 172.21.15.43:0/3817501604 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8b9600640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8b406d520 0x7fe8b40afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.242+0000 7fe8b8c00640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8b40abff0 0x7fe8b415c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 -- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8b40abff0 msgr2=0x7fe8b415c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8b40abff0 0x7fe8b415c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 -- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8b406d520 msgr2=0x7fe8b40afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8b406d520 0x7fe8b40afa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 -- 172.21.15.43:0/3817501604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0094d0 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8ba000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b970 tx=0x7fe8ac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00cb40 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ac00ccc0 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac014680 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.243+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b4152220 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.244+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b415b090 con 0x7fe8b40ad720 2024-01-16T15:00:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.245+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b4152a20 con 0x7fe8b40ad720 2024-01-16T15:00:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.246+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe8ac007370 con 0x7fe8b40ad720 2024-01-16T15:00:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.246+0000 7fe8b1600640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 0x7fe88c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.247+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8ac0897e0 con 0x7fe8b40ad720 2024-01-16T15:00:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.247+0000 7fe8b9600640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 0x7fe88c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.248+0000 7fe8b9600640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 0x7fe88c06d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4000f60 tx=0x7fe8a4000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.252+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8b4152a20 con 0x7fe8b40ad720 2024-01-16T15:00:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.519+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe8b4152a20 con 0x7fe8b40ad720 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.520+0000 7fe8b1600640 1 -- 172.21.15.43:0/3817501604 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fe8b4152a20 con 0x7fe8b40ad720 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.501 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:00:41.503 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.505 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.509 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.511 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:00:41.512 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-mds", 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlQflSKhAALeu808Po7DSZwMZZUlgnFg==", 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-mds" 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-mgr", 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlgAtTKhAAH92Ou9S5SbyqFCcGFfLLDw==", 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-osd", 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlAhxTKhAAklqrLX6DMOHvb6RRi0KN6A==", 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-osd" 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-rbd", 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlQS5TKhAAO2S+j0j0SmhP0pz0yw4fMw==", 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlVDhTKhAAzYcSc9Xxl+adpin056Ua+w==", 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.bootstrap-rgw", 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQC7maZlWkFTKhAAm3Kk0MKTFdK2iDRXgqwC6Q==", 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:00:41.515 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:00:41.516 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:00:41.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:41.517 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:00:41.517 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 msgr2=0x7fe88c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 0x7fe88c06d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4000f60 tx=0x7fe8a4000ed0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 msgr2=0x7fe8b40ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b970 tx=0x7fe8ac00be30 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 shutdown_connections 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe88c06b2c0 0x7fe88c06d780 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8b40ad720 0x7fe8b40ca750 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8b40abff0 0x7fe8b415c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 --2- 172.21.15.43:0/3817501604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8b406d520 0x7fe8b40afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.523+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 >> 172.21.15.43:0/3817501604 conn(0x7fe8b40b0d80 msgr2=0x7fe8b40cb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.524+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 shutdown_connections 2024-01-16T15:00:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.524+0000 7fe8bc000640 1 -- 172.21.15.43:0/3817501604 wait complete. 2024-01-16T15:00:41.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-16T15:00:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.755+0000 7fa526800640 1 Processor -- start 2024-01-16T15:00:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.756+0000 7fa526800640 1 -- start start 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa52015d250 0x7fa5200d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5200b8c50 0x7fa5200cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5200b67b0 con 0x7fa5200b8c50 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5200b5ec0 con 0x7fa52015d250 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa526800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200b6160 con 0x7fa5200d04b0 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.757+0000 7fa525e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5200b8c50 0x7fa5200cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5200b8c50 0x7fa5200cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53338/0 (socket says 172.21.15.43:53338) 2024-01-16T15:00:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 -- 172.21.15.43:0/953193013 learned_addr learned my addr 172.21.15.43:0/953193013 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa524a00640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525400640 1 --2- 172.21.15.43:0/953193013 >> v2:172.21.15.69:3300/0 conn(0x7fa52015d250 0x7fa5200d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 -- 172.21.15.43:0/953193013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 msgr2=0x7fa5200cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 -- 172.21.15.43:0/953193013 >> v2:172.21.15.69:3300/0 conn(0x7fa52015d250 msgr2=0x7fa5200d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 --2- 172.21.15.43:0/953193013 >> v2:172.21.15.69:3300/0 conn(0x7fa52015d250 0x7fa5200d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.758+0000 7fa525e00640 1 -- 172.21.15.43:0/953193013 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200b6440 con 0x7fa5200b8c50 2024-01-16T15:00:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.759+0000 7fa525e00640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5200b8c50 0x7fa5200cfde0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa52008b270 tx=0x7fa51802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b280f4c6968b192b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa51fe00640 1 -- 172.21.15.43:0/953193013 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51802ca70 con 0x7fa5200b8c50 2024-01-16T15:00:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa51fe00640 1 -- 172.21.15.43:0/953193013 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51802cbd0 con 0x7fa5200b8c50 2024-01-16T15:00:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa51fe00640 1 -- 172.21.15.43:0/953193013 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5180343f0 con 0x7fa5200b8c50 2024-01-16T15:00:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 -- 172.21.15.43:0/953193013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5200b8c50 msgr2=0x7fa5200cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5200b8c50 0x7fa5200cfde0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa52008b270 tx=0x7fa51802bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 -- 172.21.15.43:0/953193013 shutdown_connections 2024-01-16T15:00:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5200b8c50 0x7fa5200cfde0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 --2- 172.21.15.43:0/953193013 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 --2- 172.21.15.43:0/953193013 >> v2:172.21.15.69:3300/0 conn(0x7fa52015d250 0x7fa5200d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.760+0000 7fa526800640 1 -- 172.21.15.43:0/953193013 >> 172.21.15.43:0/953193013 conn(0x7fa52009c330 msgr2=0x7fa5200d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.761+0000 7fa526800640 1 -- 172.21.15.43:0/953193013 shutdown_connections 2024-01-16T15:00:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.761+0000 7fa526800640 1 -- 172.21.15.43:0/953193013 wait complete. 2024-01-16T15:00:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.762+0000 7fa526800640 1 Processor -- start 2024-01-16T15:00:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.762+0000 7fa526800640 1 -- start start 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200b8c50 0x7fa5201595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c6a80 con 0x7fa52015d250 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c7320 con 0x7fa5200b8c50 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.763+0000 7fa526800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200cbd80 con 0x7fa5200d04b0 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53348/0 (socket says 172.21.15.43:53348) 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 -- 172.21.15.43:0/4083379169 learned_addr learned my addr 172.21.15.43:0/4083379169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa524a00640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525400640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200b8c50 0x7fa5201595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 -- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 msgr2=0x7fa5200cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 -- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200b8c50 msgr2=0x7fa5201595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200b8c50 0x7fa5201595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.764+0000 7fa525e00640 1 -- 172.21.15.43:0/4083379169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518009b30 con 0x7fa52015d250 2024-01-16T15:00:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa525e00640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa518036040 tx=0x7fa518004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518009510 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51803e450 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51803e800 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5200cce00 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.765+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5200c6fb0 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.766+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5200c7490 con 0x7fa52015d250 2024-01-16T15:00:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.767+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa51803e960 con 0x7fa52015d250 2024-01-16T15:00:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.768+0000 7fa51d600640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 0x7fa4f806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.768+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa5180ac620 con 0x7fa52015d250 2024-01-16T15:00:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.769+0000 7fa525400640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 0x7fa4f806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.769+0000 7fa525400640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 0x7fa4f806d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:41.773+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5180777d0 con 0x7fa52015d250 2024-01-16T15:00:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.045+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fa52004e840 con 0x7fa52015d250 2024-01-16T15:00:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.049+0000 7fa51d600640 1 -- 172.21.15.43:0/4083379169 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa51807adf0 con 0x7fa52015d250 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 msgr2=0x7fa4f806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 0x7fa4f806d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510008040 comp rx=0 tx=0).stop 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 msgr2=0x7fa5200cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa518036040 tx=0x7fa518004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 shutdown_connections 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806b2c0 0x7fa4f806d780 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52015d250 0x7fa5200cc7d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5200d04b0 0x7fa5200cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 --2- 172.21.15.43:0/4083379169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200b8c50 0x7fa5201595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 >> 172.21.15.43:0/4083379169 conn(0x7fa52009c330 msgr2=0x7fa5200d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 shutdown_connections 2024-01-16T15:00:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.052+0000 7fa526800640 1 -- 172.21.15.43:0/4083379169 wait complete. 2024-01-16T15:00:42.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-16T15:00:42.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.276+0000 7fe0d0a00640 1 Processor -- start 2024-01-16T15:00:42.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.276+0000 7fe0d0a00640 1 -- start start 2024-01-16T15:00:42.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.277+0000 7fe0d0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0cc0cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0d0a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc15d810 con 0x7fe0cc15d430 2024-01-16T15:00:42.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0d0a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc0d0130 con 0x7fe0cc0d1690 2024-01-16T15:00:42.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0d0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0d09d0 con 0x7fe0cc0d3810 2024-01-16T15:00:42.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0ca800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe0cc15d430 0x7fe0cc1615e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46552/0 (socket says 172.21.15.43:46552) 2024-01-16T15:00:42.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0ca800640 1 -- 172.21.15.43:0/3208612793 learned_addr learned my addr 172.21.15.43:0/3208612793 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:42.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0c3e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0cc0cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285202947 0 0) 0x7fe0cc15d810 con 0x7fe0cc15d430 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.278+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac003520 con 0x7fe0cc15d430 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1408475036 0 0) 0x7fe0ac003520 con 0x7fe0cc15d430 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 msgr2=0x7fe0cc0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0cc0cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 >> v1:172.21.15.69:6789/0 conn(0x7fe0cc0d1690 legacy=0x7fe0cc0d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0d0410 con 0x7fe0cc15d430 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fe0bc001670 con 0x7fe0cc15d430 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 >> v1:172.21.15.43:6789/0 conn(0x7fe0cc15d430 legacy=0x7fe0cc1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0ac003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac003fe0 0x7fe0ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac004d00 0x7fe0ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0d09d0 con 0x7fe0ac004d00 2024-01-16T15:00:42.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac00bb40 con 0x7fe0ac003fe0 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac00bd00 con 0x7fe0cc0d3810 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.279+0000 7fe0cb200640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac004d00 0x7fe0ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0c3e00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac003fe0 0x7fe0ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0ca800640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0ac003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0cb200640 1 -- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 msgr2=0x7fe0ac003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0cb200640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0d3810 0x7fe0ac003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0cb200640 1 -- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac003fe0 msgr2=0x7fe0ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0cb200640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac003fe0 0x7fe0ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.280+0000 7fe0cb200640 1 -- 172.21.15.43:0/3208612793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac003520 con 0x7fe0ac004d00 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.281+0000 7fe0cb200640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac004d00 0x7fe0ac005180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4001e50 tx=0x7fe0ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff5d8ab0af2d3b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.281+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4003b70 con 0x7fe0ac004d00 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.281+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4003cd0 con 0x7fe0ac004d00 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.281+0000 7fe0c9e00640 1 -- 172.21.15.43:0/3208612793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4005310 con 0x7fe0ac004d00 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 -- 172.21.15.43:0/3208612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac004d00 msgr2=0x7fe0ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/3208612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac004d00 0x7fe0ac005180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4001e50 tx=0x7fe0ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0cb200640 1 -- 172.21.15.43:0/3208612793 reap_dead start 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 -- 172.21.15.43:0/3208612793 shutdown_connections 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 -- 172.21.15.43:0/3208612793 >> 172.21.15.43:0/3208612793 conn(0x7fe0cc08cb00 msgr2=0x7fe0cc09c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 -- 172.21.15.43:0/3208612793 shutdown_connections 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.282+0000 7fe0d0a00640 1 -- 172.21.15.43:0/3208612793 wait complete. 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.284+0000 7fe0d0a00640 1 Processor -- start 2024-01-16T15:00:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.284+0000 7fe0d0a00640 1 -- start start 2024-01-16T15:00:42.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac004d00 0x7fe0cc1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0b8b90 0x7fe0cc0b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc1f2e90 con 0x7fe0cc0b8710 2024-01-16T15:00:42.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc1f3c90 con 0x7fe0ac004d00 2024-01-16T15:00:42.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0d0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc1f3080 con 0x7fe0cc0b8b90 2024-01-16T15:00:42.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53364/0 (socket says 172.21.15.43:53364) 2024-01-16T15:00:42.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 -- 172.21.15.43:0/4267167020 learned_addr learned my addr 172.21.15.43:0/4267167020 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 -- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0b8b90 msgr2=0x7fe0cc0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0b8b90 0x7fe0cc0b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 -- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac004d00 msgr2=0x7fe0cc1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac004d00 0x7fe0cc1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.285+0000 7fe0c3e00640 1 -- 172.21.15.43:0/4267167020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc1f3360 con 0x7fe0cc0b8710 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0c3e00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4011630 tx=0x7fe0b4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4015870 con 0x7fe0cc0b8710 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b40159d0 con 0x7fe0cc0b8710 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc1f3640 con 0x7fe0cc0b8710 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4015d40 con 0x7fe0cc0b8710 2024-01-16T15:00:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.286+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc158ae0 con 0x7fe0cc0b8710 2024-01-16T15:00:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.287+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc067960 con 0x7fe0cc0b8710 2024-01-16T15:00:42.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.288+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe0b400f490 con 0x7fe0cc0b8710 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.289+0000 7fe0c2a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 0x7fe0a406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.289+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe0b408a070 con 0x7fe0cc0b8710 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.290+0000 7fe0ca800640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 0x7fe0a406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.290+0000 7fe0ca800640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 0x7fe0a406d7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc1f63e0 tx=0x7fe0bc000ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.294+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0b40bce90 con 0x7fe0cc0b8710 2024-01-16T15:00:42.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.557+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fe0cc04e840 con 0x7fe0cc0b8710 2024-01-16T15:00:42.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.561+0000 7fe0c2a00640 1 -- 172.21.15.43:0/4267167020 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe0b4055220 con 0x7fe0cc0b8710 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 msgr2=0x7fe0a406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 0x7fe0a406d7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc1f63e0 tx=0x7fe0bc000ef0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 msgr2=0x7fe0cc0b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4011630 tx=0x7fe0b4011af0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 shutdown_connections 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0a406b2f0 0x7fe0a406d7b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0cc0b8b90 0x7fe0cc0b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0cc0b8710 0x7fe0cc0b1270 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 --2- 172.21.15.43:0/4267167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac004d00 0x7fe0cc1585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 >> 172.21.15.43:0/4267167020 conn(0x7fe0cc08cb00 msgr2=0x7fe0cc0cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 shutdown_connections 2024-01-16T15:00:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.564+0000 7fe0d0a00640 1 -- 172.21.15.43:0/4267167020 wait complete. 2024-01-16T15:00:42.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-16T15:00:42.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.799+0000 7f76a7c00640 1 Processor -- start 2024-01-16T15:00:42.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.799+0000 7f76a7c00640 1 -- start start 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a7c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f76a00af890 0x7f76a006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a7c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76a006ab40 0x7f76a00b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a7c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a015d880 con 0x7f76a015d4a0 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a7c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a006af90 con 0x7f76a006ab40 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a7c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a006fba0 con 0x7f76a00af890 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a5c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76a006ab40 0x7f76a00b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a5200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f76a015d4a0 0x7f76a00af180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46564/0 (socket says 172.21.15.43:46564) 2024-01-16T15:00:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a5200640 1 -- 172.21.15.43:0/957894320 learned_addr learned my addr 172.21.15.43:0/957894320 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f76a5c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76a006ab40 0x7f76a00b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53370/0 (socket says 172.21.15.43:53370) 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.801+0000 7f7697e00640 1 --2- 172.21.15.43:0/957894320 >> v2:172.21.15.69:3300/0 conn(0x7f76a00af890 0x7f76a006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.802+0000 7f76a5c00640 1 -- 172.21.15.43:0/957894320 >> v2:172.21.15.69:3300/0 conn(0x7f76a00af890 msgr2=0x7f76a006a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.802+0000 7f76a5c00640 1 --2- 172.21.15.43:0/957894320 >> v2:172.21.15.69:3300/0 conn(0x7f76a00af890 0x7f76a006a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.802+0000 7f76a5c00640 1 -- 172.21.15.43:0/957894320 >> v1:172.21.15.43:6789/0 conn(0x7f76a015d4a0 legacy=0x7f76a00af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.802+0000 7f76a5c00640 1 -- 172.21.15.43:0/957894320 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a006eb00 con 0x7f76a006ab40 2024-01-16T15:00:42.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.802+0000 7f769fe00640 1 -- 172.21.15.43:0/957894320 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189033823 0 0) 0x7f76a015d880 con 0x7f76a015d4a0 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.803+0000 7f76a5c00640 1 --2- 172.21.15.43:0/957894320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a006ab40 0x7f76a00b3070 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76a0073c60 tx=0x7f769802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbac985411db1a38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.803+0000 7f769fe00640 1 -- 172.21.15.43:0/957894320 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769802cc80 con 0x7f76a006ab40 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.803+0000 7f769fe00640 1 -- 172.21.15.43:0/957894320 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769802cde0 con 0x7f76a006ab40 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.803+0000 7f769fe00640 1 -- 172.21.15.43:0/957894320 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698034660 con 0x7f76a006ab40 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 -- 172.21.15.43:0/957894320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a006ab40 msgr2=0x7f76a00b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 --2- 172.21.15.43:0/957894320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a006ab40 0x7f76a00b3070 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76a0073c60 tx=0x7f769802bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 -- 172.21.15.43:0/957894320 shutdown_connections 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 --2- 172.21.15.43:0/957894320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a006ab40 0x7f76a00b3070 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 --2- 172.21.15.43:0/957894320 >> v2:172.21.15.69:3300/0 conn(0x7f76a00af890 0x7f76a006a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 -- 172.21.15.43:0/957894320 >> 172.21.15.43:0/957894320 conn(0x7f76a006d490 msgr2=0x7f76a006d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 -- 172.21.15.43:0/957894320 shutdown_connections 2024-01-16T15:00:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.804+0000 7f76a7c00640 1 -- 172.21.15.43:0/957894320 wait complete. 2024-01-16T15:00:42.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.806+0000 7f76a7c00640 1 Processor -- start 2024-01-16T15:00:42.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.806+0000 7f76a7c00640 1 -- start start 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 0x7f76a0142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a015d4a0 0x7f76a0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00cdf00 con 0x7f76a00af890 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0142da0 con 0x7f76a015d4a0 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a7c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00cd280 con 0x7f76a006ab40 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f7697e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f7697e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53386/0 (socket says 172.21.15.43:53386) 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f7697e00640 1 -- 172.21.15.43:0/3728759695 learned_addr learned my addr 172.21.15.43:0/3728759695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a5c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a015d4a0 0x7f76a0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f76a5200640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 0x7f76a0142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.807+0000 7f7697e00640 1 -- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 msgr2=0x7f76a0142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f7697e00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 0x7f76a0142860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f7697e00640 1 -- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a015d4a0 msgr2=0x7f76a0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f7697e00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a015d4a0 0x7f76a0158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f7697e00640 1 -- 172.21.15.43:0/3728759695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698009a70 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f76a5200640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 0x7f76a0142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f7697e00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f768800d800 tx=0x7f768800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800b6c0 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768800b840 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800bbf0 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a00cd560 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.808+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a00cdb20 con 0x7f76a00af890 2024-01-16T15:00:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.809+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a0072ac0 con 0x7f76a00af890 2024-01-16T15:00:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.810+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f768800bd50 con 0x7f76a00af890 2024-01-16T15:00:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.811+0000 7f769d600640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 0x7f767806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.812+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7688089fb0 con 0x7f76a00af890 2024-01-16T15:00:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.812+0000 7f76a5200640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 0x7f767806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.812+0000 7f76a5200640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 0x7f767806d780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7690000f60 tx=0x7f7690009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:42.817+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7688055040 con 0x7f76a00af890 2024-01-16T15:00:43.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.079+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f76a00cdcb0 con 0x7f76a00af890 2024-01-16T15:00:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.084+0000 7f769d600640 1 -- 172.21.15.43:0/3728759695 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7688058660 con 0x7f76a00af890 2024-01-16T15:00:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 msgr2=0x7f767806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 0x7f767806d780 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7690000f60 tx=0x7f7690009040 comp rx=0 tx=0).stop 2024-01-16T15:00:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 msgr2=0x7f76a00cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f768800d800 tx=0x7f768800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 shutdown_connections 2024-01-16T15:00:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f767806b2c0 0x7f767806d780 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a015d4a0 0x7f76a0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a00af890 0x7f76a00cca10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 --2- 172.21.15.43:0/3728759695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a006ab40 0x7f76a0142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.086+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 >> 172.21.15.43:0/3728759695 conn(0x7f76a006d490 msgr2=0x7f76a00b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.087+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 shutdown_connections 2024-01-16T15:00:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.087+0000 7f76a7c00640 1 -- 172.21.15.43:0/3728759695 wait complete. 2024-01-16T15:00:43.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.325+0000 7f08c8a00640 1 Processor -- start 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.326+0000 7f08c8a00640 1 -- start start 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 0x7f08c4141e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08c415f100 0x7f08c4071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 0x7f08c406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c406cae0 con 0x7f08c415f100 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c4071e40 con 0x7f08c415ed20 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.327+0000 7f08c8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c406c1b0 con 0x7f08c4071650 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08c415f100 0x7f08c4071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08c415f100 0x7f08c4071110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43616/0 (socket says 172.21.15.43:43616) 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 -- 172.21.15.43:0/4008822377 learned_addr learned my addr 172.21.15.43:0/4008822377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c3400640 1 --2- 172.21.15.43:0/4008822377 >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 0x7f08c4141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 -- 172.21.15.43:0/4008822377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 msgr2=0x7f08c406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c3e00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 0x7f08c406cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 0x7f08c406cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 -- 172.21.15.43:0/4008822377 >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 msgr2=0x7f08c4141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 --2- 172.21.15.43:0/4008822377 >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 0x7f08c4141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c2a00640 1 -- 172.21.15.43:0/4008822377 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c406c490 con 0x7f08c415f100 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c3400640 1 --2- 172.21.15.43:0/4008822377 >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 0x7f08c4141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.328+0000 7f08c3e00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 0x7f08c406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:43.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.330+0000 7f08c2a00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c415f100 0x7f08c4071110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08c4073d10 tx=0x7f08ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189ee83b6ec5ee9b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.330+0000 7f08c2000640 1 -- 172.21.15.43:0/4008822377 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac031870 con 0x7f08c415f100 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.330+0000 7f08c2000640 1 -- 172.21.15.43:0/4008822377 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac0319d0 con 0x7f08c415f100 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.330+0000 7f08c2000640 1 -- 172.21.15.43:0/4008822377 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac031d60 con 0x7f08c415f100 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 -- 172.21.15.43:0/4008822377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c415f100 msgr2=0x7f08c4071110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c415f100 0x7f08c4071110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08c4073d10 tx=0x7f08ac02b910 comp rx=0 tx=0).stop 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 -- 172.21.15.43:0/4008822377 shutdown_connections 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c4071650 0x7f08c406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 --2- 172.21.15.43:0/4008822377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c415f100 0x7f08c4071110 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 --2- 172.21.15.43:0/4008822377 >> v2:172.21.15.69:3300/0 conn(0x7f08c415ed20 0x7f08c4141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 -- 172.21.15.43:0/4008822377 >> 172.21.15.43:0/4008822377 conn(0x7f08c40b0840 msgr2=0x7f08c40b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 -- 172.21.15.43:0/4008822377 shutdown_connections 2024-01-16T15:00:43.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.331+0000 7f08c8a00640 1 -- 172.21.15.43:0/4008822377 wait complete. 2024-01-16T15:00:43.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.333+0000 7f08c8a00640 1 Processor -- start 2024-01-16T15:00:43.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.334+0000 7f08c8a00640 1 -- start start 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 0x7f08c413a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 0x7f08c415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c413cf90 con 0x7f08c4071650 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c415ccd0 con 0x7f08c415f100 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c413c8d0 con 0x7f08c415ed20 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43622/0 (socket says 172.21.15.43:43622) 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.335+0000 7f08c3400640 1 -- 172.21.15.43:0/3760032099 learned_addr learned my addr 172.21.15.43:0/3760032099 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:43.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3e00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 0x7f08c415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c2a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 0x7f08c413a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 -- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 msgr2=0x7f08c413a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 0x7f08c413a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 -- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 msgr2=0x7f08c415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 0x7f08c415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 -- 172.21.15.43:0/3760032099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0094d0 con 0x7f08c4071650 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3e00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 0x7f08c415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c2a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 0x7f08c413a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c3400640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f08b400c7f0 tx=0x7f08b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4007a30 con 0x7f08c4071650 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b4007b90 con 0x7f08c4071650 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400ce40 con 0x7f08c4071650 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.336+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c413cbb0 con 0x7f08c4071650 2024-01-16T15:00:43.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.337+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c415cfb0 con 0x7f08c4071650 2024-01-16T15:00:43.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.338+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4067d10 con 0x7f08c4071650 2024-01-16T15:00:43.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.339+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f08b40113f0 con 0x7f08c4071650 2024-01-16T15:00:43.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.340+0000 7f08bb400640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 0x7f089c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.340+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f08b4089990 con 0x7f08c4071650 2024-01-16T15:00:43.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.340+0000 7f08c2a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 0x7f089c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.341+0000 7f08c2a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 0x7f089c06d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08ac036000 tx=0x7f08ac03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.345+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08b4054be0 con 0x7f08c4071650 2024-01-16T15:00:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.605+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f08c404e840 con 0x7f08c4071650 2024-01-16T15:00:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.609+0000 7f08bb400640 1 -- 172.21.15.43:0/3760032099 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f08b401f0a0 con 0x7f08c4071650 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 msgr2=0x7f089c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 0x7f089c06d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08ac036000 tx=0x7f08ac03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 msgr2=0x7f08c414c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f08b400c7f0 tx=0x7f08b400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 shutdown_connections 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f089c06b2c0 0x7f089c06d780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08c415f100 0x7f08c415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08c415ed20 0x7f08c413a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 --2- 172.21.15.43:0/3760032099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4071650 0x7f08c414c3a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 >> 172.21.15.43:0/3760032099 conn(0x7f08c40b0840 msgr2=0x7f08c40704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 shutdown_connections 2024-01-16T15:00:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.612+0000 7f08c8a00640 1 -- 172.21.15.43:0/3760032099 wait complete. 2024-01-16T15:00:43.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-16T15:00:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.845+0000 7f4fe0400640 1 Processor -- start 2024-01-16T15:00:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.845+0000 7f4fe0400640 1 -- start start 2024-01-16T15:00:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.846+0000 7f4fe0400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd815a090 0x7f4fd8072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.846+0000 7f4fe0400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4fd80b39a0 0x7f4fd806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.846+0000 7f4fe0400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd80ac760 con 0x7f4fd80abff0 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.846+0000 7f4fe0400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd815a7f0 con 0x7f4fd80b39a0 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.846+0000 7f4fe0400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd815aad0 con 0x7f4fd815a090 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4fd80b39a0 0x7f4fd806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4fd80b39a0 0x7f4fd806d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43636/0 (socket says 172.21.15.43:43636) 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 -- 172.21.15.43:0/3964661170 learned_addr learned my addr 172.21.15.43:0/3964661170 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fdd000640 1 --1- 172.21.15.43:0/3964661170 >> v1:172.21.15.43:6789/0 conn(0x7f4fd80abff0 0x7f4fd80b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35322/0 (socket says 172.21.15.43:35322) 2024-01-16T15:00:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fdda00640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd815a090 0x7f4fd8072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 -- 172.21.15.43:0/3964661170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd815a090 msgr2=0x7f4fd8072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd815a090 0x7f4fd8072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 -- 172.21.15.43:0/3964661170 >> v1:172.21.15.43:6789/0 conn(0x7f4fd80abff0 legacy=0x7f4fd80b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.847+0000 7f4fde400640 1 -- 172.21.15.43:0/3964661170 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd815adb0 con 0x7f4fd80b39a0 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.848+0000 7f4fde400640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd80b39a0 0x7f4fd806d520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793f724003cf8b1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fd7e00640 1 -- 172.21.15.43:0/3964661170 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0030840 con 0x7f4fd80b39a0 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fd7e00640 1 -- 172.21.15.43:0/3964661170 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd00309a0 con 0x7f4fd80b39a0 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fd7e00640 1 -- 172.21.15.43:0/3964661170 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0030d30 con 0x7f4fd80b39a0 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 -- 172.21.15.43:0/3964661170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd80b39a0 msgr2=0x7f4fd806d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd80b39a0 0x7f4fd806d520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 -- 172.21.15.43:0/3964661170 shutdown_connections 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd80b39a0 0x7f4fd806d520 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 --2- 172.21.15.43:0/3964661170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd815a090 0x7f4fd8072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 -- 172.21.15.43:0/3964661170 >> 172.21.15.43:0/3964661170 conn(0x7f4fd80b0d80 msgr2=0x7f4fd80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.849+0000 7f4fe0400640 1 -- 172.21.15.43:0/3964661170 shutdown_connections 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.850+0000 7f4fe0400640 1 -- 172.21.15.43:0/3964661170 wait complete. 2024-01-16T15:00:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.851+0000 7f4fe0400640 1 Processor -- start 2024-01-16T15:00:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.851+0000 7f4fe0400640 1 -- start start 2024-01-16T15:00:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd80abff0 0x7f4fd80ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80b39a0 0x7f4fd815cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd80ce9b0 con 0x7f4fd815a090 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd815da30 con 0x7f4fd80abff0 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.852+0000 7f4fe0400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd815d750 con 0x7f4fd80b39a0 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43650/0 (socket says 172.21.15.43:43650) 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 -- 172.21.15.43:0/897449678 learned_addr learned my addr 172.21.15.43:0/897449678 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 -- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80b39a0 msgr2=0x7f4fd815cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80b39a0 0x7f4fd815cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 -- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd80abff0 msgr2=0x7f4fd80ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd80abff0 0x7f4fd80ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 -- 172.21.15.43:0/897449678 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd00094d0 con 0x7f4fd815a090 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.853+0000 7f4fde400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.854+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0039c90 con 0x7f4fd815a090 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.854+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0039e10 con 0x7f4fd815a090 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.854+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0043640 con 0x7f4fd815a090 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.854+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd80c8530 con 0x7f4fd815a090 2024-01-16T15:00:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.854+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd80c8af0 con 0x7f4fd815a090 2024-01-16T15:00:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.855+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd8074770 con 0x7f4fd815a090 2024-01-16T15:00:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.856+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4fd00437a0 con 0x7f4fd815a090 2024-01-16T15:00:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.857+0000 7f4fd5600640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 0x7f4fb006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.857+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4fd003e070 con 0x7f4fd815a090 2024-01-16T15:00:43.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.857+0000 7f4fdda00640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 0x7f4fb006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.858+0000 7f4fdda00640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 0x7f4fb006d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006fd0 tx=0x7f4fc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:43.861+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fd0077b40 con 0x7f4fd815a090 2024-01-16T15:00:44.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.124+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f4fd80b0aa0 con 0x7f4fd815a090 2024-01-16T15:00:44.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.129+0000 7f4fd5600640 1 -- 172.21.15.43:0/897449678 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4fd007b160 con 0x7f4fd815a090 2024-01-16T15:00:44.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 msgr2=0x7f4fb006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 0x7f4fb006d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006fd0 tx=0x7f4fc0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:44.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 msgr2=0x7f4fd80c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009870 tx=0x7f4fd0004730 comp rx=0 tx=0).stop 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 shutdown_connections 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fb006b2c0 0x7f4fb006d780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fd815a090 0x7f4fd80c9010 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fd80b39a0 0x7f4fd815cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.131+0000 7f4fe0400640 1 --2- 172.21.15.43:0/897449678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fd80abff0 0x7f4fd80ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.132+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 >> 172.21.15.43:0/897449678 conn(0x7f4fd80b0d80 msgr2=0x7f4fd8071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.132+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 shutdown_connections 2024-01-16T15:00:44.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.132+0000 7f4fe0400640 1 -- 172.21.15.43:0/897449678 wait complete. 2024-01-16T15:00:44.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-16T15:00:44.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.372+0000 7f291ca00640 1 Processor -- start 2024-01-16T15:00:44.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.372+0000 7f291ca00640 1 -- start start 2024-01-16T15:00:44.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f29140b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291ca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140d1af0 con 0x7f29140b1d20 2024-01-16T15:00:44.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140b55a0 con 0x7f29140d2920 2024-01-16T15:00:44.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140d1d50 con 0x7f29140b4c80 2024-01-16T15:00:44.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f29140b1d20 0x7f29140b21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35332/0 (socket says 172.21.15.43:35332) 2024-01-16T15:00:44.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.374+0000 7f291aa00640 1 -- 172.21.15.43:0/2802478750 learned_addr learned my addr 172.21.15.43:0/2802478750 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:44.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f291a000640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f29140b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866345328 0 0) 0x7f29140d1af0 con 0x7f29140b1d20 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f4003520 con 0x7f29140b1d20 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396383452 0 0) 0x7f28f4003520 con 0x7f29140b1d20 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 msgr2=0x7f29140b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f29140b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 >> v1:172.21.15.69:6789/0 conn(0x7f29140d2920 legacy=0x7f29140d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140d2030 con 0x7f29140b1d20 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f29040019f0 con 0x7f29140b1d20 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 >> v1:172.21.15.43:6789/0 conn(0x7f29140b1d20 legacy=0x7f29140b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f28f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28f40047c0 0x7f28f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f28f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140d1d50 con 0x7f28f40039a0 2024-01-16T15:00:44.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40054b0 con 0x7f28f40047c0 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4005670 con 0x7f29140b4c80 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.375+0000 7f291a000640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f28f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f2919600640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28f40047c0 0x7f28f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291aa00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f28f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291a000640 1 -- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 msgr2=0x7f28f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291a000640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140b4c80 0x7f28f400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291a000640 1 -- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28f40047c0 msgr2=0x7f28f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291a000640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28f40047c0 0x7f28f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.376+0000 7f291a000640 1 -- 172.21.15.43:0/2802478750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4003520 con 0x7f28f40039a0 2024-01-16T15:00:44.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.377+0000 7f291a000640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f28f4003dc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f290c00ba30 tx=0x7f28f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b9e43909867eee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.377+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c032640 con 0x7f28f40039a0 2024-01-16T15:00:44.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.377+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c0327a0 con 0x7f28f40039a0 2024-01-16T15:00:44.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.377+0000 7f2918c00640 1 -- 172.21.15.43:0/2802478750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c032b30 con 0x7f28f40039a0 2024-01-16T15:00:44.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 -- 172.21.15.43:0/2802478750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 msgr2=0x7f28f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 --2- 172.21.15.43:0/2802478750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f28f4003dc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f290c00ba30 tx=0x7f28f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291aa00640 1 -- 172.21.15.43:0/2802478750 reap_dead start 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 -- 172.21.15.43:0/2802478750 shutdown_connections 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 -- 172.21.15.43:0/2802478750 >> 172.21.15.43:0/2802478750 conn(0x7f29140721a0 msgr2=0x7f29140cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 -- 172.21.15.43:0/2802478750 shutdown_connections 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.378+0000 7f291ca00640 1 -- 172.21.15.43:0/2802478750 wait complete. 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.380+0000 7f291ca00640 1 Processor -- start 2024-01-16T15:00:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.380+0000 7f291ca00640 1 -- start start 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29140c6300 0x7f2914152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140cb320 0x7f29141f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2914152cd0 con 0x7f28f40039a0 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291415e710 con 0x7f29140c6300 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140c7940 con 0x7f29140cb320 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43664/0 (socket says 172.21.15.43:43664) 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 -- 172.21.15.43:0/558366074 learned_addr learned my addr 172.21.15.43:0/558366074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291aa00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140cb320 0x7f29141f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 -- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140cb320 msgr2=0x7f29141f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f2919600640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29140c6300 0x7f2914152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140cb320 0x7f29141f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 -- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29140c6300 msgr2=0x7f2914152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29140c6300 0x7f2914152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.381+0000 7f291a000640 1 -- 172.21.15.43:0/558366074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140c7c20 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f291a000640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f290c02bec0 tx=0x7f290c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c03ba70 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c03bbd0 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29141f7520 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c030410 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.382+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29140cb7a0 con 0x7f28f40039a0 2024-01-16T15:00:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.383+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291415e4e0 con 0x7f28f40039a0 2024-01-16T15:00:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.384+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f290c030570 con 0x7f28f40039a0 2024-01-16T15:00:44.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.386+0000 7f2912000640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 0x7f28ec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.386+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f290c0ac090 con 0x7f28f40039a0 2024-01-16T15:00:44.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.386+0000 7f2919600640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 0x7f28ec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.387+0000 7f2919600640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 0x7f28ec06d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f28fc007fd0 tx=0x7f28fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.390+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f291415e4e0 con 0x7f28f40039a0 2024-01-16T15:00:44.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.656+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f291415e4e0 con 0x7f28f40039a0 2024-01-16T15:00:44.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.660+0000 7f2912000640 1 -- 172.21.15.43:0/558366074 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f291415e4e0 con 0x7f28f40039a0 2024-01-16T15:00:44.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.662+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 msgr2=0x7f28ec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.662+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 0x7f28ec06d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f28fc007fd0 tx=0x7f28fc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:44.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 msgr2=0x7f291415db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f290c02bec0 tx=0x7f290c004730 comp rx=0 tx=0).stop 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 shutdown_connections 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f28ec06b2c0 0x7f28ec06d780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29140c6300 0x7f2914152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29140cb320 0x7f29141f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 --2- 172.21.15.43:0/558366074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28f40039a0 0x7f291415db70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 >> 172.21.15.43:0/558366074 conn(0x7f29140721a0 msgr2=0x7f29140b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 shutdown_connections 2024-01-16T15:00:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.663+0000 7f291ca00640 1 -- 172.21.15.43:0/558366074 wait complete. 2024-01-16T15:00:44.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:44.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.896+0000 7f9ed0a00640 1 Processor -- start 2024-01-16T15:00:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.897+0000 7f9ed0a00640 1 -- start start 2024-01-16T15:00:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.898+0000 7f9ed0a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ecc0cf760 0x7f9ecc15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.898+0000 7f9ed0a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc071700 con 0x7f9ecc143950 2024-01-16T15:00:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.899+0000 7f9ed0a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc15b410 con 0x7f9ecc143570 2024-01-16T15:00:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.899+0000 7f9ed0a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc15b130 con 0x7f9ecc0cf760 2024-01-16T15:00:44.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.899+0000 7f9ecbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9ecc143950 0x7f9ecc0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35344/0 (socket says 172.21.15.43:35344) 2024-01-16T15:00:44.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.899+0000 7f9ecbe00640 1 -- 172.21.15.43:0/4108004319 learned_addr learned my addr 172.21.15.43:0/4108004319 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.899+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/4108004319 >> v2:172.21.15.69:3300/0 conn(0x7f9ecc0cf760 0x7f9ecc15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892044961 0 0) 0x7f9ec4001110 con 0x7f9ecc143950 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eac003520 con 0x7f9ecc143950 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139258410 0 0) 0x7f9eac003520 con 0x7f9ecc143950 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 >> v2:172.21.15.69:3300/0 conn(0x7f9ecc0cf760 msgr2=0x7f9ecc15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 --2- 172.21.15.43:0/4108004319 >> v2:172.21.15.69:3300/0 conn(0x7f9ecc0cf760 0x7f9ecc15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 >> v1:172.21.15.69:6789/0 conn(0x7f9ecc143570 legacy=0x7f9ecc0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc15dbd0 con 0x7f9ecc143950 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9ec4001800 con 0x7f9ecc143950 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 >> v1:172.21.15.43:6789/0 conn(0x7f9ecc143950 legacy=0x7f9ecc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0039e0 0x7f9eac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac0051f0 0x7f9eac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc15b130 con 0x7f9eac0047c0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac00c150 con 0x7f9eac0039e0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac00c310 con 0x7f9eac0051f0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.900+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecb400640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0039e0 0x7f9eac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac0051f0 0x7f9eac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecaa00640 1 -- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac0051f0 msgr2=0x7f9eac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eac0051f0 0x7f9eac00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecaa00640 1 -- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0039e0 msgr2=0x7f9eac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0039e0 0x7f9eac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecaa00640 1 -- 172.21.15.43:0/4108004319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac003520 con 0x7f9eac0047c0 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecbe00640 1 -- 172.21.15.43:0/4108004319 reap_dead start 2024-01-16T15:00:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.901+0000 7f9ecb400640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0039e0 0x7f9eac003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.902+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009650 tx=0x7f9eac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fce6399cea20163 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.902+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb402e890 con 0x7f9eac0047c0 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.902+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb402e9f0 con 0x7f9eac0047c0 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9eca000640 1 -- 172.21.15.43:0/4108004319 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb402ed80 con 0x7f9eac0047c0 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 -- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 msgr2=0x7f9eac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009650 tx=0x7f9eac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 -- 172.21.15.43:0/4108004319 shutdown_connections 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/4108004319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 -- 172.21.15.43:0/4108004319 >> 172.21.15.43:0/4108004319 conn(0x7f9ecc06a860 msgr2=0x7f9ecc06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 -- 172.21.15.43:0/4108004319 shutdown_connections 2024-01-16T15:00:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.903+0000 7f9ed0a00640 1 -- 172.21.15.43:0/4108004319 wait complete. 2024-01-16T15:00:44.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.905+0000 7f9ed0a00640 1 Processor -- start 2024-01-16T15:00:44.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.905+0000 7f9ed0a00640 1 -- start start 2024-01-16T15:00:44.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0047c0 0x7f9ecc0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 0x7f9ecc06c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc13c160 con 0x7f9ecc13d430 2024-01-16T15:00:44.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc13cfa0 con 0x7f9eac0047c0 2024-01-16T15:00:44.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.906+0000 7f9ed0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc13c390 con 0x7f9ecc151750 2024-01-16T15:00:44.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43680/0 (socket says 172.21.15.43:43680) 2024-01-16T15:00:44.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1508614196 learned_addr learned my addr 172.21.15.43:0/1508614196 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:44.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0047c0 0x7f9ecc0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 0x7f9ecc06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 msgr2=0x7f9ecc06c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 0x7f9ecc06c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0047c0 msgr2=0x7f9ecc0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0047c0 0x7f9ecc0ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1508614196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc13c670 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.907+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 0x7f9ecc06c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009b80 tx=0x7f9eb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4003610 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc13c950 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4003770 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4003b00 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.908+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc151bd0 con 0x7f9ecc13d430 2024-01-16T15:00:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.909+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc04e840 con 0x7f9ecc13d430 2024-01-16T15:00:44.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.910+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9eb403d070 con 0x7f9ecc13d430 2024-01-16T15:00:44.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.911+0000 7f9ec3400640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 0x7f9ea406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.911+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9eb40aaed0 con 0x7f9ecc13d430 2024-01-16T15:00:44.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.912+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 0x7f9ea406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.912+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 0x7f9ea406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc007fd0 tx=0x7f9ebc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:44.916+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9eb4075ff0 con 0x7f9ecc13d430 2024-01-16T15:00:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.119+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ecc0b3910 con 0x7f9ecc13d430 2024-01-16T15:00:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.121+0000 7f9ec3400640 1 -- 172.21.15.43:0/1508614196 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12809 (secure 0 0 0) 0x7f9eb403b090 con 0x7f9ecc13d430 2024-01-16T15:00:45.102 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:45.102 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":31,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:41.014911+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 msgr2=0x7f9ea406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 0x7f9ea406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc007fd0 tx=0x7f9ebc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 msgr2=0x7f9ecc13bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009b80 tx=0x7f9eb4004730 comp rx=0 tx=0).stop 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 shutdown_connections 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ea406b2c0 0x7f9ea406d780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ecc151750 0x7f9ecc06c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc13d430 0x7f9ecc13bc20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 --2- 172.21.15.43:0/1508614196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eac0047c0 0x7f9ecc0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 >> 172.21.15.43:0/1508614196 conn(0x7f9ecc06a860 msgr2=0x7f9ecc0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 shutdown_connections 2024-01-16T15:00:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.124+0000 7f9ed0a00640 1 -- 172.21.15.43:0/1508614196 wait complete. 2024-01-16T15:00:45.125 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:45.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.375+0000 7f5d26400640 1 Processor -- start 2024-01-16T15:00:45.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.375+0000 7f5d26400640 1 -- start start 2024-01-16T15:00:45.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d26400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d2015d350 0x7f5d200d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d26400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200cf240 con 0x7f5d200d1840 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d26400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200d1fb0 con 0x7f5d2015d730 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d26400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d200ce050 con 0x7f5d2015d350 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d1f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d2015d350 0x7f5d200d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d1f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d2015d350 0x7f5d200d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43688/0 (socket says 172.21.15.43:43688) 2024-01-16T15:00:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.377+0000 7f5d1f400640 1 -- 172.21.15.43:0/4231089636 learned_addr learned my addr 172.21.15.43:0/4231089636 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.378+0000 7f5d1f400640 1 -- 172.21.15.43:0/4231089636 >> v1:172.21.15.69:6789/0 conn(0x7f5d2015d730 legacy=0x7f5d200ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.378+0000 7f5d1f400640 1 -- 172.21.15.43:0/4231089636 >> v1:172.21.15.43:6789/0 conn(0x7f5d200d1840 legacy=0x7f5d200cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.378+0000 7f5d1f400640 1 -- 172.21.15.43:0/4231089636 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d200ce330 con 0x7f5d2015d350 2024-01-16T15:00:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.378+0000 7f5d1e000640 1 -- 172.21.15.43:0/4231089636 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699849426 0 0) 0x7f5d200cf240 con 0x7f5d200d1840 2024-01-16T15:00:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.379+0000 7f5d1f400640 1 --2- 172.21.15.43:0/4231089636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d2015d350 0x7f5d200d3d30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37738cd428b0db7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.379+0000 7f5d1e000640 1 -- 172.21.15.43:0/4231089636 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08030850 con 0x7f5d2015d350 2024-01-16T15:00:45.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.379+0000 7f5d1e000640 1 -- 172.21.15.43:0/4231089636 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d080309b0 con 0x7f5d2015d350 2024-01-16T15:00:45.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.379+0000 7f5d1e000640 1 -- 172.21.15.43:0/4231089636 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08030d40 con 0x7f5d2015d350 2024-01-16T15:00:45.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 -- 172.21.15.43:0/4231089636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d2015d350 msgr2=0x7f5d200d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 --2- 172.21.15.43:0/4231089636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d2015d350 0x7f5d200d3d30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0802b990 comp rx=0 tx=0).stop 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 -- 172.21.15.43:0/4231089636 shutdown_connections 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 --2- 172.21.15.43:0/4231089636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d2015d350 0x7f5d200d3d30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 -- 172.21.15.43:0/4231089636 >> 172.21.15.43:0/4231089636 conn(0x7f5d2009c5c0 msgr2=0x7f5d200d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 -- 172.21.15.43:0/4231089636 shutdown_connections 2024-01-16T15:00:45.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.380+0000 7f5d26400640 1 -- 172.21.15.43:0/4231089636 wait complete. 2024-01-16T15:00:45.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.382+0000 7f5d26400640 1 Processor -- start 2024-01-16T15:00:45.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.382+0000 7f5d26400640 1 -- start start 2024-01-16T15:00:45.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2015d350 0x7f5d200b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 0x7f5d200b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201f6930 con 0x7f5d200d1840 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20151c20 con 0x7f5d2015d730 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d26400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201f4090 con 0x7f5d2015d350 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43698/0 (socket says 172.21.15.43:43698) 2024-01-16T15:00:45.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 -- 172.21.15.43:0/717661802 learned_addr learned my addr 172.21.15.43:0/717661802 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:45.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1fe00640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 0x7f5d200b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1ea00640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2015d350 0x7f5d200b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 -- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2015d350 msgr2=0x7f5d200b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2015d350 0x7f5d200b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 -- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 msgr2=0x7f5d200b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 0x7f5d200b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.383+0000 7f5d1f400640 1 -- 172.21.15.43:0/717661802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080094d0 con 0x7f5d200d1840 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d1fe00640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 0x7f5d200b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d1f400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08039d60 con 0x7f5d200d1840 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08043400 con 0x7f5d200d1840 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d080437b0 con 0x7f5d200d1840 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d201f6b80 con 0x7f5d200d1840 2024-01-16T15:00:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.384+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d201f7140 con 0x7f5d200d1840 2024-01-16T15:00:45.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.386+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d200d2700 con 0x7f5d200d1840 2024-01-16T15:00:45.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.386+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5d0802e070 con 0x7f5d200d1840 2024-01-16T15:00:45.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.387+0000 7f5d17400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 0x7f5cf806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.387+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5d0803e070 con 0x7f5d200d1840 2024-01-16T15:00:45.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.388+0000 7f5d1ea00640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 0x7f5cf806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.388+0000 7f5d1ea00640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 0x7f5cf806d7d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d200cc130 tx=0x7f5d10007240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.392+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d08077990 con 0x7f5d200d1840 2024-01-16T15:00:45.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.587+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d200a1230 con 0x7f5d200d1840 2024-01-16T15:00:45.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.588+0000 7f5d17400640 1 -- 172.21.15.43:0/717661802 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d08034020 con 0x7f5d200d1840 2024-01-16T15:00:45.569 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 msgr2=0x7f5cf806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 0x7f5cf806d7d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d200cc130 tx=0x7f5d10007240 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 msgr2=0x7f5d201516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d08004730 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 shutdown_connections 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cf806b310 0x7f5cf806d7d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d2015d730 0x7f5d200b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2015d350 0x7f5d200b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 --2- 172.21.15.43:0/717661802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200d1840 0x7f5d201516e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 >> 172.21.15.43:0/717661802 conn(0x7f5d2009c5c0 msgr2=0x7f5d2015c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 shutdown_connections 2024-01-16T15:00:45.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.591+0000 7f5d26400640 1 -- 172.21.15.43:0/717661802 wait complete. 2024-01-16T15:00:45.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:00:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.831+0000 7feb0ac00640 1 Processor -- start 2024-01-16T15:00:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.831+0000 7feb0ac00640 1 -- start start 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb0406d520 0x7feb0406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb0415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feb040abff0 0x7feb0415f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb040b41a0 con 0x7feb0406d520 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0415c360 con 0x7feb040abff0 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb0ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0406de40 con 0x7feb040ad720 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb03e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb0406d520 0x7feb0406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb03e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb0406d520 0x7feb0406d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43706/0 (socket says 172.21.15.43:43706) 2024-01-16T15:00:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.833+0000 7feb03e00640 1 -- 172.21.15.43:0/2345186854 learned_addr learned my addr 172.21.15.43:0/2345186854 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb08c00640 1 --2- 172.21.15.43:0/2345186854 >> v2:172.21.15.69:3300/0 conn(0x7feb040abff0 0x7feb0415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03400640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb0415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03e00640 1 -- 172.21.15.43:0/2345186854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 msgr2=0x7feb0415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03e00640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb0415ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03e00640 1 -- 172.21.15.43:0/2345186854 >> v2:172.21.15.69:3300/0 conn(0x7feb040abff0 msgr2=0x7feb0415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03e00640 1 --2- 172.21.15.43:0/2345186854 >> v2:172.21.15.69:3300/0 conn(0x7feb040abff0 0x7feb0415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03e00640 1 -- 172.21.15.43:0/2345186854 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0415ba30 con 0x7feb0406d520 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.834+0000 7feb03400640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb0415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.835+0000 7feb03e00640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb0406d520 0x7feb0406d900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb040aca20 tx=0x7feaf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121a7f20cab6f6c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.835+0000 7feb02a00640 1 -- 172.21.15.43:0/2345186854 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4031870 con 0x7feb0406d520 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.836+0000 7feb02a00640 1 -- 172.21.15.43:0/2345186854 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf40319d0 con 0x7feb0406d520 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.836+0000 7feb02a00640 1 -- 172.21.15.43:0/2345186854 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4031d60 con 0x7feb0406d520 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.836+0000 7feb0ac00640 1 -- 172.21.15.43:0/2345186854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb0406d520 msgr2=0x7feb0406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.836+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb0406d520 0x7feb0406d900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb040aca20 tx=0x7feaf402b910 comp rx=0 tx=0).stop 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 -- 172.21.15.43:0/2345186854 shutdown_connections 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2345186854 >> v2:172.21.15.69:3300/0 conn(0x7feb040abff0 0x7feb0415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb0415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2345186854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb0406d520 0x7feb0406d900 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 -- 172.21.15.43:0/2345186854 >> 172.21.15.43:0/2345186854 conn(0x7feb040b0d80 msgr2=0x7feb040b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 -- 172.21.15.43:0/2345186854 shutdown_connections 2024-01-16T15:00:45.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.837+0000 7feb0ac00640 1 -- 172.21.15.43:0/2345186854 wait complete. 2024-01-16T15:00:45.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.838+0000 7feb0ac00640 1 Processor -- start 2024-01-16T15:00:45.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.839+0000 7feb0ac00640 1 -- start start 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.839+0000 7feb0ac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb0406d520 0x7feb0409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.839+0000 7feb0ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.839+0000 7feb0ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb040cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.839+0000 7feb0ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04148e30 con 0x7feb040abff0 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb0ac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04099170 con 0x7feb0406d520 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb0ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb040cc1a0 con 0x7feb040ad720 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43710/0 (socket says 172.21.15.43:43710) 2024-01-16T15:00:45.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 -- 172.21.15.43:0/2485519870 learned_addr learned my addr 172.21.15.43:0/2485519870 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03e00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb0406d520 0x7feb0409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb08c00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb040cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 -- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 msgr2=0x7feb040cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb040cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 -- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb0406d520 msgr2=0x7feb0409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb0406d520 0x7feb0409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb03400640 1 -- 172.21.15.43:0/2485519870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf40094d0 con 0x7feb040abff0 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.840+0000 7feb08c00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb040cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feb03400640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c7c0 tx=0x7feaec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec007820 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec007980 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec007d10 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb040cc480 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.841+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb040cca40 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.842+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb040ccc00 con 0x7feb040abff0 2024-01-16T15:00:45.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.843+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feaec0113f0 con 0x7feb040abff0 2024-01-16T15:00:45.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.844+0000 7feafbe00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 0x7feadc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.844+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feaec0898d0 con 0x7feb040abff0 2024-01-16T15:00:45.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.845+0000 7feb03e00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 0x7feadc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.845+0000 7feb03e00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 0x7feadc06d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feaf4036000 tx=0x7feaf403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:45.849+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaec054a80 con 0x7feb040abff0 2024-01-16T15:00:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.047+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feb04074770 con 0x7feb040abff0 2024-01-16T15:00:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.048+0000 7feafbe00640 1 -- 172.21.15.43:0/2485519870 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7feaec0580a0 con 0x7feb040abff0 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:46.029 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 msgr2=0x7feadc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 0x7feadc06d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feaf4036000 tx=0x7feaf403c040 comp rx=0 tx=0).stop 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 msgr2=0x7feb04157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c7c0 tx=0x7feaec00cc80 comp rx=0 tx=0).stop 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 shutdown_connections 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feadc06b2c0 0x7feadc06d780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb040ad720 0x7feb040cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb040abff0 0x7feb04157130 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 --2- 172.21.15.43:0/2485519870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb0406d520 0x7feb0409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 >> 172.21.15.43:0/2485519870 conn(0x7feb040b0d80 msgr2=0x7feb040aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 shutdown_connections 2024-01-16T15:00:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.051+0000 7feb0ac00640 1 -- 172.21.15.43:0/2485519870 wait complete. 2024-01-16T15:00:46.051 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:00:46.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:00:46.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.296+0000 7f1d82a00640 1 Processor -- start 2024-01-16T15:00:46.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.296+0000 7f1d82a00640 1 -- start start 2024-01-16T15:00:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.298+0000 7f1d82a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 0x7f1d7c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.298+0000 7f1d82a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d7c06d9e0 0x7f1d7c0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.298+0000 7f1d82a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c15f930 con 0x7f1d7c144770 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.298+0000 7f1d82a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b43b0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.298+0000 7f1d82a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b3b10 con 0x7f1d7c15f270 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 0x7f1d7c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d82000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d7c06d9e0 0x7f1d7c0cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 0x7f1d7c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34150/0 (socket says 172.21.15.43:34150) 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d80c00640 1 -- 172.21.15.43:0/593307409 learned_addr learned my addr 172.21.15.43:0/593307409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d82000640 1 -- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 msgr2=0x7f1d7c0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d82000640 1 --2- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 0x7f1d7c0cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.299+0000 7f1d82000640 1 -- 172.21.15.43:0/593307409 >> v1:172.21.15.69:6789/0 conn(0x7f1d7c144770 legacy=0x7f1d7c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.300+0000 7f1d82000640 1 -- 172.21.15.43:0/593307409 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0b3df0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.301+0000 7f1d82000640 1 --2- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c0cd9e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009d90 tx=0x7f1d7402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77fb6fed58f6a678 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.302+0000 7f1d7be00640 1 -- 172.21.15.43:0/593307409 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7402cc50 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.302+0000 7f1d7be00640 1 -- 172.21.15.43:0/593307409 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7402cdb0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.302+0000 7f1d7be00640 1 -- 172.21.15.43:0/593307409 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74034660 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 -- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c06d9e0 msgr2=0x7f1d7c0cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 --2- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c0cd9e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009d90 tx=0x7f1d7402bea0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 -- 172.21.15.43:0/593307409 shutdown_connections 2024-01-16T15:00:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 --2- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c0cd9e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 --2- 172.21.15.43:0/593307409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c15f270 0x7f1d7c0cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 -- 172.21.15.43:0/593307409 >> 172.21.15.43:0/593307409 conn(0x7f1d7c0cec10 msgr2=0x7f1d7c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 -- 172.21.15.43:0/593307409 shutdown_connections 2024-01-16T15:00:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.303+0000 7f1d82a00640 1 -- 172.21.15.43:0/593307409 wait complete. 2024-01-16T15:00:46.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.305+0000 7f1d82a00640 1 Processor -- start 2024-01-16T15:00:46.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.305+0000 7f1d82a00640 1 -- start start 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.306+0000 7f1d82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.306+0000 7f1d82a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c144770 0x7f1d7c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 0x7f1d7c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1405e0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c13f9d0 con 0x7f1d7c15f270 2024-01-16T15:00:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c13fcb0 con 0x7f1d7c144770 2024-01-16T15:00:46.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43722/0 (socket says 172.21.15.43:43722) 2024-01-16T15:00:46.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 -- 172.21.15.43:0/772387267 learned_addr learned my addr 172.21.15.43:0/772387267 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d80c00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c144770 0x7f1d7c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82000640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 0x7f1d7c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 -- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c144770 msgr2=0x7f1d7c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c144770 0x7f1d7c15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 -- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 msgr2=0x7f1d7c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 0x7f1d7c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d81600640 1 -- 172.21.15.43:0/772387267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d74009a70 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.307+0000 7f1d82000640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 0x7f1d7c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d81600640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00c450 tx=0x7f1d6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c0052d0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c005430 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c012870 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c13ff90 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.308+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c15dc80 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.309+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c04e840 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.311+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1d6c0129d0 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.312+0000 7f1d79600640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 0x7f1d5406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.312+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1d6c089450 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.313+0000 7f1d80c00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 0x7f1d5406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.313+0000 7f1d80c00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 0x7f1d5406d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0cea60 tx=0x7f1d64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.317+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d6c054570 con 0x7f1d7c06d9e0 2024-01-16T15:00:46.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:46.512+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1d7c0b2460 con 0x7f1d7c06d9e0 2024-01-16T15:00:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.021+0000 7f1d79600640 1 -- 172.21.15.43:0/772387267 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1d6c057b90 con 0x7f1d7c06d9e0 2024-01-16T15:00:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:00:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 msgr2=0x7f1d5406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 0x7f1d5406d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0cea60 tx=0x7f1d64008040 comp rx=0 tx=0).stop 2024-01-16T15:00:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 msgr2=0x7f1d7c13b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00c450 tx=0x7f1d6c00c910 comp rx=0 tx=0).stop 2024-01-16T15:00:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 shutdown_connections 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d5406b2c0 0x7f1d5406d780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.025+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d7c15f270 0x7f1d7c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.026+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d7c144770 0x7f1d7c15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.026+0000 7f1d82a00640 1 --2- 172.21.15.43:0/772387267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7c06d9e0 0x7f1d7c13b380 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.026+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 >> 172.21.15.43:0/772387267 conn(0x7f1d7c0cec10 msgr2=0x7f1d7c0ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.026+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 shutdown_connections 2024-01-16T15:00:47.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.026+0000 7f1d82a00640 1 -- 172.21.15.43:0/772387267 wait complete. 2024-01-16T15:00:47.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:00:47.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.270+0000 7f73bfa00640 1 Processor -- start 2024-01-16T15:00:47.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.270+0000 7f73bfa00640 1 -- start start 2024-01-16T15:00:47.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bfa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f73b8162730 0x7f73b8164b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bfa00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b806d700 con 0x7f73b806e550 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bfa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b80b2860 con 0x7f73b8070db0 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bfa00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b80b2b40 con 0x7f73b8162730 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bd000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f73b806e550 0x7f73b80706a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35358/0 (socket says 172.21.15.43:35358) 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bd000640 1 -- 172.21.15.43:0/3262169543 learned_addr learned my addr 172.21.15.43:0/3262169543 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.272+0000 7f73bda00640 1 --2- 172.21.15.43:0/3262169543 >> v2:172.21.15.43:3300/0 conn(0x7f73b8162730 0x7f73b8164b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129711634 0 0) 0x7f73b806d700 con 0x7f73b806e550 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7398003520 con 0x7f73b806e550 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683602358 0 0) 0x7f7398003520 con 0x7f73b806e550 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 >> v2:172.21.15.43:3300/0 conn(0x7f73b8162730 msgr2=0x7f73b8164b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 --2- 172.21.15.43:0/3262169543 >> v2:172.21.15.43:3300/0 conn(0x7f73b8162730 0x7f73b8164b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 >> v1:172.21.15.69:6789/0 conn(0x7f73b8070db0 legacy=0x7f73b8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b80b2e20 con 0x7f73b806e550 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f73a8001670 con 0x7f73b806e550 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.273+0000 7f73bda00640 1 --2- 172.21.15.43:0/3262169543 >> v2:172.21.15.43:3300/0 conn(0x7f73b8162730 0x7f73b8164b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 >> v1:172.21.15.43:6789/0 conn(0x7f73b806e550 legacy=0x7f73b80706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 0x7f7398003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7398004370 0x7f7398004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73980050e0 0x7f739800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80b2b40 con 0x7f7398003970 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398010430 con 0x7f73980050e0 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980105f0 con 0x7f7398004370 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bd000640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 0x7f7398003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bda00640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73980050e0 0x7f739800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73b7e00640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7398004370 0x7f7398004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bd000640 1 -- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7398004370 msgr2=0x7f7398004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bd000640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7398004370 0x7f7398004840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bd000640 1 -- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73980050e0 msgr2=0x7f739800fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.274+0000 7f73bd000640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73980050e0 0x7f739800fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.275+0000 7f73bda00640 1 -- 172.21.15.43:0/3262169543 reap_dead start 2024-01-16T15:00:47.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.275+0000 7f73bd000640 1 -- 172.21.15.43:0/3262169543 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7398003520 con 0x7f7398003970 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.276+0000 7f73bd000640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 0x7f7398003d50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005070 tx=0x7f73980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c35115d0f7202b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.276+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8005b90 con 0x7f7398003970 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.276+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a802f070 con 0x7f7398003970 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.276+0000 7f73b7400640 1 -- 172.21.15.43:0/3262169543 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80080d0 con 0x7f7398003970 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 -- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 msgr2=0x7f7398003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 0x7f7398003d50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005070 tx=0x7f73980057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 -- 172.21.15.43:0/3262169543 shutdown_connections 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 --2- 172.21.15.43:0/3262169543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7398003970 0x7f7398003d50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 -- 172.21.15.43:0/3262169543 >> 172.21.15.43:0/3262169543 conn(0x7f73b80b0ba0 msgr2=0x7f73b80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 -- 172.21.15.43:0/3262169543 shutdown_connections 2024-01-16T15:00:47.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.277+0000 7f73bfa00640 1 -- 172.21.15.43:0/3262169543 wait complete. 2024-01-16T15:00:47.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.279+0000 7f73bfa00640 1 Processor -- start 2024-01-16T15:00:47.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.279+0000 7f73bfa00640 1 -- start start 2024-01-16T15:00:47.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7398003970 0x7f73b8151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b806d2d0 0x7f73b806c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8152690 con 0x7f73b814ed80 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8148060 con 0x7f7398003970 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bfa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8152ab0 con 0x7f73b806d2d0 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bda00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bda00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43744/0 (socket says 172.21.15.43:43744) 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.280+0000 7f73bda00640 1 -- 172.21.15.43:0/2227554891 learned_addr learned my addr 172.21.15.43:0/2227554891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bd000640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7398003970 0x7f73b8151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73b7e00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b806d2d0 0x7f73b806c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 -- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b806d2d0 msgr2=0x7f73b806c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b806d2d0 0x7f73b806c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 -- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7398003970 msgr2=0x7f73b8151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7398003970 0x7f73b8151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 -- 172.21.15.43:0/2227554891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b80b3a80 con 0x7f73b814ed80 2024-01-16T15:00:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73bda00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f73b80cf000 tx=0x7f73b000f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0003b90 con 0x7f73b814ed80 2024-01-16T15:00:47.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0003cf0 con 0x7f73b814ed80 2024-01-16T15:00:47.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.281+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0004270 con 0x7f73b814ed80 2024-01-16T15:00:47.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.282+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b8151a10 con 0x7f73b814ed80 2024-01-16T15:00:47.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.282+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b8151fd0 con 0x7f73b814ed80 2024-01-16T15:00:47.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.283+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b81527d0 con 0x7f73b814ed80 2024-01-16T15:00:47.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.284+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f73b00043d0 con 0x7f73b814ed80 2024-01-16T15:00:47.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.285+0000 7f73b4c00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 0x7f739006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.285+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f73b008a9c0 con 0x7f73b814ed80 2024-01-16T15:00:47.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.285+0000 7f73bd000640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 0x7f739006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.286+0000 7f73bd000640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 0x7f739006d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005040 tx=0x7f73a8037040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.289+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73b0055a80 con 0x7f73b814ed80 2024-01-16T15:00:47.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:47.484+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f73b81527d0 con 0x7f73b814ed80 2024-01-16T15:00:48.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.027+0000 7f73b4c00640 1 -- 172.21.15.43:0/2227554891 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7f73b81527d0 con 0x7f73b814ed80 2024-01-16T15:00:48.008 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.031+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 msgr2=0x7f739006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.031+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 0x7f739006d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005040 tx=0x7f73a8037040 comp rx=0 tx=0).stop 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.031+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 msgr2=0x7f73b814f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.031+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f73b80cf000 tx=0x7f73b000f810 comp rx=0 tx=0).stop 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 shutdown_connections 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f739006b2c0 0x7f739006d780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73b814ed80 0x7f73b814f210 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b806d2d0 0x7f73b806c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 --2- 172.21.15.43:0/2227554891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7398003970 0x7f73b8151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 >> 172.21.15.43:0/2227554891 conn(0x7f73b80b0ba0 msgr2=0x7f73b815bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.032+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 shutdown_connections 2024-01-16T15:00:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.033+0000 7f73bfa00640 1 -- 172.21.15.43:0/2227554891 wait complete. 2024-01-16T15:00:48.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:00:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.256+0000 7f3481800640 1 Processor -- start 2024-01-16T15:00:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.256+0000 7f3481800640 1 -- start start 2024-01-16T15:00:48.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.257+0000 7f3481800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f3481800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f347c0adb60 0x7f347c0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f3481800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c15b730 con 0x7f347c09cb30 2024-01-16T15:00:48.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f3481800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c0ac740 con 0x7f347c0adb60 2024-01-16T15:00:48.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f3481800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0ac460 con 0x7f347c15b350 2024-01-16T15:00:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f347aa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f347b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f347c0adb60 0x7f347c0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f347aa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34174/0 (socket says 172.21.15.43:34174) 2024-01-16T15:00:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.258+0000 7f347aa00640 1 -- 172.21.15.43:0/3432351141 learned_addr learned my addr 172.21.15.43:0/3432351141 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.259+0000 7f347aa00640 1 -- 172.21.15.43:0/3432351141 >> v2:172.21.15.69:3300/0 conn(0x7f347c0adb60 msgr2=0x7f347c0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.259+0000 7f347aa00640 1 --2- 172.21.15.43:0/3432351141 >> v2:172.21.15.69:3300/0 conn(0x7f347c0adb60 0x7f347c0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.259+0000 7f347aa00640 1 -- 172.21.15.43:0/3432351141 >> v1:172.21.15.69:6789/0 conn(0x7f347c09cb30 legacy=0x7f347c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.259+0000 7f347aa00640 1 -- 172.21.15.43:0/3432351141 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0b2a50 con 0x7f347c15b350 2024-01-16T15:00:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.260+0000 7f347aa00640 1 --2- 172.21.15.43:0/3432351141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38beac6b79599714 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.261+0000 7f3479600640 1 -- 172.21.15.43:0/3432351141 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c030840 con 0x7f347c15b350 2024-01-16T15:00:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.261+0000 7f3479600640 1 -- 172.21.15.43:0/3432351141 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c0309a0 con 0x7f347c15b350 2024-01-16T15:00:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.261+0000 7f3479600640 1 -- 172.21.15.43:0/3432351141 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c030d30 con 0x7f347c15b350 2024-01-16T15:00:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 -- 172.21.15.43:0/3432351141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 msgr2=0x7f347c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 --2- 172.21.15.43:0/3432351141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 -- 172.21.15.43:0/3432351141 shutdown_connections 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 --2- 172.21.15.43:0/3432351141 >> v2:172.21.15.69:3300/0 conn(0x7f347c0adb60 0x7f347c0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 --2- 172.21.15.43:0/3432351141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c15b350 0x7f347c0abe80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 -- 172.21.15.43:0/3432351141 >> 172.21.15.43:0/3432351141 conn(0x7f347c0b0ba0 msgr2=0x7f347c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 -- 172.21.15.43:0/3432351141 shutdown_connections 2024-01-16T15:00:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.262+0000 7f3481800640 1 -- 172.21.15.43:0/3432351141 wait complete. 2024-01-16T15:00:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.264+0000 7f3481800640 1 Processor -- start 2024-01-16T15:00:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.264+0000 7f3481800640 1 -- start start 2024-01-16T15:00:48.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c09cb30 0x7f347c14e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f347c0adb60 0x7f347c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c14d2e0 con 0x7f347c15b350 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c071350 con 0x7f347c0adb60 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f3481800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c14c9b0 con 0x7f347c09cb30 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f347b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f347b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43752/0 (socket says 172.21.15.43:43752) 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f347b400640 1 -- 172.21.15.43:0/3199119821 learned_addr learned my addr 172.21.15.43:0/3199119821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f347aa00640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c09cb30 0x7f347c14e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.265+0000 7f347a000640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f347c0adb60 0x7f347c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 -- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c09cb30 msgr2=0x7f347c14e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c09cb30 0x7f347c14e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 -- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f347c0adb60 msgr2=0x7f347c14d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f347c0adb60 0x7f347c14d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 -- 172.21.15.43:0/3199119821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0094d0 con 0x7f347c15b350 2024-01-16T15:00:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f347b400640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f347400efc0 tx=0x7f347400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.266+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474014480 con 0x7f347c15b350 2024-01-16T15:00:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.267+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400f040 con 0x7f347c15b350 2024-01-16T15:00:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.267+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c14cc90 con 0x7f347c15b350 2024-01-16T15:00:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.267+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474014820 con 0x7f347c15b350 2024-01-16T15:00:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.267+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c14ea50 con 0x7f347c15b350 2024-01-16T15:00:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.268+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c14d490 con 0x7f347c15b350 2024-01-16T15:00:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.269+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3474014980 con 0x7f347c15b350 2024-01-16T15:00:48.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.270+0000 7f3472a00640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 0x7f345406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.270+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f347408ad30 con 0x7f347c15b350 2024-01-16T15:00:48.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.270+0000 7f347aa00640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 0x7f345406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.271+0000 7f347aa00640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 0x7f345406d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.274+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3474055c90 con 0x7f347c15b350 2024-01-16T15:00:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:48.502+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f347c09d200 con 0x7f347c15b350 2024-01-16T15:00:49.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.050+0000 7f3472a00640 1 -- 172.21.15.43:0/3199119821 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f34740180c0 con 0x7f347c15b350 2024-01-16T15:00:49.031 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 4 and data pool 5 2024-01-16T15:00:49.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.053+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 msgr2=0x7f345406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.053+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 0x7f345406d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c008030 comp rx=0 tx=0).stop 2024-01-16T15:00:49.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.053+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 msgr2=0x7f347c13e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.053+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f347400efc0 tx=0x7f347400c490 comp rx=0 tx=0).stop 2024-01-16T15:00:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 shutdown_connections 2024-01-16T15:00:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f345406b2c0 0x7f345406d780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f347c0adb60 0x7f347c14d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f347c15b350 0x7f347c13e580 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 --2- 172.21.15.43:0/3199119821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f347c09cb30 0x7f347c14e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.056+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 >> 172.21.15.43:0/3199119821 conn(0x7f347c0b0ba0 msgr2=0x7f347c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.058+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 shutdown_connections 2024-01-16T15:00:49.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.058+0000 7f3481800640 1 -- 172.21.15.43:0/3199119821 wait complete. 2024-01-16T15:00:49.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:49.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.282+0000 7f4850c00640 1 Processor -- start 2024-01-16T15:00:49.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.282+0000 7f4850c00640 1 -- start start 2024-01-16T15:00:49.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.283+0000 7f4850c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f484c143570 0x7f484c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.283+0000 7f4850c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c071700 con 0x7f484c143950 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.283+0000 7f4850c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c15b410 con 0x7f484c15e6b0 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.283+0000 7f4850c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c15b130 con 0x7f484c143570 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f4842800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f484c143950 0x7f484c15a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35372/0 (socket says 172.21.15.43:35372) 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f4842800640 1 -- 172.21.15.43:0/2515022366 learned_addr learned my addr 172.21.15.43:0/2515022366 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484b400640 1 --2- 172.21.15.43:0/2515022366 >> v2:172.21.15.69:3300/0 conn(0x7f484c143570 0x7f484c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240730257 0 0) 0x7f4834001490 con 0x7f484c143950 2024-01-16T15:00:49.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c003520 con 0x7f484c143950 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296647087 0 0) 0x7f482c003520 con 0x7f484c143950 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 >> v2:172.21.15.69:3300/0 conn(0x7f484c143570 msgr2=0x7f484c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 --2- 172.21.15.43:0/2515022366 >> v2:172.21.15.69:3300/0 conn(0x7f484c143570 0x7f484c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.284+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 >> v1:172.21.15.69:6789/0 conn(0x7f484c15e6b0 legacy=0x7f484c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c15dbd0 con 0x7f484c143950 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4834001b80 con 0x7f484c143950 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 >> v1:172.21.15.43:6789/0 conn(0x7f484c143950 legacy=0x7f484c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482c003a30 0x7f482c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0052a0 0x7f482c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c15b130 con 0x7f482c0047c0 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c00c170 con 0x7f482c003a30 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c00c330 con 0x7f482c0052a0 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f484b400640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482c003a30 0x7f482c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 -- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0052a0 msgr2=0x7f482c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0052a0 0x7f482c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 -- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482c003a30 msgr2=0x7f482c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482c003a30 0x7f482c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.285+0000 7f4842800640 1 -- 172.21.15.43:0/2515022366 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c003520 con 0x7f482c0047c0 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.286+0000 7f484be00640 1 -- 172.21.15.43:0/2515022366 reap_dead start 2024-01-16T15:00:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4842800640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f484c071f70 tx=0x7f482c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f5e3676a75110b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340072d0 con 0x7f482c0047c0 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483402f070 con 0x7f482c0047c0 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f484aa00640 1 -- 172.21.15.43:0/2515022366 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340076d0 con 0x7f482c0047c0 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4850c00640 1 -- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 msgr2=0x7f482c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4850c00640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f484c071f70 tx=0x7f482c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4850c00640 1 -- 172.21.15.43:0/2515022366 shutdown_connections 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4850c00640 1 --2- 172.21.15.43:0/2515022366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.287+0000 7f4850c00640 1 -- 172.21.15.43:0/2515022366 >> 172.21.15.43:0/2515022366 conn(0x7f484c06a860 msgr2=0x7f484c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.288+0000 7f4850c00640 1 -- 172.21.15.43:0/2515022366 shutdown_connections 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.288+0000 7f4850c00640 1 -- 172.21.15.43:0/2515022366 wait complete. 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.289+0000 7f4850c00640 1 Processor -- start 2024-01-16T15:00:49.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.289+0000 7f4850c00640 1 -- start start 2024-01-16T15:00:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0047c0 0x7f484c1599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 0x7f484c1e5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c159f20 con 0x7f484c0c9450 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0c9e10 con 0x7f484c1e4cd0 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.290+0000 7f4850c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1e5aa0 con 0x7f482c0047c0 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43770/0 (socket says 172.21.15.43:43770) 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 -- 172.21.15.43:0/1932536701 learned_addr learned my addr 172.21.15.43:0/1932536701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 -- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0047c0 msgr2=0x7f484c1599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0047c0 0x7f484c1599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 -- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 msgr2=0x7f484c1e5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f484be00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 0x7f484c1e5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 0x7f484c1e5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f4842800640 1 -- 172.21.15.43:0/1932536701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c1e5d80 con 0x7f484c0c9450 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.291+0000 7f484be00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 0x7f484c1e5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:49.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4842800640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f483402ca50 tx=0x7f4834001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834004b60 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834004cc0 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340072f0 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c1e6060 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.292+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c1e6620 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.293+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c1e67b0 con 0x7f484c0c9450 2024-01-16T15:00:49.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.294+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4834031030 con 0x7f484c0c9450 2024-01-16T15:00:49.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.295+0000 7f4843e00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 0x7f482406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.296+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48340ab170 con 0x7f484c0c9450 2024-01-16T15:00:49.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.296+0000 7f484b400640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 0x7f482406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.296+0000 7f484b400640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 0x7f482406d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f483c009990 tx=0x7f483c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.300+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4834075f40 con 0x7f484c0c9450 2024-01-16T15:00:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.489+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f484c071700 con 0x7f484c0c9450 2024-01-16T15:00:49.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.492+0000 7f4843e00640 1 -- 172.21.15.43:0/1932536701 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f484c071700 con 0x7f484c0c9450 2024-01-16T15:00:49.474 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:49.474 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":34,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:48.504649+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:46.567224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:47.540637+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:49.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.495+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 msgr2=0x7f482406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.495+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 0x7f482406d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f483c009990 tx=0x7f483c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.495+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 msgr2=0x7f484c0c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.495+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f483402ca50 tx=0x7f4834001e40 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 shutdown_connections 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f482406b2c0 0x7f482406d780 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c1e4cd0 0x7f484c1e5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f484c0c9450 0x7f484c0c98d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 --2- 172.21.15.43:0/1932536701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482c0047c0 0x7f484c1599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 >> 172.21.15.43:0/1932536701 conn(0x7f484c06a860 msgr2=0x7f484c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 shutdown_connections 2024-01-16T15:00:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.496+0000 7f4850c00640 1 -- 172.21.15.43:0/1932536701 wait complete. 2024-01-16T15:00:49.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:49.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.724+0000 7f1275800640 1 Processor -- start 2024-01-16T15:00:49.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.724+0000 7f1275800640 1 -- start start 2024-01-16T15:00:49.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f127013dfd0 0x7f127015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12700cf050 0x7f12700cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f127006c130 0x7f127006c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12700cfa10 con 0x7f127006c130 2024-01-16T15:00:49.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127015fa70 con 0x7f127013dfd0 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1275800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127015ee60 con 0x7f12700cf050 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1274e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f127006c130 0x7f127006c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1274e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f127006c130 0x7f127006c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43784/0 (socket says 172.21.15.43:43784) 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.726+0000 7f1274e00640 1 -- 172.21.15.43:0/428179196 learned_addr learned my addr 172.21.15.43:0/428179196 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1267e00640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12700cf050 0x7f12700cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f126fe00640 1 --2- 172.21.15.43:0/428179196 >> v2:172.21.15.69:3300/0 conn(0x7f127013dfd0 0x7f127015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1274e00640 1 -- 172.21.15.43:0/428179196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12700cf050 msgr2=0x7f12700cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1274e00640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12700cf050 0x7f12700cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1274e00640 1 -- 172.21.15.43:0/428179196 >> v2:172.21.15.69:3300/0 conn(0x7f127013dfd0 msgr2=0x7f127015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1274e00640 1 --2- 172.21.15.43:0/428179196 >> v2:172.21.15.69:3300/0 conn(0x7f127013dfd0 0x7f127015be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.727+0000 7f1274e00640 1 -- 172.21.15.43:0/428179196 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127015f140 con 0x7f127006c130 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.728+0000 7f1274e00640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f127006c130 0x7f127006c580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f12700cfc10 tx=0x7f126802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcce2a0b131ed8af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.728+0000 7f126f400640 1 -- 172.21.15.43:0/428179196 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126802ca80 con 0x7f127006c130 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.728+0000 7f126f400640 1 -- 172.21.15.43:0/428179196 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126802cbe0 con 0x7f127006c130 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.728+0000 7f126f400640 1 -- 172.21.15.43:0/428179196 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680343f0 con 0x7f127006c130 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 -- 172.21.15.43:0/428179196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f127006c130 msgr2=0x7f127006c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f127006c130 0x7f127006c580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f12700cfc10 tx=0x7f126802bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 -- 172.21.15.43:0/428179196 shutdown_connections 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f127006c130 0x7f127006c580 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 --2- 172.21.15.43:0/428179196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12700cf050 0x7f12700cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 --2- 172.21.15.43:0/428179196 >> v2:172.21.15.69:3300/0 conn(0x7f127013dfd0 0x7f127015be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 -- 172.21.15.43:0/428179196 >> 172.21.15.43:0/428179196 conn(0x7f127006a860 msgr2=0x7f127006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 -- 172.21.15.43:0/428179196 shutdown_connections 2024-01-16T15:00:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.729+0000 7f1275800640 1 -- 172.21.15.43:0/428179196 wait complete. 2024-01-16T15:00:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.731+0000 7f1275800640 1 Processor -- start 2024-01-16T15:00:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.731+0000 7f1275800640 1 -- start start 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f127006c130 0x7f1270156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 0x7f1270154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1270155300 con 0x7f12700cf050 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700b2bc0 con 0x7f127013dfd0 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1275800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700b2ea0 con 0x7f127006c130 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1267e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1267e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43788/0 (socket says 172.21.15.43:43788) 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1267e00640 1 -- 172.21.15.43:0/3303296923 learned_addr learned my addr 172.21.15.43:0/3303296923 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1274e00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 0x7f1270154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f126fe00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f127006c130 0x7f1270156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.732+0000 7f1267e00640 1 -- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f127006c130 msgr2=0x7f1270156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1267e00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f127006c130 0x7f1270156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1267e00640 1 -- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 msgr2=0x7f1270154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1267e00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 0x7f1270154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1267e00640 1 -- 172.21.15.43:0/3303296923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009b30 con 0x7f12700cf050 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1274e00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 0x7f1270154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1267e00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f125800d800 tx=0x7f125800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800b6c0 con 0x7f12700cf050 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800b820 con 0x7f12700cf050 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800bbb0 con 0x7f12700cf050 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12700b3180 con 0x7f12700cf050 2024-01-16T15:00:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.733+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12700b3740 con 0x7f12700cf050 2024-01-16T15:00:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.734+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127006cfd0 con 0x7f12700cf050 2024-01-16T15:00:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.735+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f125800bd10 con 0x7f12700cf050 2024-01-16T15:00:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.736+0000 7f126cc00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 0x7f124806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.736+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f125808a180 con 0x7f12700cf050 2024-01-16T15:00:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.737+0000 7f126fe00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 0x7f124806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.737+0000 7f126fe00640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 0x7f124806d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1260006fd0 tx=0x7f1260008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.741+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1258054fe0 con 0x7f12700cf050 2024-01-16T15:00:49.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.928+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12700b38d0 con 0x7f12700cf050 2024-01-16T15:00:49.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.931+0000 7f126cc00640 1 -- 172.21.15.43:0/3303296923 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f1258058600 con 0x7f12700cf050 2024-01-16T15:00:49.912 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:49.912 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":34,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:48.504649+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:46.567224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:47.540637+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:49.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 msgr2=0x7f124806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 0x7f124806d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1260006fd0 tx=0x7f1260008040 comp rx=0 tx=0).stop 2024-01-16T15:00:49.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 msgr2=0x7f1270154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f125800d800 tx=0x7f125800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 shutdown_connections 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f124806b2c0 0x7f124806d780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f127013dfd0 0x7f1270154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12700cf050 0x7f1270154880 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 --2- 172.21.15.43:0/3303296923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f127006c130 0x7f1270156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.933+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 >> 172.21.15.43:0/3303296923 conn(0x7f127006a860 msgr2=0x7f1270159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.934+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 shutdown_connections 2024-01-16T15:00:49.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:49.934+0000 7f1275800640 1 -- 172.21.15.43:0/3303296923 wait complete. 2024-01-16T15:00:49.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:00:50.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.150+0000 7fa3a6e00640 1 Processor -- start 2024-01-16T15:00:50.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.150+0000 7fa3a6e00640 1 -- start start 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a6e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3a015b350 0x7fa3a00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a6e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3a009cb30 0x7fa3a00b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a6e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a015b730 con 0x7fa3a00adb60 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a6e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a00ac740 con 0x7fa3a009cb30 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a6e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a00ac460 con 0x7fa3a015b350 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a4e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa3a00adb60 0x7fa3a00adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35386/0 (socket says 172.21.15.43:35386) 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa3a4e00640 1 -- 172.21.15.43:0/1084885763 learned_addr learned my addr 172.21.15.43:0/1084885763 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.152+0000 7fa39f400640 1 --2- 172.21.15.43:0/1084885763 >> v2:172.21.15.43:3300/0 conn(0x7fa3a009cb30 0x7fa3a00b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39fe00640 1 --2- 172.21.15.43:0/1084885763 >> v2:172.21.15.69:3300/0 conn(0x7fa3a015b350 0x7fa3a00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39ea00640 1 -- 172.21.15.43:0/1084885763 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968535323 0 0) 0x7fa398001490 con 0x7fa3a00adb60 2024-01-16T15:00:50.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39f400640 1 -- 172.21.15.43:0/1084885763 >> v2:172.21.15.69:3300/0 conn(0x7fa3a015b350 msgr2=0x7fa3a00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39f400640 1 --2- 172.21.15.43:0/1084885763 >> v2:172.21.15.69:3300/0 conn(0x7fa3a015b350 0x7fa3a00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39f400640 1 -- 172.21.15.43:0/1084885763 >> v1:172.21.15.43:6789/0 conn(0x7fa3a00adb60 legacy=0x7fa3a00adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.153+0000 7fa39f400640 1 -- 172.21.15.43:0/1084885763 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a00b2a50 con 0x7fa3a009cb30 2024-01-16T15:00:50.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.154+0000 7fa39f400640 1 --2- 172.21.15.43:0/1084885763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a009cb30 0x7fa3a00b3530 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00b5100 tx=0x7fa38802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae91cb6311731b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.154+0000 7fa39ea00640 1 -- 172.21.15.43:0/1084885763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388031830 con 0x7fa3a009cb30 2024-01-16T15:00:50.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa39ea00640 1 -- 172.21.15.43:0/1084885763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388031990 con 0x7fa3a009cb30 2024-01-16T15:00:50.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa39ea00640 1 -- 172.21.15.43:0/1084885763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388031d20 con 0x7fa3a009cb30 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 -- 172.21.15.43:0/1084885763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a009cb30 msgr2=0x7fa3a00b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/1084885763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a009cb30 0x7fa3a00b3530 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00b5100 tx=0x7fa38802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 -- 172.21.15.43:0/1084885763 shutdown_connections 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/1084885763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a009cb30 0x7fa3a00b3530 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/1084885763 >> v2:172.21.15.69:3300/0 conn(0x7fa3a015b350 0x7fa3a00abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 -- 172.21.15.43:0/1084885763 >> 172.21.15.43:0/1084885763 conn(0x7fa3a00b0ba0 msgr2=0x7fa3a00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 -- 172.21.15.43:0/1084885763 shutdown_connections 2024-01-16T15:00:50.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.155+0000 7fa3a6e00640 1 -- 172.21.15.43:0/1084885763 wait complete. 2024-01-16T15:00:50.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.157+0000 7fa3a6e00640 1 Processor -- start 2024-01-16T15:00:50.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.157+0000 7fa3a6e00640 1 -- start start 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3a009cb30 0x7fa3a0071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00adb60 0x7fa3a0156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0157a10 con 0x7fa3a015b350 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0156790 con 0x7fa3a00adb60 2024-01-16T15:00:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.158+0000 7fa3a6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0159100 con 0x7fa3a009cb30 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43802/0 (socket says 172.21.15.43:43802) 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 -- 172.21.15.43:0/2115719830 learned_addr learned my addr 172.21.15.43:0/2115719830 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa39f400640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00adb60 0x7fa3a0156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa39fe00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3a009cb30 0x7fa3a0071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 -- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3a009cb30 msgr2=0x7fa3a0071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3a009cb30 0x7fa3a0071ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 -- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00adb60 msgr2=0x7fa3a0156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00adb60 0x7fa3a0156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.159+0000 7fa3a4e00640 1 -- 172.21.15.43:0/2115719830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa388009400 con 0x7fa3a015b350 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa3a4e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa39800c7f0 tx=0x7fa39800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39800a5e0 con 0x7fa3a015b350 2024-01-16T15:00:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39800a760 con 0x7fa3a015b350 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39800ab50 con 0x7fa3a015b350 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a01593e0 con 0x7fa3a015b350 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.160+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a01563d0 con 0x7fa3a015b350 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.161+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a015cc10 con 0x7fa3a015b350 2024-01-16T15:00:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.162+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa39800acb0 con 0x7fa3a015b350 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.163+0000 7fa397e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 0x7fa37806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.163+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa398088fe0 con 0x7fa3a015b350 2024-01-16T15:00:50.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.163+0000 7fa39fe00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 0x7fa37806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.164+0000 7fa39fe00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 0x7fa37806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa390009a30 tx=0x7fa390008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.167+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3a015cc10 con 0x7fa3a015b350 2024-01-16T15:00:50.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:50.392+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa3a015b730 con 0x7fa3a015b350 2024-01-16T15:00:51.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.088+0000 7fa397e00640 1 -- 172.21.15.43:0/2115719830 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa3a015b730 con 0x7fa3a015b350 2024-01-16T15:00:51.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.091+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 msgr2=0x7fa37806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.091+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 0x7fa37806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa390009a30 tx=0x7fa390008040 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.091+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 msgr2=0x7fa3a015b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.091+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa39800c7f0 tx=0x7fa39800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 shutdown_connections 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa37806b2c0 0x7fa37806d780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00adb60 0x7fa3a0156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3a015b350 0x7fa3a015b8d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 --2- 172.21.15.43:0/2115719830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3a009cb30 0x7fa3a0071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 >> 172.21.15.43:0/2115719830 conn(0x7fa3a00b0ba0 msgr2=0x7fa3a00af830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 shutdown_connections 2024-01-16T15:00:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.092+0000 7fa3a6e00640 1 -- 172.21.15.43:0/2115719830 wait complete. 2024-01-16T15:00:51.094 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:00:51.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:00:51.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.342+0000 7f6c5b200640 1 Processor -- start 2024-01-16T15:00:51.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.342+0000 7f6c5b200640 1 -- start start 2024-01-16T15:00:51.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c15d3b0 0x7f6c5c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c5c0d1bc0 0x7f6c5c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0d2010 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0c6bf0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0d1400 con 0x7f6c5c15d3b0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c53e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c53e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43804/0 (socket says 172.21.15.43:43804) 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c53e00640 1 -- 172.21.15.43:0/4287630525 learned_addr learned my addr 172.21.15.43:0/4287630525 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c5a800640 1 --2- 172.21.15.43:0/4287630525 >> v2:172.21.15.69:3300/0 conn(0x7f6c5c0d1bc0 0x7f6c5c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.344+0000 7f6c59e00640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c15d3b0 0x7f6c5c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.345+0000 7f6c53e00640 1 -- 172.21.15.43:0/4287630525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c15d3b0 msgr2=0x7f6c5c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:51.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.345+0000 7f6c53e00640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c15d3b0 0x7f6c5c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.345+0000 7f6c53e00640 1 -- 172.21.15.43:0/4287630525 >> v2:172.21.15.69:3300/0 conn(0x7f6c5c0d1bc0 msgr2=0x7f6c5c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:51.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.345+0000 7f6c53e00640 1 --2- 172.21.15.43:0/4287630525 >> v2:172.21.15.69:3300/0 conn(0x7f6c5c0d1bc0 0x7f6c5c0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.345+0000 7f6c53e00640 1 -- 172.21.15.43:0/4287630525 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0cedc0 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.346+0000 7f6c53e00640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0cf310 tx=0x7f6c4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af6d595faa7d972 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.346+0000 7f6c59400640 1 -- 172.21.15.43:0/4287630525 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440317e0 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.346+0000 7f6c59400640 1 -- 172.21.15.43:0/4287630525 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44031940 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.346+0000 7f6c59400640 1 -- 172.21.15.43:0/4287630525 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44031cd0 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 -- 172.21.15.43:0/4287630525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d0fa0 msgr2=0x7f6c5c0d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0cf310 tx=0x7f6c4402b880 comp rx=0 tx=0).stop 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 -- 172.21.15.43:0/4287630525 shutdown_connections 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 --2- 172.21.15.43:0/4287630525 >> v2:172.21.15.69:3300/0 conn(0x7f6c5c0d1bc0 0x7f6c5c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d0fa0 0x7f6c5c0d1680 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 --2- 172.21.15.43:0/4287630525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c15d3b0 0x7f6c5c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 -- 172.21.15.43:0/4287630525 >> 172.21.15.43:0/4287630525 conn(0x7f6c5c0d30a0 msgr2=0x7f6c5c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 -- 172.21.15.43:0/4287630525 shutdown_connections 2024-01-16T15:00:51.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.347+0000 7f6c5b200640 1 -- 172.21.15.43:0/4287630525 wait complete. 2024-01-16T15:00:51.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.349+0000 7f6c5b200640 1 Processor -- start 2024-01-16T15:00:51.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.349+0000 7f6c5b200640 1 -- start start 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.350+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 0x7f6c5c15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.350+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 0x7f6c5c0b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0a0dc0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c15c490 con 0x7f6c5c15d3b0 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0ca310 con 0x7f6c5c0d0fa0 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43808/0 (socket says 172.21.15.43:43808) 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 -- 172.21.15.43:0/62424146 learned_addr learned my addr 172.21.15.43:0/62424146 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5a800640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 0x7f6c5c0b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c59e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 0x7f6c5c15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 -- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 msgr2=0x7f6c5c15bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 0x7f6c5c15bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 -- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 msgr2=0x7f6c5c0b8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 0x7f6c5c0b8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c53e00640 1 -- 172.21.15.43:0/62424146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440094d0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c5a800640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 0x7f6c5c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.351+0000 7f6c59e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 0x7f6c5c15bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c53e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000c00 tx=0x7f6c440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44032880 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44032a20 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c0ca5f0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44032dd0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.352+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c0cabb0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.353+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c04e840 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.355+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6c4404e450 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.355+0000 7f6c52000640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 0x7f6c3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.356+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c440abdf0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.356+0000 7f6c59e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 0x7f6c3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.356+0000 7f6c59e00640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 0x7f6c3406d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c006fd0 tx=0x7f6c4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.360+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c44076c80 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:51.580+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c5c0d24c0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:51.997 INFO:tasks.ceph.mds.c.smithi043.stderr:2024-01-16T15:00:52.017+0000 7f5a5d600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:00:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.100+0000 7f6c52000640 1 -- 172.21.15.43:0/62424146 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6c4407a2a0 con 0x7f6c5c0d1bc0 2024-01-16T15:00:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.104+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 msgr2=0x7f6c3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.104+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 0x7f6c3406d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c006fd0 tx=0x7f6c4c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 msgr2=0x7f6c5c0cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000c00 tx=0x7f6c440046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 shutdown_connections 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c3406b2c0 0x7f6c3406d780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c5c15d3b0 0x7f6c5c0b8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c5c0d1bc0 0x7f6c5c0cbd80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 --2- 172.21.15.43:0/62424146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c5c0d0fa0 0x7f6c5c15bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 >> 172.21.15.43:0/62424146 conn(0x7f6c5c0d30a0 msgr2=0x7f6c5c0c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 shutdown_connections 2024-01-16T15:00:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.105+0000 7f6c5b200640 1 -- 172.21.15.43:0/62424146 wait complete. 2024-01-16T15:00:52.106 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:00:52.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.344+0000 7f8f3a200640 1 Processor -- start 2024-01-16T15:00:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.344+0000 7f8f3a200640 1 -- start start 2024-01-16T15:00:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f3a200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f340cf500 0x7f8f340cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f3a200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f34140900 con 0x7f8f340cf050 2024-01-16T15:00:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f3a200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3415fa70 con 0x7f8f3413dfd0 2024-01-16T15:00:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f3a200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3415c580 con 0x7f8f340cf500 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f39800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f340cf500 0x7f8f340cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f2be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8f340cf050 0x7f8f340cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35398/0 (socket says 172.21.15.43:35398) 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f2be00640 1 -- 172.21.15.43:0/4005060221 learned_addr learned my addr 172.21.15.43:0/4005060221 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.346+0000 7f8f39800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f340cf500 0x7f8f340cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43822/0 (socket says 172.21.15.43:43822) 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.347+0000 7f8f39800640 1 -- 172.21.15.43:0/4005060221 >> v1:172.21.15.69:6789/0 conn(0x7f8f3413dfd0 legacy=0x7f8f3415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.347+0000 7f8f39800640 1 -- 172.21.15.43:0/4005060221 >> v1:172.21.15.43:6789/0 conn(0x7f8f340cf050 legacy=0x7f8f340cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.347+0000 7f8f39800640 1 -- 172.21.15.43:0/4005060221 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3415ee60 con 0x7f8f340cf500 2024-01-16T15:00:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.347+0000 7f8f33e00640 1 -- 172.21.15.43:0/4005060221 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137544619 0 0) 0x7f8f34140900 con 0x7f8f340cf050 2024-01-16T15:00:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.348+0000 7f8f39800640 1 --2- 172.21.15.43:0/4005060221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f340cf500 0x7f8f340cdec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ed80de419ce723 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.348+0000 7f8f33e00640 1 -- 172.21.15.43:0/4005060221 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c02cc60 con 0x7f8f340cf500 2024-01-16T15:00:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.348+0000 7f8f33e00640 1 -- 172.21.15.43:0/4005060221 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c02cdc0 con 0x7f8f340cf500 2024-01-16T15:00:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.348+0000 7f8f33e00640 1 -- 172.21.15.43:0/4005060221 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c034660 con 0x7f8f340cf500 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 -- 172.21.15.43:0/4005060221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f340cf500 msgr2=0x7f8f340cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 --2- 172.21.15.43:0/4005060221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f340cf500 0x7f8f340cdec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 -- 172.21.15.43:0/4005060221 shutdown_connections 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 --2- 172.21.15.43:0/4005060221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f340cf500 0x7f8f340cdec0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 -- 172.21.15.43:0/4005060221 >> 172.21.15.43:0/4005060221 conn(0x7f8f3406a860 msgr2=0x7f8f3406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 -- 172.21.15.43:0/4005060221 shutdown_connections 2024-01-16T15:00:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.349+0000 7f8f3a200640 1 -- 172.21.15.43:0/4005060221 wait complete. 2024-01-16T15:00:52.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.350+0000 7f8f3a200640 1 Processor -- start 2024-01-16T15:00:52.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.351+0000 7f8f3a200640 1 -- start start 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.351+0000 7f8f3a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f340cf050 0x7f8f3414e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.351+0000 7f8f3a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f340cf500 0x7f8f3415dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.351+0000 7f8f3a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f3a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34150b80 con 0x7f8f3413dfd0 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f3a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b0820 con 0x7f8f340cf050 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f3a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3414ed30 con 0x7f8f340cf500 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43832/0 (socket says 172.21.15.43:43832) 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 -- 172.21.15.43:0/880422617 learned_addr learned my addr 172.21.15.43:0/880422617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f38e00640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f340cf050 0x7f8f3414e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 -- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f340cf500 msgr2=0x7f8f3415dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f340cf500 0x7f8f3415dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 -- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f340cf050 msgr2=0x7f8f3414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f340cf050 0x7f8f3414e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.352+0000 7f8f39800640 1 -- 172.21.15.43:0/880422617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009a70 con 0x7f8f3413dfd0 2024-01-16T15:00:52.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f39800640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c03e610 con 0x7f8f3413dfd0 2024-01-16T15:00:52.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c03e790 con 0x7f8f3413dfd0 2024-01-16T15:00:52.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3414ff00 con 0x7f8f3413dfd0 2024-01-16T15:00:52.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c03eb40 con 0x7f8f3413dfd0 2024-01-16T15:00:52.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.353+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f341504c0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.354+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f34150cc0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.355+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8f2c03b050 con 0x7f8f3413dfd0 2024-01-16T15:00:52.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.356+0000 7f8f31600640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 0x7f8f0c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.357+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f2c0ac2c0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.357+0000 7f8f38e00640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 0x7f8f0c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.357+0000 7f8f38e00640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 0x7f8f0c06d7b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400a750 tx=0x7f8f24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.361+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f34150cc0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.592+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f34150cc0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.593+0000 7f8f31600640 1 -- 172.21.15.43:0/880422617 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4901 (secure 0 0 0) 0x7f8f34150cc0 con 0x7f8f3413dfd0 2024-01-16T15:00:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:52.574 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4691,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/543136239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":543136239},{"type":"v1","addr":"172.21.15.69:6833","nonce":543136239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4939,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/198308009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":198308009},{"type":"v1","addr":"172.21.15.69:6835","nonce":198308009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/81338476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":81338476},{"type":"v1","addr":"172.21.15.43:6835","nonce":81338476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:00:48.505147+0000","modified":"2024-01-16T15:00:51.582617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6837/3759191221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3759191221},{"type":"v1","addr":"172.21.15.43:6837","nonce":3759191221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T15:00:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-01-16T15:00:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.595+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 msgr2=0x7f8f0c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.595+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 0x7f8f0c06d7b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400a750 tx=0x7f8f24008040 comp rx=0 tx=0).stop 2024-01-16T15:00:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.595+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 msgr2=0x7f8f3415e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.595+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 shutdown_connections 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8f0c06b2f0 0x7f8f0c06d7b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f3413dfd0 0x7f8f3415e200 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f340cf500 0x7f8f3415dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 --2- 172.21.15.43:0/880422617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f340cf050 0x7f8f3414e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 >> 172.21.15.43:0/880422617 conn(0x7f8f3406a860 msgr2=0x7f8f3413d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 shutdown_connections 2024-01-16T15:00:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.596+0000 7f8f3a200640 1 -- 172.21.15.43:0/880422617 wait complete. 2024-01-16T15:00:52.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:52.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.835+0000 7f168f200640 1 Processor -- start 2024-01-16T15:00:52.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.835+0000 7f168f200640 1 -- start start 2024-01-16T15:00:52.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1690143570 0x7f16900cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1690143950 0x7f16900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1690071700 con 0x7f169015e6b0 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169015b410 con 0x7f1690143570 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169015b130 con 0x7f1690143950 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1690143570 0x7f16900cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1690143570 0x7f16900cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43842/0 (socket says 172.21.15.43:43842) 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f168de00640 1 -- 172.21.15.43:0/3543020646 learned_addr learned my addr 172.21.15.43:0/3543020646 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.837+0000 7f1687e00640 1 --2- 172.21.15.43:0/3543020646 >> v2:172.21.15.69:3300/0 conn(0x7f1690143950 0x7f16900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.838+0000 7f168de00640 1 -- 172.21.15.43:0/3543020646 >> v2:172.21.15.69:3300/0 conn(0x7f1690143950 msgr2=0x7f16900cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.838+0000 7f168de00640 1 --2- 172.21.15.43:0/3543020646 >> v2:172.21.15.69:3300/0 conn(0x7f1690143950 0x7f16900cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.838+0000 7f168de00640 1 -- 172.21.15.43:0/3543020646 >> v1:172.21.15.69:6789/0 conn(0x7f169015e6b0 legacy=0x7f169015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.838+0000 7f168de00640 1 -- 172.21.15.43:0/3543020646 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169015dbd0 con 0x7f1690143570 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.839+0000 7f168de00640 1 --2- 172.21.15.43:0/3543020646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1690143570 0x7f16900cf050 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1690070f10 tx=0x7f168002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57ac6cd493c6dd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.839+0000 7f168d400640 1 -- 172.21.15.43:0/3543020646 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16800317e0 con 0x7f1690143570 2024-01-16T15:00:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.839+0000 7f168d400640 1 -- 172.21.15.43:0/3543020646 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1680031940 con 0x7f1690143570 2024-01-16T15:00:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.839+0000 7f168d400640 1 -- 172.21.15.43:0/3543020646 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680031cd0 con 0x7f1690143570 2024-01-16T15:00:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 -- 172.21.15.43:0/3543020646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1690143570 msgr2=0x7f16900cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 --2- 172.21.15.43:0/3543020646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1690143570 0x7f16900cf050 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1690070f10 tx=0x7f168002b880 comp rx=0 tx=0).stop 2024-01-16T15:00:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 -- 172.21.15.43:0/3543020646 shutdown_connections 2024-01-16T15:00:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 --2- 172.21.15.43:0/3543020646 >> v2:172.21.15.69:3300/0 conn(0x7f1690143950 0x7f16900cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 --2- 172.21.15.43:0/3543020646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1690143570 0x7f16900cf050 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 -- 172.21.15.43:0/3543020646 >> 172.21.15.43:0/3543020646 conn(0x7f169006a860 msgr2=0x7f169006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 -- 172.21.15.43:0/3543020646 shutdown_connections 2024-01-16T15:00:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.840+0000 7f168f200640 1 -- 172.21.15.43:0/3543020646 wait complete. 2024-01-16T15:00:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.842+0000 7f168f200640 1 Processor -- start 2024-01-16T15:00:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.842+0000 7f168f200640 1 -- start start 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1690143570 0x7f1690139920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1690143950 0x7f16900ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690139e60 con 0x7f169015e6b0 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690140ad0 con 0x7f1690143950 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169013ed30 con 0x7f1690143570 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43846/0 (socket says 172.21.15.43:43846) 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168e800640 1 -- 172.21.15.43:0/1545419826 learned_addr learned my addr 172.21.15.43:0/1545419826 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f1687e00640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1690143950 0x7f16900ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168de00640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1690143570 0x7f1690139920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.843+0000 7f168e800640 1 -- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1690143570 msgr2=0x7f1690139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168e800640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1690143570 0x7f1690139920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168e800640 1 -- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1690143950 msgr2=0x7f16900ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168e800640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1690143950 0x7f16900ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168e800640 1 -- 172.21.15.43:0/1545419826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800094d0 con 0x7f169015e6b0 2024-01-16T15:00:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168e800640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f168800c7b0 tx=0x7f168800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688004b60 con 0x7f169015e6b0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688004cc0 con 0x7f169015e6b0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169013f010 con 0x7f169015e6b0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168800aa90 con 0x7f169015e6b0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.844+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690140d20 con 0x7f169015e6b0 2024-01-16T15:00:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.845+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1690140f40 con 0x7f169015e6b0 2024-01-16T15:00:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.846+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f168800abf0 con 0x7f169015e6b0 2024-01-16T15:00:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.848+0000 7f1686000640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 0x7f166806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.848+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1688089d20 con 0x7f169015e6b0 2024-01-16T15:00:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.848+0000 7f168de00640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 0x7f166806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.849+0000 7f168de00640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 0x7f166806d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1680036000 tx=0x7f168003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:52.853+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1688054b80 con 0x7f169015e6b0 2024-01-16T15:00:53.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.047+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f169004e840 con 0x7f169015e6b0 2024-01-16T15:00:53.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.049+0000 7f1686000640 1 -- 172.21.15.43:0/1545419826 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f16880581a0 con 0x7f169015e6b0 2024-01-16T15:00:53.029 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:53.030 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":35,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:00:50.068752+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:00:46.567224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:00:47.540637+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:53.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 msgr2=0x7f166806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 0x7f166806d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1680036000 tx=0x7f168003d040 comp rx=0 tx=0).stop 2024-01-16T15:00:53.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 msgr2=0x7f169013b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f168800c7b0 tx=0x7f168800cc70 comp rx=0 tx=0).stop 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 shutdown_connections 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f166806b2f0 0x7f166806d7b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1690143950 0x7f16900ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f169015e6b0 0x7f169013b150 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 --2- 172.21.15.43:0/1545419826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1690143570 0x7f1690139920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 >> 172.21.15.43:0/1545419826 conn(0x7f169006a860 msgr2=0x7f169006d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 shutdown_connections 2024-01-16T15:00:53.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.051+0000 7f168f200640 1 -- 172.21.15.43:0/1545419826 wait complete. 2024-01-16T15:00:53.053 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:00:53.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:53.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.297+0000 7f8793200640 1 Processor -- start 2024-01-16T15:00:53.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.297+0000 7f8793200640 1 -- start start 2024-01-16T15:00:53.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.298+0000 7f8793200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f878c0d1490 0x7f878c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.298+0000 7f8793200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.298+0000 7f8793200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0cf0b0 con 0x7f878c15d5a0 2024-01-16T15:00:53.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.298+0000 7f8793200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0b01a0 con 0x7f878c0d1490 2024-01-16T15:00:53.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.298+0000 7f8793200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c15d980 con 0x7f878c0b0630 2024-01-16T15:00:53.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.299+0000 7f8791e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f878c15d5a0 0x7f878c0d3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50002/0 (socket says 172.21.15.43:50002) 2024-01-16T15:00:53.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.299+0000 7f8791e00640 1 -- 172.21.15.43:0/3081599980 learned_addr learned my addr 172.21.15.43:0/3081599980 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:53.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.299+0000 7f8792800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.299+0000 7f8791400640 1 --2- 172.21.15.43:0/3081599980 >> v2:172.21.15.69:3300/0 conn(0x7f878c0d1490 0x7f878c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8791400640 1 -- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 msgr2=0x7f878c0b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8791400640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c0b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8791400640 1 -- 172.21.15.43:0/3081599980 >> v1:172.21.15.69:6789/0 conn(0x7f878c15d5a0 legacy=0x7f878c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8791400640 1 -- 172.21.15.43:0/3081599980 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0d1e30 con 0x7f878c0d1490 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8792800640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c0b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:53.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.300+0000 7f8790a00640 1 -- 172.21.15.43:0/3081599980 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180427674 0 0) 0x7f878c0cf0b0 con 0x7f878c15d5a0 2024-01-16T15:00:53.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.302+0000 7f8791400640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c0d1490 0x7f878c0d18f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f87740097a0 tx=0x7f877402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79568cb85d08db8a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.302+0000 7f8790a00640 1 -- 172.21.15.43:0/3081599980 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877402d040 con 0x7f878c0d1490 2024-01-16T15:00:53.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.302+0000 7f8790a00640 1 -- 172.21.15.43:0/3081599980 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8774031810 con 0x7f878c0d1490 2024-01-16T15:00:53.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.302+0000 7f8790a00640 1 -- 172.21.15.43:0/3081599980 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774031ba0 con 0x7f878c0d1490 2024-01-16T15:00:53.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 -- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c0d1490 msgr2=0x7f878c0d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c0d1490 0x7f878c0d18f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f87740097a0 tx=0x7f877402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 -- 172.21.15.43:0/3081599980 shutdown_connections 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c0d1490 0x7f878c0d18f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 --2- 172.21.15.43:0/3081599980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c0b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 -- 172.21.15.43:0/3081599980 >> 172.21.15.43:0/3081599980 conn(0x7f878c09c3a0 msgr2=0x7f878c0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 -- 172.21.15.43:0/3081599980 shutdown_connections 2024-01-16T15:00:53.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.303+0000 7f8793200640 1 -- 172.21.15.43:0/3081599980 wait complete. 2024-01-16T15:00:53.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.305+0000 7f8793200640 1 Processor -- start 2024-01-16T15:00:53.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.305+0000 7f8793200640 1 -- start start 2024-01-16T15:00:53.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 0x7f878c1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b3730 con 0x7f878c0d1490 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b42b0 con 0x7f878c15d5a0 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.306+0000 7f8793200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b3980 con 0x7f878c0b0630 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51862/0 (socket says 172.21.15.43:51862) 2024-01-16T15:00:53.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 -- 172.21.15.43:0/630407378 learned_addr learned my addr 172.21.15.43:0/630407378 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791e00640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8792800640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 0x7f878c1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 -- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 msgr2=0x7f878c15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 -- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 msgr2=0x7f878c1f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 0x7f878c1f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 -- 172.21.15.43:0/630407378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009a70 con 0x7f878c0d1490 2024-01-16T15:00:53.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8792800640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 0x7f878c1f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:53.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.307+0000 7f8791400640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f87740096f0 tx=0x7f87740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.308+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877403ae60 con 0x7f878c0d1490 2024-01-16T15:00:53.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.308+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87740423f0 con 0x7f878c0d1490 2024-01-16T15:00:53.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.308+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8774009400 con 0x7f878c0d1490 2024-01-16T15:00:53.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.308+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87740427a0 con 0x7f878c0d1490 2024-01-16T15:00:53.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.308+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c0b3f40 con 0x7f878c0d1490 2024-01-16T15:00:53.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.309+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c04e840 con 0x7f878c0d1490 2024-01-16T15:00:53.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.310+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f877404b450 con 0x7f878c0d1490 2024-01-16T15:00:53.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.312+0000 7f878a000640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 0x7f876406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.312+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87740ab9a0 con 0x7f878c0d1490 2024-01-16T15:00:53.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.312+0000 7f8791e00640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 0x7f876406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.313+0000 7f8791e00640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 0x7f876406d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f877c000f60 tx=0x7f877c007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.317+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87740768a0 con 0x7f878c0d1490 2024-01-16T15:00:53.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.505+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f878c15ce40 con 0x7f876406b2c0 2024-01-16T15:00:53.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.534+0000 7f878a000640 1 -- 172.21.15.43:0/630407378 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287480 (secure 0 0 0) 0x7f878c15ce40 con 0x7f876406b2c0 2024-01-16T15:00:53.518 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:53.521 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.541+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 msgr2=0x7f876406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.541+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 0x7f876406d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f877c000f60 tx=0x7f877c007b50 comp rx=0 tx=0).stop 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.541+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 msgr2=0x7f878c0c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.541+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f87740096f0 tx=0x7f87740046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 shutdown_connections 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f876406b2c0 0x7f876406d780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f878c15d5a0 0x7f878c1f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f878c0d1490 0x7f878c0c9670 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 --2- 172.21.15.43:0/630407378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f878c0b0630 0x7f878c15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 >> 172.21.15.43:0/630407378 conn(0x7f878c09c3a0 msgr2=0x7f878c15dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 shutdown_connections 2024-01-16T15:00:53.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.542+0000 7f8793200640 1 -- 172.21.15.43:0/630407378 wait complete. 2024-01-16T15:00:53.543 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-01-16T15:00:52.814644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241160,"kb_used_data":4172,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733560,"statfs":{"total":773094113280,"available":772847165440,"internally_reserved":0,"allocated":4272128,"data_stored":2362555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.789243"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103579+0000","last_change":"2024-01-16T15:00:49.100076+0000","last_active":"2024-01-16T15:00:49.103579+0000","last_peered":"2024-01-16T15:00:49.103579+0000","last_clean":"2024-01-16T15:00:49.103579+0000","last_became_active":"2024-01-16T15:00:49.099726+0000","last_became_peered":"2024-01-16T15:00:49.099726+0000","last_unstale":"2024-01-16T15:00:49.103579+0000","last_undegraded":"2024-01-16T15:00:49.103579+0000","last_fullsized":"2024-01-16T15:00:49.103579+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:47.209930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089904+0000","last_change":"2024-01-16T15:00:49.082846+0000","last_active":"2024-01-16T15:00:49.089904+0000","last_peered":"2024-01-16T15:00:49.089904+0000","last_clean":"2024-01-16T15:00:49.089904+0000","last_became_active":"2024-01-16T15:00:49.082551+0000","last_became_peered":"2024-01-16T15:00:49.082551+0000","last_unstale":"2024-01-16T15:00:49.089904+0000","last_undegraded":"2024-01-16T15:00:49.089904+0000","last_fullsized":"2024-01-16T15:00:49.089904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:46:47.690663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101858+0000","last_change":"2024-01-16T15:00:49.098483+0000","last_active":"2024-01-16T15:00:49.101858+0000","last_peered":"2024-01-16T15:00:49.101858+0000","last_clean":"2024-01-16T15:00:49.101858+0000","last_became_active":"2024-01-16T15:00:49.094570+0000","last_became_peered":"2024-01-16T15:00:49.094570+0000","last_unstale":"2024-01-16T15:00:49.101858+0000","last_undegraded":"2024-01-16T15:00:49.101858+0000","last_fullsized":"2024-01-16T15:00:49.101858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:32.023437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098152+0000","last_change":"2024-01-16T15:00:49.094940+0000","last_active":"2024-01-16T15:00:49.098152+0000","last_peered":"2024-01-16T15:00:49.098152+0000","last_clean":"2024-01-16T15:00:49.098152+0000","last_became_active":"2024-01-16T15:00:49.094526+0000","last_became_peered":"2024-01-16T15:00:49.094526+0000","last_unstale":"2024-01-16T15:00:49.098152+0000","last_undegraded":"2024-01-16T15:00:49.098152+0000","last_fullsized":"2024-01-16T15:00:49.098152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:16.494984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090578+0000","last_change":"2024-01-16T15:00:49.085022+0000","last_active":"2024-01-16T15:00:49.090578+0000","last_peered":"2024-01-16T15:00:49.090578+0000","last_clean":"2024-01-16T15:00:49.090578+0000","last_became_active":"2024-01-16T15:00:49.084708+0000","last_became_peered":"2024-01-16T15:00:49.084708+0000","last_unstale":"2024-01-16T15:00:49.090578+0000","last_undegraded":"2024-01-16T15:00:49.090578+0000","last_fullsized":"2024-01-16T15:00:49.090578+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:29.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103202+0000","last_change":"2024-01-16T15:00:49.101622+0000","last_active":"2024-01-16T15:00:49.103202+0000","last_peered":"2024-01-16T15:00:49.103202+0000","last_clean":"2024-01-16T15:00:49.103202+0000","last_became_active":"2024-01-16T15:00:49.101359+0000","last_became_peered":"2024-01-16T15:00:49.101359+0000","last_unstale":"2024-01-16T15:00:49.103202+0000","last_undegraded":"2024-01-16T15:00:49.103202+0000","last_fullsized":"2024-01-16T15:00:49.103202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:55:43.796400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107207+0000","last_change":"2024-01-16T15:00:49.103255+0000","last_active":"2024-01-16T15:00:49.107207+0000","last_peered":"2024-01-16T15:00:49.107207+0000","last_clean":"2024-01-16T15:00:49.107207+0000","last_became_active":"2024-01-16T15:00:49.100199+0000","last_became_peered":"2024-01-16T15:00:49.100199+0000","last_unstale":"2024-01-16T15:00:49.107207+0000","last_undegraded":"2024-01-16T15:00:49.107207+0000","last_fullsized":"2024-01-16T15:00:49.107207+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:53.126824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.083548+0000","last_change":"2024-01-16T15:00:49.079358+0000","last_active":"2024-01-16T15:00:49.083548+0000","last_peered":"2024-01-16T15:00:49.083548+0000","last_clean":"2024-01-16T15:00:49.083548+0000","last_became_active":"2024-01-16T15:00:49.079053+0000","last_became_peered":"2024-01-16T15:00:49.079053+0000","last_unstale":"2024-01-16T15:00:49.083548+0000","last_undegraded":"2024-01-16T15:00:49.083548+0000","last_fullsized":"2024-01-16T15:00:49.083548+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:36.045009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.087559+0000","last_change":"2024-01-16T15:00:49.082056+0000","last_active":"2024-01-16T15:00:49.087559+0000","last_peered":"2024-01-16T15:00:49.087559+0000","last_clean":"2024-01-16T15:00:49.087559+0000","last_became_active":"2024-01-16T15:00:49.081727+0000","last_became_peered":"2024-01-16T15:00:49.081727+0000","last_unstale":"2024-01-16T15:00:49.087559+0000","last_undegraded":"2024-01-16T15:00:49.087559+0000","last_fullsized":"2024-01-16T15:00:49.087559+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:35.490538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098175+0000","last_change":"2024-01-16T15:00:49.093981+0000","last_active":"2024-01-16T15:00:49.098175+0000","last_peered":"2024-01-16T15:00:49.098175+0000","last_clean":"2024-01-16T15:00:49.098175+0000","last_became_active":"2024-01-16T15:00:49.092638+0000","last_became_peered":"2024-01-16T15:00:49.092638+0000","last_unstale":"2024-01-16T15:00:49.098175+0000","last_undegraded":"2024-01-16T15:00:49.098175+0000","last_fullsized":"2024-01-16T15:00:49.098175+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:58.424871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107122+0000","last_change":"2024-01-16T15:00:49.096005+0000","last_active":"2024-01-16T15:00:49.107122+0000","last_peered":"2024-01-16T15:00:49.107122+0000","last_clean":"2024-01-16T15:00:49.107122+0000","last_became_active":"2024-01-16T15:00:49.095614+0000","last_became_peered":"2024-01-16T15:00:49.095614+0000","last_unstale":"2024-01-16T15:00:49.107122+0000","last_undegraded":"2024-01-16T15:00:49.107122+0000","last_fullsized":"2024-01-16T15:00:49.107122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:56.533665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102359+0000","last_change":"2024-01-16T15:00:49.097922+0000","last_active":"2024-01-16T15:00:49.102359+0000","last_peered":"2024-01-16T15:00:49.102359+0000","last_clean":"2024-01-16T15:00:49.102359+0000","last_became_active":"2024-01-16T15:00:49.097354+0000","last_became_peered":"2024-01-16T15:00:49.097354+0000","last_unstale":"2024-01-16T15:00:49.102359+0000","last_undegraded":"2024-01-16T15:00:49.102359+0000","last_fullsized":"2024-01-16T15:00:49.102359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:38.710890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088163+0000","last_change":"2024-01-16T15:00:49.084711+0000","last_active":"2024-01-16T15:00:49.088163+0000","last_peered":"2024-01-16T15:00:49.088163+0000","last_clean":"2024-01-16T15:00:49.088163+0000","last_became_active":"2024-01-16T15:00:49.083154+0000","last_became_peered":"2024-01-16T15:00:49.083154+0000","last_unstale":"2024-01-16T15:00:49.088163+0000","last_undegraded":"2024-01-16T15:00:49.088163+0000","last_fullsized":"2024-01-16T15:00:49.088163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.345260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089815+0000","last_change":"2024-01-16T15:00:49.082118+0000","last_active":"2024-01-16T15:00:49.089815+0000","last_peered":"2024-01-16T15:00:49.089815+0000","last_clean":"2024-01-16T15:00:49.089815+0000","last_became_active":"2024-01-16T15:00:49.079235+0000","last_became_peered":"2024-01-16T15:00:49.079235+0000","last_unstale":"2024-01-16T15:00:49.089815+0000","last_undegraded":"2024-01-16T15:00:49.089815+0000","last_fullsized":"2024-01-16T15:00:49.089815+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:57.830057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101834+0000","last_change":"2024-01-16T15:00:49.098381+0000","last_active":"2024-01-16T15:00:49.101834+0000","last_peered":"2024-01-16T15:00:49.101834+0000","last_clean":"2024-01-16T15:00:49.101834+0000","last_became_active":"2024-01-16T15:00:49.098041+0000","last_became_peered":"2024-01-16T15:00:49.098041+0000","last_unstale":"2024-01-16T15:00:49.101834+0000","last_undegraded":"2024-01-16T15:00:49.101834+0000","last_fullsized":"2024-01-16T15:00:49.101834+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:55.857087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097343+0000","last_change":"2024-01-16T15:00:49.092018+0000","last_active":"2024-01-16T15:00:49.097343+0000","last_peered":"2024-01-16T15:00:49.097343+0000","last_clean":"2024-01-16T15:00:49.097343+0000","last_became_active":"2024-01-16T15:00:49.091452+0000","last_became_peered":"2024-01-16T15:00:49.091452+0000","last_unstale":"2024-01-16T15:00:49.097343+0000","last_undegraded":"2024-01-16T15:00:49.097343+0000","last_fullsized":"2024-01-16T15:00:49.097343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:07.655096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.109703+0000","last_change":"2024-01-16T15:00:49.104805+0000","last_active":"2024-01-16T15:00:49.109703+0000","last_peered":"2024-01-16T15:00:49.109703+0000","last_clean":"2024-01-16T15:00:49.109703+0000","last_became_active":"2024-01-16T15:00:49.102663+0000","last_became_peered":"2024-01-16T15:00:49.102663+0000","last_unstale":"2024-01-16T15:00:49.109703+0000","last_undegraded":"2024-01-16T15:00:49.109703+0000","last_fullsized":"2024-01-16T15:00:49.109703+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:34:56.174799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090584+0000","last_change":"2024-01-16T15:00:49.086522+0000","last_active":"2024-01-16T15:00:49.090584+0000","last_peered":"2024-01-16T15:00:49.090584+0000","last_clean":"2024-01-16T15:00:49.090584+0000","last_became_active":"2024-01-16T15:00:49.085496+0000","last_became_peered":"2024-01-16T15:00:49.085496+0000","last_unstale":"2024-01-16T15:00:49.090584+0000","last_undegraded":"2024-01-16T15:00:49.090584+0000","last_fullsized":"2024-01-16T15:00:49.090584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:07.655096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090649+0000","last_change":"2024-01-16T15:00:49.086006+0000","last_active":"2024-01-16T15:00:49.090649+0000","last_peered":"2024-01-16T15:00:49.090649+0000","last_clean":"2024-01-16T15:00:49.090649+0000","last_became_active":"2024-01-16T15:00:49.085403+0000","last_became_peered":"2024-01-16T15:00:49.085403+0000","last_unstale":"2024-01-16T15:00:49.090649+0000","last_undegraded":"2024-01-16T15:00:49.090649+0000","last_fullsized":"2024-01-16T15:00:49.090649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:36.045009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101161+0000","last_change":"2024-01-16T15:00:49.099476+0000","last_active":"2024-01-16T15:00:49.101161+0000","last_peered":"2024-01-16T15:00:49.101161+0000","last_clean":"2024-01-16T15:00:49.101161+0000","last_became_active":"2024-01-16T15:00:49.097979+0000","last_became_peered":"2024-01-16T15:00:49.097979+0000","last_unstale":"2024-01-16T15:00:49.101161+0000","last_undegraded":"2024-01-16T15:00:49.101161+0000","last_fullsized":"2024-01-16T15:00:49.101161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:19.828148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085671+0000","last_change":"2024-01-16T15:00:49.070562+0000","last_active":"2024-01-16T15:00:49.085671+0000","last_peered":"2024-01-16T15:00:49.085671+0000","last_clean":"2024-01-16T15:00:49.085671+0000","last_became_active":"2024-01-16T15:00:49.069837+0000","last_became_peered":"2024-01-16T15:00:49.069837+0000","last_unstale":"2024-01-16T15:00:49.085671+0000","last_undegraded":"2024-01-16T15:00:49.085671+0000","last_fullsized":"2024-01-16T15:00:49.085671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:05.563630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088664+0000","last_change":"2024-01-16T15:00:49.083678+0000","last_active":"2024-01-16T15:00:49.088664+0000","last_peered":"2024-01-16T15:00:49.088664+0000","last_clean":"2024-01-16T15:00:49.088664+0000","last_became_active":"2024-01-16T15:00:49.083046+0000","last_became_peered":"2024-01-16T15:00:49.083046+0000","last_unstale":"2024-01-16T15:00:49.088664+0000","last_undegraded":"2024-01-16T15:00:49.088664+0000","last_fullsized":"2024-01-16T15:00:49.088664+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:29.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085924+0000","last_change":"2024-01-16T15:00:49.078833+0000","last_active":"2024-01-16T15:00:49.085924+0000","last_peered":"2024-01-16T15:00:49.085924+0000","last_clean":"2024-01-16T15:00:49.085924+0000","last_became_active":"2024-01-16T15:00:49.078560+0000","last_became_peered":"2024-01-16T15:00:49.078560+0000","last_unstale":"2024-01-16T15:00:49.085924+0000","last_undegraded":"2024-01-16T15:00:49.085924+0000","last_fullsized":"2024-01-16T15:00:49.085924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098693+0000","last_change":"2024-01-16T15:00:49.092981+0000","last_active":"2024-01-16T15:00:49.098693+0000","last_peered":"2024-01-16T15:00:49.098693+0000","last_clean":"2024-01-16T15:00:49.098693+0000","last_became_active":"2024-01-16T15:00:49.092371+0000","last_became_peered":"2024-01-16T15:00:49.092371+0000","last_unstale":"2024-01-16T15:00:49.098693+0000","last_undegraded":"2024-01-16T15:00:49.098693+0000","last_fullsized":"2024-01-16T15:00:49.098693+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:16.494984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092621+0000","last_change":"2024-01-16T15:00:49.082303+0000","last_active":"2024-01-16T15:00:49.092621+0000","last_peered":"2024-01-16T15:00:49.092621+0000","last_clean":"2024-01-16T15:00:49.092621+0000","last_became_active":"2024-01-16T15:00:49.081431+0000","last_became_peered":"2024-01-16T15:00:49.081431+0000","last_unstale":"2024-01-16T15:00:49.092621+0000","last_undegraded":"2024-01-16T15:00:49.092621+0000","last_fullsized":"2024-01-16T15:00:49.092621+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:24.205886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089715+0000","last_change":"2024-01-16T15:00:49.081770+0000","last_active":"2024-01-16T15:00:49.089715+0000","last_peered":"2024-01-16T15:00:49.089715+0000","last_clean":"2024-01-16T15:00:49.089715+0000","last_became_active":"2024-01-16T15:00:49.077817+0000","last_became_peered":"2024-01-16T15:00:49.077817+0000","last_unstale":"2024-01-16T15:00:49.089715+0000","last_undegraded":"2024-01-16T15:00:49.089715+0000","last_fullsized":"2024-01-16T15:00:49.089715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:11.730584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098334+0000","last_change":"2024-01-16T15:00:49.095615+0000","last_active":"2024-01-16T15:00:49.098334+0000","last_peered":"2024-01-16T15:00:49.098334+0000","last_clean":"2024-01-16T15:00:49.098334+0000","last_became_active":"2024-01-16T15:00:49.095297+0000","last_became_peered":"2024-01-16T15:00:49.095297+0000","last_unstale":"2024-01-16T15:00:49.098334+0000","last_undegraded":"2024-01-16T15:00:49.098334+0000","last_fullsized":"2024-01-16T15:00:49.098334+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102377+0000","last_change":"2024-01-16T15:00:49.098666+0000","last_active":"2024-01-16T15:00:49.102377+0000","last_peered":"2024-01-16T15:00:49.102377+0000","last_clean":"2024-01-16T15:00:49.102377+0000","last_became_active":"2024-01-16T15:00:49.093930+0000","last_became_peered":"2024-01-16T15:00:49.093930+0000","last_unstale":"2024-01-16T15:00:49.102377+0000","last_undegraded":"2024-01-16T15:00:49.102377+0000","last_fullsized":"2024-01-16T15:00:49.102377+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:48.606264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085690+0000","last_change":"2024-01-16T15:00:49.082783+0000","last_active":"2024-01-16T15:00:49.085690+0000","last_peered":"2024-01-16T15:00:49.085690+0000","last_clean":"2024-01-16T15:00:49.085690+0000","last_became_active":"2024-01-16T15:00:49.082507+0000","last_became_peered":"2024-01-16T15:00:49.082507+0000","last_unstale":"2024-01-16T15:00:49.085690+0000","last_undegraded":"2024-01-16T15:00:49.085690+0000","last_fullsized":"2024-01-16T15:00:49.085690+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:30.876555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107197+0000","last_change":"2024-01-16T15:00:49.102767+0000","last_active":"2024-01-16T15:00:49.107197+0000","last_peered":"2024-01-16T15:00:49.107197+0000","last_clean":"2024-01-16T15:00:49.107197+0000","last_became_active":"2024-01-16T15:00:49.099125+0000","last_became_peered":"2024-01-16T15:00:49.099125+0000","last_unstale":"2024-01-16T15:00:49.107197+0000","last_undegraded":"2024-01-16T15:00:49.107197+0000","last_fullsized":"2024-01-16T15:00:49.107197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:35.305458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094379+0000","last_change":"2024-01-16T15:00:49.086557+0000","last_active":"2024-01-16T15:00:49.094379+0000","last_peered":"2024-01-16T15:00:49.094379+0000","last_clean":"2024-01-16T15:00:49.094379+0000","last_became_active":"2024-01-16T15:00:49.086278+0000","last_became_peered":"2024-01-16T15:00:49.086278+0000","last_unstale":"2024-01-16T15:00:49.094379+0000","last_undegraded":"2024-01-16T15:00:49.094379+0000","last_fullsized":"2024-01-16T15:00:49.094379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097320+0000","last_change":"2024-01-16T15:00:49.091472+0000","last_active":"2024-01-16T15:00:49.097320+0000","last_peered":"2024-01-16T15:00:49.097320+0000","last_clean":"2024-01-16T15:00:49.097320+0000","last_became_active":"2024-01-16T15:00:49.091154+0000","last_became_peered":"2024-01-16T15:00:49.091154+0000","last_unstale":"2024-01-16T15:00:49.097320+0000","last_undegraded":"2024-01-16T15:00:49.097320+0000","last_fullsized":"2024-01-16T15:00:49.097320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:01.860050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082926+0000","last_change":"2024-01-16T15:00:49.077355+0000","last_active":"2024-01-16T15:00:49.082926+0000","last_peered":"2024-01-16T15:00:49.082926+0000","last_clean":"2024-01-16T15:00:49.082926+0000","last_became_active":"2024-01-16T15:00:49.077078+0000","last_became_peered":"2024-01-16T15:00:49.077078+0000","last_unstale":"2024-01-16T15:00:49.082926+0000","last_undegraded":"2024-01-16T15:00:49.082926+0000","last_fullsized":"2024-01-16T15:00:49.082926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:43.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101286+0000","last_change":"2024-01-16T15:00:48.066867+0000","last_active":"2024-01-16T15:00:49.101286+0000","last_peered":"2024-01-16T15:00:49.101286+0000","last_clean":"2024-01-16T15:00:49.101286+0000","last_became_active":"2024-01-16T15:00:48.066540+0000","last_became_peered":"2024-01-16T15:00:48.066540+0000","last_unstale":"2024-01-16T15:00:49.101286+0000","last_undegraded":"2024-01-16T15:00:49.101286+0000","last_fullsized":"2024-01-16T15:00:49.101286+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:43.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094534+0000","last_change":"2024-01-16T15:00:49.089854+0000","last_active":"2024-01-16T15:00:49.094534+0000","last_peered":"2024-01-16T15:00:49.094534+0000","last_clean":"2024-01-16T15:00:49.094534+0000","last_became_active":"2024-01-16T15:00:49.089562+0000","last_became_peered":"2024-01-16T15:00:49.089562+0000","last_unstale":"2024-01-16T15:00:49.094534+0000","last_undegraded":"2024-01-16T15:00:49.094534+0000","last_fullsized":"2024-01-16T15:00:49.094534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:58.433916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082855+0000","last_change":"2024-01-16T15:00:48.068961+0000","last_active":"2024-01-16T15:00:49.082855+0000","last_peered":"2024-01-16T15:00:49.082855+0000","last_clean":"2024-01-16T15:00:49.082855+0000","last_became_active":"2024-01-16T15:00:48.068612+0000","last_became_peered":"2024-01-16T15:00:48.068612+0000","last_unstale":"2024-01-16T15:00:49.082855+0000","last_undegraded":"2024-01-16T15:00:49.082855+0000","last_fullsized":"2024-01-16T15:00:49.082855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:43.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095130+0000","last_change":"2024-01-16T15:00:49.090176+0000","last_active":"2024-01-16T15:00:49.095130+0000","last_peered":"2024-01-16T15:00:49.095130+0000","last_clean":"2024-01-16T15:00:49.095130+0000","last_became_active":"2024-01-16T15:00:49.089727+0000","last_became_peered":"2024-01-16T15:00:49.089727+0000","last_unstale":"2024-01-16T15:00:49.095130+0000","last_undegraded":"2024-01-16T15:00:49.095130+0000","last_fullsized":"2024-01-16T15:00:49.095130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:54.523311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095142+0000","last_change":"2024-01-16T15:00:48.054284+0000","last_active":"2024-01-16T15:00:49.095142+0000","last_peered":"2024-01-16T15:00:49.095142+0000","last_clean":"2024-01-16T15:00:49.095142+0000","last_became_active":"2024-01-16T15:00:48.053923+0000","last_became_peered":"2024-01-16T15:00:48.053923+0000","last_unstale":"2024-01-16T15:00:49.095142+0000","last_undegraded":"2024-01-16T15:00:49.095142+0000","last_fullsized":"2024-01-16T15:00:49.095142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:55.528974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098354+0000","last_change":"2024-01-16T15:00:49.096237+0000","last_active":"2024-01-16T15:00:49.098354+0000","last_peered":"2024-01-16T15:00:49.098354+0000","last_clean":"2024-01-16T15:00:49.098354+0000","last_became_active":"2024-01-16T15:00:49.095903+0000","last_became_peered":"2024-01-16T15:00:49.095903+0000","last_unstale":"2024-01-16T15:00:49.098354+0000","last_undegraded":"2024-01-16T15:00:49.098354+0000","last_fullsized":"2024-01-16T15:00:49.098354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093899+0000","last_change":"2024-01-16T15:00:48.055207+0000","last_active":"2024-01-16T15:00:49.093899+0000","last_peered":"2024-01-16T15:00:49.093899+0000","last_clean":"2024-01-16T15:00:49.093899+0000","last_became_active":"2024-01-16T15:00:48.054824+0000","last_became_peered":"2024-01-16T15:00:48.054824+0000","last_unstale":"2024-01-16T15:00:49.093899+0000","last_undegraded":"2024-01-16T15:00:49.093899+0000","last_fullsized":"2024-01-16T15:00:49.093899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:53.518620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.109703+0000","last_change":"2024-01-16T15:00:49.102988+0000","last_active":"2024-01-16T15:00:49.109703+0000","last_peered":"2024-01-16T15:00:49.109703+0000","last_clean":"2024-01-16T15:00:49.109703+0000","last_became_active":"2024-01-16T15:00:49.100514+0000","last_became_peered":"2024-01-16T15:00:49.100514+0000","last_unstale":"2024-01-16T15:00:49.109703+0000","last_undegraded":"2024-01-16T15:00:49.109703+0000","last_fullsized":"2024-01-16T15:00:49.109703+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:46.867540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096783+0000","last_change":"2024-01-16T15:00:48.062992+0000","last_active":"2024-01-16T15:00:49.096783+0000","last_peered":"2024-01-16T15:00:49.096783+0000","last_clean":"2024-01-16T15:00:49.096783+0000","last_became_active":"2024-01-16T15:00:48.062375+0000","last_became_peered":"2024-01-16T15:00:48.062375+0000","last_unstale":"2024-01-16T15:00:49.096783+0000","last_undegraded":"2024-01-16T15:00:49.096783+0000","last_fullsized":"2024-01-16T15:00:49.096783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:57.429224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098786+0000","last_change":"2024-01-16T15:00:49.093002+0000","last_active":"2024-01-16T15:00:49.098786+0000","last_peered":"2024-01-16T15:00:49.098786+0000","last_clean":"2024-01-16T15:00:49.098786+0000","last_became_active":"2024-01-16T15:00:49.092636+0000","last_became_peered":"2024-01-16T15:00:49.092636+0000","last_unstale":"2024-01-16T15:00:49.098786+0000","last_undegraded":"2024-01-16T15:00:49.098786+0000","last_fullsized":"2024-01-16T15:00:49.098786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:58.424871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096303+0000","last_change":"2024-01-16T15:00:48.064120+0000","last_active":"2024-01-16T15:00:49.096303+0000","last_peered":"2024-01-16T15:00:49.096303+0000","last_clean":"2024-01-16T15:00:49.096303+0000","last_became_active":"2024-01-16T15:00:48.063828+0000","last_became_peered":"2024-01-16T15:00:48.063828+0000","last_unstale":"2024-01-16T15:00:49.096303+0000","last_undegraded":"2024-01-16T15:00:49.096303+0000","last_fullsized":"2024-01-16T15:00:49.096303+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:28.025787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.091025+0000","last_change":"2024-01-16T15:00:49.086126+0000","last_active":"2024-01-16T15:00:49.091025+0000","last_peered":"2024-01-16T15:00:49.091025+0000","last_clean":"2024-01-16T15:00:49.091025+0000","last_became_active":"2024-01-16T15:00:49.085743+0000","last_became_peered":"2024-01-16T15:00:49.085743+0000","last_unstale":"2024-01-16T15:00:49.091025+0000","last_undegraded":"2024-01-16T15:00:49.091025+0000","last_fullsized":"2024-01-16T15:00:49.091025+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:50.886550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.066285+0000","last_change":"2024-01-16T15:00:48.051772+0000","last_active":"2024-01-16T15:00:49.066285+0000","last_peered":"2024-01-16T15:00:49.066285+0000","last_clean":"2024-01-16T15:00:49.066285+0000","last_became_active":"2024-01-16T15:00:48.051466+0000","last_became_peered":"2024-01-16T15:00:48.051466+0000","last_unstale":"2024-01-16T15:00:49.066285+0000","last_undegraded":"2024-01-16T15:00:49.066285+0000","last_fullsized":"2024-01-16T15:00:49.066285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:34.300767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102464+0000","last_change":"2024-01-16T15:00:49.100036+0000","last_active":"2024-01-16T15:00:49.102464+0000","last_peered":"2024-01-16T15:00:49.102464+0000","last_clean":"2024-01-16T15:00:49.102464+0000","last_became_active":"2024-01-16T15:00:49.099324+0000","last_became_peered":"2024-01-16T15:00:49.099324+0000","last_unstale":"2024-01-16T15:00:49.102464+0000","last_undegraded":"2024-01-16T15:00:49.102464+0000","last_fullsized":"2024-01-16T15:00:49.102464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:44.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063163+0000","last_change":"2024-01-16T15:00:48.065255+0000","last_active":"2024-01-16T15:00:49.063163+0000","last_peered":"2024-01-16T15:00:49.063163+0000","last_clean":"2024-01-16T15:00:49.063163+0000","last_became_active":"2024-01-16T15:00:48.064866+0000","last_became_peered":"2024-01-16T15:00:48.064866+0000","last_unstale":"2024-01-16T15:00:49.063163+0000","last_undegraded":"2024-01-16T15:00:49.063163+0000","last_fullsized":"2024-01-16T15:00:49.063163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:56.195182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.099676+0000","last_change":"2024-01-16T15:00:49.096130+0000","last_active":"2024-01-16T15:00:49.099676+0000","last_peered":"2024-01-16T15:00:49.099676+0000","last_clean":"2024-01-16T15:00:49.099676+0000","last_became_active":"2024-01-16T15:00:49.093909+0000","last_became_peered":"2024-01-16T15:00:49.093909+0000","last_unstale":"2024-01-16T15:00:49.099676+0000","last_undegraded":"2024-01-16T15:00:49.099676+0000","last_fullsized":"2024-01-16T15:00:49.099676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:20.413171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.068433+0000","last_change":"2024-01-16T15:00:48.065291+0000","last_active":"2024-01-16T15:00:49.068433+0000","last_peered":"2024-01-16T15:00:49.068433+0000","last_clean":"2024-01-16T15:00:49.068433+0000","last_became_active":"2024-01-16T15:00:48.064941+0000","last_became_peered":"2024-01-16T15:00:48.064941+0000","last_unstale":"2024-01-16T15:00:49.068433+0000","last_undegraded":"2024-01-16T15:00:49.068433+0000","last_fullsized":"2024-01-16T15:00:49.068433+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:52.122133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.091184+0000","last_change":"2024-01-16T15:00:49.087584+0000","last_active":"2024-01-16T15:00:49.091184+0000","last_peered":"2024-01-16T15:00:49.091184+0000","last_clean":"2024-01-16T15:00:49.091184+0000","last_became_active":"2024-01-16T15:00:49.087177+0000","last_became_peered":"2024-01-16T15:00:49.087177+0000","last_unstale":"2024-01-16T15:00:49.091184+0000","last_undegraded":"2024-01-16T15:00:49.091184+0000","last_fullsized":"2024-01-16T15:00:49.091184+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:34:56.174799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062092+0000","last_change":"2024-01-16T15:00:48.056099+0000","last_active":"2024-01-16T15:00:49.062092+0000","last_peered":"2024-01-16T15:00:49.062092+0000","last_clean":"2024-01-16T15:00:49.062092+0000","last_became_active":"2024-01-16T15:00:48.055799+0000","last_became_peered":"2024-01-16T15:00:48.055799+0000","last_unstale":"2024-01-16T15:00:49.062092+0000","last_undegraded":"2024-01-16T15:00:49.062092+0000","last_fullsized":"2024-01-16T15:00:49.062092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:54.852396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089725+0000","last_change":"2024-01-16T15:00:49.082076+0000","last_active":"2024-01-16T15:00:49.089725+0000","last_peered":"2024-01-16T15:00:49.089725+0000","last_clean":"2024-01-16T15:00:49.089725+0000","last_became_active":"2024-01-16T15:00:49.081756+0000","last_became_peered":"2024-01-16T15:00:49.081756+0000","last_unstale":"2024-01-16T15:00:49.089725+0000","last_undegraded":"2024-01-16T15:00:49.089725+0000","last_fullsized":"2024-01-16T15:00:49.089725+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:54:50.522730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100728+0000","last_change":"2024-01-16T15:00:48.066297+0000","last_active":"2024-01-16T15:00:49.100728+0000","last_peered":"2024-01-16T15:00:49.100728+0000","last_clean":"2024-01-16T15:00:49.100728+0000","last_became_active":"2024-01-16T15:00:48.065970+0000","last_became_peered":"2024-01-16T15:00:48.065970+0000","last_unstale":"2024-01-16T15:00:49.100728+0000","last_undegraded":"2024-01-16T15:00:49.100728+0000","last_fullsized":"2024-01-16T15:00:49.100728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098915+0000","last_change":"2024-01-16T15:00:49.096671+0000","last_active":"2024-01-16T15:00:49.098915+0000","last_peered":"2024-01-16T15:00:49.098915+0000","last_clean":"2024-01-16T15:00:49.098915+0000","last_became_active":"2024-01-16T15:00:49.095271+0000","last_became_peered":"2024-01-16T15:00:49.095271+0000","last_unstale":"2024-01-16T15:00:49.098915+0000","last_undegraded":"2024-01-16T15:00:49.098915+0000","last_fullsized":"2024-01-16T15:00:49.098915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:03:58.554748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103528+0000","last_change":"2024-01-16T15:00:48.055829+0000","last_active":"2024-01-16T15:00:49.103528+0000","last_peered":"2024-01-16T15:00:49.103528+0000","last_clean":"2024-01-16T15:00:49.103528+0000","last_became_active":"2024-01-16T15:00:48.054343+0000","last_became_peered":"2024-01-16T15:00:48.054343+0000","last_unstale":"2024-01-16T15:00:49.103528+0000","last_undegraded":"2024-01-16T15:00:49.103528+0000","last_fullsized":"2024-01-16T15:00:49.103528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:11.601859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098563+0000","last_change":"2024-01-16T15:00:49.094720+0000","last_active":"2024-01-16T15:00:49.098563+0000","last_peered":"2024-01-16T15:00:49.098563+0000","last_clean":"2024-01-16T15:00:49.098563+0000","last_became_active":"2024-01-16T15:00:49.094448+0000","last_became_peered":"2024-01-16T15:00:49.094448+0000","last_unstale":"2024-01-16T15:00:49.098563+0000","last_undegraded":"2024-01-16T15:00:49.098563+0000","last_fullsized":"2024-01-16T15:00:49.098563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:16.352542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097890+0000","last_change":"2024-01-16T15:00:48.066210+0000","last_active":"2024-01-16T15:00:49.097890+0000","last_peered":"2024-01-16T15:00:49.097890+0000","last_clean":"2024-01-16T15:00:49.097890+0000","last_became_active":"2024-01-16T15:00:48.065896+0000","last_became_peered":"2024-01-16T15:00:48.065896+0000","last_unstale":"2024-01-16T15:00:49.097890+0000","last_undegraded":"2024-01-16T15:00:49.097890+0000","last_fullsized":"2024-01-16T15:00:49.097890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:16.944248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089921+0000","last_change":"2024-01-16T15:00:49.082844+0000","last_active":"2024-01-16T15:00:49.089921+0000","last_peered":"2024-01-16T15:00:49.089921+0000","last_clean":"2024-01-16T15:00:49.089921+0000","last_became_active":"2024-01-16T15:00:49.082545+0000","last_became_peered":"2024-01-16T15:00:49.082545+0000","last_unstale":"2024-01-16T15:00:49.089921+0000","last_undegraded":"2024-01-16T15:00:49.089921+0000","last_fullsized":"2024-01-16T15:00:49.089921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:54.713937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.064530+0000","last_change":"2024-01-16T15:00:48.066013+0000","last_active":"2024-01-16T15:00:49.064530+0000","last_peered":"2024-01-16T15:00:49.064530+0000","last_clean":"2024-01-16T15:00:49.064530+0000","last_became_active":"2024-01-16T15:00:48.065587+0000","last_became_peered":"2024-01-16T15:00:48.065587+0000","last_unstale":"2024-01-16T15:00:49.064530+0000","last_undegraded":"2024-01-16T15:00:49.064530+0000","last_fullsized":"2024-01-16T15:00:49.064530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:42.485766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107090+0000","last_change":"2024-01-16T15:00:49.100919+0000","last_active":"2024-01-16T15:00:49.107090+0000","last_peered":"2024-01-16T15:00:49.107090+0000","last_clean":"2024-01-16T15:00:49.107090+0000","last_became_active":"2024-01-16T15:00:49.100203+0000","last_became_peered":"2024-01-16T15:00:49.100203+0000","last_unstale":"2024-01-16T15:00:49.107090+0000","last_undegraded":"2024-01-16T15:00:49.107090+0000","last_fullsized":"2024-01-16T15:00:49.107090+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:05.403999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.066423+0000","last_change":"2024-01-16T15:00:48.065263+0000","last_active":"2024-01-16T15:00:49.066423+0000","last_peered":"2024-01-16T15:00:49.066423+0000","last_clean":"2024-01-16T15:00:49.066423+0000","last_became_active":"2024-01-16T15:00:48.064900+0000","last_became_peered":"2024-01-16T15:00:48.064900+0000","last_unstale":"2024-01-16T15:00:49.066423+0000","last_undegraded":"2024-01-16T15:00:49.066423+0000","last_fullsized":"2024-01-16T15:00:49.066423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:45.862848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094359+0000","last_change":"2024-01-16T15:00:49.086431+0000","last_active":"2024-01-16T15:00:49.094359+0000","last_peered":"2024-01-16T15:00:49.094359+0000","last_clean":"2024-01-16T15:00:49.094359+0000","last_became_active":"2024-01-16T15:00:49.086050+0000","last_became_peered":"2024-01-16T15:00:49.086050+0000","last_unstale":"2024-01-16T15:00:49.094359+0000","last_undegraded":"2024-01-16T15:00:49.094359+0000","last_fullsized":"2024-01-16T15:00:49.094359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100312+0000","last_change":"2024-01-16T15:00:48.053315+0000","last_active":"2024-01-16T15:00:49.100312+0000","last_peered":"2024-01-16T15:00:49.100312+0000","last_clean":"2024-01-16T15:00:49.100312+0000","last_became_active":"2024-01-16T15:00:48.052685+0000","last_became_peered":"2024-01-16T15:00:48.052685+0000","last_unstale":"2024-01-16T15:00:49.100312+0000","last_undegraded":"2024-01-16T15:00:49.100312+0000","last_fullsized":"2024-01-16T15:00:49.100312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:00.855358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098537+0000","last_change":"2024-01-16T15:00:49.096322+0000","last_active":"2024-01-16T15:00:49.098537+0000","last_peered":"2024-01-16T15:00:49.098537+0000","last_clean":"2024-01-16T15:00:49.098537+0000","last_became_active":"2024-01-16T15:00:49.095731+0000","last_became_peered":"2024-01-16T15:00:49.095731+0000","last_unstale":"2024-01-16T15:00:49.098537+0000","last_undegraded":"2024-01-16T15:00:49.098537+0000","last_fullsized":"2024-01-16T15:00:49.098537+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:49.790562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098847+0000","last_change":"2024-01-16T15:00:48.061110+0000","last_active":"2024-01-16T15:00:49.098847+0000","last_peered":"2024-01-16T15:00:49.098847+0000","last_clean":"2024-01-16T15:00:49.098847+0000","last_became_active":"2024-01-16T15:00:48.060801+0000","last_became_peered":"2024-01-16T15:00:48.060801+0000","last_unstale":"2024-01-16T15:00:49.098847+0000","last_undegraded":"2024-01-16T15:00:49.098847+0000","last_fullsized":"2024-01-16T15:00:49.098847+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:31.018746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101924+0000","last_change":"2024-01-16T15:00:49.098975+0000","last_active":"2024-01-16T15:00:49.101924+0000","last_peered":"2024-01-16T15:00:49.101924+0000","last_clean":"2024-01-16T15:00:49.101924+0000","last_became_active":"2024-01-16T15:00:49.098629+0000","last_became_peered":"2024-01-16T15:00:49.098629+0000","last_unstale":"2024-01-16T15:00:49.101924+0000","last_undegraded":"2024-01-16T15:00:49.101924+0000","last_fullsized":"2024-01-16T15:00:49.101924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:05.034889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096129+0000","last_change":"2024-01-16T15:00:48.065651+0000","last_active":"2024-01-16T15:00:49.096129+0000","last_peered":"2024-01-16T15:00:49.096129+0000","last_clean":"2024-01-16T15:00:49.096129+0000","last_became_active":"2024-01-16T15:00:48.065257+0000","last_became_peered":"2024-01-16T15:00:48.065257+0000","last_unstale":"2024-01-16T15:00:49.096129+0000","last_undegraded":"2024-01-16T15:00:49.096129+0000","last_fullsized":"2024-01-16T15:00:49.096129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:44.613107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093241+0000","last_change":"2024-01-16T15:00:49.084273+0000","last_active":"2024-01-16T15:00:49.093241+0000","last_peered":"2024-01-16T15:00:49.093241+0000","last_clean":"2024-01-16T15:00:49.093241+0000","last_became_active":"2024-01-16T15:00:49.083999+0000","last_became_peered":"2024-01-16T15:00:49.083999+0000","last_unstale":"2024-01-16T15:00:49.093241+0000","last_undegraded":"2024-01-16T15:00:49.093241+0000","last_fullsized":"2024-01-16T15:00:49.093241+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:55.857087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095383+0000","last_change":"2024-01-16T15:00:48.062649+0000","last_active":"2024-01-16T15:00:49.095383+0000","last_peered":"2024-01-16T15:00:49.095383+0000","last_clean":"2024-01-16T15:00:49.095383+0000","last_became_active":"2024-01-16T15:00:48.062318+0000","last_became_peered":"2024-01-16T15:00:48.062318+0000","last_unstale":"2024-01-16T15:00:49.095383+0000","last_undegraded":"2024-01-16T15:00:49.095383+0000","last_fullsized":"2024-01-16T15:00:49.095383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:10.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.078534+0000","last_change":"2024-01-16T15:00:49.075198+0000","last_active":"2024-01-16T15:00:49.078534+0000","last_peered":"2024-01-16T15:00:49.078534+0000","last_clean":"2024-01-16T15:00:49.078534+0000","last_became_active":"2024-01-16T15:00:49.074924+0000","last_became_peered":"2024-01-16T15:00:49.074924+0000","last_unstale":"2024-01-16T15:00:49.078534+0000","last_undegraded":"2024-01-16T15:00:49.078534+0000","last_fullsized":"2024-01-16T15:00:49.078534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:30.876555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062965+0000","last_change":"2024-01-16T15:00:48.067683+0000","last_active":"2024-01-16T15:00:49.062965+0000","last_peered":"2024-01-16T15:00:49.062965+0000","last_clean":"2024-01-16T15:00:49.062965+0000","last_became_active":"2024-01-16T15:00:48.067408+0000","last_became_peered":"2024-01-16T15:00:48.067408+0000","last_unstale":"2024-01-16T15:00:49.062965+0000","last_undegraded":"2024-01-16T15:00:49.062965+0000","last_fullsized":"2024-01-16T15:00:49.062965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:49.258940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088131+0000","last_change":"2024-01-16T15:00:49.083664+0000","last_active":"2024-01-16T15:00:49.088131+0000","last_peered":"2024-01-16T15:00:49.088131+0000","last_clean":"2024-01-16T15:00:49.088131+0000","last_became_active":"2024-01-16T15:00:49.083025+0000","last_became_peered":"2024-01-16T15:00:49.083025+0000","last_unstale":"2024-01-16T15:00:49.088131+0000","last_undegraded":"2024-01-16T15:00:49.088131+0000","last_fullsized":"2024-01-16T15:00:49.088131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:14.552571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082699+0000","last_change":"2024-01-16T15:00:48.051159+0000","last_active":"2024-01-16T15:00:49.082699+0000","last_peered":"2024-01-16T15:00:49.082699+0000","last_clean":"2024-01-16T15:00:49.082699+0000","last_became_active":"2024-01-16T15:00:48.050865+0000","last_became_peered":"2024-01-16T15:00:48.050865+0000","last_unstale":"2024-01-16T15:00:49.082699+0000","last_undegraded":"2024-01-16T15:00:49.082699+0000","last_fullsized":"2024-01-16T15:00:49.082699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:15.347850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.084463+0000","last_change":"2024-01-16T15:00:49.078806+0000","last_active":"2024-01-16T15:00:49.084463+0000","last_peered":"2024-01-16T15:00:49.084463+0000","last_clean":"2024-01-16T15:00:49.084463+0000","last_became_active":"2024-01-16T15:00:49.078517+0000","last_became_peered":"2024-01-16T15:00:49.078517+0000","last_unstale":"2024-01-16T15:00:49.084463+0000","last_undegraded":"2024-01-16T15:00:49.084463+0000","last_fullsized":"2024-01-16T15:00:49.084463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:30:02.526244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.073827+0000","last_change":"2024-01-16T15:00:48.063613+0000","last_active":"2024-01-16T15:00:49.073827+0000","last_peered":"2024-01-16T15:00:49.073827+0000","last_clean":"2024-01-16T15:00:49.073827+0000","last_became_active":"2024-01-16T15:00:48.062867+0000","last_became_peered":"2024-01-16T15:00:48.062867+0000","last_unstale":"2024-01-16T15:00:49.073827+0000","last_undegraded":"2024-01-16T15:00:49.073827+0000","last_fullsized":"2024-01-16T15:00:49.073827+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:42.485766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074922+0000","last_change":"2024-01-16T15:00:49.072810+0000","last_active":"2024-01-16T15:00:49.074922+0000","last_peered":"2024-01-16T15:00:49.074922+0000","last_clean":"2024-01-16T15:00:49.074922+0000","last_became_active":"2024-01-16T15:00:49.072509+0000","last_became_peered":"2024-01-16T15:00:49.072509+0000","last_unstale":"2024-01-16T15:00:49.074922+0000","last_undegraded":"2024-01-16T15:00:49.074922+0000","last_fullsized":"2024-01-16T15:00:49.074922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:38.202653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063637+0000","last_change":"2024-01-16T15:00:48.061152+0000","last_active":"2024-01-16T15:00:49.063637+0000","last_peered":"2024-01-16T15:00:49.063637+0000","last_clean":"2024-01-16T15:00:49.063637+0000","last_became_active":"2024-01-16T15:00:48.060236+0000","last_became_peered":"2024-01-16T15:00:48.060236+0000","last_unstale":"2024-01-16T15:00:49.063637+0000","last_undegraded":"2024-01-16T15:00:49.063637+0000","last_fullsized":"2024-01-16T15:00:49.063637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:29.850713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.105181+0000","last_change":"2024-01-16T15:00:49.100265+0000","last_active":"2024-01-16T15:00:49.105181+0000","last_peered":"2024-01-16T15:00:49.105181+0000","last_clean":"2024-01-16T15:00:49.105181+0000","last_became_active":"2024-01-16T15:00:49.099997+0000","last_became_peered":"2024-01-16T15:00:49.099997+0000","last_unstale":"2024-01-16T15:00:49.105181+0000","last_undegraded":"2024-01-16T15:00:49.105181+0000","last_fullsized":"2024-01-16T15:00:49.105181+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:12.606550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074574+0000","last_change":"2024-01-16T15:00:48.067724+0000","last_active":"2024-01-16T15:00:49.074574+0000","last_peered":"2024-01-16T15:00:49.074574+0000","last_clean":"2024-01-16T15:00:49.074574+0000","last_became_active":"2024-01-16T15:00:48.067080+0000","last_became_peered":"2024-01-16T15:00:48.067080+0000","last_unstale":"2024-01-16T15:00:49.074574+0000","last_undegraded":"2024-01-16T15:00:49.074574+0000","last_fullsized":"2024-01-16T15:00:49.074574+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:53.586410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100089+0000","last_change":"2024-01-16T15:00:49.095669+0000","last_active":"2024-01-16T15:00:49.100089+0000","last_peered":"2024-01-16T15:00:49.100089+0000","last_clean":"2024-01-16T15:00:49.100089+0000","last_became_active":"2024-01-16T15:00:49.095408+0000","last_became_peered":"2024-01-16T15:00:49.095408+0000","last_unstale":"2024-01-16T15:00:49.100089+0000","last_undegraded":"2024-01-16T15:00:49.100089+0000","last_fullsized":"2024-01-16T15:00:49.100089+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:25.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.099262+0000","last_change":"2024-01-16T15:00:48.066203+0000","last_active":"2024-01-16T15:00:49.099262+0000","last_peered":"2024-01-16T15:00:49.099262+0000","last_clean":"2024-01-16T15:00:49.099262+0000","last_became_active":"2024-01-16T15:00:48.065878+0000","last_became_peered":"2024-01-16T15:00:48.065878+0000","last_unstale":"2024-01-16T15:00:49.099262+0000","last_undegraded":"2024-01-16T15:00:49.099262+0000","last_fullsized":"2024-01-16T15:00:49.099262+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:47.601573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102550+0000","last_change":"2024-01-16T15:00:49.100710+0000","last_active":"2024-01-16T15:00:49.102550+0000","last_peered":"2024-01-16T15:00:49.102550+0000","last_clean":"2024-01-16T15:00:49.102550+0000","last_became_active":"2024-01-16T15:00:49.099074+0000","last_became_peered":"2024-01-16T15:00:49.099074+0000","last_unstale":"2024-01-16T15:00:49.102550+0000","last_undegraded":"2024-01-16T15:00:49.102550+0000","last_fullsized":"2024-01-16T15:00:49.102550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:24.205886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062227+0000","last_change":"2024-01-16T15:00:48.050860+0000","last_active":"2024-01-16T15:00:49.062227+0000","last_peered":"2024-01-16T15:00:49.062227+0000","last_clean":"2024-01-16T15:00:49.062227+0000","last_became_active":"2024-01-16T15:00:48.050522+0000","last_became_peered":"2024-01-16T15:00:48.050522+0000","last_unstale":"2024-01-16T15:00:49.062227+0000","last_undegraded":"2024-01-16T15:00:49.062227+0000","last_fullsized":"2024-01-16T15:00:49.062227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:15.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090494+0000","last_change":"2024-01-16T15:00:49.085468+0000","last_active":"2024-01-16T15:00:49.090494+0000","last_peered":"2024-01-16T15:00:49.090494+0000","last_clean":"2024-01-16T15:00:49.090494+0000","last_became_active":"2024-01-16T15:00:49.085162+0000","last_became_peered":"2024-01-16T15:00:49.085162+0000","last_unstale":"2024-01-16T15:00:49.090494+0000","last_undegraded":"2024-01-16T15:00:49.090494+0000","last_fullsized":"2024-01-16T15:00:49.090494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.068653+0000","last_change":"2024-01-16T15:00:48.052504+0000","last_active":"2024-01-16T15:00:49.068653+0000","last_peered":"2024-01-16T15:00:49.068653+0000","last_clean":"2024-01-16T15:00:49.068653+0000","last_became_active":"2024-01-16T15:00:48.050557+0000","last_became_peered":"2024-01-16T15:00:48.050557+0000","last_unstale":"2024-01-16T15:00:49.068653+0000","last_undegraded":"2024-01-16T15:00:49.068653+0000","last_fullsized":"2024-01-16T15:00:49.068653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:53.586410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094726+0000","last_change":"2024-01-16T15:00:49.088379+0000","last_active":"2024-01-16T15:00:49.094726+0000","last_peered":"2024-01-16T15:00:49.094726+0000","last_clean":"2024-01-16T15:00:49.094726+0000","last_became_active":"2024-01-16T15:00:49.088098+0000","last_became_peered":"2024-01-16T15:00:49.088098+0000","last_unstale":"2024-01-16T15:00:49.094726+0000","last_undegraded":"2024-01-16T15:00:49.094726+0000","last_fullsized":"2024-01-16T15:00:49.094726+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:16.352542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.059060+0000","last_change":"2024-01-16T15:00:48.057344+0000","last_active":"2024-01-16T15:00:49.059060+0000","last_peered":"2024-01-16T15:00:49.059060+0000","last_clean":"2024-01-16T15:00:49.059060+0000","last_became_active":"2024-01-16T15:00:48.057066+0000","last_became_peered":"2024-01-16T15:00:48.057066+0000","last_unstale":"2024-01-16T15:00:49.059060+0000","last_undegraded":"2024-01-16T15:00:49.059060+0000","last_fullsized":"2024-01-16T15:00:49.059060+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092977+0000","last_change":"2024-01-16T15:00:49.091351+0000","last_active":"2024-01-16T15:00:49.092977+0000","last_peered":"2024-01-16T15:00:49.092977+0000","last_clean":"2024-01-16T15:00:49.092977+0000","last_became_active":"2024-01-16T15:00:49.091066+0000","last_became_peered":"2024-01-16T15:00:49.091066+0000","last_unstale":"2024-01-16T15:00:49.092977+0000","last_undegraded":"2024-01-16T15:00:49.092977+0000","last_fullsized":"2024-01-16T15:00:49.092977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:47.209930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.064673+0000","last_change":"2024-01-16T15:00:48.061362+0000","last_active":"2024-01-16T15:00:49.064673+0000","last_peered":"2024-01-16T15:00:49.064673+0000","last_clean":"2024-01-16T15:00:49.064673+0000","last_became_active":"2024-01-16T15:00:48.061080+0000","last_became_peered":"2024-01-16T15:00:48.061080+0000","last_unstale":"2024-01-16T15:00:49.064673+0000","last_undegraded":"2024-01-16T15:00:49.064673+0000","last_fullsized":"2024-01-16T15:00:49.064673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092686+0000","last_change":"2024-01-16T15:00:49.088755+0000","last_active":"2024-01-16T15:00:49.092686+0000","last_peered":"2024-01-16T15:00:49.092686+0000","last_clean":"2024-01-16T15:00:49.092686+0000","last_became_active":"2024-01-16T15:00:49.088485+0000","last_became_peered":"2024-01-16T15:00:49.088485+0000","last_unstale":"2024-01-16T15:00:49.092686+0000","last_undegraded":"2024-01-16T15:00:49.092686+0000","last_fullsized":"2024-01-16T15:00:49.092686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:00.145351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063979+0000","last_change":"2024-01-16T15:00:48.065908+0000","last_active":"2024-01-16T15:00:49.063979+0000","last_peered":"2024-01-16T15:00:49.063979+0000","last_clean":"2024-01-16T15:00:49.063979+0000","last_became_active":"2024-01-16T15:00:48.065591+0000","last_became_peered":"2024-01-16T15:00:48.065591+0000","last_unstale":"2024-01-16T15:00:49.063979+0000","last_undegraded":"2024-01-16T15:00:49.063979+0000","last_fullsized":"2024-01-16T15:00:49.063979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:46.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085353+0000","last_change":"2024-01-16T15:00:49.075674+0000","last_active":"2024-01-16T15:00:49.085353+0000","last_peered":"2024-01-16T15:00:49.085353+0000","last_clean":"2024-01-16T15:00:49.085353+0000","last_became_active":"2024-01-16T15:00:49.074974+0000","last_became_peered":"2024-01-16T15:00:49.074974+0000","last_unstale":"2024-01-16T15:00:49.085353+0000","last_undegraded":"2024-01-16T15:00:49.085353+0000","last_fullsized":"2024-01-16T15:00:49.085353+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.345260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.065341+0000","last_change":"2024-01-16T15:00:48.063259+0000","last_active":"2024-01-16T15:00:49.065341+0000","last_peered":"2024-01-16T15:00:49.065341+0000","last_clean":"2024-01-16T15:00:49.065341+0000","last_became_active":"2024-01-16T15:00:48.062718+0000","last_became_peered":"2024-01-16T15:00:48.062718+0000","last_unstale":"2024-01-16T15:00:49.065341+0000","last_undegraded":"2024-01-16T15:00:49.065341+0000","last_fullsized":"2024-01-16T15:00:49.065341+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:04.399308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093379+0000","last_change":"2024-01-16T15:00:49.087595+0000","last_active":"2024-01-16T15:00:49.093379+0000","last_peered":"2024-01-16T15:00:49.093379+0000","last_clean":"2024-01-16T15:00:49.093379+0000","last_became_active":"2024-01-16T15:00:49.086730+0000","last_became_peered":"2024-01-16T15:00:49.086730+0000","last_unstale":"2024-01-16T15:00:49.093379+0000","last_undegraded":"2024-01-16T15:00:49.093379+0000","last_fullsized":"2024-01-16T15:00:49.093379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:46:47.690663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074757+0000","last_change":"2024-01-16T15:00:48.063925+0000","last_active":"2024-01-16T15:00:49.074757+0000","last_peered":"2024-01-16T15:00:49.074757+0000","last_clean":"2024-01-16T15:00:49.074757+0000","last_became_active":"2024-01-16T15:00:48.063431+0000","last_became_peered":"2024-01-16T15:00:48.063431+0000","last_unstale":"2024-01-16T15:00:49.074757+0000","last_undegraded":"2024-01-16T15:00:49.074757+0000","last_fullsized":"2024-01-16T15:00:49.074757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T15:00:48.061376+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:00:48.061376+0000","last_peered":"2024-01-16T15:00:48.061376+0000","last_clean":"2024-01-16T15:00:48.061376+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:00:48.061376+0000","last_undegraded":"2024-01-16T15:00:48.061376+0000","last_fullsized":"2024-01-16T15:00:48.061376+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":738634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.216},{"interface":"front","average":{"1min":1.204,"5min":1.204,"15min":1.204},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.2490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.7400000000000002,"5min":3.7400000000000002,"15min":3.7400000000000002},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":0.96999999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.536},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.353}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":3.8450000000000002,"5min":3.8450000000000002,"15min":3.8450000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.431}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":1.04},{"interface":"front","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":1.292}]},{"osd":5,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"last":1.619},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":1.573}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.2569999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.84599999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.7480000000000002,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.1910000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":0.92700000000000005}]},{"osd":5,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.464},{"interface":"front","average":{"1min":1.607,"5min":1.607,"15min":1.607},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.4139999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.552}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":1.204},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":1.623},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.379}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.421},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":1.343}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":3.2909999999999999,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":1.274},{"interface":"front","average":{"1min":1.47,"5min":1.47,"15min":1.47},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.514}]},{"osd":7,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.474},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.3610000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":1.671}]}]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98599999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.429},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.296}]},{"osd":2,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":1.349}]},{"osd":3,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.494}]},{"osd":5,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.8650000000000002,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.591},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.649}]},{"osd":6,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.401},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.5289999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":1.198,"5min":1.198,"15min":1.198},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.607}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.3240000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.04},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.141},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.605,"5min":2.605,"15min":2.605},"last":1.2849999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.423},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.591}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":385024,"data_stored":148246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":0.96099999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":4.6769999999999996,"5min":4.6769999999999996,"15min":4.6769999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.637}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.6120000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.429},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.4450000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.1599999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.224},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.577}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":149810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.9820000000000002,"5min":2.9820000000000002,"15min":2.9820000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":3.649,"5min":3.649,"15min":3.649},"last":0.84899999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":1.204},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":3.5619999999999998,"5min":3.5619999999999998,"15min":3.5619999999999998},"last":1.3220000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":3.722,"5min":3.722,"15min":3.722},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":1.7470000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.579}]},{"osd":6,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":3.4750000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":3.6840000000000002,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":1.7809999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.401},{"interface":"front","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.5390000000000001,"5min":3.5390000000000001,"15min":3.5390000000000001},"last":1.268}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":738544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.625}]},{"osd":2,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.77200000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.204}]},{"osd":5,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.72199999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.0589999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":1.014,"5min":1.014,"15min":1.014},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.123},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.03}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:53.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:53.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.830+0000 7fb501400640 1 Processor -- start 2024-01-16T15:00:53.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.831+0000 7fb501400640 1 -- start start 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb501400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 0x7fb4fc0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb501400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0cf0c0 con 0x7fb4fc0b0640 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb501400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0d1e40 con 0x7fb4fc15d5b0 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb501400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0b01b0 con 0x7fb4fc0d14a0 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb500a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb4fc0b0640 0x7fb4fc0b0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33234/0 (socket says 172.21.15.43:33234) 2024-01-16T15:00:53.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.832+0000 7fb500a00640 1 -- 172.21.15.43:0/2818348737 learned_addr learned my addr 172.21.15.43:0/2818348737 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4f3200640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 0x7fb4fc0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622587147 0 0) 0x7fb4fc0cf0c0 con 0x7fb4fc0b0640 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc003520 con 0x7fb4fc0b0640 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797078857 0 0) 0x7fb4dc003520 con 0x7fb4fc0b0640 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 msgr2=0x7fb4fc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 0x7fb4fc0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 >> v1:172.21.15.69:6789/0 conn(0x7fb4fc15d5b0 legacy=0x7fb4fc0d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc15d990 con 0x7fb4fc0b0640 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb4f4001670 con 0x7fb4fc0b0640 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 >> v1:172.21.15.43:6789/0 conn(0x7fb4fc0b0640 legacy=0x7fb4fc0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.833+0000 7fb4fb400640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc003a10 0x7fb4dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fb400640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc0047c0 0x7fb4dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fb400640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 0x7fb4dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0b01b0 con 0x7fb4dc003a10 2024-01-16T15:00:53.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc00ba80 con 0x7fb4dc0047c0 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc00bc40 con 0x7fb4fc0d14a0 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc003a10 0x7fb4dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 -- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 msgr2=0x7fb4dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4fc0d14a0 0x7fb4dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 -- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc0047c0 msgr2=0x7fb4dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4f3200640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc0047c0 0x7fb4dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc0047c0 0x7fb4dc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.834+0000 7fb4fbe00640 1 -- 172.21.15.43:0/2818348737 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc003520 con 0x7fb4dc003a10 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.835+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc003a10 0x7fb4dc003e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001750 tx=0x7fb4dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db490a9f43902207 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0052d0 con 0x7fb4dc003a10 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec02e050 con 0x7fb4dc003a10 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb4fb400640 1 -- 172.21.15.43:0/2818348737 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0035f0 con 0x7fb4dc003a10 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 -- 172.21.15.43:0/2818348737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc003a10 msgr2=0x7fb4dc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 --2- 172.21.15.43:0/2818348737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc003a10 0x7fb4dc003e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001750 tx=0x7fb4dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb500a00640 1 -- 172.21.15.43:0/2818348737 reap_dead start 2024-01-16T15:00:53.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 -- 172.21.15.43:0/2818348737 shutdown_connections 2024-01-16T15:00:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 -- 172.21.15.43:0/2818348737 >> 172.21.15.43:0/2818348737 conn(0x7fb4fc09c3b0 msgr2=0x7fb4fc0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 -- 172.21.15.43:0/2818348737 shutdown_connections 2024-01-16T15:00:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.836+0000 7fb501400640 1 -- 172.21.15.43:0/2818348737 wait complete. 2024-01-16T15:00:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.838+0000 7fb501400640 1 Processor -- start 2024-01-16T15:00:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.838+0000 7fb501400640 1 -- start start 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 0x7fb4fc15b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4fc0cf430 0x7fb4fc0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0cfe20 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c7d20 con 0x7fb4fc0cf430 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.839+0000 7fb501400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c7f70 con 0x7fb4dc003a10 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51888/0 (socket says 172.21.15.43:51888) 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 -- 172.21.15.43:0/407190199 learned_addr learned my addr 172.21.15.43:0/407190199 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb500a00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4fc0cf430 0x7fb4fc0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 0x7fb4fc15b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 -- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 msgr2=0x7fb4fc15b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 0x7fb4fc15b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 -- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4fc0cf430 msgr2=0x7fb4fc0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4fc0cf430 0x7fb4fc0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4f3200640 1 -- 172.21.15.43:0/407190199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0c8250 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.840+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 0x7fb4fc15b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:53.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb4f3200640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400f380 tx=0x7fb4e400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4017600 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e4011040 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc0c8530 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40179c0 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.841+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc0c8af0 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.842+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc04e840 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.843+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb4e4017b20 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.844+0000 7fb4f8c00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 0x7fb4d406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.844+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4e408a230 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.844+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 0x7fb4d406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.845+0000 7fb4fbe00640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 0x7fb4d406d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec033000 tx=0x7fb4ec009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:53.849+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4e40550c0 con 0x7fb4fc1f2d70 2024-01-16T15:00:53.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:53.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.045+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4fc0afe40 con 0x7fb4d406b2f0 2024-01-16T15:00:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.053+0000 7fb4f8c00640 1 -- 172.21.15.43:0/407190199 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287480 (secure 0 0 0) 0x7fb4ec005410 con 0x7fb4d406b2f0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:54.038 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 msgr2=0x7fb4d406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 0x7fb4d406d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec033000 tx=0x7fb4ec009030 comp rx=0 tx=0).stop 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 msgr2=0x7fb4fc0a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400f380 tx=0x7fb4e400f840 comp rx=0 tx=0).stop 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 shutdown_connections 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4d406b2f0 0x7fb4d406d7b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4fc0cf430 0x7fb4fc0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4fc1f2d70 0x7fb4fc0a0680 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 --2- 172.21.15.43:0/407190199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc003a10 0x7fb4fc15b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 >> 172.21.15.43:0/407190199 conn(0x7fb4fc09c3b0 msgr2=0x7fb4fc0b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 shutdown_connections 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.059+0000 7fb501400640 1 -- 172.21.15.43:0/407190199 wait complete. 2024-01-16T15:00:54.058 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-01-16T15:00:52.814644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241160,"kb_used_data":4172,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733560,"statfs":{"total":773094113280,"available":772847165440,"internally_reserved":0,"allocated":4272128,"data_stored":2362555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":242535980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.789243"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103579+0000","last_change":"2024-01-16T15:00:49.100076+0000","last_active":"2024-01-16T15:00:49.103579+0000","last_peered":"2024-01-16T15:00:49.103579+0000","last_clean":"2024-01-16T15:00:49.103579+0000","last_became_active":"2024-01-16T15:00:49.099726+0000","last_became_peered":"2024-01-16T15:00:49.099726+0000","last_unstale":"2024-01-16T15:00:49.103579+0000","last_undegraded":"2024-01-16T15:00:49.103579+0000","last_fullsized":"2024-01-16T15:00:49.103579+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:47.209930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089904+0000","last_change":"2024-01-16T15:00:49.082846+0000","last_active":"2024-01-16T15:00:49.089904+0000","last_peered":"2024-01-16T15:00:49.089904+0000","last_clean":"2024-01-16T15:00:49.089904+0000","last_became_active":"2024-01-16T15:00:49.082551+0000","last_became_peered":"2024-01-16T15:00:49.082551+0000","last_unstale":"2024-01-16T15:00:49.089904+0000","last_undegraded":"2024-01-16T15:00:49.089904+0000","last_fullsized":"2024-01-16T15:00:49.089904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:46:47.690663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101858+0000","last_change":"2024-01-16T15:00:49.098483+0000","last_active":"2024-01-16T15:00:49.101858+0000","last_peered":"2024-01-16T15:00:49.101858+0000","last_clean":"2024-01-16T15:00:49.101858+0000","last_became_active":"2024-01-16T15:00:49.094570+0000","last_became_peered":"2024-01-16T15:00:49.094570+0000","last_unstale":"2024-01-16T15:00:49.101858+0000","last_undegraded":"2024-01-16T15:00:49.101858+0000","last_fullsized":"2024-01-16T15:00:49.101858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:32.023437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098152+0000","last_change":"2024-01-16T15:00:49.094940+0000","last_active":"2024-01-16T15:00:49.098152+0000","last_peered":"2024-01-16T15:00:49.098152+0000","last_clean":"2024-01-16T15:00:49.098152+0000","last_became_active":"2024-01-16T15:00:49.094526+0000","last_became_peered":"2024-01-16T15:00:49.094526+0000","last_unstale":"2024-01-16T15:00:49.098152+0000","last_undegraded":"2024-01-16T15:00:49.098152+0000","last_fullsized":"2024-01-16T15:00:49.098152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:16.494984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090578+0000","last_change":"2024-01-16T15:00:49.085022+0000","last_active":"2024-01-16T15:00:49.090578+0000","last_peered":"2024-01-16T15:00:49.090578+0000","last_clean":"2024-01-16T15:00:49.090578+0000","last_became_active":"2024-01-16T15:00:49.084708+0000","last_became_peered":"2024-01-16T15:00:49.084708+0000","last_unstale":"2024-01-16T15:00:49.090578+0000","last_undegraded":"2024-01-16T15:00:49.090578+0000","last_fullsized":"2024-01-16T15:00:49.090578+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:29.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103202+0000","last_change":"2024-01-16T15:00:49.101622+0000","last_active":"2024-01-16T15:00:49.103202+0000","last_peered":"2024-01-16T15:00:49.103202+0000","last_clean":"2024-01-16T15:00:49.103202+0000","last_became_active":"2024-01-16T15:00:49.101359+0000","last_became_peered":"2024-01-16T15:00:49.101359+0000","last_unstale":"2024-01-16T15:00:49.103202+0000","last_undegraded":"2024-01-16T15:00:49.103202+0000","last_fullsized":"2024-01-16T15:00:49.103202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:55:43.796400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107207+0000","last_change":"2024-01-16T15:00:49.103255+0000","last_active":"2024-01-16T15:00:49.107207+0000","last_peered":"2024-01-16T15:00:49.107207+0000","last_clean":"2024-01-16T15:00:49.107207+0000","last_became_active":"2024-01-16T15:00:49.100199+0000","last_became_peered":"2024-01-16T15:00:49.100199+0000","last_unstale":"2024-01-16T15:00:49.107207+0000","last_undegraded":"2024-01-16T15:00:49.107207+0000","last_fullsized":"2024-01-16T15:00:49.107207+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:53.126824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.083548+0000","last_change":"2024-01-16T15:00:49.079358+0000","last_active":"2024-01-16T15:00:49.083548+0000","last_peered":"2024-01-16T15:00:49.083548+0000","last_clean":"2024-01-16T15:00:49.083548+0000","last_became_active":"2024-01-16T15:00:49.079053+0000","last_became_peered":"2024-01-16T15:00:49.079053+0000","last_unstale":"2024-01-16T15:00:49.083548+0000","last_undegraded":"2024-01-16T15:00:49.083548+0000","last_fullsized":"2024-01-16T15:00:49.083548+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:36.045009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.087559+0000","last_change":"2024-01-16T15:00:49.082056+0000","last_active":"2024-01-16T15:00:49.087559+0000","last_peered":"2024-01-16T15:00:49.087559+0000","last_clean":"2024-01-16T15:00:49.087559+0000","last_became_active":"2024-01-16T15:00:49.081727+0000","last_became_peered":"2024-01-16T15:00:49.081727+0000","last_unstale":"2024-01-16T15:00:49.087559+0000","last_undegraded":"2024-01-16T15:00:49.087559+0000","last_fullsized":"2024-01-16T15:00:49.087559+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:35.490538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098175+0000","last_change":"2024-01-16T15:00:49.093981+0000","last_active":"2024-01-16T15:00:49.098175+0000","last_peered":"2024-01-16T15:00:49.098175+0000","last_clean":"2024-01-16T15:00:49.098175+0000","last_became_active":"2024-01-16T15:00:49.092638+0000","last_became_peered":"2024-01-16T15:00:49.092638+0000","last_unstale":"2024-01-16T15:00:49.098175+0000","last_undegraded":"2024-01-16T15:00:49.098175+0000","last_fullsized":"2024-01-16T15:00:49.098175+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:58.424871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107122+0000","last_change":"2024-01-16T15:00:49.096005+0000","last_active":"2024-01-16T15:00:49.107122+0000","last_peered":"2024-01-16T15:00:49.107122+0000","last_clean":"2024-01-16T15:00:49.107122+0000","last_became_active":"2024-01-16T15:00:49.095614+0000","last_became_peered":"2024-01-16T15:00:49.095614+0000","last_unstale":"2024-01-16T15:00:49.107122+0000","last_undegraded":"2024-01-16T15:00:49.107122+0000","last_fullsized":"2024-01-16T15:00:49.107122+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:56.533665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102359+0000","last_change":"2024-01-16T15:00:49.097922+0000","last_active":"2024-01-16T15:00:49.102359+0000","last_peered":"2024-01-16T15:00:49.102359+0000","last_clean":"2024-01-16T15:00:49.102359+0000","last_became_active":"2024-01-16T15:00:49.097354+0000","last_became_peered":"2024-01-16T15:00:49.097354+0000","last_unstale":"2024-01-16T15:00:49.102359+0000","last_undegraded":"2024-01-16T15:00:49.102359+0000","last_fullsized":"2024-01-16T15:00:49.102359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:38.710890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088163+0000","last_change":"2024-01-16T15:00:49.084711+0000","last_active":"2024-01-16T15:00:49.088163+0000","last_peered":"2024-01-16T15:00:49.088163+0000","last_clean":"2024-01-16T15:00:49.088163+0000","last_became_active":"2024-01-16T15:00:49.083154+0000","last_became_peered":"2024-01-16T15:00:49.083154+0000","last_unstale":"2024-01-16T15:00:49.088163+0000","last_undegraded":"2024-01-16T15:00:49.088163+0000","last_fullsized":"2024-01-16T15:00:49.088163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.345260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089815+0000","last_change":"2024-01-16T15:00:49.082118+0000","last_active":"2024-01-16T15:00:49.089815+0000","last_peered":"2024-01-16T15:00:49.089815+0000","last_clean":"2024-01-16T15:00:49.089815+0000","last_became_active":"2024-01-16T15:00:49.079235+0000","last_became_peered":"2024-01-16T15:00:49.079235+0000","last_unstale":"2024-01-16T15:00:49.089815+0000","last_undegraded":"2024-01-16T15:00:49.089815+0000","last_fullsized":"2024-01-16T15:00:49.089815+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:57.830057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101834+0000","last_change":"2024-01-16T15:00:49.098381+0000","last_active":"2024-01-16T15:00:49.101834+0000","last_peered":"2024-01-16T15:00:49.101834+0000","last_clean":"2024-01-16T15:00:49.101834+0000","last_became_active":"2024-01-16T15:00:49.098041+0000","last_became_peered":"2024-01-16T15:00:49.098041+0000","last_unstale":"2024-01-16T15:00:49.101834+0000","last_undegraded":"2024-01-16T15:00:49.101834+0000","last_fullsized":"2024-01-16T15:00:49.101834+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:55.857087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097343+0000","last_change":"2024-01-16T15:00:49.092018+0000","last_active":"2024-01-16T15:00:49.097343+0000","last_peered":"2024-01-16T15:00:49.097343+0000","last_clean":"2024-01-16T15:00:49.097343+0000","last_became_active":"2024-01-16T15:00:49.091452+0000","last_became_peered":"2024-01-16T15:00:49.091452+0000","last_unstale":"2024-01-16T15:00:49.097343+0000","last_undegraded":"2024-01-16T15:00:49.097343+0000","last_fullsized":"2024-01-16T15:00:49.097343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:07.655096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.109703+0000","last_change":"2024-01-16T15:00:49.104805+0000","last_active":"2024-01-16T15:00:49.109703+0000","last_peered":"2024-01-16T15:00:49.109703+0000","last_clean":"2024-01-16T15:00:49.109703+0000","last_became_active":"2024-01-16T15:00:49.102663+0000","last_became_peered":"2024-01-16T15:00:49.102663+0000","last_unstale":"2024-01-16T15:00:49.109703+0000","last_undegraded":"2024-01-16T15:00:49.109703+0000","last_fullsized":"2024-01-16T15:00:49.109703+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:34:56.174799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090584+0000","last_change":"2024-01-16T15:00:49.086522+0000","last_active":"2024-01-16T15:00:49.090584+0000","last_peered":"2024-01-16T15:00:49.090584+0000","last_clean":"2024-01-16T15:00:49.090584+0000","last_became_active":"2024-01-16T15:00:49.085496+0000","last_became_peered":"2024-01-16T15:00:49.085496+0000","last_unstale":"2024-01-16T15:00:49.090584+0000","last_undegraded":"2024-01-16T15:00:49.090584+0000","last_fullsized":"2024-01-16T15:00:49.090584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:07.655096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090649+0000","last_change":"2024-01-16T15:00:49.086006+0000","last_active":"2024-01-16T15:00:49.090649+0000","last_peered":"2024-01-16T15:00:49.090649+0000","last_clean":"2024-01-16T15:00:49.090649+0000","last_became_active":"2024-01-16T15:00:49.085403+0000","last_became_peered":"2024-01-16T15:00:49.085403+0000","last_unstale":"2024-01-16T15:00:49.090649+0000","last_undegraded":"2024-01-16T15:00:49.090649+0000","last_fullsized":"2024-01-16T15:00:49.090649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:36.045009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101161+0000","last_change":"2024-01-16T15:00:49.099476+0000","last_active":"2024-01-16T15:00:49.101161+0000","last_peered":"2024-01-16T15:00:49.101161+0000","last_clean":"2024-01-16T15:00:49.101161+0000","last_became_active":"2024-01-16T15:00:49.097979+0000","last_became_peered":"2024-01-16T15:00:49.097979+0000","last_unstale":"2024-01-16T15:00:49.101161+0000","last_undegraded":"2024-01-16T15:00:49.101161+0000","last_fullsized":"2024-01-16T15:00:49.101161+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:19.828148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085671+0000","last_change":"2024-01-16T15:00:49.070562+0000","last_active":"2024-01-16T15:00:49.085671+0000","last_peered":"2024-01-16T15:00:49.085671+0000","last_clean":"2024-01-16T15:00:49.085671+0000","last_became_active":"2024-01-16T15:00:49.069837+0000","last_became_peered":"2024-01-16T15:00:49.069837+0000","last_unstale":"2024-01-16T15:00:49.085671+0000","last_undegraded":"2024-01-16T15:00:49.085671+0000","last_fullsized":"2024-01-16T15:00:49.085671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:47:05.563630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088664+0000","last_change":"2024-01-16T15:00:49.083678+0000","last_active":"2024-01-16T15:00:49.088664+0000","last_peered":"2024-01-16T15:00:49.088664+0000","last_clean":"2024-01-16T15:00:49.088664+0000","last_became_active":"2024-01-16T15:00:49.083046+0000","last_became_peered":"2024-01-16T15:00:49.083046+0000","last_unstale":"2024-01-16T15:00:49.088664+0000","last_undegraded":"2024-01-16T15:00:49.088664+0000","last_fullsized":"2024-01-16T15:00:49.088664+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:29.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085924+0000","last_change":"2024-01-16T15:00:49.078833+0000","last_active":"2024-01-16T15:00:49.085924+0000","last_peered":"2024-01-16T15:00:49.085924+0000","last_clean":"2024-01-16T15:00:49.085924+0000","last_became_active":"2024-01-16T15:00:49.078560+0000","last_became_peered":"2024-01-16T15:00:49.078560+0000","last_unstale":"2024-01-16T15:00:49.085924+0000","last_undegraded":"2024-01-16T15:00:49.085924+0000","last_fullsized":"2024-01-16T15:00:49.085924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098693+0000","last_change":"2024-01-16T15:00:49.092981+0000","last_active":"2024-01-16T15:00:49.098693+0000","last_peered":"2024-01-16T15:00:49.098693+0000","last_clean":"2024-01-16T15:00:49.098693+0000","last_became_active":"2024-01-16T15:00:49.092371+0000","last_became_peered":"2024-01-16T15:00:49.092371+0000","last_unstale":"2024-01-16T15:00:49.098693+0000","last_undegraded":"2024-01-16T15:00:49.098693+0000","last_fullsized":"2024-01-16T15:00:49.098693+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:16.494984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092621+0000","last_change":"2024-01-16T15:00:49.082303+0000","last_active":"2024-01-16T15:00:49.092621+0000","last_peered":"2024-01-16T15:00:49.092621+0000","last_clean":"2024-01-16T15:00:49.092621+0000","last_became_active":"2024-01-16T15:00:49.081431+0000","last_became_peered":"2024-01-16T15:00:49.081431+0000","last_unstale":"2024-01-16T15:00:49.092621+0000","last_undegraded":"2024-01-16T15:00:49.092621+0000","last_fullsized":"2024-01-16T15:00:49.092621+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:24.205886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089715+0000","last_change":"2024-01-16T15:00:49.081770+0000","last_active":"2024-01-16T15:00:49.089715+0000","last_peered":"2024-01-16T15:00:49.089715+0000","last_clean":"2024-01-16T15:00:49.089715+0000","last_became_active":"2024-01-16T15:00:49.077817+0000","last_became_peered":"2024-01-16T15:00:49.077817+0000","last_unstale":"2024-01-16T15:00:49.089715+0000","last_undegraded":"2024-01-16T15:00:49.089715+0000","last_fullsized":"2024-01-16T15:00:49.089715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:11.730584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098334+0000","last_change":"2024-01-16T15:00:49.095615+0000","last_active":"2024-01-16T15:00:49.098334+0000","last_peered":"2024-01-16T15:00:49.098334+0000","last_clean":"2024-01-16T15:00:49.098334+0000","last_became_active":"2024-01-16T15:00:49.095297+0000","last_became_peered":"2024-01-16T15:00:49.095297+0000","last_unstale":"2024-01-16T15:00:49.098334+0000","last_undegraded":"2024-01-16T15:00:49.098334+0000","last_fullsized":"2024-01-16T15:00:49.098334+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102377+0000","last_change":"2024-01-16T15:00:49.098666+0000","last_active":"2024-01-16T15:00:49.102377+0000","last_peered":"2024-01-16T15:00:49.102377+0000","last_clean":"2024-01-16T15:00:49.102377+0000","last_became_active":"2024-01-16T15:00:49.093930+0000","last_became_peered":"2024-01-16T15:00:49.093930+0000","last_unstale":"2024-01-16T15:00:49.102377+0000","last_undegraded":"2024-01-16T15:00:49.102377+0000","last_fullsized":"2024-01-16T15:00:49.102377+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:48.606264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085690+0000","last_change":"2024-01-16T15:00:49.082783+0000","last_active":"2024-01-16T15:00:49.085690+0000","last_peered":"2024-01-16T15:00:49.085690+0000","last_clean":"2024-01-16T15:00:49.085690+0000","last_became_active":"2024-01-16T15:00:49.082507+0000","last_became_peered":"2024-01-16T15:00:49.082507+0000","last_unstale":"2024-01-16T15:00:49.085690+0000","last_undegraded":"2024-01-16T15:00:49.085690+0000","last_fullsized":"2024-01-16T15:00:49.085690+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:30.876555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107197+0000","last_change":"2024-01-16T15:00:49.102767+0000","last_active":"2024-01-16T15:00:49.107197+0000","last_peered":"2024-01-16T15:00:49.107197+0000","last_clean":"2024-01-16T15:00:49.107197+0000","last_became_active":"2024-01-16T15:00:49.099125+0000","last_became_peered":"2024-01-16T15:00:49.099125+0000","last_unstale":"2024-01-16T15:00:49.107197+0000","last_undegraded":"2024-01-16T15:00:49.107197+0000","last_fullsized":"2024-01-16T15:00:49.107197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:35.305458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094379+0000","last_change":"2024-01-16T15:00:49.086557+0000","last_active":"2024-01-16T15:00:49.094379+0000","last_peered":"2024-01-16T15:00:49.094379+0000","last_clean":"2024-01-16T15:00:49.094379+0000","last_became_active":"2024-01-16T15:00:49.086278+0000","last_became_peered":"2024-01-16T15:00:49.086278+0000","last_unstale":"2024-01-16T15:00:49.094379+0000","last_undegraded":"2024-01-16T15:00:49.094379+0000","last_fullsized":"2024-01-16T15:00:49.094379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097320+0000","last_change":"2024-01-16T15:00:49.091472+0000","last_active":"2024-01-16T15:00:49.097320+0000","last_peered":"2024-01-16T15:00:49.097320+0000","last_clean":"2024-01-16T15:00:49.097320+0000","last_became_active":"2024-01-16T15:00:49.091154+0000","last_became_peered":"2024-01-16T15:00:49.091154+0000","last_unstale":"2024-01-16T15:00:49.097320+0000","last_undegraded":"2024-01-16T15:00:49.097320+0000","last_fullsized":"2024-01-16T15:00:49.097320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:01.860050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082926+0000","last_change":"2024-01-16T15:00:49.077355+0000","last_active":"2024-01-16T15:00:49.082926+0000","last_peered":"2024-01-16T15:00:49.082926+0000","last_clean":"2024-01-16T15:00:49.082926+0000","last_became_active":"2024-01-16T15:00:49.077078+0000","last_became_peered":"2024-01-16T15:00:49.077078+0000","last_unstale":"2024-01-16T15:00:49.082926+0000","last_undegraded":"2024-01-16T15:00:49.082926+0000","last_fullsized":"2024-01-16T15:00:49.082926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:43.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101286+0000","last_change":"2024-01-16T15:00:48.066867+0000","last_active":"2024-01-16T15:00:49.101286+0000","last_peered":"2024-01-16T15:00:49.101286+0000","last_clean":"2024-01-16T15:00:49.101286+0000","last_became_active":"2024-01-16T15:00:48.066540+0000","last_became_peered":"2024-01-16T15:00:48.066540+0000","last_unstale":"2024-01-16T15:00:49.101286+0000","last_undegraded":"2024-01-16T15:00:49.101286+0000","last_fullsized":"2024-01-16T15:00:49.101286+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:43.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094534+0000","last_change":"2024-01-16T15:00:49.089854+0000","last_active":"2024-01-16T15:00:49.094534+0000","last_peered":"2024-01-16T15:00:49.094534+0000","last_clean":"2024-01-16T15:00:49.094534+0000","last_became_active":"2024-01-16T15:00:49.089562+0000","last_became_peered":"2024-01-16T15:00:49.089562+0000","last_unstale":"2024-01-16T15:00:49.094534+0000","last_undegraded":"2024-01-16T15:00:49.094534+0000","last_fullsized":"2024-01-16T15:00:49.094534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:58.433916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082855+0000","last_change":"2024-01-16T15:00:48.068961+0000","last_active":"2024-01-16T15:00:49.082855+0000","last_peered":"2024-01-16T15:00:49.082855+0000","last_clean":"2024-01-16T15:00:49.082855+0000","last_became_active":"2024-01-16T15:00:48.068612+0000","last_became_peered":"2024-01-16T15:00:48.068612+0000","last_unstale":"2024-01-16T15:00:49.082855+0000","last_undegraded":"2024-01-16T15:00:49.082855+0000","last_fullsized":"2024-01-16T15:00:49.082855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:43.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095130+0000","last_change":"2024-01-16T15:00:49.090176+0000","last_active":"2024-01-16T15:00:49.095130+0000","last_peered":"2024-01-16T15:00:49.095130+0000","last_clean":"2024-01-16T15:00:49.095130+0000","last_became_active":"2024-01-16T15:00:49.089727+0000","last_became_peered":"2024-01-16T15:00:49.089727+0000","last_unstale":"2024-01-16T15:00:49.095130+0000","last_undegraded":"2024-01-16T15:00:49.095130+0000","last_fullsized":"2024-01-16T15:00:49.095130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:54.523311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095142+0000","last_change":"2024-01-16T15:00:48.054284+0000","last_active":"2024-01-16T15:00:49.095142+0000","last_peered":"2024-01-16T15:00:49.095142+0000","last_clean":"2024-01-16T15:00:49.095142+0000","last_became_active":"2024-01-16T15:00:48.053923+0000","last_became_peered":"2024-01-16T15:00:48.053923+0000","last_unstale":"2024-01-16T15:00:49.095142+0000","last_undegraded":"2024-01-16T15:00:49.095142+0000","last_fullsized":"2024-01-16T15:00:49.095142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:55.528974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098354+0000","last_change":"2024-01-16T15:00:49.096237+0000","last_active":"2024-01-16T15:00:49.098354+0000","last_peered":"2024-01-16T15:00:49.098354+0000","last_clean":"2024-01-16T15:00:49.098354+0000","last_became_active":"2024-01-16T15:00:49.095903+0000","last_became_peered":"2024-01-16T15:00:49.095903+0000","last_unstale":"2024-01-16T15:00:49.098354+0000","last_undegraded":"2024-01-16T15:00:49.098354+0000","last_fullsized":"2024-01-16T15:00:49.098354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:17.948940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093899+0000","last_change":"2024-01-16T15:00:48.055207+0000","last_active":"2024-01-16T15:00:49.093899+0000","last_peered":"2024-01-16T15:00:49.093899+0000","last_clean":"2024-01-16T15:00:49.093899+0000","last_became_active":"2024-01-16T15:00:48.054824+0000","last_became_peered":"2024-01-16T15:00:48.054824+0000","last_unstale":"2024-01-16T15:00:49.093899+0000","last_undegraded":"2024-01-16T15:00:49.093899+0000","last_fullsized":"2024-01-16T15:00:49.093899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:53.518620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.109703+0000","last_change":"2024-01-16T15:00:49.102988+0000","last_active":"2024-01-16T15:00:49.109703+0000","last_peered":"2024-01-16T15:00:49.109703+0000","last_clean":"2024-01-16T15:00:49.109703+0000","last_became_active":"2024-01-16T15:00:49.100514+0000","last_became_peered":"2024-01-16T15:00:49.100514+0000","last_unstale":"2024-01-16T15:00:49.109703+0000","last_undegraded":"2024-01-16T15:00:49.109703+0000","last_fullsized":"2024-01-16T15:00:49.109703+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:46.867540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096783+0000","last_change":"2024-01-16T15:00:48.062992+0000","last_active":"2024-01-16T15:00:49.096783+0000","last_peered":"2024-01-16T15:00:49.096783+0000","last_clean":"2024-01-16T15:00:49.096783+0000","last_became_active":"2024-01-16T15:00:48.062375+0000","last_became_peered":"2024-01-16T15:00:48.062375+0000","last_unstale":"2024-01-16T15:00:49.096783+0000","last_undegraded":"2024-01-16T15:00:49.096783+0000","last_fullsized":"2024-01-16T15:00:49.096783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:57.429224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098786+0000","last_change":"2024-01-16T15:00:49.093002+0000","last_active":"2024-01-16T15:00:49.098786+0000","last_peered":"2024-01-16T15:00:49.098786+0000","last_clean":"2024-01-16T15:00:49.098786+0000","last_became_active":"2024-01-16T15:00:49.092636+0000","last_became_peered":"2024-01-16T15:00:49.092636+0000","last_unstale":"2024-01-16T15:00:49.098786+0000","last_undegraded":"2024-01-16T15:00:49.098786+0000","last_fullsized":"2024-01-16T15:00:49.098786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:58.424871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096303+0000","last_change":"2024-01-16T15:00:48.064120+0000","last_active":"2024-01-16T15:00:49.096303+0000","last_peered":"2024-01-16T15:00:49.096303+0000","last_clean":"2024-01-16T15:00:49.096303+0000","last_became_active":"2024-01-16T15:00:48.063828+0000","last_became_peered":"2024-01-16T15:00:48.063828+0000","last_unstale":"2024-01-16T15:00:49.096303+0000","last_undegraded":"2024-01-16T15:00:49.096303+0000","last_fullsized":"2024-01-16T15:00:49.096303+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:28.025787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.091025+0000","last_change":"2024-01-16T15:00:49.086126+0000","last_active":"2024-01-16T15:00:49.091025+0000","last_peered":"2024-01-16T15:00:49.091025+0000","last_clean":"2024-01-16T15:00:49.091025+0000","last_became_active":"2024-01-16T15:00:49.085743+0000","last_became_peered":"2024-01-16T15:00:49.085743+0000","last_unstale":"2024-01-16T15:00:49.091025+0000","last_undegraded":"2024-01-16T15:00:49.091025+0000","last_fullsized":"2024-01-16T15:00:49.091025+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:50.886550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.066285+0000","last_change":"2024-01-16T15:00:48.051772+0000","last_active":"2024-01-16T15:00:49.066285+0000","last_peered":"2024-01-16T15:00:49.066285+0000","last_clean":"2024-01-16T15:00:49.066285+0000","last_became_active":"2024-01-16T15:00:48.051466+0000","last_became_peered":"2024-01-16T15:00:48.051466+0000","last_unstale":"2024-01-16T15:00:49.066285+0000","last_undegraded":"2024-01-16T15:00:49.066285+0000","last_fullsized":"2024-01-16T15:00:49.066285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:34.300767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102464+0000","last_change":"2024-01-16T15:00:49.100036+0000","last_active":"2024-01-16T15:00:49.102464+0000","last_peered":"2024-01-16T15:00:49.102464+0000","last_clean":"2024-01-16T15:00:49.102464+0000","last_became_active":"2024-01-16T15:00:49.099324+0000","last_became_peered":"2024-01-16T15:00:49.099324+0000","last_unstale":"2024-01-16T15:00:49.102464+0000","last_undegraded":"2024-01-16T15:00:49.102464+0000","last_fullsized":"2024-01-16T15:00:49.102464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:44.383320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063163+0000","last_change":"2024-01-16T15:00:48.065255+0000","last_active":"2024-01-16T15:00:49.063163+0000","last_peered":"2024-01-16T15:00:49.063163+0000","last_clean":"2024-01-16T15:00:49.063163+0000","last_became_active":"2024-01-16T15:00:48.064866+0000","last_became_peered":"2024-01-16T15:00:48.064866+0000","last_unstale":"2024-01-16T15:00:49.063163+0000","last_undegraded":"2024-01-16T15:00:49.063163+0000","last_fullsized":"2024-01-16T15:00:49.063163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:56.195182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.099676+0000","last_change":"2024-01-16T15:00:49.096130+0000","last_active":"2024-01-16T15:00:49.099676+0000","last_peered":"2024-01-16T15:00:49.099676+0000","last_clean":"2024-01-16T15:00:49.099676+0000","last_became_active":"2024-01-16T15:00:49.093909+0000","last_became_peered":"2024-01-16T15:00:49.093909+0000","last_unstale":"2024-01-16T15:00:49.099676+0000","last_undegraded":"2024-01-16T15:00:49.099676+0000","last_fullsized":"2024-01-16T15:00:49.099676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:20.413171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.068433+0000","last_change":"2024-01-16T15:00:48.065291+0000","last_active":"2024-01-16T15:00:49.068433+0000","last_peered":"2024-01-16T15:00:49.068433+0000","last_clean":"2024-01-16T15:00:49.068433+0000","last_became_active":"2024-01-16T15:00:48.064941+0000","last_became_peered":"2024-01-16T15:00:48.064941+0000","last_unstale":"2024-01-16T15:00:49.068433+0000","last_undegraded":"2024-01-16T15:00:49.068433+0000","last_fullsized":"2024-01-16T15:00:49.068433+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:42:52.122133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.091184+0000","last_change":"2024-01-16T15:00:49.087584+0000","last_active":"2024-01-16T15:00:49.091184+0000","last_peered":"2024-01-16T15:00:49.091184+0000","last_clean":"2024-01-16T15:00:49.091184+0000","last_became_active":"2024-01-16T15:00:49.087177+0000","last_became_peered":"2024-01-16T15:00:49.087177+0000","last_unstale":"2024-01-16T15:00:49.091184+0000","last_undegraded":"2024-01-16T15:00:49.091184+0000","last_fullsized":"2024-01-16T15:00:49.091184+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:34:56.174799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062092+0000","last_change":"2024-01-16T15:00:48.056099+0000","last_active":"2024-01-16T15:00:49.062092+0000","last_peered":"2024-01-16T15:00:49.062092+0000","last_clean":"2024-01-16T15:00:49.062092+0000","last_became_active":"2024-01-16T15:00:48.055799+0000","last_became_peered":"2024-01-16T15:00:48.055799+0000","last_unstale":"2024-01-16T15:00:49.062092+0000","last_undegraded":"2024-01-16T15:00:49.062092+0000","last_fullsized":"2024-01-16T15:00:49.062092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:54.852396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089725+0000","last_change":"2024-01-16T15:00:49.082076+0000","last_active":"2024-01-16T15:00:49.089725+0000","last_peered":"2024-01-16T15:00:49.089725+0000","last_clean":"2024-01-16T15:00:49.089725+0000","last_became_active":"2024-01-16T15:00:49.081756+0000","last_became_peered":"2024-01-16T15:00:49.081756+0000","last_unstale":"2024-01-16T15:00:49.089725+0000","last_undegraded":"2024-01-16T15:00:49.089725+0000","last_fullsized":"2024-01-16T15:00:49.089725+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:54:50.522730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100728+0000","last_change":"2024-01-16T15:00:48.066297+0000","last_active":"2024-01-16T15:00:49.100728+0000","last_peered":"2024-01-16T15:00:49.100728+0000","last_clean":"2024-01-16T15:00:49.100728+0000","last_became_active":"2024-01-16T15:00:48.065970+0000","last_became_peered":"2024-01-16T15:00:48.065970+0000","last_unstale":"2024-01-16T15:00:49.100728+0000","last_undegraded":"2024-01-16T15:00:49.100728+0000","last_fullsized":"2024-01-16T15:00:49.100728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:38.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098915+0000","last_change":"2024-01-16T15:00:49.096671+0000","last_active":"2024-01-16T15:00:49.098915+0000","last_peered":"2024-01-16T15:00:49.098915+0000","last_clean":"2024-01-16T15:00:49.098915+0000","last_became_active":"2024-01-16T15:00:49.095271+0000","last_became_peered":"2024-01-16T15:00:49.095271+0000","last_unstale":"2024-01-16T15:00:49.098915+0000","last_undegraded":"2024-01-16T15:00:49.098915+0000","last_fullsized":"2024-01-16T15:00:49.098915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:03:58.554748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.103528+0000","last_change":"2024-01-16T15:00:48.055829+0000","last_active":"2024-01-16T15:00:49.103528+0000","last_peered":"2024-01-16T15:00:49.103528+0000","last_clean":"2024-01-16T15:00:49.103528+0000","last_became_active":"2024-01-16T15:00:48.054343+0000","last_became_peered":"2024-01-16T15:00:48.054343+0000","last_unstale":"2024-01-16T15:00:49.103528+0000","last_undegraded":"2024-01-16T15:00:49.103528+0000","last_fullsized":"2024-01-16T15:00:49.103528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:11.601859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098563+0000","last_change":"2024-01-16T15:00:49.094720+0000","last_active":"2024-01-16T15:00:49.098563+0000","last_peered":"2024-01-16T15:00:49.098563+0000","last_clean":"2024-01-16T15:00:49.098563+0000","last_became_active":"2024-01-16T15:00:49.094448+0000","last_became_peered":"2024-01-16T15:00:49.094448+0000","last_unstale":"2024-01-16T15:00:49.098563+0000","last_undegraded":"2024-01-16T15:00:49.098563+0000","last_fullsized":"2024-01-16T15:00:49.098563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:16.352542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.097890+0000","last_change":"2024-01-16T15:00:48.066210+0000","last_active":"2024-01-16T15:00:49.097890+0000","last_peered":"2024-01-16T15:00:49.097890+0000","last_clean":"2024-01-16T15:00:49.097890+0000","last_became_active":"2024-01-16T15:00:48.065896+0000","last_became_peered":"2024-01-16T15:00:48.065896+0000","last_unstale":"2024-01-16T15:00:49.097890+0000","last_undegraded":"2024-01-16T15:00:49.097890+0000","last_fullsized":"2024-01-16T15:00:49.097890+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:16.944248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.089921+0000","last_change":"2024-01-16T15:00:49.082844+0000","last_active":"2024-01-16T15:00:49.089921+0000","last_peered":"2024-01-16T15:00:49.089921+0000","last_clean":"2024-01-16T15:00:49.089921+0000","last_became_active":"2024-01-16T15:00:49.082545+0000","last_became_peered":"2024-01-16T15:00:49.082545+0000","last_unstale":"2024-01-16T15:00:49.089921+0000","last_undegraded":"2024-01-16T15:00:49.089921+0000","last_fullsized":"2024-01-16T15:00:49.089921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:54.713937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.064530+0000","last_change":"2024-01-16T15:00:48.066013+0000","last_active":"2024-01-16T15:00:49.064530+0000","last_peered":"2024-01-16T15:00:49.064530+0000","last_clean":"2024-01-16T15:00:49.064530+0000","last_became_active":"2024-01-16T15:00:48.065587+0000","last_became_peered":"2024-01-16T15:00:48.065587+0000","last_unstale":"2024-01-16T15:00:49.064530+0000","last_undegraded":"2024-01-16T15:00:49.064530+0000","last_fullsized":"2024-01-16T15:00:49.064530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:42.485766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.107090+0000","last_change":"2024-01-16T15:00:49.100919+0000","last_active":"2024-01-16T15:00:49.107090+0000","last_peered":"2024-01-16T15:00:49.107090+0000","last_clean":"2024-01-16T15:00:49.107090+0000","last_became_active":"2024-01-16T15:00:49.100203+0000","last_became_peered":"2024-01-16T15:00:49.100203+0000","last_unstale":"2024-01-16T15:00:49.107090+0000","last_undegraded":"2024-01-16T15:00:49.107090+0000","last_fullsized":"2024-01-16T15:00:49.107090+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:05.403999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.066423+0000","last_change":"2024-01-16T15:00:48.065263+0000","last_active":"2024-01-16T15:00:49.066423+0000","last_peered":"2024-01-16T15:00:49.066423+0000","last_clean":"2024-01-16T15:00:49.066423+0000","last_became_active":"2024-01-16T15:00:48.064900+0000","last_became_peered":"2024-01-16T15:00:48.064900+0000","last_unstale":"2024-01-16T15:00:49.066423+0000","last_undegraded":"2024-01-16T15:00:49.066423+0000","last_fullsized":"2024-01-16T15:00:49.066423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:45.862848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094359+0000","last_change":"2024-01-16T15:00:49.086431+0000","last_active":"2024-01-16T15:00:49.094359+0000","last_peered":"2024-01-16T15:00:49.094359+0000","last_clean":"2024-01-16T15:00:49.094359+0000","last_became_active":"2024-01-16T15:00:49.086050+0000","last_became_peered":"2024-01-16T15:00:49.086050+0000","last_unstale":"2024-01-16T15:00:49.094359+0000","last_undegraded":"2024-01-16T15:00:49.094359+0000","last_fullsized":"2024-01-16T15:00:49.094359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100312+0000","last_change":"2024-01-16T15:00:48.053315+0000","last_active":"2024-01-16T15:00:49.100312+0000","last_peered":"2024-01-16T15:00:49.100312+0000","last_clean":"2024-01-16T15:00:49.100312+0000","last_became_active":"2024-01-16T15:00:48.052685+0000","last_became_peered":"2024-01-16T15:00:48.052685+0000","last_unstale":"2024-01-16T15:00:49.100312+0000","last_undegraded":"2024-01-16T15:00:49.100312+0000","last_fullsized":"2024-01-16T15:00:49.100312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:00.855358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098537+0000","last_change":"2024-01-16T15:00:49.096322+0000","last_active":"2024-01-16T15:00:49.098537+0000","last_peered":"2024-01-16T15:00:49.098537+0000","last_clean":"2024-01-16T15:00:49.098537+0000","last_became_active":"2024-01-16T15:00:49.095731+0000","last_became_peered":"2024-01-16T15:00:49.095731+0000","last_unstale":"2024-01-16T15:00:49.098537+0000","last_undegraded":"2024-01-16T15:00:49.098537+0000","last_fullsized":"2024-01-16T15:00:49.098537+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:48:49.790562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.098847+0000","last_change":"2024-01-16T15:00:48.061110+0000","last_active":"2024-01-16T15:00:49.098847+0000","last_peered":"2024-01-16T15:00:49.098847+0000","last_clean":"2024-01-16T15:00:49.098847+0000","last_became_active":"2024-01-16T15:00:48.060801+0000","last_became_peered":"2024-01-16T15:00:48.060801+0000","last_unstale":"2024-01-16T15:00:49.098847+0000","last_undegraded":"2024-01-16T15:00:49.098847+0000","last_fullsized":"2024-01-16T15:00:49.098847+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:31.018746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.101924+0000","last_change":"2024-01-16T15:00:49.098975+0000","last_active":"2024-01-16T15:00:49.101924+0000","last_peered":"2024-01-16T15:00:49.101924+0000","last_clean":"2024-01-16T15:00:49.101924+0000","last_became_active":"2024-01-16T15:00:49.098629+0000","last_became_peered":"2024-01-16T15:00:49.098629+0000","last_unstale":"2024-01-16T15:00:49.101924+0000","last_undegraded":"2024-01-16T15:00:49.101924+0000","last_fullsized":"2024-01-16T15:00:49.101924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:24:05.034889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.096129+0000","last_change":"2024-01-16T15:00:48.065651+0000","last_active":"2024-01-16T15:00:49.096129+0000","last_peered":"2024-01-16T15:00:49.096129+0000","last_clean":"2024-01-16T15:00:49.096129+0000","last_became_active":"2024-01-16T15:00:48.065257+0000","last_became_peered":"2024-01-16T15:00:48.065257+0000","last_unstale":"2024-01-16T15:00:49.096129+0000","last_undegraded":"2024-01-16T15:00:49.096129+0000","last_fullsized":"2024-01-16T15:00:49.096129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:44.613107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093241+0000","last_change":"2024-01-16T15:00:49.084273+0000","last_active":"2024-01-16T15:00:49.093241+0000","last_peered":"2024-01-16T15:00:49.093241+0000","last_clean":"2024-01-16T15:00:49.093241+0000","last_became_active":"2024-01-16T15:00:49.083999+0000","last_became_peered":"2024-01-16T15:00:49.083999+0000","last_unstale":"2024-01-16T15:00:49.093241+0000","last_undegraded":"2024-01-16T15:00:49.093241+0000","last_fullsized":"2024-01-16T15:00:49.093241+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:55.857087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.095383+0000","last_change":"2024-01-16T15:00:48.062649+0000","last_active":"2024-01-16T15:00:49.095383+0000","last_peered":"2024-01-16T15:00:49.095383+0000","last_clean":"2024-01-16T15:00:49.095383+0000","last_became_active":"2024-01-16T15:00:48.062318+0000","last_became_peered":"2024-01-16T15:00:48.062318+0000","last_unstale":"2024-01-16T15:00:49.095383+0000","last_undegraded":"2024-01-16T15:00:49.095383+0000","last_fullsized":"2024-01-16T15:00:49.095383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:31:10.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.078534+0000","last_change":"2024-01-16T15:00:49.075198+0000","last_active":"2024-01-16T15:00:49.078534+0000","last_peered":"2024-01-16T15:00:49.078534+0000","last_clean":"2024-01-16T15:00:49.078534+0000","last_became_active":"2024-01-16T15:00:49.074924+0000","last_became_peered":"2024-01-16T15:00:49.074924+0000","last_unstale":"2024-01-16T15:00:49.078534+0000","last_undegraded":"2024-01-16T15:00:49.078534+0000","last_fullsized":"2024-01-16T15:00:49.078534+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:30.876555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062965+0000","last_change":"2024-01-16T15:00:48.067683+0000","last_active":"2024-01-16T15:00:49.062965+0000","last_peered":"2024-01-16T15:00:49.062965+0000","last_clean":"2024-01-16T15:00:49.062965+0000","last_became_active":"2024-01-16T15:00:48.067408+0000","last_became_peered":"2024-01-16T15:00:48.067408+0000","last_unstale":"2024-01-16T15:00:49.062965+0000","last_undegraded":"2024-01-16T15:00:49.062965+0000","last_fullsized":"2024-01-16T15:00:49.062965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:49.258940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.088131+0000","last_change":"2024-01-16T15:00:49.083664+0000","last_active":"2024-01-16T15:00:49.088131+0000","last_peered":"2024-01-16T15:00:49.088131+0000","last_clean":"2024-01-16T15:00:49.088131+0000","last_became_active":"2024-01-16T15:00:49.083025+0000","last_became_peered":"2024-01-16T15:00:49.083025+0000","last_unstale":"2024-01-16T15:00:49.088131+0000","last_undegraded":"2024-01-16T15:00:49.088131+0000","last_fullsized":"2024-01-16T15:00:49.088131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:14.552571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.082699+0000","last_change":"2024-01-16T15:00:48.051159+0000","last_active":"2024-01-16T15:00:49.082699+0000","last_peered":"2024-01-16T15:00:49.082699+0000","last_clean":"2024-01-16T15:00:49.082699+0000","last_became_active":"2024-01-16T15:00:48.050865+0000","last_became_peered":"2024-01-16T15:00:48.050865+0000","last_unstale":"2024-01-16T15:00:49.082699+0000","last_undegraded":"2024-01-16T15:00:49.082699+0000","last_fullsized":"2024-01-16T15:00:49.082699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:15.347850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.084463+0000","last_change":"2024-01-16T15:00:49.078806+0000","last_active":"2024-01-16T15:00:49.084463+0000","last_peered":"2024-01-16T15:00:49.084463+0000","last_clean":"2024-01-16T15:00:49.084463+0000","last_became_active":"2024-01-16T15:00:49.078517+0000","last_became_peered":"2024-01-16T15:00:49.078517+0000","last_unstale":"2024-01-16T15:00:49.084463+0000","last_undegraded":"2024-01-16T15:00:49.084463+0000","last_fullsized":"2024-01-16T15:00:49.084463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:30:02.526244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.073827+0000","last_change":"2024-01-16T15:00:48.063613+0000","last_active":"2024-01-16T15:00:49.073827+0000","last_peered":"2024-01-16T15:00:49.073827+0000","last_clean":"2024-01-16T15:00:49.073827+0000","last_became_active":"2024-01-16T15:00:48.062867+0000","last_became_peered":"2024-01-16T15:00:48.062867+0000","last_unstale":"2024-01-16T15:00:49.073827+0000","last_undegraded":"2024-01-16T15:00:49.073827+0000","last_fullsized":"2024-01-16T15:00:49.073827+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:42.485766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074922+0000","last_change":"2024-01-16T15:00:49.072810+0000","last_active":"2024-01-16T15:00:49.074922+0000","last_peered":"2024-01-16T15:00:49.074922+0000","last_clean":"2024-01-16T15:00:49.074922+0000","last_became_active":"2024-01-16T15:00:49.072509+0000","last_became_peered":"2024-01-16T15:00:49.072509+0000","last_unstale":"2024-01-16T15:00:49.074922+0000","last_undegraded":"2024-01-16T15:00:49.074922+0000","last_fullsized":"2024-01-16T15:00:49.074922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:38.202653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063637+0000","last_change":"2024-01-16T15:00:48.061152+0000","last_active":"2024-01-16T15:00:49.063637+0000","last_peered":"2024-01-16T15:00:49.063637+0000","last_clean":"2024-01-16T15:00:49.063637+0000","last_became_active":"2024-01-16T15:00:48.060236+0000","last_became_peered":"2024-01-16T15:00:48.060236+0000","last_unstale":"2024-01-16T15:00:49.063637+0000","last_undegraded":"2024-01-16T15:00:49.063637+0000","last_fullsized":"2024-01-16T15:00:49.063637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:29.850713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.105181+0000","last_change":"2024-01-16T15:00:49.100265+0000","last_active":"2024-01-16T15:00:49.105181+0000","last_peered":"2024-01-16T15:00:49.105181+0000","last_clean":"2024-01-16T15:00:49.105181+0000","last_became_active":"2024-01-16T15:00:49.099997+0000","last_became_peered":"2024-01-16T15:00:49.099997+0000","last_unstale":"2024-01-16T15:00:49.105181+0000","last_undegraded":"2024-01-16T15:00:49.105181+0000","last_fullsized":"2024-01-16T15:00:49.105181+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:12.606550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074574+0000","last_change":"2024-01-16T15:00:48.067724+0000","last_active":"2024-01-16T15:00:49.074574+0000","last_peered":"2024-01-16T15:00:49.074574+0000","last_clean":"2024-01-16T15:00:49.074574+0000","last_became_active":"2024-01-16T15:00:48.067080+0000","last_became_peered":"2024-01-16T15:00:48.067080+0000","last_unstale":"2024-01-16T15:00:49.074574+0000","last_undegraded":"2024-01-16T15:00:49.074574+0000","last_fullsized":"2024-01-16T15:00:49.074574+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:53.586410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.100089+0000","last_change":"2024-01-16T15:00:49.095669+0000","last_active":"2024-01-16T15:00:49.100089+0000","last_peered":"2024-01-16T15:00:49.100089+0000","last_clean":"2024-01-16T15:00:49.100089+0000","last_became_active":"2024-01-16T15:00:49.095408+0000","last_became_peered":"2024-01-16T15:00:49.095408+0000","last_unstale":"2024-01-16T15:00:49.100089+0000","last_undegraded":"2024-01-16T15:00:49.100089+0000","last_fullsized":"2024-01-16T15:00:49.100089+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:25.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.099262+0000","last_change":"2024-01-16T15:00:48.066203+0000","last_active":"2024-01-16T15:00:49.099262+0000","last_peered":"2024-01-16T15:00:49.099262+0000","last_clean":"2024-01-16T15:00:49.099262+0000","last_became_active":"2024-01-16T15:00:48.065878+0000","last_became_peered":"2024-01-16T15:00:48.065878+0000","last_unstale":"2024-01-16T15:00:49.099262+0000","last_undegraded":"2024-01-16T15:00:49.099262+0000","last_fullsized":"2024-01-16T15:00:49.099262+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:47.601573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.102550+0000","last_change":"2024-01-16T15:00:49.100710+0000","last_active":"2024-01-16T15:00:49.102550+0000","last_peered":"2024-01-16T15:00:49.102550+0000","last_clean":"2024-01-16T15:00:49.102550+0000","last_became_active":"2024-01-16T15:00:49.099074+0000","last_became_peered":"2024-01-16T15:00:49.099074+0000","last_unstale":"2024-01-16T15:00:49.102550+0000","last_undegraded":"2024-01-16T15:00:49.102550+0000","last_fullsized":"2024-01-16T15:00:49.102550+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:24.205886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.062227+0000","last_change":"2024-01-16T15:00:48.050860+0000","last_active":"2024-01-16T15:00:49.062227+0000","last_peered":"2024-01-16T15:00:49.062227+0000","last_clean":"2024-01-16T15:00:49.062227+0000","last_became_active":"2024-01-16T15:00:48.050522+0000","last_became_peered":"2024-01-16T15:00:48.050522+0000","last_unstale":"2024-01-16T15:00:49.062227+0000","last_undegraded":"2024-01-16T15:00:49.062227+0000","last_fullsized":"2024-01-16T15:00:49.062227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:15.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.090494+0000","last_change":"2024-01-16T15:00:49.085468+0000","last_active":"2024-01-16T15:00:49.090494+0000","last_peered":"2024-01-16T15:00:49.090494+0000","last_clean":"2024-01-16T15:00:49.090494+0000","last_became_active":"2024-01-16T15:00:49.085162+0000","last_became_peered":"2024-01-16T15:00:49.085162+0000","last_unstale":"2024-01-16T15:00:49.090494+0000","last_undegraded":"2024-01-16T15:00:49.090494+0000","last_fullsized":"2024-01-16T15:00:49.090494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:45.617799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.068653+0000","last_change":"2024-01-16T15:00:48.052504+0000","last_active":"2024-01-16T15:00:49.068653+0000","last_peered":"2024-01-16T15:00:49.068653+0000","last_clean":"2024-01-16T15:00:49.068653+0000","last_became_active":"2024-01-16T15:00:48.050557+0000","last_became_peered":"2024-01-16T15:00:48.050557+0000","last_unstale":"2024-01-16T15:00:49.068653+0000","last_undegraded":"2024-01-16T15:00:49.068653+0000","last_fullsized":"2024-01-16T15:00:49.068653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:53.586410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.094726+0000","last_change":"2024-01-16T15:00:49.088379+0000","last_active":"2024-01-16T15:00:49.094726+0000","last_peered":"2024-01-16T15:00:49.094726+0000","last_clean":"2024-01-16T15:00:49.094726+0000","last_became_active":"2024-01-16T15:00:49.088098+0000","last_became_peered":"2024-01-16T15:00:49.088098+0000","last_unstale":"2024-01-16T15:00:49.094726+0000","last_undegraded":"2024-01-16T15:00:49.094726+0000","last_fullsized":"2024-01-16T15:00:49.094726+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:16.352542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.059060+0000","last_change":"2024-01-16T15:00:48.057344+0000","last_active":"2024-01-16T15:00:49.059060+0000","last_peered":"2024-01-16T15:00:49.059060+0000","last_clean":"2024-01-16T15:00:49.059060+0000","last_became_active":"2024-01-16T15:00:48.057066+0000","last_became_peered":"2024-01-16T15:00:48.057066+0000","last_unstale":"2024-01-16T15:00:49.059060+0000","last_undegraded":"2024-01-16T15:00:49.059060+0000","last_fullsized":"2024-01-16T15:00:49.059060+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092977+0000","last_change":"2024-01-16T15:00:49.091351+0000","last_active":"2024-01-16T15:00:49.092977+0000","last_peered":"2024-01-16T15:00:49.092977+0000","last_clean":"2024-01-16T15:00:49.092977+0000","last_became_active":"2024-01-16T15:00:49.091066+0000","last_became_peered":"2024-01-16T15:00:49.091066+0000","last_unstale":"2024-01-16T15:00:49.092977+0000","last_undegraded":"2024-01-16T15:00:49.092977+0000","last_fullsized":"2024-01-16T15:00:49.092977+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:47.209930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.064673+0000","last_change":"2024-01-16T15:00:48.061362+0000","last_active":"2024-01-16T15:00:49.064673+0000","last_peered":"2024-01-16T15:00:49.064673+0000","last_clean":"2024-01-16T15:00:49.064673+0000","last_became_active":"2024-01-16T15:00:48.061080+0000","last_became_peered":"2024-01-16T15:00:48.061080+0000","last_unstale":"2024-01-16T15:00:49.064673+0000","last_undegraded":"2024-01-16T15:00:49.064673+0000","last_fullsized":"2024-01-16T15:00:49.064673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.092686+0000","last_change":"2024-01-16T15:00:49.088755+0000","last_active":"2024-01-16T15:00:49.092686+0000","last_peered":"2024-01-16T15:00:49.092686+0000","last_clean":"2024-01-16T15:00:49.092686+0000","last_became_active":"2024-01-16T15:00:49.088485+0000","last_became_peered":"2024-01-16T15:00:49.088485+0000","last_unstale":"2024-01-16T15:00:49.092686+0000","last_undegraded":"2024-01-16T15:00:49.092686+0000","last_fullsized":"2024-01-16T15:00:49.092686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:00.145351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.063979+0000","last_change":"2024-01-16T15:00:48.065908+0000","last_active":"2024-01-16T15:00:49.063979+0000","last_peered":"2024-01-16T15:00:49.063979+0000","last_clean":"2024-01-16T15:00:49.063979+0000","last_became_active":"2024-01-16T15:00:48.065591+0000","last_became_peered":"2024-01-16T15:00:48.065591+0000","last_unstale":"2024-01-16T15:00:49.063979+0000","last_undegraded":"2024-01-16T15:00:49.063979+0000","last_fullsized":"2024-01-16T15:00:49.063979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:46.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.085353+0000","last_change":"2024-01-16T15:00:49.075674+0000","last_active":"2024-01-16T15:00:49.085353+0000","last_peered":"2024-01-16T15:00:49.085353+0000","last_clean":"2024-01-16T15:00:49.085353+0000","last_became_active":"2024-01-16T15:00:49.074974+0000","last_became_peered":"2024-01-16T15:00:49.074974+0000","last_unstale":"2024-01-16T15:00:49.085353+0000","last_undegraded":"2024-01-16T15:00:49.085353+0000","last_fullsized":"2024-01-16T15:00:49.085353+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.345260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.065341+0000","last_change":"2024-01-16T15:00:48.063259+0000","last_active":"2024-01-16T15:00:49.065341+0000","last_peered":"2024-01-16T15:00:49.065341+0000","last_clean":"2024-01-16T15:00:49.065341+0000","last_became_active":"2024-01-16T15:00:48.062718+0000","last_became_peered":"2024-01-16T15:00:48.062718+0000","last_unstale":"2024-01-16T15:00:49.065341+0000","last_undegraded":"2024-01-16T15:00:49.065341+0000","last_fullsized":"2024-01-16T15:00:49.065341+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:04.399308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.093379+0000","last_change":"2024-01-16T15:00:49.087595+0000","last_active":"2024-01-16T15:00:49.093379+0000","last_peered":"2024-01-16T15:00:49.093379+0000","last_clean":"2024-01-16T15:00:49.093379+0000","last_became_active":"2024-01-16T15:00:49.086730+0000","last_became_peered":"2024-01-16T15:00:49.086730+0000","last_unstale":"2024-01-16T15:00:49.093379+0000","last_undegraded":"2024-01-16T15:00:49.093379+0000","last_fullsized":"2024-01-16T15:00:49.093379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:48.021241+0000","last_clean_scrub_stamp":"2024-01-16T15:00:48.021241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:46:47.690663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:00:49.074757+0000","last_change":"2024-01-16T15:00:48.063925+0000","last_active":"2024-01-16T15:00:49.074757+0000","last_peered":"2024-01-16T15:00:49.074757+0000","last_clean":"2024-01-16T15:00:49.074757+0000","last_became_active":"2024-01-16T15:00:48.063431+0000","last_became_peered":"2024-01-16T15:00:48.063431+0000","last_unstale":"2024-01-16T15:00:49.074757+0000","last_undegraded":"2024-01-16T15:00:49.074757+0000","last_fullsized":"2024-01-16T15:00:49.074757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:00:47.016549+0000","last_clean_scrub_stamp":"2024-01-16T15:00:47.016549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:21.898481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T15:00:48.061376+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:00:48.061376+0000","last_peered":"2024-01-16T15:00:48.061376+0000","last_clean":"2024-01-16T15:00:48.061376+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:00:48.061376+0000","last_undegraded":"2024-01-16T15:00:48.061376+0000","last_fullsized":"2024-01-16T15:00:48.061376+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":738634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.216},{"interface":"front","average":{"1min":1.204,"5min":1.204,"15min":1.204},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.2490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.7400000000000002,"5min":3.7400000000000002,"15min":3.7400000000000002},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":0.96999999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.536},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.353}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":3.8450000000000002,"5min":3.8450000000000002,"15min":3.8450000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.431}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":1.04},{"interface":"front","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":1.292}]},{"osd":5,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":3.4670000000000001},"last":1.619},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":1.573}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.143,"15min":1.143},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.2569999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.472,"5min":2.472,"15min":2.472},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.84599999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.7480000000000002,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.1910000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":0.92700000000000005}]},{"osd":5,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.464},{"interface":"front","average":{"1min":1.607,"5min":1.607,"15min":1.607},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":3.1389999999999998,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.4139999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.552}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":1.204},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":1.623},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":1.379}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.421},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":1.343}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.383,"15min":1.383},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":3.2909999999999999,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":1.274},{"interface":"front","average":{"1min":1.47,"5min":1.47,"15min":1.47},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.514}]},{"osd":7,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.474},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":3.3610000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":1.671}]}]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98599999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.429},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.296}]},{"osd":2,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":1.349}]},{"osd":3,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.494}]},{"osd":5,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.8650000000000002,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.591},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.649}]},{"osd":6,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.401},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.5289999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":1.198,"5min":1.198,"15min":1.198},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.607}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.3240000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.04},{"interface":"front","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.244}]},{"osd":4,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.141},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.605,"5min":2.605,"15min":2.605},"last":1.2849999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.423},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":1.591}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":385024,"data_stored":148246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":0.96099999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":4.6769999999999996,"5min":4.6769999999999996,"15min":4.6769999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.637}]},{"osd":4,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.6120000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.429},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.4450000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":1.1599999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":1.224},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.577}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":149810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.9820000000000002,"5min":2.9820000000000002,"15min":2.9820000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":3.649,"5min":3.649,"15min":3.649},"last":0.84899999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":1.204},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":3.5619999999999998,"5min":3.5619999999999998,"15min":3.5619999999999998},"last":1.3220000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":3.722,"5min":3.722,"15min":3.722},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":3.625,"5min":3.625,"15min":3.625},"last":1.7470000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.579}]},{"osd":6,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":3.4750000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":3.6840000000000002,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":1.7809999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:19 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.401},{"interface":"front","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.5390000000000001,"5min":3.5390000000000001,"15min":3.5390000000000001},"last":1.268}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":738544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.625}]},{"osd":2,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.77200000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.204}]},{"osd":5,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.72199999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.0589999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":1.014,"5min":1.014,"15min":1.014},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.123},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.03}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:54.062 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:00:54.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:00:54.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.385+0000 7feb6fa00640 1 Processor -- start 2024-01-16T15:00:54.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.386+0000 7feb6fa00640 1 -- start start 2024-01-16T15:00:54.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb6fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb680d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb6fa00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6815d720 con 0x7feb68070af0 2024-01-16T15:00:54.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb6fa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6806d080 con 0x7feb6815d340 2024-01-16T15:00:54.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb6fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb680d0d90 con 0x7feb680ce150 2024-01-16T15:00:54.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb67e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7feb68070af0 0x7feb680cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33238/0 (socket says 172.21.15.43:33238) 2024-01-16T15:00:54.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb67e00640 1 -- 172.21.15.43:0/3509639612 learned_addr learned my addr 172.21.15.43:0/3509639612 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.388+0000 7feb6da00640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb680d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323911428 0 0) 0x7feb6815d720 con 0x7feb68070af0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb48003520 con 0x7feb68070af0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 482472118 0 0) 0x7feb48003520 con 0x7feb68070af0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 msgr2=0x7feb680d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb680d0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 >> v1:172.21.15.69:6789/0 conn(0x7feb6815d340 legacy=0x7feb680cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6806d360 con 0x7feb68070af0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7feb50001670 con 0x7feb68070af0 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 >> v1:172.21.15.43:6789/0 conn(0x7feb68070af0 legacy=0x7feb680cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb48004d00 0x7feb48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680d0d90 con 0x7feb480039a0 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4800bb70 con 0x7feb48004d00 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.389+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4800bd30 con 0x7feb680ce150 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb67e00640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6da00640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb48004d00 0x7feb48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 -- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 msgr2=0x7feb480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680ce150 0x7feb480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 -- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb48004d00 msgr2=0x7feb48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb48004d00 0x7feb48005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.390+0000 7feb6d000640 1 -- 172.21.15.43:0/3509639612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb48003520 con 0x7feb480039a0 2024-01-16T15:00:54.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.391+0000 7feb6d000640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb48003dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7feb58001750 tx=0x7feb480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53309ce5ec58686a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.391+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb580052d0 con 0x7feb480039a0 2024-01-16T15:00:54.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.391+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5802e050 con 0x7feb480039a0 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.391+0000 7feb67400640 1 -- 172.21.15.43:0/3509639612 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb580035f0 con 0x7feb480039a0 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 -- 172.21.15.43:0/3509639612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 msgr2=0x7feb48003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 --2- 172.21.15.43:0/3509639612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb48003dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7feb58001750 tx=0x7feb480057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6da00640 1 -- 172.21.15.43:0/3509639612 reap_dead start 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 -- 172.21.15.43:0/3509639612 shutdown_connections 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 -- 172.21.15.43:0/3509639612 >> 172.21.15.43:0/3509639612 conn(0x7feb680b2ba0 msgr2=0x7feb680b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 -- 172.21.15.43:0/3509639612 shutdown_connections 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.392+0000 7feb6fa00640 1 -- 172.21.15.43:0/3509639612 wait complete. 2024-01-16T15:00:54.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.394+0000 7feb6fa00640 1 Processor -- start 2024-01-16T15:00:54.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.394+0000 7feb6fa00640 1 -- start start 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680acab0 0x7feb680acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb680b2060 0x7feb680b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680c8fd0 con 0x7feb480039a0 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680b2a20 con 0x7feb680b2060 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb680c8100 con 0x7feb680acab0 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51904/0 (socket says 172.21.15.43:51904) 2024-01-16T15:00:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.395+0000 7feb6d000640 1 -- 172.21.15.43:0/1174449023 learned_addr learned my addr 172.21.15.43:0/1174449023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:54.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6da00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb680b2060 0x7feb680b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb67e00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680acab0 0x7feb680acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 -- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680acab0 msgr2=0x7feb680acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680acab0 0x7feb680acf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 -- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb680b2060 msgr2=0x7feb680b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb680b2060 0x7feb680b24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 -- 172.21.15.43:0/1174449023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb680c8350 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.396+0000 7feb6d000640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feb58008fd0 tx=0x7feb58001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.397+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58007a30 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.397+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb58007640 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.397+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58006c20 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.397+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb680c8630 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.397+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb680c8bf0 con 0x7feb480039a0 2024-01-16T15:00:54.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.398+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb680c8fd0 con 0x7feb480039a0 2024-01-16T15:00:54.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.399+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb580035f0 con 0x7feb480039a0 2024-01-16T15:00:54.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.400+0000 7feb64c00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 0x7feb4006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.400+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb580ab8d0 con 0x7feb480039a0 2024-01-16T15:00:54.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.400+0000 7feb67e00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 0x7feb4006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.401+0000 7feb67e00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 0x7feb4006d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb50001c90 tx=0x7feb50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.404+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb5807a250 con 0x7feb480039a0 2024-01-16T15:00:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.664+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feb68160c40 con 0x7feb480039a0 2024-01-16T15:00:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.669+0000 7feb64c00640 1 -- 172.21.15.43:0/1174449023 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7feb5804b0e0 con 0x7feb480039a0 2024-01-16T15:00:54.649 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:00:54.649 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:00:54.649 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:00:54.649 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:00:54.650 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:00:54.650 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:00:54.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.671+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 msgr2=0x7feb4006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.671+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 0x7feb4006d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb50001c90 tx=0x7feb50009040 comp rx=0 tx=0).stop 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.671+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 msgr2=0x7feb681595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.671+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feb58008fd0 tx=0x7feb58001120 comp rx=0 tx=0).stop 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.671+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 shutdown_connections 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb4006b2c0 0x7feb4006d780 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb680b2060 0x7feb680b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb680acab0 0x7feb680acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 --2- 172.21.15.43:0/1174449023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb480039a0 0x7feb681595f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 >> 172.21.15.43:0/1174449023 conn(0x7feb680b2ba0 msgr2=0x7feb6815f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 shutdown_connections 2024-01-16T15:00:54.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.672+0000 7feb6fa00640 1 -- 172.21.15.43:0/1174449023 wait complete. 2024-01-16T15:00:54.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:54.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.914+0000 7ff8ad200640 1 Processor -- start 2024-01-16T15:00:54.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.915+0000 7ff8ad200640 1 -- start start 2024-01-16T15:00:54.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.916+0000 7ff8ad200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 0x7ff8a806a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.916+0000 7ff8ad200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.916+0000 7ff8ad200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a806aff0 con 0x7ff8a815d4a0 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.916+0000 7ff8ad200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a815d880 con 0x7ff8a806ab40 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.916+0000 7ff8ad200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a806fba0 con 0x7ff8a80af890 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.917+0000 7ff8a6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.917+0000 7ff8a5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 0x7ff8a806a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.917+0000 7ff8a6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:48580/0 (socket says 172.21.15.43:48580) 2024-01-16T15:00:54.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.917+0000 7ff8a6800640 1 -- 172.21.15.43:0/4231371792 learned_addr learned my addr 172.21.15.43:0/4231371792 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:54.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.917+0000 7ff8a5e00640 1 --1- 172.21.15.43:0/4231371792 >> v1:172.21.15.69:6789/0 conn(0x7ff8a815d4a0 0x7ff8a80af180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50042/0 (socket says 172.21.15.43:50042) 2024-01-16T15:00:54.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a4a00640 1 -- 172.21.15.43:0/4231371792 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882534847 0 0) 0x7ff898001110 con 0x7ff8a815d4a0 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a4a00640 1 -- 172.21.15.43:0/4231371792 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888003520 con 0x7ff8a815d4a0 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a5400640 1 -- 172.21.15.43:0/4231371792 >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 msgr2=0x7ff8a80b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a5400640 1 --2- 172.21.15.43:0/4231371792 >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a5400640 1 -- 172.21.15.43:0/4231371792 >> v1:172.21.15.69:6789/0 conn(0x7ff8a815d4a0 legacy=0x7ff8a80af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a5400640 1 -- 172.21.15.43:0/4231371792 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a806eb00 con 0x7ff8a80af890 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.918+0000 7ff8a6800640 1 --2- 172.21.15.43:0/4231371792 >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.919+0000 7ff8a5400640 1 --2- 172.21.15.43:0/4231371792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 0x7ff8a806a600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8900097a0 tx=0x7ff89002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f744783c0355451d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8a4a00640 1 -- 172.21.15.43:0/4231371792 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89002d040 con 0x7ff8a80af890 2024-01-16T15:00:54.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8a4a00640 1 -- 172.21.15.43:0/4231371792 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff890031810 con 0x7ff8a80af890 2024-01-16T15:00:54.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8a4a00640 1 -- 172.21.15.43:0/4231371792 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890031ba0 con 0x7ff8a80af890 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8ad200640 1 -- 172.21.15.43:0/4231371792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 msgr2=0x7ff8a806a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4231371792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 0x7ff8a806a600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8900097a0 tx=0x7ff89002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8ad200640 1 -- 172.21.15.43:0/4231371792 shutdown_connections 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4231371792 >> v2:172.21.15.69:3300/0 conn(0x7ff8a806ab40 0x7ff8a80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.920+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4231371792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a80af890 0x7ff8a806a600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.921+0000 7ff8ad200640 1 -- 172.21.15.43:0/4231371792 >> 172.21.15.43:0/4231371792 conn(0x7ff8a806d490 msgr2=0x7ff8a806d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.921+0000 7ff8ad200640 1 -- 172.21.15.43:0/4231371792 shutdown_connections 2024-01-16T15:00:54.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.921+0000 7ff8ad200640 1 -- 172.21.15.43:0/4231371792 wait complete. 2024-01-16T15:00:54.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.922+0000 7ff8ad200640 1 Processor -- start 2024-01-16T15:00:54.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 -- start start 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a806ab40 0x7ff8a80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 0x7ff8a815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815c670 con 0x7ff8a815d4a0 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80cf320 con 0x7ff8a80af890 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.923+0000 7ff8ad200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815bbc0 con 0x7ff8a806ab40 2024-01-16T15:00:54.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51914/0 (socket says 172.21.15.43:51914) 2024-01-16T15:00:54.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 -- 172.21.15.43:0/4196868426 learned_addr learned my addr 172.21.15.43:0/4196868426 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a5e00640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a806ab40 0x7ff8a80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a5400640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 0x7ff8a815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 -- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a806ab40 msgr2=0x7ff8a80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a806ab40 0x7ff8a80cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 -- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 msgr2=0x7ff8a815b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 0x7ff8a815b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a6800640 1 -- 172.21.15.43:0/4196868426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0009b30 con 0x7ff8a815d4a0 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.924+0000 7ff8a5400640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 0x7ff8a815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff8a6800640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000efc0 tx=0x7ff8a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0018070 con 0x7ff8a815d4a0 2024-01-16T15:00:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a000f040 con 0x7ff8a815d4a0 2024-01-16T15:00:54.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000ce00 con 0x7ff8a815d4a0 2024-01-16T15:00:54.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff890009400 con 0x7ff8a815d4a0 2024-01-16T15:00:54.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.925+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a814af00 con 0x7ff8a815d4a0 2024-01-16T15:00:54.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.926+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a8074770 con 0x7ff8a815d4a0 2024-01-16T15:00:54.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.927+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff8a0014480 con 0x7ff8a815d4a0 2024-01-16T15:00:54.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.928+0000 7ff89e000640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 0x7ff88006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.928+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8a0089fa0 con 0x7ff8a815d4a0 2024-01-16T15:00:54.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.928+0000 7ff8a5e00640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 0x7ff88006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.929+0000 7ff8a5e00640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 0x7ff88006d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff898001370 tx=0x7ff8980079f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:54.933+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8a0054e00 con 0x7ff8a815d4a0 2024-01-16T15:00:55.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.169+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8a806b190 con 0x7ff8a815d4a0 2024-01-16T15:00:55.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.170+0000 7ff89e000640 1 -- 172.21.15.43:0/4196868426 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4901 (secure 0 0 0) 0x7ff8a0058420 con 0x7ff8a815d4a0 2024-01-16T15:00:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:00:55.150 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4691,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/543136239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":543136239},{"type":"v1","addr":"172.21.15.69:6833","nonce":543136239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4939,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/198308009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":198308009},{"type":"v1","addr":"172.21.15.69:6835","nonce":198308009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/81338476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":81338476},{"type":"v1","addr":"172.21.15.43:6835","nonce":81338476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:00:48.505147+0000","modified":"2024-01-16T15:00:51.582617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6837/3759191221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3759191221},{"type":"v1","addr":"172.21.15.43:6837","nonce":3759191221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T15:00:55.151 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-01-16T15:00:55.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.172+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 msgr2=0x7ff88006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.172+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 0x7ff88006d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff898001370 tx=0x7ff8980079f0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.172+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 msgr2=0x7ff8a814a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.172+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000efc0 tx=0x7ff8a000c490 comp rx=0 tx=0).stop 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 shutdown_connections 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff88006b2c0 0x7ff88006d780 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8a80af890 0x7ff8a815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8a815d4a0 0x7ff8a814a9c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 --2- 172.21.15.43:0/4196868426 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8a806ab40 0x7ff8a80cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 >> 172.21.15.43:0/4196868426 conn(0x7ff8a806d490 msgr2=0x7ff8a80ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 shutdown_connections 2024-01-16T15:00:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.173+0000 7ff8ad200640 1 -- 172.21.15.43:0/4196868426 wait complete. 2024-01-16T15:00:55.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:00:48.505147+0000', 'modified': '2024-01-16T15:00:51.582617+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5250': {'gid': 5250, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.43:6837/3759191221', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 3759191221}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 3759191221}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:00:55.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:00:55.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:00:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.827+0000 7fbff4600640 1 Processor -- start 2024-01-16T15:00:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.827+0000 7fbff4600640 1 -- start start 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff4600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfec146890 0x7fbfec06c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff4600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbfec146c70 0x7fbfec0b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff4600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0b3b60 con 0x7fbfec0b3710 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff4600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec161ec0 con 0x7fbfec146c70 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff4600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec161be0 con 0x7fbfec146890 2024-01-16T15:00:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.829+0000 7fbff1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbfec146c70 0x7fbfec0b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbfec146c70 0x7fbfec0b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51924/0 (socket says 172.21.15.43:51924) 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 -- 172.21.15.43:0/2681707580 learned_addr learned my addr 172.21.15.43:0/2681707580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1c00640 1 --2- 172.21.15.43:0/2681707580 >> v2:172.21.15.69:3300/0 conn(0x7fbfec146890 0x7fbfec06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 -- 172.21.15.43:0/2681707580 >> v2:172.21.15.69:3300/0 conn(0x7fbfec146890 msgr2=0x7fbfec06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 --2- 172.21.15.43:0/2681707580 >> v2:172.21.15.69:3300/0 conn(0x7fbfec146890 0x7fbfec06c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 -- 172.21.15.43:0/2681707580 >> v1:172.21.15.69:6789/0 conn(0x7fbfec0b3710 legacy=0x7fbfec15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.830+0000 7fbff1200640 1 -- 172.21.15.43:0/2681707580 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec06bb30 con 0x7fbfec146c70 2024-01-16T15:00:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.831+0000 7fbff1200640 1 --2- 172.21.15.43:0/2681707580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec146c70 0x7fbfec0b31d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0692a0 tx=0x7fbfd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae8f3331bd2083 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.831+0000 7fbfebe00640 1 -- 172.21.15.43:0/2681707580 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4031870 con 0x7fbfec146c70 2024-01-16T15:00:55.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbfebe00640 1 -- 172.21.15.43:0/2681707580 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd40319d0 con 0x7fbfec146c70 2024-01-16T15:00:55.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbfebe00640 1 -- 172.21.15.43:0/2681707580 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4031d60 con 0x7fbfec146c70 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 -- 172.21.15.43:0/2681707580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec146c70 msgr2=0x7fbfec0b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 --2- 172.21.15.43:0/2681707580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec146c70 0x7fbfec0b31d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0692a0 tx=0x7fbfd402b910 comp rx=0 tx=0).stop 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 -- 172.21.15.43:0/2681707580 shutdown_connections 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 --2- 172.21.15.43:0/2681707580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec146c70 0x7fbfec0b31d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 --2- 172.21.15.43:0/2681707580 >> v2:172.21.15.69:3300/0 conn(0x7fbfec146890 0x7fbfec06c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 -- 172.21.15.43:0/2681707580 >> 172.21.15.43:0/2681707580 conn(0x7fbfec066610 msgr2=0x7fbfec066a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.832+0000 7fbff4600640 1 -- 172.21.15.43:0/2681707580 shutdown_connections 2024-01-16T15:00:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.833+0000 7fbff4600640 1 -- 172.21.15.43:0/2681707580 wait complete. 2024-01-16T15:00:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.834+0000 7fbff4600640 1 Processor -- start 2024-01-16T15:00:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.834+0000 7fbff4600640 1 -- start start 2024-01-16T15:00:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfec146890 0x7fbfec142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfec146c70 0x7fbfec1e8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1432c0 con 0x7fbfec0b3710 2024-01-16T15:00:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0ceac0 con 0x7fbfec146c70 2024-01-16T15:00:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.835+0000 7fbff4600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0d0bf0 con 0x7fbfec146890 2024-01-16T15:00:55.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51928/0 (socket says 172.21.15.43:51928) 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 -- 172.21.15.43:0/3470766110 learned_addr learned my addr 172.21.15.43:0/3470766110 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff2600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfec146c70 0x7fbfec1e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 -- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfec146890 msgr2=0x7fbfec142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1200640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfec146890 0x7fbfec142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfec146890 0x7fbfec142d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 -- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfec146c70 msgr2=0x7fbfec1e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfec146c70 0x7fbfec1e8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 -- 172.21.15.43:0/3470766110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40094d0 con 0x7fbfec0b3710 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.836+0000 7fbff1c00640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc007ef0 tx=0x7fbfdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.837+0000 7fbfe9600640 1 -- 172.21.15.43:0/3470766110 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc014620 con 0x7fbfec0b3710 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.837+0000 7fbfe9600640 1 -- 172.21.15.43:0/3470766110 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc014780 con 0x7fbfec0b3710 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.837+0000 7fbfe9600640 1 -- 172.21.15.43:0/3470766110 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc014b30 con 0x7fbfec0b3710 2024-01-16T15:00:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.837+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec0d0e90 con 0x7fbfec0b3710 2024-01-16T15:00:55.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.837+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec0d1450 con 0x7fbfec0b3710 2024-01-16T15:00:55.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.839+0000 7fbfe9600640 1 -- 172.21.15.43:0/3470766110 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbfdc014c90 con 0x7fbfec0b3710 2024-01-16T15:00:55.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.840+0000 7fbfe9600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 0x7fbfc406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.840+0000 7fbfe9600640 1 -- 172.21.15.43:0/3470766110 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfdc08a690 con 0x7fbfec0b3710 2024-01-16T15:00:55.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.841+0000 7fbff1200640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 0x7fbfc406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.841+0000 7fbff7196740 1 Processor -- start 2024-01-16T15:00:55.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.841+0000 7fbff7196740 1 -- start start 2024-01-16T15:00:55.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.841+0000 7fbff1200640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 0x7fbfc406d7b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4036000 tx=0x7fbfd403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864d3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b86516560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b86431880 con 0x564b864cdba0 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b86440ae0 con 0x564b864d25e0 2024-01-16T15:00:55.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b8643e1d0 con 0x564b864d2ea0 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51934/0 (socket says 172.21.15.43:51934) 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.842+0000 7fbff1200640 1 -- 172.21.15.43:0/4280233270 learned_addr learned my addr 172.21.15.43:0/4280233270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff2600640 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b86516560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1c00640 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864d3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 -- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 msgr2=0x564b864d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864d3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 -- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 msgr2=0x564b86516560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b86516560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 -- 172.21.15.43:0/4280233270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b86401860 con 0x564b864cdba0 2024-01-16T15:00:55.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbff1200640 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4031c40 tx=0x7fbfd4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b041fc8e37526d08 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbfe2000640 1 -- 172.21.15.43:0/4280233270 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4031c70 con 0x564b864cdba0 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbfe2000640 1 -- 172.21.15.43:0/4280233270 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4031dd0 con 0x564b864cdba0 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.843+0000 7fbfe2000640 1 -- 172.21.15.43:0/4280233270 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd404d400 con 0x564b864cdba0 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 -- 172.21.15.43:0/4280233270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 msgr2=0x564b864d20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4031c40 tx=0x7fbfd4030000 comp rx=0 tx=0).stop 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 -- 172.21.15.43:0/4280233270 shutdown_connections 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b86516560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864cdba0 0x564b864d20a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 --2- 172.21.15.43:0/4280233270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864d3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 -- 172.21.15.43:0/4280233270 >> 172.21.15.43:0/4280233270 conn(0x564b86494c70 msgr2=0x564b86518990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 -- 172.21.15.43:0/4280233270 shutdown_connections 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.844+0000 7fbff7196740 1 -- 172.21.15.43:0/4280233270 wait complete. 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.845+0000 7fbff7196740 1 -- start start 2024-01-16T15:00:55.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.845+0000 7fbff7196740 1 Processor -- start 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b864f5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b86439ac0 con 0x564b864d25e0 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b863d62f0 con 0x564b864cdba0 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b863f3c70 con 0x564b864d2ea0 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.846+0000 7fbff7196740 10 client.0 _ll_register_callbacks cb 0x564b863f83b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51946/0 (socket says 172.21.15.43:51946) 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 -- 172.21.15.43:0/2202951788 learned_addr learned my addr 172.21.15.43:0/2202951788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 -- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 msgr2=0x564b864f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 -- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 msgr2=0x564b864f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b864f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 -- 172.21.15.43:0/2202951788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec06bb30 con 0x564b864d25e0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbff1200640 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbfd404f040 tx=0x7fbfd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1273320e4840e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.847+0000 7fbfdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4031c70 con 0x564b864d25e0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfd402d040 con 0x564b864d25e0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b86432aa0 con 0x564b864d25e0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfd40044f0 con 0x564b864d25e0 2024-01-16T15:00:55.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 10 client.5412 fetch_fsmap learned FSMap version 17 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564b86438140 con 0x564b864d25e0 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564b86432aa0 con 0x564b864d25e0 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbfd4003c50 con 0x564b864d25e0 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 10 client.5412 fetch_fsmap finished waiting for FSMap version 17 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 10 client.5412 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 20 client.5412 populate_metadata read hostname 'smithi043' 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b864f2310 0x564b86535ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 4 client.5412 mds_command: new command op to 5250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.848+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 --> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b864327c0 con 0x564b864f2310 2024-01-16T15:00:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.849+0000 7fbff1c00640 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b864f2310 0x564b86535ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.849+0000 7fbff1c00640 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b864f2310 0x564b86535ba0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f04cf814d9390fe server_cookie=f844bb0aef98416 in_seq=0 out_seq=0 2024-01-16T15:00:55.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.849+0000 7fbfdaa00640 10 client.5412 ms_handle_connect on v2:172.21.15.43:6836/3759191221 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbfdaa00640 1 -- 172.21.15.43:0/2202951788 <== mds.0 v2:172.21.15.43:6836/3759191221 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564b864327c0 con 0x564b864f2310 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbfdaa00640 10 client.5412 handle_command_reply: tid=0 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 client.5412 shutdown 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 msgr2=0x564b8651a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbfd404f040 tx=0x7fbfd4004730 comp rx=0 tx=0).stop 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 shutdown_connections 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b864f2310 0x564b86535ba0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d2ea0 0x564b864f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d25e0 0x564b8651a0b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 --2- 172.21.15.43:0/2202951788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b864f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbfdaa00640 0 client.5412 ms_handle_reset on v2:172.21.15.43:6836/3759191221 2024-01-16T15:00:55.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 >> 172.21.15.43:0/2202951788 conn(0x564b86494c70 msgr2=0x564b864ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 shutdown_connections 2024-01-16T15:00:55.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.850+0000 7fbff7196740 1 -- 172.21.15.43:0/2202951788 wait complete. 2024-01-16T15:00:55.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.851+0000 7fbff7196740 20 client.5412 trim_cache size 0 max 16384 2024-01-16T15:00:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.867+0000 7fbff7196740 1 Processor -- start 2024-01-16T15:00:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.867+0000 7fbff7196740 1 -- start start 2024-01-16T15:00:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864cdba0 0x564b86501790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b8651cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b8619cf20 con 0x564b864d2ea0 2024-01-16T15:00:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b864f33c0 con 0x564b864d25e0 2024-01-16T15:00:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.868+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b86532860 con 0x564b864cdba0 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51950/0 (socket says 172.21.15.43:51950) 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 -- 172.21.15.43:0/467689086 learned_addr learned my addr 172.21.15.43:0/467689086 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff1c00640 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864cdba0 0x564b86501790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff1200640 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b8651cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 -- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864cdba0 msgr2=0x564b86501790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864cdba0 0x564b86501790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 -- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 msgr2=0x564b8651cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b8651cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 -- 172.21.15.43:0/467689086 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b865196d0 con 0x564b864d2ea0 2024-01-16T15:00:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.869+0000 7fbff2600640 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4004b90 tx=0x7fbfe4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bc2192ab91b195 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbfe2a00640 1 -- 172.21.15.43:0/467689086 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400d070 con 0x564b864d2ea0 2024-01-16T15:00:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbfe2a00640 1 -- 172.21.15.43:0/467689086 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4007020 con 0x564b864d2ea0 2024-01-16T15:00:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbfe2a00640 1 -- 172.21.15.43:0/467689086 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40073d0 con 0x564b864d2ea0 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 -- 172.21.15.43:0/467689086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 msgr2=0x564b8651da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4004b90 tx=0x7fbfe4005050 comp rx=0 tx=0).stop 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 -- 172.21.15.43:0/467689086 shutdown_connections 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864d25e0 0x564b8651cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651da90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 --2- 172.21.15.43:0/467689086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864cdba0 0x564b86501790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 -- 172.21.15.43:0/467689086 >> 172.21.15.43:0/467689086 conn(0x564b86494c70 msgr2=0x564b864f1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.870+0000 7fbff7196740 1 -- 172.21.15.43:0/467689086 shutdown_connections 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.871+0000 7fbff7196740 1 -- 172.21.15.43:0/467689086 wait complete. 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.871+0000 7fbff7196740 1 -- start start 2024-01-16T15:00:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.871+0000 7fbff7196740 1 Processor -- start 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b8653a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d25e0 0x564b8651da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b864fa1b0 con 0x564b864d2ea0 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b86583960 con 0x564b864cdba0 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b864d13e0 con 0x564b864d25e0 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.872+0000 7fbff7196740 10 client.0 _ll_register_callbacks cb 0x564b864d3980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51952/0 (socket says 172.21.15.43:51952) 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 -- 172.21.15.43:0/1580771289 learned_addr learned my addr 172.21.15.43:0/1580771289 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff1c00640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b8653a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 -- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d25e0 msgr2=0x564b8651da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d25e0 0x564b8651da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 -- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 msgr2=0x564b8653a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b8653a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 -- 172.21.15.43:0/1580771289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe4004870 con 0x564b864d2ea0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbff2600640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4005620 tx=0x7fbfe4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5842a4d39dcfc59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.873+0000 7fbfdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4007660 con 0x564b864d2ea0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfe4006d90 con 0x564b864d2ea0 2024-01-16T15:00:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfd4031dd0 con 0x564b864d2ea0 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfe401d450 con 0x564b864d2ea0 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 10 client.5418 fetch_fsmap learned FSMap version 17 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564b863d5b80 con 0x564b864d2ea0 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4031dd0 con 0x564b864d2ea0 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbfe401d5b0 con 0x564b864d2ea0 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 10 client.5418 fetch_fsmap finished waiting for FSMap version 17 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 10 client.5418 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 20 client.5418 populate_metadata read hostname 'smithi043' 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b86582b10 0x564b86582ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 4 client.5418 mds_command: new command op to 5250 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.874+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 --> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564b8613a6c0 con 0x564b86582b10 2024-01-16T15:00:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbff1c00640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b86582b10 0x564b86582ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbff1c00640 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b86582b10 0x564b86582ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c665f965898741 server_cookie=d13f91df8408f03b in_seq=0 out_seq=0 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbfdaa00640 10 client.5418 ms_handle_connect on v2:172.21.15.43:6836/3759191221 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbfdaa00640 1 -- 172.21.15.43:0/1580771289 <== mds.0 v2:172.21.15.43:6836/3759191221 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x564b8613a6c0 con 0x564b86582b10 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbfdaa00640 10 client.5418 handle_command_reply: tid=0 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.875+0000 7fbff7196740 1 client.5418 shutdown 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 msgr2=0x564b8651dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4005620 tx=0x7fbfe4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 shutdown_connections 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:6836/3759191221,v1:172.21.15.43:6837/3759191221] conn(0x564b86582b10 0x564b86582ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564b864d2ea0 0x564b8651dfd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x564b864d25e0 0x564b8651da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 --2- 172.21.15.43:0/1580771289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b864cdba0 0x564b8653a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbfdaa00640 0 client.5418 ms_handle_reset on v2:172.21.15.43:6836/3759191221 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 >> 172.21.15.43:0/1580771289 conn(0x564b86494c70 msgr2=0x564b86520730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 shutdown_connections 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 1 -- 172.21.15.43:0/1580771289 wait complete. 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.876+0000 7fbff7196740 20 client.5418 trim_cache size 0 max 16384 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5250, 2024-01-16T15:00:55.860 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.8237670770000003, 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 17, 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 35, 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 35, 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15.61773342 2024-01-16T15:00:55.861 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 msgr2=0x7fbfc406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 0x7fbfc406d7b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4036000 tx=0x7fbfd403c040 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 msgr2=0x7fbfec0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc007ef0 tx=0x7fbfdc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 shutdown_connections 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfc406b2f0 0x7fbfc406d7b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfec146c70 0x7fbfec1e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfec146890 0x7fbfec142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 --2- 172.21.15.43:0/3470766110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfec0b3710 0x7fbfec0b2070 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 >> 172.21.15.43:0/3470766110 conn(0x7fbfec066610 msgr2=0x7fbfec0b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 shutdown_connections 2024-01-16T15:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:55.877+0000 7fbff4600640 1 -- 172.21.15.43:0/3470766110 wait complete. 2024-01-16T15:00:55.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.094+0000 7f9add400640 1 Processor -- start 2024-01-16T15:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.094+0000 7f9add400640 1 -- start start 2024-01-16T15:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.095+0000 7f9add400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ad806d520 0x7f9ad806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.095+0000 7f9add400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad815c360 con 0x7f9ad80ad720 2024-01-16T15:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.095+0000 7f9add400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad815faf0 con 0x7f9ad80abff0 2024-01-16T15:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9add400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad80b41a0 con 0x7f9ad806d520 2024-01-16T15:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9ad5e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9ad80ad720 0x7f9ad815ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33240/0 (socket says 172.21.15.43:33240) 2024-01-16T15:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9ad5e00640 1 -- 172.21.15.43:0/3830229591 learned_addr learned my addr 172.21.15.43:0/3830229591 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9ad6800640 1 --2- 172.21.15.43:0/3830229591 >> v2:172.21.15.69:3300/0 conn(0x7f9ad806d520 0x7f9ad806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100938651 0 0) 0x7f9ad815c360 con 0x7f9ad80ad720 2024-01-16T15:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.096+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab8003520 con 0x7f9ad80ad720 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2849404405 0 0) 0x7f9ab8003520 con 0x7f9ad80ad720 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 >> v2:172.21.15.69:3300/0 conn(0x7f9ad806d520 msgr2=0x7f9ad806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 --2- 172.21.15.43:0/3830229591 >> v2:172.21.15.69:3300/0 conn(0x7f9ad806d520 0x7f9ad806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 >> v1:172.21.15.69:6789/0 conn(0x7f9ad80abff0 legacy=0x7f9ad815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad806de40 con 0x7f9ad80ad720 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9ac0001670 con 0x7f9ad80ad720 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 >> v1:172.21.15.43:6789/0 conn(0x7f9ad80ad720 legacy=0x7f9ad815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ab8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ab80047c0 0x7f9ab8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab8005180 0x7f9ab800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab800bff0 con 0x7f9ab8003970 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab800c160 con 0x7f9ab8005180 2024-01-16T15:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab800c320 con 0x7f9ab80047c0 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.097+0000 7f9ad6800640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ab8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad5e00640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ab80047c0 0x7f9ab8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad7200640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab8005180 0x7f9ab800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad6800640 1 -- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ab80047c0 msgr2=0x7f9ab8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad6800640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ab80047c0 0x7f9ab8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad6800640 1 -- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab8005180 msgr2=0x7f9ab800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad6800640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab8005180 0x7f9ab800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad6800640 1 -- 172.21.15.43:0/3830229591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8003520 con 0x7f9ab8003970 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad7200640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab8005180 0x7f9ab800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.098+0000 7f9ad7200640 1 -- 172.21.15.43:0/3830229591 reap_dead start 2024-01-16T15:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.099+0000 7f9ad6800640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ab8003d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009ac0 tx=0x7f9ab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdacca86dfa1384 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.099+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac802c6b0 con 0x7f9ab8003970 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.099+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac802c810 con 0x7f9ab8003970 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.099+0000 7f9ad5400640 1 -- 172.21.15.43:0/3830229591 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac802cba0 con 0x7f9ab8003970 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 -- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 msgr2=0x7f9ab8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ab8003d90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009ac0 tx=0x7f9ab80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 -- 172.21.15.43:0/3830229591 shutdown_connections 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 --2- 172.21.15.43:0/3830229591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ab8003d90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 -- 172.21.15.43:0/3830229591 >> 172.21.15.43:0/3830229591 conn(0x7f9ad80b0d80 msgr2=0x7f9ad80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 -- 172.21.15.43:0/3830229591 shutdown_connections 2024-01-16T15:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.100+0000 7f9add400640 1 -- 172.21.15.43:0/3830229591 wait complete. 2024-01-16T15:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.102+0000 7f9add400640 1 Processor -- start 2024-01-16T15:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.102+0000 7f9add400640 1 -- start start 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8149ac0 0x7f9ad81f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad81f6910 0x7f9ad81f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad81f6210 con 0x7f9ab8003970 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8146540 con 0x7f9ad81f6910 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9add400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad81f5f90 con 0x7f9ad8149ac0 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51972/0 (socket says 172.21.15.43:51972) 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 -- 172.21.15.43:0/1883014165 learned_addr learned my addr 172.21.15.43:0/1883014165 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 -- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8149ac0 msgr2=0x7f9ad81f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad7200640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad81f6910 0x7f9ad81f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8149ac0 0x7f9ad81f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 -- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad81f6910 msgr2=0x7f9ad81f6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad81f6910 0x7f9ad81f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.104+0000 7f9ad6800640 1 -- 172.21.15.43:0/1883014165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8146780 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9ad6800640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8037040 tx=0x7f9ac8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8003810 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac8003970 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8003d00 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad8146a40 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.105+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81f63f0 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.106+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad8146f60 con 0x7f9ab8003970 2024-01-16T15:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.107+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9ac8034440 con 0x7f9ab8003970 2024-01-16T15:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.108+0000 7f9acea00640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 0x7f9ab006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.108+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ac80aba30 con 0x7f9ab8003970 2024-01-16T15:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.108+0000 7f9ad5e00640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 0x7f9ab006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.109+0000 7f9ad5e00640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 0x7f9ab006d7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0000f60 tx=0x7f9ac0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.113+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ad8146f60 con 0x7f9ab8003970 2024-01-16T15:00:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.346+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9ad8146f60 con 0x7f9ab8003970 2024-01-16T15:00:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.347+0000 7f9acea00640 1 -- 172.21.15.43:0/1883014165 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9ad8146f60 con 0x7f9ab8003970 2024-01-16T15:00:56.327 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.349+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 msgr2=0x7f9ab006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.349+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 0x7f9ab006d7b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0000f60 tx=0x7f9ac0009040 comp rx=0 tx=0).stop 2024-01-16T15:00:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.349+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 msgr2=0x7f9ad8146000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.349+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8037040 tx=0x7f9ac8004730 comp rx=0 tx=0).stop 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 shutdown_connections 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab006b2f0 0x7f9ab006d7b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad81f6910 0x7f9ad81f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8149ac0 0x7f9ad81f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 --2- 172.21.15.43:0/1883014165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ab8003970 0x7f9ad8146000 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 >> 172.21.15.43:0/1883014165 conn(0x7f9ad80b0d80 msgr2=0x7f9ad815ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 shutdown_connections 2024-01-16T15:00:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:56.350+0000 7f9add400640 1 -- 172.21.15.43:0/1883014165 wait complete. 2024-01-16T15:00:56.349 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:56.349 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:56.350 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:56.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.350 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42382sec preferred_lft 42382sec 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.406 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:56.498 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:00:56.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.502 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:00:56.558 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:00:56.558 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:56.558 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:00:56.558 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:00:56.558 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:56.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.559 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:00:56.560 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:00:56.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:00:56.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:00:56.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:00:56.560 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:00:56.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.754 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:00:56.807 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:00:56.807 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:00:56.808 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:56.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:00:56.912 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:00:56.912 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:56.912 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:56.966 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:56.966 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:56.966 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:57.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:00:57.108 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:00:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:00:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:00:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:00:57.165 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:57.166 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:57.167 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:57.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:57.167 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:00:57.217 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:57.217 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:00:57.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:00:57.298 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:57.342 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:00:57.345 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:57.345 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:57.345 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:57.345 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:57.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:57.346 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:00:57.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:00:57.423+0000 7f74ed436480 -1 init, newargv = 0x55bc371c5880 newargc=15 2024-01-16T15:00:57.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[48817]: starting ceph client 2024-01-16T15:00:57.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[48817]: starting fuse 2024-01-16T15:00:57.431 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:00:57.432 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:00:57.432 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:00:57.433 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:00:57.434 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:00:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.48817.asok 2024-01-16T15:00:57.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:57.648 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48817.asok 2024-01-16T15:00:57.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:57.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48817.asok status 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "48817", 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5433, 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5433 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2512787464 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2512787464 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.5433 192.168.0.1:0/2512787464", 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/2512787464", 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 17, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 35, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:00:57.900 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:00:57.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:57.914 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:00:57.914 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:00:57.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:57.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:57.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:58.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.314+0000 7fc137e00640 1 Processor -- start 2024-01-16T15:00:58.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.314+0000 7fc137e00640 1 -- start start 2024-01-16T15:00:58.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.316+0000 7fc137e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc13813dfd0 0x7fc13815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.316+0000 7fc137e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc1380cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.316+0000 7fc137e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1380ce7c0 con 0x7fc13806c130 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.316+0000 7fc137e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1380cfa10 con 0x7fc13813dfd0 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.316+0000 7fc137e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc138140900 con 0x7fc1380cf050 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.317+0000 7fc136a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc13813dfd0 0x7fc13815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.317+0000 7fc12fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc1380cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.317+0000 7fc137400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc13806c130 0x7fc13806c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50080/0 (socket says 172.21.15.43:50080) 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.317+0000 7fc136a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc13813dfd0 0x7fc13815be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:48680/0 (socket says 172.21.15.43:48680) 2024-01-16T15:00:58.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.317+0000 7fc137400640 1 -- 172.21.15.43:0/2360151797 learned_addr learned my addr 172.21.15.43:0/2360151797 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:00:58.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.318+0000 7fc136a00640 1 -- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 msgr2=0x7fc1380cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:58.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.318+0000 7fc136a00640 1 --2- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc1380cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.318+0000 7fc136a00640 1 -- 172.21.15.43:0/2360151797 >> v1:172.21.15.69:6789/0 conn(0x7fc13806c130 legacy=0x7fc13806c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.318+0000 7fc136a00640 1 -- 172.21.15.43:0/2360151797 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13815fa70 con 0x7fc13813dfd0 2024-01-16T15:00:58.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.318+0000 7fc136000640 1 -- 172.21.15.43:0/2360151797 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364167601 0 0) 0x7fc1380ce7c0 con 0x7fc13806c130 2024-01-16T15:00:58.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.320+0000 7fc136a00640 1 --2- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815be70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1280097a0 tx=0x7fc12802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5d90fc2e3d2c336 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc136000640 1 -- 172.21.15.43:0/2360151797 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128031830 con 0x7fc13813dfd0 2024-01-16T15:00:58.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc136000640 1 -- 172.21.15.43:0/2360151797 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128031990 con 0x7fc13813dfd0 2024-01-16T15:00:58.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc136000640 1 -- 172.21.15.43:0/2360151797 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128031d20 con 0x7fc13813dfd0 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 -- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 msgr2=0x7fc13815be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 --2- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815be70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1280097a0 tx=0x7fc12802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 -- 172.21.15.43:0/2360151797 shutdown_connections 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 --2- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc1380cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 --2- 172.21.15.43:0/2360151797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815be70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 -- 172.21.15.43:0/2360151797 >> 172.21.15.43:0/2360151797 conn(0x7fc13806a860 msgr2=0x7fc13806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.321+0000 7fc137e00640 1 -- 172.21.15.43:0/2360151797 shutdown_connections 2024-01-16T15:00:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.322+0000 7fc137e00640 1 -- 172.21.15.43:0/2360151797 wait complete. 2024-01-16T15:00:58.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.323+0000 7fc137e00640 1 Processor -- start 2024-01-16T15:00:58.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.323+0000 7fc137e00640 1 -- start start 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc13815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13815d250 con 0x7fc13806c130 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13815d530 con 0x7fc13813dfd0 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.324+0000 7fc137e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13815d810 con 0x7fc1380cf050 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51980/0 (socket says 172.21.15.43:51980) 2024-01-16T15:00:58.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 -- 172.21.15.43:0/956314065 learned_addr learned my addr 172.21.15.43:0/956314065 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc137400640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc12fe00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc13815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 -- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 msgr2=0x7fc13815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc13815c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 -- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 msgr2=0x7fc13815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.325+0000 7fc136a00640 1 -- 172.21.15.43:0/956314065 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128009400 con 0x7fc13806c130 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc136a00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc1280097a0 tx=0x7fc1280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128004230 con 0x7fc13806c130 2024-01-16T15:00:58.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128004390 con 0x7fc13806c130 2024-01-16T15:00:58.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12803a870 con 0x7fc13806c130 2024-01-16T15:00:58.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13815daf0 con 0x7fc13806c130 2024-01-16T15:00:58.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.326+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138138dc0 con 0x7fc13806c130 2024-01-16T15:00:58.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.328+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13815e010 con 0x7fc13806c130 2024-01-16T15:00:58.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.328+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc12803a9d0 con 0x7fc13806c130 2024-01-16T15:00:58.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.329+0000 7fc12f400640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 0x7fc11006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.329+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1280abe40 con 0x7fc13806c130 2024-01-16T15:00:58.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.330+0000 7fc12fe00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 0x7fc11006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.330+0000 7fc12fe00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 0x7fc11006d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc13815c480 tx=0x7fc120008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.334+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc128076c10 con 0x7fc13806c130 2024-01-16T15:00:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.574+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc13804e840 con 0x7fc13806c130 2024-01-16T15:00:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.575+0000 7fc12f400640 1 -- 172.21.15.43:0/956314065 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc128035020 con 0x7fc13806c130 2024-01-16T15:00:58.555 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 msgr2=0x7fc11006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 0x7fc11006d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc13815c480 tx=0x7fc120008040 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 msgr2=0x7fc138138880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc1280097a0 tx=0x7fc1280046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 shutdown_connections 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc11006b2c0 0x7fc11006d780 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc13813dfd0 0x7fc13815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1380cf050 0x7fc13815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 --2- 172.21.15.43:0/956314065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc13806c130 0x7fc138138880 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 >> 172.21.15.43:0/956314065 conn(0x7fc13806a860 msgr2=0x7fc13806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 shutdown_connections 2024-01-16T15:00:58.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:00:58.577+0000 7fc137e00640 1 -- 172.21.15.43:0/956314065 wait complete. 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:58.575 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:58.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:58.576 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.632 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42541sec preferred_lft 42541sec 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:58.633 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:00:58.634 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:58.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:58.635 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:58.725 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:00:58.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:58.730 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:00:58.780 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:00:58.781 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:58.781 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:58.781 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:58.781 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:58.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:58.782 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:00:58.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:58.955 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:00:59.003 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:00:59.003 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:00:59.004 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:59.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:59.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:00:59.094 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:00:59.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:59.095 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:59.151 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:59.151 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:59.151 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:59.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:00:59.289 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:00:59.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:00:59.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:59.346 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:59.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:59.347 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:00:59.396 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:59.396 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:00:59.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:00:59.481 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:00:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:59.526 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:59.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:59.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:59.528 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:00:59.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:00:59.611+0000 7f0f45438480 -1 init, newargv = 0x5596216dbc60 newargc=15 2024-01-16T15:00:59.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[42712]: starting ceph client 2024-01-16T15:00:59.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[42712]: starting fuse 2024-01-16T15:00:59.614 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:00:59.614 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:00:59.614 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:00:59.615 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:00:59.616 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:59.616 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:00:59.616 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:00:59.616 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:00:59.808 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.42712.asok 2024-01-16T15:00:59.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:00:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:59.825 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42712.asok 2024-01-16T15:00:59.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:59.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42712.asok status 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "42712", 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:01:00.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5092, 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5092 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2810379872 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:01:00.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2810379872 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5092 192.168.0.1:0/2810379872", 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2810379872", 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 17, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 35, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:01:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:01:00.065 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:01:00.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:00.076 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:01:00.077 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:00.128 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:01:00.129 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:00.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:00.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:00.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.471+0000 7feba5400640 1 Processor -- start 2024-01-16T15:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.471+0000 7feba5400640 1 -- start start 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.472+0000 7feba5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feba00d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.472+0000 7feba5400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba00cf240 con 0x7feba015d350 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.472+0000 7feba5400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba00d1fb0 con 0x7feba015d730 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.472+0000 7feba5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba00ce050 con 0x7feba00d0260 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.473+0000 7feb9e800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7feba015d350 0x7feba00ceb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33250/0 (socket says 172.21.15.43:33250) 2024-01-16T15:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.473+0000 7feb9e800640 1 -- 172.21.15.43:0/923079030 learned_addr learned my addr 172.21.15.43:0/923079030 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.473+0000 7feb9f200640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feba00d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.473+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891516848 0 0) 0x7feba00cf240 con 0x7feba015d350 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.473+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb80003520 con 0x7feba015d350 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884726823 0 0) 0x7feba00d1fb0 con 0x7feba015d730 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba00cf240 con 0x7feba015d730 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005074664 0 0) 0x7feb80003520 con 0x7feba015d350 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 msgr2=0x7feba00d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feba00d3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 >> v1:172.21.15.69:6789/0 conn(0x7feba015d730 legacy=0x7feba00cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00ce330 con 0x7feba015d350 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7feb90001670 con 0x7feba015d350 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 >> v1:172.21.15.43:6789/0 conn(0x7feba015d350 legacy=0x7feba00ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb80003b30 0x7feb80003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feb800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80004d00 0x7feb80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba00ce050 con 0x7feb80003b30 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8000ba60 con 0x7feb80004d00 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8000bc20 con 0x7feba00d0260 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.474+0000 7feb9e800640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb80003b30 0x7feb80003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9de00640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feb800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9f200640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80004d00 0x7feb80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9e800640 1 -- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 msgr2=0x7feb800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9e800640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba00d0260 0x7feb800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9e800640 1 -- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80004d00 msgr2=0x7feb80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9e800640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80004d00 0x7feb80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.475+0000 7feb9e800640 1 -- 172.21.15.43:0/923079030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80003520 con 0x7feb80003b30 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.476+0000 7feb9e800640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb80003b30 0x7feb80003f30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb90007f90 tx=0x7feb800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2b14c42e18141e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.476+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90005ad0 con 0x7feb80003b30 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.476+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb90005c30 con 0x7feb80003b30 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.476+0000 7feb9d400640 1 -- 172.21.15.43:0/923079030 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9002c790 con 0x7feb80003b30 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 -- 172.21.15.43:0/923079030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb80003b30 msgr2=0x7feb80003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 --2- 172.21.15.43:0/923079030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb80003b30 0x7feb80003f30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb90007f90 tx=0x7feb800057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feb9f200640 1 -- 172.21.15.43:0/923079030 reap_dead start 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 -- 172.21.15.43:0/923079030 shutdown_connections 2024-01-16T15:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 -- 172.21.15.43:0/923079030 >> 172.21.15.43:0/923079030 conn(0x7feba009c5c0 msgr2=0x7feba00d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 -- 172.21.15.43:0/923079030 shutdown_connections 2024-01-16T15:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.477+0000 7feba5400640 1 -- 172.21.15.43:0/923079030 wait complete. 2024-01-16T15:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.479+0000 7feba5400640 1 Processor -- start 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.479+0000 7feba5400640 1 -- start start 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 0x7feba00cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba0151a90 0x7feba00b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba00b3b30 con 0x7feba00b36b0 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba009daf0 con 0x7feb80003b30 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feba5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba01554c0 con 0x7feba0151a90 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feb9f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feb9f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51996/0 (socket says 172.21.15.43:51996) 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.480+0000 7feb9f200640 1 -- 172.21.15.43:0/207451506 learned_addr learned my addr 172.21.15.43:0/207451506 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9e800640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 0x7feba00cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9de00640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba0151a90 0x7feba00b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 -- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba0151a90 msgr2=0x7feba00b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba0151a90 0x7feba00b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 -- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 msgr2=0x7feba00cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 0x7feba00cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 -- 172.21.15.43:0/207451506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba009dd40 con 0x7feba00b36b0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9e800640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 0x7feba00cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb9f200640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feba00729a0 tx=0x7feb980115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.481+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9800e070 con 0x7feba00b36b0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.482+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb98011e40 con 0x7feba00b36b0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.482+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98003990 con 0x7feba00b36b0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.482+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba009e020 con 0x7feba00b36b0 2024-01-16T15:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.482+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba0151f10 con 0x7feba00b36b0 2024-01-16T15:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.483+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba00a1230 con 0x7feba00b36b0 2024-01-16T15:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.484+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb98003af0 con 0x7feba00b36b0 2024-01-16T15:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.485+0000 7feb96a00640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 0x7feb7806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.485+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb9808a290 con 0x7feba00b36b0 2024-01-16T15:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.485+0000 7feb9e800640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 0x7feb7806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.486+0000 7feb9e800640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 0x7feb7806d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb90034000 tx=0x7feb90009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.489+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb980550f0 con 0x7feba00b36b0 2024-01-16T15:01:00.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.720+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feba00ca330 con 0x7feba00b36b0 2024-01-16T15:01:00.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.756+0000 7feb96a00640 1 -- 172.21.15.43:0/207451506 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7feb98058710 con 0x7feba00b36b0 2024-01-16T15:01:00.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 msgr2=0x7feb7806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 0x7feb7806d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb90034000 tx=0x7feb90009030 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 msgr2=0x7feba0154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feba00729a0 tx=0x7feb980115a0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 shutdown_connections 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb7806b2c0 0x7feb7806d780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba00b36b0 0x7feba0154530 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feba0151a90 0x7feba00b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 --2- 172.21.15.43:0/207451506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb80003b30 0x7feba00cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 >> 172.21.15.43:0/207451506 conn(0x7feba009c5c0 msgr2=0x7feba015c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 shutdown_connections 2024-01-16T15:01:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.759+0000 7feba5400640 1 -- 172.21.15.43:0/207451506 wait complete. 2024-01-16T15:01:00.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:01:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.992+0000 7f821c600640 1 Processor -- start 2024-01-16T15:01:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.992+0000 7f821c600640 1 -- start start 2024-01-16T15:01:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f821c600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f82141660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f821c600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82140b4130 con 0x7f821415b340 2024-01-16T15:01:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f821c600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82140b2080 con 0x7f82140cbcb0 2024-01-16T15:01:00.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f821c600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82140b2360 con 0x7f821406dc30 2024-01-16T15:01:00.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f8219c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f821415b340 0x7f82140abff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33260/0 (socket says 172.21.15.43:33260) 2024-01-16T15:01:00.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.994+0000 7f8219c00640 1 -- 172.21.15.43:0/1120987470 learned_addr learned my addr 172.21.15.43:0/1120987470 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f821a600640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f82141660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527240373 0 0) 0x7f82140b4130 con 0x7f821415b340 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f4003520 con 0x7f821415b340 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266318298 0 0) 0x7f81f4003520 con 0x7f821415b340 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 msgr2=0x7f82141660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f82141660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 >> v1:172.21.15.69:6789/0 conn(0x7f82140cbcb0 legacy=0x7f821406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82140b2640 con 0x7f821415b340 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8204001670 con 0x7f821415b340 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 >> v1:172.21.15.43:6789/0 conn(0x7f821415b340 legacy=0x7f82140abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81f40039a0 0x7f81f4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f81f4004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.995+0000 7f8213e00640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f81f4005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82140b2360 con 0x7f81f4004350 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f400b9b0 con 0x7f81f40039a0 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f400bb70 con 0x7f821406dc30 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f81f4004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f821a600640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f81f4005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219c00640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81f40039a0 0x7f81f4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 -- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 msgr2=0x7f81f4005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f81f4005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 -- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81f40039a0 msgr2=0x7f81f4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81f40039a0 0x7f81f4003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f8219200640 1 -- 172.21.15.43:0/1120987470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4003520 con 0x7f81f4004350 2024-01-16T15:01:00.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.996+0000 7f821a600640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f821406dc30 0x7f81f4005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.997+0000 7f8219200640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f81f4004820 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f81fc001830 tx=0x7f81f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4638266487e1d52d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc02f050 con 0x7f81f4004350 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc033610 con 0x7f81f4004350 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f8213e00640 1 -- 172.21.15.43:0/1120987470 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0339c0 con 0x7f81f4004350 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821c600640 1 -- 172.21.15.43:0/1120987470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 msgr2=0x7f81f4004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821c600640 1 --2- 172.21.15.43:0/1120987470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f81f4004820 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f81fc001830 tx=0x7f81f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821a600640 1 -- 172.21.15.43:0/1120987470 reap_dead start 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821c600640 1 -- 172.21.15.43:0/1120987470 shutdown_connections 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821c600640 1 -- 172.21.15.43:0/1120987470 >> 172.21.15.43:0/1120987470 conn(0x7f82140b0d80 msgr2=0x7f82140b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.998+0000 7f821c600640 1 -- 172.21.15.43:0/1120987470 shutdown_connections 2024-01-16T15:01:00.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:00.999+0000 7f821c600640 1 -- 172.21.15.43:0/1120987470 wait complete. 2024-01-16T15:01:00.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.000+0000 7f821c600640 1 Processor -- start 2024-01-16T15:01:00.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.000+0000 7f821c600640 1 -- start start 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82141f0960 0x7f8214099b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82140b3020 0x7f82140b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82140b39e0 con 0x7f81f4004350 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f0eb0 con 0x7f82141f0960 2024-01-16T15:01:00.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.001+0000 7f821c600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f1190 con 0x7f82140b3020 2024-01-16T15:01:00.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52020/0 (socket says 172.21.15.43:52020) 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 -- 172.21.15.43:0/458076627 learned_addr learned my addr 172.21.15.43:0/458076627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219200640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82141f0960 0x7f8214099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f821a600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82140b3020 0x7f82140b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 -- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82140b3020 msgr2=0x7f82140b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82140b3020 0x7f82140b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 -- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82141f0960 msgr2=0x7f8214099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82141f0960 0x7f8214099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.002+0000 7f8219c00640 1 -- 172.21.15.43:0/458076627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82141f1470 con 0x7f81f4004350 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f8219c00640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f820400d6f0 tx=0x7f820400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040072e0 con 0x7f81f4004350 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8204007440 con 0x7f81f4004350 2024-01-16T15:01:00.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82141f1750 con 0x7f81f4004350 2024-01-16T15:01:00.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040047e0 con 0x7f81f4004350 2024-01-16T15:01:00.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.003+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821415e500 con 0x7f81f4004350 2024-01-16T15:01:00.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.004+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8214074770 con 0x7f81f4004350 2024-01-16T15:01:00.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.005+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8204004940 con 0x7f81f4004350 2024-01-16T15:01:00.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.006+0000 7f8211600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 0x7f81ec06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.006+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8204089840 con 0x7f81f4004350 2024-01-16T15:01:00.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.007+0000 7f8219200640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 0x7f81ec06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.007+0000 7f8219200640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 0x7f81ec06d7b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81fc036000 tx=0x7f81fc031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.011+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8204054610 con 0x7f81f4004350 2024-01-16T15:01:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.231+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f82140b2c00 con 0x7f81ec06b2f0 2024-01-16T15:01:01.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.234+0000 7f8211600640 1 -- 172.21.15.43:0/458076627 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f81fc042070 con 0x7f81ec06b2f0 2024-01-16T15:01:01.215 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:01:01.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:01:01.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:01:01.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:01.215 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:01:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 msgr2=0x7f81ec06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 0x7f81ec06d7b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81fc036000 tx=0x7f81fc031000 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 msgr2=0x7f821414f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f820400d6f0 tx=0x7f820400dbb0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 shutdown_connections 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f81ec06b2f0 0x7f81ec06d7b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82141f0960 0x7f8214099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82140b3020 0x7f82140b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 --2- 172.21.15.43:0/458076627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81f4004350 0x7f821414f460 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 >> 172.21.15.43:0/458076627 conn(0x7f82140b0d80 msgr2=0x7f82140aca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 shutdown_connections 2024-01-16T15:01:01.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.237+0000 7f821c600640 1 -- 172.21.15.43:0/458076627 wait complete. 2024-01-16T15:01:01.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:01:01.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.483+0000 7f2691800640 1 Processor -- start 2024-01-16T15:01:01.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.483+0000 7f2691800640 1 -- start start 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.484+0000 7f2691800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f268c06dae0 0x7f268c13f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.484+0000 7f2691800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 0x7f268c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.484+0000 7f2691800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c140430 con 0x7f268c13cf40 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.484+0000 7f2691800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c071220 con 0x7f268c06dae0 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.484+0000 7f2691800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c071500 con 0x7f268c13fc40 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f268be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f268c13cf40 0x7f268c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33274/0 (socket says 172.21.15.43:33274) 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f268be00640 1 -- 172.21.15.43:0/3487454703 learned_addr learned my addr 172.21.15.43:0/3487454703 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2683e00640 1 --2- 172.21.15.43:0/3487454703 >> v2:172.21.15.43:3300/0 conn(0x7f268c06dae0 0x7f268c13f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2690e00640 1 --2- 172.21.15.43:0/3487454703 >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 0x7f268c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f268b400640 1 -- 172.21.15.43:0/3487454703 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980549485 0 0) 0x7f268c140430 con 0x7f268c13cf40 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f268b400640 1 -- 172.21.15.43:0/3487454703 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c003520 con 0x7f268c13cf40 2024-01-16T15:01:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2683e00640 1 -- 172.21.15.43:0/3487454703 >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 msgr2=0x7f268c0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2683e00640 1 --2- 172.21.15.43:0/3487454703 >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 0x7f268c0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2683e00640 1 -- 172.21.15.43:0/3487454703 >> v1:172.21.15.43:6789/0 conn(0x7f268c13cf40 legacy=0x7f268c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2683e00640 1 -- 172.21.15.43:0/3487454703 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0717e0 con 0x7f268c06dae0 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.485+0000 7f2690e00640 1 --2- 172.21.15.43:0/3487454703 >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 0x7f268c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.487+0000 7f2683e00640 1 --2- 172.21.15.43:0/3487454703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c06dae0 0x7f268c13f700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f268c143bf0 tx=0x7f267402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ff7c718288b89c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.487+0000 7f268b400640 1 -- 172.21.15.43:0/3487454703 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031830 con 0x7f268c06dae0 2024-01-16T15:01:01.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.487+0000 7f268b400640 1 -- 172.21.15.43:0/3487454703 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2674031990 con 0x7f268c06dae0 2024-01-16T15:01:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.487+0000 7f268b400640 1 -- 172.21.15.43:0/3487454703 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031d20 con 0x7f268c06dae0 2024-01-16T15:01:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 -- 172.21.15.43:0/3487454703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c06dae0 msgr2=0x7f268c13f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 --2- 172.21.15.43:0/3487454703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c06dae0 0x7f268c13f700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f268c143bf0 tx=0x7f267402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 -- 172.21.15.43:0/3487454703 shutdown_connections 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 --2- 172.21.15.43:0/3487454703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c06dae0 0x7f268c13f700 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 --2- 172.21.15.43:0/3487454703 >> v2:172.21.15.69:3300/0 conn(0x7f268c13fc40 0x7f268c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 -- 172.21.15.43:0/3487454703 >> 172.21.15.43:0/3487454703 conn(0x7f268c0b0aa0 msgr2=0x7f268c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 -- 172.21.15.43:0/3487454703 shutdown_connections 2024-01-16T15:01:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.488+0000 7f2691800640 1 -- 172.21.15.43:0/3487454703 wait complete. 2024-01-16T15:01:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.490+0000 7f2691800640 1 Processor -- start 2024-01-16T15:01:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.490+0000 7f2691800640 1 -- start start 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 0x7f268c13b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 0x7f268c13e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0acbc0 con 0x7f268c13cf40 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c13be00 con 0x7f268c13fc40 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2691800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c155f20 con 0x7f268c06dae0 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2683e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2683e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52042/0 (socket says 172.21.15.43:52042) 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f2683e00640 1 -- 172.21.15.43:0/2505724892 learned_addr learned my addr 172.21.15.43:0/2505724892 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.491+0000 7f268be00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 0x7f268c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2690e00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 0x7f268c13e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 -- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 msgr2=0x7f268c13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 0x7f268c13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 -- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 msgr2=0x7f268c13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 0x7f268c13e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 -- 172.21.15.43:0/2505724892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26740093a0 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f268be00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 0x7f268c13b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2690e00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 0x7f268c13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2683e00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2674000c00 tx=0x7f26740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267403a8f0 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267403aa70 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267403ae20 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.492+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c072160 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.493+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c1564a0 con 0x7f268c13cf40 2024-01-16T15:01:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.494+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c04e840 con 0x7f268c13cf40 2024-01-16T15:01:01.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.495+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f267404d450 con 0x7f268c13cf40 2024-01-16T15:01:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.496+0000 7f2688c00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 0x7f266406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.496+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26740abc60 con 0x7f268c13cf40 2024-01-16T15:01:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.496+0000 7f268be00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 0x7f266406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.497+0000 7f268be00640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 0x7f266406d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f267c007fd0 tx=0x7f267c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.501+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2674076a30 con 0x7f268c13cf40 2024-01-16T15:01:01.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.700+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f268c0cca90 con 0x7f268c13cf40 2024-01-16T15:01:01.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.701+0000 7f2688c00640 1 -- 172.21.15.43:0/2505724892 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2674035070 con 0x7f268c13cf40 2024-01-16T15:01:01.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.703+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 msgr2=0x7f266406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.703+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 0x7f266406d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f267c007fd0 tx=0x7f267c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:01.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.703+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 msgr2=0x7f268c152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.703+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2674000c00 tx=0x7f26740046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 shutdown_connections 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f266406b2c0 0x7f266406d780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f268c13fc40 0x7f268c13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f268c13cf40 0x7f268c152150 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 --2- 172.21.15.43:0/2505724892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f268c06dae0 0x7f268c13b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 >> 172.21.15.43:0/2505724892 conn(0x7f268c0b0aa0 msgr2=0x7f268c1587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 shutdown_connections 2024-01-16T15:01:01.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.704+0000 7f2691800640 1 -- 172.21.15.43:0/2505724892 wait complete. 2024-01-16T15:01:01.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:01:01.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.947+0000 7f5973e00640 1 Processor -- start 2024-01-16T15:01:01.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.947+0000 7f5973e00640 1 -- start start 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.948+0000 7f5973e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f596c09cb30 0x7f596c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.948+0000 7f5973e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c15b730 con 0x7f596c0adb60 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.948+0000 7f5973e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c0ac740 con 0x7f596c15b350 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.948+0000 7f5973e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c0ac460 con 0x7f596c09cb30 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.949+0000 7f5971e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f596c0adb60 0x7f596c0adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33284/0 (socket says 172.21.15.43:33284) 2024-01-16T15:01:01.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.949+0000 7f5971e00640 1 -- 172.21.15.43:0/1020277764 learned_addr learned my addr 172.21.15.43:0/1020277764 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.949+0000 7f5970a00640 1 --2- 172.21.15.43:0/1020277764 >> v2:172.21.15.69:3300/0 conn(0x7f596c09cb30 0x7f596c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.949+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558616219 0 0) 0x7f5964001490 con 0x7f596c0adb60 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.949+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c003520 con 0x7f596c0adb60 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 105800089 0 0) 0x7f594c003520 con 0x7f596c0adb60 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 >> v2:172.21.15.69:3300/0 conn(0x7f596c09cb30 msgr2=0x7f596c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 --2- 172.21.15.43:0/1020277764 >> v2:172.21.15.69:3300/0 conn(0x7f596c09cb30 0x7f596c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 >> v1:172.21.15.69:6789/0 conn(0x7f596c15b350 legacy=0x7f596c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c0b2a50 con 0x7f596c0adb60 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f5964001b80 con 0x7f596c0adb60 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 >> v1:172.21.15.43:6789/0 conn(0x7f596c0adb60 legacy=0x7f596c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 0x7f594c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c0047c0 0x7f594c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c005180 0x7f594c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0ac460 con 0x7f594c003970 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c00c020 con 0x7f594c0047c0 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c00c1e0 con 0x7f594c005180 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.950+0000 7f5971400640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 0x7f594c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971e00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c005180 0x7f594c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5970a00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c0047c0 0x7f594c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971400640 1 -- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c005180 msgr2=0x7f594c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971400640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c005180 0x7f594c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971400640 1 -- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c0047c0 msgr2=0x7f594c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971400640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c0047c0 0x7f594c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971400640 1 -- 172.21.15.43:0/1020277764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c003520 con 0x7f594c003970 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971e00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c005180 0x7f594c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:01.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.951+0000 7f5971e00640 1 -- 172.21.15.43:0/1020277764 reap_dead start 2024-01-16T15:01:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.952+0000 7f5971400640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 0x7f594c003d90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f595c0015b0 tx=0x7f594c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2dc5d8ef799d5c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.952+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c0052d0 con 0x7f594c003970 2024-01-16T15:01:01.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.952+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c02e050 con 0x7f594c003970 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.952+0000 7f596be00640 1 -- 172.21.15.43:0/1020277764 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c003830 con 0x7f594c003970 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 -- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 msgr2=0x7f594c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 0x7f594c003d90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f595c0015b0 tx=0x7f594c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 -- 172.21.15.43:0/1020277764 shutdown_connections 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 --2- 172.21.15.43:0/1020277764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f594c003970 0x7f594c003d90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 -- 172.21.15.43:0/1020277764 >> 172.21.15.43:0/1020277764 conn(0x7f596c0b0ba0 msgr2=0x7f596c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 -- 172.21.15.43:0/1020277764 shutdown_connections 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.953+0000 7f5973e00640 1 -- 172.21.15.43:0/1020277764 wait complete. 2024-01-16T15:01:01.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.955+0000 7f5973e00640 1 Processor -- start 2024-01-16T15:01:01.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.955+0000 7f5973e00640 1 -- start start 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c003970 0x7f596c1f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596c156610 0x7f596c14ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c15a6c0 con 0x7f596c159d00 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c15b8d0 con 0x7f594c003970 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5973e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c15c170 con 0x7f596c156610 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5971e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5971e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52056/0 (socket says 172.21.15.43:52056) 2024-01-16T15:01:01.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5971e00640 1 -- 172.21.15.43:0/1563017206 learned_addr learned my addr 172.21.15.43:0/1563017206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5971400640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c003970 0x7f596c1f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.956+0000 7f5970a00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596c156610 0x7f596c14ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 -- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596c156610 msgr2=0x7f596c14ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596c156610 0x7f596c14ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 -- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c003970 msgr2=0x7f596c1f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c003970 0x7f596c1f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 -- 172.21.15.43:0/1563017206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1f7560 con 0x7f596c159d00 2024-01-16T15:01:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5971e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f596c072730 tx=0x7f596400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596400bdb0 con 0x7f596c159d00 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964004b60 con 0x7f596c159d00 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964007110 con 0x7f596c159d00 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c156bb0 con 0x7f596c159d00 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.957+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c15be00 con 0x7f596c159d00 2024-01-16T15:01:01.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.958+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c15c300 con 0x7f596c159d00 2024-01-16T15:01:01.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.959+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5964007270 con 0x7f596c159d00 2024-01-16T15:01:01.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.960+0000 7f5969600640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 0x7f594406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.960+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5964089dc0 con 0x7f596c159d00 2024-01-16T15:01:01.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.961+0000 7f5971400640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 0x7f594406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.962+0000 7f5971400640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 0x7f594406d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f595c033000 tx=0x7f595c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:01.965+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5964054c20 con 0x7f596c159d00 2024-01-16T15:01:02.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.189+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f596c06e720 con 0x7f594406b2c0 2024-01-16T15:01:02.182 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:02.202+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:02.182 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:02.202+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:02.182 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:02.202+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:02.182 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:02.202+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:02.182 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:02.202+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.247+0000 7f5969600640 1 -- 172.21.15.43:0/1563017206 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f595c005400 con 0x7f594406b2c0 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 msgr2=0x7f594406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 0x7f594406d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f595c033000 tx=0x7f595c009030 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 msgr2=0x7f596c15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f596c072730 tx=0x7f596400b4a0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 shutdown_connections 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f594406b2c0 0x7f594406d780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596c159d00 0x7f596c15a180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596c156610 0x7f596c14ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 --2- 172.21.15.43:0/1563017206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c003970 0x7f596c1f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 >> 172.21.15.43:0/1563017206 conn(0x7f596c0b0ba0 msgr2=0x7f596c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 shutdown_connections 2024-01-16T15:01:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.251+0000 7f5973e00640 1 -- 172.21.15.43:0/1563017206 wait complete. 2024-01-16T15:01:02.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:01:02.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.471+0000 7fc27cc00640 1 Processor -- start 2024-01-16T15:01:02.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.471+0000 7fc27cc00640 1 -- start start 2024-01-16T15:01:02.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc27cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc278071ae0 0x7fc2780cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc27cc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2780cfe50 con 0x7fc2780cd440 2024-01-16T15:01:02.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc27cc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2780cf260 con 0x7fc27806ab50 2024-01-16T15:01:02.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc27cc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc278071f20 con 0x7fc278071ae0 2024-01-16T15:01:02.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc275e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc2780cd440 0x7fc27806a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33292/0 (socket says 172.21.15.43:33292) 2024-01-16T15:01:02.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.473+0000 7fc275e00640 1 -- 172.21.15.43:0/1996315600 learned_addr learned my addr 172.21.15.43:0/1996315600 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc276800640 1 --2- 172.21.15.43:0/1996315600 >> v2:172.21.15.69:3300/0 conn(0x7fc278071ae0 0x7fc2780cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300863678 0 0) 0x7fc2780cfe50 con 0x7fc2780cd440 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258003520 con 0x7fc2780cd440 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2842322565 0 0) 0x7fc258003520 con 0x7fc2780cd440 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 >> v2:172.21.15.69:3300/0 conn(0x7fc278071ae0 msgr2=0x7fc2780cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 --2- 172.21.15.43:0/1996315600 >> v2:172.21.15.69:3300/0 conn(0x7fc278071ae0 0x7fc2780cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 >> v1:172.21.15.69:6789/0 conn(0x7fc27806ab50 legacy=0x7fc27815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780cf520 con 0x7fc2780cd440 2024-01-16T15:01:02.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc268001670 con 0x7fc2780cd440 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 >> v1:172.21.15.43:6789/0 conn(0x7fc2780cd440 legacy=0x7fc27806a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc258003970 0x7fc258003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2580047c0 0x7fc258004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 0x7fc25800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800bfa0 con 0x7fc258005180 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800c110 con 0x7fc2580047c0 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.474+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800c2d0 con 0x7fc258003970 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 0x7fc25800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc26fe00640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2580047c0 0x7fc258004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc275e00640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc258003970 0x7fc258003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 -- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc258003970 msgr2=0x7fc258003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc258003970 0x7fc258003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 -- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2580047c0 msgr2=0x7fc258004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2580047c0 0x7fc258004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc276800640 1 -- 172.21.15.43:0/1996315600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258003520 con 0x7fc258005180 2024-01-16T15:01:02.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.475+0000 7fc275e00640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc258003970 0x7fc258003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.476+0000 7fc276800640 1 -- 172.21.15.43:0/1996315600 reap_dead start 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.476+0000 7fc276800640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 0x7fc25800b980 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fc0 tx=0x7fc2580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1623f47bcc807835 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.476+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27002cc60 con 0x7fc258005180 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.476+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27000f040 con 0x7fc258005180 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.476+0000 7fc275400640 1 -- 172.21.15.43:0/1996315600 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270003990 con 0x7fc258005180 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 -- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 msgr2=0x7fc25800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 0x7fc25800b980 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc270009fc0 tx=0x7fc2580057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 -- 172.21.15.43:0/1996315600 shutdown_connections 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1996315600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc258005180 0x7fc25800b980 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 -- 172.21.15.43:0/1996315600 >> 172.21.15.43:0/1996315600 conn(0x7fc2780b1c90 msgr2=0x7fc2780b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 -- 172.21.15.43:0/1996315600 shutdown_connections 2024-01-16T15:01:02.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.477+0000 7fc27cc00640 1 -- 172.21.15.43:0/1996315600 wait complete. 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.479+0000 7fc27cc00640 1 Processor -- start 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.479+0000 7fc27cc00640 1 -- start start 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 0x7fc2780c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 0x7fc2780b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780b5700 con 0x7fc2780ad5a0 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780c4250 con 0x7fc258005180 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.480+0000 7fc27cc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780c4530 con 0x7fc2780b4d40 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52078/0 (socket says 172.21.15.43:52078) 2024-01-16T15:01:02.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 -- 172.21.15.43:0/1256101536 learned_addr learned my addr 172.21.15.43:0/1256101536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc275e00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 0x7fc2780c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 -- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 msgr2=0x7fc2780b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc276800640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 0x7fc2780b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 0x7fc2780b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 -- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 msgr2=0x7fc2780c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 0x7fc2780c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 -- 172.21.15.43:0/1256101536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780c4810 con 0x7fc2780ad5a0 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc275e00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 0x7fc2780c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:02.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc276800640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 0x7fc2780b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.481+0000 7fc26fe00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc26000e530 tx=0x7fc26000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.482+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2600052e0 con 0x7fc2780ad5a0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.482+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc260005440 con 0x7fc2780ad5a0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.482+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260003830 con 0x7fc2780ad5a0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.482+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2780c4af0 con 0x7fc2780ad5a0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.482+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2780c6a30 con 0x7fc2780ad5a0 2024-01-16T15:01:02.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.483+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc278067960 con 0x7fc2780ad5a0 2024-01-16T15:01:02.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.484+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc260003990 con 0x7fc2780ad5a0 2024-01-16T15:01:02.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.485+0000 7fc26e000640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 0x7fc25006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.485+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2600899e0 con 0x7fc2780ad5a0 2024-01-16T15:01:02.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.486+0000 7fc275e00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 0x7fc25006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.486+0000 7fc275e00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 0x7fc25006d7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc268007950 tx=0x7fc268009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.489+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc260054830 con 0x7fc2780ad5a0 2024-01-16T15:01:02.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.708+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2780c46a0 con 0x7fc25006b310 2024-01-16T15:01:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.720+0000 7fc26e000640 1 -- 172.21.15.43:0/1256101536 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fc2780c46a0 con 0x7fc25006b310 2024-01-16T15:01:02.701 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36 2024-01-16T15:01:02.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 msgr2=0x7fc25006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 0x7fc25006d7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc268007950 tx=0x7fc268009040 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 msgr2=0x7fc27815d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc26000e530 tx=0x7fc26000e9f0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 shutdown_connections 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc25006b310 0x7fc25006d7d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2780b4d40 0x7fc2780b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2780ad5a0 0x7fc27815d940 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 --2- 172.21.15.43:0/1256101536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258005180 0x7fc2780c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 >> 172.21.15.43:0/1256101536 conn(0x7fc2780b1c90 msgr2=0x7fc27806a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 shutdown_connections 2024-01-16T15:01:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:02.723+0000 7fc27cc00640 1 -- 172.21.15.43:0/1256101536 wait complete. 2024-01-16T15:01:02.724 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36 2024-01-16T15:01:02.724 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:01:02.724 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:02.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:02.774 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:01:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:01:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:01:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8625 s, 63.7 MB/s 2024-01-16T15:01:19.769 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:01:19.769 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:19.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:19.825 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:01:24.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:24.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:01:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:01:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.3972 s, 80.1 MB/s 2024-01-16T15:01:33.286 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:01:33.286 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:33.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:33.347 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:01:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:01:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:01:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.1914 s, 81.4 MB/s 2024-01-16T15:01:46.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:01:46.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.858+0000 7f4e98e00640 1 Processor -- start 2024-01-16T15:01:46.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.858+0000 7f4e98e00640 1 -- start start 2024-01-16T15:01:46.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.860+0000 7f4e98e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4e940acd70 0x7f4e9406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.860+0000 7f4e98e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94070490 0x7f4e94070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.860+0000 7f4e98e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9406d7a0 con 0x7f4e940ae490 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.860+0000 7f4e98e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9415a5b0 con 0x7f4e940acd70 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.860+0000 7f4e98e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9406ce70 con 0x7f4e94070490 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e8be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4e940ae490 0x7f4e940ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59918/0 (socket says 172.21.15.43:59918) 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e8be00640 1 -- 172.21.15.43:0/593938785 learned_addr learned my addr 172.21.15.43:0/593938785 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e91e00640 1 --2- 172.21.15.43:0/593938785 >> v2:172.21.15.43:3300/0 conn(0x7f4e940acd70 0x7f4e9406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e92800640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94070490 0x7f4e94070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e91e00640 1 -- 172.21.15.43:0/593938785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94070490 msgr2=0x7f4e94070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e91e00640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94070490 0x7f4e94070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e91e00640 1 -- 172.21.15.43:0/593938785 >> v1:172.21.15.43:6789/0 conn(0x7f4e940ae490 legacy=0x7f4e940ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.861+0000 7f4e91e00640 1 -- 172.21.15.43:0/593938785 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9406d150 con 0x7f4e940acd70 2024-01-16T15:01:46.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.862+0000 7f4e91e00640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9406dc30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097a0 tx=0x7f4e8402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bebc506d826969 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.863+0000 7f4e91400640 1 -- 172.21.15.43:0/593938785 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8402d040 con 0x7f4e940acd70 2024-01-16T15:01:46.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.863+0000 7f4e91400640 1 -- 172.21.15.43:0/593938785 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e84031810 con 0x7f4e940acd70 2024-01-16T15:01:46.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.863+0000 7f4e91400640 1 -- 172.21.15.43:0/593938785 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84031ba0 con 0x7f4e940acd70 2024-01-16T15:01:46.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.863+0000 7f4e98e00640 1 -- 172.21.15.43:0/593938785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 msgr2=0x7f4e9406dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.863+0000 7f4e98e00640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9406dc30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097a0 tx=0x7f4e8402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 -- 172.21.15.43:0/593938785 shutdown_connections 2024-01-16T15:01:46.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e94070490 0x7f4e94070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 --2- 172.21.15.43:0/593938785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9406dc30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 -- 172.21.15.43:0/593938785 >> 172.21.15.43:0/593938785 conn(0x7f4e940b0ba0 msgr2=0x7f4e940b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 -- 172.21.15.43:0/593938785 shutdown_connections 2024-01-16T15:01:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.864+0000 7f4e98e00640 1 -- 172.21.15.43:0/593938785 wait complete. 2024-01-16T15:01:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.865+0000 7f4e98e00640 1 Processor -- start 2024-01-16T15:01:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.866+0000 7f4e98e00640 1 -- start start 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e94070490 0x7f4e94156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 0x7f4e941471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941572c0 con 0x7f4e940acd70 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941575a0 con 0x7f4e94070490 2024-01-16T15:01:46.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e98e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941e2640 con 0x7f4e940ae490 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47422/0 (socket says 172.21.15.43:47422) 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 -- 172.21.15.43:0/3459286612 learned_addr learned my addr 172.21.15.43:0/3459286612 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e92800640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 0x7f4e941471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 -- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 msgr2=0x7f4e941471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 0x7f4e941471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 -- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e94070490 msgr2=0x7f4e94156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e94070490 0x7f4e94156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e8be00640 1 -- 172.21.15.43:0/3459286612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84009400 con 0x7f4e940acd70 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.867+0000 7f4e92800640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 0x7f4e941471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e8be00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00c480 tx=0x7f4e7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c005ce0 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e941e2920 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c005e40 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0072f0 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.868+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9409c380 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.869+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e941e2e40 con 0x7f4e940acd70 2024-01-16T15:01:46.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.870+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f4e7c007450 con 0x7f4e940acd70 2024-01-16T15:01:46.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.871+0000 7f4e8a000640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 0x7f4e6c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:46.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.871+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e7c0890d0 con 0x7f4e940acd70 2024-01-16T15:01:46.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.871+0000 7f4e91e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 0x7f4e6c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:46.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.872+0000 7f4e91e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 0x7f4e6c06d840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036000 tx=0x7f4e84030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:46.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:46.876+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e941e2e40 con 0x7f4e940acd70 2024-01-16T15:01:47.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.075+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f4e941e2e40 con 0x7f4e940acd70 2024-01-16T15:01:47.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.083+0000 7f4e8a000640 1 -- 172.21.15.43:0/3459286612 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4e941e2e40 con 0x7f4e940acd70 2024-01-16T15:01:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 msgr2=0x7f4e6c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 0x7f4e6c06d840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036000 tx=0x7f4e84030000 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 msgr2=0x7f4e9409be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00c480 tx=0x7f4e7c00c940 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 shutdown_connections 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e6c06b380 0x7f4e6c06d840 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e940ae490 0x7f4e941471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e940acd70 0x7f4e9409be40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 --2- 172.21.15.43:0/3459286612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e94070490 0x7f4e94156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 >> 172.21.15.43:0/3459286612 conn(0x7f4e940b0ba0 msgr2=0x7f4e94155430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 shutdown_connections 2024-01-16T15:01:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.086+0000 7f4e98e00640 1 -- 172.21.15.43:0/3459286612 wait complete. 2024-01-16T15:01:47.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:01:47.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.332+0000 7fdeab200640 1 Processor -- start 2024-01-16T15:01:47.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.332+0000 7fdeab200640 1 -- start start 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdeab200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdeac1435d0 0x7fdeac071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdeab200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdeac0b4140 0x7fdeac070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdeab200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac070d40 con 0x7fdeac1439b0 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdeab200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac071020 con 0x7fdeac1435d0 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdeab200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac071300 con 0x7fdeac0b4140 2024-01-16T15:01:47.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdea9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdeac1435d0 0x7fdeac071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdea9400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdeac1439b0 0x7fdeac0b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59930/0 (socket says 172.21.15.43:59930) 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.334+0000 7fdea9400640 1 -- 172.21.15.43:0/1674558920 learned_addr learned my addr 172.21.15.43:0/1674558920 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.335+0000 7fdea9e00640 1 -- 172.21.15.43:0/1674558920 >> v2:172.21.15.69:3300/0 conn(0x7fdeac0b4140 msgr2=0x7fdeac070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.335+0000 7fdea9e00640 1 --2- 172.21.15.43:0/1674558920 >> v2:172.21.15.69:3300/0 conn(0x7fdeac0b4140 0x7fdeac070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.335+0000 7fdea9e00640 1 -- 172.21.15.43:0/1674558920 >> v1:172.21.15.43:6789/0 conn(0x7fdeac1439b0 legacy=0x7fdeac0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.335+0000 7fdea9e00640 1 -- 172.21.15.43:0/1674558920 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0715e0 con 0x7fdeac1435d0 2024-01-16T15:01:47.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.335+0000 7fdea8a00640 1 -- 172.21.15.43:0/1674558920 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575408574 0 0) 0x7fdeac070d40 con 0x7fdeac1439b0 2024-01-16T15:01:47.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.336+0000 7fdea9e00640 1 --2- 172.21.15.43:0/1674558920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac071de0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0b4790 tx=0x7fde9c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1265843c44038383 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdea8a00640 1 -- 172.21.15.43:0/1674558920 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c02e070 con 0x7fdeac1435d0 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdea8a00640 1 -- 172.21.15.43:0/1674558920 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c0315a0 con 0x7fdeac1435d0 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdea8a00640 1 -- 172.21.15.43:0/1674558920 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c031930 con 0x7fdeac1435d0 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 -- 172.21.15.43:0/1674558920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 msgr2=0x7fdeac071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 --2- 172.21.15.43:0/1674558920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac071de0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0b4790 tx=0x7fde9c02b840 comp rx=0 tx=0).stop 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 -- 172.21.15.43:0/1674558920 shutdown_connections 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 --2- 172.21.15.43:0/1674558920 >> v2:172.21.15.69:3300/0 conn(0x7fdeac0b4140 0x7fdeac070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 --2- 172.21.15.43:0/1674558920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac071de0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 -- 172.21.15.43:0/1674558920 >> 172.21.15.43:0/1674558920 conn(0x7fdeac0cec10 msgr2=0x7fdeac0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:47.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 -- 172.21.15.43:0/1674558920 shutdown_connections 2024-01-16T15:01:47.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.337+0000 7fdeab200640 1 -- 172.21.15.43:0/1674558920 wait complete. 2024-01-16T15:01:47.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.339+0000 7fdeab200640 1 Processor -- start 2024-01-16T15:01:47.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.339+0000 7fdeab200640 1 -- start start 2024-01-16T15:01:47.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdeac0b4140 0x7fdeac142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 0x7fdeac06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac13b5d0 con 0x7fdeac1435d0 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac13b8b0 con 0x7fdeac1439b0 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.340+0000 7fdeab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac13bb90 con 0x7fdeac0b4140 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47448/0 (socket says 172.21.15.43:47448) 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 -- 172.21.15.43:0/2649059303 learned_addr learned my addr 172.21.15.43:0/2649059303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9e00640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdeac0b4140 0x7fdeac142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdeaa800640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 0x7fdeac06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 -- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdeac0b4140 msgr2=0x7fdeac142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdeac0b4140 0x7fdeac142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 -- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 msgr2=0x7fdeac06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 0x7fdeac06cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdea9400640 1 -- 172.21.15.43:0/2649059303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c009400 con 0x7fdeac1435d0 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.341+0000 7fdeaa800640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 0x7fdeac06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdea9400640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fde9400c410 tx=0x7fde9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde940074e0 con 0x7fdeac1435d0 2024-01-16T15:01:47.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde94007660 con 0x7fdeac1435d0 2024-01-16T15:01:47.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94004a20 con 0x7fdeac1435d0 2024-01-16T15:01:47.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac13be70 con 0x7fdeac1435d0 2024-01-16T15:01:47.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.342+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac1598a0 con 0x7fdeac1435d0 2024-01-16T15:01:47.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.344+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac13c390 con 0x7fdeac1435d0 2024-01-16T15:01:47.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.344+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fde94004b80 con 0x7fdeac1435d0 2024-01-16T15:01:47.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.345+0000 7fdea2000640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 0x7fde8406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.345+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde94089a40 con 0x7fdeac1435d0 2024-01-16T15:01:47.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.346+0000 7fdea9e00640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 0x7fde8406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.346+0000 7fdea9e00640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 0x7fde8406d810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde9c036000 tx=0x7fde9c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.350+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdeac13c390 con 0x7fdeac1435d0 2024-01-16T15:01:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.583+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeac13c390 con 0x7fde8406b350 2024-01-16T15:01:47.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.615+0000 7fdea2000640 1 -- 172.21.15.43:0/2649059303 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdeac13c390 con 0x7fde8406b350 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.617+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 msgr2=0x7fde8406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.617+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 0x7fde8406d810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde9c036000 tx=0x7fde9c03d040 comp rx=0 tx=0).stop 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.617+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 msgr2=0x7fdeac150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.617+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fde9400c410 tx=0x7fde9400c8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 shutdown_connections 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde8406b350 0x7fde8406d810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdeac1439b0 0x7fdeac06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1435d0 0x7fdeac150620 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 --2- 172.21.15.43:0/2649059303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdeac0b4140 0x7fdeac142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 >> 172.21.15.43:0/2649059303 conn(0x7fdeac0cec10 msgr2=0x7fdeac0cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:47.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 shutdown_connections 2024-01-16T15:01:47.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.618+0000 7fdeab200640 1 -- 172.21.15.43:0/2649059303 wait complete. 2024-01-16T15:01:47.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:01:47.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.863+0000 7f273ee00640 1 Processor -- start 2024-01-16T15:01:47.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.863+0000 7f273ee00640 1 -- start start 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f273ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 0x7f273806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f273ee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2738071780 con 0x7f273806cad0 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f273ee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27380711c0 con 0x7f2738160b10 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f273ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380b2320 con 0x7f2738072280 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f2737400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f273806cad0 0x7f27380b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59932/0 (socket says 172.21.15.43:59932) 2024-01-16T15:01:47.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.865+0000 7f2737400640 1 -- 172.21.15.43:0/892321997 learned_addr learned my addr 172.21.15.43:0/892321997 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f273ce00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 0x7f273806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919825234 0 0) 0x7f2738071780 con 0x7f273806cad0 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2718003520 con 0x7f273806cad0 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501813165 0 0) 0x7f2718003520 con 0x7f273806cad0 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 msgr2=0x7f273806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 0x7f273806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 >> v1:172.21.15.69:6789/0 conn(0x7f2738160b10 legacy=0x7f273806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380714a0 con 0x7f273806cad0 2024-01-16T15:01:47.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f27200019f0 con 0x7f273806cad0 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 >> v1:172.21.15.43:6789/0 conn(0x7f273806cad0 legacy=0x7f27380b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27180039a0 0x7f2718003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27180047c0 0x7f2718004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 0x7f27180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380b2320 con 0x7f27180039a0 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271800bab0 con 0x7f27180047c0 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.866+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271800bc70 con 0x7f2738072280 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27180039a0 0x7f2718003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737400640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27180047c0 0x7f2718004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 -- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 msgr2=0x7f27180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2738072280 0x7f27180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 -- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27180047c0 msgr2=0x7f2718004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:47.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27180047c0 0x7f2718004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.867+0000 7f2737e00640 1 -- 172.21.15.43:0/892321997 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2718003520 con 0x7f27180039a0 2024-01-16T15:01:47.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.868+0000 7f2737e00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27180039a0 0x7f2718003dc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2728001750 tx=0x7f27180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcbb047241a350 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27280052d0 con 0x7f27180039a0 2024-01-16T15:01:47.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272802e050 con 0x7f27180039a0 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f2736a00640 1 -- 172.21.15.43:0/892321997 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27280035f0 con 0x7f27180039a0 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 -- 172.21.15.43:0/892321997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27180039a0 msgr2=0x7f2718003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 --2- 172.21.15.43:0/892321997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27180039a0 0x7f2718003dc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2728001750 tx=0x7f27180057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ce00640 1 -- 172.21.15.43:0/892321997 reap_dead start 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 -- 172.21.15.43:0/892321997 shutdown_connections 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 -- 172.21.15.43:0/892321997 >> 172.21.15.43:0/892321997 conn(0x7f27380b0ba0 msgr2=0x7f27380b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 -- 172.21.15.43:0/892321997 shutdown_connections 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.869+0000 7f273ee00640 1 -- 172.21.15.43:0/892321997 wait complete. 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.871+0000 7f273ee00640 1 Processor -- start 2024-01-16T15:01:47.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.871+0000 7f273ee00640 1 -- start start 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27180039a0 0x7f27380cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 0x7f27380afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380ce880 con 0x7f27381f42c0 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27381f40d0 con 0x7f27381f4740 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.872+0000 7f273ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27381f3df0 con 0x7f27180039a0 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47456/0 (socket says 172.21.15.43:47456) 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 -- 172.21.15.43:0/3631465102 learned_addr learned my addr 172.21.15.43:0/3631465102 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:47.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f273ce00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 0x7f27380afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 -- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27180039a0 msgr2=0x7f27380cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737e00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27180039a0 0x7f27380cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27180039a0 0x7f27380cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 -- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 msgr2=0x7f27380afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 0x7f27380afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f2737400640 1 -- 172.21.15.43:0/3631465102 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380c8370 con 0x7f27381f42c0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.873+0000 7f273ce00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 0x7f27380afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f2737400640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2720007a80 tx=0x7f272000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720013420 con 0x7f27381f42c0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720013580 con 0x7f27381f42c0 2024-01-16T15:01:47.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27380c8650 con 0x7f27381f42c0 2024-01-16T15:01:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720013930 con 0x7f27381f42c0 2024-01-16T15:01:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.874+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27380cb1a0 con 0x7f27381f42c0 2024-01-16T15:01:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.875+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273809d200 con 0x7f27381f42c0 2024-01-16T15:01:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.876+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f2720013a90 con 0x7f27381f42c0 2024-01-16T15:01:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.877+0000 7f272fe00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 0x7f271006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.877+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f272008a540 con 0x7f27381f42c0 2024-01-16T15:01:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.877+0000 7f2737e00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 0x7f271006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.878+0000 7f2737e00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 0x7f271006d810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2728033000 tx=0x7f2728009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:47.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:47.882+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27200552c0 con 0x7f27381f42c0 2024-01-16T15:01:48.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.106+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2738071d50 con 0x7f271006b350 2024-01-16T15:01:48.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.119+0000 7f272fe00640 1 -- 172.21.15.43:0/3631465102 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f27280052f0 con 0x7f271006b350 2024-01-16T15:01:48.100 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 msgr2=0x7f271006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 0x7f271006d810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2728033000 tx=0x7f2728009030 comp rx=0 tx=0).stop 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 msgr2=0x7f27380af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2720007a80 tx=0x7f272000e530 comp rx=0 tx=0).stop 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 shutdown_connections 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f271006b350 0x7f271006d810 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27381f4740 0x7f27380afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27381f42c0 0x7f27380af610 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 --2- 172.21.15.43:0/3631465102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27180039a0 0x7f27380cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 >> 172.21.15.43:0/3631465102 conn(0x7f27380b0ba0 msgr2=0x7f27380abe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 shutdown_connections 2024-01-16T15:01:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:48.123+0000 7f273ee00640 1 -- 172.21.15.43:0/3631465102 wait complete. 2024-01-16T15:01:48.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:48.121 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36) 2024-01-16T15:01:48.182 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36 2024-01-16T15:01:48.182 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:01:48.182 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:01:48.183 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:01:50.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:50.184 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36) 2024-01-16T15:01:50.250 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/cbbdfa13-faa6-4ec7-a130-757211ed1f36 2024-01-16T15:01:50.250 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:01:50.250 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:01:50.251 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:01:50.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:01:50.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.497+0000 7f1a6be00640 1 Processor -- start 2024-01-16T15:01:50.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.497+0000 7f1a6be00640 1 -- start start 2024-01-16T15:01:50.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.499+0000 7f1a6be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 0x7f1a6c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.499+0000 7f1a6be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.499+0000 7f1a6be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c15f930 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.499+0000 7f1a6be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0b43b0 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.499+0000 7f1a6be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c15f510 con 0x7f1a6c144770 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47458/0 (socket says 172.21.15.43:47458) 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 -- 172.21.15.43:0/3604664480 learned_addr learned my addr 172.21.15.43:0/3604664480 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 -- 172.21.15.43:0/3604664480 >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 msgr2=0x7f1a6c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:50.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6aa00640 1 --2- 172.21.15.43:0/3604664480 >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 0x7f1a6c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 --2- 172.21.15.43:0/3604664480 >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 0x7f1a6c15eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 -- 172.21.15.43:0/3604664480 >> v1:172.21.15.69:6789/0 conn(0x7f1a6c0b35a0 legacy=0x7f1a6c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:50.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6b400640 1 -- 172.21.15.43:0/3604664480 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0ce9f0 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.500+0000 7f1a6aa00640 1 --2- 172.21.15.43:0/3604664480 >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 0x7f1a6c15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:50.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.501+0000 7f1a6b400640 1 --2- 172.21.15.43:0/3604664480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c06df90 tx=0x7f1a6402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa1c7fb0e1d077e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a69600640 1 -- 172.21.15.43:0/3604664480 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6402ca80 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a69600640 1 -- 172.21.15.43:0/3604664480 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6402cbe0 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a69600640 1 -- 172.21.15.43:0/3604664480 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a640343f0 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 -- 172.21.15.43:0/3604664480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c15f0a0 msgr2=0x7f1a6c0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 --2- 172.21.15.43:0/3604664480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c06df90 tx=0x7f1a6402bee0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 -- 172.21.15.43:0/3604664480 shutdown_connections 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 --2- 172.21.15.43:0/3604664480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c15f0a0 0x7f1a6c0ce4b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 --2- 172.21.15.43:0/3604664480 >> v2:172.21.15.69:3300/0 conn(0x7f1a6c144770 0x7f1a6c15eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.502+0000 7f1a6be00640 1 -- 172.21.15.43:0/3604664480 >> 172.21.15.43:0/3604664480 conn(0x7f1a6c0cec10 msgr2=0x7f1a6c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.503+0000 7f1a6be00640 1 -- 172.21.15.43:0/3604664480 shutdown_connections 2024-01-16T15:01:50.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.503+0000 7f1a6be00640 1 -- 172.21.15.43:0/3604664480 wait complete. 2024-01-16T15:01:50.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.504+0000 7f1a6be00640 1 Processor -- start 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.505+0000 7f1a6be00640 1 -- start start 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a6c144770 0x7f1a6c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a6c15f0a0 0x7f1a6c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15b8a0 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15cf90 con 0x7f1a6c144770 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15d270 con 0x7f1a6c15f0a0 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47474/0 (socket says 172.21.15.43:47474) 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 -- 172.21.15.43:0/1702891436 learned_addr learned my addr 172.21.15.43:0/1702891436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6a000640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a6c144770 0x7f1a6c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 -- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a6c15f0a0 msgr2=0x7f1a6c15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a6c15f0a0 0x7f1a6c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 -- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a6c144770 msgr2=0x7f1a6c15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a6c144770 0x7f1a6c15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.506+0000 7f1a6aa00640 1 -- 172.21.15.43:0/1702891436 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a64009b30 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a6aa00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c770 tx=0x7f1a5c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c007820 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c007980 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c007d10 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c15d550 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.507+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c15db10 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.508+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c15dd00 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.509+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f1a5c0113f0 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.510+0000 7f1a63400640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 0x7f1a4406d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:50.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.510+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a5c089e80 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.510+0000 7f1a6a000640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 0x7f1a4406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:50.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.511+0000 7f1a6a000640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 0x7f1a4406d860 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c13ed90 tx=0x7f1a54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:50.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.514+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a5c054c00 con 0x7f1a6c0b35a0 2024-01-16T15:01:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.748+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a6c0b40d0 con 0x7f1a4406b3a0 2024-01-16T15:01:50.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.889+0000 7f1a63400640 1 -- 172.21.15.43:0/1702891436 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1a54004020 con 0x7f1a4406b3a0 2024-01-16T15:01:50.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.892+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 msgr2=0x7f1a4406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.892+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 0x7f1a4406d860 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c13ed90 tx=0x7f1a54008040 comp rx=0 tx=0).stop 2024-01-16T15:01:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.892+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 msgr2=0x7f1a6c15e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.892+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c770 tx=0x7f1a5c00cb70 comp rx=0 tx=0).stop 2024-01-16T15:01:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 shutdown_connections 2024-01-16T15:01:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a4406b3a0 0x7f1a4406d860 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a6c144770 0x7f1a6c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a6c15f0a0 0x7f1a6c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 --2- 172.21.15.43:0/1702891436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a6c0b35a0 0x7f1a6c15e030 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:50.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 >> 172.21.15.43:0/1702891436 conn(0x7f1a6c0cec10 msgr2=0x7f1a6c0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:50.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 shutdown_connections 2024-01-16T15:01:50.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:50.893+0000 7f1a6be00640 1 -- 172.21.15.43:0/1702891436 wait complete. 2024-01-16T15:01:50.881 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.900+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.881 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.901+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.881 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.901+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.881 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.901+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.881 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.901+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.894 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:01:50.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:01:50.903 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.922+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.903 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.922+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.903 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.922+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.903 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.922+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:50.903 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:01:50.922+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:51.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.111+0000 7f42d0a00640 1 Processor -- start 2024-01-16T15:01:51.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.112+0000 7f42d0a00640 1 -- start start 2024-01-16T15:01:51.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.113+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42cc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.113+0000 7f42d0a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0c6bf0 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.113+0000 7f42d0a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0d1400 con 0x7f42cc0d1d90 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.113+0000 7f42d0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0cedc0 con 0x7f42cc15d3b0 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c1c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f42cc0d0fa0 0x7f42cc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59940/0 (socket says 172.21.15.43:59940) 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c1c00640 1 -- 172.21.15.43:0/2144368010 learned_addr learned my addr 172.21.15.43:0/2144368010 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:51.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42ca800640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42cc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619435791 0 0) 0x7f42cc0c6bf0 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ac003520 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3755945344 0 0) 0x7f42ac003520 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 msgr2=0x7f42cc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42cc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 >> v1:172.21.15.69:6789/0 conn(0x7f42cc0d1d90 legacy=0x7f42cc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.114+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0c6e40 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f42b4001670 con 0x7f42cc0d0fa0 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 >> v1:172.21.15.43:6789/0 conn(0x7f42cc0d0fa0 legacy=0x7f42cc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42ac003a10 0x7f42ac003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac004d00 0x7f42ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0cedc0 con 0x7f42ac004d00 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00bb60 con 0x7f42ac003a10 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00bd20 con 0x7f42cc15d3b0 2024-01-16T15:01:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42cb200640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac004d00 0x7f42ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42ca800640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42ac003a10 0x7f42ac003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.115+0000 7f42c1c00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.116+0000 7f42cb200640 1 -- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 msgr2=0x7f42ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.116+0000 7f42cb200640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc15d3b0 0x7f42ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.116+0000 7f42cb200640 1 -- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42ac003a10 msgr2=0x7f42ac003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.116+0000 7f42cb200640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42ac003a10 0x7f42ac003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.116+0000 7f42cb200640 1 -- 172.21.15.43:0/2144368010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac003520 con 0x7f42ac004d00 2024-01-16T15:01:51.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.117+0000 7f42cb200640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac004d00 0x7f42ac005180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f42c4001720 tx=0x7f42ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d0c9100f4190e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.117+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40052d0 con 0x7f42ac004d00 2024-01-16T15:01:51.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.117+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c402e050 con 0x7f42ac004d00 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.117+0000 7f42c9e00640 1 -- 172.21.15.43:0/2144368010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40035f0 con 0x7f42ac004d00 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 -- 172.21.15.43:0/2144368010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac004d00 msgr2=0x7f42ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 --2- 172.21.15.43:0/2144368010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac004d00 0x7f42ac005180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f42c4001720 tx=0x7f42ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42cb200640 1 -- 172.21.15.43:0/2144368010 reap_dead start 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 -- 172.21.15.43:0/2144368010 shutdown_connections 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 -- 172.21.15.43:0/2144368010 >> 172.21.15.43:0/2144368010 conn(0x7f42cc0d30a0 msgr2=0x7f42cc0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 -- 172.21.15.43:0/2144368010 shutdown_connections 2024-01-16T15:01:51.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.118+0000 7f42d0a00640 1 -- 172.21.15.43:0/2144368010 wait complete. 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.120+0000 7f42d0a00640 1 Processor -- start 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.120+0000 7f42d0a00640 1 -- start start 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 0x7f42cc157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc157fc0 0x7f42cc158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc06c910 con 0x7f42cc06b870 2024-01-16T15:01:51.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc158980 con 0x7f42cc157fc0 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42d0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0b6c70 con 0x7f42ac004d00 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42c1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42c1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47534/0 (socket says 172.21.15.43:47534) 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.121+0000 7f42c1c00640 1 -- 172.21.15.43:0/1169775179 learned_addr learned my addr 172.21.15.43:0/1169775179 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42ca800640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 0x7f42cc157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42cb200640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc157fc0 0x7f42cc158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 -- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 msgr2=0x7f42cc157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 0x7f42cc157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 -- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc157fc0 msgr2=0x7f42cc158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc157fc0 0x7f42cc158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 -- 172.21.15.43:0/1169775179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc06c350 con 0x7f42cc06b870 2024-01-16T15:01:51.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42ca800640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 0x7f42cc157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c1c00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e730 tx=0x7f42b400eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.122+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400c660 con 0x7f42cc06b870 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.123+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400c7e0 con 0x7f42cc06b870 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.123+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc06c630 con 0x7f42cc06b870 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.123+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400cb90 con 0x7f42cc06b870 2024-01-16T15:01:51.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.123+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc157650 con 0x7f42cc06b870 2024-01-16T15:01:51.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.124+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc0b6760 con 0x7f42cc06b870 2024-01-16T15:01:51.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.125+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f42b400ccf0 con 0x7f42cc06b870 2024-01-16T15:01:51.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.126+0000 7f42c3400640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 0x7f42a406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:51.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.126+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42b4089870 con 0x7f42cc06b870 2024-01-16T15:01:51.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.126+0000 7f42ca800640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 0x7f42a406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:51.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.127+0000 7f42ca800640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 0x7f42a406d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0074d0 tx=0x7f42bc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:51.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.130+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42cc0b6760 con 0x7f42cc06b870 2024-01-16T15:01:51.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:51.362+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f42cc0b5630 con 0x7f42a406b380 2024-01-16T15:01:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:53.777+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f42b40549c0 con 0x7f42cc06b870 2024-01-16T15:01:54.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:54.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:55.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.295+0000 7f42c3400640 1 -- 172.21.15.43:0/1169775179 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f42cc0b5630 con 0x7f42a406b380 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:55.276 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 msgr2=0x7f42a406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 0x7f42a406d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0074d0 tx=0x7f42bc00b040 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 msgr2=0x7f42cc06bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e730 tx=0x7f42b400eb30 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 shutdown_connections 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406b380 0x7f42a406d840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc157fc0 0x7f42cc158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06b870 0x7f42cc06bcf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 --2- 172.21.15.43:0/1169775179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42ac004d00 0x7f42cc157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.299+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 >> 172.21.15.43:0/1169775179 conn(0x7f42cc0d30a0 msgr2=0x7f42cc15e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.300+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 shutdown_connections 2024-01-16T15:01:55.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.300+0000 7f42d0a00640 1 -- 172.21.15.43:0/1169775179 wait complete. 2024-01-16T15:01:55.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:01:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.527+0000 7f14f1e00640 1 Processor -- start 2024-01-16T15:01:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.527+0000 7f14f1e00640 1 -- start start 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14ec13dfd0 0x7f14ec15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f1e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec0ce7c0 con 0x7f14ec0cf050 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f1e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec140900 con 0x7f14ec0cf500 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f1e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec06cf50 con 0x7f14ec13dfd0 2024-01-16T15:01:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14ec13dfd0 0x7f14ec15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14ec13dfd0 0x7f14ec15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50968/0 (socket says 172.21.15.43:50968) 2024-01-16T15:01:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14f0a00640 1 -- 172.21.15.43:0/37416565 learned_addr learned my addr 172.21.15.43:0/37416565 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.529+0000 7f14e3c00640 1 --1- 172.21.15.43:0/37416565 >> v1:172.21.15.43:6789/0 conn(0x7f14ec0cf050 0x7f14ec06c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41472/0 (socket says 172.21.15.43:41472) 2024-01-16T15:01:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.530+0000 7f14f0a00640 1 -- 172.21.15.43:0/37416565 >> v1:172.21.15.69:6789/0 conn(0x7f14ec0cf500 legacy=0x7f14ec06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.530+0000 7f14f0a00640 1 -- 172.21.15.43:0/37416565 >> v1:172.21.15.43:6789/0 conn(0x7f14ec0cf050 legacy=0x7f14ec06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.530+0000 7f14ebe00640 1 -- 172.21.15.43:0/37416565 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841750468 0 0) 0x7f14ec0ce7c0 con 0x7f14ec0cf050 2024-01-16T15:01:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.530+0000 7f14f0a00640 1 -- 172.21.15.43:0/37416565 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec15fa70 con 0x7f14ec13dfd0 2024-01-16T15:01:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.531+0000 7f14f0a00640 1 --2- 172.21.15.43:0/37416565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec13dfd0 0x7f14ec15be70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adaf6ceb7974bdec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.531+0000 7f14ebe00640 1 -- 172.21.15.43:0/37416565 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02f840 con 0x7f14ec13dfd0 2024-01-16T15:01:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.531+0000 7f14ebe00640 1 -- 172.21.15.43:0/37416565 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc02f9a0 con 0x7f14ec13dfd0 2024-01-16T15:01:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.531+0000 7f14ebe00640 1 -- 172.21.15.43:0/37416565 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02fd30 con 0x7f14ec13dfd0 2024-01-16T15:01:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 -- 172.21.15.43:0/37416565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec13dfd0 msgr2=0x7f14ec15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 --2- 172.21.15.43:0/37416565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec13dfd0 0x7f14ec15be70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc02ba20 comp rx=0 tx=0).stop 2024-01-16T15:01:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 -- 172.21.15.43:0/37416565 shutdown_connections 2024-01-16T15:01:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 --2- 172.21.15.43:0/37416565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec13dfd0 0x7f14ec15be70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 -- 172.21.15.43:0/37416565 >> 172.21.15.43:0/37416565 conn(0x7f14ec06a860 msgr2=0x7f14ec06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 -- 172.21.15.43:0/37416565 shutdown_connections 2024-01-16T15:01:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.532+0000 7f14f1e00640 1 -- 172.21.15.43:0/37416565 wait complete. 2024-01-16T15:01:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.534+0000 7f14f1e00640 1 Processor -- start 2024-01-16T15:01:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.534+0000 7f14f1e00640 1 -- start start 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14ec0cf500 0x7f14ec15e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14ec13dfd0 0x7f14ec070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec14e780 con 0x7f14ec0cf050 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec151050 con 0x7f14ec13dfd0 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f1e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec151330 con 0x7f14ec0cf500 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50976/0 (socket says 172.21.15.43:50976) 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.535+0000 7f14f0a00640 1 -- 172.21.15.43:0/254744112 learned_addr learned my addr 172.21.15.43:0/254744112 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f1400640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14ec13dfd0 0x7f14ec070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 -- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14ec0cf500 msgr2=0x7f14ec15e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14ec0cf500 0x7f14ec15e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 -- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14ec13dfd0 msgr2=0x7f14ec070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14ec13dfd0 0x7f14ec070ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 -- 172.21.15.43:0/254744112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc0094d0 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.536+0000 7f14f0a00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.537+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02f7f0 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.537+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc041400 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.537+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec14e130 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.537+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc041790 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.537+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec15eb60 con 0x7f14ec0cf050 2024-01-16T15:01:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.538+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec04e840 con 0x7f14ec0cf050 2024-01-16T15:01:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.539+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f14dc0418f0 con 0x7f14ec0cf050 2024-01-16T15:01:55.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.541+0000 7f14e9600640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 0x7f14c406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:55.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.541+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14dc03c070 con 0x7f14ec0cf050 2024-01-16T15:01:55.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.541+0000 7f14e3c00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 0x7f14c406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:55.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.542+0000 7f14e3c00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 0x7f14c406d930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007fd0 tx=0x7f14d4007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.546+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14dc0779e0 con 0x7f14ec0cf050 2024-01-16T15:01:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.817+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f14ec14e8c0 con 0x7f14ec0cf050 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.818+0000 7f14e9600640 1 -- 172.21.15.43:0/254744112 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1825 (secure 0 0 0) 0x7f14dc07b000 con 0x7f14ec0cf050 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:01:55.798 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 168, 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:01:55.799 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 35, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:01:55.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 793, 2024-01-16T15:01:55.801 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222033949, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 6689898496, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766404214784, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 46122820, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 11 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 2, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:01:55.802 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 2, 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5250 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:01:55.803 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7, 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:01:52.838104+0000", 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:01:55.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:55.805 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": {} 2024-01-16T15:01:55.805 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:01:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 msgr2=0x7f14c406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 0x7f14c406d930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007fd0 tx=0x7f14d4007900 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 msgr2=0x7f14ec13d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc004730 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 shutdown_connections 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14c406b470 0x7f14c406d930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14ec13dfd0 0x7f14ec070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14ec0cf500 0x7f14ec15e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 --2- 172.21.15.43:0/254744112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14ec0cf050 0x7f14ec13d740 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 >> 172.21.15.43:0/254744112 conn(0x7f14ec06a860 msgr2=0x7f14ec0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 shutdown_connections 2024-01-16T15:01:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:55.821+0000 7f14f1e00640 1 -- 172.21.15.43:0/254744112 wait complete. 2024-01-16T15:01:55.819 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:01:55.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:01:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.057+0000 7ff629600640 1 Processor -- start 2024-01-16T15:01:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.057+0000 7ff629600640 1 -- start start 2024-01-16T15:01:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.058+0000 7ff629600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff624070470 0x7ff62406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.058+0000 7ff629600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff6240cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.058+0000 7ff629600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff624142d50 con 0x7ff6240cd150 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.058+0000 7ff629600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6240cd530 con 0x7ff624070470 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.058+0000 7ff629600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62406f430 con 0x7ff62406ec60 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff622a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff6240cd150 0x7ff62406fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41474/0 (socket says 172.21.15.43:41474) 2024-01-16T15:01:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff622a00640 1 -- 172.21.15.43:0/1316380551 learned_addr learned my addr 172.21.15.43:0/1316380551 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff619e00640 1 --2- 172.21.15.43:0/1316380551 >> v2:172.21.15.69:3300/0 conn(0x7ff624070470 0x7ff62406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff623400640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff6240cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772111503 0 0) 0x7ff624142d50 con 0x7ff6240cd150 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.059+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604003520 con 0x7ff6240cd150 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2443414982 0 0) 0x7ff604003520 con 0x7ff6240cd150 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 msgr2=0x7ff6240cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff6240cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 >> v2:172.21.15.69:3300/0 conn(0x7ff624070470 msgr2=0x7ff62406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 --2- 172.21.15.43:0/1316380551 >> v2:172.21.15.69:3300/0 conn(0x7ff624070470 0x7ff62406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff624142fa0 con 0x7ff6240cd150 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff6140019f0 con 0x7ff6240cd150 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 >> v1:172.21.15.43:6789/0 conn(0x7ff6240cd150 legacy=0x7ff62406fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff604003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604003ef0 0x7ff6040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff604005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cd530 con 0x7ff604004d00 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62406f430 con 0x7ff604003ef0 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60400b9d0 con 0x7ff62406ec60 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.060+0000 7ff623400640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff604005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff619e00640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604003ef0 0x7ff6040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff622a00640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff604003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff623400640 1 -- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 msgr2=0x7ff604003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff623400640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff62406ec60 0x7ff604003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff623400640 1 -- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604003ef0 msgr2=0x7ff6040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff623400640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604003ef0 0x7ff6040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.061+0000 7ff623400640 1 -- 172.21.15.43:0/1316380551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff604003520 con 0x7ff604004d00 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.062+0000 7ff623400640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff604005180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009af0 tx=0x7ff6040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72ec812c3ea0586 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c02c6b0 con 0x7ff604004d00 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c02c810 con 0x7ff604004d00 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff622000640 1 -- 172.21.15.43:0/1316380551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c02cba0 con 0x7ff604004d00 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff629600640 1 -- 172.21.15.43:0/1316380551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 msgr2=0x7ff604005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff629600640 1 --2- 172.21.15.43:0/1316380551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff604005180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009af0 tx=0x7ff6040057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.063+0000 7ff623400640 1 -- 172.21.15.43:0/1316380551 reap_dead start 2024-01-16T15:01:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.064+0000 7ff629600640 1 -- 172.21.15.43:0/1316380551 shutdown_connections 2024-01-16T15:01:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.064+0000 7ff629600640 1 -- 172.21.15.43:0/1316380551 >> 172.21.15.43:0/1316380551 conn(0x7ff6240cdb20 msgr2=0x7ff6240cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.064+0000 7ff629600640 1 -- 172.21.15.43:0/1316380551 shutdown_connections 2024-01-16T15:01:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.064+0000 7ff629600640 1 -- 172.21.15.43:0/1316380551 wait complete. 2024-01-16T15:01:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.066+0000 7ff629600640 1 Processor -- start 2024-01-16T15:01:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.066+0000 7ff629600640 1 -- start start 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6240c9a30 0x7ff6240c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff62415a660 0x7ff62415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62415b030 con 0x7ff604004d00 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240ca5a0 con 0x7ff62415a660 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff629600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240ca880 con 0x7ff6240c9a30 2024-01-16T15:01:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff622a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff622a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51000/0 (socket says 172.21.15.43:51000) 2024-01-16T15:01:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff622a00640 1 -- 172.21.15.43:0/2320126551 learned_addr learned my addr 172.21.15.43:0/2320126551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff622a00640 1 -- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6240c9a30 msgr2=0x7ff6240c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.067+0000 7ff622a00640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6240c9a30 0x7ff6240c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff622a00640 1 -- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff62415a660 msgr2=0x7ff62415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff622a00640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff62415a660 0x7ff62415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff622a00640 1 -- 172.21.15.43:0/2320126551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240cab60 con 0x7ff604004d00 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff622a00640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff61400dfd0 tx=0x7ff61400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614004d60 con 0x7ff604004d00 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6140070d0 con 0x7ff604004d00 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614007480 con 0x7ff604004d00 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6240cae40 con 0x7ff604004d00 2024-01-16T15:01:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.068+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6241f3340 con 0x7ff604004d00 2024-01-16T15:01:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.070+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6240cb060 con 0x7ff604004d00 2024-01-16T15:01:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.070+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7ff6140075e0 con 0x7ff604004d00 2024-01-16T15:01:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.071+0000 7ff61b400640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 0x7ff5fc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:56.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.071+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff614089fe0 con 0x7ff604004d00 2024-01-16T15:01:56.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.072+0000 7ff619e00640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 0x7ff5fc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:56.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.072+0000 7ff619e00640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 0x7ff5fc06d930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff6240d20e0 tx=0x7ff60c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.076+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff614054b70 con 0x7ff604004d00 2024-01-16T15:01:56.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.307+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff62406f110 con 0x7ff5fc06b470 2024-01-16T15:01:56.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.319+0000 7ff61b400640 1 -- 172.21.15.43:0/2320126551 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7ff60c011860 con 0x7ff5fc06b470 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 0.0, 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:01:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "0/3", 2024-01-16T15:01:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:01:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:01:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:56.301 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:01:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 msgr2=0x7ff5fc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 0x7ff5fc06d930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff6240d20e0 tx=0x7ff60c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 msgr2=0x7ff624158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff61400dfd0 tx=0x7ff61400b560 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 shutdown_connections 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5fc06b470 0x7ff5fc06d930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff62415a660 0x7ff62415aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6240c9a30 0x7ff6240c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 --2- 172.21.15.43:0/2320126551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff604004d00 0x7ff624158a80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.322+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 >> 172.21.15.43:0/2320126551 conn(0x7ff6240cdb20 msgr2=0x7ff62415ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.323+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 shutdown_connections 2024-01-16T15:01:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:56.323+0000 7ff629600640 1 -- 172.21.15.43:0/2320126551 wait complete. 2024-01-16T15:01:56.321 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:01:57.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:01:57.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.606+0000 7f329e400640 1 Processor -- start 2024-01-16T15:01:57.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.607+0000 7f329e400640 1 -- start start 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f329e400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32980b48d0 0x7f32980b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f329e400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32980b23e0 con 0x7f32980b3210 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f329e400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32980d41b0 con 0x7f3298160bd0 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f329e400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980cdc70 con 0x7f32980b48d0 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f328fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32980b48d0 0x7f32980b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f328fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32980b48d0 0x7f32980b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51002/0 (socket says 172.21.15.43:51002) 2024-01-16T15:01:57.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.608+0000 7f328fe00640 1 -- 172.21.15.43:0/4072409332 learned_addr learned my addr 172.21.15.43:0/4072409332 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:57.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.609+0000 7f328fe00640 1 -- 172.21.15.43:0/4072409332 >> v1:172.21.15.69:6789/0 conn(0x7f3298160bd0 legacy=0x7f32980b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.609+0000 7f328fe00640 1 -- 172.21.15.43:0/4072409332 >> v1:172.21.15.43:6789/0 conn(0x7f32980b3210 legacy=0x7f3298167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.609+0000 7f328fe00640 1 -- 172.21.15.43:0/4072409332 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298160fb0 con 0x7f32980b48d0 2024-01-16T15:01:57.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.610+0000 7f328fe00640 1 --2- 172.21.15.43:0/4072409332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32980b1ea0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f32980cddb0 tx=0x7f328002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9032de6fb6c3aec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.611+0000 7f3296a00640 1 -- 172.21.15.43:0/4072409332 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280030840 con 0x7f32980b48d0 2024-01-16T15:01:57.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.611+0000 7f3296a00640 1 -- 172.21.15.43:0/4072409332 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32800309a0 con 0x7f32980b48d0 2024-01-16T15:01:57.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.611+0000 7f3296a00640 1 -- 172.21.15.43:0/4072409332 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280030d30 con 0x7f32980b48d0 2024-01-16T15:01:57.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 -- 172.21.15.43:0/4072409332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 msgr2=0x7f32980b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 --2- 172.21.15.43:0/4072409332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32980b1ea0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f32980cddb0 tx=0x7f328002b980 comp rx=0 tx=0).stop 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 -- 172.21.15.43:0/4072409332 shutdown_connections 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 --2- 172.21.15.43:0/4072409332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32980b1ea0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 -- 172.21.15.43:0/4072409332 >> 172.21.15.43:0/4072409332 conn(0x7f329809c5a0 msgr2=0x7f32980d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 -- 172.21.15.43:0/4072409332 shutdown_connections 2024-01-16T15:01:57.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.612+0000 7f329e400640 1 -- 172.21.15.43:0/4072409332 wait complete. 2024-01-16T15:01:57.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.614+0000 7f329e400640 1 Processor -- start 2024-01-16T15:01:57.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.615+0000 7f329e400640 1 -- start start 2024-01-16T15:01:57.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 0x7f32980b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32981f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981f76e0 con 0x7f32980b48d0 2024-01-16T15:01:57.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32980b64d0 con 0x7f32980b3210 2024-01-16T15:01:57.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.619+0000 7f329e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32980b67b0 con 0x7f3298160bd0 2024-01-16T15:01:57.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.620+0000 7f3297e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.620+0000 7f3297e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46674/0 (socket says 172.21.15.43:46674) 2024-01-16T15:01:57.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.620+0000 7f3297e00640 1 -- 172.21.15.43:0/2896217090 learned_addr learned my addr 172.21.15.43:0/2896217090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:57.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 0x7f32980b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f328fe00640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32981f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297e00640 1 -- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 msgr2=0x7f32980b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297e00640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 0x7f32980b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297e00640 1 -- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 msgr2=0x7f32981f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297e00640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32981f71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f3297e00640 1 -- 172.21.15.43:0/2896217090 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800094d0 con 0x7f3298160bd0 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.621+0000 7f328fe00640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32981f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.622+0000 7f3297e00640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f329000c710 tx=0x7f329000cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.622+0000 7f3297400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 0x7f32980b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.623+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290006d50 con 0x7f3298160bd0 2024-01-16T15:01:57.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.623+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329000ce20 con 0x7f3298160bd0 2024-01-16T15:01:57.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.623+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290003b70 con 0x7f3298160bd0 2024-01-16T15:01:57.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.623+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32980b6a90 con 0x7f3298160bd0 2024-01-16T15:01:57.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.623+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32980b7050 con 0x7f3298160bd0 2024-01-16T15:01:57.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.626+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3268005250 con 0x7f3298160bd0 2024-01-16T15:01:57.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.627+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f3290003cd0 con 0x7f3298160bd0 2024-01-16T15:01:57.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.630+0000 7f328f400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 0x7f327006d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:57.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.630+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3290089a10 con 0x7f3298160bd0 2024-01-16T15:01:57.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.633+0000 7f3297400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 0x7f327006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:57.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.635+0000 7f3297400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 0x7f327006d980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3288001660 tx=0x7f3288009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:57.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:57.637+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3290054670 con 0x7f3298160bd0 2024-01-16T15:01:57.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.015+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3268000bc0 con 0x7f327006b4c0 2024-01-16T15:01:58.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.030+0000 7f328f400640 1 -- 172.21.15.43:0/2896217090 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f3268000bc0 con 0x7f327006b4c0 2024-01-16T15:01:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 0.0, 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "0/3", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:01:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:01:58.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.034+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 msgr2=0x7f327006d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.034+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 0x7f327006d980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3288001660 tx=0x7f3288009040 comp rx=0 tx=0).stop 2024-01-16T15:01:58.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.034+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 msgr2=0x7f32980b80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:58.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.034+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f329000c710 tx=0x7f329000cbd0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 shutdown_connections 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f327006b4c0 0x7f327006d980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3298160bd0 0x7f32980b80b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32980b48d0 0x7f32981f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 --2- 172.21.15.43:0/2896217090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32980b3210 0x7f32980b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.035+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 >> 172.21.15.43:0/2896217090 conn(0x7f329809c5a0 msgr2=0x7f329806c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.036+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 shutdown_connections 2024-01-16T15:01:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:58.037+0000 7f329e400640 1 -- 172.21.15.43:0/2896217090 wait complete. 2024-01-16T15:01:58.037 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:01:59.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:01:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.427+0000 7fc59a200640 1 Processor -- start 2024-01-16T15:01:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.427+0000 7fc59a200640 1 -- start start 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc59a200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc594070470 0x7fc59406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc59a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc59406ec60 0x7fc5940cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc59a200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc594142d50 con 0x7fc5940cd150 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc59a200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5940cd530 con 0x7fc594070470 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc59a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59406f430 con 0x7fc59406ec60 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc593400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc5940cd150 0x7fc59406fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41486/0 (socket says 172.21.15.43:41486) 2024-01-16T15:01:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.429+0000 7fc593400640 1 -- 172.21.15.43:0/2972649790 learned_addr learned my addr 172.21.15.43:0/2972649790 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:01:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.430+0000 7fc593e00640 1 --2- 172.21.15.43:0/2972649790 >> v2:172.21.15.69:3300/0 conn(0x7fc59406ec60 0x7fc5940cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.430+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146432615 0 0) 0x7fc594142d50 con 0x7fc5940cd150 2024-01-16T15:01:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.430+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574003520 con 0x7fc5940cd150 2024-01-16T15:01:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.430+0000 7fc592a00640 1 --2- 172.21.15.43:0/2972649790 >> v2:172.21.15.43:3300/0 conn(0x7fc594070470 0x7fc59406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813064134 0 0) 0x7fc574003520 con 0x7fc5940cd150 2024-01-16T15:01:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 >> v2:172.21.15.69:3300/0 conn(0x7fc59406ec60 msgr2=0x7fc5940cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 --2- 172.21.15.43:0/2972649790 >> v2:172.21.15.69:3300/0 conn(0x7fc59406ec60 0x7fc5940cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 >> v2:172.21.15.43:3300/0 conn(0x7fc594070470 msgr2=0x7fc59406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 --2- 172.21.15.43:0/2972649790 >> v2:172.21.15.43:3300/0 conn(0x7fc594070470 0x7fc59406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc594142fa0 con 0x7fc5940cd150 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc584001670 con 0x7fc5940cd150 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 >> v1:172.21.15.43:6789/0 conn(0x7fc5940cd150 legacy=0x7fc59406fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574003970 0x7fc574003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5740047c0 0x7fc574004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 0x7fc57400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940cd530 con 0x7fc5740047c0 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59406f430 con 0x7fc574003970 2024-01-16T15:01:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.431+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57400bfc0 con 0x7fc574005180 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc593e00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 0x7fc57400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc592a00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5740047c0 0x7fc574004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc593400640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574003970 0x7fc574003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc593e00640 1 -- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574003970 msgr2=0x7fc574003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc593e00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574003970 0x7fc574003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.432+0000 7fc593e00640 1 -- 172.21.15.43:0/2972649790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5740047c0 msgr2=0x7fc574004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.433+0000 7fc593e00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5740047c0 0x7fc574004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.433+0000 7fc593e00640 1 -- 172.21.15.43:0/2972649790 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc594142fa0 con 0x7fc574005180 2024-01-16T15:01:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.433+0000 7fc592a00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5740047c0 0x7fc574004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.434+0000 7fc593e00640 1 -- 172.21.15.43:0/2972649790 reap_dead start 2024-01-16T15:01:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.435+0000 7fc593e00640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 0x7fc57400b980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009fc0 tx=0x7fc5740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26a6f014191c4cba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.436+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c02cc60 con 0x7fc574005180 2024-01-16T15:01:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.436+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c00f040 con 0x7fc574005180 2024-01-16T15:01:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.436+0000 7fc592000640 1 -- 172.21.15.43:0/2972649790 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c003990 con 0x7fc574005180 2024-01-16T15:01:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.437+0000 7fc59a200640 1 -- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 msgr2=0x7fc57400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.437+0000 7fc59a200640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 0x7fc57400b980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009fc0 tx=0x7fc5740057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.442+0000 7fc59a200640 1 -- 172.21.15.43:0/2972649790 shutdown_connections 2024-01-16T15:01:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.442+0000 7fc59a200640 1 --2- 172.21.15.43:0/2972649790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc574005180 0x7fc57400b980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.442+0000 7fc59a200640 1 -- 172.21.15.43:0/2972649790 >> 172.21.15.43:0/2972649790 conn(0x7fc5940cdb20 msgr2=0x7fc5940cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.443+0000 7fc59a200640 1 -- 172.21.15.43:0/2972649790 shutdown_connections 2024-01-16T15:01:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.443+0000 7fc59a200640 1 -- 172.21.15.43:0/2972649790 wait complete. 2024-01-16T15:01:59.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.449+0000 7fc59a200640 1 Processor -- start 2024-01-16T15:01:59.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.450+0000 7fc59a200640 1 -- start start 2024-01-16T15:01:59.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.451+0000 7fc59a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574005180 0x7fc594146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.453+0000 7fc59a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.454+0000 7fc59a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5941ff370 0x7fc59406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.454+0000 7fc59a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940b5970 con 0x7fc5941ff370 2024-01-16T15:01:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.454+0000 7fc59a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940b41d0 con 0x7fc574005180 2024-01-16T15:01:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.454+0000 7fc59a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940b4450 con 0x7fc5940b54f0 2024-01-16T15:01:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.455+0000 7fc592a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.455+0000 7fc592a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46694/0 (socket says 172.21.15.43:46694) 2024-01-16T15:01:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.455+0000 7fc592a00640 1 -- 172.21.15.43:0/121785102 learned_addr learned my addr 172.21.15.43:0/121785102 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:01:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.456+0000 7fc593e00640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5941ff370 0x7fc59406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.456+0000 7fc593400640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574005180 0x7fc594146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.457+0000 7fc592a00640 1 -- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574005180 msgr2=0x7fc594146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.457+0000 7fc592a00640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574005180 0x7fc594146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.457+0000 7fc592a00640 1 -- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5941ff370 msgr2=0x7fc59406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.457+0000 7fc592a00640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5941ff370 0x7fc59406c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.457+0000 7fc592a00640 1 -- 172.21.15.43:0/121785102 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940b4730 con 0x7fc5940b54f0 2024-01-16T15:01:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.459+0000 7fc592a00640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011a50 tx=0x7fc57c011f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.460+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c00fb90 con 0x7fc5940b54f0 2024-01-16T15:01:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.461+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c00fcf0 con 0x7fc5940b54f0 2024-01-16T15:01:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.461+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c014660 con 0x7fc5940b54f0 2024-01-16T15:01:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.461+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5940b4a10 con 0x7fc5940b54f0 2024-01-16T15:01:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.461+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5940b4fd0 con 0x7fc5940b54f0 2024-01-16T15:01:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.463+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564005250 con 0x7fc5940b54f0 2024-01-16T15:01:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.466+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fc57c0147c0 con 0x7fc5940b54f0 2024-01-16T15:01:59.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.468+0000 7fc58b400640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 0x7fc56c06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:59.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.469+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc57c08a890 con 0x7fc5940b54f0 2024-01-16T15:01:59.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.472+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc57c055610 con 0x7fc5940b54f0 2024-01-16T15:01:59.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.475+0000 7fc593400640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 0x7fc56c06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.480+0000 7fc593400640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 0x7fc56c06d9b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc584000fb0 tx=0x7fc584007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.741+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc564000bc0 con 0x7fc56c06b4f0 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.751+0000 7fc58b400640 1 -- 172.21.15.43:0/121785102 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fc564000bc0 con 0x7fc56c06b4f0 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:01:59.731 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "1/3", 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:01:59.732 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:01:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.753+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 msgr2=0x7fc56c06d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 0x7fc56c06d9b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc584000fb0 tx=0x7fc584007040 comp rx=0 tx=0).stop 2024-01-16T15:01:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 msgr2=0x7fc5940b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011a50 tx=0x7fc57c011f10 comp rx=0 tx=0).stop 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 shutdown_connections 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc56c06b4f0 0x7fc56c06d9b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5941ff370 0x7fc59406c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5940b54f0 0x7fc5940b3c90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 --2- 172.21.15.43:0/121785102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc574005180 0x7fc594146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 >> 172.21.15.43:0/121785102 conn(0x7fc5940cdb20 msgr2=0x7fc59415fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 shutdown_connections 2024-01-16T15:01:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:01:59.754+0000 7fc59a200640 1 -- 172.21.15.43:0/121785102 wait complete. 2024-01-16T15:01:59.751 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:00.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.013+0000 7f3cb9c00640 1 Processor -- start 2024-01-16T15:02:00.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.014+0000 7f3cb9c00640 1 -- start start 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.015+0000 7f3cb9c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3cb4160b10 0x7f3cb406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.015+0000 7f3cb9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4072280 0x7f3cb406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.015+0000 7f3cb9c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb4071780 con 0x7f3cb406c900 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.015+0000 7f3cb9c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb40b2320 con 0x7f3cb4160b10 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.015+0000 7f3cb9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb406cd90 con 0x7f3cb4072280 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3cb4160b10 0x7f3cb406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3cb4160b10 0x7f3cb406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51038/0 (socket says 172.21.15.43:51038) 2024-01-16T15:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 -- 172.21.15.43:0/3677893811 learned_addr learned my addr 172.21.15.43:0/3677893811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cabe00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4072280 0x7f3cb406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 -- 172.21.15.43:0/3677893811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4072280 msgr2=0x7f3cb406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4072280 0x7f3cb406c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.016+0000 7f3cb2a00640 1 -- 172.21.15.43:0/3677893811 >> v1:172.21.15.69:6789/0 conn(0x7f3cb406c900 legacy=0x7f3cb4160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.017+0000 7f3cb2a00640 1 -- 172.21.15.43:0/3677893811 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb40711c0 con 0x7f3cb4160b10 2024-01-16T15:02:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.018+0000 7f3cb2a00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb4160b10 0x7f3cb406e420 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009870 tx=0x7f3ca402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6799836380f9ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb2000640 1 -- 172.21.15.43:0/3677893811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4031640 con 0x7f3cb4160b10 2024-01-16T15:02:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb2000640 1 -- 172.21.15.43:0/3677893811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca40317a0 con 0x7f3cb4160b10 2024-01-16T15:02:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb2000640 1 -- 172.21.15.43:0/3677893811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4031b30 con 0x7f3cb4160b10 2024-01-16T15:02:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 -- 172.21.15.43:0/3677893811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb4160b10 msgr2=0x7f3cb406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb4160b10 0x7f3cb406e420 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009870 tx=0x7f3ca402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 -- 172.21.15.43:0/3677893811 shutdown_connections 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4072280 0x7f3cb406c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/3677893811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb4160b10 0x7f3cb406e420 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 -- 172.21.15.43:0/3677893811 >> 172.21.15.43:0/3677893811 conn(0x7f3cb40b0ba0 msgr2=0x7f3cb40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.019+0000 7f3cb9c00640 1 -- 172.21.15.43:0/3677893811 shutdown_connections 2024-01-16T15:02:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.020+0000 7f3cb9c00640 1 -- 172.21.15.43:0/3677893811 wait complete. 2024-01-16T15:02:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.021+0000 7f3cb9c00640 1 Processor -- start 2024-01-16T15:02:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.021+0000 7f3cb9c00640 1 -- start start 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.022+0000 7f3cb9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.022+0000 7f3cb9c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 0x7f3cb409a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 0x7f3cb409a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb9c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40cbf00 con 0x7f3cb406c900 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb9c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb409af80 con 0x7f3cb4072280 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb409b260 con 0x7f3cb4160b10 2024-01-16T15:02:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51044/0 (socket says 172.21.15.43:51044) 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 -- 172.21.15.43:0/2179547876 learned_addr learned my addr 172.21.15.43:0/2179547876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cabe00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 0x7f3cb409a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 -- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 msgr2=0x7f3cb409a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb3400640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 0x7f3cb409a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 0x7f3cb409a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 -- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 msgr2=0x7f3cb409a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 0x7f3cb409a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb2a00640 1 -- 172.21.15.43:0/2179547876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca40094d0 con 0x7f3cb406c900 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cabe00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 0x7f3cb409a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.023+0000 7f3cb3400640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 0x7f3cb409a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3cb2a00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009870 tx=0x7f3ca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca403ac60 con 0x7f3cb406c900 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca403ade0 con 0x7f3cb406c900 2024-01-16T15:02:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4044630 con 0x7f3cb406c900 2024-01-16T15:02:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb409b540 con 0x7f3cb406c900 2024-01-16T15:02:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.024+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb41578a0 con 0x7f3cb406c900 2024-01-16T15:02:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.025+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb409ba60 con 0x7f3cb406c900 2024-01-16T15:02:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.026+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f3ca402f050 con 0x7f3cb406c900 2024-01-16T15:02:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.027+0000 7f3caaa00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 0x7f3c8c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.027+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ca40abff0 con 0x7f3cb406c900 2024-01-16T15:02:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.027+0000 7f3cabe00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 0x7f3c8c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.028+0000 7f3cabe00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 0x7f3c8c06d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.032+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ca40dee90 con 0x7f3cb406c900 2024-01-16T15:02:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.269+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cb40b1f40 con 0x7f3c8c06b470 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.278+0000 7f3caaa00640 1 -- 172.21.15.43:0/2179547876 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f3c9c009530 con 0x7f3c8c06b470 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:02:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "1/3", 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:01.259 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.280+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 msgr2=0x7f3c8c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.280+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 0x7f3c8c06d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).stop 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.280+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 msgr2=0x7f3cb4154b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.280+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009870 tx=0x7f3ca40046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 shutdown_connections 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c8c06b470 0x7f3c8c06d930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb4072280 0x7f3cb409a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb4160b10 0x7f3cb409a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 --2- 172.21.15.43:0/2179547876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3cb406c900 0x7f3cb4154b80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 >> 172.21.15.43:0/2179547876 conn(0x7f3cb40b0ba0 msgr2=0x7f3cb40ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 shutdown_connections 2024-01-16T15:02:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:01.281+0000 7f3cb9c00640 1 -- 172.21.15.43:0/2179547876 wait complete. 2024-01-16T15:02:01.279 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:02.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:02.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.620+0000 7fadd1600640 1 Processor -- start 2024-01-16T15:02:02.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.621+0000 7fadd1600640 1 -- start start 2024-01-16T15:02:02.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.623+0000 7fadd1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fadcc162220 0x7fadcc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.623+0000 7fadd1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fadcc06cad0 0x7fadcc162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.623+0000 7fadd1600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc162fa0 con 0x7fadcc162600 2024-01-16T15:02:02.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.623+0000 7fadd1600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc0706a0 con 0x7fadcc06cad0 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.623+0000 7fadd1600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc070c60 con 0x7fadcc162220 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.624+0000 7fadcaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fadcc162220 0x7fadcc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.624+0000 7fadcaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fadcc162220 0x7fadcc0b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:57468/0 (socket says 172.21.15.43:57468) 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.624+0000 7fadcaa00640 1 -- 172.21.15.43:0/4039839965 learned_addr learned my addr 172.21.15.43:0/4039839965 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.624+0000 7fadcb400640 1 --2- 172.21.15.43:0/4039839965 >> v2:172.21.15.43:3300/0 conn(0x7fadcc06cad0 0x7fadcc162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658481474 0 0) 0x7fadcc162fa0 con 0x7fadcc162600 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac003520 con 0x7fadcc162600 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381370139 0 0) 0x7fadac003520 con 0x7fadcc162600 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 >> v2:172.21.15.69:3300/0 conn(0x7fadcc162220 msgr2=0x7fadcc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 --2- 172.21.15.43:0/4039839965 >> v2:172.21.15.69:3300/0 conn(0x7fadcc162220 0x7fadcc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.625+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 >> v2:172.21.15.43:3300/0 conn(0x7fadcc06cad0 msgr2=0x7fadcc162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 --2- 172.21.15.43:0/4039839965 >> v2:172.21.15.43:3300/0 conn(0x7fadcc06cad0 0x7fadcc162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc0722b0 con 0x7fadcc162600 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fadb4001650 con 0x7fadcc162600 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 >> v1:172.21.15.43:6789/0 conn(0x7fadcc162600 legacy=0x7fadcc06c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 0x7fadac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.626+0000 7fadc9600640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0047c0 0x7fadac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.627+0000 7fadc9600640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadac005180 0x7fadac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.627+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc0706a0 con 0x7fadac005180 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.627+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc070c60 con 0x7fadac0047c0 2024-01-16T15:02:02.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.627+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac00c190 con 0x7fadac0039c0 2024-01-16T15:02:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.627+0000 7fadcb400640 1 --2- 172.21.15.43:0/4039839965 >> v2:172.21.15.43:3300/0 conn(0x7fadcc06cad0 0x7fadcc162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.628+0000 7fadcaa00640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 0x7fadac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.628+0000 7fadcb400640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadac005180 0x7fadac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.628+0000 7fadca000640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0047c0 0x7fadac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcaa00640 1 -- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0047c0 msgr2=0x7fadac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcaa00640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0047c0 0x7fadac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcaa00640 1 -- 172.21.15.43:0/4039839965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadac005180 msgr2=0x7fadac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcaa00640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadac005180 0x7fadac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcaa00640 1 -- 172.21.15.43:0/4039839965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac003520 con 0x7fadac0039c0 2024-01-16T15:02:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.629+0000 7fadcb400640 1 -- 172.21.15.43:0/4039839965 reap_dead start 2024-01-16T15:02:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.631+0000 7fadca000640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0047c0 0x7fadac004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.631+0000 7fadcaa00640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 0x7fadac003e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0098d0 tx=0x7fadac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70ec8d4230ab5727 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.632+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc037600 con 0x7fadac0039c0 2024-01-16T15:02:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.632+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc037760 con 0x7fadac0039c0 2024-01-16T15:02:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.632+0000 7fadc9600640 1 -- 172.21.15.43:0/4039839965 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc037af0 con 0x7fadac0039c0 2024-01-16T15:02:02.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.634+0000 7fadd1600640 1 -- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 msgr2=0x7fadac003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.634+0000 7fadd1600640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 0x7fadac003e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0098d0 tx=0x7fadac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.638+0000 7fadd1600640 1 -- 172.21.15.43:0/4039839965 shutdown_connections 2024-01-16T15:02:02.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.638+0000 7fadd1600640 1 --2- 172.21.15.43:0/4039839965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadac0039c0 0x7fadac003e00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.638+0000 7fadd1600640 1 -- 172.21.15.43:0/4039839965 >> 172.21.15.43:0/4039839965 conn(0x7fadcc0b0ba0 msgr2=0x7fadcc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.639+0000 7fadd1600640 1 -- 172.21.15.43:0/4039839965 shutdown_connections 2024-01-16T15:02:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.639+0000 7fadd1600640 1 -- 172.21.15.43:0/4039839965 wait complete. 2024-01-16T15:02:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.641+0000 7fadd1600640 1 Processor -- start 2024-01-16T15:02:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.642+0000 7fadd1600640 1 -- start start 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0039c0 0x7fadcc1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadcc159fc0 0x7fadcc1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc15b800 con 0x7fadcc159b40 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc09c450 con 0x7fadac0039c0 2024-01-16T15:02:02.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.643+0000 7fadd1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc09b8d0 con 0x7fadcc159fc0 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.644+0000 7fadca000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.644+0000 7fadcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0039c0 0x7fadcc1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.644+0000 7fadca000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51078/0 (socket says 172.21.15.43:51078) 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.644+0000 7fadca000640 1 -- 172.21.15.43:0/2798027080 learned_addr learned my addr 172.21.15.43:0/2798027080 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.644+0000 7fadcb400640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadcc159fc0 0x7fadcc1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.645+0000 7fadca000640 1 -- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadcc159fc0 msgr2=0x7fadcc1594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.645+0000 7fadca000640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadcc159fc0 0x7fadcc1594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.645+0000 7fadca000640 1 -- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0039c0 msgr2=0x7fadcc1564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:02.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.645+0000 7fadca000640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0039c0 0x7fadcc1564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:02.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.645+0000 7fadca000640 1 -- 172.21.15.43:0/2798027080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc09bbb0 con 0x7fadcc159b40 2024-01-16T15:02:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.646+0000 7fadca000640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fadb4001ab0 tx=0x7fadb400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.646+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb40050d0 con 0x7fadcc159b40 2024-01-16T15:02:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.647+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb4005230 con 0x7fadcc159b40 2024-01-16T15:02:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.647+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc09be90 con 0x7fadcc159b40 2024-01-16T15:02:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.647+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb400c6c0 con 0x7fadcc159b40 2024-01-16T15:02:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.648+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc156a10 con 0x7fadcc159b40 2024-01-16T15:02:02.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.650+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fadb400c820 con 0x7fadcc159b40 2024-01-16T15:02:02.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.651+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc15b970 con 0x7fadcc159b40 2024-01-16T15:02:02.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.652+0000 7fadc2a00640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 0x7fada406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:02.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.652+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb40895a0 con 0x7fadcc159b40 2024-01-16T15:02:02.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.660+0000 7fadcaa00640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 0x7fada406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:02.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.664+0000 7fadcaa00640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 0x7fada406d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadbc036000 tx=0x7fadbc031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:02.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:02.665+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fadb4054100 con 0x7fadcc159b40 2024-01-16T15:02:03.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.094+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fadcc0ae160 con 0x7fada406b4a0 2024-01-16T15:02:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.107+0000 7fadc2a00640 1 -- 172.21.15.43:0/2798027080 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fadcc0ae160 con 0x7fada406b4a0 2024-01-16T15:02:03.088 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "1/3", 2024-01-16T15:02:03.089 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:03.090 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:03.090 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:03.090 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:03.090 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:03.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 msgr2=0x7fada406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 0x7fada406d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadbc036000 tx=0x7fadbc031000 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 msgr2=0x7fadcc158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fadb4001ab0 tx=0x7fadb400ec10 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 shutdown_connections 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fada406b4a0 0x7fada406d960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadcc159fc0 0x7fadcc1594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadcc159b40 0x7fadcc158f80 secure :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fadb4001ab0 tx=0x7fadb400ec10 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 --2- 172.21.15.43:0/2798027080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadac0039c0 0x7fadcc1564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.115+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 >> 172.21.15.43:0/2798027080 conn(0x7fadcc0b0ba0 msgr2=0x7fadcc0abe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:03.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.121+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 shutdown_connections 2024-01-16T15:02:03.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:03.121+0000 7fadd1600640 1 -- 172.21.15.43:0/2798027080 wait complete. 2024-01-16T15:02:03.135 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:04.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:04.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.387+0000 7fd659a00640 1 Processor -- start 2024-01-16T15:02:04.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.387+0000 7fd659a00640 1 -- start start 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.388+0000 7fd659a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd65415f0a0 0x7fd6540ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65415f930 con 0x7fd6540b35a0 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6540b43b0 con 0x7fd65415f0a0 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65415f530 con 0x7fd654144770 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd65415f0a0 0x7fd6540ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd65415f0a0 0x7fd6540ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43838/0 (socket says 172.21.15.43:43838) 2024-01-16T15:02:04.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd659000640 1 -- 172.21.15.43:0/1304606748 learned_addr learned my addr 172.21.15.43:0/1304606748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:04.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.389+0000 7fd653e00640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.390+0000 7fd659000640 1 -- 172.21.15.43:0/1304606748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 msgr2=0x7fd65415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:04.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.390+0000 7fd659000640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.390+0000 7fd659000640 1 -- 172.21.15.43:0/1304606748 >> v1:172.21.15.69:6789/0 conn(0x7fd6540b35a0 legacy=0x7fd65406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.390+0000 7fd659000640 1 -- 172.21.15.43:0/1304606748 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6540ce9f0 con 0x7fd65415f0a0 2024-01-16T15:02:04.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.391+0000 7fd659000640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ce4b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009e80 tx=0x7fd64c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e66ac0133ed86e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.391+0000 7fd652a00640 1 -- 172.21.15.43:0/1304606748 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c02caf0 con 0x7fd65415f0a0 2024-01-16T15:02:04.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.391+0000 7fd652a00640 1 -- 172.21.15.43:0/1304606748 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c02cc50 con 0x7fd65415f0a0 2024-01-16T15:02:04.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd652a00640 1 -- 172.21.15.43:0/1304606748 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0343f0 con 0x7fd65415f0a0 2024-01-16T15:02:04.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 -- 172.21.15.43:0/1304606748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 msgr2=0x7fd6540ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ce4b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009e80 tx=0x7fd64c02bf50 comp rx=0 tx=0).stop 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 -- 172.21.15.43:0/1304606748 shutdown_connections 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ce4b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 --2- 172.21.15.43:0/1304606748 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 -- 172.21.15.43:0/1304606748 >> 172.21.15.43:0/1304606748 conn(0x7fd6540cec10 msgr2=0x7fd6540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 -- 172.21.15.43:0/1304606748 shutdown_connections 2024-01-16T15:02:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.392+0000 7fd659a00640 1 -- 172.21.15.43:0/1304606748 wait complete. 2024-01-16T15:02:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.394+0000 7fd659a00640 1 Processor -- start 2024-01-16T15:02:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.394+0000 7fd659a00640 1 -- start start 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6540b35a0 0x7fd6541f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541f3050 con 0x7fd65415f0a0 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65415aee0 con 0x7fd6540b35a0 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.395+0000 7fd659a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65415df00 con 0x7fd654144770 2024-01-16T15:02:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43848/0 (socket says 172.21.15.43:43848) 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 -- 172.21.15.43:0/1249577807 learned_addr learned my addr 172.21.15.43:0/1249577807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 -- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 msgr2=0x7fd65415a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 -- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6540b35a0 msgr2=0x7fd6541f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd653e00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6540b35a0 0x7fd6541f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6540b35a0 0x7fd6541f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 -- 172.21.15.43:0/1249577807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c009b30 con 0x7fd65415f0a0 2024-01-16T15:02:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.396+0000 7fd659000640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036040 tx=0x7fd64c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.397+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c009510 con 0x7fd65415f0a0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.397+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c03e450 con 0x7fd65415f0a0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.397+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c03e7e0 con 0x7fd65415f0a0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.397+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65415b720 con 0x7fd65415f0a0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.397+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65415bce0 con 0x7fd65415f0a0 2024-01-16T15:02:04.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.398+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65415bed0 con 0x7fd65415f0a0 2024-01-16T15:02:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.399+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd64c03e940 con 0x7fd65415f0a0 2024-01-16T15:02:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.400+0000 7fd64be00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 0x7fd62c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.400+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd64c0aca20 con 0x7fd65415f0a0 2024-01-16T15:02:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.400+0000 7fd653e00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 0x7fd62c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.401+0000 7fd653e00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 0x7fd62c06d980 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd644009a10 tx=0x7fd644008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.404+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd64c077780 con 0x7fd65415f0a0 2024-01-16T15:02:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.624+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd65415b5a0 con 0x7fd62c06b4c0 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.634+0000 7fd64be00640 1 -- 172.21.15.43:0/1249577807 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd644010870 con 0x7fd62c06b4c0 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:02:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:04.615 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 msgr2=0x7fd62c06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 0x7fd62c06d980 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd644009a10 tx=0x7fd644008040 comp rx=0 tx=0).stop 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 msgr2=0x7fd6540ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036040 tx=0x7fd64c004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 shutdown_connections 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd62c06b4c0 0x7fd62c06d980 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd65415f0a0 0x7fd6540ad280 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd654144770 0x7fd65415a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 --2- 172.21.15.43:0/1249577807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6540b35a0 0x7fd6541f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 >> 172.21.15.43:0/1249577807 conn(0x7fd6540cec10 msgr2=0x7fd6540b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 shutdown_connections 2024-01-16T15:02:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:04.637+0000 7fd659a00640 1 -- 172.21.15.43:0/1249577807 wait complete. 2024-01-16T15:02:04.634 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:05.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.900+0000 7fd0e1000640 1 Processor -- start 2024-01-16T15:02:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.900+0000 7fd0e1000640 1 -- start start 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.901+0000 7fd0e1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0dc160bd0 0x7fd0dc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.901+0000 7fd0e1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc0b3210 0x7fd0dc167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.901+0000 7fd0e1000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0b23e0 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.901+0000 7fd0e1000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc0cdc70 con 0x7fd0dc160bd0 2024-01-16T15:02:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.901+0000 7fd0e1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0b2870 con 0x7fd0dc0b3210 2024-01-16T15:02:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.902+0000 7fd0d1200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd0dc0b48d0 0x7fd0dc0b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:34036/0 (socket says 172.21.15.43:34036) 2024-01-16T15:02:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.902+0000 7fd0d9e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0dc160bd0 0x7fd0dc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.902+0000 7fd0d1200640 1 -- 172.21.15.43:0/2549856608 learned_addr learned my addr 172.21.15.43:0/2549856608 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0da800640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc0b3210 0x7fd0dc167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9400640 1 -- 172.21.15.43:0/2549856608 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522531155 0 0) 0x7fd0dc0b23e0 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9400640 1 -- 172.21.15.43:0/2549856608 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc003520 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9e00640 1 -- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc0b3210 msgr2=0x7fd0dc167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc0b3210 0x7fd0dc167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9e00640 1 -- 172.21.15.43:0/2549856608 >> v1:172.21.15.69:6789/0 conn(0x7fd0dc0b48d0 legacy=0x7fd0dc0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.904+0000 7fd0d9e00640 1 -- 172.21.15.43:0/2549856608 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0d41b0 con 0x7fd0dc160bd0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.906+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc160bd0 0x7fd0dc0b1ea0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009d90 tx=0x7fd0cc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e19c9b3a5be622 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.907+0000 7fd0d9400640 1 -- 172.21.15.43:0/2549856608 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc02cbc0 con 0x7fd0dc160bd0 2024-01-16T15:02:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.907+0000 7fd0d9400640 1 -- 172.21.15.43:0/2549856608 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0cc02cd20 con 0x7fd0dc160bd0 2024-01-16T15:02:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.907+0000 7fd0d9400640 1 -- 172.21.15.43:0/2549856608 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc034640 con 0x7fd0dc160bd0 2024-01-16T15:02:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.907+0000 7fd0e1000640 1 -- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc160bd0 msgr2=0x7fd0dc0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.907+0000 7fd0e1000640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc160bd0 0x7fd0dc0b1ea0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009d90 tx=0x7fd0cc02be10 comp rx=0 tx=0).stop 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 -- 172.21.15.43:0/2549856608 shutdown_connections 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc0b3210 0x7fd0dc167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 --2- 172.21.15.43:0/2549856608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc160bd0 0x7fd0dc0b1ea0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 -- 172.21.15.43:0/2549856608 >> 172.21.15.43:0/2549856608 conn(0x7fd0dc09c5a0 msgr2=0x7fd0dc0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 -- 172.21.15.43:0/2549856608 shutdown_connections 2024-01-16T15:02:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.908+0000 7fd0e1000640 1 -- 172.21.15.43:0/2549856608 wait complete. 2024-01-16T15:02:05.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.909+0000 7fd0e1000640 1 Processor -- start 2024-01-16T15:02:05.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.910+0000 7fd0e1000640 1 -- start start 2024-01-16T15:02:05.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 0x7fd0dc0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc160bd0 0x7fd0dc0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0b6600 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0cb260 con 0x7fd0dc0b3210 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0e1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc158e00 con 0x7fd0dc160bd0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0d1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0d1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43852/0 (socket says 172.21.15.43:43852) 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0d1200640 1 -- 172.21.15.43:0/3376184839 learned_addr learned my addr 172.21.15.43:0/3376184839 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0da800640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc160bd0 0x7fd0dc0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.911+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 0x7fd0dc0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 -- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc160bd0 msgr2=0x7fd0dc0c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc160bd0 0x7fd0dc0c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 -- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 msgr2=0x7fd0dc0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:05.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 0x7fd0dc0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:05.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 -- 172.21.15.43:0/3376184839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc009a70 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 0x7fd0dc0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d1200640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c460 tx=0x7fd0c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4005ce0 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c4005e40 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0c9250 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4007290 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.912+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc0cb730 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.914+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc0a1210 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.915+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd0c40073f0 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.916+0000 7fd0d2a00640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 0x7fd0b406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:05.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.916+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c40894d0 con 0x7fd0dc0b48d0 2024-01-16T15:02:05.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.916+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 0x7fd0b406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:05.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.917+0000 7fd0d9e00640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 0x7fd0b406d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc036000 tx=0x7fd0cc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:05.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:05.921+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0c4054020 con 0x7fd0dc0b48d0 2024-01-16T15:02:06.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.352+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0dc0b3760 con 0x7fd0b406b470 2024-01-16T15:02:06.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.376+0000 7fd0d2a00640 1 -- 172.21.15.43:0/3376184839 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd0cc02ce90 con 0x7fd0b406b470 2024-01-16T15:02:06.356 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:06.356 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:06.357 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.383+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 msgr2=0x7fd0b406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.383+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 0x7fd0b406d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc036000 tx=0x7fd0cc030000 comp rx=0 tx=0).stop 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.383+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 msgr2=0x7fd0dc1588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.383+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c460 tx=0x7fd0c400c920 comp rx=0 tx=0).stop 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 shutdown_connections 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd0b406b470 0x7fd0b406d930 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0dc160bd0 0x7fd0dc0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0dc0b48d0 0x7fd0dc1588c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 --2- 172.21.15.43:0/3376184839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0dc0b3210 0x7fd0dc0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 >> 172.21.15.43:0/3376184839 conn(0x7fd0dc09c5a0 msgr2=0x7fd0dc0d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 shutdown_connections 2024-01-16T15:02:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:06.384+0000 7fd0e1000640 1 -- 172.21.15.43:0/3376184839 wait complete. 2024-01-16T15:02:06.408 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:07.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:07.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.761+0000 7f92aac00640 1 Processor -- start 2024-01-16T15:02:07.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.761+0000 7f92aac00640 1 -- start start 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.763+0000 7f92aac00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92a415b350 0x7f92a40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.763+0000 7f92aac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a40b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92aac00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f92a40adb60 0x7f92a40adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92aac00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a415b730 con 0x7f92a415b350 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92aac00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a40ac740 con 0x7f92a40adb60 2024-01-16T15:02:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92aac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40ac460 con 0x7f92a409cb30 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92a3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92a415b350 0x7f92a40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92a3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92a415b350 0x7f92a40abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43860/0 (socket says 172.21.15.43:43860) 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.764+0000 7f92a3e00640 1 -- 172.21.15.43:0/3314547643 learned_addr learned my addr 172.21.15.43:0/3314547643 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a8c00640 1 --2- 172.21.15.43:0/3314547643 >> v2:172.21.15.69:3300/0 conn(0x7f92a40adb60 0x7f92a40adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a3e00640 1 -- 172.21.15.43:0/3314547643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 msgr2=0x7f92a40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:07.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a3e00640 1 --2- 172.21.15.43:0/3314547643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a40b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a3e00640 1 -- 172.21.15.43:0/3314547643 >> v2:172.21.15.69:3300/0 conn(0x7f92a40adb60 msgr2=0x7f92a40adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:07.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a3e00640 1 --2- 172.21.15.43:0/3314547643 >> v2:172.21.15.69:3300/0 conn(0x7f92a40adb60 0x7f92a40adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.765+0000 7f92a3e00640 1 -- 172.21.15.43:0/3314547643 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40b2a50 con 0x7f92a415b350 2024-01-16T15:02:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.771+0000 7f92a3e00640 1 --2- 172.21.15.43:0/3314547643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a415b350 0x7f92a40abe80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f92a4072730 tx=0x7f929402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c4407a3fbb6a1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.772+0000 7f92a2a00640 1 -- 172.21.15.43:0/3314547643 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294031870 con 0x7f92a415b350 2024-01-16T15:02:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.772+0000 7f92a2a00640 1 -- 172.21.15.43:0/3314547643 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92940319d0 con 0x7f92a415b350 2024-01-16T15:02:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.772+0000 7f92a2a00640 1 -- 172.21.15.43:0/3314547643 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294031d60 con 0x7f92a415b350 2024-01-16T15:02:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.773+0000 7f92aac00640 1 -- 172.21.15.43:0/3314547643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a415b350 msgr2=0x7f92a40abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.773+0000 7f92aac00640 1 --2- 172.21.15.43:0/3314547643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a415b350 0x7f92a40abe80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f92a4072730 tx=0x7f929402b910 comp rx=0 tx=0).stop 2024-01-16T15:02:07.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 -- 172.21.15.43:0/3314547643 shutdown_connections 2024-01-16T15:02:07.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 --2- 172.21.15.43:0/3314547643 >> v2:172.21.15.69:3300/0 conn(0x7f92a40adb60 0x7f92a40adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 --2- 172.21.15.43:0/3314547643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a40b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 --2- 172.21.15.43:0/3314547643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a415b350 0x7f92a40abe80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 -- 172.21.15.43:0/3314547643 >> 172.21.15.43:0/3314547643 conn(0x7f92a40b0ba0 msgr2=0x7f92a40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:07.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 -- 172.21.15.43:0/3314547643 shutdown_connections 2024-01-16T15:02:07.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.774+0000 7f92aac00640 1 -- 172.21.15.43:0/3314547643 wait complete. 2024-01-16T15:02:07.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.778+0000 7f92aac00640 1 Processor -- start 2024-01-16T15:02:07.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.779+0000 7f92aac00640 1 -- start start 2024-01-16T15:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.781+0000 7f92aac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.781+0000 7f92aac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.782+0000 7f92aac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a415b350 0x7f92a4152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.782+0000 7f92a3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.782+0000 7f92a3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43868/0 (socket says 172.21.15.43:43868) 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 -- 172.21.15.43:0/828334138 learned_addr learned my addr 172.21.15.43:0/828334138 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a413b080 con 0x7f92a40adb60 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4139aa0 con 0x7f92a415b350 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4139d80 con 0x7f92a409cb30 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3e00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 -- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 msgr2=0x7f92a413ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a413ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 -- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a415b350 msgr2=0x7f92a4152e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a415b350 0x7f92a4152e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.783+0000 7f92a3400640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940094d0 con 0x7f92a40adb60 2024-01-16T15:02:07.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f92a3400640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f928c00b4a0 tx=0x7f928c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c010610 con 0x7f92a40adb60 2024-01-16T15:02:07.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c010770 con 0x7f92a40adb60 2024-01-16T15:02:07.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c010b20 con 0x7f92a40adb60 2024-01-16T15:02:07.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a413a060 con 0x7f92a40adb60 2024-01-16T15:02:07.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.784+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a413a620 con 0x7f92a40adb60 2024-01-16T15:02:07.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.786+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a413a840 con 0x7f92a40adb60 2024-01-16T15:02:07.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.789+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f928c010c80 con 0x7f92a40adb60 2024-01-16T15:02:07.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.789+0000 7f929be00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 0x7f927c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:07.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.789+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f928c089be0 con 0x7f92a40adb60 2024-01-16T15:02:07.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.790+0000 7f92a3e00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 0x7f927c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:07.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.791+0000 7f92a3e00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 0x7f927c06d930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9294036000 tx=0x7f929403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:07.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:07.793+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f928c054730 con 0x7f92a40adb60 2024-01-16T15:02:08.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.072+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f92a40b3a70 con 0x7f927c06b470 2024-01-16T15:02:08.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.960+0000 7f929be00640 1 -- 172.21.15.43:0/828334138 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f9294031bc0 con 0x7f927c06b470 2024-01-16T15:02:08.940 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:08.940 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:02:08.941 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:02:08.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:08.941 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 msgr2=0x7f927c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 0x7f927c06d930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9294036000 tx=0x7f929403c040 comp rx=0 tx=0).stop 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 msgr2=0x7f92a4139020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f928c00b4a0 tx=0x7f928c00b960 comp rx=0 tx=0).stop 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 shutdown_connections 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f927c06b470 0x7f927c06d930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a415b350 0x7f92a4152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92a40adb60 0x7f92a4139020 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.963+0000 7f92aac00640 1 --2- 172.21.15.43:0/828334138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92a409cb30 0x7f92a413ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.964+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 >> 172.21.15.43:0/828334138 conn(0x7f92a40b0ba0 msgr2=0x7f92a406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.964+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 shutdown_connections 2024-01-16T15:02:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:08.964+0000 7f92aac00640 1 -- 172.21.15.43:0/828334138 wait complete. 2024-01-16T15:02:08.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:09.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.213+0000 7fc6fe600640 1 Processor -- start 2024-01-16T15:02:09.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.213+0000 7fc6fe600640 1 -- start start 2024-01-16T15:02:09.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.214+0000 7fc6fe600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6f80ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fe600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6f80d1d80 0x7fc6f80cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fe600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f80d04b0 con 0x7fc6f80cf710 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fe600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f80ce4e0 con 0x7fc6f80d1d80 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fe600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80cdb80 con 0x7fc6f815d5d0 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6f7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc6f80cf710 0x7fc6f80d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60284/0 (socket says 172.21.15.43:60284) 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6f7e00640 1 -- 172.21.15.43:0/1515034523 learned_addr learned my addr 172.21.15.43:0/1515034523 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fd200640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6f80ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6fdc00640 1 --2- 172.21.15.43:0/1515034523 >> v2:172.21.15.69:3300/0 conn(0x7fc6f80d1d80 0x7fc6f80cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.215+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574022738 0 0) 0x7fc6f80d04b0 con 0x7fc6f80cf710 2024-01-16T15:02:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d8003520 con 0x7fc6f80cf710 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125777298 0 0) 0x7fc6d8003520 con 0x7fc6f80cf710 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 msgr2=0x7fc6f80ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6f80ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 >> v2:172.21.15.69:3300/0 conn(0x7fc6f80d1d80 msgr2=0x7fc6f80cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 --2- 172.21.15.43:0/1515034523 >> v2:172.21.15.69:3300/0 conn(0x7fc6f80d1d80 0x7fc6f80cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f80d01d0 con 0x7fc6f80cf710 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc6e0001670 con 0x7fc6f80cf710 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 >> v1:172.21.15.43:6789/0 conn(0x7fc6f80cf710 legacy=0x7fc6f80d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6d80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6d8003f60 0x7fc6d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80ce4e0 con 0x7fc6d8004d00 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80cdb80 con 0x7fc6d8003f60 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d800ba30 con 0x7fc6f815d5d0 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.216+0000 7fc6fdc00640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fd200640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6d80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6f7e00640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6d8003f60 0x7fc6d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fdc00640 1 -- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 msgr2=0x7fc6d80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fdc00640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f815d5d0 0x7fc6d80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fdc00640 1 -- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6d8003f60 msgr2=0x7fc6d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fdc00640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6d8003f60 0x7fc6d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.217+0000 7fc6fdc00640 1 -- 172.21.15.43:0/1515034523 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8003520 con 0x7fc6d8004d00 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.218+0000 7fc6fdc00640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6d8005180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009fc0 tx=0x7fc6d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c47eb1bfa1aaf59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.218+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f002cc60 con 0x7fc6d8004d00 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.218+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f000f040 con 0x7fc6d8004d00 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.218+0000 7fc6f7400640 1 -- 172.21.15.43:0/1515034523 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0034440 con 0x7fc6d8004d00 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 -- 172.21.15.43:0/1515034523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 msgr2=0x7fc6d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 --2- 172.21.15.43:0/1515034523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6d8005180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009fc0 tx=0x7fc6d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fdc00640 1 -- 172.21.15.43:0/1515034523 reap_dead start 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 -- 172.21.15.43:0/1515034523 shutdown_connections 2024-01-16T15:02:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 -- 172.21.15.43:0/1515034523 >> 172.21.15.43:0/1515034523 conn(0x7fc6f809c3d0 msgr2=0x7fc6f80d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 -- 172.21.15.43:0/1515034523 shutdown_connections 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.219+0000 7fc6fe600640 1 -- 172.21.15.43:0/1515034523 wait complete. 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.220+0000 7fc6fe600640 1 Processor -- start 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.221+0000 7fc6fe600640 1 -- start start 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.221+0000 7fc6fe600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.221+0000 7fc6fe600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f8147240 0x7fc6f8159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fe600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f8159ec0 0x7fc6f81efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fe600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f815f3d0 con 0x7fc6d8004d00 2024-01-16T15:02:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fe600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f81f3ce0 con 0x7fc6f8159ec0 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fe600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f815a370 con 0x7fc6f8147240 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43878/0 (socket says 172.21.15.43:43878) 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 -- 172.21.15.43:0/664729885 learned_addr learned my addr 172.21.15.43:0/664729885 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 -- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f8147240 msgr2=0x7fc6f8159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f8147240 0x7fc6f8159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 -- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f8159ec0 msgr2=0x7fc6f81efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f8159ec0 0x7fc6f81efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.222+0000 7fc6fd200640 1 -- 172.21.15.43:0/664729885 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f814a1c0 con 0x7fc6d8004d00 2024-01-16T15:02:09.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6fd200640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8011600 tx=0x7fc6e8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e800f920 con 0x7fc6d8004d00 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e800fa80 con 0x7fc6d8004d00 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f815eaa0 con 0x7fc6d8004d00 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e800fdf0 con 0x7fc6d8004d00 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.223+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f815f060 con 0x7fc6d8004d00 2024-01-16T15:02:09.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.224+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f804e840 con 0x7fc6d8004d00 2024-01-16T15:02:09.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.226+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fc6e8015460 con 0x7fc6d8004d00 2024-01-16T15:02:09.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.227+0000 7fc6f4c00640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 0x7fc6d006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.227+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6e808a410 con 0x7fc6d8004d00 2024-01-16T15:02:09.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.227+0000 7fc6f7e00640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 0x7fc6d006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.227+0000 7fc6f7e00640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 0x7fc6d006d960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00018a0 tx=0x7fc6e0001810 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.231+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6e8054f60 con 0x7fc6d8004d00 2024-01-16T15:02:09.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.454+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6f80d10a0 con 0x7fc6d006b4a0 2024-01-16T15:02:09.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.456+0000 7fc6f4c00640 1 -- 172.21.15.43:0/664729885 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc6e0005040 con 0x7fc6d006b4a0 2024-01-16T15:02:09.436 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:02:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:02:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:09.437 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:02:09.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 msgr2=0x7fc6d006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 0x7fc6d006d960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00018a0 tx=0x7fc6e0001810 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 msgr2=0x7fc6f81f37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8011600 tx=0x7fc6e8011ac0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 shutdown_connections 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc6d006b4a0 0x7fc6d006d960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f8159ec0 0x7fc6f81efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6f8147240 0x7fc6f8159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 --2- 172.21.15.43:0/664729885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6d8004d00 0x7fc6f81f37a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 >> 172.21.15.43:0/664729885 conn(0x7fc6f809c3d0 msgr2=0x7fc6f806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 shutdown_connections 2024-01-16T15:02:09.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.459+0000 7fc6fe600640 1 -- 172.21.15.43:0/664729885 wait complete. 2024-01-16T15:02:09.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:02:09.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.706+0000 7f5963200640 1 Processor -- start 2024-01-16T15:02:09.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.706+0000 7f5963200640 1 -- start start 2024-01-16T15:02:09.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.708+0000 7f5963200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f59640b3080 0x7f5964161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.708+0000 7f5963200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f596406da40 0x7f596406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.708+0000 7f5963200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596406c9a0 con 0x7f596415efa0 2024-01-16T15:02:09.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.708+0000 7f5963200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596406cc80 con 0x7f596406da40 2024-01-16T15:02:09.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.708+0000 7f5963200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596406cf60 con 0x7f59640b3080 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f596406da40 0x7f596406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f596406da40 0x7f596406c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43892/0 (socket says 172.21.15.43:43892) 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 -- 172.21.15.43:0/1612748946 learned_addr learned my addr 172.21.15.43:0/1612748946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f595be00640 1 --2- 172.21.15.43:0/1612748946 >> v2:172.21.15.69:3300/0 conn(0x7f59640b3080 0x7f5964161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 -- 172.21.15.43:0/1612748946 >> v2:172.21.15.69:3300/0 conn(0x7f59640b3080 msgr2=0x7f5964161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 --2- 172.21.15.43:0/1612748946 >> v2:172.21.15.69:3300/0 conn(0x7f59640b3080 0x7f5964161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 -- 172.21.15.43:0/1612748946 >> v1:172.21.15.69:6789/0 conn(0x7f596415efa0 legacy=0x7f59640b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.709+0000 7f5962800640 1 -- 172.21.15.43:0/1612748946 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596406d240 con 0x7f596406da40 2024-01-16T15:02:09.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.710+0000 7f5962800640 1 --2- 172.21.15.43:0/1612748946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f596406c310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f595c009e50 tx=0x7f595c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49daf919e6c5a1df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5961400640 1 -- 172.21.15.43:0/1612748946 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c02c9f0 con 0x7f596406da40 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5961400640 1 -- 172.21.15.43:0/1612748946 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c02cb50 con 0x7f596406da40 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5961400640 1 -- 172.21.15.43:0/1612748946 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c0343f0 con 0x7f596406da40 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 -- 172.21.15.43:0/1612748946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 msgr2=0x7f596406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 --2- 172.21.15.43:0/1612748946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f596406c310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f595c009e50 tx=0x7f595c02be50 comp rx=0 tx=0).stop 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 -- 172.21.15.43:0/1612748946 shutdown_connections 2024-01-16T15:02:09.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 --2- 172.21.15.43:0/1612748946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f596406c310 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 --2- 172.21.15.43:0/1612748946 >> v2:172.21.15.69:3300/0 conn(0x7f59640b3080 0x7f5964161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.711+0000 7f5963200640 1 -- 172.21.15.43:0/1612748946 >> 172.21.15.43:0/1612748946 conn(0x7f59640cec10 msgr2=0x7f59640cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.712+0000 7f5963200640 1 -- 172.21.15.43:0/1612748946 shutdown_connections 2024-01-16T15:02:09.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.712+0000 7f5963200640 1 -- 172.21.15.43:0/1612748946 wait complete. 2024-01-16T15:02:09.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.713+0000 7f5963200640 1 Processor -- start 2024-01-16T15:02:09.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.714+0000 7f5963200640 1 -- start start 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.714+0000 7f5963200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.714+0000 7f5963200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59640b3080 0x7f596414d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5963200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415efa0 0x7f5964158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5963200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596414cc60 con 0x7f596406da40 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5963200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596414bfe0 con 0x7f59640b3080 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5963200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596414c2c0 con 0x7f596415efa0 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43906/0 (socket says 172.21.15.43:43906) 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 -- 172.21.15.43:0/395144802 learned_addr learned my addr 172.21.15.43:0/395144802 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f595be00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59640b3080 0x7f596414d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5962800640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415efa0 0x7f5964158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 -- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415efa0 msgr2=0x7f5964158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415efa0 0x7f5964158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 -- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59640b3080 msgr2=0x7f596414d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59640b3080 0x7f596414d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.715+0000 7f5961e00640 1 -- 172.21.15.43:0/395144802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c009b30 con 0x7f596406da40 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f5961e00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f595400b3f0 tx=0x7f595400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954011830 con 0x7f596406da40 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954011990 con 0x7f596406da40 2024-01-16T15:02:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954011d40 con 0x7f596406da40 2024-01-16T15:02:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596414c5a0 con 0x7f596406da40 2024-01-16T15:02:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.716+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59640ae210 con 0x7f596406da40 2024-01-16T15:02:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.717+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596404e840 con 0x7f596406da40 2024-01-16T15:02:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.718+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f595401a400 con 0x7f596406da40 2024-01-16T15:02:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.719+0000 7f595a000640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 0x7f593c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.719+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5954089da0 con 0x7f596406da40 2024-01-16T15:02:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.719+0000 7f595be00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 0x7f593c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.720+0000 7f595be00640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 0x7f593c06d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5964072410 tx=0x7f594c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.723+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5954054980 con 0x7f596406da40 2024-01-16T15:02:09.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.944+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f596406c310 con 0x7f593c06b470 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.954+0000 7f595a000640 1 -- 172.21.15.43:0/395144802 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f596406c310 con 0x7f593c06b470 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1clone1" 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:09.935 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:02:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 msgr2=0x7f593c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 0x7f593c06d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5964072410 tx=0x7f594c008040 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 msgr2=0x7f59640adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f595400b3f0 tx=0x7f595400b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 shutdown_connections 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f593c06b470 0x7f593c06d930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59640b3080 0x7f596414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f596415efa0 0x7f5964158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 --2- 172.21.15.43:0/395144802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f596406da40 0x7f59640adcd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.957+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 >> 172.21.15.43:0/395144802 conn(0x7f59640cec10 msgr2=0x7f596415f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.958+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 shutdown_connections 2024-01-16T15:02:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:09.958+0000 7f5963200640 1 -- 172.21.15.43:0/395144802 wait complete. 2024-01-16T15:02:09.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:02:10.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.205+0000 7fd7cb800640 1 Processor -- start 2024-01-16T15:02:10.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.205+0000 7fd7cb800640 1 -- start start 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.206+0000 7fd7cb800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7c406cad0 0x7fd7c406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.206+0000 7fd7cb800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7c4072280 0x7fd7c40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7cb800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c4071780 con 0x7fd7c4160b10 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7cb800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c40b2320 con 0x7fd7c4072280 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7cb800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c406df30 con 0x7fd7c406cad0 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7c9800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7c4072280 0x7fd7c40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7c9800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7c4072280 0x7fd7c40b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43918/0 (socket says 172.21.15.43:43918) 2024-01-16T15:02:10.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7c9800640 1 -- 172.21.15.43:0/1995424666 learned_addr learned my addr 172.21.15.43:0/1995424666 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7bbe00640 1 --2- 172.21.15.43:0/1995424666 >> v2:172.21.15.69:3300/0 conn(0x7fd7c406cad0 0x7fd7c406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.207+0000 7fd7c9800640 1 -- 172.21.15.43:0/1995424666 >> v2:172.21.15.69:3300/0 conn(0x7fd7c406cad0 msgr2=0x7fd7c406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.208+0000 7fd7c9800640 1 --2- 172.21.15.43:0/1995424666 >> v2:172.21.15.69:3300/0 conn(0x7fd7c406cad0 0x7fd7c406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.208+0000 7fd7c9800640 1 -- 172.21.15.43:0/1995424666 >> v1:172.21.15.69:6789/0 conn(0x7fd7c4160b10 legacy=0x7fd7c406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.208+0000 7fd7c9800640 1 -- 172.21.15.43:0/1995424666 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40711c0 con 0x7fd7c4072280 2024-01-16T15:02:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.209+0000 7fd7c9800640 1 --2- 172.21.15.43:0/1995424666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4072280 0x7fd7c40b2740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40d0100 tx=0x7fd7bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cbfa3e9a05ae1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.209+0000 7fd7c3e00640 1 -- 172.21.15.43:0/1995424666 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc02c9f0 con 0x7fd7c4072280 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.209+0000 7fd7c3e00640 1 -- 172.21.15.43:0/1995424666 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc02cb50 con 0x7fd7c4072280 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.209+0000 7fd7c3e00640 1 -- 172.21.15.43:0/1995424666 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0343f0 con 0x7fd7c4072280 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 -- 172.21.15.43:0/1995424666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4072280 msgr2=0x7fd7c40b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1995424666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4072280 0x7fd7c40b2740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40d0100 tx=0x7fd7bc02be50 comp rx=0 tx=0).stop 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 -- 172.21.15.43:0/1995424666 shutdown_connections 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1995424666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4072280 0x7fd7c40b2740 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1995424666 >> v2:172.21.15.69:3300/0 conn(0x7fd7c406cad0 0x7fd7c406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 -- 172.21.15.43:0/1995424666 >> 172.21.15.43:0/1995424666 conn(0x7fd7c40b0ba0 msgr2=0x7fd7c40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 -- 172.21.15.43:0/1995424666 shutdown_connections 2024-01-16T15:02:10.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.210+0000 7fd7cb800640 1 -- 172.21.15.43:0/1995424666 wait complete. 2024-01-16T15:02:10.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.212+0000 7fd7cb800640 1 Processor -- start 2024-01-16T15:02:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.212+0000 7fd7cb800640 1 -- start start 2024-01-16T15:02:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 0x7fd7c415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7c4072280 0x7fd7c415a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c409c5b0 con 0x7fd7c4160b10 2024-01-16T15:02:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c409c890 con 0x7fd7c4072280 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7cb800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40ce2a0 con 0x7fd7c406cad0 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7c9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7c9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43930/0 (socket says 172.21.15.43:43930) 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7c9800640 1 -- 172.21.15.43:0/1950947797 learned_addr learned my addr 172.21.15.43:0/1950947797 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7bbe00640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7c4072280 0x7fd7c415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.213+0000 7fd7c8e00640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 0x7fd7c415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 -- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 msgr2=0x7fd7c415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 0x7fd7c415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 -- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7c4072280 msgr2=0x7fd7c415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7c4072280 0x7fd7c415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 -- 172.21.15.43:0/1950947797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc009b30 con 0x7fd7c4160b10 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c8e00640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 0x7fd7c415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c9800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc036040 tx=0x7fd7bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc009510 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc03e450 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc03e7e0 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c40ce580 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.214+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c414e5c0 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.216+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c409d200 con 0x7fd7c4160b10 2024-01-16T15:02:10.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.217+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd7bc03e940 con 0x7fd7c4160b10 2024-01-16T15:02:10.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.217+0000 7fd7c1600640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 0x7fd79c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.218+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7bc0acbb0 con 0x7fd7c4160b10 2024-01-16T15:02:10.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.218+0000 7fd7c8e00640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 0x7fd79c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.218+0000 7fd7c8e00640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 0x7fd79c06d980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400a830 tx=0x7fd7b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.222+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7bc0b3080 con 0x7fd7c4160b10 2024-01-16T15:02:10.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.449+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7c4071d50 con 0x7fd79c06b4c0 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.478+0000 7fd7c1600640 1 -- 172.21.15.43:0/1950947797 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd7b400adb0 con 0x7fd79c06b4c0 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:10.459 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.481+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 msgr2=0x7fd79c06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.481+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 0x7fd79c06d980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400a830 tx=0x7fd7b4008040 comp rx=0 tx=0).stop 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.481+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 msgr2=0x7fd7c415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.481+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc036040 tx=0x7fd7bc004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 shutdown_connections 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd79c06b4c0 0x7fd79c06d980 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7c4072280 0x7fd7c415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7c4160b10 0x7fd7c415ad50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 --2- 172.21.15.43:0/1950947797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7c406cad0 0x7fd7c415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 >> 172.21.15.43:0/1950947797 conn(0x7fd7c40b0ba0 msgr2=0x7fd7c40b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 shutdown_connections 2024-01-16T15:02:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.482+0000 7fd7cb800640 1 -- 172.21.15.43:0/1950947797 wait complete. 2024-01-16T15:02:10.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:02:10.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.717+0000 7f7bd5a00640 1 Processor -- start 2024-01-16T15:02:10.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.718+0000 7f7bd5a00640 1 -- start start 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.719+0000 7f7bd5a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7bd0165e20 0x7f7bd0070830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.719+0000 7f7bd5a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7bd0070d70 0x7f7bd006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.719+0000 7f7bd5a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd0070070 con 0x7f7bd015f100 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.719+0000 7f7bd5a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd01624e0 con 0x7f7bd0165e20 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.719+0000 7f7bd5a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd01627c0 con 0x7f7bd0070d70 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7bd0165e20 0x7f7bd0070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7bd0165e20 0x7f7bd0070830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43946/0 (socket says 172.21.15.43:43946) 2024-01-16T15:02:10.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 -- 172.21.15.43:0/3227359356 learned_addr learned my addr 172.21.15.43:0/3227359356 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:10.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 -- 172.21.15.43:0/3227359356 >> v2:172.21.15.69:3300/0 conn(0x7f7bd0070d70 msgr2=0x7f7bd006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 --2- 172.21.15.43:0/3227359356 >> v2:172.21.15.69:3300/0 conn(0x7f7bd0070d70 0x7f7bd006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 -- 172.21.15.43:0/3227359356 >> v1:172.21.15.69:6789/0 conn(0x7f7bd015f100 legacy=0x7f7bd0161c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:10.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.720+0000 7f7bc7200640 1 -- 172.21.15.43:0/3227359356 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd0162aa0 con 0x7f7bd0165e20 2024-01-16T15:02:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.721+0000 7f7bc7200640 1 --2- 172.21.15.43:0/3227359356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd0070830 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7bd006e4e0 tx=0x7f7bb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55e335fc188130b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.721+0000 7f7bcf400640 1 -- 172.21.15.43:0/3227359356 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8004230 con 0x7f7bd0165e20 2024-01-16T15:02:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bcf400640 1 -- 172.21.15.43:0/3227359356 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb8004390 con 0x7f7bd0165e20 2024-01-16T15:02:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bcf400640 1 -- 172.21.15.43:0/3227359356 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8003810 con 0x7f7bd0165e20 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3227359356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 msgr2=0x7f7bd0070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3227359356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd0070830 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7bd006e4e0 tx=0x7f7bb802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3227359356 shutdown_connections 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3227359356 >> v2:172.21.15.69:3300/0 conn(0x7f7bd0070d70 0x7f7bd006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3227359356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd0070830 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3227359356 >> 172.21.15.43:0/3227359356 conn(0x7f7bd00b0aa0 msgr2=0x7f7bd00b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3227359356 shutdown_connections 2024-01-16T15:02:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.722+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3227359356 wait complete. 2024-01-16T15:02:10.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.724+0000 7f7bd5a00640 1 Processor -- start 2024-01-16T15:02:10.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.724+0000 7f7bd5a00640 1 -- start start 2024-01-16T15:02:10.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd015f100 0x7f7bd01f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bd01faa40 0x7f7bd015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd01faef0 con 0x7f7bd0165e20 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd015ac90 con 0x7f7bd015f100 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.725+0000 7f7bd5a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd015b810 con 0x7f7bd01faa40 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43948/0 (socket says 172.21.15.43:43948) 2024-01-16T15:02:10.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 -- 172.21.15.43:0/3640962137 learned_addr learned my addr 172.21.15.43:0/3640962137 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:10.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 -- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bd01faa40 msgr2=0x7f7bd015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bd01faa40 0x7f7bd015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bcfe00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd015f100 0x7f7bd01f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 -- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd015f100 msgr2=0x7f7bd01f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd015f100 0x7f7bd01f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.726+0000 7f7bc7200640 1 -- 172.21.15.43:0/3640962137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb8009460 con 0x7f7bd0165e20 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bc7200640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7bb802b930 tx=0x7f7bb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8003e50 con 0x7f7bd0165e20 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb8004500 con 0x7f7bd0165e20 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd015aee0 con 0x7f7bd0165e20 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8006260 con 0x7f7bd0165e20 2024-01-16T15:02:10.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.727+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd015b4a0 con 0x7f7bd0165e20 2024-01-16T15:02:10.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.728+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd004e840 con 0x7f7bd0165e20 2024-01-16T15:02:10.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.729+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7bb8041070 con 0x7f7bd0165e20 2024-01-16T15:02:10.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.730+0000 7f7bccc00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 0x7f7ba806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:10.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.730+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bb80ac540 con 0x7f7bd0165e20 2024-01-16T15:02:10.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.730+0000 7f7bcfe00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 0x7f7ba806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:10.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.731+0000 7f7bcfe00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 0x7f7ba806d960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0003d90 tx=0x7f7bc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:10.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.734+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bb8077090 con 0x7f7bd0165e20 2024-01-16T15:02:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:10.967+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7bd00cc070 con 0x7f7ba806b4a0 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.017+0000 7f7bccc00640 1 -- 172.21.15.43:0/3640962137 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7bd00cc070 con 0x7f7ba806b4a0 2024-01-16T15:02:10.998 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.020+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 msgr2=0x7f7ba806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.020+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 0x7f7ba806d960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0003d90 tx=0x7f7bc0009040 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.020+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 msgr2=0x7f7bd01fa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.020+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7bb802b930 tx=0x7f7bb80046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 shutdown_connections 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7ba806b4a0 0x7f7ba806d960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7bd01faa40 0x7f7bd015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7bd0165e20 0x7f7bd01fa500 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 --2- 172.21.15.43:0/3640962137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd015f100 0x7f7bd01f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 >> 172.21.15.43:0/3640962137 conn(0x7f7bd00b0aa0 msgr2=0x7f7bd00b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 shutdown_connections 2024-01-16T15:02:11.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.021+0000 7f7bd5a00640 1 -- 172.21.15.43:0/3640962137 wait complete. 2024-01-16T15:02:11.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:02:11.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.250+0000 7f4246000640 1 Processor -- start 2024-01-16T15:02:11.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.250+0000 7f4246000640 1 -- start start 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f4246000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f424015b350 0x7f42400abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f4246000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42400adb60 0x7f42400adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f4246000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424015b730 con 0x7f424009cb30 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f4246000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400ac740 con 0x7f424015b350 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f4246000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400ac460 con 0x7f42400adb60 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f423f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f424015b350 0x7f42400abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f423f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f424015b350 0x7f42400abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43952/0 (socket says 172.21.15.43:43952) 2024-01-16T15:02:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.252+0000 7f423f400640 1 -- 172.21.15.43:0/2992391246 learned_addr learned my addr 172.21.15.43:0/2992391246 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.253+0000 7f423fe00640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42400adb60 0x7f42400adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.253+0000 7f423f400640 1 -- 172.21.15.43:0/2992391246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42400adb60 msgr2=0x7f42400adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.253+0000 7f423f400640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42400adb60 0x7f42400adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.253+0000 7f423f400640 1 -- 172.21.15.43:0/2992391246 >> v1:172.21.15.69:6789/0 conn(0x7f424009cb30 legacy=0x7f42400b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.253+0000 7f423f400640 1 -- 172.21.15.43:0/2992391246 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400b2a50 con 0x7f424015b350 2024-01-16T15:02:11.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.254+0000 7f423f400640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f42400abe80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4230009870 tx=0x7f423002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9829fab977a4943 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f423ea00640 1 -- 172.21.15.43:0/2992391246 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230030840 con 0x7f424015b350 2024-01-16T15:02:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f423ea00640 1 -- 172.21.15.43:0/2992391246 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300309a0 con 0x7f424015b350 2024-01-16T15:02:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f423ea00640 1 -- 172.21.15.43:0/2992391246 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230030d30 con 0x7f424015b350 2024-01-16T15:02:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 -- 172.21.15.43:0/2992391246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 msgr2=0x7f42400abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f42400abe80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4230009870 tx=0x7f423002b980 comp rx=0 tx=0).stop 2024-01-16T15:02:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 -- 172.21.15.43:0/2992391246 shutdown_connections 2024-01-16T15:02:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42400adb60 0x7f42400adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 --2- 172.21.15.43:0/2992391246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f42400abe80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 -- 172.21.15.43:0/2992391246 >> 172.21.15.43:0/2992391246 conn(0x7f42400b0ba0 msgr2=0x7f42400b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 -- 172.21.15.43:0/2992391246 shutdown_connections 2024-01-16T15:02:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.255+0000 7f4246000640 1 -- 172.21.15.43:0/2992391246 wait complete. 2024-01-16T15:02:11.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.257+0000 7f4246000640 1 Processor -- start 2024-01-16T15:02:11.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.257+0000 7f4246000640 1 -- start start 2024-01-16T15:02:11.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f424009cb30 0x7f424013dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42400adb60 0x7f42400707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240071ca0 con 0x7f424015b350 2024-01-16T15:02:11.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240071020 con 0x7f42400adb60 2024-01-16T15:02:11.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.258+0000 7f4246000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4240071300 con 0x7f424009cb30 2024-01-16T15:02:11.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.259+0000 7f423fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.259+0000 7f423fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43956/0 (socket says 172.21.15.43:43956) 2024-01-16T15:02:11.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.259+0000 7f423fe00640 1 -- 172.21.15.43:0/645557138 learned_addr learned my addr 172.21.15.43:0/645557138 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.259+0000 7f4237e00640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42400adb60 0x7f42400707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.259+0000 7f423f400640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f424009cb30 0x7f424013dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 -- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f424009cb30 msgr2=0x7f424013dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f424009cb30 0x7f424013dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 -- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42400adb60 msgr2=0x7f42400707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42400adb60 0x7f42400707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 -- 172.21.15.43:0/645557138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300094d0 con 0x7f424015b350 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f423fe00640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f423800cd70 tx=0x7f423800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238003b90 con 0x7f424015b350 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238003cf0 con 0x7f424015b350 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238004240 con 0x7f424015b350 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.260+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42400715e0 con 0x7f424015b350 2024-01-16T15:02:11.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.261+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4240070d30 con 0x7f424015b350 2024-01-16T15:02:11.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.262+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424009d200 con 0x7f424015b350 2024-01-16T15:02:11.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.263+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f42380043a0 con 0x7f424015b350 2024-01-16T15:02:11.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.264+0000 7f4237400640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 0x7f421806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.264+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4238012070 con 0x7f424015b350 2024-01-16T15:02:11.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.264+0000 7f423f400640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 0x7f421806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.265+0000 7f423f400640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 0x7f421806d930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4230036000 tx=0x7f423003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.268+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4238055470 con 0x7f424015b350 2024-01-16T15:02:11.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.490+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f42400ae4f0 con 0x7f421806b470 2024-01-16T15:02:11.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.541+0000 7f4237400640 1 -- 172.21.15.43:0/645557138 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4230030ce0 con 0x7f421806b470 2024-01-16T15:02:11.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.544+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 msgr2=0x7f421806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 0x7f421806d930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4230036000 tx=0x7f423003b040 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 msgr2=0x7f4240157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f423800cd70 tx=0x7f423800a660 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 shutdown_connections 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421806b470 0x7f421806d930 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42400adb60 0x7f42400707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f424015b350 0x7f4240157d30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 --2- 172.21.15.43:0/645557138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f424009cb30 0x7f424013dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 >> 172.21.15.43:0/645557138 conn(0x7f42400b0ba0 msgr2=0x7f424014fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 shutdown_connections 2024-01-16T15:02:11.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.545+0000 7f4246000640 1 -- 172.21.15.43:0/645557138 wait complete. 2024-01-16T15:02:11.529 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.549+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.529 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.549+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.529 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.549+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.529 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.549+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.529 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.549+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:02:11.551 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.569+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.551 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.569+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.551 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.569+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.551 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.569+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.551 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:11.569+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:11.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.761+0000 7fd996200640 1 Processor -- start 2024-01-16T15:02:11.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.761+0000 7fd996200640 1 -- start start 2024-01-16T15:02:11.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd996200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd99006d470 0x7fd9900b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd996200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9900b2d70 0x7fd990158310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd996200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd990157b30 con 0x7fd99015aae0 2024-01-16T15:02:11.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd996200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99015aec0 con 0x7fd99006d470 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd996200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd990154440 con 0x7fd9900b2d70 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd987e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd99006d470 0x7fd9900b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd98f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd99015aae0 0x7fd99006cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60288/0 (socket says 172.21.15.43:60288) 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd98f400640 1 -- 172.21.15.43:0/825165086 learned_addr learned my addr 172.21.15.43:0/825165086 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.763+0000 7fd98fe00640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9900b2d70 0x7fd990158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.764+0000 7fd987e00640 1 -- 172.21.15.43:0/825165086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9900b2d70 msgr2=0x7fd990158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.764+0000 7fd987e00640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9900b2d70 0x7fd990158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.764+0000 7fd987e00640 1 -- 172.21.15.43:0/825165086 >> v1:172.21.15.43:6789/0 conn(0x7fd99015aae0 legacy=0x7fd99006cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.764+0000 7fd987e00640 1 -- 172.21.15.43:0/825165086 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9900b34f0 con 0x7fd99006d470 2024-01-16T15:02:11.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.764+0000 7fd98ea00640 1 -- 172.21.15.43:0/825165086 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051629284 0 0) 0x7fd990157b30 con 0x7fd99015aae0 2024-01-16T15:02:11.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.765+0000 7fd987e00640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd99006d470 0x7fd9900b2830 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd9780097a0 tx=0x7fd97802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf1915447355ef0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.765+0000 7fd98ea00640 1 -- 172.21.15.43:0/825165086 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978031830 con 0x7fd99006d470 2024-01-16T15:02:11.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.765+0000 7fd98ea00640 1 -- 172.21.15.43:0/825165086 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978031990 con 0x7fd99006d470 2024-01-16T15:02:11.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.765+0000 7fd98ea00640 1 -- 172.21.15.43:0/825165086 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978031d20 con 0x7fd99006d470 2024-01-16T15:02:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 -- 172.21.15.43:0/825165086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd99006d470 msgr2=0x7fd9900b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd99006d470 0x7fd9900b2830 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd9780097a0 tx=0x7fd97802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 -- 172.21.15.43:0/825165086 shutdown_connections 2024-01-16T15:02:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9900b2d70 0x7fd990158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 --2- 172.21.15.43:0/825165086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd99006d470 0x7fd9900b2830 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 -- 172.21.15.43:0/825165086 >> 172.21.15.43:0/825165086 conn(0x7fd9900b0ba0 msgr2=0x7fd9900b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:11.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 -- 172.21.15.43:0/825165086 shutdown_connections 2024-01-16T15:02:11.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.766+0000 7fd996200640 1 -- 172.21.15.43:0/825165086 wait complete. 2024-01-16T15:02:11.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.768+0000 7fd996200640 1 Processor -- start 2024-01-16T15:02:11.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.768+0000 7fd996200640 1 -- start start 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 0x7fd990070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99015aae0 0x7fd990151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd990072090 con 0x7fd9900b2d70 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd990071320 con 0x7fd99006d470 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd996200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd990072280 con 0x7fd99015aae0 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd987e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd987e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44006/0 (socket says 172.21.15.43:44006) 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd987e00640 1 -- 172.21.15.43:0/2871276847 learned_addr learned my addr 172.21.15.43:0/2871276847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:11.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd98f400640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 0x7fd990070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.770+0000 7fd98fe00640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99015aae0 0x7fd990151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 -- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99015aae0 msgr2=0x7fd990151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99015aae0 0x7fd990151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 -- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 msgr2=0x7fd990070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 0x7fd990070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 -- 172.21.15.43:0/2871276847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978009400 con 0x7fd9900b2d70 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd98f400640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 0x7fd990070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987e00640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9780098c0 tx=0x7fd9780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97803ba70 con 0x7fd9900b2d70 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97803bbf0 con 0x7fd9900b2d70 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780453f0 con 0x7fd9900b2d70 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.771+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd990072560 con 0x7fd9900b2d70 2024-01-16T15:02:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.772+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd990143360 con 0x7fd9900b2d70 2024-01-16T15:02:11.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.773+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99014cff0 con 0x7fd9900b2d70 2024-01-16T15:02:11.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.774+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fd97804e450 con 0x7fd9900b2d70 2024-01-16T15:02:11.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.775+0000 7fd987400640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 0x7fd96806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:11.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.775+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9780ac250 con 0x7fd9900b2d70 2024-01-16T15:02:11.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.775+0000 7fd98f400640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 0x7fd96806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:11.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.776+0000 7fd98f400640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 0x7fd96806d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd980007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:11.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.779+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd99014cff0 con 0x7fd9900b2d70 2024-01-16T15:02:11.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:11.999+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd99014f340 con 0x7fd96806b470 2024-01-16T15:02:12.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.022+0000 7fd987400640 1 -- 172.21.15.43:0/2871276847 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd99014f340 con 0x7fd96806b470 2024-01-16T15:02:12.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 msgr2=0x7fd96806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 0x7fd96806d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd980007ca0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 msgr2=0x7fd990145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9780098c0 tx=0x7fd9780046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 shutdown_connections 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd96806b470 0x7fd96806d930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99015aae0 0x7fd990151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9900b2d70 0x7fd990145b40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 --2- 172.21.15.43:0/2871276847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99006d470 0x7fd990070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 >> 172.21.15.43:0/2871276847 conn(0x7fd9900b0ba0 msgr2=0x7fd990153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.025+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 shutdown_connections 2024-01-16T15:02:12.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.026+0000 7fd996200640 1 -- 172.21.15.43:0/2871276847 wait complete. 2024-01-16T15:02:12.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:12.025 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:02:12.076 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:02:12.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:02:12.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.277+0000 7faaf5600640 1 Processor -- start 2024-01-16T15:02:12.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.278+0000 7faaf5600640 1 -- start start 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaf5600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faaf00d1d90 0x7faaf00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaf5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 0x7faaf00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaf5600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf00c6bf0 con 0x7faaf015d3b0 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaf5600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf00d13f0 con 0x7faaf00d1d90 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaf5600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf00cedc0 con 0x7faaf00d0fa0 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaef400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faaf00d1d90 0x7faaf00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaef400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faaf00d1d90 0x7faaf00ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44014/0 (socket says 172.21.15.43:44014) 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.279+0000 7faaef400640 1 -- 172.21.15.43:0/2993306281 learned_addr learned my addr 172.21.15.43:0/2993306281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaef400640 1 -- 172.21.15.43:0/2993306281 >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 msgr2=0x7faaf00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaf4c00640 1 --2- 172.21.15.43:0/2993306281 >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 0x7faaf00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaef400640 1 --2- 172.21.15.43:0/2993306281 >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 0x7faaf00b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaef400640 1 -- 172.21.15.43:0/2993306281 >> v1:172.21.15.69:6789/0 conn(0x7faaf015d3b0 legacy=0x7faaf00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaef400640 1 -- 172.21.15.43:0/2993306281 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00c6e40 con 0x7faaf00d1d90 2024-01-16T15:02:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.280+0000 7faaf4c00640 1 --2- 172.21.15.43:0/2993306281 >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 0x7faaf00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.281+0000 7faaef400640 1 --2- 172.21.15.43:0/2993306281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf00ce6c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faaf0072b10 tx=0x7faad802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3eac3f024243fb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.281+0000 7faaeea00640 1 -- 172.21.15.43:0/2993306281 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad80317e0 con 0x7faaf00d1d90 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.281+0000 7faaeea00640 1 -- 172.21.15.43:0/2993306281 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8031940 con 0x7faaf00d1d90 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.281+0000 7faaeea00640 1 -- 172.21.15.43:0/2993306281 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8031cd0 con 0x7faaf00d1d90 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 -- 172.21.15.43:0/2993306281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 msgr2=0x7faaf00ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 --2- 172.21.15.43:0/2993306281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf00ce6c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faaf0072b10 tx=0x7faad802b880 comp rx=0 tx=0).stop 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 -- 172.21.15.43:0/2993306281 shutdown_connections 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 --2- 172.21.15.43:0/2993306281 >> v2:172.21.15.69:3300/0 conn(0x7faaf00d0fa0 0x7faaf00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 --2- 172.21.15.43:0/2993306281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf00ce6c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 -- 172.21.15.43:0/2993306281 >> 172.21.15.43:0/2993306281 conn(0x7faaf00d30a0 msgr2=0x7faaf00d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 -- 172.21.15.43:0/2993306281 shutdown_connections 2024-01-16T15:02:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.282+0000 7faaf5600640 1 -- 172.21.15.43:0/2993306281 wait complete. 2024-01-16T15:02:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.284+0000 7faaf5600640 1 Processor -- start 2024-01-16T15:02:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.284+0000 7faaf5600640 1 -- start start 2024-01-16T15:02:12.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 0x7faaf00b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf015d3b0 0x7faaf01f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00b6dd0 con 0x7faaf00d1d90 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00cacb0 con 0x7faaf00d0fa0 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf01f6e80 con 0x7faaf015d3b0 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44026/0 (socket says 172.21.15.43:44026) 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 -- 172.21.15.43:0/984755702 learned_addr learned my addr 172.21.15.43:0/984755702 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaefe00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 0x7faaf00b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaf4c00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf015d3b0 0x7faaf01f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 -- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf015d3b0 msgr2=0x7faaf01f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf015d3b0 0x7faaf01f5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 -- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 msgr2=0x7faaf00b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 0x7faaf00b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaef400640 1 -- 172.21.15.43:0/984755702 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80094d0 con 0x7faaf00d1d90 2024-01-16T15:02:12.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.285+0000 7faaefe00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 0x7faaf00b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:12.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faaef400640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faad8000c00 tx=0x7faad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad80329c0 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8032b40 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad80453f0 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf0159460 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.286+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf00b5db0 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.287+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf004e840 con 0x7faaf00d1d90 2024-01-16T15:02:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.288+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7faad8045550 con 0x7faaf00d1d90 2024-01-16T15:02:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.289+0000 7faae7e00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 0x7faac806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.289+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faad80ac530 con 0x7faaf00d1d90 2024-01-16T15:02:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.289+0000 7faaefe00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 0x7faac806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:12.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.290+0000 7faaefe00640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 0x7faac806d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7faae0007e80 tx=0x7faae0007390 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:12.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.293+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faad8077120 con 0x7faaf00d1d90 2024-01-16T15:02:12.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:12.506+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf00c7400 con 0x7faac806b470 2024-01-16T15:02:12.539 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:02:12.558+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:02:12.630 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:02:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:13.879+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7faad80774f0 con 0x7faaf00d1d90 2024-01-16T15:02:15.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.595+0000 7faae7e00640 1 -- 172.21.15.43:0/984755702 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7faae000cdb0 con 0x7faac806b470 2024-01-16T15:02:15.576 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:02:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.600+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 msgr2=0x7faac806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.600+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 0x7faac806d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7faae0007e80 tx=0x7faae0007390 comp rx=0 tx=0).stop 2024-01-16T15:02:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.600+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 msgr2=0x7faaf0158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.600+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faad8000c00 tx=0x7faad80046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 shutdown_connections 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faac806b470 0x7faac806d930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf015d3b0 0x7faaf01f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00d1d90 0x7faaf0158f20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 --2- 172.21.15.43:0/984755702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf00d0fa0 0x7faaf00b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 >> 172.21.15.43:0/984755702 conn(0x7faaf00d30a0 msgr2=0x7faaf00c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 shutdown_connections 2024-01-16T15:02:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:15.601+0000 7faaf5600640 1 -- 172.21.15.43:0/984755702 wait complete. 2024-01-16T15:02:15.612 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:15.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:15.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:15.919 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:15.919 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:02:15.919 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:15.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.177+0000 7f3683a00640 1 Processor -- start 2024-01-16T15:02:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.177+0000 7f3683a00640 1 -- start start 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3683a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f367c0abff0 0x7f367c071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3683a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c13da30 con 0x7f367c15a100 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3683a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0ac560 con 0x7f367c0b39a0 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3683a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0ac840 con 0x7f367c0abff0 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3681000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f367c15a100 0x7f367c0b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40676/0 (socket says 172.21.15.43:40676) 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3681000640 1 -- 172.21.15.43:0/2834219103 learned_addr learned my addr 172.21.15.43:0/2834219103 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.179+0000 7f3681a00640 1 --2- 172.21.15.43:0/2834219103 >> v2:172.21.15.43:3300/0 conn(0x7f367c0abff0 0x7f367c071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.180+0000 7f3681a00640 1 -- 172.21.15.43:0/2834219103 >> v1:172.21.15.69:6789/0 conn(0x7f367c0b39a0 legacy=0x7f367c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.180+0000 7f3681a00640 1 -- 172.21.15.43:0/2834219103 >> v1:172.21.15.43:6789/0 conn(0x7f367c15a100 legacy=0x7f367c0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.180+0000 7f3681a00640 1 -- 172.21.15.43:0/2834219103 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c070650 con 0x7f367c0abff0 2024-01-16T15:02:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.180+0000 7f367b400640 1 -- 172.21.15.43:0/2834219103 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559676048 0 0) 0x7f367c13da30 con 0x7f367c15a100 2024-01-16T15:02:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.181+0000 7f3681a00640 1 --2- 172.21.15.43:0/2834219103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c071840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3674009d90 tx=0x7f367402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6269897373d9416e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.181+0000 7f367b400640 1 -- 172.21.15.43:0/2834219103 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367402cd00 con 0x7f367c0abff0 2024-01-16T15:02:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.181+0000 7f367b400640 1 -- 172.21.15.43:0/2834219103 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367402ce60 con 0x7f367c0abff0 2024-01-16T15:02:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.181+0000 7f367b400640 1 -- 172.21.15.43:0/2834219103 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674034660 con 0x7f367c0abff0 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 -- 172.21.15.43:0/2834219103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 msgr2=0x7f367c071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 --2- 172.21.15.43:0/2834219103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c071840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3674009d90 tx=0x7f367402bf50 comp rx=0 tx=0).stop 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 -- 172.21.15.43:0/2834219103 shutdown_connections 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 --2- 172.21.15.43:0/2834219103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c071840 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 -- 172.21.15.43:0/2834219103 >> 172.21.15.43:0/2834219103 conn(0x7f367c0b0d80 msgr2=0x7f367c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 -- 172.21.15.43:0/2834219103 shutdown_connections 2024-01-16T15:02:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.182+0000 7f3683a00640 1 -- 172.21.15.43:0/2834219103 wait complete. 2024-01-16T15:02:16.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.184+0000 7f3683a00640 1 Processor -- start 2024-01-16T15:02:16.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.184+0000 7f3683a00640 1 -- start start 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f367c0b39a0 0x7f367c158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f367c15a100 0x7f367c159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c15ce80 con 0x7f367c0abff0 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c1537e0 con 0x7f367c15a100 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3683a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c15c0c0 con 0x7f367c0b39a0 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45274/0 (socket says 172.21.15.43:45274) 2024-01-16T15:02:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 -- 172.21.15.43:0/1473458175 learned_addr learned my addr 172.21.15.43:0/1473458175 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 -- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f367c0b39a0 msgr2=0x7f367c158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f367c0b39a0 0x7f367c158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 -- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f367c15a100 msgr2=0x7f367c159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f367c15a100 0x7f367c159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 -- 172.21.15.43:0/1473458175 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674009a70 con 0x7f367c0abff0 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.185+0000 7f3681000640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f366c00c4a0 tx=0x7f366c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.186+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c0074e0 con 0x7f367c0abff0 2024-01-16T15:02:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.186+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c007640 con 0x7f367c0abff0 2024-01-16T15:02:16.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.186+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c004a00 con 0x7f367c0abff0 2024-01-16T15:02:16.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.186+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c15c3a0 con 0x7f367c0abff0 2024-01-16T15:02:16.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.186+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c15c960 con 0x7f367c0abff0 2024-01-16T15:02:16.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.187+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c072750 con 0x7f367c0abff0 2024-01-16T15:02:16.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.188+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f366c004b60 con 0x7f367c0abff0 2024-01-16T15:02:16.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.189+0000 7f3678c00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 0x7f365406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.189+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f366c088f40 con 0x7f367c0abff0 2024-01-16T15:02:16.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.189+0000 7f367be00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 0x7f365406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.190+0000 7f367be00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 0x7f365406d9c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3664000f60 tx=0x7f36640079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.193+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f366c053cb0 con 0x7f367c0abff0 2024-01-16T15:02:16.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.391+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f367c15cb50 con 0x7f367c0abff0 2024-01-16T15:02:16.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.393+0000 7f3678c00640 1 -- 172.21.15.43:0/1473458175 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12938 (secure 0 0 0) 0x7f366c018020 con 0x7f367c0abff0 2024-01-16T15:02:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:16.374 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":38,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:15.572023+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3759191221":"2024-01-17T15:02:12.558684+0000","172.21.15.43:6836/3759191221":"2024-01-17T15:02:12.558684+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.395+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 msgr2=0x7f365406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.395+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 0x7f365406d9c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3664000f60 tx=0x7f36640079b0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 msgr2=0x7f367c152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f366c00c4a0 tx=0x7f366c00c960 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 shutdown_connections 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f365406b500 0x7f365406d9c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f367c15a100 0x7f367c159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f367c0b39a0 0x7f367c158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 --2- 172.21.15.43:0/1473458175 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f367c0abff0 0x7f367c152e00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 >> 172.21.15.43:0/1473458175 conn(0x7f367c0b0d80 msgr2=0x7f367c0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 shutdown_connections 2024-01-16T15:02:16.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.396+0000 7f3683a00640 1 -- 172.21.15.43:0/1473458175 wait complete. 2024-01-16T15:02:16.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:16.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:16.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.633+0000 7ff788c00640 1 Processor -- start 2024-01-16T15:02:16.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.633+0000 7ff788c00640 1 -- start start 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.634+0000 7ff788c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff7840ae490 0x7ff784070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.634+0000 7ff788c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78406d7a0 con 0x7ff7840acd70 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.634+0000 7ff788c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840ae900 con 0x7ff78406dc30 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.634+0000 7ff788c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78415a5b0 con 0x7ff7840ae490 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.635+0000 7ff781e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff7840acd70 0x7ff7840ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40678/0 (socket says 172.21.15.43:40678) 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.635+0000 7ff781e00640 1 -- 172.21.15.43:0/2174073407 learned_addr learned my addr 172.21.15.43:0/2174073407 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:16.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.635+0000 7ff782800640 1 --2- 172.21.15.43:0/2174073407 >> v2:172.21.15.43:3300/0 conn(0x7ff7840ae490 0x7ff784070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.635+0000 7ff780a00640 1 -- 172.21.15.43:0/2174073407 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998413592 0 0) 0x7ff78406d7a0 con 0x7ff7840acd70 2024-01-16T15:02:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.636+0000 7ff780a00640 1 -- 172.21.15.43:0/2174073407 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764003520 con 0x7ff7840acd70 2024-01-16T15:02:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.636+0000 7ff782800640 1 -- 172.21.15.43:0/2174073407 >> v1:172.21.15.69:6789/0 conn(0x7ff78406dc30 legacy=0x7ff784070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.636+0000 7ff782800640 1 -- 172.21.15.43:0/2174073407 >> v1:172.21.15.43:6789/0 conn(0x7ff7840acd70 legacy=0x7ff7840ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.636+0000 7ff782800640 1 -- 172.21.15.43:0/2174073407 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78406ce70 con 0x7ff7840ae490 2024-01-16T15:02:16.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff782800640 1 --2- 172.21.15.43:0/2174073407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7840ae490 0x7ff784070ba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0097a0 tx=0x7ff77c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e139c7d4abf93a16 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff780a00640 1 -- 172.21.15.43:0/2174073407 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c030610 con 0x7ff7840ae490 2024-01-16T15:02:16.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff780a00640 1 -- 172.21.15.43:0/2174073407 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c030770 con 0x7ff7840ae490 2024-01-16T15:02:16.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff780a00640 1 -- 172.21.15.43:0/2174073407 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c030b00 con 0x7ff7840ae490 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff788c00640 1 -- 172.21.15.43:0/2174073407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7840ae490 msgr2=0x7ff784070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.637+0000 7ff788c00640 1 --2- 172.21.15.43:0/2174073407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7840ae490 0x7ff784070ba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0097a0 tx=0x7ff77c02b960 comp rx=0 tx=0).stop 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.638+0000 7ff788c00640 1 -- 172.21.15.43:0/2174073407 shutdown_connections 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.638+0000 7ff788c00640 1 --2- 172.21.15.43:0/2174073407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7840ae490 0x7ff784070ba0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.638+0000 7ff788c00640 1 -- 172.21.15.43:0/2174073407 >> 172.21.15.43:0/2174073407 conn(0x7ff7840b0ba0 msgr2=0x7ff7840b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.638+0000 7ff788c00640 1 -- 172.21.15.43:0/2174073407 shutdown_connections 2024-01-16T15:02:16.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.638+0000 7ff788c00640 1 -- 172.21.15.43:0/2174073407 wait complete. 2024-01-16T15:02:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.639+0000 7ff788c00640 1 Processor -- start 2024-01-16T15:02:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.639+0000 7ff788c00640 1 -- start start 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7840acd70 0x7ff7840c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7840ae490 0x7ff78414fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840c91f0 con 0x7ff78406dc30 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78409bcb0 con 0x7ff7840acd70 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.640+0000 7ff788c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78409bf90 con 0x7ff7840ae490 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45290/0 (socket says 172.21.15.43:45290) 2024-01-16T15:02:16.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 -- 172.21.15.43:0/1322496419 learned_addr learned my addr 172.21.15.43:0/1322496419 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 -- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7840ae490 msgr2=0x7ff78414fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7840ae490 0x7ff78414fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 -- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7840acd70 msgr2=0x7ff7840c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7840acd70 0x7ff7840c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 -- 172.21.15.43:0/1322496419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c009400 con 0x7ff78406dc30 2024-01-16T15:02:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.641+0000 7ff781e00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff77400cbc0 tx=0x7ff77400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.642+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400ad40 con 0x7ff78406dc30 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.642+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774012400 con 0x7ff78406dc30 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.642+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78409c270 con 0x7ff78406dc30 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.642+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400aea0 con 0x7ff78406dc30 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.642+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78409c830 con 0x7ff78406dc30 2024-01-16T15:02:16.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.643+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78409ca20 con 0x7ff78406dc30 2024-01-16T15:02:16.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.644+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff774007070 con 0x7ff78406dc30 2024-01-16T15:02:16.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.645+0000 7ff77a000640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 0x7ff75c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:16.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.645+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff774089d60 con 0x7ff78406dc30 2024-01-16T15:02:16.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.645+0000 7ff781400640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 0x7ff75c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:16.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.646+0000 7ff781400640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 0x7ff75c06d9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000f60 tx=0x7ff76c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.649+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff774058600 con 0x7ff78406dc30 2024-01-16T15:02:16.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.836+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff78404e920 con 0x7ff78406dc30 2024-01-16T15:02:16.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.836+0000 7ff77a000640 1 -- 172.21.15.43:0/1322496419 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff774017020 con 0x7ff78406dc30 2024-01-16T15:02:16.816 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:02:16.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 msgr2=0x7ff75c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 0x7ff75c06d9f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000f60 tx=0x7ff76c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 msgr2=0x7ff78409b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff77400cbc0 tx=0x7ff77400a430 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 shutdown_connections 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff75c06b530 0x7ff75c06d9f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7840acd70 0x7ff7840c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7840ae490 0x7ff78414fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 --2- 172.21.15.43:0/1322496419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff78406dc30 0x7ff78409b5b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 >> 172.21.15.43:0/1322496419 conn(0x7ff7840b0ba0 msgr2=0x7ff7840af2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 shutdown_connections 2024-01-16T15:02:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:16.839+0000 7ff788c00640 1 -- 172.21.15.43:0/1322496419 wait complete. 2024-01-16T15:02:16.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:17.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.044+0000 7fe079200640 1 Processor -- start 2024-01-16T15:02:17.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.045+0000 7fe079200640 1 -- start start 2024-01-16T15:02:17.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.046+0000 7fe079200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0741416f0 0x7fe07415ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.046+0000 7fe079200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe074141e80 0x7fe07415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.046+0000 7fe079200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe074070bf0 con 0x7fe07406cb20 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.046+0000 7fe079200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07406db30 con 0x7fe074141e80 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.046+0000 7fe079200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07406d850 con 0x7fe0741416f0 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe074141e80 0x7fe07415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe074141e80 0x7fe07415f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45300/0 (socket says 172.21.15.43:45300) 2024-01-16T15:02:17.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 -- 172.21.15.43:0/3265505585 learned_addr learned my addr 172.21.15.43:0/3265505585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:17.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe073400640 1 --2- 172.21.15.43:0/3265505585 >> v2:172.21.15.69:3300/0 conn(0x7fe0741416f0 0x7fe07415ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 -- 172.21.15.43:0/3265505585 >> v2:172.21.15.69:3300/0 conn(0x7fe0741416f0 msgr2=0x7fe07415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 --2- 172.21.15.43:0/3265505585 >> v2:172.21.15.69:3300/0 conn(0x7fe0741416f0 0x7fe07415ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 -- 172.21.15.43:0/3265505585 >> v1:172.21.15.69:6789/0 conn(0x7fe07406cb20 legacy=0x7fe07406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.047+0000 7fe06a800640 1 -- 172.21.15.43:0/3265505585 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0740705a0 con 0x7fe074141e80 2024-01-16T15:02:17.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.048+0000 7fe06a800640 1 --2- 172.21.15.43:0/3265505585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe074141e80 0x7fe07415f290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe07415fe70 tx=0x7fe05c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0648ec51237edb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.049+0000 7fe072a00640 1 -- 172.21.15.43:0/3265505585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c031870 con 0x7fe074141e80 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.049+0000 7fe072a00640 1 -- 172.21.15.43:0/3265505585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c0319d0 con 0x7fe074141e80 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.049+0000 7fe072a00640 1 -- 172.21.15.43:0/3265505585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c031d60 con 0x7fe074141e80 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.049+0000 7fe079200640 1 -- 172.21.15.43:0/3265505585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe074141e80 msgr2=0x7fe07415f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.049+0000 7fe079200640 1 --2- 172.21.15.43:0/3265505585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe074141e80 0x7fe07415f290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe07415fe70 tx=0x7fe05c02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 -- 172.21.15.43:0/3265505585 shutdown_connections 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 --2- 172.21.15.43:0/3265505585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe074141e80 0x7fe07415f290 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 --2- 172.21.15.43:0/3265505585 >> v2:172.21.15.69:3300/0 conn(0x7fe0741416f0 0x7fe07415ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 -- 172.21.15.43:0/3265505585 >> 172.21.15.43:0/3265505585 conn(0x7fe0740b0840 msgr2=0x7fe0740b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 -- 172.21.15.43:0/3265505585 shutdown_connections 2024-01-16T15:02:17.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.050+0000 7fe079200640 1 -- 172.21.15.43:0/3265505585 wait complete. 2024-01-16T15:02:17.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.051+0000 7fe079200640 1 Processor -- start 2024-01-16T15:02:17.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.051+0000 7fe079200640 1 -- start start 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe07406cb20 0x7fe0740b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe074141e80 0x7fe07413dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe074159380 con 0x7fe0741416f0 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0740b2340 con 0x7fe074141e80 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.052+0000 7fe079200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0740b2620 con 0x7fe07406cb20 2024-01-16T15:02:17.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45316/0 (socket says 172.21.15.43:45316) 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 -- 172.21.15.43:0/3567365859 learned_addr learned my addr 172.21.15.43:0/3567365859 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe073400640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe07406cb20 0x7fe0740b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe073e00640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe074141e80 0x7fe07413dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 -- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe07406cb20 msgr2=0x7fe0740b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe07406cb20 0x7fe0740b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 -- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe074141e80 msgr2=0x7fe07413dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe074141e80 0x7fe07413dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.053+0000 7fe06a800640 1 -- 172.21.15.43:0/3567365859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0094d0 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe06a800640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe05c02b950 tx=0x7fe05c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c031820 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c0443f0 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0447a0 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0740b2900 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.054+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0740b2ec0 con 0x7fe0741416f0 2024-01-16T15:02:17.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.055+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0740cca10 con 0x7fe0741416f0 2024-01-16T15:02:17.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.056+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe05c04d450 con 0x7fe0741416f0 2024-01-16T15:02:17.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.058+0000 7fe06be00640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 0x7fe04c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.058+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe05c02d040 con 0x7fe0741416f0 2024-01-16T15:02:17.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.058+0000 7fe073400640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 0x7fe04c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.059+0000 7fe073400640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 0x7fe04c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0740ad910 tx=0x7fe064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.062+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe05c077290 con 0x7fe0741416f0 2024-01-16T15:02:17.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.267+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe0740b2760 con 0x7fe0741416f0 2024-01-16T15:02:17.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.268+0000 7fe06be00640 1 -- 172.21.15.43:0/3567365859 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7fe0740b2760 con 0x7fe0741416f0 2024-01-16T15:02:17.248 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6837/3759191221 2024-01-17T15:02:12.558684+0000 2024-01-16T15:02:17.248 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3759191221 2024-01-17T15:02:12.558684+0000 2024-01-16T15:02:17.249 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 msgr2=0x7fe04c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 0x7fe04c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0740ad910 tx=0x7fe064008040 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 msgr2=0x7fe074158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe05c02b950 tx=0x7fe05c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 shutdown_connections 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe04c06b500 0x7fe04c06d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe074141e80 0x7fe07413dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0741416f0 0x7fe074158e40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 --2- 172.21.15.43:0/3567365859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe07406cb20 0x7fe0740b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 >> 172.21.15.43:0/3567365859 conn(0x7fe0740b0840 msgr2=0x7fe0740ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 shutdown_connections 2024-01-16T15:02:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.271+0000 7fe079200640 1 -- 172.21.15.43:0/3567365859 wait complete. 2024-01-16T15:02:17.268 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:02:17.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:17.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:17.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[48817]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:17.361 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:02:17.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:17.362 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:02:17.615 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:02:17.615 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:17.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.920+0000 7f262fe00640 1 Processor -- start 2024-01-16T15:02:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.920+0000 7f262fe00640 1 -- start start 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f26280b39a0 0x7f2628071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26280abff0 0x7f2628071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262813da30 con 0x7f262815a100 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262fe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280ac840 con 0x7f26280abff0 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280ac560 con 0x7f26280b39a0 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26280abff0 0x7f2628071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26280abff0 0x7f2628071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45322/0 (socket says 172.21.15.43:45322) 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.922+0000 7f262de00640 1 -- 172.21.15.43:0/701369376 learned_addr learned my addr 172.21.15.43:0/701369376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.923+0000 7f262de00640 1 -- 172.21.15.43:0/701369376 >> v2:172.21.15.69:3300/0 conn(0x7f26280b39a0 msgr2=0x7f2628071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.923+0000 7f262de00640 1 --2- 172.21.15.43:0/701369376 >> v2:172.21.15.69:3300/0 conn(0x7f26280b39a0 0x7f2628071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.923+0000 7f262de00640 1 -- 172.21.15.43:0/701369376 >> v1:172.21.15.69:6789/0 conn(0x7f262815a100 legacy=0x7f26280b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:17.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.923+0000 7f262de00640 1 -- 172.21.15.43:0/701369376 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628070650 con 0x7f26280abff0 2024-01-16T15:02:17.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.924+0000 7f262de00640 1 --2- 172.21.15.43:0/701369376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280abff0 0x7f2628071670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f262806fe60 tx=0x7f262002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c39686ac909be4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f2627e00640 1 -- 172.21.15.43:0/701369376 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262002ca80 con 0x7f26280abff0 2024-01-16T15:02:17.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f2627e00640 1 -- 172.21.15.43:0/701369376 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262002cbe0 con 0x7f26280abff0 2024-01-16T15:02:17.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f2627e00640 1 -- 172.21.15.43:0/701369376 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26200343f0 con 0x7f26280abff0 2024-01-16T15:02:17.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 -- 172.21.15.43:0/701369376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280abff0 msgr2=0x7f2628071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 --2- 172.21.15.43:0/701369376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280abff0 0x7f2628071670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f262806fe60 tx=0x7f262002bee0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 -- 172.21.15.43:0/701369376 shutdown_connections 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 --2- 172.21.15.43:0/701369376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280abff0 0x7f2628071670 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 --2- 172.21.15.43:0/701369376 >> v2:172.21.15.69:3300/0 conn(0x7f26280b39a0 0x7f2628071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.925+0000 7f262fe00640 1 -- 172.21.15.43:0/701369376 >> 172.21.15.43:0/701369376 conn(0x7f26280b0d80 msgr2=0x7f26280b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.926+0000 7f262fe00640 1 -- 172.21.15.43:0/701369376 shutdown_connections 2024-01-16T15:02:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.926+0000 7f262fe00640 1 -- 172.21.15.43:0/701369376 wait complete. 2024-01-16T15:02:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.927+0000 7f262fe00640 1 Processor -- start 2024-01-16T15:02:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.927+0000 7f262fe00640 1 -- start start 2024-01-16T15:02:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26280abff0 0x7f26281f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 0x7f26281f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281f4100 con 0x7f26280b39a0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262815b220 con 0x7f262815a100 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.928+0000 7f262fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262815bce0 con 0x7f26280abff0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45338/0 (socket says 172.21.15.43:45338) 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 -- 172.21.15.43:0/2843804677 learned_addr learned my addr 172.21.15.43:0/2843804677 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262de00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 0x7f26281f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262d400640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26280abff0 0x7f26281f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 -- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26280abff0 msgr2=0x7f26281f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26280abff0 0x7f26281f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 -- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 msgr2=0x7f26281f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 0x7f26281f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262ca00640 1 -- 172.21.15.43:0/2843804677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2620009b30 con 0x7f26280b39a0 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.929+0000 7f262de00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 0x7f26281f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f262ca00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f261000b410 tx=0x7f261000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610011830 con 0x7f26280b39a0 2024-01-16T15:02:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26100119b0 con 0x7f26280b39a0 2024-01-16T15:02:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262815bf80 con 0x7f26280b39a0 2024-01-16T15:02:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610011d60 con 0x7f26280b39a0 2024-01-16T15:02:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.930+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262815c540 con 0x7f26280b39a0 2024-01-16T15:02:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.931+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628067960 con 0x7f26280b39a0 2024-01-16T15:02:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.932+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f261001a400 con 0x7f26280b39a0 2024-01-16T15:02:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.933+0000 7f2625600640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 0x7f260006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.933+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2610089e60 con 0x7f26280b39a0 2024-01-16T15:02:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.933+0000 7f262d400640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 0x7f260006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.934+0000 7f262d400640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 0x7f260006da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26280cb030 tx=0x7f2618009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:17.938+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2610054c50 con 0x7f26280b39a0 2024-01-16T15:02:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.134+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2628074770 con 0x7f26280b39a0 2024-01-16T15:02:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.136+0000 7f2625600640 1 -- 172.21.15.43:0/2843804677 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12938 (secure 0 0 0) 0x7f2610015020 con 0x7f26280b39a0 2024-01-16T15:02:18.116 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:18.116 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":38,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:15.572023+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3759191221":"2024-01-17T15:02:12.558684+0000","172.21.15.43:6836/3759191221":"2024-01-17T15:02:12.558684+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 msgr2=0x7f260006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 0x7f260006da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26280cb030 tx=0x7f2618009040 comp rx=0 tx=0).stop 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 msgr2=0x7f26281f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f261000b410 tx=0x7f261000b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 shutdown_connections 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.138+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f260006b550 0x7f260006da10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262815a100 0x7f26281f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26280b39a0 0x7f26281f3590 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 --2- 172.21.15.43:0/2843804677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26280abff0 0x7f26281f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 >> 172.21.15.43:0/2843804677 conn(0x7f26280b0d80 msgr2=0x7f2628158700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 shutdown_connections 2024-01-16T15:02:18.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.139+0000 7f262fe00640 1 -- 172.21.15.43:0/2843804677 wait complete. 2024-01-16T15:02:18.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:18.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:18.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.386+0000 7fa9a3000640 1 Processor -- start 2024-01-16T15:02:18.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.386+0000 7fa9a3000640 1 -- start start 2024-01-16T15:02:18.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.387+0000 7fa9a3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa99c15d3b0 0x7fa99c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.387+0000 7fa9a3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d0fa0 0x7fa99c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.387+0000 7fa9a3000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a3000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0d2010 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0c6bf0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0d1400 con 0x7fa99c0d0fa0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a2600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a2600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45348/0 (socket says 172.21.15.43:45348) 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a2600640 1 -- 172.21.15.43:0/4033377692 learned_addr learned my addr 172.21.15.43:0/4033377692 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a1200640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d0fa0 0x7fa99c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a1c00640 1 --2- 172.21.15.43:0/4033377692 >> v2:172.21.15.69:3300/0 conn(0x7fa99c15d3b0 0x7fa99c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a2600640 1 -- 172.21.15.43:0/4033377692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d0fa0 msgr2=0x7fa99c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.388+0000 7fa9a2600640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d0fa0 0x7fa99c0d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.389+0000 7fa9a2600640 1 -- 172.21.15.43:0/4033377692 >> v2:172.21.15.69:3300/0 conn(0x7fa99c15d3b0 msgr2=0x7fa99c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.389+0000 7fa9a2600640 1 --2- 172.21.15.43:0/4033377692 >> v2:172.21.15.69:3300/0 conn(0x7fa99c15d3b0 0x7fa99c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.389+0000 7fa9a2600640 1 -- 172.21.15.43:0/4033377692 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0cedc0 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.390+0000 7fa9a2600640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa99c08b2a0 tx=0x7fa99402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7367bf85733cf0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.390+0000 7fa99be00640 1 -- 172.21.15.43:0/4033377692 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9940317d0 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.390+0000 7fa99be00640 1 -- 172.21.15.43:0/4033377692 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994031930 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.390+0000 7fa99be00640 1 -- 172.21.15.43:0/4033377692 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994031cc0 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 -- 172.21.15.43:0/4033377692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c0d1bc0 msgr2=0x7fa99c0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa99c08b2a0 tx=0x7fa99402b870 comp rx=0 tx=0).stop 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 -- 172.21.15.43:0/4033377692 shutdown_connections 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c0d1bc0 0x7fa99c0b3ca0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 --2- 172.21.15.43:0/4033377692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d0fa0 0x7fa99c0d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 --2- 172.21.15.43:0/4033377692 >> v2:172.21.15.69:3300/0 conn(0x7fa99c15d3b0 0x7fa99c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 -- 172.21.15.43:0/4033377692 >> 172.21.15.43:0/4033377692 conn(0x7fa99c0d30a0 msgr2=0x7fa99c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 -- 172.21.15.43:0/4033377692 shutdown_connections 2024-01-16T15:02:18.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.391+0000 7fa9a3000640 1 -- 172.21.15.43:0/4033377692 wait complete. 2024-01-16T15:02:18.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.393+0000 7fa9a3000640 1 Processor -- start 2024-01-16T15:02:18.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.393+0000 7fa9a3000640 1 -- start start 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 0x7fa99c09ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d1bc0 0x7fa99c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c09e160 con 0x7fa99c15d3b0 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c09dcd0 con 0x7fa99c0d0fa0 2024-01-16T15:02:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.394+0000 7fa9a3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c09cde0 con 0x7fa99c0d1bc0 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45356/0 (socket says 172.21.15.43:45356) 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 -- 172.21.15.43:0/3883529684 learned_addr learned my addr 172.21.15.43:0/3883529684 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a1c00640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 0x7fa99c09ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a1200640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d1bc0 0x7fa99c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 -- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d1bc0 msgr2=0x7fa99c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d1bc0 0x7fa99c0cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 -- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 msgr2=0x7fa99c09ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 0x7fa99c09ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 -- 172.21.15.43:0/3883529684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9940094d0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a1c00640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 0x7fa99c09ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:18.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.395+0000 7fa9a2600640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa9940090b0 tx=0x7fa9940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.396+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994032870 con 0x7fa99c15d3b0 2024-01-16T15:02:18.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.396+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994032a10 con 0x7fa99c15d3b0 2024-01-16T15:02:18.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.396+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c09d0c0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.396+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994032dc0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.396+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c09d680 con 0x7fa99c15d3b0 2024-01-16T15:02:18.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.397+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c09de60 con 0x7fa99c15d3b0 2024-01-16T15:02:18.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.398+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa994030070 con 0x7fa99c15d3b0 2024-01-16T15:02:18.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.399+0000 7fa999600640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 0x7fa97406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.399+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa9940abcf0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.399+0000 7fa9a1c00640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 0x7fa97406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.400+0000 7fa9a1c00640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 0x7fa97406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa98c007330 tx=0x7fa98c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.404+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa994076ae0 con 0x7fa99c15d3b0 2024-01-16T15:02:18.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.598+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa99c04e840 con 0x7fa99c15d3b0 2024-01-16T15:02:18.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.599+0000 7fa999600640 1 -- 172.21.15.43:0/3883529684 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa99407a100 con 0x7fa99c15d3b0 2024-01-16T15:02:18.579 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.601+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 msgr2=0x7fa97406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.601+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 0x7fa97406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa98c007330 tx=0x7fa98c00b040 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.601+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 msgr2=0x7fa99c1f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.601+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa9940090b0 tx=0x7fa9940046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 shutdown_connections 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa97406b500 0x7fa97406d9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa99c15d3b0 0x7fa99c1f2fe0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa99c0d1bc0 0x7fa99c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 --2- 172.21.15.43:0/3883529684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa99c0d0fa0 0x7fa99c09ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 >> 172.21.15.43:0/3883529684 conn(0x7fa99c0d30a0 msgr2=0x7fa99c0c7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 shutdown_connections 2024-01-16T15:02:18.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.602+0000 7fa9a3000640 1 -- 172.21.15.43:0/3883529684 wait complete. 2024-01-16T15:02:18.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:18.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.841+0000 7fcc45c00640 1 Processor -- start 2024-01-16T15:02:18.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.841+0000 7fcc45c00640 1 -- start start 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.842+0000 7fcc45c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc4006d520 0x7fcc4006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.842+0000 7fcc45c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc400719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc45c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc40070dc0 con 0x7fcc40165430 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc45c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc40161d00 con 0x7fcc4006d520 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc45c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4006de40 con 0x7fcc400acb50 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc3f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcc40165430 0x7fcc401658e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40690/0 (socket says 172.21.15.43:40690) 2024-01-16T15:02:18.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc3f400640 1 -- 172.21.15.43:0/3043996929 learned_addr learned my addr 172.21.15.43:0/3043996929 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc3e000640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc400719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.843+0000 7fcc3ea00640 1 --2- 172.21.15.43:0/3043996929 >> v2:172.21.15.69:3300/0 conn(0x7fcc4006d520 0x7fcc4006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098252665 0 0) 0x7fcc40070dc0 con 0x7fcc40165430 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc20003520 con 0x7fcc40165430 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945988410 0 0) 0x7fcc20003520 con 0x7fcc40165430 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 msgr2=0x7fcc400719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc400719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 >> v2:172.21.15.69:3300/0 conn(0x7fcc4006d520 msgr2=0x7fcc4006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 --2- 172.21.15.43:0/3043996929 >> v2:172.21.15.69:3300/0 conn(0x7fcc4006d520 0x7fcc4006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400cedb0 con 0x7fcc40165430 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcc380019f0 con 0x7fcc40165430 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 >> v1:172.21.15.43:6789/0 conn(0x7fcc40165430 legacy=0x7fcc401658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc20003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc20003f40 0x7fcc200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20004d00 0x7fcc20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc40161d00 con 0x7fcc20003f40 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4006de40 con 0x7fcc20004d00 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.844+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2000b980 con 0x7fcc400acb50 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc20003f40 0x7fcc200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3f400640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20004d00 0x7fcc20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3ea00640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc20003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 -- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 msgr2=0x7fcc20003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400acb50 0x7fcc20003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 -- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20004d00 msgr2=0x7fcc20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20004d00 0x7fcc20005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3e000640 1 -- 172.21.15.43:0/3043996929 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc20003520 con 0x7fcc20003f40 2024-01-16T15:02:18.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.845+0000 7fcc3f400640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20004d00 0x7fcc20005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.846+0000 7fcc3e000640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc20003f40 0x7fcc200047c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009af0 tx=0x7fcc200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706b1fecd04f3d11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.847+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2802c6b0 con 0x7fcc20003f40 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.847+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2802c810 con 0x7fcc20003f40 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.847+0000 7fcc3d600640 1 -- 172.21.15.43:0/3043996929 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2802cba0 con 0x7fcc20003f40 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 -- 172.21.15.43:0/3043996929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc20003f40 msgr2=0x7fcc200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 --2- 172.21.15.43:0/3043996929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc20003f40 0x7fcc200047c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009af0 tx=0x7fcc200057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc3f400640 1 -- 172.21.15.43:0/3043996929 reap_dead start 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 -- 172.21.15.43:0/3043996929 shutdown_connections 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 -- 172.21.15.43:0/3043996929 >> 172.21.15.43:0/3043996929 conn(0x7fcc400b0d80 msgr2=0x7fcc400b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 -- 172.21.15.43:0/3043996929 shutdown_connections 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.848+0000 7fcc45c00640 1 -- 172.21.15.43:0/3043996929 wait complete. 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.850+0000 7fcc45c00640 1 Processor -- start 2024-01-16T15:02:18.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.850+0000 7fcc45c00640 1 -- start start 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20003f40 0x7fcc40159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 0x7fcc40145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc401462c0 con 0x7fcc40142980 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400b0830 con 0x7fcc20003f40 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc45c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc40158cf0 con 0x7fcc400b03b0 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc3e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc3e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45374/0 (socket says 172.21.15.43:45374) 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.851+0000 7fcc3e000640 1 -- 172.21.15.43:0/1640902464 learned_addr learned my addr 172.21.15.43:0/1640902464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3ea00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20003f40 0x7fcc40159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3f400640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 0x7fcc40145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 -- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 msgr2=0x7fcc40145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 0x7fcc40145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 -- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20003f40 msgr2=0x7fcc40159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20003f40 0x7fcc40159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 -- 172.21.15.43:0/1640902464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40158f90 con 0x7fcc40142980 2024-01-16T15:02:18.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3f400640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 0x7fcc40145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc3e000640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcc28037040 tx=0x7fcc28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28003810 con 0x7fcc40142980 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28003970 con 0x7fcc40142980 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28003d00 con 0x7fcc40142980 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.852+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc40159270 con 0x7fcc40142980 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.853+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc40159830 con 0x7fcc40142980 2024-01-16T15:02:18.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.854+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc40074770 con 0x7fcc40142980 2024-01-16T15:02:18.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.855+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fcc28047450 con 0x7fcc40142980 2024-01-16T15:02:18.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.855+0000 7fcc36a00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 0x7fcc1806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:18.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.856+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcc280abf90 con 0x7fcc40142980 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.856+0000 7fcc3ea00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 0x7fcc1806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:18.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.856+0000 7fcc3ea00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 0x7fcc1806d9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009a90 tx=0x7fcc30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:18.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:18.860+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc28076d20 con 0x7fcc40142980 2024-01-16T15:02:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.068+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcc40067730 con 0x7fcc40142980 2024-01-16T15:02:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.069+0000 7fcc36a00640 1 -- 172.21.15.43:0/1640902464 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7fcc2804c080 con 0x7fcc40142980 2024-01-16T15:02:19.049 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6837/3759191221 2024-01-17T15:02:12.558684+0000 2024-01-16T15:02:19.049 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3759191221 2024-01-17T15:02:12.558684+0000 2024-01-16T15:02:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:02:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 msgr2=0x7fcc1806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 0x7fcc1806d9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009a90 tx=0x7fcc30008040 comp rx=0 tx=0).stop 2024-01-16T15:02:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 msgr2=0x7fcc400afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcc28037040 tx=0x7fcc28004730 comp rx=0 tx=0).stop 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 shutdown_connections 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1806b530 0x7fcc1806d9f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400b03b0 0x7fcc40145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc40142980 0x7fcc400afe70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 --2- 172.21.15.43:0/1640902464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc20003f40 0x7fcc40159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 >> 172.21.15.43:0/1640902464 conn(0x7fcc400b0d80 msgr2=0x7fcc400b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 shutdown_connections 2024-01-16T15:02:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.072+0000 7fcc45c00640 1 -- 172.21.15.43:0/1640902464 wait complete. 2024-01-16T15:02:19.070 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:02:19.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:19.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:19.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[42712]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:19.168 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:02:19.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:19.168 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:02:19.351 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:02:19.352 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:19.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:19.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:02:19.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:19.411 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:02:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:02:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:19.466 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:02:19.466 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:19.524 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:19.524 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:02:19.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:19.524 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:02:19.575 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:02:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:19.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:19.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.834+0000 7fcdfc400640 1 Processor -- start 2024-01-16T15:02:19.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.835+0000 7fcdfc400640 1 -- start start 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.836+0000 7fcdfc400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.836+0000 7fcdfc400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcdf406efb0 0x7fcdf4160f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.836+0000 7fcdfc400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcdf41614d0 0x7fcdf40d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.836+0000 7fcdfc400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf40b4060 con 0x7fcdf41614d0 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfc400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf4147140 con 0x7fcdf406efb0 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfc400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf406f3f0 con 0x7fcdf4146d60 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcdf41614d0 0x7fcdf40d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcdf41614d0 0x7fcdf40d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45382/0 (socket says 172.21.15.43:45382) 2024-01-16T15:02:19.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 -- 172.21.15.43:0/443107876 learned_addr learned my addr 172.21.15.43:0/443107876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdf9a00640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdf9000640 1 --2- 172.21.15.43:0/443107876 >> v2:172.21.15.69:3300/0 conn(0x7fcdf406efb0 0x7fcdf4160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 -- 172.21.15.43:0/443107876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 msgr2=0x7fcdf406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf406ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 -- 172.21.15.43:0/443107876 >> v2:172.21.15.69:3300/0 conn(0x7fcdf406efb0 msgr2=0x7fcdf4160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 --2- 172.21.15.43:0/443107876 >> v2:172.21.15.69:3300/0 conn(0x7fcdf406efb0 0x7fcdf4160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.837+0000 7fcdfa400640 1 -- 172.21.15.43:0/443107876 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4161ca0 con 0x7fcdf41614d0 2024-01-16T15:02:19.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.839+0000 7fcdfa400640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf41614d0 0x7fcdf40d0fe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406f5c0 tx=0x7fcdec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1caeadc1ad68df27 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.839+0000 7fcdf3e00640 1 -- 172.21.15.43:0/443107876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec031870 con 0x7fcdf41614d0 2024-01-16T15:02:19.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.839+0000 7fcdf3e00640 1 -- 172.21.15.43:0/443107876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec0319d0 con 0x7fcdf41614d0 2024-01-16T15:02:19.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.839+0000 7fcdf3e00640 1 -- 172.21.15.43:0/443107876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec031d60 con 0x7fcdf41614d0 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 -- 172.21.15.43:0/443107876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf41614d0 msgr2=0x7fcdf40d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf41614d0 0x7fcdf40d0fe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406f5c0 tx=0x7fcdec02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 -- 172.21.15.43:0/443107876 shutdown_connections 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf41614d0 0x7fcdf40d0fe0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 --2- 172.21.15.43:0/443107876 >> v2:172.21.15.69:3300/0 conn(0x7fcdf406efb0 0x7fcdf4160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 --2- 172.21.15.43:0/443107876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 -- 172.21.15.43:0/443107876 >> 172.21.15.43:0/443107876 conn(0x7fcdf40663d0 msgr2=0x7fcdf40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 -- 172.21.15.43:0/443107876 shutdown_connections 2024-01-16T15:02:19.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.840+0000 7fcdfc400640 1 -- 172.21.15.43:0/443107876 wait complete. 2024-01-16T15:02:19.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.842+0000 7fcdfc400640 1 Processor -- start 2024-01-16T15:02:19.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.842+0000 7fcdfc400640 1 -- start start 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41614d0 0x7fcdf415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41f72a0 con 0x7fcdf406efb0 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40ce710 con 0x7fcdf41614d0 2024-01-16T15:02:19.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfc400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40ce9f0 con 0x7fcdf4146d60 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45384/0 (socket says 172.21.15.43:45384) 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdf9a00640 1 -- 172.21.15.43:0/3985909202 learned_addr learned my addr 172.21.15.43:0/3985909202 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdf9000640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.843+0000 7fcdfa400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41614d0 0x7fcdf415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 -- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 msgr2=0x7fcdf415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf415eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 -- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41614d0 msgr2=0x7fcdf415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41614d0 0x7fcdf415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 -- 172.21.15.43:0/3985909202 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0094d0 con 0x7fcdf406efb0 2024-01-16T15:02:19.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf9a00640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcde4007e40 tx=0x7fcde400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4014620 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde400ee90 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf40cecd0 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.844+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde40149c0 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.845+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf415d450 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.846+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf4070090 con 0x7fcdf406efb0 2024-01-16T15:02:19.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.847+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fcde4014b20 con 0x7fcdf406efb0 2024-01-16T15:02:19.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.848+0000 7fcdf1600640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 0x7fcdcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.848+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcde408a640 con 0x7fcdf406efb0 2024-01-16T15:02:19.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.848+0000 7fcdf9000640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 0x7fcdcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.849+0000 7fcdf9000640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 0x7fcdcc06d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcddc006fd0 tx=0x7fcddc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:19.852+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcde4055480 con 0x7fcdf406efb0 2024-01-16T15:02:20.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.083+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdf4074770 con 0x7fcdf406efb0 2024-01-16T15:02:20.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.084+0000 7fcdf1600640 1 -- 172.21.15.43:0/3985909202 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3684 (secure 0 0 0) 0x7fcde4058aa0 con 0x7fcdf406efb0 2024-01-16T15:02:20.064 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:20.064 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4691,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/543136239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":543136239},{"type":"v1","addr":"172.21.15.69:6833","nonce":543136239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4939,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/198308009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":198308009},{"type":"v1","addr":"172.21.15.69:6835","nonce":198308009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/81338476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":81338476},{"type":"v1","addr":"172.21.15.43:6835","nonce":81338476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/329886073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":329886073},{"type":"v1","addr":"172.21.15.43:6837","nonce":329886073}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-16T15:02:20.065 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 20 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 msgr2=0x7fcdcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 0x7fcdcc06d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcddc006fd0 tx=0x7fcddc008040 comp rx=0 tx=0).stop 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 msgr2=0x7fcdf415cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcde4007e40 tx=0x7fcde400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 shutdown_connections 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcdcc06b500 0x7fcdcc06d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41614d0 0x7fcdf415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdf4146d60 0x7fcdf415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 --2- 172.21.15.43:0/3985909202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdf406efb0 0x7fcdf415cf10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 >> 172.21.15.43:0/3985909202 conn(0x7fcdf40663d0 msgr2=0x7fcdf40667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 shutdown_connections 2024-01-16T15:02:20.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.087+0000 7fcdfc400640 1 -- 172.21.15.43:0/3985909202 wait complete. 2024-01-16T15:02:20.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:02:20.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.331+0000 7fc291000640 1 Processor -- start 2024-01-16T15:02:20.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.332+0000 7fc291000640 1 -- start start 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.333+0000 7fc291000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc28c06dc30 0x7fc28c0ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.333+0000 7fc291000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc28c070490 0x7fc28c070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.333+0000 7fc291000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c06d7a0 con 0x7fc28c0acd70 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.333+0000 7fc291000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c15a5b0 con 0x7fc28c06dc30 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.333+0000 7fc291000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0ae9d0 con 0x7fc28c070490 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc28c06dc30 0x7fc28c0ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc28c06dc30 0x7fc28c0ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45386/0 (socket says 172.21.15.43:45386) 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 -- 172.21.15.43:0/2890838481 learned_addr learned my addr 172.21.15.43:0/2890838481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc28a800640 1 --2- 172.21.15.43:0/2890838481 >> v2:172.21.15.69:3300/0 conn(0x7fc28c070490 0x7fc28c070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 -- 172.21.15.43:0/2890838481 >> v2:172.21.15.69:3300/0 conn(0x7fc28c070490 msgr2=0x7fc28c070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 --2- 172.21.15.43:0/2890838481 >> v2:172.21.15.69:3300/0 conn(0x7fc28c070490 0x7fc28c070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 -- 172.21.15.43:0/2890838481 >> v1:172.21.15.69:6789/0 conn(0x7fc28c0acd70 legacy=0x7fc28c0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.334+0000 7fc289400640 1 -- 172.21.15.43:0/2890838481 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c06ce70 con 0x7fc28c06dc30 2024-01-16T15:02:20.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.335+0000 7fc289400640 1 --2- 172.21.15.43:0/2890838481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c06dc30 0x7fc28c0ae490 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc28c067e70 tx=0x7fc27402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b7ee19c803df6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.336+0000 7fc288a00640 1 -- 172.21.15.43:0/2890838481 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27402ee90 con 0x7fc28c06dc30 2024-01-16T15:02:20.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.336+0000 7fc288a00640 1 -- 172.21.15.43:0/2890838481 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274036440 con 0x7fc28c06dc30 2024-01-16T15:02:20.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.336+0000 7fc288a00640 1 -- 172.21.15.43:0/2890838481 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2740367d0 con 0x7fc28c06dc30 2024-01-16T15:02:20.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.336+0000 7fc291000640 1 -- 172.21.15.43:0/2890838481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c06dc30 msgr2=0x7fc28c0ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.336+0000 7fc291000640 1 --2- 172.21.15.43:0/2890838481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c06dc30 0x7fc28c0ae490 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc28c067e70 tx=0x7fc27402e0e0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 -- 172.21.15.43:0/2890838481 shutdown_connections 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 --2- 172.21.15.43:0/2890838481 >> v2:172.21.15.69:3300/0 conn(0x7fc28c070490 0x7fc28c070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 --2- 172.21.15.43:0/2890838481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c06dc30 0x7fc28c0ae490 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 -- 172.21.15.43:0/2890838481 >> 172.21.15.43:0/2890838481 conn(0x7fc28c0b0ba0 msgr2=0x7fc28c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 -- 172.21.15.43:0/2890838481 shutdown_connections 2024-01-16T15:02:20.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.337+0000 7fc291000640 1 -- 172.21.15.43:0/2890838481 wait complete. 2024-01-16T15:02:20.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.338+0000 7fc291000640 1 Processor -- start 2024-01-16T15:02:20.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 -- start start 2024-01-16T15:02:20.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 0x7fc28c148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28c0acd70 0x7fc28c0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c149020 con 0x7fc28c070490 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0cc1d0 con 0x7fc28c06dc30 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.339+0000 7fc291000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0cc4b0 con 0x7fc28c0acd70 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45390/0 (socket says 172.21.15.43:45390) 2024-01-16T15:02:20.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 -- 172.21.15.43:0/1107512321 learned_addr learned my addr 172.21.15.43:0/1107512321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:20.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 -- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28c0acd70 msgr2=0x7fc28c0ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28c0acd70 0x7fc28c0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 -- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 msgr2=0x7fc28c148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289e00640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 0x7fc28c148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 0x7fc28c148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289400640 1 -- 172.21.15.43:0/1107512321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27400b770 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.340+0000 7fc289e00640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 0x7fc28c148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc289400640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc27402e120 tx=0x7fc2740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2740348d0 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274034a30 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c0cc790 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274034de0 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.341+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c1571b0 con 0x7fc28c070490 2024-01-16T15:02:20.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.342+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c0cccb0 con 0x7fc28c070490 2024-01-16T15:02:20.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.343+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc2740403e0 con 0x7fc28c070490 2024-01-16T15:02:20.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.344+0000 7fc282000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 0x7fc26406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.345+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc2740ac2c0 con 0x7fc28c070490 2024-01-16T15:02:20.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.345+0000 7fc289e00640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 0x7fc26406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.345+0000 7fc289e00640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 0x7fc26406d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc28c06e5b0 tx=0x7fc27c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.349+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc28c0cccb0 con 0x7fc28c070490 2024-01-16T15:02:20.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.543+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fc28c0cc5f0 con 0x7fc28c070490 2024-01-16T15:02:20.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.548+0000 7fc282000640 1 -- 172.21.15.43:0/1107512321 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc28c0cc5f0 con 0x7fc28c070490 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 msgr2=0x7fc26406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 0x7fc26406d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc28c06e5b0 tx=0x7fc27c009040 comp rx=0 tx=0).stop 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 msgr2=0x7fc28c156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc27402e120 tx=0x7fc2740046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 shutdown_connections 2024-01-16T15:02:20.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26406b500 0x7fc26406d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28c0acd70 0x7fc28c0ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28c070490 0x7fc28c156c70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 --2- 172.21.15.43:0/1107512321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c06dc30 0x7fc28c148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 >> 172.21.15.43:0/1107512321 conn(0x7fc28c0b0ba0 msgr2=0x7fc28c071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 shutdown_connections 2024-01-16T15:02:20.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.551+0000 7fc291000640 1 -- 172.21.15.43:0/1107512321 wait complete. 2024-01-16T15:02:20.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:02:20.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.788+0000 7fa342800640 1 Processor -- start 2024-01-16T15:02:20.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.789+0000 7fa342800640 1 -- start start 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.790+0000 7fa342800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c15b400 0x7fa33c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.790+0000 7fa342800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa33c071300 0x7fa33c06aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.790+0000 7fa342800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c0cece0 con 0x7fa33c071a30 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.790+0000 7fa342800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33c0cf580 con 0x7fa33c071300 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.790+0000 7fa342800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c0cefc0 con 0x7fa33c15b400 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa33c071300 0x7fa33c06aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa33c071300 0x7fa33c06aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45400/0 (socket says 172.21.15.43:45400) 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 -- 172.21.15.43:0/4248587608 learned_addr learned my addr 172.21.15.43:0/4248587608 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33b400640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c15b400 0x7fa33c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 -- 172.21.15.43:0/4248587608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c15b400 msgr2=0x7fa33c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c15b400 0x7fa33c0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 -- 172.21.15.43:0/4248587608 >> v1:172.21.15.69:6789/0 conn(0x7fa33c071a30 legacy=0x7fa33c070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.791+0000 7fa33be00640 1 -- 172.21.15.43:0/4248587608 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c0cf2a0 con 0x7fa33c071300 2024-01-16T15:02:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.793+0000 7fa33be00640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c071300 0x7fa33c06aa90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa324009870 tx=0x7fa32402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8736b0da1627f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.793+0000 7fa33a000640 1 -- 172.21.15.43:0/4248587608 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324031640 con 0x7fa33c071300 2024-01-16T15:02:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.793+0000 7fa33a000640 1 -- 172.21.15.43:0/4248587608 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3240317a0 con 0x7fa33c071300 2024-01-16T15:02:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.793+0000 7fa33a000640 1 -- 172.21.15.43:0/4248587608 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324031b30 con 0x7fa33c071300 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 -- 172.21.15.43:0/4248587608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c071300 msgr2=0x7fa33c06aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c071300 0x7fa33c06aa90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa324009870 tx=0x7fa32402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 -- 172.21.15.43:0/4248587608 shutdown_connections 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c071300 0x7fa33c06aa90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 --2- 172.21.15.43:0/4248587608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c15b400 0x7fa33c0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 -- 172.21.15.43:0/4248587608 >> 172.21.15.43:0/4248587608 conn(0x7fa33c0cf740 msgr2=0x7fa33c0cfb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 -- 172.21.15.43:0/4248587608 shutdown_connections 2024-01-16T15:02:20.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.794+0000 7fa342800640 1 -- 172.21.15.43:0/4248587608 wait complete. 2024-01-16T15:02:20.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.796+0000 7fa342800640 1 Processor -- start 2024-01-16T15:02:20.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.796+0000 7fa342800640 1 -- start start 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa33c071300 0x7fa33c152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c071a30 0x7fa33c1f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1f4250 con 0x7fa33c15b400 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c0c91b0 con 0x7fa33c071300 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa342800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c152930 con 0x7fa33c071a30 2024-01-16T15:02:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.797+0000 7fa33be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45408/0 (socket says 172.21.15.43:45408) 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 -- 172.21.15.43:0/3793126959 learned_addr learned my addr 172.21.15.43:0/3793126959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33b400640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa33c071300 0x7fa33c152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33aa00640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c071a30 0x7fa33c1f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 -- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c071a30 msgr2=0x7fa33c1f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c071a30 0x7fa33c1f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 -- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa33c071300 msgr2=0x7fa33c152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa33c071300 0x7fa33c152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 -- 172.21.15.43:0/3793126959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3240094d0 con 0x7fa33c15b400 2024-01-16T15:02:20.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.798+0000 7fa33be00640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa324009870 tx=0x7fa3240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.799+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32403a930 con 0x7fa33c15b400 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.799+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32403aab0 con 0x7fa33c15b400 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.799+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32403ae60 con 0x7fa33c15b400 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.799+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c152c10 con 0x7fa33c15b400 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.799+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c0c96e0 con 0x7fa33c15b400 2024-01-16T15:02:20.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.800+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c0c98d0 con 0x7fa33c15b400 2024-01-16T15:02:20.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.801+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa32402f050 con 0x7fa33c15b400 2024-01-16T15:02:20.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.802+0000 7fa333400640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 0x7fa31406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.802+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3240abca0 con 0x7fa33c15b400 2024-01-16T15:02:20.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.802+0000 7fa33b400640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 0x7fa31406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.803+0000 7fa33b400640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 0x7fa31406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa33c06dc00 tx=0x7fa33400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:20.806+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa324076900 con 0x7fa33c15b400 2024-01-16T15:02:20.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.004+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa33c06cbd0 con 0x7fa33c15b400 2024-01-16T15:02:20.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.009+0000 7fa333400640 1 -- 172.21.15.43:0/3793126959 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3240350a0 con 0x7fa33c15b400 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 msgr2=0x7fa31406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 0x7fa31406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa33c06dc00 tx=0x7fa33400b040 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 msgr2=0x7fa33c154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa324009870 tx=0x7fa3240046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 shutdown_connections 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31406b500 0x7fa31406d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa33c15b400 0x7fa33c154610 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa33c071a30 0x7fa33c1f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 --2- 172.21.15.43:0/3793126959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa33c071300 0x7fa33c152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.011+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 >> 172.21.15.43:0/3793126959 conn(0x7fa33c0cf740 msgr2=0x7fa33c0b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.012+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 shutdown_connections 2024-01-16T15:02:20.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.012+0000 7fa342800640 1 -- 172.21.15.43:0/3793126959 wait complete. 2024-01-16T15:02:21.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T15:02:21.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.245+0000 7fd519c00640 1 Processor -- start 2024-01-16T15:02:21.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.245+0000 7fd519c00640 1 -- start start 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd519c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd514146a80 0x7fd5140cb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd519c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd51406a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd519c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd514160d30 con 0x7fd514146590 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd519c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5140cb860 con 0x7fd514146a80 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd519c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51406ae30 con 0x7fd51406a440 2024-01-16T15:02:21.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd513e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd514146590 0x7fd514071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40702/0 (socket says 172.21.15.43:40702) 2024-01-16T15:02:21.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.246+0000 7fd513e00640 1 -- 172.21.15.43:0/1201536458 learned_addr learned my addr 172.21.15.43:0/1201536458 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:21.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd519200640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd51406a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd513400640 1 --2- 172.21.15.43:0/1201536458 >> v2:172.21.15.69:3300/0 conn(0x7fd514146a80 0x7fd5140cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727565256 0 0) 0x7fd514160d30 con 0x7fd514146590 2024-01-16T15:02:21.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4003520 con 0x7fd514146590 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2331776775 0 0) 0x7fd4f4003520 con 0x7fd514146590 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 msgr2=0x7fd51406a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd51406a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 >> v2:172.21.15.69:3300/0 conn(0x7fd514146a80 msgr2=0x7fd5140cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 --2- 172.21.15.43:0/1201536458 >> v2:172.21.15.69:3300/0 conn(0x7fd514146a80 0x7fd5140cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514161b90 con 0x7fd514146590 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd504001670 con 0x7fd514146590 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 >> v1:172.21.15.43:6789/0 conn(0x7fd514146590 legacy=0x7fd514071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd4f4003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4f4003fe0 0x7fd4f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.247+0000 7fd512a00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5140cb860 con 0x7fd4f4003fe0 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51406ae30 con 0x7fd4f4004d00 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400bae0 con 0x7fd51406a440 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4f4003fe0 0x7fd4f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513e00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd4f4003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 -- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 msgr2=0x7fd4f4003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd519200640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd51406a440 0x7fd4f4003a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 -- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4004d00 msgr2=0x7fd4f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.248+0000 7fd513400640 1 -- 172.21.15.43:0/1201536458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4003520 con 0x7fd4f4003fe0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.249+0000 7fd513400640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4f4003fe0 0x7fd4f40047c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009a90 tx=0x7fd4f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6381915010d19f70 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.249+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc02c6b0 con 0x7fd4f4003fe0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc02c810 con 0x7fd4f4003fe0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd512a00640 1 -- 172.21.15.43:0/1201536458 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc02cba0 con 0x7fd4f4003fe0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 -- 172.21.15.43:0/1201536458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4f4003fe0 msgr2=0x7fd4f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 --2- 172.21.15.43:0/1201536458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4f4003fe0 0x7fd4f40047c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009a90 tx=0x7fd4f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519200640 1 -- 172.21.15.43:0/1201536458 reap_dead start 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 -- 172.21.15.43:0/1201536458 shutdown_connections 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 -- 172.21.15.43:0/1201536458 >> 172.21.15.43:0/1201536458 conn(0x7fd5140cfdc0 msgr2=0x7fd5140d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 -- 172.21.15.43:0/1201536458 shutdown_connections 2024-01-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.250+0000 7fd519c00640 1 -- 172.21.15.43:0/1201536458 wait complete. 2024-01-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.252+0000 7fd519c00640 1 Processor -- start 2024-01-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.252+0000 7fd519c00640 1 -- start start 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 0x7fd51415e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd514141280 0x7fd514142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51415df50 con 0x7fd51413b8a0 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51415d660 con 0x7fd4f4003fe0 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd519c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd514141790 con 0x7fd514141280 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd513400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd513400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45430/0 (socket says 172.21.15.43:45430) 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.253+0000 7fd513400640 1 -- 172.21.15.43:0/3927004111 learned_addr learned my addr 172.21.15.43:0/3927004111 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513e00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 0x7fd51415e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd519200640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd514141280 0x7fd514142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 -- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd514141280 msgr2=0x7fd514142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd514141280 0x7fd514142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 -- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 msgr2=0x7fd51415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 0x7fd51415e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 -- 172.21.15.43:0/3927004111 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51415d900 con 0x7fd51413b8a0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513e00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 0x7fd51415e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd513400640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc003810 con 0x7fd51413b8a0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc003990 con 0x7fd51413b8a0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc003d40 con 0x7fd51413b8a0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.254+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51415dbe0 con 0x7fd51413b8a0 2024-01-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.255+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51413bd20 con 0x7fd51413b8a0 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.256+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51415dd70 con 0x7fd51413b8a0 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.257+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd4fc03b070 con 0x7fd51413b8a0 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.257+0000 7fd50be00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 0x7fd4ec06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.258+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd4fc0ab9e0 con 0x7fd51413b8a0 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.258+0000 7fd513e00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 0x7fd4ec06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.258+0000 7fd513e00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 0x7fd4ec06d9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd504001eb0 tx=0x7fd504009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.262+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4fc076930 con 0x7fd51413b8a0 2024-01-16T15:02:21.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.516+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fd51404e840 con 0x7fd51413b8a0 2024-01-16T15:02:21.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.792+0000 7fd50be00640 1 -- 172.21.15.43:0/3927004111 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v158) v1 ==== 152+0+0 (secure 0 0 0) 0x7fd4fc079f50 con 0x7fd51413b8a0 2024-01-16T15:02:21.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 msgr2=0x7fd4ec06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 0x7fd4ec06d9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd504001eb0 tx=0x7fd504009040 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 msgr2=0x7fd514140d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc004730 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 shutdown_connections 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4ec06b530 0x7fd4ec06d9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd514141280 0x7fd514142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51413b8a0 0x7fd514140d40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 --2- 172.21.15.43:0/3927004111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4f4003fe0 0x7fd51415e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.795+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 >> 172.21.15.43:0/3927004111 conn(0x7fd5140cfdc0 msgr2=0x7fd514071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.796+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 shutdown_connections 2024-01-16T15:02:21.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:21.796+0000 7fd519c00640 1 -- 172.21.15.43:0/3927004111 wait complete. 2024-01-16T15:02:21.797 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-16T15:02:21.798 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:02:21.798 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:02:21.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:22.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.044+0000 7f5198400640 1 Processor -- start 2024-01-16T15:02:22.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.044+0000 7f5198400640 1 -- start start 2024-01-16T15:02:22.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5198400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f519015d580 0x7f51900d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5198400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51900d1dd0 con 0x7f51900ce750 2024-01-16T15:02:22.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5198400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51900d20b0 con 0x7f51900d18c0 2024-01-16T15:02:22.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5198400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51900d2390 con 0x7f519015d580 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5195a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f519015d580 0x7f51900d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5195a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f519015d580 0x7f51900d1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45432/0 (socket says 172.21.15.43:45432) 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5195a00640 1 -- 172.21.15.43:0/1731834931 learned_addr learned my addr 172.21.15.43:0/1731834931 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.046+0000 7f5196400640 1 --1- 172.21.15.43:0/1731834931 >> v1:172.21.15.43:6789/0 conn(0x7f51900ce750 0x7f51900b6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40716/0 (socket says 172.21.15.43:40716) 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.047+0000 7f5195a00640 1 -- 172.21.15.43:0/1731834931 >> v1:172.21.15.69:6789/0 conn(0x7f51900d18c0 legacy=0x7f51900ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.047+0000 7f5195a00640 1 -- 172.21.15.43:0/1731834931 >> v1:172.21.15.43:6789/0 conn(0x7f51900ce750 legacy=0x7f51900b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.047+0000 7f5195a00640 1 -- 172.21.15.43:0/1731834931 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51900d2670 con 0x7f519015d580 2024-01-16T15:02:22.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.047+0000 7f518fe00640 1 -- 172.21.15.43:0/1731834931 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302665498 0 0) 0x7f51900d1dd0 con 0x7f51900ce750 2024-01-16T15:02:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.048+0000 7f5195a00640 1 --2- 172.21.15.43:0/1731834931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900d1380 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5180009870 tx=0x7f518002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344ff9a98be0a074 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.048+0000 7f518fe00640 1 -- 172.21.15.43:0/1731834931 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518002f840 con 0x7f519015d580 2024-01-16T15:02:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.048+0000 7f518fe00640 1 -- 172.21.15.43:0/1731834931 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518002f9a0 con 0x7f519015d580 2024-01-16T15:02:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.048+0000 7f518fe00640 1 -- 172.21.15.43:0/1731834931 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518002fd30 con 0x7f519015d580 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 -- 172.21.15.43:0/1731834931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 msgr2=0x7f51900d1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 --2- 172.21.15.43:0/1731834931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900d1380 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5180009870 tx=0x7f518002ba20 comp rx=0 tx=0).stop 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 -- 172.21.15.43:0/1731834931 shutdown_connections 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 --2- 172.21.15.43:0/1731834931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900d1380 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 -- 172.21.15.43:0/1731834931 >> 172.21.15.43:0/1731834931 conn(0x7f5190071f90 msgr2=0x7f51900723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 -- 172.21.15.43:0/1731834931 shutdown_connections 2024-01-16T15:02:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.049+0000 7f5198400640 1 -- 172.21.15.43:0/1731834931 wait complete. 2024-01-16T15:02:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.051+0000 7f5198400640 1 Processor -- start 2024-01-16T15:02:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.051+0000 7f5198400640 1 -- start start 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 0x7f5190154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51900d18c0 0x7f51901f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190154d60 con 0x7f519015d580 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51901f3ea0 con 0x7f51900d18c0 2024-01-16T15:02:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5198400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51900c6e70 con 0x7f51900ce750 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5196400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5196400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45440/0 (socket says 172.21.15.43:45440) 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.052+0000 7f5196400640 1 -- 172.21.15.43:0/3635235217 learned_addr learned my addr 172.21.15.43:0/3635235217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5195a00640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 0x7f5190154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5195000640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51900d18c0 0x7f51901f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 -- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 msgr2=0x7f5190154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 0x7f5190154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 -- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51900d18c0 msgr2=0x7f51901f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51900d18c0 0x7f51901f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 -- 172.21.15.43:0/3635235217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51800094d0 con 0x7f519015d580 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5195a00640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 0x7f5190154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5196400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5188001560 tx=0x7f518800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800a640 con 0x7f519015d580 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518800a7a0 con 0x7f519015d580 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800ab30 con 0x7f519015d580 2024-01-16T15:02:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51900d2ff0 con 0x7f519015d580 2024-01-16T15:02:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.053+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51900c7430 con 0x7f519015d580 2024-01-16T15:02:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.054+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5190074770 con 0x7f519015d580 2024-01-16T15:02:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.055+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f518800ac90 con 0x7f519015d580 2024-01-16T15:02:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.057+0000 7f518d600640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 0x7f516806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.057+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f51880892a0 con 0x7f519015d580 2024-01-16T15:02:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.057+0000 7f5195a00640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 0x7f516806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.058+0000 7f5195a00640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 0x7f516806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5180036000 tx=0x7f5180039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.061+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51880540a0 con 0x7f519015d580 2024-01-16T15:02:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.247+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519015cc10 con 0x7f519015d580 2024-01-16T15:02:22.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.249+0000 7f518d600640 1 -- 172.21.15.43:0/3635235217 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12938 (secure 0 0 0) 0x7f51880576c0 con 0x7f519015d580 2024-01-16T15:02:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:22.229 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":38,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:15.572023+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3759191221":"2024-01-17T15:02:12.558684+0000","172.21.15.43:6836/3759191221":"2024-01-17T15:02:12.558684+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:22.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 msgr2=0x7f516806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 0x7f516806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5180036000 tx=0x7f5180039040 comp rx=0 tx=0).stop 2024-01-16T15:02:22.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 msgr2=0x7f51900cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5188001560 tx=0x7f518800cc70 comp rx=0 tx=0).stop 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 shutdown_connections 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f516806b500 0x7f516806d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51900d18c0 0x7f51901f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f519015d580 0x7f51900cb0a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 --2- 172.21.15.43:0/3635235217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51900ce750 0x7f5190154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.251+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 >> 172.21.15.43:0/3635235217 conn(0x7f5190071f90 msgr2=0x7f5190160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.252+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 shutdown_connections 2024-01-16T15:02:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.252+0000 7f5198400640 1 -- 172.21.15.43:0/3635235217 wait complete. 2024-01-16T15:02:22.248 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:22.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:22.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.489+0000 7f6bdbe00640 1 Processor -- start 2024-01-16T15:02:22.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.489+0000 7f6bdbe00640 1 -- start start 2024-01-16T15:02:22.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.491+0000 7f6bdbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6bdc144770 0x7f6bdc15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.491+0000 7f6bdbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 0x7f6bdc15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.491+0000 7f6bdbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc15f930 con 0x7f6bdc06c9d0 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.491+0000 7f6bdbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0b43b0 con 0x7f6bdc144770 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.491+0000 7f6bdbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0b3b10 con 0x7f6bdc0b35a0 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6bdc144770 0x7f6bdc15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6bdc144770 0x7f6bdc15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45446/0 (socket says 172.21.15.43:45446) 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 -- 172.21.15.43:0/1809421467 learned_addr learned my addr 172.21.15.43:0/1809421467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bda000640 1 --2- 172.21.15.43:0/1809421467 >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 0x7f6bdc15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 -- 172.21.15.43:0/1809421467 >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 msgr2=0x7f6bdc15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 --2- 172.21.15.43:0/1809421467 >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 0x7f6bdc15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 -- 172.21.15.43:0/1809421467 >> v1:172.21.15.69:6789/0 conn(0x7f6bdc06c9d0 legacy=0x7f6bdc06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bdaa00640 1 -- 172.21.15.43:0/1809421467 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0b3df0 con 0x7f6bdc144770 2024-01-16T15:02:22.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.492+0000 7f6bda000640 1 --2- 172.21.15.43:0/1809421467 >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 0x7f6bdc15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bdaa00640 1 --2- 172.21.15.43:0/1809421467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc15eb60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0ae900 tx=0x7f6bcc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f005893936d59dd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bd9600640 1 -- 172.21.15.43:0/1809421467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc030860 con 0x7f6bdc144770 2024-01-16T15:02:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bd9600640 1 -- 172.21.15.43:0/1809421467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc0309c0 con 0x7f6bdc144770 2024-01-16T15:02:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bd9600640 1 -- 172.21.15.43:0/1809421467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc030d50 con 0x7f6bdc144770 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bdbe00640 1 -- 172.21.15.43:0/1809421467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 msgr2=0x7f6bdc15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.494+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/1809421467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc15eb60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0ae900 tx=0x7f6bcc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 -- 172.21.15.43:0/1809421467 shutdown_connections 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/1809421467 >> v2:172.21.15.69:3300/0 conn(0x7f6bdc0b35a0 0x7f6bdc15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/1809421467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc15eb60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 -- 172.21.15.43:0/1809421467 >> 172.21.15.43:0/1809421467 conn(0x7f6bdc0cec10 msgr2=0x7f6bdc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 -- 172.21.15.43:0/1809421467 shutdown_connections 2024-01-16T15:02:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.495+0000 7f6bdbe00640 1 -- 172.21.15.43:0/1809421467 wait complete. 2024-01-16T15:02:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.496+0000 7f6bdbe00640 1 Processor -- start 2024-01-16T15:02:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.497+0000 7f6bdbe00640 1 -- start start 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.497+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bdc06c9d0 0x7f6bdc156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bdc0b35a0 0x7f6bdc0c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1570f0 con 0x7f6bdc144770 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1421a0 con 0x7f6bdc0b35a0 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc142480 con 0x7f6bdc06c9d0 2024-01-16T15:02:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45454/0 (socket says 172.21.15.43:45454) 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 -- 172.21.15.43:0/2482394815 learned_addr learned my addr 172.21.15.43:0/2482394815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdaa00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bdc06c9d0 0x7f6bdc156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bda000640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bdc0b35a0 0x7f6bdc0c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 -- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bdc06c9d0 msgr2=0x7f6bdc156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bdc06c9d0 0x7f6bdc156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 -- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bdc0b35a0 msgr2=0x7f6bdc0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bdc0b35a0 0x7f6bdc0c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.498+0000 7f6bdb400640 1 -- 172.21.15.43:0/2482394815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0094d0 con 0x7f6bdc144770 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bdb400640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400c7a0 tx=0x7f6bd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003b70 con 0x7f6bdc144770 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4003cf0 con 0x7f6bdc144770 2024-01-16T15:02:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003ea0 con 0x7f6bdc144770 2024-01-16T15:02:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc142760 con 0x7f6bdc144770 2024-01-16T15:02:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.499+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc142d20 con 0x7f6bdc144770 2024-01-16T15:02:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.500+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc04e840 con 0x7f6bdc144770 2024-01-16T15:02:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.501+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6bd400aa90 con 0x7f6bdc144770 2024-01-16T15:02:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.502+0000 7f6bd3400640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 0x7f6bb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.502+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6bd4089bf0 con 0x7f6bdc144770 2024-01-16T15:02:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.502+0000 7f6bdaa00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 0x7f6bb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.503+0000 7f6bdaa00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 0x7f6bb406d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.506+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6bd4054960 con 0x7f6bdc144770 2024-01-16T15:02:22.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.692+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6bdc0b2460 con 0x7f6bdc144770 2024-01-16T15:02:22.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.694+0000 7f6bd3400640 1 -- 172.21.15.43:0/2482394815 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6bd4057f80 con 0x7f6bdc144770 2024-01-16T15:02:22.674 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:02:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 msgr2=0x7f6bb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 0x7f6bb406d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03b040 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 msgr2=0x7f6bdc1f3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400c7a0 tx=0x7f6bd400cc60 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 shutdown_connections 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6bb406b500 0x7f6bb406d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bdc0b35a0 0x7f6bdc0c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bdc144770 0x7f6bdc1f3120 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 --2- 172.21.15.43:0/2482394815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bdc06c9d0 0x7f6bdc156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.696+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 >> 172.21.15.43:0/2482394815 conn(0x7f6bdc0cec10 msgr2=0x7f6bdc159720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.697+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 shutdown_connections 2024-01-16T15:02:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.697+0000 7f6bdbe00640 1 -- 172.21.15.43:0/2482394815 wait complete. 2024-01-16T15:02:22.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:02:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.933+0000 7f3c45800640 1 Processor -- start 2024-01-16T15:02:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.933+0000 7f3c45800640 1 -- start start 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c45800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3c4015b120 0x7f3c400adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c45800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c400cf9e0 0x7f3c400af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c45800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4015b500 con 0x7f3c400d1550 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c45800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c400ae470 con 0x7f3c400cf9e0 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c45800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c400ae710 con 0x7f3c4015b120 2024-01-16T15:02:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c44e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c400cf9e0 0x7f3c400af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c44e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c400cf9e0 0x7f3c400af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45462/0 (socket says 172.21.15.43:45462) 2024-01-16T15:02:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.935+0000 7f3c44e00640 1 -- 172.21.15.43:0/1204213320 learned_addr learned my addr 172.21.15.43:0/1204213320 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.936+0000 7f3c3fe00640 1 --2- 172.21.15.43:0/1204213320 >> v2:172.21.15.69:3300/0 conn(0x7f3c4015b120 0x7f3c400adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.936+0000 7f3c44e00640 1 -- 172.21.15.43:0/1204213320 >> v2:172.21.15.69:3300/0 conn(0x7f3c4015b120 msgr2=0x7f3c400adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.936+0000 7f3c44e00640 1 --2- 172.21.15.43:0/1204213320 >> v2:172.21.15.69:3300/0 conn(0x7f3c4015b120 0x7f3c400adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.936+0000 7f3c44e00640 1 -- 172.21.15.43:0/1204213320 >> v1:172.21.15.69:6789/0 conn(0x7f3c400d1550 legacy=0x7f3c400cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.936+0000 7f3c44e00640 1 -- 172.21.15.43:0/1204213320 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400ae9d0 con 0x7f3c400cf9e0 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c44e00640 1 --2- 172.21.15.43:0/1204213320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400af4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3c400724d0 tx=0x7f3c3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ed8c407372f2f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c3f400640 1 -- 172.21.15.43:0/1204213320 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38031870 con 0x7f3c400cf9e0 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c3f400640 1 -- 172.21.15.43:0/1204213320 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c380319d0 con 0x7f3c400cf9e0 2024-01-16T15:02:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c3f400640 1 -- 172.21.15.43:0/1204213320 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38031d60 con 0x7f3c400cf9e0 2024-01-16T15:02:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c45800640 1 -- 172.21.15.43:0/1204213320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 msgr2=0x7f3c400af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.937+0000 7f3c45800640 1 --2- 172.21.15.43:0/1204213320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400af4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3c400724d0 tx=0x7f3c3802b910 comp rx=0 tx=0).stop 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 -- 172.21.15.43:0/1204213320 shutdown_connections 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 --2- 172.21.15.43:0/1204213320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400af4b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 --2- 172.21.15.43:0/1204213320 >> v2:172.21.15.69:3300/0 conn(0x7f3c4015b120 0x7f3c400adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 -- 172.21.15.43:0/1204213320 >> 172.21.15.43:0/1204213320 conn(0x7f3c40099a00 msgr2=0x7f3c40099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 -- 172.21.15.43:0/1204213320 shutdown_connections 2024-01-16T15:02:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.938+0000 7f3c45800640 1 -- 172.21.15.43:0/1204213320 wait complete. 2024-01-16T15:02:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.939+0000 7f3c45800640 1 Processor -- start 2024-01-16T15:02:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 -- start start 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c400d1550 0x7f3c4014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4015b120 0x7f3c400cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40159850 con 0x7f3c400cf9e0 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c400c3f20 con 0x7f3c4015b120 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.940+0000 7f3c45800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4006d560 con 0x7f3c400d1550 2024-01-16T15:02:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45476/0 (socket says 172.21.15.43:45476) 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 -- 172.21.15.43:0/1472748653 learned_addr learned my addr 172.21.15.43:0/1472748653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 -- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c400d1550 msgr2=0x7f3c4014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c400d1550 0x7f3c4014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 -- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4015b120 msgr2=0x7f3c400cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4015b120 0x7f3c400cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 -- 172.21.15.43:0/1472748653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380094d0 con 0x7f3c400cf9e0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.941+0000 7f3c3fe00640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000ccd0 tx=0x7f3c3000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.942+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30003990 con 0x7f3c400cf9e0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.942+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30003af0 con 0x7f3c400cf9e0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.942+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30004020 con 0x7f3c400cf9e0 2024-01-16T15:02:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.942+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c40158c40 con 0x7f3c400cf9e0 2024-01-16T15:02:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.942+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c40159200 con 0x7f3c400cf9e0 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.943+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4004e840 con 0x7f3c400cf9e0 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.944+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3c30004180 con 0x7f3c400cf9e0 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.945+0000 7f3c3cc00640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 0x7f3c1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.945+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3c3008a310 con 0x7f3c400cf9e0 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.945+0000 7f3c37200640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 0x7f3c1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.946+0000 7f3c37200640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 0x7f3c1806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c4015a150 tx=0x7f3c28007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:22.949+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c30054ee0 con 0x7f3c400cf9e0 2024-01-16T15:02:23.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.206+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f3c400c45c0 con 0x7f3c400cf9e0 2024-01-16T15:02:23.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.804+0000 7f3c3cc00640 1 -- 172.21.15.43:0/1472748653 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v160) v1 ==== 156+0+0 (secure 0 0 0) 0x7f3c3001d020 con 0x7f3c400cf9e0 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 msgr2=0x7f3c1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 0x7f3c1806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c4015a150 tx=0x7f3c28007990 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 msgr2=0x7f3c400c39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000ccd0 tx=0x7f3c3000a5c0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 shutdown_connections 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c1806b500 0x7f3c1806d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4015b120 0x7f3c400cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c400d1550 0x7f3c4014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 --2- 172.21.15.43:0/1472748653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c400cf9e0 0x7f3c400c39e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:23.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 >> 172.21.15.43:0/1472748653 conn(0x7f3c40099a00 msgr2=0x7f3c40099de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:23.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 shutdown_connections 2024-01-16T15:02:23.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:23.808+0000 7f3c45800640 1 -- 172.21.15.43:0/1472748653 wait complete. 2024-01-16T15:02:23.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:24.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.061+0000 7fa699800640 1 Processor -- start 2024-01-16T15:02:24.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.061+0000 7fa699800640 1 -- start start 2024-01-16T15:02:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.063+0000 7fa699800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6941448c0 0x7fa69406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.063+0000 7fa699800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.063+0000 7fa699800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69406dd50 con 0x7fa69415bbd0 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.063+0000 7fa699800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa694071080 con 0x7fa6941448c0 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.063+0000 7fa699800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa694144ca0 con 0x7fa69406d910 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa6941448c0 0x7fa69406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa698e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa69415bbd0 0x7fa69415eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58690/0 (socket says 172.21.15.43:58690) 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa698e00640 1 -- 172.21.15.43:0/4079768994 learned_addr learned my addr 172.21.15.43:0/4079768994 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693400640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693e00640 1 -- 172.21.15.43:0/4079768994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 msgr2=0x7fa69415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693e00640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693e00640 1 -- 172.21.15.43:0/4079768994 >> v1:172.21.15.43:6789/0 conn(0x7fa69415bbd0 legacy=0x7fa69415eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.064+0000 7fa693e00640 1 -- 172.21.15.43:0/4079768994 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69415c450 con 0x7fa6941448c0 2024-01-16T15:02:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.066+0000 7fa693e00640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa69406d3d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa6840097a0 tx=0x7fa68402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da87b9ef9f650006 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.066+0000 7fa692a00640 1 -- 172.21.15.43:0/4079768994 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68402d040 con 0x7fa6941448c0 2024-01-16T15:02:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.066+0000 7fa692a00640 1 -- 172.21.15.43:0/4079768994 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa684031810 con 0x7fa6941448c0 2024-01-16T15:02:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.066+0000 7fa692a00640 1 -- 172.21.15.43:0/4079768994 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684031ba0 con 0x7fa6941448c0 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 -- 172.21.15.43:0/4079768994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 msgr2=0x7fa69406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa69406d3d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa6840097a0 tx=0x7fa68402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 -- 172.21.15.43:0/4079768994 shutdown_connections 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 --2- 172.21.15.43:0/4079768994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa69406d3d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 -- 172.21.15.43:0/4079768994 >> 172.21.15.43:0/4079768994 conn(0x7fa6940b0aa0 msgr2=0x7fa6940b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 -- 172.21.15.43:0/4079768994 shutdown_connections 2024-01-16T15:02:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.067+0000 7fa699800640 1 -- 172.21.15.43:0/4079768994 wait complete. 2024-01-16T15:02:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.069+0000 7fa699800640 1 Processor -- start 2024-01-16T15:02:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.069+0000 7fa699800640 1 -- start start 2024-01-16T15:02:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa69415bbd0 0x7fa694142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694143df0 con 0x7fa6941448c0 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941431e0 con 0x7fa69415bbd0 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa699800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941434c0 con 0x7fa69406d910 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa693400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa693400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53764/0 (socket says 172.21.15.43:53764) 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.070+0000 7fa693400640 1 -- 172.21.15.43:0/317471107 learned_addr learned my addr 172.21.15.43:0/317471107 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693e00640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa698e00640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa69415bbd0 0x7fa694142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 -- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 msgr2=0x7fa69415e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 -- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa69415bbd0 msgr2=0x7fa694142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa69415bbd0 0x7fa694142990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 -- 172.21.15.43:0/317471107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684009400 con 0x7fa6941448c0 2024-01-16T15:02:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa693400640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009fc0 tx=0x7fa67c00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c014620 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c014780 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c014b30 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.071+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6941437a0 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.072+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa694142ed0 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.073+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa694143f60 con 0x7fa6941448c0 2024-01-16T15:02:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.074+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa67c014c90 con 0x7fa6941448c0 2024-01-16T15:02:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.074+0000 7fa68be00640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.075+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa67c08a6f0 con 0x7fa6941448c0 2024-01-16T15:02:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.075+0000 7fa693e00640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.075+0000 7fa693e00640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 0x7fa66c06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.081+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa67c055530 con 0x7fa6941448c0 2024-01-16T15:02:24.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.267+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa69404e840 con 0x7fa6941448c0 2024-01-16T15:02:24.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.274+0000 7fa68be00640 1 -- 172.21.15.43:0/317471107 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa67c058b50 con 0x7fa6941448c0 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 msgr2=0x7fa66c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 0x7fa66c06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403c040 comp rx=0 tx=0).stop 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 msgr2=0x7fa694132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009fc0 tx=0x7fa67c00e570 comp rx=0 tx=0).stop 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 shutdown_connections 2024-01-16T15:02:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa69415bbd0 0x7fa694142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6941448c0 0x7fa694132ad0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 --2- 172.21.15.43:0/317471107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa69406d910 0x7fa69415e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 >> 172.21.15.43:0/317471107 conn(0x7fa6940b0aa0 msgr2=0x7fa694159070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 shutdown_connections 2024-01-16T15:02:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:24.277+0000 7fa699800640 1 -- 172.21.15.43:0/317471107 wait complete. 2024-01-16T15:02:24.276 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:24.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:24.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:24.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:24.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:24.516 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:24.517 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:02:24.517 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:24.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:24.571 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:02:24.571 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:24.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:24.650 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:02:24.650 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:24.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:24.710 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:02:24.710 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:24.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:24.765 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:02:24.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:24.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.014+0000 7fc3d7200640 1 Processor -- start 2024-01-16T15:02:24.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.014+0000 7fc3d7200640 1 -- start start 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc3d80cf760 0x7fc3d815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d8071700 con 0x7fc3d8143570 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d815b410 con 0x7fc3d8143950 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d815b130 con 0x7fc3d80cf760 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc3d80cf760 0x7fc3d815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d5e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc3d8143570 0x7fc3d80cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58692/0 (socket says 172.21.15.43:58692) 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3d5e00640 1 -- 172.21.15.43:0/3530817549 learned_addr learned my addr 172.21.15.43:0/3530817549 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.016+0000 7fc3cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc3d80cf760 0x7fc3d815a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53776/0 (socket says 172.21.15.43:53776) 2024-01-16T15:02:24.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.017+0000 7fc3cfe00640 1 -- 172.21.15.43:0/3530817549 >> v1:172.21.15.69:6789/0 conn(0x7fc3d8143950 legacy=0x7fc3d80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:24.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.017+0000 7fc3cfe00640 1 -- 172.21.15.43:0/3530817549 >> v1:172.21.15.43:6789/0 conn(0x7fc3d8143570 legacy=0x7fc3d80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.017+0000 7fc3cfe00640 1 -- 172.21.15.43:0/3530817549 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d815dbd0 con 0x7fc3d80cf760 2024-01-16T15:02:24.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.018+0000 7fc3cfe00640 1 --2- 172.21.15.43:0/3530817549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d80cf760 0x7fc3d815a4b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097a0 tx=0x7fc3c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87cae1c8126aa59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:24.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.018+0000 7fc3d5400640 1 -- 172.21.15.43:0/3530817549 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c002e070 con 0x7fc3d80cf760 2024-01-16T15:02:24.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.018+0000 7fc3d5400640 1 -- 172.21.15.43:0/3530817549 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c00315a0 con 0x7fc3d80cf760 2024-01-16T15:02:24.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.018+0000 7fc3d5400640 1 -- 172.21.15.43:0/3530817549 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0031930 con 0x7fc3d80cf760 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 -- 172.21.15.43:0/3530817549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d80cf760 msgr2=0x7fc3d815a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 --2- 172.21.15.43:0/3530817549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d80cf760 0x7fc3d815a4b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00097a0 tx=0x7fc3c002b840 comp rx=0 tx=0).stop 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 -- 172.21.15.43:0/3530817549 shutdown_connections 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 --2- 172.21.15.43:0/3530817549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d80cf760 0x7fc3d815a4b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 -- 172.21.15.43:0/3530817549 >> 172.21.15.43:0/3530817549 conn(0x7fc3d806a860 msgr2=0x7fc3d806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 -- 172.21.15.43:0/3530817549 shutdown_connections 2024-01-16T15:02:24.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.019+0000 7fc3d7200640 1 -- 172.21.15.43:0/3530817549 wait complete. 2024-01-16T15:02:25.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.021+0000 7fc3d7200640 1 Processor -- start 2024-01-16T15:02:25.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.021+0000 7fc3d7200640 1 -- start start 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 0x7fc3d80ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d8143950 0x7fc3d8159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d81f7210 con 0x7fc3d8143570 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d80ce9e0 con 0x7fc3d8143950 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d81f6590 con 0x7fc3d80cf760 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3cfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3cfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53780/0 (socket says 172.21.15.43:53780) 2024-01-16T15:02:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3cfe00640 1 -- 172.21.15.43:0/885544903 learned_addr learned my addr 172.21.15.43:0/885544903 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d5e00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 0x7fc3d80ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.022+0000 7fc3d6800640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d8143950 0x7fc3d8159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 -- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 msgr2=0x7fc3d80ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 0x7fc3d80ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 -- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d8143950 msgr2=0x7fc3d8159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d8143950 0x7fc3d8159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 -- 172.21.15.43:0/885544903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0009400 con 0x7fc3d8143570 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3d5e00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 0x7fc3d80ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3cfe00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0034040 tx=0x7fc3c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0032900 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0032a80 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0032e30 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d81f6870 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.023+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d81f6e30 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.025+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d81f7350 con 0x7fc3d8143570 2024-01-16T15:02:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.025+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc3c003b3f0 con 0x7fc3d8143570 2024-01-16T15:02:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.026+0000 7fc3ce000640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 0x7fc3b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.026+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3c00abf60 con 0x7fc3d8143570 2024-01-16T15:02:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.027+0000 7fc3d5e00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 0x7fc3b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.027+0000 7fc3d5e00640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 0x7fc3b006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005880 tx=0x7fc3c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.031+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3d81f7350 con 0x7fc3d8143570 2024-01-16T15:02:25.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.262+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d81f7350 con 0x7fc3d8143570 2024-01-16T15:02:25.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.263+0000 7fc3ce000640 1 -- 172.21.15.43:0/885544903 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3684 (secure 0 0 0) 0x7fc3d81f7350 con 0x7fc3d8143570 2024-01-16T15:02:25.243 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:25.243 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4691,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/543136239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":543136239},{"type":"v1","addr":"172.21.15.69:6833","nonce":543136239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4939,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/198308009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":198308009},{"type":"v1","addr":"172.21.15.69:6835","nonce":198308009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/81338476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":81338476},{"type":"v1","addr":"172.21.15.43:6835","nonce":81338476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/329886073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":329886073},{"type":"v1","addr":"172.21.15.43:6837","nonce":329886073}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-16T15:02:25.243 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 20 2024-01-16T15:02:25.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.265+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 msgr2=0x7fc3b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.265+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 0x7fc3b006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005880 tx=0x7fc3c8009040 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.265+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 msgr2=0x7fc3d80ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.265+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0034040 tx=0x7fc3c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 shutdown_connections 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3b006b500 0x7fc3b006d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3d8143950 0x7fc3d8159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3d8143570 0x7fc3d80ce4a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 --2- 172.21.15.43:0/885544903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3d80cf760 0x7fc3d80ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 >> 172.21.15.43:0/885544903 conn(0x7fc3d806a860 msgr2=0x7fc3d806ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 shutdown_connections 2024-01-16T15:02:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.266+0000 7fc3d7200640 1 -- 172.21.15.43:0/885544903 wait complete. 2024-01-16T15:02:25.266 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:02:25.267 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:02:25.267 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:02:25.267 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:25.267 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:02:25.267 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:02:25.267 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:02:25.267 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:25.268 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:02:25.268 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:02:25.268 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:02:25.268 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:25.268 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:02:25.268 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:02:25.268 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:02:25.268 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:25.284 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:02:25.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:02:25.288 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:02:25.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:02:25.289 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:02:25.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:02:25.292 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:02:25.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:02:25.337 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:02:25.337 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:25.339 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:02:25.339 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:25.342 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:02:25.388 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:25.388 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:02:25.389 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:25.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:02:25.496 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:02:25.496 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:02:25.504 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:02:25.513 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:02:25.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.603+0000 7f378b800640 1 Processor -- start 2024-01-16T15:02:25.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.603+0000 7f378b800640 1 -- start start 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.604+0000 7f378b800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 0x7f378406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.604+0000 7f378b800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f37840b2080 0x7f37840b2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.604+0000 7f378b800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3784070dc0 con 0x7f37840acb50 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.604+0000 7f378b800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37840720e0 con 0x7f37840b2080 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.604+0000 7f378b800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3784161d00 con 0x7f378406d520 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f37840b2080 0x7f37840b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f37840b2080 0x7f37840b2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53826/0 (socket says 172.21.15.43:53826) 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3783e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f37840acb50 0x7f37840719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58716/0 (socket says 172.21.15.43:58716) 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 -- 172.21.15.43:0/2376591639 learned_addr learned my addr 172.21.15.43:0/2376591639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3788e00640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 0x7f378406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 -- 172.21.15.43:0/2376591639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 msgr2=0x7f378406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 0x7f378406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 -- 172.21.15.43:0/2376591639 >> v1:172.21.15.43:6789/0 conn(0x7f37840acb50 legacy=0x7f37840719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3783400640 1 -- 172.21.15.43:0/2376591639 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495822478 0 0) 0x7f3784070dc0 con 0x7f37840acb50 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3789800640 1 -- 172.21.15.43:0/2376591639 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378406de40 con 0x7f37840b2080 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.605+0000 7f3788e00640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 0x7f378406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.606+0000 7f3789800640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840b2080 0x7f37840b2530 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f377c009870 tx=0x7f377c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dc6a11f3ff39c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f3783400640 1 -- 172.21.15.43:0/2376591639 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c030840 con 0x7f37840b2080 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f3783400640 1 -- 172.21.15.43:0/2376591639 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c0309a0 con 0x7f37840b2080 2024-01-16T15:02:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f3783400640 1 -- 172.21.15.43:0/2376591639 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c030d30 con 0x7f37840b2080 2024-01-16T15:02:25.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 -- 172.21.15.43:0/2376591639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840b2080 msgr2=0x7f37840b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840b2080 0x7f37840b2530 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f377c009870 tx=0x7f377c02b980 comp rx=0 tx=0).stop 2024-01-16T15:02:25.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 -- 172.21.15.43:0/2376591639 shutdown_connections 2024-01-16T15:02:25.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840b2080 0x7f37840b2530 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 --2- 172.21.15.43:0/2376591639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f378406d520 0x7f378406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 -- 172.21.15.43:0/2376591639 >> 172.21.15.43:0/2376591639 conn(0x7f37840b0d80 msgr2=0x7f37840b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 -- 172.21.15.43:0/2376591639 shutdown_connections 2024-01-16T15:02:25.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.607+0000 7f378b800640 1 -- 172.21.15.43:0/2376591639 wait complete. 2024-01-16T15:02:25.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.609+0000 7f378b800640 1 Processor -- start 2024-01-16T15:02:25.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.609+0000 7f378b800640 1 -- start start 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378406d520 0x7f378415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37840b2080 0x7f3784165150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378409a220 con 0x7f37840acb50 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378409a500 con 0x7f378406d520 2024-01-16T15:02:25.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.610+0000 7f378b800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378409a7e0 con 0x7f37840b2080 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53836/0 (socket says 172.21.15.43:53836) 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 -- 172.21.15.43:0/1291510524 learned_addr learned my addr 172.21.15.43:0/1291510524 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3789800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37840b2080 0x7f3784165150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3788e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378406d520 0x7f378415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 -- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37840b2080 msgr2=0x7f3784165150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37840b2080 0x7f3784165150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 -- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378406d520 msgr2=0x7f378415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378406d520 0x7f378415c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.611+0000 7f3783e00640 1 -- 172.21.15.43:0/1291510524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0094d0 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f3783e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c4d0 tx=0x7f376c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c0074e0 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c007660 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c004a20 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378409aac0 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.612+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37841f75a0 con 0x7f37840acb50 2024-01-16T15:02:25.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.614+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37841f7820 con 0x7f37840acb50 2024-01-16T15:02:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.614+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f376c004b80 con 0x7f37840acb50 2024-01-16T15:02:25.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.616+0000 7f3780c00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 0x7f375c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.616+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f376c088e90 con 0x7f37840acb50 2024-01-16T15:02:25.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.616+0000 7f3788e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 0x7f375c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.617+0000 7f3788e00640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 0x7f375c06d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3784072000 tx=0x7f37740097f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.620+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f376c053cb0 con 0x7f37840acb50 2024-01-16T15:02:25.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:25.830+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3784164040 con 0x7f37840acb50 2024-01-16T15:02:26.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.858+0000 7f3780c00640 1 -- 172.21.15.43:0/1291510524 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7f376c018020 con 0x7f37840acb50 2024-01-16T15:02:26.838 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:02:26.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.861+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 msgr2=0x7f375c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.861+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 0x7f375c06d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3784072000 tx=0x7f37740097f0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.861+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 msgr2=0x7f37841f7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.861+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c4d0 tx=0x7f376c00c990 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.861+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 shutdown_connections 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f375c06b530 0x7f375c06d9f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f37840b2080 0x7f3784165150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37840acb50 0x7f37841f7060 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 --2- 172.21.15.43:0/1291510524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378406d520 0x7f378415c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 >> 172.21.15.43:0/1291510524 conn(0x7f37840b0d80 msgr2=0x7f3784161470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 shutdown_connections 2024-01-16T15:02:26.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:26.862+0000 7f378b800640 1 -- 172.21.15.43:0/1291510524 wait complete. 2024-01-16T15:02:26.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:02:27.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.105+0000 7ff84be00640 1 Processor -- start 2024-01-16T15:02:27.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.106+0000 7ff84be00640 1 -- start start 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff84c0d3810 0x7ff84c0cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff84c0d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0d2020 con 0x7ff84c15d430 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c0d09d0 con 0x7ff84c0d3810 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0d3ca0 con 0x7ff84c0d1690 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff84c15d430 0x7ff84c1615e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58732/0 (socket says 172.21.15.43:58732) 2024-01-16T15:02:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.107+0000 7ff84aa00640 1 -- 172.21.15.43:0/1232579451 learned_addr learned my addr 172.21.15.43:0/1232579451 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84b400640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff84c0d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff843e00640 1 --2- 172.21.15.43:0/1232579451 >> v2:172.21.15.69:3300/0 conn(0x7ff84c0d3810 0x7ff84c0cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144840218 0 0) 0x7ff83c001110 con 0x7ff84c15d430 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c003520 con 0x7ff84c15d430 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613121850 0 0) 0x7ff82c003520 con 0x7ff84c15d430 2024-01-16T15:02:27.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 msgr2=0x7ff84c0d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff84c0d1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 >> v2:172.21.15.69:3300/0 conn(0x7ff84c0d3810 msgr2=0x7ff84c0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 --2- 172.21.15.43:0/1232579451 >> v2:172.21.15.69:3300/0 conn(0x7ff84c0d3810 0x7ff84c0cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c15d810 con 0x7ff84c15d430 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff83c001820 con 0x7ff84c15d430 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 >> v1:172.21.15.43:6789/0 conn(0x7ff84c15d430 legacy=0x7ff84c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c003a10 0x7ff82c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff82c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff82c004d00 0x7ff82c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0d09d0 con 0x7ff82c004d00 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0d3ca0 con 0x7ff82c003a10 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.108+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c00b9b0 con 0x7ff84c0d1690 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff82c004d00 0x7ff82c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff843e00640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff82c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84aa00640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c003a10 0x7ff82c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 -- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 msgr2=0x7ff82c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0d1690 0x7ff82c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 -- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c003a10 msgr2=0x7ff82c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c003a10 0x7ff82c003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.109+0000 7ff84b400640 1 -- 172.21.15.43:0/1232579451 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c003520 con 0x7ff82c004d00 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.110+0000 7ff84b400640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff82c004d00 0x7ff82c005180 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff844009fc0 tx=0x7ff82c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43032a667d9e0077 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84402cc20 con 0x7ff82c004d00 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84400f040 con 0x7ff82c004d00 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84a000640 1 -- 172.21.15.43:0/1232579451 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844034440 con 0x7ff82c004d00 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 -- 172.21.15.43:0/1232579451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff82c004d00 msgr2=0x7ff82c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 --2- 172.21.15.43:0/1232579451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff82c004d00 0x7ff82c005180 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff844009fc0 tx=0x7ff82c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84b400640 1 -- 172.21.15.43:0/1232579451 reap_dead start 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 -- 172.21.15.43:0/1232579451 shutdown_connections 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 -- 172.21.15.43:0/1232579451 >> 172.21.15.43:0/1232579451 conn(0x7ff84c08cb00 msgr2=0x7ff84c09c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 -- 172.21.15.43:0/1232579451 shutdown_connections 2024-01-16T15:02:27.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.111+0000 7ff84be00640 1 -- 172.21.15.43:0/1232579451 wait complete. 2024-01-16T15:02:27.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.113+0000 7ff84be00640 1 Processor -- start 2024-01-16T15:02:27.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.114+0000 7ff84be00640 1 -- start start 2024-01-16T15:02:27.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 0x7ff84c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0b3d60 0x7ff84c0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0b4720 con 0x7ff84c06b330 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0c8b90 con 0x7ff82c004d00 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c152e30 con 0x7ff84c0b3d60 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53864/0 (socket says 172.21.15.43:53864) 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 -- 172.21.15.43:0/3634877050 learned_addr learned my addr 172.21.15.43:0/3634877050 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84aa00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 0x7ff84c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84b400640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0b3d60 0x7ff84c0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 -- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0b3d60 msgr2=0x7ff84c0b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0b3d60 0x7ff84c0b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 -- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 msgr2=0x7ff84c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 0x7ff84c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff843e00640 1 -- 172.21.15.43:0/3634877050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c0c7d20 con 0x7ff84c06b330 2024-01-16T15:02:27.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.115+0000 7ff84aa00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 0x7ff84c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff843e00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff8340116d0 tx=0x7ff834011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83400f9c0 con 0x7ff84c06b330 2024-01-16T15:02:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83400fb20 con 0x7ff84c06b330 2024-01-16T15:02:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c0c7fc0 con 0x7ff84c06b330 2024-01-16T15:02:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83400fe90 con 0x7ff84c06b330 2024-01-16T15:02:27.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.116+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0c8540 con 0x7ff84c06b330 2024-01-16T15:02:27.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.117+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c067960 con 0x7ff84c06b330 2024-01-16T15:02:27.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.118+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff834015460 con 0x7ff84c06b330 2024-01-16T15:02:27.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.120+0000 7ff843400640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 0x7ff82406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.120+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff83408a620 con 0x7ff84c06b330 2024-01-16T15:02:27.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.120+0000 7ff84aa00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 0x7ff82406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.121+0000 7ff84aa00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 0x7ff82406d9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0012b0 tx=0x7ff83c0011e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.124+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff834055490 con 0x7ff84c06b330 2024-01-16T15:02:27.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.388+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff84c0c8d00 con 0x7ff84c06b330 2024-01-16T15:02:27.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.390+0000 7ff843400640 1 -- 172.21.15.43:0/3634877050 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff834058ab0 con 0x7ff84c06b330 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.371 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.373 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:02:27.374 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.375 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:02:27.377 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:02:27.378 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:02:27.379 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.380 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:02:27.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:27.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:02:27.382 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:27.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 msgr2=0x7ff82406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 0x7ff82406d9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0012b0 tx=0x7ff83c0011e0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 msgr2=0x7ff84c0cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff8340116d0 tx=0x7ff834011b90 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 shutdown_connections 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff82406b530 0x7ff82406d9f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff84c0b3d60 0x7ff84c0b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff84c06b330 0x7ff84c0cca10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 --2- 172.21.15.43:0/3634877050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff82c004d00 0x7ff84c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.392+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 >> 172.21.15.43:0/3634877050 conn(0x7ff84c08cb00 msgr2=0x7ff84c06b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.393+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 shutdown_connections 2024-01-16T15:02:27.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.393+0000 7ff84be00640 1 -- 172.21.15.43:0/3634877050 wait complete. 2024-01-16T15:02:27.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.622+0000 7f3953200640 1 Processor -- start 2024-01-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.622+0000 7f3953200640 1 -- start start 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3953200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f395415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3953200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3954146a60 0x7f39540b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3953200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3954160da0 con 0x7f39540b29e0 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3953200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39540b2e70 con 0x7f3954146a60 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3953200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3954161920 con 0x7f39540ce950 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3952800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f39540b29e0 0x7f3954142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58736/0 (socket says 172.21.15.43:58736) 2024-01-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3952800640 1 -- 172.21.15.43:0/1507911021 learned_addr learned my addr 172.21.15.43:0/1507911021 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f3951e00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f395415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.624+0000 7f394be00640 1 --2- 172.21.15.43:0/1507911021 >> v2:172.21.15.69:3300/0 conn(0x7f3954146a60 0x7f39540b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007086383 0 0) 0x7f3954160da0 con 0x7f39540b29e0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3934003520 con 0x7f39540b29e0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569852363 0 0) 0x7f3934003520 con 0x7f39540b29e0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 msgr2=0x7f395415e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f395415e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 >> v2:172.21.15.69:3300/0 conn(0x7f3954146a60 msgr2=0x7f39540b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 --2- 172.21.15.43:0/1507911021 >> v2:172.21.15.69:3300/0 conn(0x7f3954146a60 0x7f39540b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39540667c0 con 0x7f39540b29e0 2024-01-16T15:02:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f394c0019f0 con 0x7f39540b29e0 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 >> v1:172.21.15.43:6789/0 conn(0x7f39540b29e0 legacy=0x7f3954142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39340039e0 0x7f3934003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f39340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3934004d00 0x7f3934005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39540b2e70 con 0x7f39340039e0 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3954161920 con 0x7f3934004d00 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393400ba30 con 0x7f39540ce950 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.625+0000 7f3951e00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39340039e0 0x7f3934003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f394be00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f39340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3952800640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3934004d00 0x7f3934005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3951e00640 1 -- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 msgr2=0x7f39340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3951e00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39540ce950 0x7f39340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3951e00640 1 -- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3934004d00 msgr2=0x7f3934005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3951e00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3934004d00 0x7f3934005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.626+0000 7f3951e00640 1 -- 172.21.15.43:0/1507911021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934003520 con 0x7f39340039e0 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.627+0000 7f3951e00640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39340039e0 0x7f3934003e00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3944009a90 tx=0x7f39340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de244f46cb27313 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394402c680 con 0x7f39340039e0 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394402c7e0 con 0x7f39340039e0 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3951400640 1 -- 172.21.15.43:0/1507911021 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394402cb70 con 0x7f39340039e0 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 -- 172.21.15.43:0/1507911021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39340039e0 msgr2=0x7f3934003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 --2- 172.21.15.43:0/1507911021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39340039e0 0x7f3934003e00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3944009a90 tx=0x7f39340057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3952800640 1 -- 172.21.15.43:0/1507911021 reap_dead start 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 -- 172.21.15.43:0/1507911021 shutdown_connections 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 -- 172.21.15.43:0/1507911021 >> 172.21.15.43:0/1507911021 conn(0x7f39540cf4b0 msgr2=0x7f39540cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 -- 172.21.15.43:0/1507911021 shutdown_connections 2024-01-16T15:02:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.628+0000 7f3953200640 1 -- 172.21.15.43:0/1507911021 wait complete. 2024-01-16T15:02:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.630+0000 7f3953200640 1 Processor -- start 2024-01-16T15:02:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.630+0000 7f3953200640 1 -- start start 2024-01-16T15:02:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39340039e0 0x7f395413c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 0x7f395413e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395414ef40 con 0x7f395414e580 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395413e980 con 0x7f39541386c0 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.631+0000 7f3953200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395413ec60 con 0x7f39340039e0 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53882/0 (socket says 172.21.15.43:53882) 2024-01-16T15:02:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 -- 172.21.15.43:0/2760959610 learned_addr learned my addr 172.21.15.43:0/2760959610 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f394be00640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 0x7f395413e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3951e00640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39340039e0 0x7f395413c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 -- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39340039e0 msgr2=0x7f395413c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39340039e0 0x7f395413c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 -- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 msgr2=0x7f395413e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 0x7f395413e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 -- 172.21.15.43:0/2760959610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395413ef40 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f394be00640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 0x7f395413e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.632+0000 7f3952800640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3954140930 tx=0x7f394c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.633+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c004d60 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.633+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394c00bb80 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.633+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395413f220 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.633+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c007110 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.633+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395413e5e0 con 0x7f395414e580 2024-01-16T15:02:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.634+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395404e840 con 0x7f395414e580 2024-01-16T15:02:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.635+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f394c007270 con 0x7f395414e580 2024-01-16T15:02:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.636+0000 7f394a000640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 0x7f392c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.636+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f394c08a120 con 0x7f395414e580 2024-01-16T15:02:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.636+0000 7f3951e00640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 0x7f392c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.637+0000 7f3951e00640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 0x7f392c06d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3944036000 tx=0x7f3944030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.641+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f394c054f00 con 0x7f395414e580 2024-01-16T15:02:27.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.838+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f395406b5b0 con 0x7f395414e580 2024-01-16T15:02:27.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.839+0000 7f394a000640 1 -- 172.21.15.43:0/2760959610 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+12809 (secure 0 0 0) 0x7f394c058520 con 0x7f395414e580 2024-01-16T15:02:27.820 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:27.820 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":43,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:26.853777+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 msgr2=0x7f392c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 0x7f392c06d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3944036000 tx=0x7f3944030000 comp rx=0 tx=0).stop 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 msgr2=0x7f395414ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3954140930 tx=0x7f394c00b4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 shutdown_connections 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f392c06b500 0x7f392c06d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39541386c0 0x7f395413e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f395414e580 0x7f395414ea00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 --2- 172.21.15.43:0/2760959610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39340039e0 0x7f395413c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 >> 172.21.15.43:0/2760959610 conn(0x7f39540cf4b0 msgr2=0x7f395413ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 shutdown_connections 2024-01-16T15:02:27.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:27.843+0000 7f3953200640 1 -- 172.21.15.43:0/2760959610 wait complete. 2024-01-16T15:02:27.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:27.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:28.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.093+0000 7fe558600640 1 Processor -- start 2024-01-16T15:02:28.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.093+0000 7fe558600640 1 -- start start 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe558600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe550160b10 0x7fe55006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe558600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe550072280 0x7fe55006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe558600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55006cd50 con 0x7fe55006c900 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe558600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5500b2320 con 0x7fe550072280 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe558600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55006df30 con 0x7fe550160b10 2024-01-16T15:02:28.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe555200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe550072280 0x7fe55006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe555200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe550072280 0x7fe55006c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53890/0 (socket says 172.21.15.43:53890) 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.095+0000 7fe555200640 1 -- 172.21.15.43:0/876098433 learned_addr learned my addr 172.21.15.43:0/876098433 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.096+0000 7fe555c00640 1 --2- 172.21.15.43:0/876098433 >> v2:172.21.15.69:3300/0 conn(0x7fe550160b10 0x7fe55006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.096+0000 7fe555200640 1 -- 172.21.15.43:0/876098433 >> v2:172.21.15.69:3300/0 conn(0x7fe550160b10 msgr2=0x7fe55006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.096+0000 7fe555200640 1 --2- 172.21.15.43:0/876098433 >> v2:172.21.15.69:3300/0 conn(0x7fe550160b10 0x7fe55006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.096+0000 7fe555200640 1 -- 172.21.15.43:0/876098433 >> v1:172.21.15.69:6789/0 conn(0x7fe55006c900 legacy=0x7fe550160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.096+0000 7fe555200640 1 -- 172.21.15.43:0/876098433 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550071780 con 0x7fe550072280 2024-01-16T15:02:28.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.097+0000 7fe555200640 1 --2- 172.21.15.43:0/876098433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe550072280 0x7fe55006c3c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b4070 tx=0x7fe53802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dee025c3202c710 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.097+0000 7fe54fe00640 1 -- 172.21.15.43:0/876098433 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380317e0 con 0x7fe550072280 2024-01-16T15:02:28.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.097+0000 7fe54fe00640 1 -- 172.21.15.43:0/876098433 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538031940 con 0x7fe550072280 2024-01-16T15:02:28.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.097+0000 7fe54fe00640 1 -- 172.21.15.43:0/876098433 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538031cd0 con 0x7fe550072280 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 -- 172.21.15.43:0/876098433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe550072280 msgr2=0x7fe55006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 --2- 172.21.15.43:0/876098433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe550072280 0x7fe55006c3c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b4070 tx=0x7fe53802b880 comp rx=0 tx=0).stop 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 -- 172.21.15.43:0/876098433 shutdown_connections 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 --2- 172.21.15.43:0/876098433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe550072280 0x7fe55006c3c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 --2- 172.21.15.43:0/876098433 >> v2:172.21.15.69:3300/0 conn(0x7fe550160b10 0x7fe55006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 -- 172.21.15.43:0/876098433 >> 172.21.15.43:0/876098433 conn(0x7fe5500b0ba0 msgr2=0x7fe5500b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 -- 172.21.15.43:0/876098433 shutdown_connections 2024-01-16T15:02:28.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.098+0000 7fe558600640 1 -- 172.21.15.43:0/876098433 wait complete. 2024-01-16T15:02:28.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.100+0000 7fe558600640 1 Processor -- start 2024-01-16T15:02:28.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.100+0000 7fe558600640 1 -- start start 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe550072280 0x7fe5501f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550160b10 0x7fe5501f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5500cc290 con 0x7fe55006c900 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5500cc570 con 0x7fe550160b10 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe558600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5500cc850 con 0x7fe550072280 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.101+0000 7fe555c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53904/0 (socket says 172.21.15.43:53904) 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 -- 172.21.15.43:0/2020955705 learned_addr learned my addr 172.21.15.43:0/2020955705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:28.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555200640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe550072280 0x7fe5501f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 -- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe550072280 msgr2=0x7fe5501f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe550072280 0x7fe5501f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe556600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550160b10 0x7fe5501f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 -- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550160b10 msgr2=0x7fe5501f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550160b10 0x7fe5501f2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 -- 172.21.15.43:0/2020955705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380094d0 con 0x7fe55006c900 2024-01-16T15:02:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.102+0000 7fe555c00640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe54000d5f0 tx=0x7fe54000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.103+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400125f0 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.103+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe540012750 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.103+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe540012b00 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.103+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5500cbf80 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.103+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5500ccb30 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.104+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55009d200 con 0x7fe55006c900 2024-01-16T15:02:28.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.105+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe540012c60 con 0x7fe55006c900 2024-01-16T15:02:28.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.106+0000 7fe54d600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 0x7fe52806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.106+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe54008a180 con 0x7fe55006c900 2024-01-16T15:02:28.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.106+0000 7fe555200640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 0x7fe52806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.107+0000 7fe555200640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 0x7fe52806d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe53803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.111+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe540054fd0 con 0x7fe55006c900 2024-01-16T15:02:28.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.306+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe550155cd0 con 0x7fe55006c900 2024-01-16T15:02:28.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.307+0000 7fe54d600640 1 -- 172.21.15.43:0/2020955705 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5400585f0 con 0x7fe55006c900 2024-01-16T15:02:28.287 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:02:28.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.309+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 msgr2=0x7fe52806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.309+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 0x7fe52806d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe53803d040 comp rx=0 tx=0).stop 2024-01-16T15:02:28.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.309+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 msgr2=0x7fe5500cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.309+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe54000d5f0 tx=0x7fe54000dab0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.309+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 shutdown_connections 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe52806b500 0x7fe52806d9c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe550160b10 0x7fe5501f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe550072280 0x7fe5501f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 --2- 172.21.15.43:0/2020955705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe55006c900 0x7fe5500cba40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 >> 172.21.15.43:0/2020955705 conn(0x7fe5500b0ba0 msgr2=0x7fe55015e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 shutdown_connections 2024-01-16T15:02:28.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.310+0000 7fe558600640 1 -- 172.21.15.43:0/2020955705 wait complete. 2024-01-16T15:02:28.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:02:28.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.554+0000 7f17da200640 1 Processor -- start 2024-01-16T15:02:28.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.554+0000 7f17da200640 1 -- start start 2024-01-16T15:02:28.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 0x7f17d406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17d40ad720 0x7f17d415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d40b41a0 con 0x7f17d40ad720 2024-01-16T15:02:28.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d415c360 con 0x7f17d406d520 2024-01-16T15:02:28.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.555+0000 7f17da200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d406de40 con 0x7f17d40abff0 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.556+0000 7f17d2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17d40ad720 0x7f17d415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.556+0000 7f17d2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17d40ad720 0x7f17d415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53914/0 (socket says 172.21.15.43:53914) 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.556+0000 7f17d2a00640 1 -- 172.21.15.43:0/2640754634 learned_addr learned my addr 172.21.15.43:0/2640754634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.556+0000 7f17d3e00640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.556+0000 7f17d3400640 1 --2- 172.21.15.43:0/2640754634 >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 0x7f17d406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d2a00640 1 -- 172.21.15.43:0/2640754634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 msgr2=0x7f17d415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d2a00640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d2a00640 1 -- 172.21.15.43:0/2640754634 >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 msgr2=0x7f17d406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d2a00640 1 --2- 172.21.15.43:0/2640754634 >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 0x7f17d406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d2a00640 1 -- 172.21.15.43:0/2640754634 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d415ba30 con 0x7f17d40ad720 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d3e00640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.557+0000 7f17d3400640 1 --2- 172.21.15.43:0/2640754634 >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 0x7f17d406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:28.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.558+0000 7f17d2a00640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d415ecd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f17d406c070 tx=0x7f17bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb81c5d79bdd2ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.559+0000 7f17d2000640 1 -- 172.21.15.43:0/2640754634 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031870 con 0x7f17d40ad720 2024-01-16T15:02:28.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.559+0000 7f17d2000640 1 -- 172.21.15.43:0/2640754634 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc0319d0 con 0x7f17d40ad720 2024-01-16T15:02:28.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.559+0000 7f17d2000640 1 -- 172.21.15.43:0/2640754634 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031d60 con 0x7f17d40ad720 2024-01-16T15:02:28.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.559+0000 7f17da200640 1 -- 172.21.15.43:0/2640754634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 msgr2=0x7f17d415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.559+0000 7f17da200640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d415ecd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f17d406c070 tx=0x7f17bc02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 -- 172.21.15.43:0/2640754634 shutdown_connections 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 --2- 172.21.15.43:0/2640754634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d415ecd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 --2- 172.21.15.43:0/2640754634 >> v2:172.21.15.69:3300/0 conn(0x7f17d406d520 0x7f17d406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 -- 172.21.15.43:0/2640754634 >> 172.21.15.43:0/2640754634 conn(0x7f17d40b0d80 msgr2=0x7f17d40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 -- 172.21.15.43:0/2640754634 shutdown_connections 2024-01-16T15:02:28.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.560+0000 7f17da200640 1 -- 172.21.15.43:0/2640754634 wait complete. 2024-01-16T15:02:28.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.561+0000 7f17da200640 1 Processor -- start 2024-01-16T15:02:28.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.562+0000 7f17da200640 1 -- start start 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17d406d520 0x7f17d414b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40afc90 con 0x7f17d40ad720 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40995e0 con 0x7f17d406d520 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17da200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40998c0 con 0x7f17d40abff0 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53916/0 (socket says 172.21.15.43:53916) 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 -- 172.21.15.43:0/3806113683 learned_addr learned my addr 172.21.15.43:0/3806113683 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d2a00640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3400640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17d406d520 0x7f17d414b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 -- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 msgr2=0x7f17d415ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.563+0000 7f17d3e00640 1 -- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17d406d520 msgr2=0x7f17d414b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17d3e00640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17d406d520 0x7f17d414b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17d3e00640 1 -- 172.21.15.43:0/3806113683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0094d0 con 0x7f17d40ad720 2024-01-16T15:02:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17d2a00640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17d3e00640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f17cc007e70 tx=0x7f17cc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc015600 con 0x7f17d40ad720 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc00f040 con 0x7f17d40ad720 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0159a0 con 0x7f17d40ad720 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4099ba0 con 0x7f17d40ad720 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.564+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d409a160 con 0x7f17d40ad720 2024-01-16T15:02:28.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.565+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d409a350 con 0x7f17d40ad720 2024-01-16T15:02:28.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.566+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f17cc01e3b0 con 0x7f17d40ad720 2024-01-16T15:02:28.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.568+0000 7f17cb400640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 0x7f17ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.568+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17cc08a220 con 0x7f17d40ad720 2024-01-16T15:02:28.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.568+0000 7f17d3400640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 0x7f17ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.569+0000 7f17d3400640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 0x7f17ac06d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.572+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17cc054ed0 con 0x7f17d40ad720 2024-01-16T15:02:28.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.768+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f17d4074770 con 0x7f17d40ad720 2024-01-16T15:02:28.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.769+0000 7f17cb400640 1 -- 172.21.15.43:0/3806113683 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+65 (secure 0 0 0) 0x7f17cc0584f0 con 0x7f17d40ad720 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:28.749 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.771+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 msgr2=0x7f17ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.771+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 0x7f17ac06d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.771+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 msgr2=0x7f17d409a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.771+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f17cc007e70 tx=0x7f17cc00d4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 shutdown_connections 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17ac06b500 0x7f17ac06d9c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d40ad720 0x7f17d409a680 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d40abff0 0x7f17d415ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 --2- 172.21.15.43:0/3806113683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17d406d520 0x7f17d414b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 >> 172.21.15.43:0/3806113683 conn(0x7f17d40b0d80 msgr2=0x7f17d40cb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 shutdown_connections 2024-01-16T15:02:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:28.772+0000 7f17da200640 1 -- 172.21.15.43:0/3806113683 wait complete. 2024-01-16T15:02:28.770 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:02:28.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:02:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.015+0000 7f6134800640 1 Processor -- start 2024-01-16T15:02:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.016+0000 7f6134800640 1 -- start start 2024-01-16T15:02:28.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.017+0000 7f6134800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f612c0acd70 0x7f612c06dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.017+0000 7f6134800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f612c070490 0x7f612c070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:28.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.017+0000 7f6134800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c06d7a0 con 0x7f612c0ae490 2024-01-16T15:02:28.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.017+0000 7f6134800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c15a5b0 con 0x7f612c0acd70 2024-01-16T15:02:28.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.017+0000 7f6134800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c06ce70 con 0x7f612c070490 2024-01-16T15:02:28.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f612c0acd70 0x7f612c06dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f612c0acd70 0x7f612c06dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53932/0 (socket says 172.21.15.43:53932) 2024-01-16T15:02:28.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f612c0ae490 0x7f612c0ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58746/0 (socket says 172.21.15.43:58746) 2024-01-16T15:02:28.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 -- 172.21.15.43:0/2223001759 learned_addr learned my addr 172.21.15.43:0/2223001759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6132800640 1 --2- 172.21.15.43:0/2223001759 >> v2:172.21.15.69:3300/0 conn(0x7f612c070490 0x7f612c070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 -- 172.21.15.43:0/2223001759 >> v2:172.21.15.69:3300/0 conn(0x7f612c070490 msgr2=0x7f612c070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 --2- 172.21.15.43:0/2223001759 >> v2:172.21.15.69:3300/0 conn(0x7f612c070490 0x7f612c070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 -- 172.21.15.43:0/2223001759 >> v1:172.21.15.43:6789/0 conn(0x7f612c0ae490 legacy=0x7f612c0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.018+0000 7f6131e00640 1 -- 172.21.15.43:0/2223001759 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c06d150 con 0x7f612c0acd70 2024-01-16T15:02:28.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6131e00640 1 --2- 172.21.15.43:0/2223001759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0acd70 0x7f612c06dc30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f612c067e70 tx=0x7f611c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d0850a8c5ea3d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6130a00640 1 -- 172.21.15.43:0/2223001759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c030860 con 0x7f612c0acd70 2024-01-16T15:02:29.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6130a00640 1 -- 172.21.15.43:0/2223001759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c0309c0 con 0x7f612c0acd70 2024-01-16T15:02:29.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6130a00640 1 -- 172.21.15.43:0/2223001759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c030d50 con 0x7f612c0acd70 2024-01-16T15:02:29.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6134800640 1 -- 172.21.15.43:0/2223001759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0acd70 msgr2=0x7f612c06dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.020+0000 7f6134800640 1 --2- 172.21.15.43:0/2223001759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0acd70 0x7f612c06dc30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f612c067e70 tx=0x7f611c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 -- 172.21.15.43:0/2223001759 shutdown_connections 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 --2- 172.21.15.43:0/2223001759 >> v2:172.21.15.69:3300/0 conn(0x7f612c070490 0x7f612c070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 --2- 172.21.15.43:0/2223001759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0acd70 0x7f612c06dc30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 -- 172.21.15.43:0/2223001759 >> 172.21.15.43:0/2223001759 conn(0x7f612c0b0ba0 msgr2=0x7f612c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 -- 172.21.15.43:0/2223001759 shutdown_connections 2024-01-16T15:02:29.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.021+0000 7f6134800640 1 -- 172.21.15.43:0/2223001759 wait complete. 2024-01-16T15:02:29.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.023+0000 7f6134800640 1 Processor -- start 2024-01-16T15:02:29.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.023+0000 7f6134800640 1 -- start start 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f612c070490 0x7f612c0cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 0x7f612c0ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c13b790 con 0x7f612c0ae490 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0cd210 con 0x7f612c070490 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6134800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0cd4b0 con 0x7f612c0acd70 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53948/0 (socket says 172.21.15.43:53948) 2024-01-16T15:02:29.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 -- 172.21.15.43:0/4108811011 learned_addr learned my addr 172.21.15.43:0/4108811011 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6131e00640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f612c070490 0x7f612c0cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6131400640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 0x7f612c0ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 -- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 msgr2=0x7f612c0ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 0x7f612c0ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 -- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f612c070490 msgr2=0x7f612c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.024+0000 7f6132800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f612c070490 0x7f612c0cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f6132800640 1 -- 172.21.15.43:0/4108811011 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c0094d0 con 0x7f612c0ae490 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f6131400640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 0x7f612c0ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f6132800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f612400d5d0 tx=0x7f612400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61240125f0 con 0x7f612c0ae490 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124012770 con 0x7f612c0ae490 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c0cd790 con 0x7f612c0ae490 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124012b20 con 0x7f612c0ae490 2024-01-16T15:02:29.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.025+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c0cdd50 con 0x7f612c0ae490 2024-01-16T15:02:29.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.026+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c09d200 con 0x7f612c0ae490 2024-01-16T15:02:29.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.027+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6124012c80 con 0x7f612c0ae490 2024-01-16T15:02:29.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.028+0000 7f612a000640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 0x7f610406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.028+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f612408a1c0 con 0x7f612c0ae490 2024-01-16T15:02:29.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.029+0000 7f6131e00640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 0x7f610406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.029+0000 7f6131e00640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 0x7f610406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f611c036000 tx=0x7f611c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.033+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6124054f90 con 0x7f612c0ae490 2024-01-16T15:02:29.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.231+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f612c04e920 con 0x7f612c0ae490 2024-01-16T15:02:29.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.864+0000 7f612a000640 1 -- 172.21.15.43:0/4108811011 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v44) v1 ==== 138+0+0 (secure 0 0 0) 0x7f61240585b0 con 0x7f612c0ae490 2024-01-16T15:02:29.844 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 msgr2=0x7f610406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 0x7f610406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f611c036000 tx=0x7f611c03b040 comp rx=0 tx=0).stop 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 msgr2=0x7f612c0ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f612400d5d0 tx=0x7f612400da90 comp rx=0 tx=0).stop 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 shutdown_connections 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f610406b500 0x7f610406d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f612c0ae490 0x7f612c0ca970 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f612c0acd70 0x7f612c0ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 --2- 172.21.15.43:0/4108811011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f612c070490 0x7f612c0cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 >> 172.21.15.43:0/4108811011 conn(0x7f612c0b0ba0 msgr2=0x7f612c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 shutdown_connections 2024-01-16T15:02:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:29.868+0000 7f6134800640 1 -- 172.21.15.43:0/4108811011 wait complete. 2024-01-16T15:02:29.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:02:30.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.100+0000 7f2835800640 1 Processor -- start 2024-01-16T15:02:30.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.100+0000 7f2835800640 1 -- start start 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f2835800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f283015aae0 0x7f283006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f2835800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f28300b2f40 0x7f2830158310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f2835800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283006d700 con 0x7f283006d2a0 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f2835800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283015aec0 con 0x7f28300b2f40 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f2835800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2830154440 con 0x7f283015aae0 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.102+0000 7f282f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f28300b2f40 0x7f2830158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f28300b2f40 0x7f2830158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53952/0 (socket says 172.21.15.43:53952) 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 -- 172.21.15.43:0/2128269625 learned_addr learned my addr 172.21.15.43:0/2128269625 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282ea00640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f283015aae0 0x7f283006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 -- 172.21.15.43:0/2128269625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f283015aae0 msgr2=0x7f283006cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f283015aae0 0x7f283006cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 -- 172.21.15.43:0/2128269625 >> v1:172.21.15.69:6789/0 conn(0x7f283006d2a0 legacy=0x7f28300b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.103+0000 7f282f400640 1 -- 172.21.15.43:0/2128269625 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830157b30 con 0x7f28300b2f40 2024-01-16T15:02:30.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.104+0000 7f282f400640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28300b2f40 0x7f2830158310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f282800aa10 tx=0x7f282802ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a308c0ce9b58c76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f282d600640 1 -- 172.21.15.43:0/2128269625 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282802dc10 con 0x7f28300b2f40 2024-01-16T15:02:30.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f282d600640 1 -- 172.21.15.43:0/2128269625 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282802dd70 con 0x7f28300b2f40 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f282d600640 1 -- 172.21.15.43:0/2128269625 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828035660 con 0x7f28300b2f40 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 -- 172.21.15.43:0/2128269625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28300b2f40 msgr2=0x7f2830158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28300b2f40 0x7f2830158310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f282800aa10 tx=0x7f282802ce60 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 -- 172.21.15.43:0/2128269625 shutdown_connections 2024-01-16T15:02:30.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28300b2f40 0x7f2830158310 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 --2- 172.21.15.43:0/2128269625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f283015aae0 0x7f283006cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.105+0000 7f2835800640 1 -- 172.21.15.43:0/2128269625 >> 172.21.15.43:0/2128269625 conn(0x7f28300b0ba0 msgr2=0x7f28300b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.106+0000 7f2835800640 1 -- 172.21.15.43:0/2128269625 shutdown_connections 2024-01-16T15:02:30.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.106+0000 7f2835800640 1 -- 172.21.15.43:0/2128269625 wait complete. 2024-01-16T15:02:30.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.107+0000 7f2835800640 1 Processor -- start 2024-01-16T15:02:30.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.107+0000 7f2835800640 1 -- start start 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f283006d2a0 0x7f2830151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28300b2f40 0x7f28300c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830072510 con 0x7f283015aae0 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301524d0 con 0x7f283006d2a0 2024-01-16T15:02:30.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.108+0000 7f2835800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2830156e00 con 0x7f28300b2f40 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53964/0 (socket says 172.21.15.43:53964) 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 -- 172.21.15.43:0/391691604 learned_addr learned my addr 172.21.15.43:0/391691604 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282e000640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28300b2f40 0x7f28300c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282ea00640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f283006d2a0 0x7f2830151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 -- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28300b2f40 msgr2=0x7f28300c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28300b2f40 0x7f28300c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 -- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f283006d2a0 msgr2=0x7f2830151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f283006d2a0 0x7f2830151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.109+0000 7f282f400640 1 -- 172.21.15.43:0/391691604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282800a670 con 0x7f283015aae0 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f282f400640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f282800a1c0 tx=0x7f28280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282803e630 con 0x7f283015aae0 2024-01-16T15:02:30.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282803e7b0 con 0x7f283015aae0 2024-01-16T15:02:30.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282803eb60 con 0x7f283015aae0 2024-01-16T15:02:30.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28300717b0 con 0x7f283015aae0 2024-01-16T15:02:30.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.110+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301570c0 con 0x7f283015aae0 2024-01-16T15:02:30.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.111+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28301572b0 con 0x7f283015aae0 2024-01-16T15:02:30.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.114+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f282803c040 con 0x7f283015aae0 2024-01-16T15:02:30.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.114+0000 7f2826a00640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 0x7f280806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:30.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.115+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f28280ac0c0 con 0x7f283015aae0 2024-01-16T15:02:30.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.115+0000 7f282ea00640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 0x7f280806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:30.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.116+0000 7f282ea00640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 0x7f280806da10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2820009ad0 tx=0x7f2820008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.119+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2828076d90 con 0x7f283015aae0 2024-01-16T15:02:30.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.311+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f283009d200 con 0x7f283015aae0 2024-01-16T15:02:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.879+0000 7f2826a00640 1 -- 172.21.15.43:0/391691604 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v45) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2828035e60 con 0x7f283015aae0 2024-01-16T15:02:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 msgr2=0x7f280806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 0x7f280806da10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2820009ad0 tx=0x7f2820008040 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 msgr2=0x7f2830158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f282800a1c0 tx=0x7f28280046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 shutdown_connections 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f280806b550 0x7f280806da10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f283015aae0 0x7f2830158db0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28300b2f40 0x7f28300c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 --2- 172.21.15.43:0/391691604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f283006d2a0 0x7f2830151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 >> 172.21.15.43:0/391691604 conn(0x7f28300b0ba0 msgr2=0x7f283015c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 shutdown_connections 2024-01-16T15:02:30.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:30.884+0000 7f2835800640 1 -- 172.21.15.43:0/391691604 wait complete. 2024-01-16T15:02:30.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:02:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.107+0000 7fabbfc00640 1 Processor -- start 2024-01-16T15:02:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.108+0000 7fabbfc00640 1 -- start start 2024-01-16T15:02:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbfc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fabb8160b10 0x7fabb806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbfc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fabb8072280 0x7fabb806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbfc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb806cd50 con 0x7fabb806c900 2024-01-16T15:02:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbfc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80b2320 con 0x7fabb8160b10 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbfc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb806df30 con 0x7fabb8072280 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.109+0000 7fabbd200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fabb8160b10 0x7fabb806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fabb8160b10 0x7fabb806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53974/0 (socket says 172.21.15.43:53974) 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 -- 172.21.15.43:0/3651379913 learned_addr learned my addr 172.21.15.43:0/3651379913 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabafe00640 1 --2- 172.21.15.43:0/3651379913 >> v2:172.21.15.69:3300/0 conn(0x7fabb8072280 0x7fabb806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 -- 172.21.15.43:0/3651379913 >> v2:172.21.15.69:3300/0 conn(0x7fabb8072280 msgr2=0x7fabb806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 --2- 172.21.15.43:0/3651379913 >> v2:172.21.15.69:3300/0 conn(0x7fabb8072280 0x7fabb806c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 -- 172.21.15.43:0/3651379913 >> v1:172.21.15.69:6789/0 conn(0x7fabb806c900 legacy=0x7fabb8160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.110+0000 7fabbd200640 1 -- 172.21.15.43:0/3651379913 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8071780 con 0x7fabb8160b10 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.111+0000 7fabbd200640 1 --2- 172.21.15.43:0/3651379913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb8160b10 0x7fabb806e420 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fabb80d0100 tx=0x7faba802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8e2aff411d018d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.111+0000 7fabb7e00640 1 -- 172.21.15.43:0/3651379913 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80317e0 con 0x7fabb8160b10 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabb7e00640 1 -- 172.21.15.43:0/3651379913 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8031940 con 0x7fabb8160b10 2024-01-16T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabb7e00640 1 -- 172.21.15.43:0/3651379913 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8031cd0 con 0x7fabb8160b10 2024-01-16T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 -- 172.21.15.43:0/3651379913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb8160b10 msgr2=0x7fabb806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 --2- 172.21.15.43:0/3651379913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb8160b10 0x7fabb806e420 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fabb80d0100 tx=0x7faba802b880 comp rx=0 tx=0).stop 2024-01-16T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 -- 172.21.15.43:0/3651379913 shutdown_connections 2024-01-16T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 --2- 172.21.15.43:0/3651379913 >> v2:172.21.15.69:3300/0 conn(0x7fabb8072280 0x7fabb806c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 --2- 172.21.15.43:0/3651379913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb8160b10 0x7fabb806e420 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 -- 172.21.15.43:0/3651379913 >> 172.21.15.43:0/3651379913 conn(0x7fabb80b0ba0 msgr2=0x7fabb80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 -- 172.21.15.43:0/3651379913 shutdown_connections 2024-01-16T15:02:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.112+0000 7fabbfc00640 1 -- 172.21.15.43:0/3651379913 wait complete. 2024-01-16T15:02:31.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.114+0000 7fabbfc00640 1 Processor -- start 2024-01-16T15:02:31.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.114+0000 7fabbfc00640 1 -- start start 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 0x7fabb814d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 0x7fabb809b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb80cced0 con 0x7fabb806c900 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb80cbf80 con 0x7fabb8072280 2024-01-16T15:02:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbfc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb80cc250 con 0x7fabb8160b10 2024-01-16T15:02:31.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53988/0 (socket says 172.21.15.43:53988) 2024-01-16T15:02:31.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.115+0000 7fabbd200640 1 -- 172.21.15.43:0/2855294566 learned_addr learned my addr 172.21.15.43:0/2855294566 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabafe00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 0x7fabb814d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbdc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 0x7fabb809b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 -- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 msgr2=0x7fabb809b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 0x7fabb809b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 -- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 msgr2=0x7fabb814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 0x7fabb814d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 -- 172.21.15.43:0/2855294566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba80094d0 con 0x7fabb806c900 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbdc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 0x7fabb809b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabafe00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 0x7fabb814d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.116+0000 7fabbd200640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faba8000c00 tx=0x7faba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.117+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80329c0 con 0x7fabb806c900 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.117+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8032b40 con 0x7fabb806c900 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.117+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba80453f0 con 0x7fabb806c900 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.117+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb80cc530 con 0x7fabb806c900 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.117+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb80ccaf0 con 0x7fabb806c900 2024-01-16T15:02:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.118+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb80cd010 con 0x7fabb806c900 2024-01-16T15:02:31.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.119+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7faba804e450 con 0x7fabb806c900 2024-01-16T15:02:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.120+0000 7fabb5600640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.121+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5609+0+0 (secure 0 0 0) 0x7faba80ac260 con 0x7fabb806c900 2024-01-16T15:02:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.121+0000 7fabafe00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.121+0000 7fabafe00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 0x7fab9006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faba0006fd0 tx=0x7faba0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.125+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabb80cd010 con 0x7fabb806c900 2024-01-16T15:02:31.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.353+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fabb806cd50 con 0x7fabb806c900 2024-01-16T15:02:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.884+0000 7fabb5600640 1 -- 172.21.15.43:0/2855294566 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7fabb806cd50 con 0x7fabb806c900 2024-01-16T15:02:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 6 and data pool 7 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.890+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 msgr2=0x7fab9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.890+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 0x7fab9006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faba0006fd0 tx=0x7faba0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.890+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 msgr2=0x7fabb80cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.890+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faba8000c00 tx=0x7faba80046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 shutdown_connections 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab9006b500 0x7fab9006d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabb8072280 0x7fabb814d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fabb8160b10 0x7fabb809b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 --2- 172.21.15.43:0/2855294566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabb806c900 0x7fabb80cba40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.891+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 >> 172.21.15.43:0/2855294566 conn(0x7fabb80b0ba0 msgr2=0x7fabb815fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.892+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 shutdown_connections 2024-01-16T15:02:31.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:31.892+0000 7fabbfc00640 1 -- 172.21.15.43:0/2855294566 wait complete. 2024-01-16T15:02:31.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:32.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.116+0000 7fb5bdc00640 1 Processor -- start 2024-01-16T15:02:32.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.116+0000 7fb5bdc00640 1 -- start start 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5bdc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5b8160970 0x7fb5b80b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5bdc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5b806dc30 0x7fb5b80acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5bdc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b8071cd0 con 0x7fb5b80b3140 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5bdc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b80b2120 con 0x7fb5b806dc30 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5bdc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b80b2400 con 0x7fb5b8160970 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5b7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5b806dc30 0x7fb5b80acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5afe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb5b80b3140 0x7fb5b806d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:58754/0 (socket says 172.21.15.43:58754) 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5afe00640 1 -- 172.21.15.43:0/2468516800 learned_addr learned my addr 172.21.15.43:0/2468516800 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5b6a00640 1 --2- 172.21.15.43:0/2468516800 >> v2:172.21.15.69:3300/0 conn(0x7fb5b8160970 0x7fb5b80b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.118+0000 7fb5b7400640 1 -- 172.21.15.43:0/2468516800 >> v2:172.21.15.69:3300/0 conn(0x7fb5b8160970 msgr2=0x7fb5b80b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.119+0000 7fb5b7400640 1 --2- 172.21.15.43:0/2468516800 >> v2:172.21.15.69:3300/0 conn(0x7fb5b8160970 0x7fb5b80b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.119+0000 7fb5b7400640 1 -- 172.21.15.43:0/2468516800 >> v1:172.21.15.43:6789/0 conn(0x7fb5b80b3140 legacy=0x7fb5b806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.119+0000 7fb5b7400640 1 -- 172.21.15.43:0/2468516800 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b80b26e0 con 0x7fb5b806dc30 2024-01-16T15:02:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.119+0000 7fb5b6000640 1 -- 172.21.15.43:0/2468516800 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472795490 0 0) 0x7fb5b8071cd0 con 0x7fb5b80b3140 2024-01-16T15:02:32.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.120+0000 7fb5b7400640 1 --2- 172.21.15.43:0/2468516800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b806dc30 0x7fb5b80acbe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80b5210 tx=0x7fb5b002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6790f02e9aceeb6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.120+0000 7fb5b6000640 1 -- 172.21.15.43:0/2468516800 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b002cbe0 con 0x7fb5b806dc30 2024-01-16T15:02:32.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.120+0000 7fb5b6000640 1 -- 172.21.15.43:0/2468516800 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b002cd40 con 0x7fb5b806dc30 2024-01-16T15:02:32.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.120+0000 7fb5b6000640 1 -- 172.21.15.43:0/2468516800 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0034660 con 0x7fb5b806dc30 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 -- 172.21.15.43:0/2468516800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b806dc30 msgr2=0x7fb5b80acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/2468516800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b806dc30 0x7fb5b80acbe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80b5210 tx=0x7fb5b002be30 comp rx=0 tx=0).stop 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 -- 172.21.15.43:0/2468516800 shutdown_connections 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/2468516800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b806dc30 0x7fb5b80acbe0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/2468516800 >> v2:172.21.15.69:3300/0 conn(0x7fb5b8160970 0x7fb5b80b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 -- 172.21.15.43:0/2468516800 >> 172.21.15.43:0/2468516800 conn(0x7fb5b80b0d80 msgr2=0x7fb5b80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 -- 172.21.15.43:0/2468516800 shutdown_connections 2024-01-16T15:02:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.121+0000 7fb5bdc00640 1 -- 172.21.15.43:0/2468516800 wait complete. 2024-01-16T15:02:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.123+0000 7fb5bdc00640 1 Processor -- start 2024-01-16T15:02:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.123+0000 7fb5bdc00640 1 -- start start 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5b806dc30 0x7fb5b8150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5b8160970 0x7fb5b806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b81f6f80 con 0x7fb5b80b3140 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b81f6300 con 0x7fb5b806dc30 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5bdc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b81f65e0 con 0x7fb5b8160970 2024-01-16T15:02:32.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54002/0 (socket says 172.21.15.43:54002) 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 -- 172.21.15.43:0/3738847269 learned_addr learned my addr 172.21.15.43:0/3738847269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 -- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5b8160970 msgr2=0x7fb5b806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5b8160970 0x7fb5b806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5b6a00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5b806dc30 0x7fb5b8150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.124+0000 7fb5afe00640 1 -- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5b806dc30 msgr2=0x7fb5b8150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5afe00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5b806dc30 0x7fb5b8150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5afe00640 1 -- 172.21.15.43:0/3738847269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0009a70 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5afe00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c4d0 tx=0x7fb5a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00074e0 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a0007660 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0004a20 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b81f68c0 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.125+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b81505c0 con 0x7fb5b80b3140 2024-01-16T15:02:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.126+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b8074770 con 0x7fb5b80b3140 2024-01-16T15:02:32.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.127+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb5a0004b80 con 0x7fb5b80b3140 2024-01-16T15:02:32.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.129+0000 7fb5aea00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 0x7fb59006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.129+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5a00891a0 con 0x7fb5b80b3140 2024-01-16T15:02:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.129+0000 7fb5b6a00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 0x7fb59006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.130+0000 7fb5b6a00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 0x7fb59006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8006fd0 tx=0x7fb5a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.133+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5a0053ca0 con 0x7fb5b80b3140 2024-01-16T15:02:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.321+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5b806e290 con 0x7fb5b80b3140 2024-01-16T15:02:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.324+0000 7fb5aea00640 1 -- 172.21.15.43:0/3738847269 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16478 (secure 0 0 0) 0x7fb5a00572c0 con 0x7fb5b80b3140 2024-01-16T15:02:32.304 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:32.304 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":46,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:31.355332+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:29.285810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:30.367713+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 msgr2=0x7fb59006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 0x7fb59006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8006fd0 tx=0x7fb5a8008040 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 msgr2=0x7fb5b809c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c4d0 tx=0x7fb5a000c990 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 shutdown_connections 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb59006b500 0x7fb59006d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5b8160970 0x7fb5b806c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5b80b3140 0x7fb5b809c910 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 --2- 172.21.15.43:0/3738847269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5b806dc30 0x7fb5b8150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 >> 172.21.15.43:0/3738847269 conn(0x7fb5b80b0d80 msgr2=0x7fb5b80ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 shutdown_connections 2024-01-16T15:02:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.327+0000 7fb5bdc00640 1 -- 172.21.15.43:0/3738847269 wait complete. 2024-01-16T15:02:32.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:32.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.557+0000 7f7990a00640 1 Processor -- start 2024-01-16T15:02:32.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.557+0000 7f7990a00640 1 -- start start 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f7990a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0d14a0 0x7f798c0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f7990a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f798c0b0640 0x7f798c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f7990a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0cf0c0 con 0x7f798c15d5b0 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f7990a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798c0b01b0 con 0x7f798c0b0640 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f7990a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c15d990 con 0x7f798c0d14a0 2024-01-16T15:02:32.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f798be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f798c0b0640 0x7f798c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.559+0000 7f798be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f798c0b0640 0x7f798c0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54010/0 (socket says 172.21.15.43:54010) 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f798be00640 1 -- 172.21.15.43:0/3906644287 learned_addr learned my addr 172.21.15.43:0/3906644287 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f7983e00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0d14a0 0x7f798c0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f798be00640 1 -- 172.21.15.43:0/3906644287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0d14a0 msgr2=0x7f798c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f798be00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0d14a0 0x7f798c0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f798be00640 1 -- 172.21.15.43:0/3906644287 >> v1:172.21.15.69:6789/0 conn(0x7f798c15d5b0 legacy=0x7f798c0d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.560+0000 7f798be00640 1 -- 172.21.15.43:0/3906644287 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0d1e40 con 0x7f798c0b0640 2024-01-16T15:02:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.561+0000 7f798be00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0b0640 0x7f798c0b0aa0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7984009e50 tx=0x7f798402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111cf424434d9342 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.561+0000 7f798aa00640 1 -- 172.21.15.43:0/3906644287 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798402cb00 con 0x7f798c0b0640 2024-01-16T15:02:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.561+0000 7f798aa00640 1 -- 172.21.15.43:0/3906644287 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798402cc60 con 0x7f798c0b0640 2024-01-16T15:02:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.561+0000 7f798aa00640 1 -- 172.21.15.43:0/3906644287 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79840343f0 con 0x7f798c0b0640 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 -- 172.21.15.43:0/3906644287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0b0640 msgr2=0x7f798c0b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0b0640 0x7f798c0b0aa0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7984009e50 tx=0x7f798402bf60 comp rx=0 tx=0).stop 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 -- 172.21.15.43:0/3906644287 shutdown_connections 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0b0640 0x7f798c0b0aa0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 --2- 172.21.15.43:0/3906644287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0d14a0 0x7f798c0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 -- 172.21.15.43:0/3906644287 >> 172.21.15.43:0/3906644287 conn(0x7f798c09c3b0 msgr2=0x7f798c0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 -- 172.21.15.43:0/3906644287 shutdown_connections 2024-01-16T15:02:32.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.562+0000 7f7990a00640 1 -- 172.21.15.43:0/3906644287 wait complete. 2024-01-16T15:02:32.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.564+0000 7f7990a00640 1 Processor -- start 2024-01-16T15:02:32.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.564+0000 7f7990a00640 1 -- start start 2024-01-16T15:02:32.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0b0640 0x7f798c1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 0x7f798c1f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0ca140 con 0x7f798c0d14a0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c9500 con 0x7f798c15d5b0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7990a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0ca320 con 0x7f798c0b0640 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7983e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7983e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54024/0 (socket says 172.21.15.43:54024) 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.565+0000 7f7983e00640 1 -- 172.21.15.43:0/4108576227 learned_addr learned my addr 172.21.15.43:0/4108576227 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f798b400640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0b0640 0x7f798c1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f798be00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 0x7f798c1f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 -- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0b0640 msgr2=0x7f798c1504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0b0640 0x7f798c1504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 -- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 msgr2=0x7f798c1f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 0x7f798c1f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 -- 172.21.15.43:0/4108576227 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7984009b30 con 0x7f798c0d14a0 2024-01-16T15:02:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f798be00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 0x7f798c1f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983e00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f797400d5d0 tx=0x7f797400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.566+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740125f0 con 0x7f798c0d14a0 2024-01-16T15:02:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.567+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974012770 con 0x7f798c0d14a0 2024-01-16T15:02:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.567+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c0ca600 con 0x7f798c0d14a0 2024-01-16T15:02:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.567+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974012b20 con 0x7f798c0d14a0 2024-01-16T15:02:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.567+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c0cce50 con 0x7f798c0d14a0 2024-01-16T15:02:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.568+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c0cab20 con 0x7f798c0d14a0 2024-01-16T15:02:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.569+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7974012c80 con 0x7f798c0d14a0 2024-01-16T15:02:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.570+0000 7f7983400640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 0x7f796406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.570+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f797408a520 con 0x7f798c0d14a0 2024-01-16T15:02:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.570+0000 7f798b400640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 0x7f796406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.571+0000 7f798b400640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 0x7f796406d9f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f797c000f60 tx=0x7f797c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.575+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f798c0cab20 con 0x7f798c0d14a0 2024-01-16T15:02:32.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.776+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f798c0cab20 con 0x7f798c0d14a0 2024-01-16T15:02:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.778+0000 7f7983400640 1 -- 172.21.15.43:0/4108576227 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16478 (secure 0 0 0) 0x7f798c0cab20 con 0x7f798c0d14a0 2024-01-16T15:02:32.758 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:32.758 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":46,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:31.355332+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:29.285810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:30.367713+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 msgr2=0x7f796406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 0x7f796406d9f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f797c000f60 tx=0x7f797c009040 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 msgr2=0x7f798c1f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f797400d5d0 tx=0x7f797400da90 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 shutdown_connections 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f796406b530 0x7f796406d9f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f798c15d5b0 0x7f798c1f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f798c0d14a0 0x7f798c1f2130 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 --2- 172.21.15.43:0/4108576227 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f798c0b0640 0x7f798c1504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 >> 172.21.15.43:0/4108576227 conn(0x7f798c09c3b0 msgr2=0x7f798c06b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 shutdown_connections 2024-01-16T15:02:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:32.781+0000 7f7990a00640 1 -- 172.21.15.43:0/4108576227 wait complete. 2024-01-16T15:02:32.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:02:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.012+0000 7fc000c00640 1 Processor -- start 2024-01-16T15:02:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.012+0000 7fc000c00640 1 -- start start 2024-01-16T15:02:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fc000c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbffc0acd70 0x7fbffc06dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fc000c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbffc070490 0x7fbffc070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fc000c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc06d7a0 con 0x7fbffc0ae490 2024-01-16T15:02:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fc000c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc15a5b0 con 0x7fbffc070490 2024-01-16T15:02:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fc000c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc06ce70 con 0x7fbffc0acd70 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fbffa800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbffc070490 0x7fbffc070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fbffa800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbffc070490 0x7fbffc070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36736/0 (socket says 172.21.15.43:36736) 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fbffa800640 1 -- 172.21.15.43:0/1362880722 learned_addr learned my addr 172.21.15.43:0/1362880722 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.014+0000 7fbff9e00640 1 --2- 172.21.15.43:0/1362880722 >> v2:172.21.15.69:3300/0 conn(0x7fbffc0acd70 0x7fbffc06dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.015+0000 7fbffa800640 1 -- 172.21.15.43:0/1362880722 >> v2:172.21.15.69:3300/0 conn(0x7fbffc0acd70 msgr2=0x7fbffc06dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.015+0000 7fbffa800640 1 --2- 172.21.15.43:0/1362880722 >> v2:172.21.15.69:3300/0 conn(0x7fbffc0acd70 0x7fbffc06dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.015+0000 7fbffa800640 1 -- 172.21.15.43:0/1362880722 >> v1:172.21.15.69:6789/0 conn(0x7fbffc0ae490 legacy=0x7fbffc0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.015+0000 7fbffa800640 1 -- 172.21.15.43:0/1362880722 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc06d150 con 0x7fbffc070490 2024-01-16T15:02:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.016+0000 7fbffa800640 1 --2- 172.21.15.43:0/1362880722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc070490 0x7fbffc070900 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0700f0 tx=0x7fbff402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858092bde243b7c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.016+0000 7fbff8a00640 1 -- 172.21.15.43:0/1362880722 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff402cb20 con 0x7fbffc070490 2024-01-16T15:02:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.016+0000 7fbff8a00640 1 -- 172.21.15.43:0/1362880722 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff402cc80 con 0x7fbffc070490 2024-01-16T15:02:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.016+0000 7fbff8a00640 1 -- 172.21.15.43:0/1362880722 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40343f0 con 0x7fbffc070490 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 -- 172.21.15.43:0/1362880722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc070490 msgr2=0x7fbffc070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 --2- 172.21.15.43:0/1362880722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc070490 0x7fbffc070900 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0700f0 tx=0x7fbff402bf80 comp rx=0 tx=0).stop 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 -- 172.21.15.43:0/1362880722 shutdown_connections 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 --2- 172.21.15.43:0/1362880722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc070490 0x7fbffc070900 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 --2- 172.21.15.43:0/1362880722 >> v2:172.21.15.69:3300/0 conn(0x7fbffc0acd70 0x7fbffc06dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 -- 172.21.15.43:0/1362880722 >> 172.21.15.43:0/1362880722 conn(0x7fbffc0b0ba0 msgr2=0x7fbffc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 -- 172.21.15.43:0/1362880722 shutdown_connections 2024-01-16T15:02:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.017+0000 7fc000c00640 1 -- 172.21.15.43:0/1362880722 wait complete. 2024-01-16T15:02:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.019+0000 7fc000c00640 1 Processor -- start 2024-01-16T15:02:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.019+0000 7fc000c00640 1 -- start start 2024-01-16T15:02:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbffc070490 0x7fbffc140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 0x7fbffc15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc141010 con 0x7fbffc0acd70 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0ce340 con 0x7fbffc0ae490 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fc000c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc099f10 con 0x7fbffc070490 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fbff9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fbff9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36740/0 (socket says 172.21.15.43:36740) 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.020+0000 7fbff9400640 1 -- 172.21.15.43:0/3120018738 learned_addr learned my addr 172.21.15.43:0/3120018738 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbffa800640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 0x7fbffc15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9e00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbffc070490 0x7fbffc140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 -- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbffc070490 msgr2=0x7fbffc140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbffc070490 0x7fbffc140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 -- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 msgr2=0x7fbffc15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 0x7fbffc15c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 -- 172.21.15.43:0/3120018738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4009b30 con 0x7fbffc0acd70 2024-01-16T15:02:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbffa800640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 0x7fbffc15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff9400640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c480 tx=0x7fbfe400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40052d0 con 0x7fbffc0acd70 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4005430 con 0x7fbffc0acd70 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4012870 con 0x7fbffc0acd70 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.021+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc0cdd80 con 0x7fbffc0acd70 2024-01-16T15:02:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.022+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc099ba0 con 0x7fbffc0acd70 2024-01-16T15:02:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.023+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc099d30 con 0x7fbffc0acd70 2024-01-16T15:02:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.024+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fbfe40129d0 con 0x7fbffc0acd70 2024-01-16T15:02:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.025+0000 7fbff2000640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 0x7fbfd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.025+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfe4089bc0 con 0x7fbffc0acd70 2024-01-16T15:02:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.025+0000 7fbff9e00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 0x7fbfd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.026+0000 7fbff9e00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 0x7fbfd406d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006fd0 tx=0x7fbfec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.030+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbfe40546c0 con 0x7fbffc0acd70 2024-01-16T15:02:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.299+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbffc099a00 con 0x7fbffc0acd70 2024-01-16T15:02:33.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.927+0000 7fbff2000640 1 -- 172.21.15.43:0/3120018738 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbffc099a00 con 0x7fbffc0acd70 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 msgr2=0x7fbfd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 0x7fbfd406d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006fd0 tx=0x7fbfec008040 comp rx=0 tx=0).stop 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 msgr2=0x7fbffc0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c480 tx=0x7fbfe400c940 comp rx=0 tx=0).stop 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 shutdown_connections 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbfd406b500 0x7fbfd406d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbffc0ae490 0x7fbffc15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbffc0acd70 0x7fbffc0cd300 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 --2- 172.21.15.43:0/3120018738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbffc070490 0x7fbffc140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 >> 172.21.15.43:0/3120018738 conn(0x7fbffc0b0ba0 msgr2=0x7fbffc0b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 shutdown_connections 2024-01-16T15:02:33.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:33.932+0000 7fc000c00640 1 -- 172.21.15.43:0/3120018738 wait complete. 2024-01-16T15:02:33.937 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:02:33.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:02:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.185+0000 7f98bd200640 1 Processor -- start 2024-01-16T15:02:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.185+0000 7f98bd200640 1 -- start start 2024-01-16T15:02:34.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98bd200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98b80cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98bd200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f98b80cdce0 0x7f98b80b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98bd200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b806c810 con 0x7f98b815d350 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98bd200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b80cf6c0 con 0x7f98b80cdce0 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98bd200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80ce170 con 0x7f98b806c3b0 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98b7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f98b815d350 0x7f98b80cefb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44404/0 (socket says 172.21.15.43:44404) 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.187+0000 7f98b7400640 1 -- 172.21.15.43:0/3077116452 learned_addr learned my addr 172.21.15.43:0/3077116452 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b7e00640 1 --2- 172.21.15.43:0/3077116452 >> v2:172.21.15.69:3300/0 conn(0x7f98b80cdce0 0x7f98b80b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6a00640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98b80cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490353546 0 0) 0x7f98b806c810 con 0x7f98b815d350 2024-01-16T15:02:34.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9898003520 con 0x7f98b815d350 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536517326 0 0) 0x7f9898003520 con 0x7f98b815d350 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 msgr2=0x7f98b80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98b80cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 >> v2:172.21.15.69:3300/0 conn(0x7f98b80cdce0 msgr2=0x7f98b80b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 --2- 172.21.15.43:0/3077116452 >> v2:172.21.15.69:3300/0 conn(0x7f98b80cdce0 0x7f98b80b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b815d730 con 0x7f98b815d350 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.188+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f98a8001670 con 0x7f98b815d350 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 >> v1:172.21.15.43:6789/0 conn(0x7f98b815d350 legacy=0x7f98b80cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98980039e0 0x7f9898003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9898004d00 0x7f9898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80cf6c0 con 0x7f98980039e0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80ce170 con 0x7f9898004d00 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989800b980 con 0x7f98b806c3b0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98980039e0 0x7f9898003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7e00640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9898004d00 0x7f9898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6a00640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 -- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 msgr2=0x7f98980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 -- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9898004d00 msgr2=0x7f9898005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9898004d00 0x7f9898005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b7400640 1 -- 172.21.15.43:0/3077116452 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898003520 con 0x7f98980039e0 2024-01-16T15:02:34.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.189+0000 7f98b6a00640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98b806c3b0 0x7f98980047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98b7400640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98980039e0 0x7f9898003e00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f98a800c0e0 tx=0x7f98980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcc8587f1780c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8005ad0 con 0x7f98980039e0 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a8005c50 con 0x7f98980039e0 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98b6000640 1 -- 172.21.15.43:0/3077116452 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a802c810 con 0x7f98980039e0 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98bd200640 1 -- 172.21.15.43:0/3077116452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98980039e0 msgr2=0x7f9898003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98bd200640 1 --2- 172.21.15.43:0/3077116452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98980039e0 0x7f9898003e00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f98a800c0e0 tx=0x7f98980057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98b7e00640 1 -- 172.21.15.43:0/3077116452 reap_dead start 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98bd200640 1 -- 172.21.15.43:0/3077116452 shutdown_connections 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.191+0000 7f98bd200640 1 -- 172.21.15.43:0/3077116452 >> 172.21.15.43:0/3077116452 conn(0x7f98b80b2a70 msgr2=0x7f98b80b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.192+0000 7f98bd200640 1 -- 172.21.15.43:0/3077116452 shutdown_connections 2024-01-16T15:02:34.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.192+0000 7f98bd200640 1 -- 172.21.15.43:0/3077116452 wait complete. 2024-01-16T15:02:34.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.193+0000 7f98bd200640 1 Processor -- start 2024-01-16T15:02:34.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 -- start start 2024-01-16T15:02:34.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 0x7f98b815a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98b815da90 0x7f98b815b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80b4ab0 con 0x7f98b80b5ae0 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80c8190 con 0x7f98b815da90 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.194+0000 7f98bd200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80c8470 con 0x7f98980039e0 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36772/0 (socket says 172.21.15.43:36772) 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 -- 172.21.15.43:0/4294759947 learned_addr learned my addr 172.21.15.43:0/4294759947 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b7e00640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98b815da90 0x7f98b815b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b7400640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 0x7f98b815a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 -- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 msgr2=0x7f98b815a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 0x7f98b815a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 -- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98b815da90 msgr2=0x7f98b815b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98b815da90 0x7f98b815b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b6a00640 1 -- 172.21.15.43:0/4294759947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b80c8750 con 0x7f98b80b5ae0 2024-01-16T15:02:34.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.195+0000 7f98b7400640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 0x7f98b815a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:34.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98b6a00640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f98a00108b0 tx=0x7f98a0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a000eb60 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a000ecc0 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a001c400 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b80c8a30 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.196+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b815df20 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.197+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b804e840 con 0x7f98b80b5ae0 2024-01-16T15:02:34.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.199+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f98a001c560 con 0x7f98b80b5ae0 2024-01-16T15:02:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.199+0000 7f98af400640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 0x7f989006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.200+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a008a560 con 0x7f98b80b5ae0 2024-01-16T15:02:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.200+0000 7f98b7400640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 0x7f989006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.201+0000 7f98b7400640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 0x7f989006d9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98a8034000 tx=0x7f98a8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.204+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98a00551b0 con 0x7f98b80b5ae0 2024-01-16T15:02:34.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.437+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f98b80b2050 con 0x7f98b80b5ae0 2024-01-16T15:02:34.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.937+0000 7f98af400640 1 -- 172.21.15.43:0/4294759947 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f98a00587d0 con 0x7f98b80b5ae0 2024-01-16T15:02:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 msgr2=0x7f989006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 0x7f989006d9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98a8034000 tx=0x7f98a8009030 comp rx=0 tx=0).stop 2024-01-16T15:02:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 msgr2=0x7f98b80b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f98a00108b0 tx=0x7f98a0010d70 comp rx=0 tx=0).stop 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 shutdown_connections 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f989006b530 0x7f989006d9f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98b815da90 0x7f98b815b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98b80b5ae0 0x7f98b80b4570 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 --2- 172.21.15.43:0/4294759947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98980039e0 0x7f98b815a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 >> 172.21.15.43:0/4294759947 conn(0x7f98b80b2a70 msgr2=0x7f98b806b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 shutdown_connections 2024-01-16T15:02:34.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:34.942+0000 7f98bd200640 1 -- 172.21.15.43:0/4294759947 wait complete. 2024-01-16T15:02:34.944 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:02:34.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.185+0000 7f12ee200640 1 Processor -- start 2024-01-16T15:02:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.185+0000 7f12ee200640 1 -- start start 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12ee200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12e806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12ee200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e815c360 con 0x7f12e80ad720 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12ee200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e815faf0 con 0x7f12e80abff0 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12ee200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e80b41a0 con 0x7f12e806d520 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12de800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f12e80ad720 0x7f12e815ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44412/0 (socket says 172.21.15.43:44412) 2024-01-16T15:02:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.187+0000 7f12de800640 1 -- 172.21.15.43:0/3049342766 learned_addr learned my addr 172.21.15.43:0/3049342766 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810630577 0 0) 0x7f12e815c360 con 0x7f12e80ad720 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c8003520 con 0x7f12e80ad720 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e7400640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12e806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730620616 0 0) 0x7f12c8003520 con 0x7f12e80ad720 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 msgr2=0x7f12e806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12e806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 >> v1:172.21.15.69:6789/0 conn(0x7f12e80abff0 legacy=0x7f12e815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e806de40 con 0x7f12e80ad720 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f12d0001670 con 0x7f12e80ad720 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 >> v1:172.21.15.43:6789/0 conn(0x7f12e80ad720 legacy=0x7f12e815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12c80039a0 0x7f12c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80047c0 0x7f12c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c800ba80 con 0x7f12c80039a0 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c800bbf0 con 0x7f12c80047c0 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c800bdb0 con 0x7f12e806d520 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.188+0000 7f12e7400640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12c80039a0 0x7f12c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12de800640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80047c0 0x7f12c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7e00640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7400640 1 -- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 msgr2=0x7f12c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7400640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e806d520 0x7f12c80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7400640 1 -- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80047c0 msgr2=0x7f12c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7400640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80047c0 0x7f12c8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.189+0000 7f12e7400640 1 -- 172.21.15.43:0/3049342766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8003520 con 0x7f12c80039a0 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.190+0000 7f12e7400640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12c80039a0 0x7f12c8003dc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f12d8007360 tx=0x7f12c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309873a88f415d5a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.190+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d802ccd0 con 0x7f12c80039a0 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.190+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d802ce30 con 0x7f12c80039a0 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.190+0000 7f12e6a00640 1 -- 172.21.15.43:0/3049342766 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8034660 con 0x7f12c80039a0 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 -- 172.21.15.43:0/3049342766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12c80039a0 msgr2=0x7f12c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 --2- 172.21.15.43:0/3049342766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12c80039a0 0x7f12c8003dc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f12d8007360 tx=0x7f12c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12e7e00640 1 -- 172.21.15.43:0/3049342766 reap_dead start 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 -- 172.21.15.43:0/3049342766 shutdown_connections 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 -- 172.21.15.43:0/3049342766 >> 172.21.15.43:0/3049342766 conn(0x7f12e80b0d80 msgr2=0x7f12e80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 -- 172.21.15.43:0/3049342766 shutdown_connections 2024-01-16T15:02:35.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.191+0000 7f12ee200640 1 -- 172.21.15.43:0/3049342766 wait complete. 2024-01-16T15:02:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.193+0000 7f12ee200640 1 Processor -- start 2024-01-16T15:02:35.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.193+0000 7f12ee200640 1 -- start start 2024-01-16T15:02:35.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80039a0 0x7f12e81492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e8158f80 0x7f12e8152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e815a5e0 con 0x7f12e8159400 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e81599d0 con 0x7f12c80039a0 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12ee200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e8159cb0 con 0x7f12e8158f80 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36788/0 (socket says 172.21.15.43:36788) 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.194+0000 7f12e7e00640 1 -- 172.21.15.43:0/3501408946 learned_addr learned my addr 172.21.15.43:0/3501408946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7400640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80039a0 0x7f12e81492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12de800640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e8158f80 0x7f12e8152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 -- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e8158f80 msgr2=0x7f12e8152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e8158f80 0x7f12e8152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 -- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80039a0 msgr2=0x7f12e81492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80039a0 0x7f12e81492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 -- 172.21.15.43:0/3501408946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8159f90 con 0x7f12e8159400 2024-01-16T15:02:35.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12e7e00640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f12e806fda0 tx=0x7f12e000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000b070 con 0x7f12e8159400 2024-01-16T15:02:35.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e00054e0 con 0x7f12e8159400 2024-01-16T15:02:35.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0003810 con 0x7f12e8159400 2024-01-16T15:02:35.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e815a270 con 0x7f12e8159400 2024-01-16T15:02:35.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.195+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e815aa70 con 0x7f12e8159400 2024-01-16T15:02:35.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.197+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e8074770 con 0x7f12e8159400 2024-01-16T15:02:35.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.198+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f12e0003970 con 0x7f12e8159400 2024-01-16T15:02:35.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.199+0000 7f12dfe00640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 0x7f12c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.199+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12e0089700 con 0x7f12e8159400 2024-01-16T15:02:35.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.199+0000 7f12e7400640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 0x7f12c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.200+0000 7f12e7400640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 0x7f12c006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12d8036000 tx=0x7f12d8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.203+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12e0054100 con 0x7f12e8159400 2024-01-16T15:02:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.436+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e815f840 con 0x7f12e8159400 2024-01-16T15:02:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.437+0000 7f12dfe00640 1 -- 172.21.15.43:0/3501408946 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f12e0057720 con 0x7f12e8159400 2024-01-16T15:02:35.417 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:35.417 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2588572053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2588572053},{"type":"v1","addr":"172.21.15.43:6837","nonce":2588572053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/791773791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":791773791},{"type":"v1","addr":"172.21.15.69:6833","nonce":791773791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3012792492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3012792492},{"type":"v1","addr":"172.21.15.69:6835","nonce":3012792492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:31.355647+0000","modified":"2024-01-16T15:02:34.439473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5766":{"gid":5766,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.43:6836/3947724910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3947724910},{"type":"v1","addr":"172.21.15.43:6836","nonce":3947724910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:02:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 26 2024-01-16T15:02:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 msgr2=0x7f12c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 0x7f12c006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12d8036000 tx=0x7f12d8030000 comp rx=0 tx=0).stop 2024-01-16T15:02:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 msgr2=0x7f12e8153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f12e806fda0 tx=0x7f12e000d890 comp rx=0 tx=0).stop 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 shutdown_connections 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f12c006b500 0x7f12c006d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12e8159400 0x7f12e8153120 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12e8158f80 0x7f12e8152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 --2- 172.21.15.43:0/3501408946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12c80039a0 0x7f12e81492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 >> 172.21.15.43:0/3501408946 conn(0x7f12e80b0d80 msgr2=0x7f12e815ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.440+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 shutdown_connections 2024-01-16T15:02:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.441+0000 7f12ee200640 1 -- 172.21.15.43:0/3501408946 wait complete. 2024-01-16T15:02:35.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.684+0000 7fa3cb200640 1 Processor -- start 2024-01-16T15:02:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.684+0000 7fa3cb200640 1 -- start start 2024-01-16T15:02:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3cb200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3c40cf050 0x7fa3c40cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3cb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c40cd7b0 0x7fa3c40cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3cb200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c415fa70 con 0x7fa3c413dfd0 2024-01-16T15:02:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3cb200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c40cfa40 con 0x7fa3c40cf050 2024-01-16T15:02:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3cb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c415c580 con 0x7fa3c40cd7b0 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.685+0000 7fa3c9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3c40cf050 0x7fa3c40cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3c40cf050 0x7fa3c40cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36800/0 (socket says 172.21.15.43:36800) 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 -- 172.21.15.43:0/2919055274 learned_addr learned my addr 172.21.15.43:0/2919055274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3ca800640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c40cd7b0 0x7fa3c40cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 -- 172.21.15.43:0/2919055274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c40cd7b0 msgr2=0x7fa3c40cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c40cd7b0 0x7fa3c40cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 -- 172.21.15.43:0/2919055274 >> v1:172.21.15.43:6789/0 conn(0x7fa3c413dfd0 legacy=0x7fa3c415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c9400640 1 -- 172.21.15.43:0/2919055274 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4140900 con 0x7fa3c40cf050 2024-01-16T15:02:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.686+0000 7fa3c8a00640 1 -- 172.21.15.43:0/2919055274 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015689608 0 0) 0x7fa3c415fa70 con 0x7fa3c413dfd0 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.687+0000 7fa3c9400640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c40cf500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0098a0 tx=0x7fa3ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250b9144a0aeb499 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3c8a00640 1 -- 172.21.15.43:0/2919055274 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac030840 con 0x7fa3c40cf050 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3c8a00640 1 -- 172.21.15.43:0/2919055274 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac0309a0 con 0x7fa3c40cf050 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3c8a00640 1 -- 172.21.15.43:0/2919055274 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac030d30 con 0x7fa3c40cf050 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 -- 172.21.15.43:0/2919055274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 msgr2=0x7fa3c40cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c40cf500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0098a0 tx=0x7fa3ac02b980 comp rx=0 tx=0).stop 2024-01-16T15:02:35.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 -- 172.21.15.43:0/2919055274 shutdown_connections 2024-01-16T15:02:35.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c40cd7b0 0x7fa3c40cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 --2- 172.21.15.43:0/2919055274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c40cf500 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 -- 172.21.15.43:0/2919055274 >> 172.21.15.43:0/2919055274 conn(0x7fa3c406a860 msgr2=0x7fa3c406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 -- 172.21.15.43:0/2919055274 shutdown_connections 2024-01-16T15:02:35.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.688+0000 7fa3cb200640 1 -- 172.21.15.43:0/2919055274 wait complete. 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.690+0000 7fa3cb200640 1 Processor -- start 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.690+0000 7fa3cb200640 1 -- start start 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 0x7fa3c413f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 0x7fa3c413b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c415aa20 con 0x7fa3c40cf050 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4157360 con 0x7fa3c40cd7b0 2024-01-16T15:02:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.691+0000 7fa3cb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c4157080 con 0x7fa3c413dfd0 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36812/0 (socket says 172.21.15.43:36812) 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 -- 172.21.15.43:0/3278442488 learned_addr learned my addr 172.21.15.43:0/3278442488 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3ca800640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 0x7fa3c413b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9e00640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 0x7fa3c413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 -- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 msgr2=0x7fa3c413b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 0x7fa3c413b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 -- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 msgr2=0x7fa3c413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 0x7fa3c413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9400640 1 -- 172.21.15.43:0/3278442488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0094d0 con 0x7fa3c40cf050 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3ca800640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 0x7fa3c413b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.692+0000 7fa3c9e00640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 0x7fa3c413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3c9400640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0098a0 tx=0x7fa3ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac047070 con 0x7fa3c40cf050 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac039a90 con 0x7fa3c40cf050 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac039e40 con 0x7fa3c40cf050 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c415acc0 con 0x7fa3c40cf050 2024-01-16T15:02:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.693+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c4155d50 con 0x7fa3c40cf050 2024-01-16T15:02:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.694+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c415a840 con 0x7fa3c40cf050 2024-01-16T15:02:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.695+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa3ac04c410 con 0x7fa3c40cf050 2024-01-16T15:02:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.696+0000 7fa3c2000640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 0x7fa39c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.696+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3ac03e070 con 0x7fa3c40cf050 2024-01-16T15:02:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.696+0000 7fa3c9e00640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 0x7fa39c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.697+0000 7fa3c9e00640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 0x7fa39c06da10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.701+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3ac077280 con 0x7fa3c40cf050 2024-01-16T15:02:35.816 INFO:tasks.ceph.mds.a.smithi043.stderr:2024-01-16T15:02:35.836+0000 7f2eee800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:02:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.903+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c404e840 con 0x7fa3c40cf050 2024-01-16T15:02:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.904+0000 7fa3c2000640 1 -- 172.21.15.43:0/3278442488 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16465 (secure 0 0 0) 0x7fa3ac07a8a0 con 0x7fa3c40cf050 2024-01-16T15:02:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:35.885 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":47,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:02:32.904105+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:02:29.285810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:02:30.367713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.907+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 msgr2=0x7fa39c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.907+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 0x7fa39c06da10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4009040 comp rx=0 tx=0).stop 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.907+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 msgr2=0x7fa3c4156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.907+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0098a0 tx=0x7fa3ac004730 comp rx=0 tx=0).stop 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 shutdown_connections 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa39c06b550 0x7fa39c06da10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c413dfd0 0x7fa3c413b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3c40cf050 0x7fa3c4156630 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 --2- 172.21.15.43:0/3278442488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40cd7b0 0x7fa3c413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 >> 172.21.15.43:0/3278442488 conn(0x7fa3c406a860 msgr2=0x7fa3c415e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 shutdown_connections 2024-01-16T15:02:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:35.908+0000 7fa3cb200640 1 -- 172.21.15.43:0/3278442488 wait complete. 2024-01-16T15:02:35.909 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:02:35.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.153+0000 7f7900a00640 1 Processor -- start 2024-01-16T15:02:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.153+0000 7f7900a00640 1 -- start start 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f7900a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78fc143570 0x7f78fc0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f7900a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 0x7f78fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f7900a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc071700 con 0x7f78fc15e6b0 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f7900a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc15b410 con 0x7f78fc143570 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f7900a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc15b130 con 0x7f78fc143950 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f78fbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f78fc15e6b0 0x7f78fc15a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44432/0 (socket says 172.21.15.43:44432) 2024-01-16T15:02:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f78fbe00640 1 -- 172.21.15.43:0/2089320848 learned_addr learned my addr 172.21.15.43:0/2089320848 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f78fb400640 1 --2- 172.21.15.43:0/2089320848 >> v2:172.21.15.69:3300/0 conn(0x7f78fc143570 0x7f78fc0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.155+0000 7f78f3e00640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 0x7f78fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601628134 0 0) 0x7f78f4001490 con 0x7f78fc15e6b0 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc003520 con 0x7f78fc15e6b0 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4061048882 0 0) 0x7f78dc003520 con 0x7f78fc15e6b0 2024-01-16T15:02:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78fb400640 1 -- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 msgr2=0x7f78fc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78fb400640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 0x7f78fc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78fb400640 1 -- 172.21.15.43:0/2089320848 >> v1:172.21.15.43:6789/0 conn(0x7f78fc15e6b0 legacy=0x7f78fc15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78fb400640 1 -- 172.21.15.43:0/2089320848 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc15dbd0 con 0x7f78fc143570 2024-01-16T15:02:36.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.156+0000 7f78f3e00640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 0x7f78fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:36.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.158+0000 7f78fb400640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143570 0x7f78fc0cf050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0097a0 tx=0x7f78ec02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee21aca01143f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.158+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4001b80 con 0x7f78fc143570 2024-01-16T15:02:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.158+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec02e050 con 0x7f78fc143570 2024-01-16T15:02:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.158+0000 7f78faa00640 1 -- 172.21.15.43:0/2089320848 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec030730 con 0x7f78fc143570 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 -- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143570 msgr2=0x7f78fc0cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143570 0x7f78fc0cf050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0097a0 tx=0x7f78ec02b710 comp rx=0 tx=0).stop 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 -- 172.21.15.43:0/2089320848 shutdown_connections 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc143950 0x7f78fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 --2- 172.21.15.43:0/2089320848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143570 0x7f78fc0cf050 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 -- 172.21.15.43:0/2089320848 >> 172.21.15.43:0/2089320848 conn(0x7f78fc06a860 msgr2=0x7f78fc06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 -- 172.21.15.43:0/2089320848 shutdown_connections 2024-01-16T15:02:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.159+0000 7f7900a00640 1 -- 172.21.15.43:0/2089320848 wait complete. 2024-01-16T15:02:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.161+0000 7f7900a00640 1 Processor -- start 2024-01-16T15:02:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.161+0000 7f7900a00640 1 -- start start 2024-01-16T15:02:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143950 0x7f78fc0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc15e6b0 0x7f78fc139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc13ac20 con 0x7f78fc143570 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0cd9e0 con 0x7f78fc143950 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.162+0000 7f7900a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0cde00 con 0x7f78fc15e6b0 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36820/0 (socket says 172.21.15.43:36820) 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 -- 172.21.15.43:0/2122160759 learned_addr learned my addr 172.21.15.43:0/2122160759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fbe00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc15e6b0 0x7f78fc139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78f3e00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143950 0x7f78fc0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 -- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc15e6b0 msgr2=0x7f78fc139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc15e6b0 0x7f78fc139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 -- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143950 msgr2=0x7f78fc0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143950 0x7f78fc0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.163+0000 7f78fb400640 1 -- 172.21.15.43:0/2122160759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4009400 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f78fb400640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78ec02bd20 tx=0x7f78ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec03d890 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec03da10 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec03ddc0 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ec009400 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.164+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0ccd60 con 0x7f78fc143570 2024-01-16T15:02:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.165+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc0cdc50 con 0x7f78fc143570 2024-01-16T15:02:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.166+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f78ec039040 con 0x7f78fc143570 2024-01-16T15:02:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.167+0000 7f78f3400640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 0x7f78d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.167+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78ec0ac130 con 0x7f78fc143570 2024-01-16T15:02:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.168+0000 7f78f3e00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 0x7f78d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.169+0000 7f78f3e00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 0x7f78d406d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f78fc15b800 tx=0x7f78e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.172+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78ec076c80 con 0x7f78fc143570 2024-01-16T15:02:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.365+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78fc15e190 con 0x7f78d406b500 2024-01-16T15:02:36.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.382+0000 7f78f3400640 1 -- 172.21.15.43:0/2122160759 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+273983 (secure 0 0 0) 0x7f78e4011a30 con 0x7f78d406b500 2024-01-16T15:02:36.366 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:36.367 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.388+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 msgr2=0x7f78d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.388+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 0x7f78d406d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f78fc15b800 tx=0x7f78e4008040 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.388+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 msgr2=0x7f78fc1666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.388+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78ec02bd20 tx=0x7f78ec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 shutdown_connections 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406b500 0x7f78d406d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc143950 0x7f78fc0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc15e6b0 0x7f78fc139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 --2- 172.21.15.43:0/2122160759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78fc143570 0x7f78fc1666b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 >> 172.21.15.43:0/2122160759 conn(0x7f78fc06a860 msgr2=0x7f78fc0b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 shutdown_connections 2024-01-16T15:02:36.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.389+0000 7f7900a00640 1 -- 172.21.15.43:0/2122160759 wait complete. 2024-01-16T15:02:36.388 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-16T15:02:34.851136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":262880,"kb_used_data":5064,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754711840,"statfs":{"total":773094113280,"available":772824924160,"internally_reserved":0,"allocated":5185536,"data_stored":2763328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":263835180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.984063"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.874133+0000","last_change":"2024-01-16T15:02:29.874133+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.874133+0000","last_undegraded":"2024-01-16T15:02:29.874133+0000","last_fullsized":"2024-01-16T15:02:29.874133+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.878213+0000","last_change":"2024-01-16T15:02:29.878213+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.878213+0000","last_undegraded":"2024-01-16T15:02:29.878213+0000","last_fullsized":"2024-01-16T15:02:29.878213+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.883648+0000","last_change":"2024-01-16T15:02:29.883648+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.883648+0000","last_undegraded":"2024-01-16T15:02:29.883648+0000","last_fullsized":"2024-01-16T15:02:29.883648+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.874017+0000","last_change":"2024-01-16T15:02:29.874017+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.874017+0000","last_undegraded":"2024-01-16T15:02:29.874017+0000","last_fullsized":"2024-01-16T15:02:29.874017+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.872963+0000","last_change":"2024-01-16T15:02:29.872963+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.872963+0000","last_undegraded":"2024-01-16T15:02:29.872963+0000","last_fullsized":"2024-01-16T15:02:29.872963+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.885109+0000","last_change":"2024-01-16T15:02:29.885109+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.885109+0000","last_undegraded":"2024-01-16T15:02:29.885109+0000","last_fullsized":"2024-01-16T15:02:29.885109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.883928+0000","last_change":"2024-01-16T15:02:29.883928+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.883928+0000","last_undegraded":"2024-01-16T15:02:29.883928+0000","last_fullsized":"2024-01-16T15:02:29.883928+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.873011+0000","last_change":"2024-01-16T15:02:29.873011+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.873011+0000","last_undegraded":"2024-01-16T15:02:29.873011+0000","last_fullsized":"2024-01-16T15:02:29.873011+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.883642+0000","last_change":"2024-01-16T15:02:29.883642+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.883642+0000","last_undegraded":"2024-01-16T15:02:29.883642+0000","last_fullsized":"2024-01-16T15:02:29.883642+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.883959+0000","last_change":"2024-01-16T15:02:29.883959+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.883959+0000","last_undegraded":"2024-01-16T15:02:29.883959+0000","last_fullsized":"2024-01-16T15:02:29.883959+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.878141+0000","last_change":"2024-01-16T15:02:29.878141+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.878141+0000","last_undegraded":"2024-01-16T15:02:29.878141+0000","last_fullsized":"2024-01-16T15:02:29.878141+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.884402+0000","last_change":"2024-01-16T15:02:29.884402+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.884402+0000","last_undegraded":"2024-01-16T15:02:29.884402+0000","last_fullsized":"2024-01-16T15:02:29.884402+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.872755+0000","last_change":"2024-01-16T15:02:29.872755+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.872755+0000","last_undegraded":"2024-01-16T15:02:29.872755+0000","last_fullsized":"2024-01-16T15:02:29.872755+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-16T15:02:29.877779+0000","last_change":"2024-01-16T15:02:29.877779+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.877779+0000","last_undegraded":"2024-01-16T15:02:29.877779+0000","last_fullsized":"2024-01-16T15:02:29.877779+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:30.866369+0000","last_change":"2024-01-16T15:02:30.866369+0000","last_active":"2024-01-16T15:02:30.866369+0000","last_peered":"2024-01-16T15:02:30.866369+0000","last_clean":"2024-01-16T15:02:30.866369+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:30.866369+0000","last_undegraded":"2024-01-16T15:02:30.866369+0000","last_fullsized":"2024-01-16T15:02:30.866369+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:02:29.850562+0000","last_change":"2024-01-16T15:02:29.850562+0000","last_active":"2024-01-16T15:02:29.850562+0000","last_peered":"2024-01-16T15:02:29.850562+0000","last_clean":"2024-01-16T15:02:29.850562+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:02:29.850562+0000","last_undegraded":"2024-01-16T15:02:29.850562+0000","last_fullsized":"2024-01-16T15:02:29.850562+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":182,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-16T15:02:27.081833+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:02:27.081833+0000","last_peered":"2024-01-16T15:02:27.081833+0000","last_clean":"2024-01-16T15:02:27.081833+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:02:27.081833+0000","last_undegraded":"2024-01-16T15:02:27.081833+0000","last_fullsized":"2024-01-16T15:02:27.081833+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":788192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.145,"15min":1.127},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":1.514},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.2529999999999999,"15min":1.22},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":6.2229999999999999,"5min":6.2229999999999999,"15min":6.2229999999999999},"last":1.5740000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.222,"15min":1.212},"min":{"1min":0.69899999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":1.381},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.1850000000000001,"15min":1.171},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.746}]},{"osd":2,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.4299999999999999,"15min":1.3240000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":10.707000000000001,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.331},{"interface":"front","average":{"1min":2.0489999999999999,"5min":1.4430000000000001,"15min":1.3400000000000001},"min":{"1min":0.95199999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":10.661,"5min":10.661,"15min":10.661},"last":1.7090000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.3220000000000001,"15min":1.2909999999999999},"min":{"1min":0.92700000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.4339999999999999,"15min":1.4199999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.429}]},{"osd":4,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":2.194,"5min":1.5249999999999999,"15min":1.4350000000000001},"min":{"1min":0.64900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":16.376000000000001,"5min":16.376000000000001,"15min":16.376000000000001},"last":1.246},{"interface":"front","average":{"1min":1.516,"5min":1.4570000000000001,"15min":1.4890000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":5.9720000000000004,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.609}]},{"osd":5,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5109999999999999,"15min":1.5549999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.6259999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.554,"15min":1.571},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":6.319,"5min":6.319,"15min":6.319},"last":1.4490000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.7190000000000001,"15min":1.712},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":8.2989999999999995,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.857,"5min":1.6859999999999999,"15min":1.698},"min":{"1min":0.999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":8.4979999999999993,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.675}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31684,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1679999999999999,"15min":1.1579999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.21,"15min":1.165},"min":{"1min":0.74099999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.2650000000000001,"5min":3.2650000000000001,"15min":3.2650000000000001},"last":1.254}]},{"osd":1,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.079,"15min":1.0740000000000001},"min":{"1min":0.68999999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1479999999999999,"15min":1.1259999999999999},"min":{"1min":0.77100000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.92800000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.23,"15min":1.2250000000000001},"min":{"1min":0.72999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.5089999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.641},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.3939999999999999,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3719999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":0.872}]},{"osd":3,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.2390000000000001},"min":{"1min":0.84699999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.323,"15min":1.288},"min":{"1min":0.91000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":1.4350000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":1.478,"15min":1.3700000000000001},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":15.481999999999999,"5min":15.481999999999999,"15min":15.481999999999999},"last":1.528},{"interface":"front","average":{"1min":1.329,"5min":1.4359999999999999,"15min":1.4239999999999999},"min":{"1min":0.66300000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":4.4409999999999998,"5min":6.585,"15min":6.585},"last":1.073}]},{"osd":5,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.325,"15min":1.3069999999999999},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.3860000000000001,"5min":3.3860000000000001,"15min":3.3860000000000001},"last":1.349},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5409999999999999,"15min":1.5529999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.8380000000000001,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":1.387}]},{"osd":7,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.4790000000000001,"15min":1.4550000000000001},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.147},{"interface":"front","average":{"1min":1.423,"5min":1.538,"15min":1.5840000000000001},"min":{"1min":0.80200000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.1280000000000001,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.734}]}]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31668,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.135,"15min":1.145},"min":{"1min":0.54000000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":1.069},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.1399999999999999,"15min":1.0589999999999999},"min":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":7.5789999999999997,"5min":7.5789999999999997,"15min":7.5789999999999997},"last":1.4610000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1719999999999999,"15min":1.123},"min":{"1min":0.65400000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":5.0940000000000003},"last":1.018},{"interface":"front","average":{"1min":2.3100000000000001,"5min":1.405,"15min":1.2769999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":16.113,"5min":16.113,"15min":16.113},"last":1.2070000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.258,"15min":1.2350000000000001},"min":{"1min":0.83199999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.2880000000000003,"5min":4.2880000000000003,"15min":4.2880000000000003},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.2809999999999999,"15min":1.256},"min":{"1min":0.86099999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":5.0309999999999997},"last":1.323}]},{"osd":3,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.575,"15min":1.3720000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":16.193999999999999,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.3680000000000001},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.538,"15min":1.444},"min":{"1min":0.873,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.873}]},{"osd":4,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.343,"15min":1.2509999999999999},"min":{"1min":0.93300000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":5.1440000000000001,"5min":7.234,"15min":7.234},"last":1.1240000000000001},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.526,"15min":1.399},"min":{"1min":0.92200000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":16.268000000000001,"5min":16.268000000000001,"15min":16.268000000000001},"last":1.5149999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4139999999999999,"15min":1.393},"min":{"1min":0.89200000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1989999999999998,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.5760000000000001,"15min":1.5049999999999999},"min":{"1min":0.97699999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.9299999999999997,"5min":7.335,"15min":7.335},"last":1.421}]},{"osd":7,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":1.6819999999999999,"15min":1.5780000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":12.571,"5min":12.571,"15min":12.571},"last":1.617},{"interface":"front","average":{"1min":2.3039999999999998,"5min":1.72,"15min":1.623},"min":{"1min":1.0229999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":12.673999999999999,"5min":12.673999999999999,"15min":12.673999999999999},"last":1.28}]}]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1100000000000001,"15min":1.0980000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.5760000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.085,"15min":1.0389999999999999},"min":{"1min":0.79400000000000004,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.5920000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3089999999999999,"15min":1.2909999999999999},"min":{"1min":0.95299999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":3.0609999999999999,"5min":3.0609999999999999,"15min":3.0609999999999999},"last":1.21},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.2589999999999999,"15min":1.2270000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.72599999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.23,"15min":1.1759999999999999},"min":{"1min":0.73999999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":5.5010000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.3819999999999999,"15min":1.3520000000000001},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":4.5019999999999998,"5min":4.5019999999999998,"15min":4.5019999999999998},"last":1.5169999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3959999999999999,"15min":1.4019999999999999},"min":{"1min":0.871,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.343,"15min":1.3260000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.2960000000000003,"5min":5.2960000000000003,"15min":5.2960000000000003},"last":1.0209999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4890000000000001,"15min":1.4990000000000001},"min":{"1min":0.873,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.0209999999999999,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.405},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":0.78100000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.47}]},{"osd":6,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.444,"15min":1.464},"min":{"1min":0.89400000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.53,"5min":1.571,"15min":1.595},"min":{"1min":0.81799999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.2320000000000002,"5min":3.976,"15min":3.976},"last":1.6439999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.673,"15min":1.7050000000000001},"min":{"1min":1.2789999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.2130000000000001,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.631,"5min":1.6359999999999999,"15min":1.6319999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":5.4610000000000003,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.3160000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35840,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.1930000000000001,"15min":1.0109999999999999},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":17.709,"5min":17.709,"15min":17.709},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":6.6239999999999997,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":1.0189999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.0920000000000001,"15min":1.016},"min":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2889999999999999,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.4489999999999998,"5min":10.163,"15min":10.163},"last":1.25}]},{"osd":2,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.22,"15min":1.1639999999999999},"min":{"1min":0.47599999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8439999999999994,"5min":8.8439999999999994,"15min":8.8439999999999994},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.405,"15min":1.2969999999999999},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":6.7690000000000001,"5min":10.25,"15min":10.25},"last":0.82799999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3300000000000001,"15min":1.3340000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":4.2439999999999998,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.452,"5min":1.429,"15min":1.423},"min":{"1min":0.85299999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3849999999999998,"5min":2.605,"15min":2.605},"last":1.2}]},{"osd":5,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.6639999999999999,"15min":1.53},"min":{"1min":0.80800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":12.65,"5min":12.65,"15min":12.65},"last":1.702},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.4419999999999999,"15min":1.4059999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"last":1.153}]},{"osd":6,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.4990000000000001,"15min":1.484},"min":{"1min":0.95299999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":1.29},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.6830000000000001,"15min":1.593},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":6.681,"5min":10.371,"15min":10.371},"last":1.7789999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5249999999999999,"15min":1.5369999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.5900000000000001,"15min":1.5580000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":5.1699999999999999,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":1.3440000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31672,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.0489999999999999,"15min":0.96199999999999997},"min":{"1min":0.54800000000000004,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":12.478999999999999,"5min":12.478999999999999,"15min":12.478999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.0349999999999999,"15min":1.0009999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":1.3680000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.97399999999999998,"15min":0.91900000000000004},"min":{"1min":0.68400000000000005,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.093,"5min":3.093,"15min":3.093},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2230000000000001,"15min":1.22},"min":{"1min":0.60199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8620000000000001,"5min":4.6769999999999996,"15min":4.6769999999999996},"last":1.24}]},{"osd":3,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2130000000000001,"15min":1.1699999999999999},"min":{"1min":0.69599999999999995,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.296,"15min":1.1859999999999999},"min":{"1min":0.871,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":12.198,"5min":12.198,"15min":12.198},"last":1.526}]},{"osd":4,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0179999999999998,"5min":1.446,"15min":1.3500000000000001},"min":{"1min":1.0900000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":12.287000000000001,"5min":12.287000000000001,"15min":12.287000000000001},"last":1.403},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.353,"15min":1.337},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.2759999999999998,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.665}]},{"osd":5,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.3919999999999999,"15min":1.3799999999999999},"min":{"1min":0.90800000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":3.4980000000000002,"5min":3.4980000000000002,"15min":3.4980000000000002},"last":1.9470000000000001},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.6080000000000001,"15min":1.538},"min":{"1min":1.1579999999999999,"5min":0.875,"15min":0.875},"max":{"1min":12.25,"5min":12.25,"15min":12.25},"last":1.758}]},{"osd":6,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":0.93899999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.1709999999999998,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.901},{"interface":"front","average":{"1min":2.52,"5min":1.665,"15min":1.5369999999999999},"min":{"1min":0.88900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":11.542999999999999,"5min":11.542999999999999,"15min":11.542999999999999},"last":1.8029999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.629,"15min":1.609},"min":{"1min":1.1200000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":5.4889999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":1.486},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.496,"15min":1.4670000000000001},"min":{"1min":0.96399999999999997,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.3220000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35828,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336012,"statfs":{"total":96636764160,"available":96600076288,"internally_reserved":0,"allocated":499712,"data_stored":197824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.3100000000000001,"15min":1.1160000000000001},"min":{"1min":0.69399999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":14.407,"5min":14.407,"15min":14.407},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.1599999999999999,"15min":1.1240000000000001},"min":{"1min":0.66200000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.92400000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.1499999999999999,"15min":1.0960000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":3.141,"5min":1.585,"15min":1.3300000000000001},"min":{"1min":0.65800000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":14.311,"5min":14.311,"15min":14.311},"last":1.3520000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":1.667,"15min":1.4199999999999999},"min":{"1min":1.0029999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":14.653,"5min":14.653,"15min":14.653},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.806,"5min":1.379,"15min":1.3240000000000001},"min":{"1min":0.91100000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"last":1.0449999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":1.5880000000000001,"15min":1.468},"min":{"1min":0.97799999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":12.675000000000001,"5min":12.675000000000001,"15min":12.675000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":2.3969999999999998,"5min":1.6220000000000001,"15min":1.512},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":12.731999999999999,"5min":12.731999999999999,"15min":12.731999999999999},"last":1.254}]},{"osd":5,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.599,"15min":1.5680000000000001},"min":{"1min":1.198,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.635,"5min":1.476,"15min":1.48},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":6.9569999999999999,"5min":6.9569999999999999,"15min":6.9569999999999999},"last":0.998}]},{"osd":6,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":3.367,"5min":1.9390000000000001,"15min":1.7110000000000001},"min":{"1min":1.1120000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":14.781000000000001,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.208},{"interface":"front","average":{"1min":2.1309999999999998,"5min":1.7250000000000001,"15min":1.677},"min":{"1min":1.3759999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":6.6779999999999999,"5min":6.6779999999999999,"15min":6.6779999999999999},"last":1.569}]},{"osd":7,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.583,"15min":1.5800000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":6.75,"5min":6.75,"15min":6.75},"last":1.1379999999999999},{"interface":"front","average":{"1min":3.5379999999999998,"5min":2.0310000000000001,"15min":1.7889999999999999},"min":{"1min":1.1859999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":14.884,"5min":14.884,"15min":14.884},"last":1.421}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1093632,"data_stored":788192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.94699999999999995,"15min":0.91000000000000003},"min":{"1min":0.51800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.079},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.96499999999999997,"15min":0.92100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":1.5629999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.1240000000000001,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.085,"5min":1.0249999999999999,"15min":0.99099999999999999},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.1220000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5859999999999999,"5min":1.2649999999999999,"15min":1.0329999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.357,"5min":1.2050000000000001,"15min":1.1679999999999999},"min":{"1min":0.78800000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":1.175}]},{"osd":4,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.256,"15min":1.1879999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.218},{"interface":"front","average":{"1min":1.48,"5min":1.3759999999999999,"15min":1.343},"min":{"1min":0.84399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":4.2789999999999999,"5min":4.2789999999999999,"15min":4.2789999999999999},"last":1.369}]},{"osd":5,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3009999999999999,"15min":1.2809999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.3440000000000001,"15min":1.302},"min":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":8.1489999999999991,"5min":8.1489999999999991,"15min":8.1489999999999991},"last":1.3220000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5,"15min":1.486},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":1.482},{"interface":"front","average":{"1min":1.452,"5min":1.3999999999999999,"15min":1.375},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2869999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.2769999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5,"15min":1.488},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.625},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.488,"15min":1.4379999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":8.0800000000000001,"5min":8.0800000000000001,"15min":8.0800000000000001},"last":1.506}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:36.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:02:36.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.678+0000 7f0153200640 1 Processor -- start 2024-01-16T15:02:36.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.679+0000 7f0153200640 1 -- start start 2024-01-16T15:02:36.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.680+0000 7f0153200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f014c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.680+0000 7f0153200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f014c0b0630 0x7f014c0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.680+0000 7f0153200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c0cf0b0 con 0x7f014c15d5a0 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0153200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c0b01a0 con 0x7f014c0b0630 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0153200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c15d980 con 0x7f014c0d1490 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0151e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f014c15d5a0 0x7f014c0d3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44440/0 (socket says 172.21.15.43:44440) 2024-01-16T15:02:36.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0151e00640 1 -- 172.21.15.43:0/4159373519 learned_addr learned my addr 172.21.15.43:0/4159373519 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:36.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764194290 0 0) 0x7f014c0cf0b0 con 0x7f014c15d5a0 2024-01-16T15:02:36.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c003520 con 0x7f014c15d5a0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0151400640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f014c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.681+0000 7f0152800640 1 --2- 172.21.15.43:0/4159373519 >> v2:172.21.15.69:3300/0 conn(0x7f014c0b0630 0x7f014c0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946265863 0 0) 0x7f012c003520 con 0x7f014c15d5a0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 msgr2=0x7f014c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f014c0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 >> v2:172.21.15.69:3300/0 conn(0x7f014c0b0630 msgr2=0x7f014c0b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 --2- 172.21.15.43:0/4159373519 >> v2:172.21.15.69:3300/0 conn(0x7f014c0b0630 0x7f014c0b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0d1e30 con 0x7f014c15d5a0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f013c001670 con 0x7f014c15d5a0 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 >> v1:172.21.15.43:6789/0 conn(0x7f014c15d5a0 legacy=0x7f014c0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f012c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004010 0x7f012c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f012c004d00 0x7f012c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0b01a0 con 0x7f012c004d00 2024-01-16T15:02:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c15d980 con 0x7f012c004010 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c00bb10 con 0x7f014c0d1490 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0152800640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f012c004d00 0x7f012c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0151400640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004010 0x7f012c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.682+0000 7f0151e00640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f012c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.683+0000 7f0152800640 1 -- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 msgr2=0x7f012c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.683+0000 7f0152800640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c0d1490 0x7f012c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.683+0000 7f0152800640 1 -- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004010 msgr2=0x7f012c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.683+0000 7f0152800640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004010 0x7f012c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.683+0000 7f0152800640 1 -- 172.21.15.43:0/4159373519 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c003520 con 0x7f012c004d00 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0152800640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f012c004d00 0x7f012c005180 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0144007330 tx=0x7f012c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4399587402983978 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014402ccd0 con 0x7f012c004d00 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014402ce30 con 0x7f012c004d00 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0150a00640 1 -- 172.21.15.43:0/4159373519 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144034660 con 0x7f012c004d00 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0153200640 1 -- 172.21.15.43:0/4159373519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f012c004d00 msgr2=0x7f012c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0153200640 1 --2- 172.21.15.43:0/4159373519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f012c004d00 0x7f012c005180 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0144007330 tx=0x7f012c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.684+0000 7f0152800640 1 -- 172.21.15.43:0/4159373519 reap_dead start 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.685+0000 7f0153200640 1 -- 172.21.15.43:0/4159373519 shutdown_connections 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.685+0000 7f0153200640 1 -- 172.21.15.43:0/4159373519 >> 172.21.15.43:0/4159373519 conn(0x7f014c09c3a0 msgr2=0x7f014c0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.685+0000 7f0153200640 1 -- 172.21.15.43:0/4159373519 shutdown_connections 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.685+0000 7f0153200640 1 -- 172.21.15.43:0/4159373519 wait complete. 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.687+0000 7f0153200640 1 Processor -- start 2024-01-16T15:02:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.687+0000 7f0153200640 1 -- start start 2024-01-16T15:02:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 0x7f014c0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c1570c0 0x7f014c1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0c7f20 con 0x7f014c156850 2024-01-16T15:02:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0c8aa0 con 0x7f012c004d00 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0153200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0c8170 con 0x7f014c1570c0 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36844/0 (socket says 172.21.15.43:36844) 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 -- 172.21.15.43:0/1199771090 learned_addr learned my addr 172.21.15.43:0/1199771090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0151e00640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 0x7f014c0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0151400640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c1570c0 0x7f014c1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 -- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c1570c0 msgr2=0x7f014c1f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c1570c0 0x7f014c1f2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 -- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 msgr2=0x7f014c0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 0x7f014c0ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0152800640 1 -- 172.21.15.43:0/1199771090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0c8450 con 0x7f014c156850 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.688+0000 7f0151e00640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 0x7f014c0ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:36.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f0152800640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f014402c740 tx=0x7f0144004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144045070 con 0x7f014c156850 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0144003990 con 0x7f014c156850 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144003d40 con 0x7f014c156850 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c0c8730 con 0x7f014c156850 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.689+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c157850 con 0x7f014c156850 2024-01-16T15:02:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.690+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c04e840 con 0x7f014c156850 2024-01-16T15:02:36.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.691+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f014403b040 con 0x7f014c156850 2024-01-16T15:02:36.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.693+0000 7f014a000640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.693+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01440ab350 con 0x7f014c156850 2024-01-16T15:02:36.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.693+0000 7f0151e00640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.694+0000 7f0151e00640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 0x7f012406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f013c000f60 tx=0x7f013c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.697+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0144075e50 con 0x7f014c156850 2024-01-16T15:02:36.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.889+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f014c0afe30 con 0x7f012406b500 2024-01-16T15:02:36.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.913+0000 7f014a000640 1 -- 172.21.15.43:0/1199771090 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287748 (secure 0 0 0) 0x7f013c00f7f0 con 0x7f012406b500 2024-01-16T15:02:36.897 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:36.898 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 msgr2=0x7f012406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 0x7f012406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f013c000f60 tx=0x7f013c000ed0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 msgr2=0x7f014c15e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f014402c740 tx=0x7f0144004ab0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 shutdown_connections 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f014c156850 0x7f014c15e9b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014c1570c0 0x7f014c1f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 --2- 172.21.15.43:0/1199771090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f012c004d00 0x7f014c0ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 >> 172.21.15.43:0/1199771090 conn(0x7f014c09c3a0 msgr2=0x7f014c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 shutdown_connections 2024-01-16T15:02:36.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:36.919+0000 7f0153200640 1 -- 172.21.15.43:0/1199771090 wait complete. 2024-01-16T15:02:36.920 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-01-16T15:02:36.851732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263332,"kb_used_data":5512,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754711388,"statfs":{"total":773094113280,"available":772824461312,"internally_reserved":0,"allocated":5644288,"data_stored":2969420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":263835180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001785"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157724+0000","last_change":"2024-01-16T15:02:31.926383+0000","last_active":"2024-01-16T15:02:33.157724+0000","last_peered":"2024-01-16T15:02:33.157724+0000","last_clean":"2024-01-16T15:02:33.157724+0000","last_became_active":"2024-01-16T15:02:31.926001+0000","last_became_peered":"2024-01-16T15:02:31.926001+0000","last_unstale":"2024-01-16T15:02:33.157724+0000","last_undegraded":"2024-01-16T15:02:33.157724+0000","last_fullsized":"2024-01-16T15:02:33.157724+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:09.875585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.917126+0000","last_change":"2024-01-16T15:02:31.935561+0000","last_active":"2024-01-16T15:02:32.917126+0000","last_peered":"2024-01-16T15:02:32.917126+0000","last_clean":"2024-01-16T15:02:32.917126+0000","last_became_active":"2024-01-16T15:02:31.935323+0000","last_became_peered":"2024-01-16T15:02:31.935323+0000","last_unstale":"2024-01-16T15:02:32.917126+0000","last_undegraded":"2024-01-16T15:02:32.917126+0000","last_fullsized":"2024-01-16T15:02:32.917126+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:23.531922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156658+0000","last_change":"2024-01-16T15:02:31.911920+0000","last_active":"2024-01-16T15:02:33.156658+0000","last_peered":"2024-01-16T15:02:33.156658+0000","last_clean":"2024-01-16T15:02:33.156658+0000","last_became_active":"2024-01-16T15:02:31.911473+0000","last_became_peered":"2024-01-16T15:02:31.911473+0000","last_unstale":"2024-01-16T15:02:33.156658+0000","last_undegraded":"2024-01-16T15:02:33.156658+0000","last_fullsized":"2024-01-16T15:02:33.156658+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:34.789090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.159949+0000","last_change":"2024-01-16T15:02:31.929603+0000","last_active":"2024-01-16T15:02:33.159949+0000","last_peered":"2024-01-16T15:02:33.159949+0000","last_clean":"2024-01-16T15:02:33.159949+0000","last_became_active":"2024-01-16T15:02:31.929033+0000","last_became_peered":"2024-01-16T15:02:31.929033+0000","last_unstale":"2024-01-16T15:02:33.159949+0000","last_undegraded":"2024-01-16T15:02:33.159949+0000","last_fullsized":"2024-01-16T15:02:33.159949+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:17.790196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915441+0000","last_change":"2024-01-16T15:02:31.920464+0000","last_active":"2024-01-16T15:02:32.915441+0000","last_peered":"2024-01-16T15:02:32.915441+0000","last_clean":"2024-01-16T15:02:32.915441+0000","last_became_active":"2024-01-16T15:02:31.919590+0000","last_became_peered":"2024-01-16T15:02:31.919590+0000","last_unstale":"2024-01-16T15:02:32.915441+0000","last_undegraded":"2024-01-16T15:02:32.915441+0000","last_fullsized":"2024-01-16T15:02:32.915441+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:41.209200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155453+0000","last_change":"2024-01-16T15:02:31.920669+0000","last_active":"2024-01-16T15:02:33.155453+0000","last_peered":"2024-01-16T15:02:33.155453+0000","last_clean":"2024-01-16T15:02:33.155453+0000","last_became_active":"2024-01-16T15:02:31.920331+0000","last_became_peered":"2024-01-16T15:02:31.920331+0000","last_unstale":"2024-01-16T15:02:33.155453+0000","last_undegraded":"2024-01-16T15:02:33.155453+0000","last_fullsized":"2024-01-16T15:02:33.155453+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:00.582744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155952+0000","last_change":"2024-01-16T15:02:31.921622+0000","last_active":"2024-01-16T15:02:33.155952+0000","last_peered":"2024-01-16T15:02:33.155952+0000","last_clean":"2024-01-16T15:02:33.155952+0000","last_became_active":"2024-01-16T15:02:31.921219+0000","last_became_peered":"2024-01-16T15:02:31.921219+0000","last_unstale":"2024-01-16T15:02:33.155952+0000","last_undegraded":"2024-01-16T15:02:33.155952+0000","last_fullsized":"2024-01-16T15:02:33.155952+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:30:51.883778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157678+0000","last_change":"2024-01-16T15:02:31.930899+0000","last_active":"2024-01-16T15:02:33.157678+0000","last_peered":"2024-01-16T15:02:33.157678+0000","last_clean":"2024-01-16T15:02:33.157678+0000","last_became_active":"2024-01-16T15:02:31.930492+0000","last_became_peered":"2024-01-16T15:02:31.930492+0000","last_unstale":"2024-01-16T15:02:33.157678+0000","last_undegraded":"2024-01-16T15:02:33.157678+0000","last_fullsized":"2024-01-16T15:02:33.157678+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:47.110165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160360+0000","last_change":"2024-01-16T15:02:31.925325+0000","last_active":"2024-01-16T15:02:33.160360+0000","last_peered":"2024-01-16T15:02:33.160360+0000","last_clean":"2024-01-16T15:02:33.160360+0000","last_became_active":"2024-01-16T15:02:31.924963+0000","last_became_peered":"2024-01-16T15:02:31.924963+0000","last_unstale":"2024-01-16T15:02:33.160360+0000","last_undegraded":"2024-01-16T15:02:33.160360+0000","last_fullsized":"2024-01-16T15:02:33.160360+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:23.531922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160539+0000","last_change":"2024-01-16T15:02:31.932045+0000","last_active":"2024-01-16T15:02:33.160539+0000","last_peered":"2024-01-16T15:02:33.160539+0000","last_clean":"2024-01-16T15:02:33.160539+0000","last_became_active":"2024-01-16T15:02:31.931753+0000","last_became_peered":"2024-01-16T15:02:31.931753+0000","last_unstale":"2024-01-16T15:02:33.160539+0000","last_undegraded":"2024-01-16T15:02:33.160539+0000","last_fullsized":"2024-01-16T15:02:33.160539+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:25.481867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160317+0000","last_change":"2024-01-16T15:02:31.916362+0000","last_active":"2024-01-16T15:02:33.160317+0000","last_peered":"2024-01-16T15:02:33.160317+0000","last_clean":"2024-01-16T15:02:33.160317+0000","last_became_active":"2024-01-16T15:02:31.915619+0000","last_became_peered":"2024-01-16T15:02:31.915619+0000","last_unstale":"2024-01-16T15:02:33.160317+0000","last_undegraded":"2024-01-16T15:02:33.160317+0000","last_fullsized":"2024-01-16T15:02:33.160317+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:09.635701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916479+0000","last_change":"2024-01-16T15:02:31.928420+0000","last_active":"2024-01-16T15:02:32.916479+0000","last_peered":"2024-01-16T15:02:32.916479+0000","last_clean":"2024-01-16T15:02:32.916479+0000","last_became_active":"2024-01-16T15:02:31.928009+0000","last_became_peered":"2024-01-16T15:02:31.928009+0000","last_unstale":"2024-01-16T15:02:32.916479+0000","last_undegraded":"2024-01-16T15:02:32.916479+0000","last_fullsized":"2024-01-16T15:02:32.916479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:24.838719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912022+0000","last_change":"2024-01-16T15:02:31.921543+0000","last_active":"2024-01-16T15:02:32.912022+0000","last_peered":"2024-01-16T15:02:32.912022+0000","last_clean":"2024-01-16T15:02:32.912022+0000","last_became_active":"2024-01-16T15:02:31.921251+0000","last_became_peered":"2024-01-16T15:02:31.921251+0000","last_unstale":"2024-01-16T15:02:32.912022+0000","last_undegraded":"2024-01-16T15:02:32.912022+0000","last_fullsized":"2024-01-16T15:02:32.912022+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:04.216185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158939+0000","last_change":"2024-01-16T15:02:31.928258+0000","last_active":"2024-01-16T15:02:33.158939+0000","last_peered":"2024-01-16T15:02:33.158939+0000","last_clean":"2024-01-16T15:02:33.158939+0000","last_became_active":"2024-01-16T15:02:31.927884+0000","last_became_peered":"2024-01-16T15:02:31.927884+0000","last_unstale":"2024-01-16T15:02:33.158939+0000","last_undegraded":"2024-01-16T15:02:33.158939+0000","last_fullsized":"2024-01-16T15:02:33.158939+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:48.269329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155042+0000","last_change":"2024-01-16T15:02:31.920831+0000","last_active":"2024-01-16T15:02:33.155042+0000","last_peered":"2024-01-16T15:02:33.155042+0000","last_clean":"2024-01-16T15:02:33.155042+0000","last_became_active":"2024-01-16T15:02:31.920511+0000","last_became_peered":"2024-01-16T15:02:31.920511+0000","last_unstale":"2024-01-16T15:02:33.155042+0000","last_undegraded":"2024-01-16T15:02:33.155042+0000","last_fullsized":"2024-01-16T15:02:33.155042+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:11.814450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916142+0000","last_change":"2024-01-16T15:02:31.925942+0000","last_active":"2024-01-16T15:02:32.916142+0000","last_peered":"2024-01-16T15:02:32.916142+0000","last_clean":"2024-01-16T15:02:32.916142+0000","last_became_active":"2024-01-16T15:02:31.925689+0000","last_became_peered":"2024-01-16T15:02:31.925689+0000","last_unstale":"2024-01-16T15:02:32.916142+0000","last_undegraded":"2024-01-16T15:02:32.916142+0000","last_fullsized":"2024-01-16T15:02:32.916142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:43.232601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156990+0000","last_change":"2024-01-16T15:02:31.920254+0000","last_active":"2024-01-16T15:02:33.156990+0000","last_peered":"2024-01-16T15:02:33.156990+0000","last_clean":"2024-01-16T15:02:33.156990+0000","last_became_active":"2024-01-16T15:02:31.919991+0000","last_became_peered":"2024-01-16T15:02:31.919991+0000","last_unstale":"2024-01-16T15:02:33.156990+0000","last_undegraded":"2024-01-16T15:02:33.156990+0000","last_fullsized":"2024-01-16T15:02:33.156990+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:49.314425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156677+0000","last_change":"2024-01-16T15:02:31.934897+0000","last_active":"2024-01-16T15:02:33.156677+0000","last_peered":"2024-01-16T15:02:33.156677+0000","last_clean":"2024-01-16T15:02:33.156677+0000","last_became_active":"2024-01-16T15:02:31.934612+0000","last_became_peered":"2024-01-16T15:02:31.934612+0000","last_unstale":"2024-01-16T15:02:33.156677+0000","last_undegraded":"2024-01-16T15:02:33.156677+0000","last_fullsized":"2024-01-16T15:02:33.156677+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:43.232601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915682+0000","last_change":"2024-01-16T15:02:31.922358+0000","last_active":"2024-01-16T15:02:32.915682+0000","last_peered":"2024-01-16T15:02:32.915682+0000","last_clean":"2024-01-16T15:02:32.915682+0000","last_became_active":"2024-01-16T15:02:31.920360+0000","last_became_peered":"2024-01-16T15:02:31.920360+0000","last_unstale":"2024-01-16T15:02:32.915682+0000","last_undegraded":"2024-01-16T15:02:32.915682+0000","last_fullsized":"2024-01-16T15:02:32.915682+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:29.833091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912352+0000","last_change":"2024-01-16T15:02:31.915273+0000","last_active":"2024-01-16T15:02:32.912352+0000","last_peered":"2024-01-16T15:02:32.912352+0000","last_clean":"2024-01-16T15:02:32.912352+0000","last_became_active":"2024-01-16T15:02:31.914936+0000","last_became_peered":"2024-01-16T15:02:31.914936+0000","last_unstale":"2024-01-16T15:02:32.912352+0000","last_undegraded":"2024-01-16T15:02:32.912352+0000","last_fullsized":"2024-01-16T15:02:32.912352+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:36.608115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155533+0000","last_change":"2024-01-16T15:02:31.931233+0000","last_active":"2024-01-16T15:02:33.155533+0000","last_peered":"2024-01-16T15:02:33.155533+0000","last_clean":"2024-01-16T15:02:33.155533+0000","last_became_active":"2024-01-16T15:02:31.930936+0000","last_became_peered":"2024-01-16T15:02:31.930936+0000","last_unstale":"2024-01-16T15:02:33.155533+0000","last_undegraded":"2024-01-16T15:02:33.155533+0000","last_fullsized":"2024-01-16T15:02:33.155533+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:19:17.478410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.159312+0000","last_change":"2024-01-16T15:02:31.931905+0000","last_active":"2024-01-16T15:02:33.159312+0000","last_peered":"2024-01-16T15:02:33.159312+0000","last_clean":"2024-01-16T15:02:33.159312+0000","last_became_active":"2024-01-16T15:02:31.928269+0000","last_became_peered":"2024-01-16T15:02:31.928269+0000","last_unstale":"2024-01-16T15:02:33.159312+0000","last_undegraded":"2024-01-16T15:02:33.159312+0000","last_fullsized":"2024-01-16T15:02:33.159312+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:01.772690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916117+0000","last_change":"2024-01-16T15:02:31.923257+0000","last_active":"2024-01-16T15:02:32.916117+0000","last_peered":"2024-01-16T15:02:32.916117+0000","last_clean":"2024-01-16T15:02:32.916117+0000","last_became_active":"2024-01-16T15:02:31.922701+0000","last_became_peered":"2024-01-16T15:02:31.922701+0000","last_unstale":"2024-01-16T15:02:32.916117+0000","last_undegraded":"2024-01-16T15:02:32.916117+0000","last_fullsized":"2024-01-16T15:02:32.916117+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:34.789090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916232+0000","last_change":"2024-01-16T15:02:31.923253+0000","last_active":"2024-01-16T15:02:32.916232+0000","last_peered":"2024-01-16T15:02:32.916232+0000","last_clean":"2024-01-16T15:02:32.916232+0000","last_became_active":"2024-01-16T15:02:31.922900+0000","last_became_peered":"2024-01-16T15:02:31.922900+0000","last_unstale":"2024-01-16T15:02:32.916232+0000","last_undegraded":"2024-01-16T15:02:32.916232+0000","last_fullsized":"2024-01-16T15:02:32.916232+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:16:39.017461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.914936+0000","last_change":"2024-01-16T15:02:31.918766+0000","last_active":"2024-01-16T15:02:32.914936+0000","last_peered":"2024-01-16T15:02:32.914936+0000","last_clean":"2024-01-16T15:02:32.914936+0000","last_became_active":"2024-01-16T15:02:31.918481+0000","last_became_peered":"2024-01-16T15:02:31.918481+0000","last_unstale":"2024-01-16T15:02:32.914936+0000","last_undegraded":"2024-01-16T15:02:32.914936+0000","last_fullsized":"2024-01-16T15:02:32.914936+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:08.639802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912241+0000","last_change":"2024-01-16T15:02:31.913552+0000","last_active":"2024-01-16T15:02:32.912241+0000","last_peered":"2024-01-16T15:02:32.912241+0000","last_clean":"2024-01-16T15:02:32.912241+0000","last_became_active":"2024-01-16T15:02:31.913264+0000","last_became_peered":"2024-01-16T15:02:31.913264+0000","last_unstale":"2024-01-16T15:02:32.912241+0000","last_undegraded":"2024-01-16T15:02:32.912241+0000","last_fullsized":"2024-01-16T15:02:32.912241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:27:14.495869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156559+0000","last_change":"2024-01-16T15:02:31.924124+0000","last_active":"2024-01-16T15:02:33.156559+0000","last_peered":"2024-01-16T15:02:33.156559+0000","last_clean":"2024-01-16T15:02:33.156559+0000","last_became_active":"2024-01-16T15:02:31.923813+0000","last_became_peered":"2024-01-16T15:02:31.923813+0000","last_unstale":"2024-01-16T15:02:33.156559+0000","last_undegraded":"2024-01-16T15:02:33.156559+0000","last_fullsized":"2024-01-16T15:02:33.156559+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:27.286876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916365+0000","last_change":"2024-01-16T15:02:31.927858+0000","last_active":"2024-01-16T15:02:32.916365+0000","last_peered":"2024-01-16T15:02:32.916365+0000","last_clean":"2024-01-16T15:02:32.916365+0000","last_became_active":"2024-01-16T15:02:31.927584+0000","last_became_peered":"2024-01-16T15:02:31.927584+0000","last_unstale":"2024-01-16T15:02:32.916365+0000","last_undegraded":"2024-01-16T15:02:32.916365+0000","last_fullsized":"2024-01-16T15:02:32.916365+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:17.790196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916539+0000","last_change":"2024-01-16T15:02:31.917597+0000","last_active":"2024-01-16T15:02:32.916539+0000","last_peered":"2024-01-16T15:02:32.916539+0000","last_clean":"2024-01-16T15:02:32.916539+0000","last_became_active":"2024-01-16T15:02:31.916796+0000","last_became_peered":"2024-01-16T15:02:31.916796+0000","last_unstale":"2024-01-16T15:02:32.916539+0000","last_undegraded":"2024-01-16T15:02:32.916539+0000","last_fullsized":"2024-01-16T15:02:32.916539+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:49.314425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155850+0000","last_change":"2024-01-16T15:02:31.922122+0000","last_active":"2024-01-16T15:02:33.155850+0000","last_peered":"2024-01-16T15:02:33.155850+0000","last_clean":"2024-01-16T15:02:33.155850+0000","last_became_active":"2024-01-16T15:02:31.920896+0000","last_became_peered":"2024-01-16T15:02:31.920896+0000","last_unstale":"2024-01-16T15:02:33.155850+0000","last_undegraded":"2024-01-16T15:02:33.155850+0000","last_fullsized":"2024-01-16T15:02:33.155850+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:15.732991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155408+0000","last_change":"2024-01-16T15:02:31.927431+0000","last_active":"2024-01-16T15:02:33.155408+0000","last_peered":"2024-01-16T15:02:33.155408+0000","last_clean":"2024-01-16T15:02:33.155408+0000","last_became_active":"2024-01-16T15:02:31.926873+0000","last_became_peered":"2024-01-16T15:02:31.926873+0000","last_unstale":"2024-01-16T15:02:33.155408+0000","last_undegraded":"2024-01-16T15:02:33.155408+0000","last_fullsized":"2024-01-16T15:02:33.155408+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:09.875585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.911796+0000","last_change":"2024-01-16T15:02:31.912780+0000","last_active":"2024-01-16T15:02:32.911796+0000","last_peered":"2024-01-16T15:02:32.911796+0000","last_clean":"2024-01-16T15:02:32.911796+0000","last_became_active":"2024-01-16T15:02:31.910028+0000","last_became_peered":"2024-01-16T15:02:31.910028+0000","last_unstale":"2024-01-16T15:02:32.911796+0000","last_undegraded":"2024-01-16T15:02:32.911796+0000","last_fullsized":"2024-01-16T15:02:32.911796+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:23.806240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157800+0000","last_change":"2024-01-16T15:02:31.928148+0000","last_active":"2024-01-16T15:02:33.157800+0000","last_peered":"2024-01-16T15:02:33.157800+0000","last_clean":"2024-01-16T15:02:33.157800+0000","last_became_active":"2024-01-16T15:02:31.927817+0000","last_became_peered":"2024-01-16T15:02:31.927817+0000","last_unstale":"2024-01-16T15:02:33.157800+0000","last_undegraded":"2024-01-16T15:02:33.157800+0000","last_fullsized":"2024-01-16T15:02:33.157800+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:27.286876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"46'4","reported_seq":16,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158763+0000","last_change":"2024-01-16T15:02:30.914232+0000","last_active":"2024-01-16T15:02:33.158763+0000","last_peered":"2024-01-16T15:02:33.158763+0000","last_clean":"2024-01-16T15:02:33.158763+0000","last_became_active":"2024-01-16T15:02:30.913930+0000","last_became_peered":"2024-01-16T15:02:30.913930+0000","last_unstale":"2024-01-16T15:02:33.158763+0000","last_undegraded":"2024-01-16T15:02:33.158763+0000","last_fullsized":"2024-01-16T15:02:33.158763+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:45.318685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915683+0000","last_change":"2024-01-16T15:02:31.915257+0000","last_active":"2024-01-16T15:02:32.915683+0000","last_peered":"2024-01-16T15:02:32.915683+0000","last_clean":"2024-01-16T15:02:32.915683+0000","last_became_active":"2024-01-16T15:02:31.914997+0000","last_became_peered":"2024-01-16T15:02:31.914997+0000","last_unstale":"2024-01-16T15:02:32.915683+0000","last_undegraded":"2024-01-16T15:02:32.915683+0000","last_fullsized":"2024-01-16T15:02:32.915683+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:15.732991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.923471+0000","last_change":"2024-01-16T15:02:30.914475+0000","last_active":"2024-01-16T15:02:31.923471+0000","last_peered":"2024-01-16T15:02:31.923471+0000","last_clean":"2024-01-16T15:02:31.923471+0000","last_became_active":"2024-01-16T15:02:30.914116+0000","last_became_peered":"2024-01-16T15:02:30.914116+0000","last_unstale":"2024-01-16T15:02:31.923471+0000","last_undegraded":"2024-01-16T15:02:31.923471+0000","last_fullsized":"2024-01-16T15:02:31.923471+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:07.623994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.916672+0000","last_change":"2024-01-16T15:02:31.931670+0000","last_active":"2024-01-16T15:02:32.916672+0000","last_peered":"2024-01-16T15:02:32.916672+0000","last_clean":"2024-01-16T15:02:32.916672+0000","last_became_active":"2024-01-16T15:02:31.931382+0000","last_became_peered":"2024-01-16T15:02:31.931382+0000","last_unstale":"2024-01-16T15:02:32.916672+0000","last_undegraded":"2024-01-16T15:02:32.916672+0000","last_fullsized":"2024-01-16T15:02:32.916672+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:09.635701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.930628+0000","last_change":"2024-01-16T15:02:30.917270+0000","last_active":"2024-01-16T15:02:31.930628+0000","last_peered":"2024-01-16T15:02:31.930628+0000","last_clean":"2024-01-16T15:02:31.930628+0000","last_became_active":"2024-01-16T15:02:30.916963+0000","last_became_peered":"2024-01-16T15:02:30.916963+0000","last_unstale":"2024-01-16T15:02:31.930628+0000","last_undegraded":"2024-01-16T15:02:31.930628+0000","last_fullsized":"2024-01-16T15:02:31.930628+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:42.871763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157013+0000","last_change":"2024-01-16T15:02:31.930441+0000","last_active":"2024-01-16T15:02:33.157013+0000","last_peered":"2024-01-16T15:02:33.157013+0000","last_clean":"2024-01-16T15:02:33.157013+0000","last_became_active":"2024-01-16T15:02:31.926603+0000","last_became_peered":"2024-01-16T15:02:31.926603+0000","last_unstale":"2024-01-16T15:02:33.157013+0000","last_undegraded":"2024-01-16T15:02:33.157013+0000","last_fullsized":"2024-01-16T15:02:33.157013+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:15.732991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.927709+0000","last_change":"2024-01-16T15:02:30.921988+0000","last_active":"2024-01-16T15:02:31.927709+0000","last_peered":"2024-01-16T15:02:31.927709+0000","last_clean":"2024-01-16T15:02:31.927709+0000","last_became_active":"2024-01-16T15:02:30.921668+0000","last_became_peered":"2024-01-16T15:02:30.921668+0000","last_unstale":"2024-01-16T15:02:31.927709+0000","last_undegraded":"2024-01-16T15:02:31.927709+0000","last_fullsized":"2024-01-16T15:02:31.927709+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:10.367214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156051+0000","last_change":"2024-01-16T15:02:31.922670+0000","last_active":"2024-01-16T15:02:33.156051+0000","last_peered":"2024-01-16T15:02:33.156051+0000","last_clean":"2024-01-16T15:02:33.156051+0000","last_became_active":"2024-01-16T15:02:31.922406+0000","last_became_peered":"2024-01-16T15:02:31.922406+0000","last_unstale":"2024-01-16T15:02:33.156051+0000","last_undegraded":"2024-01-16T15:02:33.156051+0000","last_fullsized":"2024-01-16T15:02:33.156051+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:29.833091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155826+0000","last_change":"2024-01-16T15:02:30.908704+0000","last_active":"2024-01-16T15:02:33.155826+0000","last_peered":"2024-01-16T15:02:33.155826+0000","last_clean":"2024-01-16T15:02:33.155826+0000","last_became_active":"2024-01-16T15:02:30.908404+0000","last_became_peered":"2024-01-16T15:02:30.908404+0000","last_unstale":"2024-01-16T15:02:33.155826+0000","last_undegraded":"2024-01-16T15:02:33.155826+0000","last_fullsized":"2024-01-16T15:02:33.155826+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:04.230373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.159354+0000","last_change":"2024-01-16T15:02:31.917238+0000","last_active":"2024-01-16T15:02:33.159354+0000","last_peered":"2024-01-16T15:02:33.159354+0000","last_clean":"2024-01-16T15:02:33.159354+0000","last_became_active":"2024-01-16T15:02:31.916934+0000","last_became_peered":"2024-01-16T15:02:31.916934+0000","last_unstale":"2024-01-16T15:02:33.159354+0000","last_undegraded":"2024-01-16T15:02:33.159354+0000","last_fullsized":"2024-01-16T15:02:33.159354+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:44.440360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.928865+0000","last_change":"2024-01-16T15:02:30.912893+0000","last_active":"2024-01-16T15:02:31.928865+0000","last_peered":"2024-01-16T15:02:31.928865+0000","last_clean":"2024-01-16T15:02:31.928865+0000","last_became_active":"2024-01-16T15:02:30.911643+0000","last_became_peered":"2024-01-16T15:02:30.911643+0000","last_unstale":"2024-01-16T15:02:31.928865+0000","last_undegraded":"2024-01-16T15:02:31.928865+0000","last_fullsized":"2024-01-16T15:02:31.928865+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:14:54.009427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158067+0000","last_change":"2024-01-16T15:02:31.930740+0000","last_active":"2024-01-16T15:02:33.158067+0000","last_peered":"2024-01-16T15:02:33.158067+0000","last_clean":"2024-01-16T15:02:33.158067+0000","last_became_active":"2024-01-16T15:02:31.930121+0000","last_became_peered":"2024-01-16T15:02:31.930121+0000","last_unstale":"2024-01-16T15:02:33.158067+0000","last_undegraded":"2024-01-16T15:02:33.158067+0000","last_fullsized":"2024-01-16T15:02:33.158067+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:06:11.675587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.914478+0000","last_change":"2024-01-16T15:02:30.917356+0000","last_active":"2024-01-16T15:02:32.914478+0000","last_peered":"2024-01-16T15:02:32.914478+0000","last_clean":"2024-01-16T15:02:32.914478+0000","last_became_active":"2024-01-16T15:02:30.916785+0000","last_became_peered":"2024-01-16T15:02:30.916785+0000","last_unstale":"2024-01-16T15:02:32.914478+0000","last_undegraded":"2024-01-16T15:02:32.914478+0000","last_fullsized":"2024-01-16T15:02:32.914478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:16:38.001653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.914938+0000","last_change":"2024-01-16T15:02:31.924398+0000","last_active":"2024-01-16T15:02:32.914938+0000","last_peered":"2024-01-16T15:02:32.914938+0000","last_clean":"2024-01-16T15:02:32.914938+0000","last_became_active":"2024-01-16T15:02:31.923690+0000","last_became_peered":"2024-01-16T15:02:31.923690+0000","last_unstale":"2024-01-16T15:02:32.914938+0000","last_undegraded":"2024-01-16T15:02:32.914938+0000","last_fullsized":"2024-01-16T15:02:32.914938+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:59.676955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157969+0000","last_change":"2024-01-16T15:02:30.910065+0000","last_active":"2024-01-16T15:02:33.157969+0000","last_peered":"2024-01-16T15:02:33.157969+0000","last_clean":"2024-01-16T15:02:33.157969+0000","last_became_active":"2024-01-16T15:02:30.909785+0000","last_became_peered":"2024-01-16T15:02:30.909785+0000","last_unstale":"2024-01-16T15:02:33.157969+0000","last_undegraded":"2024-01-16T15:02:33.157969+0000","last_fullsized":"2024-01-16T15:02:33.157969+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:19.294467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156792+0000","last_change":"2024-01-16T15:02:31.934889+0000","last_active":"2024-01-16T15:02:33.156792+0000","last_peered":"2024-01-16T15:02:33.156792+0000","last_clean":"2024-01-16T15:02:33.156792+0000","last_became_active":"2024-01-16T15:02:31.934613+0000","last_became_peered":"2024-01-16T15:02:31.934613+0000","last_unstale":"2024-01-16T15:02:33.156792+0000","last_undegraded":"2024-01-16T15:02:33.156792+0000","last_fullsized":"2024-01-16T15:02:33.156792+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:16:39.017461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.902581+0000","last_change":"2024-01-16T15:02:30.905124+0000","last_active":"2024-01-16T15:02:31.902581+0000","last_peered":"2024-01-16T15:02:31.902581+0000","last_clean":"2024-01-16T15:02:31.902581+0000","last_became_active":"2024-01-16T15:02:30.904687+0000","last_became_peered":"2024-01-16T15:02:30.904687+0000","last_unstale":"2024-01-16T15:02:31.902581+0000","last_undegraded":"2024-01-16T15:02:31.902581+0000","last_fullsized":"2024-01-16T15:02:31.902581+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:13.014312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157818+0000","last_change":"2024-01-16T15:02:31.927993+0000","last_active":"2024-01-16T15:02:33.157818+0000","last_peered":"2024-01-16T15:02:33.157818+0000","last_clean":"2024-01-16T15:02:33.157818+0000","last_became_active":"2024-01-16T15:02:31.927731+0000","last_became_peered":"2024-01-16T15:02:31.927731+0000","last_unstale":"2024-01-16T15:02:33.157818+0000","last_undegraded":"2024-01-16T15:02:33.157818+0000","last_fullsized":"2024-01-16T15:02:33.157818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:00.582744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160661+0000","last_change":"2024-01-16T15:02:30.917166+0000","last_active":"2024-01-16T15:02:33.160661+0000","last_peered":"2024-01-16T15:02:33.160661+0000","last_clean":"2024-01-16T15:02:33.160661+0000","last_became_active":"2024-01-16T15:02:30.916866+0000","last_became_peered":"2024-01-16T15:02:30.916866+0000","last_unstale":"2024-01-16T15:02:33.160661+0000","last_undegraded":"2024-01-16T15:02:33.160661+0000","last_fullsized":"2024-01-16T15:02:33.160661+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:42.216793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156208+0000","last_change":"2024-01-16T15:02:31.915572+0000","last_active":"2024-01-16T15:02:33.156208+0000","last_peered":"2024-01-16T15:02:33.156208+0000","last_clean":"2024-01-16T15:02:33.156208+0000","last_became_active":"2024-01-16T15:02:31.915227+0000","last_became_peered":"2024-01-16T15:02:31.915227+0000","last_unstale":"2024-01-16T15:02:33.156208+0000","last_undegraded":"2024-01-16T15:02:33.156208+0000","last_fullsized":"2024-01-16T15:02:33.156208+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:23:57.570666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"46'3","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.931122+0000","last_change":"2024-01-16T15:02:30.919698+0000","last_active":"2024-01-16T15:02:31.931122+0000","last_peered":"2024-01-16T15:02:31.931122+0000","last_clean":"2024-01-16T15:02:31.931122+0000","last_became_active":"2024-01-16T15:02:30.919302+0000","last_became_peered":"2024-01-16T15:02:30.919302+0000","last_unstale":"2024-01-16T15:02:31.931122+0000","last_undegraded":"2024-01-16T15:02:31.931122+0000","last_fullsized":"2024-01-16T15:02:31.931122+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:54.069880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157356+0000","last_change":"2024-01-16T15:02:31.938880+0000","last_active":"2024-01-16T15:02:33.157356+0000","last_peered":"2024-01-16T15:02:33.157356+0000","last_clean":"2024-01-16T15:02:33.157356+0000","last_became_active":"2024-01-16T15:02:31.938467+0000","last_became_peered":"2024-01-16T15:02:31.938467+0000","last_unstale":"2024-01-16T15:02:33.157356+0000","last_undegraded":"2024-01-16T15:02:33.157356+0000","last_fullsized":"2024-01-16T15:02:33.157356+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:00.619935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"46'6","reported_seq":18,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912546+0000","last_change":"2024-01-16T15:02:30.903291+0000","last_active":"2024-01-16T15:02:32.912546+0000","last_peered":"2024-01-16T15:02:32.912546+0000","last_clean":"2024-01-16T15:02:32.912546+0000","last_became_active":"2024-01-16T15:02:30.902952+0000","last_became_peered":"2024-01-16T15:02:30.902952+0000","last_unstale":"2024-01-16T15:02:32.912546+0000","last_undegraded":"2024-01-16T15:02:32.912546+0000","last_fullsized":"2024-01-16T15:02:32.912546+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:22:52.211724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.914946+0000","last_change":"2024-01-16T15:02:31.916067+0000","last_active":"2024-01-16T15:02:32.914946+0000","last_peered":"2024-01-16T15:02:32.914946+0000","last_clean":"2024-01-16T15:02:32.914946+0000","last_became_active":"2024-01-16T15:02:31.915672+0000","last_became_peered":"2024-01-16T15:02:31.915672+0000","last_unstale":"2024-01-16T15:02:32.914946+0000","last_undegraded":"2024-01-16T15:02:32.914946+0000","last_fullsized":"2024-01-16T15:02:32.914946+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:26.604551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155927+0000","last_change":"2024-01-16T15:02:30.909835+0000","last_active":"2024-01-16T15:02:33.155927+0000","last_peered":"2024-01-16T15:02:33.155927+0000","last_clean":"2024-01-16T15:02:33.155927+0000","last_became_active":"2024-01-16T15:02:30.909539+0000","last_became_peered":"2024-01-16T15:02:30.909539+0000","last_unstale":"2024-01-16T15:02:33.155927+0000","last_undegraded":"2024-01-16T15:02:33.155927+0000","last_fullsized":"2024-01-16T15:02:33.155927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:56.831958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.159551+0000","last_change":"2024-01-16T15:02:31.922844+0000","last_active":"2024-01-16T15:02:33.159551+0000","last_peered":"2024-01-16T15:02:33.159551+0000","last_clean":"2024-01-16T15:02:33.159551+0000","last_became_active":"2024-01-16T15:02:31.922121+0000","last_became_peered":"2024-01-16T15:02:31.922121+0000","last_unstale":"2024-01-16T15:02:33.159551+0000","last_undegraded":"2024-01-16T15:02:33.159551+0000","last_fullsized":"2024-01-16T15:02:33.159551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:41.209200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.913465+0000","last_change":"2024-01-16T15:02:30.919149+0000","last_active":"2024-01-16T15:02:31.913465+0000","last_peered":"2024-01-16T15:02:31.913465+0000","last_clean":"2024-01-16T15:02:31.913465+0000","last_became_active":"2024-01-16T15:02:30.918870+0000","last_became_peered":"2024-01-16T15:02:30.918870+0000","last_unstale":"2024-01-16T15:02:31.913465+0000","last_undegraded":"2024-01-16T15:02:31.913465+0000","last_fullsized":"2024-01-16T15:02:31.913465+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:46.619513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.154775+0000","last_change":"2024-01-16T15:02:31.919314+0000","last_active":"2024-01-16T15:02:33.154775+0000","last_peered":"2024-01-16T15:02:33.154775+0000","last_clean":"2024-01-16T15:02:33.154775+0000","last_became_active":"2024-01-16T15:02:31.918894+0000","last_became_peered":"2024-01-16T15:02:31.918894+0000","last_unstale":"2024-01-16T15:02:33.154775+0000","last_undegraded":"2024-01-16T15:02:33.154775+0000","last_fullsized":"2024-01-16T15:02:33.154775+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:00.619935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155754+0000","last_change":"2024-01-16T15:02:30.914398+0000","last_active":"2024-01-16T15:02:33.155754+0000","last_peered":"2024-01-16T15:02:33.155754+0000","last_clean":"2024-01-16T15:02:33.155754+0000","last_became_active":"2024-01-16T15:02:30.913286+0000","last_became_peered":"2024-01-16T15:02:30.913286+0000","last_unstale":"2024-01-16T15:02:33.155754+0000","last_undegraded":"2024-01-16T15:02:33.155754+0000","last_fullsized":"2024-01-16T15:02:33.155754+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:34.818086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156137+0000","last_change":"2024-01-16T15:02:31.926650+0000","last_active":"2024-01-16T15:02:33.156137+0000","last_peered":"2024-01-16T15:02:33.156137+0000","last_clean":"2024-01-16T15:02:33.156137+0000","last_became_active":"2024-01-16T15:02:31.926299+0000","last_became_peered":"2024-01-16T15:02:31.926299+0000","last_unstale":"2024-01-16T15:02:33.156137+0000","last_undegraded":"2024-01-16T15:02:33.156137+0000","last_fullsized":"2024-01-16T15:02:33.156137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:57.847766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"46'8","reported_seq":20,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158354+0000","last_change":"2024-01-16T15:02:30.911880+0000","last_active":"2024-01-16T15:02:33.158354+0000","last_peered":"2024-01-16T15:02:33.158354+0000","last_clean":"2024-01-16T15:02:33.158354+0000","last_became_active":"2024-01-16T15:02:30.911642+0000","last_became_peered":"2024-01-16T15:02:30.911642+0000","last_unstale":"2024-01-16T15:02:33.158354+0000","last_undegraded":"2024-01-16T15:02:33.158354+0000","last_fullsized":"2024-01-16T15:02:33.158354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:07.623994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912282+0000","last_change":"2024-01-16T15:02:31.924651+0000","last_active":"2024-01-16T15:02:32.912282+0000","last_peered":"2024-01-16T15:02:32.912282+0000","last_clean":"2024-01-16T15:02:32.912282+0000","last_became_active":"2024-01-16T15:02:31.924361+0000","last_became_peered":"2024-01-16T15:02:31.924361+0000","last_unstale":"2024-01-16T15:02:32.912282+0000","last_undegraded":"2024-01-16T15:02:32.912282+0000","last_fullsized":"2024-01-16T15:02:32.912282+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:10:40.064638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.912314+0000","last_change":"2024-01-16T15:02:30.908418+0000","last_active":"2024-01-16T15:02:32.912314+0000","last_peered":"2024-01-16T15:02:32.912314+0000","last_clean":"2024-01-16T15:02:32.912314+0000","last_became_active":"2024-01-16T15:02:30.905650+0000","last_became_peered":"2024-01-16T15:02:30.905650+0000","last_unstale":"2024-01-16T15:02:32.912314+0000","last_undegraded":"2024-01-16T15:02:32.912314+0000","last_fullsized":"2024-01-16T15:02:32.912314+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:18.132044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160154+0000","last_change":"2024-01-16T15:02:31.923375+0000","last_active":"2024-01-16T15:02:33.160154+0000","last_peered":"2024-01-16T15:02:33.160154+0000","last_clean":"2024-01-16T15:02:33.160154+0000","last_became_active":"2024-01-16T15:02:31.920898+0000","last_became_peered":"2024-01-16T15:02:31.920898+0000","last_unstale":"2024-01-16T15:02:33.160154+0000","last_undegraded":"2024-01-16T15:02:33.160154+0000","last_fullsized":"2024-01-16T15:02:33.160154+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:24.838719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.924868+0000","last_change":"2024-01-16T15:02:30.907115+0000","last_active":"2024-01-16T15:02:31.924868+0000","last_peered":"2024-01-16T15:02:31.924868+0000","last_clean":"2024-01-16T15:02:31.924868+0000","last_became_active":"2024-01-16T15:02:30.905284+0000","last_became_peered":"2024-01-16T15:02:30.905284+0000","last_unstale":"2024-01-16T15:02:31.924868+0000","last_undegraded":"2024-01-16T15:02:31.924868+0000","last_fullsized":"2024-01-16T15:02:31.924868+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:58.661148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915838+0000","last_change":"2024-01-16T15:02:31.924656+0000","last_active":"2024-01-16T15:02:32.915838+0000","last_peered":"2024-01-16T15:02:32.915838+0000","last_clean":"2024-01-16T15:02:32.915838+0000","last_became_active":"2024-01-16T15:02:31.924375+0000","last_became_peered":"2024-01-16T15:02:31.924375+0000","last_unstale":"2024-01-16T15:02:32.915838+0000","last_undegraded":"2024-01-16T15:02:32.915838+0000","last_fullsized":"2024-01-16T15:02:32.915838+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:27.286876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.917022+0000","last_change":"2024-01-16T15:02:30.916129+0000","last_active":"2024-01-16T15:02:32.917022+0000","last_peered":"2024-01-16T15:02:32.917022+0000","last_clean":"2024-01-16T15:02:32.917022+0000","last_became_active":"2024-01-16T15:02:30.915745+0000","last_became_peered":"2024-01-16T15:02:30.915745+0000","last_unstale":"2024-01-16T15:02:32.917022+0000","last_undegraded":"2024-01-16T15:02:32.917022+0000","last_fullsized":"2024-01-16T15:02:32.917022+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:47.253521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915113+0000","last_change":"2024-01-16T15:02:31.914276+0000","last_active":"2024-01-16T15:02:32.915113+0000","last_peered":"2024-01-16T15:02:32.915113+0000","last_clean":"2024-01-16T15:02:32.915113+0000","last_became_active":"2024-01-16T15:02:31.913877+0000","last_became_peered":"2024-01-16T15:02:31.913877+0000","last_unstale":"2024-01-16T15:02:32.915113+0000","last_undegraded":"2024-01-16T15:02:32.915113+0000","last_fullsized":"2024-01-16T15:02:32.915113+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:14.030119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156209+0000","last_change":"2024-01-16T15:02:30.915103+0000","last_active":"2024-01-16T15:02:33.156209+0000","last_peered":"2024-01-16T15:02:33.156209+0000","last_clean":"2024-01-16T15:02:33.156209+0000","last_became_active":"2024-01-16T15:02:30.914771+0000","last_became_peered":"2024-01-16T15:02:30.914771+0000","last_unstale":"2024-01-16T15:02:33.156209+0000","last_undegraded":"2024-01-16T15:02:33.156209+0000","last_fullsized":"2024-01-16T15:02:33.156209+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:27.854951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155844+0000","last_change":"2024-01-16T15:02:31.920433+0000","last_active":"2024-01-16T15:02:33.155844+0000","last_peered":"2024-01-16T15:02:33.155844+0000","last_clean":"2024-01-16T15:02:33.155844+0000","last_became_active":"2024-01-16T15:02:31.919915+0000","last_became_peered":"2024-01-16T15:02:31.919915+0000","last_unstale":"2024-01-16T15:02:33.155844+0000","last_undegraded":"2024-01-16T15:02:33.155844+0000","last_fullsized":"2024-01-16T15:02:33.155844+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:49.329842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156132+0000","last_change":"2024-01-16T15:02:30.913950+0000","last_active":"2024-01-16T15:02:33.156132+0000","last_peered":"2024-01-16T15:02:33.156132+0000","last_clean":"2024-01-16T15:02:33.156132+0000","last_became_active":"2024-01-16T15:02:30.913636+0000","last_became_peered":"2024-01-16T15:02:30.913636+0000","last_unstale":"2024-01-16T15:02:33.156132+0000","last_undegraded":"2024-01-16T15:02:33.156132+0000","last_fullsized":"2024-01-16T15:02:33.156132+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:46.284971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157431+0000","last_change":"2024-01-16T15:02:31.922664+0000","last_active":"2024-01-16T15:02:33.157431+0000","last_peered":"2024-01-16T15:02:33.157431+0000","last_clean":"2024-01-16T15:02:33.157431+0000","last_became_active":"2024-01-16T15:02:31.922388+0000","last_became_peered":"2024-01-16T15:02:31.922388+0000","last_unstale":"2024-01-16T15:02:33.157431+0000","last_undegraded":"2024-01-16T15:02:33.157431+0000","last_fullsized":"2024-01-16T15:02:33.157431+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:29.833091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915177+0000","last_change":"2024-01-16T15:02:30.910659+0000","last_active":"2024-01-16T15:02:32.915177+0000","last_peered":"2024-01-16T15:02:32.915177+0000","last_clean":"2024-01-16T15:02:32.915177+0000","last_became_active":"2024-01-16T15:02:30.910250+0000","last_became_peered":"2024-01-16T15:02:30.910250+0000","last_unstale":"2024-01-16T15:02:32.915177+0000","last_undegraded":"2024-01-16T15:02:32.915177+0000","last_fullsized":"2024-01-16T15:02:32.915177+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:48.298618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.911801+0000","last_change":"2024-01-16T15:02:31.932622+0000","last_active":"2024-01-16T15:02:32.911801+0000","last_peered":"2024-01-16T15:02:32.911801+0000","last_clean":"2024-01-16T15:02:32.911801+0000","last_became_active":"2024-01-16T15:02:31.931640+0000","last_became_peered":"2024-01-16T15:02:31.931640+0000","last_unstale":"2024-01-16T15:02:32.911801+0000","last_undegraded":"2024-01-16T15:02:32.911801+0000","last_fullsized":"2024-01-16T15:02:32.911801+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:29.531111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158552+0000","last_change":"2024-01-16T15:02:30.914739+0000","last_active":"2024-01-16T15:02:33.158552+0000","last_peered":"2024-01-16T15:02:33.158552+0000","last_clean":"2024-01-16T15:02:33.158552+0000","last_became_active":"2024-01-16T15:02:30.914440+0000","last_became_peered":"2024-01-16T15:02:30.914440+0000","last_unstale":"2024-01-16T15:02:33.158552+0000","last_undegraded":"2024-01-16T15:02:33.158552+0000","last_fullsized":"2024-01-16T15:02:33.158552+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:42.871763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157146+0000","last_change":"2024-01-16T15:02:31.920052+0000","last_active":"2024-01-16T15:02:33.157146+0000","last_peered":"2024-01-16T15:02:33.157146+0000","last_clean":"2024-01-16T15:02:33.157146+0000","last_became_active":"2024-01-16T15:02:31.917969+0000","last_became_peered":"2024-01-16T15:02:31.917969+0000","last_unstale":"2024-01-16T15:02:33.157146+0000","last_undegraded":"2024-01-16T15:02:33.157146+0000","last_fullsized":"2024-01-16T15:02:33.157146+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:31.514083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.912786+0000","last_change":"2024-01-16T15:02:30.917441+0000","last_active":"2024-01-16T15:02:31.912786+0000","last_peered":"2024-01-16T15:02:31.912786+0000","last_clean":"2024-01-16T15:02:31.912786+0000","last_became_active":"2024-01-16T15:02:30.917036+0000","last_became_peered":"2024-01-16T15:02:30.917036+0000","last_unstale":"2024-01-16T15:02:31.912786+0000","last_undegraded":"2024-01-16T15:02:31.912786+0000","last_fullsized":"2024-01-16T15:02:31.912786+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:34.818086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915414+0000","last_change":"2024-01-16T15:02:31.931001+0000","last_active":"2024-01-16T15:02:32.915414+0000","last_peered":"2024-01-16T15:02:32.915414+0000","last_clean":"2024-01-16T15:02:32.915414+0000","last_became_active":"2024-01-16T15:02:31.930659+0000","last_became_peered":"2024-01-16T15:02:31.930659+0000","last_unstale":"2024-01-16T15:02:32.915414+0000","last_undegraded":"2024-01-16T15:02:32.915414+0000","last_fullsized":"2024-01-16T15:02:32.915414+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:43.887571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156393+0000","last_change":"2024-01-16T15:02:30.908784+0000","last_active":"2024-01-16T15:02:33.156393+0000","last_peered":"2024-01-16T15:02:33.156393+0000","last_clean":"2024-01-16T15:02:33.156393+0000","last_became_active":"2024-01-16T15:02:30.908180+0000","last_became_peered":"2024-01-16T15:02:30.908180+0000","last_unstale":"2024-01-16T15:02:33.156393+0000","last_undegraded":"2024-01-16T15:02:33.156393+0000","last_fullsized":"2024-01-16T15:02:33.156393+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:46.619513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.159822+0000","last_change":"2024-01-16T15:02:31.929759+0000","last_active":"2024-01-16T15:02:33.159822+0000","last_peered":"2024-01-16T15:02:33.159822+0000","last_clean":"2024-01-16T15:02:33.159822+0000","last_became_active":"2024-01-16T15:02:31.929357+0000","last_became_peered":"2024-01-16T15:02:31.929357+0000","last_unstale":"2024-01-16T15:02:33.159822+0000","last_undegraded":"2024-01-16T15:02:33.159822+0000","last_fullsized":"2024-01-16T15:02:33.159822+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:00.437432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.916424+0000","last_change":"2024-01-16T15:02:30.921967+0000","last_active":"2024-01-16T15:02:31.916424+0000","last_peered":"2024-01-16T15:02:31.916424+0000","last_clean":"2024-01-16T15:02:31.916424+0000","last_became_active":"2024-01-16T15:02:30.921549+0000","last_became_peered":"2024-01-16T15:02:30.921549+0000","last_unstale":"2024-01-16T15:02:31.916424+0000","last_undegraded":"2024-01-16T15:02:31.916424+0000","last_fullsized":"2024-01-16T15:02:31.916424+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:26.946797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.157564+0000","last_change":"2024-01-16T15:02:31.934174+0000","last_active":"2024-01-16T15:02:33.157564+0000","last_peered":"2024-01-16T15:02:33.157564+0000","last_clean":"2024-01-16T15:02:33.157564+0000","last_became_active":"2024-01-16T15:02:31.933946+0000","last_became_peered":"2024-01-16T15:02:31.933946+0000","last_unstale":"2024-01-16T15:02:33.157564+0000","last_undegraded":"2024-01-16T15:02:33.157564+0000","last_fullsized":"2024-01-16T15:02:33.157564+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:11.814450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.900550+0000","last_change":"2024-01-16T15:02:30.911362+0000","last_active":"2024-01-16T15:02:31.900550+0000","last_peered":"2024-01-16T15:02:31.900550+0000","last_clean":"2024-01-16T15:02:31.900550+0000","last_became_active":"2024-01-16T15:02:30.911117+0000","last_became_peered":"2024-01-16T15:02:30.911117+0000","last_unstale":"2024-01-16T15:02:31.900550+0000","last_undegraded":"2024-01-16T15:02:31.900550+0000","last_fullsized":"2024-01-16T15:02:31.900550+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:13.014312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.156433+0000","last_change":"2024-01-16T15:02:31.920639+0000","last_active":"2024-01-16T15:02:33.156433+0000","last_peered":"2024-01-16T15:02:33.156433+0000","last_clean":"2024-01-16T15:02:33.156433+0000","last_became_active":"2024-01-16T15:02:31.920028+0000","last_became_peered":"2024-01-16T15:02:31.920028+0000","last_unstale":"2024-01-16T15:02:33.156433+0000","last_undegraded":"2024-01-16T15:02:33.156433+0000","last_fullsized":"2024-01-16T15:02:33.156433+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:25.179628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.912639+0000","last_change":"2024-01-16T15:02:30.919597+0000","last_active":"2024-01-16T15:02:31.912639+0000","last_peered":"2024-01-16T15:02:31.912639+0000","last_clean":"2024-01-16T15:02:31.912639+0000","last_became_active":"2024-01-16T15:02:30.919298+0000","last_became_peered":"2024-01-16T15:02:30.919298+0000","last_unstale":"2024-01-16T15:02:31.912639+0000","last_undegraded":"2024-01-16T15:02:31.912639+0000","last_fullsized":"2024-01-16T15:02:31.912639+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:27.854951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155724+0000","last_change":"2024-01-16T15:02:31.917030+0000","last_active":"2024-01-16T15:02:33.155724+0000","last_peered":"2024-01-16T15:02:33.155724+0000","last_clean":"2024-01-16T15:02:33.155724+0000","last_became_active":"2024-01-16T15:02:31.916718+0000","last_became_peered":"2024-01-16T15:02:31.916718+0000","last_unstale":"2024-01-16T15:02:33.155724+0000","last_undegraded":"2024-01-16T15:02:33.155724+0000","last_fullsized":"2024-01-16T15:02:33.155724+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:40.506239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.897977+0000","last_change":"2024-01-16T15:02:30.914897+0000","last_active":"2024-01-16T15:02:31.897977+0000","last_peered":"2024-01-16T15:02:31.897977+0000","last_clean":"2024-01-16T15:02:31.897977+0000","last_became_active":"2024-01-16T15:02:30.914508+0000","last_became_peered":"2024-01-16T15:02:30.914508+0000","last_unstale":"2024-01-16T15:02:31.897977+0000","last_undegraded":"2024-01-16T15:02:31.897977+0000","last_fullsized":"2024-01-16T15:02:31.897977+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:23.629145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.915540+0000","last_change":"2024-01-16T15:02:31.925171+0000","last_active":"2024-01-16T15:02:32.915540+0000","last_peered":"2024-01-16T15:02:32.915540+0000","last_clean":"2024-01-16T15:02:32.915540+0000","last_became_active":"2024-01-16T15:02:31.924914+0000","last_became_peered":"2024-01-16T15:02:31.924914+0000","last_unstale":"2024-01-16T15:02:32.915540+0000","last_undegraded":"2024-01-16T15:02:32.915540+0000","last_fullsized":"2024-01-16T15:02:32.915540+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:00.437432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.899913+0000","last_change":"2024-01-16T15:02:30.912654+0000","last_active":"2024-01-16T15:02:31.899913+0000","last_peered":"2024-01-16T15:02:31.899913+0000","last_clean":"2024-01-16T15:02:31.899913+0000","last_became_active":"2024-01-16T15:02:30.910154+0000","last_became_peered":"2024-01-16T15:02:30.910154+0000","last_unstale":"2024-01-16T15:02:31.899913+0000","last_undegraded":"2024-01-16T15:02:31.899913+0000","last_fullsized":"2024-01-16T15:02:31.899913+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:07.623994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.155721+0000","last_change":"2024-01-16T15:02:31.922738+0000","last_active":"2024-01-16T15:02:33.155721+0000","last_peered":"2024-01-16T15:02:33.155721+0000","last_clean":"2024-01-16T15:02:33.155721+0000","last_became_active":"2024-01-16T15:02:31.922398+0000","last_became_peered":"2024-01-16T15:02:31.922398+0000","last_unstale":"2024-01-16T15:02:33.155721+0000","last_undegraded":"2024-01-16T15:02:33.155721+0000","last_fullsized":"2024-01-16T15:02:33.155721+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:43.887571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.158392+0000","last_change":"2024-01-16T15:02:30.909935+0000","last_active":"2024-01-16T15:02:33.158392+0000","last_peered":"2024-01-16T15:02:33.158392+0000","last_clean":"2024-01-16T15:02:33.158392+0000","last_became_active":"2024-01-16T15:02:30.909674+0000","last_became_peered":"2024-01-16T15:02:30.909674+0000","last_unstale":"2024-01-16T15:02:33.158392+0000","last_undegraded":"2024-01-16T15:02:33.158392+0000","last_fullsized":"2024-01-16T15:02:33.158392+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:07:25.588744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:33.160540+0000","last_change":"2024-01-16T15:02:31.933459+0000","last_active":"2024-01-16T15:02:33.160540+0000","last_peered":"2024-01-16T15:02:33.160540+0000","last_clean":"2024-01-16T15:02:33.160540+0000","last_became_active":"2024-01-16T15:02:31.928625+0000","last_became_peered":"2024-01-16T15:02:31.928625+0000","last_unstale":"2024-01-16T15:02:33.160540+0000","last_undegraded":"2024-01-16T15:02:33.160540+0000","last_fullsized":"2024-01-16T15:02:33.160540+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:30.866369+0000","last_clean_scrub_stamp":"2024-01-16T15:02:30.866369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:10.625725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:02:32.917248+0000","last_change":"2024-01-16T15:02:30.917015+0000","last_active":"2024-01-16T15:02:32.917248+0000","last_peered":"2024-01-16T15:02:32.917248+0000","last_clean":"2024-01-16T15:02:32.917248+0000","last_became_active":"2024-01-16T15:02:30.916706+0000","last_became_peered":"2024-01-16T15:02:30.916706+0000","last_unstale":"2024-01-16T15:02:32.917248+0000","last_undegraded":"2024-01-16T15:02:32.917248+0000","last_fullsized":"2024-01-16T15:02:32.917248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:02:29.850562+0000","last_clean_scrub_stamp":"2024-01-16T15:02:29.850562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:05:43.424552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":185,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:02:31.912883+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:02:31.912883+0000","last_peered":"2024-01-16T15:02:31.912883+0000","last_clean":"2024-01-16T15:02:31.912883+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:02:31.912883+0000","last_undegraded":"2024-01-16T15:02:31.912883+0000","last_fullsized":"2024-01-16T15:02:31.912883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1146880,"data_stored":813441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.145,"15min":1.127},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.2529999999999999,"15min":1.22},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":6.2229999999999999,"5min":6.2229999999999999,"15min":6.2229999999999999},"last":0.95799999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.222,"15min":1.212},"min":{"1min":0.69899999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.1850000000000001,"15min":1.171},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":1.2729999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.4299999999999999,"15min":1.3240000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":10.707000000000001,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":2.0489999999999999,"5min":1.4430000000000001,"15min":1.3400000000000001},"min":{"1min":0.95199999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":10.661,"5min":10.661,"15min":10.661},"last":1.1379999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.3220000000000001,"15min":1.2909999999999999},"min":{"1min":0.92700000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.4339999999999999,"15min":1.4199999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":2.194,"5min":1.5249999999999999,"15min":1.4350000000000001},"min":{"1min":0.64900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":16.376000000000001,"5min":16.376000000000001,"15min":16.376000000000001},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.4570000000000001,"15min":1.4890000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":5.9720000000000004,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.0509999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.5109999999999999,"15min":1.5549999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.6259999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.494},{"interface":"front","average":{"1min":1.595,"5min":1.554,"15min":1.571},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":6.319,"5min":6.319,"15min":6.319},"last":1.304}]},{"osd":6,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.7190000000000001,"15min":1.712},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":8.2989999999999995,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.512},{"interface":"front","average":{"1min":1.857,"5min":1.6859999999999999,"15min":1.698},"min":{"1min":0.999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":8.4979999999999993,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.2070000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31732,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340108,"statfs":{"total":96636764160,"available":96604270592,"internally_reserved":0,"allocated":557056,"data_stored":223655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1679999999999999,"15min":1.1579999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.512},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.21,"15min":1.165},"min":{"1min":0.74099999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.2650000000000001,"5min":3.2650000000000001,"15min":3.2650000000000001},"last":1.5900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.079,"15min":1.0740000000000001},"min":{"1min":0.68999999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":1.476},{"interface":"front","average":{"1min":1.198,"5min":1.1479999999999999,"15min":1.1259999999999999},"min":{"1min":0.77100000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.3540000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.23,"15min":1.2250000000000001},"min":{"1min":0.72999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.5089999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.3939999999999999,"15min":1.411},"min":{"1min":0.83099999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3719999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":0.84199999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.2390000000000001},"min":{"1min":0.84699999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.849},{"interface":"front","average":{"1min":1.504,"5min":1.323,"15min":1.288},"min":{"1min":0.91000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":1.143}]},{"osd":4,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":1.478,"15min":1.3700000000000001},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":15.481999999999999,"5min":15.481999999999999,"15min":15.481999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.329,"5min":1.4359999999999999,"15min":1.4239999999999999},"min":{"1min":0.66300000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":4.4409999999999998,"5min":6.585,"15min":6.585},"last":1.6379999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.325,"15min":1.3069999999999999},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.3860000000000001,"5min":3.3860000000000001,"15min":3.3860000000000001},"last":1.296},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5409999999999999,"15min":1.5529999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.8380000000000001,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":1.7450000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.4790000000000001,"15min":1.4550000000000001},"min":{"1min":0.61299999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.7},{"interface":"front","average":{"1min":1.423,"5min":1.538,"15min":1.5840000000000001},"min":{"1min":0.80200000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.1280000000000001,"5min":3.1389999999999998,"15min":3.1389999999999998},"last":1.224}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":552960,"data_stored":223053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.135,"15min":1.145},"min":{"1min":0.54000000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.5489999999999999,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.1399999999999999,"15min":1.0589999999999999},"min":{"1min":0.68999999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":7.5789999999999997,"5min":7.5789999999999997,"15min":7.5789999999999997},"last":1.393}]},{"osd":1,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1719999999999999,"15min":1.123},"min":{"1min":0.65400000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":5.0940000000000003},"last":1.25},{"interface":"front","average":{"1min":2.3100000000000001,"5min":1.405,"15min":1.2769999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":16.113,"5min":16.113,"15min":16.113},"last":1.1120000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.258,"15min":1.2350000000000001},"min":{"1min":0.83199999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.2880000000000003,"5min":4.2880000000000003,"15min":4.2880000000000003},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.2809999999999999,"15min":1.256},"min":{"1min":0.86099999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":5.0309999999999997},"last":0.93600000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.575,"15min":1.3720000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":16.193999999999999,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.538,"15min":1.444},"min":{"1min":0.873,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":1.335}]},{"osd":4,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.343,"15min":1.2509999999999999},"min":{"1min":0.93300000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":5.1440000000000001,"5min":7.234,"15min":7.234},"last":1.0620000000000001},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.526,"15min":1.399},"min":{"1min":0.92200000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":16.268000000000001,"5min":16.268000000000001,"15min":16.268000000000001},"last":0.88100000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4139999999999999,"15min":1.393},"min":{"1min":0.89200000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1989999999999998,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":1.03},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.5760000000000001,"15min":1.5049999999999999},"min":{"1min":0.97699999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.9299999999999997,"5min":7.335,"15min":7.335},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":1.6819999999999999,"15min":1.5780000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":12.571,"5min":12.571,"15min":12.571},"last":1.4419999999999999},{"interface":"front","average":{"1min":2.3039999999999998,"5min":1.72,"15min":1.623},"min":{"1min":1.0229999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":12.673999999999999,"5min":12.673999999999999,"15min":12.673999999999999},"last":1.21}]}]},{"osd":4,"up_from":14,"seq":60129542185,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":561152,"data_stored":224595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1100000000000001,"15min":1.0980000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.5760000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.085,"15min":1.0389999999999999},"min":{"1min":0.79400000000000004,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":0.92600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3089999999999999,"15min":1.2909999999999999},"min":{"1min":0.95299999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":3.0609999999999999,"5min":3.0609999999999999,"15min":3.0609999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.2589999999999999,"15min":1.2270000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":1.3080000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.23,"15min":1.1759999999999999},"min":{"1min":0.73999999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":5.5010000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":1.115},{"interface":"front","average":{"1min":1.583,"5min":1.3819999999999999,"15min":1.3520000000000001},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":4.5019999999999998,"5min":4.5019999999999998,"15min":4.5019999999999998},"last":1.595}]},{"osd":3,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3959999999999999,"15min":1.4019999999999999},"min":{"1min":0.871,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.633},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.343,"15min":1.3260000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.2960000000000003,"5min":5.2960000000000003,"15min":5.2960000000000003},"last":0.98999999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4890000000000001,"15min":1.4990000000000001},"min":{"1min":0.873,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":3.0209999999999999,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.206},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":0.78100000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.5660000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.444,"15min":1.464},"min":{"1min":0.89400000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.53,"5min":1.571,"15min":1.595},"min":{"1min":0.81799999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.2320000000000002,"5min":3.976,"15min":3.976},"last":1.0589999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.673,"15min":1.7050000000000001},"min":{"1min":1.2789999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.2130000000000001,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.681},{"interface":"front","average":{"1min":1.631,"5min":1.6359999999999999,"15min":1.6319999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":5.4610000000000003,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.532}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35892,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335948,"statfs":{"total":96636764160,"available":96600010752,"internally_reserved":0,"allocated":561152,"data_stored":223621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.1930000000000001,"15min":1.0109999999999999},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":17.709,"5min":17.709,"15min":17.709},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":6.6239999999999997,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.99299999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.0920000000000001,"15min":1.016},"min":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2889999999999999,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.4489999999999998,"5min":10.163,"15min":10.163},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.22,"15min":1.1639999999999999},"min":{"1min":0.47599999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8439999999999994,"5min":8.8439999999999994,"15min":8.8439999999999994},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.405,"15min":1.2969999999999999},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":6.7690000000000001,"5min":10.25,"15min":10.25},"last":1.4530000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3300000000000001,"15min":1.3340000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":4.2439999999999998,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.429,"15min":1.423},"min":{"1min":0.85299999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3849999999999998,"5min":2.605,"15min":2.605},"last":2.0089999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.6639999999999999,"15min":1.53},"min":{"1min":0.80800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":12.65,"5min":12.65,"15min":12.65},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.4419999999999999,"15min":1.4059999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"last":1.171}]},{"osd":6,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.4990000000000001,"15min":1.484},"min":{"1min":0.95299999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":1.804},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.6830000000000001,"15min":1.593},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":6.681,"5min":10.371,"15min":10.371},"last":1.5880000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5249999999999999,"15min":1.5369999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":1.27},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.5900000000000001,"15min":1.5580000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":5.1699999999999999,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":1.907}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":565248,"data_stored":224617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.0489999999999999,"15min":0.96199999999999997},"min":{"1min":0.54800000000000004,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":12.478999999999999,"5min":12.478999999999999,"15min":12.478999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.286,"5min":1.0349999999999999,"15min":1.0009999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":1.2749999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.97399999999999998,"15min":0.91900000000000004},"min":{"1min":0.68400000000000005,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.093,"5min":3.093,"15min":3.093},"last":1.083},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2230000000000001,"15min":1.22},"min":{"1min":0.60199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8620000000000001,"5min":4.6769999999999996,"15min":4.6769999999999996},"last":1.121}]},{"osd":3,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2130000000000001,"15min":1.1699999999999999},"min":{"1min":0.69599999999999995,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.2309999999999999,"5min":3.2309999999999999,"15min":3.2309999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.296,"15min":1.1859999999999999},"min":{"1min":0.871,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":12.198,"5min":12.198,"15min":12.198},"last":0.76800000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0179999999999998,"5min":1.446,"15min":1.3500000000000001},"min":{"1min":1.0900000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":12.287000000000001,"5min":12.287000000000001,"15min":12.287000000000001},"last":1.048},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.353,"15min":1.337},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.2759999999999998,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.407}]},{"osd":5,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.3919999999999999,"15min":1.3799999999999999},"min":{"1min":0.90800000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":3.4980000000000002,"5min":3.4980000000000002,"15min":3.4980000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.6080000000000001,"15min":1.538},"min":{"1min":1.1579999999999999,"5min":0.875,"15min":0.875},"max":{"1min":12.25,"5min":12.25,"15min":12.25},"last":1.1950000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":0.93899999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.1709999999999998,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.004},{"interface":"front","average":{"1min":2.52,"5min":1.665,"15min":1.5369999999999999},"min":{"1min":0.88900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":11.542999999999999,"5min":11.542999999999999,"15min":11.542999999999999},"last":1.2410000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:02:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.629,"15min":1.609},"min":{"1min":1.1200000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":5.4889999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":1.351},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.496,"15min":1.4670000000000001},"min":{"1min":0.96399999999999997,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.3109999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35888,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335952,"statfs":{"total":96636764160,"available":96600014848,"internally_reserved":0,"allocated":552960,"data_stored":223053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.3100000000000001,"15min":1.1160000000000001},"min":{"1min":0.69399999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":14.407,"5min":14.407,"15min":14.407},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.1599999999999999,"15min":1.1240000000000001},"min":{"1min":0.66200000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.74099999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.1499999999999999,"15min":1.0960000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.89300000000000002},{"interface":"front","average":{"1min":3.141,"5min":1.585,"15min":1.3300000000000001},"min":{"1min":0.65800000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":14.311,"5min":14.311,"15min":14.311},"last":0.66600000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":1.667,"15min":1.4199999999999999},"min":{"1min":1.0029999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":14.653,"5min":14.653,"15min":14.653},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.806,"5min":1.379,"15min":1.3240000000000001},"min":{"1min":0.91100000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"last":0.82399999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":1.5880000000000001,"15min":1.468},"min":{"1min":0.97799999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":12.675000000000001,"5min":12.675000000000001,"15min":12.675000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":2.3969999999999998,"5min":1.6220000000000001,"15min":1.512},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":12.731999999999999,"5min":12.731999999999999,"15min":12.731999999999999},"last":1.117}]},{"osd":5,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.599,"15min":1.5680000000000001},"min":{"1min":1.198,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.635,"5min":1.476,"15min":1.48},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":6.9569999999999999,"5min":6.9569999999999999,"15min":6.9569999999999999},"last":0.84799999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":3.367,"5min":1.9390000000000001,"15min":1.7110000000000001},"min":{"1min":1.1120000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":14.781000000000001,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":1.7250000000000001,"15min":1.677},"min":{"1min":1.3759999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":6.6779999999999999,"5min":6.6779999999999999,"15min":6.6779999999999999},"last":1.175}]},{"osd":7,"last update":"Tue Jan 16 15:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.583,"15min":1.5800000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":6.75,"5min":6.75,"15min":6.75},"last":0.98399999999999999},{"interface":"front","average":{"1min":3.5379999999999998,"5min":2.0310000000000001,"15min":1.7889999999999999},"min":{"1min":1.1859999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":14.884,"5min":14.884,"15min":14.884},"last":1.0680000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1146880,"data_stored":813385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.94699999999999995,"15min":0.91000000000000003},"min":{"1min":0.51800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":0.995},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.96499999999999997,"15min":0.92100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.91300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.1240000000000001,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.0249999999999999,"15min":0.99099999999999999},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":0.71299999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5859999999999999,"5min":1.2649999999999999,"15min":1.0329999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":23.015999999999998,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.357,"5min":1.2050000000000001,"15min":1.1679999999999999},"min":{"1min":0.78800000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":1.224}]},{"osd":4,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.256,"15min":1.1879999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.3759999999999999,"15min":1.343},"min":{"1min":0.84399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":4.2789999999999999,"5min":4.2789999999999999,"15min":4.2789999999999999},"last":1.3999999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3009999999999999,"15min":1.2809999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":1.107},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.3440000000000001,"15min":1.302},"min":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":8.1489999999999991,"5min":8.1489999999999991,"15min":8.1489999999999991},"last":1.0609999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5,"15min":1.486},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.3999999999999999,"15min":1.375},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2869999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.171}]},{"osd":7,"last update":"Tue Jan 16 15:02:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.5,"15min":1.488},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.21},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.488,"15min":1.4379999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":8.0800000000000001,"5min":8.0800000000000001,"15min":8.0800000000000001},"last":1.2609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:02:36.925 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:02:36.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:02:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.217+0000 7f9b99400640 1 Processor -- start 2024-01-16T15:02:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.217+0000 7f9b99400640 1 -- start start 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b99400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9b9415d100 0x7f9b940b1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b99400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9b940d1490 0x7f9b940cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b99400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9406c3a0 con 0x7f9b940b1ec0 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b99400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9406c680 con 0x7f9b940d1490 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b99400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940b0ea0 con 0x7f9b9415d100 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b93400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9b940b1ec0 0x7f9b940d0d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44444/0 (socket says 172.21.15.43:44444) 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b93400640 1 -- 172.21.15.43:0/3343108687 learned_addr learned my addr 172.21.15.43:0/3343108687 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3343108687 >> v2:172.21.15.43:3300/0 conn(0x7f9b940d1490 0x7f9b940cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.219+0000 7f9b93e00640 1 --2- 172.21.15.43:0/3343108687 >> v2:172.21.15.69:3300/0 conn(0x7f9b9415d100 0x7f9b940b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.220+0000 7f9b92a00640 1 -- 172.21.15.43:0/3343108687 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949397126 0 0) 0x7f9b9406c3a0 con 0x7f9b940b1ec0 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.220+0000 7f9b98a00640 1 -- 172.21.15.43:0/3343108687 >> v2:172.21.15.69:3300/0 conn(0x7f9b9415d100 msgr2=0x7f9b940b1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.220+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3343108687 >> v2:172.21.15.69:3300/0 conn(0x7f9b9415d100 0x7f9b940b1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.220+0000 7f9b98a00640 1 -- 172.21.15.43:0/3343108687 >> v1:172.21.15.43:6789/0 conn(0x7f9b940b1ec0 legacy=0x7f9b940d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.220+0000 7f9b98a00640 1 -- 172.21.15.43:0/3343108687 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940b1180 con 0x7f9b940d1490 2024-01-16T15:02:37.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.221+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3343108687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b940d1490 0x7f9b940cef30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9b94072750 tx=0x7f9b8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2714a1255a6d2f14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.221+0000 7f9b92a00640 1 -- 172.21.15.43:0/3343108687 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c031830 con 0x7f9b940d1490 2024-01-16T15:02:37.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.221+0000 7f9b92a00640 1 -- 172.21.15.43:0/3343108687 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c031990 con 0x7f9b940d1490 2024-01-16T15:02:37.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.221+0000 7f9b92a00640 1 -- 172.21.15.43:0/3343108687 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c031d20 con 0x7f9b940d1490 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 -- 172.21.15.43:0/3343108687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b940d1490 msgr2=0x7f9b940cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 --2- 172.21.15.43:0/3343108687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b940d1490 0x7f9b940cef30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9b94072750 tx=0x7f9b8c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 -- 172.21.15.43:0/3343108687 shutdown_connections 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 --2- 172.21.15.43:0/3343108687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b940d1490 0x7f9b940cef30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 --2- 172.21.15.43:0/3343108687 >> v2:172.21.15.69:3300/0 conn(0x7f9b9415d100 0x7f9b940b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 -- 172.21.15.43:0/3343108687 >> 172.21.15.43:0/3343108687 conn(0x7f9b94099a00 msgr2=0x7f9b94099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 -- 172.21.15.43:0/3343108687 shutdown_connections 2024-01-16T15:02:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.222+0000 7f9b99400640 1 -- 172.21.15.43:0/3343108687 wait complete. 2024-01-16T15:02:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.224+0000 7f9b99400640 1 Processor -- start 2024-01-16T15:02:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.224+0000 7f9b99400640 1 -- start start 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b940b1ec0 0x7f9b9415f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b940d1490 0x7f9b94158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b941f5e50 con 0x7f9b9415d100 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94160340 con 0x7f9b940d1490 2024-01-16T15:02:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b99400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b941431b0 con 0x7f9b940b1ec0 2024-01-16T15:02:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.225+0000 7f9b98a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36872/0 (socket says 172.21.15.43:36872) 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 -- 172.21.15.43:0/3234938509 learned_addr learned my addr 172.21.15.43:0/3234938509 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b93400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b940d1490 0x7f9b94158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b93e00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b940b1ec0 0x7f9b9415f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 -- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b940b1ec0 msgr2=0x7f9b9415f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b940b1ec0 0x7f9b9415f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 -- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b940d1490 msgr2=0x7f9b94158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b940d1490 0x7f9b94158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 -- 172.21.15.43:0/3234938509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c009400 con 0x7f9b9415d100 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.226+0000 7f9b98a00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c02bee0 tx=0x7f9b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.227+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c03b9f0 con 0x7f9b9415d100 2024-01-16T15:02:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.227+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c03bb70 con 0x7f9b9415d100 2024-01-16T15:02:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.227+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0433f0 con 0x7f9b9415d100 2024-01-16T15:02:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.227+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9415faa0 con 0x7f9b9415d100 2024-01-16T15:02:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.227+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b94160060 con 0x7f9b9415d100 2024-01-16T15:02:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.228+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9404e840 con 0x7f9b9415d100 2024-01-16T15:02:37.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.229+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9b8c04c3b0 con 0x7f9b9415d100 2024-01-16T15:02:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.230+0000 7f9b8be00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 0x7f9b6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.230+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b8c0abf20 con 0x7f9b9415d100 2024-01-16T15:02:37.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.230+0000 7f9b93e00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 0x7f9b6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.231+0000 7f9b93e00640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 0x7f9b6c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006fd0 tx=0x7f9b84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.235+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b8c076b80 con 0x7f9b9415d100 2024-01-16T15:02:37.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.501+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9b941432f0 con 0x7f9b9415d100 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.505+0000 7f9b8be00640 1 -- 172.21.15.43:0/3234938509 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9b8c0350b0 con 0x7f9b9415d100 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:37.485 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:02:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 msgr2=0x7f9b6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 0x7f9b6c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006fd0 tx=0x7f9b84008040 comp rx=0 tx=0).stop 2024-01-16T15:02:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 msgr2=0x7f9b94159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c02bee0 tx=0x7f9b8c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 shutdown_connections 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b6c06b500 0x7f9b6c06d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b940d1490 0x7f9b94158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9415d100 0x7f9b94159400 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 --2- 172.21.15.43:0/3234938509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b940b1ec0 0x7f9b9415f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 >> 172.21.15.43:0/3234938509 conn(0x7f9b94099a00 msgr2=0x7f9b940ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.508+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 shutdown_connections 2024-01-16T15:02:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.509+0000 7f9b99400640 1 -- 172.21.15.43:0/3234938509 wait complete. 2024-01-16T15:02:37.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:37.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.740+0000 7fbccae00640 1 Processor -- start 2024-01-16T15:02:37.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.741+0000 7fbccae00640 1 -- start start 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbccae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbcc40acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbccae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc4071cd0 con 0x7fbcc4070780 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbccae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40b2120 con 0x7fbcc4160970 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbccae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40b2400 con 0x7fbcc406dc30 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbcbbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbcc4070780 0x7fbcc406d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44456/0 (socket says 172.21.15.43:44456) 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbcbbe00640 1 -- 172.21.15.43:0/4279608219 learned_addr learned my addr 172.21.15.43:0/4279608219 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.743+0000 7fbcc8e00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbcc40acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378705304 0 0) 0x7fbcc4071cd0 con 0x7fbcc4070780 2024-01-16T15:02:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca4003520 con 0x7fbcc4070780 2024-01-16T15:02:37.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1346528030 0 0) 0x7fbca4003520 con 0x7fbcc4070780 2024-01-16T15:02:37.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 msgr2=0x7fbcc40acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbcc40acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 >> v1:172.21.15.69:6789/0 conn(0x7fbcc4160970 legacy=0x7fbcc40b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40b26e0 con 0x7fbcc4070780 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fbcac001670 con 0x7fbcc4070780 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 >> v1:172.21.15.43:6789/0 conn(0x7fbcc4070780 legacy=0x7fbcc406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbca40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbca4003f70 0x7fbca40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca4004d00 0x7fbca4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40b2400 con 0x7fbca4003f70 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca400bb70 con 0x7fbca4004d00 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca400bd30 con 0x7fbcc406dc30 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.744+0000 7fbcbbe00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbca4003f70 0x7fbca40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcc3e00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbca40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcbbe00640 1 -- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 msgr2=0x7fbca40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcc8e00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca4004d00 0x7fbca4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcbbe00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbca40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcbbe00640 1 -- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca4004d00 msgr2=0x7fbca4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcbbe00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca4004d00 0x7fbca4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcbbe00640 1 -- 172.21.15.43:0/4279608219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4003520 con 0x7fbca4003f70 2024-01-16T15:02:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.745+0000 7fbcc3e00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcc406dc30 0x7fbca40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.746+0000 7fbcbbe00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbca4003f70 0x7fbca40047c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00c0e0 tx=0x7fbca40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09034083451d433 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.746+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac005ad0 con 0x7fbca4003f70 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.746+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac005c30 con 0x7fbca4003f70 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.746+0000 7fbcc3400640 1 -- 172.21.15.43:0/4279608219 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac02ca30 con 0x7fbca4003f70 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 -- 172.21.15.43:0/4279608219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbca4003f70 msgr2=0x7fbca40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 --2- 172.21.15.43:0/4279608219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbca4003f70 0x7fbca40047c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00c0e0 tx=0x7fbca40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbcc8e00640 1 -- 172.21.15.43:0/4279608219 reap_dead start 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 -- 172.21.15.43:0/4279608219 shutdown_connections 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 -- 172.21.15.43:0/4279608219 >> 172.21.15.43:0/4279608219 conn(0x7fbcc40b0d80 msgr2=0x7fbcc40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 -- 172.21.15.43:0/4279608219 shutdown_connections 2024-01-16T15:02:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.747+0000 7fbccae00640 1 -- 172.21.15.43:0/4279608219 wait complete. 2024-01-16T15:02:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.749+0000 7fbccae00640 1 Processor -- start 2024-01-16T15:02:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.749+0000 7fbccae00640 1 -- start start 2024-01-16T15:02:37.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbca4003f70 0x7fbcc4144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc413ed90 0x7fbcc41403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc413ff80 con 0x7fbcc41408e0 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc4144c00 con 0x7fbcc413ed90 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbccae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc413f300 con 0x7fbca4003f70 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.750+0000 7fbcc8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36892/0 (socket says 172.21.15.43:36892) 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 -- 172.21.15.43:0/206850280 learned_addr learned my addr 172.21.15.43:0/206850280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc3e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbca4003f70 0x7fbcc4144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcbbe00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc413ed90 0x7fbcc41403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 -- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbca4003f70 msgr2=0x7fbcc4144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbca4003f70 0x7fbcc4144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 -- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc413ed90 msgr2=0x7fbcc41403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc413ed90 0x7fbcc41403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 -- 172.21.15.43:0/206850280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc413f5e0 con 0x7fbcc41408e0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.751+0000 7fbcc8e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4072930 tx=0x7fbcbc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.752+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc017400 con 0x7fbcc41408e0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.752+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc011040 con 0x7fbcc41408e0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.752+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc0177c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.752+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc413f8c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.752+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc409ce20 con 0x7fbcc41408e0 2024-01-16T15:02:37.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.753+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc41400c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.754+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fbcbc017920 con 0x7fbcc41408e0 2024-01-16T15:02:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.755+0000 7fbcc0c00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 0x7fbc9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.755+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcbc08a5c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.755+0000 7fbcc3e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 0x7fbc9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.756+0000 7fbcc3e00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 0x7fbc9c06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4001750 tx=0x7fbcb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.759+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcc41400c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.994+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcc41400c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.995+0000 7fbcc0c00640 1 -- 172.21.15.43:0/206850280 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4910 (secure 0 0 0) 0x7fbcc41400c0 con 0x7fbcc41408e0 2024-01-16T15:02:37.975 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:02:37.976 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2588572053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2588572053},{"type":"v1","addr":"172.21.15.43:6837","nonce":2588572053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/791773791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":791773791},{"type":"v1","addr":"172.21.15.69:6833","nonce":791773791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3012792492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3012792492},{"type":"v1","addr":"172.21.15.69:6835","nonce":3012792492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:02:31.355647+0000","modified":"2024-01-16T15:02:34.439473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5766":{"gid":5766,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.43:6836/3947724910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3947724910},{"type":"v1","addr":"172.21.15.43:6836","nonce":3947724910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:02:37.976 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 26 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 msgr2=0x7fbc9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 0x7fbc9c06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4001750 tx=0x7fbcb4009040 comp rx=0 tx=0).stop 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 msgr2=0x7fbcc409c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4072930 tx=0x7fbcbc00f4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 shutdown_connections 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc9c06b500 0x7fbc9c06d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcc413ed90 0x7fbcc41403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcc41408e0 0x7fbcc409c8e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 --2- 172.21.15.43:0/206850280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbca4003f70 0x7fbcc4144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 >> 172.21.15.43:0/206850280 conn(0x7fbcc40b0d80 msgr2=0x7fbcc406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 shutdown_connections 2024-01-16T15:02:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:37.999+0000 7fbccae00640 1 -- 172.21.15.43:0/206850280 wait complete. 2024-01-16T15:02:38.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:02:31.355647+0000', 'modified': '2024-01-16T15:02:34.439473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5766}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5766': {'gid': 5766, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.43:6836/3947724910', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 3947724910}, {'type': 'v1', 'addr': '172.21.15.43:6836', 'nonce': 3947724910}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:02:38.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:02:38.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T15:02:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.659+0000 7fc465800640 1 Processor -- start 2024-01-16T15:02:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.659+0000 7fc465800640 1 -- start start 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc465800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4600b2550 0x7fc460070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc465800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc460161700 0x7fc4600ac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc465800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc460070a90 con 0x7fc4600b2930 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc465800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc460070cf0 con 0x7fc460161700 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc465800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc460070fd0 con 0x7fc4600b2550 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc464e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc460161700 0x7fc4600ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc464e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc460161700 0x7fc4600ac900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36908/0 (socket says 172.21.15.43:36908) 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.661+0000 7fc464e00640 1 -- 172.21.15.43:0/3703605226 learned_addr learned my addr 172.21.15.43:0/3703605226 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.662+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3703605226 >> v2:172.21.15.69:3300/0 conn(0x7fc4600b2550 0x7fc460070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.662+0000 7fc464e00640 1 -- 172.21.15.43:0/3703605226 >> v2:172.21.15.69:3300/0 conn(0x7fc4600b2550 msgr2=0x7fc460070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.662+0000 7fc464e00640 1 --2- 172.21.15.43:0/3703605226 >> v2:172.21.15.69:3300/0 conn(0x7fc4600b2550 0x7fc460070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.662+0000 7fc464e00640 1 -- 172.21.15.43:0/3703605226 >> v1:172.21.15.69:6789/0 conn(0x7fc4600b2930 legacy=0x7fc460160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.662+0000 7fc464e00640 1 -- 172.21.15.43:0/3703605226 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600712b0 con 0x7fc460161700 2024-01-16T15:02:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.663+0000 7fc464e00640 1 --2- 172.21.15.43:0/3703605226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc460161700 0x7fc4600ac900 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc46006fdc0 tx=0x7fc45802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c1d925e42a02b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc45f400640 1 -- 172.21.15.43:0/3703605226 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45802cb20 con 0x7fc460161700 2024-01-16T15:02:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc45f400640 1 -- 172.21.15.43:0/3703605226 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45802cc80 con 0x7fc460161700 2024-01-16T15:02:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc45f400640 1 -- 172.21.15.43:0/3703605226 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580343f0 con 0x7fc460161700 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 -- 172.21.15.43:0/3703605226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc460161700 msgr2=0x7fc4600ac900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 --2- 172.21.15.43:0/3703605226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc460161700 0x7fc4600ac900 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc46006fdc0 tx=0x7fc45802bf80 comp rx=0 tx=0).stop 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 -- 172.21.15.43:0/3703605226 shutdown_connections 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 --2- 172.21.15.43:0/3703605226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc460161700 0x7fc4600ac900 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 --2- 172.21.15.43:0/3703605226 >> v2:172.21.15.69:3300/0 conn(0x7fc4600b2550 0x7fc460070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 -- 172.21.15.43:0/3703605226 >> 172.21.15.43:0/3703605226 conn(0x7fc4600b0840 msgr2=0x7fc4600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 -- 172.21.15.43:0/3703605226 shutdown_connections 2024-01-16T15:02:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.664+0000 7fc465800640 1 -- 172.21.15.43:0/3703605226 wait complete. 2024-01-16T15:02:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.666+0000 7fc465800640 1 Processor -- start 2024-01-16T15:02:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.666+0000 7fc465800640 1 -- start start 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 0x7fc460154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc460161700 0x7fc46014e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46014e5f0 con 0x7fc4600b2550 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460152e10 con 0x7fc460161700 2024-01-16T15:02:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.667+0000 7fc465800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc460154f10 con 0x7fc4600b2930 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36924/0 (socket says 172.21.15.43:36924) 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 -- 172.21.15.43:0/3714683039 learned_addr learned my addr 172.21.15.43:0/3714683039 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc464e00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc460161700 0x7fc46014e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc457e00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 0x7fc460154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 -- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 msgr2=0x7fc460154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 0x7fc460154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 -- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc460161700 msgr2=0x7fc46014e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc460161700 0x7fc46014e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc45fe00640 1 -- 172.21.15.43:0/3714683039 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458009b30 con 0x7fc4600b2550 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.668+0000 7fc457e00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 0x7fc460154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e870 tx=0x7fc45000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc45cc00640 1 -- 172.21.15.43:0/3714683039 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500094f0 con 0x7fc4600b2550 2024-01-16T15:02:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc45cc00640 1 -- 172.21.15.43:0/3714683039 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45000c940 con 0x7fc4600b2550 2024-01-16T15:02:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc45cc00640 1 -- 172.21.15.43:0/3714683039 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000ccf0 con 0x7fc4600b2550 2024-01-16T15:02:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc460153020 con 0x7fc4600b2550 2024-01-16T15:02:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.669+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4601535e0 con 0x7fc4600b2550 2024-01-16T15:02:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.672+0000 7fc45cc00640 1 -- 172.21.15.43:0/3714683039 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc450013470 con 0x7fc4600b2550 2024-01-16T15:02:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.673+0000 7fc45cc00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 0x7fc43806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.673+0000 7fc45cc00640 1 -- 172.21.15.43:0/3714683039 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc45008a100 con 0x7fc4600b2550 2024-01-16T15:02:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.673+0000 7fc457e00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 0x7fc43806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.674+0000 7fc469b8b740 1 Processor -- start 2024-01-16T15:02:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.674+0000 7fc457e00640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 0x7fc43806da10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4601f42e0 tx=0x7fc448009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.674+0000 7fc469b8b740 1 -- start start 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d89047d70 0x558d8904c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89090730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d88fab880 con 0x558d8904d070 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d88fbaae0 con 0x558d8904c7b0 2024-01-16T15:02:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d88fb81d0 con 0x558d89047d70 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36932/0 (socket says 172.21.15.43:36932) 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 -- 172.21.15.43:0/3219373186 learned_addr learned my addr 172.21.15.43:0/3219373186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 -- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d89047d70 msgr2=0x558d8904c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d89047d70 0x558d8904c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 -- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 msgr2=0x558d89090730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89090730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.675+0000 7fc45fe00640 1 -- 172.21.15.43:0/3219373186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d88f7b860 con 0x558d8904d070 2024-01-16T15:02:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.676+0000 7fc45fe00640 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc450058840 tx=0x7fc450059070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9b198476e5f7e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.676+0000 7fc454c00640 1 -- 172.21.15.43:0/3219373186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450060bb0 con 0x558d8904d070 2024-01-16T15:02:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.676+0000 7fc454c00640 1 -- 172.21.15.43:0/3219373186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450056c50 con 0x558d8904d070 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.676+0000 7fc454c00640 1 -- 172.21.15.43:0/3219373186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450059be0 con 0x558d8904d070 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 -- 172.21.15.43:0/3219373186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 msgr2=0x558d8904d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc450058840 tx=0x7fc450059070 comp rx=0 tx=0).stop 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 -- 172.21.15.43:0/3219373186 shutdown_connections 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89090730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d89047d70 0x558d8904c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 --2- 172.21.15.43:0/3219373186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d8904d450 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 -- 172.21.15.43:0/3219373186 >> 172.21.15.43:0/3219373186 conn(0x558d8900ec70 msgr2=0x558d89092b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 -- 172.21.15.43:0/3219373186 shutdown_connections 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.677+0000 7fc469b8b740 1 -- 172.21.15.43:0/3219373186 wait complete. 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.678+0000 7fc469b8b740 1 -- start start 2024-01-16T15:02:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.678+0000 7fc469b8b740 1 Processor -- start 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 0x558d89098150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d89071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d88fb3ac0 con 0x558d8904d070 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d88f502f0 con 0x558d89047d70 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d88f6dc70 con 0x558d8904c7b0 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc469b8b740 10 client.0 _ll_register_callbacks cb 0x558d88f723b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc464e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc464e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36946/0 (socket says 172.21.15.43:36946) 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc464e00640 1 -- 172.21.15.43:0/299575703 learned_addr learned my addr 172.21.15.43:0/299575703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.679+0000 7fc45fe00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 0x558d89098150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc457e00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d89071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc464e00640 1 -- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 msgr2=0x558d89071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc464e00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d89071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc464e00640 1 -- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 msgr2=0x558d89098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc464e00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 0x558d89098150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc464e00640 1 -- 172.21.15.43:0/299575703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4601535e0 con 0x558d8904d070 2024-01-16T15:02:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.680+0000 7fc45fe00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 0x558d89098150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc464e00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc45802c590 tx=0x7fc458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07c6e76108415c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc44d600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458034940 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc458034aa0 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558d88facaa0 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc458034c60 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc469b8b740 10 client.5913 fetch_fsmap learned FSMap version 26 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.681+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x558d88fb2140 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558d88facaa0 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc45803d420 con 0x558d8904d070 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 10 client.5913 fetch_fsmap finished waiting for FSMap version 26 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 10 client.5913 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 20 client.5913 populate_metadata read hostname 'smithi043' 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d890a6af0 0x558d8909a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 4 client.5913 mds_command: new command op to 5766 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 --> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d88fac7c0 con 0x558d890a6af0 2024-01-16T15:02:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.682+0000 7fc45fe00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d890a6af0 0x558d8909a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.683+0000 7fc45fe00640 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d890a6af0 0x558d8909a250 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5ca574437286107 server_cookie=c4f840cd78ef3e6f in_seq=0 out_seq=0 2024-01-16T15:02:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.683+0000 7fc44d600640 10 client.5913 ms_handle_connect on v2:172.21.15.43:6834/3947724910 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.683+0000 7fc44d600640 1 -- 172.21.15.43:0/299575703 <== mds.0 v2:172.21.15.43:6834/3947724910 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558d88fac7c0 con 0x558d890a6af0 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.683+0000 7fc44d600640 10 client.5913 handle_command_reply: tid=0 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.683+0000 7fc469b8b740 1 client.5913 shutdown 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 msgr2=0x558d89075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc45802c590 tx=0x7fc458004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 shutdown_connections 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d8904d070 0x558d89075f90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d89071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d890a6af0 0x558d8909a250 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 --2- 172.21.15.43:0/299575703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d89047d70 0x558d89098150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc44d600640 0 client.5913 ms_handle_reset on v2:172.21.15.43:6834/3947724910 2024-01-16T15:02:38.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 >> 172.21.15.43:0/299575703 conn(0x558d8900ec70 msgr2=0x558d89092b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 shutdown_connections 2024-01-16T15:02:38.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 1 -- 172.21.15.43:0/299575703 wait complete. 2024-01-16T15:02:38.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.684+0000 7fc469b8b740 20 client.5913 trim_cache size 0 max 16384 2024-01-16T15:02:38.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.701+0000 7fc469b8b740 1 Processor -- start 2024-01-16T15:02:38.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 -- start start 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89075470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904d070 0x558d890759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d88d16f20 con 0x558d89047d70 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d8904bd00 con 0x558d8904c7b0 2024-01-16T15:02:38.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.702+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d890990f0 con 0x558d8904d070 2024-01-16T15:02:38.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36962/0 (socket says 172.21.15.43:36962) 2024-01-16T15:02:38.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 -- 172.21.15.43:0/2166046664 learned_addr learned my addr 172.21.15.43:0/2166046664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 -- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904d070 msgr2=0x558d890759b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904d070 0x558d890759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 -- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 msgr2=0x558d89075470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89075470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 -- 172.21.15.43:0/2166046664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d88f4fb80 con 0x558d89047d70 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.703+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc450059160 tx=0x7fc45005b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476d35ab880995f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc455600640 1 -- 172.21.15.43:0/2166046664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450087040 con 0x558d89047d70 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc455600640 1 -- 172.21.15.43:0/2166046664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450075e90 con 0x558d89047d70 2024-01-16T15:02:38.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc455600640 1 -- 172.21.15.43:0/2166046664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500770b0 con 0x558d89047d70 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 -- 172.21.15.43:0/2166046664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 msgr2=0x558d89098140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc450059160 tx=0x7fc45005b000 comp rx=0 tx=0).stop 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 -- 172.21.15.43:0/2166046664 shutdown_connections 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904c7b0 0x558d89075470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904d070 0x558d890759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2166046664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d89098140 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 -- 172.21.15.43:0/2166046664 >> 172.21.15.43:0/2166046664 conn(0x558d8900ec70 msgr2=0x558d89046dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 -- 172.21.15.43:0/2166046664 shutdown_connections 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.704+0000 7fc469b8b740 1 -- 172.21.15.43:0/2166046664 wait complete. 2024-01-16T15:02:38.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.705+0000 7fc469b8b740 1 -- start start 2024-01-16T15:02:38.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.705+0000 7fc469b8b740 1 Processor -- start 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d8908e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904d070 0x558d8908ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d890a2b70 con 0x558d89047d70 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d89072ee0 con 0x558d8904d070 2024-01-16T15:02:38.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d890a2930 con 0x558d8904c7b0 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc469b8b740 10 client.0 _ll_register_callbacks cb 0x7fc450009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36978/0 (socket says 172.21.15.43:36978) 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.706+0000 7fc45fe00640 1 -- 172.21.15.43:0/2377515767 learned_addr learned my addr 172.21.15.43:0/2377515767 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 -- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 msgr2=0x558d8908e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d8908e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 -- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904d070 msgr2=0x558d8908ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904d070 0x558d8908ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 -- 172.21.15.43:0/2377515767 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d88f7b860 con 0x558d89047d70 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc45fe00640 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e840 tx=0x7fc450009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd18e962e8a4f55e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc44d600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500770b0 con 0x558d89047d70 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc450086070 con 0x558d89047d70 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.707+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc450056c50 con 0x558d89047d70 2024-01-16T15:02:38.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc450077210 con 0x558d89047d70 2024-01-16T15:02:38.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 10 client.5919 fetch_fsmap learned FSMap version 26 2024-01-16T15:02:38.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x558d88f75170 con 0x558d89047d70 2024-01-16T15:02:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450056c50 con 0x558d89047d70 2024-01-16T15:02:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc450011070 con 0x558d89047d70 2024-01-16T15:02:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 10 client.5919 fetch_fsmap finished waiting for FSMap version 26 2024-01-16T15:02:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 10 client.5919 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 20 client.5919 populate_metadata read hostname 'smithi043' 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d89095ca0 0x558d89096080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 4 client.5919 mds_command: new command op to 5766 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 --> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558d88cb46c0 con 0x558d89095ca0 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.708+0000 7fc457e00640 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d89095ca0 0x558d89096080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc457e00640 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d89095ca0 0x558d89096080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=846ec7ef7857ad1e server_cookie=9d4ab5f0b2b5578d in_seq=0 out_seq=0 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc44d600640 10 client.5919 ms_handle_connect on v2:172.21.15.43:6834/3947724910 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc44d600640 1 -- 172.21.15.43:0/2377515767 <== mds.0 v2:172.21.15.43:6834/3947724910 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x558d88cb46c0 con 0x558d89095ca0 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc44d600640 10 client.5919 handle_command_reply: tid=0 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 client.5919 shutdown 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 msgr2=0x558d890e59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e840 tx=0x7fc450009f20 comp rx=0 tx=0).stop 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 shutdown_connections 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:6834/3947724910,v1:172.21.15.43:6836/3947724910] conn(0x558d89095ca0 0x558d89096080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d8904d070 0x558d8908ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558d8904c7b0 0x558d8908e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc44d600640 0 client.5919 ms_handle_reset on v2:172.21.15.43:6834/3947724910 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 --2- 172.21.15.43:0/2377515767 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558d89047d70 0x558d890e59c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.709+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 >> 172.21.15.43:0/2377515767 conn(0x558d8900ec70 msgr2=0x558d8904d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 shutdown_connections 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc469b8b740 1 -- 172.21.15.43:0/2377515767 wait complete. 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc469b8b740 20 client.5919 trim_cache size 0 max 16384 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5766, 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:02:38.692 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.8239820059999996, 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 26, 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 47, 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 47, 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 13.188658945 2024-01-16T15:02:38.693 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 msgr2=0x7fc43806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 0x7fc43806da10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4601f42e0 tx=0x7fc448009040 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 msgr2=0x7fc4601f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.710+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e870 tx=0x7fc45000ed30 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 shutdown_connections 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc43806b550 0x7fc43806da10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc460161700 0x7fc46014e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4600b2930 0x7fc460154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 --2- 172.21.15.43:0/3714683039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4600b2550 0x7fc4601f6f20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 >> 172.21.15.43:0/3714683039 conn(0x7fc4600b0840 msgr2=0x7fc4600b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 shutdown_connections 2024-01-16T15:02:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.711+0000 7fc465800640 1 -- 172.21.15.43:0/3714683039 wait complete. 2024-01-16T15:02:38.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.948+0000 7f01f8200640 1 Processor -- start 2024-01-16T15:02:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.948+0000 7f01f8200640 1 -- start start 2024-01-16T15:02:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.949+0000 7f01f8200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01f006d520 0x7f01f006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.949+0000 7f01f8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01f015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.949+0000 7f01f8200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f015c360 con 0x7f01f00ad720 2024-01-16T15:02:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f8200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00b41a0 con 0x7f01f006d520 2024-01-16T15:02:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f006de40 con 0x7f01f00abff0 2024-01-16T15:02:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f4e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f01f00ad720 0x7f01f015ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44466/0 (socket says 172.21.15.43:44466) 2024-01-16T15:02:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f4e00640 1 -- 172.21.15.43:0/1969087125 learned_addr learned my addr 172.21.15.43:0/1969087125 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:38.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f5800640 1 --2- 172.21.15.43:0/1969087125 >> v2:172.21.15.69:3300/0 conn(0x7f01f006d520 0x7f01f006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01f6200640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01f015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544871049 0 0) 0x7f01f015c360 con 0x7f01f00ad720 2024-01-16T15:02:38.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.950+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d0003520 con 0x7f01f00ad720 2024-01-16T15:02:38.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7981519 0 0) 0x7f01d0003520 con 0x7f01f00ad720 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 msgr2=0x7f01f015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01f015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 >> v2:172.21.15.69:3300/0 conn(0x7f01f006d520 msgr2=0x7f01f006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 --2- 172.21.15.43:0/1969087125 >> v2:172.21.15.69:3300/0 conn(0x7f01f006d520 0x7f01f006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f015ba30 con 0x7f01f00ad720 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f01d8001670 con 0x7f01f00ad720 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 >> v1:172.21.15.43:6789/0 conn(0x7f01f00ad720 legacy=0x7f01f015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01d0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d0003f40 0x7f01d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d0004d00 0x7f01d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f006de40 con 0x7f01d0003f40 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d000b980 con 0x7f01d0004d00 2024-01-16T15:02:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d000bac0 con 0x7f01f00abff0 2024-01-16T15:02:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.951+0000 7f01f4e00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d0003f40 0x7f01d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f6200640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d0004d00 0x7f01d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f5800640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01d0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f4e00640 1 -- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 msgr2=0x7f01d0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f4e00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01f00abff0 0x7f01d0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f4e00640 1 -- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d0004d00 msgr2=0x7f01d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f4e00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d0004d00 0x7f01d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.952+0000 7f01f4e00640 1 -- 172.21.15.43:0/1969087125 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0003520 con 0x7f01d0003f40 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.953+0000 7f01f4e00640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d0003f40 0x7f01d00047c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c0e0 tx=0x7f01d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ced1e47930aacac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.953+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8005ad0 con 0x7f01d0003f40 2024-01-16T15:02:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.953+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d8005c30 con 0x7f01d0003f40 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.953+0000 7f01efe00640 1 -- 172.21.15.43:0/1969087125 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d802c7c0 con 0x7f01d0003f40 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 -- 172.21.15.43:0/1969087125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d0003f40 msgr2=0x7f01d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 --2- 172.21.15.43:0/1969087125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d0003f40 0x7f01d00047c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c0e0 tx=0x7f01d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f6200640 1 -- 172.21.15.43:0/1969087125 reap_dead start 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 -- 172.21.15.43:0/1969087125 shutdown_connections 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 -- 172.21.15.43:0/1969087125 >> 172.21.15.43:0/1969087125 conn(0x7f01f00b0d80 msgr2=0x7f01f00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 -- 172.21.15.43:0/1969087125 shutdown_connections 2024-01-16T15:02:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.954+0000 7f01f8200640 1 -- 172.21.15.43:0/1969087125 wait complete. 2024-01-16T15:02:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.956+0000 7f01f8200640 1 Processor -- start 2024-01-16T15:02:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.956+0000 7f01f8200640 1 -- start start 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.957+0000 7f01f8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d0003f40 0x7f01f014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.957+0000 7f01f8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f0070f30 0x7f01f0071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.957+0000 7f01f8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0072370 con 0x7f01f00718d0 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f8200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f015da90 con 0x7f01f0070f30 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f015dd70 con 0x7f01d0003f40 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37002/0 (socket says 172.21.15.43:37002) 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 -- 172.21.15.43:0/9676622 learned_addr learned my addr 172.21.15.43:0/9676622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f4e00640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f0070f30 0x7f01f0071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f5800640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d0003f40 0x7f01f014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 -- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d0003f40 msgr2=0x7f01f014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d0003f40 0x7f01f014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 -- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f0070f30 msgr2=0x7f01f0071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f0070f30 0x7f01f0071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.958+0000 7f01f6200640 1 -- 172.21.15.43:0/9676622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f015e050 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01f6200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f01f00b1e40 tx=0x7f01e80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800e070 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e8011e10 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f015e330 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8003990 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.959+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f015d720 con 0x7f01f00718d0 2024-01-16T15:02:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.960+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f0074770 con 0x7f01f00718d0 2024-01-16T15:02:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.962+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f01e8003af0 con 0x7f01f00718d0 2024-01-16T15:02:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.963+0000 7f01ed600640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 0x7f01c806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.963+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e808a710 con 0x7f01f00718d0 2024-01-16T15:02:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.963+0000 7f01f5800640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 0x7f01c806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.964+0000 7f01f5800640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 0x7f01c806d9e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009a90 tx=0x7f01e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:38.969+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01e8054fe0 con 0x7f01f00718d0 2024-01-16T15:02:39.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.207+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f01f00008d0 con 0x7f01f00718d0 2024-01-16T15:02:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.208+0000 7f01ed600640 1 -- 172.21.15.43:0/9676622 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f01e8058600 con 0x7f01f00718d0 2024-01-16T15:02:39.188 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.211+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 msgr2=0x7f01c806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.211+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 0x7f01c806d9e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009a90 tx=0x7f01e0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.211+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 msgr2=0x7f01f00cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.211+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f01f00b1e40 tx=0x7f01e80115a0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 shutdown_connections 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01c806b520 0x7f01c806d9e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01f0070f30 0x7f01f0071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01f00718d0 0x7f01f00cbee0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 --2- 172.21.15.43:0/9676622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d0003f40 0x7f01f014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 >> 172.21.15.43:0/9676622 conn(0x7f01f00b0d80 msgr2=0x7f01f0158450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 shutdown_connections 2024-01-16T15:02:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:39.212+0000 7f01f8200640 1 -- 172.21.15.43:0/9676622 wait complete. 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:39.210 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:39.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.210 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:02:39.259 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:02:39.260 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42279sec preferred_lft 42279sec 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:39.261 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:39.262 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:02:39.263 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:39.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.264 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:39.355 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:02:39.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.360 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:02:39.417 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:02:39.417 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:39.417 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:02:39.417 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:02:39.417 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:39.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:02:39.419 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:02:39.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.610 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:02:39.661 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:02:39.661 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:02:39.662 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:39.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:39.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:02:39.758 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:02:39.759 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:39.759 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:39.814 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:02:39.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:39.814 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:39.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:02:39.971 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:40.024 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:02:40.025 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:40.026 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:40.027 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:40.027 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:40.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:40.028 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:02:40.079 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:40.080 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:02:40.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:02:40.159 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:40.199 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:02:40.199 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:02:40.199 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:02:40.199 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:40.200 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:40.201 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:40.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:40.202 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:02:40.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:02:40.300+0000 7f6fa4052480 -1 init, newargv = 0x55f706664ab0 newargc=15 2024-01-16T15:02:40.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[52388]: starting ceph client 2024-01-16T15:02:40.289 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[52388]: starting fuse 2024-01-16T15:02:41.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:02:41.497 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:02:41.555 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:41.556 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:02:41.557 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:41.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.558 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:02:41.612 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:02:41.613 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:02:41.613 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:02:41.614 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:02:41.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.52388.asok 2024-01-16T15:02:41.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:41.820 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52388.asok 2024-01-16T15:02:41.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:41.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52388.asok status 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "52388", 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:02:42.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5934, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5934 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2696355439 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:02:42.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2696355439 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.5934 192.168.0.1:0/2696355439", 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/2696355439", 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 26, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 47, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:02:42.070 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:02:42.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.083 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:02:42.083 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:42.138 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:02:42.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:42.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:42.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:02:42.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.476+0000 7f77f2400640 1 Processor -- start 2024-01-16T15:02:42.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.477+0000 7f77f2400640 1 -- start start 2024-01-16T15:02:42.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77f2400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77ec15d5a0 0x7f77ec0d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77f2400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f77ec0d1490 0x7f77ec0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77f2400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec0cf0b0 con 0x7f77ec0cf790 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77f2400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ec0b01a0 con 0x7f77ec0d1490 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77f2400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ec15d980 con 0x7f77ec15d5a0 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.478+0000 7f77ebe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f77ec0d1490 0x7f77ec0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f77ec0d1490 0x7f77ec0d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37012/0 (socket says 172.21.15.43:37012) 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 -- 172.21.15.43:0/2736807736 learned_addr learned my addr 172.21.15.43:0/2736807736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77f1a00640 1 --1- 172.21.15.43:0/2736807736 >> v1:172.21.15.43:6789/0 conn(0x7f77ec0cf790 0x7f77ec0cfc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44478/0 (socket says 172.21.15.43:44478) 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77f1000640 1 --2- 172.21.15.43:0/2736807736 >> v2:172.21.15.69:3300/0 conn(0x7f77ec15d5a0 0x7f77ec0d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 -- 172.21.15.43:0/2736807736 >> v2:172.21.15.69:3300/0 conn(0x7f77ec15d5a0 msgr2=0x7f77ec0d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 --2- 172.21.15.43:0/2736807736 >> v2:172.21.15.69:3300/0 conn(0x7f77ec15d5a0 0x7f77ec0d3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 -- 172.21.15.43:0/2736807736 >> v1:172.21.15.43:6789/0 conn(0x7f77ec0cf790 legacy=0x7f77ec0cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77eb400640 1 -- 172.21.15.43:0/2736807736 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083277519 0 0) 0x7f77ec0cf0b0 con 0x7f77ec0cf790 2024-01-16T15:02:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.479+0000 7f77ebe00640 1 -- 172.21.15.43:0/2736807736 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec0d1e30 con 0x7f77ec0d1490 2024-01-16T15:02:42.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.480+0000 7f77ebe00640 1 --2- 172.21.15.43:0/2736807736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0d1490 0x7f77ec0d18f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f77ec08c9d0 tx=0x7f77d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4345d8142fc3fd62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.480+0000 7f77eb400640 1 -- 172.21.15.43:0/2736807736 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4030860 con 0x7f77ec0d1490 2024-01-16T15:02:42.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.480+0000 7f77eb400640 1 -- 172.21.15.43:0/2736807736 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d40309c0 con 0x7f77ec0d1490 2024-01-16T15:02:42.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77eb400640 1 -- 172.21.15.43:0/2736807736 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4030d50 con 0x7f77ec0d1490 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 -- 172.21.15.43:0/2736807736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0d1490 msgr2=0x7f77ec0d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 --2- 172.21.15.43:0/2736807736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0d1490 0x7f77ec0d18f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f77ec08c9d0 tx=0x7f77d402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 -- 172.21.15.43:0/2736807736 shutdown_connections 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 --2- 172.21.15.43:0/2736807736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0d1490 0x7f77ec0d18f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 --2- 172.21.15.43:0/2736807736 >> v2:172.21.15.69:3300/0 conn(0x7f77ec15d5a0 0x7f77ec0d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 -- 172.21.15.43:0/2736807736 >> 172.21.15.43:0/2736807736 conn(0x7f77ec09c3a0 msgr2=0x7f77ec0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 -- 172.21.15.43:0/2736807736 shutdown_connections 2024-01-16T15:02:42.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.481+0000 7f77f2400640 1 -- 172.21.15.43:0/2736807736 wait complete. 2024-01-16T15:02:42.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.483+0000 7f77f2400640 1 Processor -- start 2024-01-16T15:02:42.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.483+0000 7f77f2400640 1 -- start start 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77ec0d1490 0x7f77ec0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 0x7f77ec1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b3420 con 0x7f77ec0cf790 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b4090 con 0x7f77ec15d5a0 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f2400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b4370 con 0x7f77ec0d1490 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37024/0 (socket says 172.21.15.43:37024) 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.484+0000 7f77f1000640 1 -- 172.21.15.43:0/3987667077 learned_addr learned my addr 172.21.15.43:0/3987667077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77ebe00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77ec0d1490 0x7f77ec0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1a00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 0x7f77ec1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 -- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77ec0d1490 msgr2=0x7f77ec0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77ec0d1490 0x7f77ec0b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 -- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 msgr2=0x7f77ec1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 0x7f77ec1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 -- 172.21.15.43:0/3987667077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40094d0 con 0x7f77ec0cf790 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1a00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 0x7f77ec1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f1000640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c7f0 tx=0x7f77dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc004220 con 0x7f77ec0cf790 2024-01-16T15:02:42.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc0043a0 con 0x7f77ec0cf790 2024-01-16T15:02:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc007860 con 0x7f77ec0cf790 2024-01-16T15:02:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.485+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec0b2dd0 con 0x7f77ec0cf790 2024-01-16T15:02:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.486+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec155e10 con 0x7f77ec0cf790 2024-01-16T15:02:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.487+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec04e840 con 0x7f77ec0cf790 2024-01-16T15:02:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.488+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f77dc007bd0 con 0x7f77ec0cf790 2024-01-16T15:02:42.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.489+0000 7f77e8c00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 0x7f77c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.489+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77dc08a0a0 con 0x7f77ec0cf790 2024-01-16T15:02:42.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.489+0000 7f77ebe00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 0x7f77c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.490+0000 7f77ebe00640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 0x7f77c406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77d4036000 tx=0x7f77d403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.494+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77dc054a80 con 0x7f77ec0cf790 2024-01-16T15:02:42.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.728+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f77ec0d3e90 con 0x7f77ec0cf790 2024-01-16T15:02:42.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.729+0000 7f77e8c00640 1 -- 172.21.15.43:0/3987667077 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f77dc0580a0 con 0x7f77ec0cf790 2024-01-16T15:02:42.709 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:02:42.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 msgr2=0x7f77c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 0x7f77c406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77d4036000 tx=0x7f77d403b040 comp rx=0 tx=0).stop 2024-01-16T15:02:42.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 msgr2=0x7f77ec156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c7f0 tx=0x7f77dc00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 shutdown_connections 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f77c406b500 0x7f77c406d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77ec15d5a0 0x7f77ec1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77ec0d1490 0x7f77ec0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 --2- 172.21.15.43:0/3987667077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f77ec0cf790 0x7f77ec156600 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.732+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 >> 172.21.15.43:0/3987667077 conn(0x7f77ec09c3a0 msgr2=0x7f77ec06c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.733+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 shutdown_connections 2024-01-16T15:02:42.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:42.733+0000 7f77f2400640 1 -- 172.21.15.43:0/3987667077 wait complete. 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:02:42.731 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:42.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.731 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42437sec preferred_lft 42437sec 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:02:42.789 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:02:42.790 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:02:42.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.791 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:02:42.876 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:02:42.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.880 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:02:42.931 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:02:42.931 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:02:42.931 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:02:42.931 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:02:42.931 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:02:42.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:02:42.933 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:02:43.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.108 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:02:43.157 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:02:43.158 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:02:43.158 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:43.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:02:43.247 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:02:43.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:43.248 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:43.298 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:02:43.298 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:02:43.298 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:43.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:02:43.440 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:43.494 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:02:43.494 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:02:43.495 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:43.496 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:43.497 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:43.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.497 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:02:43.552 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:02:43.552 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:02:43.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:02:43.632 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:02:43.672 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:02:43.673 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:02:43.674 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:43.675 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:02:43.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.675 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:02:43.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:02:43.753+0000 7f08aac0b480 -1 init, newargv = 0x559392710880 newargc=15 2024-01-16T15:02:43.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[43404]: starting ceph client 2024-01-16T15:02:43.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[43404]: starting fuse 2024-01-16T15:02:43.759 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:02:43.760 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:02:43.760 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:02:43.761 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:02:43.959 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.43404.asok 2024-01-16T15:02:43.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:02:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:02:43.973 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43404.asok 2024-01-16T15:02:43.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43404.asok status 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "43404", 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:02:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5383, 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5383 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2516733296 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:02:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2516733296 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5383 192.168.0.1:0/2516733296", 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2516733296", 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 26, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 47, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:02:44.212 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:02:44.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.225 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:02:44.225 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:44.276 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:02:44.276 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:44.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:44.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:44.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:02:44.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.625+0000 7ff878a00640 1 Processor -- start 2024-01-16T15:02:44.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.626+0000 7ff878a00640 1 -- start start 2024-01-16T15:02:44.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.627+0000 7ff878a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff87415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.627+0000 7ff878a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8740b35a0 0x7ff87415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.627+0000 7ff878a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87415f930 con 0x7ff87406c9d0 2024-01-16T15:02:44.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.627+0000 7ff878a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740b43b0 con 0x7ff8740b35a0 2024-01-16T15:02:44.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.627+0000 7ff878a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740b3b10 con 0x7ff874144770 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.628+0000 7ff873e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff87406c9d0 0x7ff87406ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38964/0 (socket says 172.21.15.43:38964) 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.628+0000 7ff873e00640 1 -- 172.21.15.43:0/3903531420 learned_addr learned my addr 172.21.15.43:0/3903531420 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.628+0000 7ff86be00640 1 --2- 172.21.15.43:0/3903531420 >> v2:172.21.15.69:3300/0 conn(0x7ff8740b35a0 0x7ff87415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.628+0000 7ff873400640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff87415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.628+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446295260 0 0) 0x7ff87415f930 con 0x7ff87406c9d0 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff854003520 con 0x7ff87406c9d0 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3888205861 0 0) 0x7ff854003520 con 0x7ff87406c9d0 2024-01-16T15:02:44.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 msgr2=0x7ff87415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff87415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 >> v2:172.21.15.69:3300/0 conn(0x7ff8740b35a0 msgr2=0x7ff87415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 --2- 172.21.15.43:0/3903531420 >> v2:172.21.15.69:3300/0 conn(0x7ff8740b35a0 0x7ff87415f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740b3df0 con 0x7ff87406c9d0 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff86c0019f0 con 0x7ff87406c9d0 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 >> v1:172.21.15.43:6789/0 conn(0x7ff87406c9d0 legacy=0x7ff87406ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff8540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff854003fb0 0x7ff8540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff854004d00 0x7ff854005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740b43b0 con 0x7ff854003fb0 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740b3b10 con 0x7ff854004d00 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.629+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85400ba00 con 0x7ff874144770 2024-01-16T15:02:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff854003fb0 0x7ff8540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff873e00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff854004d00 0x7ff854005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff873400640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff8540039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 -- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 msgr2=0x7ff8540039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff874144770 0x7ff8540039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 -- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff854004d00 msgr2=0x7ff854005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff854004d00 0x7ff854005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff86be00640 1 -- 172.21.15.43:0/3903531420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff854003520 con 0x7ff854003fb0 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.630+0000 7ff873e00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff854004d00 0x7ff854005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.632+0000 7ff86be00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff854003fb0 0x7ff8540047c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009ac0 tx=0x7ff8540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e144360f0926ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.632+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c02c710 con 0x7ff854003fb0 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.632+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c02c870 con 0x7ff854003fb0 2024-01-16T15:02:44.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.632+0000 7ff872a00640 1 -- 172.21.15.43:0/3903531420 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c02cc00 con 0x7ff854003fb0 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 -- 172.21.15.43:0/3903531420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff854003fb0 msgr2=0x7ff8540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 --2- 172.21.15.43:0/3903531420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff854003fb0 0x7ff8540047c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009ac0 tx=0x7ff8540057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff873e00640 1 -- 172.21.15.43:0/3903531420 reap_dead start 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 -- 172.21.15.43:0/3903531420 shutdown_connections 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 -- 172.21.15.43:0/3903531420 >> 172.21.15.43:0/3903531420 conn(0x7ff8740cec10 msgr2=0x7ff8740cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 -- 172.21.15.43:0/3903531420 shutdown_connections 2024-01-16T15:02:44.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.633+0000 7ff878a00640 1 -- 172.21.15.43:0/3903531420 wait complete. 2024-01-16T15:02:44.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.635+0000 7ff878a00640 1 Processor -- start 2024-01-16T15:02:44.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.635+0000 7ff878a00640 1 -- start start 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff854003fb0 0x7ff87415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8740c3c30 0x7ff8740c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff874130670 con 0x7ff874132d90 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740c45f0 con 0x7ff8740c3c30 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff878a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8741310a0 con 0x7ff854003fb0 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff873e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff873e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45200/0 (socket says 172.21.15.43:45200) 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff873e00640 1 -- 172.21.15.43:0/87444696 learned_addr learned my addr 172.21.15.43:0/87444696 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff86be00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8740c3c30 0x7ff8740c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.636+0000 7ff873400640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff854003fb0 0x7ff87415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 -- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff854003fb0 msgr2=0x7ff87415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff854003fb0 0x7ff87415cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 -- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8740c3c30 msgr2=0x7ff8740c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8740c3c30 0x7ff8740c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 -- 172.21.15.43:0/87444696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87415d480 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff873e00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff8740cfc40 tx=0x7ff86c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c00eb90 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c00ecf0 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff874130a50 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c014630 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.637+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87415d190 con 0x7ff874132d90 2024-01-16T15:02:44.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.638+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff874131230 con 0x7ff874132d90 2024-01-16T15:02:44.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.640+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff86c014790 con 0x7ff874132d90 2024-01-16T15:02:44.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.641+0000 7ff86b400640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 0x7ff84c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.641+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff86c08a610 con 0x7ff874132d90 2024-01-16T15:02:44.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.641+0000 7ff873400640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 0x7ff84c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.642+0000 7ff873400640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 0x7ff84c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff864009930 tx=0x7ff864008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.645+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff86c055100 con 0x7ff874132d90 2024-01-16T15:02:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:44.882+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff87404e840 con 0x7ff874132d90 2024-01-16T15:02:45.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.786+0000 7ff86b400640 1 -- 172.21.15.43:0/87444696 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff86c058720 con 0x7ff874132d90 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 msgr2=0x7ff84c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 0x7ff84c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff864009930 tx=0x7ff864008040 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 msgr2=0x7ff874133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff8740cfc40 tx=0x7ff86c00e530 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 shutdown_connections 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff84c06b500 0x7ff84c06d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8740c3c30 0x7ff8740c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff874132d90 0x7ff874133210 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 --2- 172.21.15.43:0/87444696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff854003fb0 0x7ff87415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 >> 172.21.15.43:0/87444696 conn(0x7ff8740cec10 msgr2=0x7ff8740ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 shutdown_connections 2024-01-16T15:02:45.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:45.790+0000 7ff878a00640 1 -- 172.21.15.43:0/87444696 wait complete. 2024-01-16T15:02:45.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:46.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.043+0000 7fe7cd600640 1 Processor -- start 2024-01-16T15:02:46.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.043+0000 7fe7cd600640 1 -- start start 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.044+0000 7fe7cd600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c815b340 0x7fe7c80abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.044+0000 7fe7cd600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7c80cbcb0 0x7fe7c806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.044+0000 7fe7cd600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c80b4130 con 0x7fe7c80ce880 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7cd600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c80ac530 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7cd600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c80b2080 con 0x7fe7c815b340 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7c80cbcb0 0x7fe7c806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7c80cbcb0 0x7fe7c806d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45204/0 (socket says 172.21.15.43:45204) 2024-01-16T15:02:46.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 -- 172.21.15.43:0/3607374351 learned_addr learned my addr 172.21.15.43:0/3607374351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c7400640 1 --1- 172.21.15.43:0/3607374351 >> v1:172.21.15.43:6789/0 conn(0x7fe7c80ce880 0x7fe7c815ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38978/0 (socket says 172.21.15.43:38978) 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6a00640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c815b340 0x7fe7c80abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 -- 172.21.15.43:0/3607374351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c815b340 msgr2=0x7fe7c80abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c815b340 0x7fe7c80abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 -- 172.21.15.43:0/3607374351 >> v1:172.21.15.43:6789/0 conn(0x7fe7c80ce880 legacy=0x7fe7c815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c6000640 1 -- 172.21.15.43:0/3607374351 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c80b2360 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.045+0000 7fe7c5600640 1 -- 172.21.15.43:0/3607374351 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535168002 0 0) 0x7fe7c80b4130 con 0x7fe7c80ce880 2024-01-16T15:02:46.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7c6000640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c806d520 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009870 tx=0x7fe7b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26878b32ff6a0090 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7c5600640 1 -- 172.21.15.43:0/3607374351 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0030840 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7c5600640 1 -- 172.21.15.43:0/3607374351 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b00309a0 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7c5600640 1 -- 172.21.15.43:0/3607374351 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0030d30 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7cd600640 1 -- 172.21.15.43:0/3607374351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 msgr2=0x7fe7c806d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.047+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c806d520 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009870 tx=0x7fe7b002b980 comp rx=0 tx=0).stop 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 -- 172.21.15.43:0/3607374351 shutdown_connections 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c806d520 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3607374351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c815b340 0x7fe7c80abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 -- 172.21.15.43:0/3607374351 >> 172.21.15.43:0/3607374351 conn(0x7fe7c80b0d80 msgr2=0x7fe7c80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 -- 172.21.15.43:0/3607374351 shutdown_connections 2024-01-16T15:02:46.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.048+0000 7fe7cd600640 1 -- 172.21.15.43:0/3607374351 wait complete. 2024-01-16T15:02:46.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.049+0000 7fe7cd600640 1 Processor -- start 2024-01-16T15:02:46.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.050+0000 7fe7cd600640 1 -- start start 2024-01-16T15:02:46.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.050+0000 7fe7cd600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.050+0000 7fe7cd600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c80ce880 0x7fe7c8159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7cd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 0x7fe7c815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7cd600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8145680 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7cd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c815a2b0 con 0x7fe7c815b340 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7cd600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c815a590 con 0x7fe7c80ce880 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45218/0 (socket says 172.21.15.43:45218) 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 -- 172.21.15.43:0/3359558213 learned_addr learned my addr 172.21.15.43:0/3359558213 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6000640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c80ce880 0x7fe7c8159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 -- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c80ce880 msgr2=0x7fe7c8159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c7400640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 0x7fe7c815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c80ce880 0x7fe7c8159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 -- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 msgr2=0x7fe7c815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 0x7fe7c815c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c6a00640 1 -- 172.21.15.43:0/3359558213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00094d0 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.051+0000 7fe7c7400640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 0x7fe7c815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7c6a00640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800d5f0 tx=0x7fe7b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80125f0 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b8012750 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8012b00 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c815b8c0 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.052+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c8159fa0 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.054+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c815b720 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.054+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe7b8012c60 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.056+0000 7fe7bea00640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 0x7fe7a006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.056+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7b808a450 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.056+0000 7fe7c6000640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 0x7fe7a006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.057+0000 7fe7c6000640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 0x7fe7a006da10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036000 tx=0x7fe7b003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.061+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7c815b720 con 0x7fe7c80cbcb0 2024-01-16T15:02:46.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.284+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c80b2c00 con 0x7fe7a006b550 2024-01-16T15:02:46.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.286+0000 7fe7bea00640 1 -- 172.21.15.43:0/3359558213 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe7b0030b90 con 0x7fe7a006b550 2024-01-16T15:02:46.266 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:02:46.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:02:46.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:02:46.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:02:46.267 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:02:46.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 msgr2=0x7fe7a006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 0x7fe7a006da10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0036000 tx=0x7fe7b003b040 comp rx=0 tx=0).stop 2024-01-16T15:02:46.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 msgr2=0x7fe7c8145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800d5f0 tx=0x7fe7b800dab0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 shutdown_connections 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7a006b550 0x7fe7a006da10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c815b340 0x7fe7c815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7c80ce880 0x7fe7c8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 --2- 172.21.15.43:0/3359558213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c80cbcb0 0x7fe7c8145140 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 >> 172.21.15.43:0/3359558213 conn(0x7fe7c80b0d80 msgr2=0x7fe7c80adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.289+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 shutdown_connections 2024-01-16T15:02:46.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.290+0000 7fe7cd600640 1 -- 172.21.15.43:0/3359558213 wait complete. 2024-01-16T15:02:46.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:46.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.534+0000 7f79e3400640 1 Processor -- start 2024-01-16T15:02:46.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.535+0000 7f79e3400640 1 -- start start 2024-01-16T15:02:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.536+0000 7f79e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc1448c0 0x7f79dc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.536+0000 7f79e3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f79dc06d910 0x7f79dc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.536+0000 7f79e3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79dc15bbd0 0x7f79dc15eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc06dd50 con 0x7f79dc06d910 2024-01-16T15:02:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc144ca0 con 0x7f79dc15bbd0 2024-01-16T15:02:46.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc15c450 con 0x7f79dc1448c0 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f79dc06d910 0x7f79dc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f79dc06d910 0x7f79dc15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45232/0 (socket says 172.21.15.43:45232) 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e1600640 1 -- 172.21.15.43:0/2035344871 learned_addr learned my addr 172.21.15.43:0/2035344871 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e2a00640 1 --2- 172.21.15.43:0/2035344871 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15bbd0 0x7f79dc15eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.537+0000 7f79e2000640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc1448c0 0x7f79dc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.538+0000 7f79e1600640 1 -- 172.21.15.43:0/2035344871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc1448c0 msgr2=0x7f79dc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.538+0000 7f79e1600640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc1448c0 0x7f79dc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.538+0000 7f79e1600640 1 -- 172.21.15.43:0/2035344871 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15bbd0 msgr2=0x7f79dc15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.538+0000 7f79e1600640 1 --2- 172.21.15.43:0/2035344871 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15bbd0 0x7f79dc15eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.538+0000 7f79e1600640 1 -- 172.21.15.43:0/2035344871 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0715b0 con 0x7f79dc06d910 2024-01-16T15:02:46.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.540+0000 7f79e1600640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc06d910 0x7f79dc15b690 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0cc090 tx=0x7f79c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c6b936343a7c45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.540+0000 7f79e0c00640 1 -- 172.21.15.43:0/2035344871 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4004230 con 0x7f79dc06d910 2024-01-16T15:02:46.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.540+0000 7f79e0c00640 1 -- 172.21.15.43:0/2035344871 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c4004390 con 0x7f79dc06d910 2024-01-16T15:02:46.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.540+0000 7f79e0c00640 1 -- 172.21.15.43:0/2035344871 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4003810 con 0x7f79dc06d910 2024-01-16T15:02:46.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 -- 172.21.15.43:0/2035344871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc06d910 msgr2=0x7f79dc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc06d910 0x7f79dc15b690 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0cc090 tx=0x7f79c402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 -- 172.21.15.43:0/2035344871 shutdown_connections 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 --2- 172.21.15.43:0/2035344871 >> v2:172.21.15.69:3300/0 conn(0x7f79dc15bbd0 0x7f79dc15eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc06d910 0x7f79dc15b690 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 --2- 172.21.15.43:0/2035344871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc1448c0 0x7f79dc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 -- 172.21.15.43:0/2035344871 >> 172.21.15.43:0/2035344871 conn(0x7f79dc0b0aa0 msgr2=0x7f79dc0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 -- 172.21.15.43:0/2035344871 shutdown_connections 2024-01-16T15:02:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.541+0000 7f79e3400640 1 -- 172.21.15.43:0/2035344871 wait complete. 2024-01-16T15:02:46.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.543+0000 7f79e3400640 1 Processor -- start 2024-01-16T15:02:46.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.543+0000 7f79e3400640 1 -- start start 2024-01-16T15:02:46.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc06d910 0x7f79dc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15bbd0 0x7f79dc1555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0a8690 con 0x7f79dc1448c0 2024-01-16T15:02:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0b25a0 con 0x7f79dc15bbd0 2024-01-16T15:02:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.544+0000 7f79e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0b2880 con 0x7f79dc06d910 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45248/0 (socket says 172.21.15.43:45248) 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 -- 172.21.15.43:0/1947970644 learned_addr learned my addr 172.21.15.43:0/1947970644 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e2000640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc06d910 0x7f79dc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e2a00640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15bbd0 0x7f79dc1555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 -- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc06d910 msgr2=0x7f79dc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc06d910 0x7f79dc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 -- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15bbd0 msgr2=0x7f79dc1555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15bbd0 0x7f79dc1555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.545+0000 7f79e1600640 1 -- 172.21.15.43:0/1947970644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4009460 con 0x7f79dc1448c0 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79e1600640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f79c4000c00 tx=0x7f79c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4003c40 con 0x7f79dc1448c0 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c403f070 con 0x7f79dc1448c0 2024-01-16T15:02:46.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4004500 con 0x7f79dc1448c0 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc0b2b60 con 0x7f79dc1448c0 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.546+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc0ac1f0 con 0x7f79dc1448c0 2024-01-16T15:02:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.547+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc0b2da0 con 0x7f79dc1448c0 2024-01-16T15:02:46.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.548+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f79c4041030 con 0x7f79dc1448c0 2024-01-16T15:02:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.549+0000 7f79da000640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 0x7f79b406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.549+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79c40abfe0 con 0x7f79dc1448c0 2024-01-16T15:02:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.549+0000 7f79e2000640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 0x7f79b406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.550+0000 7f79e2000640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 0x7f79b406d9f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f79dc1591e0 tx=0x7f79cc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.554+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79dc0b2da0 con 0x7f79dc1448c0 2024-01-16T15:02:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.747+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f79dc0b2da0 con 0x7f79dc1448c0 2024-01-16T15:02:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.747+0000 7f79da000640 1 -- 172.21.15.43:0/1947970644 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f79dc0b2da0 con 0x7f79dc1448c0 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.750+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 msgr2=0x7f79b406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.750+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 0x7f79b406d9f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f79dc1591e0 tx=0x7f79cc009190 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.750+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 msgr2=0x7f79dc0adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.750+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f79c4000c00 tx=0x7f79c40046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 shutdown_connections 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f79b406b530 0x7f79b406d9f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15bbd0 0x7f79dc1555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79dc1448c0 0x7f79dc0adf70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 --2- 172.21.15.43:0/1947970644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79dc06d910 0x7f79dc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 >> 172.21.15.43:0/1947970644 conn(0x7f79dc0b0aa0 msgr2=0x7f79dc13e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 shutdown_connections 2024-01-16T15:02:46.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:46.751+0000 7f79e3400640 1 -- 172.21.15.43:0/1947970644 wait complete. 2024-01-16T15:02:46.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:02:46.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.000+0000 7fba89600640 1 Processor -- start 2024-01-16T15:02:46.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.000+0000 7fba89600640 1 -- start start 2024-01-16T15:02:46.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.002+0000 7fba89600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 0x7fba84071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.002+0000 7fba89600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba841439b0 0x7fba840b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.002+0000 7fba89600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba84070d40 con 0x7fba840b3f70 2024-01-16T15:02:46.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.002+0000 7fba89600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840b4400 con 0x7fba841439b0 2024-01-16T15:02:46.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.002+0000 7fba89600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba84071020 con 0x7fba841435d0 2024-01-16T15:02:46.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.003+0000 7fba83e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 0x7fba84071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.003+0000 7fba83e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 0x7fba84071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:43062/0 (socket says 172.21.15.43:43062) 2024-01-16T15:02:46.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.003+0000 7fba7b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba841439b0 0x7fba840b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.003+0000 7fba83e00640 1 -- 172.21.15.43:0/2020387145 learned_addr learned my addr 172.21.15.43:0/2020387145 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.004+0000 7fba7b200640 1 -- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 msgr2=0x7fba84071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:46.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.004+0000 7fba7b200640 1 --2- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 0x7fba84071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.004+0000 7fba7b200640 1 -- 172.21.15.43:0/2020387145 >> v1:172.21.15.69:6789/0 conn(0x7fba840b3f70 legacy=0x7fba84070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.004+0000 7fba7b200640 1 -- 172.21.15.43:0/2020387145 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84071300 con 0x7fba841439b0 2024-01-16T15:02:46.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.005+0000 7fba7b200640 1 --2- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841439b0 0x7fba840b3a30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0097d0 tx=0x7fba6c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99045b86f61af286 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba83400640 1 -- 172.21.15.43:0/2020387145 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c031800 con 0x7fba841439b0 2024-01-16T15:02:46.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba83400640 1 -- 172.21.15.43:0/2020387145 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c031960 con 0x7fba841439b0 2024-01-16T15:02:46.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba83400640 1 -- 172.21.15.43:0/2020387145 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c031cf0 con 0x7fba841439b0 2024-01-16T15:02:46.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 -- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841439b0 msgr2=0x7fba840b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 --2- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841439b0 0x7fba840b3a30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0097d0 tx=0x7fba6c02b8a0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 -- 172.21.15.43:0/2020387145 shutdown_connections 2024-01-16T15:02:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 --2- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841439b0 0x7fba840b3a30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 --2- 172.21.15.43:0/2020387145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841435d0 0x7fba84071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.006+0000 7fba89600640 1 -- 172.21.15.43:0/2020387145 >> 172.21.15.43:0/2020387145 conn(0x7fba840cec10 msgr2=0x7fba840cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.007+0000 7fba89600640 1 -- 172.21.15.43:0/2020387145 shutdown_connections 2024-01-16T15:02:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.007+0000 7fba89600640 1 -- 172.21.15.43:0/2020387145 wait complete. 2024-01-16T15:02:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.008+0000 7fba89600640 1 Processor -- start 2024-01-16T15:02:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.009+0000 7fba89600640 1 -- start start 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.009+0000 7fba89600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.009+0000 7fba89600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841435d0 0x7fba840cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba89600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841439b0 0x7fba840cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba89600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba840b05c0 con 0x7fba840b3f70 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba89600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba840ccfd0 con 0x7fba841435d0 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba89600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8413a600 con 0x7fba841439b0 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45260/0 (socket says 172.21.15.43:45260) 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba83e00640 1 -- 172.21.15.43:0/215553884 learned_addr learned my addr 172.21.15.43:0/215553884 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba88c00640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841439b0 0x7fba840cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba7b200640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841435d0 0x7fba840cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.010+0000 7fba83e00640 1 -- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841439b0 msgr2=0x7fba840cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba83e00640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841439b0 0x7fba840cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba83e00640 1 -- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841435d0 msgr2=0x7fba840cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba83e00640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841435d0 0x7fba840cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba83e00640 1 -- 172.21.15.43:0/215553884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c009400 con 0x7fba840b3f70 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba83e00640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fba74009fd0 tx=0x7fba7400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7400c940 con 0x7fba840b3f70 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7400caa0 con 0x7fba840b3f70 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7400ce30 con 0x7fba840b3f70 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba840cd270 con 0x7fba840b3f70 2024-01-16T15:02:46.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.011+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba840b01e0 con 0x7fba840b3f70 2024-01-16T15:02:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.013+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba840cde30 con 0x7fba840b3f70 2024-01-16T15:02:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.013+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fba74013470 con 0x7fba840b3f70 2024-01-16T15:02:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.015+0000 7fba80c00640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.015+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba7408a180 con 0x7fba840b3f70 2024-01-16T15:02:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.015+0000 7fba7b200640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.016+0000 7fba7b200640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 0x7fba5c06d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fba6c036000 tx=0x7fba6c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.019+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba74054c90 con 0x7fba840b3f70 2024-01-16T15:02:47.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.241+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fba840718c0 con 0x7fba5c06b500 2024-01-16T15:02:47.230 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:47.250+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.230 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:47.250+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.230 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:47.250+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.230 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:47.250+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.230 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:02:47.250+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.300+0000 7fba80c00640 1 -- 172.21.15.43:0/215553884 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba6c031ad0 con 0x7fba5c06b500 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 msgr2=0x7fba5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 0x7fba5c06d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fba6c036000 tx=0x7fba6c03c040 comp rx=0 tx=0).stop 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 msgr2=0x7fba8413a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fba74009fd0 tx=0x7fba7400ed40 comp rx=0 tx=0).stop 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 shutdown_connections 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba841435d0 0x7fba840cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba841439b0 0x7fba840cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 --2- 172.21.15.43:0/215553884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba840b3f70 0x7fba8413a0c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.303+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 >> 172.21.15.43:0/215553884 conn(0x7fba840cec10 msgr2=0x7fba840ad0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.304+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 shutdown_connections 2024-01-16T15:02:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.304+0000 7fba89600640 1 -- 172.21.15.43:0/215553884 wait complete. 2024-01-16T15:02:47.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:02:47.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.529+0000 7f452d000640 1 Processor -- start 2024-01-16T15:02:47.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.529+0000 7f452d000640 1 -- start start 2024-01-16T15:02:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.530+0000 7f452d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45280b48d0 0x7f45280b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f452d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 0x7f4528167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f452d000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45280b23e0 con 0x7f4528160bd0 2024-01-16T15:02:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f452d000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45280cdc70 con 0x7f45280b48d0 2024-01-16T15:02:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f452d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280b2870 con 0x7f45280b3210 2024-01-16T15:02:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f4525400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45280b48d0 0x7f45280b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f4525400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45280b48d0 0x7f45280b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45292/0 (socket says 172.21.15.43:45292) 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f4525400640 1 -- 172.21.15.43:0/198353453 learned_addr learned my addr 172.21.15.43:0/198353453 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.531+0000 7f4526800640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 0x7f4528167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.532+0000 7f4525400640 1 -- 172.21.15.43:0/198353453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 msgr2=0x7f4528167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.532+0000 7f4525400640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 0x7f4528167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.532+0000 7f4525400640 1 -- 172.21.15.43:0/198353453 >> v1:172.21.15.69:6789/0 conn(0x7f4528160bd0 legacy=0x7f45280b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.532+0000 7f4525400640 1 -- 172.21.15.43:0/198353453 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45280d41b0 con 0x7f45280b48d0 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.532+0000 7f4526800640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 0x7f4528167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.533+0000 7f4525400640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45280b48d0 0x7f45280b1ea0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4518009e50 tx=0x7f451802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ac00f66639355f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.533+0000 7f4524a00640 1 -- 172.21.15.43:0/198353453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451802ca70 con 0x7f45280b48d0 2024-01-16T15:02:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f4524a00640 1 -- 172.21.15.43:0/198353453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451802cbd0 con 0x7f45280b48d0 2024-01-16T15:02:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f4524a00640 1 -- 172.21.15.43:0/198353453 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180343f0 con 0x7f45280b48d0 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 -- 172.21.15.43:0/198353453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45280b48d0 msgr2=0x7f45280b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45280b48d0 0x7f45280b1ea0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4518009e50 tx=0x7f451802bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 -- 172.21.15.43:0/198353453 shutdown_connections 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45280b48d0 0x7f45280b1ea0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 --2- 172.21.15.43:0/198353453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b3210 0x7f4528167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 -- 172.21.15.43:0/198353453 >> 172.21.15.43:0/198353453 conn(0x7f452809c5a0 msgr2=0x7f45280d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 -- 172.21.15.43:0/198353453 shutdown_connections 2024-01-16T15:02:47.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.534+0000 7f452d000640 1 -- 172.21.15.43:0/198353453 wait complete. 2024-01-16T15:02:47.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.536+0000 7f452d000640 1 Processor -- start 2024-01-16T15:02:47.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.536+0000 7f452d000640 1 -- start start 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45280b3210 0x7f45280b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b48d0 0x7f45280c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280c86a0 con 0x7f4528160bd0 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280c7a90 con 0x7f45280b3210 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.537+0000 7f452d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280c7d70 con 0x7f45280b48d0 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45306/0 (socket says 172.21.15.43:45306) 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 -- 172.21.15.43:0/4213608763 learned_addr learned my addr 172.21.15.43:0/4213608763 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4525e00640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45280b3210 0x7f45280b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 -- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b48d0 msgr2=0x7f45280c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4525400640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b48d0 0x7f45280c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b48d0 0x7f45280c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 -- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45280b3210 msgr2=0x7f45280b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45280b3210 0x7f45280b87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 -- 172.21.15.43:0/4213608763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4518009b30 con 0x7f4528160bd0 2024-01-16T15:02:47.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.538+0000 7f4526800640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f452000e7e0 tx=0x7f452000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.539+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520004030 con 0x7f4528160bd0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.539+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4520004190 con 0x7f4528160bd0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.539+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520004500 con 0x7f4528160bd0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.539+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45280c8050 con 0x7f4528160bd0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.539+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4528158b20 con 0x7f4528160bd0 2024-01-16T15:02:47.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.540+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528066d20 con 0x7f4528160bd0 2024-01-16T15:02:47.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.541+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f452000c6d0 con 0x7f4528160bd0 2024-01-16T15:02:47.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.542+0000 7f451e000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 0x7f450006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:47.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.543+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4520089ce0 con 0x7f4528160bd0 2024-01-16T15:02:47.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.543+0000 7f4525e00640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 0x7f450006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:47.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.543+0000 7f4525e00640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 0x7f450006d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4510000f60 tx=0x7f4510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:47.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.547+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45200546d0 con 0x7f4528160bd0 2024-01-16T15:02:47.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.769+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f45280b3760 con 0x7f450006b530 2024-01-16T15:02:47.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.777+0000 7f451e000640 1 -- 172.21.15.43:0/4213608763 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f45100050d0 con 0x7f450006b530 2024-01-16T15:02:47.757 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 msgr2=0x7f450006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 0x7f450006d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4510000f60 tx=0x7f4510009040 comp rx=0 tx=0).stop 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 msgr2=0x7f452814b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f452000e7e0 tx=0x7f452000eca0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 shutdown_connections 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f450006b530 0x7f450006d9f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4528160bd0 0x7f452814b360 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45280b48d0 0x7f45280c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 --2- 172.21.15.43:0/4213608763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45280b3210 0x7f45280b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 >> 172.21.15.43:0/4213608763 conn(0x7f452809c5a0 msgr2=0x7f45280ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 shutdown_connections 2024-01-16T15:02:47.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:02:47.780+0000 7f452d000640 1 -- 172.21.15.43:0/4213608763 wait complete. 2024-01-16T15:02:47.780 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8 2024-01-16T15:02:47.780 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:02:47.780 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:47.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:47.828 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:54.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:54.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:03.713 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:03:03.713 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:03:03.713 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.7493 s, 68.2 MB/s 2024-01-16T15:03:03.714 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:03:03.715 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:03.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:03.772 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:03:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:03:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.8272 s, 72.4 MB/s 2024-01-16T15:03:18.657 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:03:18.657 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:18.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:18.713 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:24.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:24.712 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:03:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:03:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.588 s, 79.0 MB/s 2024-01-16T15:03:32.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:03:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.615+0000 7fa44b200640 1 Processor -- start 2024-01-16T15:03:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.615+0000 7fa44b200640 1 -- start start 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa44b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa44c06d9e0 0x7fa44c165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa44b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c15f930 con 0x7fa44c15f270 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa44b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c0b3b10 con 0x7fa44c144770 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa44b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c0b43b0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa44a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa44c06d9e0 0x7fa44c165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa449400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa44c15f270 0x7fa44c0cd4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57464/0 (socket says 172.21.15.43:57464) 2024-01-16T15:03:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.617+0000 7fa449400640 1 -- 172.21.15.43:0/2284344864 learned_addr learned my addr 172.21.15.43:0/2284344864 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:03:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.618+0000 7fa44a800640 1 -- 172.21.15.43:0/2284344864 >> v1:172.21.15.69:6789/0 conn(0x7fa44c144770 legacy=0x7fa44c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.618+0000 7fa44a800640 1 -- 172.21.15.43:0/2284344864 >> v1:172.21.15.43:6789/0 conn(0x7fa44c15f270 legacy=0x7fa44c0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.618+0000 7fa44a800640 1 -- 172.21.15.43:0/2284344864 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0b3df0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.619+0000 7fa44a800640 1 --2- 172.21.15.43:0/2284344864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c165ef0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa4440097a0 tx=0x7fa44402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fab0120a90298ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.619+0000 7fa448a00640 1 -- 172.21.15.43:0/2284344864 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444031830 con 0x7fa44c06d9e0 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.619+0000 7fa448a00640 1 -- 172.21.15.43:0/2284344864 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa444031990 con 0x7fa44c06d9e0 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.619+0000 7fa448a00640 1 -- 172.21.15.43:0/2284344864 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444031d20 con 0x7fa44c06d9e0 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 -- 172.21.15.43:0/2284344864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 msgr2=0x7fa44c165ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 --2- 172.21.15.43:0/2284344864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c165ef0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa4440097a0 tx=0x7fa44402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 -- 172.21.15.43:0/2284344864 shutdown_connections 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 --2- 172.21.15.43:0/2284344864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c165ef0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 -- 172.21.15.43:0/2284344864 >> 172.21.15.43:0/2284344864 conn(0x7fa44c0cec10 msgr2=0x7fa44c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 -- 172.21.15.43:0/2284344864 shutdown_connections 2024-01-16T15:03:32.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.620+0000 7fa44b200640 1 -- 172.21.15.43:0/2284344864 wait complete. 2024-01-16T15:03:32.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.622+0000 7fa44b200640 1 Processor -- start 2024-01-16T15:03:32.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.622+0000 7fa44b200640 1 -- start start 2024-01-16T15:03:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c144770 0x7fa44c0ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 0x7fa44c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0714d0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0acf00 con 0x7fa44c15f270 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0ad1e0 con 0x7fa44c144770 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa449e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa449e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52626/0 (socket says 172.21.15.43:52626) 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa449e00640 1 -- 172.21.15.43:0/979421583 learned_addr learned my addr 172.21.15.43:0/979421583 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa449400640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c144770 0x7fa44c0ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.623+0000 7fa44a800640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 0x7fa44c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 -- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c144770 msgr2=0x7fa44c0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c144770 0x7fa44c0ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 -- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 msgr2=0x7fa44c070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 0x7fa44c070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 -- 172.21.15.43:0/979421583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa444009400 con 0x7fa44c06d9e0 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa44a800640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 0x7fa44c070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa449e00640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa43400c430 tx=0x7fa43400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434011790 con 0x7fa44c06d9e0 2024-01-16T15:03:32.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4340118f0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434011ce0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c0ad4c0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.624+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c0ada80 con 0x7fa44c06d9e0 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.625+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c0adc70 con 0x7fa44c06d9e0 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.627+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fa43401c460 con 0x7fa44c06d9e0 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.627+0000 7fa442000640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 0x7fa42406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.628+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa43408a0d0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.628+0000 7fa449400640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 0x7fa42406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:32.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.629+0000 7fa449400640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 0x7fa42406da50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa44c15aa00 tx=0x7fa43c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:32.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.633+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa434054ae0 con 0x7fa44c06d9e0 2024-01-16T15:03:32.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.836+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fa44c04e840 con 0x7fa44c06d9e0 2024-01-16T15:03:32.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.843+0000 7fa442000640 1 -- 172.21.15.43:0/979421583 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7fa434058100 con 0x7fa44c06d9e0 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 msgr2=0x7fa42406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 0x7fa42406da50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa44c15aa00 tx=0x7fa43c009040 comp rx=0 tx=0).stop 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 msgr2=0x7fa44c159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa43400c430 tx=0x7fa43400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 shutdown_connections 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406b590 0x7fa42406da50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c15f270 0x7fa44c070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c144770 0x7fa44c0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 --2- 172.21.15.43:0/979421583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c06d9e0 0x7fa44c159900 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:32.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.846+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 >> 172.21.15.43:0/979421583 conn(0x7fa44c0cec10 msgr2=0x7fa44c0aeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:32.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.847+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 shutdown_connections 2024-01-16T15:03:32.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:32.847+0000 7fa44b200640 1 -- 172.21.15.43:0/979421583 wait complete. 2024-01-16T15:03:32.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:03:33.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.084+0000 7f6296c00640 1 Processor -- start 2024-01-16T15:03:33.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.084+0000 7f6296c00640 1 -- start start 2024-01-16T15:03:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.086+0000 7f6296c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.086+0000 7f6296c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62900d1d90 0x7f62900b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.086+0000 7f6296c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900c6bf0 con 0x7f62900d0fa0 2024-01-16T15:03:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.086+0000 7f6296c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900d1400 con 0x7f62900d1d90 2024-01-16T15:03:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.086+0000 7f6296c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900cedc0 con 0x7f629015d3b0 2024-01-16T15:03:33.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.087+0000 7f6295800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.087+0000 7f6295800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:40894/0 (socket says 172.21.15.43:40894) 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.087+0000 7f6295800640 1 -- 172.21.15.43:0/1195591059 learned_addr learned my addr 172.21.15.43:0/1195591059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.087+0000 7f6296200640 1 --2- 172.21.15.43:0/1195591059 >> v2:172.21.15.69:3300/0 conn(0x7f62900d1d90 0x7f62900b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.088+0000 7f6296200640 1 -- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 msgr2=0x7f62900ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.088+0000 7f6296200640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.088+0000 7f6296200640 1 -- 172.21.15.43:0/1195591059 >> v1:172.21.15.69:6789/0 conn(0x7f62900d0fa0 legacy=0x7f62900d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.088+0000 7f6296200640 1 -- 172.21.15.43:0/1195591059 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900c6e40 con 0x7f62900d1d90 2024-01-16T15:03:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.088+0000 7f6295800640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:03:33.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.089+0000 7f6296200640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62900d1d90 0x7f62900b3ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f62880097a0 tx=0x7f628802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac0dcc75cb9ff823 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.090+0000 7f628fe00640 1 -- 172.21.15.43:0/1195591059 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288031780 con 0x7f62900d1d90 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.090+0000 7f628fe00640 1 -- 172.21.15.43:0/1195591059 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62880318e0 con 0x7f62900d1d90 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.090+0000 7f628fe00640 1 -- 172.21.15.43:0/1195591059 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288031c70 con 0x7f62900d1d90 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.090+0000 7f6296c00640 1 -- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62900d1d90 msgr2=0x7f62900b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.090+0000 7f6296c00640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62900d1d90 0x7f62900b3ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f62880097a0 tx=0x7f628802b820 comp rx=0 tx=0).stop 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 -- 172.21.15.43:0/1195591059 shutdown_connections 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62900d1d90 0x7f62900b3ca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 --2- 172.21.15.43:0/1195591059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f629015d3b0 0x7f62900ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 -- 172.21.15.43:0/1195591059 >> 172.21.15.43:0/1195591059 conn(0x7f62900d30a0 msgr2=0x7f62900d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:33.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 -- 172.21.15.43:0/1195591059 shutdown_connections 2024-01-16T15:03:33.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.091+0000 7f6296c00640 1 -- 172.21.15.43:0/1195591059 wait complete. 2024-01-16T15:03:33.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.092+0000 7f6296c00640 1 Processor -- start 2024-01-16T15:03:33.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.093+0000 7f6296c00640 1 -- start start 2024-01-16T15:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 0x7f629015a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629015d3b0 0x7f629015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629014dc90 con 0x7f62900d0fa0 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62900cba40 con 0x7f629015d3b0 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900cbed0 con 0x7f62900d1d90 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45952/0 (socket says 172.21.15.43:45952) 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 -- 172.21.15.43:0/3947043461 learned_addr learned my addr 172.21.15.43:0/3947043461 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6296200640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629015d3b0 0x7f629015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6294e00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 0x7f629015a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 -- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 msgr2=0x7f629015a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 0x7f629015a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 -- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629015d3b0 msgr2=0x7f629015ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.094+0000 7f6295800640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629015d3b0 0x7f629015ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f6295800640 1 -- 172.21.15.43:0/3947043461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280009b30 con 0x7f62900d0fa0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f6294e00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 0x7f629015a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f6295800640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f628000efc0 tx=0x7f628000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000ce60 con 0x7f62900d0fa0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628000f040 con 0x7f62900d0fa0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280014680 con 0x7f62900d0fa0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6288009400 con 0x7f62900d0fa0 2024-01-16T15:03:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.095+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62900cb110 con 0x7f62900d0fa0 2024-01-16T15:03:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.097+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629004e840 con 0x7f62900d0fa0 2024-01-16T15:03:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.098+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f62800147e0 con 0x7f62900d0fa0 2024-01-16T15:03:33.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.099+0000 7f628d600640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 0x7f626806da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.099+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f628008a980 con 0x7f62900d0fa0 2024-01-16T15:03:33.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.099+0000 7f6294e00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 0x7f626806da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.100+0000 7f6294e00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 0x7f626806da80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6290072ab0 tx=0x7f6278009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.104+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62800553a0 con 0x7f62900d0fa0 2024-01-16T15:03:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.299+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f629015c9d0 con 0x7f62900d0fa0 2024-01-16T15:03:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.304+0000 7f628d600640 1 -- 172.21.15.43:0/3947043461 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7f62800589c0 con 0x7f62900d0fa0 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.307+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 msgr2=0x7f626806da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.307+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 0x7f626806da80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6290072ab0 tx=0x7f6278009040 comp rx=0 tx=0).stop 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.307+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 msgr2=0x7f629014d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.307+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f628000efc0 tx=0x7f628000c490 comp rx=0 tx=0).stop 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 shutdown_connections 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626806b5c0 0x7f626806da80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629015d3b0 0x7f629015ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62900d1d90 0x7f629015a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 --2- 172.21.15.43:0/3947043461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62900d0fa0 0x7f629014d750 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 >> 172.21.15.43:0/3947043461 conn(0x7f62900d30a0 msgr2=0x7f62900c7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 shutdown_connections 2024-01-16T15:03:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.308+0000 7f6296c00640 1 -- 172.21.15.43:0/3947043461 wait complete. 2024-01-16T15:03:33.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:03:33.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.551+0000 7efce2e00640 1 Processor -- start 2024-01-16T15:03:33.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.552+0000 7efce2e00640 1 -- start start 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.553+0000 7efce2e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efcdc0ae270 0x7efcdc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.553+0000 7efce2e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc15b730 con 0x7efcdc09cb30 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce2e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0ac740 con 0x7efcdc15b350 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce2e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0ac460 con 0x7efcdc0ae270 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efcdc0ae270 0x7efcdc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efcdc0ae270 0x7efcdc0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45964/0 (socket says 172.21.15.43:45964) 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efcdb400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efcdc09cb30 0x7efcdc0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57164/0 (socket says 172.21.15.43:57164) 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 -- 172.21.15.43:0/4017456995 learned_addr learned my addr 172.21.15.43:0/4017456995 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 -- 172.21.15.43:0/4017456995 >> v1:172.21.15.69:6789/0 conn(0x7efcdc15b350 legacy=0x7efcdc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 -- 172.21.15.43:0/4017456995 >> v1:172.21.15.43:6789/0 conn(0x7efcdc09cb30 legacy=0x7efcdc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.554+0000 7efce0e00640 1 -- 172.21.15.43:0/4017456995 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0b2a50 con 0x7efcdc0ae270 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efce0e00640 1 --2- 172.21.15.43:0/4017456995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc0ae270 0x7efcdc0abe80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c6173dcb27c6d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efcdaa00640 1 -- 172.21.15.43:0/4017456995 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4030860 con 0x7efcdc0ae270 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efcdaa00640 1 -- 172.21.15.43:0/4017456995 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd40309c0 con 0x7efcdc0ae270 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efcdaa00640 1 -- 172.21.15.43:0/4017456995 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4030d50 con 0x7efcdc0ae270 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efce2e00640 1 -- 172.21.15.43:0/4017456995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc0ae270 msgr2=0x7efcdc0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.556+0000 7efce2e00640 1 --2- 172.21.15.43:0/4017456995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc0ae270 0x7efcdc0abe80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.557+0000 7efce2e00640 1 -- 172.21.15.43:0/4017456995 shutdown_connections 2024-01-16T15:03:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.557+0000 7efce2e00640 1 --2- 172.21.15.43:0/4017456995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc0ae270 0x7efcdc0abe80 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.557+0000 7efce2e00640 1 -- 172.21.15.43:0/4017456995 >> 172.21.15.43:0/4017456995 conn(0x7efcdc0b0ba0 msgr2=0x7efcdc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.557+0000 7efce2e00640 1 -- 172.21.15.43:0/4017456995 shutdown_connections 2024-01-16T15:03:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.557+0000 7efce2e00640 1 -- 172.21.15.43:0/4017456995 wait complete. 2024-01-16T15:03:33.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.558+0000 7efce2e00640 1 Processor -- start 2024-01-16T15:03:33.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.559+0000 7efce2e00640 1 -- start start 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.559+0000 7efce2e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcdc09cb30 0x7efcdc13e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce2e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 0x7efcdc13b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce2e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce2e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc152c60 con 0x7efcdc15b350 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce2e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc151bc0 con 0x7efcdc0ae270 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce2e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc151ea0 con 0x7efcdc09cb30 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45966/0 (socket says 172.21.15.43:45966) 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 -- 172.21.15.43:0/2339133058 learned_addr learned my addr 172.21.15.43:0/2339133058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efcdb400640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 0x7efcdc13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efcdbe00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcdc09cb30 0x7efcdc13e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 -- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcdc09cb30 msgr2=0x7efcdc13e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcdc09cb30 0x7efcdc13e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 -- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 msgr2=0x7efcdc13b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 0x7efcdc13b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efce0e00640 1 -- 172.21.15.43:0/2339133058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40094d0 con 0x7efcdc15b350 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.560+0000 7efcdb400640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 0x7efcdc13b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efce0e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4039da0 con 0x7efcdc15b350 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd4043400 con 0x7efcdc15b350 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4043770 con 0x7efcdc15b350 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc152180 con 0x7efcdc15b350 2024-01-16T15:03:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.561+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc152740 con 0x7efcdc15b350 2024-01-16T15:03:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.562+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc09d200 con 0x7efcdc15b350 2024-01-16T15:03:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.563+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7efcd40438d0 con 0x7efcdc15b350 2024-01-16T15:03:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.564+0000 7efcd3e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 0x7efcb406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.564+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcd403e070 con 0x7efcdc15b350 2024-01-16T15:03:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.565+0000 7efcdbe00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 0x7efcb406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.565+0000 7efcdbe00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 0x7efcb406da50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efccc007bf0 tx=0x7efccc007240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:33.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.569+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcd4077b70 con 0x7efcdc15b350 2024-01-16T15:03:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.794+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7efcdc06e720 con 0x7efcb406b590 2024-01-16T15:03:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.826+0000 7efcd3e00640 1 -- 172.21.15.43:0/2339133058 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efccc00c9d0 con 0x7efcb406b590 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 msgr2=0x7efcb406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 0x7efcb406da50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efccc007bf0 tx=0x7efccc007240 comp rx=0 tx=0).stop 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 msgr2=0x7efcdc14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd4004730 comp rx=0 tx=0).stop 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 shutdown_connections 2024-01-16T15:03:33.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efcb406b590 0x7efcb406da50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcdc0ae270 0x7efcdc13b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcdc15b350 0x7efcdc14f8d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 --2- 172.21.15.43:0/2339133058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcdc09cb30 0x7efcdc13e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 >> 172.21.15.43:0/2339133058 conn(0x7efcdc0b0ba0 msgr2=0x7efcdc15a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 shutdown_connections 2024-01-16T15:03:33.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:33.829+0000 7efce2e00640 1 -- 172.21.15.43:0/2339133058 wait complete. 2024-01-16T15:03:33.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:03:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.072+0000 7ff963800640 1 Processor -- start 2024-01-16T15:03:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.073+0000 7ff963800640 1 -- start start 2024-01-16T15:03:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.074+0000 7ff963800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff95c0706a0 0x7ff95c070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.074+0000 7ff963800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff95c0b1ea0 0x7ff95c06c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.074+0000 7ff963800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95c070fc0 con 0x7ff95c06d4c0 2024-01-16T15:03:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.074+0000 7ff963800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c0b2860 con 0x7ff95c0706a0 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.074+0000 7ff963800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c0b2b40 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff95c0706a0 0x7ff95c070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff95c0706a0 0x7ff95c070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45970/0 (socket says 172.21.15.43:45970) 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 -- 172.21.15.43:0/3509140294 learned_addr learned my addr 172.21.15.43:0/3509140294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff953c00640 1 --2- 172.21.15.43:0/3509140294 >> v2:172.21.15.69:3300/0 conn(0x7ff95c0b1ea0 0x7ff95c06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 -- 172.21.15.43:0/3509140294 >> v2:172.21.15.69:3300/0 conn(0x7ff95c0b1ea0 msgr2=0x7ff95c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 --2- 172.21.15.43:0/3509140294 >> v2:172.21.15.69:3300/0 conn(0x7ff95c0b1ea0 0x7ff95c06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 -- 172.21.15.43:0/3509140294 >> v1:172.21.15.43:6789/0 conn(0x7ff95c06d4c0 legacy=0x7ff95c163020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff960e00640 1 -- 172.21.15.43:0/3509140294 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c0b2e20 con 0x7ff95c0706a0 2024-01-16T15:03:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.075+0000 7ff95be00640 1 -- 172.21.15.43:0/3509140294 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887027220 0 0) 0x7ff95c070fc0 con 0x7ff95c06d4c0 2024-01-16T15:03:34.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.077+0000 7ff960e00640 1 --2- 172.21.15.43:0/3509140294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0706a0 0x7ff95c070a80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0aeae0 tx=0x7ff94c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1850782635fe50 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.077+0000 7ff95be00640 1 -- 172.21.15.43:0/3509140294 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c038400 con 0x7ff95c0706a0 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.077+0000 7ff95be00640 1 -- 172.21.15.43:0/3509140294 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c032070 con 0x7ff95c0706a0 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.077+0000 7ff95be00640 1 -- 172.21.15.43:0/3509140294 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c038780 con 0x7ff95c0706a0 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 -- 172.21.15.43:0/3509140294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0706a0 msgr2=0x7ff95c070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 --2- 172.21.15.43:0/3509140294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0706a0 0x7ff95c070a80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0aeae0 tx=0x7ff94c02e180 comp rx=0 tx=0).stop 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 -- 172.21.15.43:0/3509140294 shutdown_connections 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 --2- 172.21.15.43:0/3509140294 >> v2:172.21.15.69:3300/0 conn(0x7ff95c0b1ea0 0x7ff95c06c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 --2- 172.21.15.43:0/3509140294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0706a0 0x7ff95c070a80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 -- 172.21.15.43:0/3509140294 >> 172.21.15.43:0/3509140294 conn(0x7ff95c0b0ba0 msgr2=0x7ff95c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 -- 172.21.15.43:0/3509140294 shutdown_connections 2024-01-16T15:03:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.078+0000 7ff963800640 1 -- 172.21.15.43:0/3509140294 wait complete. 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.079+0000 7ff963800640 1 Processor -- start 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.080+0000 7ff963800640 1 -- start start 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 0x7ff95c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff95c0cb610 0x7ff95c1e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0cce30 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0cc9a0 con 0x7ff95c0cb610 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff963800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0cbd90 con 0x7ff95c0706a0 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45982/0 (socket says 172.21.15.43:45982) 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 -- 172.21.15.43:0/351725787 learned_addr learned my addr 172.21.15.43:0/351725787 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff960e00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 0x7ff95c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff961800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff95c0cb610 0x7ff95c1e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 -- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 msgr2=0x7ff95c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 0x7ff95c06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 -- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff95c0cb610 msgr2=0x7ff95c1e5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff95c0cb610 0x7ff95c1e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff953c00640 1 -- 172.21.15.43:0/351725787 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c00bd40 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.081+0000 7ff960e00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 0x7ff95c06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff953c00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff94400d5f0 tx=0x7ff94400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440125f0 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944012750 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c0b3a80 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944012b00 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.082+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c0cc120 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.084+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c0ccb30 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.085+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7ff944012c60 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.086+0000 7ff959600640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 0x7ff93406da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:34.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.086+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff94408a5b0 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.086+0000 7ff960e00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 0x7ff93406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:34.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.087+0000 7ff960e00640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 0x7ff93406da80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff94c039000 tx=0x7ff94c034000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:34.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.091+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff944054fc0 con 0x7ff95c0b1ea0 2024-01-16T15:03:34.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.311+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff95c0b3900 con 0x7ff93406b5c0 2024-01-16T15:03:34.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.316+0000 7ff959600640 1 -- 172.21.15.43:0/351725787 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ff94c00b260 con 0x7ff93406b5c0 2024-01-16T15:03:34.296 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 msgr2=0x7ff93406da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 0x7ff93406da80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff94c039000 tx=0x7ff94c034000 comp rx=0 tx=0).stop 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 msgr2=0x7ff95c0cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff94400d5f0 tx=0x7ff94400dab0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 shutdown_connections 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff93406b5c0 0x7ff93406da80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff95c0cb610 0x7ff95c1e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff95c0b1ea0 0x7ff95c0cb0d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 --2- 172.21.15.43:0/351725787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff95c0706a0 0x7ff95c06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:34.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 >> 172.21.15.43:0/351725787 conn(0x7ff95c0b0ba0 msgr2=0x7ff95c15b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:34.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 shutdown_connections 2024-01-16T15:03:34.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:34.319+0000 7ff963800640 1 -- 172.21.15.43:0/351725787 wait complete. 2024-01-16T15:03:34.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:34.319 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8) 2024-01-16T15:03:34.383 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8 2024-01-16T15:03:34.384 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:03:34.384 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:03:34.385 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:03:36.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:36.387 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8) 2024-01-16T15:03:36.450 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/17f9d8ce-5934-42f2-905e-9d75891132d8 2024-01-16T15:03:36.450 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:03:36.451 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:03:36.451 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:03:36.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone24921 2024-01-16T15:03:36.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.698+0000 7f5c00c00640 1 Processor -- start 2024-01-16T15:03:36.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.698+0000 7f5c00c00640 1 -- start start 2024-01-16T15:03:36.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.699+0000 7f5c00c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5bfc153ea0 0x7f5bfc0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:36.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.699+0000 7f5c00c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0ce650 con 0x7f5bfc070850 2024-01-16T15:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.699+0000 7f5c00c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc154710 con 0x7f5bfc154280 2024-01-16T15:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.699+0000 7f5c00c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc15e0a0 con 0x7f5bfc153ea0 2024-01-16T15:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5bfc070850 0x7f5bfc071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57180/0 (socket says 172.21.15.43:57180) 2024-01-16T15:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfbe00640 1 -- 172.21.15.43:0/291205322 learned_addr learned my addr 172.21.15.43:0/291205322 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfb400640 1 --2- 172.21.15.43:0/291205322 >> v2:172.21.15.43:3300/0 conn(0x7f5bfc153ea0 0x7f5bfc0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:36.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfa000640 1 -- 172.21.15.43:0/291205322 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625315147 0 0) 0x7f5bfc0ce650 con 0x7f5bfc070850 2024-01-16T15:03:36.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfb400640 1 -- 172.21.15.43:0/291205322 >> v1:172.21.15.69:6789/0 conn(0x7f5bfc154280 legacy=0x7f5bfc15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:36.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfb400640 1 -- 172.21.15.43:0/291205322 >> v1:172.21.15.43:6789/0 conn(0x7f5bfc070850 legacy=0x7f5bfc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:36.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.700+0000 7f5bfb400640 1 -- 172.21.15.43:0/291205322 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc15e380 con 0x7f5bfc153ea0 2024-01-16T15:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.702+0000 7f5bfb400640 1 --2- 172.21.15.43:0/291205322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc153ea0 0x7f5bfc0ad450 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009d90 tx=0x7f5bec02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d42be7afcc55667 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.702+0000 7f5bfa000640 1 -- 172.21.15.43:0/291205322 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec02cd00 con 0x7f5bfc153ea0 2024-01-16T15:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.702+0000 7f5bfa000640 1 -- 172.21.15.43:0/291205322 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec02ce60 con 0x7f5bfc153ea0 2024-01-16T15:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.702+0000 7f5bfa000640 1 -- 172.21.15.43:0/291205322 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec034660 con 0x7f5bfc153ea0 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 -- 172.21.15.43:0/291205322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc153ea0 msgr2=0x7f5bfc0ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 --2- 172.21.15.43:0/291205322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc153ea0 0x7f5bfc0ad450 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009d90 tx=0x7f5bec02bf50 comp rx=0 tx=0).stop 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 -- 172.21.15.43:0/291205322 shutdown_connections 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 --2- 172.21.15.43:0/291205322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc153ea0 0x7f5bfc0ad450 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 -- 172.21.15.43:0/291205322 >> 172.21.15.43:0/291205322 conn(0x7f5bfc0cec10 msgr2=0x7f5bfc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 -- 172.21.15.43:0/291205322 shutdown_connections 2024-01-16T15:03:36.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.703+0000 7f5c00c00640 1 -- 172.21.15.43:0/291205322 wait complete. 2024-01-16T15:03:36.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.705+0000 7f5c00c00640 1 Processor -- start 2024-01-16T15:03:36.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.705+0000 7f5c00c00640 1 -- start start 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc070850 0x7f5bfc14bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc153ea0 0x7f5bfc137a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc137fc0 con 0x7f5bfc154280 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc14e1b0 con 0x7f5bfc070850 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5c00c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc14e490 con 0x7f5bfc153ea0 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5bfbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5bfbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45998/0 (socket says 172.21.15.43:45998) 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5bfbe00640 1 -- 172.21.15.43:0/1381922387 learned_addr learned my addr 172.21.15.43:0/1381922387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5bfb400640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc070850 0x7f5bfc14bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.706+0000 7f5bfaa00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc153ea0 0x7f5bfc137a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:36.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 -- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc153ea0 msgr2=0x7f5bfc137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:36.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc153ea0 0x7f5bfc137a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:36.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 -- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc070850 msgr2=0x7f5bfc14bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:36.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc070850 0x7f5bfc14bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:36.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 -- 172.21.15.43:0/1381922387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec009a70 con 0x7f5bfc154280 2024-01-16T15:03:36.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bfbe00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c450 tx=0x7f5bf400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40074e0 con 0x7f5bfc154280 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf4007660 con 0x7f5bfc154280 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4004a20 con 0x7f5bfc154280 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc14e770 con 0x7f5bfc154280 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.707+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc14ed30 con 0x7f5bfc154280 2024-01-16T15:03:36.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.709+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc14ef50 con 0x7f5bfc154280 2024-01-16T15:03:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.710+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f5bf4004b80 con 0x7f5bfc154280 2024-01-16T15:03:36.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.710+0000 7f5bf3400640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 0x7f5bd406da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:36.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.711+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5bf40890f0 con 0x7f5bfc154280 2024-01-16T15:03:36.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.711+0000 7f5bfb400640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 0x7f5bd406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:36.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.712+0000 7f5bfb400640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 0x7f5bd406da80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5bec036000 tx=0x7f5bec030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.715+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bf4053b90 con 0x7f5bfc154280 2024-01-16T15:03:36.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:36.950+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone24921", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bfc070330 con 0x7f5bd406b5c0 2024-01-16T15:03:37.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.109+0000 7f5bf3400640 1 -- 172.21.15.43:0/1381922387 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5bec034610 con 0x7f5bd406b5c0 2024-01-16T15:03:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.112+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 msgr2=0x7f5bd406da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.112+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 0x7f5bd406da80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5bec036000 tx=0x7f5bec030000 comp rx=0 tx=0).stop 2024-01-16T15:03:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.112+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 msgr2=0x7f5bfc14d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.112+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c450 tx=0x7f5bf400c910 comp rx=0 tx=0).stop 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 shutdown_connections 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bd406b5c0 0x7f5bd406da80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bfc154280 0x7f5bfc14d910 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc153ea0 0x7f5bfc137a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 --2- 172.21.15.43:0/1381922387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc070850 0x7f5bfc14bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 >> 172.21.15.43:0/1381922387 conn(0x7f5bfc0cec10 msgr2=0x7f5bfc0b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 shutdown_connections 2024-01-16T15:03:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.113+0000 7f5c00c00640 1 -- 172.21.15.43:0/1381922387 wait complete. 2024-01-16T15:03:37.095 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.116+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.095 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.116+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.095 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.116+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.095 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.116+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.095 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.116+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone24922 2024-01-16T15:03:37.121 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.138+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.121 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.138+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.121 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.138+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.121 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.138+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.122 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:37.138+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.333+0000 7f1f7d000640 1 Processor -- start 2024-01-16T15:03:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.333+0000 7f1f7d000640 1 -- start start 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.334+0000 7f1f7d000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 0x7f1f7815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.334+0000 7f1f7d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f780b35a0 0x7f1f7815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f7d000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780b43b0 con 0x7f1f780b35a0 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f7d000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7815f930 con 0x7f1f78144770 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f7d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780b3b10 con 0x7f1f7806c9d0 2024-01-16T15:03:37.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f76a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f780b35a0 0x7f1f7815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f76a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f780b35a0 0x7f1f7815f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46040/0 (socket says 172.21.15.43:46040) 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f76a00640 1 -- 172.21.15.43:0/3398114189 learned_addr learned my addr 172.21.15.43:0/3398114189 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f77400640 1 --2- 172.21.15.43:0/3398114189 >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 0x7f1f7815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.335+0000 7f1f77e00640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f76a00640 1 -- 172.21.15.43:0/3398114189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 msgr2=0x7f1f7806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f76a00640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7806ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f76a00640 1 -- 172.21.15.43:0/3398114189 >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 msgr2=0x7f1f7815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f76a00640 1 --2- 172.21.15.43:0/3398114189 >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 0x7f1f7815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f76a00640 1 -- 172.21.15.43:0/3398114189 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780b3df0 con 0x7f1f780b35a0 2024-01-16T15:03:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.336+0000 7f1f77400640 1 --2- 172.21.15.43:0/3398114189 >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 0x7f1f7815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.337+0000 7f1f76a00640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f780b35a0 0x7f1f7815f0a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1f7806df90 tx=0x7f1f6002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05d837c5554802e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.337+0000 7f1f76000640 1 -- 172.21.15.43:0/3398114189 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60031860 con 0x7f1f780b35a0 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.337+0000 7f1f76000640 1 -- 172.21.15.43:0/3398114189 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f600319c0 con 0x7f1f780b35a0 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.337+0000 7f1f76000640 1 -- 172.21.15.43:0/3398114189 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60031d50 con 0x7f1f780b35a0 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 -- 172.21.15.43:0/3398114189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f780b35a0 msgr2=0x7f1f7815f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f780b35a0 0x7f1f7815f0a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1f7806df90 tx=0x7f1f6002b900 comp rx=0 tx=0).stop 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 -- 172.21.15.43:0/3398114189 shutdown_connections 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 --2- 172.21.15.43:0/3398114189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f780b35a0 0x7f1f7815f0a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 --2- 172.21.15.43:0/3398114189 >> v2:172.21.15.69:3300/0 conn(0x7f1f78144770 0x7f1f7815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 -- 172.21.15.43:0/3398114189 >> 172.21.15.43:0/3398114189 conn(0x7f1f780cec10 msgr2=0x7f1f780cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 -- 172.21.15.43:0/3398114189 shutdown_connections 2024-01-16T15:03:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.338+0000 7f1f7d000640 1 -- 172.21.15.43:0/3398114189 wait complete. 2024-01-16T15:03:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.340+0000 7f1f7d000640 1 Processor -- start 2024-01-16T15:03:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.340+0000 7f1f7d000640 1 -- start start 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f780b35a0 0x7f1f780a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815b1b0 con 0x7f1f78144770 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781588c0 con 0x7f1f780b35a0 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f7d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78157cb0 con 0x7f1f7806c9d0 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46046/0 (socket says 172.21.15.43:46046) 2024-01-16T15:03:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f77e00640 1 -- 172.21.15.43:0/1170752005 learned_addr learned my addr 172.21.15.43:0/1170752005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f77400640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.341+0000 7f1f76a00640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f780b35a0 0x7f1f780a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 -- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 msgr2=0x7f1f7815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7815ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 -- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f780b35a0 msgr2=0x7f1f780a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f780b35a0 0x7f1f780a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 -- 172.21.15.43:0/1170752005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600094d0 con 0x7f1f78144770 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f77e00640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007e60 tx=0x7f1f7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70014620 con 0x7f1f78144770 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7000ee90 con 0x7f1f78144770 2024-01-16T15:03:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700149e0 con 0x7f1f78144770 2024-01-16T15:03:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.342+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78157f90 con 0x7f1f78144770 2024-01-16T15:03:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.343+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f78158550 con 0x7f1f78144770 2024-01-16T15:03:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.344+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7804e840 con 0x7f1f78144770 2024-01-16T15:03:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.345+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f1f70014b40 con 0x7f1f78144770 2024-01-16T15:03:37.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.345+0000 7f1f6f400640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 0x7f1f5006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.346+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f7008a9b0 con 0x7f1f78144770 2024-01-16T15:03:37.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.346+0000 7f1f77400640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 0x7f1f5006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.347+0000 7f1f77400640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 0x7f1f5006da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.350+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f70055450 con 0x7f1f78144770 2024-01-16T15:03:37.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.579+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone24922", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f78073b80 con 0x7f1f5006b590 2024-01-16T15:03:37.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.725+0000 7f1f6f400640 1 -- 172.21.15.43:0/1170752005 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1f78073b80 con 0x7f1f5006b590 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 msgr2=0x7f1f5006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 0x7f1f5006da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 msgr2=0x7f1f78155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007e60 tx=0x7f1f7000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 shutdown_connections 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f5006b590 0x7f1f5006da50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f780b35a0 0x7f1f780a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f78144770 0x7f1f78155280 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 --2- 172.21.15.43:0/1170752005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f7806c9d0 0x7f1f7815ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.729+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 >> 172.21.15.43:0/1170752005 conn(0x7f1f780cec10 msgr2=0x7f1f780ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.730+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 shutdown_connections 2024-01-16T15:03:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.730+0000 7f1f7d000640 1 -- 172.21.15.43:0/1170752005 wait complete. 2024-01-16T15:03:37.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone24923 2024-01-16T15:03:37.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.980+0000 7f7e37200640 1 Processor -- start 2024-01-16T15:03:37.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.980+0000 7f7e37200640 1 -- start start 2024-01-16T15:03:37.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e381435d0 0x7f7e38071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e381439b0 0x7f7e380b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380b3f70 0x7f7e38070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e380b43e0 con 0x7f7e381439b0 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e38070d40 con 0x7f7e381435d0 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e37200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e38071020 con 0x7f7e380b3f70 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e35400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e381439b0 0x7f7e380b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e35400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e381439b0 0x7f7e380b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46050/0 (socket says 172.21.15.43:46050) 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.982+0000 7f7e35400640 1 -- 172.21.15.43:0/3075174435 learned_addr learned my addr 172.21.15.43:0/3075174435 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e36800640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380b3f70 0x7f7e38070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35e00640 1 --2- 172.21.15.43:0/3075174435 >> v2:172.21.15.69:3300/0 conn(0x7f7e381435d0 0x7f7e38071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35400640 1 -- 172.21.15.43:0/3075174435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380b3f70 msgr2=0x7f7e38070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35400640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380b3f70 0x7f7e38070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35400640 1 -- 172.21.15.43:0/3075174435 >> v2:172.21.15.69:3300/0 conn(0x7f7e381435d0 msgr2=0x7f7e38071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:37.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35400640 1 --2- 172.21.15.43:0/3075174435 >> v2:172.21.15.69:3300/0 conn(0x7f7e381435d0 0x7f7e38071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.983+0000 7f7e35400640 1 -- 172.21.15.43:0/3075174435 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38071300 con 0x7f7e381439b0 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.984+0000 7f7e35400640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e381439b0 0x7f7e380b3a30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7e380cfc10 tx=0x7f7e2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb673c189fcdce8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.984+0000 7f7e34a00640 1 -- 172.21.15.43:0/3075174435 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20031870 con 0x7f7e381439b0 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.984+0000 7f7e34a00640 1 -- 172.21.15.43:0/3075174435 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e200319d0 con 0x7f7e381439b0 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.984+0000 7f7e34a00640 1 -- 172.21.15.43:0/3075174435 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20031d60 con 0x7f7e381439b0 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 -- 172.21.15.43:0/3075174435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e381439b0 msgr2=0x7f7e380b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e381439b0 0x7f7e380b3a30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7e380cfc10 tx=0x7f7e2002b910 comp rx=0 tx=0).stop 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 -- 172.21.15.43:0/3075174435 shutdown_connections 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380b3f70 0x7f7e38070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 --2- 172.21.15.43:0/3075174435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e381439b0 0x7f7e380b3a30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 --2- 172.21.15.43:0/3075174435 >> v2:172.21.15.69:3300/0 conn(0x7f7e381435d0 0x7f7e38071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 -- 172.21.15.43:0/3075174435 >> 172.21.15.43:0/3075174435 conn(0x7f7e380cec10 msgr2=0x7f7e380cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:37.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 -- 172.21.15.43:0/3075174435 shutdown_connections 2024-01-16T15:03:37.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.985+0000 7f7e37200640 1 -- 172.21.15.43:0/3075174435 wait complete. 2024-01-16T15:03:37.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.987+0000 7f7e37200640 1 Processor -- start 2024-01-16T15:03:37.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.987+0000 7f7e37200640 1 -- start start 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 0x7f7e380af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e381439b0 0x7f7e38158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380cdf90 con 0x7f7e380b3f70 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380cd380 con 0x7f7e381435d0 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e37200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380cd660 con 0x7f7e381439b0 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e35e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.988+0000 7f7e35e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46060/0 (socket says 172.21.15.43:46060) 2024-01-16T15:03:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 -- 172.21.15.43:0/3070954084 learned_addr learned my addr 172.21.15.43:0/3070954084 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 -- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e381439b0 msgr2=0x7f7e38158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e381439b0 0x7f7e38158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 -- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 msgr2=0x7f7e380af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35400640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 0x7f7e380af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 0x7f7e380af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 -- 172.21.15.43:0/3070954084 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200094d0 con 0x7f7e380b3f70 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35400640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 0x7f7e380af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.989+0000 7f7e35e00640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800d800 tx=0x7f7e2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.990+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800b6c0 con 0x7f7e380b3f70 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.990+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800b820 con 0x7f7e380b3f70 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.990+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800bbb0 con 0x7f7e380b3f70 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.990+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e380cd940 con 0x7f7e380b3f70 2024-01-16T15:03:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.990+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e380ccfe0 con 0x7f7e380b3f70 2024-01-16T15:03:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.991+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3804e840 con 0x7f7e380b3f70 2024-01-16T15:03:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.992+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f7e2800bd10 con 0x7f7e380b3f70 2024-01-16T15:03:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.993+0000 7f7e2e000640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 0x7f7e1006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.993+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e2808a640 con 0x7f7e380b3f70 2024-01-16T15:03:37.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.993+0000 7f7e35400640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 0x7f7e1006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:37.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.994+0000 7f7e35400640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 0x7f7e1006da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7e20036000 tx=0x7f7e2003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:37.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:37.998+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e280550e0 con 0x7f7e380b3f70 2024-01-16T15:03:38.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:38.226+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone24923", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e380cd170 con 0x7f7e1006b590 2024-01-16T15:03:39.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:39.823+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f7e280554b0 con 0x7f7e380b3f70 2024-01-16T15:03:40.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.711+0000 7f7e2e000640 1 -- 172.21.15.43:0/3070954084 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e380cd170 con 0x7f7e1006b590 2024-01-16T15:03:40.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 msgr2=0x7f7e1006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 0x7f7e1006da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7e20036000 tx=0x7f7e2003c040 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 msgr2=0x7f7e380ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800d800 tx=0x7f7e2800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 shutdown_connections 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e1006b590 0x7f7e1006da50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e381435d0 0x7f7e380af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e381439b0 0x7f7e38158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 --2- 172.21.15.43:0/3070954084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e380b3f70 0x7f7e380ccaa0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 >> 172.21.15.43:0/3070954084 conn(0x7f7e380cec10 msgr2=0x7f7e380ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.716+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 shutdown_connections 2024-01-16T15:03:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.717+0000 7f7e37200640 1 -- 172.21.15.43:0/3070954084 wait complete. 2024-01-16T15:03:40.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone24924 2024-01-16T15:03:40.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.965+0000 7facdac00640 1 Processor -- start 2024-01-16T15:03:40.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.965+0000 7facdac00640 1 -- start start 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.966+0000 7facdac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 0x7facd40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.966+0000 7facdac00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7facd409cb30 0x7facd40b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.966+0000 7facdac00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facd40adb60 0x7facd40adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.966+0000 7facdac00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd415b730 con 0x7facd40adb60 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facdac00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd40ac740 con 0x7facd409cb30 2024-01-16T15:03:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facdac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40ac460 con 0x7facd415b350 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facd40adb60 0x7facd40adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facd40adb60 0x7facd40adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46064/0 (socket says 172.21.15.43:46064) 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 -- 172.21.15.43:0/2573137772 learned_addr learned my addr 172.21.15.43:0/2573137772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd3400640 1 --2- 172.21.15.43:0/2573137772 >> v2:172.21.15.69:3300/0 conn(0x7facd409cb30 0x7facd40b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd3e00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 0x7facd40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 -- 172.21.15.43:0/2573137772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 msgr2=0x7facd40abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 0x7facd40abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 -- 172.21.15.43:0/2573137772 >> v2:172.21.15.69:3300/0 conn(0x7facd409cb30 msgr2=0x7facd40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 --2- 172.21.15.43:0/2573137772 >> v2:172.21.15.69:3300/0 conn(0x7facd409cb30 0x7facd40b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd8c00640 1 -- 172.21.15.43:0/2573137772 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40b2a50 con 0x7facd40adb60 2024-01-16T15:03:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.967+0000 7facd3e00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 0x7facd40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facd8c00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd40adfb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7facd40b5130 tx=0x7faccc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17483accb130ec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facd2a00640 1 -- 172.21.15.43:0/2573137772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc02ca80 con 0x7facd40adb60 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facd2a00640 1 -- 172.21.15.43:0/2573137772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc02cbe0 con 0x7facd40adb60 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facd2a00640 1 -- 172.21.15.43:0/2573137772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0343f0 con 0x7facd40adb60 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facdac00640 1 -- 172.21.15.43:0/2573137772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 msgr2=0x7facd40adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.969+0000 7facdac00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd40adfb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7facd40b5130 tx=0x7faccc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 -- 172.21.15.43:0/2573137772 shutdown_connections 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd40adfb0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 --2- 172.21.15.43:0/2573137772 >> v2:172.21.15.69:3300/0 conn(0x7facd409cb30 0x7facd40b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 --2- 172.21.15.43:0/2573137772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd415b350 0x7facd40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 -- 172.21.15.43:0/2573137772 >> 172.21.15.43:0/2573137772 conn(0x7facd40b0ba0 msgr2=0x7facd40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 -- 172.21.15.43:0/2573137772 shutdown_connections 2024-01-16T15:03:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.970+0000 7facdac00640 1 -- 172.21.15.43:0/2573137772 wait complete. 2024-01-16T15:03:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.971+0000 7facdac00640 1 Processor -- start 2024-01-16T15:03:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.972+0000 7facdac00640 1 -- start start 2024-01-16T15:03:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.972+0000 7facdac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd409cb30 0x7facd4141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facdac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facdac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 0x7facd4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facdac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4148510 con 0x7facd40adb60 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facdac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4142090 con 0x7facd415b350 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facdac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd4142370 con 0x7facd409cb30 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46074/0 (socket says 172.21.15.43:46074) 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 -- 172.21.15.43:0/1472734496 learned_addr learned my addr 172.21.15.43:0/1472734496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3e00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd409cb30 0x7facd4141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd8c00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 0x7facd4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 -- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd409cb30 msgr2=0x7facd4141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd409cb30 0x7facd4141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 -- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 msgr2=0x7facd4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 0x7facd4159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd3400640 1 -- 172.21.15.43:0/1472734496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc009b30 con 0x7facd40adb60 2024-01-16T15:03:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.973+0000 7facd8c00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 0x7facd4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7facd3400640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7facbc00d800 tx=0x7facbc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc0096a0 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc009800 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc009990 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4142650 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.974+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd4141d80 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.975+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd409d200 con 0x7facd40adb60 2024-01-16T15:03:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.977+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7facbc009af0 con 0x7facd40adb60 2024-01-16T15:03:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.977+0000 7faccbe00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 0x7facac06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.978+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7facbc08a780 con 0x7facd40adb60 2024-01-16T15:03:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.978+0000 7facd3e00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 0x7facac06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.978+0000 7facd3e00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 0x7facac06db70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7facd414d3c0 tx=0x7facc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:40.982+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facbc055080 con 0x7facd40adb60 2024-01-16T15:03:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:41.214+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone24924", "target": ["mon-mgr", ""]}) v1 -- 0x7facd4142200 con 0x7facac06b6b0 2024-01-16T15:03:45.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.717+0000 7faccbe00640 1 -- 172.21.15.43:0/1472734496 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7facd4142200 con 0x7facac06b6b0 2024-01-16T15:03:45.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.721+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 msgr2=0x7facac06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.721+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 0x7facac06db70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7facd414d3c0 tx=0x7facc4008040 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.721+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 msgr2=0x7facd4159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.721+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7facbc00d800 tx=0x7facbc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 shutdown_connections 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facac06b6b0 0x7facac06db70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facd415b350 0x7facd4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd40adb60 0x7facd4159950 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 --2- 172.21.15.43:0/1472734496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facd409cb30 0x7facd4141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 >> 172.21.15.43:0/1472734496 conn(0x7facd40b0ba0 msgr2=0x7facd4149730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 shutdown_connections 2024-01-16T15:03:45.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.722+0000 7facdac00640 1 -- 172.21.15.43:0/1472734496 wait complete. 2024-01-16T15:03:45.726 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:03:45.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24921 2024-01-16T15:03:45.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.969+0000 7f13ac800640 1 Processor -- start 2024-01-16T15:03:45.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.969+0000 7f13ac800640 1 -- start start 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13ac800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13a40ae270 0x7f13a40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13ac800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a415b730 con 0x7f13a409cb30 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13ac800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40ac740 con 0x7f13a415b350 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13ac800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40ac460 con 0x7f13a40ae270 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13a9400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f13a409cb30 0x7f13a40adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54038/0 (socket says 172.21.15.43:54038) 2024-01-16T15:03:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.971+0000 7f13a9400640 1 -- 172.21.15.43:0/2251996852 learned_addr learned my addr 172.21.15.43:0/2251996852 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13aa800640 1 --2- 172.21.15.43:0/2251996852 >> v2:172.21.15.69:3300/0 conn(0x7f13a40ae270 0x7f13a40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495905578 0 0) 0x7f138c001110 con 0x7f13a409cb30 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1384003520 con 0x7f13a409cb30 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165358558 0 0) 0x7f1384003520 con 0x7f13a409cb30 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 >> v2:172.21.15.69:3300/0 conn(0x7f13a40ae270 msgr2=0x7f13a40abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 --2- 172.21.15.43:0/2251996852 >> v2:172.21.15.69:3300/0 conn(0x7f13a40ae270 0x7f13a40abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 >> v1:172.21.15.69:6789/0 conn(0x7f13a415b350 legacy=0x7f13a40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.972+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40b2a50 con 0x7f13a409cb30 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f138c001800 con 0x7f13a409cb30 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 >> v1:172.21.15.43:6789/0 conn(0x7f13a409cb30 legacy=0x7f13a40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1384003970 0x7f1384003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 0x7f1384004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1384005180 0x7f138400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40ac460 con 0x7f13840047c0 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138400c020 con 0x7f1384005180 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138400c1e0 con 0x7f1384003970 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a9400640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 0x7f1384004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13aa800640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1384005180 0x7f138400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a9e00640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1384003970 0x7f1384003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.973+0000 7f13a9400640 1 -- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1384003970 msgr2=0x7f1384003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13a9400640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1384003970 0x7f1384003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13a9400640 1 -- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1384005180 msgr2=0x7f138400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13a9400640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1384005180 0x7f138400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13a9400640 1 -- 172.21.15.43:0/2251996852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384003520 con 0x7f13840047c0 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13aa800640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1384005180 0x7f138400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:45.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.974+0000 7f13aa800640 1 -- 172.21.15.43:0/2251996852 reap_dead start 2024-01-16T15:03:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13a9400640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 0x7f1384004c40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c240 tx=0x7f13840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1e5cf9945e51fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c005ad0 con 0x7f13840047c0 2024-01-16T15:03:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c005c50 con 0x7f13840047c0 2024-01-16T15:03:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13a8a00640 1 -- 172.21.15.43:0/2251996852 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c02c830 con 0x7f13840047c0 2024-01-16T15:03:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13ac800640 1 -- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 msgr2=0x7f1384004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.975+0000 7f13ac800640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 0x7f1384004c40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f138c00c240 tx=0x7f13840057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.976+0000 7f13ac800640 1 -- 172.21.15.43:0/2251996852 shutdown_connections 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.976+0000 7f13ac800640 1 --2- 172.21.15.43:0/2251996852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13840047c0 0x7f1384004c40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.976+0000 7f13ac800640 1 -- 172.21.15.43:0/2251996852 >> 172.21.15.43:0/2251996852 conn(0x7f13a40b0ba0 msgr2=0x7f13a40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.976+0000 7f13ac800640 1 -- 172.21.15.43:0/2251996852 shutdown_connections 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.976+0000 7f13ac800640 1 -- 172.21.15.43:0/2251996852 wait complete. 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.978+0000 7f13ac800640 1 Processor -- start 2024-01-16T15:03:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.978+0000 7f13ac800640 1 -- start start 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13840047c0 0x7f13a4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a413bba0 0x7f13a4157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a4155710 con 0x7f13a4154d50 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a414d8e0 con 0x7f13a413bba0 2024-01-16T15:03:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13ac800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41546e0 con 0x7f13840047c0 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13aa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13aa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48652/0 (socket says 172.21.15.43:48652) 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13aa800640 1 -- 172.21.15.43:0/3804419866 learned_addr learned my addr 172.21.15.43:0/3804419866 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13a9400640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a413bba0 0x7f13a4157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.979+0000 7f13a9e00640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13840047c0 0x7f13a4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 -- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13840047c0 msgr2=0x7f13a4153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13840047c0 0x7f13a4153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 -- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a413bba0 msgr2=0x7f13a4157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a413bba0 0x7f13a4157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 -- 172.21.15.43:0/3804419866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a4158470 con 0x7f13a4154d50 2024-01-16T15:03:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13aa800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f13a40b1bf0 tx=0x7f139c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c016420 con 0x7f13a4154d50 2024-01-16T15:03:45.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c016580 con 0x7f13a4154d50 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c0168f0 con 0x7f13a4154d50 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.980+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4153e40 con 0x7f13a4154d50 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.981+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a4154400 con 0x7f13a4154d50 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.982+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a413bf80 con 0x7f13a4154d50 2024-01-16T15:03:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.983+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f139c016a50 con 0x7f13a4154d50 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.984+0000 7f13a2000640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 0x7f137c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.984+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f139c08b250 con 0x7f13a4154d50 2024-01-16T15:03:45.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.984+0000 7f13a9e00640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 0x7f137c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:45.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.985+0000 7f13a9e00640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 0x7f137c06dba0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1394000c00 tx=0x7f1394009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:45.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:45.989+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f139c055950 con 0x7f13a4154d50 2024-01-16T15:03:46.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:46.226+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24921", "target": ["mon-mgr", ""]}) v1 -- 0x7f13a414d740 con 0x7f137c06b6e0 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.567+0000 7f13a2000640 1 -- 172.21.15.43:0/3804419866 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f139400d050 con 0x7f137c06b6e0 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:03:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:03:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:03:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:03:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:50.548 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 msgr2=0x7f137c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 0x7f137c06dba0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1394000c00 tx=0x7f1394009040 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 msgr2=0x7f13a41551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f13a40b1bf0 tx=0x7f139c00f520 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 shutdown_connections 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f137c06b6e0 0x7f137c06dba0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a413bba0 0x7f13a4157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a4154d50 0x7f13a41551d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 --2- 172.21.15.43:0/3804419866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13840047c0 0x7f13a4153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.571+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 >> 172.21.15.43:0/3804419866 conn(0x7f13a40b0ba0 msgr2=0x7f13a40b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:50.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.572+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 shutdown_connections 2024-01-16T15:03:50.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:50.572+0000 7f13ac800640 1 -- 172.21.15.43:0/3804419866 wait complete. 2024-01-16T15:03:50.573 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:03:51.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24921 2024-01-16T15:03:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.831+0000 7f72c5c00640 1 Processor -- start 2024-01-16T15:03:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.832+0000 7f72c5c00640 1 -- start start 2024-01-16T15:03:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.833+0000 7f72c5c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72c00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72c5c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c015b730 con 0x7f72c00adb60 2024-01-16T15:03:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72c5c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c00ac740 con 0x7f72c009cb30 2024-01-16T15:03:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72c5c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c00ac460 con 0x7f72c015b350 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72bf400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f72c00adb60 0x7f72c00adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54052/0 (socket says 172.21.15.43:54052) 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72bf400640 1 -- 172.21.15.43:0/2998000772 learned_addr learned my addr 172.21.15.43:0/2998000772 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.834+0000 7f72bea00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72c00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335101704 0 0) 0x7f72b8001490 con 0x7f72c00adb60 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a0003520 con 0x7f72c00adb60 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545548625 0 0) 0x7f72a0003520 con 0x7f72c00adb60 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 msgr2=0x7f72c00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72c00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 >> v1:172.21.15.69:6789/0 conn(0x7f72c009cb30 legacy=0x7f72c00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c00b2a50 con 0x7f72c00adb60 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f72b8001b80 con 0x7f72c00adb60 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 >> v1:172.21.15.43:6789/0 conn(0x7f72c00adb60 legacy=0x7f72c00adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72a00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72a0003f20 0x7f72a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0004d00 0x7f72a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c00ac460 con 0x7f72a0003f20 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a000ba60 con 0x7f72a0004d00 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a000bc20 con 0x7f72c015b350 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.835+0000 7f72b7e00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72a0003f20 0x7f72a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72bea00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72a00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72bf400640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0004d00 0x7f72a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72b7e00640 1 -- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 msgr2=0x7f72a00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72b7e00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c015b350 0x7f72a00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72b7e00640 1 -- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0004d00 msgr2=0x7f72a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72b7e00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0004d00 0x7f72a0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.836+0000 7f72b7e00640 1 -- 172.21.15.43:0/2998000772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0003520 con 0x7f72a0003f20 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.837+0000 7f72b7e00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72a0003f20 0x7f72a00047c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f72a80015b0 tx=0x7f72a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6422d59c4562420d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.837+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80052d0 con 0x7f72a0003f20 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.837+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a802e050 con 0x7f72a0003f20 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.837+0000 7f72be000640 1 -- 172.21.15.43:0/2998000772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8003830 con 0x7f72a0003f20 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 -- 172.21.15.43:0/2998000772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72a0003f20 msgr2=0x7f72a00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 --2- 172.21.15.43:0/2998000772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72a0003f20 0x7f72a00047c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f72a80015b0 tx=0x7f72a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72bf400640 1 -- 172.21.15.43:0/2998000772 reap_dead start 2024-01-16T15:03:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 -- 172.21.15.43:0/2998000772 shutdown_connections 2024-01-16T15:03:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 -- 172.21.15.43:0/2998000772 >> 172.21.15.43:0/2998000772 conn(0x7f72c00b0ba0 msgr2=0x7f72c00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 -- 172.21.15.43:0/2998000772 shutdown_connections 2024-01-16T15:03:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.838+0000 7f72c5c00640 1 -- 172.21.15.43:0/2998000772 wait complete. 2024-01-16T15:03:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.840+0000 7f72c5c00640 1 Processor -- start 2024-01-16T15:03:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.840+0000 7f72c5c00640 1 -- start start 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0003f20 0x7f72c01e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c01594a0 0x7f72c01f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01e3be0 con 0x7f72c0071530 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01e2fd0 con 0x7f72a0003f20 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72c5c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01e32b0 con 0x7f72c01594a0 2024-01-16T15:03:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48662/0 (socket says 172.21.15.43:48662) 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.841+0000 7f72b7e00640 1 -- 172.21.15.43:0/3724566482 learned_addr learned my addr 172.21.15.43:0/3724566482 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72bea00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0003f20 0x7f72c01e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72bf400640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c01594a0 0x7f72c01f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 -- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c01594a0 msgr2=0x7f72c01f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c01594a0 0x7f72c01f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 -- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0003f20 msgr2=0x7f72c01e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0003f20 0x7f72c01e4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 -- 172.21.15.43:0/3724566482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c01e3590 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.842+0000 7f72b7e00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f72a8008fd0 tx=0x7f72a8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.843+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a80076c0 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.843+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8007a30 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.843+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8006be0 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.843+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c01e3870 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.843+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c01e2da0 con 0x7f72c0071530 2024-01-16T15:03:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.844+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c01e3d90 con 0x7f72c0071530 2024-01-16T15:03:51.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.845+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f72a8006d40 con 0x7f72c0071530 2024-01-16T15:03:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.846+0000 7f72b6a00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 0x7f729806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.846+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f72a80ac630 con 0x7f72c0071530 2024-01-16T15:03:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.846+0000 7f72bea00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 0x7f729806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:51.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.847+0000 7f72bea00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 0x7f729806dba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009a10 tx=0x7f72b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:51.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:51.851+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72c01e3d90 con 0x7f72c0071530 2024-01-16T15:03:52.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.089+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24921", "target": ["mon-mgr", ""]}) v1 -- 0x7f72c01e3420 con 0x7f729806b6e0 2024-01-16T15:03:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.107+0000 7f72b6a00640 1 -- 172.21.15.43:0/3724566482 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f72c01e3420 con 0x7f729806b6e0 2024-01-16T15:03:52.086 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 0.0, 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "0/3", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.088 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:03:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 msgr2=0x7f729806dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 0x7f729806dba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009a10 tx=0x7f72b0008040 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 msgr2=0x7f72c01e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f72a8008fd0 tx=0x7f72a8001120 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 shutdown_connections 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f729806b6e0 0x7f729806dba0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72c01594a0 0x7f72c01f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72c0071530 0x7f72c01e2680 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 --2- 172.21.15.43:0/3724566482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a0003f20 0x7f72c01e4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 >> 172.21.15.43:0/3724566482 conn(0x7f72c00b0ba0 msgr2=0x7f72c00afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 shutdown_connections 2024-01-16T15:03:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.110+0000 7f72c5c00640 1 -- 172.21.15.43:0/3724566482 wait complete. 2024-01-16T15:03:52.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:52.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.340+0000 7f32cfe00640 1 Processor -- start 2024-01-16T15:03:52.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.340+0000 7f32cfe00640 1 -- start start 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.341+0000 7f32cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32d0143570 0x7f32d00cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.341+0000 7f32cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 0x7f32d00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.341+0000 7f32cfe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d0071700 con 0x7f32d0143950 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.341+0000 7f32cfe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d015b410 con 0x7f32d0143570 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.341+0000 7f32cfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d015b130 con 0x7f32d015e6b0 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32d0143570 0x7f32d00cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f32d0143570 0x7f32d00cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48666/0 (socket says 172.21.15.43:48666) 2024-01-16T15:03:52.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 -- 172.21.15.43:0/1381477377 learned_addr learned my addr 172.21.15.43:0/1381477377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:52.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 -- 172.21.15.43:0/1381477377 >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 msgr2=0x7f32d00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:52.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cf400640 1 --2- 172.21.15.43:0/1381477377 >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 0x7f32d00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 --2- 172.21.15.43:0/1381477377 >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 0x7f32d00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 -- 172.21.15.43:0/1381477377 >> v1:172.21.15.69:6789/0 conn(0x7f32d0143950 legacy=0x7f32d015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cea00640 1 -- 172.21.15.43:0/1381477377 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d015dbd0 con 0x7f32d0143570 2024-01-16T15:03:52.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.342+0000 7f32cf400640 1 --2- 172.21.15.43:0/1381477377 >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 0x7f32d00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:52.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.344+0000 7f32cea00640 1 --2- 172.21.15.43:0/1381477377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d00cf050 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f32d00cbe10 tx=0x7f32c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50458ac5ca1ec82c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:52.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.344+0000 7f32ce000640 1 -- 172.21.15.43:0/1381477377 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0031870 con 0x7f32d0143570 2024-01-16T15:03:52.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.344+0000 7f32ce000640 1 -- 172.21.15.43:0/1381477377 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c00319d0 con 0x7f32d0143570 2024-01-16T15:03:52.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.344+0000 7f32ce000640 1 -- 172.21.15.43:0/1381477377 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0031d60 con 0x7f32d0143570 2024-01-16T15:03:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 -- 172.21.15.43:0/1381477377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 msgr2=0x7f32d00cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 --2- 172.21.15.43:0/1381477377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d00cf050 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f32d00cbe10 tx=0x7f32c002b910 comp rx=0 tx=0).stop 2024-01-16T15:03:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 -- 172.21.15.43:0/1381477377 shutdown_connections 2024-01-16T15:03:52.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 --2- 172.21.15.43:0/1381477377 >> v2:172.21.15.69:3300/0 conn(0x7f32d015e6b0 0x7f32d00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 --2- 172.21.15.43:0/1381477377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d00cf050 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 -- 172.21.15.43:0/1381477377 >> 172.21.15.43:0/1381477377 conn(0x7f32d006a860 msgr2=0x7f32d006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:52.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 -- 172.21.15.43:0/1381477377 shutdown_connections 2024-01-16T15:03:52.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.345+0000 7f32cfe00640 1 -- 172.21.15.43:0/1381477377 wait complete. 2024-01-16T15:03:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.347+0000 7f32cfe00640 1 Processor -- start 2024-01-16T15:03:52.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.347+0000 7f32cfe00640 1 -- start start 2024-01-16T15:03:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 0x7f32d0141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d015e6b0 0x7f32d00cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d013e2e0 con 0x7f32d0143570 2024-01-16T15:03:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d013e5c0 con 0x7f32d015e6b0 2024-01-16T15:03:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d013e8a0 con 0x7f32d0143950 2024-01-16T15:03:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48668/0 (socket says 172.21.15.43:48668) 2024-01-16T15:03:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32cea00640 1 -- 172.21.15.43:0/2714413172 learned_addr learned my addr 172.21.15.43:0/2714413172 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.348+0000 7f32c7e00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 0x7f32d0141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 -- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 msgr2=0x7f32d0141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cf400640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d015e6b0 0x7f32d00cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 0x7f32d0141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 -- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d015e6b0 msgr2=0x7f32d00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d015e6b0 0x7f32d00cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 -- 172.21.15.43:0/2714413172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00094d0 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32c7e00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 0x7f32d0141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cea00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32c00090b0 tx=0x7f32c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c003adf0 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c00443f0 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c00447a0 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d013eb80 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.349+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d013dfb0 con 0x7f32d0143570 2024-01-16T15:03:52.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.350+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d013f0a0 con 0x7f32d0143570 2024-01-16T15:03:52.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.352+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f32c004d450 con 0x7f32d0143570 2024-01-16T15:03:52.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.353+0000 7f32c7400640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 0x7f32a806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:52.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.353+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f32c002d040 con 0x7f32d0143570 2024-01-16T15:03:52.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.353+0000 7f32c7e00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 0x7f32a806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.354+0000 7f32c7e00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 0x7f32a806db70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32d013c380 tx=0x7f32b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:52.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.357+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32d013f0a0 con 0x7f32d0143570 2024-01-16T15:03:52.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.622+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f32d004e840 con 0x7f32d0143570 2024-01-16T15:03:52.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.623+0000 7f32c7400640 1 -- 172.21.15.43:0/2714413172 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2092 (secure 0 0 0) 0x7f32c0035090 con 0x7f32d0143570 2024-01-16T15:03:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:03:52.602 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:03:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 284, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:03:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 800, 2024-01-16T15:03:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222208026, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 6721560576, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766372552704, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 597, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 18594, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 5 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:03:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:03:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:03:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:03:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:52.609 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.625+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 msgr2=0x7f32a806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.625+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 0x7f32a806db70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32d013c380 tx=0x7f32b8009040 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 msgr2=0x7f32d013f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32c00090b0 tx=0x7f32c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 shutdown_connections 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f32a806b6b0 0x7f32a806db70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d015e6b0 0x7f32d00cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0143950 0x7f32d0141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 --2- 172.21.15.43:0/2714413172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32d0143570 0x7f32d013f380 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 >> 172.21.15.43:0/2714413172 conn(0x7f32d006a860 msgr2=0x7f32d00b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 shutdown_connections 2024-01-16T15:03:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:52.626+0000 7f32cfe00640 1 -- 172.21.15.43:0/2714413172 wait complete. 2024-01-16T15:03:54.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:54.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:54.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.095+0000 7fde3c800640 1 Processor -- start 2024-01-16T15:03:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.097+0000 7fde3c800640 1 -- start start 2024-01-16T15:03:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.101+0000 7fde3c800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde340719c0 0x7fde340ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.101+0000 7fde3c800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340ad510 con 0x7fde3406d520 2024-01-16T15:03:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.102+0000 7fde39e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fde34161930 0x7fde340ad9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:47860/0 (socket says 172.21.15.43:47860) 2024-01-16T15:03:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.102+0000 7fde39e00640 1 -- 172.21.15.43:0/1058672804 learned_addr learned my addr 172.21.15.43:0/1058672804 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:03:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.102+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340acc50 con 0x7fde34161930 2024-01-16T15:03:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.102+0000 7fde39400640 1 --2- 172.21.15.43:0/1058672804 >> v2:172.21.15.69:3300/0 conn(0x7fde340719c0 0x7fde340ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.103+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde34071e60 con 0x7fde340719c0 2024-01-16T15:03:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.103+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107822208 0 0) 0x7fde340acc50 con 0x7fde34161930 2024-01-16T15:03:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.104+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde14003520 con 0x7fde34161930 2024-01-16T15:03:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.105+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017042646 0 0) 0x7fde340ad510 con 0x7fde3406d520 2024-01-16T15:03:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.105+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde340acc50 con 0x7fde3406d520 2024-01-16T15:03:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.106+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658524960 0 0) 0x7fde14003520 con 0x7fde34161930 2024-01-16T15:03:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.106+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 >> v2:172.21.15.69:3300/0 conn(0x7fde340719c0 msgr2=0x7fde340ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.106+0000 7fde38a00640 1 --2- 172.21.15.43:0/1058672804 >> v2:172.21.15.69:3300/0 conn(0x7fde340719c0 0x7fde340ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.106+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 >> v1:172.21.15.43:6789/0 conn(0x7fde3406d520 legacy=0x7fde3406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.106+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde340acec0 con 0x7fde34161930 2024-01-16T15:03:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fde2c0019f0 con 0x7fde34161930 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 >> v1:172.21.15.69:6789/0 conn(0x7fde34161930 legacy=0x7fde340ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14003b00 0x7fde14003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 0x7fde14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde14005180 0x7fde1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c001230 con 0x7fde14003b00 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34071e60 con 0x7fde14005180 2024-01-16T15:03:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.107+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400c050 con 0x7fde140047c0 2024-01-16T15:03:55.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.108+0000 7fde3a800640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde14005180 0x7fde1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.108+0000 7fde39400640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 0x7fde14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.108+0000 7fde39e00640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14003b00 0x7fde14003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.109+0000 7fde39400640 1 -- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde14005180 msgr2=0x7fde1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.109+0000 7fde39400640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde14005180 0x7fde1400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.109+0000 7fde39400640 1 -- 172.21.15.43:0/1058672804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14003b00 msgr2=0x7fde14003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.109+0000 7fde39400640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14003b00 0x7fde14003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.109+0000 7fde39400640 1 -- 172.21.15.43:0/1058672804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14003520 con 0x7fde140047c0 2024-01-16T15:03:55.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.110+0000 7fde3a800640 1 -- 172.21.15.43:0/1058672804 reap_dead start 2024-01-16T15:03:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.115+0000 7fde39400640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 0x7fde14004c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde24009aa0 tx=0x7fde140057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=155f49df705910ae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.116+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2402c6b0 con 0x7fde140047c0 2024-01-16T15:03:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.116+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2402c810 con 0x7fde140047c0 2024-01-16T15:03:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.117+0000 7fde38a00640 1 -- 172.21.15.43:0/1058672804 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2402cc00 con 0x7fde140047c0 2024-01-16T15:03:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 msgr2=0x7fde14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 0x7fde14004c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde24009aa0 tx=0x7fde140057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 shutdown_connections 2024-01-16T15:03:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 --2- 172.21.15.43:0/1058672804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde140047c0 0x7fde14004c40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 >> 172.21.15.43:0/1058672804 conn(0x7fde340b0d80 msgr2=0x7fde340b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 shutdown_connections 2024-01-16T15:03:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.118+0000 7fde3c800640 1 -- 172.21.15.43:0/1058672804 wait complete. 2024-01-16T15:03:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.120+0000 7fde3c800640 1 Processor -- start 2024-01-16T15:03:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.121+0000 7fde3c800640 1 -- start start 2024-01-16T15:03:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde140047c0 0x7fde341f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde34154840 0x7fde34164240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde341fb9f0 con 0x7fde34164780 2024-01-16T15:03:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34164c30 con 0x7fde140047c0 2024-01-16T15:03:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3c800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde34154cc0 con 0x7fde34154840 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52736/0 (socket says 172.21.15.43:52736) 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.122+0000 7fde3a800640 1 -- 172.21.15.43:0/3027810090 learned_addr learned my addr 172.21.15.43:0/3027810090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde39e00640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde140047c0 0x7fde341f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde39400640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde34154840 0x7fde34164240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 -- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde34154840 msgr2=0x7fde34164240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde34154840 0x7fde34164240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 -- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde140047c0 msgr2=0x7fde341f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde140047c0 0x7fde341f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 -- 172.21.15.43:0/3027810090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34162210 con 0x7fde34164780 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.123+0000 7fde3a800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fde340cb7d0 tx=0x7fde1c00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.124+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0052e0 con 0x7fde34164780 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.124+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c005440 con 0x7fde34164780 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.124+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde341624f0 con 0x7fde34164780 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.124+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c00d8c0 con 0x7fde34164780 2024-01-16T15:03:55.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.124+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde34162ab0 con 0x7fde34164780 2024-01-16T15:03:55.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.125+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde34074770 con 0x7fde34164780 2024-01-16T15:03:55.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.126+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 10) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fde1c00da20 con 0x7fde34164780 2024-01-16T15:03:55.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.130+0000 7fde32000640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 0x7fde0c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:55.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.130+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fde1c08a290 con 0x7fde34164780 2024-01-16T15:03:55.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.134+0000 7fde39e00640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 0x7fde0c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:55.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.135+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde1c054aa0 con 0x7fde34164780 2024-01-16T15:03:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.141+0000 7fde39e00640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 0x7fde0c06dba0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fde34068d90 tx=0x7fde2c006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:55.630 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:55.651+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:55.630 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:55.651+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:55.631 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:55.651+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:55.631 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:55.651+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:55.631 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:03:55.651+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:55.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.791+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fde3406e290 con 0x7fde34164780 2024-01-16T15:03:55.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.792+0000 7fde32000640 1 -- 172.21.15.43:0/3027810090 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2092 (secure 0 0 0) 0x7fde1c0580c0 con 0x7fde34164780 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.772 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 288, 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:03:55.773 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:03:55.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 800, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222208026, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 6721560576, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766372552704, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 426, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 10491, 2024-01-16T15:03:55.775 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 3 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:03:55.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:03:55.777 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:03:55.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:55.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:55.779 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:03:55.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 msgr2=0x7fde0c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 0x7fde0c06dba0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fde34068d90 tx=0x7fde2c006040 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 msgr2=0x7fde341fac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fde340cb7d0 tx=0x7fde1c00fc00 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 shutdown_connections 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde0c06b6e0 0x7fde0c06dba0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde34164780 0x7fde341fac40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde34154840 0x7fde34164240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 --2- 172.21.15.43:0/3027810090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde140047c0 0x7fde341f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 >> 172.21.15.43:0/3027810090 conn(0x7fde340b0d80 msgr2=0x7fde340b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 shutdown_connections 2024-01-16T15:03:55.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:55.798+0000 7fde3c800640 1 -- 172.21.15.43:0/3027810090 wait complete. 2024-01-16T15:03:57.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:03:58.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.070+0000 7f2b91400640 1 Processor -- start 2024-01-16T15:03:58.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.070+0000 7f2b91400640 1 -- start start 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.071+0000 7f2b91400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 0x7f2b8c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.071+0000 7f2b91400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.071+0000 7f2b91400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c15c360 con 0x7f2b8c0abff0 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.071+0000 7f2b91400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c0b41a0 con 0x7f2b8c0ad720 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.071+0000 7f2b91400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c06de40 con 0x7f2b8c06d520 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52770/0 (socket says 172.21.15.43:52770) 2024-01-16T15:03:58.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 -- 172.21.15.43:0/3542098438 learned_addr learned my addr 172.21.15.43:0/3542098438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b8a800640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 0x7f2b8c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 -- 172.21.15.43:0/3542098438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 msgr2=0x7f2b8c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 0x7f2b8c06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 -- 172.21.15.43:0/3542098438 >> v1:172.21.15.69:6789/0 conn(0x7f2b8c0abff0 legacy=0x7f2b8c15f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b83e00640 1 -- 172.21.15.43:0/3542098438 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c15ba30 con 0x7f2b8c0ad720 2024-01-16T15:03:58.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.072+0000 7f2b8a800640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 0x7f2b8c06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:58.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b83e00640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009870 tx=0x7f2b7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffd22a887ed1d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b89e00640 1 -- 172.21.15.43:0/3542098438 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74031640 con 0x7f2b8c0ad720 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b89e00640 1 -- 172.21.15.43:0/3542098438 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b740317a0 con 0x7f2b8c0ad720 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b89e00640 1 -- 172.21.15.43:0/3542098438 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74031b30 con 0x7f2b8c0ad720 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b91400640 1 -- 172.21.15.43:0/3542098438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c0ad720 msgr2=0x7f2b8c15ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.074+0000 7f2b91400640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009870 tx=0x7f2b7402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 -- 172.21.15.43:0/3542098438 shutdown_connections 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c0ad720 0x7f2b8c15ecd0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 --2- 172.21.15.43:0/3542098438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06d520 0x7f2b8c06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 -- 172.21.15.43:0/3542098438 >> 172.21.15.43:0/3542098438 conn(0x7f2b8c0b0d80 msgr2=0x7f2b8c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 -- 172.21.15.43:0/3542098438 shutdown_connections 2024-01-16T15:03:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.075+0000 7f2b91400640 1 -- 172.21.15.43:0/3542098438 wait complete. 2024-01-16T15:03:58.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.077+0000 7f2b91400640 1 Processor -- start 2024-01-16T15:03:58.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.077+0000 7f2b91400640 1 -- start start 2024-01-16T15:03:58.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c0abff0 0x7f2b8c15c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c0ad720 0x7f2b8c15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c072500 con 0x7f2b8c06d520 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c1f2f60 con 0x7f2b8c0abff0 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.078+0000 7f2b91400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0c9160 con 0x7f2b8c0ad720 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52780/0 (socket says 172.21.15.43:52780) 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 -- 172.21.15.43:0/4108220870 learned_addr learned my addr 172.21.15.43:0/4108220870 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 -- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c0ad720 msgr2=0x7f2b8c15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c0ad720 0x7f2b8c15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 -- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c0abff0 msgr2=0x7f2b8c15c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c0abff0 0x7f2b8c15c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 -- 172.21.15.43:0/4108220870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740094d0 con 0x7f2b8c06d520 2024-01-16T15:03:58.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.079+0000 7f2b8a800640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c7f0 tx=0x7f2b7c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.080+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c007a30 con 0x7f2b8c06d520 2024-01-16T15:03:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.080+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c007b90 con 0x7f2b8c06d520 2024-01-16T15:03:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.080+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00ce40 con 0x7f2b8c06d520 2024-01-16T15:03:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.080+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c0c9400 con 0x7f2b8c06d520 2024-01-16T15:03:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.080+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c071f40 con 0x7f2b8c06d520 2024-01-16T15:03:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.081+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c074770 con 0x7f2b8c06d520 2024-01-16T15:03:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.082+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 11) v1 ==== 89947+0+0 (secure 0 0 0) 0x7f2b7c0113f0 con 0x7f2b8c06d520 2024-01-16T15:03:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.083+0000 7f2b82a00640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 0x7f2b6406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.083+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b7c08a6c0 con 0x7f2b8c06d520 2024-01-16T15:03:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.083+0000 7f2b83e00640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 0x7f2b6406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.084+0000 7f2b83e00640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 0x7f2b6406dc00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2b74036000 tx=0x7f2b74030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:58.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.087+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b7c054d20 con 0x7f2b8c06d520 2024-01-16T15:03:58.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.419+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2b8c15deb0 con 0x7f2b8c06d520 2024-01-16T15:03:58.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.422+0000 7f2b82a00640 1 -- 172.21.15.43:0/4108220870 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2141 (secure 0 0 0) 0x7f2b7c058340 con 0x7f2b8c06d520 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:03:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:03:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 290, 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:03:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1096, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 4463817625, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 8353325056, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 764740788224, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 111107261, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 103439341, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 27, 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 31 2024-01-16T15:03:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:03:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:03:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:03:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 19s)", 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.22220000624656677, 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:03:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 msgr2=0x7f2b6406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 0x7f2b6406dc00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2b74036000 tx=0x7f2b74030000 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 msgr2=0x7f2b8c1f2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c7f0 tx=0x7f2b7c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 shutdown_connections 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b6406b740 0x7f2b6406dc00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c0abff0 0x7f2b8c15c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c0ad720 0x7f2b8c15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 --2- 172.21.15.43:0/4108220870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b8c06d520 0x7f2b8c1f2a20 secure :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c7f0 tx=0x7f2b7c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:03:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.434+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 >> 172.21.15.43:0/4108220870 conn(0x7f2b8c0b0d80 msgr2=0x7f2b8c0cb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:58.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.440+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 shutdown_connections 2024-01-16T15:03:58.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:03:58.441+0000 7f2b91400640 1 -- 172.21.15.43:0/4108220870 wait complete. 2024-01-16T15:04:00.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:00.573 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:00.594+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:00.574 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:00.594+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:00.574 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:00.594+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:00.574 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:00.594+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:00.574 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:00.594+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:00.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.941+0000 7f712d600640 1 Processor -- start 2024-01-16T15:04:00.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.943+0000 7f712d600640 1 -- start start 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.944+0000 7f712d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7128071fb0 0x7f71280adc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.944+0000 7f712d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7128159d00 0x7f712815a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.944+0000 7f712d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71280ae9e0 con 0x7f712815abe0 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.944+0000 7f712d600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7128153f70 con 0x7f7128159d00 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.944+0000 7f712d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712806c550 con 0x7f7128071fb0 2024-01-16T15:04:00.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.945+0000 7f7126000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f712815abe0 0x7f71280ae160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:47874/0 (socket says 172.21.15.43:47874) 2024-01-16T15:04:00.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.945+0000 7f7126000640 1 -- 172.21.15.43:0/89388028 learned_addr learned my addr 172.21.15.43:0/89388028 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:00.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.946+0000 7f7127400640 1 --2- 172.21.15.43:0/89388028 >> v2:172.21.15.43:3300/0 conn(0x7f7128159d00 0x7f712815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.946+0000 7f7127400640 1 -- 172.21.15.43:0/89388028 >> v2:172.21.15.69:3300/0 conn(0x7f7128071fb0 msgr2=0x7f71280adc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:04:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.948+0000 7f7125600640 1 -- 172.21.15.43:0/89388028 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448060136 0 0) 0x7f71280ae9e0 con 0x7f712815abe0 2024-01-16T15:04:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.949+0000 7f7126a00640 1 --2- 172.21.15.43:0/89388028 >> v2:172.21.15.69:3300/0 conn(0x7f7128071fb0 0x7f71280adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.949+0000 7f7127400640 1 --2- 172.21.15.43:0/89388028 >> v2:172.21.15.69:3300/0 conn(0x7f7128071fb0 0x7f71280adc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.949+0000 7f7127400640 1 -- 172.21.15.43:0/89388028 >> v1:172.21.15.69:6789/0 conn(0x7f712815abe0 legacy=0x7f71280ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.949+0000 7f7127400640 1 -- 172.21.15.43:0/89388028 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712806c830 con 0x7f7128159d00 2024-01-16T15:04:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.953+0000 7f7127400640 1 --2- 172.21.15.43:0/89388028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f712815a150 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7128070f60 tx=0x7f712002bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31345ebc05c0aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.955+0000 7f7125600640 1 -- 172.21.15.43:0/89388028 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712002ccb0 con 0x7f7128159d00 2024-01-16T15:04:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.956+0000 7f7125600640 1 -- 172.21.15.43:0/89388028 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712002ce10 con 0x7f7128159d00 2024-01-16T15:04:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.956+0000 7f7125600640 1 -- 172.21.15.43:0/89388028 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712002ccb0 con 0x7f7128159d00 2024-01-16T15:04:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 -- 172.21.15.43:0/89388028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 msgr2=0x7f712815a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 --2- 172.21.15.43:0/89388028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f712815a150 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7128070f60 tx=0x7f712002bf00 comp rx=0 tx=0).stop 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 -- 172.21.15.43:0/89388028 shutdown_connections 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 --2- 172.21.15.43:0/89388028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f712815a150 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 --2- 172.21.15.43:0/89388028 >> v2:172.21.15.69:3300/0 conn(0x7f7128071fb0 0x7f71280adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.958+0000 7f712d600640 1 -- 172.21.15.43:0/89388028 >> 172.21.15.43:0/89388028 conn(0x7f71280709f0 msgr2=0x7f71280718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.959+0000 7f712d600640 1 -- 172.21.15.43:0/89388028 shutdown_connections 2024-01-16T15:04:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.959+0000 7f712d600640 1 -- 172.21.15.43:0/89388028 wait complete. 2024-01-16T15:04:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.962+0000 7f712d600640 1 Processor -- start 2024-01-16T15:04:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.962+0000 7f712d600640 1 -- start start 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128071fb0 0x7f712806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f71281562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712809b210 con 0x7f7128159d00 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712809a970 con 0x7f7128071fb0 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.964+0000 7f712d600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712809af30 con 0x7f712815abe0 2024-01-16T15:04:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.965+0000 7f7127400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.965+0000 7f7127400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46552/0 (socket says 172.21.15.43:46552) 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.965+0000 7f7127400640 1 -- 172.21.15.43:0/434896355 learned_addr learned my addr 172.21.15.43:0/434896355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.965+0000 7f7126a00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128071fb0 0x7f712806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.967+0000 7f7127400640 1 -- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128071fb0 msgr2=0x7f712806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.967+0000 7f7127400640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128071fb0 0x7f712806d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.967+0000 7f7127400640 1 -- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 msgr2=0x7f71281562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.967+0000 7f7127400640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f71281562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.967+0000 7f7127400640 1 -- 172.21.15.43:0/434896355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120009a70 con 0x7f712815abe0 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.968+0000 7f7127400640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7120037040 tx=0x7f7120004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.969+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120004600 con 0x7f712815abe0 2024-01-16T15:04:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.969+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120004760 con 0x7f712815abe0 2024-01-16T15:04:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.969+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120040650 con 0x7f712815abe0 2024-01-16T15:04:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.969+0000 7f712d600640 1 -- 172.21.15.43:0/434896355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712809cf30 con 0x7f712815abe0 2024-01-16T15:04:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.969+0000 7f712d600640 1 -- 172.21.15.43:0/434896355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712814ba20 con 0x7f712815abe0 2024-01-16T15:04:00.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.972+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f8005250 con 0x7f712815abe0 2024-01-16T15:04:00.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.977+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 11) v1 ==== 89947+0+0 (secure 0 0 0) 0x7f71200407b0 con 0x7f712815abe0 2024-01-16T15:04:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.979+0000 7f711ea00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 0x7f710006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.979+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f71200ad270 con 0x7f712815abe0 2024-01-16T15:04:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.995+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7120077a20 con 0x7f712815abe0 2024-01-16T15:04:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:00.995+0000 7f7126a00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 0x7f710006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.020+0000 7f7126a00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 0x7f710006dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f7110008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:01.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.039+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f7120077df0 con 0x7f712815abe0 2024-01-16T15:04:01.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.627+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f70f80057d0 con 0x7f712815abe0 2024-01-16T15:04:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.629+0000 7f711ea00640 1 -- 172.21.15.43:0/434896355 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2142 (secure 0 0 0) 0x7f712007ed60 con 0x7f712815abe0 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:04:01.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:04:01.613 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 293, 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.614 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:04:01.616 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1545, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6347168822, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12178722816, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760915390464, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 270440714, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 260331931, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 65, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 70 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.617 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:04:01.618 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:04:01.619 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:04:01.620 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 19s)", 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.22220000624656677, 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:01.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.639+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 msgr2=0x7f710006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.639+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 0x7f710006dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f7110008040 comp rx=0 tx=0).stop 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.639+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 msgr2=0x7f71281567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.639+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7120037040 tx=0x7f7120004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 shutdown_connections 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f710006b810 0x7f710006dcd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f712815abe0 0x7f71281567f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7128159d00 0x7f71281562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 --2- 172.21.15.43:0/434896355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128071fb0 0x7f712806d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.640+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 >> 172.21.15.43:0/434896355 conn(0x7f71280709f0 msgr2=0x7f7128157f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.641+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 shutdown_connections 2024-01-16T15:04:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:01.641+0000 7f7117e00640 1 -- 172.21.15.43:0/434896355 wait complete. 2024-01-16T15:04:03.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:03.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.943+0000 7fd6bf400640 1 Processor -- start 2024-01-16T15:04:03.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.944+0000 7fd6bf400640 1 -- start start 2024-01-16T15:04:03.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bf400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd6b806bf60 0x7fd6b80cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bf400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b8160c90 con 0x7fd6b806b760 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bf400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b80b0ba0 con 0x7fd6b80b2040 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bf400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b80ce7a0 con 0x7fd6b806bf60 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bd600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd6b806b760 0x7fd6b806c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51166/0 (socket says 172.21.15.43:51166) 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.945+0000 7fd6bd600640 1 -- 172.21.15.43:0/959185565 learned_addr learned my addr 172.21.15.43:0/959185565 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:03.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6be000640 1 --2- 172.21.15.43:0/959185565 >> v2:172.21.15.43:3300/0 conn(0x7fd6b806bf60 0x7fd6b80cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617142316 0 0) 0x7fd6a8001110 con 0x7fd6b806b760 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd698003520 con 0x7fd6b806b760 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030636231 0 0) 0x7fd698003520 con 0x7fd6b806b760 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 >> v2:172.21.15.43:3300/0 conn(0x7fd6b806bf60 msgr2=0x7fd6b80cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 --2- 172.21.15.43:0/959185565 >> v2:172.21.15.43:3300/0 conn(0x7fd6b806bf60 0x7fd6b80cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 >> v1:172.21.15.69:6789/0 conn(0x7fd6b80b2040 legacy=0x7fd6b8166ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80d0350 con 0x7fd6b806b760 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd6a8001800 con 0x7fd6b806b760 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.946+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 >> v1:172.21.15.43:6789/0 conn(0x7fd6b806b760 legacy=0x7fd6b806c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6980039e0 0x7fd698003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd69800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80ce7a0 con 0x7fd6980052a0 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800c1c0 con 0x7fd6980039e0 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800c380 con 0x7fd6980047c0 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bea00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd69800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bea00640 1 -- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6980047c0 msgr2=0x7fd698004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bea00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bea00640 1 -- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6980039e0 msgr2=0x7fd698003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.947+0000 7fd6bea00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6980039e0 0x7fd698003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.948+0000 7fd6bea00640 1 -- 172.21.15.43:0/959185565 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698003520 con 0x7fd6980052a0 2024-01-16T15:04:03.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.949+0000 7fd6bea00640 1 -- 172.21.15.43:0/959185565 reap_dead start 2024-01-16T15:04:03.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.949+0000 7fd6bea00640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd69800b980 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0001d50 tx=0x7fd6980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd8d53a95ce481b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.949+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0034070 con 0x7fd6980052a0 2024-01-16T15:04:03.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.949+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0003990 con 0x7fd6980052a0 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.949+0000 7fd6bcc00640 1 -- 172.21.15.43:0/959185565 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0003b40 con 0x7fd6980052a0 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 -- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 msgr2=0x7fd69800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd69800b980 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0001d50 tx=0x7fd6980057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 -- 172.21.15.43:0/959185565 shutdown_connections 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 --2- 172.21.15.43:0/959185565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd69800b980 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 -- 172.21.15.43:0/959185565 >> 172.21.15.43:0/959185565 conn(0x7fd6b80d0e50 msgr2=0x7fd6b80d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 -- 172.21.15.43:0/959185565 shutdown_connections 2024-01-16T15:04:03.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.950+0000 7fd6bf400640 1 -- 172.21.15.43:0/959185565 wait complete. 2024-01-16T15:04:03.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.954+0000 7fd6bf400640 1 Processor -- start 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.955+0000 7fd6bf400640 1 -- start start 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.956+0000 7fd6bf400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd6b8146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.956+0000 7fd6bf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6b80cab80 0x7fd6b8147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.956+0000 7fd6bf400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 0x7fd6b81497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6bf400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b8149d20 con 0x7fd6980052a0 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6bf400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b8147840 con 0x7fd6b80cab80 2024-01-16T15:04:03.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6bf400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80cafc0 con 0x7fd6b8149330 2024-01-16T15:04:03.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6be000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd6b8146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6be000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd6b8146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39820/0 (socket says 172.21.15.43:39820) 2024-01-16T15:04:03.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.957+0000 7fd6be000640 1 -- 172.21.15.43:0/2820620722 learned_addr learned my addr 172.21.15.43:0/2820620722 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:03.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.958+0000 7fd6bea00640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 0x7fd6b81497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.958+0000 7fd6bd600640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6b80cab80 0x7fd6b8147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.959+0000 7fd6bea00640 1 -- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6b80cab80 msgr2=0x7fd6b8147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.959+0000 7fd6bea00640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6b80cab80 0x7fd6b8147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.959+0000 7fd6bea00640 1 -- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 msgr2=0x7fd6b8146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.959+0000 7fd6bea00640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd6b8146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:03.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.959+0000 7fd6bea00640 1 -- 172.21.15.43:0/2820620722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b815b1d0 con 0x7fd6b8149330 2024-01-16T15:04:03.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.960+0000 7fd6bea00640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 0x7fd6b81497e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80c4930 tx=0x7fd6b0001330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.961+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0034040 con 0x7fd6b8149330 2024-01-16T15:04:03.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.961+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0006b30 con 0x7fd6b8149330 2024-01-16T15:04:03.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.961+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0006ea0 con 0x7fd6b8149330 2024-01-16T15:04:03.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.961+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b80ae550 con 0x7fd6b8149330 2024-01-16T15:04:03.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.961+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b80aeb10 con 0x7fd6b8149330 2024-01-16T15:04:03.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.962+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b804e840 con 0x7fd6b8149330 2024-01-16T15:04:03.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.965+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd6b0005420 con 0x7fd6b8149330 2024-01-16T15:04:03.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.966+0000 7fd6b6000640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 0x7fd69006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:03.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.966+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6b00aae70 con 0x7fd6b8149330 2024-01-16T15:04:03.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.967+0000 7fd6be000640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 0x7fd69006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:03.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.968+0000 7fd6be000640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 0x7fd69006dce0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006a90 tx=0x7fd6a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:03.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:03.984+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6b00756f0 con 0x7fd6b8149330 2024-01-16T15:04:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.393+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd6b806bd40 con 0x7fd6b8149330 2024-01-16T15:04:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.395+0000 7fd6b6000640 1 -- 172.21.15.43:0/2820620722 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2144 (secure 0 0 0) 0x7fd6b003f020 con 0x7fd6b8149330 2024-01-16T15:04:04.374 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:04:04.375 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 296, 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:04:04.376 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.377 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:04:04.378 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:04:04.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:04.378 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1545, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6347168822, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12178722816, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760915390464, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 270373142, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 260261250, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 65, 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 68 2024-01-16T15:04:04.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:04:04.380 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.381 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (11s)\n [====........................] (remaining: 55s)", 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:04.382 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 msgr2=0x7fd69006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 0x7fd69006dce0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0006a90 tx=0x7fd6a0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 msgr2=0x7fd6b81497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 0x7fd6b81497e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80c4930 tx=0x7fd6b0001330 comp rx=0 tx=0).stop 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 shutdown_connections 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd69006b820 0x7fd69006dce0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6b80cab80 0x7fd6b8147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6b8149330 0x7fd6b81497e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 --2- 172.21.15.43:0/2820620722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6980052a0 0x7fd6b8146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 >> 172.21.15.43:0/2820620722 conn(0x7fd6b80d0e50 msgr2=0x7fd6b80d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 shutdown_connections 2024-01-16T15:04:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:04.399+0000 7fd6bf400640 1 -- 172.21.15.43:0/2820620722 wait complete. 2024-01-16T15:04:06.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.917+0000 7fb682000640 1 Processor -- start 2024-01-16T15:04:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.919+0000 7fb682000640 1 -- start start 2024-01-16T15:04:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.921+0000 7fb682000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 0x7fb6740e00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.921+0000 7fb682000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb674053070 0x7fb67404fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.922+0000 7fb67be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb674053070 0x7fb67404fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.922+0000 7fb67be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb674053070 0x7fb67404fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:56074/0 (socket says 172.21.15.43:56074) 2024-01-16T15:04:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.923+0000 7fb680c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 0x7fb6740e00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.923+0000 7fb680c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 0x7fb6740e00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:53810/0 (socket says 172.21.15.43:53810) 2024-01-16T15:04:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.925+0000 7fb681600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb67400aaa0 0x7fb6740e34d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51182/0 (socket says 172.21.15.43:51182) 2024-01-16T15:04:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.925+0000 7fb67be00640 1 -- 172.21.15.43:0/803929221 learned_addr learned my addr 172.21.15.43:0/803929221 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.925+0000 7fb682000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6740df4b0 con 0x7fb67400aaa0 2024-01-16T15:04:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.925+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67404f810 con 0x7fb674053070 2024-01-16T15:04:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.925+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6740e0610 con 0x7fb6740df0d0 2024-01-16T15:04:06.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.927+0000 7fb67be00640 1 -- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 msgr2=0x7fb6740e00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.927+0000 7fb67be00640 1 --2- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 0x7fb6740e00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.927+0000 7fb67be00640 1 -- 172.21.15.43:0/803929221 >> v1:172.21.15.43:6789/0 conn(0x7fb67400aaa0 legacy=0x7fb6740e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.927+0000 7fb67be00640 1 -- 172.21.15.43:0/803929221 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6740535c0 con 0x7fb674053070 2024-01-16T15:04:06.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.929+0000 7fb67be00640 1 --2- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb674053070 0x7fb67404fa40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009870 tx=0x7fb66c02dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1aea9c55d60391a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.931+0000 7fb67b400640 1 -- 172.21.15.43:0/803929221 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c02e960 con 0x7fb674053070 2024-01-16T15:04:06.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.931+0000 7fb67b400640 1 -- 172.21.15.43:0/803929221 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c02eac0 con 0x7fb674053070 2024-01-16T15:04:06.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.931+0000 7fb67b400640 1 -- 172.21.15.43:0/803929221 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c02ee90 con 0x7fb674053070 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb674053070 msgr2=0x7fb67404fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 --2- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb674053070 0x7fb67404fa40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009870 tx=0x7fb66c02dbb0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 shutdown_connections 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 --2- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb674053070 0x7fb67404fa40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 --2- 172.21.15.43:0/803929221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6740df0d0 0x7fb6740e00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.932+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 >> 172.21.15.43:0/803929221 conn(0x7fb674009f80 msgr2=0x7fb67400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.933+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 shutdown_connections 2024-01-16T15:04:06.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.933+0000 7fb682000640 1 -- 172.21.15.43:0/803929221 wait complete. 2024-01-16T15:04:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.941+0000 7fb682000640 1 Processor -- start 2024-01-16T15:04:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.943+0000 7fb682000640 1 -- start start 2024-01-16T15:04:06.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.948+0000 7fb682000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.948+0000 7fb682000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 0x7fb674006ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.953+0000 7fb682000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.952+0000 7fb680c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.954+0000 7fb67be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 0x7fb674006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.954+0000 7fb680c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:53826/0 (socket says 172.21.15.43:53826) 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb681600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb681600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:56082/0 (socket says 172.21.15.43:56082) 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb681600640 1 -- 172.21.15.43:0/3172332780 learned_addr learned my addr 172.21.15.43:0/3172332780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6740d3530 con 0x7fb674053070 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67404cbb0 con 0x7fb6740df0d0 2024-01-16T15:04:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.956+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb674048310 con 0x7fb67400aaa0 2024-01-16T15:04:06.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.958+0000 7fb681600640 1 -- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 msgr2=0x7fb67404b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.961+0000 7fb681600640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.961+0000 7fb681600640 1 -- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 msgr2=0x7fb674006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.961+0000 7fb681600640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 0x7fb674006ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.961+0000 7fb681600640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0094d0 con 0x7fb6740df0d0 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.961+0000 7fb67be00640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 0x7fb674006ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:06.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.962+0000 7fb680c00640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.966+0000 7fb681600640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c3c0 tx=0x7fb66400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.967+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6640074e0 con 0x7fb6740df0d0 2024-01-16T15:04:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.968+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb664007640 con 0x7fb6740df0d0 2024-01-16T15:04:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.968+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664004a20 con 0x7fb6740df0d0 2024-01-16T15:04:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.970+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6740d3f40 con 0x7fb6740df0d0 2024-01-16T15:04:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.973+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67404ce00 con 0x7fb6740df0d0 2024-01-16T15:04:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.978+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb674005290 con 0x7fb6740df0d0 2024-01-16T15:04:06.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.980+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fb664004b80 con 0x7fb6740df0d0 2024-01-16T15:04:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.982+0000 7fb678c00640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 0x7fb65406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.982+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb66408a180 con 0x7fb6740df0d0 2024-01-16T15:04:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.983+0000 7fb680c00640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 0x7fb65406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.987+0000 7fb680c00640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 0x7fb65406dce0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb67c064180 tx=0x7fb67c06e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:06.990+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb664054af0 con 0x7fb6740df0d0 2024-01-16T15:04:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.614+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb674037970 con 0x7fb6740df0d0 2024-01-16T15:04:07.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.615+0000 7fb678c00640 1 -- 172.21.15.43:0/3172332780 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2146 (secure 0 0 0) 0x7fb664058110 con 0x7fb6740df0d0 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:04:07.596 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 299, 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:04:07.597 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:04:07.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2192, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 9060980814, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 18034167808, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 755059945472, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 491287378, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 486077149, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 118, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 125 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:04:07.599 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.600 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:04:07.601 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 17s)", 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.41667500138282776, 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:07.602 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.621+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 msgr2=0x7fb65406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.621+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 0x7fb65406dce0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb67c064180 tx=0x7fb67c06e040 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 msgr2=0x7fb6740d2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c3c0 tx=0x7fb66400c880 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 shutdown_connections 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb65406b820 0x7fb65406dce0 secure :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb67c064180 tx=0x7fb67c06e040 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6740df0d0 0x7fb6740d2ff0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb674053070 0x7fb674006ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 --2- 172.21.15.43:0/3172332780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67400aaa0 0x7fb67404b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.622+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 >> 172.21.15.43:0/3172332780 conn(0x7fb674009f80 msgr2=0x7fb674036560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.623+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 shutdown_connections 2024-01-16T15:04:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:07.623+0000 7fb682000640 1 -- 172.21.15.43:0/3172332780 wait complete. 2024-01-16T15:04:09.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:10.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.151+0000 7efdcb800640 1 Processor -- start 2024-01-16T15:04:10.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.152+0000 7efdcb800640 1 -- start start 2024-01-16T15:04:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.153+0000 7efdcb800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efdc4142f20 0x7efdc406f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.153+0000 7efdcb800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc40cfdc0 con 0x7efdc406f070 2024-01-16T15:04:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.153+0000 7efdcb800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc415e9e0 con 0x7efdc40cf930 2024-01-16T15:04:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.153+0000 7efdcb800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc40cee50 con 0x7efdc4142f20 2024-01-16T15:04:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.154+0000 7efdc9800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efdc406f070 0x7efdc40b3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51186/0 (socket says 172.21.15.43:51186) 2024-01-16T15:04:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.154+0000 7efdc9800640 1 -- 172.21.15.43:0/548498476 learned_addr learned my addr 172.21.15.43:0/548498476 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.154+0000 7efdc8e00640 1 --2- 172.21.15.43:0/548498476 >> v2:172.21.15.69:3300/0 conn(0x7efdc4142f20 0x7efdc406f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.155+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761993040 0 0) 0x7efdc40cfdc0 con 0x7efdc406f070 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.155+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4007930 con 0x7efdc406f070 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.155+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330663207 0 0) 0x7efda4007930 con 0x7efdc406f070 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.155+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 >> v2:172.21.15.69:3300/0 conn(0x7efdc4142f20 msgr2=0x7efdc406f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.158+0000 7efdc3400640 1 --2- 172.21.15.43:0/548498476 >> v2:172.21.15.69:3300/0 conn(0x7efdc4142f20 0x7efdc406f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 >> v1:172.21.15.69:6789/0 conn(0x7efdc40cf930 legacy=0x7efdc406e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc40cf130 con 0x7efdc406f070 2024-01-16T15:04:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7efdbc0019f0 con 0x7efdc406f070 2024-01-16T15:04:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 >> v1:172.21.15.43:6789/0 conn(0x7efdc406f070 legacy=0x7efdc40b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efda4007d80 0x7efda4008160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efda4008bd0 0x7efda4009050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efda400fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40cee50 con 0x7efda4008bd0 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda40104c0 con 0x7efda4009590 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda4010680 con 0x7efda4007d80 2024-01-16T15:04:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.159+0000 7efdc8e00640 1 --2- 172.21.15.43:0/548498476 >> v2:172.21.15.69:3300/0 conn(0x7efdc4142f20 0x7efdc406f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.160+0000 7efdc9800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efda400fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.160+0000 7efdc8e00640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efda4007d80 0x7efda4008160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.160+0000 7efdc3e00640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efda4008bd0 0x7efda4009050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.162+0000 7efdc9800640 1 -- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efda4007d80 msgr2=0x7efda4008160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.162+0000 7efdc9800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efda4007d80 0x7efda4008160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.162+0000 7efdc9800640 1 -- 172.21.15.43:0/548498476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efda4008bd0 msgr2=0x7efda4009050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.162+0000 7efdc9800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efda4008bd0 0x7efda4009050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.162+0000 7efdc9800640 1 -- 172.21.15.43:0/548498476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4007930 con 0x7efda4009590 2024-01-16T15:04:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.164+0000 7efdc9800640 1 -- 172.21.15.43:0/548498476 reap_dead start 2024-01-16T15:04:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.164+0000 7efdc9800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efda400fd90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a700 tx=0x7efda4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90e059feaee99f37 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.165+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc034630 con 0x7efda4009590 2024-01-16T15:04:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.165+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc02f070 con 0x7efda4009590 2024-01-16T15:04:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.165+0000 7efdc3400640 1 -- 172.21.15.43:0/548498476 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc0349f0 con 0x7efda4009590 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.167+0000 7efdcb800640 1 -- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 msgr2=0x7efda400fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.167+0000 7efdcb800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efda400fd90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a700 tx=0x7efda4009c00 comp rx=0 tx=0).stop 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.168+0000 7efdcb800640 1 -- 172.21.15.43:0/548498476 shutdown_connections 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.168+0000 7efdcb800640 1 --2- 172.21.15.43:0/548498476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efda400fd90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.168+0000 7efdcb800640 1 -- 172.21.15.43:0/548498476 >> 172.21.15.43:0/548498476 conn(0x7efdc406d490 msgr2=0x7efdc406d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.168+0000 7efdcb800640 1 -- 172.21.15.43:0/548498476 shutdown_connections 2024-01-16T15:04:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.168+0000 7efdcb800640 1 -- 172.21.15.43:0/548498476 wait complete. 2024-01-16T15:04:10.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.175+0000 7efdcb800640 1 Processor -- start 2024-01-16T15:04:10.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.176+0000 7efdcb800640 1 -- start start 2024-01-16T15:04:10.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efdc41ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdc4157fc0 0x7efdc4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4158960 con 0x7efdc41482b0 2024-01-16T15:04:10.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc41f1d50 con 0x7efda4009590 2024-01-16T15:04:10.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdcb800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc41f1fa0 con 0x7efdc4157fc0 2024-01-16T15:04:10.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.179+0000 7efdc9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.180+0000 7efdc8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efdc41ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.180+0000 7efdc9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39846/0 (socket says 172.21.15.43:39846) 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.180+0000 7efdc9800640 1 -- 172.21.15.43:0/3988587778 learned_addr learned my addr 172.21.15.43:0/3988587778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.180+0000 7efdc3e00640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdc4157fc0 0x7efdc4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.181+0000 7efdc9800640 1 -- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdc4157fc0 msgr2=0x7efdc4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.181+0000 7efdc9800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdc4157fc0 0x7efdc4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.181+0000 7efdc9800640 1 -- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 msgr2=0x7efdc41ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.181+0000 7efdc9800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efdc41ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.181+0000 7efdc9800640 1 -- 172.21.15.43:0/3988587778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc41f2280 con 0x7efdc41482b0 2024-01-16T15:04:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.182+0000 7efdc9800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a820 tx=0x7efdbc001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.183+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc004ea0 con 0x7efdc41482b0 2024-01-16T15:04:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.183+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc41f2560 con 0x7efdc41482b0 2024-01-16T15:04:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.183+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc03f470 con 0x7efdc41482b0 2024-01-16T15:04:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.183+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4148c70 con 0x7efdc41482b0 2024-01-16T15:04:10.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.184+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc004ea0 con 0x7efdc41482b0 2024-01-16T15:04:10.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.186+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd94005250 con 0x7efdc41482b0 2024-01-16T15:04:10.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.190+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7efdbc031030 con 0x7efdc41482b0 2024-01-16T15:04:10.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.192+0000 7efdc0c00640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 0x7efd9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.193+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efdbc0ab840 con 0x7efdc41482b0 2024-01-16T15:04:10.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.202+0000 7efdc8e00640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 0x7efd9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.206+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdbc075f60 con 0x7efdc41482b0 2024-01-16T15:04:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.226+0000 7efdc8e00640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 0x7efd9c06dd20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efdac006fd0 tx=0x7efdac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.839+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efd940057d0 con 0x7efdc41482b0 2024-01-16T15:04:10.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.842+0000 7efdc0c00640 1 -- 172.21.15.43:0/3988587778 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2126 (secure 0 0 0) 0x7efdbc079580 con 0x7efdc41482b0 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:04:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 303, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:04:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2192, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 9060980814, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 18034167808, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 755059945472, 2024-01-16T15:04:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 380242319, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 382705235, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 91, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 97 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:04:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:04:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "5a22eb75-7689-4f4a-bd33-9546ebe9e0ec": { 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (16s)\n [==============..............] (remaining: 16s)", 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.5, 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:04:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:10.835 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:10.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.847+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 msgr2=0x7efd9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.847+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 0x7efd9c06dd20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efdac006fd0 tx=0x7efdac008040 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.847+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 msgr2=0x7efdc4148730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.847+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a820 tx=0x7efdbc001bf0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 shutdown_connections 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd9c06b860 0x7efd9c06dd20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdc41482b0 0x7efdc4148730 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdc4157fc0 0x7efdc4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 --2- 172.21.15.43:0/3988587778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4009590 0x7efdc41ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 >> 172.21.15.43:0/3988587778 conn(0x7efdc406d490 msgr2=0x7efdc41596e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.849+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 shutdown_connections 2024-01-16T15:04:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:10.850+0000 7efdcb800640 1 -- 172.21.15.43:0/3988587778 wait complete. 2024-01-16T15:04:12.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.213+0000 7fb806400640 1 Processor -- start 2024-01-16T15:04:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.213+0000 7fb806400640 1 -- start start 2024-01-16T15:04:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.216+0000 7fb806400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb800153f10 0x7fb80015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb806400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb80006ddb0 0x7fb80006e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb806400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8000ad6b0 con 0x7fb80015b050 2024-01-16T15:04:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb806400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80006e7a0 con 0x7fb800153f10 2024-01-16T15:04:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb806400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8000b2860 con 0x7fb80006ddb0 2024-01-16T15:04:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb7ff400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb800153f10 0x7fb80015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb7fea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb80015b050 0x7fb8000b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:38976/0 (socket says 172.21.15.43:38976) 2024-01-16T15:04:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb7fea00640 1 -- 172.21.15.43:0/939804941 learned_addr learned my addr 172.21.15.43:0/939804941 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.217+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/939804941 >> v2:172.21.15.69:3300/0 conn(0x7fb80006ddb0 0x7fb80006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.218+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995311256 0 0) 0x7fb8000ad6b0 con 0x7fb80015b050 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.218+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e0003520 con 0x7fb80015b050 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.219+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063974009 0 0) 0x7fb7e0003520 con 0x7fb80015b050 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.219+0000 7fb7ffe00640 1 -- 172.21.15.43:0/939804941 >> v2:172.21.15.43:3300/0 conn(0x7fb800153f10 msgr2=0x7fb80015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.219+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/939804941 >> v2:172.21.15.43:3300/0 conn(0x7fb800153f10 0x7fb80015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.219+0000 7fb7ffe00640 1 -- 172.21.15.43:0/939804941 >> v1:172.21.15.69:6789/0 conn(0x7fb80015b050 legacy=0x7fb8000b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.219+0000 7fb7ffe00640 1 -- 172.21.15.43:0/939804941 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8000b2b40 con 0x7fb80006ddb0 2024-01-16T15:04:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.221+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/939804941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb80006ddb0 0x7fb80006e260 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8000cec80 tx=0x7fb7f802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03d5a5556f89f2f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.223+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8001580 con 0x7fb80006ddb0 2024-01-16T15:04:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.223+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f802c8c0 con 0x7fb80006ddb0 2024-01-16T15:04:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.224+0000 7fb7fe000640 1 -- 172.21.15.43:0/939804941 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f802cc50 con 0x7fb80006ddb0 2024-01-16T15:04:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.224+0000 7fb806400640 1 -- 172.21.15.43:0/939804941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb80006ddb0 msgr2=0x7fb80006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.224+0000 7fb806400640 1 --2- 172.21.15.43:0/939804941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb80006ddb0 0x7fb80006e260 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8000cec80 tx=0x7fb7f802bd90 comp rx=0 tx=0).stop 2024-01-16T15:04:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 -- 172.21.15.43:0/939804941 shutdown_connections 2024-01-16T15:04:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 --2- 172.21.15.43:0/939804941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb80006ddb0 0x7fb80006e260 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 --2- 172.21.15.43:0/939804941 >> v2:172.21.15.43:3300/0 conn(0x7fb800153f10 0x7fb80015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 -- 172.21.15.43:0/939804941 >> 172.21.15.43:0/939804941 conn(0x7fb8000b0d80 msgr2=0x7fb8000b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 -- 172.21.15.43:0/939804941 shutdown_connections 2024-01-16T15:04:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.225+0000 7fb806400640 1 -- 172.21.15.43:0/939804941 wait complete. 2024-01-16T15:04:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.227+0000 7fb806400640 1 Processor -- start 2024-01-16T15:04:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.228+0000 7fb806400640 1 -- start start 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb80006ddb0 0x7fb80013f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb800153f10 0x7fb800150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800152670 con 0x7fb80015b050 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80013fc00 con 0x7fb800153f10 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb806400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8001515d0 con 0x7fb80006ddb0 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb7ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb7ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45904/0 (socket says 172.21.15.43:45904) 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.229+0000 7fb7ffe00640 1 -- 172.21.15.43:0/2052032174 learned_addr learned my addr 172.21.15.43:0/2052032174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 -- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb80006ddb0 msgr2=0x7fb80013f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb80006ddb0 0x7fb80013f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 -- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb800153f10 msgr2=0x7fb800150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb800153f10 0x7fb800150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 -- 172.21.15.43:0/2052032174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8009a70 con 0x7fb80015b050 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7ffe00640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e80 tx=0x7fb7f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8034e50 con 0x7fb80015b050 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8009450 con 0x7fb80015b050 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.230+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f803e650 con 0x7fb80015b050 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.231+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8001518b0 con 0x7fb80015b050 2024-01-16T15:04:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.231+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb800151e70 con 0x7fb80015b050 2024-01-16T15:04:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.232+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb800074770 con 0x7fb80015b050 2024-01-16T15:04:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.233+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fb7f803b050 con 0x7fb80015b050 2024-01-16T15:04:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.234+0000 7fb7f7400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 0x7fb7d806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.234+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7f80acdf0 con 0x7fb80015b050 2024-01-16T15:04:13.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.245+0000 7fb7ff400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 0x7fb7d806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:13.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.246+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7f8077590 con 0x7fb80015b050 2024-01-16T15:04:13.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.246+0000 7fb7ff400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 0x7fb7d806dc90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b240 tx=0x7fb7f0006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.739+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb80015ccc0 con 0x7fb80015b050 2024-01-16T15:04:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.741+0000 7fb7f7400640 1 -- 172.21.15.43:0/2052032174 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1869 (secure 0 0 0) 0x7fb7f804e090 con 0x7fb80015b050 2024-01-16T15:04:13.720 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:13.720 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:04:13.721 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 306, 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:04:13.722 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:04:13.723 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:13.724 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2993, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 12420735573, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 24623783936, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 748470329344, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 666574027, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 662413006, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 159, 2024-01-16T15:04:13.725 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 167 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 27, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 3, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:13.726 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 3, 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5766 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:04:13.727 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:04:13.728 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-01-16T15:04:13.729 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:03:36.875584+0000", 2024-01-16T15:04:13.729 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:04:13.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:13.729 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": {} 2024-01-16T15:04:13.729 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.746+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 msgr2=0x7fb7d806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.746+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 0x7fb7d806dc90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b240 tx=0x7fb7f0006a20 comp rx=0 tx=0).stop 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.746+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 msgr2=0x7fb8000cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.746+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e80 tx=0x7fb7f8004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 shutdown_connections 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7d806b7d0 0x7fb7d806dc90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb800153f10 0x7fb800150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb80015b050 0x7fb8000cc220 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 --2- 172.21.15.43:0/2052032174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb80006ddb0 0x7fb80013f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 >> 172.21.15.43:0/2052032174 conn(0x7fb8000b0d80 msgr2=0x7fb80015dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 shutdown_connections 2024-01-16T15:04:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:13.747+0000 7fb806400640 1 -- 172.21.15.43:0/2052032174 wait complete. 2024-01-16T15:04:13.769 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:13.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24921 2024-01-16T15:04:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.120+0000 7fad41e00640 1 Processor -- start 2024-01-16T15:04:14.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.120+0000 7fad41e00640 1 -- start start 2024-01-16T15:04:14.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.121+0000 7fad41e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0cfa50 0x7fad3c162a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.121+0000 7fad41e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fad3c0c6140 0x7fad3c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.121+0000 7fad41e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c162fd0 con 0x7fad3c0d1cc0 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.121+0000 7fad41e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3c0d3f80 con 0x7fad3c0c6140 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.121+0000 7fad41e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0d3770 con 0x7fad3c0cfa50 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fad3c0c6140 0x7fad3c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fad3c0c6140 0x7fad3c0cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45920/0 (socket says 172.21.15.43:45920) 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 -- 172.21.15.43:0/2738398779 learned_addr learned my addr 172.21.15.43:0/2738398779 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad40a00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0cfa50 0x7fad3c162a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 -- 172.21.15.43:0/2738398779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0cfa50 msgr2=0x7fad3c162a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0cfa50 0x7fad3c162a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 -- 172.21.15.43:0/2738398779 >> v1:172.21.15.69:6789/0 conn(0x7fad3c0d1cc0 legacy=0x7fad3c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.122+0000 7fad3be00640 1 -- 172.21.15.43:0/2738398779 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c0d1830 con 0x7fad3c0c6140 2024-01-16T15:04:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.123+0000 7fad3be00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0cfe30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009870 tx=0x7fad2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b19f2dc7ce0fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.124+0000 7fad3b400640 1 -- 172.21.15.43:0/2738398779 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c030840 con 0x7fad3c0c6140 2024-01-16T15:04:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.124+0000 7fad3b400640 1 -- 172.21.15.43:0/2738398779 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c0309a0 con 0x7fad3c0c6140 2024-01-16T15:04:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.124+0000 7fad3b400640 1 -- 172.21.15.43:0/2738398779 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c030d30 con 0x7fad3c0c6140 2024-01-16T15:04:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 -- 172.21.15.43:0/2738398779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 msgr2=0x7fad3c0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0cfe30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009870 tx=0x7fad2c02b980 comp rx=0 tx=0).stop 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 -- 172.21.15.43:0/2738398779 shutdown_connections 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0cfe30 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 --2- 172.21.15.43:0/2738398779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0cfa50 0x7fad3c162a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 -- 172.21.15.43:0/2738398779 >> 172.21.15.43:0/2738398779 conn(0x7fad3c09c390 msgr2=0x7fad3c0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 -- 172.21.15.43:0/2738398779 shutdown_connections 2024-01-16T15:04:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.125+0000 7fad41e00640 1 -- 172.21.15.43:0/2738398779 wait complete. 2024-01-16T15:04:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.127+0000 7fad41e00640 1 Processor -- start 2024-01-16T15:04:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.127+0000 7fad41e00640 1 -- start start 2024-01-16T15:04:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0d1cc0 0x7fad3c0c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0c7bb0 con 0x7fad3c0c6140 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0c6fa0 con 0x7fad3c0cfa50 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0c7280 con 0x7fad3c0d1cc0 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad40a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad3be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad3be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:35920/0 (socket says 172.21.15.43:35920) 2024-01-16T15:04:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad3be00640 1 -- 172.21.15.43:0/940521867 learned_addr learned my addr 172.21.15.43:0/940521867 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.129+0000 7fad41400640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0d1cc0 0x7fad3c0c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad3be00640 1 -- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0d1cc0 msgr2=0x7fad3c0c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad3be00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0d1cc0 0x7fad3c0c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad3be00640 1 -- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 msgr2=0x7fad3c0b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad3be00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad3be00640 1 -- 172.21.15.43:0/940521867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0094d0 con 0x7fad3c0cfa50 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.130+0000 7fad40a00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.131+0000 7fad3be00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fad2c000c00 tx=0x7fad2c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.132+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c039c90 con 0x7fad3c0cfa50 2024-01-16T15:04:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.132+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c039df0 con 0x7fad3c0cfa50 2024-01-16T15:04:14.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.132+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c043640 con 0x7fad3c0cfa50 2024-01-16T15:04:14.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.132+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c0c7560 con 0x7fad3c0cfa50 2024-01-16T15:04:14.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.132+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c0b7ad0 con 0x7fad3c0cfa50 2024-01-16T15:04:14.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.135+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fad2c0437a0 con 0x7fad3c0cfa50 2024-01-16T15:04:14.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.136+0000 7fad38c00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 0x7fad1406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.136+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c161e70 con 0x7fad3c0cfa50 2024-01-16T15:04:14.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.136+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad2c03e070 con 0x7fad3c0cfa50 2024-01-16T15:04:14.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.142+0000 7fad40a00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 0x7fad1406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.143+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad3c161e70 con 0x7fad3c0cfa50 2024-01-16T15:04:14.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.148+0000 7fad40a00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 0x7fad1406dd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad24006fd0 tx=0x7fad24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.493+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24921", "target": ["mon-mgr", ""]}) v1 -- 0x7fad3c0d39c0 con 0x7fad1406b850 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.540+0000 7fad38c00640 1 -- 172.21.15.43:0/940521867 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fad24007ae0 con 0x7fad1406b850 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:14.520 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.543+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 msgr2=0x7fad1406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.543+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 0x7fad1406dd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad24006fd0 tx=0x7fad24008040 comp rx=0 tx=0).stop 2024-01-16T15:04:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.543+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 msgr2=0x7fad3c1610b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.543+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fad2c000c00 tx=0x7fad2c004730 comp rx=0 tx=0).stop 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 shutdown_connections 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad1406b850 0x7fad1406dd10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad3c0cfa50 0x7fad3c1610b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad3c0d1cc0 0x7fad3c0c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 --2- 172.21.15.43:0/940521867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad3c0c6140 0x7fad3c0b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 >> 172.21.15.43:0/940521867 conn(0x7fad3c09c390 msgr2=0x7fad3c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 shutdown_connections 2024-01-16T15:04:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.544+0000 7fad41e00640 1 -- 172.21.15.43:0/940521867 wait complete. 2024-01-16T15:04:14.541 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:14.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24922 2024-01-16T15:04:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.768+0000 7f0e1ea00640 1 Processor -- start 2024-01-16T15:04:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.768+0000 7f0e1ea00640 1 -- start start 2024-01-16T15:04:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.770+0000 7f0e1ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.770+0000 7f0e1ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e1806ab50 0x7f0e180af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.770+0000 7f0e1ea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e180cfe50 con 0x7f0e180cc920 2024-01-16T15:04:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.770+0000 7f0e1ea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180723c0 con 0x7f0e1806ab50 2024-01-16T15:04:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.770+0000 7f0e1ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180ccd60 con 0x7f0e180cd440 2024-01-16T15:04:14.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.771+0000 7f0e17e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.771+0000 7f0e17e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34268/0 (socket says 172.21.15.43:34268) 2024-01-16T15:04:14.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.771+0000 7f0e17e00640 1 -- 172.21.15.43:0/4190479054 learned_addr learned my addr 172.21.15.43:0/4190479054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:14.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.771+0000 7f0e1ca00640 1 --2- 172.21.15.43:0/4190479054 >> v2:172.21.15.69:3300/0 conn(0x7f0e1806ab50 0x7f0e180af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.772+0000 7f0e1ca00640 1 -- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 msgr2=0x7f0e18071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.772+0000 7f0e1ca00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.772+0000 7f0e1ca00640 1 -- 172.21.15.43:0/4190479054 >> v1:172.21.15.69:6789/0 conn(0x7f0e180cc920 legacy=0x7f0e1806a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.772+0000 7f0e1ca00640 1 -- 172.21.15.43:0/4190479054 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180cf260 con 0x7f0e1806ab50 2024-01-16T15:04:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.772+0000 7f0e17e00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.774+0000 7f0e1ca00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1806ab50 0x7f0e180af1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009d90 tx=0x7f0e1002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7809e9bfc57f671 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.774+0000 7f0e16a00640 1 -- 172.21.15.43:0/4190479054 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1002cc60 con 0x7f0e1806ab50 2024-01-16T15:04:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.774+0000 7f0e16a00640 1 -- 172.21.15.43:0/4190479054 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1002cdc0 con 0x7f0e1806ab50 2024-01-16T15:04:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.774+0000 7f0e16a00640 1 -- 172.21.15.43:0/4190479054 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10034660 con 0x7f0e1806ab50 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 -- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1806ab50 msgr2=0x7f0e180af1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1806ab50 0x7f0e180af1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009d90 tx=0x7f0e1002beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 -- 172.21.15.43:0/4190479054 shutdown_connections 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1806ab50 0x7f0e180af1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/4190479054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cd440 0x7f0e18071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.775+0000 7f0e1ea00640 1 -- 172.21.15.43:0/4190479054 >> 172.21.15.43:0/4190479054 conn(0x7f0e180b1c90 msgr2=0x7f0e180b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.776+0000 7f0e1ea00640 1 -- 172.21.15.43:0/4190479054 shutdown_connections 2024-01-16T15:04:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.776+0000 7f0e1ea00640 1 -- 172.21.15.43:0/4190479054 wait complete. 2024-01-16T15:04:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.778+0000 7f0e1ea00640 1 Processor -- start 2024-01-16T15:04:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.778+0000 7f0e1ea00640 1 -- start start 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cc920 0x7f0e180b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e180cd440 0x7f0e1815be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180b5210 con 0x7f0e1806ab50 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815b0c0 con 0x7f0e180cd440 2024-01-16T15:04:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.780+0000 7f0e1ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815b3a0 con 0x7f0e180cc920 2024-01-16T15:04:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.781+0000 7f0e17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.781+0000 7f0e17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45940/0 (socket says 172.21.15.43:45940) 2024-01-16T15:04:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.781+0000 7f0e17e00640 1 -- 172.21.15.43:0/1035929107 learned_addr learned my addr 172.21.15.43:0/1035929107 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.781+0000 7f0e17400640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cc920 0x7f0e180b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.781+0000 7f0e1ca00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e180cd440 0x7f0e1815be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 -- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cc920 msgr2=0x7f0e180b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cc920 0x7f0e180b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 -- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e180cd440 msgr2=0x7f0e1815be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e180cd440 0x7f0e1815be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 -- 172.21.15.43:0/1035929107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080094d0 con 0x7f0e1806ab50 2024-01-16T15:04:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.782+0000 7f0e17e00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800e9d0 tx=0x7f0e0800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.786+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0800ca60 con 0x7f0e1806ab50 2024-01-16T15:04:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.786+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0800cbc0 con 0x7f0e1806ab50 2024-01-16T15:04:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.786+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08013470 con 0x7f0e1806ab50 2024-01-16T15:04:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.787+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e10009a70 con 0x7f0e1806ab50 2024-01-16T15:04:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.788+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1815b960 con 0x7f0e1806ab50 2024-01-16T15:04:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.789+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1809e2f0 con 0x7f0e1806ab50 2024-01-16T15:04:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.792+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f0e080135d0 con 0x7f0e1806ab50 2024-01-16T15:04:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.794+0000 7f0e0fe00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 0x7f0df006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.795+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e0808a6c0 con 0x7f0e1806ab50 2024-01-16T15:04:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.796+0000 7f0e17400640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 0x7f0df006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.799+0000 7f0e17400640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 0x7f0df006dd60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0e180b5d80 tx=0x7f0e00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:14.804+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e08054f40 con 0x7f0e1806ab50 2024-01-16T15:04:15.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.252+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24922", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e180cbe80 con 0x7f0df006b8a0 2024-01-16T15:04:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.290+0000 7f0e0fe00640 1 -- 172.21.15.43:0/1035929107 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f0e180cbe80 con 0x7f0df006b8a0 2024-01-16T15:04:15.269 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:15.269 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:04:15.269 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:04:15.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:15.270 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 msgr2=0x7f0df006dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 0x7f0df006dd60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0e180b5d80 tx=0x7f0e00009040 comp rx=0 tx=0).stop 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 msgr2=0x7f0e181514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800e9d0 tx=0x7f0e0800ee90 comp rx=0 tx=0).stop 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 shutdown_connections 2024-01-16T15:04:15.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0df006b8a0 0x7f0df006dd60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e180cd440 0x7f0e1815be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e180cc920 0x7f0e180b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 --2- 172.21.15.43:0/1035929107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1806ab50 0x7f0e181514c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.293+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 >> 172.21.15.43:0/1035929107 conn(0x7f0e180b1c90 msgr2=0x7f0e1806a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.294+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 shutdown_connections 2024-01-16T15:04:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.294+0000 7f0e1ea00640 1 -- 172.21.15.43:0/1035929107 wait complete. 2024-01-16T15:04:15.291 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:15.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24923 2024-01-16T15:04:15.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.676+0000 7f76aca00640 1 Processor -- start 2024-01-16T15:04:15.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.676+0000 7f76aca00640 1 -- start start 2024-01-16T15:04:15.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.678+0000 7f76aca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f76a815d3b0 0x7f76a80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.678+0000 7f76aca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.678+0000 7f76aca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80d2010 con 0x7f76a80d1bc0 2024-01-16T15:04:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.678+0000 7f76aca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a80c6bf0 con 0x7f76a815d3b0 2024-01-16T15:04:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.678+0000 7f76aca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80d1400 con 0x7f76a80d0fa0 2024-01-16T15:04:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.679+0000 7f769fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.679+0000 7f769fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34278/0 (socket says 172.21.15.43:34278) 2024-01-16T15:04:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.679+0000 7f769fe00640 1 -- 172.21.15.43:0/242244202 learned_addr learned my addr 172.21.15.43:0/242244202 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.681+0000 7f769fe00640 1 -- 172.21.15.43:0/242244202 >> v2:172.21.15.69:3300/0 conn(0x7f76a815d3b0 msgr2=0x7f76a80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.681+0000 7f769fe00640 1 --2- 172.21.15.43:0/242244202 >> v2:172.21.15.69:3300/0 conn(0x7f76a815d3b0 0x7f76a80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.681+0000 7f769fe00640 1 -- 172.21.15.43:0/242244202 >> v1:172.21.15.69:6789/0 conn(0x7f76a80d1bc0 legacy=0x7f76a80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:15.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.681+0000 7f769fe00640 1 -- 172.21.15.43:0/242244202 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a80cedc0 con 0x7f76a80d0fa0 2024-01-16T15:04:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.683+0000 7f769fe00640 1 --2- 172.21.15.43:0/242244202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7690009870 tx=0x7f769002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecae72052ec9495b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.684+0000 7f76a5e00640 1 -- 172.21.15.43:0/242244202 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76900317c0 con 0x7f76a80d0fa0 2024-01-16T15:04:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.684+0000 7f76a5e00640 1 -- 172.21.15.43:0/242244202 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7690031920 con 0x7f76a80d0fa0 2024-01-16T15:04:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.684+0000 7f76a5e00640 1 -- 172.21.15.43:0/242244202 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690031cb0 con 0x7f76a80d0fa0 2024-01-16T15:04:15.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 -- 172.21.15.43:0/242244202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 msgr2=0x7f76a80d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 --2- 172.21.15.43:0/242244202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7690009870 tx=0x7f769002b860 comp rx=0 tx=0).stop 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 -- 172.21.15.43:0/242244202 shutdown_connections 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 --2- 172.21.15.43:0/242244202 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a80d1680 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 --2- 172.21.15.43:0/242244202 >> v2:172.21.15.69:3300/0 conn(0x7f76a815d3b0 0x7f76a80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 -- 172.21.15.43:0/242244202 >> 172.21.15.43:0/242244202 conn(0x7f76a80d30a0 msgr2=0x7f76a80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 -- 172.21.15.43:0/242244202 shutdown_connections 2024-01-16T15:04:15.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.685+0000 7f76aca00640 1 -- 172.21.15.43:0/242244202 wait complete. 2024-01-16T15:04:15.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.687+0000 7f76aca00640 1 Processor -- start 2024-01-16T15:04:15.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.687+0000 7f76aca00640 1 -- start start 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a806b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a815d3b0 0x7f76a814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a815ad40 con 0x7f76a80d1bc0 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80b6030 con 0x7f76a815d3b0 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f76aca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a809d130 con 0x7f76a80d0fa0 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f769fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f769fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45946/0 (socket says 172.21.15.43:45946) 2024-01-16T15:04:15.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.688+0000 7f769fe00640 1 -- 172.21.15.43:0/1320314387 learned_addr learned my addr 172.21.15.43:0/1320314387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:15.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f76a6800640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a806b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f769fe00640 1 -- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 msgr2=0x7f76a806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f769fe00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a806b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f769fe00640 1 -- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a815d3b0 msgr2=0x7f76a814e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:15.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f769fe00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a815d3b0 0x7f76a814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:15.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.689+0000 7f769fe00640 1 -- 172.21.15.43:0/1320314387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76900094d0 con 0x7f76a80d1bc0 2024-01-16T15:04:15.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.690+0000 7f769fe00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f769002b8a0 tx=0x7f76900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.691+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690032860 con 0x7f76a80d1bc0 2024-01-16T15:04:15.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.691+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76900329c0 con 0x7f76a80d1bc0 2024-01-16T15:04:15.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.691+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690032d50 con 0x7f76a80d1bc0 2024-01-16T15:04:15.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.691+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a809d410 con 0x7f76a80d1bc0 2024-01-16T15:04:15.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.692+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a80b5cc0 con 0x7f76a80d1bc0 2024-01-16T15:04:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.697+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f769004e450 con 0x7f76a80d1bc0 2024-01-16T15:04:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.698+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a804e840 con 0x7f76a80d1bc0 2024-01-16T15:04:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.699+0000 7f769ea00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 0x7f768006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.700+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76900ac550 con 0x7f76a80d1bc0 2024-01-16T15:04:15.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.704+0000 7f76a6800640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 0x7f768006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.705+0000 7f76a6800640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 0x7f768006dd60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7698009890 tx=0x7f7698008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:15.710+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7690076e20 con 0x7f76a80d1bc0 2024-01-16T15:04:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.146+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24923", "target": ["mon-mgr", ""]}) v1 -- 0x7f76a80b4e60 con 0x7f768006b8a0 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.636+0000 7f769ea00640 1 -- 172.21.15.43:0/1320314387 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f76a80b4e60 con 0x7f768006b8a0 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:16.616 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.639+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 msgr2=0x7f768006dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.639+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 0x7f768006dd60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7698009890 tx=0x7f7698008040 comp rx=0 tx=0).stop 2024-01-16T15:04:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.639+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 msgr2=0x7f76a80cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.639+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f769002b8a0 tx=0x7f76900046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 shutdown_connections 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768006b8a0 0x7f768006dd60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76a815d3b0 0x7f76a814e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76a80d1bc0 0x7f76a80cc5b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 --2- 172.21.15.43:0/1320314387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76a80d0fa0 0x7f76a806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 >> 172.21.15.43:0/1320314387 conn(0x7f76a80d30a0 msgr2=0x7f76a8160280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 shutdown_connections 2024-01-16T15:04:16.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.640+0000 7f76aca00640 1 -- 172.21.15.43:0/1320314387 wait complete. 2024-01-16T15:04:16.635 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:16.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone24924 2024-01-16T15:04:16.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.845+0000 7f06a2c00640 1 Processor -- start 2024-01-16T15:04:16.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.845+0000 7f06a2c00640 1 -- start start 2024-01-16T15:04:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.847+0000 7f06a2c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f069c070e10 0x7f069c0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.847+0000 7f06a2c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.847+0000 7f06a2c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0cd400 con 0x7f069c0cf920 2024-01-16T15:04:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.847+0000 7f06a2c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c0cfdd0 con 0x7f069c070e10 2024-01-16T15:04:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.847+0000 7f06a2c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0b1f10 con 0x7f069c0ce720 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.848+0000 7f069b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.848+0000 7f069b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34310/0 (socket says 172.21.15.43:34310) 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.848+0000 7f069b400640 1 -- 172.21.15.43:0/2132011620 learned_addr learned my addr 172.21.15.43:0/2132011620 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.848+0000 7f069be00640 1 --2- 172.21.15.43:0/2132011620 >> v2:172.21.15.69:3300/0 conn(0x7f069c070e10 0x7f069c0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.849+0000 7f069b400640 1 -- 172.21.15.43:0/2132011620 >> v2:172.21.15.69:3300/0 conn(0x7f069c070e10 msgr2=0x7f069c0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.849+0000 7f069b400640 1 --2- 172.21.15.43:0/2132011620 >> v2:172.21.15.69:3300/0 conn(0x7f069c070e10 0x7f069c0ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.849+0000 7f069b400640 1 -- 172.21.15.43:0/2132011620 >> v1:172.21.15.69:6789/0 conn(0x7f069c0cf920 legacy=0x7f069c06c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.849+0000 7f069b400640 1 -- 172.21.15.43:0/2132011620 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0711f0 con 0x7f069c0ce720 2024-01-16T15:04:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.849+0000 7f069aa00640 1 -- 172.21.15.43:0/2132011620 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652417823 0 0) 0x7f0694001490 con 0x7f069c0cf920 2024-01-16T15:04:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.850+0000 7f069b400640 1 --2- 172.21.15.43:0/2132011620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0684009870 tx=0x7f068402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eec3cd84b6c7fcd9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.850+0000 7f069aa00640 1 -- 172.21.15.43:0/2132011620 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684030840 con 0x7f069c0ce720 2024-01-16T15:04:16.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.850+0000 7f069aa00640 1 -- 172.21.15.43:0/2132011620 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06840309a0 con 0x7f069c0ce720 2024-01-16T15:04:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.850+0000 7f069aa00640 1 -- 172.21.15.43:0/2132011620 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684030d30 con 0x7f069c0ce720 2024-01-16T15:04:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 -- 172.21.15.43:0/2132011620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 msgr2=0x7f069c0cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 --2- 172.21.15.43:0/2132011620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0684009870 tx=0x7f068402b980 comp rx=0 tx=0).stop 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 -- 172.21.15.43:0/2132011620 shutdown_connections 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 --2- 172.21.15.43:0/2132011620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c0cf3e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 --2- 172.21.15.43:0/2132011620 >> v2:172.21.15.69:3300/0 conn(0x7f069c070e10 0x7f069c0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 -- 172.21.15.43:0/2132011620 >> 172.21.15.43:0/2132011620 conn(0x7f069c06a860 msgr2=0x7f069c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 -- 172.21.15.43:0/2132011620 shutdown_connections 2024-01-16T15:04:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.851+0000 7f06a2c00640 1 -- 172.21.15.43:0/2132011620 wait complete. 2024-01-16T15:04:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.853+0000 7f06a2c00640 1 Processor -- start 2024-01-16T15:04:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.853+0000 7f06a2c00640 1 -- start start 2024-01-16T15:04:16.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 0x7f069c099f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c9d70 con 0x7f069c070e10 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c93d0 con 0x7f069c0cf920 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a2c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c96b0 con 0x7f069c0ce720 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45956/0 (socket says 172.21.15.43:45956) 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 -- 172.21.15.43:0/986343931 learned_addr learned my addr 172.21.15.43:0/986343931 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069b400640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 -- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 msgr2=0x7f069c099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f069be00640 1 -- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 msgr2=0x7f069c099f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.854+0000 7f06a0c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 0x7f069c099f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f069be00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 0x7f069c099f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f069be00640 1 -- 172.21.15.43:0/986343931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840094d0 con 0x7f069c070e10 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f06a0c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 0x7f069c099f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f069be00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f068c00e9b0 tx=0x7f068c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c009670 con 0x7f069c070e10 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068c00c9a0 con 0x7f069c070e10 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c00cd50 con 0x7f069c070e10 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c0c9990 con 0x7f069c070e10 2024-01-16T15:04:16.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.855+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c15a890 con 0x7f069c070e10 2024-01-16T15:04:16.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.856+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c067d60 con 0x7f069c070e10 2024-01-16T15:04:16.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.857+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f068c013470 con 0x7f069c070e10 2024-01-16T15:04:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.858+0000 7f0693e00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 0x7f067406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.858+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f068c08a5e0 con 0x7f069c070e10 2024-01-16T15:04:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.859+0000 7f069b400640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 0x7f067406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.859+0000 7f069b400640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 0x7f067406dc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0684036000 tx=0x7f068403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:16.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:16.863+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f068c054e10 con 0x7f069c070e10 2024-01-16T15:04:17.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.090+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone24924", "target": ["mon-mgr", ""]}) v1 -- 0x7f069c0c9540 con 0x7f067406b7d0 2024-01-16T15:04:17.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.770+0000 7f0693e00640 1 -- 172.21.15.43:0/986343931 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f069c0c9540 con 0x7f067406b7d0 2024-01-16T15:04:17.750 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:17.750 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:04:17.750 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:04:17.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:17.750 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:17.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 msgr2=0x7f067406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 0x7f067406dc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0684036000 tx=0x7f068403b040 comp rx=0 tx=0).stop 2024-01-16T15:04:17.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 msgr2=0x7f069c15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f068c00e9b0 tx=0x7f068c00ee70 comp rx=0 tx=0).stop 2024-01-16T15:04:17.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 shutdown_connections 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f067406b7d0 0x7f067406dc90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f069c0cf920 0x7f069c099f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f069c0ce720 0x7f069c099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 --2- 172.21.15.43:0/986343931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f069c070e10 0x7f069c15b0e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 >> 172.21.15.43:0/986343931 conn(0x7f069c06a860 msgr2=0x7f069c0b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 shutdown_connections 2024-01-16T15:04:17.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:17.774+0000 7f06a2c00640 1 -- 172.21.15.43:0/986343931 wait complete. 2024-01-16T15:04:17.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:04:17.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.012+0000 7f9140a00640 1 Processor -- start 2024-01-16T15:04:17.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.012+0000 7f9140a00640 1 -- start start 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f9140a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f913c141e80 0x7f913c06d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f9140a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c070bf0 con 0x7f913c1416f0 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f9140a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c06d850 con 0x7f913c15f460 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f9140a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c06db30 con 0x7f913c141e80 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f913a800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f913c1416f0 0x7f913c15ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32874/0 (socket says 172.21.15.43:32874) 2024-01-16T15:04:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.014+0000 7f913a800640 1 -- 172.21.15.43:0/1757001311 learned_addr learned my addr 172.21.15.43:0/1757001311 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:17.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f913b200640 1 --2- 172.21.15.43:0/1757001311 >> v2:172.21.15.69:3300/0 conn(0x7f913c141e80 0x7f913c06d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488733997 0 0) 0x7f913c070bf0 con 0x7f913c1416f0 2024-01-16T15:04:17.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c003520 con 0x7f913c1416f0 2024-01-16T15:04:17.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845081114 0 0) 0x7f911c003520 con 0x7f913c1416f0 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 >> v2:172.21.15.69:3300/0 conn(0x7f913c141e80 msgr2=0x7f913c06d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 --2- 172.21.15.43:0/1757001311 >> v2:172.21.15.69:3300/0 conn(0x7f913c141e80 0x7f913c06d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 >> v1:172.21.15.69:6789/0 conn(0x7f913c15f460 legacy=0x7f913c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0705a0 con 0x7f913c1416f0 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f912c001670 con 0x7f913c1416f0 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 >> v1:172.21.15.43:6789/0 conn(0x7f913c1416f0 legacy=0x7f913c15ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.015+0000 7f9139e00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f911c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9139e00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9139e00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f911c005180 0x7f911c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c06db30 con 0x7f911c0039e0 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c00c0b0 con 0x7f911c005180 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c00c270 con 0x7f911c0047c0 2024-01-16T15:04:17.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f911c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9133e00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 -- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f911c0047c0 msgr2=0x7f911c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 -- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f911c005180 msgr2=0x7f911c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f911c005180 0x7f911c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913a800640 1 -- 172.21.15.43:0/1757001311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c003520 con 0x7f911c0039e0 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f913b200640 1 -- 172.21.15.43:0/1757001311 reap_dead start 2024-01-16T15:04:17.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.016+0000 7f9133e00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.018+0000 7f913a800640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f911c003e00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c240 tx=0x7f911c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619512bb7db00fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.018+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c005ad0 con 0x7f911c0039e0 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.018+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c005c50 con 0x7f911c0039e0 2024-01-16T15:04:17.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.018+0000 7f9139e00640 1 -- 172.21.15.43:0/1757001311 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c02c8b0 con 0x7f911c0039e0 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 -- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 msgr2=0x7f911c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f911c003e00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c240 tx=0x7f911c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 -- 172.21.15.43:0/1757001311 shutdown_connections 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 --2- 172.21.15.43:0/1757001311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f911c003e00 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 -- 172.21.15.43:0/1757001311 >> 172.21.15.43:0/1757001311 conn(0x7f913c0b0840 msgr2=0x7f913c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 -- 172.21.15.43:0/1757001311 shutdown_connections 2024-01-16T15:04:17.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.019+0000 7f9140a00640 1 -- 172.21.15.43:0/1757001311 wait complete. 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.021+0000 7f9140a00640 1 Processor -- start 2024-01-16T15:04:18.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.021+0000 7f9140a00640 1 -- start start 2024-01-16T15:04:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f913c15c440 0x7f913c13c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f913c13c9b0 0x7f913c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15b710 con 0x7f911c0039e0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c150f40 con 0x7f913c13c9b0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9140a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15ddc0 con 0x7f913c15c440 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f913a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f913a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45984/0 (socket says 172.21.15.43:45984) 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f913a800640 1 -- 172.21.15.43:0/3067540400 learned_addr learned my addr 172.21.15.43:0/3067540400 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f9133e00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f913c15c440 0x7f913c13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.022+0000 7f913a800640 1 -- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f913c15c440 msgr2=0x7f913c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f913a800640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f913c15c440 0x7f913c13c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f913a800640 1 -- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f913c13c9b0 msgr2=0x7f913c0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f913a800640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f913c13c9b0 0x7f913c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f913a800640 1 -- 172.21.15.43:0/3067540400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c13cfe0 con 0x7f911c0039e0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f913a800640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f912c02c450 tx=0x7f912c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c03c040 con 0x7f911c0039e0 2024-01-16T15:04:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c02ccc0 con 0x7f911c0039e0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c007290 con 0x7f911c0039e0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c15b960 con 0x7f911c0039e0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.023+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c15bf20 con 0x7f911c0039e0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.024+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c15c0e0 con 0x7f911c0039e0 2024-01-16T15:04:18.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.025+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f912c032070 con 0x7f911c0039e0 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.027+0000 7f9132a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 0x7f911406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.027+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f912c0ab0c0 con 0x7f911c0039e0 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.027+0000 7f9133e00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 0x7f911406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.027+0000 7f9133e00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 0x7f911406dcc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9124000c00 tx=0x7f9124009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.031+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f912c0dde90 con 0x7f911c0039e0 2024-01-16T15:04:18.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.248+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f913c13cdd0 con 0x7f911406b800 2024-01-16T15:04:18.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.250+0000 7f9132a00640 1 -- 172.21.15.43:0/3067540400 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f913c13cdd0 con 0x7f911406b800 2024-01-16T15:04:18.229 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:04:18.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:04:18.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:18.230 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.252+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 msgr2=0x7f911406dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.252+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 0x7f911406dcc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9124000c00 tx=0x7f9124009040 comp rx=0 tx=0).stop 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.252+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 msgr2=0x7f913c15d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.252+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f912c02c450 tx=0x7f912c02e070 comp rx=0 tx=0).stop 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 shutdown_connections 2024-01-16T15:04:18.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f911406b800 0x7f911406dcc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f913c13c9b0 0x7f913c0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f913c15c440 0x7f913c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 --2- 172.21.15.43:0/3067540400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f911c0039e0 0x7f913c15d880 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 >> 172.21.15.43:0/3067540400 conn(0x7f913c0b0840 msgr2=0x7f913c144430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 shutdown_connections 2024-01-16T15:04:18.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.253+0000 7f9140a00640 1 -- 172.21.15.43:0/3067540400 wait complete. 2024-01-16T15:04:18.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:04:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.492+0000 7f1e81800640 1 Processor -- start 2024-01-16T15:04:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.493+0000 7f1e81800640 1 -- start start 2024-01-16T15:04:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.494+0000 7f1e81800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 0x7f1e7c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.494+0000 7f1e81800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.494+0000 7f1e81800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c15b730 con 0x7f1e7c15b350 2024-01-16T15:04:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.494+0000 7f1e81800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0ac740 con 0x7f1e7c09cb30 2024-01-16T15:04:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.494+0000 7f1e81800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0ac460 con 0x7f1e7c0adb60 2024-01-16T15:04:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.495+0000 7f1e73e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 0x7f1e7c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.495+0000 7f1e7b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.495+0000 7f1e7aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1e7c15b350 0x7f1e7c0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:39042/0 (socket says 172.21.15.43:39042) 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.495+0000 7f1e7aa00640 1 -- 172.21.15.43:0/3636269036 learned_addr learned my addr 172.21.15.43:0/3636269036 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.496+0000 7f1e7b400640 1 -- 172.21.15.43:0/3636269036 >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 msgr2=0x7f1e7c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.496+0000 7f1e7b400640 1 --2- 172.21.15.43:0/3636269036 >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 0x7f1e7c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.496+0000 7f1e7b400640 1 -- 172.21.15.43:0/3636269036 >> v1:172.21.15.69:6789/0 conn(0x7f1e7c15b350 legacy=0x7f1e7c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.496+0000 7f1e7b400640 1 -- 172.21.15.43:0/3636269036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c0b2a50 con 0x7f1e7c0adb60 2024-01-16T15:04:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.496+0000 7f1e73e00640 1 --2- 172.21.15.43:0/3636269036 >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 0x7f1e7c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.498+0000 7f1e7b400640 1 --2- 172.21.15.43:0/3636269036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c0adfb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268a2869569e142e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.498+0000 7f1e7a000640 1 -- 172.21.15.43:0/3636269036 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7402cc60 con 0x7f1e7c0adb60 2024-01-16T15:04:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.498+0000 7f1e7a000640 1 -- 172.21.15.43:0/3636269036 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7402cdc0 con 0x7f1e7c0adb60 2024-01-16T15:04:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.498+0000 7f1e7a000640 1 -- 172.21.15.43:0/3636269036 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74034660 con 0x7f1e7c0adb60 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 -- 172.21.15.43:0/3636269036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 msgr2=0x7f1e7c0adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 --2- 172.21.15.43:0/3636269036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c0adfb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7402beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 -- 172.21.15.43:0/3636269036 shutdown_connections 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 --2- 172.21.15.43:0/3636269036 >> v2:172.21.15.69:3300/0 conn(0x7f1e7c09cb30 0x7f1e7c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 --2- 172.21.15.43:0/3636269036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c0adfb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 -- 172.21.15.43:0/3636269036 >> 172.21.15.43:0/3636269036 conn(0x7f1e7c0b0ba0 msgr2=0x7f1e7c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 -- 172.21.15.43:0/3636269036 shutdown_connections 2024-01-16T15:04:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.499+0000 7f1e81800640 1 -- 172.21.15.43:0/3636269036 wait complete. 2024-01-16T15:04:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.501+0000 7f1e81800640 1 Processor -- start 2024-01-16T15:04:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.502+0000 7f1e81800640 1 -- start start 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c15b350 0x7f1e7c13e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c13c670 con 0x7f1e7c09cb30 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c14bb40 con 0x7f1e7c15b350 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e81800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c14be20 con 0x7f1e7c0adb60 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e7aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e7aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46000/0 (socket says 172.21.15.43:46000) 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e7aa00640 1 -- 172.21.15.43:0/50234447 learned_addr learned my addr 172.21.15.43:0/50234447 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.503+0000 7f1e73e00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 -- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 msgr2=0x7f1e7c149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 -- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c15b350 msgr2=0x7f1e7c13e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c15b350 0x7f1e7c13e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 -- 172.21.15.43:0/50234447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64009400 con 0x7f1e7c09cb30 2024-01-16T15:04:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e73e00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e7aa00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00c400 tx=0x7f1e6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c005ce0 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c005e40 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0072f0 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e74009a70 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.504+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c14c3e0 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.505+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c09d200 con 0x7f1e7c09cb30 2024-01-16T15:04:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.506+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f1e6c0074d0 con 0x7f1e7c09cb30 2024-01-16T15:04:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.507+0000 7f1e72a00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 0x7f1e5406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.507+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1e6c089760 con 0x7f1e7c09cb30 2024-01-16T15:04:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.508+0000 7f1e73e00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 0x7f1e5406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.508+0000 7f1e73e00640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 0x7f1e5406dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c1f40f0 tx=0x7f1e64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.511+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e6c053fc0 con 0x7f1e7c09cb30 2024-01-16T15:04:18.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.738+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e7c0ae4f0 con 0x7f1e5406b7d0 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.750+0000 7f1e72a00640 1 -- 172.21.15.43:0/50234447 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7f1e640109a0 con 0x7f1e5406b7d0 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone24922" 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone24924" 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone24921" 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone24923" 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:18.731 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:04:18.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.753+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 msgr2=0x7f1e5406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.753+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 0x7f1e5406dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c1f40f0 tx=0x7f1e64008040 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 msgr2=0x7f1e7c13c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00c400 tx=0x7f1e6c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 shutdown_connections 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e5406b7d0 0x7f1e5406dc90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c15b350 0x7f1e7c13e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e7c0adb60 0x7f1e7c149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 --2- 172.21.15.43:0/50234447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c09cb30 0x7f1e7c13c130 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 >> 172.21.15.43:0/50234447 conn(0x7f1e7c0b0ba0 msgr2=0x7f1e7c156fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 shutdown_connections 2024-01-16T15:04:18.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.754+0000 7f1e81800640 1 -- 172.21.15.43:0/50234447 wait complete. 2024-01-16T15:04:18.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:04:18.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.997+0000 7fbd85400640 1 Processor -- start 2024-01-16T15:04:18.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.998+0000 7fbd85400640 1 -- start start 2024-01-16T15:04:18.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.999+0000 7fbd85400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbd800acd70 0x7fbd8006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.999+0000 7fbd85400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd800ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.999+0000 7fbd85400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8006d7a0 con 0x7fbd800ad690 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.999+0000 7fbd85400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8015a5b0 con 0x7fbd800acd70 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:18.999+0000 7fbd85400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006ce70 con 0x7fbd800ae490 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7f200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbd800ad690 0x7fbd80070490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32888/0 (socket says 172.21.15.43:32888) 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7f200640 1 -- 172.21.15.43:0/1529969677 learned_addr learned my addr 172.21.15.43:0/1529969677 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7de00640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd800ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7e800640 1 --2- 172.21.15.43:0/1529969677 >> v2:172.21.15.69:3300/0 conn(0x7fbd800acd70 0x7fbd8006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563427710 0 0) 0x7fbd8006d7a0 con 0x7fbd800ad690 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.000+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd60003520 con 0x7fbd800ad690 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 128575147 0 0) 0x7fbd60003520 con 0x7fbd800ad690 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 msgr2=0x7fbd800ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd800ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 >> v2:172.21.15.69:3300/0 conn(0x7fbd800acd70 msgr2=0x7fbd8006dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 --2- 172.21.15.43:0/1529969677 >> v2:172.21.15.69:3300/0 conn(0x7fbd800acd70 0x7fbd8006dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8006d150 con 0x7fbd800ad690 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fbd780019f0 con 0x7fbd800ad690 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 >> v1:172.21.15.43:6789/0 conn(0x7fbd800ad690 legacy=0x7fbd80070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd60003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd60003ef0 0x7fbd600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60004d00 0x7fbd60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8015a5b0 con 0x7fbd60003ef0 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006ce70 con 0x7fbd60004d00 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6000db90 con 0x7fbd800ae490 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7de00640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd60003ef0 0x7fbd600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.001+0000 7fbd7e800640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd60003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7f200640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60004d00 0x7fbd60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7de00640 1 -- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 msgr2=0x7fbd60003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7de00640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd60003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7de00640 1 -- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60004d00 msgr2=0x7fbd60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7de00640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60004d00 0x7fbd60005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7de00640 1 -- 172.21.15.43:0/1529969677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60003520 con 0x7fbd60003ef0 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7e800640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800ae490 0x7fbd60003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.002+0000 7fbd7f200640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60004d00 0x7fbd60005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:18.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.003+0000 7fbd7de00640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd60003ef0 0x7fbd600047c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbd68009ac0 tx=0x7fbd600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661cf579d5fdd29b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.003+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6802c6b0 con 0x7fbd60003ef0 2024-01-16T15:04:18.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.003+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6802c810 con 0x7fbd60003ef0 2024-01-16T15:04:18.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.003+0000 7fbd7d400640 1 -- 172.21.15.43:0/1529969677 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6802cba0 con 0x7fbd60003ef0 2024-01-16T15:04:18.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 -- 172.21.15.43:0/1529969677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd60003ef0 msgr2=0x7fbd600047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 --2- 172.21.15.43:0/1529969677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd60003ef0 0x7fbd600047c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbd68009ac0 tx=0x7fbd600057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd7f200640 1 -- 172.21.15.43:0/1529969677 reap_dead start 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 -- 172.21.15.43:0/1529969677 shutdown_connections 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 -- 172.21.15.43:0/1529969677 >> 172.21.15.43:0/1529969677 conn(0x7fbd800b0ba0 msgr2=0x7fbd800b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 -- 172.21.15.43:0/1529969677 shutdown_connections 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.004+0000 7fbd85400640 1 -- 172.21.15.43:0/1529969677 wait complete. 2024-01-16T15:04:18.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.006+0000 7fbd85400640 1 Processor -- start 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.006+0000 7fbd85400640 1 -- start start 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60003ef0 0x7fbd800cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800cba80 0x7fbd8015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800cd570 con 0x7fbd80146a30 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800cbf90 con 0x7fbd60003ef0 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd85400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800cc270 con 0x7fbd800cba80 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd7de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd7de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46016/0 (socket says 172.21.15.43:46016) 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd7de00640 1 -- 172.21.15.43:0/2538675064 learned_addr learned my addr 172.21.15.43:0/2538675064 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.007+0000 7fbd7f200640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800cba80 0x7fbd8015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7e800640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60003ef0 0x7fbd800cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 -- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800cba80 msgr2=0x7fbd8015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800cba80 0x7fbd8015c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 -- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60003ef0 msgr2=0x7fbd800cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60003ef0 0x7fbd800cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 -- 172.21.15.43:0/2538675064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd800cc550 con 0x7fbd80146a30 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd7de00640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbd6802c2b0 tx=0x7fbd68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68003810 con 0x7fbd80146a30 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd68003970 con 0x7fbd80146a30 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.008+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68003ce0 con 0x7fbd80146a30 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.009+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd800cd570 con 0x7fbd80146a30 2024-01-16T15:04:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.009+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd800ccb10 con 0x7fbd80146a30 2024-01-16T15:04:18.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.010+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd800ccca0 con 0x7fbd80146a30 2024-01-16T15:04:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.011+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fbd68034440 con 0x7fbd80146a30 2024-01-16T15:04:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.012+0000 7fbd76a00640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 0x7fbd5806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.012+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd680ac6c0 con 0x7fbd80146a30 2024-01-16T15:04:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.012+0000 7fbd7e800640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 0x7fbd5806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.013+0000 7fbd7e800640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 0x7fbd5806dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009a40 tx=0x7fbd70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:18.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.016+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd68076ec0 con 0x7fbd80146a30 2024-01-16T15:04:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:19.244+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd800b3d90 con 0x7fbd5806b7d0 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.568+0000 7fbd76a00640 1 -- 172.21.15.43:0/2538675064 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fbd700119a0 con 0x7fbd5806b7d0 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:04:20.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:20.548 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:04:20.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.571+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 msgr2=0x7fbd5806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.571+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 0x7fbd5806dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009a40 tx=0x7fbd70008040 comp rx=0 tx=0).stop 2024-01-16T15:04:20.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.571+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 msgr2=0x7fbd800cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.571+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbd6802c2b0 tx=0x7fbd68004730 comp rx=0 tx=0).stop 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 shutdown_connections 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd5806b7d0 0x7fbd5806dc90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd800cba80 0x7fbd8015c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80146a30 0x7fbd800cb540 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 --2- 172.21.15.43:0/2538675064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd60003ef0 0x7fbd800cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 >> 172.21.15.43:0/2538675064 conn(0x7fbd800b0ba0 msgr2=0x7fbd800aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 shutdown_connections 2024-01-16T15:04:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.572+0000 7fbd85400640 1 -- 172.21.15.43:0/2538675064 wait complete. 2024-01-16T15:04:20.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:04:20.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.813+0000 7ff820a00640 1 Processor -- start 2024-01-16T15:04:20.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.814+0000 7ff820a00640 1 -- start start 2024-01-16T15:04:20.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.815+0000 7ff820a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff81815b350 0x7ff8180abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.815+0000 7ff820a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8180adb60 0x7ff8180adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.815+0000 7ff820a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81815b730 con 0x7ff81809cb30 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.815+0000 7ff820a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8180ac740 con 0x7ff81815b350 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.815+0000 7ff820a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8180ac460 con 0x7ff8180adb60 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff81815b350 0x7ff8180abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff81815b350 0x7ff8180abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46018/0 (socket says 172.21.15.43:46018) 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 -- 172.21.15.43:0/950235642 learned_addr learned my addr 172.21.15.43:0/950235642 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 -- 172.21.15.43:0/950235642 >> v2:172.21.15.69:3300/0 conn(0x7ff8180adb60 msgr2=0x7ff8180adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 --2- 172.21.15.43:0/950235642 >> v2:172.21.15.69:3300/0 conn(0x7ff8180adb60 0x7ff8180adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 -- 172.21.15.43:0/950235642 >> v1:172.21.15.69:6789/0 conn(0x7ff81809cb30 legacy=0x7ff8180b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:20.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.816+0000 7ff81e000640 1 -- 172.21.15.43:0/950235642 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8180b2a50 con 0x7ff81815b350 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.817+0000 7ff81e000640 1 --2- 172.21.15.43:0/950235642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff8180abe80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff81806dac0 tx=0x7ff80802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594990b8f453e2a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff81cc00640 1 -- 172.21.15.43:0/950235642 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808031890 con 0x7ff81815b350 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff81cc00640 1 -- 172.21.15.43:0/950235642 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8080319f0 con 0x7ff81815b350 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff81cc00640 1 -- 172.21.15.43:0/950235642 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808031d80 con 0x7ff81815b350 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 -- 172.21.15.43:0/950235642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 msgr2=0x7ff8180abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 --2- 172.21.15.43:0/950235642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff8180abe80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff81806dac0 tx=0x7ff80802b930 comp rx=0 tx=0).stop 2024-01-16T15:04:20.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 -- 172.21.15.43:0/950235642 shutdown_connections 2024-01-16T15:04:20.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 --2- 172.21.15.43:0/950235642 >> v2:172.21.15.69:3300/0 conn(0x7ff8180adb60 0x7ff8180adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 --2- 172.21.15.43:0/950235642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff8180abe80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.818+0000 7ff820a00640 1 -- 172.21.15.43:0/950235642 >> 172.21.15.43:0/950235642 conn(0x7ff8180b0ba0 msgr2=0x7ff8180b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:20.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.819+0000 7ff820a00640 1 -- 172.21.15.43:0/950235642 shutdown_connections 2024-01-16T15:04:20.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.819+0000 7ff820a00640 1 -- 172.21.15.43:0/950235642 wait complete. 2024-01-16T15:04:20.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.820+0000 7ff820a00640 1 Processor -- start 2024-01-16T15:04:20.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 -- start start 2024-01-16T15:04:20.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81809cb30 0x7ff81813b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8180adb60 0x7ff81814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81814d610 con 0x7ff81815b350 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.821+0000 7ff820a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81814c780 con 0x7ff81809cb30 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff820a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81814c990 con 0x7ff8180adb60 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46032/0 (socket says 172.21.15.43:46032) 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 -- 172.21.15.43:0/4130477796 learned_addr learned my addr 172.21.15.43:0/4130477796 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81e000640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81809cb30 0x7ff81813b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81d600640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8180adb60 0x7ff81814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 -- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8180adb60 msgr2=0x7ff81814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8180adb60 0x7ff81814da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 -- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81809cb30 msgr2=0x7ff81813b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.822+0000 7ff81ea00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81809cb30 0x7ff81813b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff81ea00640 1 -- 172.21.15.43:0/4130477796 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8080094d0 con 0x7ff81815b350 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff81ea00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b970 tx=0x7ff81000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81000cb40 con 0x7ff81815b350 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81000ccc0 con 0x7ff81815b350 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810014680 con 0x7ff81815b350 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81814cc70 con 0x7ff81815b350 2024-01-16T15:04:20.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.823+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81814d230 con 0x7ff81815b350 2024-01-16T15:04:20.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.824+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81814d750 con 0x7ff81815b350 2024-01-16T15:04:20.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.826+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7ff81001d3b0 con 0x7ff81815b350 2024-01-16T15:04:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.827+0000 7ff816000640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 0x7ff7f006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.827+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff81008a580 con 0x7ff81815b350 2024-01-16T15:04:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.827+0000 7ff81e000640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 0x7ff7f006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:20.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.828+0000 7ff81e000640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 0x7ff7f006dcc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff808036000 tx=0x7ff80803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:20.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:20.831+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8100588f0 con 0x7ff81815b350 2024-01-16T15:04:21.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.061+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff81806e720 con 0x7ff7f006b800 2024-01-16T15:04:21.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.133+0000 7ff816000640 1 -- 172.21.15.43:0/4130477796 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff808031ce0 con 0x7ff7f006b800 2024-01-16T15:04:21.113 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:21.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 msgr2=0x7ff7f006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 0x7ff7f006dcc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff808036000 tx=0x7ff80803c040 comp rx=0 tx=0).stop 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 msgr2=0x7ff818155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b970 tx=0x7ff81000be30 comp rx=0 tx=0).stop 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 shutdown_connections 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7f006b800 0x7ff7f006dcc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff81815b350 0x7ff818155170 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8180adb60 0x7ff81814da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 --2- 172.21.15.43:0/4130477796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff81809cb30 0x7ff81813b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 >> 172.21.15.43:0/4130477796 conn(0x7ff8180b0ba0 msgr2=0x7ff8180b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:21.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 shutdown_connections 2024-01-16T15:04:21.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.137+0000 7ff820a00640 1 -- 172.21.15.43:0/4130477796 wait complete. 2024-01-16T15:04:21.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:04:21.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.368+0000 7fd90c400640 1 Processor -- start 2024-01-16T15:04:21.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.369+0000 7fd90c400640 1 -- start start 2024-01-16T15:04:21.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.370+0000 7fd90c400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd90415d350 0x7fd9040d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.370+0000 7fd90c400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd9040d1840 0x7fd9040cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.370+0000 7fd90c400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040cf240 con 0x7fd90415d730 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.370+0000 7fd90c400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040d1fb0 con 0x7fd9040d1840 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.370+0000 7fd90c400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040ce050 con 0x7fd90415d350 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd9040d1840 0x7fd9040cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd9040d1840 0x7fd9040cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46042/0 (socket says 172.21.15.43:46042) 2024-01-16T15:04:21.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 -- 172.21.15.43:0/973923673 learned_addr learned my addr 172.21.15.43:0/973923673 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:21.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd909a00640 1 --2- 172.21.15.43:0/973923673 >> v2:172.21.15.69:3300/0 conn(0x7fd90415d350 0x7fd9040d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 -- 172.21.15.43:0/973923673 >> v2:172.21.15.69:3300/0 conn(0x7fd90415d350 msgr2=0x7fd9040d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:21.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 --2- 172.21.15.43:0/973923673 >> v2:172.21.15.69:3300/0 conn(0x7fd90415d350 0x7fd9040d3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 -- 172.21.15.43:0/973923673 >> v1:172.21.15.69:6789/0 conn(0x7fd90415d730 legacy=0x7fd9040ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:21.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.371+0000 7fd90a400640 1 -- 172.21.15.43:0/973923673 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9040ce330 con 0x7fd9040d1840 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.372+0000 7fd90a400640 1 --2- 172.21.15.43:0/973923673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040cfb50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd90408bab0 tx=0x7fd8fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d249256a374db87a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.373+0000 7fd903e00640 1 -- 172.21.15.43:0/973923673 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031870 con 0x7fd9040d1840 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.373+0000 7fd903e00640 1 -- 172.21.15.43:0/973923673 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc0319d0 con 0x7fd9040d1840 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.373+0000 7fd903e00640 1 -- 172.21.15.43:0/973923673 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031d60 con 0x7fd9040d1840 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 -- 172.21.15.43:0/973923673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 msgr2=0x7fd9040cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 --2- 172.21.15.43:0/973923673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040cfb50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd90408bab0 tx=0x7fd8fc02b910 comp rx=0 tx=0).stop 2024-01-16T15:04:21.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 -- 172.21.15.43:0/973923673 shutdown_connections 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 --2- 172.21.15.43:0/973923673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040cfb50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 --2- 172.21.15.43:0/973923673 >> v2:172.21.15.69:3300/0 conn(0x7fd90415d350 0x7fd9040d3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 -- 172.21.15.43:0/973923673 >> 172.21.15.43:0/973923673 conn(0x7fd90409c5c0 msgr2=0x7fd9040d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 -- 172.21.15.43:0/973923673 shutdown_connections 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.374+0000 7fd90c400640 1 -- 172.21.15.43:0/973923673 wait complete. 2024-01-16T15:04:21.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.376+0000 7fd90c400640 1 Processor -- start 2024-01-16T15:04:21.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.376+0000 7fd90c400640 1 -- start start 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.377+0000 7fd90c400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.377+0000 7fd90c400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 0x7fd90414a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 0x7fd9041f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90c400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90414acb0 con 0x7fd9040d1840 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9041f43b0 con 0x7fd90415d730 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90c400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040b6e30 con 0x7fd90415d350 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46054/0 (socket says 172.21.15.43:46054) 2024-01-16T15:04:21.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 -- 172.21.15.43:0/2522180811 learned_addr learned my addr 172.21.15.43:0/2522180811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 -- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 msgr2=0x7fd90414a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909000640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 0x7fd90414a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90a400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 0x7fd9041f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 0x7fd90414a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 -- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 msgr2=0x7fd9041f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 0x7fd9041f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909a00640 1 -- 172.21.15.43:0/2522180811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0094d0 con 0x7fd9040d1840 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd909000640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 0x7fd90414a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:21.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.378+0000 7fd90a400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 0x7fd9041f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd909a00640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4007ef0 tx=0x7fd8f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4014620 con 0x7fd9040d1840 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4014780 con 0x7fd9040d1840 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9040b7110 con 0x7fd9040d1840 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4014b30 con 0x7fd9040d1840 2024-01-16T15:04:21.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.379+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90415ee00 con 0x7fd9040d1840 2024-01-16T15:04:21.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.381+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9040a1230 con 0x7fd9040d1840 2024-01-16T15:04:21.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.382+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd8f4014c90 con 0x7fd9040d1840 2024-01-16T15:04:21.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.382+0000 7fd901600640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.383+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8f408aef0 con 0x7fd9040d1840 2024-01-16T15:04:21.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.383+0000 7fd909000640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.384+0000 7fd909000640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd9040cfb50 tx=0x7fd8ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.387+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8f4055610 con 0x7fd9040d1840 2024-01-16T15:04:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.610+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9040cdd00 con 0x7fd8dc06b7d0 2024-01-16T15:04:21.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.663+0000 7fd901600640 1 -- 172.21.15.43:0/2522180811 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd8ec00ad80 con 0x7fd8dc06b7d0 2024-01-16T15:04:21.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 msgr2=0x7fd8dc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd9040cfb50 tx=0x7fd8ec008040 comp rx=0 tx=0).stop 2024-01-16T15:04:21.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 msgr2=0x7fd9040b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4007ef0 tx=0x7fd8f400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 shutdown_connections 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd90415d730 0x7fd9041f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd90415d350 0x7fd90414a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 --2- 172.21.15.43:0/2522180811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9040d1840 0x7fd9040b6850 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 >> 172.21.15.43:0/2522180811 conn(0x7fd90409c5c0 msgr2=0x7fd90406c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 shutdown_connections 2024-01-16T15:04:21.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.667+0000 7fd90c400640 1 -- 172.21.15.43:0/2522180811 wait complete. 2024-01-16T15:04:21.649 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.670+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.650 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.670+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.650 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.670+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.650 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.670+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.650 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.670+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone24922 2024-01-16T15:04:21.678 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.696+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.678 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.696+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.678 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.696+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.678 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.696+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.678 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:21.696+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.891+0000 7f8efd600640 1 Processor -- start 2024-01-16T15:04:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.891+0000 7f8efd600640 1 -- start start 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.892+0000 7f8efd600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ef806dc30 0x7f8ef80ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.892+0000 7f8efd600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef806d7a0 con 0x7f8ef8070490 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8efd600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef806ce70 con 0x7f8ef80acd70 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8efd600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef815a5b0 con 0x7f8ef806dc30 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8ef7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8ef8070490 0x7f8ef8070900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32902/0 (socket says 172.21.15.43:32902) 2024-01-16T15:04:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8ef7400640 1 -- 172.21.15.43:0/1146724338 learned_addr learned my addr 172.21.15.43:0/1146724338 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8eefe00640 1 --2- 172.21.15.43:0/1146724338 >> v2:172.21.15.69:3300/0 conn(0x7f8ef806dc30 0x7f8ef80ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255292015 0 0) 0x7f8ef806d7a0 con 0x7f8ef8070490 2024-01-16T15:04:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.893+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed8005730 con 0x7f8ef8070490 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946383242 0 0) 0x7f8ed8005730 con 0x7f8ef8070490 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 >> v2:172.21.15.69:3300/0 conn(0x7f8ef806dc30 msgr2=0x7f8ef80ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 --2- 172.21.15.43:0/1146724338 >> v2:172.21.15.69:3300/0 conn(0x7f8ef806dc30 0x7f8ef80ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 >> v1:172.21.15.69:6789/0 conn(0x7f8ef80acd70 legacy=0x7f8ef80ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef806d150 con 0x7f8ef8070490 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8ef00019f0 con 0x7f8ef8070490 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 >> v1:172.21.15.43:6789/0 conn(0x7f8ef8070490 legacy=0x7f8ef8070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed8005bd0 0x7f8ed8005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed80069d0 0x7f8ed8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 0x7f8ed800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef815a5b0 con 0x7f8ed8007390 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800e290 con 0x7f8ed80069d0 2024-01-16T15:04:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800e450 con 0x7f8ed8005bd0 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.894+0000 7f8ef7400640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 0x7f8ed800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef6a00640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed8005bd0 0x7f8ed8005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8eefe00640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed80069d0 0x7f8ed8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef7400640 1 -- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed8005bd0 msgr2=0x7f8ed8005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef7400640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed8005bd0 0x7f8ed8005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef7400640 1 -- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed80069d0 msgr2=0x7f8ed8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef7400640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed80069d0 0x7f8ed8006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.895+0000 7f8ef7400640 1 -- 172.21.15.43:0/1146724338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8005730 con 0x7f8ed8007390 2024-01-16T15:04:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.896+0000 7f8ef7400640 1 -- 172.21.15.43:0/1146724338 reap_dead start 2024-01-16T15:04:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.896+0000 7f8ef7400640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 0x7f8ed800db90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000a760 tx=0x7f8ed8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72a0dff575547f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0034630 con 0x7f8ed8007390 2024-01-16T15:04:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef002f070 con 0x7f8ed8007390 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8ef6000640 1 -- 172.21.15.43:0/1146724338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0034a10 con 0x7f8ed8007390 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 -- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 msgr2=0x7f8ed800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 0x7f8ed800db90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000a760 tx=0x7f8ed8007a00 comp rx=0 tx=0).stop 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 -- 172.21.15.43:0/1146724338 shutdown_connections 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 --2- 172.21.15.43:0/1146724338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed8007390 0x7f8ed800db90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 -- 172.21.15.43:0/1146724338 >> 172.21.15.43:0/1146724338 conn(0x7f8ef80b0ba0 msgr2=0x7f8ef80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 -- 172.21.15.43:0/1146724338 shutdown_connections 2024-01-16T15:04:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.897+0000 7f8efd600640 1 -- 172.21.15.43:0/1146724338 wait complete. 2024-01-16T15:04:21.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.899+0000 7f8efd600640 1 Processor -- start 2024-01-16T15:04:21.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.899+0000 7f8efd600640 1 -- start start 2024-01-16T15:04:21.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed8007390 0x7f8ef815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 0x7f8ef814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80ae9d0 con 0x7f8ef80ccb30 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef81eece0 con 0x7f8ed8007390 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.900+0000 7f8efd600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef809be80 con 0x7f8ef80cc6b0 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46096/0 (socket says 172.21.15.43:46096) 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 -- 172.21.15.43:0/3606379731 learned_addr learned my addr 172.21.15.43:0/3606379731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef6a00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed8007390 0x7f8ef815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8eefe00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 0x7f8ef814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 -- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 msgr2=0x7f8ef814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 0x7f8ef814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 -- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed8007390 msgr2=0x7f8ef815cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed8007390 0x7f8ef815cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8ef7400640 1 -- 172.21.15.43:0/3606379731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef809c160 con 0x7f8ef80ccb30 2024-01-16T15:04:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.901+0000 7f8eefe00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 0x7f8ef814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8ef7400640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0002230 tx=0x7f8ef0001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0042070 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef00345f0 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0004d80 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef809c440 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.902+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef809ba70 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.903+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef81eef00 con 0x7f8ef80ccb30 2024-01-16T15:04:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.904+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f8ef0031030 con 0x7f8ef80ccb30 2024-01-16T15:04:21.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.905+0000 7f8eeea00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 0x7f8ed006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:21.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.905+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8ef00ab1d0 con 0x7f8ef80ccb30 2024-01-16T15:04:21.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.905+0000 7f8ef6a00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 0x7f8ed006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:21.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.906+0000 7f8ef6a00640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 0x7f8ed006dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:21.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:21.909+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ef81eef00 con 0x7f8ef80ccb30 2024-01-16T15:04:22.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.132+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone24922", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ef80cb290 con 0x7f8ed006b7d0 2024-01-16T15:04:22.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.169+0000 7f8eeea00640 1 -- 172.21.15.43:0/3606379731 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8ee8005020 con 0x7f8ed006b7d0 2024-01-16T15:04:22.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.172+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 msgr2=0x7f8ed006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.172+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 0x7f8ed006dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8000c00 tx=0x7f8ee8009040 comp rx=0 tx=0).stop 2024-01-16T15:04:22.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 msgr2=0x7f8ef814fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0002230 tx=0x7f8ef0001cb0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 shutdown_connections 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ed006b7d0 0x7f8ed006dc90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ef80ccb30 0x7f8ef814fd90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ef80cc6b0 0x7f8ef814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 --2- 172.21.15.43:0/3606379731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed8007390 0x7f8ef815cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 >> 172.21.15.43:0/3606379731 conn(0x7f8ef80b0ba0 msgr2=0x7f8ef80aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 shutdown_connections 2024-01-16T15:04:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.173+0000 7f8efd600640 1 -- 172.21.15.43:0/3606379731 wait complete. 2024-01-16T15:04:22.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone24924 2024-01-16T15:04:22.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.380+0000 7f5bf0200640 1 Processor -- start 2024-01-16T15:04:22.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.380+0000 7f5bf0200640 1 -- start start 2024-01-16T15:04:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.381+0000 7f5bf0200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5be80acd70 0x7f5be806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.381+0000 7f5bf0200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 0x7f5be8070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bf0200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be806d7a0 con 0x7f5be80ae490 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bf0200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be815a5b0 con 0x7f5be80acd70 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bf0200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be806ce70 con 0x7f5be8070490 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5be80acd70 0x7f5be806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5be80acd70 0x7f5be806dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46102/0 (socket says 172.21.15.43:46102) 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 -- 172.21.15.43:0/3437636136 learned_addr learned my addr 172.21.15.43:0/3437636136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bee200640 1 --2- 172.21.15.43:0/3437636136 >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 0x7f5be8070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 -- 172.21.15.43:0/3437636136 >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 msgr2=0x7f5be8070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 --2- 172.21.15.43:0/3437636136 >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 0x7f5be8070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 -- 172.21.15.43:0/3437636136 >> v1:172.21.15.43:6789/0 conn(0x7f5be80ae490 legacy=0x7f5be80ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bed800640 1 -- 172.21.15.43:0/3437636136 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be806d150 con 0x7f5be80acd70 2024-01-16T15:04:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5be7e00640 1 -- 172.21.15.43:0/3437636136 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642433352 0 0) 0x7f5be806d7a0 con 0x7f5be80ae490 2024-01-16T15:04:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.382+0000 7f5bee200640 1 --2- 172.21.15.43:0/3437636136 >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 0x7f5be8070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.383+0000 7f5bed800640 1 --2- 172.21.15.43:0/3437636136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be806dc30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5be8067e70 tx=0x7f5bd802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d39fa7f3df31dfe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5be7e00640 1 -- 172.21.15.43:0/3437636136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802cb20 con 0x7f5be80acd70 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5be7e00640 1 -- 172.21.15.43:0/3437636136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd802cc80 con 0x7f5be80acd70 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5be7e00640 1 -- 172.21.15.43:0/3437636136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80343f0 con 0x7f5be80acd70 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 -- 172.21.15.43:0/3437636136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 msgr2=0x7f5be806dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 --2- 172.21.15.43:0/3437636136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be806dc30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5be8067e70 tx=0x7f5bd802bf80 comp rx=0 tx=0).stop 2024-01-16T15:04:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 -- 172.21.15.43:0/3437636136 shutdown_connections 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 --2- 172.21.15.43:0/3437636136 >> v2:172.21.15.69:3300/0 conn(0x7f5be8070490 0x7f5be8070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 --2- 172.21.15.43:0/3437636136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be806dc30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 -- 172.21.15.43:0/3437636136 >> 172.21.15.43:0/3437636136 conn(0x7f5be80b0ba0 msgr2=0x7f5be80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 -- 172.21.15.43:0/3437636136 shutdown_connections 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.384+0000 7f5bf0200640 1 -- 172.21.15.43:0/3437636136 wait complete. 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.386+0000 7f5bf0200640 1 Processor -- start 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.386+0000 7f5bf0200640 1 -- start start 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5be8070490 0x7f5be815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5be80ae490 0x7f5be814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815ce00 con 0x7f5be80acd70 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be81f3810 con 0x7f5be80ae490 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bf0200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8157b30 con 0x7f5be8070490 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.387+0000 7f5bece00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46116/0 (socket says 172.21.15.43:46116) 2024-01-16T15:04:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 -- 172.21.15.43:0/1732294157 learned_addr learned my addr 172.21.15.43:0/1732294157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bee200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5be80ae490 0x7f5be814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 -- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5be8070490 msgr2=0x7f5be815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5be8070490 0x7f5be815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 -- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5be80ae490 msgr2=0x7f5be814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5be80ae490 0x7f5be814ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 -- 172.21.15.43:0/1732294157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8009b30 con 0x7f5be80acd70 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.388+0000 7f5bece00640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c4b0 tx=0x7f5bd000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.389+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0005480 con 0x7f5be80acd70 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.389+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd00035f0 con 0x7f5be80acd70 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.389+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00039e0 con 0x7f5be80acd70 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.389+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be8157e10 con 0x7f5be80acd70 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.389+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81583d0 con 0x7f5be80acd70 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.390+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be81585f0 con 0x7f5be80acd70 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.391+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f5bd0003b40 con 0x7f5be80acd70 2024-01-16T15:04:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.392+0000 7f5be5600640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 0x7f5bc006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.392+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5bd00899f0 con 0x7f5be80acd70 2024-01-16T15:04:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.392+0000 7f5bed800640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 0x7f5bc006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.393+0000 7f5bed800640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 0x7f5bc006dcc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8035000 tx=0x7f5bd802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.396+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bd0054190 con 0x7f5be80acd70 2024-01-16T15:04:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.616+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone24924", "target": ["mon-mgr", ""]}) v1 -- 0x7f5be80b3d90 con 0x7f5bc006b800 2024-01-16T15:04:22.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.657+0000 7f5be5600640 1 -- 172.21.15.43:0/1732294157 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5bd8040480 con 0x7f5bc006b800 2024-01-16T15:04:22.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 msgr2=0x7f5bc006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 0x7f5bc006dcc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8035000 tx=0x7f5bd802e000 comp rx=0 tx=0).stop 2024-01-16T15:04:22.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 msgr2=0x7f5be81f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c4b0 tx=0x7f5bd000c970 comp rx=0 tx=0).stop 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 shutdown_connections 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bc006b800 0x7f5bc006dcc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5be80ae490 0x7f5be814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5be80acd70 0x7f5be81f32d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 --2- 172.21.15.43:0/1732294157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5be8070490 0x7f5be815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 >> 172.21.15.43:0/1732294157 conn(0x7f5be80b0ba0 msgr2=0x7f5be80ad150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 shutdown_connections 2024-01-16T15:04:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.661+0000 7f5bf0200640 1 -- 172.21.15.43:0/1732294157 wait complete. 2024-01-16T15:04:22.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone24921 2024-01-16T15:04:22.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.901+0000 7faf12000640 1 Processor -- start 2024-01-16T15:04:22.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.901+0000 7faf12000640 1 -- start start 2024-01-16T15:04:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf12000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf12000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 0x7faf0c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf12000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0c0c6bf0 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf12000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0c0d1400 con 0x7faf0c0d1d90 2024-01-16T15:04:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf12000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0cedc0 con 0x7faf0c15d3b0 2024-01-16T15:04:22.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf10c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf10c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34446/0 (socket says 172.21.15.43:34446) 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf10c00640 1 -- 172.21.15.43:0/3528241070 learned_addr learned my addr 172.21.15.43:0/3528241070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.903+0000 7faf11600640 1 --2- 172.21.15.43:0/3528241070 >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 0x7faf0c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.904+0000 7faf10c00640 1 -- 172.21.15.43:0/3528241070 >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 msgr2=0x7faf0c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.904+0000 7faf10c00640 1 --2- 172.21.15.43:0/3528241070 >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 0x7faf0c0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.904+0000 7faf10c00640 1 -- 172.21.15.43:0/3528241070 >> v1:172.21.15.69:6789/0 conn(0x7faf0c0d0fa0 legacy=0x7faf0c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.904+0000 7faf10c00640 1 -- 172.21.15.43:0/3528241070 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c0c6e40 con 0x7faf0c15d3b0 2024-01-16T15:04:22.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.904+0000 7faf11600640 1 --2- 172.21.15.43:0/3528241070 >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 0x7faf0c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:22.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.906+0000 7faf10c00640 1 --2- 172.21.15.43:0/3528241070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faefc009e50 tx=0x7faefc02bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca01caf54f8d77e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.906+0000 7faf0be00640 1 -- 172.21.15.43:0/3528241070 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc02ca80 con 0x7faf0c15d3b0 2024-01-16T15:04:22.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.906+0000 7faf0be00640 1 -- 172.21.15.43:0/3528241070 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc02cbe0 con 0x7faf0c15d3b0 2024-01-16T15:04:22.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.906+0000 7faf0be00640 1 -- 172.21.15.43:0/3528241070 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc0343f0 con 0x7faf0c15d3b0 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 -- 172.21.15.43:0/3528241070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 msgr2=0x7faf0c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 --2- 172.21.15.43:0/3528241070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faefc009e50 tx=0x7faefc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 -- 172.21.15.43:0/3528241070 shutdown_connections 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 --2- 172.21.15.43:0/3528241070 >> v2:172.21.15.69:3300/0 conn(0x7faf0c0d1d90 0x7faf0c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 --2- 172.21.15.43:0/3528241070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c15d3b0 0x7faf0c0ce6c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 -- 172.21.15.43:0/3528241070 >> 172.21.15.43:0/3528241070 conn(0x7faf0c0d30a0 msgr2=0x7faf0c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.907+0000 7faf12000640 1 -- 172.21.15.43:0/3528241070 shutdown_connections 2024-01-16T15:04:22.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.908+0000 7faf12000640 1 -- 172.21.15.43:0/3528241070 wait complete. 2024-01-16T15:04:22.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.909+0000 7faf12000640 1 Processor -- start 2024-01-16T15:04:22.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.909+0000 7faf12000640 1 -- start start 2024-01-16T15:04:22.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 0x7faf0c0b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf0c15d3b0 0x7faf0c15a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0b3480 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0b5cc0 con 0x7faf0c15d3b0 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.910+0000 7faf12000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0b61f0 con 0x7faf0c0d1d90 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46132/0 (socket says 172.21.15.43:46132) 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 -- 172.21.15.43:0/2507058832 learned_addr learned my addr 172.21.15.43:0/2507058832 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf11600640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf0c15d3b0 0x7faf0c15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf03c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 0x7faf0c0b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 -- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 msgr2=0x7faf0c0b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 0x7faf0c0b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 -- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf0c15d3b0 msgr2=0x7faf0c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf0c15d3b0 0x7faf0c15a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf10c00640 1 -- 172.21.15.43:0/2507058832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04009400 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.911+0000 7faf03c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 0x7faf0c0b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf10c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faefc036040 tx=0x7faefc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc040670 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc0407d0 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc009b30 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc040b80 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.912+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c0ca8f0 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.913+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c04e840 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.914+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) v1 ==== 89998+0+0 (secure 0 0 0) 0x7faefc02cd50 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.915+0000 7faf09600640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 0x7faee406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:22.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.915+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faefc0ad140 con 0x7faf0c0d0fa0 2024-01-16T15:04:22.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.916+0000 7faf03c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 0x7faee406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:22.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.917+0000 7faf03c00640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 0x7faee406dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faf0c15a0c0 tx=0x7faef4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:22.921+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faefc077930 con 0x7faf0c0d0fa0 2024-01-16T15:04:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:23.145+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone24921", "target": ["mon-mgr", ""]}) v1 -- 0x7faf0c0b8bd0 con 0x7faee406b7d0 2024-01-16T15:04:23.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:23.839+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7faefc077d00 con 0x7faf0c0d0fa0 2024-01-16T15:04:25.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:25.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:25.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.541+0000 7faf09600640 1 -- 172.21.15.43:0/2507058832 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faf0c0b8bd0 con 0x7faee406b7d0 2024-01-16T15:04:25.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 msgr2=0x7faee406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 0x7faee406dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faf0c15a0c0 tx=0x7faef4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 msgr2=0x7faf0c159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faefc036040 tx=0x7faefc004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 shutdown_connections 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faee406b7d0 0x7faee406dc90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf0c15d3b0 0x7faf0c15a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf0c0d1d90 0x7faf0c0b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 --2- 172.21.15.43:0/2507058832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf0c0d0fa0 0x7faf0c159b80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 >> 172.21.15.43:0/2507058832 conn(0x7faf0c0d30a0 msgr2=0x7faf0c0c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 shutdown_connections 2024-01-16T15:04:25.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.545+0000 7faf12000640 1 -- 172.21.15.43:0/2507058832 wait complete. 2024-01-16T15:04:25.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone24923 2024-01-16T15:04:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.862+0000 7f6c8a200640 1 Processor -- start 2024-01-16T15:04:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.862+0000 7f6c8a200640 1 -- start start 2024-01-16T15:04:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.864+0000 7f6c8a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.864+0000 7f6c8a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c840abe80 0x7f6c840b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.864+0000 7f6c8a200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840b1ea0 con 0x7f6c8406e040 2024-01-16T15:04:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.864+0000 7f6c8a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840b2740 con 0x7f6c840abe80 2024-01-16T15:04:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.864+0000 7f6c8a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84070db0 con 0x7f6c84070490 2024-01-16T15:04:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.866+0000 7f6c83400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.866+0000 7f6c83400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:45448/0 (socket says 172.21.15.43:45448) 2024-01-16T15:04:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.866+0000 7f6c83400640 1 -- 172.21.15.43:0/1905949602 learned_addr learned my addr 172.21.15.43:0/1905949602 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.866+0000 7f6c82a00640 1 --2- 172.21.15.43:0/1905949602 >> v2:172.21.15.69:3300/0 conn(0x7f6c840abe80 0x7f6c840b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.867+0000 7f6c83400640 1 -- 172.21.15.43:0/1905949602 >> v2:172.21.15.69:3300/0 conn(0x7f6c840abe80 msgr2=0x7f6c840b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.867+0000 7f6c83400640 1 --2- 172.21.15.43:0/1905949602 >> v2:172.21.15.69:3300/0 conn(0x7f6c840abe80 0x7f6c840b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.867+0000 7f6c83400640 1 -- 172.21.15.43:0/1905949602 >> v1:172.21.15.69:6789/0 conn(0x7f6c8406e040 legacy=0x7f6c840b3110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:04:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.867+0000 7f6c83400640 1 -- 172.21.15.43:0/1905949602 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8406d520 con 0x7f6c84070490 2024-01-16T15:04:25.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.868+0000 7f6c83400640 1 --2- 172.21.15.43:0/1905949602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcff9a726def123f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.869+0000 7f6c82000640 1 -- 172.21.15.43:0/1905949602 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74031870 con 0x7f6c84070490 2024-01-16T15:04:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.869+0000 7f6c82000640 1 -- 172.21.15.43:0/1905949602 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c740319d0 con 0x7f6c84070490 2024-01-16T15:04:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.869+0000 7f6c82000640 1 -- 172.21.15.43:0/1905949602 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74031d40 con 0x7f6c84070490 2024-01-16T15:04:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 -- 172.21.15.43:0/1905949602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 msgr2=0x7f6c84070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 --2- 172.21.15.43:0/1905949602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402b910 comp rx=0 tx=0).stop 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 -- 172.21.15.43:0/1905949602 shutdown_connections 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 --2- 172.21.15.43:0/1905949602 >> v2:172.21.15.69:3300/0 conn(0x7f6c840abe80 0x7f6c840b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 --2- 172.21.15.43:0/1905949602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c84070870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 -- 172.21.15.43:0/1905949602 >> 172.21.15.43:0/1905949602 conn(0x7f6c840b0ba0 msgr2=0x7f6c840b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 -- 172.21.15.43:0/1905949602 shutdown_connections 2024-01-16T15:04:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.872+0000 7f6c8a200640 1 -- 172.21.15.43:0/1905949602 wait complete. 2024-01-16T15:04:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.874+0000 7f6c8a200640 1 Processor -- start 2024-01-16T15:04:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.875+0000 7f6c8a200640 1 -- start start 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 0x7f6c8414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c841566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c840abe80 0x7f6c84156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c841588d0 con 0x7f6c8406e040 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84158bb0 con 0x7f6c840abe80 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.876+0000 7f6c8a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84158e90 con 0x7f6c84070490 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c83400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 0x7f6c8414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c840abe80 0x7f6c84156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c83400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 0x7f6c8414ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34462/0 (socket says 172.21.15.43:34462) 2024-01-16T15:04:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c83400640 1 -- 172.21.15.43:0/2389837648 learned_addr learned my addr 172.21.15.43:0/2389837648 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c841566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 -- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c840abe80 msgr2=0x7f6c84156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c840abe80 0x7f6c84156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 -- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 msgr2=0x7f6c8414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 0x7f6c8414ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.877+0000 7f6c82a00640 1 -- 172.21.15.43:0/2389837648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740094d0 con 0x7f6c84070490 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c82a00640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c841566e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00cc00 tx=0x7f6c7c00a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00ce10 con 0x7f6c84070490 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c003990 con 0x7f6c84070490 2024-01-16T15:04:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c003d40 con 0x7f6c84070490 2024-01-16T15:04:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c84159170 con 0x7f6c84070490 2024-01-16T15:04:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.878+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8414b1d0 con 0x7f6c84070490 2024-01-16T15:04:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.881+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c54005250 con 0x7f6c84070490 2024-01-16T15:04:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.882+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f6c7c004030 con 0x7f6c84070490 2024-01-16T15:04:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.883+0000 7f6c7b400640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 0x7f6c5c06de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.883+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6c7c08aa30 con 0x7f6c84070490 2024-01-16T15:04:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.885+0000 7f6c83400640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 0x7f6c5c06de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.887+0000 7f6c83400640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 0x7f6c5c06de00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6c74034040 tx=0x7f6c7403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:25.893+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c7c054f60 con 0x7f6c84070490 2024-01-16T15:04:26.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.188+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone24923", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c54000bc0 con 0x7f6c5c06b940 2024-01-16T15:04:26.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.223+0000 7f6c7b400640 1 -- 172.21.15.43:0/2389837648 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6c54000bc0 con 0x7f6c5c06b940 2024-01-16T15:04:26.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 msgr2=0x7f6c5c06de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 0x7f6c5c06de00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6c74034040 tx=0x7f6c7403c040 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 msgr2=0x7f6c841566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c841566e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00cc00 tx=0x7f6c7c00a5c0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 shutdown_connections 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c5c06b940 0x7f6c5c06de00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c840abe80 0x7f6c84156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c84070490 0x7f6c841566e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 --2- 172.21.15.43:0/2389837648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c8406e040 0x7f6c8414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 >> 172.21.15.43:0/2389837648 conn(0x7f6c840b0ba0 msgr2=0x7f6c840b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 shutdown_connections 2024-01-16T15:04:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.226+0000 7f6c8a200640 1 -- 172.21.15.43:0/2389837648 wait complete. 2024-01-16T15:04:26.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:26.227 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:04:26.284 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:04:26.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:04:26.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.495+0000 7f8f03e00640 1 Processor -- start 2024-01-16T15:04:26.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.495+0000 7f8f03e00640 1 -- start start 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f041435d0 0x7f8f04071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f041439b0 0x7f8f040b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f04070d40 con 0x7f8f040b3f70 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f040b43e0 con 0x7f8f041435d0 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f04071020 con 0x7f8f041439b0 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f02a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8f041435d0 0x7f8f04071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8f040b3f70 0x7f8f04070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40300/0 (socket says 172.21.15.43:40300) 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.497+0000 7f8f03400640 1 -- 172.21.15.43:0/16495541 learned_addr learned my addr 172.21.15.43:0/16495541 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f02000640 1 --2- 172.21.15.43:0/16495541 >> v2:172.21.15.69:3300/0 conn(0x7f8f041439b0 0x7f8f040b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f02a00640 1 -- 172.21.15.43:0/16495541 >> v2:172.21.15.69:3300/0 conn(0x7f8f041439b0 msgr2=0x7f8f040b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f02a00640 1 --2- 172.21.15.43:0/16495541 >> v2:172.21.15.69:3300/0 conn(0x7f8f041439b0 0x7f8f040b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f02a00640 1 -- 172.21.15.43:0/16495541 >> v1:172.21.15.43:6789/0 conn(0x7f8f040b3f70 legacy=0x7f8f04070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f02a00640 1 -- 172.21.15.43:0/16495541 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f04071300 con 0x7f8f041435d0 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.498+0000 7f8f01600640 1 -- 172.21.15.43:0/16495541 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743235018 0 0) 0x7f8f04070d40 con 0x7f8f040b3f70 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.499+0000 7f8f02a00640 1 --2- 172.21.15.43:0/16495541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f04071de0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8f040cbe80 tx=0x7f8ef402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c505bc54e30b0882 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.499+0000 7f8f01600640 1 -- 172.21.15.43:0/16495541 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef402e070 con 0x7f8f041435d0 2024-01-16T15:04:26.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.499+0000 7f8f01600640 1 -- 172.21.15.43:0/16495541 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef40315a0 con 0x7f8f041435d0 2024-01-16T15:04:26.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.499+0000 7f8f01600640 1 -- 172.21.15.43:0/16495541 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4031930 con 0x7f8f041435d0 2024-01-16T15:04:26.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 -- 172.21.15.43:0/16495541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 msgr2=0x7f8f04071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 --2- 172.21.15.43:0/16495541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f04071de0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8f040cbe80 tx=0x7f8ef402b840 comp rx=0 tx=0).stop 2024-01-16T15:04:26.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 -- 172.21.15.43:0/16495541 shutdown_connections 2024-01-16T15:04:26.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 --2- 172.21.15.43:0/16495541 >> v2:172.21.15.69:3300/0 conn(0x7f8f041439b0 0x7f8f040b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 --2- 172.21.15.43:0/16495541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f04071de0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 -- 172.21.15.43:0/16495541 >> 172.21.15.43:0/16495541 conn(0x7f8f040cec10 msgr2=0x7f8f040cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 -- 172.21.15.43:0/16495541 shutdown_connections 2024-01-16T15:04:26.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.500+0000 7f8f03e00640 1 -- 172.21.15.43:0/16495541 wait complete. 2024-01-16T15:04:26.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.502+0000 7f8f03e00640 1 Processor -- start 2024-01-16T15:04:26.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.502+0000 7f8f03e00640 1 -- start start 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f040b3f70 0x7f8f041f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 0x7f8f0414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040cc710 con 0x7f8f041435d0 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f041f2960 con 0x7f8f041439b0 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f03e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040c80e0 con 0x7f8f040b3f70 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f02000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f02000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34480/0 (socket says 172.21.15.43:34480) 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.503+0000 7f8f02000640 1 -- 172.21.15.43:0/4281248897 learned_addr learned my addr 172.21.15.43:0/4281248897 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f03400640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 0x7f8f0414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02a00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f040b3f70 0x7f8f041f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 -- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f040b3f70 msgr2=0x7f8f041f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f040b3f70 0x7f8f041f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 -- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 msgr2=0x7f8f0414f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 0x7f8f0414f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 -- 172.21.15.43:0/4281248897 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4009400 con 0x7f8f041435d0 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f03400640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 0x7f8f0414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f02000640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c7c0 tx=0x7f8eec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007820 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec007980 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f040c8380 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.504+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007cf0 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.505+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f040cac90 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.506+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0404e840 con 0x7f8f041435d0 2024-01-16T15:04:26.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.507+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f8eec0113f0 con 0x7f8f041435d0 2024-01-16T15:04:26.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.508+0000 7f8efaa00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 0x7f8edc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.508+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8eec08a5b0 con 0x7f8f041435d0 2024-01-16T15:04:26.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.508+0000 7f8f02a00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 0x7f8edc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.509+0000 7f8f02a00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 0x7f8edc06ddb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4036000 tx=0x7f8ef403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.512+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8eec054920 con 0x7f8f041435d0 2024-01-16T15:04:26.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:26.726+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f040718c0 con 0x7f8edc06b8f0 2024-01-16T15:04:26.747 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:04:26.767+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:04:26.844 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:04:27.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:27.869+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8eec054cf0 con 0x7f8f041435d0 2024-01-16T15:04:29.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.798+0000 7f8efaa00640 1 -- 172.21.15.43:0/4281248897 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f8ef4031b10 con 0x7f8edc06b8f0 2024-01-16T15:04:29.777 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:04:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.802+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 msgr2=0x7f8edc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.802+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 0x7f8edc06ddb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4036000 tx=0x7f8ef403d040 comp rx=0 tx=0).stop 2024-01-16T15:04:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.802+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 msgr2=0x7f8f040ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.802+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c7c0 tx=0x7f8eec00cc80 comp rx=0 tx=0).stop 2024-01-16T15:04:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 shutdown_connections 2024-01-16T15:04:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8edc06b8f0 0x7f8edc06ddb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f041439b0 0x7f8f0414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f041435d0 0x7f8f040ca750 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 --2- 172.21.15.43:0/4281248897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f040b3f70 0x7f8f041f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.803+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 >> 172.21.15.43:0/4281248897 conn(0x7f8f040cec10 msgr2=0x7f8f040aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.805+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 shutdown_connections 2024-01-16T15:04:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:29.805+0000 7f8f03e00640 1 -- 172.21.15.43:0/4281248897 wait complete. 2024-01-16T15:04:29.815 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:04:29.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:29.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:30.109 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:04:30.109 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:04:30.110 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:30.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:30.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.401+0000 7feb88c00640 1 Processor -- start 2024-01-16T15:04:30.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.401+0000 7feb88c00640 1 -- start start 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.402+0000 7feb88c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb8415a100 0x7feb840b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.402+0000 7feb88c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feb840abff0 0x7feb840b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.402+0000 7feb88c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8413da30 con 0x7feb840ae7e0 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.402+0000 7feb88c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840ac840 con 0x7feb8415a100 2024-01-16T15:04:30.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.402+0000 7feb88c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840ac560 con 0x7feb840abff0 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb8415a100 0x7feb840b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feb8415a100 0x7feb840b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34490/0 (socket says 172.21.15.43:34490) 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 -- 172.21.15.43:0/258483211 learned_addr learned my addr 172.21.15.43:0/258483211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 -- 172.21.15.43:0/258483211 >> v2:172.21.15.69:3300/0 conn(0x7feb840abff0 msgr2=0x7feb840b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 --2- 172.21.15.43:0/258483211 >> v2:172.21.15.69:3300/0 conn(0x7feb840abff0 0x7feb840b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 -- 172.21.15.43:0/258483211 >> v1:172.21.15.69:6789/0 conn(0x7feb840ae7e0 legacy=0x7feb8406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.403+0000 7feb81e00640 1 -- 172.21.15.43:0/258483211 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84070650 con 0x7feb8415a100 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.404+0000 7feb81e00640 1 --2- 172.21.15.43:0/258483211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb8415a100 0x7feb840b3290 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb8406fe60 tx=0x7feb7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf627a3e5708b6c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.404+0000 7feb81400640 1 -- 172.21.15.43:0/258483211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74031870 con 0x7feb8415a100 2024-01-16T15:04:30.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.404+0000 7feb81400640 1 -- 172.21.15.43:0/258483211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb740319d0 con 0x7feb8415a100 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.404+0000 7feb81400640 1 -- 172.21.15.43:0/258483211 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74031d60 con 0x7feb8415a100 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 -- 172.21.15.43:0/258483211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb8415a100 msgr2=0x7feb840b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 --2- 172.21.15.43:0/258483211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb8415a100 0x7feb840b3290 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb8406fe60 tx=0x7feb7402b910 comp rx=0 tx=0).stop 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 -- 172.21.15.43:0/258483211 shutdown_connections 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 --2- 172.21.15.43:0/258483211 >> v2:172.21.15.69:3300/0 conn(0x7feb840abff0 0x7feb840b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 --2- 172.21.15.43:0/258483211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb8415a100 0x7feb840b3290 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 -- 172.21.15.43:0/258483211 >> 172.21.15.43:0/258483211 conn(0x7feb840b0d80 msgr2=0x7feb840b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 -- 172.21.15.43:0/258483211 shutdown_connections 2024-01-16T15:04:30.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.405+0000 7feb88c00640 1 -- 172.21.15.43:0/258483211 wait complete. 2024-01-16T15:04:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.407+0000 7feb88c00640 1 Processor -- start 2024-01-16T15:04:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.407+0000 7feb88c00640 1 -- start start 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.408+0000 7feb88c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb88c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb840ae7e0 0x7feb840989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb88c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb8415a100 0x7feb840afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb88c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84098f20 con 0x7feb840abff0 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb88c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8415b310 con 0x7feb8415a100 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb88c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8415a700 con 0x7feb840ae7e0 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34506/0 (socket says 172.21.15.43:34506) 2024-01-16T15:04:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 -- 172.21.15.43:0/1423450498 learned_addr learned my addr 172.21.15.43:0/1423450498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 -- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb840ae7e0 msgr2=0x7feb840989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb7be00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb840ae7e0 0x7feb840989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb840ae7e0 0x7feb840989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 -- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb8415a100 msgr2=0x7feb840afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb8415a100 0x7feb840afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.409+0000 7feb81e00640 1 -- 172.21.15.43:0/1423450498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740094d0 con 0x7feb840abff0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb81e00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7feb74000c00 tx=0x7feb740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7403adf0 con 0x7feb840abff0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb740443f0 con 0x7feb840abff0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb740447a0 con 0x7feb840abff0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8415a9e0 con 0x7feb840abff0 2024-01-16T15:04:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.410+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8415afa0 con 0x7feb840abff0 2024-01-16T15:04:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.411+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb84067960 con 0x7feb840abff0 2024-01-16T15:04:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.412+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7feb7404d450 con 0x7feb840abff0 2024-01-16T15:04:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.413+0000 7feb7a000640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 0x7feb5c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.413+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7feb7402d040 con 0x7feb840abff0 2024-01-16T15:04:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.413+0000 7feb7be00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 0x7feb5c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.414+0000 7feb7be00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 0x7feb5c06dd20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feb8406d5d0 tx=0x7feb6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.418+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb74077360 con 0x7feb840abff0 2024-01-16T15:04:30.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.608+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb8406d620 con 0x7feb840abff0 2024-01-16T15:04:30.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.610+0000 7feb7a000640 1 -- 172.21.15.43:0/1423450498 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12981 (secure 0 0 0) 0x7feb7407a980 con 0x7feb840abff0 2024-01-16T15:04:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:30.590 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":53,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:29.783590+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6836/3947724910":"2024-01-17T15:04:26.768522+0000","172.21.15.43:6834/3947724910":"2024-01-17T15:04:26.768522+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 msgr2=0x7feb5c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 0x7feb5c06dd20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feb8406d5d0 tx=0x7feb6c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 msgr2=0x7feb8415b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7feb74000c00 tx=0x7feb740046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 shutdown_connections 2024-01-16T15:04:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feb5c06b860 0x7feb5c06dd20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb8415a100 0x7feb840afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feb840ae7e0 0x7feb840989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 --2- 172.21.15.43:0/1423450498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb840abff0 0x7feb8415b7a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 >> 172.21.15.43:0/1423450498 conn(0x7feb840b0d80 msgr2=0x7feb840aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 shutdown_connections 2024-01-16T15:04:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.613+0000 7feb88c00640 1 -- 172.21.15.43:0/1423450498 wait complete. 2024-01-16T15:04:30.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:30.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:30.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.851+0000 7f1bd3400640 1 Processor -- start 2024-01-16T15:04:30.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.851+0000 7f1bd3400640 1 -- start start 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1bcc15b350 0x7f1bcc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc09cb30 0x7f1bcc0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc15b730 con 0x7f1bcc0adb60 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc0ac740 con 0x7f1bcc15b350 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0ac460 con 0x7f1bcc09cb30 2024-01-16T15:04:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1bcc15b350 0x7f1bcc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1bcc15b350 0x7f1bcc0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34522/0 (socket says 172.21.15.43:34522) 2024-01-16T15:04:30.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.853+0000 7f1bd0a00640 1 -- 172.21.15.43:0/4027365866 learned_addr learned my addr 172.21.15.43:0/4027365866 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:30.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.854+0000 7f1bcbe00640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc09cb30 0x7f1bcc0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.854+0000 7f1bd0a00640 1 -- 172.21.15.43:0/4027365866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc09cb30 msgr2=0x7f1bcc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.854+0000 7f1bd0a00640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc09cb30 0x7f1bcc0b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.854+0000 7f1bd0a00640 1 -- 172.21.15.43:0/4027365866 >> v1:172.21.15.69:6789/0 conn(0x7f1bcc0adb60 legacy=0x7f1bcc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.854+0000 7f1bd0a00640 1 -- 172.21.15.43:0/4027365866 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0b2a50 con 0x7f1bcc15b350 2024-01-16T15:04:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.855+0000 7f1bd0a00640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc15b350 0x7f1bcc0abe80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009870 tx=0x7f1bbc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7229bef85d060f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.855+0000 7f1bcb400640 1 -- 172.21.15.43:0/4027365866 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc031640 con 0x7f1bcc15b350 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.855+0000 7f1bcb400640 1 -- 172.21.15.43:0/4027365866 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0317a0 con 0x7f1bcc15b350 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.855+0000 7f1bcb400640 1 -- 172.21.15.43:0/4027365866 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc031b30 con 0x7f1bcc15b350 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 -- 172.21.15.43:0/4027365866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc15b350 msgr2=0x7f1bcc0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc15b350 0x7f1bcc0abe80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009870 tx=0x7f1bbc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 -- 172.21.15.43:0/4027365866 shutdown_connections 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc09cb30 0x7f1bcc0b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 --2- 172.21.15.43:0/4027365866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc15b350 0x7f1bcc0abe80 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 -- 172.21.15.43:0/4027365866 >> 172.21.15.43:0/4027365866 conn(0x7f1bcc0b0ba0 msgr2=0x7f1bcc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 -- 172.21.15.43:0/4027365866 shutdown_connections 2024-01-16T15:04:30.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.856+0000 7f1bd3400640 1 -- 172.21.15.43:0/4027365866 wait complete. 2024-01-16T15:04:30.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.857+0000 7f1bd3400640 1 Processor -- start 2024-01-16T15:04:30.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.858+0000 7f1bd3400640 1 -- start start 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.858+0000 7f1bd3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bcc09cb30 0x7f1bcc146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.858+0000 7f1bd3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc15b350 0x7f1bcc1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc1472b0 con 0x7f1bcc0adb60 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc1f6810 con 0x7f1bcc09cb30 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc14e240 con 0x7f1bcc15b350 2024-01-16T15:04:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34538/0 (socket says 172.21.15.43:34538) 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 -- 172.21.15.43:0/3241396890 learned_addr learned my addr 172.21.15.43:0/3241396890 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd1400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc15b350 0x7f1bcc1f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bd0a00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bcc09cb30 0x7f1bcc146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 -- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc15b350 msgr2=0x7f1bcc1f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc15b350 0x7f1bcc1f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 -- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bcc09cb30 msgr2=0x7f1bcc146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bcc09cb30 0x7f1bcc146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.859+0000 7f1bcbe00640 1 -- 172.21.15.43:0/3241396890 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0094d0 con 0x7f1bcc0adb60 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bcbe00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c7f0 tx=0x7f1bb400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4004220 con 0x7f1bcc0adb60 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb40043a0 con 0x7f1bcc0adb60 2024-01-16T15:04:30.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4007860 con 0x7f1bcc0adb60 2024-01-16T15:04:30.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc14e520 con 0x7f1bcc0adb60 2024-01-16T15:04:30.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.860+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc14e800 con 0x7f1bcc0adb60 2024-01-16T15:04:30.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.861+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc09d200 con 0x7f1bcc0adb60 2024-01-16T15:04:30.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.862+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1bb4007bd0 con 0x7f1bcc0adb60 2024-01-16T15:04:30.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.863+0000 7f1bc8c00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 0x7f1ba406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:30.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.863+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1bb408a390 con 0x7f1bcc0adb60 2024-01-16T15:04:30.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.863+0000 7f1bd0a00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 0x7f1ba406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:30.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.864+0000 7f1bd0a00640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 0x7f1ba406dd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc036000 tx=0x7f1bbc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:30.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:30.868+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bb4054c70 con 0x7f1bcc0adb60 2024-01-16T15:04:31.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.055+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1bcc1473f0 con 0x7f1bcc0adb60 2024-01-16T15:04:31.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.056+0000 7f1bc8c00640 1 -- 172.21.15.43:0/3241396890 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1bb4058290 con 0x7f1bcc0adb60 2024-01-16T15:04:31.035 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 msgr2=0x7f1ba406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 0x7f1ba406dd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc036000 tx=0x7f1bbc03c040 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 msgr2=0x7f1bcc14eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c7f0 tx=0x7f1bb400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 shutdown_connections 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ba406b860 0x7f1ba406dd20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bcc15b350 0x7f1bcc1f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bcc0adb60 0x7f1bcc14eaa0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 --2- 172.21.15.43:0/3241396890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bcc09cb30 0x7f1bcc146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 >> 172.21.15.43:0/3241396890 conn(0x7f1bcc0b0ba0 msgr2=0x7f1bcc070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 shutdown_connections 2024-01-16T15:04:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.059+0000 7f1bd3400640 1 -- 172.21.15.43:0/3241396890 wait complete. 2024-01-16T15:04:31.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:31.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.296+0000 7fc815c00640 1 Processor -- start 2024-01-16T15:04:31.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.296+0000 7fc815c00640 1 -- start start 2024-01-16T15:04:31.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc815c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc81006cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc815c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc810141e80 0x7fc81006d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc815c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc810070bf0 con 0x7fc8101416f0 2024-01-16T15:04:31.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc815c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006db30 con 0x7fc810141e80 2024-01-16T15:04:31.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc815c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006d5a0 con 0x7fc81015f460 2024-01-16T15:04:31.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc80fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc8101416f0 0x7fc81015ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40318/0 (socket says 172.21.15.43:40318) 2024-01-16T15:04:31.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.297+0000 7fc80fe00640 1 -- 172.21.15.43:0/2517923842 learned_addr learned my addr 172.21.15.43:0/2517923842 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:31.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350087626 0 0) 0x7fc810070bf0 con 0x7fc8101416f0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc815200640 1 --2- 172.21.15.43:0/2517923842 >> v2:172.21.15.69:3300/0 conn(0x7fc810141e80 0x7fc81006d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0003520 con 0x7fc8101416f0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc81006cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213746433 0 0) 0x7fc7f0003520 con 0x7fc8101416f0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 msgr2=0x7fc81006cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc81006cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 >> v2:172.21.15.69:3300/0 conn(0x7fc810141e80 msgr2=0x7fc81006d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 --2- 172.21.15.43:0/2517923842 >> v2:172.21.15.69:3300/0 conn(0x7fc810141e80 0x7fc81006d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81006d850 con 0x7fc8101416f0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc800001670 con 0x7fc8101416f0 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc81006cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.298+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 >> v1:172.21.15.43:6789/0 conn(0x7fc8101416f0 legacy=0x7fc81015ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc7f00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc7f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f0004d00 0x7fc7f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006db30 con 0x7fc7f0003f20 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006d5a0 con 0x7fc7f0004d00 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f000ba00 con 0x7fc81015f460 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc7f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 -- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 msgr2=0x7fc7f00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80fe00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc7f00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc7f00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 -- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f0004d00 msgr2=0x7fc7f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f0004d00 0x7fc7f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80f400640 1 -- 172.21.15.43:0/2517923842 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0003520 con 0x7fc7f0003f20 2024-01-16T15:04:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.299+0000 7fc80fe00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc81015f460 0x7fc7f00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:31.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.300+0000 7fc80f400640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc7f00047c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098a0 tx=0x7fc7f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c900356736ef7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:31.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8004230 con 0x7fc7f0003f20 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f8004390 con 0x7fc7f0003f20 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc80ea00640 1 -- 172.21.15.43:0/2517923842 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8003830 con 0x7fc7f0003f20 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc815c00640 1 -- 172.21.15.43:0/2517923842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 msgr2=0x7fc7f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc815c00640 1 --2- 172.21.15.43:0/2517923842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc7f00047c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098a0 tx=0x7fc7f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc815200640 1 -- 172.21.15.43:0/2517923842 reap_dead start 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc815c00640 1 -- 172.21.15.43:0/2517923842 shutdown_connections 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.301+0000 7fc815c00640 1 -- 172.21.15.43:0/2517923842 >> 172.21.15.43:0/2517923842 conn(0x7fc8100b0840 msgr2=0x7fc8100b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.302+0000 7fc815c00640 1 -- 172.21.15.43:0/2517923842 shutdown_connections 2024-01-16T15:04:31.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.302+0000 7fc815c00640 1 -- 172.21.15.43:0/2517923842 wait complete. 2024-01-16T15:04:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.303+0000 7fc815c00640 1 Processor -- start 2024-01-16T15:04:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.304+0000 7fc815c00640 1 -- start start 2024-01-16T15:04:31.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 0x7fc81014aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8100a8460 0x7fc8100a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81014be90 con 0x7fc7f0003f20 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81014af80 con 0x7fc81014a5c0 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81014b210 con 0x7fc8100a8460 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34552/0 (socket says 172.21.15.43:34552) 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 -- 172.21.15.43:0/1353236763 learned_addr learned my addr 172.21.15.43:0/1353236763 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc815200640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8100a8460 0x7fc8100a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80f400640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 0x7fc81014aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 -- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8100a8460 msgr2=0x7fc8100a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8100a8460 0x7fc8100a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 -- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 msgr2=0x7fc81014aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 0x7fc81014aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.305+0000 7fc80fe00640 1 -- 172.21.15.43:0/1353236763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81014b4f0 con 0x7fc7f0003f20 2024-01-16T15:04:31.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc80f400640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 0x7fc81014aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc80fe00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc80000d810 tx=0x7fc80000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800007560 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000047e0 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800004b90 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81014b7d0 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.306+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81014c2b0 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.308+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100cc990 con 0x7fc7f0003f20 2024-01-16T15:04:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.308+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc800004cf0 con 0x7fc7f0003f20 2024-01-16T15:04:31.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.309+0000 7fc807e00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 0x7fc7e806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.309+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc800089c90 con 0x7fc7f0003f20 2024-01-16T15:04:31.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.309+0000 7fc80f400640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 0x7fc7e806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.310+0000 7fc80f400640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 0x7fc7e806dd20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8035000 tx=0x7fc7f803a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:31.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.314+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc80004de90 con 0x7fc7f0003f20 2024-01-16T15:04:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.522+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc81004e840 con 0x7fc7f0003f20 2024-01-16T15:04:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.523+0000 7fc807e00640 1 -- 172.21.15.43:0/1353236763 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v53) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc800019080 con 0x7fc7f0003f20 2024-01-16T15:04:31.502 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3947724910 2024-01-17T15:04:26.768522+0000 2024-01-16T15:04:31.502 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6834/3947724910 2024-01-17T15:04:26.768522+0000 2024-01-16T15:04:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.525+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 msgr2=0x7fc7e806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.525+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 0x7fc7e806dd20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8035000 tx=0x7fc7f803a040 comp rx=0 tx=0).stop 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.525+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 msgr2=0x7fc8100a9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.525+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc80000d810 tx=0x7fc80000dcd0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 shutdown_connections 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7e806b860 0x7fc7e806dd20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc81014a5c0 0x7fc81014aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8100a8460 0x7fc8100a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 --2- 172.21.15.43:0/1353236763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f0003f20 0x7fc8100a9710 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 >> 172.21.15.43:0/1353236763 conn(0x7fc8100b0840 msgr2=0x7fc810144330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 shutdown_connections 2024-01-16T15:04:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:31.526+0000 7fc815c00640 1 -- 172.21.15.43:0/1353236763 wait complete. 2024-01-16T15:04:31.525 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:04:31.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:31.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:31.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[52388]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:31.625 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:04:31.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:31.625 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:04:31.846 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:04:31.847 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:31.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:32.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.157+0000 7f8a40800640 1 Processor -- start 2024-01-16T15:04:32.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.157+0000 7f8a40800640 1 -- start start 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a40800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a38160970 0x7f8a380b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a40800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a38071cd0 con 0x7f8a3806dc30 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a40800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a380b2120 con 0x7f8a380b3140 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a40800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a380b2400 con 0x7f8a38160970 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a3de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a38160970 0x7f8a380b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a3e800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8a3806dc30 0x7f8a380acbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40324/0 (socket says 172.21.15.43:40324) 2024-01-16T15:04:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.159+0000 7f8a3e800640 1 -- 172.21.15.43:0/1901254006 learned_addr learned my addr 172.21.15.43:0/1901254006 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:32.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.160+0000 7f8a3de00640 1 -- 172.21.15.43:0/1901254006 >> v1:172.21.15.69:6789/0 conn(0x7f8a380b3140 legacy=0x7f8a3806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.160+0000 7f8a3de00640 1 -- 172.21.15.43:0/1901254006 >> v1:172.21.15.43:6789/0 conn(0x7f8a3806dc30 legacy=0x7f8a380acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.160+0000 7f8a3de00640 1 -- 172.21.15.43:0/1901254006 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a380b26e0 con 0x7f8a38160970 2024-01-16T15:04:32.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.160+0000 7f8a3ca00640 1 -- 172.21.15.43:0/1901254006 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293748617 0 0) 0x7f8a38071cd0 con 0x7f8a3806dc30 2024-01-16T15:04:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.161+0000 7f8a3de00640 1 --2- 172.21.15.43:0/1901254006 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a38160970 0x7f8a380b2c00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097a0 tx=0x7f8a2802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf9f73ce9c66011 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.161+0000 7f8a3ca00640 1 -- 172.21.15.43:0/1901254006 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2802e070 con 0x7f8a38160970 2024-01-16T15:04:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.161+0000 7f8a3ca00640 1 -- 172.21.15.43:0/1901254006 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a280315a0 con 0x7f8a38160970 2024-01-16T15:04:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.161+0000 7f8a3ca00640 1 -- 172.21.15.43:0/1901254006 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28031930 con 0x7f8a38160970 2024-01-16T15:04:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 -- 172.21.15.43:0/1901254006 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a38160970 msgr2=0x7f8a380b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 --2- 172.21.15.43:0/1901254006 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a38160970 0x7f8a380b2c00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097a0 tx=0x7f8a2802b840 comp rx=0 tx=0).stop 2024-01-16T15:04:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 -- 172.21.15.43:0/1901254006 shutdown_connections 2024-01-16T15:04:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 --2- 172.21.15.43:0/1901254006 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a38160970 0x7f8a380b2c00 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 -- 172.21.15.43:0/1901254006 >> 172.21.15.43:0/1901254006 conn(0x7f8a380b0d80 msgr2=0x7f8a380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 -- 172.21.15.43:0/1901254006 shutdown_connections 2024-01-16T15:04:32.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.162+0000 7f8a40800640 1 -- 172.21.15.43:0/1901254006 wait complete. 2024-01-16T15:04:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.164+0000 7f8a40800640 1 Processor -- start 2024-01-16T15:04:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.164+0000 7f8a40800640 1 -- start start 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a380b3140 0x7f8a381533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a38160970 0x7f8a38157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3809cdf0 con 0x7f8a3806dc30 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3809d0d0 con 0x7f8a38160970 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a40800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3809ac30 con 0x7f8a380b3140 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.165+0000 7f8a3de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34572/0 (socket says 172.21.15.43:34572) 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 -- 172.21.15.43:0/2624599572 learned_addr learned my addr 172.21.15.43:0/2624599572 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3e800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a38160970 0x7f8a38157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3d400640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a380b3140 0x7f8a381533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 -- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a380b3140 msgr2=0x7f8a381533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a380b3140 0x7f8a381533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 -- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a38160970 msgr2=0x7f8a38157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a38160970 0x7f8a38157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 -- 172.21.15.43:0/2624599572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28009400 con 0x7f8a3806dc30 2024-01-16T15:04:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.166+0000 7f8a3de00640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097a0 tx=0x7f8a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.167+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28032840 con 0x7f8a3806dc30 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.167+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a280329a0 con 0x7f8a3806dc30 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.167+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28032d50 con 0x7f8a3806dc30 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.167+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3809af10 con 0x7f8a3806dc30 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.167+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a381f7530 con 0x7f8a3806dc30 2024-01-16T15:04:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.168+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a38074770 con 0x7f8a3806dc30 2024-01-16T15:04:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.169+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8a2803b3f0 con 0x7f8a3806dc30 2024-01-16T15:04:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.170+0000 7f8a36000640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.170+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8a280ac5d0 con 0x7f8a3806dc30 2024-01-16T15:04:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.170+0000 7f8a3d400640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.171+0000 7f8a3d400640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 0x7f8a1006dd20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ade90 tx=0x7f8a20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.174+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a28076d60 con 0x7f8a3806dc30 2024-01-16T15:04:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.374+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a3806cd00 con 0x7f8a3806dc30 2024-01-16T15:04:32.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.376+0000 7f8a36000640 1 -- 172.21.15.43:0/2624599572 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12981 (secure 0 0 0) 0x7f8a2807a380 con 0x7f8a3806dc30 2024-01-16T15:04:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":53,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:29.783590+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6836/3947724910":"2024-01-17T15:04:26.768522+0000","172.21.15.43:6834/3947724910":"2024-01-17T15:04:26.768522+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 msgr2=0x7f8a1006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 0x7f8a1006dd20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ade90 tx=0x7f8a20009040 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 msgr2=0x7f8a381f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097a0 tx=0x7f8a280046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 shutdown_connections 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8a1006b860 0x7f8a1006dd20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a38160970 0x7f8a38157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a380b3140 0x7f8a381533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.379+0000 7f8a40800640 1 --2- 172.21.15.43:0/2624599572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a3806dc30 0x7f8a381f31a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.380+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 >> 172.21.15.43:0/2624599572 conn(0x7f8a380b0d80 msgr2=0x7f8a380af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.380+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 shutdown_connections 2024-01-16T15:04:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.380+0000 7f8a40800640 1 -- 172.21.15.43:0/2624599572 wait complete. 2024-01-16T15:04:32.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:32.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:32.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.608+0000 7f05b7e00640 1 Processor -- start 2024-01-16T15:04:32.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.608+0000 7f05b7e00640 1 -- start start 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.609+0000 7f05b7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05b815d3b0 0x7f05b80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.609+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d1d90 0x7f05b80b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.609+0000 7f05b7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b80c6bf0 con 0x7f05b80d0fa0 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.609+0000 7f05b7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b80d1430 con 0x7f05b815d3b0 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.609+0000 7f05b7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80cedc0 con 0x7f05b80d1d90 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05b815d3b0 0x7f05b80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05b815d3b0 0x7f05b80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34578/0 (socket says 172.21.15.43:34578) 2024-01-16T15:04:32.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 -- 172.21.15.43:0/3570960467 learned_addr learned my addr 172.21.15.43:0/3570960467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:32.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 -- 172.21.15.43:0/3570960467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d1d90 msgr2=0x7f05b80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 --2- 172.21.15.43:0/3570960467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d1d90 0x7f05b80b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 -- 172.21.15.43:0/3570960467 >> v1:172.21.15.69:6789/0 conn(0x7f05b80d0fa0 legacy=0x7f05b80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:32.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.610+0000 7f05b6a00640 1 -- 172.21.15.43:0/3570960467 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b80c6e40 con 0x7f05b815d3b0 2024-01-16T15:04:32.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.611+0000 7f05b6a00640 1 --2- 172.21.15.43:0/3570960467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80ce6c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009870 tx=0x7f05a802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64115608cc0ae6d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b5600640 1 -- 172.21.15.43:0/3570960467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a80317c0 con 0x7f05b815d3b0 2024-01-16T15:04:32.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b5600640 1 -- 172.21.15.43:0/3570960467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8031920 con 0x7f05b815d3b0 2024-01-16T15:04:32.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b5600640 1 -- 172.21.15.43:0/3570960467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8031cb0 con 0x7f05b815d3b0 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 -- 172.21.15.43:0/3570960467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 msgr2=0x7f05b80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 --2- 172.21.15.43:0/3570960467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80ce6c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009870 tx=0x7f05a802b860 comp rx=0 tx=0).stop 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 -- 172.21.15.43:0/3570960467 shutdown_connections 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 --2- 172.21.15.43:0/3570960467 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d1d90 0x7f05b80b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 --2- 172.21.15.43:0/3570960467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80ce6c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.612+0000 7f05b7e00640 1 -- 172.21.15.43:0/3570960467 >> 172.21.15.43:0/3570960467 conn(0x7f05b80d30a0 msgr2=0x7f05b80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.613+0000 7f05b7e00640 1 -- 172.21.15.43:0/3570960467 shutdown_connections 2024-01-16T15:04:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.613+0000 7f05b7e00640 1 -- 172.21.15.43:0/3570960467 wait complete. 2024-01-16T15:04:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.614+0000 7f05b7e00640 1 Processor -- start 2024-01-16T15:04:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 -- start start 2024-01-16T15:04:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d0fa0 0x7f05b815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b80d1d90 0x7f05b81590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b815fe90 con 0x7f05b815d3b0 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b815edf0 con 0x7f05b80d1d90 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.615+0000 7f05b7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b815f0d0 con 0x7f05b80d0fa0 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34580/0 (socket says 172.21.15.43:34580) 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 -- 172.21.15.43:0/409825855 learned_addr learned my addr 172.21.15.43:0/409825855 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b6a00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d0fa0 0x7f05b815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 -- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d0fa0 msgr2=0x7f05b815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d0fa0 0x7f05b815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 -- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b80d1d90 msgr2=0x7f05b81590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b80d1d90 0x7f05b81590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.616+0000 7f05b7400640 1 -- 172.21.15.43:0/409825855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80094d0 con 0x7f05b815d3b0 2024-01-16T15:04:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05b7400640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f05b000b9e0 tx=0x7f05b000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b000cb20 con 0x7f05b815d3b0 2024-01-16T15:04:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b000cc80 con 0x7f05b815d3b0 2024-01-16T15:04:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0014680 con 0x7f05b815d3b0 2024-01-16T15:04:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b815f3b0 con 0x7f05b815d3b0 2024-01-16T15:04:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.617+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b815f970 con 0x7f05b815d3b0 2024-01-16T15:04:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.619+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b815fb60 con 0x7f05b815d3b0 2024-01-16T15:04:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.619+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f05b001d3b0 con 0x7f05b815d3b0 2024-01-16T15:04:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.621+0000 7f05af400640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 0x7f059006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.621+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f05b008a260 con 0x7f05b815d3b0 2024-01-16T15:04:32.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.621+0000 7f05b6a00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 0x7f059006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:32.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.622+0000 7f05b6a00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 0x7f059006dd20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f05a8036000 tx=0x7f05a803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.625+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05b0054b90 con 0x7f05b815d3b0 2024-01-16T15:04:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.822+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05b804e840 con 0x7f05b815d3b0 2024-01-16T15:04:32.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.822+0000 7f05af400640 1 -- 172.21.15.43:0/409825855 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05b0019020 con 0x7f05b815d3b0 2024-01-16T15:04:32.801 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:04:32.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 msgr2=0x7f059006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 0x7f059006dd20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f05a8036000 tx=0x7f05a803d040 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 msgr2=0x7f05b80cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f05b000b9e0 tx=0x7f05b000bea0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 shutdown_connections 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f059006b860 0x7f059006dd20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b80d1d90 0x7f05b81590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b815d3b0 0x7f05b80cb220 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 --2- 172.21.15.43:0/409825855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05b80d0fa0 0x7f05b815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.825+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 >> 172.21.15.43:0/409825855 conn(0x7f05b80d30a0 msgr2=0x7f05b80c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.826+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 shutdown_connections 2024-01-16T15:04:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:32.826+0000 7f05b7e00640 1 -- 172.21.15.43:0/409825855 wait complete. 2024-01-16T15:04:32.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:04:33.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.070+0000 7f715b200640 1 Processor -- start 2024-01-16T15:04:33.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.071+0000 7f715b200640 1 -- start start 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.072+0000 7f715b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7154146aa0 0x7f71540b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.072+0000 7f715b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f715406ab50 0x7f71540ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.072+0000 7f715b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154146e80 con 0x7f71540b40c0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f715b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715406f470 con 0x7f7154146aa0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f715b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71540b36f0 con 0x7f715406ab50 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f7159400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f71540b40c0 0x7f715406a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45134/0 (socket says 172.21.15.43:45134) 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f7159400640 1 -- 172.21.15.43:0/2383657488 learned_addr learned my addr 172.21.15.43:0/2383657488 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f7159e00640 1 --2- 172.21.15.43:0/2383657488 >> v2:172.21.15.43:3300/0 conn(0x7f7154146aa0 0x7f71540b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f715a800640 1 --2- 172.21.15.43:0/2383657488 >> v2:172.21.15.69:3300/0 conn(0x7f715406ab50 0x7f71540ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.073+0000 7f7158a00640 1 -- 172.21.15.43:0/2383657488 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130543785 0 0) 0x7f7154146e80 con 0x7f71540b40c0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.074+0000 7f7158a00640 1 -- 172.21.15.43:0/2383657488 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7134003520 con 0x7f71540b40c0 2024-01-16T15:04:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.074+0000 7f7159e00640 1 -- 172.21.15.43:0/2383657488 >> v2:172.21.15.69:3300/0 conn(0x7f715406ab50 msgr2=0x7f71540ad4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.074+0000 7f7159e00640 1 --2- 172.21.15.43:0/2383657488 >> v2:172.21.15.69:3300/0 conn(0x7f715406ab50 0x7f71540ad4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.074+0000 7f7159e00640 1 -- 172.21.15.43:0/2383657488 >> v1:172.21.15.43:6789/0 conn(0x7f71540b40c0 legacy=0x7f715406a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.074+0000 7f7159e00640 1 -- 172.21.15.43:0/2383657488 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715406f190 con 0x7f7154146aa0 2024-01-16T15:04:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.075+0000 7f7159e00640 1 --2- 172.21.15.43:0/2383657488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540b3b80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f715406fe40 tx=0x7f714402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa94f108a68591c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.075+0000 7f7158a00640 1 -- 172.21.15.43:0/2383657488 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714402e070 con 0x7f7154146aa0 2024-01-16T15:04:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.075+0000 7f7158a00640 1 -- 172.21.15.43:0/2383657488 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71440315a0 con 0x7f7154146aa0 2024-01-16T15:04:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f7158a00640 1 -- 172.21.15.43:0/2383657488 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144031910 con 0x7f7154146aa0 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 -- 172.21.15.43:0/2383657488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 msgr2=0x7f71540b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 --2- 172.21.15.43:0/2383657488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540b3b80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f715406fe40 tx=0x7f714402b840 comp rx=0 tx=0).stop 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 -- 172.21.15.43:0/2383657488 shutdown_connections 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 --2- 172.21.15.43:0/2383657488 >> v2:172.21.15.69:3300/0 conn(0x7f715406ab50 0x7f71540ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 --2- 172.21.15.43:0/2383657488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540b3b80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 -- 172.21.15.43:0/2383657488 >> 172.21.15.43:0/2383657488 conn(0x7f71540712b0 msgr2=0x7f71540716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 -- 172.21.15.43:0/2383657488 shutdown_connections 2024-01-16T15:04:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.076+0000 7f715b200640 1 -- 172.21.15.43:0/2383657488 wait complete. 2024-01-16T15:04:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.078+0000 7f715b200640 1 Processor -- start 2024-01-16T15:04:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.078+0000 7f715b200640 1 -- start start 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f715406ab50 0x7f7154154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71540b40c0 0x7f71540c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71541424c0 con 0x7f7154146aa0 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71540c6340 con 0x7f71540b40c0 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7154142710 con 0x7f715406ab50 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46408/0 (socket says 172.21.15.43:46408) 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.079+0000 7f715a800640 1 -- 172.21.15.43:0/1383771977 learned_addr learned my addr 172.21.15.43:0/1383771977 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f7159400640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71540b40c0 0x7f71540c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f7159e00640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f715406ab50 0x7f7154154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 -- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f715406ab50 msgr2=0x7f7154154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f715406ab50 0x7f7154154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 -- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71540b40c0 msgr2=0x7f71540c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71540b40c0 0x7f71540c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 -- 172.21.15.43:0/1383771977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009400 con 0x7f7154146aa0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f715a800640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f714c00efc0 tx=0x7f714c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.080+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c009ca0 con 0x7f7154146aa0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.081+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c00f040 con 0x7f7154146aa0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.081+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71541429f0 con 0x7f7154146aa0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.081+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0146c0 con 0x7f7154146aa0 2024-01-16T15:04:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.081+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71541553d0 con 0x7f7154146aa0 2024-01-16T15:04:33.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.082+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715404e840 con 0x7f7154146aa0 2024-01-16T15:04:33.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.083+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f714c014820 con 0x7f7154146aa0 2024-01-16T15:04:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.084+0000 7f7152000640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 0x7f712c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.084+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f714c08b3b0 con 0x7f7154146aa0 2024-01-16T15:04:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.084+0000 7f7159e00640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 0x7f712c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.085+0000 7f7159e00640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 0x7f712c06dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7144036000 tx=0x7f7144008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.089+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f714c055b80 con 0x7f7154146aa0 2024-01-16T15:04:33.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.296+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f715406a780 con 0x7f7154146aa0 2024-01-16T15:04:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.297+0000 7f7152000640 1 -- 172.21.15.43:0/1383771977 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v53) v1 ==== 80+0+122 (secure 0 0 0) 0x7f714c0591a0 con 0x7f7154146aa0 2024-01-16T15:04:33.276 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3947724910 2024-01-17T15:04:26.768522+0000 2024-01-16T15:04:33.276 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6834/3947724910 2024-01-17T15:04:26.768522+0000 2024-01-16T15:04:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:04:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 msgr2=0x7f712c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 0x7f712c06dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7144036000 tx=0x7f7144008030 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 msgr2=0x7f71540c6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f714c00efc0 tx=0x7f714c00c490 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 shutdown_connections 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f712c06b860 0x7f712c06dd20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71540b40c0 0x7f71540c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7154146aa0 0x7f71540c6800 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 --2- 172.21.15.43:0/1383771977 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f715406ab50 0x7f7154154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 >> 172.21.15.43:0/1383771977 conn(0x7f71540712b0 msgr2=0x7f71540b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 shutdown_connections 2024-01-16T15:04:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:33.300+0000 7f715b200640 1 -- 172.21.15.43:0/1383771977 wait complete. 2024-01-16T15:04:33.297 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:04:33.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:33.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:33.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[43404]: fuse finished with error 0 and tester_r 0 2024-01-16T15:04:33.397 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:04:33.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:33.397 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:04:33.576 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:04:33.577 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:33.637 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:33.638 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:04:33.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:33.638 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:04:33.695 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:04:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:33.697 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:04:33.698 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:33.757 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:33.757 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:04:33.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:33.757 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:04:33.813 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:04:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:04:33.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:34.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.058+0000 7ff6f7e00640 1 Processor -- start 2024-01-16T15:04:34.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.059+0000 7ff6f7e00640 1 -- start start 2024-01-16T15:04:34.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f80d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff6f80b1650 0x7ff6f80d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80cf0c0 con 0x7ff6f80d14a0 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80b01b0 con 0x7ff6f80b1650 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f815d990 con 0x7ff6f815d5b0 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.060+0000 7ff6f7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff6f80b1650 0x7ff6f80d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff6f80b1650 0x7ff6f80d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46416/0 (socket says 172.21.15.43:46416) 2024-01-16T15:04:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 -- 172.21.15.43:0/3774694899 learned_addr learned my addr 172.21.15.43:0/3774694899 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f6a00640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f80d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 -- 172.21.15.43:0/3774694899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 msgr2=0x7ff6f80d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f80d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 -- 172.21.15.43:0/3774694899 >> v1:172.21.15.69:6789/0 conn(0x7ff6f80d14a0 legacy=0x7ff6f80cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f7400640 1 -- 172.21.15.43:0/3774694899 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80d1e40 con 0x7ff6f80b1650 2024-01-16T15:04:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.061+0000 7ff6f6a00640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f80d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:34.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.062+0000 7ff6f7400640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80b1650 0x7ff6f80d1900 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009e50 tx=0x7ff6f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7a0958780f5cf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f5600640 1 -- 172.21.15.43:0/3774694899 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002cb00 con 0x7ff6f80b1650 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f5600640 1 -- 172.21.15.43:0/3774694899 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f002cc60 con 0x7ff6f80b1650 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f5600640 1 -- 172.21.15.43:0/3774694899 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f00343f0 con 0x7ff6f80b1650 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 -- 172.21.15.43:0/3774694899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80b1650 msgr2=0x7ff6f80d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80b1650 0x7ff6f80d1900 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009e50 tx=0x7ff6f002bf60 comp rx=0 tx=0).stop 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 -- 172.21.15.43:0/3774694899 shutdown_connections 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80b1650 0x7ff6f80d1900 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/3774694899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f80d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.063+0000 7ff6f7e00640 1 -- 172.21.15.43:0/3774694899 >> 172.21.15.43:0/3774694899 conn(0x7ff6f809c3b0 msgr2=0x7ff6f80d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:34.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.064+0000 7ff6f7e00640 1 -- 172.21.15.43:0/3774694899 shutdown_connections 2024-01-16T15:04:34.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.064+0000 7ff6f7e00640 1 -- 172.21.15.43:0/3774694899 wait complete. 2024-01-16T15:04:34.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.065+0000 7ff6f7e00640 1 Processor -- start 2024-01-16T15:04:34.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.066+0000 7ff6f7e00640 1 -- start start 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.066+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6f80b1650 0x7ff6f8160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80cc370 con 0x7ff6f80d14a0 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80c6ba0 con 0x7ff6f80b1650 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80cc560 con 0x7ff6f815d5b0 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46422/0 (socket says 172.21.15.43:46422) 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 -- 172.21.15.43:0/2319341270 learned_addr learned my addr 172.21.15.43:0/2319341270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6a00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6f80b1650 0x7ff6f8160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7400640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 -- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 msgr2=0x7ff6f8152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f8152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 -- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6f80b1650 msgr2=0x7ff6f8160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6f80b1650 0x7ff6f8160570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f6000640 1 -- 172.21.15.43:0/2319341270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009b30 con 0x7ff6f80d14a0 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.067+0000 7ff6f7400640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f8152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6f6000640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c430 tx=0x7ff6e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00052d0 con 0x7ff6f80d14a0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0005430 con 0x7ff6f80d14a0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0003830 con 0x7ff6f80d14a0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f80cc840 con 0x7ff6f80d14a0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.068+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f80c70d0 con 0x7ff6f80d14a0 2024-01-16T15:04:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.069+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f804e840 con 0x7ff6f80d14a0 2024-01-16T15:04:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.070+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff6e0003990 con 0x7ff6f80d14a0 2024-01-16T15:04:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.072+0000 7ff6eea00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 0x7ff6d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.072+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff6e0089870 con 0x7ff6f80d14a0 2024-01-16T15:04:34.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.072+0000 7ff6f6a00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 0x7ff6d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.073+0000 7ff6f6a00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 0x7ff6d006dd20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8006fd0 tx=0x7ff6e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.077+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6e00541a0 con 0x7ff6f80d14a0 2024-01-16T15:04:34.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.313+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f80678d0 con 0x7ff6f80d14a0 2024-01-16T15:04:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.314+0000 7ff6eea00640 1 -- 172.21.15.43:0/2319341270 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3696 (secure 0 0 0) 0x7ff6e0017e80 con 0x7ff6f80d14a0 2024-01-16T15:04:34.294 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:34.294 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2588572053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2588572053},{"type":"v1","addr":"172.21.15.43:6837","nonce":2588572053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/791773791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":791773791},{"type":"v1","addr":"172.21.15.69:6833","nonce":791773791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3012792492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3012792492},{"type":"v1","addr":"172.21.15.69:6835","nonce":3012792492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/2054901873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2054901873},{"type":"v1","addr":"172.21.15.43:6836","nonce":2054901873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2024-01-16T15:04:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-01-16T15:04:34.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 msgr2=0x7ff6d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 0x7ff6d006dd20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8006fd0 tx=0x7ff6e8008040 comp rx=0 tx=0).stop 2024-01-16T15:04:34.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 msgr2=0x7ff6f80cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c430 tx=0x7ff6e000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 shutdown_connections 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6d006b860 0x7ff6d006dd20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6f815d5b0 0x7ff6f8152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6f80d14a0 0x7ff6f80cbe30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 --2- 172.21.15.43:0/2319341270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6f80b1650 0x7ff6f8160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.317+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 >> 172.21.15.43:0/2319341270 conn(0x7ff6f809c3b0 msgr2=0x7ff6f80cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.318+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 shutdown_connections 2024-01-16T15:04:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.318+0000 7ff6f7e00640 1 -- 172.21.15.43:0/2319341270 wait complete. 2024-01-16T15:04:34.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:04:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.562+0000 7efde5600640 1 Processor -- start 2024-01-16T15:04:34.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.562+0000 7efde5600640 1 -- start start 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.563+0000 7efde5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 0x7efde00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde5600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efde00d1d90 0x7efde00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde5600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00c6bf0 con 0x7efde00d0fa0 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde5600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00d1430 con 0x7efde00d1d90 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde5600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00cedc0 con 0x7efde015d3b0 2024-01-16T15:04:34.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde4c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efde00d1d90 0x7efde00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde4c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efde00d1d90 0x7efde00b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46424/0 (socket says 172.21.15.43:46424) 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efde4c00640 1 -- 172.21.15.43:0/1552041892 learned_addr learned my addr 172.21.15.43:0/1552041892 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.564+0000 7efddfe00640 1 --2- 172.21.15.43:0/1552041892 >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 0x7efde00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.565+0000 7efde4c00640 1 -- 172.21.15.43:0/1552041892 >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 msgr2=0x7efde00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.565+0000 7efde4c00640 1 --2- 172.21.15.43:0/1552041892 >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 0x7efde00ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.565+0000 7efde4c00640 1 -- 172.21.15.43:0/1552041892 >> v1:172.21.15.43:6789/0 conn(0x7efde00d0fa0 legacy=0x7efde00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.565+0000 7efde4c00640 1 -- 172.21.15.43:0/1552041892 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00c6e40 con 0x7efde00d1d90 2024-01-16T15:04:34.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.565+0000 7efddfe00640 1 --2- 172.21.15.43:0/1552041892 >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 0x7efde00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.566+0000 7efde4c00640 1 --2- 172.21.15.43:0/1552041892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde00d1d90 0x7efde00b3ca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efde008b2a0 tx=0x7efdd802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a09f89de152575 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.566+0000 7efddf400640 1 -- 172.21.15.43:0/1552041892 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd802ca80 con 0x7efde00d1d90 2024-01-16T15:04:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.566+0000 7efddf400640 1 -- 172.21.15.43:0/1552041892 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd802cbe0 con 0x7efde00d1d90 2024-01-16T15:04:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.566+0000 7efddf400640 1 -- 172.21.15.43:0/1552041892 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd80343f0 con 0x7efde00d1d90 2024-01-16T15:04:34.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 -- 172.21.15.43:0/1552041892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde00d1d90 msgr2=0x7efde00b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 --2- 172.21.15.43:0/1552041892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde00d1d90 0x7efde00b3ca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efde008b2a0 tx=0x7efdd802bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 -- 172.21.15.43:0/1552041892 shutdown_connections 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 --2- 172.21.15.43:0/1552041892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde00d1d90 0x7efde00b3ca0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 --2- 172.21.15.43:0/1552041892 >> v2:172.21.15.69:3300/0 conn(0x7efde015d3b0 0x7efde00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 -- 172.21.15.43:0/1552041892 >> 172.21.15.43:0/1552041892 conn(0x7efde00d30a0 msgr2=0x7efde00d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 -- 172.21.15.43:0/1552041892 shutdown_connections 2024-01-16T15:04:34.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.567+0000 7efde5600640 1 -- 172.21.15.43:0/1552041892 wait complete. 2024-01-16T15:04:34.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.569+0000 7efde5600640 1 Processor -- start 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.569+0000 7efde5600640 1 -- start start 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00d0fa0 0x7efde01f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efde00d1d90 0x7efde00cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00ca9d0 con 0x7efde015d3b0 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde0151ba0 con 0x7efde00d0fa0 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde01606d0 con 0x7efde00d1d90 2024-01-16T15:04:34.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46440/0 (socket says 172.21.15.43:46440) 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efde4c00640 1 -- 172.21.15.43:0/2485256502 learned_addr learned my addr 172.21.15.43:0/2485256502 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.570+0000 7efddfe00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00d0fa0 0x7efde01f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efdd7e00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efde00d1d90 0x7efde00cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 -- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efde00d1d90 msgr2=0x7efde00cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efde00d1d90 0x7efde00cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 -- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00d0fa0 msgr2=0x7efde01f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00d0fa0 0x7efde01f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 -- 172.21.15.43:0/2485256502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009b30 con 0x7efde015d3b0 2024-01-16T15:04:34.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde4c00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efdd8036040 tx=0x7efdd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8009510 con 0x7efde015d3b0 2024-01-16T15:04:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd803e450 con 0x7efde015d3b0 2024-01-16T15:04:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd803e800 con 0x7efde015d3b0 2024-01-16T15:04:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.571+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde00ca310 con 0x7efde015d3b0 2024-01-16T15:04:34.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.572+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde01f2790 con 0x7efde015d3b0 2024-01-16T15:04:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.573+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde00ca170 con 0x7efde015d3b0 2024-01-16T15:04:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.574+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7efdd802cd50 con 0x7efde015d3b0 2024-01-16T15:04:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.575+0000 7efddcc00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 0x7efdb806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.575+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7efdd80acfc0 con 0x7efde015d3b0 2024-01-16T15:04:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.575+0000 7efddfe00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 0x7efdb806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.576+0000 7efddfe00640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 0x7efdb806dd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdd000a950 tx=0x7efdd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.579+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efde00ca170 con 0x7efde015d3b0 2024-01-16T15:04:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.773+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7efde00cab10 con 0x7efde015d3b0 2024-01-16T15:04:34.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.777+0000 7efddcc00640 1 -- 172.21.15.43:0/2485256502 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v17) v1 ==== 120+0+0 (secure 0 0 0) 0x7efde00cab10 con 0x7efde015d3b0 2024-01-16T15:04:34.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.780+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 msgr2=0x7efdb806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.780+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 0x7efdb806dd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdd000a950 tx=0x7efdd0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:34.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 msgr2=0x7efde00b7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efdd8036040 tx=0x7efdd8004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 shutdown_connections 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efdb806b860 0x7efdb806dd20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efde015d3b0 0x7efde00b7c90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efde00d1d90 0x7efde00cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 --2- 172.21.15.43:0/2485256502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00d0fa0 0x7efde01f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 >> 172.21.15.43:0/2485256502 conn(0x7efde00d30a0 msgr2=0x7efde00d1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 shutdown_connections 2024-01-16T15:04:34.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:34.781+0000 7efde5600640 1 -- 172.21.15.43:0/2485256502 wait complete. 2024-01-16T15:04:34.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:04:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.018+0000 7f7a76e00640 1 Processor -- start 2024-01-16T15:04:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.018+0000 7f7a76e00640 1 -- start start 2024-01-16T15:04:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a76e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7a7015f270 0x7f7a700cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a76e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7015f930 con 0x7f7a70144770 2024-01-16T15:04:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a76e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700b3b10 con 0x7f7a7006d9e0 2024-01-16T15:04:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a76e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a700b43b0 con 0x7f7a7015f270 2024-01-16T15:04:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a75a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7a70144770 0x7f7a7015eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45142/0 (socket says 172.21.15.43:45142) 2024-01-16T15:04:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a75a00640 1 -- 172.21.15.43:0/2165342511 learned_addr learned my addr 172.21.15.43:0/2165342511 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.020+0000 7f7a75000640 1 --2- 172.21.15.43:0/2165342511 >> v2:172.21.15.43:3300/0 conn(0x7f7a7015f270 0x7f7a700cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.021+0000 7f7a6fe00640 1 -- 172.21.15.43:0/2165342511 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300941530 0 0) 0x7f7a7015f930 con 0x7f7a70144770 2024-01-16T15:04:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.021+0000 7f7a6fe00640 1 -- 172.21.15.43:0/2165342511 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50003520 con 0x7f7a70144770 2024-01-16T15:04:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.021+0000 7f7a75000640 1 -- 172.21.15.43:0/2165342511 >> v1:172.21.15.69:6789/0 conn(0x7f7a7006d9e0 legacy=0x7f7a700cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.021+0000 7f7a75000640 1 -- 172.21.15.43:0/2165342511 >> v1:172.21.15.43:6789/0 conn(0x7f7a70144770 legacy=0x7f7a7015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.021+0000 7f7a75000640 1 -- 172.21.15.43:0/2165342511 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700b3df0 con 0x7f7a7015f270 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.022+0000 7f7a75000640 1 --2- 172.21.15.43:0/2165342511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a7015f270 0x7f7a700cd4a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097a0 tx=0x7f7a5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8331ff5c022236b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.022+0000 7f7a6fe00640 1 -- 172.21.15.43:0/2165342511 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58030610 con 0x7f7a7015f270 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.022+0000 7f7a6fe00640 1 -- 172.21.15.43:0/2165342511 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a58030770 con 0x7f7a7015f270 2024-01-16T15:04:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.022+0000 7f7a6fe00640 1 -- 172.21.15.43:0/2165342511 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58030b00 con 0x7f7a7015f270 2024-01-16T15:04:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 -- 172.21.15.43:0/2165342511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a7015f270 msgr2=0x7f7a700cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2165342511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a7015f270 0x7f7a700cd4a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097a0 tx=0x7f7a5802b960 comp rx=0 tx=0).stop 2024-01-16T15:04:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 -- 172.21.15.43:0/2165342511 shutdown_connections 2024-01-16T15:04:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2165342511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a7015f270 0x7f7a700cd4a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 -- 172.21.15.43:0/2165342511 >> 172.21.15.43:0/2165342511 conn(0x7f7a700cec10 msgr2=0x7f7a700cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 -- 172.21.15.43:0/2165342511 shutdown_connections 2024-01-16T15:04:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.023+0000 7f7a76e00640 1 -- 172.21.15.43:0/2165342511 wait complete. 2024-01-16T15:04:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.025+0000 7f7a76e00640 1 Processor -- start 2024-01-16T15:04:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.025+0000 7f7a76e00640 1 -- start start 2024-01-16T15:04:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7006d9e0 0x7f7a701eca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7015f270 0x7f7a700cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7015a6c0 con 0x7f7a70144770 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7015d4c0 con 0x7f7a7015f270 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a76e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7013d6f0 con 0x7f7a7006d9e0 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.026+0000 7f7a75000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46462/0 (socket says 172.21.15.43:46462) 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 -- 172.21.15.43:0/2476317832 learned_addr learned my addr 172.21.15.43:0/2476317832 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a76400640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7015f270 0x7f7a700cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75a00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7006d9e0 0x7f7a701eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 -- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7006d9e0 msgr2=0x7f7a701eca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7006d9e0 0x7f7a701eca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 -- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7015f270 msgr2=0x7f7a700cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7015f270 0x7f7a700cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 -- 172.21.15.43:0/2476317832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a58009400 con 0x7f7a70144770 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.027+0000 7f7a75000640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7a5802bf70 tx=0x7f7a580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.028+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5803e070 con 0x7f7a70144770 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.028+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a58030e10 con 0x7f7a70144770 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.028+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58039c30 con 0x7f7a70144770 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.028+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7013d9d0 con 0x7f7a70144770 2024-01-16T15:04:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.028+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7013df90 con 0x7f7a70144770 2024-01-16T15:04:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.029+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7013e180 con 0x7f7a70144770 2024-01-16T15:04:35.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.030+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7a5804b400 con 0x7f7a70144770 2024-01-16T15:04:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.031+0000 7f7a6d600640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 0x7f7a4806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.031+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7a580ac840 con 0x7f7a70144770 2024-01-16T15:04:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.031+0000 7f7a75a00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 0x7f7a4806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.032+0000 7f7a75a00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 0x7f7a4806dd20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7a600011e0 tx=0x7f7a60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.035+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a58076fd0 con 0x7f7a70144770 2024-01-16T15:04:35.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.233+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f7a7004e840 con 0x7f7a70144770 2024-01-16T15:04:35.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.239+0000 7f7a6d600640 1 -- 172.21.15.43:0/2476317832 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v18) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7a5807a5f0 con 0x7f7a70144770 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.242+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 msgr2=0x7f7a4806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.242+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 0x7f7a4806dd20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7a600011e0 tx=0x7f7a60009040 comp rx=0 tx=0).stop 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.242+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 msgr2=0x7f7a701ecf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.242+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7a5802bf70 tx=0x7f7a580046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 shutdown_connections 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a4806b860 0x7f7a4806dd20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7015f270 0x7f7a700cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a70144770 0x7f7a701ecf80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 --2- 172.21.15.43:0/2476317832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7006d9e0 0x7f7a701eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 >> 172.21.15.43:0/2476317832 conn(0x7f7a700cec10 msgr2=0x7f7a70070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 shutdown_connections 2024-01-16T15:04:35.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.243+0000 7f7a76e00640 1 -- 172.21.15.43:0/2476317832 wait complete. 2024-01-16T15:04:35.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:04:35.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.482+0000 7f9639e00640 1 Processor -- start 2024-01-16T15:04:35.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.482+0000 7f9639e00640 1 -- start start 2024-01-16T15:04:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.483+0000 7f9639e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 0x7f96340ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.483+0000 7f9639e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9634070490 0x7f9634070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9639e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963406d7a0 con 0x7f96340acd70 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9639e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963415a5b0 con 0x7f9634070490 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9639e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340ae9d0 con 0x7f963406dc30 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9632a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f96340acd70 0x7f96340ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45152/0 (socket says 172.21.15.43:45152) 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9633400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9634070490 0x7f9634070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9632a00640 1 -- 172.21.15.43:0/2105598849 learned_addr learned my addr 172.21.15.43:0/2105598849 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9632000640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 0x7f96340ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9633400640 1 -- 172.21.15.43:0/2105598849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 msgr2=0x7f96340ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9633400640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 0x7f96340ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9633400640 1 -- 172.21.15.43:0/2105598849 >> v1:172.21.15.43:6789/0 conn(0x7f96340acd70 legacy=0x7f96340ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9633400640 1 -- 172.21.15.43:0/2105598849 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963406ce70 con 0x7f9634070490 2024-01-16T15:04:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9631600640 1 -- 172.21.15.43:0/2105598849 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402660765 0 0) 0x7f963406d7a0 con 0x7f96340acd70 2024-01-16T15:04:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.484+0000 7f9632000640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 0x7f96340ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9633400640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634070920 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f962c009d90 tx=0x7f962c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4adb6bbdebc70fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9631600640 1 -- 172.21.15.43:0/2105598849 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c02cc60 con 0x7f9634070490 2024-01-16T15:04:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9631600640 1 -- 172.21.15.43:0/2105598849 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c02cdc0 con 0x7f9634070490 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9631600640 1 -- 172.21.15.43:0/2105598849 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c034660 con 0x7f9634070490 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9639e00640 1 -- 172.21.15.43:0/2105598849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 msgr2=0x7f9634070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.486+0000 7f9639e00640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634070920 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f962c009d90 tx=0x7f962c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 -- 172.21.15.43:0/2105598849 shutdown_connections 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634070920 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 --2- 172.21.15.43:0/2105598849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f963406dc30 0x7f96340ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 -- 172.21.15.43:0/2105598849 >> 172.21.15.43:0/2105598849 conn(0x7f96340b0ba0 msgr2=0x7f96340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 -- 172.21.15.43:0/2105598849 shutdown_connections 2024-01-16T15:04:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.487+0000 7f9639e00640 1 -- 172.21.15.43:0/2105598849 wait complete. 2024-01-16T15:04:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.488+0000 7f9639e00640 1 Processor -- start 2024-01-16T15:04:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.489+0000 7f9639e00640 1 -- start start 2024-01-16T15:04:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.489+0000 7f9639e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 0x7f96341e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.489+0000 7f9639e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9639e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96340acd70 0x7f96341e30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9639e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634145070 con 0x7f9634070490 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9639e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96341e6540 con 0x7f963406dc30 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9639e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96341e5ca0 con 0x7f96340acd70 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46478/0 (socket says 172.21.15.43:46478) 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 -- 172.21.15.43:0/2434519775 learned_addr learned my addr 172.21.15.43:0/2434519775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632a00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 0x7f96341e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 -- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96340acd70 msgr2=0x7f96341e30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9633400640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96340acd70 0x7f96341e30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96340acd70 0x7f96341e30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 -- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 msgr2=0x7f96341e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 0x7f96341e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632000640 1 -- 172.21.15.43:0/2434519775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c009a70 con 0x7f9634070490 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.490+0000 7f9632a00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 0x7f96341e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f9632000640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c820 tx=0x7f961c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c007a30 con 0x7f9634070490 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c007b90 con 0x7f9634070490 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c00ce70 con 0x7f9634070490 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96341e5f80 con 0x7f9634070490 2024-01-16T15:04:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.491+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9634157de0 con 0x7f9634070490 2024-01-16T15:04:35.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.492+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9634157fd0 con 0x7f9634070490 2024-01-16T15:04:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.493+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f961c0113f0 con 0x7f9634070490 2024-01-16T15:04:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.494+0000 7f962b400640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 0x7f960c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.494+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f961c08a580 con 0x7f9634070490 2024-01-16T15:04:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.495+0000 7f9632a00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 0x7f960c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.495+0000 7f9632a00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 0x7f960c06dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9624000f60 tx=0x7f9624009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.499+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f961c054de0 con 0x7f9634070490 2024-01-16T15:04:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.694+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f963409d200 con 0x7f9634070490 2024-01-16T15:04:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.699+0000 7f962b400640 1 -- 172.21.15.43:0/2434519775 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f961c058400 con 0x7f9634070490 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 msgr2=0x7f960c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 0x7f960c06dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9624000f60 tx=0x7f9624009040 comp rx=0 tx=0).stop 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 msgr2=0x7f9634144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c820 tx=0x7f961c00cc20 comp rx=0 tx=0).stop 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 shutdown_connections 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f960c06b860 0x7f960c06dd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96340acd70 0x7f96341e30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9634070490 0x7f9634144b30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 --2- 172.21.15.43:0/2434519775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f963406dc30 0x7f96341e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 >> 172.21.15.43:0/2434519775 conn(0x7f96340b0ba0 msgr2=0x7f9634070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 shutdown_connections 2024-01-16T15:04:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.702+0000 7f9639e00640 1 -- 172.21.15.43:0/2434519775 wait complete. 2024-01-16T15:04:35.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:04:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.937+0000 7f089da00640 1 Processor -- start 2024-01-16T15:04:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.938+0000 7f089da00640 1 -- start start 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f089da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0898070e10 0x7f0898161b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f089da00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08981628b0 con 0x7f089806d440 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f089da00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898070050 con 0x7f08981620a0 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f089da00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08980b4840 con 0x7f0898070e10 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f0897400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0898070e10 0x7f0898161b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f0897e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f089806d440 0x7f089806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45158/0 (socket says 172.21.15.43:45158) 2024-01-16T15:04:35.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f0897e00640 1 -- 172.21.15.43:0/205648252 learned_addr learned my addr 172.21.15.43:0/205648252 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.940+0000 7f0896a00640 1 -- 172.21.15.43:0/205648252 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675443135 0 0) 0x7f08981628b0 con 0x7f089806d440 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.941+0000 7f0897400640 1 -- 172.21.15.43:0/205648252 >> v1:172.21.15.69:6789/0 conn(0x7f08981620a0 legacy=0x7f08980b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.941+0000 7f0897400640 1 -- 172.21.15.43:0/205648252 >> v1:172.21.15.43:6789/0 conn(0x7f089806d440 legacy=0x7f089806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.941+0000 7f0897400640 1 -- 172.21.15.43:0/205648252 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898070330 con 0x7f0898070e10 2024-01-16T15:04:35.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.942+0000 7f0897400640 1 --2- 172.21.15.43:0/205648252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0898070e10 0x7f0898161b60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f08980b4980 tx=0x7f088002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff0e2f09fc6f6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.942+0000 7f0896a00640 1 -- 172.21.15.43:0/205648252 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880031830 con 0x7f0898070e10 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.942+0000 7f0896a00640 1 -- 172.21.15.43:0/205648252 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880031990 con 0x7f0898070e10 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.942+0000 7f0896a00640 1 -- 172.21.15.43:0/205648252 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880031d20 con 0x7f0898070e10 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 -- 172.21.15.43:0/205648252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0898070e10 msgr2=0x7f0898161b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 --2- 172.21.15.43:0/205648252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0898070e10 0x7f0898161b60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f08980b4980 tx=0x7f088002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 -- 172.21.15.43:0/205648252 shutdown_connections 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 --2- 172.21.15.43:0/205648252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0898070e10 0x7f0898161b60 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 -- 172.21.15.43:0/205648252 >> 172.21.15.43:0/205648252 conn(0x7f08980b0b10 msgr2=0x7f08980b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 -- 172.21.15.43:0/205648252 shutdown_connections 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.943+0000 7f089da00640 1 -- 172.21.15.43:0/205648252 wait complete. 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.945+0000 7f089da00640 1 Processor -- start 2024-01-16T15:04:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.945+0000 7f089da00640 1 -- start start 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 0x7f08981faf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 0x7f089815ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981fb470 con 0x7f08981620a0 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089815efb0 con 0x7f0898070e10 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089da00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08981f9110 con 0x7f089806d440 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46510/0 (socket says 172.21.15.43:46510) 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f089d000640 1 -- 172.21.15.43:0/429700192 learned_addr learned my addr 172.21.15.43:0/429700192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f0897e00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 0x7f08981faf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.946+0000 7f0897400640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 0x7f089815ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 -- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 msgr2=0x7f08981faf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 0x7f08981faf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 -- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 msgr2=0x7f089815ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 0x7f089815ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 -- 172.21.15.43:0/429700192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880009400 con 0x7f08981620a0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f0897400640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 0x7f089815ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f0897e00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 0x7f08981faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089d000640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f089000c440 tx=0x7f089000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890011790 con 0x7f08981620a0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08900118f0 con 0x7f08981620a0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890011ca0 con 0x7f08981620a0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08980b34d0 con 0x7f08981620a0 2024-01-16T15:04:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.947+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981f87e0 con 0x7f08981620a0 2024-01-16T15:04:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.949+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089804e840 con 0x7f08981620a0 2024-01-16T15:04:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.949+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f089001c460 con 0x7f08981620a0 2024-01-16T15:04:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.950+0000 7f088fe00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 0x7f087006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.951+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f089008a220 con 0x7f08981620a0 2024-01-16T15:04:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.951+0000 7f0897e00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 0x7f087006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.951+0000 7f0897e00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 0x7f087006dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0888007fd0 tx=0x7f0888009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:35.955+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0890054b50 con 0x7f08981620a0 2024-01-16T15:04:36.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.217+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f08981f9250 con 0x7f08981620a0 2024-01-16T15:04:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.840+0000 7f088fe00640 1 -- 172.21.15.43:0/429700192 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v255) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0890058170 con 0x7f08981620a0 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 msgr2=0x7f087006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 0x7f087006dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0888007fd0 tx=0x7f0888009040 comp rx=0 tx=0).stop 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 msgr2=0x7f08981551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f089000c440 tx=0x7f089000c900 comp rx=0 tx=0).stop 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 shutdown_connections 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f087006b860 0x7f087006dd20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0898070e10 0x7f089815ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08981620a0 0x7f08981551c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 --2- 172.21.15.43:0/429700192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089806d440 0x7f08981faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.843+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 >> 172.21.15.43:0/429700192 conn(0x7f08980b0b10 msgr2=0x7f08980b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.844+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 shutdown_connections 2024-01-16T15:04:36.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:36.844+0000 7f089da00640 1 -- 172.21.15.43:0/429700192 wait complete. 2024-01-16T15:04:36.846 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:04:36.846 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:04:36.846 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:04:36.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:37.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.085+0000 7fc172600640 1 Processor -- start 2024-01-16T15:04:37.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.086+0000 7fc172600640 1 -- start start 2024-01-16T15:04:37.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.087+0000 7fc172600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc16c0d14a0 0x7fc16c0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.087+0000 7fc172600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0cf0c0 con 0x7fc16c0b0640 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.087+0000 7fc172600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0d1e40 con 0x7fc16c15d5b0 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.087+0000 7fc172600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0b01b0 con 0x7fc16c0d14a0 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc171c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc16c0b0640 0x7fc16c0b0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45170/0 (socket says 172.21.15.43:45170) 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc171c00640 1 -- 172.21.15.43:0/2374540837 learned_addr learned my addr 172.21.15.43:0/2374540837 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16be00640 1 --2- 172.21.15.43:0/2374540837 >> v2:172.21.15.43:3300/0 conn(0x7fc16c0d14a0 0x7fc16c0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16b400640 1 -- 172.21.15.43:0/2374540837 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075047538 0 0) 0x7fc16c0cf0c0 con 0x7fc16c0b0640 2024-01-16T15:04:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16b400640 1 -- 172.21.15.43:0/2374540837 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc14c003520 con 0x7fc16c0b0640 2024-01-16T15:04:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16be00640 1 -- 172.21.15.43:0/2374540837 >> v1:172.21.15.69:6789/0 conn(0x7fc16c15d5b0 legacy=0x7fc16c0d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16be00640 1 -- 172.21.15.43:0/2374540837 >> v1:172.21.15.43:6789/0 conn(0x7fc16c0b0640 legacy=0x7fc16c0b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.088+0000 7fc16be00640 1 -- 172.21.15.43:0/2374540837 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c15d990 con 0x7fc16c0d14a0 2024-01-16T15:04:37.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc16be00640 1 --2- 172.21.15.43:0/2374540837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0d14a0 0x7fc16c0d1900 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc1540097a0 tx=0x7fc15402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45db96730f77a47f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc16b400640 1 -- 172.21.15.43:0/2374540837 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154030610 con 0x7fc16c0d14a0 2024-01-16T15:04:37.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc16b400640 1 -- 172.21.15.43:0/2374540837 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154030770 con 0x7fc16c0d14a0 2024-01-16T15:04:37.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc16b400640 1 -- 172.21.15.43:0/2374540837 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154030b00 con 0x7fc16c0d14a0 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc172600640 1 -- 172.21.15.43:0/2374540837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0d14a0 msgr2=0x7fc16c0d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.090+0000 7fc172600640 1 --2- 172.21.15.43:0/2374540837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0d14a0 0x7fc16c0d1900 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc1540097a0 tx=0x7fc15402b960 comp rx=0 tx=0).stop 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.091+0000 7fc172600640 1 -- 172.21.15.43:0/2374540837 shutdown_connections 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.091+0000 7fc172600640 1 --2- 172.21.15.43:0/2374540837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0d14a0 0x7fc16c0d1900 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.091+0000 7fc172600640 1 -- 172.21.15.43:0/2374540837 >> 172.21.15.43:0/2374540837 conn(0x7fc16c09c3b0 msgr2=0x7fc16c0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.091+0000 7fc172600640 1 -- 172.21.15.43:0/2374540837 shutdown_connections 2024-01-16T15:04:37.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.091+0000 7fc172600640 1 -- 172.21.15.43:0/2374540837 wait complete. 2024-01-16T15:04:37.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.093+0000 7fc172600640 1 Processor -- start 2024-01-16T15:04:37.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.093+0000 7fc172600640 1 -- start start 2024-01-16T15:04:37.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 0x7fc16c1f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c15d5b0 0x7fc16c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1557b0 con 0x7fc16c0b0640 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0ca140 con 0x7fc16c15d5b0 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc172600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1f5f90 con 0x7fc16c0d14a0 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc171200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc171200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46524/0 (socket says 172.21.15.43:46524) 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.094+0000 7fc171200640 1 -- 172.21.15.43:0/1686495240 learned_addr learned my addr 172.21.15.43:0/1686495240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171c00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c15d5b0 0x7fc16c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc16be00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 0x7fc16c1f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 -- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 msgr2=0x7fc16c1f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 0x7fc16c1f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 -- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c15d5b0 msgr2=0x7fc16c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c15d5b0 0x7fc16c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 -- 172.21.15.43:0/1686495240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc154009400 con 0x7fc16c0b0640 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc16be00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 0x7fc16c1f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc171200640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c400 tx=0x7fc15c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0052d0 con 0x7fc16c0b0640 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c005430 con 0x7fc16c0b0640 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c003830 con 0x7fc16c0b0640 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c1554d0 con 0x7fc16c0b0640 2024-01-16T15:04:37.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.095+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c0ca670 con 0x7fc16c0b0640 2024-01-16T15:04:37.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.097+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c04e840 con 0x7fc16c0b0640 2024-01-16T15:04:37.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.097+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc15c003990 con 0x7fc16c0b0640 2024-01-16T15:04:37.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.098+0000 7fc168c00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 0x7fc14406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.098+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc15c0898d0 con 0x7fc16c0b0640 2024-01-16T15:04:37.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.099+0000 7fc16be00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 0x7fc14406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.100+0000 7fc16be00640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 0x7fc14406dd50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc154036000 tx=0x7fc15403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.103+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc15c0541b0 con 0x7fc16c0b0640 2024-01-16T15:04:37.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.303+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc16c0678d0 con 0x7fc16c0b0640 2024-01-16T15:04:37.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.304+0000 7fc168c00640 1 -- 172.21.15.43:0/1686495240 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12981 (secure 0 0 0) 0x7fc15c0577d0 con 0x7fc16c0b0640 2024-01-16T15:04:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:37.284 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":53,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:29.783590+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6836/3947724910":"2024-01-17T15:04:26.768522+0000","172.21.15.43:6834/3947724910":"2024-01-17T15:04:26.768522+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:37.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 msgr2=0x7fc14406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 0x7fc14406dd50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc154036000 tx=0x7fc15403b040 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 msgr2=0x7fc16c1f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c400 tx=0x7fc15c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 shutdown_connections 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc14406b890 0x7fc14406dd50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc16c15d5b0 0x7fc16c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16c0d14a0 0x7fc16c1f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 --2- 172.21.15.43:0/1686495240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16c0b0640 0x7fc16c1f3e00 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.307+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 >> 172.21.15.43:0/1686495240 conn(0x7fc16c09c3b0 msgr2=0x7fc16c15c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.308+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 shutdown_connections 2024-01-16T15:04:37.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.308+0000 7fc172600640 1 -- 172.21.15.43:0/1686495240 wait complete. 2024-01-16T15:04:37.306 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:37.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.549+0000 7fda48400640 1 Processor -- start 2024-01-16T15:04:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.550+0000 7fda48400640 1 -- start start 2024-01-16T15:04:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.551+0000 7fda48400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda400abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.551+0000 7fda48400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fda400adb60 0x7fda400adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.551+0000 7fda48400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4015b730 con 0x7fda4015b350 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.551+0000 7fda48400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda400ac740 con 0x7fda400adb60 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.551+0000 7fda48400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda400ac460 con 0x7fda4009cb30 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fda400adb60 0x7fda400adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fda400adb60 0x7fda400adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46526/0 (socket says 172.21.15.43:46526) 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 -- 172.21.15.43:0/221892229 learned_addr learned my addr 172.21.15.43:0/221892229 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda45000640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda400abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 -- 172.21.15.43:0/221892229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 msgr2=0x7fda400abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda400abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 -- 172.21.15.43:0/221892229 >> v1:172.21.15.69:6789/0 conn(0x7fda4015b350 legacy=0x7fda400b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.552+0000 7fda46400640 1 -- 172.21.15.43:0/221892229 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda400b2a50 con 0x7fda400adb60 2024-01-16T15:04:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda46400640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400adfb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda38009e50 tx=0x7fda3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafec5c16cdb0207 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda3fe00640 1 -- 172.21.15.43:0/221892229 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3802cb00 con 0x7fda400adb60 2024-01-16T15:04:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda3fe00640 1 -- 172.21.15.43:0/221892229 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3802cc60 con 0x7fda400adb60 2024-01-16T15:04:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda3fe00640 1 -- 172.21.15.43:0/221892229 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380343f0 con 0x7fda400adb60 2024-01-16T15:04:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 -- 172.21.15.43:0/221892229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 msgr2=0x7fda400adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400adfb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda38009e50 tx=0x7fda3802bf60 comp rx=0 tx=0).stop 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 -- 172.21.15.43:0/221892229 shutdown_connections 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400adfb0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 --2- 172.21.15.43:0/221892229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda400abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 -- 172.21.15.43:0/221892229 >> 172.21.15.43:0/221892229 conn(0x7fda400b0ba0 msgr2=0x7fda400b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 -- 172.21.15.43:0/221892229 shutdown_connections 2024-01-16T15:04:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.555+0000 7fda48400640 1 -- 172.21.15.43:0/221892229 wait complete. 2024-01-16T15:04:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.557+0000 7fda48400640 1 Processor -- start 2024-01-16T15:04:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.557+0000 7fda48400640 1 -- start start 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda4014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 0x7fda400afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4009bc90 con 0x7fda400adb60 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4009b6d0 con 0x7fda4015b350 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda48400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4009b9b0 con 0x7fda4009cb30 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda45000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda45000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46528/0 (socket says 172.21.15.43:46528) 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.558+0000 7fda45000640 1 -- 172.21.15.43:0/3262153912 learned_addr learned my addr 172.21.15.43:0/3262153912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda46400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 0x7fda400afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45a00640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda4014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 -- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 msgr2=0x7fda4014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda4014b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 -- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 msgr2=0x7fda400afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 0x7fda400afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 -- 172.21.15.43:0/3262153912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38009b30 con 0x7fda400adb60 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda46400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 0x7fda400afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda45000640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda2800d800 tx=0x7fda2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2800b6c0 con 0x7fda400adb60 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2800b820 con 0x7fda400adb60 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.559+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2800bbb0 con 0x7fda400adb60 2024-01-16T15:04:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.560+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda401f2ef0 con 0x7fda400adb60 2024-01-16T15:04:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.560+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4014be20 con 0x7fda400adb60 2024-01-16T15:04:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.561+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4009d200 con 0x7fda400adb60 2024-01-16T15:04:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.562+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fda2800bd10 con 0x7fda400adb60 2024-01-16T15:04:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.563+0000 7fda3d600640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 0x7fda1806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.563+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fda2808a900 con 0x7fda400adb60 2024-01-16T15:04:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.563+0000 7fda45a00640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 0x7fda1806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.564+0000 7fda45a00640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 0x7fda1806dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fda401597e0 tx=0x7fda30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.567+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda280551e0 con 0x7fda400adb60 2024-01-16T15:04:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.759+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda4004e920 con 0x7fda400adb60 2024-01-16T15:04:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.759+0000 7fda3d600640 1 -- 172.21.15.43:0/3262153912 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda28058800 con 0x7fda400adb60 2024-01-16T15:04:37.739 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:04:37.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 msgr2=0x7fda1806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 0x7fda1806dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fda401597e0 tx=0x7fda30009040 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 msgr2=0x7fda400af5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda2800d800 tx=0x7fda2800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 shutdown_connections 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda1806b860 0x7fda1806dd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4015b350 0x7fda400afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda400adb60 0x7fda400af5b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 --2- 172.21.15.43:0/3262153912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4009cb30 0x7fda4014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 >> 172.21.15.43:0/3262153912 conn(0x7fda400b0ba0 msgr2=0x7fda4015a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.762+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 shutdown_connections 2024-01-16T15:04:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:37.763+0000 7fda48400640 1 -- 172.21.15.43:0/3262153912 wait complete. 2024-01-16T15:04:37.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:04:37.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.006+0000 7f570a200640 1 Processor -- start 2024-01-16T15:04:37.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.006+0000 7f570a200640 1 -- start start 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.007+0000 7f570a200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f57040cf050 0x7f57040cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.007+0000 7f570a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f57040cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.007+0000 7f570a200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570415fa70 con 0x7f570413dfd0 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.007+0000 7f570a200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040cfa40 con 0x7f57040cf050 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.008+0000 7f570a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570415c580 con 0x7f57040cd7b0 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.008+0000 7f56fbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f57040cf050 0x7f57040cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.008+0000 7f5708e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f570413dfd0 0x7f570415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45184/0 (socket says 172.21.15.43:45184) 2024-01-16T15:04:37.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.008+0000 7f5708e00640 1 -- 172.21.15.43:0/2169442211 learned_addr learned my addr 172.21.15.43:0/2169442211 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:37.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.008+0000 7f5709800640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f57040cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.009+0000 7f56fbe00640 1 -- 172.21.15.43:0/2169442211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 msgr2=0x7f57040cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:37.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.009+0000 7f56fbe00640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f57040cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.009+0000 7f56fbe00640 1 -- 172.21.15.43:0/2169442211 >> v1:172.21.15.43:6789/0 conn(0x7f570413dfd0 legacy=0x7f570415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.009+0000 7f5703e00640 1 -- 172.21.15.43:0/2169442211 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616980711 0 0) 0x7f570415fa70 con 0x7f570413dfd0 2024-01-16T15:04:37.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.009+0000 7f56fbe00640 1 -- 172.21.15.43:0/2169442211 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704140900 con 0x7f57040cf050 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.010+0000 7f56fbe00640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57040cf050 0x7f57040cf500 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0097d0 tx=0x7f56ec02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae62d4befb748cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.010+0000 7f5703e00640 1 -- 172.21.15.43:0/2169442211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec031800 con 0x7f57040cf050 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.010+0000 7f5703e00640 1 -- 172.21.15.43:0/2169442211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec031960 con 0x7f57040cf050 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.010+0000 7f5703e00640 1 -- 172.21.15.43:0/2169442211 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec031cf0 con 0x7f57040cf050 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 -- 172.21.15.43:0/2169442211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57040cf050 msgr2=0x7f57040cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57040cf050 0x7f57040cf500 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0097d0 tx=0x7f56ec02b8a0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 -- 172.21.15.43:0/2169442211 shutdown_connections 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f57040cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 --2- 172.21.15.43:0/2169442211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57040cf050 0x7f57040cf500 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 -- 172.21.15.43:0/2169442211 >> 172.21.15.43:0/2169442211 conn(0x7f570406a860 msgr2=0x7f570406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 -- 172.21.15.43:0/2169442211 shutdown_connections 2024-01-16T15:04:37.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.011+0000 7f570a200640 1 -- 172.21.15.43:0/2169442211 wait complete. 2024-01-16T15:04:37.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.013+0000 7f570a200640 1 Processor -- start 2024-01-16T15:04:37.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.013+0000 7f570a200640 1 -- start start 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f5704155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57040cf050 0x7f570413a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570413a830 con 0x7f570413dfd0 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570413aad0 con 0x7f57040cf050 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f570a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570413adb0 con 0x7f57040cd7b0 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f5709800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f5709800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46542/0 (socket says 172.21.15.43:46542) 2024-01-16T15:04:37.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f5709800640 1 -- 172.21.15.43:0/2192746141 learned_addr learned my addr 172.21.15.43:0/2192746141 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.014+0000 7f5708e00640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f5704155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f56fbe00640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57040cf050 0x7f570413a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 -- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 msgr2=0x7f5704155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f5704155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 -- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57040cf050 msgr2=0x7f570413a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57040cf050 0x7f570413a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 -- 172.21.15.43:0/2192746141 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec009400 con 0x7f570413dfd0 2024-01-16T15:04:37.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5709800640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f56fc007e40 tx=0x7f56fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc014620 con 0x7f570413dfd0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc00ee90 con 0x7f570413dfd0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0149e0 con 0x7f570413dfd0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.015+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570413b090 con 0x7f570413dfd0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.016+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570413b650 con 0x7f570413dfd0 2024-01-16T15:04:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.017+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57040cf710 con 0x7f570413dfd0 2024-01-16T15:04:37.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.018+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f56fc014b40 con 0x7f570413dfd0 2024-01-16T15:04:37.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.020+0000 7f5701600640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 0x7f56dc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:37.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.020+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f56fc08ade0 con 0x7f570413dfd0 2024-01-16T15:04:37.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.020+0000 7f5708e00640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 0x7f56dc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:37.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.021+0000 7f5708e00640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 0x7f56dc06dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000f60 tx=0x7f56f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.025+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56fc090020 con 0x7f570413dfd0 2024-01-16T15:04:38.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.296+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f570404e840 con 0x7f570413dfd0 2024-01-16T15:04:38.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.850+0000 7f5701600640 1 -- 172.21.15.43:0/2192746141 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v257) v1 ==== 157+0+0 (secure 0 0 0) 0x7f56fc0555f0 con 0x7f570413dfd0 2024-01-16T15:04:38.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.853+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 msgr2=0x7f56dc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.853+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 0x7f56dc06dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000f60 tx=0x7f56f4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:38.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.853+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 msgr2=0x7f57041568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.853+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f56fc007e40 tx=0x7f56fc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 shutdown_connections 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56dc06b890 0x7f56dc06dd50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57040cf050 0x7f570413a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f570413dfd0 0x7f57041568d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 --2- 172.21.15.43:0/2192746141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57040cd7b0 0x7f5704155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 >> 172.21.15.43:0/2192746141 conn(0x7f570406a860 msgr2=0x7f57040cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 shutdown_connections 2024-01-16T15:04:38.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:38.854+0000 7f570a200640 1 -- 172.21.15.43:0/2192746141 wait complete. 2024-01-16T15:04:38.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:04:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.104+0000 7f1f4a400640 1 Processor -- start 2024-01-16T15:04:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.104+0000 7f1f4a400640 1 -- start start 2024-01-16T15:04:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f4a400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f440d1d90 0x7f1f440d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f4a400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f440cef30 con 0x7f1f440cdbb0 2024-01-16T15:04:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f4a400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f440d11a0 con 0x7f1f4415d340 2024-01-16T15:04:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f4a400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f440ce320 con 0x7f1f440d1d90 2024-01-16T15:04:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f43e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f440d1d90 0x7f1f440d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f49a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1f440cdbb0 0x7f1f440b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45192/0 (socket says 172.21.15.43:45192) 2024-01-16T15:04:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.106+0000 7f1f49a00640 1 -- 172.21.15.43:0/2772072075 learned_addr learned my addr 172.21.15.43:0/2772072075 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.107+0000 7f1f43e00640 1 -- 172.21.15.43:0/2772072075 >> v1:172.21.15.69:6789/0 conn(0x7f1f4415d340 legacy=0x7f1f440d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.107+0000 7f1f43e00640 1 -- 172.21.15.43:0/2772072075 >> v1:172.21.15.43:6789/0 conn(0x7f1f440cdbb0 legacy=0x7f1f440b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.107+0000 7f1f43e00640 1 -- 172.21.15.43:0/2772072075 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f440ce600 con 0x7f1f440d1d90 2024-01-16T15:04:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.108+0000 7f1f43e00640 1 --2- 172.21.15.43:0/2772072075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440d1d90 0x7f1f440d00b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009d90 tx=0x7f1f2c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81281636d8f69f47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.108+0000 7f1f43400640 1 -- 172.21.15.43:0/2772072075 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c02ccf0 con 0x7f1f440d1d90 2024-01-16T15:04:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.108+0000 7f1f43400640 1 -- 172.21.15.43:0/2772072075 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c02ce50 con 0x7f1f440d1d90 2024-01-16T15:04:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.108+0000 7f1f43400640 1 -- 172.21.15.43:0/2772072075 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c034660 con 0x7f1f440d1d90 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.109+0000 7f1f4a400640 1 -- 172.21.15.43:0/2772072075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440d1d90 msgr2=0x7f1f440d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.109+0000 7f1f4a400640 1 --2- 172.21.15.43:0/2772072075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440d1d90 0x7f1f440d00b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009d90 tx=0x7f1f2c02bf40 comp rx=0 tx=0).stop 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.110+0000 7f1f4a400640 1 -- 172.21.15.43:0/2772072075 shutdown_connections 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.110+0000 7f1f4a400640 1 --2- 172.21.15.43:0/2772072075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440d1d90 0x7f1f440d00b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.110+0000 7f1f4a400640 1 -- 172.21.15.43:0/2772072075 >> 172.21.15.43:0/2772072075 conn(0x7f1f4409c370 msgr2=0x7f1f440d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.110+0000 7f1f4a400640 1 -- 172.21.15.43:0/2772072075 shutdown_connections 2024-01-16T15:04:39.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.110+0000 7f1f4a400640 1 -- 172.21.15.43:0/2772072075 wait complete. 2024-01-16T15:04:39.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.111+0000 7f1f4a400640 1 Processor -- start 2024-01-16T15:04:39.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 -- start start 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 0x7f1f440b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f4415d340 0x7f1f44141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440b6190 con 0x7f1f440cdbb0 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4415ec50 con 0x7f1f440d1d90 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.112+0000 7f1f4a400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4415ef30 con 0x7f1f4415d340 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46554/0 (socket says 172.21.15.43:46554) 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 -- 172.21.15.43:0/3864944010 learned_addr learned my addr 172.21.15.43:0/3864944010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f43e00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 0x7f1f440b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49a00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f4415d340 0x7f1f44141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 -- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f4415d340 msgr2=0x7f1f44141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f4415d340 0x7f1f44141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 -- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 msgr2=0x7f1f440b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 0x7f1f440b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f49000640 1 -- 172.21.15.43:0/3864944010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c009a70 con 0x7f1f440cdbb0 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.113+0000 7f1f43e00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 0x7f1f440b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f49000640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400c450 tx=0x7f1f3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340052d0 con 0x7f1f440cdbb0 2024-01-16T15:04:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f34005430 con 0x7f1f440cdbb0 2024-01-16T15:04:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34012870 con 0x7f1f440cdbb0 2024-01-16T15:04:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4415f210 con 0x7f1f440cdbb0 2024-01-16T15:04:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.114+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4415f7d0 con 0x7f1f440cdbb0 2024-01-16T15:04:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.115+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4413fd30 con 0x7f1f440cdbb0 2024-01-16T15:04:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.116+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1f340129d0 con 0x7f1f440cdbb0 2024-01-16T15:04:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.117+0000 7f1f40c00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 0x7f1f1c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.117+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1f34089ee0 con 0x7f1f440cdbb0 2024-01-16T15:04:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.118+0000 7f1f43e00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 0x7f1f1c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.119+0000 7f1f43e00640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 0x7f1f1c06dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c036000 tx=0x7f1f2c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.122+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f3408d030 con 0x7f1f440cdbb0 2024-01-16T15:04:39.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.319+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1f44067d70 con 0x7f1f440cdbb0 2024-01-16T15:04:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.326+0000 7f1f40c00640 1 -- 172.21.15.43:0/3864944010 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1f34054780 con 0x7f1f440cdbb0 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 msgr2=0x7f1f1c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 0x7f1f1c06dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c036000 tx=0x7f1f2c030000 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 msgr2=0x7f1f440b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400c450 tx=0x7f1f3400c910 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 shutdown_connections 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f1c06b860 0x7f1f1c06dd20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f440d1d90 0x7f1f440b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f4415d340 0x7f1f44141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 --2- 172.21.15.43:0/3864944010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f440cdbb0 0x7f1f440b50b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 >> 172.21.15.43:0/3864944010 conn(0x7f1f4409c370 msgr2=0x7f1f440b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 shutdown_connections 2024-01-16T15:04:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:39.329+0000 7f1f4a400640 1 -- 172.21.15.43:0/3864944010 wait complete. 2024-01-16T15:04:39.328 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:04:39.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:39.448 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:04:39.568 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:04:39.569 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:04:39.569 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:39.628 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:39.628 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:04:39.628 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:39.682 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:39.682 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:04:39.682 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:39.742 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:04:39.742 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:04:39.743 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:04:39.799 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:04:39.799 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:04:39.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:40.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.059+0000 7fb263000640 1 Processor -- start 2024-01-16T15:04:40.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.059+0000 7fb263000640 1 -- start start 2024-01-16T15:04:40.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 0x7fb25c06a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 0x7fb25c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb25c15b870 0x7fb25c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c144600 con 0x7fb25c15b870 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c06e540 con 0x7fb25c143f10 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb263000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c06eb20 con 0x7fb25c0af8e0 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.061+0000 7fb261000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb25c15b870 0x7fb25c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb25c15b870 0x7fb25c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46566/0 (socket says 172.21.15.43:46566) 2024-01-16T15:04:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 -- 172.21.15.43:0/3778375054 learned_addr learned my addr 172.21.15.43:0/3778375054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb25b400640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 0x7fb25c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb25be00640 1 --2- 172.21.15.43:0/3778375054 >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 0x7fb25c06a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 -- 172.21.15.43:0/3778375054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 msgr2=0x7fb25c06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 0x7fb25c06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 -- 172.21.15.43:0/3778375054 >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 msgr2=0x7fb25c06a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 --2- 172.21.15.43:0/3778375054 >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 0x7fb25c06a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb261000640 1 -- 172.21.15.43:0/3778375054 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c06b330 con 0x7fb25c15b870 2024-01-16T15:04:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb25b400640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 0x7fb25c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.062+0000 7fb25be00640 1 --2- 172.21.15.43:0/3778375054 >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 0x7fb25c06a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.064+0000 7fb261000640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c15b870 0x7fb25c0cc9f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb254009e50 tx=0x7fb25402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399a517c1e4f01e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.064+0000 7fb25aa00640 1 -- 172.21.15.43:0/3778375054 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25402c9f0 con 0x7fb25c15b870 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.064+0000 7fb25aa00640 1 -- 172.21.15.43:0/3778375054 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25402cb50 con 0x7fb25c15b870 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.064+0000 7fb25aa00640 1 -- 172.21.15.43:0/3778375054 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2540343f0 con 0x7fb25c15b870 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 -- 172.21.15.43:0/3778375054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c15b870 msgr2=0x7fb25c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c15b870 0x7fb25c0cc9f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb254009e50 tx=0x7fb25402be50 comp rx=0 tx=0).stop 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 -- 172.21.15.43:0/3778375054 shutdown_connections 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c15b870 0x7fb25c0cc9f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 --2- 172.21.15.43:0/3778375054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c0af8e0 0x7fb25c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 --2- 172.21.15.43:0/3778375054 >> v2:172.21.15.69:3300/0 conn(0x7fb25c143f10 0x7fb25c06a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 -- 172.21.15.43:0/3778375054 >> 172.21.15.43:0/3778375054 conn(0x7fb25c06d490 msgr2=0x7fb25c06d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 -- 172.21.15.43:0/3778375054 shutdown_connections 2024-01-16T15:04:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.065+0000 7fb263000640 1 -- 172.21.15.43:0/3778375054 wait complete. 2024-01-16T15:04:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.067+0000 7fb263000640 1 Processor -- start 2024-01-16T15:04:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.067+0000 7fb263000640 1 -- start start 2024-01-16T15:04:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c143f10 0x7fb25c1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25c15b870 0x7fb25c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1e45c0 con 0x7fb25c0af8e0 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1e48a0 con 0x7fb25c15b870 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb263000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1e4b80 con 0x7fb25c143f10 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46578/0 (socket says 172.21.15.43:46578) 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 -- 172.21.15.43:0/3363380498 learned_addr learned my addr 172.21.15.43:0/3363380498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 -- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c143f10 msgr2=0x7fb25c1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c143f10 0x7fb25c1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 -- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25c15b870 msgr2=0x7fb25c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25c15b870 0x7fb25c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.068+0000 7fb25be00640 1 -- 172.21.15.43:0/3363380498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb254009b30 con 0x7fb25c0af8e0 2024-01-16T15:04:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb25be00640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00c7f0 tx=0x7fb24c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c007a30 con 0x7fb25c0af8e0 2024-01-16T15:04:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c007b90 con 0x7fb25c0af8e0 2024-01-16T15:04:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c1e4e60 con 0x7fb25c0af8e0 2024-01-16T15:04:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c00ce40 con 0x7fb25c0af8e0 2024-01-16T15:04:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.069+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c15a860 con 0x7fb25c0af8e0 2024-01-16T15:04:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.070+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c067960 con 0x7fb25c0af8e0 2024-01-16T15:04:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.071+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb24c0113f0 con 0x7fb25c0af8e0 2024-01-16T15:04:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.072+0000 7fb253e00640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 0x7fb23406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.072+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb24c08a5a0 con 0x7fb25c0af8e0 2024-01-16T15:04:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.073+0000 7fb25b400640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 0x7fb23406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.073+0000 7fb25b400640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 0x7fb23406dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb244006fd0 tx=0x7fb244008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.077+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb24c054d30 con 0x7fb25c0af8e0 2024-01-16T15:04:40.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.309+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb25c1e5380 con 0x7fb25c0af8e0 2024-01-16T15:04:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.310+0000 7fb253e00640 1 -- 172.21.15.43:0/3363380498 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fb25c1e5380 con 0x7fb25c0af8e0 2024-01-16T15:04:40.289 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:40.289 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2588572053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2588572053},{"type":"v1","addr":"172.21.15.43:6837","nonce":2588572053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/791773791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":791773791},{"type":"v1","addr":"172.21.15.69:6833","nonce":791773791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3012792492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3012792492},{"type":"v1","addr":"172.21.15.69:6835","nonce":3012792492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/2054901873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2054901873},{"type":"v1","addr":"172.21.15.43:6836","nonce":2054901873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2024-01-16T15:04:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-01-16T15:04:40.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 msgr2=0x7fb23406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 0x7fb23406dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb244006fd0 tx=0x7fb244008040 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 msgr2=0x7fb25c15a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00c7f0 tx=0x7fb24c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 shutdown_connections 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb23406b860 0x7fb23406dd20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25c15b870 0x7fb25c0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25c143f10 0x7fb25c1f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 --2- 172.21.15.43:0/3363380498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25c0af8e0 0x7fb25c15a320 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 >> 172.21.15.43:0/3363380498 conn(0x7fb25c06d490 msgr2=0x7fb25c0afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 shutdown_connections 2024-01-16T15:04:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.313+0000 7fb263000640 1 -- 172.21.15.43:0/3363380498 wait complete. 2024-01-16T15:04:40.313 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:04:40.313 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:04:40.314 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:04:40.314 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:40.314 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:04:40.314 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:04:40.314 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:04:40.314 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:40.315 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:04:40.315 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:04:40.315 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:04:40.315 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:40.315 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:04:40.315 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:04:40.315 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:04:40.316 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:04:40.335 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:04:40.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:04:40.336 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:04:40.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:04:40.337 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:04:40.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:04:40.338 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:04:40.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:04:40.382 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:04:40.382 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:40.384 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:04:40.385 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:40.388 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:04:40.388 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:40.390 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:04:40.390 DEBUG:teuthology.parallel:result is None 2024-01-16T15:04:40.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:04:40.539 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:04:40.539 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:04:40.548 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:04:40.549 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:04:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.601+0000 7fa597600640 1 Processor -- start 2024-01-16T15:04:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.601+0000 7fa597600640 1 -- start start 2024-01-16T15:04:40.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa597600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 0x7fa5900b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa597600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa59015dc50 0x7fa59015e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa597600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5900ce230 con 0x7fa5900ccc00 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa597600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa590160fe0 con 0x7fa59015dc50 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa597600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa590161e40 con 0x7fa5900cddd0 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa594c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa5900ccc00 0x7fa5900ccfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45216/0 (socket says 172.21.15.43:45216) 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa594c00640 1 -- 172.21.15.43:0/3920892917 learned_addr learned my addr 172.21.15.43:0/3920892917 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa587e00640 1 --2- 172.21.15.43:0/3920892917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 0x7fa5900b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa595600640 1 --2- 172.21.15.43:0/3920892917 >> v2:172.21.15.69:3300/0 conn(0x7fa59015dc50 0x7fa59015e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.603+0000 7fa58fe00640 1 -- 172.21.15.43:0/3920892917 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619337165 0 0) 0x7fa5900ce230 con 0x7fa5900ccc00 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.604+0000 7fa58fe00640 1 -- 172.21.15.43:0/3920892917 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003520 con 0x7fa5900ccc00 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.604+0000 7fa587e00640 1 -- 172.21.15.43:0/3920892917 >> v2:172.21.15.69:3300/0 conn(0x7fa59015dc50 msgr2=0x7fa59015e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.604+0000 7fa587e00640 1 --2- 172.21.15.43:0/3920892917 >> v2:172.21.15.69:3300/0 conn(0x7fa59015dc50 0x7fa59015e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.604+0000 7fa587e00640 1 -- 172.21.15.43:0/3920892917 >> v1:172.21.15.43:6789/0 conn(0x7fa5900ccc00 legacy=0x7fa5900ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.604+0000 7fa587e00640 1 -- 172.21.15.43:0/3920892917 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5901611c0 con 0x7fa5900cddd0 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.605+0000 7fa587e00640 1 --2- 172.21.15.43:0/3920892917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 0x7fa5900b2aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5780097d0 tx=0x7fa57802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3267773603e9ce9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.605+0000 7fa58fe00640 1 -- 172.21.15.43:0/3920892917 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578031800 con 0x7fa5900cddd0 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.605+0000 7fa58fe00640 1 -- 172.21.15.43:0/3920892917 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa578031960 con 0x7fa5900cddd0 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.605+0000 7fa58fe00640 1 -- 172.21.15.43:0/3920892917 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578031cf0 con 0x7fa5900cddd0 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 -- 172.21.15.43:0/3920892917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 msgr2=0x7fa5900b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 --2- 172.21.15.43:0/3920892917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 0x7fa5900b2aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5780097d0 tx=0x7fa57802b8a0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 -- 172.21.15.43:0/3920892917 shutdown_connections 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 --2- 172.21.15.43:0/3920892917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5900cddd0 0x7fa5900b2aa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 --2- 172.21.15.43:0/3920892917 >> v2:172.21.15.69:3300/0 conn(0x7fa59015dc50 0x7fa59015e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 -- 172.21.15.43:0/3920892917 >> 172.21.15.43:0/3920892917 conn(0x7fa5900b23e0 msgr2=0x7fa5900cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 -- 172.21.15.43:0/3920892917 shutdown_connections 2024-01-16T15:04:40.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.606+0000 7fa597600640 1 -- 172.21.15.43:0/3920892917 wait complete. 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.608+0000 7fa597600640 1 Processor -- start 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.608+0000 7fa597600640 1 -- start start 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5900cddd0 0x7fa590156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa59015dc50 0x7fa590157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59006f210 con 0x7fa5900ccc00 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa590157060 con 0x7fa5900cddd0 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa597600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900aefb0 con 0x7fa59015dc50 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa594c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa594c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46612/0 (socket says 172.21.15.43:46612) 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.609+0000 7fa594c00640 1 -- 172.21.15.43:0/1663751813 learned_addr learned my addr 172.21.15.43:0/1663751813 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa587e00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5900cddd0 0x7fa590156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa595600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa59015dc50 0x7fa590157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 -- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa59015dc50 msgr2=0x7fa590157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa59015dc50 0x7fa590157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 -- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5900cddd0 msgr2=0x7fa590156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5900cddd0 0x7fa590156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 -- 172.21.15.43:0/1663751813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578009400 con 0x7fa5900ccc00 2024-01-16T15:04:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa594c00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c390 tx=0x7fa58000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.610+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580005ce0 con 0x7fa5900ccc00 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.611+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa580005e40 con 0x7fa5900ccc00 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.611+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5800072f0 con 0x7fa5900ccc00 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.611+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5900addf0 con 0x7fa5900ccc00 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.611+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59006ef30 con 0x7fa5900ccc00 2024-01-16T15:04:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.612+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590074770 con 0x7fa5900ccc00 2024-01-16T15:04:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.613+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa580007450 con 0x7fa5900ccc00 2024-01-16T15:04:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.614+0000 7fa58d600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 0x7fa56806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.614+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fa580089950 con 0x7fa5900ccc00 2024-01-16T15:04:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.614+0000 7fa587e00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 0x7fa56806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.615+0000 7fa587e00640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 0x7fa56806dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa578036000 tx=0x7fa57803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.618+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa580054220 con 0x7fa5900ccc00 2024-01-16T15:04:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.825+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa5900b0110 con 0x7fa5900ccc00 2024-01-16T15:04:40.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.879+0000 7fa58d600640 1 -- 172.21.15.43:0/1663751813 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa580057840 con 0x7fa5900ccc00 2024-01-16T15:04:40.859 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.882+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 msgr2=0x7fa56806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.882+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 0x7fa56806dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa578036000 tx=0x7fa57803c040 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.882+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 msgr2=0x7fa5900c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.882+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c390 tx=0x7fa58000c850 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 shutdown_connections 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa56806b860 0x7fa56806dd20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5900cddd0 0x7fa590156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa59015dc50 0x7fa590157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 --2- 172.21.15.43:0/1663751813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5900ccc00 0x7fa5900c4a10 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 >> 172.21.15.43:0/1663751813 conn(0x7fa5900b23e0 msgr2=0x7fa5900b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 shutdown_connections 2024-01-16T15:04:40.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:40.883+0000 7fa597600640 1 -- 172.21.15.43:0/1663751813 wait complete. 2024-01-16T15:04:40.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:04:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.117+0000 7f1399400640 1 Processor -- start 2024-01-16T15:04:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.117+0000 7f1399400640 1 -- start start 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.118+0000 7f1399400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f139415b050 0x7f139415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.118+0000 7f1399400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1394153f10 0x7f13940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.118+0000 7f1399400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13940ad6b0 con 0x7f139415d480 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1399400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1394154390 con 0x7f139415b050 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1399400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13940b25c0 con 0x7f1394153f10 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f139415b050 0x7f139415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f139415b050 0x7f139415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46618/0 (socket says 172.21.15.43:46618) 2024-01-16T15:04:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 -- 172.21.15.43:0/2890510755 learned_addr learned my addr 172.21.15.43:0/2890510755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1391e00640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1394153f10 0x7f13940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 -- 172.21.15.43:0/2890510755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1394153f10 msgr2=0x7f13940b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1394153f10 0x7f13940b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 -- 172.21.15.43:0/2890510755 >> v1:172.21.15.69:6789/0 conn(0x7f139415d480 legacy=0x7f139415d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.119+0000 7f1392800640 1 -- 172.21.15.43:0/2890510755 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940b2860 con 0x7f139415b050 2024-01-16T15:04:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.121+0000 7f1392800640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415b050 0x7f139415aaa0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1384009870 tx=0x7f138402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b43e790c121978a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.121+0000 7f1391400640 1 -- 172.21.15.43:0/2890510755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384031640 con 0x7f139415b050 2024-01-16T15:04:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.121+0000 7f1391400640 1 -- 172.21.15.43:0/2890510755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13840317a0 con 0x7f139415b050 2024-01-16T15:04:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.121+0000 7f1391400640 1 -- 172.21.15.43:0/2890510755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384031b30 con 0x7f139415b050 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 -- 172.21.15.43:0/2890510755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415b050 msgr2=0x7f139415aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415b050 0x7f139415aaa0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1384009870 tx=0x7f138402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 -- 172.21.15.43:0/2890510755 shutdown_connections 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1394153f10 0x7f13940b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 --2- 172.21.15.43:0/2890510755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415b050 0x7f139415aaa0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 -- 172.21.15.43:0/2890510755 >> 172.21.15.43:0/2890510755 conn(0x7f13940b0d80 msgr2=0x7f13940b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 -- 172.21.15.43:0/2890510755 shutdown_connections 2024-01-16T15:04:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.122+0000 7f1399400640 1 -- 172.21.15.43:0/2890510755 wait complete. 2024-01-16T15:04:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.124+0000 7f1399400640 1 Processor -- start 2024-01-16T15:04:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.124+0000 7f1399400640 1 -- start start 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1394153f10 0x7f139414c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f139415b050 0x7f139414eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139414f3b0 con 0x7f139415d480 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139414f690 con 0x7f1394153f10 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1399400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139414f970 con 0x7f139415b050 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1393200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1393200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46630/0 (socket says 172.21.15.43:46630) 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.125+0000 7f1393200640 1 -- 172.21.15.43:0/108977271 learned_addr learned my addr 172.21.15.43:0/108977271 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1391e00640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f139415b050 0x7f139414eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1392800640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1394153f10 0x7f139414c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 -- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f139415b050 msgr2=0x7f139414eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f139415b050 0x7f139414eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 -- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1394153f10 msgr2=0x7f139414c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1394153f10 0x7f139414c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 -- 172.21.15.43:0/108977271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13840094d0 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.126+0000 7f1393200640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f138c00b970 tx=0x7f138c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.127+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c00cb40 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.127+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c00cca0 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.127+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139414fc50 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.127+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c014680 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.127+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13940cc3f0 con 0x7f139415d480 2024-01-16T15:04:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.128+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13940cc670 con 0x7f139415d480 2024-01-16T15:04:41.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.129+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f138c01d3b0 con 0x7f139415d480 2024-01-16T15:04:41.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.130+0000 7f138aa00640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 0x7f136c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.130+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f138c08a160 con 0x7f139415d480 2024-01-16T15:04:41.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.130+0000 7f1392800640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 0x7f136c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.131+0000 7f1392800640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 0x7f136c06dd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1384036000 tx=0x7f138403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.134+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f138c054a50 con 0x7f139415d480 2024-01-16T15:04:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.395+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f139414fe10 con 0x7f139415d480 2024-01-16T15:04:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.396+0000 7f138aa00640 1 -- 172.21.15.43:0/108977271 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f138c058070 con 0x7f139415d480 2024-01-16T15:04:41.375 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:41.375 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.376 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:04:41.377 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:04:41.378 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:04:41.379 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.381 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.382 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.384 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:41.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:04:41.388 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 msgr2=0x7f136c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 0x7f136c06dd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1384036000 tx=0x7f138403c040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 msgr2=0x7f1394150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f138c00b970 tx=0x7f138c00be30 comp rx=0 tx=0).stop 2024-01-16T15:04:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 shutdown_connections 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f136c06b890 0x7f136c06dd50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139415d480 0x7f1394150c80 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f139415b050 0x7f139414eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 --2- 172.21.15.43:0/108977271 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1394153f10 0x7f139414c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 >> 172.21.15.43:0/108977271 conn(0x7f13940b0d80 msgr2=0x7f1394158960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 shutdown_connections 2024-01-16T15:04:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.399+0000 7f1399400640 1 -- 172.21.15.43:0/108977271 wait complete. 2024-01-16T15:04:41.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.641+0000 7f58bd800640 1 Processor -- start 2024-01-16T15:04:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.642+0000 7f58bd800640 1 -- start start 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b8143570 0x7f58b80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58b8143950 0x7f58b80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b815e6b0 0x7f58b815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b8071700 con 0x7f58b8143570 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b815b410 con 0x7f58b8143950 2024-01-16T15:04:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.643+0000 7f58bd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b815b130 con 0x7f58b815e6b0 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.644+0000 7f58b7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b8143570 0x7f58b80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.644+0000 7f58b7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b8143570 0x7f58b80cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46634/0 (socket says 172.21.15.43:46634) 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.644+0000 7f58b7e00640 1 -- 172.21.15.43:0/4246002560 learned_addr learned my addr 172.21.15.43:0/4246002560 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.644+0000 7f58bce00640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b815e6b0 0x7f58b815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.644+0000 7f58afe00640 1 --2- 172.21.15.43:0/4246002560 >> v2:172.21.15.69:3300/0 conn(0x7f58b8143950 0x7f58b80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.645+0000 7f58b7e00640 1 -- 172.21.15.43:0/4246002560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b815e6b0 msgr2=0x7f58b815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.645+0000 7f58b7e00640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b815e6b0 0x7f58b815a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.645+0000 7f58b7e00640 1 -- 172.21.15.43:0/4246002560 >> v2:172.21.15.69:3300/0 conn(0x7f58b8143950 msgr2=0x7f58b80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.645+0000 7f58b7e00640 1 --2- 172.21.15.43:0/4246002560 >> v2:172.21.15.69:3300/0 conn(0x7f58b8143950 0x7f58b80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.645+0000 7f58b7e00640 1 -- 172.21.15.43:0/4246002560 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b815dbd0 con 0x7f58b8143570 2024-01-16T15:04:41.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.648+0000 7f58b7e00640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b80cf050 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f58b8070f10 tx=0x7f58a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd6cbca185f88e63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.648+0000 7f58b7400640 1 -- 172.21.15.43:0/4246002560 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80317e0 con 0x7f58b8143570 2024-01-16T15:04:41.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.648+0000 7f58b7400640 1 -- 172.21.15.43:0/4246002560 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8031940 con 0x7f58b8143570 2024-01-16T15:04:41.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.648+0000 7f58b7400640 1 -- 172.21.15.43:0/4246002560 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8031cd0 con 0x7f58b8143570 2024-01-16T15:04:41.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 -- 172.21.15.43:0/4246002560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 msgr2=0x7f58b80cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b80cf050 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f58b8070f10 tx=0x7f58a802b880 comp rx=0 tx=0).stop 2024-01-16T15:04:41.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 -- 172.21.15.43:0/4246002560 shutdown_connections 2024-01-16T15:04:41.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b815e6b0 0x7f58b815a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 --2- 172.21.15.43:0/4246002560 >> v2:172.21.15.69:3300/0 conn(0x7f58b8143950 0x7f58b80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 --2- 172.21.15.43:0/4246002560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b80cf050 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 -- 172.21.15.43:0/4246002560 >> 172.21.15.43:0/4246002560 conn(0x7f58b806a860 msgr2=0x7f58b806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 -- 172.21.15.43:0/4246002560 shutdown_connections 2024-01-16T15:04:41.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.649+0000 7f58bd800640 1 -- 172.21.15.43:0/4246002560 wait complete. 2024-01-16T15:04:41.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.651+0000 7f58bd800640 1 Processor -- start 2024-01-16T15:04:41.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.651+0000 7f58bd800640 1 -- start start 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b8143950 0x7f58b81f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b815e6b0 0x7f58b813ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8159e60 con 0x7f58b8143570 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80c91b0 con 0x7f58b815e6b0 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.652+0000 7f58bd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80c9490 con 0x7f58b8143950 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46636/0 (socket says 172.21.15.43:46636) 2024-01-16T15:04:41.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 -- 172.21.15.43:0/3635146805 learned_addr learned my addr 172.21.15.43:0/3635146805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:41.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58bce00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b815e6b0 0x7f58b813ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58afe00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b8143950 0x7f58b81f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 -- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b8143950 msgr2=0x7f58b81f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b8143950 0x7f58b81f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 -- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b815e6b0 msgr2=0x7f58b813ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b815e6b0 0x7f58b813ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.653+0000 7f58b7e00640 1 -- 172.21.15.43:0/3635146805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80094d0 con 0x7f58b8143570 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.654+0000 7f58b7e00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f58a8000c00 tx=0x7f58a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.654+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8032880 con 0x7f58b8143570 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.654+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a80329e0 con 0x7f58b8143570 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.654+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b80c9770 con 0x7f58b8143570 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.654+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8032d90 con 0x7f58b8143570 2024-01-16T15:04:41.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.655+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b80c9d30 con 0x7f58b8143570 2024-01-16T15:04:41.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.656+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b80c9f20 con 0x7f58b8143570 2024-01-16T15:04:41.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.657+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f58a804e450 con 0x7f58b8143570 2024-01-16T15:04:41.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.658+0000 7f58b4c00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 0x7f589006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:41.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.658+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f58a80ac390 con 0x7f58b8143570 2024-01-16T15:04:41.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.658+0000 7f58afe00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 0x7f589006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:41.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.659+0000 7f58afe00640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 0x7f589006dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f58b813c1f0 tx=0x7f58a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.663+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58a8076dc0 con 0x7f58b8143570 2024-01-16T15:04:41.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.857+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58b80c95d0 con 0x7f58b8143570 2024-01-16T15:04:41.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.859+0000 7f58b4c00640 1 -- 172.21.15.43:0/3635146805 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+12852 (secure 0 0 0) 0x7f58b80c95d0 con 0x7f58b8143570 2024-01-16T15:04:41.839 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:41.839 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":57,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:40.890701+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 msgr2=0x7f589006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 0x7f589006dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f58b813c1f0 tx=0x7f58a0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 msgr2=0x7f58b8159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f58a8000c00 tx=0x7f58a80046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 shutdown_connections 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f589006b860 0x7f589006dd20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b815e6b0 0x7f58b813ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b8143950 0x7f58b81f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 --2- 172.21.15.43:0/3635146805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b8143570 0x7f58b8159370 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.862+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 >> 172.21.15.43:0/3635146805 conn(0x7f58b806a860 msgr2=0x7f58b80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.863+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 shutdown_connections 2024-01-16T15:04:41.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:41.863+0000 7f58bd800640 1 -- 172.21.15.43:0/3635146805 wait complete. 2024-01-16T15:04:41.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:04:41.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:04:42.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.104+0000 7f4b9e200640 1 Processor -- start 2024-01-16T15:04:42.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.104+0000 7f4b9e200640 1 -- start start 2024-01-16T15:04:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4b9815b350 0x7f4b980abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b9809cb30 0x7f4b980b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4b980adb60 0x7f4b980adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9815b730 con 0x7f4b9815b350 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980ac740 con 0x7f4b980adb60 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.106+0000 7f4b9e200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980ac460 con 0x7f4b9809cb30 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4b9815b350 0x7f4b980abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4b9815b350 0x7f4b980abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46652/0 (socket says 172.21.15.43:46652) 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 -- 172.21.15.43:0/1829488912 learned_addr learned my addr 172.21.15.43:0/1829488912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b8fe00640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b9809cb30 0x7f4b980b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97e00640 1 --2- 172.21.15.43:0/1829488912 >> v2:172.21.15.69:3300/0 conn(0x7f4b980adb60 0x7f4b980adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 -- 172.21.15.43:0/1829488912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b9809cb30 msgr2=0x7f4b980b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b9809cb30 0x7f4b980b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 -- 172.21.15.43:0/1829488912 >> v2:172.21.15.69:3300/0 conn(0x7f4b980adb60 msgr2=0x7f4b980adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 --2- 172.21.15.43:0/1829488912 >> v2:172.21.15.69:3300/0 conn(0x7f4b980adb60 0x7f4b980adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.107+0000 7f4b97400640 1 -- 172.21.15.43:0/1829488912 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980b2a50 con 0x7f4b9815b350 2024-01-16T15:04:42.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.108+0000 7f4b97400640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b980abe80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4b98072730 tx=0x7f4b8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309a3aacf89d38d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.109+0000 7f4b96a00640 1 -- 172.21.15.43:0/1829488912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88031870 con 0x7f4b9815b350 2024-01-16T15:04:42.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.109+0000 7f4b96a00640 1 -- 172.21.15.43:0/1829488912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b880319d0 con 0x7f4b9815b350 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.109+0000 7f4b96a00640 1 -- 172.21.15.43:0/1829488912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88031d60 con 0x7f4b9815b350 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.109+0000 7f4b9e200640 1 -- 172.21.15.43:0/1829488912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 msgr2=0x7f4b980abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.109+0000 7f4b9e200640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b980abe80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4b98072730 tx=0x7f4b8802b910 comp rx=0 tx=0).stop 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 -- 172.21.15.43:0/1829488912 shutdown_connections 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 --2- 172.21.15.43:0/1829488912 >> v2:172.21.15.69:3300/0 conn(0x7f4b980adb60 0x7f4b980adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b9809cb30 0x7f4b980b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 --2- 172.21.15.43:0/1829488912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b980abe80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 -- 172.21.15.43:0/1829488912 >> 172.21.15.43:0/1829488912 conn(0x7f4b980b0ba0 msgr2=0x7f4b980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 -- 172.21.15.43:0/1829488912 shutdown_connections 2024-01-16T15:04:42.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.110+0000 7f4b9e200640 1 -- 172.21.15.43:0/1829488912 wait complete. 2024-01-16T15:04:42.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.112+0000 7f4b9e200640 1 Processor -- start 2024-01-16T15:04:42.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.112+0000 7f4b9e200640 1 -- start start 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 0x7f4b98151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b980adb60 0x7f4b9814ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98151fa0 con 0x7f4b9815b350 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9814e410 con 0x7f4b9809cb30 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b9e200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98155ad0 con 0x7f4b980adb60 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b97e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b97e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46656/0 (socket says 172.21.15.43:46656) 2024-01-16T15:04:42.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b97e00640 1 -- 172.21.15.43:0/2203262776 learned_addr learned my addr 172.21.15.43:0/2203262776 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b97400640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 0x7f4b98151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.114+0000 7f4b8fe00640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b980adb60 0x7f4b9814ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 -- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b980adb60 msgr2=0x7f4b9814ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b980adb60 0x7f4b9814ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 -- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 msgr2=0x7f4b98151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 0x7f4b98151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 -- 172.21.15.43:0/2203262776 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880094d0 con 0x7f4b9815b350 2024-01-16T15:04:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97400640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 0x7f4b98151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b97e00640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4b90007e40 tx=0x7f4b9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90014620 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9000ee90 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.115+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900149e0 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.116+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b98155db0 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.116+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b98156370 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.117+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9809d200 con 0x7f4b9815b350 2024-01-16T15:04:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.118+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4b90014b40 con 0x7f4b9815b350 2024-01-16T15:04:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.119+0000 7f4b8f400640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 0x7f4b7006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.119+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f4b9008ac10 con 0x7f4b9815b350 2024-01-16T15:04:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.119+0000 7f4b97400640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 0x7f4b7006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.120+0000 7f4b97400640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 0x7f4b7006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b88036000 tx=0x7f4b8803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.124+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b90055590 con 0x7f4b9815b350 2024-01-16T15:04:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.317+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b9804e920 con 0x7f4b9815b350 2024-01-16T15:04:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.317+0000 7f4b8f400640 1 -- 172.21.15.43:0/2203262776 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b90058bb0 con 0x7f4b9815b350 2024-01-16T15:04:42.297 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 msgr2=0x7f4b7006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 0x7f4b7006dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b88036000 tx=0x7f4b8803c040 comp rx=0 tx=0).stop 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 msgr2=0x7f4b9814e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4b90007e40 tx=0x7f4b9000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 shutdown_connections 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4b7006b860 0x7f4b7006dd20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b9815b350 0x7f4b9814e760 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b980adb60 0x7f4b9814ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 --2- 172.21.15.43:0/2203262776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b9809cb30 0x7f4b98151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 >> 172.21.15.43:0/2203262776 conn(0x7f4b980b0ba0 msgr2=0x7f4b9806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 shutdown_connections 2024-01-16T15:04:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.320+0000 7f4b9e200640 1 -- 172.21.15.43:0/2203262776 wait complete. 2024-01-16T15:04:42.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:04:42.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.559+0000 7ff176400640 1 Processor -- start 2024-01-16T15:04:42.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.559+0000 7ff176400640 1 -- start start 2024-01-16T15:04:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.561+0000 7ff176400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17015a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.561+0000 7ff176400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff170143950 0x7ff17015a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.561+0000 7ff176400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff170071700 con 0x7ff170143570 2024-01-16T15:04:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff176400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff17015af30 con 0x7ff170143950 2024-01-16T15:04:42.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff176400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17015b410 con 0x7ff1700cf760 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff170143950 0x7ff17015a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff170143950 0x7ff17015a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46658/0 (socket says 172.21.15.43:46658) 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 -- 172.21.15.43:0/4197154621 learned_addr learned my addr 172.21.15.43:0/4197154621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff167e00640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17015a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 -- 172.21.15.43:0/4197154621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 msgr2=0x7ff17015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17015a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 -- 172.21.15.43:0/4197154621 >> v1:172.21.15.43:6789/0 conn(0x7ff170143570 legacy=0x7ff1700cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.562+0000 7ff175a00640 1 -- 172.21.15.43:0/4197154621 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17015b130 con 0x7ff170143950 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.564+0000 7ff175a00640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff17015a9f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d150 tx=0x7ff16802e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fa3613ec033d73 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.564+0000 7ff16fe00640 1 -- 172.21.15.43:0/4197154621 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16802ea70 con 0x7ff170143950 2024-01-16T15:04:42.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.564+0000 7ff16fe00640 1 -- 172.21.15.43:0/4197154621 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16802ebd0 con 0x7ff170143950 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.564+0000 7ff16fe00640 1 -- 172.21.15.43:0/4197154621 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168009600 con 0x7ff170143950 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 -- 172.21.15.43:0/4197154621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 msgr2=0x7ff17015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff17015a9f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d150 tx=0x7ff16802e0d0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 -- 172.21.15.43:0/4197154621 shutdown_connections 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff17015a9f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 --2- 172.21.15.43:0/4197154621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17015a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 -- 172.21.15.43:0/4197154621 >> 172.21.15.43:0/4197154621 conn(0x7ff17006a860 msgr2=0x7ff17006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 -- 172.21.15.43:0/4197154621 shutdown_connections 2024-01-16T15:04:42.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.565+0000 7ff176400640 1 -- 172.21.15.43:0/4197154621 wait complete. 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.567+0000 7ff176400640 1 Processor -- start 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.567+0000 7ff176400640 1 -- start start 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17013d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff170143570 0x7ff1700cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700cc7b0 con 0x7ff170143950 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700cca90 con 0x7ff170143570 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff176400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700ccd70 con 0x7ff1700cf760 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46670/0 (socket says 172.21.15.43:46670) 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175a00640 1 -- 172.21.15.43:0/2939900262 learned_addr learned my addr 172.21.15.43:0/2939900262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175000640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17013d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff167e00640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff170143570 0x7ff1700cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175a00640 1 -- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 msgr2=0x7ff17013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.568+0000 7ff175a00640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17013d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff175a00640 1 -- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff170143570 msgr2=0x7ff1700cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff175a00640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff170143570 0x7ff1700cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff175a00640 1 -- 172.21.15.43:0/2939900262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16800bd40 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff175a00640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d150 tx=0x7ff168004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16802ea70 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16800b720 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1680408a0 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1700cd050 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.569+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff170159c20 con 0x7ff170143950 2024-01-16T15:04:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.571+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17004e840 con 0x7ff170143950 2024-01-16T15:04:42.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.572+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff168040a00 con 0x7ff170143950 2024-01-16T15:04:42.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.573+0000 7ff16d600640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 0x7ff14806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.573+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4885+0+0 (secure 0 0 0) 0x7ff1680af300 con 0x7ff170143950 2024-01-16T15:04:42.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.573+0000 7ff175000640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 0x7ff14806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.574+0000 7ff175000640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 0x7ff14806dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1700703d0 tx=0x7ff160009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.577+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff168079d20 con 0x7ff170143950 2024-01-16T15:04:42.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.773+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff1700b3910 con 0x7ff170143950 2024-01-16T15:04:42.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.774+0000 7ff16d600640 1 -- 172.21.15.43:0/2939900262 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff168040cb0 con 0x7ff170143950 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:42.754 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 msgr2=0x7ff14806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 0x7ff14806dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1700703d0 tx=0x7ff160009040 comp rx=0 tx=0).stop 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 msgr2=0x7ff1701596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d150 tx=0x7ff168004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 shutdown_connections 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff14806b860 0x7ff14806dd20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff170143570 0x7ff1700cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff170143950 0x7ff1701596e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 --2- 172.21.15.43:0/2939900262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1700cf760 0x7ff17013d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.777+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 >> 172.21.15.43:0/2939900262 conn(0x7ff17006a860 msgr2=0x7ff17006ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.778+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 shutdown_connections 2024-01-16T15:04:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:42.778+0000 7ff176400640 1 -- 172.21.15.43:0/2939900262 wait complete. 2024-01-16T15:04:42.777 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:04:42.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:04:43.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.031+0000 7f834c400640 1 Processor -- start 2024-01-16T15:04:43.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.032+0000 7f834c400640 1 -- start start 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.033+0000 7f834c400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f834415b120 0x7f83440cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.033+0000 7f834c400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f83440cf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.033+0000 7f834c400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83440cbd70 con 0x7f83440cf680 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.033+0000 7f834c400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83440cb7b0 con 0x7f834415b120 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.033+0000 7f834c400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83440cba90 con 0x7f8344070a60 2024-01-16T15:04:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f834a400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f83440cf680 0x7f834415ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56210/0 (socket says 172.21.15.43:56210) 2024-01-16T15:04:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f834a400640 1 -- 172.21.15.43:0/3307594514 learned_addr learned my addr 172.21.15.43:0/3307594514 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f8349a00640 1 --2- 172.21.15.43:0/3307594514 >> v2:172.21.15.69:3300/0 conn(0x7f834415b120 0x7f83440cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f8349000640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f83440cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012967591 0 0) 0x7f83440cbd70 con 0x7f83440cf680 2024-01-16T15:04:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.034+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8324003520 con 0x7f83440cf680 2024-01-16T15:04:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986718180 0 0) 0x7f8324003520 con 0x7f83440cf680 2024-01-16T15:04:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 msgr2=0x7f83440cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f83440cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 >> v2:172.21.15.69:3300/0 conn(0x7f834415b120 msgr2=0x7f83440cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 --2- 172.21.15.43:0/3307594514 >> v2:172.21.15.69:3300/0 conn(0x7f834415b120 0x7f83440cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83440d0840 con 0x7f83440cf680 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f833c0019f0 con 0x7f83440cf680 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 >> v1:172.21.15.43:6789/0 conn(0x7f83440cf680 legacy=0x7f834415ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f8324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440cb7b0 con 0x7f8324003970 2024-01-16T15:04:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440cba90 con 0x7f83240047c0 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832400b9d0 con 0x7f8344070a60 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8349a00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f834a400640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f8324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.035+0000 7f8349000640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.036+0000 7f8349a00640 1 -- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 msgr2=0x7f8324005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.036+0000 7f8349a00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8344070a60 0x7f8324005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.036+0000 7f8349a00640 1 -- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83240047c0 msgr2=0x7f8324004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.036+0000 7f8349a00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.036+0000 7f8349a00640 1 -- 172.21.15.43:0/3307594514 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324003520 con 0x7f8324003970 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.037+0000 7f8349a00640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8324003d90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8334009af0 tx=0x7f83240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a705ba77361583e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.037+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402c710 con 0x7f8324003970 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.037+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833402c870 con 0x7f8324003970 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.037+0000 7f8343e00640 1 -- 172.21.15.43:0/3307594514 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402cc00 con 0x7f8324003970 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 -- 172.21.15.43:0/3307594514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 msgr2=0x7f8324003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 --2- 172.21.15.43:0/3307594514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8324003d90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8334009af0 tx=0x7f83240057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834a400640 1 -- 172.21.15.43:0/3307594514 reap_dead start 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 -- 172.21.15.43:0/3307594514 shutdown_connections 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 -- 172.21.15.43:0/3307594514 >> 172.21.15.43:0/3307594514 conn(0x7f83440b2ba0 msgr2=0x7f83440b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 -- 172.21.15.43:0/3307594514 shutdown_connections 2024-01-16T15:04:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.038+0000 7f834c400640 1 -- 172.21.15.43:0/3307594514 wait complete. 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.040+0000 7f834c400640 1 Processor -- start 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.040+0000 7f834c400640 1 -- start start 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83441e45c0 0x7f83440caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83441551c0 0x7f83440b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83441e4a40 con 0x7f8324003970 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83441563e0 con 0x7f83441551c0 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834c400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83441560b0 con 0x7f83441e45c0 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50718/0 (socket says 172.21.15.43:50718) 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 -- 172.21.15.43:0/899952538 learned_addr learned my addr 172.21.15.43:0/899952538 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349000640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83441e45c0 0x7f83440caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f834a400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83441551c0 0x7f83440b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 -- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83441e45c0 msgr2=0x7f83440caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83441e45c0 0x7f83440caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 -- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83441551c0 msgr2=0x7f83440b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.041+0000 7f8349a00640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83441551c0 0x7f83440b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f8349a00640 1 -- 172.21.15.43:0/899952538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8344158420 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f8349a00640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8334009af0 tx=0x7f8334004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003810 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833402de90 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003b70 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.042+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8344158700 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.043+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83441589e0 con 0x7f8324003970 2024-01-16T15:04:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.044+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83441e49a0 con 0x7f8324003970 2024-01-16T15:04:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.045+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8334003cd0 con 0x7f8324003970 2024-01-16T15:04:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.045+0000 7f8341600640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 0x7f831c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.046+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f83340ac190 con 0x7f8324003970 2024-01-16T15:04:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.046+0000 7f8349000640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 0x7f831c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.047+0000 7f8349000640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 0x7f831c06dda0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83440cb200 tx=0x7f832c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.051+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8334076cd0 con 0x7f8324003970 2024-01-16T15:04:43.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.247+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f83441561f0 con 0x7f8324003970 2024-01-16T15:04:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.901+0000 7f8341600640 1 -- 172.21.15.43:0/899952538 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) v1 ==== 138+0+0 (secure 0 0 0) 0x7f833407a2f0 con 0x7f8324003970 2024-01-16T15:04:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.905+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 msgr2=0x7f831c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.905+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 0x7f831c06dda0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83440cb200 tx=0x7f832c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.905+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 msgr2=0x7f8344157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.905+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8334009af0 tx=0x7f8334004730 comp rx=0 tx=0).stop 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 shutdown_connections 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831c06b8e0 0x7f831c06dda0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83441551c0 0x7f83440b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83441e45c0 0x7f83440caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 --2- 172.21.15.43:0/899952538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8324003970 0x7f8344157e40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 >> 172.21.15.43:0/899952538 conn(0x7f83440b2ba0 msgr2=0x7f834415d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.906+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 shutdown_connections 2024-01-16T15:04:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:43.907+0000 7f834c400640 1 -- 172.21.15.43:0/899952538 wait complete. 2024-01-16T15:04:43.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:04:44.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.152+0000 7ff1a6a00640 1 Processor -- start 2024-01-16T15:04:44.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.152+0000 7ff1a6a00640 1 -- start start 2024-01-16T15:04:44.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1a00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff1a00d1d90 0x7ff1a00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a6a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a00c6bf0 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a6a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00d1400 con 0x7ff1a00d1d90 2024-01-16T15:04:44.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00cedc0 con 0x7ff1a015d3b0 2024-01-16T15:04:44.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a4c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff1a00d0fa0 0x7ff1a00d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56218/0 (socket says 172.21.15.43:56218) 2024-01-16T15:04:44.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.154+0000 7ff1a4c00640 1 -- 172.21.15.43:0/1338645620 learned_addr learned my addr 172.21.15.43:0/1338645620 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff1a5600640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1a00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff1a6000640 1 --2- 172.21.15.43:0/1338645620 >> v2:172.21.15.69:3300/0 conn(0x7ff1a00d1d90 0x7ff1a00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363205740 0 0) 0x7ff1a00c6bf0 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff180003520 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259094492 0 0) 0x7ff180003520 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 msgr2=0x7ff1a00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1a00ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 >> v2:172.21.15.69:3300/0 conn(0x7ff1a00d1d90 msgr2=0x7ff1a00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 --2- 172.21.15.43:0/1338645620 >> v2:172.21.15.69:3300/0 conn(0x7ff1a00d1d90 0x7ff1a00b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00c6e40 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff1880019f0 con 0x7ff1a00d0fa0 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 >> v1:172.21.15.43:6789/0 conn(0x7ff1a00d0fa0 legacy=0x7ff1a00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1800039e0 0x7ff180003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180004d00 0x7ff180005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00d1400 con 0x7ff1800039e0 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00cedc0 con 0x7ff180004d00 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.155+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18000b9d0 con 0x7ff1a015d3b0 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1800039e0 0x7ff180003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a4c00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a6000640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180004d00 0x7ff180005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 -- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 msgr2=0x7ff1800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1a015d3b0 0x7ff1800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 -- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180004d00 msgr2=0x7ff180005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180004d00 0x7ff180005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.156+0000 7ff1a5600640 1 -- 172.21.15.43:0/1338645620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180003520 con 0x7ff1800039e0 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.157+0000 7ff1a5600640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1800039e0 0x7ff180003e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff190009ac0 tx=0x7ff1800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1f71a1364002f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.157+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002c6b0 con 0x7ff1800039e0 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.157+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19002c810 con 0x7ff1800039e0 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.157+0000 7ff19fe00640 1 -- 172.21.15.43:0/1338645620 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002cba0 con 0x7ff1800039e0 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 -- 172.21.15.43:0/1338645620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1800039e0 msgr2=0x7ff180003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/1338645620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1800039e0 0x7ff180003e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff190009ac0 tx=0x7ff1800057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6000640 1 -- 172.21.15.43:0/1338645620 reap_dead start 2024-01-16T15:04:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 -- 172.21.15.43:0/1338645620 shutdown_connections 2024-01-16T15:04:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 -- 172.21.15.43:0/1338645620 >> 172.21.15.43:0/1338645620 conn(0x7ff1a00d30a0 msgr2=0x7ff1a00d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 -- 172.21.15.43:0/1338645620 shutdown_connections 2024-01-16T15:04:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.158+0000 7ff1a6a00640 1 -- 172.21.15.43:0/1338645620 wait complete. 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.160+0000 7ff1a6a00640 1 Processor -- start 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.160+0000 7ff1a6a00640 1 -- start start 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1800039e0 0x7ff1a01597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a00a0560 0x7ff1a00a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a009e070 con 0x7ff1a00cad80 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a009e350 con 0x7ff1a00a0560 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a009e630 con 0x7ff1800039e0 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.161+0000 7ff1a4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50740/0 (socket says 172.21.15.43:50740) 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 -- 172.21.15.43:0/2635055660 learned_addr learned my addr 172.21.15.43:0/2635055660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a5600640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1800039e0 0x7ff1a01597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a6000640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a00a0560 0x7ff1a00a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 -- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1800039e0 msgr2=0x7ff1a01597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1800039e0 0x7ff1a01597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 -- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a00a0560 msgr2=0x7ff1a00a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a00a0560 0x7ff1a00a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 -- 172.21.15.43:0/2635055660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00a0040 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.162+0000 7ff1a4c00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff188007a80 tx=0x7ff18800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.163+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800ee50 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.163+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188013420 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.163+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a009dd80 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.163+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1880137d0 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.163+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a00cbae0 con 0x7ff1a00cad80 2024-01-16T15:04:44.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.164+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a00ce6a0 con 0x7ff1a00cad80 2024-01-16T15:04:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.166+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff188013930 con 0x7ff1a00cad80 2024-01-16T15:04:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.167+0000 7ff19d600640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 0x7ff17806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.167+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5255+0+0 (secure 0 0 0) 0x7ff18808b4b0 con 0x7ff1a00cad80 2024-01-16T15:04:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.167+0000 7ff1a5600640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 0x7ff17806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.168+0000 7ff1a5600640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 0x7ff17806dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff190036000 tx=0x7ff190030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.171+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff188055e00 con 0x7ff1a00cad80 2024-01-16T15:04:44.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.364+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff1a00ce6a0 con 0x7ff1a00cad80 2024-01-16T15:04:44.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.917+0000 7ff19d600640 1 -- 172.21.15.43:0/2635055660 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff1a00ce6a0 con 0x7ff1a00cad80 2024-01-16T15:04:44.897 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:04:44.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 msgr2=0x7ff17806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 0x7ff17806dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff190036000 tx=0x7ff190030000 comp rx=0 tx=0).stop 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 msgr2=0x7ff1a009d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff188007a80 tx=0x7ff18800e530 comp rx=0 tx=0).stop 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 shutdown_connections 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff17806b860 0x7ff17806dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1a00a0560 0x7ff1a00a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a00cad80 0x7ff1a009d840 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 --2- 172.21.15.43:0/2635055660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1800039e0 0x7ff1a01597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 >> 172.21.15.43:0/2635055660 conn(0x7ff1a00d30a0 msgr2=0x7ff1a006c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 shutdown_connections 2024-01-16T15:04:44.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:44.922+0000 7ff1a6a00640 1 -- 172.21.15.43:0/2635055660 wait complete. 2024-01-16T15:04:44.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:04:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.147+0000 7ff386600640 1 Processor -- start 2024-01-16T15:04:45.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.147+0000 7ff386600640 1 -- start start 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff386600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 0x7ff38015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff386600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38015f930 con 0x7ff3800b35a0 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff386600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800b3b10 con 0x7ff38015f0a0 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff386600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800b43b0 con 0x7ff380144770 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff377e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff3800b35a0 0x7ff38006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56230/0 (socket says 172.21.15.43:56230) 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.149+0000 7ff377e00640 1 -- 172.21.15.43:0/3482756504 learned_addr learned my addr 172.21.15.43:0/3482756504 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff385200640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 0x7ff38015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359740535 0 0) 0x7ff38015f930 con 0x7ff3800b35a0 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff360003520 con 0x7ff3800b35a0 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 422102561 0 0) 0x7ff360003520 con 0x7ff3800b35a0 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 msgr2=0x7ff38015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 0x7ff38015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 >> v1:172.21.15.69:6789/0 conn(0x7ff38015f0a0 legacy=0x7ff3800ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800b3df0 con 0x7ff3800b35a0 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff368001670 con 0x7ff3800b35a0 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.150+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 >> v1:172.21.15.43:6789/0 conn(0x7ff3800b35a0 legacy=0x7ff38006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff360003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 0x7ff360005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800b43b0 con 0x7ff360003a10 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36000ba80 con 0x7ff3600047c0 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36000bc40 con 0x7ff380144770 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff360003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff377e00640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 -- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 msgr2=0x7ff360005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:45.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380144770 0x7ff360005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 -- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600047c0 msgr2=0x7ff360004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:45.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.151+0000 7ff385200640 1 -- 172.21.15.43:0/3482756504 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360003520 con 0x7ff360003a10 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.153+0000 7ff385200640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff360003e30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff370009ac0 tx=0x7ff3600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28258eaed2df366b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.153+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37002c710 con 0x7ff360003a10 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.153+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37002c870 con 0x7ff360003a10 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.153+0000 7ff37fe00640 1 -- 172.21.15.43:0/3482756504 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37002cc00 con 0x7ff360003a10 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 -- 172.21.15.43:0/3482756504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 msgr2=0x7ff360003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 --2- 172.21.15.43:0/3482756504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff360003e30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff370009ac0 tx=0x7ff3600057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff385c00640 1 -- 172.21.15.43:0/3482756504 reap_dead start 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 -- 172.21.15.43:0/3482756504 shutdown_connections 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 -- 172.21.15.43:0/3482756504 >> 172.21.15.43:0/3482756504 conn(0x7ff3800cec10 msgr2=0x7ff3800cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 -- 172.21.15.43:0/3482756504 shutdown_connections 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.154+0000 7ff386600640 1 -- 172.21.15.43:0/3482756504 wait complete. 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.156+0000 7ff386600640 1 Processor -- start 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.156+0000 7ff386600640 1 -- start start 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 0x7ff3801424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 0x7ff380138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380142ea0 con 0x7ff360003a10 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801f4890 con 0x7ff3800c4640 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff386600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800cbfc0 con 0x7ff380142a20 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff385200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff385200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50766/0 (socket says 172.21.15.43:50766) 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff385200640 1 -- 172.21.15.43:0/2763650289 learned_addr learned my addr 172.21.15.43:0/2763650289 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff377e00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 0x7ff3801424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff385c00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 0x7ff380138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.157+0000 7ff385200640 1 -- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 msgr2=0x7ff380138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385200640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 0x7ff380138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385200640 1 -- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 msgr2=0x7ff3801424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385200640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 0x7ff3801424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385200640 1 -- 172.21.15.43:0/2763650289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38015aed0 con 0x7ff360003a10 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff377e00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 0x7ff3801424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385c00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 0x7ff380138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff385200640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff37002c2b0 tx=0x7ff370004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370003810 con 0x7ff360003a10 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37002de90 con 0x7ff360003a10 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3801f4d20 con 0x7ff360003a10 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370003b70 con 0x7ff360003a10 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.158+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3801f4240 con 0x7ff360003a10 2024-01-16T15:04:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.160+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38004e840 con 0x7ff360003a10 2024-01-16T15:04:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.161+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff370003cd0 con 0x7ff360003a10 2024-01-16T15:04:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.162+0000 7ff37d600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 0x7ff35806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.162+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5637+0+0 (secure 0 0 0) 0x7ff3700ac470 con 0x7ff360003a10 2024-01-16T15:04:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.163+0000 7ff377e00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 0x7ff35806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.163+0000 7ff377e00640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 0x7ff35806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff380072630 tx=0x7ff368001520 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.168+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff370076b90 con 0x7ff360003a10 2024-01-16T15:04:45.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.403+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff3800cc200 con 0x7ff360003a10 2024-01-16T15:04:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.932+0000 7ff37d600640 1 -- 172.21.15.43:0/2763650289 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v32) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff37007a1b0 con 0x7ff360003a10 2024-01-16T15:04:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 8 and data pool 9 2024-01-16T15:04:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.942+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 msgr2=0x7ff35806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.942+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 0x7ff35806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff380072630 tx=0x7ff368001520 comp rx=0 tx=0).stop 2024-01-16T15:04:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.942+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 msgr2=0x7ff38015a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.942+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff37002c2b0 tx=0x7ff370004730 comp rx=0 tx=0).stop 2024-01-16T15:04:45.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 shutdown_connections 2024-01-16T15:04:45.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff35806b860 0x7ff35806dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3800c4640 0x7ff3801424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff380142a20 0x7ff380138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 --2- 172.21.15.43:0/2763650289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff360003a10 0x7ff38015a990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:45.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.943+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 >> 172.21.15.43:0/2763650289 conn(0x7ff3800cec10 msgr2=0x7ff38015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:45.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.944+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 shutdown_connections 2024-01-16T15:04:45.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:45.945+0000 7ff386600640 1 -- 172.21.15.43:0/2763650289 wait complete. 2024-01-16T15:04:45.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:46.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.159+0000 7f05a1400640 1 Processor -- start 2024-01-16T15:04:46.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.159+0000 7f05a1400640 1 -- start start 2024-01-16T15:04:46.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.160+0000 7f05a1400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f059c160970 0x7f059c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f05a1400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c0acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f05a1400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c071cd0 con 0x7f059c0b3140 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f05a1400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0b2120 con 0x7f059c160970 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f05a1400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0b2400 con 0x7f059c06dc30 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f059a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f059c160970 0x7f059c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f059a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f059c160970 0x7f059c0b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50780/0 (socket says 172.21.15.43:50780) 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f059a800640 1 -- 172.21.15.43:0/472270436 learned_addr learned my addr 172.21.15.43:0/472270436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f0593e00640 1 --1- 172.21.15.43:0/472270436 >> v1:172.21.15.43:6789/0 conn(0x7f059c0b3140 0x7f059c06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56240/0 (socket says 172.21.15.43:56240) 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.161+0000 7f059b200640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c0acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.162+0000 7f059a800640 1 -- 172.21.15.43:0/472270436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 msgr2=0x7f059c0acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.162+0000 7f059a800640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c0acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.162+0000 7f059a800640 1 -- 172.21.15.43:0/472270436 >> v1:172.21.15.43:6789/0 conn(0x7f059c0b3140 legacy=0x7f059c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.162+0000 7f059a800640 1 -- 172.21.15.43:0/472270436 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0b26e0 con 0x7f059c160970 2024-01-16T15:04:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.163+0000 7f059a800640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c0b2c00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f058c009870 tx=0x7f058c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910b5610ba53b297 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.163+0000 7f0599e00640 1 -- 172.21.15.43:0/472270436 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c031640 con 0x7f059c160970 2024-01-16T15:04:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.163+0000 7f0599e00640 1 -- 172.21.15.43:0/472270436 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c0317a0 con 0x7f059c160970 2024-01-16T15:04:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.163+0000 7f0599e00640 1 -- 172.21.15.43:0/472270436 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c031b30 con 0x7f059c160970 2024-01-16T15:04:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 -- 172.21.15.43:0/472270436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 msgr2=0x7f059c0b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c0b2c00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f058c009870 tx=0x7f058c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 -- 172.21.15.43:0/472270436 shutdown_connections 2024-01-16T15:04:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c0acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 --2- 172.21.15.43:0/472270436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c0b2c00 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 -- 172.21.15.43:0/472270436 >> 172.21.15.43:0/472270436 conn(0x7f059c0b0d80 msgr2=0x7f059c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 -- 172.21.15.43:0/472270436 shutdown_connections 2024-01-16T15:04:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.164+0000 7f05a1400640 1 -- 172.21.15.43:0/472270436 wait complete. 2024-01-16T15:04:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.166+0000 7f05a1400640 1 Processor -- start 2024-01-16T15:04:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.166+0000 7f05a1400640 1 -- start start 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c1512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f059c0b3140 0x7f059c14a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c152b80 con 0x7f059c160970 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c152e60 con 0x7f059c0b3140 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f05a1400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c153140 con 0x7f059c06dc30 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f059b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f059b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50796/0 (socket says 172.21.15.43:50796) 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.167+0000 7f059b200640 1 -- 172.21.15.43:0/287976777 learned_addr learned my addr 172.21.15.43:0/287976777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f0593e00640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f059c0b3140 0x7f059c14a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059a800640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c1512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 -- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 msgr2=0x7f059c1512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c1512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 -- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f059c0b3140 msgr2=0x7f059c14a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f059c0b3140 0x7f059c14a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 -- 172.21.15.43:0/287976777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0094d0 con 0x7f059c160970 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.168+0000 7f059b200640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0594007de0 tx=0x7f059400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.169+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594014620 con 0x7f059c160970 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.169+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059400ee90 con 0x7f059c160970 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.169+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c150ae0 con 0x7f059c160970 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.169+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05940149e0 con 0x7f059c160970 2024-01-16T15:04:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.169+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c14cc10 con 0x7f059c160970 2024-01-16T15:04:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.170+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c151000 con 0x7f059c160970 2024-01-16T15:04:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.171+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0594014b40 con 0x7f059c160970 2024-01-16T15:04:46.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.172+0000 7f0592a00640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 0x7f057406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.172+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f059408bd30 con 0x7f059c160970 2024-01-16T15:04:46.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.172+0000 7f059a800640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 0x7f057406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.173+0000 7f059a800640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 0x7f057406dd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f058c036000 tx=0x7f058c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.176+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0594056410 con 0x7f059c160970 2024-01-16T15:04:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.374+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f059c151000 con 0x7f059c160970 2024-01-16T15:04:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.377+0000 7f0592a00640 1 -- 172.21.15.43:0/287976777 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f059c151000 con 0x7f059c160970 2024-01-16T15:04:46.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:46.357 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":60,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:45.405048+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:04:43.301469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:04:44.420985+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.380+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 msgr2=0x7f057406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.380+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 0x7f057406dd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f058c036000 tx=0x7f058c03c040 comp rx=0 tx=0).stop 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.380+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 msgr2=0x7f059c14a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.380+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0594007de0 tx=0x7f059400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 shutdown_connections 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f057406b860 0x7f057406dd20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f059c0b3140 0x7f059c14a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f059c160970 0x7f059c14a9b0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 --2- 172.21.15.43:0/287976777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f059c06dc30 0x7f059c1512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 >> 172.21.15.43:0/287976777 conn(0x7f059c0b0d80 msgr2=0x7f059c161a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 shutdown_connections 2024-01-16T15:04:46.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.381+0000 7f05a1400640 1 -- 172.21.15.43:0/287976777 wait complete. 2024-01-16T15:04:46.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:46.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.618+0000 7fc7fbe00640 1 Processor -- start 2024-01-16T15:04:46.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.618+0000 7fc7fbe00640 1 -- start start 2024-01-16T15:04:46.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.620+0000 7fc7fbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7f406d520 0x7fc7f406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.620+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40abff0 0x7fc7f415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.620+0000 7fc7fbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f415c360 con 0x7fc7f40ad720 2024-01-16T15:04:46.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.620+0000 7fc7fbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f40b41a0 con 0x7fc7f406d520 2024-01-16T15:04:46.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.620+0000 7fc7fbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f406de40 con 0x7fc7f40abff0 2024-01-16T15:04:46.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7f406d520 0x7fc7f406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40abff0 0x7fc7f415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7f406d520 0x7fc7f406d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:47396/0 (socket says 172.21.15.43:47396) 2024-01-16T15:04:46.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 -- 172.21.15.43:0/112588017 learned_addr learned my addr 172.21.15.43:0/112588017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:46.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 -- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40abff0 msgr2=0x7fc7f415f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:46.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 --2- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40abff0 0x7fc7f415f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 -- 172.21.15.43:0/112588017 >> v1:172.21.15.69:6789/0 conn(0x7fc7f40ad720 legacy=0x7fc7f415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.621+0000 7fc7f9400640 1 -- 172.21.15.43:0/112588017 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f415ba30 con 0x7fc7f406d520 2024-01-16T15:04:46.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.623+0000 7fc7f9400640 1 --2- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f406d520 0x7fc7f406d900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009e50 tx=0x7fc7e402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b5fd06371568b96 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.623+0000 7fc7f3e00640 1 -- 172.21.15.43:0/112588017 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e402cb00 con 0x7fc7f406d520 2024-01-16T15:04:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.623+0000 7fc7f3e00640 1 -- 172.21.15.43:0/112588017 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e402cc60 con 0x7fc7f406d520 2024-01-16T15:04:46.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.623+0000 7fc7f3e00640 1 -- 172.21.15.43:0/112588017 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40343f0 con 0x7fc7f406d520 2024-01-16T15:04:46.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 -- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f406d520 msgr2=0x7fc7f406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f406d520 0x7fc7f406d900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009e50 tx=0x7fc7e402bf60 comp rx=0 tx=0).stop 2024-01-16T15:04:46.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 -- 172.21.15.43:0/112588017 shutdown_connections 2024-01-16T15:04:46.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40abff0 0x7fc7f415f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/112588017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f406d520 0x7fc7f406d900 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 -- 172.21.15.43:0/112588017 >> 172.21.15.43:0/112588017 conn(0x7fc7f40b0d80 msgr2=0x7fc7f40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 -- 172.21.15.43:0/112588017 shutdown_connections 2024-01-16T15:04:46.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.624+0000 7fc7fbe00640 1 -- 172.21.15.43:0/112588017 wait complete. 2024-01-16T15:04:46.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.627+0000 7fc7fbe00640 1 Processor -- start 2024-01-16T15:04:46.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.627+0000 7fc7fbe00640 1 -- start start 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f40abff0 0x7fc7f4158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40ad720 0x7fc7f415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f40cca90 con 0x7fc7f406d520 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f4158850 con 0x7fc7f40abff0 2024-01-16T15:04:46.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7fbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f40cbe10 con 0x7fc7f40ad720 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7f9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7f9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50808/0 (socket says 172.21.15.43:50808) 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.628+0000 7fc7f9400640 1 -- 172.21.15.43:0/255482174 learned_addr learned my addr 172.21.15.43:0/255482174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f40abff0 0x7fc7f4158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9e00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40ad720 0x7fc7f415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 -- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40ad720 msgr2=0x7fc7f415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40ad720 0x7fc7f415a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 -- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f40abff0 msgr2=0x7fc7f4158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f40abff0 0x7fc7f4158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 -- 172.21.15.43:0/255482174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e4009b30 con 0x7fc7f406d520 2024-01-16T15:04:46.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.629+0000 7fc7f9400640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4000f80 tx=0x7fc7e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.630+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4009510 con 0x7fc7f406d520 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.630+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f40cc0f0 con 0x7fc7f406d520 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.630+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e403e450 con 0x7fc7f406d520 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.630+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f40cc6b0 con 0x7fc7f406d520 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.630+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e403e800 con 0x7fc7f406d520 2024-01-16T15:04:46.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.631+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f40ccd00 con 0x7fc7f406d520 2024-01-16T15:04:46.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.633+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc7e403e960 con 0x7fc7f406d520 2024-01-16T15:04:46.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.634+0000 7fc7f1600640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 0x7fc7cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.634+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc7e40ad3b0 con 0x7fc7f406d520 2024-01-16T15:04:46.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.634+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 0x7fc7cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.635+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 0x7fc7cc06dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc7f406de10 tx=0x7fc7dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.639+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7e4077960 con 0x7fc7f406d520 2024-01-16T15:04:46.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.831+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7f4074770 con 0x7fc7f406d520 2024-01-16T15:04:46.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.834+0000 7fc7f1600640 1 -- 172.21.15.43:0/255482174 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16573 (secure 0 0 0) 0x7fc7e407af80 con 0x7fc7f406d520 2024-01-16T15:04:46.814 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:46.814 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":60,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:45.405048+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:04:43.301469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:04:44.420985+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:46.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 msgr2=0x7fc7cc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 0x7fc7cc06dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc7f406de10 tx=0x7fc7dc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 msgr2=0x7fc7f40cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4000f80 tx=0x7fc7e4004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 shutdown_connections 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc7cc06b890 0x7fc7cc06dd50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7f40abff0 0x7fc7f4158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7f40ad720 0x7fc7f415a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 --2- 172.21.15.43:0/255482174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f406d520 0x7fc7f40cceb0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 >> 172.21.15.43:0/255482174 conn(0x7fc7f40b0d80 msgr2=0x7fc7f40b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 shutdown_connections 2024-01-16T15:04:46.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:46.837+0000 7fc7fbe00640 1 -- 172.21.15.43:0/255482174 wait complete. 2024-01-16T15:04:46.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:04:47.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.083+0000 7fed69e00640 1 Processor -- start 2024-01-16T15:04:47.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.083+0000 7fed69e00640 1 -- start start 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed69e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fed6415d3b0 0x7fed640ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed69e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed640d1d90 0x7fed640b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed69e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640c6bf0 con 0x7fed640d0fa0 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed69e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed640d1430 con 0x7fed6415d3b0 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed69e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640cedc0 con 0x7fed640d1d90 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed68a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fed6415d3b0 0x7fed640ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed68a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fed6415d3b0 0x7fed640ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50814/0 (socket says 172.21.15.43:50814) 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.085+0000 7fed68a00640 1 -- 172.21.15.43:0/3495346263 learned_addr learned my addr 172.21.15.43:0/3495346263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.086+0000 7fed69400640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed640d1d90 0x7fed640b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.086+0000 7fed68a00640 1 -- 172.21.15.43:0/3495346263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed640d1d90 msgr2=0x7fed640b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.086+0000 7fed68a00640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed640d1d90 0x7fed640b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.086+0000 7fed68a00640 1 -- 172.21.15.43:0/3495346263 >> v1:172.21.15.69:6789/0 conn(0x7fed640d0fa0 legacy=0x7fed640d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.086+0000 7fed68a00640 1 -- 172.21.15.43:0/3495346263 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640c6e40 con 0x7fed6415d3b0 2024-01-16T15:04:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.087+0000 7fed68a00640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed6415d3b0 0x7fed640ce6c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fed54009870 tx=0x7fed5402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d61fa0f6148f93e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.087+0000 7fed63400640 1 -- 172.21.15.43:0/3495346263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed540317c0 con 0x7fed6415d3b0 2024-01-16T15:04:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.087+0000 7fed63400640 1 -- 172.21.15.43:0/3495346263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed54031920 con 0x7fed6415d3b0 2024-01-16T15:04:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.087+0000 7fed63400640 1 -- 172.21.15.43:0/3495346263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54031cb0 con 0x7fed6415d3b0 2024-01-16T15:04:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 -- 172.21.15.43:0/3495346263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed6415d3b0 msgr2=0x7fed640ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed6415d3b0 0x7fed640ce6c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fed54009870 tx=0x7fed5402b860 comp rx=0 tx=0).stop 2024-01-16T15:04:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 -- 172.21.15.43:0/3495346263 shutdown_connections 2024-01-16T15:04:47.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed640d1d90 0x7fed640b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 --2- 172.21.15.43:0/3495346263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed6415d3b0 0x7fed640ce6c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 -- 172.21.15.43:0/3495346263 >> 172.21.15.43:0/3495346263 conn(0x7fed640d30a0 msgr2=0x7fed640d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:47.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 -- 172.21.15.43:0/3495346263 shutdown_connections 2024-01-16T15:04:47.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.088+0000 7fed69e00640 1 -- 172.21.15.43:0/3495346263 wait complete. 2024-01-16T15:04:47.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.090+0000 7fed69e00640 1 Processor -- start 2024-01-16T15:04:47.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.091+0000 7fed69e00640 1 -- start start 2024-01-16T15:04:47.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed640d0fa0 0x7fed640b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed6415d3b0 0x7fed64152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640b2a20 con 0x7fed640d1d90 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640b6510 con 0x7fed640d0fa0 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640b0580 con 0x7fed6415d3b0 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50818/0 (socket says 172.21.15.43:50818) 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 -- 172.21.15.43:0/2084142291 learned_addr learned my addr 172.21.15.43:0/2084142291 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed68a00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed640d0fa0 0x7fed640b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed69400640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed6415d3b0 0x7fed64152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 -- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed6415d3b0 msgr2=0x7fed64152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed6415d3b0 0x7fed64152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 -- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed640d0fa0 msgr2=0x7fed640b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed640d0fa0 0x7fed640b24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.092+0000 7fed63e00640 1 -- 172.21.15.43:0/2084142291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540094d0 con 0x7fed640d1d90 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed63e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fed4c00c460 tx=0x7fed4c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c012830 con 0x7fed640d1d90 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c0129d0 con 0x7fed640d1d90 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c012d80 con 0x7fed640d1d90 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed640b2c30 con 0x7fed640d1d90 2024-01-16T15:04:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.093+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed640b31f0 con 0x7fed640d1d90 2024-01-16T15:04:47.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.095+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed640b66a0 con 0x7fed640d1d90 2024-01-16T15:04:47.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.096+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fed4c01a3b0 con 0x7fed640d1d90 2024-01-16T15:04:47.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.097+0000 7fed60c00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 0x7fed3c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:47.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.097+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fed4c08a5b0 con 0x7fed640d1d90 2024-01-16T15:04:47.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.097+0000 7fed68a00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 0x7fed3c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:47.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.098+0000 7fed68a00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 0x7fed3c06dd20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed54036000 tx=0x7fed5403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:47.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.104+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed4c054b50 con 0x7fed640d1d90 2024-01-16T15:04:47.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.344+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fed6404e840 con 0x7fed640d1d90 2024-01-16T15:04:47.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.958+0000 7fed60c00640 1 -- 172.21.15.43:0/2084142291 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fed4c058170 con 0x7fed640d1d90 2024-01-16T15:04:47.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.963+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 msgr2=0x7fed3c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.963+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 0x7fed3c06dd20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed54036000 tx=0x7fed5403d040 comp rx=0 tx=0).stop 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.963+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 msgr2=0x7fed640b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fed4c00c460 tx=0x7fed4c00c920 comp rx=0 tx=0).stop 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 shutdown_connections 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fed3c06b860 0x7fed3c06dd20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed6415d3b0 0x7fed64152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed640d1d90 0x7fed640b0040 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 --2- 172.21.15.43:0/2084142291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed640d0fa0 0x7fed640b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:47.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 >> 172.21.15.43:0/2084142291 conn(0x7fed640d30a0 msgr2=0x7fed640c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:47.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 shutdown_connections 2024-01-16T15:04:47.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:47.964+0000 7fed69e00640 1 -- 172.21.15.43:0/2084142291 wait complete. 2024-01-16T15:04:47.967 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:04:47.968 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:04:48.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.215+0000 7fcdb8200640 1 Processor -- start 2024-01-16T15:04:48.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.215+0000 7fcdb8200640 1 -- start start 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.216+0000 7fcdb8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 0x7fcdb00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.216+0000 7fcdb8200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcdb00adb60 0x7fcdb00adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.216+0000 7fcdb8200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb015b730 con 0x7fcdb009cb30 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.216+0000 7fcdb8200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb00ac740 con 0x7fcdb00adb60 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.216+0000 7fcdb8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00ac460 con 0x7fcdb015b350 2024-01-16T15:04:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdb4e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcdb009cb30 0x7fcdb00b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56242/0 (socket says 172.21.15.43:56242) 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdb4e00640 1 -- 172.21.15.43:0/321357318 learned_addr learned my addr 172.21.15.43:0/321357318 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdb5800640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 0x7fcdb00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdb6200640 1 --2- 172.21.15.43:0/321357318 >> v2:172.21.15.69:3300/0 conn(0x7fcdb00adb60 0x7fcdb00adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690952796 0 0) 0x7fcd98001110 con 0x7fcdb009cb30 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.217+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd90003520 con 0x7fcdb009cb30 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445525060 0 0) 0x7fcd90003520 con 0x7fcdb009cb30 2024-01-16T15:04:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 msgr2=0x7fcdb00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 0x7fcdb00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 >> v2:172.21.15.69:3300/0 conn(0x7fcdb00adb60 msgr2=0x7fcdb00adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 --2- 172.21.15.43:0/321357318 >> v2:172.21.15.69:3300/0 conn(0x7fcdb00adb60 0x7fcdb00adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb00b2a50 con 0x7fcdb009cb30 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcd98001800 con 0x7fcdb009cb30 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 >> v1:172.21.15.43:6789/0 conn(0x7fcdb009cb30 legacy=0x7fcdb00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd90003970 0x7fcd90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 0x7fcd900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd90004d00 0x7fcd90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00ac740 con 0x7fcd90003970 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00ac460 con 0x7fcd90004d00 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9000b980 con 0x7fcdb015b350 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd90003970 0x7fcd90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 -- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 msgr2=0x7fcd900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb015b350 0x7fcd900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 -- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd90004d00 msgr2=0x7fcd90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd90004d00 0x7fcd90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.218+0000 7fcdb5800640 1 -- 172.21.15.43:0/321357318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90003520 con 0x7fcd90003970 2024-01-16T15:04:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdb5800640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd90003970 0x7fcd90003d90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009ac0 tx=0x7fcd900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76513d8fc5fe3d3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda002c710 con 0x7fcd90003970 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda002c870 con 0x7fcd90003970 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdafe00640 1 -- 172.21.15.43:0/321357318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda002cc00 con 0x7fcd90003970 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdb8200640 1 -- 172.21.15.43:0/321357318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd90003970 msgr2=0x7fcd90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdb8200640 1 --2- 172.21.15.43:0/321357318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd90003970 0x7fcd90003d90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009ac0 tx=0x7fcd900057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.220+0000 7fcdb6200640 1 -- 172.21.15.43:0/321357318 reap_dead start 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.221+0000 7fcdb8200640 1 -- 172.21.15.43:0/321357318 shutdown_connections 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.221+0000 7fcdb8200640 1 -- 172.21.15.43:0/321357318 >> 172.21.15.43:0/321357318 conn(0x7fcdb00b0ba0 msgr2=0x7fcdb00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.221+0000 7fcdb8200640 1 -- 172.21.15.43:0/321357318 shutdown_connections 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.221+0000 7fcdb8200640 1 -- 172.21.15.43:0/321357318 wait complete. 2024-01-16T15:04:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.222+0000 7fcdb8200640 1 Processor -- start 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.223+0000 7fcdb8200640 1 -- start start 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb01e3890 0x7fcdb01543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb0154910 0x7fcdb0141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb01553c0 con 0x7fcdb0144d40 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb01e53c0 con 0x7fcdb01e3890 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb8200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb01e3cf0 con 0x7fcdb0154910 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50826/0 (socket says 172.21.15.43:50826) 2024-01-16T15:04:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 -- 172.21.15.43:0/62481351 learned_addr learned my addr 172.21.15.43:0/62481351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 -- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb0154910 msgr2=0x7fcdb0141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb0154910 0x7fcdb0141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 -- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb01e3890 msgr2=0x7fcdb01543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb01e3890 0x7fcdb01543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.224+0000 7fcdb5800640 1 -- 172.21.15.43:0/62481351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb01550e0 con 0x7fcdb0144d40 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdb5800640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fcda0000c00 tx=0x7fcda0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0003810 con 0x7fcdb0144d40 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda002de90 con 0x7fcdb0144d40 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0003b90 con 0x7fcdb0144d40 2024-01-16T15:04:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb01e5660 con 0x7fcdb0144d40 2024-01-16T15:04:48.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.225+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb01e5c20 con 0x7fcdb0144d40 2024-01-16T15:04:48.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.226+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb009d200 con 0x7fcdb0144d40 2024-01-16T15:04:48.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.227+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fcda0003cf0 con 0x7fcdb0144d40 2024-01-16T15:04:48.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.229+0000 7fcdad600640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 0x7fcd8806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.229+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcda00abfd0 con 0x7fcdb0144d40 2024-01-16T15:04:48.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.229+0000 7fcdb4e00640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 0x7fcd8806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.230+0000 7fcdb4e00640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 0x7fcd8806dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcd98001eb0 tx=0x7fcd98001c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.234+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcda0076570 con 0x7fcdb0144d40 2024-01-16T15:04:48.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.464+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcdb004e920 con 0x7fcdb0144d40 2024-01-16T15:04:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.977+0000 7fcdad600640 1 -- 172.21.15.43:0/62481351 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcda0079b90 con 0x7fcdb0144d40 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.980+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 msgr2=0x7fcd8806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.980+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 0x7fcd8806dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcd98001eb0 tx=0x7fcd98001c00 comp rx=0 tx=0).stop 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 msgr2=0x7fcdb01e3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fcda0000c00 tx=0x7fcda0004730 comp rx=0 tx=0).stop 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 shutdown_connections 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd8806b860 0x7fcd8806dd20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb01e3890 0x7fcdb01543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdb0154910 0x7fcdb0141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 --2- 172.21.15.43:0/62481351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdb0144d40 0x7fcdb01e3350 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 >> 172.21.15.43:0/62481351 conn(0x7fcdb00b0ba0 msgr2=0x7fcdb0159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 shutdown_connections 2024-01-16T15:04:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:48.981+0000 7fcdb8200640 1 -- 172.21.15.43:0/62481351 wait complete. 2024-01-16T15:04:48.982 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:04:48.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:49.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.230+0000 7f7e0f200640 1 Processor -- start 2024-01-16T15:04:49.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.230+0000 7f7e0f200640 1 -- start start 2024-01-16T15:04:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.232+0000 7f7e0f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e100d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.232+0000 7f7e0f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.232+0000 7f7e0f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100d3990 con 0x7f7e100b2400 2024-01-16T15:04:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.232+0000 7f7e0f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100d3c70 con 0x7f7e1015d620 2024-01-16T15:04:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.232+0000 7f7e0f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100d3f50 con 0x7f7e1015d240 2024-01-16T15:04:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.233+0000 7f7e07e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.233+0000 7f7e0de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e100d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.233+0000 7f7e07e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:47448/0 (socket says 172.21.15.43:47448) 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.233+0000 7f7e07e00640 1 -- 172.21.15.43:0/1743736130 learned_addr learned my addr 172.21.15.43:0/1743736130 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.233+0000 7f7e0de00640 1 -- 172.21.15.43:0/1743736130 >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 msgr2=0x7f7e100d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.234+0000 7f7e0de00640 1 --2- 172.21.15.43:0/1743736130 >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.234+0000 7f7e0de00640 1 -- 172.21.15.43:0/1743736130 >> v1:172.21.15.69:6789/0 conn(0x7f7e100b2400 legacy=0x7f7e100d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.234+0000 7f7e0de00640 1 -- 172.21.15.43:0/1743736130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100b29f0 con 0x7f7e1015d240 2024-01-16T15:04:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.234+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1743736130 >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:49.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.235+0000 7f7e0de00640 1 --2- 172.21.15.43:0/1743736130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e100d0cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e000097a0 tx=0x7f7e0002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=640fc6917f2d8971 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0d400640 1 -- 172.21.15.43:0/1743736130 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002d040 con 0x7f7e1015d240 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0d400640 1 -- 172.21.15.43:0/1743736130 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00031810 con 0x7f7e1015d240 2024-01-16T15:04:49.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0d400640 1 -- 172.21.15.43:0/1743736130 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00031ba0 con 0x7f7e1015d240 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 -- 172.21.15.43:0/1743736130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 msgr2=0x7f7e100d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 --2- 172.21.15.43:0/1743736130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e100d0cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e000097a0 tx=0x7f7e0002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 -- 172.21.15.43:0/1743736130 shutdown_connections 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 --2- 172.21.15.43:0/1743736130 >> v2:172.21.15.69:3300/0 conn(0x7f7e1015d620 0x7f7e100d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 --2- 172.21.15.43:0/1743736130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e100d0cb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 -- 172.21.15.43:0/1743736130 >> 172.21.15.43:0/1743736130 conn(0x7f7e1009c3f0 msgr2=0x7f7e100d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.236+0000 7f7e0f200640 1 -- 172.21.15.43:0/1743736130 shutdown_connections 2024-01-16T15:04:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.237+0000 7f7e0f200640 1 -- 172.21.15.43:0/1743736130 wait complete. 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.238+0000 7f7e0f200640 1 Processor -- start 2024-01-16T15:04:49.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.238+0000 7f7e0f200640 1 -- start start 2024-01-16T15:04:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100b2400 0x7f7e10152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e101f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100cb910 con 0x7f7e1015d620 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100cbbf0 con 0x7f7e100b2400 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.239+0000 7f7e0f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100cbed0 con 0x7f7e1015d240 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50828/0 (socket says 172.21.15.43:50828) 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 -- 172.21.15.43:0/3765174147 learned_addr learned my addr 172.21.15.43:0/3765174147 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0de00640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100b2400 0x7f7e10152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e07e00640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e101f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 -- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 msgr2=0x7f7e101f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e101f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 -- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100b2400 msgr2=0x7f7e10152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100b2400 0x7f7e10152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.240+0000 7f7e0e800640 1 -- 172.21.15.43:0/3765174147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df80094d0 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e0e800640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800c7a0 tx=0x7f7e0800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08003b70 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08003cf0 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e00009400 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08003ea0 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.241+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e100cc490 con 0x7f7e1015d620 2024-01-16T15:04:49.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.243+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1004e840 con 0x7f7e1015d620 2024-01-16T15:04:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.244+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7e0800aa90 con 0x7f7e1015d620 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.245+0000 7f7e06000640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 0x7f7de806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.245+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7e0808a440 con 0x7f7e1015d620 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.245+0000 7f7e0de00640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 0x7f7de806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.246+0000 7f7e0de00640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 0x7f7de806dda0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e00036000 tx=0x7f7e0003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.249+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e08054b10 con 0x7f7e1015d620 2024-01-16T15:04:49.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.483+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e101568e0 con 0x7f7e1015d620 2024-01-16T15:04:49.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.484+0000 7f7e06000640 1 -- 172.21.15.43:0/3765174147 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4904 (secure 0 0 0) 0x7f7e08058130 con 0x7f7e1015d620 2024-01-16T15:04:49.463 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:49.463 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/986973826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":986973826},{"type":"v1","addr":"172.21.15.69:6835","nonce":986973826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5644,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/607340672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":607340672},{"type":"v1","addr":"172.21.15.43:6836","nonce":607340672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/154390899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":154390899},{"type":"v1","addr":"172.21.15.43:6837","nonce":154390899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:04:45.405522+0000","modified":"2024-01-16T15:04:48.466693+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5650":{"gid":5650,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/4073013627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4073013627},{"type":"v1","addr":"172.21.15.69:6834","nonce":4073013627}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:04:49.463 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 36 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 msgr2=0x7f7de806dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 0x7f7de806dda0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e00036000 tx=0x7f7e0003c040 comp rx=0 tx=0).stop 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 msgr2=0x7f7e100cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800c7a0 tx=0x7f7e0800cc60 comp rx=0 tx=0).stop 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 shutdown_connections 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7de806b8e0 0x7f7de806dda0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e1015d620 0x7f7e100cb200 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1015d240 0x7f7e101f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 --2- 172.21.15.43:0/3765174147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e100b2400 0x7f7e10152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 >> 172.21.15.43:0/3765174147 conn(0x7f7e1009c3f0 msgr2=0x7f7e100afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 shutdown_connections 2024-01-16T15:04:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.487+0000 7f7e0f200640 1 -- 172.21.15.43:0/3765174147 wait complete. 2024-01-16T15:04:49.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.730+0000 7f8ade000640 1 Processor -- start 2024-01-16T15:04:49.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.731+0000 7f8ade000640 1 -- start start 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.732+0000 7f8ade000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 0x7f8ad815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.732+0000 7f8ade000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ad80cf050 0x7f8ad80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.732+0000 7f8ade000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad80ce7c0 con 0x7f8ad806c130 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.732+0000 7f8ade000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad80cfa10 con 0x7f8ad80cf050 2024-01-16T15:04:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.732+0000 7f8ade000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad8140900 con 0x7f8ad813dfd0 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.733+0000 7f8acfc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ad80cf050 0x7f8ad80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.733+0000 7f8adcc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 0x7f8ad815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.733+0000 7f8acfc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ad80cf050 0x7f8ad80cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:47468/0 (socket says 172.21.15.43:47468) 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.733+0000 7f8acfc00640 1 -- 172.21.15.43:0/1984646523 learned_addr learned my addr 172.21.15.43:0/1984646523 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.734+0000 7f8acfc00640 1 -- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 msgr2=0x7f8ad815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.734+0000 7f8acfc00640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 0x7f8ad815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.734+0000 7f8acfc00640 1 -- 172.21.15.43:0/1984646523 >> v1:172.21.15.69:6789/0 conn(0x7f8ad806c130 legacy=0x7f8ad806c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.734+0000 7f8acfc00640 1 -- 172.21.15.43:0/1984646523 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad815fa70 con 0x7f8ad80cf050 2024-01-16T15:04:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.734+0000 7f8adcc00640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 0x7f8ad815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.735+0000 7f8acfc00640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad80cf050 0x7f8ad80cf4d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009870 tx=0x7f8ac002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ac79550ff3538b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ad7e00640 1 -- 172.21.15.43:0/1984646523 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0030840 con 0x7f8ad80cf050 2024-01-16T15:04:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ad7e00640 1 -- 172.21.15.43:0/1984646523 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac00309a0 con 0x7f8ad80cf050 2024-01-16T15:04:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ad7e00640 1 -- 172.21.15.43:0/1984646523 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0030d30 con 0x7f8ad80cf050 2024-01-16T15:04:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 -- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad80cf050 msgr2=0x7f8ad80cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad80cf050 0x7f8ad80cf4d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009870 tx=0x7f8ac002b980 comp rx=0 tx=0).stop 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 -- 172.21.15.43:0/1984646523 shutdown_connections 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad80cf050 0x7f8ad80cf4d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 --2- 172.21.15.43:0/1984646523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad813dfd0 0x7f8ad815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 -- 172.21.15.43:0/1984646523 >> 172.21.15.43:0/1984646523 conn(0x7f8ad806a860 msgr2=0x7f8ad806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 -- 172.21.15.43:0/1984646523 shutdown_connections 2024-01-16T15:04:49.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.736+0000 7f8ade000640 1 -- 172.21.15.43:0/1984646523 wait complete. 2024-01-16T15:04:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.738+0000 7f8ade000640 1 Processor -- start 2024-01-16T15:04:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.738+0000 7f8ade000640 1 -- start start 2024-01-16T15:04:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad806c130 0x7f8ad80ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 0x7f8ad813c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80c82f0 con 0x7f8ad80cf050 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81560c0 con 0x7f8ad813dfd0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.739+0000 7f8ade000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81563a0 con 0x7f8ad806c130 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8acfc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8acfc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50832/0 (socket says 172.21.15.43:50832) 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8acfc00640 1 -- 172.21.15.43:0/1977908653 learned_addr learned my addr 172.21.15.43:0/1977908653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8add600640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 0x7f8ad813c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8adcc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad806c130 0x7f8ad80ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8acfc00640 1 -- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad806c130 msgr2=0x7f8ad80ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.740+0000 7f8acfc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad806c130 0x7f8ad80ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.741+0000 7f8acfc00640 1 -- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 msgr2=0x7f8ad813c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.741+0000 7f8acfc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 0x7f8ad813c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.741+0000 7f8acfc00640 1 -- 172.21.15.43:0/1977908653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8009400 con 0x7f8ad80cf050 2024-01-16T15:04:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.741+0000 7f8add600640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 0x7f8ad813c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.741+0000 7f8acfc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.742+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0039c90 con 0x7f8ad80cf050 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.742+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac0039df0 con 0x7f8ad80cf050 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.742+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0043640 con 0x7f8ad80cf050 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.742+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac00094d0 con 0x7f8ad80cf050 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.742+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad8156960 con 0x7f8ad80cf050 2024-01-16T15:04:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.743+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad804e840 con 0x7f8ad80cf050 2024-01-16T15:04:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.745+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f8ac00437a0 con 0x7f8ad80cf050 2024-01-16T15:04:49.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.745+0000 7f8ad5600640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 0x7f8ab006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:49.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.746+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8ac003e070 con 0x7f8ad80cf050 2024-01-16T15:04:49.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.746+0000 7f8adcc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 0x7f8ab006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:49.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.746+0000 7f8adcc00640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 0x7f8ab006dd50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8ad815ded0 tx=0x7f8ac8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:49.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.751+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ac0077c60 con 0x7f8ad80cf050 2024-01-16T15:04:49.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.949+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ad815b900 con 0x7f8ad80cf050 2024-01-16T15:04:49.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.951+0000 7f8ad5600640 1 -- 172.21.15.43:0/1977908653 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f8ac007b280 con 0x7f8ad80cf050 2024-01-16T15:04:49.931 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:49.931 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":61,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:04:46.932697+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:04:43.301469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:04:44.420985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.954+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 msgr2=0x7f8ab006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.954+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 0x7f8ab006dd50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8ad815ded0 tx=0x7f8ac8008040 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.954+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 msgr2=0x7f8ad80b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.954+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0004730 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.954+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 shutdown_connections 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ab006b890 0x7f8ab006dd50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ad813dfd0 0x7f8ad813c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad80cf050 0x7f8ad80b2f20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 --2- 172.21.15.43:0/1977908653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ad806c130 0x7f8ad80ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 >> 172.21.15.43:0/1977908653 conn(0x7f8ad806a860 msgr2=0x7f8ad813e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:49.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 shutdown_connections 2024-01-16T15:04:49.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:49.955+0000 7f8ade000640 1 -- 172.21.15.43:0/1977908653 wait complete. 2024-01-16T15:04:49.955 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:04:49.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:04:50.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.198+0000 7fe853e00640 1 Processor -- start 2024-01-16T15:04:50.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.198+0000 7fe853e00640 1 -- start start 2024-01-16T15:04:50.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe853e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8540cf050 0x7fe8540cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe853e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 0x7fe8540cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe853e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85415fa70 con 0x7fe85413dfd0 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe853e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8540cfa40 con 0x7fe8540cf050 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe853e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85415c580 con 0x7fe8540cd7b0 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe84be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8540cf050 0x7fe8540cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe84be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8540cf050 0x7fe8540cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50838/0 (socket says 172.21.15.43:50838) 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.200+0000 7fe84be00640 1 -- 172.21.15.43:0/2261998368 learned_addr learned my addr 172.21.15.43:0/2261998368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe853400640 1 --2- 172.21.15.43:0/2261998368 >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 0x7fe8540cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe84be00640 1 -- 172.21.15.43:0/2261998368 >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 msgr2=0x7fe8540cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe84be00640 1 --2- 172.21.15.43:0/2261998368 >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 0x7fe8540cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe84be00640 1 -- 172.21.15.43:0/2261998368 >> v1:172.21.15.43:6789/0 conn(0x7fe85413dfd0 legacy=0x7fe85415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe84be00640 1 -- 172.21.15.43:0/2261998368 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854140900 con 0x7fe8540cf050 2024-01-16T15:04:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.201+0000 7fe853400640 1 --2- 172.21.15.43:0/2261998368 >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 0x7fe8540cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.202+0000 7fe84be00640 1 --2- 172.21.15.43:0/2261998368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8540cf500 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe8541416f0 tx=0x7fe83c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f83d3c1f1ac5fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.202+0000 7fe852000640 1 -- 172.21.15.43:0/2261998368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c030870 con 0x7fe8540cf050 2024-01-16T15:04:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.202+0000 7fe852000640 1 -- 172.21.15.43:0/2261998368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c0309d0 con 0x7fe8540cf050 2024-01-16T15:04:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.202+0000 7fe852000640 1 -- 172.21.15.43:0/2261998368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c030d60 con 0x7fe8540cf050 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 -- 172.21.15.43:0/2261998368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 msgr2=0x7fe8540cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 --2- 172.21.15.43:0/2261998368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8540cf500 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe8541416f0 tx=0x7fe83c02b9b0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 -- 172.21.15.43:0/2261998368 shutdown_connections 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 --2- 172.21.15.43:0/2261998368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8540cf500 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 --2- 172.21.15.43:0/2261998368 >> v2:172.21.15.69:3300/0 conn(0x7fe8540cd7b0 0x7fe8540cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 -- 172.21.15.43:0/2261998368 >> 172.21.15.43:0/2261998368 conn(0x7fe85406a860 msgr2=0x7fe85406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 -- 172.21.15.43:0/2261998368 shutdown_connections 2024-01-16T15:04:50.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.203+0000 7fe853e00640 1 -- 172.21.15.43:0/2261998368 wait complete. 2024-01-16T15:04:50.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.205+0000 7fe853e00640 1 Processor -- start 2024-01-16T15:04:50.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.205+0000 7fe853e00640 1 -- start start 2024-01-16T15:04:50.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8540cd7b0 0x7fe85413b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe85413dfd0 0x7fe854151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe854151bc0 con 0x7fe8540cf050 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe854158990 con 0x7fe8540cd7b0 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe853e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe854159910 con 0x7fe85413dfd0 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe84be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe84be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50846/0 (socket says 172.21.15.43:50846) 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.206+0000 7fe84be00640 1 -- 172.21.15.43:0/2413014429 learned_addr learned my addr 172.21.15.43:0/2413014429 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe852a00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8540cd7b0 0x7fe85413b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe853400640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe85413dfd0 0x7fe854151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 -- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe85413dfd0 msgr2=0x7fe854151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe85413dfd0 0x7fe854151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 -- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8540cd7b0 msgr2=0x7fe85413b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8540cd7b0 0x7fe85413b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 -- 172.21.15.43:0/2413014429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0094d0 con 0x7fe8540cf050 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.207+0000 7fe84be00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe83c02b9f0 tx=0x7fe83c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.208+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c030860 con 0x7fe8540cf050 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.208+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c043400 con 0x7fe8540cf050 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.208+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0437b0 con 0x7fe8540cf050 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.208+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe854159bf0 con 0x7fe8540cf050 2024-01-16T15:04:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.208+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85415a1b0 con 0x7fe8540cf050 2024-01-16T15:04:50.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.209+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85404e840 con 0x7fe8540cf050 2024-01-16T15:04:50.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.210+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe83c043910 con 0x7fe8540cf050 2024-01-16T15:04:50.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.211+0000 7fe84aa00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 0x7fe82c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.211+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe83c03e070 con 0x7fe8540cf050 2024-01-16T15:04:50.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.211+0000 7fe852a00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 0x7fe82c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.212+0000 7fe852a00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 0x7fe82c06dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe854158c00 tx=0x7fe844009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.215+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe83c078150 con 0x7fe8540cf050 2024-01-16T15:04:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.405+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe854159a80 con 0x7fe82c06b860 2024-01-16T15:04:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.432+0000 7fe84aa00640 1 -- 172.21.15.43:0/2413014429 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281115 (secure 0 0 0) 0x7fe854159a80 con 0x7fe82c06b860 2024-01-16T15:04:50.415 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:04:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 msgr2=0x7fe82c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 0x7fe82c06dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe854158c00 tx=0x7fe844009040 comp rx=0 tx=0).stop 2024-01-16T15:04:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 msgr2=0x7fe8541582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe83c02b9f0 tx=0x7fe83c004730 comp rx=0 tx=0).stop 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 shutdown_connections 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe82c06b860 0x7fe82c06dd20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe85413dfd0 0x7fe854151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8540cf050 0x7fe8541582b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 --2- 172.21.15.43:0/2413014429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8540cd7b0 0x7fe85413b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 >> 172.21.15.43:0/2413014429 conn(0x7fe85406a860 msgr2=0x7fe854071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 shutdown_connections 2024-01-16T15:04:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.437+0000 7fe853e00640 1 -- 172.21.15.43:0/2413014429 wait complete. 2024-01-16T15:04:50.438 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-16T15:04:48.907105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":168,"acting":168,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310104,"kb_used_data":6580,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754664616,"statfs":{"total":773094113280,"available":772776566784,"internally_reserved":0,"allocated":6737920,"data_stored":3472287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":310693420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.003297"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.927505+0000","last_change":"2024-01-16T15:04:44.927505+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.927505+0000","last_undegraded":"2024-01-16T15:04:44.927505+0000","last_fullsized":"2024-01-16T15:04:44.927505+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.937784+0000","last_change":"2024-01-16T15:04:44.937784+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.937784+0000","last_undegraded":"2024-01-16T15:04:44.937784+0000","last_fullsized":"2024-01-16T15:04:44.937784+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.948080+0000","last_change":"2024-01-16T15:04:44.948080+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.948080+0000","last_undegraded":"2024-01-16T15:04:44.948080+0000","last_fullsized":"2024-01-16T15:04:44.948080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.927413+0000","last_change":"2024-01-16T15:04:44.927413+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.927413+0000","last_undegraded":"2024-01-16T15:04:44.927413+0000","last_fullsized":"2024-01-16T15:04:44.927413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.935773+0000","last_change":"2024-01-16T15:04:44.935773+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.935773+0000","last_undegraded":"2024-01-16T15:04:44.935773+0000","last_fullsized":"2024-01-16T15:04:44.935773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.929917+0000","last_change":"2024-01-16T15:04:44.929917+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.929917+0000","last_undegraded":"2024-01-16T15:04:44.929917+0000","last_fullsized":"2024-01-16T15:04:44.929917+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.930402+0000","last_change":"2024-01-16T15:04:44.930402+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.930402+0000","last_undegraded":"2024-01-16T15:04:44.930402+0000","last_fullsized":"2024-01-16T15:04:44.930402+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.938032+0000","last_change":"2024-01-16T15:04:44.938032+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.938032+0000","last_undegraded":"2024-01-16T15:04:44.938032+0000","last_fullsized":"2024-01-16T15:04:44.938032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936455+0000","last_change":"2024-01-16T15:04:44.936455+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936455+0000","last_undegraded":"2024-01-16T15:04:44.936455+0000","last_fullsized":"2024-01-16T15:04:44.936455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.947558+0000","last_change":"2024-01-16T15:04:44.947558+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.947558+0000","last_undegraded":"2024-01-16T15:04:44.947558+0000","last_fullsized":"2024-01-16T15:04:44.947558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.932600+0000","last_change":"2024-01-16T15:04:44.932600+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.932600+0000","last_undegraded":"2024-01-16T15:04:44.932600+0000","last_fullsized":"2024-01-16T15:04:44.932600+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.931024+0000","last_change":"2024-01-16T15:04:44.931024+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.931024+0000","last_undegraded":"2024-01-16T15:04:44.931024+0000","last_fullsized":"2024-01-16T15:04:44.931024+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.928599+0000","last_change":"2024-01-16T15:04:44.928599+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.928599+0000","last_undegraded":"2024-01-16T15:04:44.928599+0000","last_fullsized":"2024-01-16T15:04:44.928599+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.933550+0000","last_change":"2024-01-16T15:04:44.933550+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.933550+0000","last_undegraded":"2024-01-16T15:04:44.933550+0000","last_fullsized":"2024-01-16T15:04:44.933550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.937883+0000","last_change":"2024-01-16T15:04:44.937883+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.937883+0000","last_undegraded":"2024-01-16T15:04:44.937883+0000","last_fullsized":"2024-01-16T15:04:44.937883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.926506+0000","last_change":"2024-01-16T15:04:44.926506+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.926506+0000","last_undegraded":"2024-01-16T15:04:44.926506+0000","last_fullsized":"2024-01-16T15:04:44.926506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.938941+0000","last_change":"2024-01-16T15:04:44.938941+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.938941+0000","last_undegraded":"2024-01-16T15:04:44.938941+0000","last_fullsized":"2024-01-16T15:04:44.938941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936151+0000","last_change":"2024-01-16T15:04:44.936151+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936151+0000","last_undegraded":"2024-01-16T15:04:44.936151+0000","last_fullsized":"2024-01-16T15:04:44.936151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.932386+0000","last_change":"2024-01-16T15:04:44.932386+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.932386+0000","last_undegraded":"2024-01-16T15:04:44.932386+0000","last_fullsized":"2024-01-16T15:04:44.932386+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.926345+0000","last_change":"2024-01-16T15:04:44.926345+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.926345+0000","last_undegraded":"2024-01-16T15:04:44.926345+0000","last_fullsized":"2024-01-16T15:04:44.926345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.943213+0000","last_change":"2024-01-16T15:04:44.943213+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.943213+0000","last_undegraded":"2024-01-16T15:04:44.943213+0000","last_fullsized":"2024-01-16T15:04:44.943213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.933545+0000","last_change":"2024-01-16T15:04:44.933545+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.933545+0000","last_undegraded":"2024-01-16T15:04:44.933545+0000","last_fullsized":"2024-01-16T15:04:44.933545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.927666+0000","last_change":"2024-01-16T15:04:44.927666+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.927666+0000","last_undegraded":"2024-01-16T15:04:44.927666+0000","last_fullsized":"2024-01-16T15:04:44.927666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.937080+0000","last_change":"2024-01-16T15:04:44.937080+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.937080+0000","last_undegraded":"2024-01-16T15:04:44.937080+0000","last_fullsized":"2024-01-16T15:04:44.937080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.926814+0000","last_change":"2024-01-16T15:04:44.926814+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.926814+0000","last_undegraded":"2024-01-16T15:04:44.926814+0000","last_fullsized":"2024-01-16T15:04:44.926814+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.930462+0000","last_change":"2024-01-16T15:04:44.930462+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.930462+0000","last_undegraded":"2024-01-16T15:04:44.930462+0000","last_fullsized":"2024-01-16T15:04:44.930462+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.932228+0000","last_change":"2024-01-16T15:04:44.932228+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.932228+0000","last_undegraded":"2024-01-16T15:04:44.932228+0000","last_fullsized":"2024-01-16T15:04:44.932228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.960099+0000","last_change":"2024-01-16T15:04:44.958382+0000","last_active":"2024-01-16T15:04:44.960099+0000","last_peered":"2024-01-16T15:04:44.960099+0000","last_clean":"2024-01-16T15:04:44.960099+0000","last_became_active":"2024-01-16T15:04:44.958066+0000","last_became_peered":"2024-01-16T15:04:44.958066+0000","last_unstale":"2024-01-16T15:04:44.960099+0000","last_undegraded":"2024-01-16T15:04:44.960099+0000","last_fullsized":"2024-01-16T15:04:44.960099+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:30:22.338288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.937389+0000","last_change":"2024-01-16T15:04:44.937389+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.937389+0000","last_undegraded":"2024-01-16T15:04:44.937389+0000","last_fullsized":"2024-01-16T15:04:44.937389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.914932+0000","last_change":"2024-01-16T15:04:43.914932+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.914932+0000","last_undegraded":"2024-01-16T15:04:43.914932+0000","last_fullsized":"2024-01-16T15:04:43.914932+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.930088+0000","last_change":"2024-01-16T15:04:44.930088+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.930088+0000","last_undegraded":"2024-01-16T15:04:44.930088+0000","last_fullsized":"2024-01-16T15:04:44.930088+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.954602+0000","last_change":"2024-01-16T15:04:44.941385+0000","last_active":"2024-01-16T15:04:44.954602+0000","last_peered":"2024-01-16T15:04:44.954602+0000","last_clean":"2024-01-16T15:04:44.954602+0000","last_became_active":"2024-01-16T15:04:44.941068+0000","last_became_peered":"2024-01-16T15:04:44.941068+0000","last_unstale":"2024-01-16T15:04:44.954602+0000","last_undegraded":"2024-01-16T15:04:44.954602+0000","last_fullsized":"2024-01-16T15:04:44.954602+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:18:46.123235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.933125+0000","last_change":"2024-01-16T15:04:44.933125+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.933125+0000","last_undegraded":"2024-01-16T15:04:44.933125+0000","last_fullsized":"2024-01-16T15:04:44.933125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.956578+0000","last_change":"2024-01-16T15:04:44.951546+0000","last_active":"2024-01-16T15:04:44.956578+0000","last_peered":"2024-01-16T15:04:44.956578+0000","last_clean":"2024-01-16T15:04:44.956578+0000","last_became_active":"2024-01-16T15:04:44.949430+0000","last_became_peered":"2024-01-16T15:04:44.949430+0000","last_unstale":"2024-01-16T15:04:44.956578+0000","last_undegraded":"2024-01-16T15:04:44.956578+0000","last_fullsized":"2024-01-16T15:04:44.956578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:24.260411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.944877+0000","last_change":"2024-01-16T15:04:44.940904+0000","last_active":"2024-01-16T15:04:44.944877+0000","last_peered":"2024-01-16T15:04:44.944877+0000","last_clean":"2024-01-16T15:04:44.944877+0000","last_became_active":"2024-01-16T15:04:44.939860+0000","last_became_peered":"2024-01-16T15:04:44.939860+0000","last_unstale":"2024-01-16T15:04:44.944877+0000","last_undegraded":"2024-01-16T15:04:44.944877+0000","last_fullsized":"2024-01-16T15:04:44.944877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:26.385417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.956335+0000","last_change":"2024-01-16T15:04:44.949930+0000","last_active":"2024-01-16T15:04:44.956335+0000","last_peered":"2024-01-16T15:04:44.956335+0000","last_clean":"2024-01-16T15:04:44.956335+0000","last_became_active":"2024-01-16T15:04:44.949670+0000","last_became_peered":"2024-01-16T15:04:44.949670+0000","last_unstale":"2024-01-16T15:04:44.956335+0000","last_undegraded":"2024-01-16T15:04:44.956335+0000","last_fullsized":"2024-01-16T15:04:44.956335+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:56.796316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.931126+0000","last_change":"2024-01-16T15:04:44.931126+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.931126+0000","last_undegraded":"2024-01-16T15:04:44.931126+0000","last_fullsized":"2024-01-16T15:04:44.931126+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957771+0000","last_change":"2024-01-16T15:04:44.952580+0000","last_active":"2024-01-16T15:04:44.957771+0000","last_peered":"2024-01-16T15:04:44.957771+0000","last_clean":"2024-01-16T15:04:44.957771+0000","last_became_active":"2024-01-16T15:04:44.951072+0000","last_became_peered":"2024-01-16T15:04:44.951072+0000","last_unstale":"2024-01-16T15:04:44.957771+0000","last_undegraded":"2024-01-16T15:04:44.957771+0000","last_fullsized":"2024-01-16T15:04:44.957771+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:05.252202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.958797+0000","last_change":"2024-01-16T15:04:44.954744+0000","last_active":"2024-01-16T15:04:44.958797+0000","last_peered":"2024-01-16T15:04:44.958797+0000","last_clean":"2024-01-16T15:04:44.958797+0000","last_became_active":"2024-01-16T15:04:44.954114+0000","last_became_peered":"2024-01-16T15:04:44.954114+0000","last_unstale":"2024-01-16T15:04:44.958797+0000","last_undegraded":"2024-01-16T15:04:44.958797+0000","last_fullsized":"2024-01-16T15:04:44.958797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:35.538247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.933302+0000","last_change":"2024-01-16T15:04:44.933302+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.933302+0000","last_undegraded":"2024-01-16T15:04:44.933302+0000","last_fullsized":"2024-01-16T15:04:44.933302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.958955+0000","last_change":"2024-01-16T15:04:44.956126+0000","last_active":"2024-01-16T15:04:44.958955+0000","last_peered":"2024-01-16T15:04:44.958955+0000","last_clean":"2024-01-16T15:04:44.958955+0000","last_became_active":"2024-01-16T15:04:44.955841+0000","last_became_peered":"2024-01-16T15:04:44.955841+0000","last_unstale":"2024-01-16T15:04:44.958955+0000","last_undegraded":"2024-01-16T15:04:44.958955+0000","last_fullsized":"2024-01-16T15:04:44.958955+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:17.959767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936750+0000","last_change":"2024-01-16T15:04:44.936750+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936750+0000","last_undegraded":"2024-01-16T15:04:44.936750+0000","last_fullsized":"2024-01-16T15:04:44.936750+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.912382+0000","last_change":"2024-01-16T15:04:43.912382+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.912382+0000","last_undegraded":"2024-01-16T15:04:43.912382+0000","last_fullsized":"2024-01-16T15:04:43.912382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.939803+0000","last_change":"2024-01-16T15:04:44.939803+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.939803+0000","last_undegraded":"2024-01-16T15:04:44.939803+0000","last_fullsized":"2024-01-16T15:04:44.939803+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.952875+0000","last_change":"2024-01-16T15:04:44.950674+0000","last_active":"2024-01-16T15:04:44.952875+0000","last_peered":"2024-01-16T15:04:44.952875+0000","last_clean":"2024-01-16T15:04:44.952875+0000","last_became_active":"2024-01-16T15:04:44.950287+0000","last_became_peered":"2024-01-16T15:04:44.950287+0000","last_unstale":"2024-01-16T15:04:44.952875+0000","last_undegraded":"2024-01-16T15:04:44.952875+0000","last_fullsized":"2024-01-16T15:04:44.952875+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:05.946555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936734+0000","last_change":"2024-01-16T15:04:44.936734+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936734+0000","last_undegraded":"2024-01-16T15:04:44.936734+0000","last_fullsized":"2024-01-16T15:04:44.936734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.946216+0000","last_change":"2024-01-16T15:04:44.944375+0000","last_active":"2024-01-16T15:04:44.946216+0000","last_peered":"2024-01-16T15:04:44.946216+0000","last_clean":"2024-01-16T15:04:44.946216+0000","last_became_active":"2024-01-16T15:04:44.943448+0000","last_became_peered":"2024-01-16T15:04:44.943448+0000","last_unstale":"2024-01-16T15:04:44.946216+0000","last_undegraded":"2024-01-16T15:04:44.946216+0000","last_fullsized":"2024-01-16T15:04:44.946216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:01.560365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.950057+0000","last_change":"2024-01-16T15:04:44.950057+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.950057+0000","last_undegraded":"2024-01-16T15:04:44.950057+0000","last_fullsized":"2024-01-16T15:04:44.950057+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.913113+0000","last_change":"2024-01-16T15:04:43.913113+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.913113+0000","last_undegraded":"2024-01-16T15:04:43.913113+0000","last_fullsized":"2024-01-16T15:04:43.913113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936592+0000","last_change":"2024-01-16T15:04:44.936592+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936592+0000","last_undegraded":"2024-01-16T15:04:44.936592+0000","last_fullsized":"2024-01-16T15:04:44.936592+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.948023+0000","last_change":"2024-01-16T15:04:44.946889+0000","last_active":"2024-01-16T15:04:44.948023+0000","last_peered":"2024-01-16T15:04:44.948023+0000","last_clean":"2024-01-16T15:04:44.948023+0000","last_became_active":"2024-01-16T15:04:44.946529+0000","last_became_peered":"2024-01-16T15:04:44.946529+0000","last_unstale":"2024-01-16T15:04:44.948023+0000","last_undegraded":"2024-01-16T15:04:44.948023+0000","last_fullsized":"2024-01-16T15:04:44.948023+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:15.650249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.928675+0000","last_change":"2024-01-16T15:04:44.928675+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.928675+0000","last_undegraded":"2024-01-16T15:04:44.928675+0000","last_fullsized":"2024-01-16T15:04:44.928675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.954577+0000","last_change":"2024-01-16T15:04:44.950600+0000","last_active":"2024-01-16T15:04:44.954577+0000","last_peered":"2024-01-16T15:04:44.954577+0000","last_clean":"2024-01-16T15:04:44.954577+0000","last_became_active":"2024-01-16T15:04:44.949790+0000","last_became_peered":"2024-01-16T15:04:44.949790+0000","last_unstale":"2024-01-16T15:04:44.954577+0000","last_undegraded":"2024-01-16T15:04:44.954577+0000","last_fullsized":"2024-01-16T15:04:44.954577+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:32.257346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.933914+0000","last_change":"2024-01-16T15:04:44.933914+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.933914+0000","last_undegraded":"2024-01-16T15:04:44.933914+0000","last_fullsized":"2024-01-16T15:04:44.933914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.958934+0000","last_change":"2024-01-16T15:04:44.955658+0000","last_active":"2024-01-16T15:04:44.958934+0000","last_peered":"2024-01-16T15:04:44.958934+0000","last_clean":"2024-01-16T15:04:44.958934+0000","last_became_active":"2024-01-16T15:04:44.955007+0000","last_became_peered":"2024-01-16T15:04:44.955007+0000","last_unstale":"2024-01-16T15:04:44.958934+0000","last_undegraded":"2024-01-16T15:04:44.958934+0000","last_fullsized":"2024-01-16T15:04:44.958934+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:47.851218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.936214+0000","last_change":"2024-01-16T15:04:44.936214+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.936214+0000","last_undegraded":"2024-01-16T15:04:44.936214+0000","last_fullsized":"2024-01-16T15:04:44.936214+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.912565+0000","last_change":"2024-01-16T15:04:43.912565+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.912565+0000","last_undegraded":"2024-01-16T15:04:43.912565+0000","last_fullsized":"2024-01-16T15:04:43.912565+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.949153+0000","last_change":"2024-01-16T15:04:44.949153+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.949153+0000","last_undegraded":"2024-01-16T15:04:44.949153+0000","last_fullsized":"2024-01-16T15:04:44.949153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.954176+0000","last_change":"2024-01-16T15:04:44.944854+0000","last_active":"2024-01-16T15:04:44.954176+0000","last_peered":"2024-01-16T15:04:44.954176+0000","last_clean":"2024-01-16T15:04:44.954176+0000","last_became_active":"2024-01-16T15:04:44.942010+0000","last_became_peered":"2024-01-16T15:04:44.942010+0000","last_unstale":"2024-01-16T15:04:44.954176+0000","last_undegraded":"2024-01-16T15:04:44.954176+0000","last_fullsized":"2024-01-16T15:04:44.954176+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:07.504682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.958180+0000","last_change":"2024-01-16T15:04:44.953363+0000","last_active":"2024-01-16T15:04:44.958180+0000","last_peered":"2024-01-16T15:04:44.958180+0000","last_clean":"2024-01-16T15:04:44.958180+0000","last_became_active":"2024-01-16T15:04:44.952992+0000","last_became_peered":"2024-01-16T15:04:44.952992+0000","last_unstale":"2024-01-16T15:04:44.958180+0000","last_undegraded":"2024-01-16T15:04:44.958180+0000","last_fullsized":"2024-01-16T15:04:44.958180+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:25:34.942172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.934529+0000","last_change":"2024-01-16T15:04:44.934529+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.934529+0000","last_undegraded":"2024-01-16T15:04:44.934529+0000","last_fullsized":"2024-01-16T15:04:44.934529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.955495+0000","last_change":"2024-01-16T15:04:44.949082+0000","last_active":"2024-01-16T15:04:44.955495+0000","last_peered":"2024-01-16T15:04:44.955495+0000","last_clean":"2024-01-16T15:04:44.955495+0000","last_became_active":"2024-01-16T15:04:44.948716+0000","last_became_peered":"2024-01-16T15:04:44.948716+0000","last_unstale":"2024-01-16T15:04:44.955495+0000","last_undegraded":"2024-01-16T15:04:44.955495+0000","last_fullsized":"2024-01-16T15:04:44.955495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:11.012196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.949450+0000","last_change":"2024-01-16T15:04:44.949450+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.949450+0000","last_undegraded":"2024-01-16T15:04:44.949450+0000","last_fullsized":"2024-01-16T15:04:44.949450+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.912147+0000","last_change":"2024-01-16T15:04:43.912147+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.912147+0000","last_undegraded":"2024-01-16T15:04:43.912147+0000","last_fullsized":"2024-01-16T15:04:43.912147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.929815+0000","last_change":"2024-01-16T15:04:44.929815+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.929815+0000","last_undegraded":"2024-01-16T15:04:44.929815+0000","last_fullsized":"2024-01-16T15:04:44.929815+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.944226+0000","last_change":"2024-01-16T15:04:44.938607+0000","last_active":"2024-01-16T15:04:44.944226+0000","last_peered":"2024-01-16T15:04:44.944226+0000","last_clean":"2024-01-16T15:04:44.944226+0000","last_became_active":"2024-01-16T15:04:44.938328+0000","last_became_peered":"2024-01-16T15:04:44.938328+0000","last_unstale":"2024-01-16T15:04:44.944226+0000","last_undegraded":"2024-01-16T15:04:44.944226+0000","last_fullsized":"2024-01-16T15:04:44.944226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:13.930160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.929257+0000","last_change":"2024-01-16T15:04:44.929257+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.929257+0000","last_undegraded":"2024-01-16T15:04:44.929257+0000","last_fullsized":"2024-01-16T15:04:44.929257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.958586+0000","last_change":"2024-01-16T15:04:44.955973+0000","last_active":"2024-01-16T15:04:44.958586+0000","last_peered":"2024-01-16T15:04:44.958586+0000","last_clean":"2024-01-16T15:04:44.958586+0000","last_became_active":"2024-01-16T15:04:44.955709+0000","last_became_peered":"2024-01-16T15:04:44.955709+0000","last_unstale":"2024-01-16T15:04:44.958586+0000","last_undegraded":"2024-01-16T15:04:44.958586+0000","last_fullsized":"2024-01-16T15:04:44.958586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:39:55.171997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.942616+0000","last_change":"2024-01-16T15:04:44.942616+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.942616+0000","last_undegraded":"2024-01-16T15:04:44.942616+0000","last_fullsized":"2024-01-16T15:04:44.942616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957087+0000","last_change":"2024-01-16T15:04:44.954224+0000","last_active":"2024-01-16T15:04:44.957087+0000","last_peered":"2024-01-16T15:04:44.957087+0000","last_clean":"2024-01-16T15:04:44.957087+0000","last_became_active":"2024-01-16T15:04:44.953706+0000","last_became_peered":"2024-01-16T15:04:44.953706+0000","last_unstale":"2024-01-16T15:04:44.957087+0000","last_undegraded":"2024-01-16T15:04:44.957087+0000","last_fullsized":"2024-01-16T15:04:44.957087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:54.445394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.948210+0000","last_change":"2024-01-16T15:04:44.948210+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.948210+0000","last_undegraded":"2024-01-16T15:04:44.948210+0000","last_fullsized":"2024-01-16T15:04:44.948210+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957671+0000","last_change":"2024-01-16T15:04:44.956833+0000","last_active":"2024-01-16T15:04:44.957671+0000","last_peered":"2024-01-16T15:04:44.957671+0000","last_clean":"2024-01-16T15:04:44.957671+0000","last_became_active":"2024-01-16T15:04:44.956562+0000","last_became_peered":"2024-01-16T15:04:44.956562+0000","last_unstale":"2024-01-16T15:04:44.957671+0000","last_undegraded":"2024-01-16T15:04:44.957671+0000","last_fullsized":"2024-01-16T15:04:44.957671+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:41.054530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.937960+0000","last_change":"2024-01-16T15:04:44.937960+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.937960+0000","last_undegraded":"2024-01-16T15:04:44.937960+0000","last_fullsized":"2024-01-16T15:04:44.937960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.945562+0000","last_change":"2024-01-16T15:04:44.942876+0000","last_active":"2024-01-16T15:04:44.945562+0000","last_peered":"2024-01-16T15:04:44.945562+0000","last_clean":"2024-01-16T15:04:44.945562+0000","last_became_active":"2024-01-16T15:04:44.942274+0000","last_became_peered":"2024-01-16T15:04:44.942274+0000","last_unstale":"2024-01-16T15:04:44.945562+0000","last_undegraded":"2024-01-16T15:04:44.945562+0000","last_fullsized":"2024-01-16T15:04:44.945562+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:28.317748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.932207+0000","last_change":"2024-01-16T15:04:44.932207+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.932207+0000","last_undegraded":"2024-01-16T15:04:44.932207+0000","last_fullsized":"2024-01-16T15:04:44.932207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.954290+0000","last_change":"2024-01-16T15:04:44.949594+0000","last_active":"2024-01-16T15:04:44.954290+0000","last_peered":"2024-01-16T15:04:44.954290+0000","last_clean":"2024-01-16T15:04:44.954290+0000","last_became_active":"2024-01-16T15:04:44.949254+0000","last_became_peered":"2024-01-16T15:04:44.949254+0000","last_unstale":"2024-01-16T15:04:44.954290+0000","last_undegraded":"2024-01-16T15:04:44.954290+0000","last_fullsized":"2024-01-16T15:04:44.954290+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.214978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957873+0000","last_change":"2024-01-16T15:04:44.956069+0000","last_active":"2024-01-16T15:04:44.957873+0000","last_peered":"2024-01-16T15:04:44.957873+0000","last_clean":"2024-01-16T15:04:44.957873+0000","last_became_active":"2024-01-16T15:04:44.955054+0000","last_became_peered":"2024-01-16T15:04:44.955054+0000","last_unstale":"2024-01-16T15:04:44.957873+0000","last_undegraded":"2024-01-16T15:04:44.957873+0000","last_fullsized":"2024-01-16T15:04:44.957873+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:19.957830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.934564+0000","last_change":"2024-01-16T15:04:44.934564+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.934564+0000","last_undegraded":"2024-01-16T15:04:44.934564+0000","last_fullsized":"2024-01-16T15:04:44.934564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957904+0000","last_change":"2024-01-16T15:04:44.953091+0000","last_active":"2024-01-16T15:04:44.957904+0000","last_peered":"2024-01-16T15:04:44.957904+0000","last_clean":"2024-01-16T15:04:44.957904+0000","last_became_active":"2024-01-16T15:04:44.951937+0000","last_became_peered":"2024-01-16T15:04:44.951937+0000","last_unstale":"2024-01-16T15:04:44.957904+0000","last_undegraded":"2024-01-16T15:04:44.957904+0000","last_fullsized":"2024-01-16T15:04:44.957904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:39.494697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":59,"state":"creating+peering","last_fresh":"2024-01-16T15:04:44.935177+0000","last_change":"2024-01-16T15:04:44.935177+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.935177+0000","last_undegraded":"2024-01-16T15:04:44.935177+0000","last_fullsized":"2024-01-16T15:04:44.935177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.957967+0000","last_change":"2024-01-16T15:04:44.955674+0000","last_active":"2024-01-16T15:04:44.957967+0000","last_peered":"2024-01-16T15:04:44.957967+0000","last_clean":"2024-01-16T15:04:44.957967+0000","last_became_active":"2024-01-16T15:04:44.955359+0000","last_became_peered":"2024-01-16T15:04:44.955359+0000","last_unstale":"2024-01-16T15:04:44.957967+0000","last_undegraded":"2024-01-16T15:04:44.957967+0000","last_fullsized":"2024-01-16T15:04:44.957967+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:36.388668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":58,"state":"creating+peering","last_fresh":"2024-01-16T15:04:43.912184+0000","last_change":"2024-01-16T15:04:43.912184+0000","last_active":"2024-01-16T15:04:43.887806+0000","last_peered":"2024-01-16T15:04:43.887806+0000","last_clean":"2024-01-16T15:04:43.887806+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:43.912184+0000","last_undegraded":"2024-01-16T15:04:43.912184+0000","last_fullsized":"2024-01-16T15:04:43.912184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T15:04:44.902434+0000","last_change":"2024-01-16T15:04:44.902434+0000","last_active":"2024-01-16T15:04:44.902434+0000","last_peered":"2024-01-16T15:04:44.902434+0000","last_clean":"2024-01-16T15:04:44.902434+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:04:44.902434+0000","last_undegraded":"2024-01-16T15:04:44.902434+0000","last_fullsized":"2024-01-16T15:04:44.902434+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:04:44.959017+0000","last_change":"2024-01-16T15:04:44.955479+0000","last_active":"2024-01-16T15:04:44.959017+0000","last_peered":"2024-01-16T15:04:44.959017+0000","last_clean":"2024-01-16T15:04:44.959017+0000","last_became_active":"2024-01-16T15:04:44.955081+0000","last_became_peered":"2024-01-16T15:04:44.955081+0000","last_unstale":"2024-01-16T15:04:44.959017+0000","last_undegraded":"2024-01-16T15:04:44.959017+0000","last_fullsized":"2024-01-16T15:04:44.959017+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:57.903933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":196,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:04:43.922305+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:04:43.922305+0000","last_peered":"2024-01-16T15:04:43.922305+0000","last_clean":"2024-01-16T15:04:43.922305+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:04:43.922305+0000","last_undegraded":"2024-01-16T15:04:43.922305+0000","last_fullsized":"2024-01-16T15:04:43.922305+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":102,"acting":102,"num_store_stats":7},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1286144,"data_stored":876025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.343,"15min":1.1930000000000001},"min":{"1min":0.63700000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3740000000000001,"15min":1.26},"min":{"1min":0.65700000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":7.8460000000000001,"5min":7.8460000000000001,"15min":7.8460000000000001},"last":1.0469999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.252,"15min":1.222},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":4.1379999999999999,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":1.141},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.2470000000000001,"15min":1.1910000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":5.694,"5min":5.694,"15min":5.694},"last":1.454}]},{"osd":2,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.454,"15min":1.3320000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":3.7149999999999999,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.022},{"interface":"front","average":{"1min":1.361,"5min":1.4630000000000001,"15min":1.347},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":3.6899999999999999,"5min":10.661,"15min":10.661},"last":1.321}]},{"osd":3,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2210000000000001,"5min":1.52,"15min":1.357},"min":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":8.0909999999999993,"5min":8.0909999999999993,"15min":8.0909999999999993},"last":1.256},{"interface":"front","average":{"1min":1.794,"5min":1.6459999999999999,"15min":1.4910000000000001},"min":{"1min":0.72699999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":8.3699999999999992,"5min":13.193,"15min":13.193},"last":1.2989999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.452,"15min":1.4099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.0430000000000001,"5min":16.376000000000001,"15min":16.376000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4330000000000001,"15min":1.4810000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.8039999999999998,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.3520000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.464,"15min":1.54},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":1.399},{"interface":"front","average":{"1min":1.421,"5min":1.474,"15min":1.544},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":3.3050000000000002,"5min":6.319,"15min":6.319},"last":1.21}]},{"osd":6,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.635,"15min":1.6839999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.234,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.6339999999999999,"15min":1.6799999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.8650000000000002,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.4930000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":692224,"data_stored":285657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":4.9279999999999999,"5min":1.909,"15min":1.405},"min":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":28.327999999999999,"5min":28.327999999999999,"15min":28.327999999999999},"last":1.5449999999999999},{"interface":"front","average":{"1min":5.0990000000000002,"5min":2.0059999999999998,"15min":1.4299999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":30.280999999999999,"5min":30.280999999999999,"15min":30.280999999999999},"last":1.3899999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":3.298,"5min":1.54,"15min":1.228},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":26.344000000000001,"5min":26.344000000000001,"15min":26.344000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.1679999999999999,"15min":1.1319999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":1.2889999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":3.4729999999999999,"5min":1.6759999999999999,"15min":1.3740000000000001},"min":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":17.991,"5min":17.991,"15min":17.991},"last":1.5089999999999999},{"interface":"front","average":{"1min":3.6499999999999999,"5min":1.8220000000000001,"15min":1.554},"min":{"1min":0.67100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":18.02,"5min":18.02,"15min":18.02},"last":1.5740000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":1.579,"15min":1.347},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":13.092000000000001,"5min":13.092000000000001,"15min":13.092000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":2.6080000000000001,"5min":1.5940000000000001,"15min":1.3779999999999999},"min":{"1min":0.69399999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":13.058999999999999,"5min":13.058999999999999,"15min":13.058999999999999},"last":1.1970000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.6060000000000001,"15min":1.413},"min":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":10.018000000000001,"5min":15.481999999999999,"15min":15.481999999999999},"last":1.337},{"interface":"front","average":{"1min":1.9610000000000001,"5min":1.512,"15min":1.45},"min":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.401999999999999,"5min":13.401999999999999,"15min":13.401999999999999},"last":1.141}]},{"osd":5,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3420000000000001,"15min":1.3129999999999999},"min":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.21,"5min":4.21,"15min":4.21},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4950000000000001,"15min":1.538},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.861,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.6719999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.5229999999999999,"15min":1.4690000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"last":1.474},{"interface":"front","average":{"1min":2.044,"5min":1.603,"15min":1.605},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":10.087,"5min":10.087,"15min":10.087},"last":1.6379999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":692224,"data_stored":285657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":3.1030000000000002,"5min":1.5129999999999999,"15min":1.2709999999999999},"min":{"1min":0.495,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":22.077000000000002,"5min":22.077000000000002,"15min":22.077000000000002},"last":0.98399999999999999},{"interface":"front","average":{"1min":6.0510000000000002,"5min":2.1549999999999998,"15min":1.397},"min":{"1min":0.61599999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":56.011000000000003,"5min":56.011000000000003,"15min":56.011000000000003},"last":0.90500000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.27,"5min":1.4059999999999999,"15min":1.2010000000000001},"min":{"1min":0.745,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":16.687999999999999,"5min":16.687999999999999,"15min":16.687999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":5.3109999999999999,"5min":2.2360000000000002,"15min":1.554},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":67.930000000000007,"5min":67.930000000000007,"15min":67.930000000000007},"last":1.3999999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.4490000000000001,"15min":1.2989999999999999},"min":{"1min":0.80900000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":1.1060000000000001},{"interface":"front","average":{"1min":2.343,"5min":1.484,"15min":1.3240000000000001},"min":{"1min":0.92300000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":11.09,"5min":11.09,"15min":11.09},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.4590000000000001,"5min":1.8320000000000001,"15min":1.458},"min":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":13.711,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.4340000000000002,"5min":1.7230000000000001,"15min":1.506},"min":{"1min":0.80800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":11.923,"5min":11.923,"15min":11.923},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4530000000000001,"15min":1.288},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.2909999999999999,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":1.054},{"interface":"front","average":{"1min":1.919,"5min":1.6259999999999999,"15min":1.4319999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":13.686,"5min":16.268000000000001,"15min":16.268000000000001},"last":1.5209999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7949999999999999,"5min":1.4730000000000001,"15min":1.413},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":11.706,"5min":11.706,"15min":11.706},"last":1.361},{"interface":"front","average":{"1min":1.593,"5min":1.585,"15min":1.508},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.206,"5min":7.335,"15min":7.335},"last":1.615}]},{"osd":7,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.786,"15min":1.613},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":14.345000000000001,"5min":14.345000000000001,"15min":14.345000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":2.0590000000000002,"5min":1.8009999999999999,"15min":1.6499999999999999},"min":{"1min":0.88200000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":11.954000000000001,"5min":12.673999999999999,"15min":12.673999999999999},"last":1.758}]}]},{"osd":4,"up_from":14,"seq":60129542211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":692224,"data_stored":285657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.2609999999999999,"15min":1.1479999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":12.587999999999999,"5min":12.587999999999999,"15min":12.587999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":4.5940000000000003,"5min":1.819,"15min":1.2829999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":60.106000000000002,"5min":60.106000000000002,"15min":60.106000000000002},"last":1.498}]},{"osd":1,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4449999999999998,"5min":1.643,"15min":1.403},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":9.6340000000000003,"5min":9.6340000000000003,"15min":9.6340000000000003},"last":0.84699999999999998},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.6040000000000001,"15min":1.3420000000000001},"min":{"1min":0.83799999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":9.5589999999999993,"5min":9.5589999999999993,"15min":9.5589999999999993},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.2769999999999999,"15min":1.1919999999999999},"min":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0860000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":3.7509999999999999,"5min":1.859,"15min":1.5109999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":35.024999999999999,"5min":35.024999999999999,"15min":35.024999999999999},"last":1.012}]},{"osd":3,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.494,"15min":1.4350000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.6980000000000004,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.2},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.379,"15min":1.3380000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":7.8140000000000001,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":0.91000000000000003}]},{"osd":5,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.415,"15min":1.4750000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.1190000000000002,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4970000000000001,"15min":1.5620000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.2989999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.4670000000000001,"15min":1.472},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":9.6630000000000003,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.571,"5min":1.5269999999999999,"15min":1.581},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":4.9649999999999999},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5780000000000001,"15min":1.673},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.2050000000000001,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.5800000000000001,"15min":1.613},"min":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.3540000000000001,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.3500000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44336,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327504,"statfs":{"total":96636764160,"available":96591364096,"internally_reserved":0,"allocated":692224,"data_stored":285657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":1.4810000000000001,"15min":1.107},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":13.241,"5min":17.709,"15min":17.709},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.216,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.2690000000000001,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.68899999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.2110000000000001,"15min":1.056},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":8.9689999999999994,"5min":8.9689999999999994,"15min":8.9689999999999994},"last":0.72899999999999998},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.5049999999999999,"15min":1.2889999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":14.893000000000001,"5min":14.893000000000001,"15min":14.893000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3320000000000001,"15min":1.2010000000000001},"min":{"1min":0.51900000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9780000000000002,"5min":8.8439999999999994,"15min":8.8439999999999994},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.575,"5min":1.4970000000000001,"15min":1.3280000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.444,"5min":10.25,"15min":10.25},"last":1.216}]},{"osd":4,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0939999999999999,"5min":1.5,"15min":1.391},"min":{"1min":0.69899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":7.0620000000000003,"5min":7.0620000000000003,"15min":7.0620000000000003},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.474,"15min":1.4379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.7200000000000002,"5min":3.7200000000000002,"15min":3.7200000000000002},"last":1.2649999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7350000000000001,"15min":1.5529999999999999},"min":{"1min":0.72399999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.758,"5min":12.65,"15min":12.65},"last":1.429},{"interface":"front","average":{"1min":1.9379999999999999,"5min":1.5940000000000001,"15min":1.456},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":6.5789999999999997,"5min":6.5789999999999997,"15min":6.5789999999999997},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.5309999999999999,"15min":1.4950000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.367,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":1.661},{"interface":"front","average":{"1min":1.962,"5min":1.792,"15min":1.629},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.8029999999999999,"5min":10.371,"15min":10.371},"last":1.5640000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":1.6319999999999999,"15min":1.5720000000000001},"min":{"1min":0.64600000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":6.6289999999999996,"5min":6.6289999999999996,"15min":6.6289999999999996},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.6499999999999999,"15min":1.5780000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":3.6669999999999998,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":1.762}]}]},{"osd":2,"up_from":14,"seq":60129542211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40176,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":692224,"data_stored":285657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":6.5640000000000001,"5min":2.2210000000000001,"15min":1.353},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":64.387,"5min":64.387,"15min":64.387},"last":1.357},{"interface":"front","average":{"1min":2.548,"5min":1.3859999999999999,"15min":1.1180000000000001},"min":{"1min":0.47399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":11.339,"5min":11.339,"15min":11.339},"last":1.53}]},{"osd":1,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":4.5309999999999997,"5min":1.845,"15min":1.2090000000000001},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":27.263000000000002,"5min":27.263000000000002,"15min":27.263000000000002},"last":1.155},{"interface":"front","average":{"1min":3.9129999999999998,"5min":1.877,"15min":1.4379999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":23.547999999999998,"5min":23.547999999999998,"15min":23.547999999999998},"last":1.216}]},{"osd":3,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":1.7350000000000001,"15min":1.3440000000000001},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":27.311,"5min":27.311,"15min":27.311},"last":1.4019999999999999},{"interface":"front","average":{"1min":3.4239999999999999,"5min":1.7909999999999999,"15min":1.351},"min":{"1min":0.47999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":26.555,"5min":26.555,"15min":26.555},"last":1.8500000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.0139999999999998,"5min":1.8029999999999999,"15min":1.4690000000000001},"min":{"1min":1.0149999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":26.503,"5min":26.503,"15min":26.503},"last":3.032},{"interface":"front","average":{"1min":2.383,"5min":1.585,"15min":1.4139999999999999},"min":{"1min":0.82699999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":11.188000000000001,"5min":11.188000000000001,"15min":11.188000000000001},"last":3.5630000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.8340000000000001,"5min":1.9099999999999999,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":27.149000000000001,"5min":27.149000000000001,"15min":27.149000000000001},"last":2.4159999999999999},{"interface":"front","average":{"1min":2.903,"5min":1.907,"15min":1.637},"min":{"1min":0.93000000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":26.349,"5min":26.349,"15min":26.349},"last":2.2850000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.4409999999999998,"5min":1.6399999999999999,"15min":1.488},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":11.266,"5min":11.266,"15min":11.266},"last":3.891},{"interface":"front","average":{"1min":3.831,"5min":2.1469999999999998,"15min":1.6970000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":27.076000000000001,"5min":27.076000000000001,"15min":27.076000000000001},"last":3.1850000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":1.823,"15min":1.6739999999999999},"min":{"1min":1.0649999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":11.303000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":3.7989999999999999},{"interface":"front","average":{"1min":4.0570000000000004,"5min":2.0459999999999998,"15min":1.6499999999999999},"min":{"1min":0.873,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":27.207999999999998,"5min":27.207999999999998,"15min":27.207999999999998},"last":4.1879999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542211,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40192,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331648,"statfs":{"total":96636764160,"available":96595607552,"internally_reserved":0,"allocated":704512,"data_stored":291952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.359,"15min":1.133},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1310000000000002,"5min":14.407,"15min":14.407},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.2230000000000001,"15min":1.145},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":8.5090000000000003,"5min":8.5090000000000003,"15min":8.5090000000000003},"last":0.70199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1519999999999999,"15min":1.097},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.3050000000000002,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":2.3420000000000001,"5min":1.8029999999999999,"15min":1.403},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":22.550999999999998,"5min":22.550999999999998,"15min":22.550999999999998},"last":0.89500000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.8180000000000001,"15min":1.47},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":5.9969999999999999,"5min":14.653,"15min":14.653},"last":1.081},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.5309999999999999,"15min":1.375},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":9.4169999999999998,"5min":9.4169999999999998,"15min":9.4169999999999998},"last":0.86399999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.637,"15min":1.4850000000000001},"min":{"1min":0.84099999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":5.3579999999999997,"5min":12.675000000000001,"15min":12.675000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.639,"15min":1.5169999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":4.7969999999999997,"5min":12.731999999999999,"15min":12.731999999999999},"last":1.2549999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.591,"15min":1.5649999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":5.4660000000000002,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.234},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.4870000000000001,"15min":1.484},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.9589999999999996,"5min":6.9569999999999999,"15min":6.9569999999999999},"last":1.3520000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.982,"15min":1.726},"min":{"1min":1.014,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":5.0549999999999997,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.45},{"interface":"front","average":{"1min":1.75,"5min":1.726,"15min":1.677},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":5.4290000000000003,"5min":6.6779999999999999,"15min":6.6779999999999999},"last":1.3129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.6020000000000001,"15min":1.5860000000000001},"min":{"1min":0.99299999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.9169999999999998,"5min":6.75,"15min":6.75},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.819,"5min":2.0630000000000002,"15min":1.8},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":5.0010000000000003,"5min":14.884,"15min":14.884},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542212,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40756,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331084,"statfs":{"total":96636764160,"available":96595030016,"internally_reserved":0,"allocated":1286144,"data_stored":876025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.321,"15min":1.0349999999999999},"min":{"1min":0.60799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.53600000000000003},{"interface":"front","average":{"1min":3.871,"5min":1.6439999999999999,"15min":1.1479999999999999},"min":{"1min":0.81599999999999995,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.414000000000001,"5min":24.414000000000001,"15min":24.414000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.3380000000000001,"15min":1.1120000000000001},"min":{"1min":0.57699999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":5.0460000000000003,"5min":5.444,"15min":5.444},"last":0.40400000000000003},{"interface":"front","average":{"1min":4.3010000000000002,"5min":1.758,"15min":1.2350000000000001},"min":{"1min":0.81100000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":38.893000000000001,"5min":38.893000000000001,"15min":38.893000000000001},"last":0.50900000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.829,"15min":1.2210000000000001},"min":{"1min":0.63900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":18.436,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.99,"5min":1.4850000000000001,"15min":1.2609999999999999},"min":{"1min":0.86799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":4.407,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.444,"5min":1.573,"15min":1.2929999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":20.327000000000002,"5min":20.327000000000002,"15min":20.327000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":2.0659999999999998,"5min":1.6319999999999999,"15min":1.4279999999999999},"min":{"1min":0.92500000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":4.7599999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.85099999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":4.8040000000000003,"5min":2.0630000000000002,"15min":1.5349999999999999},"min":{"1min":1.171,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":24.611000000000001,"5min":24.611000000000001,"15min":24.611000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":2.9119999999999999,"5min":1.7450000000000001,"15min":1.4350000000000001},"min":{"1min":0.98899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":20.199000000000002,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.873}]},{"osd":6,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":1.734,"15min":1.5640000000000001},"min":{"1min":0.99099999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.754},{"interface":"front","average":{"1min":4.3659999999999997,"5min":2.1139999999999999,"15min":1.613},"min":{"1min":1.1970000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":24.478999999999999,"5min":24.478999999999999,"15min":24.478999999999999},"last":0.78800000000000003}]},{"osd":7,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.423,"5min":1.95,"15min":1.6379999999999999},"min":{"1min":1.3300000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.957000000000001,"5min":17.957000000000001,"15min":17.957000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":2.6480000000000001,"5min":1.8140000000000001,"15min":1.546},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":20.279,"5min":20.279,"15min":20.279},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:04:50.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:04:50.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.724+0000 7f7743a00640 1 Processor -- start 2024-01-16T15:04:50.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.724+0000 7f7743a00640 1 -- start start 2024-01-16T15:04:50.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.725+0000 7f7743a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f773c153f10 0x7f773c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.725+0000 7f7743a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7743a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0ad6b0 con 0x7f773c06ddb0 2024-01-16T15:04:50.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7743a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c0b25c0 con 0x7f773c153f10 2024-01-16T15:04:50.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7743a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0b2860 con 0x7f773c15b050 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f773c153f10 0x7f773c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f773c153f10 0x7f773c15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50856/0 (socket says 172.21.15.43:50856) 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 -- 172.21.15.43:0/728626876 learned_addr learned my addr 172.21.15.43:0/728626876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f773be00640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 -- 172.21.15.43:0/728626876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 msgr2=0x7f773c0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c0b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 -- 172.21.15.43:0/728626876 >> v1:172.21.15.43:6789/0 conn(0x7f773c06ddb0 legacy=0x7f773c15d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f7741000640 1 -- 172.21.15.43:0/728626876 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0b2b40 con 0x7f773c153f10 2024-01-16T15:04:50.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.726+0000 7f773be00640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c0b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:50.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.728+0000 7f7741000640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c15aaa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f772400bb10 tx=0x7f772402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523685b6aa30652e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.728+0000 7f773b400640 1 -- 172.21.15.43:0/728626876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724036440 con 0x7f773c153f10 2024-01-16T15:04:50.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.728+0000 7f773b400640 1 -- 172.21.15.43:0/728626876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77240365a0 con 0x7f773c153f10 2024-01-16T15:04:50.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.728+0000 7f773b400640 1 -- 172.21.15.43:0/728626876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724036930 con 0x7f773c153f10 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 -- 172.21.15.43:0/728626876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 msgr2=0x7f773c15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c15aaa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f772400bb10 tx=0x7f772402e150 comp rx=0 tx=0).stop 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 -- 172.21.15.43:0/728626876 shutdown_connections 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c0b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 --2- 172.21.15.43:0/728626876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c15aaa0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 -- 172.21.15.43:0/728626876 >> 172.21.15.43:0/728626876 conn(0x7f773c0b0d80 msgr2=0x7f773c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 -- 172.21.15.43:0/728626876 shutdown_connections 2024-01-16T15:04:50.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.729+0000 7f7743a00640 1 -- 172.21.15.43:0/728626876 wait complete. 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.731+0000 7f7743a00640 1 Processor -- start 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.731+0000 7f7743a00640 1 -- start start 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f773c06ddb0 0x7f773c14b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c14aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c14e550 con 0x7f773c153f10 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c14f240 con 0x7f773c06ddb0 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f7743a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c14d080 con 0x7f773c15b050 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f773be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f773be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50864/0 (socket says 172.21.15.43:50864) 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.732+0000 7f773be00640 1 -- 172.21.15.43:0/1860383544 learned_addr learned my addr 172.21.15.43:0/1860383544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7741000640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f773c06ddb0 0x7f773c14b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 -- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 msgr2=0x7f773c14aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c14aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 -- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f773c06ddb0 msgr2=0x7f773c14b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f773c06ddb0 0x7f773c14b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 -- 172.21.15.43:0/1860383544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772400b770 con 0x7f773c153f10 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f773be00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f772c007f20 tx=0x7f772c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c00dee0 con 0x7f773c153f10 2024-01-16T15:04:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c00f040 con 0x7f773c153f10 2024-01-16T15:04:50.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c003db0 con 0x7f773c153f10 2024-01-16T15:04:50.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c14cda0 con 0x7f773c153f10 2024-01-16T15:04:50.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.733+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c14ed20 con 0x7f773c153f10 2024-01-16T15:04:50.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.735+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c14e880 con 0x7f773c153f10 2024-01-16T15:04:50.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.736+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f772c004030 con 0x7f773c153f10 2024-01-16T15:04:50.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.736+0000 7f7738c00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 0x7f771406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.737+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f772c08a900 con 0x7f773c153f10 2024-01-16T15:04:50.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.737+0000 7f7741000640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 0x7f771406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.737+0000 7f7741000640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 0x7f771406dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7724038000 tx=0x7f7724008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.741+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f772c054f10 con 0x7f773c153f10 2024-01-16T15:04:50.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.934+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f773c1542f0 con 0x7f771406b890 2024-01-16T15:04:50.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.959+0000 7f7738c00640 1 -- 172.21.15.43:0/1860383544 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287661 (secure 0 0 0) 0x7f772403e3b0 con 0x7f771406b890 2024-01-16T15:04:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 msgr2=0x7f771406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 0x7f771406dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7724038000 tx=0x7f7724008040 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 msgr2=0x7f773c14c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f772c007f20 tx=0x7f772c00d560 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 shutdown_connections 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f771406b890 0x7f771406dd50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f773c15b050 0x7f773c14aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f773c153f10 0x7f773c14c550 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 --2- 172.21.15.43:0/1860383544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f773c06ddb0 0x7f773c14b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 >> 172.21.15.43:0/1860383544 conn(0x7f773c0b0d80 msgr2=0x7f773c151fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 shutdown_connections 2024-01-16T15:04:50.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:50.966+0000 7f7743a00640 1 -- 172.21.15.43:0/1860383544 wait complete. 2024-01-16T15:04:50.966 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-01-16T15:04:50.907686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310448,"kb_used_data":6920,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754664272,"statfs":{"total":773094113280,"available":772776214528,"internally_reserved":0,"allocated":7086080,"data_stored":3626220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":310693420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001882"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075381+0000","last_change":"2024-01-16T15:04:45.948263+0000","last_active":"2024-01-16T15:04:47.075381+0000","last_peered":"2024-01-16T15:04:47.075381+0000","last_clean":"2024-01-16T15:04:47.075381+0000","last_became_active":"2024-01-16T15:04:45.947956+0000","last_became_peered":"2024-01-16T15:04:45.947956+0000","last_unstale":"2024-01-16T15:04:47.075381+0000","last_undegraded":"2024-01-16T15:04:47.075381+0000","last_fullsized":"2024-01-16T15:04:47.075381+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:14.944788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941570+0000","last_change":"2024-01-16T15:04:45.946057+0000","last_active":"2024-01-16T15:04:46.941570+0000","last_peered":"2024-01-16T15:04:46.941570+0000","last_clean":"2024-01-16T15:04:46.941570+0000","last_became_active":"2024-01-16T15:04:45.945739+0000","last_became_peered":"2024-01-16T15:04:45.945739+0000","last_unstale":"2024-01-16T15:04:46.941570+0000","last_undegraded":"2024-01-16T15:04:46.941570+0000","last_fullsized":"2024-01-16T15:04:46.941570+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:35.605765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077232+0000","last_change":"2024-01-16T15:04:45.962793+0000","last_active":"2024-01-16T15:04:47.077232+0000","last_peered":"2024-01-16T15:04:47.077232+0000","last_clean":"2024-01-16T15:04:47.077232+0000","last_became_active":"2024-01-16T15:04:45.962504+0000","last_became_peered":"2024-01-16T15:04:45.962504+0000","last_unstale":"2024-01-16T15:04:47.077232+0000","last_undegraded":"2024-01-16T15:04:47.077232+0000","last_fullsized":"2024-01-16T15:04:47.077232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:13.592506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.945587+0000","last_change":"2024-01-16T15:04:45.945143+0000","last_active":"2024-01-16T15:04:46.945587+0000","last_peered":"2024-01-16T15:04:46.945587+0000","last_clean":"2024-01-16T15:04:46.945587+0000","last_became_active":"2024-01-16T15:04:45.944753+0000","last_became_peered":"2024-01-16T15:04:45.944753+0000","last_unstale":"2024-01-16T15:04:46.945587+0000","last_undegraded":"2024-01-16T15:04:46.945587+0000","last_fullsized":"2024-01-16T15:04:46.945587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:37.416515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075170+0000","last_change":"2024-01-16T15:04:45.951476+0000","last_active":"2024-01-16T15:04:47.075170+0000","last_peered":"2024-01-16T15:04:47.075170+0000","last_clean":"2024-01-16T15:04:47.075170+0000","last_became_active":"2024-01-16T15:04:45.951205+0000","last_became_peered":"2024-01-16T15:04:45.951205+0000","last_unstale":"2024-01-16T15:04:47.075170+0000","last_undegraded":"2024-01-16T15:04:47.075170+0000","last_fullsized":"2024-01-16T15:04:47.075170+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:37.403296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.074485+0000","last_change":"2024-01-16T15:04:45.941384+0000","last_active":"2024-01-16T15:04:47.074485+0000","last_peered":"2024-01-16T15:04:47.074485+0000","last_clean":"2024-01-16T15:04:47.074485+0000","last_became_active":"2024-01-16T15:04:45.940095+0000","last_became_peered":"2024-01-16T15:04:45.940095+0000","last_unstale":"2024-01-16T15:04:47.074485+0000","last_undegraded":"2024-01-16T15:04:47.074485+0000","last_fullsized":"2024-01-16T15:04:47.074485+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:57:17.950508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075074+0000","last_change":"2024-01-16T15:04:45.945824+0000","last_active":"2024-01-16T15:04:47.075074+0000","last_peered":"2024-01-16T15:04:47.075074+0000","last_clean":"2024-01-16T15:04:47.075074+0000","last_became_active":"2024-01-16T15:04:45.945487+0000","last_became_peered":"2024-01-16T15:04:45.945487+0000","last_unstale":"2024-01-16T15:04:47.075074+0000","last_undegraded":"2024-01-16T15:04:47.075074+0000","last_fullsized":"2024-01-16T15:04:47.075074+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:25:35.956800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940578+0000","last_change":"2024-01-16T15:04:45.941570+0000","last_active":"2024-01-16T15:04:46.940578+0000","last_peered":"2024-01-16T15:04:46.940578+0000","last_clean":"2024-01-16T15:04:46.940578+0000","last_became_active":"2024-01-16T15:04:45.941141+0000","last_became_peered":"2024-01-16T15:04:45.941141+0000","last_unstale":"2024-01-16T15:04:46.940578+0000","last_undegraded":"2024-01-16T15:04:46.940578+0000","last_fullsized":"2024-01-16T15:04:46.940578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:38.720922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076761+0000","last_change":"2024-01-16T15:04:45.939036+0000","last_active":"2024-01-16T15:04:47.076761+0000","last_peered":"2024-01-16T15:04:47.076761+0000","last_clean":"2024-01-16T15:04:47.076761+0000","last_became_active":"2024-01-16T15:04:45.938598+0000","last_became_peered":"2024-01-16T15:04:45.938598+0000","last_unstale":"2024-01-16T15:04:47.076761+0000","last_undegraded":"2024-01-16T15:04:47.076761+0000","last_fullsized":"2024-01-16T15:04:47.076761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:12.026824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077137+0000","last_change":"2024-01-16T15:04:45.953712+0000","last_active":"2024-01-16T15:04:47.077137+0000","last_peered":"2024-01-16T15:04:47.077137+0000","last_clean":"2024-01-16T15:04:47.077137+0000","last_became_active":"2024-01-16T15:04:45.953437+0000","last_became_peered":"2024-01-16T15:04:45.953437+0000","last_unstale":"2024-01-16T15:04:47.077137+0000","last_undegraded":"2024-01-16T15:04:47.077137+0000","last_fullsized":"2024-01-16T15:04:47.077137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:16.664877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076803+0000","last_change":"2024-01-16T15:04:45.946177+0000","last_active":"2024-01-16T15:04:47.076803+0000","last_peered":"2024-01-16T15:04:47.076803+0000","last_clean":"2024-01-16T15:04:47.076803+0000","last_became_active":"2024-01-16T15:04:45.945870+0000","last_became_peered":"2024-01-16T15:04:45.945870+0000","last_unstale":"2024-01-16T15:04:47.076803+0000","last_undegraded":"2024-01-16T15:04:47.076803+0000","last_fullsized":"2024-01-16T15:04:47.076803+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:58.918561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.945549+0000","last_change":"2024-01-16T15:04:45.951641+0000","last_active":"2024-01-16T15:04:46.945549+0000","last_peered":"2024-01-16T15:04:46.945549+0000","last_clean":"2024-01-16T15:04:46.945549+0000","last_became_active":"2024-01-16T15:04:45.951362+0000","last_became_peered":"2024-01-16T15:04:45.951362+0000","last_unstale":"2024-01-16T15:04:46.945549+0000","last_undegraded":"2024-01-16T15:04:46.945549+0000","last_fullsized":"2024-01-16T15:04:46.945549+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:56.091757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.943502+0000","last_change":"2024-01-16T15:04:45.943932+0000","last_active":"2024-01-16T15:04:46.943502+0000","last_peered":"2024-01-16T15:04:46.943502+0000","last_clean":"2024-01-16T15:04:46.943502+0000","last_became_active":"2024-01-16T15:04:45.942609+0000","last_became_peered":"2024-01-16T15:04:45.942609+0000","last_unstale":"2024-01-16T15:04:46.943502+0000","last_undegraded":"2024-01-16T15:04:46.943502+0000","last_fullsized":"2024-01-16T15:04:46.943502+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:25.275039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075553+0000","last_change":"2024-01-16T15:04:45.940662+0000","last_active":"2024-01-16T15:04:47.075553+0000","last_peered":"2024-01-16T15:04:47.075553+0000","last_clean":"2024-01-16T15:04:47.075553+0000","last_became_active":"2024-01-16T15:04:45.940173+0000","last_became_peered":"2024-01-16T15:04:45.940173+0000","last_unstale":"2024-01-16T15:04:47.075553+0000","last_undegraded":"2024-01-16T15:04:47.075553+0000","last_fullsized":"2024-01-16T15:04:47.075553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:30.852075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075297+0000","last_change":"2024-01-16T15:04:45.943418+0000","last_active":"2024-01-16T15:04:47.075297+0000","last_peered":"2024-01-16T15:04:47.075297+0000","last_clean":"2024-01-16T15:04:47.075297+0000","last_became_active":"2024-01-16T15:04:45.941398+0000","last_became_peered":"2024-01-16T15:04:45.941398+0000","last_unstale":"2024-01-16T15:04:47.075297+0000","last_undegraded":"2024-01-16T15:04:47.075297+0000","last_fullsized":"2024-01-16T15:04:47.075297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:10:47.253335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.073366+0000","last_change":"2024-01-16T15:04:45.935832+0000","last_active":"2024-01-16T15:04:47.073366+0000","last_peered":"2024-01-16T15:04:47.073366+0000","last_clean":"2024-01-16T15:04:47.073366+0000","last_became_active":"2024-01-16T15:04:45.934129+0000","last_became_peered":"2024-01-16T15:04:45.934129+0000","last_unstale":"2024-01-16T15:04:47.073366+0000","last_undegraded":"2024-01-16T15:04:47.073366+0000","last_fullsized":"2024-01-16T15:04:47.073366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:56.586132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941637+0000","last_change":"2024-01-16T15:04:45.941980+0000","last_active":"2024-01-16T15:04:46.941637+0000","last_peered":"2024-01-16T15:04:46.941637+0000","last_clean":"2024-01-16T15:04:46.941637+0000","last_became_active":"2024-01-16T15:04:45.941615+0000","last_became_peered":"2024-01-16T15:04:45.941615+0000","last_unstale":"2024-01-16T15:04:46.941637+0000","last_undegraded":"2024-01-16T15:04:46.941637+0000","last_fullsized":"2024-01-16T15:04:46.941637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:20.286760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.078057+0000","last_change":"2024-01-16T15:04:45.966558+0000","last_active":"2024-01-16T15:04:47.078057+0000","last_peered":"2024-01-16T15:04:47.078057+0000","last_clean":"2024-01-16T15:04:47.078057+0000","last_became_active":"2024-01-16T15:04:45.965760+0000","last_became_peered":"2024-01-16T15:04:45.965760+0000","last_unstale":"2024-01-16T15:04:47.078057+0000","last_undegraded":"2024-01-16T15:04:47.078057+0000","last_fullsized":"2024-01-16T15:04:47.078057+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:42:17.162841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.073806+0000","last_change":"2024-01-16T15:04:45.945773+0000","last_active":"2024-01-16T15:04:47.073806+0000","last_peered":"2024-01-16T15:04:47.073806+0000","last_clean":"2024-01-16T15:04:47.073806+0000","last_became_active":"2024-01-16T15:04:45.945474+0000","last_became_peered":"2024-01-16T15:04:45.945474+0000","last_unstale":"2024-01-16T15:04:47.073806+0000","last_undegraded":"2024-01-16T15:04:47.073806+0000","last_fullsized":"2024-01-16T15:04:47.073806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:20.972458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941437+0000","last_change":"2024-01-16T15:04:45.942793+0000","last_active":"2024-01-16T15:04:46.941437+0000","last_peered":"2024-01-16T15:04:46.941437+0000","last_clean":"2024-01-16T15:04:46.941437+0000","last_became_active":"2024-01-16T15:04:45.942356+0000","last_became_peered":"2024-01-16T15:04:45.942356+0000","last_unstale":"2024-01-16T15:04:46.941437+0000","last_undegraded":"2024-01-16T15:04:46.941437+0000","last_fullsized":"2024-01-16T15:04:46.941437+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:39.501141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076325+0000","last_change":"2024-01-16T15:04:45.954702+0000","last_active":"2024-01-16T15:04:47.076325+0000","last_peered":"2024-01-16T15:04:47.076325+0000","last_clean":"2024-01-16T15:04:47.076325+0000","last_became_active":"2024-01-16T15:04:45.954423+0000","last_became_peered":"2024-01-16T15:04:45.954423+0000","last_unstale":"2024-01-16T15:04:47.076325+0000","last_undegraded":"2024-01-16T15:04:47.076325+0000","last_fullsized":"2024-01-16T15:04:47.076325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:51.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940803+0000","last_change":"2024-01-16T15:04:45.942670+0000","last_active":"2024-01-16T15:04:46.940803+0000","last_peered":"2024-01-16T15:04:46.940803+0000","last_clean":"2024-01-16T15:04:46.940803+0000","last_became_active":"2024-01-16T15:04:45.942374+0000","last_became_peered":"2024-01-16T15:04:45.942374+0000","last_unstale":"2024-01-16T15:04:46.940803+0000","last_undegraded":"2024-01-16T15:04:46.940803+0000","last_fullsized":"2024-01-16T15:04:46.940803+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:26:36.076649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076027+0000","last_change":"2024-01-16T15:04:45.953750+0000","last_active":"2024-01-16T15:04:47.076027+0000","last_peered":"2024-01-16T15:04:47.076027+0000","last_clean":"2024-01-16T15:04:47.076027+0000","last_became_active":"2024-01-16T15:04:45.953453+0000","last_became_peered":"2024-01-16T15:04:45.953453+0000","last_unstale":"2024-01-16T15:04:47.076027+0000","last_undegraded":"2024-01-16T15:04:47.076027+0000","last_fullsized":"2024-01-16T15:04:47.076027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:38:18.550233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.943119+0000","last_change":"2024-01-16T15:04:45.943278+0000","last_active":"2024-01-16T15:04:46.943119+0000","last_peered":"2024-01-16T15:04:46.943119+0000","last_clean":"2024-01-16T15:04:46.943119+0000","last_became_active":"2024-01-16T15:04:45.942939+0000","last_became_peered":"2024-01-16T15:04:45.942939+0000","last_unstale":"2024-01-16T15:04:46.943119+0000","last_undegraded":"2024-01-16T15:04:46.943119+0000","last_fullsized":"2024-01-16T15:04:46.943119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:10.229606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075823+0000","last_change":"2024-01-16T15:04:45.945364+0000","last_active":"2024-01-16T15:04:47.075823+0000","last_peered":"2024-01-16T15:04:47.075823+0000","last_clean":"2024-01-16T15:04:47.075823+0000","last_became_active":"2024-01-16T15:04:45.944886+0000","last_became_peered":"2024-01-16T15:04:45.944886+0000","last_unstale":"2024-01-16T15:04:47.075823+0000","last_undegraded":"2024-01-16T15:04:47.075823+0000","last_fullsized":"2024-01-16T15:04:47.075823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:34.229373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.945781+0000","last_change":"2024-01-16T15:04:45.946104+0000","last_active":"2024-01-16T15:04:46.945781+0000","last_peered":"2024-01-16T15:04:46.945781+0000","last_clean":"2024-01-16T15:04:46.945781+0000","last_became_active":"2024-01-16T15:04:45.945789+0000","last_became_peered":"2024-01-16T15:04:45.945789+0000","last_unstale":"2024-01-16T15:04:46.945781+0000","last_undegraded":"2024-01-16T15:04:46.945781+0000","last_fullsized":"2024-01-16T15:04:46.945781+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:52.405525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.074907+0000","last_change":"2024-01-16T15:04:45.935481+0000","last_active":"2024-01-16T15:04:47.074907+0000","last_peered":"2024-01-16T15:04:47.074907+0000","last_clean":"2024-01-16T15:04:47.074907+0000","last_became_active":"2024-01-16T15:04:45.935202+0000","last_became_peered":"2024-01-16T15:04:45.935202+0000","last_unstale":"2024-01-16T15:04:47.074907+0000","last_undegraded":"2024-01-16T15:04:47.074907+0000","last_fullsized":"2024-01-16T15:04:47.074907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:35.116397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.074178+0000","last_change":"2024-01-16T15:04:45.951373+0000","last_active":"2024-01-16T15:04:47.074178+0000","last_peered":"2024-01-16T15:04:47.074178+0000","last_clean":"2024-01-16T15:04:47.074178+0000","last_became_active":"2024-01-16T15:04:45.951037+0000","last_became_peered":"2024-01-16T15:04:45.951037+0000","last_unstale":"2024-01-16T15:04:47.074178+0000","last_undegraded":"2024-01-16T15:04:47.074178+0000","last_fullsized":"2024-01-16T15:04:47.074178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:55.460022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941933+0000","last_change":"2024-01-16T15:04:45.951254+0000","last_active":"2024-01-16T15:04:46.941933+0000","last_peered":"2024-01-16T15:04:46.941933+0000","last_clean":"2024-01-16T15:04:46.941933+0000","last_became_active":"2024-01-16T15:04:45.950947+0000","last_became_peered":"2024-01-16T15:04:45.950947+0000","last_unstale":"2024-01-16T15:04:46.941933+0000","last_undegraded":"2024-01-16T15:04:46.941933+0000","last_fullsized":"2024-01-16T15:04:46.941933+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:31:28.718866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076113+0000","last_change":"2024-01-16T15:04:45.953610+0000","last_active":"2024-01-16T15:04:47.076113+0000","last_peered":"2024-01-16T15:04:47.076113+0000","last_clean":"2024-01-16T15:04:47.076113+0000","last_became_active":"2024-01-16T15:04:45.953256+0000","last_became_peered":"2024-01-16T15:04:45.953256+0000","last_unstale":"2024-01-16T15:04:47.076113+0000","last_undegraded":"2024-01-16T15:04:47.076113+0000","last_fullsized":"2024-01-16T15:04:47.076113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:14.199899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.073274+0000","last_change":"2024-01-16T15:04:45.948023+0000","last_active":"2024-01-16T15:04:47.073274+0000","last_peered":"2024-01-16T15:04:47.073274+0000","last_clean":"2024-01-16T15:04:47.073274+0000","last_became_active":"2024-01-16T15:04:45.947725+0000","last_became_peered":"2024-01-16T15:04:45.947725+0000","last_unstale":"2024-01-16T15:04:47.073274+0000","last_undegraded":"2024-01-16T15:04:47.073274+0000","last_fullsized":"2024-01-16T15:04:47.073274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:50.479489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.073513+0000","last_change":"2024-01-16T15:04:45.957282+0000","last_active":"2024-01-16T15:04:47.073513+0000","last_peered":"2024-01-16T15:04:47.073513+0000","last_clean":"2024-01-16T15:04:47.073513+0000","last_became_active":"2024-01-16T15:04:45.946074+0000","last_became_peered":"2024-01-16T15:04:45.946074+0000","last_unstale":"2024-01-16T15:04:47.073513+0000","last_undegraded":"2024-01-16T15:04:47.073513+0000","last_fullsized":"2024-01-16T15:04:47.073513+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:47.391710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075565+0000","last_change":"2024-01-16T15:04:45.958123+0000","last_active":"2024-01-16T15:04:47.075565+0000","last_peered":"2024-01-16T15:04:47.075565+0000","last_clean":"2024-01-16T15:04:47.075565+0000","last_became_active":"2024-01-16T15:04:45.957899+0000","last_became_peered":"2024-01-16T15:04:45.957899+0000","last_unstale":"2024-01-16T15:04:47.075565+0000","last_undegraded":"2024-01-16T15:04:47.075565+0000","last_fullsized":"2024-01-16T15:04:47.075565+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:33.209939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"60'4","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.970857+0000","last_change":"2024-01-16T15:04:44.958382+0000","last_active":"2024-01-16T15:04:45.970857+0000","last_peered":"2024-01-16T15:04:45.970857+0000","last_clean":"2024-01-16T15:04:45.970857+0000","last_became_active":"2024-01-16T15:04:44.958066+0000","last_became_peered":"2024-01-16T15:04:44.958066+0000","last_unstale":"2024-01-16T15:04:45.970857+0000","last_undegraded":"2024-01-16T15:04:45.970857+0000","last_fullsized":"2024-01-16T15:04:45.970857+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:37.374750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077212+0000","last_change":"2024-01-16T15:04:45.956586+0000","last_active":"2024-01-16T15:04:47.077212+0000","last_peered":"2024-01-16T15:04:47.077212+0000","last_clean":"2024-01-16T15:04:47.077212+0000","last_became_active":"2024-01-16T15:04:45.956294+0000","last_became_peered":"2024-01-16T15:04:45.956294+0000","last_unstale":"2024-01-16T15:04:47.077212+0000","last_undegraded":"2024-01-16T15:04:47.077212+0000","last_fullsized":"2024-01-16T15:04:47.077212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:29.332376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"60'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.969633+0000","last_change":"2024-01-16T15:04:44.952284+0000","last_active":"2024-01-16T15:04:45.969633+0000","last_peered":"2024-01-16T15:04:45.969633+0000","last_clean":"2024-01-16T15:04:45.969633+0000","last_became_active":"2024-01-16T15:04:44.951936+0000","last_became_peered":"2024-01-16T15:04:44.951936+0000","last_unstale":"2024-01-16T15:04:45.969633+0000","last_undegraded":"2024-01-16T15:04:45.969633+0000","last_fullsized":"2024-01-16T15:04:45.969633+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:03.634333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.074907+0000","last_change":"2024-01-16T15:04:45.941823+0000","last_active":"2024-01-16T15:04:47.074907+0000","last_peered":"2024-01-16T15:04:47.074907+0000","last_clean":"2024-01-16T15:04:47.074907+0000","last_became_active":"2024-01-16T15:04:45.941449+0000","last_became_peered":"2024-01-16T15:04:45.941449+0000","last_unstale":"2024-01-16T15:04:47.074907+0000","last_undegraded":"2024-01-16T15:04:47.074907+0000","last_fullsized":"2024-01-16T15:04:47.074907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:04:27.764295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.970257+0000","last_change":"2024-01-16T15:04:44.941385+0000","last_active":"2024-01-16T15:04:45.970257+0000","last_peered":"2024-01-16T15:04:45.970257+0000","last_clean":"2024-01-16T15:04:45.970257+0000","last_became_active":"2024-01-16T15:04:44.941068+0000","last_became_peered":"2024-01-16T15:04:44.941068+0000","last_unstale":"2024-01-16T15:04:45.970257+0000","last_undegraded":"2024-01-16T15:04:45.970257+0000","last_fullsized":"2024-01-16T15:04:45.970257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:58.007542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075041+0000","last_change":"2024-01-16T15:04:45.959408+0000","last_active":"2024-01-16T15:04:47.075041+0000","last_peered":"2024-01-16T15:04:47.075041+0000","last_clean":"2024-01-16T15:04:47.075041+0000","last_became_active":"2024-01-16T15:04:45.957180+0000","last_became_peered":"2024-01-16T15:04:45.957180+0000","last_unstale":"2024-01-16T15:04:47.075041+0000","last_undegraded":"2024-01-16T15:04:47.075041+0000","last_fullsized":"2024-01-16T15:04:47.075041+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:13.713936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.969931+0000","last_change":"2024-01-16T15:04:44.951546+0000","last_active":"2024-01-16T15:04:45.969931+0000","last_peered":"2024-01-16T15:04:45.969931+0000","last_clean":"2024-01-16T15:04:45.969931+0000","last_became_active":"2024-01-16T15:04:44.949430+0000","last_became_peered":"2024-01-16T15:04:44.949430+0000","last_unstale":"2024-01-16T15:04:45.969931+0000","last_undegraded":"2024-01-16T15:04:45.969931+0000","last_fullsized":"2024-01-16T15:04:45.969931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.144535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940798+0000","last_change":"2024-01-16T15:04:45.942097+0000","last_active":"2024-01-16T15:04:46.940798+0000","last_peered":"2024-01-16T15:04:46.940798+0000","last_clean":"2024-01-16T15:04:46.940798+0000","last_became_active":"2024-01-16T15:04:45.941781+0000","last_became_peered":"2024-01-16T15:04:45.941781+0000","last_unstale":"2024-01-16T15:04:46.940798+0000","last_undegraded":"2024-01-16T15:04:46.940798+0000","last_fullsized":"2024-01-16T15:04:46.940798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:22.607811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"60'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.965325+0000","last_change":"2024-01-16T15:04:44.940904+0000","last_active":"2024-01-16T15:04:45.965325+0000","last_peered":"2024-01-16T15:04:45.965325+0000","last_clean":"2024-01-16T15:04:45.965325+0000","last_became_active":"2024-01-16T15:04:44.939860+0000","last_became_peered":"2024-01-16T15:04:44.939860+0000","last_unstale":"2024-01-16T15:04:45.965325+0000","last_undegraded":"2024-01-16T15:04:45.965325+0000","last_fullsized":"2024-01-16T15:04:45.965325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:08:09.214978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940837+0000","last_change":"2024-01-16T15:04:45.947950+0000","last_active":"2024-01-16T15:04:46.940837+0000","last_peered":"2024-01-16T15:04:46.940837+0000","last_clean":"2024-01-16T15:04:46.940837+0000","last_became_active":"2024-01-16T15:04:45.946965+0000","last_became_peered":"2024-01-16T15:04:45.946965+0000","last_unstale":"2024-01-16T15:04:46.940837+0000","last_undegraded":"2024-01-16T15:04:46.940837+0000","last_fullsized":"2024-01-16T15:04:46.940837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:37.327038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"60'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.969726+0000","last_change":"2024-01-16T15:04:44.949930+0000","last_active":"2024-01-16T15:04:45.969726+0000","last_peered":"2024-01-16T15:04:45.969726+0000","last_clean":"2024-01-16T15:04:45.969726+0000","last_became_active":"2024-01-16T15:04:44.949670+0000","last_became_peered":"2024-01-16T15:04:44.949670+0000","last_unstale":"2024-01-16T15:04:45.969726+0000","last_undegraded":"2024-01-16T15:04:45.969726+0000","last_fullsized":"2024-01-16T15:04:45.969726+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:12.579518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.943281+0000","last_change":"2024-01-16T15:04:45.943710+0000","last_active":"2024-01-16T15:04:46.943281+0000","last_peered":"2024-01-16T15:04:46.943281+0000","last_clean":"2024-01-16T15:04:46.943281+0000","last_became_active":"2024-01-16T15:04:45.943016+0000","last_became_peered":"2024-01-16T15:04:45.943016+0000","last_unstale":"2024-01-16T15:04:46.943281+0000","last_undegraded":"2024-01-16T15:04:46.943281+0000","last_fullsized":"2024-01-16T15:04:46.943281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:06.266830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.933768+0000","last_change":"2024-01-16T15:04:44.952580+0000","last_active":"2024-01-16T15:04:45.933768+0000","last_peered":"2024-01-16T15:04:45.933768+0000","last_clean":"2024-01-16T15:04:45.933768+0000","last_became_active":"2024-01-16T15:04:44.951072+0000","last_became_peered":"2024-01-16T15:04:44.951072+0000","last_unstale":"2024-01-16T15:04:45.933768+0000","last_undegraded":"2024-01-16T15:04:45.933768+0000","last_fullsized":"2024-01-16T15:04:45.933768+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:21:05.712361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940609+0000","last_change":"2024-01-16T15:04:45.944418+0000","last_active":"2024-01-16T15:04:46.940609+0000","last_peered":"2024-01-16T15:04:46.940609+0000","last_clean":"2024-01-16T15:04:46.940609+0000","last_became_active":"2024-01-16T15:04:45.944139+0000","last_became_peered":"2024-01-16T15:04:45.944139+0000","last_unstale":"2024-01-16T15:04:46.940609+0000","last_undegraded":"2024-01-16T15:04:46.940609+0000","last_fullsized":"2024-01-16T15:04:46.940609+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:02.699162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.920884+0000","last_change":"2024-01-16T15:04:44.954744+0000","last_active":"2024-01-16T15:04:45.920884+0000","last_peered":"2024-01-16T15:04:45.920884+0000","last_clean":"2024-01-16T15:04:45.920884+0000","last_became_active":"2024-01-16T15:04:44.954114+0000","last_became_peered":"2024-01-16T15:04:44.954114+0000","last_unstale":"2024-01-16T15:04:45.920884+0000","last_undegraded":"2024-01-16T15:04:45.920884+0000","last_fullsized":"2024-01-16T15:04:45.920884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:00.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076944+0000","last_change":"2024-01-16T15:04:45.946501+0000","last_active":"2024-01-16T15:04:47.076944+0000","last_peered":"2024-01-16T15:04:47.076944+0000","last_clean":"2024-01-16T15:04:47.076944+0000","last_became_active":"2024-01-16T15:04:45.946148+0000","last_became_peered":"2024-01-16T15:04:45.946148+0000","last_unstale":"2024-01-16T15:04:47.076944+0000","last_undegraded":"2024-01-16T15:04:47.076944+0000","last_fullsized":"2024-01-16T15:04:47.076944+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:13:57.810944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.944774+0000","last_change":"2024-01-16T15:04:44.956126+0000","last_active":"2024-01-16T15:04:45.944774+0000","last_peered":"2024-01-16T15:04:45.944774+0000","last_clean":"2024-01-16T15:04:45.944774+0000","last_became_active":"2024-01-16T15:04:44.955841+0000","last_became_peered":"2024-01-16T15:04:44.955841+0000","last_unstale":"2024-01-16T15:04:45.944774+0000","last_undegraded":"2024-01-16T15:04:45.944774+0000","last_fullsized":"2024-01-16T15:04:45.944774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:50:59.091270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.078423+0000","last_change":"2024-01-16T15:04:45.959742+0000","last_active":"2024-01-16T15:04:47.078423+0000","last_peered":"2024-01-16T15:04:47.078423+0000","last_clean":"2024-01-16T15:04:47.078423+0000","last_became_active":"2024-01-16T15:04:45.959422+0000","last_became_peered":"2024-01-16T15:04:45.959422+0000","last_unstale":"2024-01-16T15:04:47.078423+0000","last_undegraded":"2024-01-16T15:04:47.078423+0000","last_fullsized":"2024-01-16T15:04:47.078423+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:51.909148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.931574+0000","last_change":"2024-01-16T15:04:44.950230+0000","last_active":"2024-01-16T15:04:45.931574+0000","last_peered":"2024-01-16T15:04:45.931574+0000","last_clean":"2024-01-16T15:04:45.931574+0000","last_became_active":"2024-01-16T15:04:44.949784+0000","last_became_peered":"2024-01-16T15:04:44.949784+0000","last_unstale":"2024-01-16T15:04:45.931574+0000","last_undegraded":"2024-01-16T15:04:45.931574+0000","last_fullsized":"2024-01-16T15:04:45.931574+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:06:20.111605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077651+0000","last_change":"2024-01-16T15:04:45.950408+0000","last_active":"2024-01-16T15:04:47.077651+0000","last_peered":"2024-01-16T15:04:47.077651+0000","last_clean":"2024-01-16T15:04:47.077651+0000","last_became_active":"2024-01-16T15:04:45.948912+0000","last_became_peered":"2024-01-16T15:04:45.948912+0000","last_unstale":"2024-01-16T15:04:47.077651+0000","last_undegraded":"2024-01-16T15:04:47.077651+0000","last_fullsized":"2024-01-16T15:04:47.077651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:54.228850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"60'3","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.970822+0000","last_change":"2024-01-16T15:04:44.950674+0000","last_active":"2024-01-16T15:04:45.970822+0000","last_peered":"2024-01-16T15:04:45.970822+0000","last_clean":"2024-01-16T15:04:45.970822+0000","last_became_active":"2024-01-16T15:04:44.950287+0000","last_became_peered":"2024-01-16T15:04:44.950287+0000","last_unstale":"2024-01-16T15:04:45.970822+0000","last_undegraded":"2024-01-16T15:04:45.970822+0000","last_fullsized":"2024-01-16T15:04:45.970822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:25:32.886153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077504+0000","last_change":"2024-01-16T15:04:45.955944+0000","last_active":"2024-01-16T15:04:47.077504+0000","last_peered":"2024-01-16T15:04:47.077504+0000","last_clean":"2024-01-16T15:04:47.077504+0000","last_became_active":"2024-01-16T15:04:45.955674+0000","last_became_peered":"2024-01-16T15:04:45.955674+0000","last_unstale":"2024-01-16T15:04:47.077504+0000","last_undegraded":"2024-01-16T15:04:47.077504+0000","last_fullsized":"2024-01-16T15:04:47.077504+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:16.429156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"60'6","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.972271+0000","last_change":"2024-01-16T15:04:44.944375+0000","last_active":"2024-01-16T15:04:45.972271+0000","last_peered":"2024-01-16T15:04:45.972271+0000","last_clean":"2024-01-16T15:04:45.972271+0000","last_became_active":"2024-01-16T15:04:44.943448+0000","last_became_peered":"2024-01-16T15:04:44.943448+0000","last_unstale":"2024-01-16T15:04:45.972271+0000","last_undegraded":"2024-01-16T15:04:45.972271+0000","last_fullsized":"2024-01-16T15:04:45.972271+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:39:55.171997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.944604+0000","last_change":"2024-01-16T15:04:45.947151+0000","last_active":"2024-01-16T15:04:46.944604+0000","last_peered":"2024-01-16T15:04:46.944604+0000","last_clean":"2024-01-16T15:04:46.944604+0000","last_became_active":"2024-01-16T15:04:45.946377+0000","last_became_peered":"2024-01-16T15:04:45.946377+0000","last_unstale":"2024-01-16T15:04:46.944604+0000","last_undegraded":"2024-01-16T15:04:46.944604+0000","last_fullsized":"2024-01-16T15:04:46.944604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:41.809624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.965901+0000","last_change":"2024-01-16T15:04:44.953633+0000","last_active":"2024-01-16T15:04:45.965901+0000","last_peered":"2024-01-16T15:04:45.965901+0000","last_clean":"2024-01-16T15:04:45.965901+0000","last_became_active":"2024-01-16T15:04:44.953323+0000","last_became_peered":"2024-01-16T15:04:44.953323+0000","last_unstale":"2024-01-16T15:04:45.965901+0000","last_undegraded":"2024-01-16T15:04:45.965901+0000","last_fullsized":"2024-01-16T15:04:45.965901+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:51.687038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076380+0000","last_change":"2024-01-16T15:04:45.953701+0000","last_active":"2024-01-16T15:04:47.076380+0000","last_peered":"2024-01-16T15:04:47.076380+0000","last_clean":"2024-01-16T15:04:47.076380+0000","last_became_active":"2024-01-16T15:04:45.953353+0000","last_became_peered":"2024-01-16T15:04:45.953353+0000","last_unstale":"2024-01-16T15:04:47.076380+0000","last_undegraded":"2024-01-16T15:04:47.076380+0000","last_fullsized":"2024-01-16T15:04:47.076380+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:02.156245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.918168+0000","last_change":"2024-01-16T15:04:44.946889+0000","last_active":"2024-01-16T15:04:45.918168+0000","last_peered":"2024-01-16T15:04:45.918168+0000","last_clean":"2024-01-16T15:04:45.918168+0000","last_became_active":"2024-01-16T15:04:44.946529+0000","last_became_peered":"2024-01-16T15:04:44.946529+0000","last_unstale":"2024-01-16T15:04:45.918168+0000","last_undegraded":"2024-01-16T15:04:45.918168+0000","last_fullsized":"2024-01-16T15:04:45.918168+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:00.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075030+0000","last_change":"2024-01-16T15:04:45.938994+0000","last_active":"2024-01-16T15:04:47.075030+0000","last_peered":"2024-01-16T15:04:47.075030+0000","last_clean":"2024-01-16T15:04:47.075030+0000","last_became_active":"2024-01-16T15:04:45.938712+0000","last_became_peered":"2024-01-16T15:04:45.938712+0000","last_unstale":"2024-01-16T15:04:47.075030+0000","last_undegraded":"2024-01-16T15:04:47.075030+0000","last_fullsized":"2024-01-16T15:04:47.075030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:55:25.275039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.933975+0000","last_change":"2024-01-16T15:04:44.950600+0000","last_active":"2024-01-16T15:04:45.933975+0000","last_peered":"2024-01-16T15:04:45.933975+0000","last_clean":"2024-01-16T15:04:45.933975+0000","last_became_active":"2024-01-16T15:04:44.949790+0000","last_became_peered":"2024-01-16T15:04:44.949790+0000","last_unstale":"2024-01-16T15:04:45.933975+0000","last_undegraded":"2024-01-16T15:04:45.933975+0000","last_fullsized":"2024-01-16T15:04:45.933975+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:15.209510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076099+0000","last_change":"2024-01-16T15:04:45.954238+0000","last_active":"2024-01-16T15:04:47.076099+0000","last_peered":"2024-01-16T15:04:47.076099+0000","last_clean":"2024-01-16T15:04:47.076099+0000","last_became_active":"2024-01-16T15:04:45.953974+0000","last_became_peered":"2024-01-16T15:04:45.953974+0000","last_unstale":"2024-01-16T15:04:47.076099+0000","last_undegraded":"2024-01-16T15:04:47.076099+0000","last_fullsized":"2024-01-16T15:04:47.076099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:36.339721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"60'8","reported_seq":20,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.970073+0000","last_change":"2024-01-16T15:04:44.955658+0000","last_active":"2024-01-16T15:04:45.970073+0000","last_peered":"2024-01-16T15:04:45.970073+0000","last_clean":"2024-01-16T15:04:45.970073+0000","last_became_active":"2024-01-16T15:04:44.955007+0000","last_became_peered":"2024-01-16T15:04:44.955007+0000","last_unstale":"2024-01-16T15:04:45.970073+0000","last_undegraded":"2024-01-16T15:04:45.970073+0000","last_fullsized":"2024-01-16T15:04:45.970073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:25.460735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.078018+0000","last_change":"2024-01-16T15:04:45.955017+0000","last_active":"2024-01-16T15:04:47.078018+0000","last_peered":"2024-01-16T15:04:47.078018+0000","last_clean":"2024-01-16T15:04:47.078018+0000","last_became_active":"2024-01-16T15:04:45.954190+0000","last_became_peered":"2024-01-16T15:04:45.954190+0000","last_unstale":"2024-01-16T15:04:47.078018+0000","last_undegraded":"2024-01-16T15:04:47.078018+0000","last_fullsized":"2024-01-16T15:04:47.078018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:52.701665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"60'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.967171+0000","last_change":"2024-01-16T15:04:44.949726+0000","last_active":"2024-01-16T15:04:45.967171+0000","last_peered":"2024-01-16T15:04:45.967171+0000","last_clean":"2024-01-16T15:04:45.967171+0000","last_became_active":"2024-01-16T15:04:44.948106+0000","last_became_peered":"2024-01-16T15:04:44.948106+0000","last_unstale":"2024-01-16T15:04:45.967171+0000","last_undegraded":"2024-01-16T15:04:45.967171+0000","last_fullsized":"2024-01-16T15:04:45.967171+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:04.666517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.946386+0000","last_change":"2024-01-16T15:04:45.948049+0000","last_active":"2024-01-16T15:04:46.946386+0000","last_peered":"2024-01-16T15:04:46.946386+0000","last_clean":"2024-01-16T15:04:46.946386+0000","last_became_active":"2024-01-16T15:04:45.946461+0000","last_became_peered":"2024-01-16T15:04:45.946461+0000","last_unstale":"2024-01-16T15:04:46.946386+0000","last_undegraded":"2024-01-16T15:04:46.946386+0000","last_fullsized":"2024-01-16T15:04:46.946386+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:28:01.731921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.969244+0000","last_change":"2024-01-16T15:04:44.944854+0000","last_active":"2024-01-16T15:04:45.969244+0000","last_peered":"2024-01-16T15:04:45.969244+0000","last_clean":"2024-01-16T15:04:45.969244+0000","last_became_active":"2024-01-16T15:04:44.942010+0000","last_became_peered":"2024-01-16T15:04:44.942010+0000","last_unstale":"2024-01-16T15:04:45.969244+0000","last_undegraded":"2024-01-16T15:04:45.969244+0000","last_fullsized":"2024-01-16T15:04:45.969244+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:26.839552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941515+0000","last_change":"2024-01-16T15:04:45.947528+0000","last_active":"2024-01-16T15:04:46.941515+0000","last_peered":"2024-01-16T15:04:46.941515+0000","last_clean":"2024-01-16T15:04:46.941515+0000","last_became_active":"2024-01-16T15:04:45.947124+0000","last_became_peered":"2024-01-16T15:04:45.947124+0000","last_unstale":"2024-01-16T15:04:46.941515+0000","last_undegraded":"2024-01-16T15:04:46.941515+0000","last_fullsized":"2024-01-16T15:04:46.941515+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:03.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"60'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.964487+0000","last_change":"2024-01-16T15:04:44.953363+0000","last_active":"2024-01-16T15:04:45.964487+0000","last_peered":"2024-01-16T15:04:45.964487+0000","last_clean":"2024-01-16T15:04:45.964487+0000","last_became_active":"2024-01-16T15:04:44.952992+0000","last_became_peered":"2024-01-16T15:04:44.952992+0000","last_unstale":"2024-01-16T15:04:45.964487+0000","last_undegraded":"2024-01-16T15:04:45.964487+0000","last_fullsized":"2024-01-16T15:04:45.964487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:06.325558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.077603+0000","last_change":"2024-01-16T15:04:45.957536+0000","last_active":"2024-01-16T15:04:47.077603+0000","last_peered":"2024-01-16T15:04:47.077603+0000","last_clean":"2024-01-16T15:04:47.077603+0000","last_became_active":"2024-01-16T15:04:45.957231+0000","last_became_peered":"2024-01-16T15:04:45.957231+0000","last_unstale":"2024-01-16T15:04:47.077603+0000","last_undegraded":"2024-01-16T15:04:47.077603+0000","last_fullsized":"2024-01-16T15:04:47.077603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:05.681145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.918471+0000","last_change":"2024-01-16T15:04:44.949082+0000","last_active":"2024-01-16T15:04:45.918471+0000","last_peered":"2024-01-16T15:04:45.918471+0000","last_clean":"2024-01-16T15:04:45.918471+0000","last_became_active":"2024-01-16T15:04:44.948716+0000","last_became_peered":"2024-01-16T15:04:44.948716+0000","last_unstale":"2024-01-16T15:04:45.918471+0000","last_undegraded":"2024-01-16T15:04:45.918471+0000","last_fullsized":"2024-01-16T15:04:45.918471+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:36.401887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.945091+0000","last_change":"2024-01-16T15:04:45.947405+0000","last_active":"2024-01-16T15:04:46.945091+0000","last_peered":"2024-01-16T15:04:46.945091+0000","last_clean":"2024-01-16T15:04:46.945091+0000","last_became_active":"2024-01-16T15:04:45.946866+0000","last_became_peered":"2024-01-16T15:04:45.946866+0000","last_unstale":"2024-01-16T15:04:46.945091+0000","last_undegraded":"2024-01-16T15:04:46.945091+0000","last_fullsized":"2024-01-16T15:04:46.945091+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:21:38.389378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.967896+0000","last_change":"2024-01-16T15:04:44.951578+0000","last_active":"2024-01-16T15:04:45.967896+0000","last_peered":"2024-01-16T15:04:45.967896+0000","last_clean":"2024-01-16T15:04:45.967896+0000","last_became_active":"2024-01-16T15:04:44.951281+0000","last_became_peered":"2024-01-16T15:04:44.951281+0000","last_unstale":"2024-01-16T15:04:45.967896+0000","last_undegraded":"2024-01-16T15:04:45.967896+0000","last_fullsized":"2024-01-16T15:04:45.967896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:50.894520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.074908+0000","last_change":"2024-01-16T15:04:45.943663+0000","last_active":"2024-01-16T15:04:47.074908+0000","last_peered":"2024-01-16T15:04:47.074908+0000","last_clean":"2024-01-16T15:04:47.074908+0000","last_became_active":"2024-01-16T15:04:45.943370+0000","last_became_peered":"2024-01-16T15:04:45.943370+0000","last_unstale":"2024-01-16T15:04:47.074908+0000","last_undegraded":"2024-01-16T15:04:47.074908+0000","last_fullsized":"2024-01-16T15:04:47.074908+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:20.301429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.930702+0000","last_change":"2024-01-16T15:04:44.938607+0000","last_active":"2024-01-16T15:04:45.930702+0000","last_peered":"2024-01-16T15:04:45.930702+0000","last_clean":"2024-01-16T15:04:45.930702+0000","last_became_active":"2024-01-16T15:04:44.938328+0000","last_became_peered":"2024-01-16T15:04:44.938328+0000","last_unstale":"2024-01-16T15:04:45.930702+0000","last_undegraded":"2024-01-16T15:04:45.930702+0000","last_fullsized":"2024-01-16T15:04:45.930702+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:13:42.708682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.073563+0000","last_change":"2024-01-16T15:04:45.956899+0000","last_active":"2024-01-16T15:04:47.073563+0000","last_peered":"2024-01-16T15:04:47.073563+0000","last_clean":"2024-01-16T15:04:47.073563+0000","last_became_active":"2024-01-16T15:04:45.949486+0000","last_became_peered":"2024-01-16T15:04:45.949486+0000","last_unstale":"2024-01-16T15:04:47.073563+0000","last_undegraded":"2024-01-16T15:04:47.073563+0000","last_fullsized":"2024-01-16T15:04:47.073563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:44.062031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.934713+0000","last_change":"2024-01-16T15:04:44.955973+0000","last_active":"2024-01-16T15:04:45.934713+0000","last_peered":"2024-01-16T15:04:45.934713+0000","last_clean":"2024-01-16T15:04:45.934713+0000","last_became_active":"2024-01-16T15:04:44.955709+0000","last_became_peered":"2024-01-16T15:04:44.955709+0000","last_unstale":"2024-01-16T15:04:45.934713+0000","last_undegraded":"2024-01-16T15:04:45.934713+0000","last_fullsized":"2024-01-16T15:04:45.934713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:12.579518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.946165+0000","last_change":"2024-01-16T15:04:45.946062+0000","last_active":"2024-01-16T15:04:46.946165+0000","last_peered":"2024-01-16T15:04:46.946165+0000","last_clean":"2024-01-16T15:04:46.946165+0000","last_became_active":"2024-01-16T15:04:45.945766+0000","last_became_peered":"2024-01-16T15:04:45.945766+0000","last_unstale":"2024-01-16T15:04:46.946165+0000","last_undegraded":"2024-01-16T15:04:46.946165+0000","last_fullsized":"2024-01-16T15:04:46.946165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:27.854180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"60'2","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.969722+0000","last_change":"2024-01-16T15:04:44.954224+0000","last_active":"2024-01-16T15:04:45.969722+0000","last_peered":"2024-01-16T15:04:45.969722+0000","last_clean":"2024-01-16T15:04:45.969722+0000","last_became_active":"2024-01-16T15:04:44.953706+0000","last_became_peered":"2024-01-16T15:04:44.953706+0000","last_unstale":"2024-01-16T15:04:45.969722+0000","last_undegraded":"2024-01-16T15:04:45.969722+0000","last_fullsized":"2024-01-16T15:04:45.969722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:35.538247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.946368+0000","last_change":"2024-01-16T15:04:45.956978+0000","last_active":"2024-01-16T15:04:46.946368+0000","last_peered":"2024-01-16T15:04:46.946368+0000","last_clean":"2024-01-16T15:04:46.946368+0000","last_became_active":"2024-01-16T15:04:45.956682+0000","last_became_peered":"2024-01-16T15:04:45.956682+0000","last_unstale":"2024-01-16T15:04:46.946368+0000","last_undegraded":"2024-01-16T15:04:46.946368+0000","last_fullsized":"2024-01-16T15:04:46.946368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:21.846436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.919607+0000","last_change":"2024-01-16T15:04:44.956833+0000","last_active":"2024-01-16T15:04:45.919607+0000","last_peered":"2024-01-16T15:04:45.919607+0000","last_clean":"2024-01-16T15:04:45.919607+0000","last_became_active":"2024-01-16T15:04:44.956562+0000","last_became_peered":"2024-01-16T15:04:44.956562+0000","last_unstale":"2024-01-16T15:04:45.919607+0000","last_undegraded":"2024-01-16T15:04:45.919607+0000","last_fullsized":"2024-01-16T15:04:45.919607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:20.831808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.078149+0000","last_change":"2024-01-16T15:04:45.949151+0000","last_active":"2024-01-16T15:04:47.078149+0000","last_peered":"2024-01-16T15:04:47.078149+0000","last_clean":"2024-01-16T15:04:47.078149+0000","last_became_active":"2024-01-16T15:04:45.947540+0000","last_became_peered":"2024-01-16T15:04:45.947540+0000","last_unstale":"2024-01-16T15:04:47.078149+0000","last_undegraded":"2024-01-16T15:04:47.078149+0000","last_fullsized":"2024-01-16T15:04:47.078149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:02.350195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.918134+0000","last_change":"2024-01-16T15:04:44.942876+0000","last_active":"2024-01-16T15:04:45.918134+0000","last_peered":"2024-01-16T15:04:45.918134+0000","last_clean":"2024-01-16T15:04:45.918134+0000","last_became_active":"2024-01-16T15:04:44.942274+0000","last_became_peered":"2024-01-16T15:04:44.942274+0000","last_unstale":"2024-01-16T15:04:45.918134+0000","last_undegraded":"2024-01-16T15:04:45.918134+0000","last_fullsized":"2024-01-16T15:04:45.918134+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:39:55.171997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075131+0000","last_change":"2024-01-16T15:04:45.959140+0000","last_active":"2024-01-16T15:04:47.075131+0000","last_peered":"2024-01-16T15:04:47.075131+0000","last_clean":"2024-01-16T15:04:47.075131+0000","last_became_active":"2024-01-16T15:04:45.951189+0000","last_became_peered":"2024-01-16T15:04:45.951189+0000","last_unstale":"2024-01-16T15:04:47.075131+0000","last_undegraded":"2024-01-16T15:04:47.075131+0000","last_fullsized":"2024-01-16T15:04:47.075131+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:02.240767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.934683+0000","last_change":"2024-01-16T15:04:44.949594+0000","last_active":"2024-01-16T15:04:45.934683+0000","last_peered":"2024-01-16T15:04:45.934683+0000","last_clean":"2024-01-16T15:04:45.934683+0000","last_became_active":"2024-01-16T15:04:44.949254+0000","last_became_peered":"2024-01-16T15:04:44.949254+0000","last_unstale":"2024-01-16T15:04:45.934683+0000","last_undegraded":"2024-01-16T15:04:45.934683+0000","last_fullsized":"2024-01-16T15:04:45.934683+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:19.702211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941295+0000","last_change":"2024-01-16T15:04:45.952305+0000","last_active":"2024-01-16T15:04:46.941295+0000","last_peered":"2024-01-16T15:04:46.941295+0000","last_clean":"2024-01-16T15:04:46.941295+0000","last_became_active":"2024-01-16T15:04:45.949414+0000","last_became_peered":"2024-01-16T15:04:45.949414+0000","last_unstale":"2024-01-16T15:04:46.941295+0000","last_undegraded":"2024-01-16T15:04:46.941295+0000","last_fullsized":"2024-01-16T15:04:46.941295+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:36.943632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.928334+0000","last_change":"2024-01-16T15:04:44.956069+0000","last_active":"2024-01-16T15:04:45.928334+0000","last_peered":"2024-01-16T15:04:45.928334+0000","last_clean":"2024-01-16T15:04:45.928334+0000","last_became_active":"2024-01-16T15:04:44.955054+0000","last_became_peered":"2024-01-16T15:04:44.955054+0000","last_unstale":"2024-01-16T15:04:45.928334+0000","last_undegraded":"2024-01-16T15:04:45.928334+0000","last_fullsized":"2024-01-16T15:04:45.928334+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:29.054200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.076681+0000","last_change":"2024-01-16T15:04:45.954263+0000","last_active":"2024-01-16T15:04:47.076681+0000","last_peered":"2024-01-16T15:04:47.076681+0000","last_clean":"2024-01-16T15:04:47.076681+0000","last_became_active":"2024-01-16T15:04:45.953966+0000","last_became_peered":"2024-01-16T15:04:45.953966+0000","last_unstale":"2024-01-16T15:04:47.076681+0000","last_undegraded":"2024-01-16T15:04:47.076681+0000","last_fullsized":"2024-01-16T15:04:47.076681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:39:14.767237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.923200+0000","last_change":"2024-01-16T15:04:44.953091+0000","last_active":"2024-01-16T15:04:45.923200+0000","last_peered":"2024-01-16T15:04:45.923200+0000","last_clean":"2024-01-16T15:04:45.923200+0000","last_became_active":"2024-01-16T15:04:44.951937+0000","last_became_peered":"2024-01-16T15:04:44.951937+0000","last_unstale":"2024-01-16T15:04:45.923200+0000","last_undegraded":"2024-01-16T15:04:45.923200+0000","last_fullsized":"2024-01-16T15:04:45.923200+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:32.257346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:47.075264+0000","last_change":"2024-01-16T15:04:45.953487+0000","last_active":"2024-01-16T15:04:47.075264+0000","last_peered":"2024-01-16T15:04:47.075264+0000","last_clean":"2024-01-16T15:04:47.075264+0000","last_became_active":"2024-01-16T15:04:45.953189+0000","last_became_peered":"2024-01-16T15:04:45.953189+0000","last_unstale":"2024-01-16T15:04:47.075264+0000","last_undegraded":"2024-01-16T15:04:47.075264+0000","last_fullsized":"2024-01-16T15:04:47.075264+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:40.509325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.932940+0000","last_change":"2024-01-16T15:04:44.955674+0000","last_active":"2024-01-16T15:04:45.932940+0000","last_peered":"2024-01-16T15:04:45.932940+0000","last_clean":"2024-01-16T15:04:45.932940+0000","last_became_active":"2024-01-16T15:04:44.955359+0000","last_became_peered":"2024-01-16T15:04:44.955359+0000","last_unstale":"2024-01-16T15:04:45.932940+0000","last_undegraded":"2024-01-16T15:04:45.932940+0000","last_fullsized":"2024-01-16T15:04:45.932940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:25:48.380507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.940806+0000","last_change":"2024-01-16T15:04:45.943585+0000","last_active":"2024-01-16T15:04:46.940806+0000","last_peered":"2024-01-16T15:04:46.940806+0000","last_clean":"2024-01-16T15:04:46.940806+0000","last_became_active":"2024-01-16T15:04:45.943177+0000","last_became_peered":"2024-01-16T15:04:45.943177+0000","last_unstale":"2024-01-16T15:04:46.940806+0000","last_undegraded":"2024-01-16T15:04:46.940806+0000","last_fullsized":"2024-01-16T15:04:46.940806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:14.669876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.929138+0000","last_change":"2024-01-16T15:04:44.956478+0000","last_active":"2024-01-16T15:04:45.929138+0000","last_peered":"2024-01-16T15:04:45.929138+0000","last_clean":"2024-01-16T15:04:45.929138+0000","last_became_active":"2024-01-16T15:04:44.955596+0000","last_became_peered":"2024-01-16T15:04:44.955596+0000","last_unstale":"2024-01-16T15:04:45.929138+0000","last_undegraded":"2024-01-16T15:04:45.929138+0000","last_fullsized":"2024-01-16T15:04:45.929138+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:01.141617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-16T15:04:46.941026+0000","last_change":"2024-01-16T15:04:45.957938+0000","last_active":"2024-01-16T15:04:46.941026+0000","last_peered":"2024-01-16T15:04:46.941026+0000","last_clean":"2024-01-16T15:04:46.941026+0000","last_became_active":"2024-01-16T15:04:45.957640+0000","last_became_peered":"2024-01-16T15:04:45.957640+0000","last_unstale":"2024-01-16T15:04:46.941026+0000","last_undegraded":"2024-01-16T15:04:46.941026+0000","last_fullsized":"2024-01-16T15:04:46.941026+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:44.902434+0000","last_clean_scrub_stamp":"2024-01-16T15:04:44.902434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:03:18.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.917016+0000","last_change":"2024-01-16T15:04:44.955479+0000","last_active":"2024-01-16T15:04:45.917016+0000","last_peered":"2024-01-16T15:04:45.917016+0000","last_clean":"2024-01-16T15:04:45.917016+0000","last_became_active":"2024-01-16T15:04:44.955081+0000","last_became_peered":"2024-01-16T15:04:44.955081+0000","last_unstale":"2024-01-16T15:04:45.917016+0000","last_undegraded":"2024-01-16T15:04:45.917016+0000","last_fullsized":"2024-01-16T15:04:45.917016+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:04:43.887806+0000","last_clean_scrub_stamp":"2024-01-16T15:04:43.887806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:53.214222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":198,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:04:45.931102+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:04:45.931102+0000","last_peered":"2024-01-16T15:04:45.931102+0000","last_clean":"2024-01-16T15:04:45.931102+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:04:45.931102+0000","last_undegraded":"2024-01-16T15:04:45.931102+0000","last_fullsized":"2024-01-16T15:04:45.931102+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36640,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335200,"statfs":{"total":96636764160,"available":96599244800,"internally_reserved":0,"allocated":1331200,"data_stored":895611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.343,"15min":1.1930000000000001},"min":{"1min":0.63700000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3740000000000001,"15min":1.26},"min":{"1min":0.65700000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":7.8460000000000001,"5min":7.8460000000000001,"15min":7.8460000000000001},"last":1.5680000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.252,"15min":1.222},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":4.1379999999999999,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":1.016},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.2470000000000001,"15min":1.1910000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":5.694,"5min":5.694,"15min":5.694},"last":0.84299999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.454,"15min":1.3320000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":3.7149999999999999,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.351},{"interface":"front","average":{"1min":1.361,"5min":1.4630000000000001,"15min":1.347},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":3.6899999999999999,"5min":10.661,"15min":10.661},"last":1.3899999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2210000000000001,"5min":1.52,"15min":1.357},"min":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":8.0909999999999993,"5min":8.0909999999999993,"15min":8.0909999999999993},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.794,"5min":1.6459999999999999,"15min":1.4910000000000001},"min":{"1min":0.72699999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":8.3699999999999992,"5min":13.193,"15min":13.193},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.452,"15min":1.4099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.0430000000000001,"5min":16.376000000000001,"15min":16.376000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4330000000000001,"15min":1.4810000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.8039999999999998,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.3009999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.464,"15min":1.54},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.474,"15min":1.544},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":3.3050000000000002,"5min":6.319,"15min":6.319},"last":1.5109999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.635,"15min":1.6839999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.234,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.6339999999999999,"15min":1.6799999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.8650000000000002,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.5960000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36052,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":729088,"data_stored":305083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":4.9279999999999999,"5min":1.909,"15min":1.405},"min":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":28.327999999999999,"5min":28.327999999999999,"15min":28.327999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":5.0990000000000002,"5min":2.0059999999999998,"15min":1.4299999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":30.280999999999999,"5min":30.280999999999999,"15min":30.280999999999999},"last":1.506}]},{"osd":1,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":3.298,"5min":1.54,"15min":1.228},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":26.344000000000001,"5min":26.344000000000001,"15min":26.344000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.272,"5min":1.1679999999999999,"15min":1.1319999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":0.93000000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":3.4729999999999999,"5min":1.6759999999999999,"15min":1.3740000000000001},"min":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":17.991,"5min":17.991,"15min":17.991},"last":1.0880000000000001},{"interface":"front","average":{"1min":3.6499999999999999,"5min":1.8220000000000001,"15min":1.554},"min":{"1min":0.67100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":18.02,"5min":18.02,"15min":18.02},"last":1.4330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":1.579,"15min":1.347},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":13.092000000000001,"5min":13.092000000000001,"15min":13.092000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":2.6080000000000001,"5min":1.5940000000000001,"15min":1.3779999999999999},"min":{"1min":0.69399999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":13.058999999999999,"5min":13.058999999999999,"15min":13.058999999999999},"last":1.3460000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.6060000000000001,"15min":1.413},"min":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":10.018000000000001,"5min":15.481999999999999,"15min":15.481999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.9610000000000001,"5min":1.512,"15min":1.45},"min":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":13.401999999999999,"5min":13.401999999999999,"15min":13.401999999999999},"last":1.1519999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3420000000000001,"15min":1.3129999999999999},"min":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.21,"5min":4.21,"15min":4.21},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4950000000000001,"15min":1.538},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.861,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.3979999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.5229999999999999,"15min":1.4690000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"last":1.581},{"interface":"front","average":{"1min":2.044,"5min":1.603,"15min":1.605},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":10.087,"5min":10.087,"15min":10.087},"last":1.605}]}]},{"osd":5,"up_from":14,"seq":60129542213,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36060,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335780,"statfs":{"total":96636764160,"available":96599838720,"internally_reserved":0,"allocated":737280,"data_stored":305735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":3.1030000000000002,"5min":1.5129999999999999,"15min":1.2709999999999999},"min":{"1min":0.495,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":22.077000000000002,"5min":22.077000000000002,"15min":22.077000000000002},"last":1.171},{"interface":"front","average":{"1min":6.0510000000000002,"5min":2.1549999999999998,"15min":1.397},"min":{"1min":0.61599999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":56.011000000000003,"5min":56.011000000000003,"15min":56.011000000000003},"last":0.86499999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.27,"5min":1.4059999999999999,"15min":1.2010000000000001},"min":{"1min":0.745,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":16.687999999999999,"5min":16.687999999999999,"15min":16.687999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":5.3109999999999999,"5min":2.2360000000000002,"15min":1.554},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":67.930000000000007,"5min":67.930000000000007,"15min":67.930000000000007},"last":0.753}]},{"osd":2,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.4490000000000001,"15min":1.2989999999999999},"min":{"1min":0.80900000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":1.224},{"interface":"front","average":{"1min":2.343,"5min":1.484,"15min":1.3240000000000001},"min":{"1min":0.92300000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":11.09,"5min":11.09,"15min":11.09},"last":1.03}]},{"osd":3,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.4590000000000001,"5min":1.8320000000000001,"15min":1.458},"min":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":13.711,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":2.4340000000000002,"5min":1.7230000000000001,"15min":1.506},"min":{"1min":0.80800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":11.923,"5min":11.923,"15min":11.923},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4530000000000001,"15min":1.288},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.2909999999999999,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.919,"5min":1.6259999999999999,"15min":1.4319999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":13.686,"5min":16.268000000000001,"15min":16.268000000000001},"last":1.2829999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7949999999999999,"5min":1.4730000000000001,"15min":1.413},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":11.706,"5min":11.706,"15min":11.706},"last":1.696},{"interface":"front","average":{"1min":1.593,"5min":1.585,"15min":1.508},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.206,"5min":7.335,"15min":7.335},"last":1.4259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.786,"15min":1.613},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":14.345000000000001,"5min":14.345000000000001,"15min":14.345000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":2.0590000000000002,"5min":1.8009999999999999,"15min":1.6499999999999999},"min":{"1min":0.88200000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":11.954000000000001,"5min":12.673999999999999,"15min":12.673999999999999},"last":1.633}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36064,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335776,"statfs":{"total":96636764160,"available":96599834624,"internally_reserved":0,"allocated":741376,"data_stored":306695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.2609999999999999,"15min":1.1479999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":12.587999999999999,"5min":12.587999999999999,"15min":12.587999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":4.5940000000000003,"5min":1.819,"15min":1.2829999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":60.106000000000002,"5min":60.106000000000002,"15min":60.106000000000002},"last":1.2130000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4449999999999998,"5min":1.643,"15min":1.403},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":9.6340000000000003,"5min":9.6340000000000003,"15min":9.6340000000000003},"last":1.5640000000000001},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.6040000000000001,"15min":1.3420000000000001},"min":{"1min":0.83799999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":9.5589999999999993,"5min":9.5589999999999993,"15min":9.5589999999999993},"last":1.5289999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.2769999999999999,"15min":1.1919999999999999},"min":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":4.0860000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":1.1100000000000001},{"interface":"front","average":{"1min":3.7509999999999999,"5min":1.859,"15min":1.5109999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":35.024999999999999,"5min":35.024999999999999,"15min":35.024999999999999},"last":1.6619999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.494,"15min":1.4350000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.6980000000000004,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.379,"15min":1.3380000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":7.8140000000000001,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":1.2569999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.415,"15min":1.4750000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.1190000000000002,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":1.431},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4970000000000001,"15min":1.5620000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.4750000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.4670000000000001,"15min":1.472},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":9.6630000000000003,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.571,"5min":1.5269999999999999,"15min":1.581},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":4.9649999999999999},"last":0.89900000000000002}]},{"osd":7,"last update":"Tue Jan 16 15:04:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5780000000000001,"15min":1.673},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.2050000000000001,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.5800000000000001,"15min":1.613},"min":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.3540000000000001,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.3180000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44388,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327452,"statfs":{"total":96636764160,"available":96591310848,"internally_reserved":0,"allocated":737280,"data_stored":305139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":1.4810000000000001,"15min":1.107},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":13.241,"5min":17.709,"15min":17.709},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.216,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.2690000000000001,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.98699999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.2110000000000001,"15min":1.056},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":8.9689999999999994,"5min":8.9689999999999994,"15min":8.9689999999999994},"last":1.431},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.5049999999999999,"15min":1.2889999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":14.893000000000001,"5min":14.893000000000001,"15min":14.893000000000001},"last":1.4610000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.3320000000000001,"15min":1.2010000000000001},"min":{"1min":0.51900000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.9780000000000002,"5min":8.8439999999999994,"15min":8.8439999999999994},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.4970000000000001,"15min":1.3280000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.444,"5min":10.25,"15min":10.25},"last":1.3600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0939999999999999,"5min":1.5,"15min":1.391},"min":{"1min":0.69899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":7.0620000000000003,"5min":7.0620000000000003,"15min":7.0620000000000003},"last":1.385},{"interface":"front","average":{"1min":1.444,"5min":1.474,"15min":1.4379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.7200000000000002,"5min":3.7200000000000002,"15min":3.7200000000000002},"last":1.5600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7350000000000001,"15min":1.5529999999999999},"min":{"1min":0.72399999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.758,"5min":12.65,"15min":12.65},"last":1.306},{"interface":"front","average":{"1min":1.9379999999999999,"5min":1.5940000000000001,"15min":1.456},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":6.5789999999999997,"5min":6.5789999999999997,"15min":6.5789999999999997},"last":1.6799999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.5309999999999999,"15min":1.4950000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.367,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":1.738},{"interface":"front","average":{"1min":1.962,"5min":1.792,"15min":1.629},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.8029999999999999,"5min":10.371,"15min":10.371},"last":1.4179999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":1.6319999999999999,"15min":1.5720000000000001},"min":{"1min":0.64600000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":6.6289999999999996,"5min":6.6289999999999996,"15min":6.6289999999999996},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.6499999999999999,"15min":1.5780000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":3.6669999999999998,"5min":5.1699999999999999,"15min":5.1699999999999999},"last":1.919}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":737280,"data_stored":305139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":6.5640000000000001,"5min":2.2210000000000001,"15min":1.353},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":64.387,"5min":64.387,"15min":64.387},"last":1.357},{"interface":"front","average":{"1min":2.548,"5min":1.3859999999999999,"15min":1.1180000000000001},"min":{"1min":0.47399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":11.339,"5min":11.339,"15min":11.339},"last":1.53}]},{"osd":1,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":4.5309999999999997,"5min":1.845,"15min":1.2090000000000001},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":27.263000000000002,"5min":27.263000000000002,"15min":27.263000000000002},"last":1.155},{"interface":"front","average":{"1min":3.9129999999999998,"5min":1.877,"15min":1.4379999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":23.547999999999998,"5min":23.547999999999998,"15min":23.547999999999998},"last":1.216}]},{"osd":3,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":1.7350000000000001,"15min":1.3440000000000001},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":27.311,"5min":27.311,"15min":27.311},"last":1.4019999999999999},{"interface":"front","average":{"1min":3.4239999999999999,"5min":1.7909999999999999,"15min":1.351},"min":{"1min":0.47999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":26.555,"5min":26.555,"15min":26.555},"last":1.8500000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.0139999999999998,"5min":1.8029999999999999,"15min":1.4690000000000001},"min":{"1min":1.0149999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":26.503,"5min":26.503,"15min":26.503},"last":3.032},{"interface":"front","average":{"1min":2.383,"5min":1.585,"15min":1.4139999999999999},"min":{"1min":0.82699999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":11.188000000000001,"5min":11.188000000000001,"15min":11.188000000000001},"last":3.5630000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.8340000000000001,"5min":1.9099999999999999,"15min":1.552},"min":{"1min":0.80800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":27.149000000000001,"5min":27.149000000000001,"15min":27.149000000000001},"last":2.4159999999999999},{"interface":"front","average":{"1min":2.903,"5min":1.907,"15min":1.637},"min":{"1min":0.93000000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":26.349,"5min":26.349,"15min":26.349},"last":2.2850000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.4409999999999998,"5min":1.6399999999999999,"15min":1.488},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":11.266,"5min":11.266,"15min":11.266},"last":3.891},{"interface":"front","average":{"1min":3.831,"5min":2.1469999999999998,"15min":1.6970000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":27.076000000000001,"5min":27.076000000000001,"15min":27.076000000000001},"last":3.1850000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":1.823,"15min":1.6739999999999999},"min":{"1min":1.0649999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":11.303000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":3.7989999999999999},{"interface":"front","average":{"1min":4.0570000000000004,"5min":2.0459999999999998,"15min":1.6499999999999999},"min":{"1min":0.873,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":27.207999999999998,"5min":27.207999999999998,"15min":27.207999999999998},"last":4.1879999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542212,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40216,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331624,"statfs":{"total":96636764160,"available":96595582976,"internally_reserved":0,"allocated":733184,"data_stored":305173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.359,"15min":1.133},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1310000000000002,"5min":14.407,"15min":14.407},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.2230000000000001,"15min":1.145},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":8.5090000000000003,"5min":8.5090000000000003,"15min":8.5090000000000003},"last":0.78500000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1519999999999999,"15min":1.097},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.3050000000000002,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.371},{"interface":"front","average":{"1min":2.3420000000000001,"5min":1.8029999999999999,"15min":1.403},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":22.550999999999998,"5min":22.550999999999998,"15min":22.550999999999998},"last":0.70199999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.8180000000000001,"15min":1.47},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":5.9969999999999999,"5min":14.653,"15min":14.653},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.5309999999999999,"15min":1.375},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":9.4169999999999998,"5min":9.4169999999999998,"15min":9.4169999999999998},"last":0.90600000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.637,"15min":1.4850000000000001},"min":{"1min":0.84099999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":5.3579999999999997,"5min":12.675000000000001,"15min":12.675000000000001},"last":1.107},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.639,"15min":1.5169999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":4.7969999999999997,"5min":12.731999999999999,"15min":12.731999999999999},"last":0.68000000000000005}]},{"osd":5,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.591,"15min":1.5649999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":5.4660000000000002,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.4870000000000001,"15min":1.484},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.9589999999999996,"5min":6.9569999999999999,"15min":6.9569999999999999},"last":0.83199999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.982,"15min":1.726},"min":{"1min":1.014,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":5.0549999999999997,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.75,"5min":1.726,"15min":1.677},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":5.4290000000000003,"5min":6.6779999999999999,"15min":6.6779999999999999},"last":1.242}]},{"osd":7,"last update":"Tue Jan 16 15:04:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.6020000000000001,"15min":1.5860000000000001},"min":{"1min":0.99299999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.9169999999999998,"5min":6.75,"15min":6.75},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.819,"5min":2.0630000000000002,"15min":1.8},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":5.0010000000000003,"5min":14.884,"15min":14.884},"last":1.0409999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1339392,"data_stored":897645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.321,"15min":1.0349999999999999},"min":{"1min":0.60799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.53600000000000003},{"interface":"front","average":{"1min":3.871,"5min":1.6439999999999999,"15min":1.1479999999999999},"min":{"1min":0.81599999999999995,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.414000000000001,"5min":24.414000000000001,"15min":24.414000000000001},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.3380000000000001,"15min":1.1120000000000001},"min":{"1min":0.57699999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":5.0460000000000003,"5min":5.444,"15min":5.444},"last":0.40400000000000003},{"interface":"front","average":{"1min":4.3010000000000002,"5min":1.758,"15min":1.2350000000000001},"min":{"1min":0.81100000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":38.893000000000001,"5min":38.893000000000001,"15min":38.893000000000001},"last":0.50900000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.829,"15min":1.2210000000000001},"min":{"1min":0.63900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":18.436,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.99,"5min":1.4850000000000001,"15min":1.2609999999999999},"min":{"1min":0.86799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":4.407,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.444,"5min":1.573,"15min":1.2929999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":20.327000000000002,"5min":20.327000000000002,"15min":20.327000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":2.0659999999999998,"5min":1.6319999999999999,"15min":1.4279999999999999},"min":{"1min":0.92500000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":4.7599999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.85099999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":4.8040000000000003,"5min":2.0630000000000002,"15min":1.5349999999999999},"min":{"1min":1.171,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":24.611000000000001,"5min":24.611000000000001,"15min":24.611000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":2.9119999999999999,"5min":1.7450000000000001,"15min":1.4350000000000001},"min":{"1min":0.98899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":20.199000000000002,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.873}]},{"osd":6,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":1.734,"15min":1.5640000000000001},"min":{"1min":0.99099999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.754},{"interface":"front","average":{"1min":4.3659999999999997,"5min":2.1139999999999999,"15min":1.613},"min":{"1min":1.1970000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":24.478999999999999,"5min":24.478999999999999,"15min":24.478999999999999},"last":0.78800000000000003}]},{"osd":7,"last update":"Tue Jan 16 15:04:27 2024","interfaces":[{"interface":"back","average":{"1min":3.423,"5min":1.95,"15min":1.6379999999999999},"min":{"1min":1.3300000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":17.957000000000001,"5min":17.957000000000001,"15min":17.957000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":2.6480000000000001,"5min":1.8140000000000001,"15min":1.546},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":20.279,"5min":20.279,"15min":20.279},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:04:50.971 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:04:50.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:04:51.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.264+0000 7f3c7f200640 1 Processor -- start 2024-01-16T15:04:51.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.265+0000 7f3c7f200640 1 -- start start 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.266+0000 7f3c7f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c8015bda0 0x7f3c80165ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.266+0000 7f3c7f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80071080 con 0x7f3c801448c0 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.266+0000 7f3c7f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8015c1f0 con 0x7f3c8006dae0 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.266+0000 7f3c7f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c80144ca0 con 0x7f3c8015bda0 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c8015bda0 0x7f3c80165ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c8015bda0 0x7f3c80165ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50872/0 (socket says 172.21.15.43:50872) 2024-01-16T15:04:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 -- 172.21.15.43:0/1681900664 learned_addr learned my addr 172.21.15.43:0/1681900664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 -- 172.21.15.43:0/1681900664 >> v1:172.21.15.69:6789/0 conn(0x7f3c8006dae0 legacy=0x7f3c8015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 -- 172.21.15.43:0/1681900664 >> v1:172.21.15.43:6789/0 conn(0x7f3c801448c0 legacy=0x7f3c8006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.267+0000 7f3c7e800640 1 -- 172.21.15.43:0/1681900664 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8015c450 con 0x7f3c8015bda0 2024-01-16T15:04:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.268+0000 7f3c7e800640 1 --2- 172.21.15.43:0/1681900664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80165ce0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c7802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f06bc988a75411e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.269+0000 7f3c7d400640 1 -- 172.21.15.43:0/1681900664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7802cb20 con 0x7f3c8015bda0 2024-01-16T15:04:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.269+0000 7f3c7d400640 1 -- 172.21.15.43:0/1681900664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7802cc80 con 0x7f3c8015bda0 2024-01-16T15:04:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.269+0000 7f3c7d400640 1 -- 172.21.15.43:0/1681900664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c780343f0 con 0x7f3c8015bda0 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.269+0000 7f3c7f200640 1 -- 172.21.15.43:0/1681900664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 msgr2=0x7f3c80165ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.269+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1681900664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80165ce0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c7802bf80 comp rx=0 tx=0).stop 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.270+0000 7f3c7f200640 1 -- 172.21.15.43:0/1681900664 shutdown_connections 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.270+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1681900664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80165ce0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.270+0000 7f3c7f200640 1 -- 172.21.15.43:0/1681900664 >> 172.21.15.43:0/1681900664 conn(0x7f3c800b0aa0 msgr2=0x7f3c800b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.270+0000 7f3c7f200640 1 -- 172.21.15.43:0/1681900664 shutdown_connections 2024-01-16T15:04:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.270+0000 7f3c7f200640 1 -- 172.21.15.43:0/1681900664 wait complete. 2024-01-16T15:04:51.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.271+0000 7f3c7f200640 1 Processor -- start 2024-01-16T15:04:51.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.272+0000 7f3c7f200640 1 -- start start 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 0x7f3c80143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 0x7f3c801587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80143680 con 0x7f3c8015bda0 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801399f0 con 0x7f3c8006dae0 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80142a00 con 0x7f3c801448c0 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50888/0 (socket says 172.21.15.43:50888) 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7e800640 1 -- 172.21.15.43:0/1166697410 learned_addr learned my addr 172.21.15.43:0/1166697410 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c7de00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 0x7f3c80143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.273+0000 7f3c77e00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 0x7f3c801587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 -- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 msgr2=0x7f3c801587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 0x7f3c801587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 -- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 msgr2=0x7f3c80143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 0x7f3c80143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 -- 172.21.15.43:0/1166697410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78009b30 con 0x7f3c8015bda0 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7de00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 0x7f3c80143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c77e00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 0x7f3c801587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7e800640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78009510 con 0x7f3c8015bda0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78007e50 con 0x7f3c8015bda0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7803e690 con 0x7f3c8015bda0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.274+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c80142ce0 con 0x7f3c8015bda0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.275+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c801432a0 con 0x7f3c8015bda0 2024-01-16T15:04:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.276+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c801437c0 con 0x7f3c8015bda0 2024-01-16T15:04:51.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.277+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f3c7803e7f0 con 0x7f3c8015bda0 2024-01-16T15:04:51.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.277+0000 7f3c76000640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 0x7f3c5806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.278+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3c780acf90 con 0x7f3c8015bda0 2024-01-16T15:04:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.278+0000 7f3c7de00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 0x7f3c5806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.278+0000 7f3c7de00640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 0x7f3c5806dd50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000f60 tx=0x7f3c70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.282+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c801437c0 con 0x7f3c8015bda0 2024-01-16T15:04:51.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.549+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3c801437c0 con 0x7f3c8015bda0 2024-01-16T15:04:51.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.553+0000 7f3c76000640 1 -- 172.21.15.43:0/1166697410 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3c801437c0 con 0x7f3c8015bda0 2024-01-16T15:04:51.532 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:04:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:04:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:04:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:04:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:04:51.533 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.555+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 msgr2=0x7f3c5806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.555+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 0x7f3c5806dd50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000f60 tx=0x7f3c70009040 comp rx=0 tx=0).stop 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 msgr2=0x7f3c80141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009e50 tx=0x7f3c78004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 shutdown_connections 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3c5806b890 0x7f3c5806dd50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c8015bda0 0x7f3c80141eb0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c801448c0 0x7f3c801587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 --2- 172.21.15.43:0/1166697410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c8006dae0 0x7f3c80143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 >> 172.21.15.43:0/1166697410 conn(0x7f3c800b0aa0 msgr2=0x7f3c800cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 shutdown_connections 2024-01-16T15:04:51.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.556+0000 7f3c7f200640 1 -- 172.21.15.43:0/1166697410 wait complete. 2024-01-16T15:04:51.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:04:51.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.798+0000 7f6e5e600640 1 Processor -- start 2024-01-16T15:04:51.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.798+0000 7f6e5e600640 1 -- start start 2024-01-16T15:04:51.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6e5815d5b0 0x7f6e580d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 0x7f6e580cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e580cf0c0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e580d1e40 con 0x7f6e580cf7a0 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5e600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580b01b0 con 0x7f6e580d14a0 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.800+0000 7f6e5d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6e5815d5b0 0x7f6e580d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6e5815d5b0 0x7f6e580d3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50898/0 (socket says 172.21.15.43:50898) 2024-01-16T15:04:51.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 -- 172.21.15.43:0/2745997038 learned_addr learned my addr 172.21.15.43:0/2745997038 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:51.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5dc00640 1 --2- 172.21.15.43:0/2745997038 >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 0x7f6e580cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e4fe00640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 -- 172.21.15.43:0/2745997038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 msgr2=0x7f6e580d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 -- 172.21.15.43:0/2745997038 >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 msgr2=0x7f6e580cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 --2- 172.21.15.43:0/2745997038 >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 0x7f6e580cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5d200640 1 -- 172.21.15.43:0/2745997038 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5815d990 con 0x7f6e5815d5b0 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e5dc00640 1 --2- 172.21.15.43:0/2745997038 >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 0x7f6e580cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.801+0000 7f6e4fe00640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.803+0000 7f6e5d200640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e580d3990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6e580d2d60 tx=0x7f6e4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c849f0fa3bb2f433 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.803+0000 7f6e57e00640 1 -- 172.21.15.43:0/2745997038 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48031640 con 0x7f6e5815d5b0 2024-01-16T15:04:51.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.803+0000 7f6e57e00640 1 -- 172.21.15.43:0/2745997038 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e480317a0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e57e00640 1 -- 172.21.15.43:0/2745997038 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48031b30 con 0x7f6e5815d5b0 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 -- 172.21.15.43:0/2745997038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 msgr2=0x7f6e580d3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e580d3990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6e580d2d60 tx=0x7f6e4802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 -- 172.21.15.43:0/2745997038 shutdown_connections 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 --2- 172.21.15.43:0/2745997038 >> v2:172.21.15.69:3300/0 conn(0x7f6e580cf7a0 0x7f6e580cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 --2- 172.21.15.43:0/2745997038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e580d3990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 -- 172.21.15.43:0/2745997038 >> 172.21.15.43:0/2745997038 conn(0x7f6e5809c3b0 msgr2=0x7f6e580d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 -- 172.21.15.43:0/2745997038 shutdown_connections 2024-01-16T15:04:51.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.804+0000 7f6e5e600640 1 -- 172.21.15.43:0/2745997038 wait complete. 2024-01-16T15:04:51.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.806+0000 7f6e5e600640 1 Processor -- start 2024-01-16T15:04:51.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.806+0000 7f6e5e600640 1 -- start start 2024-01-16T15:04:51.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e580cf7a0 0x7f6e581f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580b3b70 con 0x7f6e5815d5b0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581f3b60 con 0x7f6e580cf7a0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.807+0000 7f6e5e600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581f32c0 con 0x7f6e580d14a0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50904/0 (socket says 172.21.15.43:50904) 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 -- 172.21.15.43:0/1743166149 learned_addr learned my addr 172.21.15.43:0/1743166149 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5d200640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e580cf7a0 0x7f6e581f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e4fe00640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 -- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 msgr2=0x7f6e580b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 -- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e580cf7a0 msgr2=0x7f6e581f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e580cf7a0 0x7f6e581f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.808+0000 7f6e5dc00640 1 -- 172.21.15.43:0/1743166149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480094d0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e5dc00640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000cd50 tx=0x7f6e5000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5000f680 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5000f800 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5000fbb0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e581f35a0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.809+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e580c85b0 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.811+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5804e840 con 0x7f6e5815d5b0 2024-01-16T15:04:51.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.811+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6e5000b040 con 0x7f6e5815d5b0 2024-01-16T15:04:51.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.813+0000 7f6e55600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 0x7f6e3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.813+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6e5008b170 con 0x7f6e5815d5b0 2024-01-16T15:04:51.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.813+0000 7f6e5d200640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 0x7f6e3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.814+0000 7f6e5d200640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 0x7f6e3006dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6e48036000 tx=0x7f6e4803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:51.817+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e50055640 con 0x7f6e5815d5b0 2024-01-16T15:04:51.878 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-01-16T15:04:51.900+0000 7f4e0c600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:04:52.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.051+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e580678d0 con 0x7f6e5815d5b0 2024-01-16T15:04:52.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.052+0000 7f6e55600640 1 -- 172.21.15.43:0/1743166149 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4904 (secure 0 0 0) 0x7f6e50058c60 con 0x7f6e5815d5b0 2024-01-16T15:04:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:04:52.032 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/986973826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":986973826},{"type":"v1","addr":"172.21.15.69:6835","nonce":986973826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5644,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/607340672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":607340672},{"type":"v1","addr":"172.21.15.43:6836","nonce":607340672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/154390899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":154390899},{"type":"v1","addr":"172.21.15.43:6837","nonce":154390899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:04:45.405522+0000","modified":"2024-01-16T15:04:48.466693+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5650":{"gid":5650,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/4073013627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4073013627},{"type":"v1","addr":"172.21.15.69:6834","nonce":4073013627}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:04:52.032 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 36 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 msgr2=0x7f6e3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 0x7f6e3006dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6e48036000 tx=0x7f6e4803c040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 msgr2=0x7f6e5815a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000cd50 tx=0x7f6e5000a650 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 shutdown_connections 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6e3006b860 0x7f6e3006dd20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e5815d5b0 0x7f6e5815a5f0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e580d14a0 0x7f6e580b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 --2- 172.21.15.43:0/1743166149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e580cf7a0 0x7f6e581f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.055+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 >> 172.21.15.43:0/1743166149 conn(0x7f6e5809c3b0 msgr2=0x7f6e580d36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.056+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 shutdown_connections 2024-01-16T15:04:52.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.056+0000 7f6e5e600640 1 -- 172.21.15.43:0/1743166149 wait complete. 2024-01-16T15:04:52.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:04:45.405522+0000', 'modified': '2024-01-16T15:04:48.466693+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5650}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5650': {'gid': 5650, 'name': 'd', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6834/4073013627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 4073013627}, {'type': 'v1', 'addr': '172.21.15.69:6834', 'nonce': 4073013627}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:04:52.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:04:52.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:04:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.712+0000 7f58b7400640 1 Processor -- start 2024-01-16T15:04:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.713+0000 7f58b7400640 1 -- start start 2024-01-16T15:04:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.714+0000 7f58b7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b013db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.714+0000 7f58b7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b015c430 0x7f58b015f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.714+0000 7f58b7400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b015faa0 con 0x7f58b00b3730 2024-01-16T15:04:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.714+0000 7f58b7400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b015b780 con 0x7f58b015c430 2024-01-16T15:04:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.714+0000 7f58b7400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b015b970 con 0x7f58b00b3350 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.715+0000 7f58b5400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b015c430 0x7f58b015f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.715+0000 7f58b4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b013db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.715+0000 7f58b5400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f58b015c430 0x7f58b015f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50910/0 (socket says 172.21.15.43:50910) 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.715+0000 7f58b5400640 1 -- 172.21.15.43:0/1694421179 learned_addr learned my addr 172.21.15.43:0/1694421179 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.715+0000 7f58afe00640 1 --1- 172.21.15.43:0/1694421179 >> v1:172.21.15.69:6789/0 conn(0x7f58b00b3730 0x7f58b015ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:39596/0 (socket says 172.21.15.43:39596) 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.716+0000 7f58b5400640 1 -- 172.21.15.43:0/1694421179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 msgr2=0x7f58b013db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.716+0000 7f58b5400640 1 --2- 172.21.15.43:0/1694421179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b013db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.716+0000 7f58b5400640 1 -- 172.21.15.43:0/1694421179 >> v1:172.21.15.69:6789/0 conn(0x7f58b00b3730 legacy=0x7f58b015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.716+0000 7f58af400640 1 -- 172.21.15.43:0/1694421179 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578594646 0 0) 0x7f58b015faa0 con 0x7f58b00b3730 2024-01-16T15:04:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.716+0000 7f58b5400640 1 -- 172.21.15.43:0/1694421179 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b015bc50 con 0x7f58b015c430 2024-01-16T15:04:52.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.717+0000 7f58b5400640 1 --2- 172.21.15.43:0/1694421179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b015f480 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5a8c656806e15a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.718+0000 7f58af400640 1 -- 172.21.15.43:0/1694421179 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8030840 con 0x7f58b015c430 2024-01-16T15:04:52.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.718+0000 7f58af400640 1 -- 172.21.15.43:0/1694421179 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a80309a0 con 0x7f58b015c430 2024-01-16T15:04:52.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.718+0000 7f58af400640 1 -- 172.21.15.43:0/1694421179 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8030d30 con 0x7f58b015c430 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.718+0000 7f58b7400640 1 -- 172.21.15.43:0/1694421179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 msgr2=0x7f58b015f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.718+0000 7f58b7400640 1 --2- 172.21.15.43:0/1694421179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b015f480 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a802b980 comp rx=0 tx=0).stop 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 -- 172.21.15.43:0/1694421179 shutdown_connections 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 --2- 172.21.15.43:0/1694421179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b015f480 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 --2- 172.21.15.43:0/1694421179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b013db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 -- 172.21.15.43:0/1694421179 >> 172.21.15.43:0/1694421179 conn(0x7f58b00663d0 msgr2=0x7f58b00667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 -- 172.21.15.43:0/1694421179 shutdown_connections 2024-01-16T15:04:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.719+0000 7f58b7400640 1 -- 172.21.15.43:0/1694421179 wait complete. 2024-01-16T15:04:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.721+0000 7f58b7400640 1 Processor -- start 2024-01-16T15:04:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.721+0000 7f58b7400640 1 -- start start 2024-01-16T15:04:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3730 0x7f58b00b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00b64a0 con 0x7f58b015c430 2024-01-16T15:04:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b0158d10 con 0x7f58b00b3730 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b7400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b0158140 con 0x7f58b00b3350 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50912/0 (socket says 172.21.15.43:50912) 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b5400640 1 -- 172.21.15.43:0/3346430718 learned_addr learned my addr 172.21.15.43:0/3346430718 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58afe00640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3730 0x7f58b00b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.722+0000 7f58b4a00640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 -- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 msgr2=0x7f58b015a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b015a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 -- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3730 msgr2=0x7f58b00b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3730 0x7f58b00b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 -- 172.21.15.43:0/3346430718 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80094d0 con 0x7f58b015c430 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b5400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58acc00640 1 -- 172.21.15.43:0/3346430718 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a803e070 con 0x7f58b015c430 2024-01-16T15:04:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58acc00640 1 -- 172.21.15.43:0/3346430718 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8004030 con 0x7f58b015c430 2024-01-16T15:04:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58acc00640 1 -- 172.21.15.43:0/3346430718 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80043e0 con 0x7f58b015c430 2024-01-16T15:04:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.723+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b00ac870 con 0x7f58b015c430 2024-01-16T15:04:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.724+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b01586c0 con 0x7f58b015c430 2024-01-16T15:04:52.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.726+0000 7f58acc00640 1 -- 172.21.15.43:0/3346430718 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f58a8042070 con 0x7f58b015c430 2024-01-16T15:04:52.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.727+0000 7f58acc00640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 0x7f588806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.727+0000 7f58acc00640 1 -- 172.21.15.43:0/3346430718 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58a80ac450 con 0x7f58b015c430 2024-01-16T15:04:52.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.727+0000 7f58b4a00640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 0x7f588806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.728+0000 7f58b4a00640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 0x7f588806dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009ad0 tx=0x7f58a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.728+0000 7f58ba17e740 1 Processor -- start 2024-01-16T15:04:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.728+0000 7f58ba17e740 1 -- start start 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab342ea0 0x561cab343280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab386560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab2a1880 con 0x561cab33dba0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab2b0ae0 con 0x561cab342ea0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.729+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cab2ae1d0 con 0x561cab3425e0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50922/0 (socket says 172.21.15.43:50922) 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 -- 172.21.15.43:0/3614405862 learned_addr learned my addr 172.21.15.43:0/3614405862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58b5400640 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab386560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58b4a00640 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab342ea0 0x561cab343280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 -- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 msgr2=0x561cab386560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab386560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 -- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab342ea0 msgr2=0x561cab343280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab342ea0 0x561cab343280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 -- 172.21.15.43:0/3614405862 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cab271860 con 0x561cab33dba0 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.730+0000 7f58afe00640 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5898001be0 tx=0x7f58980020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f64921d89d8ff1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.731+0000 7f58a5600640 1 -- 172.21.15.43:0/3614405862 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898017070 con 0x561cab33dba0 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.731+0000 7f58a5600640 1 -- 172.21.15.43:0/3614405862 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5898007910 con 0x561cab33dba0 2024-01-16T15:04:52.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.731+0000 7f58a5600640 1 -- 172.21.15.43:0/3614405862 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58980133f0 con 0x561cab33dba0 2024-01-16T15:04:52.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.731+0000 7f58ba17e740 1 -- 172.21.15.43:0/3614405862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 msgr2=0x561cab3420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.731+0000 7f58ba17e740 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5898001be0 tx=0x7f58980020a0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 -- 172.21.15.43:0/3614405862 shutdown_connections 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab386560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab33dba0 0x561cab3420a0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 --2- 172.21.15.43:0/3614405862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab342ea0 0x561cab343280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 -- 172.21.15.43:0/3614405862 >> 172.21.15.43:0/3614405862 conn(0x561cab304c70 msgr2=0x561cab388990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 -- 172.21.15.43:0/3614405862 shutdown_connections 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 -- 172.21.15.43:0/3614405862 wait complete. 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.732+0000 7f58ba17e740 1 -- start start 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.733+0000 7f58ba17e740 1 Processor -- start 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab367750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab36b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab2a9ac0 con 0x561cab342ea0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab2462f0 con 0x561cab33dba0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cab263c70 con 0x561cab3425e0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58ba17e740 10 client.0 _ll_register_callbacks cb 0x561cab2683b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50930/0 (socket says 172.21.15.43:50930) 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 -- 172.21.15.43:0/1788559552 learned_addr learned my addr 172.21.15.43:0/1788559552 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58afe00640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab36b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b4a00640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab367750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 -- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 msgr2=0x561cab36b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab36b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 -- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 msgr2=0x561cab367750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab367750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58b5400640 1 -- 172.21.15.43:0/1788559552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5898004dd0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.734+0000 7f58afe00640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab36b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58b5400640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f58a80aa300 tx=0x7f58a807a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87c84852bd1bb95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80a9eb0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58a80828e0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561cab2a2aa0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58a80ac2c0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58ba17e740 10 client.6480 fetch_fsmap learned FSMap version 36 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x561cab2a8140 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561cab2a2aa0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f58a80763d0 con 0x561cab342ea0 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58ba17e740 10 client.6480 fetch_fsmap finished waiting for FSMap version 36 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.735+0000 7f58ba17e740 10 client.6480 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.736+0000 7f58ba17e740 20 client.6480 populate_metadata read hostname 'smithi043' 2024-01-16T15:04:52.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.736+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab3905a0 0x561cab367e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.736+0000 7f58ba17e740 4 client.6480 mds_command: new command op to 5650 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:04:52.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.736+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 --> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561cab2a27c0 con 0x561cab3905a0 2024-01-16T15:04:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.736+0000 7f58b4a00640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab3905a0 0x561cab367e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.737+0000 7f58b4a00640 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab3905a0 0x561cab367e70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a386b39f32e0950 server_cookie=c3c05c1884be81dd in_seq=0 out_seq=0 2024-01-16T15:04:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.737+0000 7f589e000640 10 client.6480 ms_handle_connect on v2:172.21.15.69:6832/4073013627 2024-01-16T15:04:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.739+0000 7f589e000640 1 -- 172.21.15.43:0/1788559552 <== mds.0 v2:172.21.15.69:6832/4073013627 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561cab2a27c0 con 0x561cab3905a0 2024-01-16T15:04:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.739+0000 7f589e000640 10 client.6480 handle_command_reply: tid=0 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.739+0000 7f58ba17e740 1 client.6480 shutdown 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.739+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 msgr2=0x561cab36b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.739+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f58a80aa300 tx=0x7f58a807a030 comp rx=0 tx=0).stop 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 shutdown_connections 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab3905a0 0x561cab367e70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab36b810 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab36b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 --2- 172.21.15.43:0/1788559552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab367750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 >> 172.21.15.43:0/1788559552 conn(0x561cab304c70 msgr2=0x561cab386320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f589e000640 0 client.6480 ms_handle_reset on v2:172.21.15.69:6832/4073013627 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 shutdown_connections 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 1 -- 172.21.15.43:0/1788559552 wait complete. 2024-01-16T15:04:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.740+0000 7f58ba17e740 20 client.6480 trim_cache size 0 max 16384 2024-01-16T15:04:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.757+0000 7f58ba17e740 1 Processor -- start 2024-01-16T15:04:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.757+0000 7f58ba17e740 1 -- start start 2024-01-16T15:04:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab38ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab33cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab00cf20 con 0x561cab342ea0 2024-01-16T15:04:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab340a20 con 0x561cab33dba0 2024-01-16T15:04:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.758+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cab39b3d0 con 0x561cab3425e0 2024-01-16T15:04:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50934/0 (socket says 172.21.15.43:50934) 2024-01-16T15:04:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 -- 172.21.15.43:0/4242332001 learned_addr learned my addr 172.21.15.43:0/4242332001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58afe00640 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab33cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b4a00640 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab38ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 -- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 msgr2=0x561cab33cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab33cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 -- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 msgr2=0x561cab38ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab38ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 -- 172.21.15.43:0/4242332001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cab245b80 con 0x561cab342ea0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.759+0000 7f58b5400640 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f58a807a570 tx=0x7f58a808f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc0c2619132219e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58a6000640 1 -- 172.21.15.43:0/4242332001 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80ac2c0 con 0x561cab342ea0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58a6000640 1 -- 172.21.15.43:0/4242332001 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a807a5e0 con 0x561cab342ea0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58a6000640 1 -- 172.21.15.43:0/4242332001 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8082690 con 0x561cab342ea0 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 -- 172.21.15.43:0/4242332001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 msgr2=0x561cab33d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f58a807a570 tx=0x7f58a808f000 comp rx=0 tx=0).stop 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 -- 172.21.15.43:0/4242332001 shutdown_connections 2024-01-16T15:04:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab33d0c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab3425e0 0x561cab33cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 --2- 172.21.15.43:0/4242332001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab33dba0 0x561cab38ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 -- 172.21.15.43:0/4242332001 >> 172.21.15.43:0/4242332001 conn(0x561cab304c70 msgr2=0x561cab343280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 -- 172.21.15.43:0/4242332001 shutdown_connections 2024-01-16T15:04:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.760+0000 7f58ba17e740 1 -- 172.21.15.43:0/4242332001 wait complete. 2024-01-16T15:04:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.761+0000 7f58ba17e740 1 -- start start 2024-01-16T15:04:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.761+0000 7f58ba17e740 1 Processor -- start 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab33dba0 0x561cab38c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab3425e0 0x561cab3419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab38d3f0 con 0x561cab342ea0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cab3a1c60 con 0x561cab3425e0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cab33d120 con 0x561cab33dba0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58ba17e740 10 client.0 _ll_register_callbacks cb 0x7f58a8006bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50942/0 (socket says 172.21.15.43:50942) 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 -- 172.21.15.43:0/300272318 learned_addr learned my addr 172.21.15.43:0/300272318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b4a00640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab33dba0 0x561cab38c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 -- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab33dba0 msgr2=0x561cab38c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58afe00640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab3425e0 0x561cab3419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab33dba0 0x561cab38c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 -- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab3425e0 msgr2=0x561cab3419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab3425e0 0x561cab3419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.763+0000 7f58b5400640 1 -- 172.21.15.43:0/300272318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561cab2a8140 con 0x561cab342ea0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f58b5400640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f58a807a570 tx=0x7f58a807d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c75756c550bdc4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f589e000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8082d30 con 0x561cab342ea0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58a80828e0 con 0x561cab342ea0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5898007910 con 0x561cab342ea0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58a8082e90 con 0x561cab342ea0 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f58ba17e740 10 client.6486 fetch_fsmap learned FSMap version 36 2024-01-16T15:04:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x561cab26b170 con 0x561cab342ea0 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.764+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5898007910 con 0x561cab342ea0 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f58a8079540 con 0x561cab342ea0 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 10 client.6486 fetch_fsmap finished waiting for FSMap version 36 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 10 client.6486 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 20 client.6486 populate_metadata read hostname 'smithi043' 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab38f6c0 0x561cab367d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 4 client.6486 mds_command: new command op to 5650 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:04:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 --> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561caafaa6c0 con 0x561cab38f6c0 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.765+0000 7f58b4a00640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab38f6c0 0x561cab367d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.766+0000 7f58b4a00640 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab38f6c0 0x561cab367d10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7fa3136d586c9c server_cookie=68f1916f7341cc9f in_seq=0 out_seq=0 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.766+0000 7f589e000640 10 client.6486 ms_handle_connect on v2:172.21.15.69:6832/4073013627 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f589e000640 1 -- 172.21.15.43:0/300272318 <== mds.0 v2:172.21.15.69:6832/4073013627 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561caafaa6c0 con 0x561cab38f6c0 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f589e000640 10 client.6486 handle_command_reply: tid=0 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 client.6486 shutdown 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 msgr2=0x561cab341f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f58a807a570 tx=0x7f58a807d040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 shutdown_connections 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:6832/4073013627,v1:172.21.15.69:6834/4073013627] conn(0x561cab38f6c0 0x561cab367d10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561cab3425e0 0x561cab3419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cab342ea0 0x561cab341f00 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 --2- 172.21.15.43:0/300272318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561cab33dba0 0x561cab38c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f589e000640 0 client.6486 ms_handle_reset on v2:172.21.15.69:6832/4073013627 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 >> 172.21.15.43:0/300272318 conn(0x561cab304c70 msgr2=0x561cab371680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 shutdown_connections 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.767+0000 7f58ba17e740 1 -- 172.21.15.43:0/300272318 wait complete. 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58ba17e740 20 client.6486 trim_cache size 0 max 16384 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:04:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5650, 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.8314009320000002, 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 36, 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 61, 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 61, 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 12.193627463 2024-01-16T15:04:52.750 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 msgr2=0x7f588806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 0x7f588806dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009ad0 tx=0x7f58a0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 msgr2=0x7f58b0150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a80046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 shutdown_connections 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f588806b860 0x7f588806dd20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3730 0x7f58b00b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b015c430 0x7f58b0150870 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 --2- 172.21.15.43:0/3346430718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58b00b3350 0x7f58b015a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 >> 172.21.15.43:0/3346430718 conn(0x7f58b00663d0 msgr2=0x7f58b0151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 shutdown_connections 2024-01-16T15:04:52.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.768+0000 7f58b7400640 1 -- 172.21.15.43:0/3346430718 wait complete. 2024-01-16T15:04:52.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:04:52.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.998+0000 7f93d6c00640 1 Processor -- start 2024-01-16T15:04:52.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.998+0000 7f93d6c00640 1 -- start start 2024-01-16T15:04:52.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.999+0000 7f93d6c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93d015b2c0 0x7f93d00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.999+0000 7f93d6c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 0x7f93d015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.999+0000 7f93d6c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d00d00f0 con 0x7f93d015bbe0 2024-01-16T15:04:52.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.999+0000 7f93d6c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d00d03d0 con 0x7f93d015b2c0 2024-01-16T15:04:52.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:52.999+0000 7f93d6c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d00d06b0 con 0x7f93d00b46a0 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93d015b2c0 0x7f93d00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93c7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 0x7f93d015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93d015b2c0 0x7f93d00cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50188/0 (socket says 172.21.15.43:50188) 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 -- 172.21.15.43:0/2618624537 learned_addr learned my addr 172.21.15.43:0/2618624537 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 -- 172.21.15.43:0/2618624537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 msgr2=0x7f93d015b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 0x7f93d015b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 -- 172.21.15.43:0/2618624537 >> v1:172.21.15.43:6789/0 conn(0x7f93d015bbe0 legacy=0x7f93d006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93cfe00640 1 -- 172.21.15.43:0/2618624537 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d00d0990 con 0x7f93d015b2c0 2024-01-16T15:04:52.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.000+0000 7f93c7e00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 0x7f93d015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:52.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.002+0000 7f93cfe00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d00cfaf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009870 tx=0x7f93c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92deb4cb6ae6c785 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93cf400640 1 -- 172.21.15.43:0/2618624537 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0031640 con 0x7f93d015b2c0 2024-01-16T15:04:52.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93cf400640 1 -- 172.21.15.43:0/2618624537 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c00317a0 con 0x7f93d015b2c0 2024-01-16T15:04:52.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93cf400640 1 -- 172.21.15.43:0/2618624537 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0031b30 con 0x7f93d015b2c0 2024-01-16T15:04:52.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 -- 172.21.15.43:0/2618624537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 msgr2=0x7f93d00cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d00cfaf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009870 tx=0x7f93c002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 -- 172.21.15.43:0/2618624537 shutdown_connections 2024-01-16T15:04:52.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d00b46a0 0x7f93d015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2618624537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d00cfaf0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.003+0000 7f93d6c00640 1 -- 172.21.15.43:0/2618624537 >> 172.21.15.43:0/2618624537 conn(0x7f93d00b2e00 msgr2=0x7f93d00b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.004+0000 7f93d6c00640 1 -- 172.21.15.43:0/2618624537 shutdown_connections 2024-01-16T15:04:52.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.004+0000 7f93d6c00640 1 -- 172.21.15.43:0/2618624537 wait complete. 2024-01-16T15:04:52.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.005+0000 7f93d6c00640 1 Processor -- start 2024-01-16T15:04:52.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.006+0000 7f93d6c00640 1 -- start start 2024-01-16T15:04:52.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.006+0000 7f93d6c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93d00b46a0 0x7f93d0142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.006+0000 7f93d6c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93d6c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d015bbe0 0x7f93d015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93d6c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0143140 con 0x7f93d015b2c0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93d6c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01517f0 con 0x7f93d00b46a0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93d6c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0151ad0 con 0x7f93d015bbe0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50198/0 (socket says 172.21.15.43:50198) 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 -- 172.21.15.43:0/2578284512 learned_addr learned my addr 172.21.15.43:0/2578284512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93d4c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d015bbe0 0x7f93d015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93cfe00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93d00b46a0 0x7f93d0142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 -- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d015bbe0 msgr2=0x7f93d015c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d015bbe0 0x7f93d015c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 -- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93d00b46a0 msgr2=0x7f93d0142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93d00b46a0 0x7f93d0142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.007+0000 7f93c7e00640 1 -- 172.21.15.43:0/2578284512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8009400 con 0x7f93d015b2c0 2024-01-16T15:04:52.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93c7e00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f93b800ebe0 tx=0x7f93b800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b8009760 con 0x7f93d015b2c0 2024-01-16T15:04:52.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b800cc30 con 0x7f93d015b2c0 2024-01-16T15:04:52.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c00094d0 con 0x7f93d015b2c0 2024-01-16T15:04:52.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800eee0 con 0x7f93d015b2c0 2024-01-16T15:04:52.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.008+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d0152090 con 0x7f93d015b2c0 2024-01-16T15:04:52.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.010+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d0074770 con 0x7f93d015b2c0 2024-01-16T15:04:52.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.010+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f93b8013470 con 0x7f93d015b2c0 2024-01-16T15:04:52.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.012+0000 7f93ccc00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.012+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f93b808a990 con 0x7f93d015b2c0 2024-01-16T15:04:52.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.012+0000 7f93cfe00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.013+0000 7f93cfe00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 0x7f93a806dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f93c0036000 tx=0x7f93c003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.016+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93b8054f30 con 0x7f93d015b2c0 2024-01-16T15:04:53.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.251+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93d00b3c00 con 0x7f93d015b2c0 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.252+0000 7f93ccc00640 1 -- 172.21.15.43:0/2578284512 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7f93b8017060 con 0x7f93d015b2c0 2024-01-16T15:04:53.231 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 msgr2=0x7f93a806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 0x7f93a806dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f93c0036000 tx=0x7f93c003c040 comp rx=0 tx=0).stop 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 msgr2=0x7f93d015c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f93b800ebe0 tx=0x7f93b800c520 comp rx=0 tx=0).stop 2024-01-16T15:04:53.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 shutdown_connections 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93d015bbe0 0x7f93d015c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93d015b2c0 0x7f93d015c0d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 --2- 172.21.15.43:0/2578284512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93d00b46a0 0x7f93d0142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 >> 172.21.15.43:0/2578284512 conn(0x7f93d00b2e00 msgr2=0x7f93d00b4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 shutdown_connections 2024-01-16T15:04:53.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:53.255+0000 7f93d6c00640 1 -- 172.21.15.43:0/2578284512 wait complete. 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:04:53.253 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:04:53.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.253 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:04:53.304 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:04:53.304 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:04:53.304 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:04:53.304 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.304 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42145sec preferred_lft 42145sec 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:53.305 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:04:53.306 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:53.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.307 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:04:53.403 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:04:53.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.407 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:04:53.465 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:04:53.465 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:04:53.465 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:04:53.465 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:04:53.465 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:04:53.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:04:53.466 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:04:53.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.662 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:04:53.716 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:04:53.716 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:04:53.717 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:04:53.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:53.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:04:53.823 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:04:53.869 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:04:53.869 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:53.933 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:04:53.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:04:53.933 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:53.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:04:54.081 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:04:54.142 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:04:54.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.144 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:04:54.202 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:04:54.202 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:04:54.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:04:54.290 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:04:54.334 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:04:54.335 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:04:54.336 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:04:54.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.337 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:04:54.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:04:54.422+0000 7fa065442480 -1 init, newargv = 0x561c0c1df880 newargc=15 2024-01-16T15:04:54.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[56534]: starting ceph client 2024-01-16T15:04:54.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[56534]: starting fuse 2024-01-16T15:04:54.423 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:04:54.423 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:04:54.423 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:04:54.424 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:04:54.425 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:04:54.629 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.56534.asok 2024-01-16T15:04:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:04:54.647 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56534.asok 2024-01-16T15:04:54.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56534.asok status 2024-01-16T15:04:54.893 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:04:54.893 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "56534", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 6501, 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:04:54.894 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 6501 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2475926428 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2475926428 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.6501 192.168.0.1:0/2475926428", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/2475926428", 2024-01-16T15:04:54.895 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 36, 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 61, 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:04:54.896 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:04:54.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.909 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:04:54.909 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:54.965 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:04:54.965 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:54.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:54.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:04:55.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:04:55.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:55.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:55.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.307+0000 7f2b3f000640 1 Processor -- start 2024-01-16T15:04:55.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.307+0000 7f2b3f000640 1 -- start start 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.308+0000 7f2b3f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 0x7f2b3806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.308+0000 7f2b3f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b380abff0 0x7f2b3815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.308+0000 7f2b3f000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3815c360 con 0x7f2b380ad720 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.308+0000 7f2b3f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b380b41a0 con 0x7f2b380abff0 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.308+0000 7f2b3f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3806de40 con 0x7f2b3806d520 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b2f200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2b380ad720 0x7f2b3815ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52102/0 (socket says 172.21.15.43:52102) 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b2f200640 1 -- 172.21.15.43:0/1877383148 learned_addr learned my addr 172.21.15.43:0/1877383148 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b3d000640 1 --2- 172.21.15.43:0/1877383148 >> v2:172.21.15.69:3300/0 conn(0x7f2b380abff0 0x7f2b3815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b37e00640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 0x7f2b3806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b3d000640 1 -- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 msgr2=0x7f2b3806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.310+0000 7f2b3d000640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 0x7f2b3806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.309+0000 7f2b37400640 1 -- 172.21.15.43:0/1877383148 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612624037 0 0) 0x7f2b3815c360 con 0x7f2b380ad720 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.310+0000 7f2b3d000640 1 -- 172.21.15.43:0/1877383148 >> v1:172.21.15.43:6789/0 conn(0x7f2b380ad720 legacy=0x7f2b3815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.310+0000 7f2b3d000640 1 -- 172.21.15.43:0/1877383148 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3815ba30 con 0x7f2b380abff0 2024-01-16T15:04:55.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.310+0000 7f2b37e00640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 0x7f2b3806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:55.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.311+0000 7f2b3d000640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b3815f3e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b300097a0 tx=0x7f2b3002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ccfbb1960cbea24 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.311+0000 7f2b37400640 1 -- 172.21.15.43:0/1877383148 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3002d040 con 0x7f2b380abff0 2024-01-16T15:04:55.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.311+0000 7f2b37400640 1 -- 172.21.15.43:0/1877383148 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b30031810 con 0x7f2b380abff0 2024-01-16T15:04:55.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.311+0000 7f2b37400640 1 -- 172.21.15.43:0/1877383148 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30031ba0 con 0x7f2b380abff0 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 -- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 msgr2=0x7f2b3815f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b3815f3e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b300097a0 tx=0x7f2b3002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 -- 172.21.15.43:0/1877383148 shutdown_connections 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b3815f3e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 --2- 172.21.15.43:0/1877383148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b3806d520 0x7f2b3806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 -- 172.21.15.43:0/1877383148 >> 172.21.15.43:0/1877383148 conn(0x7f2b380b0d80 msgr2=0x7f2b380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 -- 172.21.15.43:0/1877383148 shutdown_connections 2024-01-16T15:04:55.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.312+0000 7f2b3f000640 1 -- 172.21.15.43:0/1877383148 wait complete. 2024-01-16T15:04:55.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.314+0000 7f2b3f000640 1 Processor -- start 2024-01-16T15:04:55.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.314+0000 7f2b3f000640 1 -- start start 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b381f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b380ad720 0x7f2b380c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381f5f00 con 0x7f2b3806d520 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380cb5e0 con 0x7f2b380abff0 2024-01-16T15:04:55.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b38070630 con 0x7f2b380ad720 2024-01-16T15:04:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b37e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b37e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50214/0 (socket says 172.21.15.43:50214) 2024-01-16T15:04:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b2f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b381f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b37e00640 1 -- 172.21.15.43:0/249982573 learned_addr learned my addr 172.21.15.43:0/249982573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.315+0000 7f2b3d000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b380ad720 0x7f2b380c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 -- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b380ad720 msgr2=0x7f2b380c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b380ad720 0x7f2b380c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 -- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 msgr2=0x7f2b381f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b381f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 -- 172.21.15.43:0/249982573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b28009a70 con 0x7f2b3806d520 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.316+0000 7f2b37e00640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800efc0 tx=0x7f2b2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.317+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28014680 con 0x7f2b3806d520 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.317+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2800f040 con 0x7f2b3806d520 2024-01-16T15:04:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.317+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28014a00 con 0x7f2b3806d520 2024-01-16T15:04:55.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.317+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b30009400 con 0x7f2b3806d520 2024-01-16T15:04:55.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.317+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b38070bf0 con 0x7f2b3806d520 2024-01-16T15:04:55.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.318+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b38070490 con 0x7f2b3806d520 2024-01-16T15:04:55.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.319+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2b28014b60 con 0x7f2b3806d520 2024-01-16T15:04:55.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.320+0000 7f2b34c00640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 0x7f2b1006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:55.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.320+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2b2808b630 con 0x7f2b3806d520 2024-01-16T15:04:55.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.321+0000 7f2b2f200640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 0x7f2b1006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:55.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.321+0000 7f2b2f200640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 0x7f2b1006dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b380ce570 tx=0x7f2b20007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:55.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.328+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b38070490 con 0x7f2b3806d520 2024-01-16T15:04:55.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.564+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2b38070490 con 0x7f2b3806d520 2024-01-16T15:04:55.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.565+0000 7f2b34c00640 1 -- 172.21.15.43:0/249982573 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2b38070490 con 0x7f2b3806d520 2024-01-16T15:04:55.544 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:04:55.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 msgr2=0x7f2b1006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 0x7f2b1006dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b380ce570 tx=0x7f2b20007de0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 msgr2=0x7f2b381f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800efc0 tx=0x7f2b2800c490 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 shutdown_connections 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2b1006b860 0x7f2b1006dd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b380abff0 0x7f2b381f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b380ad720 0x7f2b380c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 --2- 172.21.15.43:0/249982573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b3806d520 0x7f2b381f3d60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 >> 172.21.15.43:0/249982573 conn(0x7f2b380b0d80 msgr2=0x7f2b381573d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 shutdown_connections 2024-01-16T15:04:55.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:55.568+0000 7f2b3f000640 1 -- 172.21.15.43:0/249982573 wait complete. 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:04:55.568 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:04:55.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.569 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:04:55.625 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:04:55.626 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42304sec preferred_lft 42304sec 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:04:55.627 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:04:55.628 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:04:55.628 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:04:55.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.629 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:04:55.721 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:04:55.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.726 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:04:55.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:04:55.775 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:04:55.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.948 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:04:55.998 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:04:55.998 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:04:55.999 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:04:55.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:55.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:04:56.093 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:04:56.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:04:56.093 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:56.148 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:04:56.148 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:04:56.148 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:56.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:04:56.288 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:04:56.343 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:04:56.344 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:04:56.345 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:04:56.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:56.346 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:04:56.395 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:04:56.396 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:04:56.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:04:56.479 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:04:56.520 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:04:56.521 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:04:56.522 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:04:56.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:56.523 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:04:56.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:04:56.611+0000 7fc1fb43d480 -1 init, newargv = 0x563e8d36d930 newargc=15 2024-01-16T15:04:56.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44117]: starting ceph client 2024-01-16T15:04:56.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44117]: starting fuse 2024-01-16T15:04:56.609 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:04:56.609 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:04:56.610 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:04:56.611 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:04:56.802 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.44117.asok 2024-01-16T15:04:56.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:04:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:04:56.819 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44117.asok 2024-01-16T15:04:56.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:56.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44117.asok status 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "44117", 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:04:57.055 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5285, 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5285 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1818723633 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:04:57.056 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1818723633 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5285 192.168.0.1:0/1818723633", 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/1818723633", 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 36, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 61, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:04:57.057 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:04:57.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:57.070 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:04:57.070 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:57.122 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:04:57.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:57.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:57.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:04:57.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:04:57.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.472+0000 7fd7a4a00640 1 Processor -- start 2024-01-16T15:04:57.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.472+0000 7fd7a4a00640 1 -- start start 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd7a4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7a00ad450 0x7fd7a0154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a0070850 0x7fd7a0071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd7a4a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00ce650 con 0x7fd7a0153ea0 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd7a4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a00ad8e0 con 0x7fd7a00ad450 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd7a4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01547c0 con 0x7fd7a0070850 2024-01-16T15:04:57.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd79f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd7a0153ea0 0x7fd7a015d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52104/0 (socket says 172.21.15.43:52104) 2024-01-16T15:04:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.474+0000 7fd79f400640 1 -- 172.21.15.43:0/299598317 learned_addr learned my addr 172.21.15.43:0/299598317 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd797e00640 1 --2- 172.21.15.43:0/299598317 >> v2:172.21.15.69:3300/0 conn(0x7fd7a00ad450 0x7fd7a0154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79fe00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a0070850 0x7fd7a0071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287626542 0 0) 0x7fd7a00ce650 con 0x7fd7a0153ea0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd780003520 con 0x7fd7a0153ea0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215943860 0 0) 0x7fd780003520 con 0x7fd7a0153ea0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a0070850 msgr2=0x7fd7a0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a0070850 0x7fd7a0071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 >> v2:172.21.15.69:3300/0 conn(0x7fd7a00ad450 msgr2=0x7fd7a0154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 --2- 172.21.15.43:0/299598317 >> v2:172.21.15.69:3300/0 conn(0x7fd7a00ad450 0x7fd7a0154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a015e0a0 con 0x7fd7a0153ea0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd790001670 con 0x7fd7a0153ea0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 >> v1:172.21.15.43:6789/0 conn(0x7fd7a0153ea0 legacy=0x7fd7a015d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.475+0000 7fd79ea00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd780003a10 0x7fd780003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79ea00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79ea00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01547c0 con 0x7fd7800047c0 2024-01-16T15:04:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78000bf20 con 0x7fd780003a10 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78000c0e0 con 0x7fd780005180 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79fe00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a0070850 0x7fd7a0071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79f400640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd780003a10 0x7fd780003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 -- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd780005180 msgr2=0x7fd78000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 -- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd780003a10 msgr2=0x7fd780003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd780003a10 0x7fd780003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd797e00640 1 -- 172.21.15.43:0/299598317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780003520 con 0x7fd7800047c0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.476+0000 7fd79f400640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd780003a10 0x7fd780003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.477+0000 7fd797e00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd788009ac0 tx=0x7fd7800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e876833dd777e78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78802c710 con 0x7fd7800047c0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78802c870 con 0x7fd7800047c0 2024-01-16T15:04:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd79ea00640 1 -- 172.21.15.43:0/299598317 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78802cc00 con 0x7fd7800047c0 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd7a4a00640 1 -- 172.21.15.43:0/299598317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 msgr2=0x7fd780004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/299598317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd788009ac0 tx=0x7fd7800057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd79fe00640 1 -- 172.21.15.43:0/299598317 reap_dead start 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd7a4a00640 1 -- 172.21.15.43:0/299598317 shutdown_connections 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd7a4a00640 1 -- 172.21.15.43:0/299598317 >> 172.21.15.43:0/299598317 conn(0x7fd7a00cec10 msgr2=0x7fd7a00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.478+0000 7fd7a4a00640 1 -- 172.21.15.43:0/299598317 shutdown_connections 2024-01-16T15:04:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.479+0000 7fd7a4a00640 1 -- 172.21.15.43:0/299598317 wait complete. 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.481+0000 7fd7a4a00640 1 Processor -- start 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.481+0000 7fd7a4a00640 1 -- start start 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a015caa0 0x7fd7a014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a00b30b0 0x7fd7a00b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01f2760 con 0x7fd7800047c0 2024-01-16T15:04:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a0155950 con 0x7fd7a00b30b0 2024-01-16T15:04:57.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd7a4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a014bb50 con 0x7fd7a015caa0 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.482+0000 7fd79f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50232/0 (socket says 172.21.15.43:50232) 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 -- 172.21.15.43:0/2818019892 learned_addr learned my addr 172.21.15.43:0/2818019892 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79fe00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a00b30b0 0x7fd7a00b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd797e00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a015caa0 0x7fd7a014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 -- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a015caa0 msgr2=0x7fd7a014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a015caa0 0x7fd7a014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 -- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a00b30b0 msgr2=0x7fd7a00b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a00b30b0 0x7fd7a00b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 -- 172.21.15.43:0/2818019892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a014fba0 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.483+0000 7fd79f400640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d8e0 tx=0x7fd79000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.484+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900074e0 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.484+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790007640 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.484+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790004a00 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.484+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a0155ba0 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.484+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a0156160 con 0x7fd7800047c0 2024-01-16T15:04:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.485+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a004e840 con 0x7fd7800047c0 2024-01-16T15:04:57.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.486+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fd790004b60 con 0x7fd7800047c0 2024-01-16T15:04:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.487+0000 7fd797400640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 0x7fd77806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.487+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd79008a400 con 0x7fd7800047c0 2024-01-16T15:04:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.487+0000 7fd797e00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 0x7fd77806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.488+0000 7fd797e00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 0x7fd77806dd70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd788036000 tx=0x7fd788030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.491+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd790058450 con 0x7fd7800047c0 2024-01-16T15:04:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:57.723+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd7a006b450 con 0x7fd7800047c0 2024-01-16T15:04:57.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.015+0000 7fd797400640 1 -- 172.21.15.43:0/2818019892 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd7900190c0 con 0x7fd7800047c0 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 msgr2=0x7fd77806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 0x7fd77806dd70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd788036000 tx=0x7fd788030000 comp rx=0 tx=0).stop 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 msgr2=0x7fd7a01f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d8e0 tx=0x7fd79000dda0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 shutdown_connections 2024-01-16T15:04:57.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd77806b8b0 0x7fd77806dd70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7a00b30b0 0x7fd7a00b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7a015caa0 0x7fd7a014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 --2- 172.21.15.43:0/2818019892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7800047c0 0x7fd7a01f2220 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 >> 172.21.15.43:0/2818019892 conn(0x7fd7a00cec10 msgr2=0x7fd7a00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 shutdown_connections 2024-01-16T15:04:57.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.019+0000 7fd7a4a00640 1 -- 172.21.15.43:0/2818019892 wait complete. 2024-01-16T15:04:58.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:04:58.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.267+0000 7fb16b200640 1 Processor -- start 2024-01-16T15:04:58.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.267+0000 7fb16b200640 1 -- start start 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.268+0000 7fb16b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1641429c0 0x7fb164071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb16b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb16415e060 0x7fb1640b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb16b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb164070e70 con 0x7fb164071f10 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb16b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb164070050 con 0x7fb16415e060 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb16b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb164070260 con 0x7fb1641429c0 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb169400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb164071f10 0x7fb16415d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52112/0 (socket says 172.21.15.43:52112) 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb169400640 1 -- 172.21.15.43:0/4179127948 learned_addr learned my addr 172.21.15.43:0/4179127948 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb16a800640 1 --2- 172.21.15.43:0/4179127948 >> v2:172.21.15.43:3300/0 conn(0x7fb16415e060 0x7fb1640b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb169e00640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1641429c0 0x7fb164071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.269+0000 7fb168a00640 1 -- 172.21.15.43:0/4179127948 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041856767 0 0) 0x7fb164070e70 con 0x7fb164071f10 2024-01-16T15:04:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.270+0000 7fb168a00640 1 -- 172.21.15.43:0/4179127948 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144005730 con 0x7fb164071f10 2024-01-16T15:04:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.270+0000 7fb16a800640 1 -- 172.21.15.43:0/4179127948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1641429c0 msgr2=0x7fb164071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.270+0000 7fb16a800640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1641429c0 0x7fb164071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.270+0000 7fb16a800640 1 -- 172.21.15.43:0/4179127948 >> v1:172.21.15.43:6789/0 conn(0x7fb164071f10 legacy=0x7fb16415d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.270+0000 7fb16a800640 1 -- 172.21.15.43:0/4179127948 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164070540 con 0x7fb16415e060 2024-01-16T15:04:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.271+0000 7fb16a800640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb16415e060 0x7fb1640b1da0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009dc0 tx=0x7fb15c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40895c86f23ab51 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.271+0000 7fb168a00640 1 -- 172.21.15.43:0/4179127948 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02cc40 con 0x7fb16415e060 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.271+0000 7fb168a00640 1 -- 172.21.15.43:0/4179127948 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c02cda0 con 0x7fb16415e060 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.271+0000 7fb168a00640 1 -- 172.21.15.43:0/4179127948 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c034660 con 0x7fb16415e060 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 -- 172.21.15.43:0/4179127948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb16415e060 msgr2=0x7fb1640b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb16415e060 0x7fb1640b1da0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009dc0 tx=0x7fb15c02be90 comp rx=0 tx=0).stop 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 -- 172.21.15.43:0/4179127948 shutdown_connections 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb16415e060 0x7fb1640b1da0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 --2- 172.21.15.43:0/4179127948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1641429c0 0x7fb164071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 -- 172.21.15.43:0/4179127948 >> 172.21.15.43:0/4179127948 conn(0x7fb16406a620 msgr2=0x7fb16406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 -- 172.21.15.43:0/4179127948 shutdown_connections 2024-01-16T15:04:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.272+0000 7fb16b200640 1 -- 172.21.15.43:0/4179127948 wait complete. 2024-01-16T15:04:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.274+0000 7fb16b200640 1 Processor -- start 2024-01-16T15:04:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.274+0000 7fb16b200640 1 -- start start 2024-01-16T15:04:58.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1641429c0 0x7fb164137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb16415e060 0x7fb164134730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415b370 con 0x7fb164071f10 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415a410 con 0x7fb1641429c0 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb16b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415a6f0 con 0x7fb16415e060 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb169e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb169e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50244/0 (socket says 172.21.15.43:50244) 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.275+0000 7fb169e00640 1 -- 172.21.15.43:0/759287546 learned_addr learned my addr 172.21.15.43:0/759287546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 -- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb16415e060 msgr2=0x7fb164134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb16415e060 0x7fb164134730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169400640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1641429c0 0x7fb164137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 -- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1641429c0 msgr2=0x7fb164137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1641429c0 0x7fb164137410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 -- 172.21.15.43:0/759287546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009a70 con 0x7fb164071f10 2024-01-16T15:04:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb169e00640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb15400d800 tx=0x7fb15400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15400b6c0 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15400b820 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15400bbb0 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.276+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16415a9d0 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.277+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16415af90 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.278+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16404e840 con 0x7fb164071f10 2024-01-16T15:04:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.278+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb15400bd10 con 0x7fb164071f10 2024-01-16T15:04:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.279+0000 7fb162000640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 0x7fb13c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.280+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb15408ac40 con 0x7fb164071f10 2024-01-16T15:04:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.280+0000 7fb169400640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 0x7fb13c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.281+0000 7fb169400640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 0x7fb13c06dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb16413c620 tx=0x7fb14c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.285+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1540551e0 con 0x7fb164071f10 2024-01-16T15:04:58.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.505+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb164070b00 con 0x7fb13c06b860 2024-01-16T15:04:58.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.508+0000 7fb162000640 1 -- 172.21.15.43:0/759287546 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb14c001c40 con 0x7fb13c06b860 2024-01-16T15:04:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:04:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:04:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:04:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:04:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:04:58.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 msgr2=0x7fb13c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 0x7fb13c06dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb16413c620 tx=0x7fb14c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 msgr2=0x7fb164139860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb15400d800 tx=0x7fb15400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 shutdown_connections 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb13c06b860 0x7fb13c06dd20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1641429c0 0x7fb164137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb16415e060 0x7fb164134730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 --2- 172.21.15.43:0/759287546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb164071f10 0x7fb164139860 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 >> 172.21.15.43:0/759287546 conn(0x7fb16406a620 msgr2=0x7fb16413dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.510+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 shutdown_connections 2024-01-16T15:04:58.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.511+0000 7fb16b200640 1 -- 172.21.15.43:0/759287546 wait complete. 2024-01-16T15:04:58.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:04:58.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.751+0000 7fb943200640 1 Processor -- start 2024-01-16T15:04:58.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.752+0000 7fb943200640 1 -- start start 2024-01-16T15:04:58.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.753+0000 7fb943200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 0x7fb94415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.753+0000 7fb943200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944144ca0 con 0x7fb94415bbd0 2024-01-16T15:04:58.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.753+0000 7fb943200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94415c450 con 0x7fb9441448c0 2024-01-16T15:04:58.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.753+0000 7fb943200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440715b0 con 0x7fb94406dae0 2024-01-16T15:04:58.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb942800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb94415bbd0 0x7fb944070070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52126/0 (socket says 172.21.15.43:52126) 2024-01-16T15:04:58.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb942800640 1 -- 172.21.15.43:0/2921661773 learned_addr learned my addr 172.21.15.43:0/2921661773 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:58.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb941400640 1 --2- 172.21.15.43:0/2921661773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 0x7fb94415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032953066 0 0) 0x7fb944144ca0 con 0x7fb94415bbd0 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb924003520 con 0x7fb94415bbd0 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb941400640 1 -- 172.21.15.43:0/2921661773 >> v1:172.21.15.69:6789/0 conn(0x7fb9441448c0 legacy=0x7fb94406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002467511 0 0) 0x7fb94415c450 con 0x7fb9441448c0 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb941400640 1 -- 172.21.15.43:0/2921661773 >> v1:172.21.15.43:6789/0 conn(0x7fb94415bbd0 legacy=0x7fb944070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.754+0000 7fb941400640 1 -- 172.21.15.43:0/2921661773 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944071890 con 0x7fb94406dae0 2024-01-16T15:04:58.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.756+0000 7fb941400640 1 --2- 172.21.15.43:0/2921661773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 0x7fb94415b690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0097c0 tx=0x7fb92c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90c86134e3ac9860 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.756+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c030800 con 0x7fb94406dae0 2024-01-16T15:04:58.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.756+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c030960 con 0x7fb94406dae0 2024-01-16T15:04:58.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.756+0000 7fb940a00640 1 -- 172.21.15.43:0/2921661773 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c030cf0 con 0x7fb94406dae0 2024-01-16T15:04:58.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 -- 172.21.15.43:0/2921661773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 msgr2=0x7fb94415b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 --2- 172.21.15.43:0/2921661773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 0x7fb94415b690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0097c0 tx=0x7fb92c02b940 comp rx=0 tx=0).stop 2024-01-16T15:04:58.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 -- 172.21.15.43:0/2921661773 shutdown_connections 2024-01-16T15:04:58.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 --2- 172.21.15.43:0/2921661773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb94406dae0 0x7fb94415b690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 -- 172.21.15.43:0/2921661773 >> 172.21.15.43:0/2921661773 conn(0x7fb9440b0aa0 msgr2=0x7fb9440b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 -- 172.21.15.43:0/2921661773 shutdown_connections 2024-01-16T15:04:58.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.757+0000 7fb943200640 1 -- 172.21.15.43:0/2921661773 wait complete. 2024-01-16T15:04:58.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.759+0000 7fb943200640 1 Processor -- start 2024-01-16T15:04:58.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.759+0000 7fb943200640 1 -- start start 2024-01-16T15:04:58.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94406dae0 0x7fb944155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 0x7fb94415e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94415c6c0 con 0x7fb94415bbd0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944155cd0 con 0x7fb94406dae0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb943200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94415c930 con 0x7fb9441448c0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.760+0000 7fb942800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50250/0 (socket says 172.21.15.43:50250) 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 -- 172.21.15.43:0/2765388682 learned_addr learned my addr 172.21.15.43:0/2765388682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb941400640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 0x7fb94415e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb941e00640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94406dae0 0x7fb944155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 -- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 msgr2=0x7fb94415e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 0x7fb94415e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 -- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94406dae0 msgr2=0x7fb944155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94406dae0 0x7fb944155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 -- 172.21.15.43:0/2765388682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0093a0 con 0x7fb94415bbd0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb941400640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 0x7fb94415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb942800640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c3c0 tx=0x7fb93c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c005ce0 con 0x7fb94415bbd0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.761+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c005e40 con 0x7fb94415bbd0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.762+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9440724f0 con 0x7fb94415bbd0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.762+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c007310 con 0x7fb94415bbd0 2024-01-16T15:04:58.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.762+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944156490 con 0x7fb94415bbd0 2024-01-16T15:04:58.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.763+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94404e840 con 0x7fb94415bbd0 2024-01-16T15:04:58.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.764+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb93c007470 con 0x7fb94415bbd0 2024-01-16T15:04:58.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.765+0000 7fb93a000640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 0x7fb91c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.765+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb93c089bd0 con 0x7fb94415bbd0 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.765+0000 7fb941e00640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 0x7fb91c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.766+0000 7fb941e00640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 0x7fb91c06dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb9340012e0 tx=0x7fb934009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.769+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb93c08be60 con 0x7fb94415bbd0 2024-01-16T15:04:58.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.967+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb94415ca70 con 0x7fb94415bbd0 2024-01-16T15:04:58.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.968+0000 7fb93a000640 1 -- 172.21.15.43:0/2765388682 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb93c0170c0 con 0x7fb94415bbd0 2024-01-16T15:04:58.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 msgr2=0x7fb91c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 0x7fb91c06dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb9340012e0 tx=0x7fb934009040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 msgr2=0x7fb944155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c3c0 tx=0x7fb93c00c880 comp rx=0 tx=0).stop 2024-01-16T15:04:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 shutdown_connections 2024-01-16T15:04:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb91c06b860 0x7fb91c06dd20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94415bbd0 0x7fb944155790 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9441448c0 0x7fb94415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 --2- 172.21.15.43:0/2765388682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb94406dae0 0x7fb944155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 >> 172.21.15.43:0/2765388682 conn(0x7fb9440b0aa0 msgr2=0x7fb9440b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 shutdown_connections 2024-01-16T15:04:58.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:58.971+0000 7fb943200640 1 -- 172.21.15.43:0/2765388682 wait complete. 2024-01-16T15:04:58.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:04:59.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.221+0000 7f19a5e00640 1 Processor -- start 2024-01-16T15:04:59.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.221+0000 7f19a5e00640 1 -- start start 2024-01-16T15:04:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f19a5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19a009cb30 0x7f19a00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f19a5e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f19a00adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f19a5e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a015b730 con 0x7f19a015b350 2024-01-16T15:04:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f19a5e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a00ac740 con 0x7f19a009cb30 2024-01-16T15:04:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f19a5e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00ac460 con 0x7f19a00adb60 2024-01-16T15:04:59.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f199ea00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f19a015b350 0x7f19a00b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52134/0 (socket says 172.21.15.43:52134) 2024-01-16T15:04:59.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.223+0000 7f199ea00640 1 -- 172.21.15.43:0/2151169793 learned_addr learned my addr 172.21.15.43:0/2151169793 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:59.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f1995e00640 1 --2- 172.21.15.43:0/2151169793 >> v2:172.21.15.69:3300/0 conn(0x7f19a009cb30 0x7f19a00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199f400640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f19a00adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828578612 0 0) 0x7f1990001110 con 0x7f19a015b350 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003520 con 0x7f19a015b350 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031467760 0 0) 0x7f1980003520 con 0x7f19a015b350 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 msgr2=0x7f19a00adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f19a00adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 >> v2:172.21.15.69:3300/0 conn(0x7f19a009cb30 msgr2=0x7f19a00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 --2- 172.21.15.43:0/2151169793 >> v2:172.21.15.69:3300/0 conn(0x7f19a009cb30 0x7f19a00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00b2a50 con 0x7f19a015b350 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.224+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f1990001800 con 0x7f19a015b350 2024-01-16T15:04:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 >> v1:172.21.15.43:6789/0 conn(0x7f19a015b350 legacy=0x7f19a00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1980003970 0x7f1980003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19800047c0 0x7f1980004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f1980005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00ac740 con 0x7f1980003970 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00ac460 con 0x7f19800047c0 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198000b980 con 0x7f19a00adb60 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1980003970 0x7f1980003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f1995e00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19800047c0 0x7f1980004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199f400640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f1980005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 -- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 msgr2=0x7f1980005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19a00adb60 0x7f1980005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 -- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19800047c0 msgr2=0x7f1980004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19800047c0 0x7f1980004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.225+0000 7f199ea00640 1 -- 172.21.15.43:0/2151169793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1980003520 con 0x7f1980003970 2024-01-16T15:04:59.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.227+0000 7f199ea00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1980003970 0x7f1980003d90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f199000c210 tx=0x7f19800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c98abdce446ea5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.227+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199002cae0 con 0x7f1980003970 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.227+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199002cc60 con 0x7f1980003970 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.227+0000 7f199e000640 1 -- 172.21.15.43:0/2151169793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900058d0 con 0x7f1980003970 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 -- 172.21.15.43:0/2151169793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1980003970 msgr2=0x7f1980003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2151169793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1980003970 0x7f1980003d90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f199000c210 tx=0x7f19800057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f199f400640 1 -- 172.21.15.43:0/2151169793 reap_dead start 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 -- 172.21.15.43:0/2151169793 shutdown_connections 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 -- 172.21.15.43:0/2151169793 >> 172.21.15.43:0/2151169793 conn(0x7f19a00b0ba0 msgr2=0x7f19a00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 -- 172.21.15.43:0/2151169793 shutdown_connections 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.228+0000 7f19a5e00640 1 -- 172.21.15.43:0/2151169793 wait complete. 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.230+0000 7f19a5e00640 1 Processor -- start 2024-01-16T15:04:59.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.230+0000 7f19a5e00640 1 -- start start 2024-01-16T15:04:59.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1980003970 0x7f19a0148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19a01f6770 0x7f19a01f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00cfed0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a0099a00 con 0x7f19a01f6770 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f19a5e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a01f7250 con 0x7f1980003970 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50272/0 (socket says 172.21.15.43:50272) 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 -- 172.21.15.43:0/2200692472 learned_addr learned my addr 172.21.15.43:0/2200692472 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199ea00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1980003970 0x7f19a0148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f1995e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19a01f6770 0x7f19a01f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 -- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1980003970 msgr2=0x7f19a0148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1980003970 0x7f19a0148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 -- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19a01f6770 msgr2=0x7f19a01f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.231+0000 7f199f400640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19a01f6770 0x7f19a01f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f199f400640 1 -- 172.21.15.43:0/2200692472 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a01f7530 con 0x7f19a00cf4e0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f199f400640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f19a0067e90 tx=0x7f199800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998017400 con 0x7f19a00cf4e0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998011040 con 0x7f19a00cf4e0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19980177c0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a0099cc0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.232+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a01f3440 con 0x7f19a00cf4e0 2024-01-16T15:04:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.233+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a009a1e0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.234+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1998017920 con 0x7f19a00cf4e0 2024-01-16T15:04:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.235+0000 7f1997e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 0x7f197806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.235+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f199808aba0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.236+0000 7f199ea00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 0x7f197806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.236+0000 7f199ea00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 0x7f197806dd50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1990034000 tx=0x7f1990007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.240+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19a009a1e0 con 0x7f19a00cf4e0 2024-01-16T15:04:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.461+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f19a006e720 con 0x7f197806b890 2024-01-16T15:04:59.451 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:59.472+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:59.451 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:59.472+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:59.451 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:59.472+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:59.451 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:59.472+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:59.451 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:04:59.472+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.552+0000 7f1997e00640 1 -- 172.21.15.43:0/2200692472 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1990001230 con 0x7f197806b890 2024-01-16T15:04:59.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.555+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 msgr2=0x7f197806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.555+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 0x7f197806dd50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1990034000 tx=0x7f1990007000 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.555+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 msgr2=0x7f19a00cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.555+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f19a0067e90 tx=0x7f199800f4a0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.555+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 shutdown_connections 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197806b890 0x7f197806dd50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19a01f6770 0x7f19a01f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19a00cf4e0 0x7f19a00cf990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 --2- 172.21.15.43:0/2200692472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1980003970 0x7f19a0148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 >> 172.21.15.43:0/2200692472 conn(0x7f19a00b0ba0 msgr2=0x7f19a0071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 shutdown_connections 2024-01-16T15:04:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.556+0000 7f19a5e00640 1 -- 172.21.15.43:0/2200692472 wait complete. 2024-01-16T15:04:59.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:04:59.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.797+0000 7fe0b2800640 1 Processor -- start 2024-01-16T15:04:59.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.798+0000 7fe0b2800640 1 -- start start 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.799+0000 7fe0b2800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe0ac153ea0 0x7fe0ac0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.799+0000 7fe0b2800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac070850 0x7fe0ac071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.799+0000 7fe0b2800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0ce650 con 0x7fe0ac154280 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.799+0000 7fe0b2800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac154710 con 0x7fe0ac153ea0 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.799+0000 7fe0b2800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac15e0a0 con 0x7fe0ac070850 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b1400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe0ac153ea0 0x7fe0ac0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b0a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe0ac154280 0x7fe0ac15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52136/0 (socket says 172.21.15.43:52136) 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b0a00640 1 -- 172.21.15.43:0/3308064067 learned_addr learned my addr 172.21.15.43:0/3308064067 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b1e00640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac070850 0x7fe0ac071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b1400640 1 -- 172.21.15.43:0/3308064067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac070850 msgr2=0x7fe0ac071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b1400640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac070850 0x7fe0ac071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.800+0000 7fe0b1400640 1 -- 172.21.15.43:0/3308064067 >> v1:172.21.15.43:6789/0 conn(0x7fe0ac154280 legacy=0x7fe0ac15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.801+0000 7fe0b1400640 1 -- 172.21.15.43:0/3308064067 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac15e380 con 0x7fe0ac153ea0 2024-01-16T15:04:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.801+0000 7fe0abe00640 1 -- 172.21.15.43:0/3308064067 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752547932 0 0) 0x7fe0ac0ce650 con 0x7fe0ac154280 2024-01-16T15:04:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.802+0000 7fe0b1400640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0ad450 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0097a0 tx=0x7fe09c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74d8c8e2b5a55e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.802+0000 7fe0abe00640 1 -- 172.21.15.43:0/3308064067 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c02d040 con 0x7fe0ac153ea0 2024-01-16T15:04:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.802+0000 7fe0abe00640 1 -- 172.21.15.43:0/3308064067 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c031810 con 0x7fe0ac153ea0 2024-01-16T15:04:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.802+0000 7fe0abe00640 1 -- 172.21.15.43:0/3308064067 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c031ba0 con 0x7fe0ac153ea0 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 -- 172.21.15.43:0/3308064067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac153ea0 msgr2=0x7fe0ac0ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0ad450 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0097a0 tx=0x7fe09c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 -- 172.21.15.43:0/3308064067 shutdown_connections 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac070850 0x7fe0ac071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 --2- 172.21.15.43:0/3308064067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0ad450 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 -- 172.21.15.43:0/3308064067 >> 172.21.15.43:0/3308064067 conn(0x7fe0ac0cec10 msgr2=0x7fe0ac0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 -- 172.21.15.43:0/3308064067 shutdown_connections 2024-01-16T15:04:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.803+0000 7fe0b2800640 1 -- 172.21.15.43:0/3308064067 wait complete. 2024-01-16T15:04:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.805+0000 7fe0b2800640 1 Processor -- start 2024-01-16T15:04:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.805+0000 7fe0b2800640 1 -- start start 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.806+0000 7fe0b2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.806+0000 7fe0b2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b2800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac154280 0x7fe0ac0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b2800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac150be0 con 0x7fe0ac070850 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b2800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac157f60 con 0x7fe0ac153ea0 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b2800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac158240 con 0x7fe0ac154280 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50324/0 (socket says 172.21.15.43:50324) 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 -- 172.21.15.43:0/920592756 learned_addr learned my addr 172.21.15.43:0/920592756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 -- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac154280 msgr2=0x7fe0ac0b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac154280 0x7fe0ac0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 -- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac153ea0 msgr2=0x7fe0ac0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.807+0000 7fe0b1400640 1 -- 172.21.15.43:0/920592756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c009400 con 0x7fe0ac070850 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0b1400640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0098c0 tx=0x7fe09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c03ad60 con 0x7fe0ac070850 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c044400 con 0x7fe0ac070850 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0447b0 con 0x7fe0ac070850 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac158520 con 0x7fe0ac070850 2024-01-16T15:04:59.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.808+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac159d70 con 0x7fe0ac070850 2024-01-16T15:04:59.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.809+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac158a40 con 0x7fe0ac070850 2024-01-16T15:04:59.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.810+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 14) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe09c04d440 con 0x7fe0ac070850 2024-01-16T15:04:59.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.811+0000 7fe0a9600640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 0x7fe08406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.811+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe09c0acbc0 con 0x7fe0ac070850 2024-01-16T15:04:59.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.812+0000 7fe0b0a00640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 0x7fe08406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.812+0000 7fe0b0a00640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 0x7fe08406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe094007fd0 tx=0x7fe094007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:04:59.816+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0ac158a40 con 0x7fe0ac070850 2024-01-16T15:05:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.045+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0ac070330 con 0x7fe08406b860 2024-01-16T15:05:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.056+0000 7fe0a9600640 1 -- 172.21.15.43:0/920592756 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fe0940070d0 con 0x7fe08406b860 2024-01-16T15:05:00.035 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.058+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 msgr2=0x7fe08406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.058+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 0x7fe08406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe094007fd0 tx=0x7fe094007960 comp rx=0 tx=0).stop 2024-01-16T15:05:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.058+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 msgr2=0x7fe0ac1506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.058+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0098c0 tx=0x7fe09c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 shutdown_connections 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe08406b860 0x7fe08406dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0ac153ea0 0x7fe0ac0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe0ac154280 0x7fe0ac0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 --2- 172.21.15.43:0/920592756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0ac070850 0x7fe0ac1506a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 >> 172.21.15.43:0/920592756 conn(0x7fe0ac0cec10 msgr2=0x7fe0ac0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 shutdown_connections 2024-01-16T15:05:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:00.059+0000 7fe0b2800640 1 -- 172.21.15.43:0/920592756 wait complete. 2024-01-16T15:05:00.056 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:00.056 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:05:00.056 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:00.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:00.108 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:16.079 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:05:16.079 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:05:16.080 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.8846 s, 67.6 MB/s 2024-01-16T15:05:16.081 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:05:16.082 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:16.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:16.138 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:25.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:25.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:30.686 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:05:30.687 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:05:30.687 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.4912 s, 74.1 MB/s 2024-01-16T15:05:30.688 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:05:30.688 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:30.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.744 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:05:44.617 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:05:44.617 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.8059 s, 77.8 MB/s 2024-01-16T15:05:44.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:05:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.867+0000 7f8935600640 1 Processor -- start 2024-01-16T15:05:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.868+0000 7f8935600640 1 -- start start 2024-01-16T15:05:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.869+0000 7f8935600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89300abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f8935600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893015b730 con 0x7f893009cb30 2024-01-16T15:05:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f8935600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300ac740 con 0x7f893015b350 2024-01-16T15:05:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f8935600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89300ac460 con 0x7f89300ae270 2024-01-16T15:05:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f892e000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f893009cb30 0x7f89300adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39204/0 (socket says 172.21.15.43:39204) 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f892e000640 1 -- 172.21.15.43:0/2321686647 learned_addr learned my addr 172.21.15.43:0/2321686647 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f892f400640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89300abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122902674 0 0) 0x7f8918001110 con 0x7f893009cb30 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.870+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910003520 con 0x7f893009cb30 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183666923 0 0) 0x7f8910003520 con 0x7f893009cb30 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 msgr2=0x7f89300abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89300abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 >> v1:172.21.15.69:6789/0 conn(0x7f893015b350 legacy=0x7f89300b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300b2a50 con 0x7f893009cb30 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8918001800 con 0x7f893009cb30 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 >> v1:172.21.15.43:6789/0 conn(0x7f893009cb30 legacy=0x7f89300adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89100039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f89100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8910004d00 0x7f8910005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300ac460 con 0x7f8910003f20 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891000ba60 con 0x7f8910004d00 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891000bc20 con 0x7f89300ae270 2024-01-16T15:05:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892e000640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f89100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.871+0000 7f892ea00640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89100039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892f400640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8910004d00 0x7f8910005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892e000640 1 -- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 msgr2=0x7f89100039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892e000640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89300ae270 0x7f89100039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892e000640 1 -- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8910004d00 msgr2=0x7f8910005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892e000640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8910004d00 0x7f8910005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.872+0000 7f892e000640 1 -- 172.21.15.43:0/2321686647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910003520 con 0x7f8910003f20 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.873+0000 7f892e000640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f89100047c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f891800c240 tx=0x7f89100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fed18b16e67035 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.873+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918005ad0 con 0x7f8910003f20 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.873+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918005c50 con 0x7f8910003f20 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.873+0000 7f892d600640 1 -- 172.21.15.43:0/2321686647 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891802c8b0 con 0x7f8910003f20 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 -- 172.21.15.43:0/2321686647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 msgr2=0x7f89100047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 --2- 172.21.15.43:0/2321686647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f89100047c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f891800c240 tx=0x7f89100057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f892f400640 1 -- 172.21.15.43:0/2321686647 reap_dead start 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 -- 172.21.15.43:0/2321686647 shutdown_connections 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 -- 172.21.15.43:0/2321686647 >> 172.21.15.43:0/2321686647 conn(0x7f89300b0ba0 msgr2=0x7f89300b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 -- 172.21.15.43:0/2321686647 shutdown_connections 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.874+0000 7f8935600640 1 -- 172.21.15.43:0/2321686647 wait complete. 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.876+0000 7f8935600640 1 Processor -- start 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.876+0000 7f8935600640 1 -- start start 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89301e7e80 0x7f89301e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f893015d7d0 0x7f893015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89301e8840 con 0x7f8910003f20 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89301f77b0 con 0x7f893015d7d0 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f8935600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89301f6ba0 con 0x7f89301e7e80 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f892ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f892ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59504/0 (socket says 172.21.15.43:59504) 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.877+0000 7f892ea00640 1 -- 172.21.15.43:0/3550014076 learned_addr learned my addr 172.21.15.43:0/3550014076 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892e000640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89301e7e80 0x7f89301e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892f400640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f893015d7d0 0x7f893015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 -- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89301e7e80 msgr2=0x7f89301e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89301e7e80 0x7f89301e8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 -- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f893015d7d0 msgr2=0x7f893015dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f893015d7d0 0x7f893015dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 -- 172.21.15.43:0/3550014076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89301f6e80 con 0x7f8910003f20 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f892ea00640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f892000e580 tx=0x7f892000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89200052e0 con 0x7f8910003f20 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920005440 con 0x7f8910003f20 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920003810 con 0x7f8910003f20 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.878+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89301f7160 con 0x7f8910003f20 2024-01-16T15:05:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.879+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893015e190 con 0x7f8910003f20 2024-01-16T15:05:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.880+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893006e700 con 0x7f8910003f20 2024-01-16T15:05:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.880+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f8920003970 con 0x7f8910003f20 2024-01-16T15:05:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.881+0000 7f8926a00640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 0x7f890806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.881+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f892008a3b0 con 0x7f8910003f20 2024-01-16T15:05:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.882+0000 7f892e000640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 0x7f890806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.882+0000 7f892e000640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 0x7f890806ddb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8918034000 tx=0x7f8918009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:44.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:44.886+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f893006e700 con 0x7f8910003f20 2024-01-16T15:05:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.081+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f89300ac740 con 0x7f8910003f20 2024-01-16T15:05:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.089+0000 7f8926a00640 1 -- 172.21.15.43:0/3550014076 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7f89300ac740 con 0x7f8910003f20 2024-01-16T15:05:45.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.091+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 msgr2=0x7f890806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.091+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 0x7f890806ddb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8918034000 tx=0x7f8918009030 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.091+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 msgr2=0x7f893013c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.091+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f892000e580 tx=0x7f892000ea40 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 shutdown_connections 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f890806b8f0 0x7f890806ddb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f893015d7d0 0x7f893015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89301e7e80 0x7f89301e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 --2- 172.21.15.43:0/3550014076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8910003f20 0x7f893013c4a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 >> 172.21.15.43:0/3550014076 conn(0x7f89300b0ba0 msgr2=0x7f89300b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 shutdown_connections 2024-01-16T15:05:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.092+0000 7f8935600640 1 -- 172.21.15.43:0/3550014076 wait complete. 2024-01-16T15:05:45.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:05:45.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.332+0000 7faf9e200640 1 Processor -- start 2024-01-16T15:05:45.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.332+0000 7faf9e200640 1 -- start start 2024-01-16T15:05:45.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf9e200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf9806dc30 0x7faf980ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf9e200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 0x7faf98070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf9e200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9806d7a0 con 0x7faf980acd70 2024-01-16T15:05:45.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf9e200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9815a5b0 con 0x7faf9806dc30 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf9e200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf980ae9d0 con 0x7faf98070490 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.333+0000 7faf96a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf9806dc30 0x7faf980ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf9806dc30 0x7faf980ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59510/0 (socket says 172.21.15.43:59510) 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 -- 172.21.15.43:0/3269624616 learned_addr learned my addr 172.21.15.43:0/3269624616 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf97400640 1 --1- 172.21.15.43:0/3269624616 >> v1:172.21.15.43:6789/0 conn(0x7faf980acd70 0x7faf980ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39208/0 (socket says 172.21.15.43:39208) 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf97e00640 1 --2- 172.21.15.43:0/3269624616 >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 0x7faf98070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 -- 172.21.15.43:0/3269624616 >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 msgr2=0x7faf98070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:45.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 --2- 172.21.15.43:0/3269624616 >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 0x7faf98070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 -- 172.21.15.43:0/3269624616 >> v1:172.21.15.43:6789/0 conn(0x7faf980acd70 legacy=0x7faf980ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96a00640 1 -- 172.21.15.43:0/3269624616 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9806ce70 con 0x7faf9806dc30 2024-01-16T15:05:45.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf96000640 1 -- 172.21.15.43:0/3269624616 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886175116 0 0) 0x7faf9806d7a0 con 0x7faf980acd70 2024-01-16T15:05:45.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.334+0000 7faf97e00640 1 --2- 172.21.15.43:0/3269624616 >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 0x7faf98070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.335+0000 7faf96a00640 1 --2- 172.21.15.43:0/3269624616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf980ae490 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7faf98067e70 tx=0x7faf8002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8982d710b3e1f4dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.335+0000 7faf96000640 1 -- 172.21.15.43:0/3269624616 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80030870 con 0x7faf9806dc30 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf96000640 1 -- 172.21.15.43:0/3269624616 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf800309d0 con 0x7faf9806dc30 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf96000640 1 -- 172.21.15.43:0/3269624616 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80030d60 con 0x7faf9806dc30 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 -- 172.21.15.43:0/3269624616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 msgr2=0x7faf980ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 --2- 172.21.15.43:0/3269624616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf980ae490 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7faf98067e70 tx=0x7faf8002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 -- 172.21.15.43:0/3269624616 shutdown_connections 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 --2- 172.21.15.43:0/3269624616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf980ae490 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 --2- 172.21.15.43:0/3269624616 >> v2:172.21.15.69:3300/0 conn(0x7faf98070490 0x7faf98070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 -- 172.21.15.43:0/3269624616 >> 172.21.15.43:0/3269624616 conn(0x7faf980b0ba0 msgr2=0x7faf980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 -- 172.21.15.43:0/3269624616 shutdown_connections 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.336+0000 7faf9e200640 1 -- 172.21.15.43:0/3269624616 wait complete. 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.338+0000 7faf9e200640 1 Processor -- start 2024-01-16T15:05:45.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.338+0000 7faf9e200640 1 -- start start 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf98070490 0x7faf980afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf980acd70 0x7faf980b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9813c440 con 0x7faf9806dc30 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9813b830 con 0x7faf980acd70 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf9e200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9813bb10 con 0x7faf98070490 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf97400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.339+0000 7faf97400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59516/0 (socket says 172.21.15.43:59516) 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 -- 172.21.15.43:0/4119477411 learned_addr learned my addr 172.21.15.43:0/4119477411 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf96a00640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf98070490 0x7faf980afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97e00640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf980acd70 0x7faf980b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 -- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf98070490 msgr2=0x7faf980afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf98070490 0x7faf980afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 -- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf980acd70 msgr2=0x7faf980b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf980acd70 0x7faf980b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 -- 172.21.15.43:0/4119477411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800094d0 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf97400640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c890 tx=0x7faf8800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.340+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800a640 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.341+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8800a7c0 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.341+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800ab70 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.341+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9813bdf0 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.341+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9813de00 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.342+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9809d200 con 0x7faf9806dc30 2024-01-16T15:05:45.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.343+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7faf8800acd0 con 0x7faf9806dc30 2024-01-16T15:05:45.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.344+0000 7faf8f400640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 0x7faf7006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.344+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faf88089cb0 con 0x7faf9806dc30 2024-01-16T15:05:45.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.344+0000 7faf96a00640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 0x7faf7006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.345+0000 7faf96a00640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 0x7faf7006dde0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faf80036000 tx=0x7faf8003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.348+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf88054180 con 0x7faf9806dc30 2024-01-16T15:05:45.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.541+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7faf980ad810 con 0x7faf9806dc30 2024-01-16T15:05:45.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.548+0000 7faf8f400640 1 -- 172.21.15.43:0/4119477411 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7faf880180c0 con 0x7faf9806dc30 2024-01-16T15:05:45.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 msgr2=0x7faf7006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 0x7faf7006dde0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faf80036000 tx=0x7faf8003b040 comp rx=0 tx=0).stop 2024-01-16T15:05:45.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 msgr2=0x7faf9813d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c890 tx=0x7faf8800cd50 comp rx=0 tx=0).stop 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 shutdown_connections 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf7006b920 0x7faf7006dde0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf980acd70 0x7faf980b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf98070490 0x7faf980afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 --2- 172.21.15.43:0/4119477411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf9806dc30 0x7faf9813d8c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 >> 172.21.15.43:0/4119477411 conn(0x7faf980b0ba0 msgr2=0x7faf9815c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:45.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 shutdown_connections 2024-01-16T15:05:45.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.551+0000 7faf9e200640 1 -- 172.21.15.43:0/4119477411 wait complete. 2024-01-16T15:05:45.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:05:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.791+0000 7f13c7200640 1 Processor -- start 2024-01-16T15:05:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.792+0000 7f13c7200640 1 -- start start 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.793+0000 7f13c7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13c015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.793+0000 7f13c7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13c015f0a0 0x7f13c00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.793+0000 7f13c7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c015f930 con 0x7f13c00b35a0 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.793+0000 7f13c7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c00b43b0 con 0x7f13c015f0a0 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.793+0000 7f13c7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015f530 con 0x7f13c0144770 2024-01-16T15:05:45.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c5400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f13c00b35a0 0x7f13c006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39214/0 (socket says 172.21.15.43:39214) 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c5400640 1 -- 172.21.15.43:0/1364628675 learned_addr learned my addr 172.21.15.43:0/1364628675 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c6800640 1 --2- 172.21.15.43:0/1364628675 >> v2:172.21.15.69:3300/0 conn(0x7f13c015f0a0 0x7f13c00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c5e00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13c015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83826211 0 0) 0x7f13c015f930 con 0x7f13c00b35a0 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.794+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0003520 con 0x7f13c00b35a0 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180669584 0 0) 0x7f13a0003520 con 0x7f13c00b35a0 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 msgr2=0x7f13c015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13c015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 >> v2:172.21.15.69:3300/0 conn(0x7f13c015f0a0 msgr2=0x7f13c00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 --2- 172.21.15.43:0/1364628675 >> v2:172.21.15.69:3300/0 conn(0x7f13c015f0a0 0x7f13c00ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00ce9f0 con 0x7f13c00b35a0 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f13a8001670 con 0x7f13c00b35a0 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 >> v1:172.21.15.43:6789/0 conn(0x7f13c00b35a0 legacy=0x7f13c006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0003f60 0x7f13a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004d00 0x7f13a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00b43b0 con 0x7f13a0003f60 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015f530 con 0x7f13a0004d00 2024-01-16T15:05:45.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000b9b0 con 0x7f13c0144770 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c5400640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0003f60 0x7f13a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c5e00640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.795+0000 7f13c6800640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004d00 0x7f13a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.796+0000 7f13c5400640 1 -- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 msgr2=0x7f13a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.796+0000 7f13c5400640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c0144770 0x7f13a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.796+0000 7f13c5400640 1 -- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004d00 msgr2=0x7f13a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.796+0000 7f13c5400640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0004d00 0x7f13a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.796+0000 7f13c5400640 1 -- 172.21.15.43:0/1364628675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0003520 con 0x7f13a0003f60 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.797+0000 7f13c5400640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0003f60 0x7f13a00047c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c240 tx=0x7f13a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce58f68aabe387af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.797+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8005ad0 con 0x7f13a0003f60 2024-01-16T15:05:45.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.797+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8005c30 con 0x7f13a0003f60 2024-01-16T15:05:45.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.797+0000 7f13c4a00640 1 -- 172.21.15.43:0/1364628675 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a802caf0 con 0x7f13a0003f60 2024-01-16T15:05:45.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 -- 172.21.15.43:0/1364628675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0003f60 msgr2=0x7f13a00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 --2- 172.21.15.43:0/1364628675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13a0003f60 0x7f13a00047c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c240 tx=0x7f13a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c6800640 1 -- 172.21.15.43:0/1364628675 reap_dead start 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 -- 172.21.15.43:0/1364628675 shutdown_connections 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 -- 172.21.15.43:0/1364628675 >> 172.21.15.43:0/1364628675 conn(0x7f13c00cec10 msgr2=0x7f13c00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 -- 172.21.15.43:0/1364628675 shutdown_connections 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.798+0000 7f13c7200640 1 -- 172.21.15.43:0/1364628675 wait complete. 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.800+0000 7f13c7200640 1 Processor -- start 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.800+0000 7f13c7200640 1 -- start start 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 0x7f13c015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006cc90 0x7f13c006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c006d680 con 0x7f13c013f500 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c012d720 con 0x7f13a0003f60 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c01302b0 con 0x7f13c006cc90 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59542/0 (socket says 172.21.15.43:59542) 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.801+0000 7f13c5400640 1 -- 172.21.15.43:0/2300989602 learned_addr learned my addr 172.21.15.43:0/2300989602 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c6800640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006cc90 0x7f13c006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5e00640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 0x7f13c015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 -- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006cc90 msgr2=0x7f13c006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006cc90 0x7f13c006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 -- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 msgr2=0x7f13c015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 0x7f13c015a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 -- 172.21.15.43:0/2300989602 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c012fcf0 con 0x7f13c013f500 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5e00640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 0x7f13c015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.802+0000 7f13c5400640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c240 tx=0x7f13a8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.803+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a803c040 con 0x7f13c013f500 2024-01-16T15:05:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.803+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a802dd50 con 0x7f13c013f500 2024-01-16T15:05:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.803+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8007290 con 0x7f13c013f500 2024-01-16T15:05:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.803+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c012ffd0 con 0x7f13c013f500 2024-01-16T15:05:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.803+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c015ad50 con 0x7f13c013f500 2024-01-16T15:05:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.804+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c004e840 con 0x7f13c013f500 2024-01-16T15:05:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.805+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f13a8032070 con 0x7f13c013f500 2024-01-16T15:05:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.806+0000 7f13be000640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 0x7f139806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.806+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f13a80ac2b0 con 0x7f13c013f500 2024-01-16T15:05:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.807+0000 7f13c5e00640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 0x7f139806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.807+0000 7f13c5e00640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 0x7f139806dde0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13c015de40 tx=0x7f13b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:45.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:45.811+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13a8076780 con 0x7f13c013f500 2024-01-16T15:05:46.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.036+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f13c00b40d0 con 0x7f139806b920 2024-01-16T15:05:46.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.074+0000 7f13be000640 1 -- 172.21.15.43:0/2300989602 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13b0013930 con 0x7f139806b920 2024-01-16T15:05:46.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 msgr2=0x7f139806dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 0x7f139806dde0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13c015de40 tx=0x7f13b0008040 comp rx=0 tx=0).stop 2024-01-16T15:05:46.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 msgr2=0x7f13c012f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c240 tx=0x7f13a8001a00 comp rx=0 tx=0).stop 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 shutdown_connections 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f139806b920 0x7f139806dde0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13c006cc90 0x7f13c006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13c013f500 0x7f13c012f340 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 --2- 172.21.15.43:0/2300989602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13a0003f60 0x7f13c015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.076+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 >> 172.21.15.43:0/2300989602 conn(0x7f13c00cec10 msgr2=0x7f13c0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.077+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 shutdown_connections 2024-01-16T15:05:46.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.077+0000 7f13c7200640 1 -- 172.21.15.43:0/2300989602 wait complete. 2024-01-16T15:05:46.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:46.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.323+0000 7f5ab4a00640 1 Processor -- start 2024-01-16T15:05:46.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.324+0000 7f5ab4a00640 1 -- start start 2024-01-16T15:05:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.325+0000 7f5ab4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 0x7f5ab015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.325+0000 7f5ab4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.325+0000 7f5ab4a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab015f930 con 0x7f5ab00b35a0 2024-01-16T15:05:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.325+0000 7f5ab4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00b43b0 con 0x7f5ab0144770 2024-01-16T15:05:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.325+0000 7f5ab4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab015f530 con 0x7f5ab015f0a0 2024-01-16T15:05:46.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.326+0000 7f5aae800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 0x7f5ab015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.326+0000 7f5aaf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.326+0000 7f5aaf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37132/0 (socket says 172.21.15.43:37132) 2024-01-16T15:05:46.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.326+0000 7f5aaf200640 1 -- 172.21.15.43:0/1452531674 learned_addr learned my addr 172.21.15.43:0/1452531674 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:46.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.326+0000 7f5aae800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 0x7f5ab015eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:42826/0 (socket says 172.21.15.43:42826) 2024-01-16T15:05:46.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.327+0000 7f5aaf200640 1 -- 172.21.15.43:0/1452531674 >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 msgr2=0x7f5ab015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:46.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.327+0000 7f5aaf200640 1 --2- 172.21.15.43:0/1452531674 >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 0x7f5ab015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.327+0000 7f5aaf200640 1 -- 172.21.15.43:0/1452531674 >> v1:172.21.15.69:6789/0 conn(0x7f5ab00b35a0 legacy=0x7f5ab006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.327+0000 7f5aaf200640 1 -- 172.21.15.43:0/1452531674 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00ce9f0 con 0x7f5ab015f0a0 2024-01-16T15:05:46.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.328+0000 7f5aaf200640 1 --2- 172.21.15.43:0/1452531674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009e50 tx=0x7f5aa802bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd56f20b7932484 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:46.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.329+0000 7f5aad400640 1 -- 172.21.15.43:0/1452531674 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa802caf0 con 0x7f5ab015f0a0 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.329+0000 7f5aad400640 1 -- 172.21.15.43:0/1452531674 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa802cc50 con 0x7f5ab015f0a0 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.329+0000 7f5aad400640 1 -- 172.21.15.43:0/1452531674 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80343f0 con 0x7f5ab015f0a0 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 -- 172.21.15.43:0/1452531674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 msgr2=0x7f5ab00ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/1452531674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009e50 tx=0x7f5aa802bf50 comp rx=0 tx=0).stop 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 -- 172.21.15.43:0/1452531674 shutdown_connections 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/1452531674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00ce4b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/1452531674 >> v2:172.21.15.69:3300/0 conn(0x7f5ab0144770 0x7f5ab015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 -- 172.21.15.43:0/1452531674 >> 172.21.15.43:0/1452531674 conn(0x7f5ab00cec10 msgr2=0x7f5ab00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 -- 172.21.15.43:0/1452531674 shutdown_connections 2024-01-16T15:05:46.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.330+0000 7f5ab4a00640 1 -- 172.21.15.43:0/1452531674 wait complete. 2024-01-16T15:05:46.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.332+0000 7f5ab4a00640 1 Processor -- start 2024-01-16T15:05:46.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.332+0000 7f5ab4a00640 1 -- start start 2024-01-16T15:05:46.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab00b35a0 0x7f5ab01f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab015c910 con 0x7f5ab0144770 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0139e90 con 0x7f5ab00b35a0 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5ab4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab01fb670 con 0x7f5ab015f0a0 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.333+0000 7f5aade00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59556/0 (socket says 172.21.15.43:59556) 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 -- 172.21.15.43:0/2570926924 learned_addr learned my addr 172.21.15.43:0/2570926924 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aae800640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab00b35a0 0x7f5ab01f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aaf200640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 -- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 msgr2=0x7f5ab00b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 -- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab00b35a0 msgr2=0x7f5ab01f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab00b35a0 0x7f5ab01f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 -- 172.21.15.43:0/2570926924 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8009b30 con 0x7f5ab0144770 2024-01-16T15:05:46.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.334+0000 7f5aade00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c400 tx=0x7f5a9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:46.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.335+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980052d0 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.335+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98005450 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.335+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98007260 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.335+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab01fb390 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.335+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab01f7580 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.336+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab01f7100 con 0x7f5ab0144770 2024-01-16T15:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.337+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f5a980073c0 con 0x7f5ab0144770 2024-01-16T15:05:46.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.338+0000 7f5aa6a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 0x7f5a8806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:46.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.338+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5a98089b00 con 0x7f5ab0144770 2024-01-16T15:05:46.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.338+0000 7f5aae800640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 0x7f5a8806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:46.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.339+0000 7f5aae800640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 0x7f5a8806ddb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009ad0 tx=0x7f5aa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:46.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.342+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ab01f7100 con 0x7f5ab0144770 2024-01-16T15:05:46.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.565+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab00b40d0 con 0x7f5a8806b8f0 2024-01-16T15:05:46.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.574+0000 7f5aa6a00640 1 -- 172.21.15.43:0/2570926924 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f5aa00119a0 con 0x7f5a8806b8f0 2024-01-16T15:05:46.552 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 msgr2=0x7f5a8806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 0x7f5a8806ddb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009ad0 tx=0x7f5aa0008040 comp rx=0 tx=0).stop 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 msgr2=0x7f5ab00afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c400 tx=0x7f5a9800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 shutdown_connections 2024-01-16T15:05:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8806b8f0 0x7f5a8806ddb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab015f0a0 0x7f5ab00b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab0144770 0x7f5ab00afdc0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 --2- 172.21.15.43:0/2570926924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab00b35a0 0x7f5ab01f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 >> 172.21.15.43:0/2570926924 conn(0x7f5ab00cec10 msgr2=0x7f5ab00cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.576+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 shutdown_connections 2024-01-16T15:05:46.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:46.577+0000 7f5ab4a00640 1 -- 172.21.15.43:0/2570926924 wait complete. 2024-01-16T15:05:46.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:46.573 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab) 2024-01-16T15:05:46.639 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:46.639 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:05:46.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:05:46.640 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:05:48.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:48.642 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab) 2024-01-16T15:05:48.710 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:48.710 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:05:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:05:48.711 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:05:50.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:50.713 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab) 2024-01-16T15:05:50.780 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/a29dc5ec-2ed6-4e9c-8e50-d437ce912aab 2024-01-16T15:05:50.780 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:05:50.780 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:05:50.782 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:05:50.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53191 2024-01-16T15:05:51.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.040+0000 7f63cd600640 1 Processor -- start 2024-01-16T15:05:51.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.040+0000 7f63cd600640 1 -- start start 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63cd600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63c80c8930 0x7f63c80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63cd600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80c6bf0 con 0x7f63c80d1d90 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63cd600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80cedc0 con 0x7f63c815d3b0 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63cd600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c80c6e40 con 0x7f63c80c8930 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63c7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f63c80d1d90 0x7f63c80b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39216/0 (socket says 172.21.15.43:39216) 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63c7400640 1 -- 172.21.15.43:0/3391079840 learned_addr learned my addr 172.21.15.43:0/3391079840 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63c7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f63c815d3b0 0x7f63c80d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:54564/0 (socket says 172.21.15.43:54564) 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.042+0000 7f63ccc00640 1 --2- 172.21.15.43:0/3391079840 >> v2:172.21.15.69:3300/0 conn(0x7f63c80c8930 0x7f63c80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459325887 0 0) 0x7f63c80c6bf0 con 0x7f63c80d1d90 2024-01-16T15:05:51.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a8003520 con 0x7f63c80d1d90 2024-01-16T15:05:51.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63ccc00640 1 -- 172.21.15.43:0/3391079840 >> v1:172.21.15.69:6789/0 conn(0x7f63c815d3b0 legacy=0x7f63c80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63ccc00640 1 -- 172.21.15.43:0/3391079840 >> v1:172.21.15.43:6789/0 conn(0x7f63c80d1d90 legacy=0x7f63c80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141730751 0 0) 0x7f63a8003520 con 0x7f63c80d1d90 2024-01-16T15:05:51.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.043+0000 7f63ccc00640 1 -- 172.21.15.43:0/3391079840 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80c7120 con 0x7f63c80c8930 2024-01-16T15:05:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.045+0000 7f63ccc00640 1 --2- 172.21.15.43:0/3391079840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80c8930 0x7f63c80ce6c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f63c00097a0 tx=0x7f63c002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88121cf99a1263bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.045+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80019f0 con 0x7f63c80c8930 2024-01-16T15:05:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.045+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c002e050 con 0x7f63c80c8930 2024-01-16T15:05:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.045+0000 7f63c6a00640 1 -- 172.21.15.43:0/3391079840 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0030730 con 0x7f63c80c8930 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 -- 172.21.15.43:0/3391079840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80c8930 msgr2=0x7f63c80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 --2- 172.21.15.43:0/3391079840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80c8930 0x7f63c80ce6c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f63c00097a0 tx=0x7f63c002b790 comp rx=0 tx=0).stop 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 -- 172.21.15.43:0/3391079840 shutdown_connections 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 --2- 172.21.15.43:0/3391079840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80c8930 0x7f63c80ce6c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 -- 172.21.15.43:0/3391079840 >> 172.21.15.43:0/3391079840 conn(0x7f63c80d30a0 msgr2=0x7f63c80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 -- 172.21.15.43:0/3391079840 shutdown_connections 2024-01-16T15:05:51.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.046+0000 7f63cd600640 1 -- 172.21.15.43:0/3391079840 wait complete. 2024-01-16T15:05:51.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.048+0000 7f63cd600640 1 Processor -- start 2024-01-16T15:05:51.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.048+0000 7f63cd600640 1 -- start start 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 0x7f63c814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 0x7f63c809d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cb500 con 0x7f63c815d3b0 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cb770 con 0x7f63c80d1d90 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.049+0000 7f63cd600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cba50 con 0x7f63c80c8930 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59572/0 (socket says 172.21.15.43:59572) 2024-01-16T15:05:51.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 -- 172.21.15.43:0/101874798 learned_addr learned my addr 172.21.15.43:0/101874798 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63c7e00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 0x7f63c814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63c7400640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 0x7f63c809d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 -- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 msgr2=0x7f63c814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 0x7f63c814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 -- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 msgr2=0x7f63c809d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 0x7f63c809d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 -- 172.21.15.43:0/101874798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0009400 con 0x7f63c815d3b0 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63c7e00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 0x7f63c814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63c7400640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 0x7f63c809d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:51.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.050+0000 7f63ccc00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f63c00097a0 tx=0x7f63c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.051+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004230 con 0x7f63c815d3b0 2024-01-16T15:05:51.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.051+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0004390 con 0x7f63c815d3b0 2024-01-16T15:05:51.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.051+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c003d8d0 con 0x7f63c815d3b0 2024-01-16T15:05:51.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.051+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c80cbd30 con 0x7f63c815d3b0 2024-01-16T15:05:51.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.051+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c814f9b0 con 0x7f63c815d3b0 2024-01-16T15:05:51.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.052+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c80cc250 con 0x7f63c815d3b0 2024-01-16T15:05:51.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.053+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f63c0039040 con 0x7f63c815d3b0 2024-01-16T15:05:51.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.055+0000 7f63bfe00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 0x7f63a006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.055+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f63c00ac580 con 0x7f63c815d3b0 2024-01-16T15:05:51.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.055+0000 7f63c7e00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 0x7f63a006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.056+0000 7f63c7e00640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 0x7f63a006ddb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000f60 tx=0x7f63b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.060+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63c00769d0 con 0x7f63c815d3b0 2024-01-16T15:05:51.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.288+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53191", "target": ["mon-mgr", ""]}) v1 -- 0x7f63c815d790 con 0x7f63a006b8f0 2024-01-16T15:05:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.440+0000 7f63bfe00640 1 -- 172.21.15.43:0/101874798 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f63b0001310 con 0x7f63a006b8f0 2024-01-16T15:05:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 msgr2=0x7f63a006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 0x7f63a006ddb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000f60 tx=0x7f63b0009040 comp rx=0 tx=0).stop 2024-01-16T15:05:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 msgr2=0x7f63c80b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f63c00097a0 tx=0x7f63c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 shutdown_connections 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f63a006b8f0 0x7f63a006ddb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c80d1d90 0x7f63c809d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c815d3b0 0x7f63c80b6da0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 --2- 172.21.15.43:0/101874798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c80c8930 0x7f63c814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 >> 172.21.15.43:0/101874798 conn(0x7f63c80d30a0 msgr2=0x7f63c80d1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 shutdown_connections 2024-01-16T15:05:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.443+0000 7f63cd600640 1 -- 172.21.15.43:0/101874798 wait complete. 2024-01-16T15:05:51.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.447+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.447+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.447+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.447+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.426 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.447+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53192 2024-01-16T15:05:51.455 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.477+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.455 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.477+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.455 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.477+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.456 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.477+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.456 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:05:51.477+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.666+0000 7fc7c5600640 1 Processor -- start 2024-01-16T15:05:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.666+0000 7fc7c5600640 1 -- start start 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7c5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7c0072280 0x7fc7c006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7c5600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c0071780 con 0x7fc7c006c900 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7c5600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c00b2320 con 0x7fc7c0072280 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7c5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c006cd90 con 0x7fc7c0160b10 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.668+0000 7fc7bf400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc7c006c900 0x7fc7c0160fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39224/0 (socket says 172.21.15.43:39224) 2024-01-16T15:05:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7bf400640 1 -- 172.21.15.43:0/1677475336 learned_addr learned my addr 172.21.15.43:0/1677475336 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7b7e00640 1 --2- 172.21.15.43:0/1677475336 >> v2:172.21.15.69:3300/0 conn(0x7fc7c0072280 0x7fc7c006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7bea00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584374275 0 0) 0x7fc7c0071780 con 0x7fc7c006c900 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0003520 con 0x7fc7c006c900 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79794134 0 0) 0x7fc7a0003520 con 0x7fc7c006c900 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 msgr2=0x7fc7c006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7c006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 >> v2:172.21.15.69:3300/0 conn(0x7fc7c0072280 msgr2=0x7fc7c006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 --2- 172.21.15.43:0/1677475336 >> v2:172.21.15.69:3300/0 conn(0x7fc7c0072280 0x7fc7c006c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00711c0 con 0x7fc7c006c900 2024-01-16T15:05:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.669+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc7b80019f0 con 0x7fc7c006c900 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 >> v1:172.21.15.43:6789/0 conn(0x7fc7c006c900 legacy=0x7fc7c0160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7a0003970 0x7fc7a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a00047c0 0x7fc7a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c00b2320 con 0x7fc7a0003970 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c006cd90 con 0x7fc7a00047c0 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a000b980 con 0x7fc7c0160b10 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bea00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7a0003970 0x7fc7a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7b7e00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a00047c0 0x7fc7a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bf400640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bea00640 1 -- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 msgr2=0x7fc7a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bea00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7c0160b10 0x7fc7a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bea00640 1 -- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a00047c0 msgr2=0x7fc7a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.670+0000 7fc7bea00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a00047c0 0x7fc7a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.671+0000 7fc7bea00640 1 -- 172.21.15.43:0/1677475336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0003520 con 0x7fc7a0003970 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.672+0000 7fc7bea00640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7a0003970 0x7fc7a0003d90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009af0 tx=0x7fc7a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa6552331a6d849 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.672+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b002c6b0 con 0x7fc7a0003970 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.672+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b002c810 con 0x7fc7a0003970 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.672+0000 7fc7be000640 1 -- 172.21.15.43:0/1677475336 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b002cba0 con 0x7fc7a0003970 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 -- 172.21.15.43:0/1677475336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7a0003970 msgr2=0x7fc7a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 --2- 172.21.15.43:0/1677475336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7a0003970 0x7fc7a0003d90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009af0 tx=0x7fc7a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7bf400640 1 -- 172.21.15.43:0/1677475336 reap_dead start 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 -- 172.21.15.43:0/1677475336 shutdown_connections 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 -- 172.21.15.43:0/1677475336 >> 172.21.15.43:0/1677475336 conn(0x7fc7c00b0ba0 msgr2=0x7fc7c00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 -- 172.21.15.43:0/1677475336 shutdown_connections 2024-01-16T15:05:51.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.673+0000 7fc7c5600640 1 -- 172.21.15.43:0/1677475336 wait complete. 2024-01-16T15:05:51.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.675+0000 7fc7c5600640 1 Processor -- start 2024-01-16T15:05:51.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.675+0000 7fc7c5600640 1 -- start start 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0003970 0x7fc7c0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7c00b3010 0x7fc7c00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01551e0 con 0x7fc7c00c82d0 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c00b39d0 con 0x7fc7c00b3010 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7c5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c00c98a0 con 0x7fc7a0003970 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7bf400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7bf400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59620/0 (socket says 172.21.15.43:59620) 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.676+0000 7fc7bf400640 1 -- 172.21.15.43:0/3427946056 learned_addr learned my addr 172.21.15.43:0/3427946056 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bea00640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0003970 0x7fc7c0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7b7e00640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7c00b3010 0x7fc7c00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 -- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0003970 msgr2=0x7fc7c0154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0003970 0x7fc7c0154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 -- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7c00b3010 msgr2=0x7fc7c00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7c00b3010 0x7fc7c00b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 -- 172.21.15.43:0/3427946056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00cc390 con 0x7fc7c00c82d0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7bf400640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8000f80 tx=0x7fc7b800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.677+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8007a80 con 0x7fc7c00c82d0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.678+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b800edc0 con 0x7fc7c00c82d0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.678+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8014670 con 0x7fc7c00c82d0 2024-01-16T15:05:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.678+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c00c9000 con 0x7fc7c00c82d0 2024-01-16T15:05:51.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.678+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c00c95c0 con 0x7fc7c00c82d0 2024-01-16T15:05:51.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.679+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c006c3c0 con 0x7fc7c00c82d0 2024-01-16T15:05:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.680+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fc7b80147d0 con 0x7fc7c00c82d0 2024-01-16T15:05:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.681+0000 7fc7b6a00640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 0x7fc79806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.681+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc7b808a560 con 0x7fc7c00c82d0 2024-01-16T15:05:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.681+0000 7fc7bea00640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 0x7fc79806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.682+0000 7fc7bea00640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 0x7fc79806ddb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0036000 tx=0x7fc7b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:51.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.685+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7b8054a30 con 0x7fc7c00c82d0 2024-01-16T15:05:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:51.914+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53192", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7c00b1f40 con 0x7fc79806b8f0 2024-01-16T15:05:52.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.090+0000 7fc7b6a00640 1 -- 172.21.15.43:0/3427946056 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc7b0004140 con 0x7fc79806b8f0 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 msgr2=0x7fc79806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 0x7fc79806ddb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0036000 tx=0x7fc7b0030000 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 msgr2=0x7fc7c00c8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8000f80 tx=0x7fc7b800e530 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 shutdown_connections 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc79806b8f0 0x7fc79806ddb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7c00b3010 0x7fc7c00b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7c00c82d0 0x7fc7c00c8750 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 --2- 172.21.15.43:0/3427946056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0003970 0x7fc7c0154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 >> 172.21.15.43:0/3427946056 conn(0x7fc7c00b0ba0 msgr2=0x7fc7c0070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:52.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 shutdown_connections 2024-01-16T15:05:52.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.094+0000 7fc7c5600640 1 -- 172.21.15.43:0/3427946056 wait complete. 2024-01-16T15:05:52.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53193 2024-01-16T15:05:52.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.389+0000 7f8eb1c00640 1 Processor -- start 2024-01-16T15:05:52.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.390+0000 7f8eb1c00640 1 -- start start 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eb1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d1d90 0x7f8eac0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eb1c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8eac0d0fa0 0x7f8eac0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eb1c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0c6bf0 con 0x7f8eac15d3b0 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eb1c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac0d13d0 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eb1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0cedc0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eabe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8eac15d3b0 0x7f8eac0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39238/0 (socket says 172.21.15.43:39238) 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.391+0000 7f8eabe00640 1 -- 172.21.15.43:0/1670667712 learned_addr learned my addr 172.21.15.43:0/1670667712 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:05:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eb1200640 1 --2- 172.21.15.43:0/1670667712 >> v2:172.21.15.43:3300/0 conn(0x7f8eac0d0fa0 0x7f8eac0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eab400640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d1d90 0x7f8eac0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eaaa00640 1 -- 172.21.15.43:0/1670667712 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681708010 0 0) 0x7f8eac0c6bf0 con 0x7f8eac15d3b0 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eaaa00640 1 -- 172.21.15.43:0/1670667712 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c003520 con 0x7f8eac15d3b0 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eb1200640 1 -- 172.21.15.43:0/1670667712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d1d90 msgr2=0x7f8eac0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eb1200640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d1d90 0x7f8eac0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eb1200640 1 -- 172.21.15.43:0/1670667712 >> v1:172.21.15.43:6789/0 conn(0x7f8eac15d3b0 legacy=0x7f8eac0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.392+0000 7f8eb1200640 1 -- 172.21.15.43:0/1670667712 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0c6e40 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.393+0000 7f8eb1200640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d0fa0 0x7f8eac0b3ca0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097a0 tx=0x7f8ea402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5ab7c3544286b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eaaa00640 1 -- 172.21.15.43:0/1670667712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea402e070 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eaaa00640 1 -- 172.21.15.43:0/1670667712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea40315a0 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eaaa00640 1 -- 172.21.15.43:0/1670667712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4031930 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 -- 172.21.15.43:0/1670667712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d0fa0 msgr2=0x7f8eac0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d0fa0 0x7f8eac0b3ca0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097a0 tx=0x7f8ea402b840 comp rx=0 tx=0).stop 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 -- 172.21.15.43:0/1670667712 shutdown_connections 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d0fa0 0x7f8eac0b3ca0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/1670667712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d1d90 0x7f8eac0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 -- 172.21.15.43:0/1670667712 >> 172.21.15.43:0/1670667712 conn(0x7f8eac0d30a0 msgr2=0x7f8eac0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 -- 172.21.15.43:0/1670667712 shutdown_connections 2024-01-16T15:05:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.394+0000 7f8eb1c00640 1 -- 172.21.15.43:0/1670667712 wait complete. 2024-01-16T15:05:52.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.396+0000 7f8eb1c00640 1 Processor -- start 2024-01-16T15:05:52.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.396+0000 7f8eb1c00640 1 -- start start 2024-01-16T15:05:52.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 0x7f8eac1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 0x7f8eac15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac1568f0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac1f5ed0 con 0x7f8eac15d3b0 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.397+0000 7f8eb1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac1f61b0 con 0x7f8eac0d0fa0 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59644/0 (socket says 172.21.15.43:59644) 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 -- 172.21.15.43:0/3162210801 learned_addr learned my addr 172.21.15.43:0/3162210801 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eabe00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 0x7f8eac1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eb1200640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 0x7f8eac15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 -- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 msgr2=0x7f8eac1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 0x7f8eac1563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 -- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 msgr2=0x7f8eac15a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 0x7f8eac15a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 -- 172.21.15.43:0/3162210801 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4009400 con 0x7f8eac0d1d90 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eabe00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 0x7f8eac1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eb1200640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 0x7f8eac15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:52.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.398+0000 7f8eab400640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400d800 tx=0x7f8e9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.399+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400b6c0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.399+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9400b820 con 0x7f8eac0d1d90 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.399+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400bbb0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.399+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac1f6490 con 0x7f8eac0d1d90 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.399+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac0cb670 con 0x7f8eac0d1d90 2024-01-16T15:05:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.400+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac0c73c0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.401+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f8e9400bd10 con 0x7f8eac0d1d90 2024-01-16T15:05:52.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.402+0000 7f8ea3e00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 0x7f8e8406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:52.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.403+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8e9408ac90 con 0x7f8eac0d1d90 2024-01-16T15:05:52.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.403+0000 7f8eabe00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 0x7f8e8406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:52.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.404+0000 7f8eabe00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 0x7f8e8406ddb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c001560 tx=0x7f8e9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:52.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.407+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e940551f0 con 0x7f8eac0d1d90 2024-01-16T15:05:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:52.640+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53193", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eac15d790 con 0x7f8e8406b8f0 2024-01-16T15:05:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:53.872+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f8e940555c0 con 0x7f8eac0d1d90 2024-01-16T15:05:55.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:55.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:55.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.741+0000 7f8ea3e00640 1 -- 172.21.15.43:0/3162210801 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8e9c001d90 con 0x7f8e8406b8f0 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 msgr2=0x7f8e8406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 0x7f8e8406ddb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c001560 tx=0x7f8e9c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 msgr2=0x7f8eac0b8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400d800 tx=0x7f8e9400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 shutdown_connections 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8e8406b8f0 0x7f8e8406ddb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eac15d3b0 0x7f8eac15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8eac0d1d90 0x7f8eac0b8740 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 --2- 172.21.15.43:0/3162210801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8eac0d0fa0 0x7f8eac1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 >> 172.21.15.43:0/3162210801 conn(0x7f8eac0d30a0 msgr2=0x7f8eac15c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.745+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 shutdown_connections 2024-01-16T15:05:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.746+0000 7f8eb1c00640 1 -- 172.21.15.43:0/3162210801 wait complete. 2024-01-16T15:05:55.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53194 2024-01-16T15:05:55.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.993+0000 7fa453800640 1 Processor -- start 2024-01-16T15:05:55.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.993+0000 7fa453800640 1 -- start start 2024-01-16T15:05:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa453800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa44c160b10 0x7fa44c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa453800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa44c072280 0x7fa44c06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa453800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c06cd50 con 0x7fa44c06c900 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa453800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c0b2320 con 0x7fa44c160b10 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa453800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c06df30 con 0x7fa44c072280 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa450e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa44c160b10 0x7fa44c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa450e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa44c160b10 0x7fa44c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46310/0 (socket says 172.21.15.43:46310) 2024-01-16T15:05:55.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa450e00640 1 -- 172.21.15.43:0/2157609665 learned_addr learned my addr 172.21.15.43:0/2157609665 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.995+0000 7fa44be00640 1 --2- 172.21.15.43:0/2157609665 >> v2:172.21.15.69:3300/0 conn(0x7fa44c072280 0x7fa44c06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.996+0000 7fa450e00640 1 -- 172.21.15.43:0/2157609665 >> v2:172.21.15.69:3300/0 conn(0x7fa44c072280 msgr2=0x7fa44c06c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.996+0000 7fa450e00640 1 --2- 172.21.15.43:0/2157609665 >> v2:172.21.15.69:3300/0 conn(0x7fa44c072280 0x7fa44c06c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.996+0000 7fa450e00640 1 -- 172.21.15.43:0/2157609665 >> v1:172.21.15.69:6789/0 conn(0x7fa44c06c900 legacy=0x7fa44c160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.996+0000 7fa450e00640 1 -- 172.21.15.43:0/2157609665 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c071780 con 0x7fa44c160b10 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.997+0000 7fa450e00640 1 --2- 172.21.15.43:0/2157609665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c160b10 0x7fa44c06e420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0d0100 tx=0x7fa43c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4af4c7def0dd4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:55.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.997+0000 7fa44b400640 1 -- 172.21.15.43:0/2157609665 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0317e0 con 0x7fa44c160b10 2024-01-16T15:05:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.997+0000 7fa44b400640 1 -- 172.21.15.43:0/2157609665 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c031940 con 0x7fa44c160b10 2024-01-16T15:05:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.997+0000 7fa44b400640 1 -- 172.21.15.43:0/2157609665 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c031cd0 con 0x7fa44c160b10 2024-01-16T15:05:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 -- 172.21.15.43:0/2157609665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c160b10 msgr2=0x7fa44c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 --2- 172.21.15.43:0/2157609665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c160b10 0x7fa44c06e420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0d0100 tx=0x7fa43c02b880 comp rx=0 tx=0).stop 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 -- 172.21.15.43:0/2157609665 shutdown_connections 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 --2- 172.21.15.43:0/2157609665 >> v2:172.21.15.69:3300/0 conn(0x7fa44c072280 0x7fa44c06c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 --2- 172.21.15.43:0/2157609665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c160b10 0x7fa44c06e420 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 -- 172.21.15.43:0/2157609665 >> 172.21.15.43:0/2157609665 conn(0x7fa44c0b0ba0 msgr2=0x7fa44c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 -- 172.21.15.43:0/2157609665 shutdown_connections 2024-01-16T15:05:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:55.998+0000 7fa453800640 1 -- 172.21.15.43:0/2157609665 wait complete. 2024-01-16T15:05:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.000+0000 7fa453800640 1 Processor -- start 2024-01-16T15:05:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.000+0000 7fa453800640 1 -- start start 2024-01-16T15:05:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 0x7fa44c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c160b10 0x7fa44c09b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0cced0 con 0x7fa44c072280 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0cbf80 con 0x7fa44c06c900 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa453800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0cc250 con 0x7fa44c160b10 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46314/0 (socket says 172.21.15.43:46314) 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 -- 172.21.15.43:0/3922651066 learned_addr learned my addr 172.21.15.43:0/3922651066 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa450e00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 0x7fa44c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa451800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c160b10 0x7fa44c09b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 -- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c160b10 msgr2=0x7fa44c09b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c160b10 0x7fa44c09b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 -- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 msgr2=0x7fa44c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 0x7fa44c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa44be00640 1 -- 172.21.15.43:0/3922651066 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c0094d0 con 0x7fa44c072280 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.001+0000 7fa450e00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 0x7fa44c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa44be00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d5f0 tx=0x7fa43400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4340125f0 con 0x7fa44c072280 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434012750 con 0x7fa44c072280 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434012b00 con 0x7fa44c072280 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c0cc530 con 0x7fa44c072280 2024-01-16T15:05:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.002+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c0ccaf0 con 0x7fa44c072280 2024-01-16T15:05:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.003+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c0b51b0 con 0x7fa44c072280 2024-01-16T15:05:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.004+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fa434012c60 con 0x7fa44c072280 2024-01-16T15:05:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.005+0000 7fa448c00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 0x7fa42406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.005+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa43408b250 con 0x7fa44c072280 2024-01-16T15:05:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.006+0000 7fa450e00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 0x7fa42406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.006+0000 7fa450e00640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 0x7fa42406ded0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa43c036000 tx=0x7fa43c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.010+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4340886f0 con 0x7fa44c072280 2024-01-16T15:05:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:05:56.242+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53194", "target": ["mon-mgr", ""]}) v1 -- 0x7fa44c0b1f40 con 0x7fa42406ba10 2024-01-16T15:06:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.768+0000 7fa448c00640 1 -- 172.21.15.43:0/3922651066 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa43c031b30 con 0x7fa42406ba10 2024-01-16T15:06:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.771+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 msgr2=0x7fa42406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.771+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 0x7fa42406ded0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa43c036000 tx=0x7fa43c03d040 comp rx=0 tx=0).stop 2024-01-16T15:06:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.771+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 msgr2=0x7fa44c14d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.771+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d5f0 tx=0x7fa43400dab0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 shutdown_connections 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa42406ba10 0x7fa42406ded0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa44c160b10 0x7fa44c09b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa44c072280 0x7fa44c14d9e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 --2- 172.21.15.43:0/3922651066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa44c06c900 0x7fa44c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 >> 172.21.15.43:0/3922651066 conn(0x7fa44c0b0ba0 msgr2=0x7fa44c15fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 shutdown_connections 2024-01-16T15:06:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:00.772+0000 7fa453800640 1 -- 172.21.15.43:0/3922651066 wait complete. 2024-01-16T15:06:00.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53195 2024-01-16T15:06:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.021+0000 7fa43b200640 1 Processor -- start 2024-01-16T15:06:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.021+0000 7fa43b200640 1 -- start start 2024-01-16T15:06:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.022+0000 7fa43b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa4340d1d90 0x7fa4340ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.022+0000 7fa43b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340c6bf0 con 0x7fa43415d3b0 2024-01-16T15:06:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.022+0000 7fa43b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340d13f0 con 0x7fa4340d0fa0 2024-01-16T15:06:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.022+0000 7fa43b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4340cedc0 con 0x7fa4340d1d90 2024-01-16T15:06:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.023+0000 7fa439e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa43415d3b0 0x7fa4340d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39082/0 (socket says 172.21.15.43:39082) 2024-01-16T15:06:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.023+0000 7fa439e00640 1 -- 172.21.15.43:0/3110638210 learned_addr learned my addr 172.21.15.43:0/3110638210 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.023+0000 7fa439400640 1 --2- 172.21.15.43:0/3110638210 >> v2:172.21.15.69:3300/0 conn(0x7fa4340d1d90 0x7fa4340ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.023+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628589631 0 0) 0x7fa4340c6bf0 con 0x7fa43415d3b0 2024-01-16T15:06:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.023+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa414003520 con 0x7fa43415d3b0 2024-01-16T15:06:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094041773 0 0) 0x7fa414003520 con 0x7fa43415d3b0 2024-01-16T15:06:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 >> v2:172.21.15.69:3300/0 conn(0x7fa4340d1d90 msgr2=0x7fa4340ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 --2- 172.21.15.43:0/3110638210 >> v2:172.21.15.69:3300/0 conn(0x7fa4340d1d90 0x7fa4340ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 >> v1:172.21.15.69:6789/0 conn(0x7fa4340d0fa0 legacy=0x7fa4340b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340c6e40 con 0x7fa43415d3b0 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa424001670 con 0x7fa43415d3b0 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 >> v1:172.21.15.43:6789/0 conn(0x7fa43415d3b0 legacy=0x7fa4340d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 0x7fa414003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4140047c0 0x7fa414004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140051f0 0x7fa41400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340cedc0 con 0x7fa4140039e0 2024-01-16T15:06:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41400c150 con 0x7fa4140047c0 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41400c310 con 0x7fa4140051f0 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.024+0000 7fa439e00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 0x7fa414003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439400640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4140047c0 0x7fa414004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439e00640 1 -- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140051f0 msgr2=0x7fa41400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439e00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140051f0 0x7fa41400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439e00640 1 -- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4140047c0 msgr2=0x7fa414004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439e00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4140047c0 0x7fa414004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa439e00640 1 -- 172.21.15.43:0/3110638210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414003520 con 0x7fa4140039e0 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.025+0000 7fa43a800640 1 -- 172.21.15.43:0/3110638210 reap_dead start 2024-01-16T15:06:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.026+0000 7fa439e00640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 0x7fa414003e00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c240 tx=0x7fa4140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ff95111d2b7d46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.026+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424005ad0 con 0x7fa4140039e0 2024-01-16T15:06:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.026+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa424005c30 con 0x7fa4140039e0 2024-01-16T15:06:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.026+0000 7fa438a00640 1 -- 172.21.15.43:0/3110638210 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42402ca50 con 0x7fa4140039e0 2024-01-16T15:06:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 -- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 msgr2=0x7fa414003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 0x7fa414003e00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c240 tx=0x7fa4140057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 -- 172.21.15.43:0/3110638210 shutdown_connections 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 --2- 172.21.15.43:0/3110638210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4140039e0 0x7fa414003e00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 -- 172.21.15.43:0/3110638210 >> 172.21.15.43:0/3110638210 conn(0x7fa4340d30a0 msgr2=0x7fa4340d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 -- 172.21.15.43:0/3110638210 shutdown_connections 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.027+0000 7fa43b200640 1 -- 172.21.15.43:0/3110638210 wait complete. 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.029+0000 7fa43b200640 1 Processor -- start 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.029+0000 7fa43b200640 1 -- start start 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140039e0 0x7fa434153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 0x7fa43409ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43409f4a0 con 0x7fa4340b8140 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4340b8c90 con 0x7fa43409eae0 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa43b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43415cac0 con 0x7fa4140039e0 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.030+0000 7fa439400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46342/0 (socket says 172.21.15.43:46342) 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 -- 172.21.15.43:0/1427863217 learned_addr learned my addr 172.21.15.43:0/1427863217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439e00640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140039e0 0x7fa434153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa43a800640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 0x7fa43409ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 -- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140039e0 msgr2=0x7fa434153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140039e0 0x7fa434153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 -- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 msgr2=0x7fa43409ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 0x7fa43409ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 -- 172.21.15.43:0/1427863217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340b6800 con 0x7fa4340b8140 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa43a800640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 0x7fa43409ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.031+0000 7fa439400640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0108e0 tx=0x7fa41c010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.032+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00eb60 con 0x7fa4340b8140 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.032+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c00ecc0 con 0x7fa4340b8140 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.032+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4340b7580 con 0x7fa4340b8140 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.032+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c01c400 con 0x7fa4340b8140 2024-01-16T15:06:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.032+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4340b6d80 con 0x7fa4340b8140 2024-01-16T15:06:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.033+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43415fd60 con 0x7fa4340b8140 2024-01-16T15:06:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.034+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fa41c01c560 con 0x7fa4340b8140 2024-01-16T15:06:01.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.035+0000 7fa432000640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 0x7fa40c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:01.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.035+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa41c08b470 con 0x7fa4340b8140 2024-01-16T15:06:01.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.036+0000 7fa439e00640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 0x7fa40c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:01.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.036+0000 7fa439e00640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 0x7fa40c06ded0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa424034000 tx=0x7fa424009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.040+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa41c088910 con 0x7fa4340b8140 2024-01-16T15:06:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:01.272+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7fa43415d790 con 0x7fa40c06ba10 2024-01-16T15:06:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.769+0000 7fa432000640 1 -- 172.21.15.43:0/1427863217 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa42403d040 con 0x7fa40c06ba10 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 msgr2=0x7fa40c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 0x7fa40c06ded0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa424034000 tx=0x7fa424009030 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 msgr2=0x7fa4340b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0108e0 tx=0x7fa41c010da0 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 shutdown_connections 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa40c06ba10 0x7fa40c06ded0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa43409eae0 0x7fa43409ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4340b8140 0x7fa4340b85a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 --2- 172.21.15.43:0/1427863217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa4140039e0 0x7fa434153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.772+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 >> 172.21.15.43:0/1427863217 conn(0x7fa4340d30a0 msgr2=0x7fa43406c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.773+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 shutdown_connections 2024-01-16T15:06:05.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:05.773+0000 7fa43b200640 1 -- 172.21.15.43:0/1427863217 wait complete. 2024-01-16T15:06:05.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53196 2024-01-16T15:06:06.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.021+0000 7f9a6f600640 1 Processor -- start 2024-01-16T15:06:06.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.022+0000 7f9a6f600640 1 -- start start 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.023+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a6806a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.023+0000 7f9a6f600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a680cba40 con 0x7f9a6806e240 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.023+0000 7f9a6f600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a680cbce0 con 0x7f9a680cd640 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.023+0000 7f9a6f600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68066900 con 0x7f9a68066490 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9a6806e240 0x7f9a680b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46878/0 (socket says 172.21.15.43:46878) 2024-01-16T15:06:06.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67e00640 1 -- 172.21.15.43:0/1663493928 learned_addr learned my addr 172.21.15.43:0/1663493928 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:06.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a6d600640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a6806a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182672035 0 0) 0x7f9a680cba40 con 0x7f9a6806e240 2024-01-16T15:06:06.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a48003520 con 0x7f9a6806e240 2024-01-16T15:06:06.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260693475 0 0) 0x7f9a48003520 con 0x7f9a6806e240 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 msgr2=0x7f9a6806a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a6806a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.024+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 >> v1:172.21.15.69:6789/0 conn(0x7f9a680cd640 legacy=0x7f9a680cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a680b45a0 con 0x7f9a6806e240 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9a500019f0 con 0x7f9a6806e240 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 >> v1:172.21.15.43:6789/0 conn(0x7f9a6806e240 legacy=0x7f9a680b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68066900 con 0x7f9a480039a0 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800bab0 con 0x7f9a480047c0 2024-01-16T15:06:06.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800de80 con 0x7f9a68066490 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a67e00640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 -- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 msgr2=0x7f9a480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6d600640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68066490 0x7f9a480051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 -- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480047c0 msgr2=0x7f9a48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.025+0000 7f9a6cc00640 1 -- 172.21.15.43:0/1663493928 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48003520 con 0x7f9a480039a0 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.027+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a48003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f9a58001750 tx=0x7f9a480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8eecb7247314f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.027+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580052d0 con 0x7f9a480039a0 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.027+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5802e050 con 0x7f9a480039a0 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.027+0000 7f9a67400640 1 -- 172.21.15.43:0/1663493928 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580035f0 con 0x7f9a480039a0 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 -- 172.21.15.43:0/1663493928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 msgr2=0x7f9a48003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 --2- 172.21.15.43:0/1663493928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a48003dc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f9a58001750 tx=0x7f9a480057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6d600640 1 -- 172.21.15.43:0/1663493928 reap_dead start 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 -- 172.21.15.43:0/1663493928 shutdown_connections 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 -- 172.21.15.43:0/1663493928 >> 172.21.15.43:0/1663493928 conn(0x7f9a680cf740 msgr2=0x7f9a680cfb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 -- 172.21.15.43:0/1663493928 shutdown_connections 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.028+0000 7f9a6f600640 1 -- 172.21.15.43:0/1663493928 wait complete. 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.030+0000 7f9a6f600640 1 Processor -- start 2024-01-16T15:06:06.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.030+0000 7f9a6f600640 1 -- start start 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68158320 0x7f9a68145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a68158780 0x7f9a681461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68153ae0 con 0x7f9a480039a0 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68153d30 con 0x7f9a68158780 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6f600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68154010 con 0x7f9a68158320 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34346/0 (socket says 172.21.15.43:34346) 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 -- 172.21.15.43:0/315001819 learned_addr learned my addr 172.21.15.43:0/315001819 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 -- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68158320 msgr2=0x7f9a68145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68158320 0x7f9a68145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 -- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a68158780 msgr2=0x7f9a681461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a68158780 0x7f9a681461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.031+0000 7f9a6cc00640 1 -- 172.21.15.43:0/315001819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a681542f0 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a6cc00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9a58008fd0 tx=0x7f9a58001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007a30 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58007640 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58006c20 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a681545d0 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.032+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a68155450 con 0x7f9a480039a0 2024-01-16T15:06:06.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.033+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68072510 con 0x7f9a480039a0 2024-01-16T15:06:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.034+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f9a580035f0 con 0x7f9a480039a0 2024-01-16T15:06:06.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.036+0000 7f9a64c00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 0x7f9a4006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.036+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9a580acce0 con 0x7f9a480039a0 2024-01-16T15:06:06.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.036+0000 7f9a67e00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 0x7f9a4006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.037+0000 7f9a67e00640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 0x7f9a4006df00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9a50001580 tx=0x7f9a500014f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.041+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a580aa070 con 0x7f9a480039a0 2024-01-16T15:06:06.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.270+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53196", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a68153ae0 con 0x7f9a4006ba40 2024-01-16T15:06:06.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.432+0000 7f9a64c00640 1 -- 172.21.15.43:0/315001819 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9a68153ae0 con 0x7f9a4006ba40 2024-01-16T15:06:06.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 msgr2=0x7f9a4006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 0x7f9a4006df00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9a50001580 tx=0x7f9a500014f0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 msgr2=0x7f9a68154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9a58008fd0 tx=0x7f9a58001120 comp rx=0 tx=0).stop 2024-01-16T15:06:06.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 shutdown_connections 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a4006ba40 0x7f9a4006df00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a68158780 0x7f9a681461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a68158320 0x7f9a68145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 --2- 172.21.15.43:0/315001819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a480039a0 0x7f9a68154af0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 >> 172.21.15.43:0/315001819 conn(0x7f9a680cf740 msgr2=0x7f9a68159c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.435+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 shutdown_connections 2024-01-16T15:06:06.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.436+0000 7f9a6f600640 1 -- 172.21.15.43:0/315001819 wait complete. 2024-01-16T15:06:06.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone53197 2024-01-16T15:06:06.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.679+0000 7fb3dde00640 1 Processor -- start 2024-01-16T15:06:06.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.680+0000 7fb3dde00640 1 -- start start 2024-01-16T15:06:06.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3dde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3dde00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d80b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3dde00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d80b23e0 con 0x7fb3d80b3040 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3dde00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d80cdc70 con 0x7fb3d8160bd0 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3dde00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80b2870 con 0x7fb3d80b48d0 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.681+0000 7fb3d6a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34356/0 (socket says 172.21.15.43:34356) 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 -- 172.21.15.43:0/644442457 learned_addr learned my addr 172.21.15.43:0/644442457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3cfe00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d80b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 -- 172.21.15.43:0/644442457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 msgr2=0x7fb3d80b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d80b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 -- 172.21.15.43:0/644442457 >> v1:172.21.15.69:6789/0 conn(0x7fb3d80b3040 legacy=0x7fb3d8166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:06.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.682+0000 7fb3d6a00640 1 -- 172.21.15.43:0/644442457 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d80d41b0 con 0x7fb3d8160bd0 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.683+0000 7fb3d6a00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009870 tx=0x7fb3c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b7b8f598c4e084 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3d6000640 1 -- 172.21.15.43:0/644442457 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80317c0 con 0x7fb3d8160bd0 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3d6000640 1 -- 172.21.15.43:0/644442457 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8031920 con 0x7fb3d8160bd0 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3d6000640 1 -- 172.21.15.43:0/644442457 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8031cb0 con 0x7fb3d8160bd0 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 -- 172.21.15.43:0/644442457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d8160bd0 msgr2=0x7fb3d80b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009870 tx=0x7fb3c802b860 comp rx=0 tx=0).stop 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 -- 172.21.15.43:0/644442457 shutdown_connections 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d80b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 --2- 172.21.15.43:0/644442457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b1ea0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.684+0000 7fb3dde00640 1 -- 172.21.15.43:0/644442457 >> 172.21.15.43:0/644442457 conn(0x7fb3d809c5a0 msgr2=0x7fb3d80d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.685+0000 7fb3dde00640 1 -- 172.21.15.43:0/644442457 shutdown_connections 2024-01-16T15:06:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.685+0000 7fb3dde00640 1 -- 172.21.15.43:0/644442457 wait complete. 2024-01-16T15:06:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.686+0000 7fb3dde00640 1 Processor -- start 2024-01-16T15:06:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.687+0000 7fb3dde00640 1 -- start start 2024-01-16T15:06:06.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.687+0000 7fb3dde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3dde00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d8155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3dde00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80c7a20 con 0x7fb3d80b3040 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3dde00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80c6e10 con 0x7fb3d8160bd0 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3dde00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80c70f0 con 0x7fb3d80b48d0 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34364/0 (socket says 172.21.15.43:34364) 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 -- 172.21.15.43:0/3965185117 learned_addr learned my addr 172.21.15.43:0/3965185117 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3cfe00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d8155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d7400640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 -- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 msgr2=0x7fb3d8155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d8155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 -- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 msgr2=0x7fb3d80b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d6a00640 1 -- 172.21.15.43:0/3965185117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80094d0 con 0x7fb3d80b3040 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.688+0000 7fb3d7400640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3d6a00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009870 tx=0x7fb3c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8032960 con 0x7fb3d80b3040 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8032ac0 con 0x7fb3d80b3040 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d80c73d0 con 0x7fb3d80b3040 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8032e70 con 0x7fb3d80b3040 2024-01-16T15:06:06.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.689+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d8156290 con 0x7fb3d80b3040 2024-01-16T15:06:06.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.690+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d80cf020 con 0x7fb3d80b3040 2024-01-16T15:06:06.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.691+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fb3c804e450 con 0x7fb3d80b3040 2024-01-16T15:06:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.692+0000 7fb3cf400640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 0x7fb3b006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.692+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb3c80ad250 con 0x7fb3d80b3040 2024-01-16T15:06:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.693+0000 7fb3cfe00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 0x7fb3b006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.693+0000 7fb3cfe00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 0x7fb3b006ded0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80cd3a0 tx=0x7fb3c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.697+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3c80aa6f0 con 0x7fb3d80b3040 2024-01-16T15:06:06.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:06.925+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone53197", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3d80b3760 con 0x7fb3b006ba10 2024-01-16T15:06:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.119+0000 7fb3cf400640 1 -- 172.21.15.43:0/3965185117 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb3c000c050 con 0x7fb3b006ba10 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 msgr2=0x7fb3b006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 0x7fb3b006ded0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80cd3a0 tx=0x7fb3c0008040 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 msgr2=0x7fb3d80c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009870 tx=0x7fb3c80046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 shutdown_connections 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb3b006ba10 0x7fb3b006ded0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3d8160bd0 0x7fb3d80b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb3d80b48d0 0x7fb3d8155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 --2- 172.21.15.43:0/3965185117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3d80b3040 0x7fb3d80c7eb0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 >> 172.21.15.43:0/3965185117 conn(0x7fb3d809c5a0 msgr2=0x7fb3d806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 shutdown_connections 2024-01-16T15:06:07.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.123+0000 7fb3dde00640 1 -- 172.21.15.43:0/3965185117 wait complete. 2024-01-16T15:06:07.126 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:06:07.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53191 2024-01-16T15:06:07.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.375+0000 7f5937c00640 1 Processor -- start 2024-01-16T15:06:07.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.376+0000 7f5937c00640 1 -- start start 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.377+0000 7f5937c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5930146bd0 0x7f59300b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5937c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5930072510 0x7f59301616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5937c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f59300b2360 0x7f59300b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5937c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300b3650 con 0x7f5930072510 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5937c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59300b2d30 con 0x7f59300b2360 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5937c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59300ada70 con 0x7f5930146bd0 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f592fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5930072510 0x7f59301616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f592fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5930072510 0x7f59301616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34374/0 (socket says 172.21.15.43:34374) 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f592fe00640 1 -- 172.21.15.43:0/200254152 learned_addr learned my addr 172.21.15.43:0/200254152 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5935200640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5930146bd0 0x7f59300b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.378+0000 7f5935c00640 1 --2- 172.21.15.43:0/200254152 >> v2:172.21.15.69:3300/0 conn(0x7f59300b2360 0x7f59300b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.379+0000 7f592fe00640 1 -- 172.21.15.43:0/200254152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5930146bd0 msgr2=0x7f59300b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.379+0000 7f592fe00640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5930146bd0 0x7f59300b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.379+0000 7f592fe00640 1 -- 172.21.15.43:0/200254152 >> v2:172.21.15.69:3300/0 conn(0x7f59300b2360 msgr2=0x7f59300b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.379+0000 7f592fe00640 1 --2- 172.21.15.43:0/200254152 >> v2:172.21.15.69:3300/0 conn(0x7f59300b2360 0x7f59300b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.379+0000 7f592fe00640 1 -- 172.21.15.43:0/200254152 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59300add50 con 0x7f5930072510 2024-01-16T15:06:07.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.380+0000 7f592fe00640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930072510 0x7f59301616f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5930072930 tx=0x7f591802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d327609bafd4e887 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.380+0000 7f592f400640 1 -- 172.21.15.43:0/200254152 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918031870 con 0x7f5930072510 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.380+0000 7f592f400640 1 -- 172.21.15.43:0/200254152 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59180319d0 con 0x7f5930072510 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.380+0000 7f592f400640 1 -- 172.21.15.43:0/200254152 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918031d60 con 0x7f5930072510 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 -- 172.21.15.43:0/200254152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930072510 msgr2=0x7f59301616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930072510 0x7f59301616f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5930072930 tx=0x7f591802b910 comp rx=0 tx=0).stop 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 -- 172.21.15.43:0/200254152 shutdown_connections 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 --2- 172.21.15.43:0/200254152 >> v2:172.21.15.69:3300/0 conn(0x7f59300b2360 0x7f59300b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930072510 0x7f59301616f0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 --2- 172.21.15.43:0/200254152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5930146bd0 0x7f59300b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 -- 172.21.15.43:0/200254152 >> 172.21.15.43:0/200254152 conn(0x7f59300661f0 msgr2=0x7f59300665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 -- 172.21.15.43:0/200254152 shutdown_connections 2024-01-16T15:06:07.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.381+0000 7f5937c00640 1 -- 172.21.15.43:0/200254152 wait complete. 2024-01-16T15:06:07.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.383+0000 7f5937c00640 1 Processor -- start 2024-01-16T15:06:07.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.383+0000 7f5937c00640 1 -- start start 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5930072510 0x7f5930160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59300b2360 0x7f5930140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59301e74b0 con 0x7f5930146bd0 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59301e7700 con 0x7f5930072510 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5937c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59301e79e0 con 0x7f59300b2360 2024-01-16T15:06:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5935c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5935c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34378/0 (socket says 172.21.15.43:34378) 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.384+0000 7f5935c00640 1 -- 172.21.15.43:0/2716194339 learned_addr learned my addr 172.21.15.43:0/2716194339 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935200640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5930072510 0x7f5930160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f592fe00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59300b2360 0x7f5930140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 -- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59300b2360 msgr2=0x7f5930140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59300b2360 0x7f5930140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 -- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5930072510 msgr2=0x7f5930160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5930072510 0x7f5930160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 -- 172.21.15.43:0/2716194339 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180094d0 con 0x7f5930146bd0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f5935c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5928007e60 tx=0x7f592800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928014620 con 0x7f5930146bd0 2024-01-16T15:06:07.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592800ee90 con 0x7f5930146bd0 2024-01-16T15:06:07.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.385+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59280149e0 con 0x7f5930146bd0 2024-01-16T15:06:07.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.386+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59301e7cc0 con 0x7f5930146bd0 2024-01-16T15:06:07.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.386+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59301e8280 con 0x7f5930146bd0 2024-01-16T15:06:07.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.387+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593006e800 con 0x7f5930146bd0 2024-01-16T15:06:07.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.388+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f5928014b40 con 0x7f5930146bd0 2024-01-16T15:06:07.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.389+0000 7f592cc00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 0x7f590806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.389+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f592808b660 con 0x7f5930146bd0 2024-01-16T15:06:07.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.389+0000 7f5935200640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 0x7f590806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.390+0000 7f5935200640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 0x7f590806ded0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5920006fd0 tx=0x7f5920008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.393+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5928088bc0 con 0x7f5930146bd0 2024-01-16T15:06:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.624+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53191", "target": ["mon-mgr", ""]}) v1 -- 0x7f5930146fb0 con 0x7f590806ba10 2024-01-16T15:06:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.645+0000 7f592cc00640 1 -- 172.21.15.43:0/2716194339 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f592000f860 con 0x7f590806ba10 2024-01-16T15:06:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 0.0, 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "0/3", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:07.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:07.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:07.625 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 msgr2=0x7f590806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 0x7f590806ded0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5920006fd0 tx=0x7f5920008040 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 msgr2=0x7f59301411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5928007e60 tx=0x7f592800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 shutdown_connections 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f590806ba10 0x7f590806ded0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5930146bd0 0x7f59301411d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59300b2360 0x7f5930140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 --2- 172.21.15.43:0/2716194339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5930072510 0x7f5930160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 >> 172.21.15.43:0/2716194339 conn(0x7f59300661f0 msgr2=0x7f59300ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 shutdown_connections 2024-01-16T15:06:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.648+0000 7f5937c00640 1 -- 172.21.15.43:0/2716194339 wait complete. 2024-01-16T15:06:07.647 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:06:07.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53197 2024-01-16T15:06:07.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.899+0000 7f85b8800640 1 Processor -- start 2024-01-16T15:06:07.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.899+0000 7f85b8800640 1 -- start start 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b8800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85b0162220 0x7f85b00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b8800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b00706a0 con 0x7f85b006cad0 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b8800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b0070c60 con 0x7f85b0162600 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b8800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b00722b0 con 0x7f85b0162220 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b6800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f85b006cad0 0x7f85b00710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46894/0 (socket says 172.21.15.43:46894) 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b6800640 1 -- 172.21.15.43:0/1031157032 learned_addr learned my addr 172.21.15.43:0/1031157032 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b5e00640 1 --2- 172.21.15.43:0/1031157032 >> v2:172.21.15.69:3300/0 conn(0x7f85b0162220 0x7f85b00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.901+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732681170 0 0) 0x7f85b00706a0 con 0x7f85b006cad0 2024-01-16T15:06:07.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8590003520 con 0x7f85b006cad0 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454283174 0 0) 0x7f8590003520 con 0x7f85b006cad0 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 >> v2:172.21.15.69:3300/0 conn(0x7f85b0162220 msgr2=0x7f85b00b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 --2- 172.21.15.43:0/1031157032 >> v2:172.21.15.69:3300/0 conn(0x7f85b0162220 0x7f85b00b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 >> v1:172.21.15.69:6789/0 conn(0x7f85b0162600 legacy=0x7f85b006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0072590 con 0x7f85b006cad0 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f85a80019f0 con 0x7f85b006cad0 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 >> v1:172.21.15.43:6789/0 conn(0x7f85b006cad0 legacy=0x7f85b00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8590003970 0x7f8590003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85900047c0 0x7f8590004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 0x7f859000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b00722b0 con 0x7f8590005180 2024-01-16T15:06:07.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859000c020 con 0x7f85900047c0 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859000c1e0 con 0x7f8590003970 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.902+0000 7f85b6800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 0x7f859000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b5e00640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8590003970 0x7f8590003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b5400640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85900047c0 0x7f8590004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b6800640 1 -- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8590003970 msgr2=0x7f8590003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b6800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8590003970 0x7f8590003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b6800640 1 -- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85900047c0 msgr2=0x7f8590004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b6800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85900047c0 0x7f8590004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b6800640 1 -- 172.21.15.43:0/1031157032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590003520 con 0x7f8590005180 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.903+0000 7f85b5400640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85900047c0 0x7f8590004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.904+0000 7f85b6800640 1 -- 172.21.15.43:0/1031157032 reap_dead start 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.904+0000 7f85b6800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 0x7f859000b980 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f85a800a800 tx=0x7f85900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea4265ebb16ca14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.904+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a80072d0 con 0x7f8590005180 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.904+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a802f070 con 0x7f8590005180 2024-01-16T15:06:07.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.904+0000 7f85b4a00640 1 -- 172.21.15.43:0/1031157032 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8007670 con 0x7f8590005180 2024-01-16T15:06:07.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 -- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 msgr2=0x7f859000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 0x7f859000b980 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f85a800a800 tx=0x7f85900057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 -- 172.21.15.43:0/1031157032 shutdown_connections 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 --2- 172.21.15.43:0/1031157032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8590005180 0x7f859000b980 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 -- 172.21.15.43:0/1031157032 >> 172.21.15.43:0/1031157032 conn(0x7f85b00b0ba0 msgr2=0x7f85b00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 -- 172.21.15.43:0/1031157032 shutdown_connections 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.905+0000 7f85b8800640 1 -- 172.21.15.43:0/1031157032 wait complete. 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.907+0000 7f85b8800640 1 Processor -- start 2024-01-16T15:06:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.907+0000 7f85b8800640 1 -- start start 2024-01-16T15:06:07.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8590005180 0x7f85b00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b009bda0 0x7f85b015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b00b2cd0 con 0x7f85b00b3b20 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b009b910 con 0x7f8590005180 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b8800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b009afe0 con 0x7f85b009bda0 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.908+0000 7f85b5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34390/0 (socket says 172.21.15.43:34390) 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 -- 172.21.15.43:0/3430790435 learned_addr learned my addr 172.21.15.43:0/3430790435 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5e00640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8590005180 0x7f85b00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b6800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b009bda0 0x7f85b015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 -- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b009bda0 msgr2=0x7f85b015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b009bda0 0x7f85b015ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 -- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8590005180 msgr2=0x7f85b00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8590005180 0x7f85b00b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 -- 172.21.15.43:0/3430790435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b009b2c0 con 0x7f85b00b3b20 2024-01-16T15:06:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.909+0000 7f85b5400640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f859800d770 tx=0x7f859800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.910+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598005270 con 0x7f85b00b3b20 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.910+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85980053d0 con 0x7f85b00b3b20 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.910+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859800b8f0 con 0x7f85b00b3b20 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.910+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b009b5a0 con 0x7f85b00b3b20 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.910+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b00cdcf0 con 0x7f85b00b3b20 2024-01-16T15:06:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.911+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b0162df0 con 0x7f85b00b3b20 2024-01-16T15:06:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.912+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f859800ba50 con 0x7f85b00b3b20 2024-01-16T15:06:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.913+0000 7f85ae000640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 0x7f858806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.913+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f859808ace0 con 0x7f85b00b3b20 2024-01-16T15:06:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.913+0000 7f85b5e00640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 0x7f858806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:07.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.914+0000 7f85b5e00640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 0x7f858806df20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009990 tx=0x7f85a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:07.918+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8598088180 con 0x7f85b00b3b20 2024-01-16T15:06:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.252+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53197", "target": ["mon-mgr", ""]}) v1 -- 0x7f85b0162a60 con 0x7f858806ba60 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.326+0000 7f85ae000640 1 -- 172.21.15.43:0/3430790435 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f85a0013960 con 0x7f858806ba60 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:08.306 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 msgr2=0x7f858806df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 0x7f858806df20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009990 tx=0x7f85a0008040 comp rx=0 tx=0).stop 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 msgr2=0x7f85b00cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f859800d770 tx=0x7f859800dc30 comp rx=0 tx=0).stop 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 shutdown_connections 2024-01-16T15:06:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f858806ba60 0x7f858806df20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b009bda0 0x7f85b015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85b00b3b20 0x7f85b00cd7b0 secure :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f859800d770 tx=0x7f859800dc30 comp rx=0 tx=0).stop 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 --2- 172.21.15.43:0/3430790435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8590005180 0x7f85b00b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.330+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 >> 172.21.15.43:0/3430790435 conn(0x7f85b00b0ba0 msgr2=0x7f85b006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.332+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 shutdown_connections 2024-01-16T15:06:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.332+0000 7f85b8800640 1 -- 172.21.15.43:0/3430790435 wait complete. 2024-01-16T15:06:08.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.756+0000 7f0e68400640 1 Processor -- start 2024-01-16T15:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.756+0000 7f0e68400640 1 -- start start 2024-01-16T15:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.758+0000 7f0e68400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e6015d4d0 0x7f0e600cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.758+0000 7f0e68400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e600ce260 0x7f0e600d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.758+0000 7f0e68400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6015d8b0 con 0x7f0e600ce6c0 2024-01-16T15:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.758+0000 7f0e68400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e600b91b0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.758+0000 7f0e68400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e600cdfb0 con 0x7f0e600ce260 2024-01-16T15:06:08.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.759+0000 7f0e65a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e6015d4d0 0x7f0e600cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:08.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.759+0000 7f0e65a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e6015d4d0 0x7f0e600cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34396/0 (socket says 172.21.15.43:34396) 2024-01-16T15:06:08.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.759+0000 7f0e65a00640 1 -- 172.21.15.43:0/3031994012 learned_addr learned my addr 172.21.15.43:0/3031994012 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:08.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.760+0000 7f0e65a00640 1 -- 172.21.15.43:0/3031994012 >> v2:172.21.15.69:3300/0 conn(0x7f0e600ce260 msgr2=0x7f0e600d1850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:08.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.760+0000 7f0e65a00640 1 --2- 172.21.15.43:0/3031994012 >> v2:172.21.15.69:3300/0 conn(0x7f0e600ce260 0x7f0e600d1850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.760+0000 7f0e65a00640 1 -- 172.21.15.43:0/3031994012 >> v1:172.21.15.43:6789/0 conn(0x7f0e600ce6c0 legacy=0x7f0e600b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.760+0000 7f0e65a00640 1 -- 172.21.15.43:0/3031994012 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e600b2400 con 0x7f0e6015d4d0 2024-01-16T15:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.762+0000 7f0e65a00640 1 --2- 172.21.15.43:0/3031994012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600cda70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0e6006d2a0 tx=0x7f0e5002b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a3c5b1fb819140 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.762+0000 7f0e5fe00640 1 -- 172.21.15.43:0/3031994012 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50030710 con 0x7f0e6015d4d0 2024-01-16T15:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.762+0000 7f0e5fe00640 1 -- 172.21.15.43:0/3031994012 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5002e050 con 0x7f0e6015d4d0 2024-01-16T15:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.762+0000 7f0e5fe00640 1 -- 172.21.15.43:0/3031994012 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50030a90 con 0x7f0e6015d4d0 2024-01-16T15:06:08.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.763+0000 7f0e68400640 1 -- 172.21.15.43:0/3031994012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 msgr2=0x7f0e600cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.763+0000 7f0e68400640 1 --2- 172.21.15.43:0/3031994012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600cda70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0e6006d2a0 tx=0x7f0e5002b800 comp rx=0 tx=0).stop 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 -- 172.21.15.43:0/3031994012 shutdown_connections 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 --2- 172.21.15.43:0/3031994012 >> v2:172.21.15.69:3300/0 conn(0x7f0e600ce260 0x7f0e600d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 --2- 172.21.15.43:0/3031994012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600cda70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 -- 172.21.15.43:0/3031994012 >> 172.21.15.43:0/3031994012 conn(0x7f0e60071f90 msgr2=0x7f0e600723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 -- 172.21.15.43:0/3031994012 shutdown_connections 2024-01-16T15:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.764+0000 7f0e68400640 1 -- 172.21.15.43:0/3031994012 wait complete. 2024-01-16T15:06:08.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.766+0000 7f0e68400640 1 Processor -- start 2024-01-16T15:06:08.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.766+0000 7f0e68400640 1 -- start start 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.767+0000 7f0e68400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e600ce260 0x7f0e601519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.767+0000 7f0e68400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 0x7f0e60148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.767+0000 7f0e68400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.767+0000 7f0e68400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601517b0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e68400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60148a70 con 0x7f0e600ce6c0 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e68400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e600c96e0 con 0x7f0e600ce260 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e66400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:08.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e66400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34410/0 (socket says 172.21.15.43:34410) 2024-01-16T15:06:08.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e66400640 1 -- 172.21.15.43:0/2201552435 learned_addr learned my addr 172.21.15.43:0/2201552435 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:08.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.768+0000 7f0e65000640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 0x7f0e60148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:08.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 -- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e600ce260 msgr2=0x7f0e601519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e600ce260 0x7f0e601519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 -- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 msgr2=0x7f0e60148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 0x7f0e60148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 -- 172.21.15.43:0/2201552435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e500094d0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e65000640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 0x7f0e60148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.769+0000 7f0e66400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800caf0 tx=0x7f0e58002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.770+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58003da0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.770+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e58005070 con 0x7f0e6015d4d0 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.770+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58005420 con 0x7f0e6015d4d0 2024-01-16T15:06:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.771+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e600c8880 con 0x7f0e6015d4d0 2024-01-16T15:06:08.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.771+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e600c9bd0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.773+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e600b2270 con 0x7f0e6015d4d0 2024-01-16T15:06:08.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.777+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f0e58005580 con 0x7f0e6015d4d0 2024-01-16T15:06:08.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.779+0000 7f0e5d600640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 0x7f0e3806df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:08.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.779+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0e580883b0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.786+0000 7f0e65a00640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 0x7f0e3806df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:08.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.790+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e580880a0 con 0x7f0e6015d4d0 2024-01-16T15:06:08.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:08.793+0000 7f0e65a00640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 0x7f0e3806df50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e50034000 tx=0x7f0e5002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:09.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.351+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0e600c9850 con 0x7f0e6015d4d0 2024-01-16T15:06:09.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.355+0000 7f0e5d600640 1 -- 172.21.15.43:0/2201552435 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2336 (secure 0 0 0) 0x7f0e58017030 con 0x7f0e6015d4d0 2024-01-16T15:06:09.337 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:09.337 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:09.337 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:09.337 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:09.338 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 421, 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:09.339 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:09.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:09.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 800, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222215179, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 6756892672, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766337220608, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 341, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 9638, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 2 2024-01-16T15:06:09.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:09.343 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:09.344 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:09.345 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:09.346 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:09.347 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:09.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.365+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 msgr2=0x7f0e3806df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.365+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 0x7f0e3806df50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0e50034000 tx=0x7f0e5002f000 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.365+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 msgr2=0x7f0e600c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.365+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800caf0 tx=0x7f0e58002080 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 shutdown_connections 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e3806ba90 0x7f0e3806df50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e600ce6c0 0x7f0e60148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e6015d4d0 0x7f0e600c8340 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 --2- 172.21.15.43:0/2201552435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e600ce260 0x7f0e601519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.368+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 >> 172.21.15.43:0/2201552435 conn(0x7f0e60071f90 msgr2=0x7f0e600b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.370+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 shutdown_connections 2024-01-16T15:06:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:09.370+0000 7f0e68400640 1 -- 172.21.15.43:0/2201552435 wait complete. 2024-01-16T15:06:10.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:10.718 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:10.740+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:10.719 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:10.740+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:10.719 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:10.740+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:10.719 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:10.740+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:10.719 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:10.740+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.824+0000 7fdde3a00640 1 Processor -- start 2024-01-16T15:06:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.824+0000 7fdde3a00640 1 -- start start 2024-01-16T15:06:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.827+0000 7fdde3a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 0x7fdddc0b2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde3a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0ad520 con 0x7fdddc0ad9b0 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde3a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0b2e20 con 0x7fdddc06e5a0 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde3a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b3100 con 0x7fdddc0b20f0 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde1a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdddc0ad9b0 0x7fdddc0ade40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46908/0 (socket says 172.21.15.43:46908) 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde1000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdddc06e5a0 0x7fdddc156ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50838/0 (socket says 172.21.15.43:50838) 2024-01-16T15:06:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.829+0000 7fdde1000640 1 -- 172.21.15.43:0/3428300756 learned_addr learned my addr 172.21.15.43:0/3428300756 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.830+0000 7fdddbe00640 1 --2- 172.21.15.43:0/3428300756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 0x7fdddc0b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.832+0000 7fdddbe00640 1 -- 172.21.15.43:0/3428300756 >> v1:172.21.15.69:6789/0 conn(0x7fdddc06e5a0 legacy=0x7fdddc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.832+0000 7fdddbe00640 1 -- 172.21.15.43:0/3428300756 >> v1:172.21.15.43:6789/0 conn(0x7fdddc0ad9b0 legacy=0x7fdddc0ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:10.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.833+0000 7fdddbe00640 1 -- 172.21.15.43:0/3428300756 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0ad1b0 con 0x7fdddc0b20f0 2024-01-16T15:06:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.837+0000 7fdddbe00640 1 --2- 172.21.15.43:0/3428300756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 0x7fdddc0b2530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009dc0 tx=0x7fddcc02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d87281e12c622b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:10.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.839+0000 7fdddb400640 1 -- 172.21.15.43:0/3428300756 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc02caa0 con 0x7fdddc0b20f0 2024-01-16T15:06:10.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.839+0000 7fdddb400640 1 -- 172.21.15.43:0/3428300756 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc02cc00 con 0x7fdddc0b20f0 2024-01-16T15:06:10.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.839+0000 7fdddb400640 1 -- 172.21.15.43:0/3428300756 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc034440 con 0x7fdddc0b20f0 2024-01-16T15:06:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.844+0000 7fdde3a00640 1 -- 172.21.15.43:0/3428300756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 msgr2=0x7fdddc0b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.844+0000 7fdde3a00640 1 --2- 172.21.15.43:0/3428300756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 0x7fdddc0b2530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009dc0 tx=0x7fddcc02bd70 comp rx=0 tx=0).stop 2024-01-16T15:06:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.846+0000 7fdde3a00640 1 -- 172.21.15.43:0/3428300756 shutdown_connections 2024-01-16T15:06:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.846+0000 7fdde3a00640 1 --2- 172.21.15.43:0/3428300756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0b20f0 0x7fdddc0b2530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.846+0000 7fdde3a00640 1 -- 172.21.15.43:0/3428300756 >> 172.21.15.43:0/3428300756 conn(0x7fdddc0b0d80 msgr2=0x7fdddc0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.848+0000 7fdde3a00640 1 -- 172.21.15.43:0/3428300756 shutdown_connections 2024-01-16T15:06:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.850+0000 7fdde3a00640 1 -- 172.21.15.43:0/3428300756 wait complete. 2024-01-16T15:06:10.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.854+0000 7fdde3a00640 1 Processor -- start 2024-01-16T15:06:10.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.854+0000 7fdde3a00640 1 -- start start 2024-01-16T15:06:10.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 0x7fdddc0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:10.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc0b20f0 0x7fdddc14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd050 con 0x7fdddc06e5a0 2024-01-16T15:06:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc14cea0 con 0x7fdddc0b20f0 2024-01-16T15:06:10.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.856+0000 7fdde3a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc14d110 con 0x7fdddc0ad9b0 2024-01-16T15:06:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.859+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.859+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:53886/0 (socket says 172.21.15.43:53886) 2024-01-16T15:06:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.859+0000 7fdddbe00640 1 -- 172.21.15.43:0/1734405192 learned_addr learned my addr 172.21.15.43:0/1734405192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.859+0000 7fdde1000640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 0x7fdddc0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdde1000640 1 -- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 msgr2=0x7fdddc14c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdde1000640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdde1000640 1 -- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc0b20f0 msgr2=0x7fdddc14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdde1000640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc0b20f0 0x7fdddc14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdde1000640 1 -- 172.21.15.43:0/1734405192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc009a70 con 0x7fdddc06e5a0 2024-01-16T15:06:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.861+0000 7fdddbe00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.862+0000 7fdde1000640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 0x7fdddc0afda0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fddc400c470 tx=0x7fddc400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.862+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4005ce0 con 0x7fdddc06e5a0 2024-01-16T15:06:10.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.862+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4005e40 con 0x7fdddc06e5a0 2024-01-16T15:06:10.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.862+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40072d0 con 0x7fdddc06e5a0 2024-01-16T15:06:10.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.863+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc0b2bf0 con 0x7fdddc06e5a0 2024-01-16T15:06:10.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.863+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc14d6d0 con 0x7fdddc06e5a0 2024-01-16T15:06:10.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.868+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fddc4007430 con 0x7fdddc06e5a0 2024-01-16T15:06:10.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.869+0000 7fddd8c00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 0x7fddb406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:10.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.869+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fddc4087970 con 0x7fdddc06e5a0 2024-01-16T15:06:10.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.874+0000 7fdddbe00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 0x7fddb406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:10.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.877+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc074770 con 0x7fdddc06e5a0 2024-01-16T15:06:10.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.877+0000 7fdddbe00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 0x7fddb406df20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fddcc000f80 tx=0x7fddcc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:10.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:10.886+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddc4087660 con 0x7fdddc06e5a0 2024-01-16T15:06:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.105+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 17) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fddc4054550 con 0x7fdddc06e5a0 2024-01-16T15:06:11.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.358+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdddc0adf40 con 0x7fdddc06e5a0 2024-01-16T15:06:11.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.358+0000 7fddd8c00640 1 -- 172.21.15.43:0/1734405192 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2350 (secure 0 0 0) 0x7fddc407c300 con 0x7fdddc06e5a0 2024-01-16T15:06:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:11.337 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 423, 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1122, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 4556209674, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 8751120384, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 764342992896, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 119449364, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 111095425, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 29, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 34 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:11.347 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 msgr2=0x7fddb406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 0x7fddb406df20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fddcc000f80 tx=0x7fddcc030000 comp rx=0 tx=0).stop 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 msgr2=0x7fdddc0afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 0x7fdddc0afda0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fddc400c470 tx=0x7fddc400c930 comp rx=0 tx=0).stop 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 shutdown_connections 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fddb406ba60 0x7fddb406df20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdddc0b20f0 0x7fdddc14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdddc0ad9b0 0x7fdddc14c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 --2- 172.21.15.43:0/1734405192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdddc06e5a0 0x7fdddc0afda0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:11.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 >> 172.21.15.43:0/1734405192 conn(0x7fdddc0b0d80 msgr2=0x7fdddc158220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:11.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 shutdown_connections 2024-01-16T15:06:11.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:11.361+0000 7fdde3a00640 1 -- 172.21.15.43:0/1734405192 wait complete. 2024-01-16T15:06:13.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.845+0000 7f5e94c00640 1 Processor -- start 2024-01-16T15:06:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.845+0000 7f5e94c00640 1 -- start start 2024-01-16T15:06:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.848+0000 7f5e94c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e900d1f60 0x7f5e900b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.848+0000 7f5e94c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9015d8b0 con 0x7f5e9015d4d0 2024-01-16T15:06:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.848+0000 7f5e94c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900b2400 con 0x7f5e900ce260 2024-01-16T15:06:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.848+0000 7f5e94c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900b91b0 con 0x7f5e900d1f60 2024-01-16T15:06:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.849+0000 7f5e8e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e900d1f60 0x7f5e900b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.849+0000 7f5e8e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e900d1f60 0x7f5e900b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43872/0 (socket says 172.21.15.43:43872) 2024-01-16T15:06:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.849+0000 7f5e8e800640 1 -- 172.21.15.43:0/4099920580 learned_addr learned my addr 172.21.15.43:0/4099920580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.854+0000 7f5e8e800640 1 -- 172.21.15.43:0/4099920580 >> v1:172.21.15.69:6789/0 conn(0x7f5e900ce260 legacy=0x7f5e900d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.854+0000 7f5e8e800640 1 -- 172.21.15.43:0/4099920580 >> v1:172.21.15.43:6789/0 conn(0x7f5e9015d4d0 legacy=0x7f5e900cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.854+0000 7f5e8e800640 1 -- 172.21.15.43:0/4099920580 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900b26e0 con 0x7f5e900d1f60 2024-01-16T15:06:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.854+0000 7f5e8ca00640 1 -- 172.21.15.43:0/4099920580 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389739226 0 0) 0x7f5e9015d8b0 con 0x7f5e9015d4d0 2024-01-16T15:06:13.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.857+0000 7f5e8e800640 1 --2- 172.21.15.43:0/4099920580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900b4cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009e50 tx=0x7f5e8802be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d420116bba58280 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:13.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.861+0000 7f5e8ca00640 1 -- 172.21.15.43:0/4099920580 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8802c990 con 0x7f5e900d1f60 2024-01-16T15:06:13.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.861+0000 7f5e8ca00640 1 -- 172.21.15.43:0/4099920580 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8802caf0 con 0x7f5e900d1f60 2024-01-16T15:06:13.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.862+0000 7f5e8ca00640 1 -- 172.21.15.43:0/4099920580 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8802ce80 con 0x7f5e900d1f60 2024-01-16T15:06:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.863+0000 7f5e94c00640 1 -- 172.21.15.43:0/4099920580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 msgr2=0x7f5e900b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.863+0000 7f5e94c00640 1 --2- 172.21.15.43:0/4099920580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900b4cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009e50 tx=0x7f5e8802be70 comp rx=0 tx=0).stop 2024-01-16T15:06:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.865+0000 7f5e94c00640 1 -- 172.21.15.43:0/4099920580 shutdown_connections 2024-01-16T15:06:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.865+0000 7f5e94c00640 1 --2- 172.21.15.43:0/4099920580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900b4cf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.865+0000 7f5e94c00640 1 -- 172.21.15.43:0/4099920580 >> 172.21.15.43:0/4099920580 conn(0x7f5e90071f90 msgr2=0x7f5e900723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:13.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.865+0000 7f5e94c00640 1 -- 172.21.15.43:0/4099920580 shutdown_connections 2024-01-16T15:06:13.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.866+0000 7f5e94c00640 1 -- 172.21.15.43:0/4099920580 wait complete. 2024-01-16T15:06:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.877+0000 7f5e94c00640 1 Processor -- start 2024-01-16T15:06:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.877+0000 7f5e94c00640 1 -- start start 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e900ce260 0x7f5e900b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e9015d4d0 0x7f5e900cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900c9d70 con 0x7f5e9015d4d0 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e901f7080 con 0x7f5e900d1f60 2024-01-16T15:06:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.882+0000 7f5e94c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900c7180 con 0x7f5e900ce260 2024-01-16T15:06:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.884+0000 7f5e8d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.884+0000 7f5e8d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43882/0 (socket says 172.21.15.43:43882) 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.884+0000 7f5e8d400640 1 -- 172.21.15.43:0/1337220209 learned_addr learned my addr 172.21.15.43:0/1337220209 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.885+0000 7f5e8de00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e900ce260 0x7f5e900b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.886+0000 7f5e8d400640 1 -- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e900ce260 msgr2=0x7f5e900b8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.886+0000 7f5e8d400640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e900ce260 0x7f5e900b8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.886+0000 7f5e8d400640 1 -- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e9015d4d0 msgr2=0x7f5e900cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.886+0000 7f5e8d400640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e9015d4d0 0x7f5e900cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.886+0000 7f5e8d400640 1 -- 172.21.15.43:0/1337220209 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88009b30 con 0x7f5e900d1f60 2024-01-16T15:06:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.890+0000 7f5e8d400640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c520 tx=0x7f5e7800c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.894+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78005480 con 0x7f5e900d1f60 2024-01-16T15:06:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.894+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e780035f0 con 0x7f5e900d1f60 2024-01-16T15:06:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.894+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780039a0 con 0x7f5e900d1f60 2024-01-16T15:06:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.894+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900c7460 con 0x7f5e900d1f60 2024-01-16T15:06:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.894+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e900c7a20 con 0x7f5e900d1f60 2024-01-16T15:06:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.896+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e900c7c40 con 0x7f5e900d1f60 2024-01-16T15:06:13.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.900+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f5e78003b00 con 0x7f5e900d1f60 2024-01-16T15:06:13.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.902+0000 7f5e86000640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 0x7f5e6806e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:13.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.902+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5e78087ee0 con 0x7f5e900d1f60 2024-01-16T15:06:13.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.905+0000 7f5e8de00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 0x7f5e6806e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:13.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.914+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e78054950 con 0x7f5e900d1f60 2024-01-16T15:06:13.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:13.919+0000 7f5e8de00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 0x7f5e6806e030 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5e80001120 tx=0x7f5e80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:14.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.574+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5e90074770 con 0x7f5e900d1f60 2024-01-16T15:06:14.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.577+0000 7f5e86000640 1 -- 172.21.15.43:0/1337220209 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2423 (secure 0 0 0) 0x7f5e78017030 con 0x7f5e900d1f60 2024-01-16T15:06:14.563 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:14.563 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:14.563 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:14.563 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:14.564 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:14.565 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 426, 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:14.567 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1122, 2024-01-16T15:06:14.568 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 4556209674, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 8751120384, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 764342992896, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 119445161, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 111087411, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 29, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 33 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:14.569 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:14.570 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:14.571 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:14.572 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 51s)", 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-16T15:06:14.573 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.22220000624656677, 2024-01-16T15:06:14.574 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:14.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:14.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:14.574 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.587+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 msgr2=0x7f5e6806e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.587+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 0x7f5e6806e030 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5e80001120 tx=0x7f5e80009040 comp rx=0 tx=0).stop 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.587+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 msgr2=0x7f5e900c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.587+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c520 tx=0x7f5e7800c9e0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 shutdown_connections 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5e6806bb70 0x7f5e6806e030 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e900d1f60 0x7f5e900c9830 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e9015d4d0 0x7f5e900cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 --2- 172.21.15.43:0/1337220209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e900ce260 0x7f5e900b8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.588+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 >> 172.21.15.43:0/1337220209 conn(0x7f5e90071f90 msgr2=0x7f5e9015a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.591+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 shutdown_connections 2024-01-16T15:06:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:14.593+0000 7f5e94c00640 1 -- 172.21.15.43:0/1337220209 wait complete. 2024-01-16T15:06:15.665 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:15.686+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:15.665 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:15.686+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:15.665 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:15.686+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:15.665 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:15.686+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:15.665 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:15.686+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:16.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:17.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.109+0000 7fcc3e800640 1 Processor -- start 2024-01-16T15:06:17.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.109+0000 7fcc3e800640 1 -- start start 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc3e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc380b39a0 0x7fcc38071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc3e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc380abff0 0x7fcc38071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc3e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3813da30 con 0x7fcc3815a100 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc3e800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc380ac840 con 0x7fcc380b39a0 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc3e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc380ac560 con 0x7fcc380abff0 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.111+0000 7fcc36a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc380b39a0 0x7fcc38071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc380b39a0 0x7fcc38071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40530/0 (socket says 172.21.15.43:40530) 2024-01-16T15:06:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 -- 172.21.15.43:0/2823902709 learned_addr learned my addr 172.21.15.43:0/2823902709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:17.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 -- 172.21.15.43:0/2823902709 >> v2:172.21.15.69:3300/0 conn(0x7fcc380abff0 msgr2=0x7fcc38071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:17.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 --2- 172.21.15.43:0/2823902709 >> v2:172.21.15.69:3300/0 conn(0x7fcc380abff0 0x7fcc38071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 -- 172.21.15.43:0/2823902709 >> v1:172.21.15.69:6789/0 conn(0x7fcc3815a100 legacy=0x7fcc380b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:17.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.112+0000 7fcc36a00640 1 -- 172.21.15.43:0/2823902709 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38070650 con 0x7fcc380b39a0 2024-01-16T15:06:17.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.114+0000 7fcc36a00640 1 --2- 172.21.15.43:0/2823902709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc380b39a0 0x7fcc38071130 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcc380b3e50 tx=0x7fcc2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894da4a86bb62a97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.115+0000 7fcc36000640 1 -- 172.21.15.43:0/2823902709 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20031870 con 0x7fcc380b39a0 2024-01-16T15:06:17.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.115+0000 7fcc36000640 1 -- 172.21.15.43:0/2823902709 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc200319d0 con 0x7fcc380b39a0 2024-01-16T15:06:17.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.115+0000 7fcc36000640 1 -- 172.21.15.43:0/2823902709 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20031d60 con 0x7fcc380b39a0 2024-01-16T15:06:17.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.116+0000 7fcc3e800640 1 -- 172.21.15.43:0/2823902709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc380b39a0 msgr2=0x7fcc38071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.116+0000 7fcc3e800640 1 --2- 172.21.15.43:0/2823902709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc380b39a0 0x7fcc38071130 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcc380b3e50 tx=0x7fcc2002b910 comp rx=0 tx=0).stop 2024-01-16T15:06:17.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 -- 172.21.15.43:0/2823902709 shutdown_connections 2024-01-16T15:06:17.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 --2- 172.21.15.43:0/2823902709 >> v2:172.21.15.69:3300/0 conn(0x7fcc380abff0 0x7fcc38071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 --2- 172.21.15.43:0/2823902709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc380b39a0 0x7fcc38071130 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 -- 172.21.15.43:0/2823902709 >> 172.21.15.43:0/2823902709 conn(0x7fcc380b0d80 msgr2=0x7fcc380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 -- 172.21.15.43:0/2823902709 shutdown_connections 2024-01-16T15:06:17.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.117+0000 7fcc3e800640 1 -- 172.21.15.43:0/2823902709 wait complete. 2024-01-16T15:06:17.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.120+0000 7fcc3e800640 1 Processor -- start 2024-01-16T15:06:17.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.121+0000 7fcc3e800640 1 -- start start 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 0x7fcc38150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 0x7fcc38167620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 0x7fcc38155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380b0020 con 0x7fcc3815a100 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38167b60 con 0x7fcc380abff0 2024-01-16T15:06:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.123+0000 7fcc3e800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380af3a0 con 0x7fcc380b39a0 2024-01-16T15:06:17.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.124+0000 7fcc37e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 0x7fcc38155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.125+0000 7fcc36a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 0x7fcc38167620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.125+0000 7fcc36a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 0x7fcc38167620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46292/0 (socket says 172.21.15.43:46292) 2024-01-16T15:06:17.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.125+0000 7fcc36a00640 1 -- 172.21.15.43:0/4181778432 learned_addr learned my addr 172.21.15.43:0/4181778432 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:17.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.125+0000 7fcc37400640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 0x7fcc38150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37400640 1 -- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 msgr2=0x7fcc38167620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37400640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 0x7fcc38167620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37400640 1 -- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 msgr2=0x7fcc38155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37400640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 0x7fcc38155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37400640 1 -- 172.21.15.43:0/4181778432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200094d0 con 0x7fcc380abff0 2024-01-16T15:06:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.126+0000 7fcc37e00640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 0x7fcc38155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.127+0000 7fcc37400640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 0x7fcc38150040 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c410 tx=0x7fcc2800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.127+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc280035f0 con 0x7fcc380abff0 2024-01-16T15:06:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.128+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28003770 con 0x7fcc380abff0 2024-01-16T15:06:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.128+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28003b60 con 0x7fcc380abff0 2024-01-16T15:06:17.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.129+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc380af680 con 0x7fcc380abff0 2024-01-16T15:06:17.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.129+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc380afc40 con 0x7fcc380abff0 2024-01-16T15:06:17.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.131+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc380ad910 con 0x7fcc380abff0 2024-01-16T15:06:17.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.135+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fcc28003cc0 con 0x7fcc380abff0 2024-01-16T15:06:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.137+0000 7fcc2f400640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 0x7fcc1006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.137+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcc28087890 con 0x7fcc380abff0 2024-01-16T15:06:17.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.143+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc28054130 con 0x7fcc380abff0 2024-01-16T15:06:17.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.144+0000 7fcc36a00640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 0x7fcc1006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:17.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.220+0000 7fcc36a00640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 0x7fcc1006dff0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc380ad760 tx=0x7fcc2003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.707+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcc38074770 con 0x7fcc380abff0 2024-01-16T15:06:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.709+0000 7fcc2f400640 1 -- 172.21.15.43:0/4181778432 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7fcc28017090 con 0x7fcc380abff0 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:17.688 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 430, 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:17.689 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:17.690 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1595, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6544433811, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12720947200, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760373166080, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 290773501, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 276497231, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 70, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 77 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:17.691 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:17.692 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:17.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (10s)\n [==..........................] (remaining: 101s)", 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (10s)\n [====........................] (remaining: 53s)", 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:06:17.694 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:17.695 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:17.695 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:17.695 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 msgr2=0x7fcc1006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 0x7fcc1006dff0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc380ad760 tx=0x7fcc2003c040 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 msgr2=0x7fcc38150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 0x7fcc38150040 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c410 tx=0x7fcc2800c810 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 shutdown_connections 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc1006bb30 0x7fcc1006dff0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc3815a100 0x7fcc38155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc380b39a0 0x7fcc38167620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 --2- 172.21.15.43:0/4181778432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc380abff0 0x7fcc38150040 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 >> 172.21.15.43:0/4181778432 conn(0x7fcc380b0d80 msgr2=0x7fcc38152580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 shutdown_connections 2024-01-16T15:06:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:17.713+0000 7fcc3e800640 1 -- 172.21.15.43:0/4181778432 wait complete. 2024-01-16T15:06:19.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:20.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.218+0000 7f88a6000640 1 Processor -- start 2024-01-16T15:06:20.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.218+0000 7f88a6000640 1 -- start start 2024-01-16T15:06:20.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.221+0000 7f88a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.222+0000 7f88a6000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a015d640 con 0x7f88a015d260 2024-01-16T15:06:20.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.222+0000 7f88a6000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00acf80 con 0x7f88a0160d80 2024-01-16T15:06:20.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.226+0000 7f889fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.226+0000 7f889fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46302/0 (socket says 172.21.15.43:46302) 2024-01-16T15:06:20.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.226+0000 7f889fe00640 1 -- 172.21.15.43:0/2275685242 learned_addr learned my addr 172.21.15.43:0/2275685242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:20.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.226+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00b2390 con 0x7f88a0071c90 2024-01-16T15:06:20.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.229+0000 7f889f400640 1 -- 172.21.15.43:0/2275685242 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745243102 0 0) 0x7f88a00acf80 con 0x7f88a0160d80 2024-01-16T15:06:20.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.229+0000 7f889f400640 1 -- 172.21.15.43:0/2275685242 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880003520 con 0x7f88a0160d80 2024-01-16T15:06:20.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.229+0000 7f889fe00640 1 -- 172.21.15.43:0/2275685242 >> v1:172.21.15.69:6789/0 conn(0x7f88a0160d80 legacy=0x7f88a006d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.229+0000 7f889fe00640 1 -- 172.21.15.43:0/2275685242 >> v1:172.21.15.43:6789/0 conn(0x7f88a015d260 legacy=0x7f88a00ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.229+0000 7f889fe00640 1 -- 172.21.15.43:0/2275685242 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a0072270 con 0x7f88a0071c90 2024-01-16T15:06:20.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.234+0000 7f889fe00640 1 --2- 172.21.15.43:0/2275685242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f88900098d0 tx=0x7f889002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fab40d1e9aded7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.237+0000 7f889f400640 1 -- 172.21.15.43:0/2275685242 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890030840 con 0x7f88a0071c90 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.237+0000 7f889f400640 1 -- 172.21.15.43:0/2275685242 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900309a0 con 0x7f88a0071c90 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.237+0000 7f889f400640 1 -- 172.21.15.43:0/2275685242 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890030d10 con 0x7f88a0071c90 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.239+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 msgr2=0x7f88a00b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.239+0000 7f88a6000640 1 --2- 172.21.15.43:0/2275685242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f88900098d0 tx=0x7f889002b980 comp rx=0 tx=0).stop 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.243+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 shutdown_connections 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.243+0000 7f88a6000640 1 --2- 172.21.15.43:0/2275685242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a00b1e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.243+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 >> 172.21.15.43:0/2275685242 conn(0x7f88a00b0840 msgr2=0x7f88a00b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.243+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 shutdown_connections 2024-01-16T15:06:20.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.243+0000 7f88a6000640 1 -- 172.21.15.43:0/2275685242 wait complete. 2024-01-16T15:06:20.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.246+0000 7f88a6000640 1 Processor -- start 2024-01-16T15:06:20.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.247+0000 7f88a6000640 1 -- start start 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.250+0000 7f88a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a01418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.251+0000 7f88a6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 0x7f88a0141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.251+0000 7f88a6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 0x7f88a0144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.251+0000 7f88a6000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a015a290 con 0x7f88a015d260 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.251+0000 7f88a6000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a015f940 con 0x7f88a0160d80 2024-01-16T15:06:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.251+0000 7f88a6000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0141650 con 0x7f88a0071c90 2024-01-16T15:06:20.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.253+0000 7f88a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 0x7f88a0144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.253+0000 7f88a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 0x7f88a0144340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43912/0 (socket says 172.21.15.43:43912) 2024-01-16T15:06:20.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.253+0000 7f88a5600640 1 -- 172.21.15.43:0/2114587464 learned_addr learned my addr 172.21.15.43:0/2114587464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:20.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.254+0000 7f889fe00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 0x7f88a0141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.256+0000 7f889fe00640 1 -- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 msgr2=0x7f88a01418e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.256+0000 7f889fe00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a01418e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.256+0000 7f889fe00640 1 -- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 msgr2=0x7f88a0144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.256+0000 7f889fe00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 0x7f88a0144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.256+0000 7f889fe00640 1 -- 172.21.15.43:0/2114587464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900094d0 con 0x7f88a015d260 2024-01-16T15:06:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.258+0000 7f889fe00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 0x7f88a0141110 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8890033000 tx=0x7f88900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.259+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890003610 con 0x7f88a015d260 2024-01-16T15:06:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.259+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890003770 con 0x7f88a015d260 2024-01-16T15:06:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.259+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890003b20 con 0x7f88a015d260 2024-01-16T15:06:20.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.259+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a0151d20 con 0x7f88a015d260 2024-01-16T15:06:20.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.259+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0144880 con 0x7f88a015d260 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.263+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a015fa80 con 0x7f88a015d260 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.264+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f8890003c80 con 0x7f88a015d260 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.266+0000 7f889cc00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 0x7f887806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.267+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f88900aa400 con 0x7f88a015d260 2024-01-16T15:06:20.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.269+0000 7f88a4c00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 0x7f887806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.270+0000 7f88a4c00640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 0x7f887806e040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88980012e0 tx=0x7f8898007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.278+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88a015fa80 con 0x7f88a015d260 2024-01-16T15:06:20.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.953+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f88a0151d20 con 0x7f88a015d260 2024-01-16T15:06:20.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.958+0000 7f889cc00640 1 -- 172.21.15.43:0/2114587464 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f88a0151d20 con 0x7f88a015d260 2024-01-16T15:06:20.940 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:20.940 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:20.940 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:20.940 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:20.941 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 433, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1595, 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6544433811, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12720947200, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760373166080, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 290767795, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 276486598, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 70, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 75 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:20.944 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:20.945 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 38s)", 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.23810000717639923, 2024-01-16T15:06:20.946 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.41667500138282776, 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:20.947 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 msgr2=0x7f887806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 0x7f887806e040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88980012e0 tx=0x7f8898007d30 comp rx=0 tx=0).stop 2024-01-16T15:06:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 msgr2=0x7f88a0141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 0x7f88a0141110 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8890033000 tx=0x7f88900046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 shutdown_connections 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f887806bb80 0x7f887806e040 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88980012e0 tx=0x7f8898007d30 comp rx=0 tx=0).stop 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88a0160d80 0x7f88a0144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88a015d260 0x7f88a0141110 secure :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8890033000 tx=0x7f88900046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 --2- 172.21.15.43:0/2114587464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88a0071c90 0x7f88a01418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.977+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 >> 172.21.15.43:0/2114587464 conn(0x7f88a00b0840 msgr2=0x7f88a0156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.980+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 shutdown_connections 2024-01-16T15:06:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:20.982+0000 7f88a6000640 1 -- 172.21.15.43:0/2114587464 wait complete. 2024-01-16T15:06:23.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.501+0000 7faa6e000640 1 Processor -- start 2024-01-16T15:06:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.501+0000 7faa6e000640 1 -- start start 2024-01-16T15:06:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.506+0000 7faa6e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa680ce260 0x7faa680d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.506+0000 7faa6e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6815d8b0 con 0x7faa680ce6a0 2024-01-16T15:06:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.506+0000 7faa6e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680b2400 con 0x7faa6815d4d0 2024-01-16T15:06:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.506+0000 7faa6e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa680b91b0 con 0x7faa680ce260 2024-01-16T15:06:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.508+0000 7faa67400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7faa6815d4d0 0x7faa680cda70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:42392/0 (socket says 172.21.15.43:42392) 2024-01-16T15:06:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.508+0000 7faa67400640 1 -- 172.21.15.43:0/1502193769 learned_addr learned my addr 172.21.15.43:0/1502193769 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.511+0000 7faa5fe00640 1 --2- 172.21.15.43:0/1502193769 >> v2:172.21.15.69:3300/0 conn(0x7faa680ce260 0x7faa680d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.512+0000 7faa5fe00640 1 -- 172.21.15.43:0/1502193769 >> v1:172.21.15.69:6789/0 conn(0x7faa6815d4d0 legacy=0x7faa680cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.512+0000 7faa5fe00640 1 -- 172.21.15.43:0/1502193769 >> v1:172.21.15.43:6789/0 conn(0x7faa680ce6a0 legacy=0x7faa680b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.512+0000 7faa5fe00640 1 -- 172.21.15.43:0/1502193769 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680b26e0 con 0x7faa680ce260 2024-01-16T15:06:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.514+0000 7faa5fe00640 1 --2- 172.21.15.43:0/1502193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa680ce260 0x7faa680d1850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faa500097a0 tx=0x7faa5002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a8dd0549e9ac8ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.515+0000 7faa66a00640 1 -- 172.21.15.43:0/1502193769 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5002e070 con 0x7faa680ce260 2024-01-16T15:06:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.516+0000 7faa66a00640 1 -- 172.21.15.43:0/1502193769 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa500315a0 con 0x7faa680ce260 2024-01-16T15:06:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.517+0000 7faa6e000640 1 -- 172.21.15.43:0/1502193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa680ce260 msgr2=0x7faa680d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.517+0000 7faa66a00640 1 -- 172.21.15.43:0/1502193769 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5002e070 con 0x7faa680ce260 2024-01-16T15:06:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.518+0000 7faa6e000640 1 --2- 172.21.15.43:0/1502193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa680ce260 0x7faa680d1850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faa500097a0 tx=0x7faa5002b840 comp rx=0 tx=0).stop 2024-01-16T15:06:23.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.518+0000 7faa6e000640 1 -- 172.21.15.43:0/1502193769 shutdown_connections 2024-01-16T15:06:23.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.518+0000 7faa6e000640 1 --2- 172.21.15.43:0/1502193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa680ce260 0x7faa680d1850 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.518+0000 7faa6e000640 1 -- 172.21.15.43:0/1502193769 >> 172.21.15.43:0/1502193769 conn(0x7faa68071f90 msgr2=0x7faa680723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:23.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.519+0000 7faa6e000640 1 -- 172.21.15.43:0/1502193769 shutdown_connections 2024-01-16T15:06:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.519+0000 7faa6e000640 1 -- 172.21.15.43:0/1502193769 wait complete. 2024-01-16T15:06:23.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.525+0000 7faa6e000640 1 Processor -- start 2024-01-16T15:06:23.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.525+0000 7faa6e000640 1 -- start start 2024-01-16T15:06:23.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.528+0000 7faa6e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa680ce260 0x7faa6815f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:23.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.529+0000 7faa6e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:23.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.530+0000 7faa5fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:23.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.530+0000 7faa6e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:23.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.530+0000 7faa5fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45916/0 (socket says 172.21.15.43:45916) 2024-01-16T15:06:23.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.531+0000 7faa67400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa680ce260 0x7faa6815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:23.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.531+0000 7faa67e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:23.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.531+0000 7faa67e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:55726/0 (socket says 172.21.15.43:55726) 2024-01-16T15:06:23.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.532+0000 7faa6e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680caad0 con 0x7faa680ce6a0 2024-01-16T15:06:23.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.532+0000 7faa6e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681f6e90 con 0x7faa6815d4d0 2024-01-16T15:06:23.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.532+0000 7faa6e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa681f5f70 con 0x7faa680ce260 2024-01-16T15:06:23.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.532+0000 7faa5fe00640 1 -- 172.21.15.43:0/624884211 learned_addr learned my addr 172.21.15.43:0/624884211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa67e00640 1 -- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa680ce260 msgr2=0x7faa6815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa67e00640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa680ce260 0x7faa6815f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa67e00640 1 -- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 msgr2=0x7faa681516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa67e00640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa67e00640 1 -- 172.21.15.43:0/624884211 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa50009400 con 0x7faa6815d4d0 2024-01-16T15:06:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.536+0000 7faa5fe00640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.538+0000 7faa67e00640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faa6000c780 tx=0x7faa6000cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.540+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60003b70 con 0x7faa6815d4d0 2024-01-16T15:06:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.540+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa681f6210 con 0x7faa6815d4d0 2024-01-16T15:06:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.541+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa681f67d0 con 0x7faa6815d4d0 2024-01-16T15:06:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.544+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa68074770 con 0x7faa6815d4d0 2024-01-16T15:06:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.545+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60003cd0 con 0x7faa6815d4d0 2024-01-16T15:06:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.545+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60003b70 con 0x7faa6815d4d0 2024-01-16T15:06:23.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.556+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7faa60005070 con 0x7faa6815d4d0 2024-01-16T15:06:23.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.558+0000 7faa5f400640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 0x7faa4006e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:23.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.558+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faa6008a6a0 con 0x7faa6815d4d0 2024-01-16T15:06:23.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.560+0000 7faa67400640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 0x7faa4006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:23.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.562+0000 7faa67400640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 0x7faa4006e050 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa5800b830 tx=0x7faa58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:23.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:23.567+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa60054640 con 0x7faa6815d4d0 2024-01-16T15:06:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.235+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faa68154a00 con 0x7faa6815d4d0 2024-01-16T15:06:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.239+0000 7faa5f400640 1 -- 172.21.15.43:0/624884211 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7faa60017030 con 0x7faa6815d4d0 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:24.219 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:24.220 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 436, 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:24.221 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:24.222 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:24.223 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2243, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 9262463064, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 18421915648, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 754672197632, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 390940424, 2024-01-16T15:06:24.224 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 391660688, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 93, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 100 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:24.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:24.226 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:24.227 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 38s)", 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.23810000717639923, 2024-01-16T15:06:24.228 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.41667500138282776, 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:24.229 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 msgr2=0x7faa4006e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 0x7faa4006e050 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa5800b830 tx=0x7faa58008040 comp rx=0 tx=0).stop 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 msgr2=0x7faa680b7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faa6000c780 tx=0x7faa6000cc40 comp rx=0 tx=0).stop 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 shutdown_connections 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa4006bb90 0x7faa4006e050 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa6815d4d0 0x7faa680b7ff0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:24.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa680ce6a0 0x7faa681516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:24.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 --2- 172.21.15.43:0/624884211 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa680ce260 0x7faa6815f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:24.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.250+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 >> 172.21.15.43:0/624884211 conn(0x7faa68071f90 msgr2=0x7faa6815a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.259+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 shutdown_connections 2024-01-16T15:06:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:24.262+0000 7faa6e000640 1 -- 172.21.15.43:0/624884211 wait complete. 2024-01-16T15:06:25.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:25.672 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:26.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:26.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.707+0000 7f48bb400640 1 Processor -- start 2024-01-16T15:06:26.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.710+0000 7f48bb400640 1 -- start start 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.714+0000 7f48bb400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 0x7f48b40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.714+0000 7f48bb400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48b40abff0 0x7f48b40b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.714+0000 7f48bb400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b415c360 con 0x7f48b406d520 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.714+0000 7f48bb400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b40b41a0 con 0x7f48b40abff0 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.714+0000 7f48bb400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b415ba30 con 0x7f48b40ad720 2024-01-16T15:06:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.716+0000 7f48b8a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f48b406d520 0x7f48b406d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48802/0 (socket says 172.21.15.43:48802) 2024-01-16T15:06:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.716+0000 7f48b8a00640 1 -- 172.21.15.43:0/2653741233 learned_addr learned my addr 172.21.15.43:0/2653741233 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.717+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2653741233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 0x7f48b40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.718+0000 7f48b9400640 1 --2- 172.21.15.43:0/2653741233 >> v2:172.21.15.69:3300/0 conn(0x7f48b40abff0 0x7f48b40b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.718+0000 7f48b3400640 1 -- 172.21.15.43:0/2653741233 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399663639 0 0) 0x7f48b415c360 con 0x7f48b406d520 2024-01-16T15:06:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.718+0000 7f48b3400640 1 -- 172.21.15.43:0/2653741233 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4894003520 con 0x7f48b406d520 2024-01-16T15:06:26.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.718+0000 7f48b3e00640 1 -- 172.21.15.43:0/2653741233 >> v2:172.21.15.69:3300/0 conn(0x7f48b40abff0 msgr2=0x7f48b40b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.718+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2653741233 >> v2:172.21.15.69:3300/0 conn(0x7f48b40abff0 0x7f48b40b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.719+0000 7f48b3e00640 1 -- 172.21.15.43:0/2653741233 >> v1:172.21.15.43:6789/0 conn(0x7f48b406d520 legacy=0x7f48b406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.719+0000 7f48b3e00640 1 -- 172.21.15.43:0/2653741233 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b415bd10 con 0x7f48b40ad720 2024-01-16T15:06:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.721+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2653741233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 0x7f48b40b2080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f489c0097a0 tx=0x7f489c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8c91f9db57eccf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.723+0000 7f48b3400640 1 -- 172.21.15.43:0/2653741233 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c031640 con 0x7f48b40ad720 2024-01-16T15:06:26.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.723+0000 7f48b3400640 1 -- 172.21.15.43:0/2653741233 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0317a0 con 0x7f48b40ad720 2024-01-16T15:06:26.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.723+0000 7f48b3400640 1 -- 172.21.15.43:0/2653741233 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c031b50 con 0x7f48b40ad720 2024-01-16T15:06:26.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.724+0000 7f48bb400640 1 -- 172.21.15.43:0/2653741233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 msgr2=0x7f48b40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.724+0000 7f48bb400640 1 --2- 172.21.15.43:0/2653741233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 0x7f48b40b2080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f489c0097a0 tx=0x7f489c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.725+0000 7f48bb400640 1 -- 172.21.15.43:0/2653741233 shutdown_connections 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.725+0000 7f48bb400640 1 --2- 172.21.15.43:0/2653741233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40ad720 0x7f48b40b2080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.725+0000 7f48bb400640 1 --2- 172.21.15.43:0/2653741233 >> v2:172.21.15.69:3300/0 conn(0x7f48b40abff0 0x7f48b40b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.725+0000 7f48bb400640 1 -- 172.21.15.43:0/2653741233 >> 172.21.15.43:0/2653741233 conn(0x7f48b40b0d80 msgr2=0x7f48b40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.726+0000 7f48bb400640 1 -- 172.21.15.43:0/2653741233 shutdown_connections 2024-01-16T15:06:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.726+0000 7f48bb400640 1 -- 172.21.15.43:0/2653741233 wait complete. 2024-01-16T15:06:26.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.731+0000 7f48bb400640 1 Processor -- start 2024-01-16T15:06:26.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.731+0000 7f48bb400640 1 -- start start 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 0x7f48b40cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b40ad720 0x7f48b40704b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b40c9ac0 con 0x7f48b406d520 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b41f6160 con 0x7f48b40ad720 2024-01-16T15:06:26.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.733+0000 7f48bb400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b41f63b0 con 0x7f48b40abff0 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.734+0000 7f48b8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 0x7f48b40cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.734+0000 7f48b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.734+0000 7f48b3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:40750/0 (socket says 172.21.15.43:40750) 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.734+0000 7f48b3e00640 1 -- 172.21.15.43:0/2777822906 learned_addr learned my addr 172.21.15.43:0/2777822906 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.734+0000 7f48b9400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b40ad720 0x7f48b40704b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b3e00640 1 -- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b40ad720 msgr2=0x7f48b40704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b40ad720 0x7f48b40704b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b3e00640 1 -- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 msgr2=0x7f48b40cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 0x7f48b40cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b3e00640 1 -- 172.21.15.43:0/2777822906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c009400 con 0x7f48b40abff0 2024-01-16T15:06:26.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.735+0000 7f48b8a00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 0x7f48b40cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:26.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.736+0000 7f48b3e00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f489c000c00 tx=0x7f489c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.736+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c003810 con 0x7f48b40abff0 2024-01-16T15:06:26.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.736+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c03f070 con 0x7f48b40abff0 2024-01-16T15:06:26.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.736+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c003bb0 con 0x7f48b40abff0 2024-01-16T15:06:26.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.737+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b41f6690 con 0x7f48b40abff0 2024-01-16T15:06:26.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.737+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b41f6c50 con 0x7f48b40abff0 2024-01-16T15:06:26.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.739+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f489c02f050 con 0x7f48b40abff0 2024-01-16T15:06:26.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.742+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b4074770 con 0x7f48b40abff0 2024-01-16T15:06:26.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.743+0000 7f48b0c00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 0x7f488c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.744+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f489c041070 con 0x7f48b40abff0 2024-01-16T15:06:26.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.745+0000 7f48b8a00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 0x7f488c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.752+0000 7f48b8a00640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 0x7f488c06e040 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f48a4007fd0 tx=0x7f48a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:26.755+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f489c076dc0 con 0x7f48b40abff0 2024-01-16T15:06:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.183+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f48b41f64f0 con 0x7f48b40abff0 2024-01-16T15:06:27.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.184+0000 7f48b0c00640 1 -- 172.21.15.43:0/2777822906 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f489c035980 con 0x7f48b40abff0 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:27.163 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 439, 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:27.164 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:27.165 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 3073, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 12731316366, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 25277521920, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 747816591360, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 679058047, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 680141531, 2024-01-16T15:06:27.166 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 162, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 173 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:27.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (19s)\n [============................] (remaining: 26s)", 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.4285714328289032, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (19s)\n [==============..............] (remaining: 19s)", 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.5, 2024-01-16T15:06:27.169 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:27.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:27.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:27.170 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 msgr2=0x7f488c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 0x7f488c06e040 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f48a4007fd0 tx=0x7f48a4009040 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 msgr2=0x7f48b415ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f489c000c00 tx=0x7f489c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 shutdown_connections 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f488c06bb80 0x7f488c06e040 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b40ad720 0x7f48b40704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b40abff0 0x7f48b415ab30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 --2- 172.21.15.43:0/2777822906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f48b406d520 0x7f48b40cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 >> 172.21.15.43:0/2777822906 conn(0x7f48b40b0d80 msgr2=0x7f48b40acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 shutdown_connections 2024-01-16T15:06:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:27.187+0000 7f48bb400640 1 -- 172.21.15.43:0/2777822906 wait complete. 2024-01-16T15:06:29.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.613+0000 7fa967e00640 1 Processor -- start 2024-01-16T15:06:29.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.617+0000 7fa967e00640 1 -- start start 2024-01-16T15:06:29.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.619+0000 7fa967e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa960005eb0 0x7fa9600062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.619+0000 7fa967e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa960006830 0x7fa960006cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.619+0000 7fa967e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa960054300 con 0x7fa9600eacd0 2024-01-16T15:06:29.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.619+0000 7fa967e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9600536f0 con 0x7fa960005eb0 2024-01-16T15:06:29.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.619+0000 7fa967e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600539d0 con 0x7fa960006830 2024-01-16T15:06:29.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.622+0000 7fa966000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa960005eb0 0x7fa9600062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.622+0000 7fa966000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa960005eb0 0x7fa9600062f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:55756/0 (socket says 172.21.15.43:55756) 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.622+0000 7fa966a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa9600eacd0 0x7fa96004fa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:42406/0 (socket says 172.21.15.43:42406) 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.622+0000 7fa966a00640 1 -- 172.21.15.43:0/730921480 learned_addr learned my addr 172.21.15.43:0/730921480 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:29.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.623+0000 7fa967400640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa960006830 0x7fa960006cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.627+0000 7fa965600640 1 -- 172.21.15.43:0/730921480 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18596018 0 0) 0x7fa960054300 con 0x7fa9600eacd0 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.627+0000 7fa965600640 1 -- 172.21.15.43:0/730921480 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa948003520 con 0x7fa9600eacd0 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.629+0000 7fa966000640 1 -- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa960006830 msgr2=0x7fa960006cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.629+0000 7fa966000640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa960006830 0x7fa960006cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.629+0000 7fa966000640 1 -- 172.21.15.43:0/730921480 >> v1:172.21.15.69:6789/0 conn(0x7fa9600eacd0 legacy=0x7fa96004fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.629+0000 7fa966000640 1 -- 172.21.15.43:0/730921480 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960053cb0 con 0x7fa960005eb0 2024-01-16T15:06:29.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.636+0000 7fa966000640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960005eb0 0x7fa9600062f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9500097a0 tx=0x7fa95002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b986fbe333ed05c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.638+0000 7fa965600640 1 -- 172.21.15.43:0/730921480 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95002d040 con 0x7fa960005eb0 2024-01-16T15:06:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.638+0000 7fa965600640 1 -- 172.21.15.43:0/730921480 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa950031810 con 0x7fa960005eb0 2024-01-16T15:06:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.638+0000 7fa965600640 1 -- 172.21.15.43:0/730921480 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95002d040 con 0x7fa960005eb0 2024-01-16T15:06:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 -- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960005eb0 msgr2=0x7fa9600062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960005eb0 0x7fa9600062f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9500097a0 tx=0x7fa95002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 -- 172.21.15.43:0/730921480 shutdown_connections 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960005eb0 0x7fa9600062f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 --2- 172.21.15.43:0/730921480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa960006830 0x7fa960006cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.640+0000 7fa967e00640 1 -- 172.21.15.43:0/730921480 >> 172.21.15.43:0/730921480 conn(0x7fa960009f80 msgr2=0x7fa96000a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.641+0000 7fa967e00640 1 -- 172.21.15.43:0/730921480 shutdown_connections 2024-01-16T15:06:29.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.643+0000 7fa967e00640 1 -- 172.21.15.43:0/730921480 wait complete. 2024-01-16T15:06:29.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.645+0000 7fa967e00640 1 Processor -- start 2024-01-16T15:06:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.646+0000 7fa967e00640 1 -- start start 2024-01-16T15:06:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.648+0000 7fa967e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 0x7fa9600dd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.649+0000 7fa967e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960006830 0x7fa960038ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.649+0000 7fa967e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 0x7fa960039530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.649+0000 7fa966a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 0x7fa9600dd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa966a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 0x7fa9600dd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45934/0 (socket says 172.21.15.43:45934) 2024-01-16T15:06:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa966a00640 1 -- 172.21.15.43:0/3948400629 learned_addr learned my addr 172.21.15.43:0/3948400629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa967400640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 0x7fa960039530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600dc220 con 0x7fa960005eb0 2024-01-16T15:06:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600dc500 con 0x7fa960006830 2024-01-16T15:06:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.650+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600dc7e0 con 0x7fa9600eacd0 2024-01-16T15:06:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.651+0000 7fa966000640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960006830 0x7fa960038ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 -- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960006830 msgr2=0x7fa960038ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960006830 0x7fa960038ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 -- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 msgr2=0x7fa9600dd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 0x7fa9600dd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa950009400 con 0x7fa9600eacd0 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.652+0000 7fa967400640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 0x7fa960039530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa968069dd0 tx=0x7fa968067fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:29.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.653+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96806a440 con 0x7fa9600eacd0 2024-01-16T15:06:29.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.653+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96806a5a0 con 0x7fa9600eacd0 2024-01-16T15:06:29.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.653+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96806a930 con 0x7fa9600eacd0 2024-01-16T15:06:29.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.653+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9600dcac0 con 0x7fa9600eacd0 2024-01-16T15:06:29.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.653+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa960035c50 con 0x7fa9600eacd0 2024-01-16T15:06:29.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.656+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9600dcfe0 con 0x7fa9600eacd0 2024-01-16T15:06:29.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.660+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa96806aa90 con 0x7fa9600eacd0 2024-01-16T15:06:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.664+0000 7fa95ea00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 0x7fa94006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:29.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.665+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa968102620 con 0x7fa9600eacd0 2024-01-16T15:06:29.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.671+0000 7fa966a00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 0x7fa94006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:29.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.671+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9680cedc0 con 0x7fa9600eacd0 2024-01-16T15:06:29.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:29.681+0000 7fa966a00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 0x7fa94006e0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa9580011e0 tx=0x7fa958009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.253+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa9600dcfe0 con 0x7fa9600eacd0 2024-01-16T15:06:30.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.255+0000 7fa95ea00640 1 -- 172.21.15.43:0/3948400629 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7fa9600dcfe0 con 0x7fa9600eacd0 2024-01-16T15:06:30.234 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:30.234 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:30.234 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:30.234 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:30.235 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 442, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:30.236 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 3073, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 12731316366, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 25277521920, 2024-01-16T15:06:30.237 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 747816591360, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 507953986, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 514961649, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 121, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 130 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:30.238 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:30.239 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 47.62% (21s)\n [=============...............] (remaining: 23s)", 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.47620001435279846, 2024-01-16T15:06:30.240 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 66.67% (21s)\n [==================..........] (remaining: 10s)", 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.66670000553131104, 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:30.241 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:30.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 msgr2=0x7fa94006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 0x7fa94006e0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa9580011e0 tx=0x7fa958009040 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 msgr2=0x7fa960039530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 0x7fa960039530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa968069dd0 tx=0x7fa968067fe0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 shutdown_connections 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa94006bc00 0x7fa94006e0c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa960006830 0x7fa960038ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9600eacd0 0x7fa960039530 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 --2- 172.21.15.43:0/3948400629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa960005eb0 0x7fa9600dd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.262+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 >> 172.21.15.43:0/3948400629 conn(0x7fa960009f80 msgr2=0x7fa96004f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.263+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 shutdown_connections 2024-01-16T15:06:30.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:30.263+0000 7fa967e00640 1 -- 172.21.15.43:0/3948400629 wait complete. 2024-01-16T15:06:32.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:32.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.535+0000 7fd3b3600640 1 Processor -- start 2024-01-16T15:06:32.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.535+0000 7fd3b3600640 1 -- start start 2024-01-16T15:06:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.536+0000 7fd3b3600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd3ac06d520 0x7fd3ac06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.536+0000 7fd3b3600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 0x7fd3ac15f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.536+0000 7fd3b3600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac15c360 con 0x7fd3ac0ad720 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.536+0000 7fd3b3600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac0b41a0 con 0x7fd3ac06d520 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.536+0000 7fd3b3600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac06de40 con 0x7fd3ac0abff0 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd3ac06d520 0x7fd3ac06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3abe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd3ac0ad720 0x7fd3ac15ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48804/0 (socket says 172.21.15.43:48804) 2024-01-16T15:06:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3abe00640 1 -- 172.21.15.43:0/3995147984 learned_addr learned my addr 172.21.15.43:0/3995147984 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b0c00640 1 -- 172.21.15.43:0/3995147984 >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 msgr2=0x7fd3ac15f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b1600640 1 --2- 172.21.15.43:0/3995147984 >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 0x7fd3ac15f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b0c00640 1 --2- 172.21.15.43:0/3995147984 >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 0x7fd3ac15f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b0c00640 1 -- 172.21.15.43:0/3995147984 >> v1:172.21.15.43:6789/0 conn(0x7fd3ac0ad720 legacy=0x7fd3ac15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b0c00640 1 -- 172.21.15.43:0/3995147984 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac15ba30 con 0x7fd3ac06d520 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3b1600640 1 --2- 172.21.15.43:0/3995147984 >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 0x7fd3ac15f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.537+0000 7fd3ab400640 1 -- 172.21.15.43:0/3995147984 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57431123 0 0) 0x7fd3ac15c360 con 0x7fd3ac0ad720 2024-01-16T15:06:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.538+0000 7fd3b0c00640 1 --2- 172.21.15.43:0/3995147984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac06d900 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac072960 tx=0x7fd39c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bda14b965a5aad0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3ab400640 1 -- 172.21.15.43:0/3995147984 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031830 con 0x7fd3ac06d520 2024-01-16T15:06:32.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3ab400640 1 -- 172.21.15.43:0/3995147984 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c031990 con 0x7fd3ac06d520 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3ab400640 1 -- 172.21.15.43:0/3995147984 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031d20 con 0x7fd3ac06d520 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 -- 172.21.15.43:0/3995147984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 msgr2=0x7fd3ac06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 --2- 172.21.15.43:0/3995147984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac06d900 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac072960 tx=0x7fd39c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 -- 172.21.15.43:0/3995147984 shutdown_connections 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 --2- 172.21.15.43:0/3995147984 >> v2:172.21.15.69:3300/0 conn(0x7fd3ac0abff0 0x7fd3ac15f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 --2- 172.21.15.43:0/3995147984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac06d900 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 -- 172.21.15.43:0/3995147984 >> 172.21.15.43:0/3995147984 conn(0x7fd3ac0b0d80 msgr2=0x7fd3ac0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.539+0000 7fd3b3600640 1 -- 172.21.15.43:0/3995147984 shutdown_connections 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.540+0000 7fd3b3600640 1 -- 172.21.15.43:0/3995147984 wait complete. 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.541+0000 7fd3b3600640 1 Processor -- start 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.541+0000 7fd3b3600640 1 -- start start 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3ac0abff0 0x7fd3ac147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3ac0ad720 0x7fd3ac1469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac14d750 con 0x7fd3ac06d520 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac0cd300 con 0x7fd3ac0abff0 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b3600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1f4f10 con 0x7fd3ac0ad720 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45962/0 (socket says 172.21.15.43:45962) 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.542+0000 7fd3b0c00640 1 -- 172.21.15.43:0/1384560370 learned_addr learned my addr 172.21.15.43:0/1384560370 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3abe00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3ac0abff0 0x7fd3ac147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 -- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3ac0ad720 msgr2=0x7fd3ac1469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3ac0ad720 0x7fd3ac1469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 -- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3ac0abff0 msgr2=0x7fd3ac147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3ac0abff0 0x7fd3ac147360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 -- 172.21.15.43:0/1384560370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c009400 con 0x7fd3ac06d520 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3b0c00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009890 tx=0x7fd39c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c03ba70 con 0x7fd3ac06d520 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.543+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c03bbd0 con 0x7fd3ac06d520 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.544+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0453f0 con 0x7fd3ac06d520 2024-01-16T15:06:32.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.544+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac0cd570 con 0x7fd3ac06d520 2024-01-16T15:06:32.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.544+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac0cdb30 con 0x7fd3ac06d520 2024-01-16T15:06:32.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.545+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac0cdd20 con 0x7fd3ac06d520 2024-01-16T15:06:32.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.546+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fd39c045550 con 0x7fd3ac06d520 2024-01-16T15:06:32.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.546+0000 7fd3a8c00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 0x7fd38406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:32.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.547+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd39c0aa920 con 0x7fd3ac06d520 2024-01-16T15:06:32.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.547+0000 7fd3abe00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 0x7fd38406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:32.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.548+0000 7fd3abe00640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 0x7fd38406dff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac14da70 tx=0x7fd394009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:32.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.551+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd39c077010 con 0x7fd3ac06d520 2024-01-16T15:06:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.817+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd3ac074770 con 0x7fd3ac06d520 2024-01-16T15:06:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.818+0000 7fd3a8c00640 1 -- 172.21.15.43:0/1384560370 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2429 (secure 0 0 0) 0x7fd39c07a630 con 0x7fd3ac06d520 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:32.797 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:32.798 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 445, 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:32.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:32.801 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 3710, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 15407457488, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 30723694592, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 742370418688, 2024-01-16T15:06:32.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 727453153, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 737617465, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 174, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 188 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:32.804 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:32.805 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:32.806 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 52.381% (23s)\n [==============..............] (remaining: 21s)", 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.52381426095962524, 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:32.807 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "2 ongoing cloning jobs; Avg progress = 83.335% (23s)\n [=======================.....] (remaining: 4s)", 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.83335000276565552, 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:32.808 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:32.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 msgr2=0x7fd38406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 0x7fd38406dff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac14da70 tx=0x7fd394009040 comp rx=0 tx=0).stop 2024-01-16T15:06:32.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 msgr2=0x7fd3ac1f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009890 tx=0x7fd39c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 shutdown_connections 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38406bb30 0x7fd38406dff0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3ac0abff0 0x7fd3ac147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3ac0ad720 0x7fd3ac1469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 --2- 172.21.15.43:0/1384560370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3ac06d520 0x7fd3ac1f49d0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 >> 172.21.15.43:0/1384560370 conn(0x7fd3ac0b0d80 msgr2=0x7fd3ac0acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 shutdown_connections 2024-01-16T15:06:32.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:32.821+0000 7fd3b3600640 1 -- 172.21.15.43:0/1384560370 wait complete. 2024-01-16T15:06:34.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:35.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.076+0000 7f476dc00640 1 Processor -- start 2024-01-16T15:06:35.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.076+0000 7f476dc00640 1 -- start start 2024-01-16T15:06:35.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476dc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f47680cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476dc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f47680cd7b0 0x7f47680cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476dc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476815fa70 con 0x7f476813dfd0 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476dc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f476815c580 con 0x7f47680cd7b0 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476dc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47680cfa10 con 0x7f47680cf050 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f4767e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f476813dfd0 0x7f476815be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54312/0 (socket says 172.21.15.43:54312) 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.078+0000 7f476d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f47680cd7b0 0x7f47680cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f4767e00640 1 -- 172.21.15.43:0/3576100605 learned_addr learned my addr 172.21.15.43:0/3576100605 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f4767400640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f47680cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f476d200640 1 -- 172.21.15.43:0/3576100605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 msgr2=0x7f47680cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f476d200640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f47680cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f476d200640 1 -- 172.21.15.43:0/3576100605 >> v1:172.21.15.43:6789/0 conn(0x7f476813dfd0 legacy=0x7f476815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.079+0000 7f476d200640 1 -- 172.21.15.43:0/3576100605 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4768140900 con 0x7f47680cd7b0 2024-01-16T15:06:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.080+0000 7f476d200640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47680cd7b0 0x7f47680cdc00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4760009d90 tx=0x7f476002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7579827726abe8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.080+0000 7f4766a00640 1 -- 172.21.15.43:0/3576100605 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476002cc60 con 0x7f47680cd7b0 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.080+0000 7f4766a00640 1 -- 172.21.15.43:0/3576100605 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476002cdc0 con 0x7f47680cd7b0 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.080+0000 7f4766a00640 1 -- 172.21.15.43:0/3576100605 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760034660 con 0x7f47680cd7b0 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 -- 172.21.15.43:0/3576100605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47680cd7b0 msgr2=0x7f47680cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47680cd7b0 0x7f47680cdc00 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4760009d90 tx=0x7f476002beb0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 -- 172.21.15.43:0/3576100605 shutdown_connections 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47680cd7b0 0x7f47680cdc00 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 --2- 172.21.15.43:0/3576100605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f47680cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 -- 172.21.15.43:0/3576100605 >> 172.21.15.43:0/3576100605 conn(0x7f476806a860 msgr2=0x7f476806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 -- 172.21.15.43:0/3576100605 shutdown_connections 2024-01-16T15:06:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.081+0000 7f476dc00640 1 -- 172.21.15.43:0/3576100605 wait complete. 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.083+0000 7f476dc00640 1 Processor -- start 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.083+0000 7f476dc00640 1 -- start start 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.084+0000 7f476dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47680cd7b0 0x7f476813e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.084+0000 7f476dc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f476812e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476dc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476813fc70 con 0x7f476813dfd0 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476dc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476813f060 con 0x7f47680cd7b0 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476dc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476813f340 con 0x7f47680cf050 2024-01-16T15:06:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57310/0 (socket says 172.21.15.43:57310) 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 -- 172.21.15.43:0/3945429041 learned_addr learned my addr 172.21.15.43:0/3945429041 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f4767e00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47680cd7b0 0x7f476813e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f4767400640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f476812e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 -- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 msgr2=0x7f476812e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f476812e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 -- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47680cd7b0 msgr2=0x7f476813e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47680cd7b0 0x7f476813e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.085+0000 7f476d200640 1 -- 172.21.15.43:0/3945429041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4760009a70 con 0x7f476813dfd0 2024-01-16T15:06:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f476d200640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4760009d90 tx=0x7f4760004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760040650 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47600407d0 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760040b80 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476813f620 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.086+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47681589e0 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.087+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476813fe20 con 0x7f476813dfd0 2024-01-16T15:06:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.088+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4760040ce0 con 0x7f476813dfd0 2024-01-16T15:06:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.089+0000 7f475fe00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 0x7f474006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.089+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f47600ab420 con 0x7f476813dfd0 2024-01-16T15:06:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.090+0000 7f4767e00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 0x7f474006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.090+0000 7f4767e00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 0x7f474006dff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4758005880 tx=0x7f4758009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.094+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f476813fe20 con 0x7f476813dfd0 2024-01-16T15:06:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.360+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f476804e840 con 0x7f476813dfd0 2024-01-16T15:06:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.361+0000 7f475fe00640 1 -- 172.21.15.43:0/3945429041 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2404 (secure 0 0 0) 0x7f4760077b10 con 0x7f476813dfd0 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:06:35.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:06:35.341 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 447, 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.342 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 61, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:06:35.343 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 3710, 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 15407457488, 2024-01-16T15:06:35.344 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 30723694592, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 742370418688, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 508070656, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 511592981, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 121, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 130 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37, 2024-01-16T15:06:35.345 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 4, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 4, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 5650 2024-01-16T15:06:35.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:06:35.347 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:05:52.934035+0000", 2024-01-16T15:06:35.348 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "30f3c314-5f0d-4752-8d01-8327229bcf7b": { 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (25s)\n [================............] (remaining: 18s)", 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.57142859697341919, 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "7e934886-a6f9-46be-9b23-db2643dbf055": { 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "Subvolume \"subvol_clone53197\" has been 100.0% cloned (25s)\n [============================] ", 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 1, 2024-01-16T15:06:35.349 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:06:35.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:35.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:35.350 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 msgr2=0x7f474006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 0x7f474006dff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4758005880 tx=0x7f4758009040 comp rx=0 tx=0).stop 2024-01-16T15:06:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 msgr2=0x7f4768140100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4760009d90 tx=0x7f4760004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 shutdown_connections 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474006bb30 0x7f474006dff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f476813dfd0 0x7f4768140100 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47680cf050 0x7f476812e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 --2- 172.21.15.43:0/3945429041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47680cd7b0 0x7f476813e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 >> 172.21.15.43:0/3945429041 conn(0x7f476806a860 msgr2=0x7f476815c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 shutdown_connections 2024-01-16T15:06:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:35.364+0000 7f476dc00640 1 -- 172.21.15.43:0/3945429041 wait complete. 2024-01-16T15:06:37.362 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:37.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53191 2024-01-16T15:06:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.615+0000 7f947be00640 1 Processor -- start 2024-01-16T15:06:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.615+0000 7f947be00640 1 -- start start 2024-01-16T15:06:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.616+0000 7f947be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f947c0c8930 0x7f947c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.616+0000 7f947be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0c6bf0 con 0x7f947c0d1d90 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f947be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0cedc0 con 0x7f947c15d3b0 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f947be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c0c6e40 con 0x7f947c0c8930 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f947a000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f947c0d1d90 0x7f947c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54324/0 (socket says 172.21.15.43:54324) 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f947a000640 1 -- 172.21.15.43:0/3487484573 learned_addr learned my addr 172.21.15.43:0/3487484573 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f947b400640 1 --2- 172.21.15.43:0/3487484573 >> v2:172.21.15.69:3300/0 conn(0x7f947c0c8930 0x7f947c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722219499 0 0) 0x7f947c0c6bf0 con 0x7f947c0d1d90 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.617+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c003520 con 0x7f947c0d1d90 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2248992060 0 0) 0x7f945c003520 con 0x7f947c0d1d90 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 >> v2:172.21.15.69:3300/0 conn(0x7f947c0c8930 msgr2=0x7f947c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 --2- 172.21.15.43:0/3487484573 >> v2:172.21.15.69:3300/0 conn(0x7f947c0c8930 0x7f947c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 >> v1:172.21.15.69:6789/0 conn(0x7f947c15d3b0 legacy=0x7f947c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0c7120 con 0x7f947c0d1d90 2024-01-16T15:06:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9464001670 con 0x7f947c0d1d90 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 >> v1:172.21.15.43:6789/0 conn(0x7f947c0d1d90 legacy=0x7f947c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c003a30 0x7f945c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c0047c0 0x7f945c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f945c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0c6e40 con 0x7f945c0052a0 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c00c170 con 0x7f945c0047c0 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c00c330 con 0x7f945c003a30 2024-01-16T15:06:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.618+0000 7f947b400640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f945c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947a000640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c0047c0 0x7f945c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947aa00640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c003a30 0x7f945c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947b400640 1 -- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c003a30 msgr2=0x7f945c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947b400640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c003a30 0x7f945c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947b400640 1 -- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c0047c0 msgr2=0x7f945c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947b400640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c0047c0 0x7f945c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947b400640 1 -- 172.21.15.43:0/3487484573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c003520 con 0x7f945c0052a0 2024-01-16T15:06:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.619+0000 7f947aa00640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c003a30 0x7f945c003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.620+0000 7f947b400640 1 -- 172.21.15.43:0/3487484573 reap_dead start 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.620+0000 7f947b400640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f945c00b980 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9474009ac0 tx=0x7f945c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2dca6f49d5fd73 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.620+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474004020 con 0x7f945c0052a0 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.620+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9474004180 con 0x7f945c0052a0 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.620+0000 7f9479600640 1 -- 172.21.15.43:0/3487484573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947402c710 con 0x7f945c0052a0 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 -- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 msgr2=0x7f945c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f945c00b980 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9474009ac0 tx=0x7f945c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 -- 172.21.15.43:0/3487484573 shutdown_connections 2024-01-16T15:06:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 --2- 172.21.15.43:0/3487484573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f945c00b980 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 -- 172.21.15.43:0/3487484573 >> 172.21.15.43:0/3487484573 conn(0x7f947c0d30a0 msgr2=0x7f947c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 -- 172.21.15.43:0/3487484573 shutdown_connections 2024-01-16T15:06:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.621+0000 7f947be00640 1 -- 172.21.15.43:0/3487484573 wait complete. 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.623+0000 7f947be00640 1 Processor -- start 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.623+0000 7f947be00640 1 -- start start 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 0x7f947c09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f947c09d320 0x7f947c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c09d7a0 con 0x7f945c0052a0 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0cb9b0 con 0x7f947c15a960 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0cbc90 con 0x7f947c09d320 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57322/0 (socket says 172.21.15.43:57322) 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947aa00640 1 -- 172.21.15.43:0/2882326186 learned_addr learned my addr 172.21.15.43:0/2882326186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.624+0000 7f947b400640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f947c09d320 0x7f947c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947a000640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 0x7f947c09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 -- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f947c09d320 msgr2=0x7f947c09f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f947c09d320 0x7f947c09f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 -- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 msgr2=0x7f947c09cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 0x7f947c09cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 -- 172.21.15.43:0/2882326186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0cbf70 con 0x7f945c0052a0 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947a000640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 0x7f947c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947aa00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f946c00db10 tx=0x7f946c0020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c003da0 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c006b50 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c0cc250 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00b740 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.625+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c0cb680 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.627+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c0b6d10 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.628+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f946c00b8a0 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.628+0000 7f9472a00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 0x7f945406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.629+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f946c088030 con 0x7f945c0052a0 2024-01-16T15:06:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.629+0000 7f947a000640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 0x7f945406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.630+0000 7f947a000640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 0x7f945406dff0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9464000c00 tx=0x7f9464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.634+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f946c054760 con 0x7f945c0052a0 2024-01-16T15:06:37.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:37.865+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53191", "target": ["mon-mgr", ""]}) v1 -- 0x7f947c15d790 con 0x7f945406bb30 2024-01-16T15:06:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.513+0000 7f9472a00640 1 -- 172.21.15.43:0/2882326186 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f946400d040 con 0x7f945406bb30 2024-01-16T15:06:38.492 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:38.492 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:38.492 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:38.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:38.492 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.519+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 msgr2=0x7f945406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.519+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 0x7f945406dff0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9464000c00 tx=0x7f9464009040 comp rx=0 tx=0).stop 2024-01-16T15:06:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.519+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 msgr2=0x7f947c0ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.519+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f946c00db10 tx=0x7f946c0020c0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 shutdown_connections 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f945406bb30 0x7f945406dff0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f947c15a960 0x7f947c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f947c09d320 0x7f947c09f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 --2- 172.21.15.43:0/2882326186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c0052a0 0x7f947c0ccaf0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.520+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 >> 172.21.15.43:0/2882326186 conn(0x7f947c0d30a0 msgr2=0x7f947c0cf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:38.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.522+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 shutdown_connections 2024-01-16T15:06:38.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.522+0000 7f947be00640 1 -- 172.21.15.43:0/2882326186 wait complete. 2024-01-16T15:06:38.539 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:38.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53192 2024-01-16T15:06:38.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.873+0000 7f8a07200640 1 Processor -- start 2024-01-16T15:06:38.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.873+0000 7f8a07200640 1 -- start start 2024-01-16T15:06:38.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.875+0000 7f8a07200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a0006dc30 0x7f8a000ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.875+0000 7f8a07200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a00070490 0x7f8a00070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.875+0000 7f8a07200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0006d7a0 con 0x7f8a000acd70 2024-01-16T15:06:38.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.875+0000 7f8a07200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0015a5b0 con 0x7f8a0006dc30 2024-01-16T15:06:38.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.875+0000 7f8a07200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000ae9d0 con 0x7f8a00070490 2024-01-16T15:06:38.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.876+0000 7f89ffe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8a000acd70 0x7f8a000ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:53534/0 (socket says 172.21.15.43:53534) 2024-01-16T15:06:38.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.876+0000 7f89ffe00640 1 -- 172.21.15.43:0/2993471206 learned_addr learned my addr 172.21.15.43:0/2993471206 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:38.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.877+0000 7f89ff400640 1 --2- 172.21.15.43:0/2993471206 >> v2:172.21.15.43:3300/0 conn(0x7f8a0006dc30 0x7f8a000ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.877+0000 7f8a05200640 1 --2- 172.21.15.43:0/2993471206 >> v2:172.21.15.69:3300/0 conn(0x7f8a00070490 0x7f8a00070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.877+0000 7f89fea00640 1 -- 172.21.15.43:0/2993471206 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273771690 0 0) 0x7f8a0006d7a0 con 0x7f8a000acd70 2024-01-16T15:06:38.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.877+0000 7f89fea00640 1 -- 172.21.15.43:0/2993471206 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e0003520 con 0x7f8a000acd70 2024-01-16T15:06:38.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.878+0000 7f8a05200640 1 -- 172.21.15.43:0/2993471206 >> v2:172.21.15.43:3300/0 conn(0x7f8a0006dc30 msgr2=0x7f8a000ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:38.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.878+0000 7f8a05200640 1 --2- 172.21.15.43:0/2993471206 >> v2:172.21.15.43:3300/0 conn(0x7f8a0006dc30 0x7f8a000ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.878+0000 7f8a05200640 1 -- 172.21.15.43:0/2993471206 >> v1:172.21.15.69:6789/0 conn(0x7f8a000acd70 legacy=0x7f8a000ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.878+0000 7f8a05200640 1 -- 172.21.15.43:0/2993471206 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0006ce70 con 0x7f8a00070490 2024-01-16T15:06:38.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.882+0000 7f8a05200640 1 --2- 172.21.15.43:0/2993471206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00070900 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8a00067e70 tx=0x7f89f80302c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31a03c7bd745a60c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.882+0000 7f89fea00640 1 -- 172.21.15.43:0/2993471206 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800d2d0 con 0x7f8a00070490 2024-01-16T15:06:38.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.883+0000 7f89fea00640 1 -- 172.21.15.43:0/2993471206 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f800d430 con 0x7f8a00070490 2024-01-16T15:06:38.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.883+0000 7f89fea00640 1 -- 172.21.15.43:0/2993471206 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800b960 con 0x7f8a00070490 2024-01-16T15:06:38.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.884+0000 7f8a07200640 1 -- 172.21.15.43:0/2993471206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 msgr2=0x7f8a00070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.884+0000 7f8a07200640 1 --2- 172.21.15.43:0/2993471206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00070900 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8a00067e70 tx=0x7f89f80302c0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 -- 172.21.15.43:0/2993471206 shutdown_connections 2024-01-16T15:06:38.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 --2- 172.21.15.43:0/2993471206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00070900 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 --2- 172.21.15.43:0/2993471206 >> v2:172.21.15.43:3300/0 conn(0x7f8a0006dc30 0x7f8a000ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 -- 172.21.15.43:0/2993471206 >> 172.21.15.43:0/2993471206 conn(0x7f8a000b0ba0 msgr2=0x7f8a000b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:38.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 -- 172.21.15.43:0/2993471206 shutdown_connections 2024-01-16T15:06:38.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.885+0000 7f8a07200640 1 -- 172.21.15.43:0/2993471206 wait complete. 2024-01-16T15:06:38.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.888+0000 7f8a07200640 1 Processor -- start 2024-01-16T15:06:38.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.889+0000 7f8a07200640 1 -- start start 2024-01-16T15:06:38.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001f2bf0 con 0x7f8a0006dc30 2024-01-16T15:06:38.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001f1e30 con 0x7f8a00070490 2024-01-16T15:06:38.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.891+0000 7f8a07200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001f2110 con 0x7f8a000acd70 2024-01-16T15:06:38.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.892+0000 7f89ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.893+0000 7f8a05200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.893+0000 7f89ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57348/0 (socket says 172.21.15.43:57348) 2024-01-16T15:06:38.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.893+0000 7f8a05200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37270/0 (socket says 172.21.15.43:37270) 2024-01-16T15:06:38.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.893+0000 7f8a05200640 1 -- 172.21.15.43:0/1101036718 learned_addr learned my addr 172.21.15.43:0/1101036718 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:38.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.894+0000 7f89ff400640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.895+0000 7f8a05200640 1 -- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 msgr2=0x7f8a00159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.896+0000 7f8a05200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.896+0000 7f8a05200640 1 -- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 msgr2=0x7f8a000c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:38.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.896+0000 7f8a05200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:38.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.896+0000 7f8a05200640 1 -- 172.21.15.43:0/1101036718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f800f050 con 0x7f8a000acd70 2024-01-16T15:06:38.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.897+0000 7f89ffe00640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:38.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.898+0000 7f8a05200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89f803a040 tx=0x7f89f8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.898+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800d5a0 con 0x7f8a000acd70 2024-01-16T15:06:38.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.899+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f8042660 con 0x7f8a000acd70 2024-01-16T15:06:38.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.899+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8042a10 con 0x7f8a000acd70 2024-01-16T15:06:38.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.899+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a001f23f0 con 0x7f8a000acd70 2024-01-16T15:06:38.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.900+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a000c4430 con 0x7f8a000acd70 2024-01-16T15:06:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.901+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a001f28b0 con 0x7f8a000acd70 2024-01-16T15:06:38.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.904+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f89f8042b70 con 0x7f8a000acd70 2024-01-16T15:06:38.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.905+0000 7f89f7e00640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 0x7f89d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:38.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.905+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f89f80af530 con 0x7f8a000acd70 2024-01-16T15:06:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.910+0000 7f89ffe00640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 0x7f89d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.911+0000 7f89ffe00640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 0x7f89d806e0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89f0005880 tx=0x7f89f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:38.919+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89f807bc20 con 0x7f8a000acd70 2024-01-16T15:06:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.380+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53192", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a000b3d90 con 0x7f89d806bc00 2024-01-16T15:06:39.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.420+0000 7f89f7e00640 1 -- 172.21.15.43:0/1101036718 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f89f00018a0 con 0x7f89d806bc00 2024-01-16T15:06:39.400 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:39.400 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 msgr2=0x7f89d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 0x7f89d806e0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89f0005880 tx=0x7f89f0009040 comp rx=0 tx=0).stop 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 msgr2=0x7f8a001edf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f89f803a040 tx=0x7f89f8004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 shutdown_connections 2024-01-16T15:06:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d806bc00 0x7f89d806e0c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a00070490 0x7f8a00159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a000acd70 0x7f8a001edf60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 --2- 172.21.15.43:0/1101036718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a0006dc30 0x7f8a000c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.428+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 >> 172.21.15.43:0/1101036718 conn(0x7f8a000b0ba0 msgr2=0x7f8a0015c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.429+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 shutdown_connections 2024-01-16T15:06:39.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.429+0000 7f8a07200640 1 -- 172.21.15.43:0/1101036718 wait complete. 2024-01-16T15:06:39.455 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:39.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53193 2024-01-16T15:06:39.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.832+0000 7f5139000640 1 Processor -- start 2024-01-16T15:06:39.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.833+0000 7f5139000640 1 -- start start 2024-01-16T15:06:39.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.836+0000 7f5139000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f51340b2110 0x7f513406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.836+0000 7f5139000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f513406c710 0x7f5134070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5131400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f513406c710 0x7f5134070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5132800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f51340b24f0 0x7f51340abe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54332/0 (socket says 172.21.15.43:54332) 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5131400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f513406c710 0x7f5134070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:59258/0 (socket says 172.21.15.43:59258) 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5132800640 1 -- 172.21.15.43:0/956857222 learned_addr learned my addr 172.21.15.43:0/956857222 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5131e00640 1 --2- 172.21.15.43:0/956857222 >> v2:172.21.15.43:3300/0 conn(0x7f51340b2110 0x7f513406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513406cc10 con 0x7f51340b24f0 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513415cd60 con 0x7f51340b2110 2024-01-16T15:06:39.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.837+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513406cef0 con 0x7f513406c710 2024-01-16T15:06:39.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.838+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665938840 0 0) 0x7f513406cc10 con 0x7f51340b24f0 2024-01-16T15:06:39.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.838+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5114003520 con 0x7f51340b24f0 2024-01-16T15:06:39.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2083893783 0 0) 0x7f5114003520 con 0x7f51340b24f0 2024-01-16T15:06:39.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 >> v2:172.21.15.69:3300/0 conn(0x7f513406c710 msgr2=0x7f5134070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:39.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 --2- 172.21.15.43:0/956857222 >> v2:172.21.15.69:3300/0 conn(0x7f513406c710 0x7f5134070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 >> v2:172.21.15.43:3300/0 conn(0x7f51340b2110 msgr2=0x7f513406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:39.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 --2- 172.21.15.43:0/956857222 >> v2:172.21.15.43:3300/0 conn(0x7f51340b2110 0x7f513406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.839+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513406d1d0 con 0x7f51340b24f0 2024-01-16T15:06:39.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f512c001670 con 0x7f51340b24f0 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 >> v1:172.21.15.43:6789/0 conn(0x7f51340b24f0 legacy=0x7f51340abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5114003970 0x7f5114003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 0x7f5114007960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513415cd60 con 0x7f5114003970 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513406cef0 con 0x7f5114005180 2024-01-16T15:06:39.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.840+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5114007f80 con 0x7f51140047c0 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 0x7f5114007960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5131400640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5131e00640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5114003970 0x7f5114003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 -- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51140047c0 msgr2=0x7f5114004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 -- 172.21.15.43:0/956857222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5114003970 msgr2=0x7f5114003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:39.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5114003970 0x7f5114003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.841+0000 7f5132800640 1 -- 172.21.15.43:0/956857222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114003520 con 0x7f5114005180 2024-01-16T15:06:39.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.843+0000 7f5132800640 1 -- 172.21.15.43:0/956857222 reap_dead start 2024-01-16T15:06:39.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.843+0000 7f5132800640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 0x7f5114007960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f512c004f90 tx=0x7f51140057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccd323414015d660 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.844+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c005b90 con 0x7f5114005180 2024-01-16T15:06:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.844+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c02f070 con 0x7f5114005180 2024-01-16T15:06:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.844+0000 7f5130a00640 1 -- 172.21.15.43:0/956857222 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0080d0 con 0x7f5114005180 2024-01-16T15:06:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 msgr2=0x7f5114007960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 0x7f5114007960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f512c004f90 tx=0x7f51140057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 shutdown_connections 2024-01-16T15:06:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 --2- 172.21.15.43:0/956857222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5114005180 0x7f5114007960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 >> 172.21.15.43:0/956857222 conn(0x7f51340b0ba0 msgr2=0x7f51340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 shutdown_connections 2024-01-16T15:06:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.845+0000 7f5139000640 1 -- 172.21.15.43:0/956857222 wait complete. 2024-01-16T15:06:39.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.849+0000 7f5139000640 1 Processor -- start 2024-01-16T15:06:39.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.850+0000 7f5139000640 1 -- start start 2024-01-16T15:06:39.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5114005180 0x7f5134159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f513409af90 0x7f513409b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134159ab0 con 0x7f5134147fb0 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513415c270 con 0x7f513409af90 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.851+0000 7f5139000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513415c550 con 0x7f5114005180 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.852+0000 7f5131400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.852+0000 7f5131400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57378/0 (socket says 172.21.15.43:57378) 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.852+0000 7f5131400640 1 -- 172.21.15.43:0/552200659 learned_addr learned my addr 172.21.15.43:0/552200659 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.852+0000 7f5131e00640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5114005180 0x7f5134159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.852+0000 7f5132800640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f513409af90 0x7f513409b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 -- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5114005180 msgr2=0x7f5134159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5114005180 0x7f5134159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 -- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f513409af90 msgr2=0x7f513409b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f513409af90 0x7f513409b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 -- 172.21.15.43:0/552200659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51340cbbf0 con 0x7f5134147fb0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.853+0000 7f5131400640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f511c00e7e0 tx=0x7f511c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.854+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c00c6d0 con 0x7f5134147fb0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.854+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c00c830 con 0x7f5134147fb0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.854+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c00cba0 con 0x7f5134147fb0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.854+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51340cbed0 con 0x7f5134147fb0 2024-01-16T15:06:39.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.854+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51341eeb10 con 0x7f5134147fb0 2024-01-16T15:06:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.856+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513415c3b0 con 0x7f5134147fb0 2024-01-16T15:06:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.858+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f511c00cd00 con 0x7f5134147fb0 2024-01-16T15:06:39.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.859+0000 7f512a000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 0x7f510c06e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:39.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.860+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f511c0883b0 con 0x7f5134147fb0 2024-01-16T15:06:39.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.863+0000 7f5131e00640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 0x7f510c06e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.866+0000 7f5131e00640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 0x7f510c06e070 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5124003b80 tx=0x7f512400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:39.867+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f513415c3b0 con 0x7f5134147fb0 2024-01-16T15:06:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.269+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53193", "target": ["mon-mgr", ""]}) v1 -- 0x7f513406d790 con 0x7f510c06bbb0 2024-01-16T15:06:40.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.391+0000 7f512a000640 1 -- 172.21.15.43:0/552200659 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f512400f040 con 0x7f510c06bbb0 2024-01-16T15:06:40.369 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:40.370 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:40.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 msgr2=0x7f510c06e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 0x7f510c06e070 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5124003b80 tx=0x7f512400b040 comp rx=0 tx=0).stop 2024-01-16T15:06:40.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 msgr2=0x7f513409aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f511c00e7e0 tx=0x7f511c00eca0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 shutdown_connections 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f510c06bbb0 0x7f510c06e070 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f513409af90 0x7f513409b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5134147fb0 0x7f513409aa50 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 --2- 172.21.15.43:0/552200659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5114005180 0x7f5134159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 >> 172.21.15.43:0/552200659 conn(0x7f51340b0ba0 msgr2=0x7f5134070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 shutdown_connections 2024-01-16T15:06:40.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.394+0000 7f5139000640 1 -- 172.21.15.43:0/552200659 wait complete. 2024-01-16T15:06:40.389 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:40.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53194 2024-01-16T15:06:40.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.587+0000 7fb2e7400640 1 Processor -- start 2024-01-16T15:06:40.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.588+0000 7fb2e7400640 1 -- start start 2024-01-16T15:06:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb2e015efa0 0x7fb2e0161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb2e00b2970 0x7fb2e00b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e00b32e0 con 0x7fb2e015efa0 2024-01-16T15:06:40.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e006c9a0 con 0x7fb2e00b2970 2024-01-16T15:06:40.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e7400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e006cc80 con 0x7fb2e006da40 2024-01-16T15:06:40.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e6000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb2e015efa0 0x7fb2e0161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e6000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb2e015efa0 0x7fb2e0161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57390/0 (socket says 172.21.15.43:57390) 2024-01-16T15:06:40.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.589+0000 7fb2e6000640 1 -- 172.21.15.43:0/2096180662 learned_addr learned my addr 172.21.15.43:0/2096180662 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.590+0000 7fb2e6000640 1 -- 172.21.15.43:0/2096180662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 msgr2=0x7fb2e006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.590+0000 7fb2e6000640 1 --2- 172.21.15.43:0/2096180662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.590+0000 7fb2e6000640 1 -- 172.21.15.43:0/2096180662 >> v2:172.21.15.69:3300/0 conn(0x7fb2e00b2970 msgr2=0x7fb2e00b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.590+0000 7fb2e6000640 1 --2- 172.21.15.43:0/2096180662 >> v2:172.21.15.69:3300/0 conn(0x7fb2e00b2970 0x7fb2e00b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.590+0000 7fb2e6000640 1 -- 172.21.15.43:0/2096180662 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e006cf60 con 0x7fb2e015efa0 2024-01-16T15:06:40.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.591+0000 7fb2e6000640 1 --2- 172.21.15.43:0/2096180662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e015efa0 0x7fb2e0161ac0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0162000 tx=0x7fb2c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342fd8be91bedb65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.591+0000 7fb2e4c00640 1 -- 172.21.15.43:0/2096180662 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4031870 con 0x7fb2e015efa0 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.591+0000 7fb2e4c00640 1 -- 172.21.15.43:0/2096180662 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40319d0 con 0x7fb2e015efa0 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.591+0000 7fb2e4c00640 1 -- 172.21.15.43:0/2096180662 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4031d60 con 0x7fb2e015efa0 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 -- 172.21.15.43:0/2096180662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e015efa0 msgr2=0x7fb2e0161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2096180662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e015efa0 0x7fb2e0161ac0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0162000 tx=0x7fb2c402b910 comp rx=0 tx=0).stop 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 -- 172.21.15.43:0/2096180662 shutdown_connections 2024-01-16T15:06:40.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2096180662 >> v2:172.21.15.69:3300/0 conn(0x7fb2e00b2970 0x7fb2e00b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2096180662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2096180662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e015efa0 0x7fb2e0161ac0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 -- 172.21.15.43:0/2096180662 >> 172.21.15.43:0/2096180662 conn(0x7fb2e00cec10 msgr2=0x7fb2e00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 -- 172.21.15.43:0/2096180662 shutdown_connections 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.592+0000 7fb2e7400640 1 -- 172.21.15.43:0/2096180662 wait complete. 2024-01-16T15:06:40.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.593+0000 7fb2e7400640 1 Processor -- start 2024-01-16T15:06:40.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 -- start start 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e0161630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2e015efa0 0x7fb2e01598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0154990 con 0x7fb2e00b2970 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e01587c0 con 0x7fb2e015efa0 2024-01-16T15:06:40.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.594+0000 7fb2e7400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0153d10 con 0x7fb2e006da40 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57398/0 (socket says 172.21.15.43:57398) 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 -- 172.21.15.43:0/2481263194 learned_addr learned my addr 172.21.15.43:0/2481263194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e6a00640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2e015efa0 0x7fb2e01598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 -- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 msgr2=0x7fb2e0161630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e0161630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 -- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2e015efa0 msgr2=0x7fb2e01598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2e015efa0 0x7fb2e01598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.595+0000 7fb2e5600640 1 -- 172.21.15.43:0/2481263194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40094d0 con 0x7fb2e00b2970 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2e5600640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00b9c0 tx=0x7fb2cc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00cba0 con 0x7fb2e00b2970 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc00cd20 con 0x7fb2e00b2970 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e0153ff0 con 0x7fb2e00b2970 2024-01-16T15:06:40.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc014680 con 0x7fb2e00b2970 2024-01-16T15:06:40.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.596+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e01545b0 con 0x7fb2e00b2970 2024-01-16T15:06:40.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.597+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e0154ad0 con 0x7fb2e00b2970 2024-01-16T15:06:40.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.598+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb2cc01d3b0 con 0x7fb2e00b2970 2024-01-16T15:06:40.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.599+0000 7fb2de000640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 0x7fb2b406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.599+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb2cc0885d0 con 0x7fb2e00b2970 2024-01-16T15:06:40.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.599+0000 7fb2e6000640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 0x7fb2b406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.600+0000 7fb2e6000640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 0x7fb2b406dff0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4036000 tx=0x7fb2c4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.603+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2e0154ad0 con 0x7fb2e00b2970 2024-01-16T15:06:40.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.895+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53194", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2e00b4d20 con 0x7fb2b406bb30 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.952+0000 7fb2de000640 1 -- 172.21.15.43:0/2481263194 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fb2c4031c40 con 0x7fb2b406bb30 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:40.931 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.964+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 msgr2=0x7fb2b406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.964+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 0x7fb2b406dff0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4036000 tx=0x7fb2c4030000 comp rx=0 tx=0).stop 2024-01-16T15:06:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.964+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 msgr2=0x7fb2e01593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.964+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00b9c0 tx=0x7fb2cc00be80 comp rx=0 tx=0).stop 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 shutdown_connections 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb2b406bb30 0x7fb2b406dff0 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4036000 tx=0x7fb2c4030000 comp rx=0 tx=0).stop 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2e015efa0 0x7fb2e01598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2e00b2970 0x7fb2e01593b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 --2- 172.21.15.43:0/2481263194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2e006da40 0x7fb2e0161630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.965+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 >> 172.21.15.43:0/2481263194 conn(0x7fb2e00cec10 msgr2=0x7fb2e01606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.966+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 shutdown_connections 2024-01-16T15:06:40.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:40.966+0000 7fb2e7400640 1 -- 172.21.15.43:0/2481263194 wait complete. 2024-01-16T15:06:40.987 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:40.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53195 2024-01-16T15:06:41.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.358+0000 7f049d800640 1 Processor -- start 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.358+0000 7f049d800640 1 -- start start 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.360+0000 7f049d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049808a230 0x7f0498070490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f049d800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0498070e20 con 0x7f049816d740 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f049d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049816db20 con 0x7f049808a690 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f049d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980b1ea0 con 0x7f049808a230 2024-01-16T15:06:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f0496000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049808a230 0x7f0498070490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f0496000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049808a230 0x7f0498070490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37304/0 (socket says 172.21.15.43:37304) 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.361+0000 7f0496000640 1 -- 172.21.15.43:0/1444736777 learned_addr learned my addr 172.21.15.43:0/1444736777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034319851 0 0) 0x7f049816db20 con 0x7f049808a690 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0478003520 con 0x7f049808a690 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771538546 0 0) 0x7f0490001490 con 0x7f049816d740 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049816db20 con 0x7f049816d740 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1392073329 0 0) 0x7f0478003520 con 0x7f049808a690 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049808a230 msgr2=0x7f0498070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049808a230 0x7f0498070490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 >> v1:172.21.15.43:6789/0 conn(0x7f049816d740 legacy=0x7f04980abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980b2180 con 0x7f049808a690 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.363+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0488001670 con 0x7f049808a690 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 >> v1:172.21.15.69:6789/0 conn(0x7f049808a690 legacy=0x7f04980b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478003b00 0x7f0478003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04780047c0 0x7f0478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0478005180 0x7f047800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980b1ea0 con 0x7f0478003b00 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478010490 con 0x7f0478005180 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.364+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0478010650 con 0x7f04780047c0 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.365+0000 7f0496a00640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478003b00 0x7f0478003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.365+0000 7f0497400640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0478005180 0x7f047800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.366+0000 7f0497400640 1 -- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04780047c0 msgr2=0x7f0478004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:06:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.366+0000 7f0497400640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04780047c0 0x7f0478004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.366+0000 7f0497400640 1 -- 172.21.15.43:0/1444736777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478003b00 msgr2=0x7f0478003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.366+0000 7f0497400640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478003b00 0x7f0478003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.366+0000 7f0497400640 1 -- 172.21.15.43:0/1444736777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980b2180 con 0x7f0478005180 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.368+0000 7f0497400640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0478005180 0x7f047800fd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f048800c160 tx=0x7f04780057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3097ea23cc6c0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.369+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048802c970 con 0x7f0478005180 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.369+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048802caf0 con 0x7f0478005180 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.369+0000 7f0495600640 1 -- 172.21.15.43:0/1444736777 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488034440 con 0x7f0478005180 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.370+0000 7f049d800640 1 -- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0478005180 msgr2=0x7f047800fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.370+0000 7f049d800640 1 --2- 172.21.15.43:0/1444736777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0478005180 0x7f047800fd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f048800c160 tx=0x7f04780057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.371+0000 7f0497400640 1 -- 172.21.15.43:0/1444736777 reap_dead start 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.371+0000 7f049d800640 1 -- 172.21.15.43:0/1444736777 shutdown_connections 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.371+0000 7f049d800640 1 -- 172.21.15.43:0/1444736777 >> 172.21.15.43:0/1444736777 conn(0x7f04980b0ba0 msgr2=0x7f04980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.371+0000 7f049d800640 1 -- 172.21.15.43:0/1444736777 shutdown_connections 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.372+0000 7f049d800640 1 -- 172.21.15.43:0/1444736777 wait complete. 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.376+0000 7f049d800640 1 Processor -- start 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.377+0000 7f049d800640 1 -- start start 2024-01-16T15:06:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0498159d40 0x7f049815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049806d270 0x7f049806d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049806dc60 con 0x7f0478005180 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0498154c30 con 0x7f049806d270 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.378+0000 7f049d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0498086a20 con 0x7f0498159d40 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.379+0000 7f0496a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.379+0000 7f0496a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57420/0 (socket says 172.21.15.43:57420) 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.379+0000 7f0496000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0498159d40 0x7f049815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.379+0000 7f0496a00640 1 -- 172.21.15.43:0/1299835143 learned_addr learned my addr 172.21.15.43:0/1299835143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 -- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0498159d40 msgr2=0x7f049815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0498159d40 0x7f049815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 -- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049806d270 msgr2=0x7f049806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049806d270 0x7f049806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 -- 172.21.15.43:0/1299835143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049815a1d0 con 0x7f0478005180 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.380+0000 7f0496a00640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0490007fd0 tx=0x7f049000ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.381+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490003b80 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.381+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0490003d00 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.381+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049000c9c0 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.381+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049815a4b0 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.381+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049815aa70 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.384+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0498071840 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.387+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f049000cb20 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.388+0000 7f048ea00640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 0x7f047006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.389+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0490059c60 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.391+0000 7f0496000640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 0x7f047006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.395+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0490056030 con 0x7f0478005180 2024-01-16T15:06:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.397+0000 7f0496000640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 0x7f047006dff0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f048000b240 tx=0x7f0480008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.839+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7f04980b2a20 con 0x7f047006bb30 2024-01-16T15:06:41.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.869+0000 7f048ea00640 1 -- 172.21.15.43:0/1299835143 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f048000f9c0 con 0x7f047006bb30 2024-01-16T15:06:41.847 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:41.847 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "1/3", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:41.848 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:41.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:41.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:41.849 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:41.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.871+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 msgr2=0x7f047006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.871+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 0x7f047006dff0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f048000b240 tx=0x7f0480008040 comp rx=0 tx=0).stop 2024-01-16T15:06:41.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 msgr2=0x7f04980864e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0490007fd0 tx=0x7f049000ef80 comp rx=0 tx=0).stop 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 shutdown_connections 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047006bb30 0x7f047006dff0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049806d270 0x7f049806d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0498159d40 0x7f049815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 --2- 172.21.15.43:0/1299835143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0478005180 0x7f04980864e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 >> 172.21.15.43:0/1299835143 conn(0x7f04980b0ba0 msgr2=0x7f049816b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 shutdown_connections 2024-01-16T15:06:41.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:41.872+0000 7f049d800640 1 -- 172.21.15.43:0/1299835143 wait complete. 2024-01-16T15:06:41.883 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:42.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53195 2024-01-16T15:06:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.231+0000 7fc189c00640 1 Processor -- start 2024-01-16T15:06:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.231+0000 7fc189c00640 1 -- start start 2024-01-16T15:06:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.234+0000 7fc189c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 0x7fc184070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.234+0000 7fc189c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1840b3870 0x7fc1840b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.234+0000 7fc189c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184143c60 con 0x7fc184143880 2024-01-16T15:06:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.234+0000 7fc189c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840cc9e0 con 0x7fc1840b3870 2024-01-16T15:06:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.234+0000 7fc189c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840b4220 con 0x7fc1840b2da0 2024-01-16T15:06:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.235+0000 7fc189200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1840b3870 0x7fc1840b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.235+0000 7fc183400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 0x7fc184070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.235+0000 7fc183400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 0x7fc184070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37348/0 (socket says 172.21.15.43:37348) 2024-01-16T15:06:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.235+0000 7fc183400640 1 -- 172.21.15.43:0/3860274901 learned_addr learned my addr 172.21.15.43:0/3860274901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.236+0000 7fc189200640 1 -- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 msgr2=0x7fc184070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.236+0000 7fc189200640 1 --2- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 0x7fc184070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.236+0000 7fc189200640 1 -- 172.21.15.43:0/3860274901 >> v1:172.21.15.43:6789/0 conn(0x7fc184143880 legacy=0x7fc1840cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.236+0000 7fc189200640 1 -- 172.21.15.43:0/3860274901 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840cd7f0 con 0x7fc1840b3870 2024-01-16T15:06:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.238+0000 7fc189200640 1 --2- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b3870 0x7fc1840b3ce0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009d90 tx=0x7fc17c02bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=352df98465f9a7a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.238+0000 7fc182a00640 1 -- 172.21.15.43:0/3860274901 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c02cb60 con 0x7fc1840b3870 2024-01-16T15:06:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.239+0000 7fc182a00640 1 -- 172.21.15.43:0/3860274901 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c02ccc0 con 0x7fc1840b3870 2024-01-16T15:06:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.239+0000 7fc182a00640 1 -- 172.21.15.43:0/3860274901 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c034660 con 0x7fc1840b3870 2024-01-16T15:06:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.240+0000 7fc189c00640 1 -- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b3870 msgr2=0x7fc1840b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.240+0000 7fc189c00640 1 --2- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b3870 0x7fc1840b3ce0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009d90 tx=0x7fc17c02bdb0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.241+0000 7fc189c00640 1 -- 172.21.15.43:0/3860274901 shutdown_connections 2024-01-16T15:06:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.241+0000 7fc189c00640 1 --2- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b2da0 0x7fc184070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.241+0000 7fc189c00640 1 --2- 172.21.15.43:0/3860274901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b3870 0x7fc1840b3ce0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.241+0000 7fc189c00640 1 -- 172.21.15.43:0/3860274901 >> 172.21.15.43:0/3860274901 conn(0x7fc18406ae50 msgr2=0x7fc1840cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.245+0000 7fc189c00640 1 -- 172.21.15.43:0/3860274901 shutdown_connections 2024-01-16T15:06:43.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.245+0000 7fc189c00640 1 -- 172.21.15.43:0/3860274901 wait complete. 2024-01-16T15:06:43.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.249+0000 7fc189c00640 1 Processor -- start 2024-01-16T15:06:43.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.249+0000 7fc189c00640 1 -- start start 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.252+0000 7fc189c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 0x7fc18412f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.252+0000 7fc189c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b3870 0x7fc184071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.252+0000 7fc189c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 0x7fc184153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.252+0000 7fc189c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18412f7f0 con 0x7fc184143880 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.252+0000 7fc189c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18412fad0 con 0x7fc1840b2da0 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc189c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18412fdb0 con 0x7fc1840b3870 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc189200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 0x7fc184153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc183e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 0x7fc18412f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc183e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 0x7fc18412f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33550/0 (socket says 172.21.15.43:33550) 2024-01-16T15:06:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc183e00640 1 -- 172.21.15.43:0/138918641 learned_addr learned my addr 172.21.15.43:0/138918641 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.253+0000 7fc183400640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b3870 0x7fc184071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.257+0000 7fc189200640 1 -- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b3870 msgr2=0x7fc184071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.257+0000 7fc189200640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b3870 0x7fc184071f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.257+0000 7fc189200640 1 -- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 msgr2=0x7fc18412f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.257+0000 7fc189200640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 0x7fc18412f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.257+0000 7fc189200640 1 -- 172.21.15.43:0/138918641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c009a70 con 0x7fc184143880 2024-01-16T15:06:43.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.258+0000 7fc189200640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 0x7fc184153b10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc17c033000 tx=0x7fc17c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.261+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c03e610 con 0x7fc184143880 2024-01-16T15:06:43.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.263+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c03e790 con 0x7fc184143880 2024-01-16T15:06:43.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.263+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc184130090 con 0x7fc184143880 2024-01-16T15:06:43.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.263+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c03eb40 con 0x7fc184143880 2024-01-16T15:06:43.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.263+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc184072450 con 0x7fc184143880 2024-01-16T15:06:43.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.265+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1841305b0 con 0x7fc184143880 2024-01-16T15:06:43.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.268+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc17c03b050 con 0x7fc184143880 2024-01-16T15:06:43.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.269+0000 7fc17be00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 0x7fc15c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:43.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.270+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc17c0aafc0 con 0x7fc184143880 2024-01-16T15:06:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.276+0000 7fc183e00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 0x7fc15c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:43.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.280+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1841305b0 con 0x7fc184143880 2024-01-16T15:06:43.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.291+0000 7fc183e00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 0x7fc15c06dff0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009990 tx=0x7fc16c009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.698+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1840cd1a0 con 0x7fc15c06bb30 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.729+0000 7fc17be00640 1 -- 172.21.15.43:0/138918641 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fc16c0095e0 con 0x7fc15c06bb30 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:06:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "1/3", 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:43.709 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 msgr2=0x7fc15c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 0x7fc15c06dff0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009990 tx=0x7fc16c009110 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 msgr2=0x7fc184153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 0x7fc184153b10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc17c033000 tx=0x7fc17c004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 shutdown_connections 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc15c06bb30 0x7fc15c06dff0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc184143880 0x7fc184153b10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1840b3870 0x7fc184071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 --2- 172.21.15.43:0/138918641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1840b2da0 0x7fc18412f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 >> 172.21.15.43:0/138918641 conn(0x7fc18406ae50 msgr2=0x7fc184135130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.736+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 shutdown_connections 2024-01-16T15:06:43.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:43.737+0000 7fc189c00640 1 -- 172.21.15.43:0/138918641 wait complete. 2024-01-16T15:06:43.774 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:44.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53195 2024-01-16T15:06:45.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.220+0000 7fbd60a00640 1 Processor -- start 2024-01-16T15:06:45.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.221+0000 7fbd60a00640 1 -- start start 2024-01-16T15:06:45.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.223+0000 7fbd60a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd58162600 0x7fbd58162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.223+0000 7fbd60a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbd5806c3c0 0x7fbd5806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.224+0000 7fbd5e000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbd58162220 0x7fbd580b1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:33744/0 (socket says 172.21.15.43:33744) 2024-01-16T15:06:45.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.224+0000 7fbd5e000640 1 -- 172.21.15.43:0/934403015 learned_addr learned my addr 172.21.15.43:0/934403015 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:45.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.225+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58162fa0 con 0x7fbd58162220 2024-01-16T15:06:45.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.225+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5806cd90 con 0x7fbd58162600 2024-01-16T15:06:45.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.225+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd580706a0 con 0x7fbd5806c3c0 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5ea00640 1 --2- 172.21.15.43:0/934403015 >> v2:172.21.15.69:3300/0 conn(0x7fbd5806c3c0 0x7fbd5806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016024322 0 0) 0x7fbd58162fa0 con 0x7fbd58162220 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd38003520 con 0x7fbd58162220 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 65606196 0 0) 0x7fbd38003520 con 0x7fbd58162220 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 >> v2:172.21.15.69:3300/0 conn(0x7fbd5806c3c0 msgr2=0x7fbd5806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 --2- 172.21.15.43:0/934403015 >> v2:172.21.15.69:3300/0 conn(0x7fbd5806c3c0 0x7fbd5806c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.226+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 >> v2:172.21.15.43:3300/0 conn(0x7fbd58162600 msgr2=0x7fbd58162a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:06:45.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 --2- 172.21.15.43:0/934403015 >> v2:172.21.15.43:3300/0 conn(0x7fbd58162600 0x7fbd58162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58070c60 con 0x7fbd58162220 2024-01-16T15:06:45.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fbd40001670 con 0x7fbd58162220 2024-01-16T15:06:45.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 >> v1:172.21.15.69:6789/0 conn(0x7fbd58162220 legacy=0x7fbd580b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 0x7fbd38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd380047c0 0x7fbd38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd38005180 0x7fbd3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5806cd90 con 0x7fbd38003970 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580706a0 con 0x7fbd38005180 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3800c150 con 0x7fbd380047c0 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.227+0000 7fbd5ea00640 1 --2- 172.21.15.43:0/934403015 >> v2:172.21.15.69:3300/0 conn(0x7fbd5806c3c0 0x7fbd5806c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5d600640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd380047c0 0x7fbd38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5e000640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 0x7fbd38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5e000640 1 -- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd380047c0 msgr2=0x7fbd38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5e000640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd380047c0 0x7fbd38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5e000640 1 -- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd38005180 msgr2=0x7fbd3800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.229+0000 7fbd5e000640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd38005180 0x7fbd3800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.230+0000 7fbd5e000640 1 -- 172.21.15.43:0/934403015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58070c60 con 0x7fbd38003970 2024-01-16T15:06:45.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.230+0000 7fbd5ea00640 1 -- 172.21.15.43:0/934403015 reap_dead start 2024-01-16T15:06:45.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.235+0000 7fbd5e000640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 0x7fbd38003d90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c210 tx=0x7fbd380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bccf8844c6bb81cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.237+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4002cae0 con 0x7fbd38003970 2024-01-16T15:06:45.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.237+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4002cc40 con 0x7fbd38003970 2024-01-16T15:06:45.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.237+0000 7fbd5cc00640 1 -- 172.21.15.43:0/934403015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40005af0 con 0x7fbd38003970 2024-01-16T15:06:45.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.239+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 msgr2=0x7fbd38003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.239+0000 7fbd60a00640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 0x7fbd38003d90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c210 tx=0x7fbd380057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.243+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 shutdown_connections 2024-01-16T15:06:45.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.243+0000 7fbd60a00640 1 --2- 172.21.15.43:0/934403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd38003970 0x7fbd38003d90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.243+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 >> 172.21.15.43:0/934403015 conn(0x7fbd580b0ba0 msgr2=0x7fbd580b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.244+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 shutdown_connections 2024-01-16T15:06:45.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.245+0000 7fbd60a00640 1 -- 172.21.15.43:0/934403015 wait complete. 2024-01-16T15:06:45.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.250+0000 7fbd60a00640 1 Processor -- start 2024-01-16T15:06:45.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.251+0000 7fbd60a00640 1 -- start start 2024-01-16T15:06:45.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.254+0000 7fbd60a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 0x7fbd5809c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.255+0000 7fbd60a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 0x7fbd581fa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.256+0000 7fbd60a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.256+0000 7fbd60a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581fafa0 con 0x7fbd58165050 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.256+0000 7fbd5e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 0x7fbd5809c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.256+0000 7fbd5e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 0x7fbd5809c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37368/0 (socket says 172.21.15.43:37368) 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.257+0000 7fbd60a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58099ae0 con 0x7fbd58164bf0 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.257+0000 7fbd60a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58099dc0 con 0x7fbd38003970 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.257+0000 7fbd5ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.257+0000 7fbd5ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43764/0 (socket says 172.21.15.43:43764) 2024-01-16T15:06:45.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.257+0000 7fbd5ea00640 1 -- 172.21.15.43:0/3646158424 learned_addr learned my addr 172.21.15.43:0/3646158424 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:45.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.258+0000 7fbd5ea00640 1 -- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 msgr2=0x7fbd5809c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.258+0000 7fbd5ea00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 0x7fbd5809c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.258+0000 7fbd5ea00640 1 -- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 msgr2=0x7fbd581fa520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:06:45.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.262+0000 7fbd5d600640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 0x7fbd581fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.262+0000 7fbd5ea00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 0x7fbd581fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.262+0000 7fbd5ea00640 1 -- 172.21.15.43:0/3646158424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5809c850 con 0x7fbd58165050 2024-01-16T15:06:45.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.262+0000 7fbd5d600640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 0x7fbd581fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:45.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.263+0000 7fbd5ea00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806e970 tx=0x7fbd5000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.264+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000dd00 con 0x7fbd58165050 2024-01-16T15:06:45.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.265+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5000f040 con 0x7fbd58165050 2024-01-16T15:06:45.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.265+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50003bd0 con 0x7fbd58165050 2024-01-16T15:06:45.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.266+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd580ad4e0 con 0x7fbd58165050 2024-01-16T15:06:45.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.266+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd580adaa0 con 0x7fbd58165050 2024-01-16T15:06:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.268+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5809d200 con 0x7fbd58165050 2024-01-16T15:06:45.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.273+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fbd50003d30 con 0x7fbd58165050 2024-01-16T15:06:45.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.275+0000 7fbd56000640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 0x7fbd3006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:45.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.276+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbd50088a30 con 0x7fbd58165050 2024-01-16T15:06:45.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.282+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd50055090 con 0x7fbd58165050 2024-01-16T15:06:45.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.291+0000 7fbd5e000640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 0x7fbd3006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:45.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.292+0000 7fbd5e000640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 0x7fbd3006e0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbd40034000 tx=0x7fbd40007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:45.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.767+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd580708f0 con 0x7fbd3006bc00 2024-01-16T15:06:45.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.830+0000 7fbd56000640 1 -- 172.21.15.43:0/3646158424 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fbd4003a040 con 0x7fbd3006bc00 2024-01-16T15:06:45.809 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:45.809 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:06:45.810 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:06:45.811 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:45.811 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:45.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:45.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:45.811 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 msgr2=0x7fbd3006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 0x7fbd3006e0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbd40034000 tx=0x7fbd40007000 comp rx=0 tx=0).stop 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 msgr2=0x7fbd581faa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806e970 tx=0x7fbd5000d4a0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 shutdown_connections 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd3006bc00 0x7fbd3006e0c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd58164bf0 0x7fbd581fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd58165050 0x7fbd581faa60 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 --2- 172.21.15.43:0/3646158424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd38003970 0x7fbd5809c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.835+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 >> 172.21.15.43:0/3646158424 conn(0x7fbd580b0ba0 msgr2=0x7fbd580b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.836+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 shutdown_connections 2024-01-16T15:06:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:45.836+0000 7fbd60a00640 1 -- 172.21.15.43:0/3646158424 wait complete. 2024-01-16T15:06:45.862 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:46.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53195 2024-01-16T15:06:47.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.223+0000 7f41e2000640 1 Processor -- start 2024-01-16T15:06:47.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.223+0000 7f41e2000640 1 -- start start 2024-01-16T15:06:47.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.224+0000 7f41e2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 0x7f41dc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41e2000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0706a0 con 0x7f41dc06cad0 2024-01-16T15:06:47.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41e2000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc070c60 con 0x7f41dc162600 2024-01-16T15:06:47.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41e2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0722b0 con 0x7f41dc162220 2024-01-16T15:06:47.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41daa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f41dc162600 0x7f41dc06c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:33748/0 (socket says 172.21.15.43:33748) 2024-01-16T15:06:47.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41daa00640 1 -- 172.21.15.43:0/2272749981 learned_addr learned my addr 172.21.15.43:0/2272749981 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.225+0000 7f41db400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 0x7f41dc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.226+0000 7f41da000640 1 -- 172.21.15.43:0/2272749981 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537619917 0 0) 0x7f41dc070c60 con 0x7f41dc162600 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.226+0000 7f41da000640 1 -- 172.21.15.43:0/2272749981 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41bc003520 con 0x7f41dc162600 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.226+0000 7f41db400640 1 -- 172.21.15.43:0/2272749981 >> v1:172.21.15.69:6789/0 conn(0x7f41dc162600 legacy=0x7f41dc06c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.226+0000 7f41db400640 1 -- 172.21.15.43:0/2272749981 >> v1:172.21.15.43:6789/0 conn(0x7f41dc06cad0 legacy=0x7f41dc0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.226+0000 7f41db400640 1 -- 172.21.15.43:0/2272749981 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc072590 con 0x7f41dc162220 2024-01-16T15:06:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.228+0000 7f41db400640 1 --2- 172.21.15.43:0/2272749981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 0x7f41dc0b1ea0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097d0 tx=0x7f41cc02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8665ee4305e7cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.229+0000 7f41da000640 1 -- 172.21.15.43:0/2272749981 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc030800 con 0x7f41dc162220 2024-01-16T15:06:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.229+0000 7f41da000640 1 -- 172.21.15.43:0/2272749981 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc030960 con 0x7f41dc162220 2024-01-16T15:06:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.229+0000 7f41da000640 1 -- 172.21.15.43:0/2272749981 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc030cf0 con 0x7f41dc162220 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.233+0000 7f41e2000640 1 -- 172.21.15.43:0/2272749981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 msgr2=0x7f41dc0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.233+0000 7f41e2000640 1 --2- 172.21.15.43:0/2272749981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 0x7f41dc0b1ea0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097d0 tx=0x7f41cc02b940 comp rx=0 tx=0).stop 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.233+0000 7f41e2000640 1 -- 172.21.15.43:0/2272749981 shutdown_connections 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.233+0000 7f41e2000640 1 --2- 172.21.15.43:0/2272749981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162220 0x7f41dc0b1ea0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.233+0000 7f41e2000640 1 -- 172.21.15.43:0/2272749981 >> 172.21.15.43:0/2272749981 conn(0x7f41dc0b0ba0 msgr2=0x7f41dc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.234+0000 7f41e2000640 1 -- 172.21.15.43:0/2272749981 shutdown_connections 2024-01-16T15:06:47.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.236+0000 7f41e2000640 1 -- 172.21.15.43:0/2272749981 wait complete. 2024-01-16T15:06:47.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.240+0000 7f41e2000640 1 Processor -- start 2024-01-16T15:06:47.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.240+0000 7f41e2000640 1 -- start start 2024-01-16T15:06:47.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 0x7f41dc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162600 0x7f41dc0ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc09c890 con 0x7f41dc162220 2024-01-16T15:06:47.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc153950 con 0x7f41dc06cad0 2024-01-16T15:06:47.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.242+0000 7f41e2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc153400 con 0x7f41dc162600 2024-01-16T15:06:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.243+0000 7f41daa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 0x7f41dc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.244+0000 7f41db400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.244+0000 7f41db400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:33596/0 (socket says 172.21.15.43:33596) 2024-01-16T15:06:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.244+0000 7f41db400640 1 -- 172.21.15.43:0/801061635 learned_addr learned my addr 172.21.15.43:0/801061635 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.244+0000 7f41dbe00640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162600 0x7f41dc0ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.246+0000 7f41db400640 1 -- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162600 msgr2=0x7f41dc0ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.246+0000 7f41db400640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162600 0x7f41dc0ac3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.246+0000 7f41db400640 1 -- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 msgr2=0x7f41dc0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.246+0000 7f41db400640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 0x7f41dc0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.246+0000 7f41db400640 1 -- 172.21.15.43:0/801061635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc009400 con 0x7f41dc06cad0 2024-01-16T15:06:47.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.247+0000 7f41daa00640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 0x7f41dc0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:47.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.247+0000 7f41db400640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0098f0 tx=0x7f41cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.248+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc03e070 con 0x7f41dc06cad0 2024-01-16T15:06:47.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.248+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc039a10 con 0x7f41dc06cad0 2024-01-16T15:06:47.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.248+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc039dc0 con 0x7f41dc06cad0 2024-01-16T15:06:47.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.249+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc09c5b0 con 0x7f41dc06cad0 2024-01-16T15:06:47.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.249+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc156090 con 0x7f41dc06cad0 2024-01-16T15:06:47.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.252+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc070db0 con 0x7f41dc06cad0 2024-01-16T15:06:47.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.256+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f41cc069410 con 0x7f41dc06cad0 2024-01-16T15:06:47.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.257+0000 7f41d3400640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 0x7f41b406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.258+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f41cc0aaf00 con 0x7f41dc06cad0 2024-01-16T15:06:47.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.272+0000 7f41daa00640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 0x7f41b406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.276+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41cc0776e0 con 0x7f41dc06cad0 2024-01-16T15:06:47.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.282+0000 7f41daa00640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 0x7f41b406e040 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f41dc1fb030 tx=0x7f41c4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.869+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7f41dc15f2a0 con 0x7f41b406bb80 2024-01-16T15:06:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.986+0000 7f41d3400640 1 -- 172.21.15.43:0/801061635 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f41dc15f2a0 con 0x7f41b406bb80 2024-01-16T15:06:47.965 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:47.968 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.992+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 msgr2=0x7f41b406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.992+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 0x7f41b406e040 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f41dc1fb030 tx=0x7f41c4007d30 comp rx=0 tx=0).stop 2024-01-16T15:06:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.992+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 msgr2=0x7f41dc152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.992+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0098f0 tx=0x7f41cc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 shutdown_connections 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f41b406bb80 0x7f41b406e040 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc162600 0x7f41dc0ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41dc162220 0x7f41dc0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 --2- 172.21.15.43:0/801061635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc06cad0 0x7f41dc152ec0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.993+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 >> 172.21.15.43:0/801061635 conn(0x7f41dc0b0ba0 msgr2=0x7f41dc1606a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.994+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 shutdown_connections 2024-01-16T15:06:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:47.994+0000 7f41e2000640 1 -- 172.21.15.43:0/801061635 wait complete. 2024-01-16T15:06:48.015 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:49.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53195 2024-01-16T15:06:49.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.461+0000 7f7902c00640 1 Processor -- start 2024-01-16T15:06:49.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.461+0000 7f7902c00640 1 -- start start 2024-01-16T15:06:49.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7902c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78fc160bd0 0x7f78fc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7902c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc0b23e0 con 0x7f78fc0b3210 2024-01-16T15:06:49.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7902c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc0d41b0 con 0x7f78fc0b48d0 2024-01-16T15:06:49.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7902c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0cdc70 con 0x7f78fc160bd0 2024-01-16T15:06:49.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f78fbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78fc160bd0 0x7f78fc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7900c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f78fc0b3210 0x7f78fc167290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56562/0 (socket says 172.21.15.43:56562) 2024-01-16T15:06:49.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.464+0000 7f7900c00640 1 -- 172.21.15.43:0/1741481495 learned_addr learned my addr 172.21.15.43:0/1741481495 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:49.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.465+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815129818 0 0) 0x7f78fc0b23e0 con 0x7f78fc0b3210 2024-01-16T15:06:49.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.467+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc003520 con 0x7f78fc0b3210 2024-01-16T15:06:49.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.467+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843553231 0 0) 0x7f78fc0d41b0 con 0x7f78fc0b48d0 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.467+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc0b23e0 con 0x7f78fc0b48d0 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4233274232 0 0) 0x7f78dc003520 con 0x7f78fc0b3210 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 >> v2:172.21.15.69:3300/0 conn(0x7f78fc160bd0 msgr2=0x7f78fc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 --2- 172.21.15.43:0/1741481495 >> v2:172.21.15.69:3300/0 conn(0x7f78fc160bd0 0x7f78fc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 >> v1:172.21.15.69:6789/0 conn(0x7f78fc0b48d0 legacy=0x7f78fc0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc160fb0 con 0x7f78fc0b3210 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f78f40019f0 con 0x7f78fc0b3210 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 >> v1:172.21.15.43:6789/0 conn(0x7f78fc0b3210 legacy=0x7f78fc167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78dc004b90 0x7f78dc004f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78dc003760 0x7f78dc003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec001230 con 0x7f78dc004120 2024-01-16T15:06:49.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0cdc70 con 0x7f78dc004b90 2024-01-16T15:06:49.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc00c100 con 0x7f78dc003760 2024-01-16T15:06:49.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.468+0000 7f78fbe00640 1 --2- 172.21.15.43:0/1741481495 >> v2:172.21.15.69:3300/0 conn(0x7f78fc160bd0 0x7f78fc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:49.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.470+0000 7f78fb400640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78dc003760 0x7f78dc003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.471+0000 7f7900c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.471+0000 7f78fbe00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78dc004b90 0x7f78dc004f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f7900c00640 1 -- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78dc003760 msgr2=0x7f78dc003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f7900c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78dc003760 0x7f78dc003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f7900c00640 1 -- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78dc004b90 msgr2=0x7f78dc004f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f7900c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78dc004b90 0x7f78dc004f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f7900c00640 1 -- 172.21.15.43:0/1741481495 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc160fb0 con 0x7f78dc004120 2024-01-16T15:06:49.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.472+0000 7f78fbe00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78dc004b90 0x7f78dc004f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:49.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.475+0000 7f7900c00640 1 -- 172.21.15.43:0/1741481495 reap_dead start 2024-01-16T15:06:49.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.476+0000 7f7900c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78dc00b980 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f78f400aaf0 tx=0x7f78dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608c3b97ab096b92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.476+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f40072d0 con 0x7f78dc004120 2024-01-16T15:06:49.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.476+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f4007450 con 0x7f78dc004120 2024-01-16T15:06:49.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.476+0000 7f78faa00640 1 -- 172.21.15.43:0/1741481495 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4007800 con 0x7f78dc004120 2024-01-16T15:06:49.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.478+0000 7f7902c00640 1 -- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 msgr2=0x7f78dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:49.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.478+0000 7f7902c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78dc00b980 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f78f400aaf0 tx=0x7f78dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.479+0000 7f7902c00640 1 -- 172.21.15.43:0/1741481495 shutdown_connections 2024-01-16T15:06:49.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.479+0000 7f7902c00640 1 --2- 172.21.15.43:0/1741481495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78dc00b980 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.479+0000 7f7902c00640 1 -- 172.21.15.43:0/1741481495 >> 172.21.15.43:0/1741481495 conn(0x7f78fc09c5a0 msgr2=0x7f78fc0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:49.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.479+0000 7f7902c00640 1 -- 172.21.15.43:0/1741481495 shutdown_connections 2024-01-16T15:06:49.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.479+0000 7f7902c00640 1 -- 172.21.15.43:0/1741481495 wait complete. 2024-01-16T15:06:49.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.483+0000 7f7902c00640 1 Processor -- start 2024-01-16T15:06:49.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.484+0000 7f7902c00640 1 -- start start 2024-01-16T15:06:49.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.487+0000 7f7902c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.487+0000 7f7902c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc0c6d70 0x7f78fc147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f7902c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc148100 0x7f78fc0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f7902c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1477a0 con 0x7f78dc004120 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f7902c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0c7210 con 0x7f78fc148100 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f7902c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1470e0 con 0x7f78fc0c6d70 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f78fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f78fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43786/0 (socket says 172.21.15.43:43786) 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.488+0000 7f78fbe00640 1 -- 172.21.15.43:0/3992917674 learned_addr learned my addr 172.21.15.43:0/3992917674 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 -- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc0c6d70 msgr2=0x7f78fc147bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc0c6d70 0x7f78fc147bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 -- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc148100 msgr2=0x7f78fc0cddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc148100 0x7f78fc0cddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 -- 172.21.15.43:0/3992917674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc1473c0 con 0x7f78dc004120 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.489+0000 7f78fbe00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.490+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e401b040 con 0x7f78dc004120 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.490+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e401e790 con 0x7f78dc004120 2024-01-16T15:06:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.490+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e401eb00 con 0x7f78dc004120 2024-01-16T15:06:49.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.490+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0c74f0 con 0x7f78dc004120 2024-01-16T15:06:49.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.490+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0c9ca0 con 0x7f78dc004120 2024-01-16T15:06:49.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.496+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f78e401ec60 con 0x7f78dc004120 2024-01-16T15:06:49.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.497+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc005250 con 0x7f78dc004120 2024-01-16T15:06:49.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.498+0000 7f78f3e00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 0x7f78d406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:49.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.498+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f78e4091340 con 0x7f78dc004120 2024-01-16T15:06:49.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.513+0000 7f78fb400640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 0x7f78d406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:49.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.515+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78e405db80 con 0x7f78dc004120 2024-01-16T15:06:49.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.532+0000 7f78fb400640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 0x7f78d406dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f78fc15a160 tx=0x7f78ec001450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:49.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:49.956+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7f78cc000bc0 con 0x7f78d406bb30 2024-01-16T15:06:50.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.290+0000 7f78f3e00640 1 -- 172.21.15.43:0/3992917674 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f78cc000bc0 con 0x7f78d406bb30 2024-01-16T15:06:50.269 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:50.269 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 msgr2=0x7f78d406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 0x7f78d406dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f78fc15a160 tx=0x7f78ec001450 comp rx=0 tx=0).stop 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 msgr2=0x7f78fc146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009d90 tx=0x7f78e40039e0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 shutdown_connections 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f78d406bb30 0x7f78d406dff0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78fc148100 0x7f78fc0cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78fc0c6d70 0x7f78fc147bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 --2- 172.21.15.43:0/3992917674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78dc004120 0x7f78fc146830 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.297+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 >> 172.21.15.43:0/3992917674 conn(0x7f78fc09c5a0 msgr2=0x7f78fc15f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.298+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 shutdown_connections 2024-01-16T15:06:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.298+0000 7f7902c00640 1 -- 172.21.15.43:0/3992917674 wait complete. 2024-01-16T15:06:50.339 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:50.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53196 2024-01-16T15:06:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.810+0000 7f9463c00640 1 Processor -- start 2024-01-16T15:06:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.810+0000 7f9463c00640 1 -- start start 2024-01-16T15:06:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f945c06a440 0x7f945c06a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f945c071be0 0x7f945c072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c15c410 con 0x7f945c071be0 2024-01-16T15:06:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0ce830 con 0x7f945c06a440 2024-01-16T15:06:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.812+0000 7f9463c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c06ad60 con 0x7f945c06d4d0 2024-01-16T15:06:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.813+0000 7f945be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f945c071be0 0x7f945c072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.813+0000 7f945be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f945c071be0 0x7f945c072090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43792/0 (socket says 172.21.15.43:43792) 2024-01-16T15:06:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.813+0000 7f945be00640 1 -- 172.21.15.43:0/1200543082 learned_addr learned my addr 172.21.15.43:0/1200543082 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f9461200640 1 --2- 172.21.15.43:0/1200543082 >> v2:172.21.15.69:3300/0 conn(0x7f945c06a440 0x7f945c06a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f9461c00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f945be00640 1 -- 172.21.15.43:0/1200543082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 msgr2=0x7f945c06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f945be00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c06d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f945be00640 1 -- 172.21.15.43:0/1200543082 >> v2:172.21.15.69:3300/0 conn(0x7f945c06a440 msgr2=0x7f945c06a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f945be00640 1 --2- 172.21.15.43:0/1200543082 >> v2:172.21.15.69:3300/0 conn(0x7f945c06a440 0x7f945c06a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.814+0000 7f945be00640 1 -- 172.21.15.43:0/1200543082 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0725d0 con 0x7f945c071be0 2024-01-16T15:06:50.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.818+0000 7f945be00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c072090 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f945c0b5620 tx=0x7f944c02bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7277eec07602cde server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.819+0000 7f945b400640 1 -- 172.21.15.43:0/1200543082 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c02d690 con 0x7f945c071be0 2024-01-16T15:06:50.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.819+0000 7f945b400640 1 -- 172.21.15.43:0/1200543082 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c02d7f0 con 0x7f945c071be0 2024-01-16T15:06:50.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.819+0000 7f945b400640 1 -- 172.21.15.43:0/1200543082 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c02db80 con 0x7f945c071be0 2024-01-16T15:06:50.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.820+0000 7f9463c00640 1 -- 172.21.15.43:0/1200543082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 msgr2=0x7f945c072090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.820+0000 7f9463c00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c072090 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f945c0b5620 tx=0x7f944c02bae0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 -- 172.21.15.43:0/1200543082 shutdown_connections 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 --2- 172.21.15.43:0/1200543082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c072090 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 --2- 172.21.15.43:0/1200543082 >> v2:172.21.15.69:3300/0 conn(0x7f945c06a440 0x7f945c06a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 -- 172.21.15.43:0/1200543082 >> 172.21.15.43:0/1200543082 conn(0x7f945c06b550 msgr2=0x7f945c06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.822+0000 7f9463c00640 1 -- 172.21.15.43:0/1200543082 shutdown_connections 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.823+0000 7f9463c00640 1 -- 172.21.15.43:0/1200543082 wait complete. 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.826+0000 7f9463c00640 1 Processor -- start 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.827+0000 7f9463c00640 1 -- start start 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c06a440 0x7f945c14b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c145470 con 0x7f945c071be0 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1445c0 con 0x7f945c06a440 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9463c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c144860 con 0x7f945c06d4d0 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9461c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9461c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43798/0 (socket says 172.21.15.43:43798) 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.828+0000 7f9461c00640 1 -- 172.21.15.43:0/658038959 learned_addr learned my addr 172.21.15.43:0/658038959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.829+0000 7f9461200640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c06a440 0x7f945c14b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.829+0000 7f945be00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.830+0000 7f945be00640 1 -- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c06a440 msgr2=0x7f945c14b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.830+0000 7f945be00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c06a440 0x7f945c14b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.830+0000 7f945be00640 1 -- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 msgr2=0x7f945c144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.831+0000 7f945be00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.831+0000 7f945be00640 1 -- 172.21.15.43:0/658038959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c0094d0 con 0x7f945c06d4d0 2024-01-16T15:06:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.831+0000 7f9461c00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.832+0000 7f945be00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c1568b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f944c033000 tx=0x7f944c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.834+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c03d070 con 0x7f945c06d4d0 2024-01-16T15:06:50.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.834+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c037690 con 0x7f945c06d4d0 2024-01-16T15:06:50.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.834+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c037a20 con 0x7f945c06d4d0 2024-01-16T15:06:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.835+0000 7f9463c00640 1 -- 172.21.15.43:0/658038959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c144b40 con 0x7f945c06d4d0 2024-01-16T15:06:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.839+0000 7f9463c00640 1 -- 172.21.15.43:0/658038959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c145100 con 0x7f945c06d4d0 2024-01-16T15:06:50.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.841+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9424005250 con 0x7f945c06d4d0 2024-01-16T15:06:50.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.849+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f944c037b80 con 0x7f945c06d4d0 2024-01-16T15:06:50.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.852+0000 7f9458c00640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 0x7f943406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.853+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f944c032070 con 0x7f945c06d4d0 2024-01-16T15:06:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.855+0000 7f9461200640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 0x7f943406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.857+0000 7f9461200640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 0x7f943406e040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f944400a9b0 tx=0x7f9444008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:50.858+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f944c076c10 con 0x7f945c06d4d0 2024-01-16T15:06:51.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.419+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53196", "target": ["mon-mgr", ""]}) v1 -- 0x7f9424000bc0 con 0x7f943406bb80 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.543+0000 7f9458c00640 1 -- 172.21.15.43:0/658038959 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f9424000bc0 con 0x7f943406bb80 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:51.522 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 msgr2=0x7f943406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 0x7f943406e040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f944400a9b0 tx=0x7f9444008040 comp rx=0 tx=0).stop 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 msgr2=0x7f945c1568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c1568b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f944c033000 tx=0x7f944c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 shutdown_connections 2024-01-16T15:06:51.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f943406bb80 0x7f943406e040 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f945c071be0 0x7f945c144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f945c06d4d0 0x7f945c1568b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 --2- 172.21.15.43:0/658038959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f945c06a440 0x7f945c14b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.547+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 >> 172.21.15.43:0/658038959 conn(0x7f945c06b550 msgr2=0x7f945c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.549+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 shutdown_connections 2024-01-16T15:06:51.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:51.549+0000 7f9452000640 1 -- 172.21.15.43:0/658038959 wait complete. 2024-01-16T15:06:51.569 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:52.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53196 2024-01-16T15:06:52.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.916+0000 7f75a5200640 1 Processor -- start 2024-01-16T15:06:52.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.917+0000 7f75a5200640 1 -- start start 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f75a5200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75a015d3b0 0x7f75a00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f75a5200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a00d1d90 0x7f75a00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f75a5200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a00c6bf0 con 0x7f75a00d0fa0 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f75a5200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a00d1430 con 0x7f75a015d3b0 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f75a5200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00cedc0 con 0x7f75a00d1d90 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f759f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75a015d3b0 0x7f75a00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f759f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f75a015d3b0 0x7f75a00ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43814/0 (socket says 172.21.15.43:43814) 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f759f400640 1 -- 172.21.15.43:0/542876343 learned_addr learned my addr 172.21.15.43:0/542876343 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.918+0000 7f759fe00640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a00d1d90 0x7f75a00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.919+0000 7f759f400640 1 -- 172.21.15.43:0/542876343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a00d1d90 msgr2=0x7f75a00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.919+0000 7f759f400640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a00d1d90 0x7f75a00b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.919+0000 7f759f400640 1 -- 172.21.15.43:0/542876343 >> v1:172.21.15.69:6789/0 conn(0x7f75a00d0fa0 legacy=0x7f75a00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.919+0000 7f759f400640 1 -- 172.21.15.43:0/542876343 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00c6e40 con 0x7f75a015d3b0 2024-01-16T15:06:52.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.920+0000 7f759f400640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a015d3b0 0x7f75a00ce6c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7590009870 tx=0x7f759002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f925db6fe43f9013 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.920+0000 7f759e000640 1 -- 172.21.15.43:0/542876343 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900317c0 con 0x7f75a015d3b0 2024-01-16T15:06:52.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.920+0000 7f759e000640 1 -- 172.21.15.43:0/542876343 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590031920 con 0x7f75a015d3b0 2024-01-16T15:06:52.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.920+0000 7f759e000640 1 -- 172.21.15.43:0/542876343 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590031cb0 con 0x7f75a015d3b0 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 -- 172.21.15.43:0/542876343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a015d3b0 msgr2=0x7f75a00ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a015d3b0 0x7f75a00ce6c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7590009870 tx=0x7f759002b860 comp rx=0 tx=0).stop 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 -- 172.21.15.43:0/542876343 shutdown_connections 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a00d1d90 0x7f75a00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 --2- 172.21.15.43:0/542876343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a015d3b0 0x7f75a00ce6c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 -- 172.21.15.43:0/542876343 >> 172.21.15.43:0/542876343 conn(0x7f75a00d30a0 msgr2=0x7f75a00d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 -- 172.21.15.43:0/542876343 shutdown_connections 2024-01-16T15:06:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.921+0000 7f75a5200640 1 -- 172.21.15.43:0/542876343 wait complete. 2024-01-16T15:06:52.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.923+0000 7f75a5200640 1 Processor -- start 2024-01-16T15:06:52.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.923+0000 7f75a5200640 1 -- start start 2024-01-16T15:06:52.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75a00d0fa0 0x7f75a00b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a015d3b0 0x7f75a009d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00b8da0 con 0x7f75a00d1d90 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00cbd00 con 0x7f75a00d0fa0 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f75a5200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00cbfe0 con 0x7f75a015d3b0 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43816/0 (socket says 172.21.15.43:43816) 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 -- 172.21.15.43:0/3191379457 learned_addr learned my addr 172.21.15.43:0/3191379457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759fe00640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a015d3b0 0x7f75a009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759f400640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75a00d0fa0 0x7f75a00b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 -- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a015d3b0 msgr2=0x7f75a009d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a015d3b0 0x7f75a009d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 -- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75a00d0fa0 msgr2=0x7f75a00b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75a00d0fa0 0x7f75a00b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:52.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.924+0000 7f759ea00640 1 -- 172.21.15.43:0/3191379457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75900094d0 con 0x7f75a00d1d90 2024-01-16T15:06:52.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f759ea00640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f758800c430 tx=0x7f758800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588011790 con 0x7f75a00d1d90 2024-01-16T15:06:52.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880118f0 con 0x7f75a00d1d90 2024-01-16T15:06:52.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588011ce0 con 0x7f75a00d1d90 2024-01-16T15:06:52.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a00cc2c0 con 0x7f75a00d1d90 2024-01-16T15:06:52.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.925+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a00cc880 con 0x7f75a00d1d90 2024-01-16T15:06:52.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.926+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a004e840 con 0x7f75a00d1d90 2024-01-16T15:06:52.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.927+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f758801c460 con 0x7f75a00d1d90 2024-01-16T15:06:52.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.928+0000 7f7597400640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 0x7f757806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:52.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.928+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7588088470 con 0x7f75a00d1d90 2024-01-16T15:06:52.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.928+0000 7f759f400640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 0x7f757806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:52.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.929+0000 7f759f400640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 0x7f757806dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7590036000 tx=0x7f759003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:52.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:52.933+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7588054bc0 con 0x7f75a00d1d90 2024-01-16T15:06:53.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:53.162+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53196", "target": ["mon-mgr", ""]}) v1 -- 0x7f75a00b5a40 con 0x7f757806bb30 2024-01-16T15:06:53.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.021+0000 7f7597400640 1 -- 172.21.15.43:0/3191379457 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f75a00b5a40 con 0x7f757806bb30 2024-01-16T15:06:53.999 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:54.000 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 msgr2=0x7f757806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 0x7f757806dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7590036000 tx=0x7f759003d040 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 msgr2=0x7f75a009cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f758800c430 tx=0x7f758800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 shutdown_connections 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757806bb30 0x7f757806dff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75a015d3b0 0x7f75a009d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75a00d1d90 0x7f75a009cde0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 --2- 172.21.15.43:0/3191379457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75a00d0fa0 0x7f75a00b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 >> 172.21.15.43:0/3191379457 conn(0x7f75a00d30a0 msgr2=0x7f75a006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.024+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 shutdown_connections 2024-01-16T15:06:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.025+0000 7f75a5200640 1 -- 172.21.15.43:0/3191379457 wait complete. 2024-01-16T15:06:54.022 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:06:54.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53197 2024-01-16T15:06:54.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.328+0000 7fb82bc00640 1 Processor -- start 2024-01-16T15:06:54.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.328+0000 7fb82bc00640 1 -- start start 2024-01-16T15:06:54.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.330+0000 7fb82bc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8240abe80 0x7fb824164f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.330+0000 7fb82bc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb82406e040 0x7fb8241674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.330+0000 7fb82bc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240b1ea0 con 0x7fb824070490 2024-01-16T15:06:54.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.330+0000 7fb82bc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8240b2740 con 0x7fb82406e040 2024-01-16T15:06:54.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.330+0000 7fb82bc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82406d520 con 0x7fb8240abe80 2024-01-16T15:06:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.331+0000 7fb829200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb824070490 0x7fb824070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56274/0 (socket says 172.21.15.43:56274) 2024-01-16T15:06:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.331+0000 7fb829200640 1 -- 172.21.15.43:0/3767871703 learned_addr learned my addr 172.21.15.43:0/3767871703 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.331+0000 7fb829c00640 1 --2- 172.21.15.43:0/3767871703 >> v2:172.21.15.43:3300/0 conn(0x7fb82406e040 0x7fb8241674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb81bc00640 1 --2- 172.21.15.43:0/3767871703 >> v2:172.21.15.69:3300/0 conn(0x7fb8240abe80 0x7fb824164f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb823e00640 1 -- 172.21.15.43:0/3767871703 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840200884 0 0) 0x7fb8240b1ea0 con 0x7fb824070490 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb823e00640 1 -- 172.21.15.43:0/3767871703 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb804003520 con 0x7fb824070490 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb829c00640 1 -- 172.21.15.43:0/3767871703 >> v2:172.21.15.69:3300/0 conn(0x7fb8240abe80 msgr2=0x7fb824164f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb829c00640 1 --2- 172.21.15.43:0/3767871703 >> v2:172.21.15.69:3300/0 conn(0x7fb8240abe80 0x7fb824164f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb829c00640 1 -- 172.21.15.43:0/3767871703 >> v1:172.21.15.43:6789/0 conn(0x7fb824070490 legacy=0x7fb824070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.332+0000 7fb829c00640 1 -- 172.21.15.43:0/3767871703 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240b20f0 con 0x7fb82406e040 2024-01-16T15:06:54.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.335+0000 7fb829c00640 1 --2- 172.21.15.43:0/3767871703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb82406e040 0x7fb8241674c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb82406b8b0 tx=0x7fb81c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d272d01315ad46d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.335+0000 7fb823e00640 1 -- 172.21.15.43:0/3767871703 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c02cc70 con 0x7fb82406e040 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.336+0000 7fb823e00640 1 -- 172.21.15.43:0/3767871703 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c02cdd0 con 0x7fb82406e040 2024-01-16T15:06:54.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.336+0000 7fb823e00640 1 -- 172.21.15.43:0/3767871703 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c034660 con 0x7fb82406e040 2024-01-16T15:06:54.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.337+0000 7fb82bc00640 1 -- 172.21.15.43:0/3767871703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb82406e040 msgr2=0x7fb8241674c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.337+0000 7fb82bc00640 1 --2- 172.21.15.43:0/3767871703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb82406e040 0x7fb8241674c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb82406b8b0 tx=0x7fb81c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.338+0000 7fb82bc00640 1 -- 172.21.15.43:0/3767871703 shutdown_connections 2024-01-16T15:06:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.338+0000 7fb82bc00640 1 --2- 172.21.15.43:0/3767871703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb82406e040 0x7fb8241674c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.338+0000 7fb82bc00640 1 --2- 172.21.15.43:0/3767871703 >> v2:172.21.15.69:3300/0 conn(0x7fb8240abe80 0x7fb824164f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.338+0000 7fb82bc00640 1 -- 172.21.15.43:0/3767871703 >> 172.21.15.43:0/3767871703 conn(0x7fb8240b0ba0 msgr2=0x7fb8240b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:54.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.339+0000 7fb82bc00640 1 -- 172.21.15.43:0/3767871703 shutdown_connections 2024-01-16T15:06:54.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.339+0000 7fb82bc00640 1 -- 172.21.15.43:0/3767871703 wait complete. 2024-01-16T15:06:54.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.343+0000 7fb82bc00640 1 Processor -- start 2024-01-16T15:06:54.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.344+0000 7fb82bc00640 1 -- start start 2024-01-16T15:06:54.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.346+0000 7fb82bc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb82406e040 0x7fb82414be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.347+0000 7fb82bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.347+0000 7fb82bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 0x7fb82414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.347+0000 7fb82bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82415f110 con 0x7fb824070490 2024-01-16T15:06:54.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.348+0000 7fb82bc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240ce3d0 con 0x7fb8240abe80 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.348+0000 7fb81bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.348+0000 7fb81bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54826/0 (socket says 172.21.15.43:54826) 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.348+0000 7fb81bc00640 1 -- 172.21.15.43:0/1767819290 learned_addr learned my addr 172.21.15.43:0/1767819290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.349+0000 7fb829c00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 0x7fb82414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.349+0000 7fb829200640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb82406e040 0x7fb82414be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.350+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82414f3d0 con 0x7fb82406e040 2024-01-16T15:06:54.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb81bc00640 1 -- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb82406e040 msgr2=0x7fb82414be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb81bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb82406e040 0x7fb82414be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb81bc00640 1 -- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 msgr2=0x7fb82414ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb81bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 0x7fb82414ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb81bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c009a70 con 0x7fb824070490 2024-01-16T15:06:54.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.351+0000 7fb829c00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 0x7fb82414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:54.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.353+0000 7fb81bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00d5d0 tx=0x7fb80c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.353+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c004240 con 0x7fb824070490 2024-01-16T15:06:54.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.353+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c0043a0 con 0x7fb824070490 2024-01-16T15:06:54.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.353+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c003830 con 0x7fb824070490 2024-01-16T15:06:54.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.354+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82414f6b0 con 0x7fb824070490 2024-01-16T15:06:54.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.355+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82415ed30 con 0x7fb824070490 2024-01-16T15:06:54.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.356+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82415f250 con 0x7fb824070490 2024-01-16T15:06:54.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.361+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb80c003990 con 0x7fb824070490 2024-01-16T15:06:54.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.366+0000 7fb821600640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 0x7fb7fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:54.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.366+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb80c089100 con 0x7fb824070490 2024-01-16T15:06:54.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.368+0000 7fb829200640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 0x7fb7fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:54.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.369+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb80c0557f0 con 0x7fb824070490 2024-01-16T15:06:54.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.371+0000 7fb829200640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 0x7fb7fc06e0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb814007fd0 tx=0x7fb814000bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:54.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.811+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53197", "target": ["mon-mgr", ""]}) v1 -- 0x7fb82406d1b0 con 0x7fb7fc06bc00 2024-01-16T15:06:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.847+0000 7fb821600640 1 -- 172.21.15.43:0/1767819290 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb814018040 con 0x7fb7fc06bc00 2024-01-16T15:06:54.825 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:54.825 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:54.825 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:06:54.825 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "2/3", 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:54.826 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 msgr2=0x7fb7fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 0x7fb7fc06e0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb814007fd0 tx=0x7fb814000bc0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 msgr2=0x7fb82414e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00d5d0 tx=0x7fb80c00da90 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 shutdown_connections 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb7fc06bc00 0x7fb7fc06e0c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8240abe80 0x7fb82414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb824070490 0x7fb82414e7c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 --2- 172.21.15.43:0/1767819290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb82406e040 0x7fb82414be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.853+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 >> 172.21.15.43:0/1767819290 conn(0x7fb8240b0ba0 msgr2=0x7fb8240711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.854+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 shutdown_connections 2024-01-16T15:06:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:54.854+0000 7fb82bc00640 1 -- 172.21.15.43:0/1767819290 wait complete. 2024-01-16T15:06:54.873 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:06:55.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone53197 2024-01-16T15:06:56.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.127+0000 7ff9fae00640 1 Processor -- start 2024-01-16T15:06:56.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.127+0000 7ff9fae00640 1 -- start start 2024-01-16T15:06:56.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.128+0000 7ff9fae00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff9f4153ea0 0x7ff9f40ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.128+0000 7ff9fae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 0x7ff9f415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9fae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4070850 0x7ff9f4071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9fae00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f40ce650 con 0x7ff9f4153ea0 2024-01-16T15:06:56.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9fae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f4154710 con 0x7ff9f4154280 2024-01-16T15:06:56.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9fae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f415ddd0 con 0x7ff9f4070850 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff9f4153ea0 0x7ff9f40ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff9f4153ea0 0x7ff9f40ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54840/0 (socket says 172.21.15.43:54840) 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 -- 172.21.15.43:0/3793572628 learned_addr learned my addr 172.21.15.43:0/3793572628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9000640 1 --2- 172.21.15.43:0/3793572628 >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 0x7ff9f415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 -- 172.21.15.43:0/3793572628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4070850 msgr2=0x7ff9f4071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9fa400640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4070850 0x7ff9f4071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4070850 0x7ff9f4071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 -- 172.21.15.43:0/3793572628 >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 msgr2=0x7ff9f415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/3793572628 >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 0x7ff9f415d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9a00640 1 -- 172.21.15.43:0/3793572628 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f415e0a0 con 0x7ff9f4153ea0 2024-01-16T15:06:56.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.129+0000 7ff9f9000640 1 --2- 172.21.15.43:0/3793572628 >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 0x7ff9f415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:56.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.130+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4153ea0 0x7ff9f40ad450 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4157e50 tx=0x7ff9e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef84390de5a1fb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9f3e00640 1 -- 172.21.15.43:0/3793572628 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4031860 con 0x7ff9f4153ea0 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9f3e00640 1 -- 172.21.15.43:0/3793572628 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e40319c0 con 0x7ff9f4153ea0 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9f3e00640 1 -- 172.21.15.43:0/3793572628 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4031d50 con 0x7ff9f4153ea0 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 -- 172.21.15.43:0/3793572628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4153ea0 msgr2=0x7ff9f40ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4153ea0 0x7ff9f40ad450 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4157e50 tx=0x7ff9e402b900 comp rx=0 tx=0).stop 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 -- 172.21.15.43:0/3793572628 shutdown_connections 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4070850 0x7ff9f4071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 --2- 172.21.15.43:0/3793572628 >> v2:172.21.15.69:3300/0 conn(0x7ff9f4154280 0x7ff9f415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 --2- 172.21.15.43:0/3793572628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4153ea0 0x7ff9f40ad450 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.131+0000 7ff9fae00640 1 -- 172.21.15.43:0/3793572628 >> 172.21.15.43:0/3793572628 conn(0x7ff9f40cec10 msgr2=0x7ff9f40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.132+0000 7ff9fae00640 1 -- 172.21.15.43:0/3793572628 shutdown_connections 2024-01-16T15:06:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.132+0000 7ff9fae00640 1 -- 172.21.15.43:0/3793572628 wait complete. 2024-01-16T15:06:56.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.133+0000 7ff9fae00640 1 Processor -- start 2024-01-16T15:06:56.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.133+0000 7ff9fae00640 1 -- start start 2024-01-16T15:06:56.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4153ea0 0x7ff9f40b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f4154280 0x7ff9f413d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40afe30 con 0x7ff9f4070850 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40af210 con 0x7ff9f4154280 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9fae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40af490 con 0x7ff9f4153ea0 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.134+0000 7ff9f9a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54848/0 (socket says 172.21.15.43:54848) 2024-01-16T15:06:56.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 -- 172.21.15.43:0/1777638946 learned_addr learned my addr 172.21.15.43:0/1777638946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 -- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4153ea0 msgr2=0x7ff9f40b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4153ea0 0x7ff9f40b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 -- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f4154280 msgr2=0x7ff9f413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f4154280 0x7ff9f413d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 -- 172.21.15.43:0/1777638946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40094d0 con 0x7ff9f4070850 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.135+0000 7ff9f9a00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff9e402b940 tx=0x7ff9e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.136+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4031850 con 0x7ff9f4070850 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.136+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e40443f0 con 0x7ff9f4070850 2024-01-16T15:06:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.136+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40447a0 con 0x7ff9f4070850 2024-01-16T15:06:56.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.136+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f40af770 con 0x7ff9f4070850 2024-01-16T15:06:56.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.136+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f413dad0 con 0x7ff9f4070850 2024-01-16T15:06:56.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.137+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f40aff70 con 0x7ff9f4070850 2024-01-16T15:06:56.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.138+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7ff9e404d450 con 0x7ff9f4070850 2024-01-16T15:06:56.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.139+0000 7ff9f1600640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 0x7ff9cc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.139+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff9e40ad690 con 0x7ff9f4070850 2024-01-16T15:06:56.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.139+0000 7ff9f9000640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 0x7ff9cc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.140+0000 7ff9f9000640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 0x7ff9cc06dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006fd0 tx=0x7ff9dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.143+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9f40aff70 con 0x7ff9f4070850 2024-01-16T15:06:56.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:56.365+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone53197", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9f4070050 con 0x7ff9cc06bb30 2024-01-16T15:06:56.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:56.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.152+0000 7ff9f1600640 1 -- 172.21.15.43:0/1777638946 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7ff9dc00ae40 con 0x7ff9cc06bb30 2024-01-16T15:06:57.131 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:06:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:06:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:06:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:57.131 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:06:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 msgr2=0x7ff9cc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 0x7ff9cc06dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006fd0 tx=0x7ff9dc008040 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 msgr2=0x7ff9f40b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff9e402b940 tx=0x7ff9e40046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 shutdown_connections 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9cc06bb30 0x7ff9cc06dff0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f4154280 0x7ff9f413d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f4153ea0 0x7ff9f40b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 --2- 172.21.15.43:0/1777638946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9f4070850 0x7ff9f40b0250 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 >> 172.21.15.43:0/1777638946 conn(0x7ff9f40cec10 msgr2=0x7ff9f406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.156+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 shutdown_connections 2024-01-16T15:06:57.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.157+0000 7ff9fae00640 1 -- 172.21.15.43:0/1777638946 wait complete. 2024-01-16T15:06:57.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:06:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.403+0000 7fe97d800640 1 Processor -- start 2024-01-16T15:06:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.403+0000 7fe97d800640 1 -- start start 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.404+0000 7fe97d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.404+0000 7fe97d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 0x7fe9780cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe97d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780d04b0 con 0x7fe9780d1670 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe97d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9780ce4e0 con 0x7fe9780cf710 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe97d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780cdb80 con 0x7fe97815d5d0 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe96fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 0x7fe9780cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe977e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe977e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:37980/0 (socket says 172.21.15.43:37980) 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.405+0000 7fe977e00640 1 -- 172.21.15.43:0/4195189777 learned_addr learned my addr 172.21.15.43:0/4195189777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.406+0000 7fe977e00640 1 -- 172.21.15.43:0/4195189777 >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 msgr2=0x7fe9780cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.406+0000 7fe977e00640 1 --2- 172.21.15.43:0/4195189777 >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 0x7fe9780cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.406+0000 7fe977e00640 1 -- 172.21.15.43:0/4195189777 >> v1:172.21.15.69:6789/0 conn(0x7fe9780d1670 legacy=0x7fe9780d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.406+0000 7fe977e00640 1 -- 172.21.15.43:0/4195189777 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780d01d0 con 0x7fe97815d5d0 2024-01-16T15:06:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.406+0000 7fe96fe00640 1 --2- 172.21.15.43:0/4195189777 >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 0x7fe9780cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:57.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.407+0000 7fe977e00640 1 --2- 172.21.15.43:0/4195189777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe968009870 tx=0x7fe96802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47cc2f31bd2d490d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.408+0000 7fe977400640 1 -- 172.21.15.43:0/4195189777 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968031640 con 0x7fe97815d5d0 2024-01-16T15:06:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.408+0000 7fe977400640 1 -- 172.21.15.43:0/4195189777 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9680317a0 con 0x7fe97815d5d0 2024-01-16T15:06:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.408+0000 7fe977400640 1 -- 172.21.15.43:0/4195189777 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968031b30 con 0x7fe97815d5d0 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 -- 172.21.15.43:0/4195189777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 msgr2=0x7fe9780ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 --2- 172.21.15.43:0/4195189777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe968009870 tx=0x7fe96802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 -- 172.21.15.43:0/4195189777 shutdown_connections 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 --2- 172.21.15.43:0/4195189777 >> v2:172.21.15.69:3300/0 conn(0x7fe9780cf710 0x7fe9780cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 --2- 172.21.15.43:0/4195189777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe97815d5d0 0x7fe9780ce900 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 -- 172.21.15.43:0/4195189777 >> 172.21.15.43:0/4195189777 conn(0x7fe97809c3d0 msgr2=0x7fe9780d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 -- 172.21.15.43:0/4195189777 shutdown_connections 2024-01-16T15:06:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.409+0000 7fe97d800640 1 -- 172.21.15.43:0/4195189777 wait complete. 2024-01-16T15:06:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.411+0000 7fe97d800640 1 Processor -- start 2024-01-16T15:06:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.411+0000 7fe97d800640 1 -- start start 2024-01-16T15:06:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9780cf710 0x7fe978157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9780d1670 0x7fe9780cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe978157dd0 con 0x7fe97815d5d0 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780b8bf0 con 0x7fe9780d1670 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780ca160 con 0x7fe9780cf710 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54858/0 (socket says 172.21.15.43:54858) 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe97ce00640 1 -- 172.21.15.43:0/160099244 learned_addr learned my addr 172.21.15.43:0/160099244 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe977e00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9780cf710 0x7fe978157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.413+0000 7fe96fe00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9780d1670 0x7fe9780cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 -- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9780cf710 msgr2=0x7fe978157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9780cf710 0x7fe978157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 -- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9780d1670 msgr2=0x7fe9780cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9780d1670 0x7fe9780cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 -- 172.21.15.43:0/160099244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960009400 con 0x7fe97815d5d0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97ce00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe97000c6e0 tx=0x7fe97000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970003b70 con 0x7fe97815d5d0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe970003cf0 con 0x7fe97815d5d0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970005310 con 0x7fe97815d5d0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9680094d0 con 0x7fe97815d5d0 2024-01-16T15:06:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.414+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9780ca3b0 con 0x7fe97815d5d0 2024-01-16T15:06:57.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.416+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97804e840 con 0x7fe97815d5d0 2024-01-16T15:06:57.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.416+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe970005470 con 0x7fe97815d5d0 2024-01-16T15:06:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.418+0000 7fe974c00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 0x7fe95006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.418+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe970088380 con 0x7fe97815d5d0 2024-01-16T15:06:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.418+0000 7fe977e00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 0x7fe95006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.418+0000 7fe977e00640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 0x7fe95006dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe968036000 tx=0x7fe96803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.422+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe970054a70 con 0x7fe97815d5d0 2024-01-16T15:06:57.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.642+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9780cee40 con 0x7fe95006bb30 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.644+0000 7fe974c00640 1 -- 172.21.15.43:0/160099244 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe968031a50 con 0x7fe95006bb30 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:57.622 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:06:57.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.646+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 msgr2=0x7fe95006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.646+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 0x7fe95006dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe968036000 tx=0x7fe96803c040 comp rx=0 tx=0).stop 2024-01-16T15:06:57.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.646+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 msgr2=0x7fe97806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.646+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe97000c6e0 tx=0x7fe97000cba0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 shutdown_connections 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe95006bb30 0x7fe95006dff0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9780d1670 0x7fe9780cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe97815d5d0 0x7fe97806c220 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 --2- 172.21.15.43:0/160099244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9780cf710 0x7fe978157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 >> 172.21.15.43:0/160099244 conn(0x7fe97809c3d0 msgr2=0x7fe97806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 shutdown_connections 2024-01-16T15:06:57.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.647+0000 7fe97d800640 1 -- 172.21.15.43:0/160099244 wait complete. 2024-01-16T15:06:57.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.890+0000 7f7d27e00640 1 Processor -- start 2024-01-16T15:06:57.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.890+0000 7f7d27e00640 1 -- start start 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d2815d3b0 0x7f7d280ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 0x7f7d280d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d280d2010 con 0x7f7d280d1bc0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d280c6bf0 con 0x7f7d2815d3b0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280d1400 con 0x7f7d280d0fa0 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7d280d1bc0 0x7f7d280b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56282/0 (socket says 172.21.15.43:56282) 2024-01-16T15:06:57.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.892+0000 7f7d27400640 1 -- 172.21.15.43:0/516012908 learned_addr learned my addr 172.21.15.43:0/516012908 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:57.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d26000640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 0x7f7d280d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d26a00640 1 --2- 172.21.15.43:0/516012908 >> v2:172.21.15.69:3300/0 conn(0x7f7d2815d3b0 0x7f7d280ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851627277 0 0) 0x7f7d20001490 con 0x7f7d280d1bc0 2024-01-16T15:06:57.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d08003520 con 0x7f7d280d1bc0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3759304959 0 0) 0x7f7d08003520 con 0x7f7d280d1bc0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 msgr2=0x7f7d280d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 0x7f7d280d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 >> v2:172.21.15.69:3300/0 conn(0x7f7d2815d3b0 msgr2=0x7f7d280ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 --2- 172.21.15.43:0/516012908 >> v2:172.21.15.69:3300/0 conn(0x7f7d2815d3b0 0x7f7d280ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280cedc0 con 0x7f7d280d1bc0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7d20001b80 con 0x7f7d280d1bc0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 >> v1:172.21.15.43:6789/0 conn(0x7f7d280d1bc0 legacy=0x7f7d280b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d080039e0 0x7f7d08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d080047c0 0x7f7d08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.893+0000 7f7d25600640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 0x7f7d08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280c6bf0 con 0x7f7d080039e0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280d1400 con 0x7f7d080047c0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0800ba60 con 0x7f7d280d0fa0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d080039e0 0x7f7d08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 -- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 msgr2=0x7f7d08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d280d0fa0 0x7f7d08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 -- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d080047c0 msgr2=0x7f7d08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26000640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d080047c0 0x7f7d08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d080047c0 0x7f7d08004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.894+0000 7f7d26a00640 1 -- 172.21.15.43:0/516012908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08003520 con 0x7f7d080039e0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.895+0000 7f7d26a00640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d080039e0 0x7f7d08003e00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009a90 tx=0x7f7d080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4f7068eb7f9a63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1802c6b0 con 0x7f7d080039e0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1802c810 con 0x7f7d080039e0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d25600640 1 -- 172.21.15.43:0/516012908 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1802cba0 con 0x7f7d080039e0 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d27e00640 1 -- 172.21.15.43:0/516012908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d080039e0 msgr2=0x7f7d08003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d27e00640 1 --2- 172.21.15.43:0/516012908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d080039e0 0x7f7d08003e00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009a90 tx=0x7f7d080057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d27400640 1 -- 172.21.15.43:0/516012908 reap_dead start 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d27e00640 1 -- 172.21.15.43:0/516012908 shutdown_connections 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.896+0000 7f7d27e00640 1 -- 172.21.15.43:0/516012908 >> 172.21.15.43:0/516012908 conn(0x7f7d280d30a0 msgr2=0x7f7d280d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:57.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.897+0000 7f7d27e00640 1 -- 172.21.15.43:0/516012908 shutdown_connections 2024-01-16T15:06:57.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.897+0000 7f7d27e00640 1 -- 172.21.15.43:0/516012908 wait complete. 2024-01-16T15:06:57.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.898+0000 7f7d27e00640 1 Processor -- start 2024-01-16T15:06:57.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.899+0000 7f7d27e00640 1 -- start start 2024-01-16T15:06:57.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 0x7f7d2815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d281f6600 0x7f7d280cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281f6a80 con 0x7f7d2814de00 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28147ff0 con 0x7f7d281f6600 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280c69d0 con 0x7f7d080039e0 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d26000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d26000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54876/0 (socket says 172.21.15.43:54876) 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d26000640 1 -- 172.21.15.43:0/584236263 learned_addr learned my addr 172.21.15.43:0/584236263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d26a00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 0x7f7d2815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.900+0000 7f7d27400640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d281f6600 0x7f7d280cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 -- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 msgr2=0x7f7d2815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 0x7f7d2815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 -- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d281f6600 msgr2=0x7f7d280cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:57.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d281f6600 0x7f7d280cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:57.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 -- 172.21.15.43:0/584236263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2815aa10 con 0x7f7d2814de00 2024-01-16T15:06:57.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26a00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 0x7f7d2815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:57.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d26000640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7d10010860 tx=0x7f7d10010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10009610 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1000eb60 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1001c400 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2815acf0 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.901+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d280cb1c0 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.903+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2804e840 con 0x7f7d2814de00 2024-01-16T15:06:57.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.903+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f7d1001c560 con 0x7f7d2814de00 2024-01-16T15:06:57.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.904+0000 7f7d1ea00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 0x7f7d0006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:57.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.904+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7d10088c50 con 0x7f7d2814de00 2024-01-16T15:06:57.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.905+0000 7f7d26a00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 0x7f7d0006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:57.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.906+0000 7f7d26a00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 0x7f7d0006e040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d18036000 tx=0x7f7d18030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:57.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:57.909+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d100553f0 con 0x7f7d2814de00 2024-01-16T15:06:58.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.132+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d280c7400 con 0x7f7d0006bb80 2024-01-16T15:06:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.152+0000 7f7d1ea00640 1 -- 172.21.15.43:0/584236263 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7f7d18004210 con 0x7f7d0006bb80 2024-01-16T15:06:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:06:58.130 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:06:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53191" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53192" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53194" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53197" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53196" 2024-01-16T15:06:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53195" 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone53193" 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:06:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 msgr2=0x7f7d0006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 0x7f7d0006e040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d18036000 tx=0x7f7d18030000 comp rx=0 tx=0).stop 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 msgr2=0x7f7d281f60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7d10010860 tx=0x7f7d10010d20 comp rx=0 tx=0).stop 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 shutdown_connections 2024-01-16T15:06:58.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d0006bb80 0x7f7d0006e040 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d281f6600 0x7f7d280cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d2814de00 0x7f7d281f60c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 --2- 172.21.15.43:0/584236263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d080039e0 0x7f7d2815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 >> 172.21.15.43:0/584236263 conn(0x7f7d280d30a0 msgr2=0x7f7d280c76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 shutdown_connections 2024-01-16T15:06:58.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.155+0000 7f7d27e00640 1 -- 172.21.15.43:0/584236263 wait complete. 2024-01-16T15:06:58.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53191 2024-01-16T15:06:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.400+0000 7f569be00640 1 Processor -- start 2024-01-16T15:06:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.400+0000 7f569be00640 1 -- start start 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.401+0000 7f569be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f569c15d3b0 0x7f569c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.401+0000 7f569be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0c6bf0 con 0x7f569c0d1d90 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.401+0000 7f569be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0d1430 con 0x7f569c0d0fa0 2024-01-16T15:06:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.401+0000 7f569be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c0cedc0 con 0x7f569c15d3b0 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.401+0000 7f569aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f569c15d3b0 0x7f569c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f569c15d3b0 0x7f569c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54888/0 (socket says 172.21.15.43:54888) 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f569c0d1d90 0x7f569c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56284/0 (socket says 172.21.15.43:56284) 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569aa00640 1 -- 172.21.15.43:0/1297086621 learned_addr learned my addr 172.21.15.43:0/1297086621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569aa00640 1 -- 172.21.15.43:0/1297086621 >> v1:172.21.15.69:6789/0 conn(0x7f569c0d0fa0 legacy=0x7f569c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569aa00640 1 -- 172.21.15.43:0/1297086621 >> v1:172.21.15.43:6789/0 conn(0x7f569c0d1d90 legacy=0x7f569c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f569aa00640 1 -- 172.21.15.43:0/1297086621 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c0c6e40 con 0x7f569c15d3b0 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.402+0000 7f5699600640 1 -- 172.21.15.43:0/1297086621 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538646334 0 0) 0x7f569c0c6bf0 con 0x7f569c0d1d90 2024-01-16T15:06:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.403+0000 7f569aa00640 1 --2- 172.21.15.43:0/1297086621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c15d3b0 0x7f569c0ce6c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f568c009870 tx=0x7f568c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7b1b0cda6f3047 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f5699600640 1 -- 172.21.15.43:0/1297086621 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c031870 con 0x7f569c15d3b0 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f5699600640 1 -- 172.21.15.43:0/1297086621 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c0319d0 con 0x7f569c15d3b0 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f5699600640 1 -- 172.21.15.43:0/1297086621 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c031d60 con 0x7f569c15d3b0 2024-01-16T15:06:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f569be00640 1 -- 172.21.15.43:0/1297086621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c15d3b0 msgr2=0x7f569c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f569be00640 1 --2- 172.21.15.43:0/1297086621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c15d3b0 0x7f569c0ce6c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f568c009870 tx=0x7f568c02b910 comp rx=0 tx=0).stop 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f569be00640 1 -- 172.21.15.43:0/1297086621 shutdown_connections 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f569be00640 1 --2- 172.21.15.43:0/1297086621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c15d3b0 0x7f569c0ce6c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.404+0000 7f569be00640 1 -- 172.21.15.43:0/1297086621 >> 172.21.15.43:0/1297086621 conn(0x7f569c0d30a0 msgr2=0x7f569c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.405+0000 7f569be00640 1 -- 172.21.15.43:0/1297086621 shutdown_connections 2024-01-16T15:06:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.405+0000 7f569be00640 1 -- 172.21.15.43:0/1297086621 wait complete. 2024-01-16T15:06:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.406+0000 7f569be00640 1 Processor -- start 2024-01-16T15:06:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.407+0000 7f569be00640 1 -- start start 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.407+0000 7f569be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 0x7f569c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.407+0000 7f569be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 0x7f569c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c157b30 con 0x7f569c0d1d90 2024-01-16T15:06:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0b6200 con 0x7f569c0d0fa0 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c158e10 con 0x7f569c15d3b0 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54898/0 (socket says 172.21.15.43:54898) 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 -- 172.21.15.43:0/3526039342 learned_addr learned my addr 172.21.15.43:0/3526039342 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569aa00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 0x7f569c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 -- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 msgr2=0x7f569c1588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569b400640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 0x7f569c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 0x7f569c1588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 -- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 msgr2=0x7f569c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 0x7f569c1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569a000640 1 -- 172.21.15.43:0/3526039342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0094d0 con 0x7f569c0d1d90 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569aa00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 0x7f569c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.408+0000 7f569b400640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 0x7f569c1588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f569a000640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f568400c460 tx=0x7f568400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684012830 con 0x7f569c0d1d90 2024-01-16T15:06:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56840129d0 con 0x7f569c0d1d90 2024-01-16T15:06:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c0b6410 con 0x7f569c0d1d90 2024-01-16T15:06:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684012d80 con 0x7f569c0d1d90 2024-01-16T15:06:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.409+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c0b69d0 con 0x7f569c0d1d90 2024-01-16T15:06:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.410+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c04e840 con 0x7f569c0d1d90 2024-01-16T15:06:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.411+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f568401a3b0 con 0x7f569c0d1d90 2024-01-16T15:06:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.412+0000 7f5692a00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 0x7f567406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.412+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f56840882a0 con 0x7f569c0d1d90 2024-01-16T15:06:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.412+0000 7f569aa00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 0x7f567406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.413+0000 7f569aa00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 0x7f567406dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f568c036000 tx=0x7f568c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.417+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5684054990 con 0x7f569c0d1d90 2024-01-16T15:06:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.635+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53191", "target": ["mon-mgr", ""]}) v1 -- 0x7f569c15d790 con 0x7f567406bb30 2024-01-16T15:06:58.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.670+0000 7f5692a00640 1 -- 172.21.15.43:0/3526039342 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f568c031db0 con 0x7f567406bb30 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 msgr2=0x7f567406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 0x7f567406dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f568c036000 tx=0x7f568c03c040 comp rx=0 tx=0).stop 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 msgr2=0x7f569c0b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f568400c460 tx=0x7f568400c920 comp rx=0 tx=0).stop 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 shutdown_connections 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f567406bb30 0x7f567406dff0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f569c15d3b0 0x7f569c1588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f569c0d1d90 0x7f569c0b5cc0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 --2- 172.21.15.43:0/3526039342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f569c0d0fa0 0x7f569c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 >> 172.21.15.43:0/3526039342 conn(0x7f569c0d30a0 msgr2=0x7f569c0b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 shutdown_connections 2024-01-16T15:06:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.673+0000 7f569be00640 1 -- 172.21.15.43:0/3526039342 wait complete. 2024-01-16T15:06:58.653 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.675+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.653 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.675+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.654 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.675+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.654 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.675+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.654 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.675+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53192 2024-01-16T15:06:58.684 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.706+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.684 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.706+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.684 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.706+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.684 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.706+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.684 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:06:58.706+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.880+0000 7f6358e00640 1 Processor -- start 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.881+0000 7f6358e00640 1 -- start start 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.882+0000 7f6358e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 0x7f635406e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.882+0000 7f6358e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.882+0000 7f6358e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63540c79a0 con 0x7f63540c5ef0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.882+0000 7f6358e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63540c7ed0 con 0x7f6354144650 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.882+0000 7f6358e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540ced50 con 0x7f6354143ff0 2024-01-16T15:06:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.883+0000 7f6351400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.883+0000 7f6351e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 0x7f635406e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.883+0000 7f6351400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:46606/0 (socket says 172.21.15.43:46606) 2024-01-16T15:06:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.883+0000 7f6351400640 1 -- 172.21.15.43:0/851442251 learned_addr learned my addr 172.21.15.43:0/851442251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.884+0000 7f6351e00640 1 -- 172.21.15.43:0/851442251 >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 msgr2=0x7f635406ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.884+0000 7f6351e00640 1 --2- 172.21.15.43:0/851442251 >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.884+0000 7f6351e00640 1 -- 172.21.15.43:0/851442251 >> v1:172.21.15.69:6789/0 conn(0x7f63540c5ef0 legacy=0x7f63540c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.884+0000 7f6351e00640 1 -- 172.21.15.43:0/851442251 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63541443d0 con 0x7f6354143ff0 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.884+0000 7f6351400640 1 --2- 172.21.15.43:0/851442251 >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.886+0000 7f6351e00640 1 --2- 172.21.15.43:0/851442251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 0x7f635406e700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63440097a0 tx=0x7f634402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1b52df674fa5b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.886+0000 7f6350a00640 1 -- 172.21.15.43:0/851442251 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344031830 con 0x7f6354143ff0 2024-01-16T15:06:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6350a00640 1 -- 172.21.15.43:0/851442251 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6344031990 con 0x7f6354143ff0 2024-01-16T15:06:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6350a00640 1 -- 172.21.15.43:0/851442251 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344031d20 con 0x7f6354143ff0 2024-01-16T15:06:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 -- 172.21.15.43:0/851442251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 msgr2=0x7f635406e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 --2- 172.21.15.43:0/851442251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 0x7f635406e700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63440097a0 tx=0x7f634402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 -- 172.21.15.43:0/851442251 shutdown_connections 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 --2- 172.21.15.43:0/851442251 >> v2:172.21.15.69:3300/0 conn(0x7f6354144650 0x7f635406ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 --2- 172.21.15.43:0/851442251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354143ff0 0x7f635406e700 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 -- 172.21.15.43:0/851442251 >> 172.21.15.43:0/851442251 conn(0x7f63540661f0 msgr2=0x7f63540665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 -- 172.21.15.43:0/851442251 shutdown_connections 2024-01-16T15:06:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.887+0000 7f6358e00640 1 -- 172.21.15.43:0/851442251 wait complete. 2024-01-16T15:06:58.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.889+0000 7f6358e00640 1 Processor -- start 2024-01-16T15:06:58.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.889+0000 7f6358e00640 1 -- start start 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63540c5ef0 0x7f6354142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 0x7f63541e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541e8ec0 con 0x7f6354143ff0 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541e82b0 con 0x7f63540c5ef0 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6358e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63541e8590 con 0x7f6354144650 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6351400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6351400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54946/0 (socket says 172.21.15.43:54946) 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.890+0000 7f6351400640 1 -- 172.21.15.43:0/2379588188 learned_addr learned my addr 172.21.15.43:0/2379588188 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63540c5ef0 0x7f6354142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 -- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 msgr2=0x7f63541e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6352800640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 0x7f63541e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 0x7f63541e9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 -- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63540c5ef0 msgr2=0x7f6354142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63540c5ef0 0x7f6354142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 -- 172.21.15.43:0/2379588188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c009b30 con 0x7f6354143ff0 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6352800640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 0x7f63541e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6351400640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f633c00efc0 tx=0x7f633c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c014480 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c00f040 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c0147e0 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344009400 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.891+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541e8b50 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.893+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63541e8ce0 con 0x7f6354143ff0 2024-01-16T15:06:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.893+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f633c014940 con 0x7f6354143ff0 2024-01-16T15:06:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.894+0000 7f634a000640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 0x7f632c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.894+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f633c088f20 con 0x7f6354143ff0 2024-01-16T15:06:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.895+0000 7f6351e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 0x7f632c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.895+0000 7f6351e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 0x7f632c06dff0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6344036000 tx=0x7f6344008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:58.898+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f633c055670 con 0x7f6354143ff0 2024-01-16T15:06:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.117+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53192", "target": ["mon-mgr", ""]}) v1 -- 0x7f635406cd10 con 0x7f632c06bb30 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.159+0000 7f634a000640 1 -- 172.21.15.43:0/2379588188 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f635406cd10 con 0x7f632c06bb30 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 msgr2=0x7f632c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 0x7f632c06dff0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6344036000 tx=0x7f6344008030 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 msgr2=0x7f635415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f633c00efc0 tx=0x7f633c00c490 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 shutdown_connections 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f632c06bb30 0x7f632c06dff0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354144650 0x7f63541e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6354143ff0 0x7f635415a130 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 --2- 172.21.15.43:0/2379588188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63540c5ef0 0x7f6354142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 >> 172.21.15.43:0/2379588188 conn(0x7f63540661f0 msgr2=0x7f635415ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 shutdown_connections 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.162+0000 7f6358e00640 1 -- 172.21.15.43:0/2379588188 wait complete. 2024-01-16T15:06:59.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53194 2024-01-16T15:06:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.448+0000 7fa3e1600640 1 Processor -- start 2024-01-16T15:06:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.448+0000 7fa3e1600640 1 -- start start 2024-01-16T15:06:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.450+0000 7fa3e1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3dc153ea0 0x7fa3dc0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.450+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 0x7fa3dc071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.450+0000 7fa3e1600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0ce650 con 0x7fa3dc154280 2024-01-16T15:06:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.450+0000 7fa3e1600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc154710 con 0x7fa3dc153ea0 2024-01-16T15:06:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.450+0000 7fa3e1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc15e0a0 con 0x7fa3dc070850 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3db400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa3dc154280 0x7fa3dc15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56298/0 (socket says 172.21.15.43:56298) 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3db400640 1 -- 172.21.15.43:0/2160790672 learned_addr learned my addr 172.21.15.43:0/2160790672 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2160790672 >> v2:172.21.15.43:3300/0 conn(0x7fa3dc153ea0 0x7fa3dc0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3e0c00640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 0x7fa3dc071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3daa00640 1 -- 172.21.15.43:0/2160790672 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871033884 0 0) 0x7fa3dc0ce650 con 0x7fa3dc154280 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3daa00640 1 -- 172.21.15.43:0/2160790672 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3bc003520 con 0x7fa3dc154280 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3dbe00640 1 -- 172.21.15.43:0/2160790672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 msgr2=0x7fa3dc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 0x7fa3dc071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3dbe00640 1 -- 172.21.15.43:0/2160790672 >> v1:172.21.15.43:6789/0 conn(0x7fa3dc154280 legacy=0x7fa3dc15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3dbe00640 1 -- 172.21.15.43:0/2160790672 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc15e380 con 0x7fa3dc153ea0 2024-01-16T15:06:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.451+0000 7fa3e0c00640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 0x7fa3dc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:59.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc0ad450 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0097a0 tx=0x7fa3cc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aaf6767d8a075dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3daa00640 1 -- 172.21.15.43:0/2160790672 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc02d040 con 0x7fa3dc153ea0 2024-01-16T15:06:59.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3daa00640 1 -- 172.21.15.43:0/2160790672 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc031810 con 0x7fa3dc153ea0 2024-01-16T15:06:59.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3daa00640 1 -- 172.21.15.43:0/2160790672 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc031ba0 con 0x7fa3dc153ea0 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3e1600640 1 -- 172.21.15.43:0/2160790672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 msgr2=0x7fa3dc0ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.453+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc0ad450 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0097a0 tx=0x7fa3cc02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 -- 172.21.15.43:0/2160790672 shutdown_connections 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc070850 0x7fa3dc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2160790672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc0ad450 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 -- 172.21.15.43:0/2160790672 >> 172.21.15.43:0/2160790672 conn(0x7fa3dc0cec10 msgr2=0x7fa3dc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 -- 172.21.15.43:0/2160790672 shutdown_connections 2024-01-16T15:06:59.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.454+0000 7fa3e1600640 1 -- 172.21.15.43:0/2160790672 wait complete. 2024-01-16T15:06:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.456+0000 7fa3e1600640 1 Processor -- start 2024-01-16T15:06:59.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.456+0000 7fa3e1600640 1 -- start start 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3dc070850 0x7fa3dc13f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 0x7fa3dc0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0b05e0 con 0x7fa3dc153ea0 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc158350 con 0x7fa3dc070850 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3e1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0b0290 con 0x7fa3dc154280 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.457+0000 7fa3db400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54972/0 (socket says 172.21.15.43:54972) 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 -- 172.21.15.43:0/2913124156 learned_addr learned my addr 172.21.15.43:0/2913124156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3e0c00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 0x7fa3dc0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3dc070850 0x7fa3dc13f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 -- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 msgr2=0x7fa3dc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 0x7fa3dc0b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 -- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3dc070850 msgr2=0x7fa3dc13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3dc070850 0x7fa3dc13f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 -- 172.21.15.43:0/2913124156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc009400 con 0x7fa3dc153ea0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3e0c00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 0x7fa3dc0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.458+0000 7fa3db400640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c440 tx=0x7fa3c400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.459+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4005ce0 con 0x7fa3dc153ea0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.459+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4005e40 con 0x7fa3dc153ea0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.459+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40072b0 con 0x7fa3dc153ea0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.459+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc154dc0 con 0x7fa3dc153ea0 2024-01-16T15:06:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.459+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc14eb10 con 0x7fa3dc153ea0 2024-01-16T15:06:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.460+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc0b0720 con 0x7fa3dc153ea0 2024-01-16T15:06:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.461+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa3c4007410 con 0x7fa3dc153ea0 2024-01-16T15:06:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.462+0000 7fa3d3e00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 0x7fa3b406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.462+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa3c4087b10 con 0x7fa3dc153ea0 2024-01-16T15:06:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.462+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 0x7fa3b406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.463+0000 7fa3dbe00640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 0x7fa3b406e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc036000 tx=0x7fa3cc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.466+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3dc0b0720 con 0x7fa3dc153ea0 2024-01-16T15:06:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.686+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53194", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3dc0ad350 con 0x7fa3b406bb60 2024-01-16T15:06:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.727+0000 7fa3d3e00640 1 -- 172.21.15.43:0/2913124156 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3dc0ad350 con 0x7fa3b406bb60 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.730+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 msgr2=0x7fa3b406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.730+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 0x7fa3b406e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc036000 tx=0x7fa3cc03c040 comp rx=0 tx=0).stop 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.730+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 msgr2=0x7fa3dc157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.730+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c440 tx=0x7fa3c400c900 comp rx=0 tx=0).stop 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 shutdown_connections 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa3b406bb60 0x7fa3b406e020 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3dc154280 0x7fa3dc0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3dc153ea0 0x7fa3dc157680 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 --2- 172.21.15.43:0/2913124156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3dc070850 0x7fa3dc13f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 >> 172.21.15.43:0/2913124156 conn(0x7fa3dc0cec10 msgr2=0x7fa3dc0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 shutdown_connections 2024-01-16T15:06:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.731+0000 7fa3e1600640 1 -- 172.21.15.43:0/2913124156 wait complete. 2024-01-16T15:06:59.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:06:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.980+0000 7f6ecd000640 1 Processor -- start 2024-01-16T15:06:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.980+0000 7f6ecd000640 1 -- start start 2024-01-16T15:06:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.982+0000 7f6ecd000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ec8162220 0x7f6ec80b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.982+0000 7f6ecd000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162600 0x7f6ec8162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.982+0000 7f6ecd000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec8162fa0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.982+0000 7f6ecd000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec80706a0 con 0x7f6ec8162220 2024-01-16T15:06:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.982+0000 7f6ecd000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8070c60 con 0x7f6ec8162600 2024-01-16T15:06:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ec8162220 0x7f6ec80b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ec8162220 0x7f6ec80b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54982/0 (socket says 172.21.15.43:54982) 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 -- 172.21.15.43:0/1606314876 learned_addr learned my addr 172.21.15.43:0/1606314876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5400640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162600 0x7f6ec8162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 -- 172.21.15.43:0/1606314876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162600 msgr2=0x7f6ec8162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162600 0x7f6ec8162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 -- 172.21.15.43:0/1606314876 >> v1:172.21.15.43:6789/0 conn(0x7f6ec806c3c0 legacy=0x7f6ec80710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec5e00640 1 -- 172.21.15.43:0/1606314876 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80722b0 con 0x7f6ec8162220 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.983+0000 7f6ec4a00640 1 -- 172.21.15.43:0/1606314876 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23682429 0 0) 0x7f6ec8162fa0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.985+0000 7f6ec5e00640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec8162220 0x7f6ec80b1ea0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02830411c528b62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.985+0000 7f6ec4a00640 1 -- 172.21.15.43:0/1606314876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8030840 con 0x7f6ec8162220 2024-01-16T15:06:59.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.985+0000 7f6ec4a00640 1 -- 172.21.15.43:0/1606314876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb80309a0 con 0x7f6ec8162220 2024-01-16T15:06:59.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.985+0000 7f6ec4a00640 1 -- 172.21.15.43:0/1606314876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8030d30 con 0x7f6ec8162220 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 -- 172.21.15.43:0/1606314876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec8162220 msgr2=0x7f6ec80b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec8162220 0x7f6ec80b1ea0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb802b980 comp rx=0 tx=0).stop 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 -- 172.21.15.43:0/1606314876 shutdown_connections 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162600 0x7f6ec8162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 --2- 172.21.15.43:0/1606314876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec8162220 0x7f6ec80b1ea0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 -- 172.21.15.43:0/1606314876 >> 172.21.15.43:0/1606314876 conn(0x7f6ec80b0ba0 msgr2=0x7f6ec80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 -- 172.21.15.43:0/1606314876 shutdown_connections 2024-01-16T15:06:59.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.986+0000 7f6ecd000640 1 -- 172.21.15.43:0/1606314876 wait complete. 2024-01-16T15:06:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.988+0000 7f6ecd000640 1 Processor -- start 2024-01-16T15:06:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.988+0000 7f6ecd000640 1 -- start start 2024-01-16T15:06:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162220 0x7f6ec809c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ec8162600 0x7f6ec80cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ccd90 con 0x7f6ec806c3c0 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ccab0 con 0x7f6ec8162600 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ecd000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec809b7c0 con 0x7f6ec8162220 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ec5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ec5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54986/0 (socket says 172.21.15.43:54986) 2024-01-16T15:06:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.989+0000 7f6ec5e00640 1 -- 172.21.15.43:0/4017086409 learned_addr learned my addr 172.21.15.43:0/4017086409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 -- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162220 msgr2=0x7f6ec809c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162220 0x7f6ec809c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 -- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ec8162600 msgr2=0x7f6ec80cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ec8162600 0x7f6ec80cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 -- 172.21.15.43:0/4017086409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb80094d0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ec5e00640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80090b0 tx=0x7f6eb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.990+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8039b50 con 0x7f6ec806c3c0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.991+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8039cf0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.991+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8043650 con 0x7f6ec806c3c0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.991+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec809baa0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.991+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec809c7e0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.992+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec809bfc0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.993+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f6eb80437b0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.994+0000 7f6ebe000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 0x7f6ea006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.994+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6eb80aa8a0 con 0x7f6ec806c3c0 2024-01-16T15:06:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.994+0000 7f6ec5400640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 0x7f6ea006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.995+0000 7f6ec5400640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 0x7f6ea006e110 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:06:59.999+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ec809bfc0 con 0x7f6ec806c3c0 2024-01-16T15:07:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.226+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ec80b23e0 con 0x7f6ea006bc50 2024-01-16T15:07:00.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.610+0000 7f6ebe000640 1 -- 172.21.15.43:0/4017086409 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f6eb000be20 con 0x7f6ea006bc50 2024-01-16T15:07:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:00.588 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:07:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:07:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:00.589 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 msgr2=0x7f6ea006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 0x7f6ea006e110 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0008040 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 msgr2=0x7f6ec815bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80090b0 tx=0x7f6eb8004730 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 shutdown_connections 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6ea006bc50 0x7f6ea006e110 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ec8162600 0x7f6ec80cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ec8162220 0x7f6ec809c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 --2- 172.21.15.43:0/4017086409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec806c3c0 0x7f6ec815bbe0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 >> 172.21.15.43:0/4017086409 conn(0x7f6ec80b0ba0 msgr2=0x7f6ec8160f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.613+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 shutdown_connections 2024-01-16T15:07:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.614+0000 7f6ecd000640 1 -- 172.21.15.43:0/4017086409 wait complete. 2024-01-16T15:07:00.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:07:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.865+0000 7f500be00640 1 Processor -- start 2024-01-16T15:07:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.866+0000 7f500be00640 1 -- start start 2024-01-16T15:07:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.867+0000 7f500be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f500c0d3750 0x7f500c0afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.867+0000 7f500be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f500c0b0230 0x7f500c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.867+0000 7f500be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0d3d30 con 0x7f500c160a70 2024-01-16T15:07:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.867+0000 7f500be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0cdf30 con 0x7f500c0b0230 2024-01-16T15:07:00.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.867+0000 7f500be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0ce210 con 0x7f500c0d3750 2024-01-16T15:07:00.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f500c0b0230 0x7f500c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f500c0b0230 0x7f500c161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54990/0 (socket says 172.21.15.43:54990) 2024-01-16T15:07:00.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 -- 172.21.15.43:0/121717551 learned_addr learned my addr 172.21.15.43:0/121717551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500a000640 1 --2- 172.21.15.43:0/121717551 >> v2:172.21.15.69:3300/0 conn(0x7f500c0d3750 0x7f500c0afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 -- 172.21.15.43:0/121717551 >> v2:172.21.15.69:3300/0 conn(0x7f500c0d3750 msgr2=0x7f500c0afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 --2- 172.21.15.43:0/121717551 >> v2:172.21.15.69:3300/0 conn(0x7f500c0d3750 0x7f500c0afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 -- 172.21.15.43:0/121717551 >> v1:172.21.15.69:6789/0 conn(0x7f500c160a70 legacy=0x7f500c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.868+0000 7f500b400640 1 -- 172.21.15.43:0/121717551 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0ce4f0 con 0x7f500c0b0230 2024-01-16T15:07:00.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.869+0000 7f500b400640 1 --2- 172.21.15.43:0/121717551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c0b0230 0x7f500c161130 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f500c0b5bf0 tx=0x7f500402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454f02ddf74fbbdd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:00.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.870+0000 7f5009600640 1 -- 172.21.15.43:0/121717551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50040317e0 con 0x7f500c0b0230 2024-01-16T15:07:00.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.870+0000 7f5009600640 1 -- 172.21.15.43:0/121717551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004031940 con 0x7f500c0b0230 2024-01-16T15:07:00.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.870+0000 7f5009600640 1 -- 172.21.15.43:0/121717551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004031cd0 con 0x7f500c0b0230 2024-01-16T15:07:00.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 -- 172.21.15.43:0/121717551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c0b0230 msgr2=0x7f500c161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:00.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 --2- 172.21.15.43:0/121717551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c0b0230 0x7f500c161130 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f500c0b5bf0 tx=0x7f500402b880 comp rx=0 tx=0).stop 2024-01-16T15:07:00.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 -- 172.21.15.43:0/121717551 shutdown_connections 2024-01-16T15:07:00.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 --2- 172.21.15.43:0/121717551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c0b0230 0x7f500c161130 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 --2- 172.21.15.43:0/121717551 >> v2:172.21.15.69:3300/0 conn(0x7f500c0d3750 0x7f500c0afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 -- 172.21.15.43:0/121717551 >> 172.21.15.43:0/121717551 conn(0x7f500c09c390 msgr2=0x7f500c0d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:00.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 -- 172.21.15.43:0/121717551 shutdown_connections 2024-01-16T15:07:00.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.871+0000 7f500be00640 1 -- 172.21.15.43:0/121717551 wait complete. 2024-01-16T15:07:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.873+0000 7f500be00640 1 Processor -- start 2024-01-16T15:07:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.873+0000 7f500be00640 1 -- start start 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500c0b0230 0x7f500c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f500c0d3750 0x7f500c0cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15e100 con 0x7f500c160a70 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0ccce0 con 0x7f500c0d3750 2024-01-16T15:07:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15e3a0 con 0x7f500c0b0230 2024-01-16T15:07:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54994/0 (socket says 172.21.15.43:54994) 2024-01-16T15:07:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500b400640 1 -- 172.21.15.43:0/135661790 learned_addr learned my addr 172.21.15.43:0/135661790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500a000640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f500c0d3750 0x7f500c0cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.874+0000 7f500aa00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500c0b0230 0x7f500c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 -- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500c0b0230 msgr2=0x7f500c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500c0b0230 0x7f500c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 -- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f500c0d3750 msgr2=0x7f500c0cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f500c0d3750 0x7f500c0cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 -- 172.21.15.43:0/135661790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040094d0 con 0x7f500c160a70 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500b400640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f50040090b0 tx=0x7f50040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004032880 con 0x7f500c160a70 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004032a00 con 0x7f500c160a70 2024-01-16T15:07:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c15e680 con 0x7f500c160a70 2024-01-16T15:07:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.875+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004032db0 con 0x7f500c160a70 2024-01-16T15:07:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.876+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c0caa10 con 0x7f500c160a70 2024-01-16T15:07:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.876+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c15e890 con 0x7f500c160a70 2024-01-16T15:07:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.878+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f500404e450 con 0x7f500c160a70 2024-01-16T15:07:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.879+0000 7f5002a00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 0x7f4fe406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.879+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f50040aa760 con 0x7f500c160a70 2024-01-16T15:07:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.879+0000 7f500aa00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 0x7f4fe406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.880+0000 7f500aa00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 0x7f4fe406e110 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc005520 tx=0x7f4ffc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:00.883+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5004076e20 con 0x7f500c160a70 2024-01-16T15:07:01.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.115+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f500c0cdbe0 con 0x7f4fe406bc50 2024-01-16T15:07:01.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.176+0000 7f5002a00640 1 -- 172.21.15.43:0/135661790 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ffc005110 con 0x7f4fe406bc50 2024-01-16T15:07:01.154 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:01.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.179+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 msgr2=0x7f4fe406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.179+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 0x7f4fe406e110 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc005520 tx=0x7f4ffc009040 comp rx=0 tx=0).stop 2024-01-16T15:07:01.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.179+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 msgr2=0x7f500c1f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.179+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f50040090b0 tx=0x7f50040046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 shutdown_connections 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4fe406bc50 0x7f4fe406e110 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f500c0d3750 0x7f500c0cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c160a70 0x7f500c1f5700 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 --2- 172.21.15.43:0/135661790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f500c0b0230 0x7f500c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.181+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 >> 172.21.15.43:0/135661790 conn(0x7f500c09c390 msgr2=0x7f500c0b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.182+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 shutdown_connections 2024-01-16T15:07:01.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.183+0000 7f500be00640 1 -- 172.21.15.43:0/135661790 wait complete. 2024-01-16T15:07:01.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:07:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.500+0000 7fafc1a00640 1 Processor -- start 2024-01-16T15:07:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.500+0000 7fafc1a00640 1 -- start start 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.501+0000 7fafc1a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc160b10 0x7fafbc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafc1a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fafbc072280 0x7fafbc06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafc1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fafbc06c900 0x7fafbc160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafc1a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc0b2320 con 0x7fafbc072280 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafc1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc06cd90 con 0x7fafbc06c900 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafc1a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc06df30 con 0x7fafbc160b10 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafba000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fafbc072280 0x7fafbc06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafba000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fafbc072280 0x7fafbc06c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55008/0 (socket says 172.21.15.43:55008) 2024-01-16T15:07:01.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafba000640 1 -- 172.21.15.43:0/1448173768 learned_addr learned my addr 172.21.15.43:0/1448173768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:01.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.502+0000 7fafbaa00640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc160b10 0x7fafbc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.503+0000 7fafba000640 1 -- 172.21.15.43:0/1448173768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc160b10 msgr2=0x7fafbc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.503+0000 7fafba000640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc160b10 0x7fafbc06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.503+0000 7fafba000640 1 -- 172.21.15.43:0/1448173768 >> v2:172.21.15.69:3300/0 conn(0x7fafbc06c900 msgr2=0x7fafbc160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.503+0000 7fafba000640 1 --2- 172.21.15.43:0/1448173768 >> v2:172.21.15.69:3300/0 conn(0x7fafbc06c900 0x7fafbc160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.503+0000 7fafba000640 1 -- 172.21.15.43:0/1448173768 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc071780 con 0x7fafbc072280 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.504+0000 7fafba000640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc072280 0x7fafbc06c3c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fafbc06bec0 tx=0x7fafa402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6739bcbe62c565 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.505+0000 7fafb9600640 1 -- 172.21.15.43:0/1448173768 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40317c0 con 0x7fafbc072280 2024-01-16T15:07:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.505+0000 7fafb9600640 1 -- 172.21.15.43:0/1448173768 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4031920 con 0x7fafbc072280 2024-01-16T15:07:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.505+0000 7fafb9600640 1 -- 172.21.15.43:0/1448173768 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4031cb0 con 0x7fafbc072280 2024-01-16T15:07:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 -- 172.21.15.43:0/1448173768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc072280 msgr2=0x7fafbc06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc072280 0x7fafbc06c3c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fafbc06bec0 tx=0x7fafa402b860 comp rx=0 tx=0).stop 2024-01-16T15:07:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 -- 172.21.15.43:0/1448173768 shutdown_connections 2024-01-16T15:07:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 --2- 172.21.15.43:0/1448173768 >> v2:172.21.15.69:3300/0 conn(0x7fafbc06c900 0x7fafbc160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc072280 0x7fafbc06c3c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 --2- 172.21.15.43:0/1448173768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc160b10 0x7fafbc06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 -- 172.21.15.43:0/1448173768 >> 172.21.15.43:0/1448173768 conn(0x7fafbc0b0ba0 msgr2=0x7fafbc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 -- 172.21.15.43:0/1448173768 shutdown_connections 2024-01-16T15:07:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.506+0000 7fafc1a00640 1 -- 172.21.15.43:0/1448173768 wait complete. 2024-01-16T15:07:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.509+0000 7fafc1a00640 1 Processor -- start 2024-01-16T15:07:01.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.509+0000 7fafc1a00640 1 -- start start 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 0x7fafbc1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 0x7fafbc154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc160b10 0x7fafbc099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc157c90 con 0x7fafbc160b10 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0ce740 con 0x7fafbc06c900 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafc1a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc09a860 con 0x7fafbc072280 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafbb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc160b10 0x7fafbc099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafba000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 0x7fafbc154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 0x7fafbc1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 0x7fafbc1580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:46670/0 (socket says 172.21.15.43:46670) 2024-01-16T15:07:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.510+0000 7fafbaa00640 1 -- 172.21.15.43:0/2689458604 learned_addr learned my addr 172.21.15.43:0/2689458604 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 -- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 msgr2=0x7fafbc1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 0x7fafbc1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 -- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc160b10 msgr2=0x7fafbc099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc160b10 0x7fafbc099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 -- 172.21.15.43:0/2689458604 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40094d0 con 0x7fafbc072280 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafba000640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 0x7fafbc154a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fafa4033000 tx=0x7fafa40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4032860 con 0x7fafbc072280 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.511+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa40329c0 con 0x7fafbc072280 2024-01-16T15:07:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.512+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4032d70 con 0x7fafbc072280 2024-01-16T15:07:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.513+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc09ab40 con 0x7fafbc072280 2024-01-16T15:07:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.513+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc154fb0 con 0x7fafbc072280 2024-01-16T15:07:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.514+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc157dd0 con 0x7fafbc072280 2024-01-16T15:07:01.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.516+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fafa404d3f0 con 0x7fafbc072280 2024-01-16T15:07:01.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.518+0000 7fafb2a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 0x7faf9406e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.518+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fafa40aa260 con 0x7fafbc072280 2024-01-16T15:07:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.519+0000 7fafbaa00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 0x7faf9406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.521+0000 7fafbaa00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 0x7faf9406e140 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fafbc1588f0 tx=0x7fafac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.526+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafbc157dd0 con 0x7fafbc072280 2024-01-16T15:07:01.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.935+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fafbc157730 con 0x7faf9406bc80 2024-01-16T15:07:01.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.968+0000 7fafb2a00640 1 -- 172.21.15.43:0/2689458604 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fafac010990 con 0x7faf9406bc80 2024-01-16T15:07:01.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.972+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 msgr2=0x7faf9406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.972+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 0x7faf9406e140 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fafbc1588f0 tx=0x7fafac008040 comp rx=0 tx=0).stop 2024-01-16T15:07:01.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.972+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 msgr2=0x7fafbc154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.972+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 0x7fafbc154a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fafa4033000 tx=0x7fafa40046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 shutdown_connections 2024-01-16T15:07:01.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf9406bc80 0x7faf9406e140 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafbc160b10 0x7fafbc099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafbc072280 0x7fafbc154a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 --2- 172.21.15.43:0/2689458604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafbc06c900 0x7fafbc1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.973+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 >> 172.21.15.43:0/2689458604 conn(0x7fafbc0b0ba0 msgr2=0x7fafbc160f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.974+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 shutdown_connections 2024-01-16T15:07:01.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:01.974+0000 7fafc1a00640 1 -- 172.21.15.43:0/2689458604 wait complete. 2024-01-16T15:07:01.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53197 2024-01-16T15:07:02.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.273+0000 7f133c200640 1 Processor -- start 2024-01-16T15:07:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.273+0000 7f133c200640 1 -- start start 2024-01-16T15:07:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.276+0000 7f133c200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1334160b10 0x7f133406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.276+0000 7f133c200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 0x7f133406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.276+0000 7f133c200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133406cd50 con 0x7f133406c900 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.276+0000 7f133c200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13340b2320 con 0x7f1334072280 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f133c200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133406df30 con 0x7f1334160b10 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1338e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 0x7f133406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1338e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 0x7f133406c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55024/0 (socket says 172.21.15.43:55024) 2024-01-16T15:07:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1338e00640 1 -- 172.21.15.43:0/1167141259 learned_addr learned my addr 172.21.15.43:0/1167141259 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1339800640 1 --2- 172.21.15.43:0/1167141259 >> v2:172.21.15.69:3300/0 conn(0x7f1334160b10 0x7f133406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1339800640 1 -- 172.21.15.43:0/1167141259 >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 msgr2=0x7f133406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1339800640 1 --2- 172.21.15.43:0/1167141259 >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 0x7f133406c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.277+0000 7f1339800640 1 -- 172.21.15.43:0/1167141259 >> v1:172.21.15.69:6789/0 conn(0x7f133406c900 legacy=0x7f1334160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.278+0000 7f1339800640 1 -- 172.21.15.43:0/1167141259 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334071780 con 0x7f1334160b10 2024-01-16T15:07:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.279+0000 7f1339800640 1 --2- 172.21.15.43:0/1167141259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334160b10 0x7f133406e420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f13340b4070 tx=0x7f132c02be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cd714b1e5c370b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.280+0000 7f1333e00640 1 -- 172.21.15.43:0/1167141259 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c02cbe0 con 0x7f1334160b10 2024-01-16T15:07:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.280+0000 7f1333e00640 1 -- 172.21.15.43:0/1167141259 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c02cd40 con 0x7f1334160b10 2024-01-16T15:07:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.280+0000 7f1333e00640 1 -- 172.21.15.43:0/1167141259 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c034660 con 0x7f1334160b10 2024-01-16T15:07:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 -- 172.21.15.43:0/1167141259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334160b10 msgr2=0x7f133406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 --2- 172.21.15.43:0/1167141259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334160b10 0x7f133406e420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f13340b4070 tx=0x7f132c02be30 comp rx=0 tx=0).stop 2024-01-16T15:07:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 -- 172.21.15.43:0/1167141259 shutdown_connections 2024-01-16T15:07:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 --2- 172.21.15.43:0/1167141259 >> v2:172.21.15.43:3300/0 conn(0x7f1334072280 0x7f133406c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 --2- 172.21.15.43:0/1167141259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334160b10 0x7f133406e420 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 -- 172.21.15.43:0/1167141259 >> 172.21.15.43:0/1167141259 conn(0x7f13340b0ba0 msgr2=0x7f13340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:02.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.281+0000 7f133c200640 1 -- 172.21.15.43:0/1167141259 shutdown_connections 2024-01-16T15:07:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.282+0000 7f133c200640 1 -- 172.21.15.43:0/1167141259 wait complete. 2024-01-16T15:07:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.284+0000 7f133c200640 1 Processor -- start 2024-01-16T15:07:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.285+0000 7f133c200640 1 -- start start 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 0x7f13340cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1334160b10 0x7f13340af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13340cda90 con 0x7f133406c900 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13340cdd70 con 0x7f1334072280 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.286+0000 7f133c200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13340ce050 con 0x7f1334160b10 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.287+0000 7f1338e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.287+0000 7f1338e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:46698/0 (socket says 172.21.15.43:46698) 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.287+0000 7f1338e00640 1 -- 172.21.15.43:0/2739727682 learned_addr learned my addr 172.21.15.43:0/2739727682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.287+0000 7f1339800640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 0x7f13340cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.287+0000 7f133a200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1334160b10 0x7f13340af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1339800640 1 -- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1334160b10 msgr2=0x7f13340af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1339800640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1334160b10 0x7f13340af7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1339800640 1 -- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 msgr2=0x7f13340af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1339800640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1339800640 1 -- 172.21.15.43:0/2739727682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c009a70 con 0x7f133406c900 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.288+0000 7f1338e00640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.289+0000 7f1339800640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 0x7f13340cd390 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f13340d0090 tx=0x7f132c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.290+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c03e610 con 0x7f133406c900 2024-01-16T15:07:02.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.291+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c03e770 con 0x7f133406c900 2024-01-16T15:07:02.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.291+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c03e610 con 0x7f133406c900 2024-01-16T15:07:02.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.291+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13340ce330 con 0x7f133406c900 2024-01-16T15:07:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.292+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13340cbf20 con 0x7f133406c900 2024-01-16T15:07:02.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.293+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334099a00 con 0x7f133406c900 2024-01-16T15:07:02.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.302+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f132c03b050 con 0x7f133406c900 2024-01-16T15:07:02.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.307+0000 7f1331600640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 0x7f130c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.308+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f132c0aae70 con 0x7f133406c900 2024-01-16T15:07:02.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.308+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f132c0ada40 con 0x7f133406c900 2024-01-16T15:07:02.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.309+0000 7f1338e00640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 0x7f130c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.310+0000 7f1338e00640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 0x7f130c06e1a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f133415fed0 tx=0x7f131c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.713+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53197", "target": ["mon-mgr", ""]}) v1 -- 0x7f13340b1f40 con 0x7f130c06bce0 2024-01-16T15:07:02.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.739+0000 7f1331600640 1 -- 172.21.15.43:0/2739727682 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f131c014440 con 0x7f130c06bce0 2024-01-16T15:07:02.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 msgr2=0x7f130c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 0x7f130c06e1a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f133415fed0 tx=0x7f131c009240 comp rx=0 tx=0).stop 2024-01-16T15:07:02.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 msgr2=0x7f13340cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 0x7f13340cd390 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f13340d0090 tx=0x7f132c004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:02.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 shutdown_connections 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130c06bce0 0x7f130c06e1a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1334072280 0x7f13340af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1334160b10 0x7f13340af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 --2- 172.21.15.43:0/2739727682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f133406c900 0x7f13340cd390 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.741+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 >> 172.21.15.43:0/2739727682 conn(0x7f13340b0ba0 msgr2=0x7f133415eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.742+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 shutdown_connections 2024-01-16T15:07:02.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.742+0000 7f133c200640 1 -- 172.21.15.43:0/2739727682 wait complete. 2024-01-16T15:07:02.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53196 2024-01-16T15:07:02.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.941+0000 7f32abe00640 1 Processor -- start 2024-01-16T15:07:02.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.941+0000 7f32abe00640 1 -- start start 2024-01-16T15:07:02.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32abe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac143ac0 0x7f32ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32abe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32ac15e130 0x7f32ac0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32abe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac0cf920 con 0x7f32ac15dd50 2024-01-16T15:07:02.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32abe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ac0cfc00 con 0x7f32ac15e130 2024-01-16T15:07:02.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32abe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac06d780 con 0x7f32ac143ac0 2024-01-16T15:07:02.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.943+0000 7f32ab400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32ac15e130 0x7f32ac0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32aaa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f32ac15dd50 0x7f32ac1433b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:48752/0 (socket says 172.21.15.43:48752) 2024-01-16T15:07:02.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32aaa00640 1 -- 172.21.15.43:0/3881504521 learned_addr learned my addr 172.21.15.43:0/3881504521 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:02.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32a1e00640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac143ac0 0x7f32ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32ab400640 1 -- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac143ac0 msgr2=0x7f32ac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:02.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32ab400640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac143ac0 0x7f32ac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32ab400640 1 -- 172.21.15.43:0/3881504521 >> v1:172.21.15.69:6789/0 conn(0x7f32ac15dd50 legacy=0x7f32ac1433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.944+0000 7f32ab400640 1 -- 172.21.15.43:0/3881504521 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac06da60 con 0x7f32ac15e130 2024-01-16T15:07:02.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.946+0000 7f32ab400640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15e130 0x7f32ac0b2fe0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f32a40097a0 tx=0x7f32a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756ff9d5bf387a91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.946+0000 7f32aa000640 1 -- 172.21.15.43:0/3881504521 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a4031830 con 0x7f32ac15e130 2024-01-16T15:07:02.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.946+0000 7f32aa000640 1 -- 172.21.15.43:0/3881504521 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a4031990 con 0x7f32ac15e130 2024-01-16T15:07:02.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.946+0000 7f32aa000640 1 -- 172.21.15.43:0/3881504521 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a4031d20 con 0x7f32ac15e130 2024-01-16T15:07:02.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 -- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15e130 msgr2=0x7f32ac0b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15e130 0x7f32ac0b2fe0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f32a40097a0 tx=0x7f32a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 -- 172.21.15.43:0/3881504521 shutdown_connections 2024-01-16T15:07:02.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15e130 0x7f32ac0b2fe0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 --2- 172.21.15.43:0/3881504521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac143ac0 0x7f32ac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 -- 172.21.15.43:0/3881504521 >> 172.21.15.43:0/3881504521 conn(0x7f32ac06bbb0 msgr2=0x7f32ac0b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:02.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 -- 172.21.15.43:0/3881504521 shutdown_connections 2024-01-16T15:07:02.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.947+0000 7f32abe00640 1 -- 172.21.15.43:0/3881504521 wait complete. 2024-01-16T15:07:02.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.948+0000 7f32abe00640 1 Processor -- start 2024-01-16T15:07:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.949+0000 7f32abe00640 1 -- start start 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.949+0000 7f32abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15dd50 0x7f32ac158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32abe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac15e130 0x7f32ac159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32abe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac13f8c0 con 0x7f32ac143ac0 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32abe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac155990 con 0x7f32ac15dd50 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32abe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac155f50 con 0x7f32ac15e130 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38062/0 (socket says 172.21.15.43:38062) 2024-01-16T15:07:02.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 -- 172.21.15.43:0/1889838378 learned_addr learned my addr 172.21.15.43:0/1889838378 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32ab400640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac15e130 0x7f32ac159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 -- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac15e130 msgr2=0x7f32ac159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac15e130 0x7f32ac159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 -- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15dd50 msgr2=0x7f32ac158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15dd50 0x7f32ac158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.950+0000 7f32aaa00640 1 -- 172.21.15.43:0/1889838378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4009400 con 0x7f32ac143ac0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32aaa00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f329c00cb40 tx=0x7f329c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c00aca0 con 0x7f32ac143ac0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c00ae00 con 0x7f32ac143ac0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c012620 con 0x7f32ac143ac0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac13fd60 con 0x7f32ac143ac0 2024-01-16T15:07:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.951+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac071e30 con 0x7f32ac143ac0 2024-01-16T15:07:02.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.952+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac04e840 con 0x7f32ac143ac0 2024-01-16T15:07:02.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.953+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f329c007070 con 0x7f32ac143ac0 2024-01-16T15:07:02.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.954+0000 7f32a3400640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 0x7f328406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:02.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.954+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f329c088700 con 0x7f32ac143ac0 2024-01-16T15:07:02.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.954+0000 7f32a1e00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 0x7f328406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:02.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.955+0000 7f32a1e00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 0x7f328406e160 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3294006a90 tx=0x7f3294006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:02.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:02.958+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f329c054dc0 con 0x7f32ac143ac0 2024-01-16T15:07:03.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.171+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53196", "target": ["mon-mgr", ""]}) v1 -- 0x7f32ac156090 con 0x7f328406bca0 2024-01-16T15:07:03.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.208+0000 7f32a3400640 1 -- 172.21.15.43:0/1889838378 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f329400e860 con 0x7f328406bca0 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 msgr2=0x7f328406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 0x7f328406e160 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3294006a90 tx=0x7f3294006a20 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 msgr2=0x7f32ac13f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f329c00cb40 tx=0x7f329c00a060 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 shutdown_connections 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f328406bca0 0x7f328406e160 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32ac15dd50 0x7f32ac158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32ac15e130 0x7f32ac159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 --2- 172.21.15.43:0/1889838378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32ac143ac0 0x7f32ac13f380 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.211+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 >> 172.21.15.43:0/1889838378 conn(0x7f32ac06bbb0 msgr2=0x7f32ac141420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.212+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 shutdown_connections 2024-01-16T15:07:03.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.212+0000 7f32abe00640 1 -- 172.21.15.43:0/1889838378 wait complete. 2024-01-16T15:07:03.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53195 2024-01-16T15:07:03.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.423+0000 7fa75e800640 1 Processor -- start 2024-01-16T15:07:03.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.424+0000 7fa75e800640 1 -- start start 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.425+0000 7fa75e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa75815ae80 0x7fa758070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.425+0000 7fa75e800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75815a850 con 0x7fa758071310 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.425+0000 7fa75e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa758071770 con 0x7fa7580b3450 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.425+0000 7fa75e800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7580b4330 con 0x7fa75815ae80 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa75815ae80 0x7fa758070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa75815ae80 0x7fa758070dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38070/0 (socket says 172.21.15.43:38070) 2024-01-16T15:07:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 -- 172.21.15.43:0/2118867243 learned_addr learned my addr 172.21.15.43:0/2118867243 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:03.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 -- 172.21.15.43:0/2118867243 >> v1:172.21.15.69:6789/0 conn(0x7fa7580b3450 legacy=0x7fa7580cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 -- 172.21.15.43:0/2118867243 >> v1:172.21.15.43:6789/0 conn(0x7fa758071310 legacy=0x7fa7580b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.426+0000 7fa75d400640 1 -- 172.21.15.43:0/2118867243 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580b3a90 con 0x7fa75815ae80 2024-01-16T15:07:03.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.427+0000 7fa75d400640 1 --2- 172.21.15.43:0/2118867243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758070dd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa740009870 tx=0x7fa74002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c075e43bf9cebe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa757e00640 1 -- 172.21.15.43:0/2118867243 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740030850 con 0x7fa75815ae80 2024-01-16T15:07:03.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa757e00640 1 -- 172.21.15.43:0/2118867243 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7400309b0 con 0x7fa75815ae80 2024-01-16T15:07:03.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa757e00640 1 -- 172.21.15.43:0/2118867243 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740030d40 con 0x7fa75815ae80 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 -- 172.21.15.43:0/2118867243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 msgr2=0x7fa758070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 --2- 172.21.15.43:0/2118867243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758070dd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa740009870 tx=0x7fa74002b990 comp rx=0 tx=0).stop 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 -- 172.21.15.43:0/2118867243 shutdown_connections 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 --2- 172.21.15.43:0/2118867243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758070dd0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 -- 172.21.15.43:0/2118867243 >> 172.21.15.43:0/2118867243 conn(0x7fa7580b15c0 msgr2=0x7fa7580b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 -- 172.21.15.43:0/2118867243 shutdown_connections 2024-01-16T15:07:03.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.428+0000 7fa75e800640 1 -- 172.21.15.43:0/2118867243 wait complete. 2024-01-16T15:07:03.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.430+0000 7fa75e800640 1 Processor -- start 2024-01-16T15:07:03.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.430+0000 7fa75e800640 1 -- start start 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa758071310 0x7fa7581377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7580b3450 0x7fa75814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75814fd90 con 0x7fa75815ae80 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa758134000 con 0x7fa758071310 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75e800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa758134490 con 0x7fa7580b3450 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38074/0 (socket says 172.21.15.43:38074) 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.431+0000 7fa75de00640 1 -- 172.21.15.43:0/3368759321 learned_addr learned my addr 172.21.15.43:0/3368759321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75d400640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa758071310 0x7fa7581377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 -- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7580b3450 msgr2=0x7fa75814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7580b3450 0x7fa75814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 -- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa758071310 msgr2=0x7fa7581377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa758071310 0x7fa7581377f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 -- 172.21.15.43:0/3368759321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400094d0 con 0x7fa75815ae80 2024-01-16T15:07:03.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75de00640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa75000c390 tx=0x7fa75000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7500035f0 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750003750 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750003b00 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7581339f0 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.432+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa758169a80 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.433+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7580b4010 con 0x7fa75815ae80 2024-01-16T15:07:03.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.434+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fa750003c60 con 0x7fa75815ae80 2024-01-16T15:07:03.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.435+0000 7fa755600640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 0x7fa73006e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.435+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa750087830 con 0x7fa75815ae80 2024-01-16T15:07:03.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.435+0000 7fa75d400640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 0x7fa73006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.436+0000 7fa75d400640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 0x7fa73006e140 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa740036000 tx=0x7fa74003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.439+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa750053ef0 con 0x7fa75815ae80 2024-01-16T15:07:03.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.650+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53195", "target": ["mon-mgr", ""]}) v1 -- 0x7fa75806cb70 con 0x7fa73006bc80 2024-01-16T15:07:03.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.690+0000 7fa755600640 1 -- 172.21.15.43:0/3368759321 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa740030c90 con 0x7fa73006bc80 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 msgr2=0x7fa73006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 0x7fa73006e140 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa740036000 tx=0x7fa74003b040 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 msgr2=0x7fa758169280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa75000c390 tx=0x7fa75000c850 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 shutdown_connections 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa73006bc80 0x7fa73006e140 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa75815ae80 0x7fa758169280 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7580b3450 0x7fa75814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 --2- 172.21.15.43:0/3368759321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa758071310 0x7fa7581377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 >> 172.21.15.43:0/3368759321 conn(0x7fa7580b15c0 msgr2=0x7fa7580af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.692+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 shutdown_connections 2024-01-16T15:07:03.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.693+0000 7fa75e800640 1 -- 172.21.15.43:0/3368759321 wait complete. 2024-01-16T15:07:03.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone53193 2024-01-16T15:07:03.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.920+0000 7f4919200640 1 Processor -- start 2024-01-16T15:07:03.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.921+0000 7f4919200640 1 -- start start 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.922+0000 7f4919200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 0x7f49140abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.922+0000 7f4919200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f49140adb60 0x7f49140adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.922+0000 7f4919200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491415b730 con 0x7f491415b350 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.922+0000 7f4919200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49140ac740 con 0x7f49140adb60 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.922+0000 7f4919200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49140ac460 con 0x7f491409cb30 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4911e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f491415b350 0x7f49140b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51684/0 (socket says 172.21.15.43:51684) 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4911e00640 1 -- 172.21.15.43:0/884341878 learned_addr learned my addr 172.21.15.43:0/884341878 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4912800640 1 --2- 172.21.15.43:0/884341878 >> v2:172.21.15.43:3300/0 conn(0x7f49140adb60 0x7f49140adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4911400640 1 -- 172.21.15.43:0/884341878 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959831149 0 0) 0x7f4904001490 con 0x7f491415b350 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4911400640 1 -- 172.21.15.43:0/884341878 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f4003520 con 0x7f491415b350 2024-01-16T15:07:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4909200640 1 --2- 172.21.15.43:0/884341878 >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 0x7f49140abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4912800640 1 -- 172.21.15.43:0/884341878 >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 msgr2=0x7f49140abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:03.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4912800640 1 --2- 172.21.15.43:0/884341878 >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 0x7f49140abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4912800640 1 -- 172.21.15.43:0/884341878 >> v1:172.21.15.43:6789/0 conn(0x7f491415b350 legacy=0x7f49140b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4912800640 1 -- 172.21.15.43:0/884341878 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140b2a50 con 0x7f49140adb60 2024-01-16T15:07:03.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.923+0000 7f4909200640 1 --2- 172.21.15.43:0/884341878 >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 0x7f49140abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:03.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.924+0000 7f4912800640 1 --2- 172.21.15.43:0/884341878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140adb60 0x7f49140adf70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f49140d0060 tx=0x7f490c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fbacba8224bf48 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.925+0000 7f4911400640 1 -- 172.21.15.43:0/884341878 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c031830 con 0x7f49140adb60 2024-01-16T15:07:03.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.925+0000 7f4911400640 1 -- 172.21.15.43:0/884341878 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c031990 con 0x7f49140adb60 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.925+0000 7f4911400640 1 -- 172.21.15.43:0/884341878 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c031d20 con 0x7f49140adb60 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.925+0000 7f4919200640 1 -- 172.21.15.43:0/884341878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140adb60 msgr2=0x7f49140adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.925+0000 7f4919200640 1 --2- 172.21.15.43:0/884341878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140adb60 0x7f49140adf70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f49140d0060 tx=0x7f490c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 -- 172.21.15.43:0/884341878 shutdown_connections 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 --2- 172.21.15.43:0/884341878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49140adb60 0x7f49140adf70 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 --2- 172.21.15.43:0/884341878 >> v2:172.21.15.69:3300/0 conn(0x7f491409cb30 0x7f49140abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 -- 172.21.15.43:0/884341878 >> 172.21.15.43:0/884341878 conn(0x7f49140b0ba0 msgr2=0x7f49140b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 -- 172.21.15.43:0/884341878 shutdown_connections 2024-01-16T15:07:03.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.926+0000 7f4919200640 1 -- 172.21.15.43:0/884341878 wait complete. 2024-01-16T15:07:03.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.927+0000 7f4919200640 1 Processor -- start 2024-01-16T15:07:03.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.927+0000 7f4919200640 1 -- start start 2024-01-16T15:07:03.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491409cb30 0x7f4914141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140adb60 0x7f4914154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4914156470 con 0x7f491415b350 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491415a170 con 0x7f491409cb30 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.928+0000 7f4919200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491415a410 con 0x7f49140adb60 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38094/0 (socket says 172.21.15.43:38094) 2024-01-16T15:07:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 -- 172.21.15.43:0/2363729983 learned_addr learned my addr 172.21.15.43:0/2363729983 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:03.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 -- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140adb60 msgr2=0x7f4914154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140adb60 0x7f4914154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 -- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491409cb30 msgr2=0x7f4914141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:03.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491409cb30 0x7f4914141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 -- 172.21.15.43:0/2363729983 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c009400 con 0x7f491415b350 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.929+0000 7f4912800640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f490c02bee0 tx=0x7f490c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.930+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c03a630 con 0x7f491415b350 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.930+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c03a7b0 con 0x7f491415b350 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.930+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491415a6f0 con 0x7f491415b350 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.930+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c03ab60 con 0x7f491415b350 2024-01-16T15:07:03.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.930+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4914141e80 con 0x7f491415b350 2024-01-16T15:07:03.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.931+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491415abb0 con 0x7f491415b350 2024-01-16T15:07:03.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.932+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f490c03acc0 con 0x7f491415b350 2024-01-16T15:07:03.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.933+0000 7f490aa00640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 0x7f48ec06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:03.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.933+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f490c0aa380 con 0x7f491415b350 2024-01-16T15:07:03.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.933+0000 7f4911e00640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 0x7f48ec06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:03.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.934+0000 7f4911e00640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 0x7f48ec06e110 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4904005c00 tx=0x7f49040014d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:03.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:03.937+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f490c076b60 con 0x7f491415b350 2024-01-16T15:07:04.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.153+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone53193", "target": ["mon-mgr", ""]}) v1 -- 0x7f491406e720 con 0x7f48ec06bc50 2024-01-16T15:07:04.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.196+0000 7f490aa00640 1 -- 172.21.15.43:0/2363729983 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f49040020e0 con 0x7f48ec06bc50 2024-01-16T15:07:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 msgr2=0x7f48ec06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 0x7f48ec06e110 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4904005c00 tx=0x7f49040014d0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 msgr2=0x7f4914151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f490c02bee0 tx=0x7f490c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 shutdown_connections 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48ec06bc50 0x7f48ec06e110 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f491415b350 0x7f4914151aa0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49140adb60 0x7f4914154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 --2- 172.21.15.43:0/2363729983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f491409cb30 0x7f4914141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 >> 172.21.15.43:0/2363729983 conn(0x7f49140b0ba0 msgr2=0x7f49140b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.199+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 shutdown_connections 2024-01-16T15:07:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.200+0000 7f4919200640 1 -- 172.21.15.43:0/2363729983 wait complete. 2024-01-16T15:07:04.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:04.203 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:07:04.285 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:07:04.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:07:04.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.514+0000 7f25d3000640 1 Processor -- start 2024-01-16T15:07:04.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.514+0000 7f25d3000640 1 -- start start 2024-01-16T15:07:04.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.515+0000 7f25d3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.515+0000 7f25d3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25cc0adb60 0x7f25cc0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.515+0000 7f25d3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc15b730 con 0x7f25cc09cb30 2024-01-16T15:07:04.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.515+0000 7f25d3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0ac740 con 0x7f25cc0adb60 2024-01-16T15:07:04.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.515+0000 7f25d3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0ac460 con 0x7f25cc15b350 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.516+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.516+0000 7f25d1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25cc0adb60 0x7f25cc0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.516+0000 7f25d1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25cc0adb60 0x7f25cc0adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43046/0 (socket says 172.21.15.43:43046) 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.516+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36004/0 (socket says 172.21.15.43:36004) 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.516+0000 7f25c3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f25cc09cb30 0x7f25cc0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:48764/0 (socket says 172.21.15.43:48764) 2024-01-16T15:07:04.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25cbe00640 1 -- 172.21.15.43:0/3674077681 learned_addr learned my addr 172.21.15.43:0/3674077681 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25d1000640 1 -- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 msgr2=0x7f25cc0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25d1000640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25d1000640 1 -- 172.21.15.43:0/3674077681 >> v1:172.21.15.69:6789/0 conn(0x7f25cc09cb30 legacy=0x7f25cc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25d1000640 1 -- 172.21.15.43:0/3674077681 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0b2a50 con 0x7f25cc0adb60 2024-01-16T15:07:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.517+0000 7f25cbe00640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:04.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.519+0000 7f25d1000640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc0adfb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009d90 tx=0x7f25c402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd0438410c26d2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.519+0000 7f25cb400640 1 -- 172.21.15.43:0/3674077681 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c402cc50 con 0x7f25cc0adb60 2024-01-16T15:07:04.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.519+0000 7f25cb400640 1 -- 172.21.15.43:0/3674077681 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c402cdb0 con 0x7f25cc0adb60 2024-01-16T15:07:04.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.519+0000 7f25cb400640 1 -- 172.21.15.43:0/3674077681 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4034660 con 0x7f25cc0adb60 2024-01-16T15:07:04.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 -- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 msgr2=0x7f25cc0adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc0adfb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009d90 tx=0x7f25c402bea0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 -- 172.21.15.43:0/3674077681 shutdown_connections 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc0adfb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 --2- 172.21.15.43:0/3674077681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 -- 172.21.15.43:0/3674077681 >> 172.21.15.43:0/3674077681 conn(0x7f25cc0b0ba0 msgr2=0x7f25cc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 -- 172.21.15.43:0/3674077681 shutdown_connections 2024-01-16T15:07:04.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.520+0000 7f25d3000640 1 -- 172.21.15.43:0/3674077681 wait complete. 2024-01-16T15:07:04.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.522+0000 7f25d3000640 1 Processor -- start 2024-01-16T15:07:04.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.522+0000 7f25d3000640 1 -- start start 2024-01-16T15:07:04.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc070e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc157f70 con 0x7f25cc09cb30 2024-01-16T15:07:04.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc152ef0 con 0x7f25cc0adb60 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25d3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc1531d0 con 0x7f25cc15b350 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38106/0 (socket says 172.21.15.43:38106) 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.523+0000 7f25cbe00640 1 -- 172.21.15.43:0/2475126466 learned_addr learned my addr 172.21.15.43:0/2475126466 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25c3e00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25d1000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 -- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 msgr2=0x7f25cc157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 -- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 msgr2=0x7f25cc070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 -- 172.21.15.43:0/2475126466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0094d0 con 0x7f25cc09cb30 2024-01-16T15:07:04.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25d1000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:04.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.524+0000 7f25cbe00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00e9d0 tx=0x7f25bc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.525+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00ca60 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.525+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00cbc0 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.525+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc013470 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.525+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c4009a70 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.525+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc158250 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.526+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc06eda0 con 0x7f25cc09cb30 2024-01-16T15:07:04.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.527+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f25bc0135d0 con 0x7f25cc09cb30 2024-01-16T15:07:04.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.528+0000 7f25c8c00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 0x7f25a406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.528+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f25bc0889b0 con 0x7f25cc09cb30 2024-01-16T15:07:04.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.528+0000 7f25c3e00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 0x7f25a406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.529+0000 7f25c3e00640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 0x7f25a406e110 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25cc159550 tx=0x7f25b4007ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.533+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25bc055070 con 0x7f25cc09cb30 2024-01-16T15:07:04.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:04.750+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f25cc0ae4f0 con 0x7f25a406bc50 2024-01-16T15:07:04.787 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:07:04.806+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:07:04.878 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:07:05.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:06.014+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f25bc055440 con 0x7f25cc09cb30 2024-01-16T15:07:07.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.847+0000 7f25c8c00640 1 -- 172.21.15.43:0/2475126466 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f25b40074e0 con 0x7f25a406bc50 2024-01-16T15:07:07.825 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:07:07.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.851+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 msgr2=0x7f25a406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.851+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 0x7f25a406e110 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25cc159550 tx=0x7f25b4007ce0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.851+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 msgr2=0x7f25cc1539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.851+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00e9d0 tx=0x7f25bc00ee90 comp rx=0 tx=0).stop 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 shutdown_connections 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25a406bc50 0x7f25a406e110 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25cc0adb60 0x7f25cc070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25cc15b350 0x7f25cc157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 --2- 172.21.15.43:0/2475126466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25cc09cb30 0x7f25cc1539d0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.852+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 >> 172.21.15.43:0/2475126466 conn(0x7f25cc0b0ba0 msgr2=0x7f25cc150a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:07.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.853+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 shutdown_connections 2024-01-16T15:07:07.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:07.853+0000 7f25d3000640 1 -- 172.21.15.43:0/2475126466 wait complete. 2024-01-16T15:07:07.862 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:07:07.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:08.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:08.128 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:07:08.128 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:08.128 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:08.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.492+0000 7fc3ade00640 1 Processor -- start 2024-01-16T15:07:08.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.492+0000 7fc3ade00640 1 -- start start 2024-01-16T15:07:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.494+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.494+0000 7fc3ade00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 0x7fc3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3ade00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80ad510 con 0x7fc3a8161930 2024-01-16T15:07:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3ade00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a806ded0 con 0x7fc3a806d520 2024-01-16T15:07:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3ade00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80acc50 con 0x7fc3a80719c0 2024-01-16T15:07:08.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36028/0 (socket says 172.21.15.43:36028) 2024-01-16T15:07:08.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.495+0000 7fc3a6000640 1 -- 172.21.15.43:0/2787145804 learned_addr learned my addr 172.21.15.43:0/2787145804 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:08.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.496+0000 7fc3a7400640 1 --2- 172.21.15.43:0/2787145804 >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 0x7fc3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.496+0000 7fc3a5600640 1 -- 172.21.15.43:0/2787145804 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259351160 0 0) 0x7fc3a80ad510 con 0x7fc3a8161930 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.496+0000 7fc3a5600640 1 -- 172.21.15.43:0/2787145804 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388003520 con 0x7fc3a8161930 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.497+0000 7fc3a6000640 1 -- 172.21.15.43:0/2787145804 >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 msgr2=0x7fc3a806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.497+0000 7fc3a6000640 1 --2- 172.21.15.43:0/2787145804 >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 0x7fc3a806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.497+0000 7fc3a6000640 1 -- 172.21.15.43:0/2787145804 >> v1:172.21.15.43:6789/0 conn(0x7fc3a8161930 legacy=0x7fc3a80ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.497+0000 7fc3a6000640 1 -- 172.21.15.43:0/2787145804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8071e60 con 0x7fc3a80719c0 2024-01-16T15:07:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.497+0000 7fc3a7400640 1 --2- 172.21.15.43:0/2787145804 >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 0x7fc3a806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.499+0000 7fc3a6000640 1 --2- 172.21.15.43:0/2787145804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc39802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b23ea99596f0c6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.500+0000 7fc3a5600640 1 -- 172.21.15.43:0/2787145804 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398030840 con 0x7fc3a80719c0 2024-01-16T15:07:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.500+0000 7fc3a5600640 1 -- 172.21.15.43:0/2787145804 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3980309a0 con 0x7fc3a80719c0 2024-01-16T15:07:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.500+0000 7fc3a5600640 1 -- 172.21.15.43:0/2787145804 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398030d30 con 0x7fc3a80719c0 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 -- 172.21.15.43:0/2787145804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 msgr2=0x7fc3a80ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 --2- 172.21.15.43:0/2787145804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc39802b980 comp rx=0 tx=0).stop 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 -- 172.21.15.43:0/2787145804 shutdown_connections 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 --2- 172.21.15.43:0/2787145804 >> v2:172.21.15.43:3300/0 conn(0x7fc3a806d520 0x7fc3a806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 --2- 172.21.15.43:0/2787145804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a80ae6a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 -- 172.21.15.43:0/2787145804 >> 172.21.15.43:0/2787145804 conn(0x7fc3a80b0d80 msgr2=0x7fc3a80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 -- 172.21.15.43:0/2787145804 shutdown_connections 2024-01-16T15:07:08.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.501+0000 7fc3ade00640 1 -- 172.21.15.43:0/2787145804 wait complete. 2024-01-16T15:07:08.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.504+0000 7fc3ade00640 1 Processor -- start 2024-01-16T15:07:08.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.504+0000 7fc3ade00640 1 -- start start 2024-01-16T15:07:08.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.506+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 0x7fc3a815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.506+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.506+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 0x7fc3a809c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3ade00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a809b810 con 0x7fc3a8161930 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3ade00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a815b9b0 con 0x7fc3a806d520 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3ade00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a815d220 con 0x7fc3a80719c0 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 0x7fc3a815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 0x7fc3a809c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36036/0 (socket says 172.21.15.43:36036) 2024-01-16T15:07:08.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.507+0000 7fc3a6000640 1 -- 172.21.15.43:0/410410663 learned_addr learned my addr 172.21.15.43:0/410410663 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:08.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a6a00640 1 -- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 msgr2=0x7fc3a809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a6a00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a6a00640 1 -- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 msgr2=0x7fc3a809c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a6a00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 0x7fc3a809c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a6a00640 1 -- 172.21.15.43:0/410410663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3980094d0 con 0x7fc3a806d520 2024-01-16T15:07:08.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.509+0000 7fc3a7400640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 0x7fc3a809c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:08.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.510+0000 7fc3a6000640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:08.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.511+0000 7fc3a6a00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 0x7fc3a815d720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c400 tx=0x7fc39000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.512+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3900052d0 con 0x7fc3a806d520 2024-01-16T15:07:08.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.512+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc390005430 con 0x7fc3a806d520 2024-01-16T15:07:08.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.512+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc390003830 con 0x7fc3a806d520 2024-01-16T15:07:08.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.512+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a809ac00 con 0x7fc3a806d520 2024-01-16T15:07:08.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.512+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a809b1c0 con 0x7fc3a806d520 2024-01-16T15:07:08.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.516+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc390003990 con 0x7fc3a806d520 2024-01-16T15:07:08.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.517+0000 7fc39f400640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 0x7fc38006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:08.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.517+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc3900879e0 con 0x7fc3a806d520 2024-01-16T15:07:08.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.518+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a8166510 con 0x7fc3a806d520 2024-01-16T15:07:08.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.519+0000 7fc3a6000640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 0x7fc38006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:08.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.523+0000 7fc3a6000640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 0x7fc38006e080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc398008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:08.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.532+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc390054350 con 0x7fc3a806d520 2024-01-16T15:07:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.891+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a8074770 con 0x7fc3a806d520 2024-01-16T15:07:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.893+0000 7fc39f400640 1 -- 172.21.15.43:0/410410663 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12981 (secure 0 0 0) 0x7fc390017070 con 0x7fc3a806d520 2024-01-16T15:07:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:08.871 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":64,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:07.828633+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/4073013627":"2024-01-17T15:07:04.807372+0000","172.21.15.69:6832/4073013627":"2024-01-17T15:07:04.807372+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 msgr2=0x7fc38006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 0x7fc38006e080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc398008030 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 msgr2=0x7fc3a815d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 0x7fc3a815d720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c400 tx=0x7fc39000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 shutdown_connections 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbc0 0x7fc38006e080 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3a8161930 0x7fc3a809c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a80719c0 0x7fc3a809bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 --2- 172.21.15.43:0/410410663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a806d520 0x7fc3a815d720 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 >> 172.21.15.43:0/410410663 conn(0x7fc3a80b0d80 msgr2=0x7fc3a80b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 shutdown_connections 2024-01-16T15:07:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:08.895+0000 7fc3ade00640 1 -- 172.21.15.43:0/410410663 wait complete. 2024-01-16T15:07:08.893 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:08.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:09.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.101+0000 7fb9e3400640 1 Processor -- start 2024-01-16T15:07:09.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.101+0000 7fb9e3400640 1 -- start start 2024-01-16T15:07:09.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.102+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc15d3b0 0x7fb9dc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.102+0000 7fb9e3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0d2010 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc0c6bf0 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d1400 con 0x7fb9dc15d3b0 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38134/0 (socket says 172.21.15.43:38134) 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 -- 172.21.15.43:0/363158489 learned_addr learned my addr 172.21.15.43:0/363158489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e2000640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc15d3b0 0x7fb9dc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 -- 172.21.15.43:0/363158489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc15d3b0 msgr2=0x7fb9dc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc15d3b0 0x7fb9dc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 -- 172.21.15.43:0/363158489 >> v1:172.21.15.69:6789/0 conn(0x7fb9dc0d1bc0 legacy=0x7fb9dc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.103+0000 7fb9e1600640 1 -- 172.21.15.43:0/363158489 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0cedc0 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.104+0000 7fb9e1600640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009870 tx=0x7fb9c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebb3cdfb248a71d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e0c00640 1 -- 172.21.15.43:0/363158489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c40317c0 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e0c00640 1 -- 172.21.15.43:0/363158489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4031920 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e0c00640 1 -- 172.21.15.43:0/363158489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4031cb0 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 -- 172.21.15.43:0/363158489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d0fa0 msgr2=0x7fb9dc0d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009870 tx=0x7fb9c402b860 comp rx=0 tx=0).stop 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 -- 172.21.15.43:0/363158489 shutdown_connections 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d0fa0 0x7fb9dc0d1680 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 --2- 172.21.15.43:0/363158489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc15d3b0 0x7fb9dc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 -- 172.21.15.43:0/363158489 >> 172.21.15.43:0/363158489 conn(0x7fb9dc0d30a0 msgr2=0x7fb9dc0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 -- 172.21.15.43:0/363158489 shutdown_connections 2024-01-16T15:07:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.105+0000 7fb9e3400640 1 -- 172.21.15.43:0/363158489 wait complete. 2024-01-16T15:07:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.107+0000 7fb9e3400640 1 Processor -- start 2024-01-16T15:07:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.107+0000 7fb9e3400640 1 -- start start 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc0d0fa0 0x7fb9dc15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9dc15d3b0 0x7fb9dc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0b5d30 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc09dcd0 con 0x7fb9dc15d3b0 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc09d150 con 0x7fb9dc0d0fa0 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38146/0 (socket says 172.21.15.43:38146) 2024-01-16T15:07:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.108+0000 7fb9e1600640 1 -- 172.21.15.43:0/667179834 learned_addr learned my addr 172.21.15.43:0/667179834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e2000640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc0d0fa0 0x7fb9dc15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 -- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc0d0fa0 msgr2=0x7fb9dc15a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc0d0fa0 0x7fb9dc15a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e2a00640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9dc15d3b0 0x7fb9dc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 -- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9dc15d3b0 msgr2=0x7fb9dc0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9dc15d3b0 0x7fb9dc0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 -- 172.21.15.43:0/667179834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c40094d0 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e1600640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4000c00 tx=0x7fb9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4032860 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c40329c0 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4032d70 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.109+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc09d430 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.110+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc09d9f0 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.111+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc0b5e70 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.112+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb9c404e450 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.112+0000 7fb9da000640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 0x7fb9b406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.112+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb9c40aa520 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.113+0000 7fb9e2000640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 0x7fb9b406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.113+0000 7fb9e2000640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 0x7fb9b406e080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009f30 tx=0x7fb9cc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.117+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9dc0b5e70 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.301+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9dc0d2010 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.302+0000 7fb9da000640 1 -- 172.21.15.43:0/667179834 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9dc0d2010 con 0x7fb9dc0d1bc0 2024-01-16T15:07:09.280 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 msgr2=0x7fb9b406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 0x7fb9b406e080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009f30 tx=0x7fb9cc00b040 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 msgr2=0x7fb9dc0b8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4000c00 tx=0x7fb9c40046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 shutdown_connections 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb9b406bbc0 0x7fb9b406e080 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9dc15d3b0 0x7fb9dc0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9dc0d1bc0 0x7fb9dc0b8640 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 --2- 172.21.15.43:0/667179834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9dc0d0fa0 0x7fb9dc15a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.304+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 >> 172.21.15.43:0/667179834 conn(0x7fb9dc0d30a0 msgr2=0x7fb9dc0d3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.305+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 shutdown_connections 2024-01-16T15:07:09.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.305+0000 7fb9e3400640 1 -- 172.21.15.43:0/667179834 wait complete. 2024-01-16T15:07:09.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:07:09.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.550+0000 7f7a9e800640 1 Processor -- start 2024-01-16T15:07:09.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.551+0000 7f7a9e800640 1 -- start start 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a980b2510 0x7f7a98070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9e800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980b2a40 con 0x7f7a980cb990 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980b2d20 con 0x7f7a980cbd70 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a980b3a60 con 0x7f7a980b2510 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9d400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7a980cb990 0x7f7a9815af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51692/0 (socket says 172.21.15.43:51692) 2024-01-16T15:07:09.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9d400640 1 -- 172.21.15.43:0/734242199 learned_addr learned my addr 172.21.15.43:0/734242199 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555682187 0 0) 0x7f7a980b2a40 con 0x7f7a980cb990 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a78003520 con 0x7f7a980cb990 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.553+0000 7f7a9ca00640 1 --2- 172.21.15.43:0/734242199 >> v2:172.21.15.69:3300/0 conn(0x7f7a980b2510 0x7f7a98070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757936180 0 0) 0x7f7a78003520 con 0x7f7a980cb990 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 >> v2:172.21.15.69:3300/0 conn(0x7f7a980b2510 msgr2=0x7f7a98070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 --2- 172.21.15.43:0/734242199 >> v2:172.21.15.69:3300/0 conn(0x7f7a980b2510 0x7f7a98070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 >> v1:172.21.15.69:6789/0 conn(0x7f7a980cbd70 legacy=0x7f7a9806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980b3000 con 0x7f7a980cb990 2024-01-16T15:07:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7a80001690 con 0x7f7a980cb990 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 >> v1:172.21.15.43:6789/0 conn(0x7f7a980cb990 legacy=0x7f7a9815af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a78003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a780047c0 0x7f7a78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a780052a0 0x7f7a7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a980b3a60 con 0x7f7a78003a30 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7800c1a0 con 0x7f7a780052a0 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7800c360 con 0x7f7a780047c0 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.554+0000 7f7a9d400640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a78003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9de00640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a780052a0 0x7f7a7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9d400640 1 -- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a780047c0 msgr2=0x7f7a78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9ca00640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a780047c0 0x7f7a78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9d400640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a780047c0 0x7f7a78004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9d400640 1 -- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a780052a0 msgr2=0x7f7a7800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9d400640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a780052a0 0x7f7a7800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9d400640 1 -- 172.21.15.43:0/734242199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78003520 con 0x7f7a78003a30 2024-01-16T15:07:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.555+0000 7f7a9de00640 1 -- 172.21.15.43:0/734242199 reap_dead start 2024-01-16T15:07:09.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a9d400640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a78003e50 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a980cc210 tx=0x7f7a780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69a2e0d0e25b56f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80005ad0 con 0x7f7a78003a30 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80005c30 con 0x7f7a78003a30 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a97e00640 1 -- 172.21.15.43:0/734242199 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002c790 con 0x7f7a78003a30 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a9e800640 1 -- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 msgr2=0x7f7a78003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.556+0000 7f7a9e800640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a78003e50 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a980cc210 tx=0x7f7a780057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.557+0000 7f7a9e800640 1 -- 172.21.15.43:0/734242199 shutdown_connections 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.557+0000 7f7a9e800640 1 --2- 172.21.15.43:0/734242199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a78003e50 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.557+0000 7f7a9e800640 1 -- 172.21.15.43:0/734242199 >> 172.21.15.43:0/734242199 conn(0x7f7a980b1630 msgr2=0x7f7a980b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.557+0000 7f7a9e800640 1 -- 172.21.15.43:0/734242199 shutdown_connections 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.557+0000 7f7a9e800640 1 -- 172.21.15.43:0/734242199 wait complete. 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.558+0000 7f7a9e800640 1 Processor -- start 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.558+0000 7f7a9e800640 1 -- start start 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9813e750 0x7f7a98141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a981421c0 0x7f7a98158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a98142640 con 0x7f7a78003a30 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9815b5d0 con 0x7f7a9813e750 2024-01-16T15:07:09.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.559+0000 7f7a9e800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9815c430 con 0x7f7a981421c0 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38166/0 (socket says 172.21.15.43:38166) 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 -- 172.21.15.43:0/3489647670 learned_addr learned my addr 172.21.15.43:0/3489647670 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 -- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a981421c0 msgr2=0x7f7a98158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a981421c0 0x7f7a98158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 -- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9813e750 msgr2=0x7f7a98141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9813e750 0x7f7a98141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 -- 172.21.15.43:0/3489647670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9815b820 con 0x7f7a78003a30 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.560+0000 7f7a9d400640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c1a0 tx=0x7f7a980b35c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.561+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8003c070 con 0x7f7a78003a30 2024-01-16T15:07:09.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.561+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8002ca60 con 0x7f7a78003a30 2024-01-16T15:07:09.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.561+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002cdf0 con 0x7f7a78003a30 2024-01-16T15:07:09.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.561+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9815bb00 con 0x7f7a78003a30 2024-01-16T15:07:09.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.561+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9815c0c0 con 0x7f7a78003a30 2024-01-16T15:07:09.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.562+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9806e950 con 0x7f7a78003a30 2024-01-16T15:07:09.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.563+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7a80032070 con 0x7f7a78003a30 2024-01-16T15:07:09.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.564+0000 7f7a95600640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 0x7f7a7006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.564+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7a800a9580 con 0x7f7a78003a30 2024-01-16T15:07:09.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.564+0000 7f7a9ca00640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 0x7f7a7006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.564+0000 7f7a9ca00640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 0x7f7a7006e080 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a88006a90 tx=0x7f7a88006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.568+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a80075e40 con 0x7f7a78003a30 2024-01-16T15:07:09.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.765+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7a9815c250 con 0x7f7a78003a30 2024-01-16T15:07:09.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.766+0000 7f7a95600640 1 -- 172.21.15.43:0/3489647670 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+122 (secure 0 0 0) 0x7f7a80079460 con 0x7f7a78003a30 2024-01-16T15:07:09.744 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6834/4073013627 2024-01-17T15:07:04.807372+0000 2024-01-16T15:07:09.744 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/4073013627 2024-01-17T15:07:04.807372+0000 2024-01-16T15:07:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:07:09.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.768+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 msgr2=0x7f7a7006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.768+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 0x7f7a7006e080 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a88006a90 tx=0x7f7a88006a20 comp rx=0 tx=0).stop 2024-01-16T15:07:09.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 msgr2=0x7f7a9813b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c1a0 tx=0x7f7a980b35c0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 shutdown_connections 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7a7006bbc0 0x7f7a7006e080 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9813e750 0x7f7a98141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a981421c0 0x7f7a98158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 --2- 172.21.15.43:0/3489647670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a78003a30 0x7f7a9813b900 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 >> 172.21.15.43:0/3489647670 conn(0x7f7a980b1630 msgr2=0x7f7a9815da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 shutdown_connections 2024-01-16T15:07:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:09.769+0000 7f7a9e800640 1 -- 172.21.15.43:0/3489647670 wait complete. 2024-01-16T15:07:09.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:07:09.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:09.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:09.861 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:07:09.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:09.862 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:07:09.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[56534]: fuse finished with error 0 and tester_r 0 2024-01-16T15:07:10.111 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:07:10.111 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:10.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.419+0000 7f0857200640 1 Processor -- start 2024-01-16T15:07:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.419+0000 7f0857200640 1 -- start start 2024-01-16T15:07:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0857200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 0x7f08580cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0857200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08580cd7b0 0x7f08580cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0857200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085815fa70 con 0x7f085813dfd0 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0857200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08580cfa40 con 0x7f08580cd7b0 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0857200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085815c580 con 0x7f08580cf050 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0856800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08580cd7b0 0x7f08580cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0856800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08580cd7b0 0x7f08580cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38176/0 (socket says 172.21.15.43:38176) 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.421+0000 7f0856800640 1 -- 172.21.15.43:0/2473615477 learned_addr learned my addr 172.21.15.43:0/2473615477 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f084d200640 1 --2- 172.21.15.43:0/2473615477 >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 0x7f08580cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f0856800640 1 -- 172.21.15.43:0/2473615477 >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 msgr2=0x7f08580cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f0856800640 1 --2- 172.21.15.43:0/2473615477 >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 0x7f08580cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f0856800640 1 -- 172.21.15.43:0/2473615477 >> v1:172.21.15.69:6789/0 conn(0x7f085813dfd0 legacy=0x7f085815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f0856800640 1 -- 172.21.15.43:0/2473615477 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0858140900 con 0x7f08580cd7b0 2024-01-16T15:07:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.422+0000 7f084d200640 1 --2- 172.21.15.43:0/2473615477 >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 0x7f08580cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.423+0000 7f0856800640 1 --2- 172.21.15.43:0/2473615477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08580cd7b0 0x7f08580cdbb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f085806e070 tx=0x7f085002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b543ce27181a5365 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.423+0000 7f0855400640 1 -- 172.21.15.43:0/2473615477 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085002ca80 con 0x7f08580cd7b0 2024-01-16T15:07:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.423+0000 7f0855400640 1 -- 172.21.15.43:0/2473615477 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085002cbe0 con 0x7f08580cd7b0 2024-01-16T15:07:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.423+0000 7f0855400640 1 -- 172.21.15.43:0/2473615477 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500343f0 con 0x7f08580cd7b0 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 -- 172.21.15.43:0/2473615477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08580cd7b0 msgr2=0x7f08580cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 --2- 172.21.15.43:0/2473615477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08580cd7b0 0x7f08580cdbb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f085806e070 tx=0x7f085002bee0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 -- 172.21.15.43:0/2473615477 shutdown_connections 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 --2- 172.21.15.43:0/2473615477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08580cd7b0 0x7f08580cdbb0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 --2- 172.21.15.43:0/2473615477 >> v2:172.21.15.69:3300/0 conn(0x7f08580cf050 0x7f08580cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 -- 172.21.15.43:0/2473615477 >> 172.21.15.43:0/2473615477 conn(0x7f085806a860 msgr2=0x7f085806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 -- 172.21.15.43:0/2473615477 shutdown_connections 2024-01-16T15:07:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.424+0000 7f0857200640 1 -- 172.21.15.43:0/2473615477 wait complete. 2024-01-16T15:07:10.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.426+0000 7f0857200640 1 Processor -- start 2024-01-16T15:07:10.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.426+0000 7f0857200640 1 -- start start 2024-01-16T15:07:10.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 0x7f0858139700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 0x7f0858071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858139c40 con 0x7f085813dfd0 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085815dc10 con 0x7f08580cf050 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0857200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085815def0 con 0x7f08580cd7b0 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38190/0 (socket says 172.21.15.43:38190) 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 -- 172.21.15.43:0/2568322902 learned_addr learned my addr 172.21.15.43:0/2568322902 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f084d200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 0x7f0858071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 -- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 msgr2=0x7f0858139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0855e00640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 0x7f0858139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 0x7f0858139700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 -- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 msgr2=0x7f0858071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 0x7f0858071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0856800640 1 -- 172.21.15.43:0/2568322902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850009b30 con 0x7f085813dfd0 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f084d200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 0x7f0858071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.427+0000 7f0855e00640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 0x7f0858139700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f0856800640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0850036040 tx=0x7f0850004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085003e450 con 0x7f085813dfd0 2024-01-16T15:07:10.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085003e5d0 con 0x7f085813dfd0 2024-01-16T15:07:10.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085003e980 con 0x7f085813dfd0 2024-01-16T15:07:10.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085815e1d0 con 0x7f085813dfd0 2024-01-16T15:07:10.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.428+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085815e790 con 0x7f085813dfd0 2024-01-16T15:07:10.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.429+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085804e840 con 0x7f085813dfd0 2024-01-16T15:07:10.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.430+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f085002cd50 con 0x7f085813dfd0 2024-01-16T15:07:10.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.431+0000 7f084ea00640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 0x7f083006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.431+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f08500ab100 con 0x7f085813dfd0 2024-01-16T15:07:10.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.431+0000 7f0855e00640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 0x7f083006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.432+0000 7f0855e00640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 0x7f083006e080 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f084800a830 tx=0x7f0848008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.435+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08500779e0 con 0x7f085813dfd0 2024-01-16T15:07:10.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.634+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08580b3b10 con 0x7f085813dfd0 2024-01-16T15:07:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.635+0000 7f084ea00640 1 -- 172.21.15.43:0/2568322902 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12981 (secure 0 0 0) 0x7f0850043090 con 0x7f085813dfd0 2024-01-16T15:07:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:10.614 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":64,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:07.828633+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/4073013627":"2024-01-17T15:07:04.807372+0000","172.21.15.69:6832/4073013627":"2024-01-17T15:07:04.807372+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:10.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 msgr2=0x7f083006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 0x7f083006e080 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f084800a830 tx=0x7f0848008040 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 msgr2=0x7f085815a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0850036040 tx=0x7f0850004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 shutdown_connections 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f083006bbc0 0x7f083006e080 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08580cf050 0x7f0858071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f085813dfd0 0x7f085815a9e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 --2- 172.21.15.43:0/2568322902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08580cd7b0 0x7f0858139700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 >> 172.21.15.43:0/2568322902 conn(0x7f085806a860 msgr2=0x7f085813fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 shutdown_connections 2024-01-16T15:07:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.638+0000 7f0857200640 1 -- 172.21.15.43:0/2568322902 wait complete. 2024-01-16T15:07:10.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:10.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:10.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.886+0000 7f91edc00640 1 Processor -- start 2024-01-16T15:07:10.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.887+0000 7f91edc00640 1 -- start start 2024-01-16T15:07:10.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.888+0000 7f91edc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.888+0000 7f91edc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.888+0000 7f91edc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e8070d40 con 0x7f91e81435d0 2024-01-16T15:07:10.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.888+0000 7f91edc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e8071020 con 0x7f91e81439b0 2024-01-16T15:07:10.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.888+0000 7f91edc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e8071300 con 0x7f91e80b4140 2024-01-16T15:07:10.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.889+0000 7f91e7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.889+0000 7f91ed200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.889+0000 7f91e7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36066/0 (socket says 172.21.15.43:36066) 2024-01-16T15:07:10.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.889+0000 7f91e7400640 1 -- 172.21.15.43:0/1085486162 learned_addr learned my addr 172.21.15.43:0/1085486162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:10.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.889+0000 7f91ed200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43126/0 (socket says 172.21.15.43:43126) 2024-01-16T15:07:10.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.890+0000 7f91e7400640 1 -- 172.21.15.43:0/1085486162 >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 msgr2=0x7f91e8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:10.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.890+0000 7f91e7400640 1 --2- 172.21.15.43:0/1085486162 >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.890+0000 7f91e7400640 1 -- 172.21.15.43:0/1085486162 >> v1:172.21.15.69:6789/0 conn(0x7f91e81435d0 legacy=0x7f91e80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.890+0000 7f91e7400640 1 -- 172.21.15.43:0/1085486162 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80715e0 con 0x7f91e80b4140 2024-01-16T15:07:10.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.890+0000 7f91ed200640 1 --2- 172.21.15.43:0/1085486162 >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:10.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.891+0000 7f91e7400640 1 --2- 172.21.15.43:0/1085486162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009870 tx=0x7f91d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1d3874405609f6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.892+0000 7f91e6a00640 1 -- 172.21.15.43:0/1085486162 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0030840 con 0x7f91e80b4140 2024-01-16T15:07:10.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.892+0000 7f91e6a00640 1 -- 172.21.15.43:0/1085486162 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d00309a0 con 0x7f91e80b4140 2024-01-16T15:07:10.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.892+0000 7f91e6a00640 1 -- 172.21.15.43:0/1085486162 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0030d30 con 0x7f91e80b4140 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.892+0000 7f91edc00640 1 -- 172.21.15.43:0/1085486162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 msgr2=0x7f91e8071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.892+0000 7f91edc00640 1 --2- 172.21.15.43:0/1085486162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009870 tx=0x7f91d002b980 comp rx=0 tx=0).stop 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 -- 172.21.15.43:0/1085486162 shutdown_connections 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 --2- 172.21.15.43:0/1085486162 >> v2:172.21.15.69:3300/0 conn(0x7f91e81439b0 0x7f91e8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 --2- 172.21.15.43:0/1085486162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e80b4140 0x7f91e8071de0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 -- 172.21.15.43:0/1085486162 >> 172.21.15.43:0/1085486162 conn(0x7f91e80cec10 msgr2=0x7f91e80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 -- 172.21.15.43:0/1085486162 shutdown_connections 2024-01-16T15:07:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.893+0000 7f91edc00640 1 -- 172.21.15.43:0/1085486162 wait complete. 2024-01-16T15:07:10.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.895+0000 7f91edc00640 1 Processor -- start 2024-01-16T15:07:10.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.895+0000 7f91edc00640 1 -- start start 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b4140 0x7f91e80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e81439b0 0x7f91e813e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80aefd0 con 0x7f91e81435d0 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e813a8b0 con 0x7f91e80b4140 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91edc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80b1810 con 0x7f91e81439b0 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91e7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91e7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38202/0 (socket says 172.21.15.43:38202) 2024-01-16T15:07:10.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91e7400640 1 -- 172.21.15.43:0/2954534069 learned_addr learned my addr 172.21.15.43:0/2954534069 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91e7e00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b4140 0x7f91e80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.897+0000 7f91ed200640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e81439b0 0x7f91e813e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 -- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e81439b0 msgr2=0x7f91e813e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e81439b0 0x7f91e813e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 -- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b4140 msgr2=0x7f91e80b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b4140 0x7f91e80b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 -- 172.21.15.43:0/2954534069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e000bc80 con 0x7f91e81435d0 2024-01-16T15:07:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91e7400640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f91d0000c00 tx=0x7f91d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0039c90 con 0x7f91e81435d0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0039e10 con 0x7f91e81435d0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.898+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d00094d0 con 0x7f91e81435d0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.899+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0043640 con 0x7f91e81435d0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.899+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e813a260 con 0x7f91e81435d0 2024-01-16T15:07:10.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.900+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e813aa60 con 0x7f91e81435d0 2024-01-16T15:07:10.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.901+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f91d00437a0 con 0x7f91e81435d0 2024-01-16T15:07:10.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.902+0000 7f91dfe00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 0x7f91c006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:10.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.903+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f91d003e070 con 0x7f91e81435d0 2024-01-16T15:07:10.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.903+0000 7f91e7e00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 0x7f91c006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:10.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.904+0000 7f91e7e00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 0x7f91c006e080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f91d8001640 tx=0x7f91d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:10.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:10.907+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91e813aa60 con 0x7f91e81435d0 2024-01-16T15:07:11.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.101+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91e813aa20 con 0x7f91e81435d0 2024-01-16T15:07:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.102+0000 7f91dfe00640 1 -- 172.21.15.43:0/2954534069 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91d0077d30 con 0x7f91e81435d0 2024-01-16T15:07:11.080 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:07:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 msgr2=0x7f91c006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 0x7f91c006e080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f91d8001640 tx=0x7f91d8009040 comp rx=0 tx=0).stop 2024-01-16T15:07:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 msgr2=0x7f91e815bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f91d0000c00 tx=0x7f91d0004730 comp rx=0 tx=0).stop 2024-01-16T15:07:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 shutdown_connections 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f91c006bbc0 0x7f91c006e080 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91e81439b0 0x7f91e813e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91e81435d0 0x7f91e815bf90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 --2- 172.21.15.43:0/2954534069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b4140 0x7f91e80b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 >> 172.21.15.43:0/2954534069 conn(0x7f91e80cec10 msgr2=0x7f91e80ace40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 shutdown_connections 2024-01-16T15:07:11.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.105+0000 7f91edc00640 1 -- 172.21.15.43:0/2954534069 wait complete. 2024-01-16T15:07:11.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:07:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.346+0000 7fce4d800640 1 Processor -- start 2024-01-16T15:07:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.346+0000 7fce4d800640 1 -- start start 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce480abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fce480cbcb0 0x7fce4806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fce480ce880 0x7fce4806da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480ac530 con 0x7fce480ce880 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480b2080 con 0x7fce480cbcb0 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce4d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480b2360 con 0x7fce4815b340 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce47400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fce480ce880 0x7fce4806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce47400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fce480ce880 0x7fce4806da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38208/0 (socket says 172.21.15.43:38208) 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.347+0000 7fce47400640 1 -- 172.21.15.43:0/1069901101 learned_addr learned my addr 172.21.15.43:0/1069901101 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce46a00640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce480abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce3de00640 1 --2- 172.21.15.43:0/1069901101 >> v2:172.21.15.69:3300/0 conn(0x7fce480cbcb0 0x7fce4806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce47400640 1 -- 172.21.15.43:0/1069901101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 msgr2=0x7fce480abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce47400640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce480abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce47400640 1 -- 172.21.15.43:0/1069901101 >> v2:172.21.15.69:3300/0 conn(0x7fce480cbcb0 msgr2=0x7fce4806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce47400640 1 --2- 172.21.15.43:0/1069901101 >> v2:172.21.15.69:3300/0 conn(0x7fce480cbcb0 0x7fce4806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.348+0000 7fce47400640 1 -- 172.21.15.43:0/1069901101 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480b2640 con 0x7fce480ce880 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.349+0000 7fce47400640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480ce880 0x7fce4806da60 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fce4806bdb0 tx=0x7fce4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0b4e7cee6f690c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.349+0000 7fce46000640 1 -- 172.21.15.43:0/1069901101 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4002ca70 con 0x7fce480ce880 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.349+0000 7fce46000640 1 -- 172.21.15.43:0/1069901101 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4002cbd0 con 0x7fce480ce880 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.349+0000 7fce46000640 1 -- 172.21.15.43:0/1069901101 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce400343f0 con 0x7fce480ce880 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 -- 172.21.15.43:0/1069901101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480ce880 msgr2=0x7fce4806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480ce880 0x7fce4806da60 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fce4806bdb0 tx=0x7fce4002bed0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 -- 172.21.15.43:0/1069901101 shutdown_connections 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480ce880 0x7fce4806da60 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 --2- 172.21.15.43:0/1069901101 >> v2:172.21.15.69:3300/0 conn(0x7fce480cbcb0 0x7fce4806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 --2- 172.21.15.43:0/1069901101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce480abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 -- 172.21.15.43:0/1069901101 >> 172.21.15.43:0/1069901101 conn(0x7fce480b0d80 msgr2=0x7fce480b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 -- 172.21.15.43:0/1069901101 shutdown_connections 2024-01-16T15:07:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.350+0000 7fce4d800640 1 -- 172.21.15.43:0/1069901101 wait complete. 2024-01-16T15:07:11.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.352+0000 7fce4d800640 1 Processor -- start 2024-01-16T15:07:11.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.352+0000 7fce4d800640 1 -- start start 2024-01-16T15:07:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce480ce880 0x7fce4814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce4815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c9d20 con 0x7fce480cbcb0 2024-01-16T15:07:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c9760 con 0x7fce480ce880 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce4d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480c9a40 con 0x7fce4815b340 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce46a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce46a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38224/0 (socket says 172.21.15.43:38224) 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.353+0000 7fce46a00640 1 -- 172.21.15.43:0/3397045479 learned_addr learned my addr 172.21.15.43:0/3397045479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce47400640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce4815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 -- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 msgr2=0x7fce4815dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce3de00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce480ce880 0x7fce4814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce4815dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 -- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce480ce880 msgr2=0x7fce4814e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce480ce880 0x7fce4814e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 -- 172.21.15.43:0/3397045479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009b30 con 0x7fce480cbcb0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce46a00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fce3800d5f0 tx=0x7fce3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380125f0 con 0x7fce480cbcb0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce38012750 con 0x7fce480cbcb0 2024-01-16T15:07:11.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38012b00 con 0x7fce480cbcb0 2024-01-16T15:07:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4815b7c0 con 0x7fce480cbcb0 2024-01-16T15:07:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.354+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce480c9350 con 0x7fce480cbcb0 2024-01-16T15:07:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.356+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce480c9510 con 0x7fce480cbcb0 2024-01-16T15:07:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.356+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fce38012c60 con 0x7fce480cbcb0 2024-01-16T15:07:11.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.357+0000 7fce3f400640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 0x7fce2006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:11.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.357+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fce38088aa0 con 0x7fce480cbcb0 2024-01-16T15:07:11.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.357+0000 7fce3de00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 0x7fce2006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:11.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.358+0000 7fce3de00640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 0x7fce2006e080 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fce48155ab0 tx=0x7fce30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:11.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.362+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce38055170 con 0x7fce480cbcb0 2024-01-16T15:07:11.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.569+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fce48074770 con 0x7fce480cbcb0 2024-01-16T15:07:11.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.570+0000 7fce3f400640 1 -- 172.21.15.43:0/3397045479 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+122 (secure 0 0 0) 0x7fce38058790 con 0x7fce480cbcb0 2024-01-16T15:07:11.548 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6834/4073013627 2024-01-17T15:07:04.807372+0000 2024-01-16T15:07:11.548 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/4073013627 2024-01-17T15:07:04.807372+0000 2024-01-16T15:07:11.548 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 msgr2=0x7fce2006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 0x7fce2006e080 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fce48155ab0 tx=0x7fce30008040 comp rx=0 tx=0).stop 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 msgr2=0x7fce480c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fce3800d5f0 tx=0x7fce3800dab0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 shutdown_connections 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fce2006bbc0 0x7fce2006e080 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fce480ce880 0x7fce4814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fce4815b340 0x7fce4815dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 --2- 172.21.15.43:0/3397045479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce480cbcb0 0x7fce480c8e10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 >> 172.21.15.43:0/3397045479 conn(0x7fce480b0d80 msgr2=0x7fce48156000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 shutdown_connections 2024-01-16T15:07:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:11.572+0000 7fce4d800640 1 -- 172.21.15.43:0/3397045479 wait complete. 2024-01-16T15:07:11.570 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:07:11.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:11.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:11.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44117]: fuse finished with error 0 and tester_r 0 2024-01-16T15:07:11.672 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:07:11.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:11.672 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:07:11.854 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:11.854 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:11.917 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:11.917 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:07:11.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:11.917 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:07:11.975 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:07:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:07:11.976 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:07:11.977 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:12.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:12.033 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:07:12.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:12.034 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:07:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:07:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:07:12.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:12.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.337+0000 7fb5b0e00640 1 Processor -- start 2024-01-16T15:07:12.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.338+0000 7fb5b0e00640 1 -- start start 2024-01-16T15:07:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5b0e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5ac13dfd0 0x7fb5ac15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5b0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5ac0cf050 0x7fb5ac0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5b0e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0ce7c0 con 0x7fb5ac06c130 2024-01-16T15:07:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5b0e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac0cfa40 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5b0e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac140900 con 0x7fb5ac0cf050 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5ab400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5ac13dfd0 0x7fb5ac15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5ab400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5ac13dfd0 0x7fb5ac15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38234/0 (socket says 172.21.15.43:38234) 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.339+0000 7fb5ab400640 1 -- 172.21.15.43:0/2204899516 learned_addr learned my addr 172.21.15.43:0/2204899516 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.340+0000 7fb5aaa00640 1 --2- 172.21.15.43:0/2204899516 >> v2:172.21.15.69:3300/0 conn(0x7fb5ac0cf050 0x7fb5ac0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.340+0000 7fb5ab400640 1 -- 172.21.15.43:0/2204899516 >> v2:172.21.15.69:3300/0 conn(0x7fb5ac0cf050 msgr2=0x7fb5ac0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.340+0000 7fb5ab400640 1 --2- 172.21.15.43:0/2204899516 >> v2:172.21.15.69:3300/0 conn(0x7fb5ac0cf050 0x7fb5ac0cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.340+0000 7fb5ab400640 1 -- 172.21.15.43:0/2204899516 >> v1:172.21.15.69:6789/0 conn(0x7fb5ac06c130 legacy=0x7fb5ac06c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.340+0000 7fb5ab400640 1 -- 172.21.15.43:0/2204899516 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac15fa70 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.341+0000 7fb5ab400640 1 --2- 172.21.15.43:0/2204899516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac15be70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac06e070 tx=0x7fb59c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e8d5c912b45d98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.341+0000 7fb5aa000640 1 -- 172.21.15.43:0/2204899516 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c031870 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.341+0000 7fb5aa000640 1 -- 172.21.15.43:0/2204899516 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c0319d0 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.341+0000 7fb5aa000640 1 -- 172.21.15.43:0/2204899516 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c031d60 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 -- 172.21.15.43:0/2204899516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac13dfd0 msgr2=0x7fb5ac15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/2204899516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac15be70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac06e070 tx=0x7fb59c02b910 comp rx=0 tx=0).stop 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 -- 172.21.15.43:0/2204899516 shutdown_connections 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/2204899516 >> v2:172.21.15.69:3300/0 conn(0x7fb5ac0cf050 0x7fb5ac0cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/2204899516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac15be70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 -- 172.21.15.43:0/2204899516 >> 172.21.15.43:0/2204899516 conn(0x7fb5ac06a860 msgr2=0x7fb5ac06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 -- 172.21.15.43:0/2204899516 shutdown_connections 2024-01-16T15:07:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.342+0000 7fb5b0e00640 1 -- 172.21.15.43:0/2204899516 wait complete. 2024-01-16T15:07:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.344+0000 7fb5b0e00640 1 Processor -- start 2024-01-16T15:07:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.344+0000 7fb5b0e00640 1 -- start start 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5ac0cf050 0x7fb5ac0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac1f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0c8400 con 0x7fb5ac06c130 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0ca3b0 con 0x7fb5ac13dfd0 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5b0e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0ca650 con 0x7fb5ac0cf050 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38250/0 (socket says 172.21.15.43:38250) 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.345+0000 7fb5ab400640 1 -- 172.21.15.43:0/3681099614 learned_addr learned my addr 172.21.15.43:0/3681099614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5aaa00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5ac0cf050 0x7fb5ac0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5abe00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac1f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 -- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5ac0cf050 msgr2=0x7fb5ac0c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5ac0cf050 0x7fb5ac0c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 -- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5ac13dfd0 msgr2=0x7fb5ac1f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac1f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 -- 172.21.15.43:0/3681099614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0094d0 con 0x7fb5ac06c130 2024-01-16T15:07:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5ab400640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0090b0 tx=0x7fb59c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c03ad20 con 0x7fb5ac06c130 2024-01-16T15:07:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c0443f0 con 0x7fb5ac06c130 2024-01-16T15:07:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c03aea0 con 0x7fb5ac06c130 2024-01-16T15:07:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.346+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac0ca930 con 0x7fb5ac06c130 2024-01-16T15:07:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.347+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1589d0 con 0x7fb5ac06c130 2024-01-16T15:07:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.348+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac0cae50 con 0x7fb5ac06c130 2024-01-16T15:07:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.349+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb59c04d450 con 0x7fb5ac06c130 2024-01-16T15:07:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.350+0000 7fb5a3400640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 0x7fb58406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.350+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb59c02d040 con 0x7fb5ac06c130 2024-01-16T15:07:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.350+0000 7fb5aaa00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 0x7fb58406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.351+0000 7fb5aaa00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 0x7fb58406e080 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac1f6410 tx=0x7fb594008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.354+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5ac0cae50 con 0x7fb5ac06c130 2024-01-16T15:07:12.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.586+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5ac0ca790 con 0x7fb5ac06c130 2024-01-16T15:07:12.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.588+0000 7fb5a3400640 1 -- 172.21.15.43:0/3681099614 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+3690 (secure 0 0 0) 0x7fb5ac0ca790 con 0x7fb5ac06c130 2024-01-16T15:07:12.566 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:12.566 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/986973826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":986973826},{"type":"v1","addr":"172.21.15.69:6835","nonce":986973826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5644,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/607340672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":607340672},{"type":"v1","addr":"172.21.15.43:6836","nonce":607340672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/154390899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":154390899},{"type":"v1","addr":"172.21.15.43:6837","nonce":154390899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6016,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2149841884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2149841884},{"type":"v1","addr":"172.21.15.69:6834","nonce":2149841884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-16T15:07:12.567 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 39 2024-01-16T15:07:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.590+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 msgr2=0x7fb58406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.590+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 0x7fb58406e080 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac1f6410 tx=0x7fb594008040 comp rx=0 tx=0).stop 2024-01-16T15:07:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.590+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 msgr2=0x7fb5ac0cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.590+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0090b0 tx=0x7fb59c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 shutdown_connections 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb58406bbc0 0x7fb58406e080 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5ac13dfd0 0x7fb5ac1f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5ac0cf050 0x7fb5ac0c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 --2- 172.21.15.43:0/3681099614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac06c130 0x7fb5ac0cb130 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 >> 172.21.15.43:0/3681099614 conn(0x7fb5ac06a860 msgr2=0x7fb5ac13e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 shutdown_connections 2024-01-16T15:07:12.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.591+0000 7fb5b0e00640 1 -- 172.21.15.43:0/3681099614 wait complete. 2024-01-16T15:07:12.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:07:12.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.838+0000 7ff70ac00640 1 Processor -- start 2024-01-16T15:07:12.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.838+0000 7ff70ac00640 1 -- start start 2024-01-16T15:07:12.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff7040b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff70ac00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7040d13b0 0x7ff70415c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff70ac00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040b13b0 con 0x7ff7040b2450 2024-01-16T15:07:12.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff70ac00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7040b11c0 con 0x7ff7040d13b0 2024-01-16T15:07:12.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff70ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70415c640 con 0x7ff70415af50 2024-01-16T15:07:12.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.839+0000 7ff708e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff7040b2450 0x7ff7040d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51706/0 (socket says 172.21.15.43:51706) 2024-01-16T15:07:12.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff708e00640 1 -- 172.21.15.43:0/2393383280 learned_addr learned my addr 172.21.15.43:0/2393383280 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff709800640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff7040b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff70a200640 1 --2- 172.21.15.43:0/2393383280 >> v2:172.21.15.69:3300/0 conn(0x7ff7040d13b0 0x7ff70415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977894629 0 0) 0x7ff7040b13b0 con 0x7ff7040b2450 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e4003520 con 0x7ff7040b2450 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1565762918 0 0) 0x7ff6e4003520 con 0x7ff7040b2450 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 msgr2=0x7ff7040b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff7040b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 >> v2:172.21.15.69:3300/0 conn(0x7ff7040d13b0 msgr2=0x7ff70415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 --2- 172.21.15.43:0/2393383280 >> v2:172.21.15.69:3300/0 conn(0x7ff7040d13b0 0x7ff70415c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7040cc830 con 0x7ff7040b2450 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff6ec001670 con 0x7ff7040b2450 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.840+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 >> v1:172.21.15.43:6789/0 conn(0x7ff7040b2450 legacy=0x7ff7040d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6e40039e0 0x7ff6e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40047c0 0x7ff6e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff6e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e400ba50 con 0x7ff6e40039e0 2024-01-16T15:07:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e400bbc0 con 0x7ff6e40047c0 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e400bd80 con 0x7ff70415af50 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6e40039e0 0x7ff6e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff708e00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40047c0 0x7ff6e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff70a200640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff6e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 -- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 msgr2=0x7ff6e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff70415af50 0x7ff6e4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 -- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40047c0 msgr2=0x7ff6e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40047c0 0x7ff6e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.841+0000 7ff709800640 1 -- 172.21.15.43:0/2393383280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4003520 con 0x7ff6e40039e0 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff709800640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6e40039e0 0x7ff6e4003e00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400ba30 tx=0x7ff6e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f4e8e8a1fe1a1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4032640 con 0x7ff6e40039e0 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f40327a0 con 0x7ff6e40039e0 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff703e00640 1 -- 172.21.15.43:0/2393383280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4032b30 con 0x7ff6e40039e0 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff70ac00640 1 -- 172.21.15.43:0/2393383280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6e40039e0 msgr2=0x7ff6e4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff70ac00640 1 --2- 172.21.15.43:0/2393383280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6e40039e0 0x7ff6e4003e00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400ba30 tx=0x7ff6e40057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.843+0000 7ff70a200640 1 -- 172.21.15.43:0/2393383280 reap_dead start 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.844+0000 7ff70ac00640 1 -- 172.21.15.43:0/2393383280 shutdown_connections 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.844+0000 7ff70ac00640 1 -- 172.21.15.43:0/2393383280 >> 172.21.15.43:0/2393383280 conn(0x7ff704099a00 msgr2=0x7ff704099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.844+0000 7ff70ac00640 1 -- 172.21.15.43:0/2393383280 shutdown_connections 2024-01-16T15:07:12.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.844+0000 7ff70ac00640 1 -- 172.21.15.43:0/2393383280 wait complete. 2024-01-16T15:07:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.845+0000 7ff70ac00640 1 Processor -- start 2024-01-16T15:07:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.846+0000 7ff70ac00640 1 -- start start 2024-01-16T15:07:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.846+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 0x7ff704159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7040c88f0 0x7ff7040c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70415bf60 con 0x7ff7040c8d70 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70ac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040c9320 con 0x7ff6e40039e0 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040c9600 con 0x7ff7040c88f0 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38266/0 (socket says 172.21.15.43:38266) 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 -- 172.21.15.43:0/806944580 learned_addr learned my addr 172.21.15.43:0/806944580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff709800640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 0x7ff704159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 -- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7040c88f0 msgr2=0x7ff7040c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff708e00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7040c88f0 0x7ff7040c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7040c88f0 0x7ff7040c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 -- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 msgr2=0x7ff704159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 0x7ff704159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff70a200640 1 -- 172.21.15.43:0/806944580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7040c98e0 con 0x7ff7040c8d70 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.847+0000 7ff709800640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 0x7ff704159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff70a200640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004740 tx=0x7ff6fc00e510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc007e20 con 0x7ff7040c8d70 2024-01-16T15:07:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc00f040 con 0x7ff7040c8d70 2024-01-16T15:07:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc015650 con 0x7ff7040c8d70 2024-01-16T15:07:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70415bb10 con 0x7ff7040c8d70 2024-01-16T15:07:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.848+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7040c9d20 con 0x7ff7040c8d70 2024-01-16T15:07:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.849+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7040c9f40 con 0x7ff7040c8d70 2024-01-16T15:07:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.850+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7ff6fc0157b0 con 0x7ff7040c8d70 2024-01-16T15:07:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.851+0000 7ff701600640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 0x7ff6dc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.851+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff6fc088f70 con 0x7ff7040c8d70 2024-01-16T15:07:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.851+0000 7ff709800640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 0x7ff6dc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.852+0000 7ff709800640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 0x7ff6dc06e080 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400ba00 tx=0x7ff6f403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:12.855+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7040c9f40 con 0x7ff7040c8d70 2024-01-16T15:07:13.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.054+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7ff7040c9f40 con 0x7ff7040c8d70 2024-01-16T15:07:13.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.061+0000 7ff701600640 1 -- 172.21.15.43:0/806944580 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v23) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff7040c9f40 con 0x7ff7040c8d70 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 msgr2=0x7ff6dc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 0x7ff6dc06e080 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400ba00 tx=0x7ff6f403d040 comp rx=0 tx=0).stop 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 msgr2=0x7ff7040c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004740 tx=0x7ff6fc00e510 comp rx=0 tx=0).stop 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 shutdown_connections 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff6dc06bbc0 0x7ff6dc06e080 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7040c8d70 0x7ff7040c64c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7040c88f0 0x7ff7040c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 --2- 172.21.15.43:0/806944580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e40039e0 0x7ff704159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 >> 172.21.15.43:0/806944580 conn(0x7ff704099a00 msgr2=0x7ff7040b2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 shutdown_connections 2024-01-16T15:07:13.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.065+0000 7ff70ac00640 1 -- 172.21.15.43:0/806944580 wait complete. 2024-01-16T15:07:13.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:07:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.313+0000 7fe502400640 1 Processor -- start 2024-01-16T15:07:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.313+0000 7fe502400640 1 -- start start 2024-01-16T15:07:13.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe502400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4fc1435d0 0x7fe4fc071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe502400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe502400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc070d40 con 0x7fe4fc0b3f70 2024-01-16T15:07:13.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe502400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc0b43e0 con 0x7fe4fc1439b0 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe502400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc071020 con 0x7fe4fc1435d0 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.315+0000 7fe4fbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53212/0 (socket says 172.21.15.43:53212) 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 -- 172.21.15.43:0/4131433343 learned_addr learned my addr 172.21.15.43:0/4131433343 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe501000640 1 --2- 172.21.15.43:0/4131433343 >> v2:172.21.15.69:3300/0 conn(0x7fe4fc1435d0 0x7fe4fc071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 -- 172.21.15.43:0/4131433343 >> v2:172.21.15.69:3300/0 conn(0x7fe4fc1435d0 msgr2=0x7fe4fc071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 --2- 172.21.15.43:0/4131433343 >> v2:172.21.15.69:3300/0 conn(0x7fe4fc1435d0 0x7fe4fc071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 -- 172.21.15.43:0/4131433343 >> v1:172.21.15.69:6789/0 conn(0x7fe4fc0b3f70 legacy=0x7fe4fc070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.316+0000 7fe4fbe00640 1 -- 172.21.15.43:0/4131433343 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc071300 con 0x7fe4fc1439b0 2024-01-16T15:07:13.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.317+0000 7fe4fbe00640 1 --2- 172.21.15.43:0/4131433343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0cfc10 tx=0x7fe4e402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc95fb9b4355c1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.317+0000 7fe4fb400640 1 -- 172.21.15.43:0/4131433343 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e402ec00 con 0x7fe4fc1439b0 2024-01-16T15:07:13.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe4fb400640 1 -- 172.21.15.43:0/4131433343 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e402ed60 con 0x7fe4fc1439b0 2024-01-16T15:07:13.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe4fb400640 1 -- 172.21.15.43:0/4131433343 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4036610 con 0x7fe4fc1439b0 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 -- 172.21.15.43:0/4131433343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1439b0 msgr2=0x7fe4fc0b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 --2- 172.21.15.43:0/4131433343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0cfc10 tx=0x7fe4e402e060 comp rx=0 tx=0).stop 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 -- 172.21.15.43:0/4131433343 shutdown_connections 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 --2- 172.21.15.43:0/4131433343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1439b0 0x7fe4fc0b3a30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 --2- 172.21.15.43:0/4131433343 >> v2:172.21.15.69:3300/0 conn(0x7fe4fc1435d0 0x7fe4fc071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 -- 172.21.15.43:0/4131433343 >> 172.21.15.43:0/4131433343 conn(0x7fe4fc0cec10 msgr2=0x7fe4fc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.318+0000 7fe502400640 1 -- 172.21.15.43:0/4131433343 shutdown_connections 2024-01-16T15:07:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.319+0000 7fe502400640 1 -- 172.21.15.43:0/4131433343 wait complete. 2024-01-16T15:07:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.320+0000 7fe502400640 1 Processor -- start 2024-01-16T15:07:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.320+0000 7fe502400640 1 -- start start 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 0x7fe4fc1f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4fc1439b0 0x7fe4fc0c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0c5620 con 0x7fe4fc1435d0 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0acba0 con 0x7fe4fc0b3f70 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.321+0000 7fe502400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0ace80 con 0x7fe4fc1439b0 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53228/0 (socket says 172.21.15.43:53228) 2024-01-16T15:07:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 -- 172.21.15.43:0/2668027116 learned_addr learned my addr 172.21.15.43:0/2668027116 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe501000640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 0x7fe4fc1f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 -- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4fc1439b0 msgr2=0x7fe4fc0c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe501a00640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4fc1439b0 0x7fe4fc0c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4fc1439b0 0x7fe4fc0c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 -- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 msgr2=0x7fe4fc1f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 0x7fe4fc1f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe4fbe00640 1 -- 172.21.15.43:0/2668027116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4009b30 con 0x7fe4fc1435d0 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.322+0000 7fe501000640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 0x7fe4fc1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe4fbe00640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4038040 tx=0x7fe4e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e402ec00 con 0x7fe4fc1435d0 2024-01-16T15:07:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4009510 con 0x7fe4fc1435d0 2024-01-16T15:07:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e403f690 con 0x7fe4fc1435d0 2024-01-16T15:07:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc0ad160 con 0x7fe4fc1435d0 2024-01-16T15:07:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.323+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc1f4460 con 0x7fe4fc1435d0 2024-01-16T15:07:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.324+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc0ad680 con 0x7fe4fc1435d0 2024-01-16T15:07:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.325+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe4e403f7f0 con 0x7fe4fc1435d0 2024-01-16T15:07:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.326+0000 7fe4f8c00640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 0x7fe4d406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.326+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe4e40aacf0 con 0x7fe4fc1435d0 2024-01-16T15:07:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.326+0000 7fe501000640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 0x7fe4d406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.327+0000 7fe501000640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 0x7fe4d406e080 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec006fd0 tx=0x7fe4ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.331+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4fc0ad680 con 0x7fe4fc1435d0 2024-01-16T15:07:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.523+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fe4fc0ad680 con 0x7fe4fc1435d0 2024-01-16T15:07:13.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.530+0000 7fe4f8c00640 1 -- 172.21.15.43:0/2668027116 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v24) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe4fc0ad680 con 0x7fe4fc1435d0 2024-01-16T15:07:13.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 msgr2=0x7fe4d406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 0x7fe4d406e080 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec006fd0 tx=0x7fe4ec008040 comp rx=0 tx=0).stop 2024-01-16T15:07:13.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 msgr2=0x7fe4fc1f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4038040 tx=0x7fe4e4004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:13.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 shutdown_connections 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4d406bbc0 0x7fe4d406e080 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4fc1439b0 0x7fe4fc0c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4fc1435d0 0x7fe4fc1f3520 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 --2- 172.21.15.43:0/2668027116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4fc0b3f70 0x7fe4fc1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.533+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 >> 172.21.15.43:0/2668027116 conn(0x7fe4fc0cec10 msgr2=0x7fe4fc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.534+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 shutdown_connections 2024-01-16T15:07:13.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.534+0000 7fe502400640 1 -- 172.21.15.43:0/2668027116 wait complete. 2024-01-16T15:07:13.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:07:13.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.774+0000 7fa2a7e00640 1 Processor -- start 2024-01-16T15:07:13.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.774+0000 7fa2a7e00640 1 -- start start 2024-01-16T15:07:13.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa2a80ad450 0x7fa2a8154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8070850 0x7fa2a8071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a7e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a80ce650 con 0x7fa2a8153ea0 2024-01-16T15:07:13.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a80ad8e0 con 0x7fa2a80ad450 2024-01-16T15:07:13.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a81547c0 con 0x7fa2a8070850 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a6a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa2a8153ea0 0x7fa2a815d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42516/0 (socket says 172.21.15.43:42516) 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a6a00640 1 -- 172.21.15.43:0/2966564278 learned_addr learned my addr 172.21.15.43:0/2966564278 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.776+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2966564278 >> v2:172.21.15.43:3300/0 conn(0x7fa2a80ad450 0x7fa2a8154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.777+0000 7fa2a7400640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8070850 0x7fa2a8071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.777+0000 7fa2a6000640 1 -- 172.21.15.43:0/2966564278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8070850 msgr2=0x7fa2a8071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.777+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8070850 0x7fa2a8071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.777+0000 7fa2a6000640 1 -- 172.21.15.43:0/2966564278 >> v1:172.21.15.43:6789/0 conn(0x7fa2a8153ea0 legacy=0x7fa2a815d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.777+0000 7fa2a6000640 1 -- 172.21.15.43:0/2966564278 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a815e0a0 con 0x7fa2a80ad450 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.778+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a80ad450 0x7fa2a8154280 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097d0 tx=0x7fa29002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9088e486d48583d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.778+0000 7fa2a5600640 1 -- 172.21.15.43:0/2966564278 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290031800 con 0x7fa2a80ad450 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.778+0000 7fa2a5600640 1 -- 172.21.15.43:0/2966564278 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa290031960 con 0x7fa2a80ad450 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.778+0000 7fa2a5600640 1 -- 172.21.15.43:0/2966564278 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290031cf0 con 0x7fa2a80ad450 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2966564278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a80ad450 msgr2=0x7fa2a8154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a80ad450 0x7fa2a8154280 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097d0 tx=0x7fa29002b8a0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2966564278 shutdown_connections 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8070850 0x7fa2a8071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2966564278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a80ad450 0x7fa2a8154280 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2966564278 >> 172.21.15.43:0/2966564278 conn(0x7fa2a80cec10 msgr2=0x7fa2a80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2966564278 shutdown_connections 2024-01-16T15:07:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.779+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2966564278 wait complete. 2024-01-16T15:07:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.782+0000 7fa2a7e00640 1 Processor -- start 2024-01-16T15:07:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.782+0000 7fa2a7e00640 1 -- start start 2024-01-16T15:07:13.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2a80ad450 0x7fa2a80ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8153ea0 0x7fa2a813d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a813daf0 con 0x7fa2a8070850 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80aef30 con 0x7fa2a80ad450 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.783+0000 7fa2a7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80af210 con 0x7fa2a8153ea0 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53258/0 (socket says 172.21.15.43:53258) 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 -- 172.21.15.43:0/2156455442 learned_addr learned my addr 172.21.15.43:0/2156455442 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2a80ad450 0x7fa2a80ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 -- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8153ea0 msgr2=0x7fa2a813d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8153ea0 0x7fa2a813d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 -- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2a80ad450 msgr2=0x7fa2a80ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2a80ad450 0x7fa2a80ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.784+0000 7fa2a6a00640 1 -- 172.21.15.43:0/2156455442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290009400 con 0x7fa2a8070850 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa2a6a00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa29800c3c0 tx=0x7fa29800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298005ce0 con 0x7fa2a8070850 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa298005e40 con 0x7fa2a8070850 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2980072d0 con 0x7fa2a8070850 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a80af4f0 con 0x7fa2a8070850 2024-01-16T15:07:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.785+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a80afab0 con 0x7fa2a8070850 2024-01-16T15:07:13.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.786+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a80ad130 con 0x7fa2a8070850 2024-01-16T15:07:13.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.787+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa2980074f0 con 0x7fa2a8070850 2024-01-16T15:07:13.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.789+0000 7fa29f400640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 0x7fa28006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.789+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fa2980879d0 con 0x7fa2a8070850 2024-01-16T15:07:13.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.789+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 0x7fa28006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.790+0000 7fa2a6000640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 0x7fa28006e080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa290036000 tx=0x7fa29003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.793+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2980542f0 con 0x7fa2a8070850 2024-01-16T15:07:13.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.992+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa2a804e840 con 0x7fa2a8070850 2024-01-16T15:07:13.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:13.999+0000 7fa29f400640 1 -- 172.21.15.43:0/2156455442 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa298057910 con 0x7fa2a8070850 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 msgr2=0x7fa28006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 0x7fa28006e080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa290036000 tx=0x7fa29003c040 comp rx=0 tx=0).stop 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 msgr2=0x7fa2a814f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa29800c3c0 tx=0x7fa29800c880 comp rx=0 tx=0).stop 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 shutdown_connections 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa28006bbc0 0x7fa28006e080 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2a80ad450 0x7fa2a80ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2a8153ea0 0x7fa2a813d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 --2- 172.21.15.43:0/2156455442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2a8070850 0x7fa2a814f910 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 >> 172.21.15.43:0/2156455442 conn(0x7fa2a80cec10 msgr2=0x7fa2a80b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 shutdown_connections 2024-01-16T15:07:13.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.002+0000 7fa2a7e00640 1 -- 172.21.15.43:0/2156455442 wait complete. 2024-01-16T15:07:13.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:07:14.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.235+0000 7f0410a00640 1 Processor -- start 2024-01-16T15:07:14.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.235+0000 7f0410a00640 1 -- start start 2024-01-16T15:07:14.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f0410a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f0410a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f0410a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040815c360 con 0x7f04080ad720 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f0410a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04080b41a0 con 0x7f04080abff0 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f0410a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040806de40 con 0x7f040806d520 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f040ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f040e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f040e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:52090/0 (socket says 172.21.15.43:52090) 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f040e000640 1 -- 172.21.15.43:0/1593506599 learned_addr learned my addr 172.21.15.43:0/1593506599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.237+0000 7f040ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:39262/0 (socket says 172.21.15.43:39262) 2024-01-16T15:07:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.238+0000 7f040e000640 1 -- 172.21.15.43:0/1593506599 >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 msgr2=0x7f040815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.238+0000 7f040e000640 1 --2- 172.21.15.43:0/1593506599 >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.238+0000 7f040e000640 1 -- 172.21.15.43:0/1593506599 >> v1:172.21.15.69:6789/0 conn(0x7f04080ad720 legacy=0x7f040815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.238+0000 7f040e000640 1 -- 172.21.15.43:0/1593506599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040815ba30 con 0x7f040806d520 2024-01-16T15:07:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.238+0000 7f040ea00640 1 --2- 172.21.15.43:0/1593506599 >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.240+0000 7f040e000640 1 --2- 172.21.15.43:0/1593506599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0400009e50 tx=0x7f040002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82367c70856fdb21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.240+0000 7f040cc00640 1 -- 172.21.15.43:0/1593506599 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040002cb00 con 0x7f040806d520 2024-01-16T15:07:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.240+0000 7f040cc00640 1 -- 172.21.15.43:0/1593506599 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040002cc60 con 0x7f040806d520 2024-01-16T15:07:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.240+0000 7f040cc00640 1 -- 172.21.15.43:0/1593506599 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000343f0 con 0x7f040806d520 2024-01-16T15:07:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 -- 172.21.15.43:0/1593506599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 msgr2=0x7f040806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 --2- 172.21.15.43:0/1593506599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0400009e50 tx=0x7f040002bf60 comp rx=0 tx=0).stop 2024-01-16T15:07:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 -- 172.21.15.43:0/1593506599 shutdown_connections 2024-01-16T15:07:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 --2- 172.21.15.43:0/1593506599 >> v2:172.21.15.69:3300/0 conn(0x7f04080abff0 0x7f040815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 --2- 172.21.15.43:0/1593506599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f040806d520 0x7f040806d900 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.241+0000 7f0410a00640 1 -- 172.21.15.43:0/1593506599 >> 172.21.15.43:0/1593506599 conn(0x7f04080b0d80 msgr2=0x7f04080b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.242+0000 7f0410a00640 1 -- 172.21.15.43:0/1593506599 shutdown_connections 2024-01-16T15:07:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.242+0000 7f0410a00640 1 -- 172.21.15.43:0/1593506599 wait complete. 2024-01-16T15:07:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.244+0000 7f0410a00640 1 Processor -- start 2024-01-16T15:07:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.244+0000 7f0410a00640 1 -- start start 2024-01-16T15:07:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f040806d520 0x7f040809cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04080abff0 0x7f04080cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408154ff0 con 0x7f04080ad720 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04080c4190 con 0x7f040806d520 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.245+0000 7f0410a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04080cc910 con 0x7f04080abff0 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53262/0 (socket says 172.21.15.43:53262) 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 -- 172.21.15.43:0/1291181825 learned_addr learned my addr 172.21.15.43:0/1291181825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040e000640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f040806d520 0x7f040809cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040d600640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04080abff0 0x7f04080cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 -- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04080abff0 msgr2=0x7f04080cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04080abff0 0x7f04080cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 -- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f040806d520 msgr2=0x7f040809cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f040806d520 0x7f040809cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 -- 172.21.15.43:0/1291181825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8009400 con 0x7f04080ad720 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.246+0000 7f040ea00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e930 tx=0x7f03f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.247+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80096e0 con 0x7f04080ad720 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.247+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f800ca00 con 0x7f04080ad720 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.247+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f800cd90 con 0x7f04080ad720 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.247+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0400009b30 con 0x7f04080ad720 2024-01-16T15:07:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.247+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04080cd1b0 con 0x7f04080ad720 2024-01-16T15:07:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.248+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04080ccd10 con 0x7f04080ad720 2024-01-16T15:07:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.249+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f03f8013470 con 0x7f04080ad720 2024-01-16T15:07:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.250+0000 7f0406000640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 0x7f03e006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.250+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f03f80889c0 con 0x7f04080ad720 2024-01-16T15:07:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.251+0000 7f040e000640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 0x7f03e006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.251+0000 7f040e000640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 0x7f03e006e080 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0400035000 tx=0x7f0400030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.255+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03f8055060 con 0x7f04080ad720 2024-01-16T15:07:14.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.517+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f0408074770 con 0x7f04080ad720 2024-01-16T15:07:14.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.908+0000 7f0406000640 1 -- 172.21.15.43:0/1291181825 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v366) v1 ==== 154+0+0 (secure 0 0 0) 0x7f03f80170e0 con 0x7f04080ad720 2024-01-16T15:07:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.911+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 msgr2=0x7f03e006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.911+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 0x7f03e006e080 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0400035000 tx=0x7f0400030000 comp rx=0 tx=0).stop 2024-01-16T15:07:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.911+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 msgr2=0x7f0408157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.911+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e930 tx=0x7f03f800edf0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 shutdown_connections 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f03e006bbc0 0x7f03e006e080 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04080ad720 0x7f0408157a10 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04080abff0 0x7f04080cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 --2- 172.21.15.43:0/1291181825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f040806d520 0x7f040809cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 >> 172.21.15.43:0/1291181825 conn(0x7f04080b0d80 msgr2=0x7f04080ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 shutdown_connections 2024-01-16T15:07:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:14.912+0000 7f0410a00640 1 -- 172.21.15.43:0/1291181825 wait complete. 2024-01-16T15:07:14.912 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:07:14.912 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:07:14.912 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:07:14.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:15.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.141+0000 7f9399600640 1 Processor -- start 2024-01-16T15:07:15.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.141+0000 7f9399600640 1 -- start start 2024-01-16T15:07:15.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9399600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93940ad720 0x7f93940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9399600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93940abff0 0x7f93940b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9399600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939415c360 con 0x7f939406d520 2024-01-16T15:07:15.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9399600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940b41a0 con 0x7f93940abff0 2024-01-16T15:07:15.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9399600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939415ba30 con 0x7f93940ad720 2024-01-16T15:07:15.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9393400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93940abff0 0x7f93940b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9392a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f939406d520 0x7f939406d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42528/0 (socket says 172.21.15.43:42528) 2024-01-16T15:07:15.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f9392a00640 1 -- 172.21.15.43:0/3430812629 learned_addr learned my addr 172.21.15.43:0/3430812629 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.143+0000 7f938be00640 1 --2- 172.21.15.43:0/3430812629 >> v2:172.21.15.69:3300/0 conn(0x7f93940ad720 0x7f93940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9392000640 1 -- 172.21.15.43:0/3430812629 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709910546 0 0) 0x7f939415c360 con 0x7f939406d520 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9392000640 1 -- 172.21.15.43:0/3430812629 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9374003520 con 0x7f939406d520 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9393400640 1 -- 172.21.15.43:0/3430812629 >> v2:172.21.15.69:3300/0 conn(0x7f93940ad720 msgr2=0x7f93940b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9393400640 1 --2- 172.21.15.43:0/3430812629 >> v2:172.21.15.69:3300/0 conn(0x7f93940ad720 0x7f93940b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9393400640 1 -- 172.21.15.43:0/3430812629 >> v1:172.21.15.43:6789/0 conn(0x7f939406d520 legacy=0x7f939406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.144+0000 7f9393400640 1 -- 172.21.15.43:0/3430812629 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939415bd10 con 0x7f93940abff0 2024-01-16T15:07:15.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.145+0000 7f9393400640 1 --2- 172.21.15.43:0/3430812629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940abff0 0x7f93940b25c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f939406e1e0 tx=0x7f938c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e744eb501e59b47c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.145+0000 7f9392000640 1 -- 172.21.15.43:0/3430812629 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c02cc70 con 0x7f93940abff0 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.145+0000 7f9392000640 1 -- 172.21.15.43:0/3430812629 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c02cdd0 con 0x7f93940abff0 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.145+0000 7f9392000640 1 -- 172.21.15.43:0/3430812629 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c034660 con 0x7f93940abff0 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 -- 172.21.15.43:0/3430812629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940abff0 msgr2=0x7f93940b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 --2- 172.21.15.43:0/3430812629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940abff0 0x7f93940b25c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f939406e1e0 tx=0x7f938c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 -- 172.21.15.43:0/3430812629 shutdown_connections 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 --2- 172.21.15.43:0/3430812629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940abff0 0x7f93940b25c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 --2- 172.21.15.43:0/3430812629 >> v2:172.21.15.69:3300/0 conn(0x7f93940ad720 0x7f93940b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 -- 172.21.15.43:0/3430812629 >> 172.21.15.43:0/3430812629 conn(0x7f93940b0d80 msgr2=0x7f93940b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 -- 172.21.15.43:0/3430812629 shutdown_connections 2024-01-16T15:07:15.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.146+0000 7f9399600640 1 -- 172.21.15.43:0/3430812629 wait complete. 2024-01-16T15:07:15.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.147+0000 7f9399600640 1 Processor -- start 2024-01-16T15:07:15.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 -- start start 2024-01-16T15:07:15.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f939406d520 0x7f93941f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93940abff0 0x7f93940cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940ce9c0 con 0x7f93940ad720 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394154e20 con 0x7f93940abff0 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.148+0000 7f9399600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9394154060 con 0x7f939406d520 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53288/0 (socket says 172.21.15.43:53288) 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 -- 172.21.15.43:0/3992322712 learned_addr learned my addr 172.21.15.43:0/3992322712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:15.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 -- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f939406d520 msgr2=0x7f93941f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f939406d520 0x7f93941f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 -- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93940abff0 msgr2=0x7f93940cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93940abff0 0x7f93940cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.149+0000 7f9393400640 1 -- 172.21.15.43:0/3992322712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c009a70 con 0x7f93940ad720 2024-01-16T15:07:15.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f9393400640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e80 tx=0x7f938c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c040650 con 0x7f93940ad720 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c0407b0 con 0x7f93940ad720 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9394154340 con 0x7f93940ad720 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c040b60 con 0x7f93940ad720 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.150+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9394154900 con 0x7f93940ad720 2024-01-16T15:07:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.151+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9394074770 con 0x7f93940ad720 2024-01-16T15:07:15.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.152+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f938c040cc0 con 0x7f93940ad720 2024-01-16T15:07:15.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.153+0000 7f938aa00640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 0x7f936c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.153+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f938c0ab200 con 0x7f93940ad720 2024-01-16T15:07:15.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.153+0000 7f9392a00640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 0x7f936c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.154+0000 7f9392a00640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 0x7f936c06e080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9384001610 tx=0x7f9384007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.157+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f938c077ac0 con 0x7f93940ad720 2024-01-16T15:07:15.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.352+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93940b3cd0 con 0x7f93940ad720 2024-01-16T15:07:15.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.353+0000 7f938aa00640 1 -- 172.21.15.43:0/3992322712 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12981 (secure 0 0 0) 0x7f938c0a4e60 con 0x7f93940ad720 2024-01-16T15:07:15.332 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:15.332 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":64,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:07.828633+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/4073013627":"2024-01-17T15:07:04.807372+0000","172.21.15.69:6832/4073013627":"2024-01-17T15:07:04.807372+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 msgr2=0x7f936c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 0x7f936c06e080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9384001610 tx=0x7f9384007930 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 msgr2=0x7f93940ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e80 tx=0x7f938c004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 shutdown_connections 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f936c06bbc0 0x7f936c06e080 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93940abff0 0x7f93940cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93940ad720 0x7f93940ce390 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 --2- 172.21.15.43:0/3992322712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f939406d520 0x7f93941f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.356+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 >> 172.21.15.43:0/3992322712 conn(0x7f93940b0d80 msgr2=0x7f93940acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.357+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 shutdown_connections 2024-01-16T15:07:15.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.357+0000 7f9399600640 1 -- 172.21.15.43:0/3992322712 wait complete. 2024-01-16T15:07:15.354 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:15.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.602+0000 7f54f3200640 1 Processor -- start 2024-01-16T15:07:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.602+0000 7f54f3200640 1 -- start start 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.603+0000 7f54f3200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f54f415efa0 0x7f54f4161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.603+0000 7f54f3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f54f40b3080 0x7f54f406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.603+0000 7f54f3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f406c9a0 con 0x7f54f406da40 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.603+0000 7f54f3200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f406cc80 con 0x7f54f415efa0 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.603+0000 7f54f3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f406cf60 con 0x7f54f40b3080 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f54f415efa0 0x7f54f4161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f54f415efa0 0x7f54f4161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53296/0 (socket says 172.21.15.43:53296) 2024-01-16T15:07:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 -- 172.21.15.43:0/2457754456 learned_addr learned my addr 172.21.15.43:0/2457754456 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 -- 172.21.15.43:0/2457754456 >> v2:172.21.15.69:3300/0 conn(0x7f54f40b3080 msgr2=0x7f54f406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 --2- 172.21.15.43:0/2457754456 >> v2:172.21.15.69:3300/0 conn(0x7f54f40b3080 0x7f54f406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 -- 172.21.15.43:0/2457754456 >> v1:172.21.15.69:6789/0 conn(0x7f54f406da40 legacy=0x7f54f40b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.604+0000 7f54f1e00640 1 -- 172.21.15.43:0/2457754456 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f406d240 con 0x7f54f415efa0 2024-01-16T15:07:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.605+0000 7f54f1e00640 1 --2- 172.21.15.43:0/2457754456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f415efa0 0x7f54f4161ac0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f54f40b4390 tx=0x7f54e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16038ce9ae22d1e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f1400640 1 -- 172.21.15.43:0/2457754456 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4031870 con 0x7f54f415efa0 2024-01-16T15:07:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f1400640 1 -- 172.21.15.43:0/2457754456 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e40319d0 con 0x7f54f415efa0 2024-01-16T15:07:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f1400640 1 -- 172.21.15.43:0/2457754456 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4031d60 con 0x7f54f415efa0 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 -- 172.21.15.43:0/2457754456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f415efa0 msgr2=0x7f54f4161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 --2- 172.21.15.43:0/2457754456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f415efa0 0x7f54f4161ac0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f54f40b4390 tx=0x7f54e402b910 comp rx=0 tx=0).stop 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 -- 172.21.15.43:0/2457754456 shutdown_connections 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 --2- 172.21.15.43:0/2457754456 >> v2:172.21.15.69:3300/0 conn(0x7f54f40b3080 0x7f54f406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 --2- 172.21.15.43:0/2457754456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f415efa0 0x7f54f4161ac0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 -- 172.21.15.43:0/2457754456 >> 172.21.15.43:0/2457754456 conn(0x7f54f40cec10 msgr2=0x7f54f40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 -- 172.21.15.43:0/2457754456 shutdown_connections 2024-01-16T15:07:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.606+0000 7f54f3200640 1 -- 172.21.15.43:0/2457754456 wait complete. 2024-01-16T15:07:15.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.608+0000 7f54f3200640 1 Processor -- start 2024-01-16T15:07:15.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.608+0000 7f54f3200640 1 -- start start 2024-01-16T15:07:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54f40b3080 0x7f54f415cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54f415efa0 0x7f54f4145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f41560d0 con 0x7f54f406da40 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f41f7af0 con 0x7f54f40b3080 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f4156390 con 0x7f54f415efa0 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53308/0 (socket says 172.21.15.43:53308) 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.609+0000 7f54f1e00640 1 -- 172.21.15.43:0/2246494228 learned_addr learned my addr 172.21.15.43:0/2246494228 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 -- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54f415efa0 msgr2=0x7f54f4145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54f415efa0 0x7f54f4145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 -- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54f40b3080 msgr2=0x7f54f415cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54f40b3080 0x7f54f415cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 -- 172.21.15.43:0/2246494228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e40094d0 con 0x7f54f406da40 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.610+0000 7f54f1e00640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f54e4000c00 tx=0x7f54e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.611+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4031840 con 0x7f54f406da40 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.611+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e40443f0 con 0x7f54f406da40 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.611+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e40447a0 con 0x7f54f406da40 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.611+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f4156670 con 0x7f54f406da40 2024-01-16T15:07:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.611+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f415d440 con 0x7f54f406da40 2024-01-16T15:07:15.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.612+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f415d630 con 0x7f54f406da40 2024-01-16T15:07:15.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.613+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f54e404d450 con 0x7f54f406da40 2024-01-16T15:07:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.614+0000 7f54ea000640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 0x7f54cc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.614+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f54e402d040 con 0x7f54f406da40 2024-01-16T15:07:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.614+0000 7f54ebe00640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 0x7f54cc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.615+0000 7f54ebe00640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 0x7f54cc06e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f54dc000f60 tx=0x7f54dc0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:15.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.618+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54e40dfe90 con 0x7f54f406da40 2024-01-16T15:07:15.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.800+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54f4156b90 con 0x7f54f406da40 2024-01-16T15:07:15.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.801+0000 7f54ea000640 1 -- 172.21.15.43:0/2246494228 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54e40774d0 con 0x7f54f406da40 2024-01-16T15:07:15.780 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:07:15.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 msgr2=0x7f54cc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 0x7f54cc06e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f54dc000f60 tx=0x7f54dc0079b0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 msgr2=0x7f54f41f7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f54e4000c00 tx=0x7f54e40046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 shutdown_connections 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f54cc06bbc0 0x7f54cc06e080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54f40b3080 0x7f54f415cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54f415efa0 0x7f54f4145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 --2- 172.21.15.43:0/2246494228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54f406da40 0x7f54f41f7f10 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 >> 172.21.15.43:0/2246494228 conn(0x7f54f40cec10 msgr2=0x7f54f4163fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 shutdown_connections 2024-01-16T15:07:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:15.804+0000 7f54f3200640 1 -- 172.21.15.43:0/2246494228 wait complete. 2024-01-16T15:07:15.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:07:16.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.042+0000 7fa216000640 1 Processor -- start 2024-01-16T15:07:16.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.042+0000 7fa216000640 1 -- start start 2024-01-16T15:07:16.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa216000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2100abff0 0x7fa2100b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa216000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21015c360 con 0x7fa21006d520 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa216000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21015ba30 con 0x7fa2100ad720 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa216000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2100b41a0 con 0x7fa2100abff0 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa20f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa21006d520 0x7fa21006d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42542/0 (socket says 172.21.15.43:42542) 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa20f400640 1 -- 172.21.15.43:0/1812018428 learned_addr learned my addr 172.21.15.43:0/1812018428 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.044+0000 7fa20fe00640 1 --2- 172.21.15.43:0/1812018428 >> v2:172.21.15.69:3300/0 conn(0x7fa2100abff0 0x7fa2100b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750159 0 0) 0x7fa21015c360 con 0x7fa21006d520 2024-01-16T15:07:16.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f0003520 con 0x7fa21006d520 2024-01-16T15:07:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353383374 0 0) 0x7fa1f0003520 con 0x7fa21006d520 2024-01-16T15:07:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 >> v2:172.21.15.69:3300/0 conn(0x7fa2100abff0 msgr2=0x7fa2100b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 --2- 172.21.15.43:0/1812018428 >> v2:172.21.15.69:3300/0 conn(0x7fa2100abff0 0x7fa2100b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 >> v1:172.21.15.69:6789/0 conn(0x7fa2100ad720 legacy=0x7fa2100b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21015bd10 con 0x7fa21006d520 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa200001670 con 0x7fa21006d520 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 >> v1:172.21.15.43:6789/0 conn(0x7fa21006d520 legacy=0x7fa21006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1f0003970 0x7fa1f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f00047c0 0x7fa1f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 0x7fa1f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f000bff0 con 0x7fa1f0005180 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f000c160 con 0x7fa1f00047c0 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f000c320 con 0x7fa1f0003970 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.045+0000 7fa20fe00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 0x7fa1f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20f400640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1f0003970 0x7fa1f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa206800640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f00047c0 0x7fa1f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20fe00640 1 -- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1f0003970 msgr2=0x7fa1f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20fe00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa1f0003970 0x7fa1f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20fe00640 1 -- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f00047c0 msgr2=0x7fa1f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20fe00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f00047c0 0x7fa1f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa20fe00640 1 -- 172.21.15.43:0/1812018428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0003520 con 0x7fa1f0005180 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.046+0000 7fa206800640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f00047c0 0x7fa1f0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.047+0000 7fa20fe00640 1 -- 172.21.15.43:0/1812018428 reap_dead start 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.047+0000 7fa20fe00640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 0x7fa1f000b980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa208009fc0 tx=0x7fa1f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3918902512dd578f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.047+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20802cc60 con 0x7fa1f0005180 2024-01-16T15:07:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.047+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20800f040 con 0x7fa1f0005180 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.047+0000 7fa20ea00640 1 -- 172.21.15.43:0/1812018428 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208003990 con 0x7fa1f0005180 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 -- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 msgr2=0x7fa1f000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 0x7fa1f000b980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa208009fc0 tx=0x7fa1f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 -- 172.21.15.43:0/1812018428 shutdown_connections 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 --2- 172.21.15.43:0/1812018428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1f0005180 0x7fa1f000b980 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 -- 172.21.15.43:0/1812018428 >> 172.21.15.43:0/1812018428 conn(0x7fa2100b0d80 msgr2=0x7fa2100b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 -- 172.21.15.43:0/1812018428 shutdown_connections 2024-01-16T15:07:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.048+0000 7fa216000640 1 -- 172.21.15.43:0/1812018428 wait complete. 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.050+0000 7fa216000640 1 Processor -- start 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.050+0000 7fa216000640 1 -- start start 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 0x7fa2101f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa210146d80 0x7fa210147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101f1c00 con 0x7fa210151e30 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa210152f20 con 0x7fa1f0005180 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.051+0000 7fa216000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100cc110 con 0x7fa210146d80 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53334/0 (socket says 172.21.15.43:53334) 2024-01-16T15:07:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 -- 172.21.15.43:0/2061513845 learned_addr learned my addr 172.21.15.43:0/2061513845 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa20fe00640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa210146d80 0x7fa210147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa20f400640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 0x7fa2101f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 -- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa210146d80 msgr2=0x7fa210147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa210146d80 0x7fa210147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 -- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 msgr2=0x7fa2101f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 0x7fa2101f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa206800640 1 -- 172.21.15.43:0/2061513845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210152980 con 0x7fa210151e30 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.052+0000 7fa20f400640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 0x7fa2101f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa206800640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800d770 tx=0x7fa1f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8005270 con 0x7fa210151e30 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f80053f0 con 0x7fa210151e30 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa210152c40 con 0x7fa210151e30 2024-01-16T15:07:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800b8f0 con 0x7fa210151e30 2024-01-16T15:07:16.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.053+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa210147740 con 0x7fa210151e30 2024-01-16T15:07:16.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.054+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210074770 con 0x7fa210151e30 2024-01-16T15:07:16.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.055+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa1f800ba50 con 0x7fa210151e30 2024-01-16T15:07:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.057+0000 7fa207e00640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 0x7fa1e806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.057+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fa1f8088370 con 0x7fa210151e30 2024-01-16T15:07:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.057+0000 7fa20f400640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 0x7fa1e806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.058+0000 7fa20f400640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 0x7fa1e806e080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa200001eb0 tx=0x7fa200000f50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.061+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1f8054c90 con 0x7fa210151e30 2024-01-16T15:07:16.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.332+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7fa2100cc250 con 0x7fa210151e30 2024-01-16T15:07:16.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.921+0000 7fa207e00640 1 -- 172.21.15.43:0/2061513845 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v368) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa1f8019310 con 0x7fa210151e30 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 msgr2=0x7fa1e806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 0x7fa1e806e080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa200001eb0 tx=0x7fa200000f50 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 msgr2=0x7fa210154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800d770 tx=0x7fa1f800dc30 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 shutdown_connections 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa1e806bbc0 0x7fa1e806e080 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa210146d80 0x7fa210147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210151e30 0x7fa210154790 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 --2- 172.21.15.43:0/2061513845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1f0005180 0x7fa2101f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.924+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 >> 172.21.15.43:0/2061513845 conn(0x7fa2100b0d80 msgr2=0x7fa2100acca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.925+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 shutdown_connections 2024-01-16T15:07:16.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:16.925+0000 7fa216000640 1 -- 172.21.15.43:0/2061513845 wait complete. 2024-01-16T15:07:16.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:07:17.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.169+0000 7fbc18800640 1 Processor -- start 2024-01-16T15:07:17.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.169+0000 7fbc18800640 1 -- start start 2024-01-16T15:07:17.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.170+0000 7fbc18800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbc10160bd0 0x7fbc100b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc18800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 0x7fbc10167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc18800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100b23e0 con 0x7fbc100b48d0 2024-01-16T15:07:17.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc18800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc100cdc70 con 0x7fbc10160bd0 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc18800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100b2870 con 0x7fbc100b3210 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbc10160bd0 0x7fbc100b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbc100b48d0 0x7fbc100b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42558/0 (socket says 172.21.15.43:42558) 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15400640 1 -- 172.21.15.43:0/3881333244 learned_addr learned my addr 172.21.15.43:0/3881333244 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc16800640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 0x7fbc10167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15e00640 1 -- 172.21.15.43:0/3881333244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 msgr2=0x7fbc10167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15e00640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 0x7fbc10167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15e00640 1 -- 172.21.15.43:0/3881333244 >> v1:172.21.15.43:6789/0 conn(0x7fbc100b48d0 legacy=0x7fbc100b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc15e00640 1 -- 172.21.15.43:0/3881333244 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc100d41b0 con 0x7fbc10160bd0 2024-01-16T15:07:17.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.171+0000 7fbc16800640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 0x7fbc10167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:17.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.173+0000 7fbc15e00640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100b1ea0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80097a0 tx=0x7fbbf802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935b99d1f8334f66 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.173+0000 7fbc14a00640 1 -- 172.21.15.43:0/3881333244 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8031780 con 0x7fbc10160bd0 2024-01-16T15:07:17.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.173+0000 7fbc14a00640 1 -- 172.21.15.43:0/3881333244 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf80318e0 con 0x7fbc10160bd0 2024-01-16T15:07:17.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.173+0000 7fbc14a00640 1 -- 172.21.15.43:0/3881333244 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8031c70 con 0x7fbc10160bd0 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 -- 172.21.15.43:0/3881333244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 msgr2=0x7fbc100b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100b1ea0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80097a0 tx=0x7fbbf802b820 comp rx=0 tx=0).stop 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 -- 172.21.15.43:0/3881333244 shutdown_connections 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b3210 0x7fbc10167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 --2- 172.21.15.43:0/3881333244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100b1ea0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 -- 172.21.15.43:0/3881333244 >> 172.21.15.43:0/3881333244 conn(0x7fbc1009c5a0 msgr2=0x7fbc100d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 -- 172.21.15.43:0/3881333244 shutdown_connections 2024-01-16T15:07:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.174+0000 7fbc18800640 1 -- 172.21.15.43:0/3881333244 wait complete. 2024-01-16T15:07:17.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.176+0000 7fbc18800640 1 Processor -- start 2024-01-16T15:07:17.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.176+0000 7fbc18800640 1 -- start start 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc100b3210 0x7fbc1015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b48d0 0x7fbc100b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100c7fd0 con 0x7fbc10160bd0 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100b76e0 con 0x7fbc100b3210 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc18800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100b7950 con 0x7fbc100b48d0 2024-01-16T15:07:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc16800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc16800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53358/0 (socket says 172.21.15.43:53358) 2024-01-16T15:07:17.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc16800640 1 -- 172.21.15.43:0/2340061966 learned_addr learned my addr 172.21.15.43:0/2340061966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:17.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc15e00640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc100b3210 0x7fbc1015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.177+0000 7fbc15400640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b48d0 0x7fbc100b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 -- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b48d0 msgr2=0x7fbc100b8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b48d0 0x7fbc100b8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 -- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc100b3210 msgr2=0x7fbc1015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc100b3210 0x7fbc1015d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 -- 172.21.15.43:0/2340061966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8009400 con 0x7fbc10160bd0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc16800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800c820 tx=0x7fbc0800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08007a30 con 0x7fbc10160bd0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc08007b90 con 0x7fbc10160bd0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0800ce10 con 0x7fbc10160bd0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc100b7c30 con 0x7fbc10160bd0 2024-01-16T15:07:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.178+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc100caac0 con 0x7fbc10160bd0 2024-01-16T15:07:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.180+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc100c8180 con 0x7fbc10160bd0 2024-01-16T15:07:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.180+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fbc080113f0 con 0x7fbc10160bd0 2024-01-16T15:07:17.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.182+0000 7fbc0e000640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 0x7fbbe806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:17.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.182+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fbc080885f0 con 0x7fbc10160bd0 2024-01-16T15:07:17.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.182+0000 7fbc15e00640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 0x7fbbe806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.183+0000 7fbc15e00640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 0x7fbbe806e080 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8036000 tx=0x7fbbf803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.186+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc100c8180 con 0x7fbc10160bd0 2024-01-16T15:07:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.382+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbc100c8180 con 0x7fbc10160bd0 2024-01-16T15:07:17.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.389+0000 7fbc0e000640 1 -- 172.21.15.43:0/2340061966 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbc100c8180 con 0x7fbc10160bd0 2024-01-16T15:07:17.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 msgr2=0x7fbbe806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 0x7fbbe806e080 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8036000 tx=0x7fbbf803d040 comp rx=0 tx=0).stop 2024-01-16T15:07:17.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 msgr2=0x7fbc100ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:17.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800c820 tx=0x7fbc0800cce0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 shutdown_connections 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbbe806bbc0 0x7fbbe806e080 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc10160bd0 0x7fbc100ca580 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc100b48d0 0x7fbc100b8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 --2- 172.21.15.43:0/2340061966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc100b3210 0x7fbc1015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 >> 172.21.15.43:0/2340061966 conn(0x7fbc1009c5a0 msgr2=0x7fbc100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 shutdown_connections 2024-01-16T15:07:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:17.392+0000 7fbc18800640 1 -- 172.21.15.43:0/2340061966 wait complete. 2024-01-16T15:07:17.391 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:07:17.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:17.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:17.633 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:07:17.633 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:17.633 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:17.693 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:17.693 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:07:17.693 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:17.750 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:17.750 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:17.750 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:17.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:17.810 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:07:17.810 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:17.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:17.867 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:07:17.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.111+0000 7f7c3d000640 1 Processor -- start 2024-01-16T15:07:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.111+0000 7f7c3d000640 1 -- start start 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c3d000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c380ad720 0x7f7c380b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c3d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7c380abff0 0x7f7c380b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c3d000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3815c360 con 0x7f7c3806d520 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c3d000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c380b41a0 con 0x7f7c380abff0 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c3d000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3815ba30 con 0x7f7c380ad720 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c36800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7c380abff0 0x7f7c380b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c36800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7c380abff0 0x7f7c380b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53364/0 (socket says 172.21.15.43:53364) 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c36800640 1 -- 172.21.15.43:0/2724846858 learned_addr learned my addr 172.21.15.43:0/2724846858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.113+0000 7f7c35e00640 1 --1- 172.21.15.43:0/2724846858 >> v1:172.21.15.43:6789/0 conn(0x7f7c3806d520 0x7f7c3806d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42574/0 (socket says 172.21.15.43:42574) 2024-01-16T15:07:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.114+0000 7f7c35400640 1 --2- 172.21.15.43:0/2724846858 >> v2:172.21.15.69:3300/0 conn(0x7f7c380ad720 0x7f7c380b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.114+0000 7f7c36800640 1 -- 172.21.15.43:0/2724846858 >> v2:172.21.15.69:3300/0 conn(0x7f7c380ad720 msgr2=0x7f7c380b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.114+0000 7f7c36800640 1 --2- 172.21.15.43:0/2724846858 >> v2:172.21.15.69:3300/0 conn(0x7f7c380ad720 0x7f7c380b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.114+0000 7f7c36800640 1 -- 172.21.15.43:0/2724846858 >> v1:172.21.15.43:6789/0 conn(0x7f7c3806d520 legacy=0x7f7c3806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.114+0000 7f7c36800640 1 -- 172.21.15.43:0/2724846858 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3815bd10 con 0x7f7c380abff0 2024-01-16T15:07:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.115+0000 7f7c36800640 1 --2- 172.21.15.43:0/2724846858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380abff0 0x7f7c380b25c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7c3806e1e0 tx=0x7f7c3002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91905a7af23138f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.115+0000 7f7c34a00640 1 -- 172.21.15.43:0/2724846858 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30030860 con 0x7f7c380abff0 2024-01-16T15:07:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c34a00640 1 -- 172.21.15.43:0/2724846858 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c300309c0 con 0x7f7c380abff0 2024-01-16T15:07:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c34a00640 1 -- 172.21.15.43:0/2724846858 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30030d50 con 0x7f7c380abff0 2024-01-16T15:07:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 -- 172.21.15.43:0/2724846858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380abff0 msgr2=0x7f7c380b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 --2- 172.21.15.43:0/2724846858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380abff0 0x7f7c380b25c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7c3806e1e0 tx=0x7f7c3002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 -- 172.21.15.43:0/2724846858 shutdown_connections 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 --2- 172.21.15.43:0/2724846858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380abff0 0x7f7c380b25c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 --2- 172.21.15.43:0/2724846858 >> v2:172.21.15.69:3300/0 conn(0x7f7c380ad720 0x7f7c380b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.116+0000 7f7c3d000640 1 -- 172.21.15.43:0/2724846858 >> 172.21.15.43:0/2724846858 conn(0x7f7c380b0d80 msgr2=0x7f7c380b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.117+0000 7f7c3d000640 1 -- 172.21.15.43:0/2724846858 shutdown_connections 2024-01-16T15:07:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.117+0000 7f7c3d000640 1 -- 172.21.15.43:0/2724846858 wait complete. 2024-01-16T15:07:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.118+0000 7f7c3d000640 1 Processor -- start 2024-01-16T15:07:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.118+0000 7f7c3d000640 1 -- start start 2024-01-16T15:07:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c3806d520 0x7f7c381578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380abff0 0x7f7c3815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38157e10 con 0x7f7c380ad720 2024-01-16T15:07:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38155de0 con 0x7f7c380abff0 2024-01-16T15:07:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.119+0000 7f7c3d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c381560c0 con 0x7f7c3806d520 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53378/0 (socket says 172.21.15.43:53378) 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 -- 172.21.15.43:0/8482759 learned_addr learned my addr 172.21.15.43:0/8482759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c35e00640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c3806d520 0x7f7c381578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c35400640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380abff0 0x7f7c3815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 -- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c3806d520 msgr2=0x7f7c381578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c3806d520 0x7f7c381578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 -- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380abff0 msgr2=0x7f7c3815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380abff0 0x7f7c3815dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.120+0000 7f7c36800640 1 -- 172.21.15.43:0/8482759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300094d0 con 0x7f7c380ad720 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c36800640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7c300090b0 tx=0x7f7c30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30039c90 con 0x7f7c380ad720 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c30039df0 con 0x7f7c380ad720 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30043640 con 0x7f7c380ad720 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c380cc260 con 0x7f7c380ad720 2024-01-16T15:07:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.121+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c381542d0 con 0x7f7c380ad720 2024-01-16T15:07:18.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.122+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c38074770 con 0x7f7c380ad720 2024-01-16T15:07:18.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.123+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7c300437a0 con 0x7f7c380ad720 2024-01-16T15:07:18.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.124+0000 7f7c2e000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 0x7f7c1006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.124+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7c3003e070 con 0x7f7c380ad720 2024-01-16T15:07:18.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.125+0000 7f7c35e00640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 0x7f7c1006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.125+0000 7f7c35e00640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 0x7f7c1006e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7c28000f60 tx=0x7f7c28007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.130+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c30077ed0 con 0x7f7c380ad720 2024-01-16T15:07:18.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.359+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c3806dbe0 con 0x7f7c380ad720 2024-01-16T15:07:18.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.360+0000 7f7c2e000640 1 -- 172.21.15.43:0/8482759 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+3690 (secure 0 0 0) 0x7f7c30034070 con 0x7f7c380ad720 2024-01-16T15:07:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:18.339 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5180,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/986973826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":986973826},{"type":"v1","addr":"172.21.15.69:6835","nonce":986973826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5644,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/607340672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":607340672},{"type":"v1","addr":"172.21.15.43:6836","nonce":607340672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/154390899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":154390899},{"type":"v1","addr":"172.21.15.43:6837","nonce":154390899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6016,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2149841884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2149841884},{"type":"v1","addr":"172.21.15.69:6834","nonce":2149841884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-16T15:07:18.339 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 39 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.362+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 msgr2=0x7f7c1006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 0x7f7c1006e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7c28000f60 tx=0x7f7c28007c80 comp rx=0 tx=0).stop 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 msgr2=0x7f7c38098470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7c300090b0 tx=0x7f7c30004730 comp rx=0 tx=0).stop 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 shutdown_connections 2024-01-16T15:07:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c1006bbc0 0x7f7c1006e080 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380abff0 0x7f7c3815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c380ad720 0x7f7c38098470 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 --2- 172.21.15.43:0/8482759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c3806d520 0x7f7c381578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 >> 172.21.15.43:0/8482759 conn(0x7f7c380b0d80 msgr2=0x7f7c380cb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 shutdown_connections 2024-01-16T15:07:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.363+0000 7f7c3d000640 1 -- 172.21.15.43:0/8482759 wait complete. 2024-01-16T15:07:18.363 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:07:18.363 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:07:18.363 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:07:18.364 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:18.364 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:07:18.364 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:07:18.364 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:07:18.364 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:18.364 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:07:18.364 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:07:18.365 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:07:18.365 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:18.365 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:07:18.365 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:07:18.365 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:07:18.365 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:18.387 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:07:18.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:07:18.390 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:07:18.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:07:18.391 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:07:18.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:07:18.392 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:07:18.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:07:18.437 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:07:18.437 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:18.439 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:07:18.439 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:18.443 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:07:18.443 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:18.445 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:07:18.445 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:18.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:07:18.594 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:07:18.594 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:07:18.602 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:07:18.603 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:07:18.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.667+0000 7fe434e00640 1 Processor -- start 2024-01-16T15:07:18.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.667+0000 7fe434e00640 1 -- start start 2024-01-16T15:07:18.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.668+0000 7fe434e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4300cf710 0x7fe4300cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.668+0000 7fe434e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4300d1670 0x7fe4300d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.668+0000 7fe434e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4300d04b0 con 0x7fe43015d5d0 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.668+0000 7fe434e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300ce4e0 con 0x7fe4300d1670 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.668+0000 7fe434e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300cf010 con 0x7fe4300cf710 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe43015d5d0 0x7fe4300ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42590/0 (socket says 172.21.15.43:42590) 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42f400640 1 -- 172.21.15.43:0/4045962162 learned_addr learned my addr 172.21.15.43:0/4045962162 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42fe00640 1 --2- 172.21.15.43:0/4045962162 >> v2:172.21.15.43:3300/0 conn(0x7fe4300d1670 0x7fe4300d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42ea00640 1 --2- 172.21.15.43:0/4045962162 >> v2:172.21.15.69:3300/0 conn(0x7fe4300cf710 0x7fe4300cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42e000640 1 -- 172.21.15.43:0/4045962162 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516195311 0 0) 0x7fe4300d04b0 con 0x7fe43015d5d0 2024-01-16T15:07:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42fe00640 1 -- 172.21.15.43:0/4045962162 >> v2:172.21.15.69:3300/0 conn(0x7fe4300cf710 msgr2=0x7fe4300cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42fe00640 1 --2- 172.21.15.43:0/4045962162 >> v2:172.21.15.69:3300/0 conn(0x7fe4300cf710 0x7fe4300cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42fe00640 1 -- 172.21.15.43:0/4045962162 >> v1:172.21.15.43:6789/0 conn(0x7fe43015d5d0 legacy=0x7fe4300ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.669+0000 7fe42fe00640 1 -- 172.21.15.43:0/4045962162 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300cdb80 con 0x7fe4300d1670 2024-01-16T15:07:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.670+0000 7fe42fe00640 1 --2- 172.21.15.43:0/4045962162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300d1a80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe4300cf370 tx=0x7fe42802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf8be897f6a2dd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe42e000640 1 -- 172.21.15.43:0/4045962162 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42802cbf0 con 0x7fe4300d1670 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe42e000640 1 -- 172.21.15.43:0/4045962162 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42802cd50 con 0x7fe4300d1670 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe42e000640 1 -- 172.21.15.43:0/4045962162 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428034660 con 0x7fe4300d1670 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 -- 172.21.15.43:0/4045962162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 msgr2=0x7fe4300d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 --2- 172.21.15.43:0/4045962162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300d1a80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe4300cf370 tx=0x7fe42802be40 comp rx=0 tx=0).stop 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 -- 172.21.15.43:0/4045962162 shutdown_connections 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 --2- 172.21.15.43:0/4045962162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300d1a80 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 --2- 172.21.15.43:0/4045962162 >> v2:172.21.15.69:3300/0 conn(0x7fe4300cf710 0x7fe4300cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 -- 172.21.15.43:0/4045962162 >> 172.21.15.43:0/4045962162 conn(0x7fe43009c3d0 msgr2=0x7fe4300d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 -- 172.21.15.43:0/4045962162 shutdown_connections 2024-01-16T15:07:18.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.671+0000 7fe434e00640 1 -- 172.21.15.43:0/4045962162 wait complete. 2024-01-16T15:07:18.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.673+0000 7fe434e00640 1 Processor -- start 2024-01-16T15:07:18.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.673+0000 7fe434e00640 1 -- start start 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4300cf710 0x7fe430154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe43015d5d0 0x7fe43015ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c7670 con 0x7fe4300d1670 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c7950 con 0x7fe43015d5d0 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe434e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c7c30 con 0x7fe4300cf710 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.674+0000 7fe42ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53426/0 (socket says 172.21.15.43:53426) 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 -- 172.21.15.43:0/2066052464 learned_addr learned my addr 172.21.15.43:0/2066052464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42f400640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4300cf710 0x7fe430154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42fe00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe43015d5d0 0x7fe43015ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 -- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4300cf710 msgr2=0x7fe430154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4300cf710 0x7fe430154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 -- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe43015d5d0 msgr2=0x7fe43015ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe43015d5d0 0x7fe43015ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 -- 172.21.15.43:0/2066052464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428009a70 con 0x7fe4300d1670 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.675+0000 7fe42ea00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe41800e870 tx=0x7fe41800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.676+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4180094f0 con 0x7fe4300d1670 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.676+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41800c940 con 0x7fe4300d1670 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.676+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41800ccf0 con 0x7fe4300d1670 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.676+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4300c7f10 con 0x7fe4300d1670 2024-01-16T15:07:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.676+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe430154bb0 con 0x7fe4300d1670 2024-01-16T15:07:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.677+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43004e840 con 0x7fe4300d1670 2024-01-16T15:07:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.678+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe418013470 con 0x7fe4300d1670 2024-01-16T15:07:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.680+0000 7fe427400640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 0x7fe40806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.680+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe418088710 con 0x7fe4300d1670 2024-01-16T15:07:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.680+0000 7fe42f400640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 0x7fe40806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.681+0000 7fe42f400640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 0x7fe40806e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe420000f60 tx=0x7fe420007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.685+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe418054e90 con 0x7fe4300d1670 2024-01-16T15:07:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.892+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe4300678d0 con 0x7fe4300d1670 2024-01-16T15:07:18.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.951+0000 7fe427400640 1 -- 172.21.15.43:0/2066052464 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v66) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe418017080 con 0x7fe4300d1670 2024-01-16T15:07:18.930 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.954+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 msgr2=0x7fe40806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.954+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 0x7fe40806e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe420000f60 tx=0x7fe420007d50 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.954+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 msgr2=0x7fe4300c8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.954+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe41800e870 tx=0x7fe41800ed30 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.954+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 shutdown_connections 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe40806bbc0 0x7fe40806e080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe43015d5d0 0x7fe43015ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4300d1670 0x7fe4300c8710 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 --2- 172.21.15.43:0/2066052464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4300cf710 0x7fe430154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 >> 172.21.15.43:0/2066052464 conn(0x7fe43009c3d0 msgr2=0x7fe4300b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 shutdown_connections 2024-01-16T15:07:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:18.955+0000 7fe434e00640 1 -- 172.21.15.43:0/2066052464 wait complete. 2024-01-16T15:07:18.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:07:19.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.197+0000 7f89ba200640 1 Processor -- start 2024-01-16T15:07:19.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.197+0000 7f89ba200640 1 -- start start 2024-01-16T15:07:19.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.199+0000 7f89ba200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89b415d430 0x7f89b40d3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.199+0000 7f89ba200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 0x7f89b40d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.199+0000 7f89ba200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b40d2020 con 0x7f89b40cfaa0 2024-01-16T15:07:19.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.199+0000 7f89ba200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b40d09d0 con 0x7f89b415d430 2024-01-16T15:07:19.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.199+0000 7f89ba200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b40d1030 con 0x7f89b40d1690 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.200+0000 7f89abe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f89b40cfaa0 0x7f89b41615e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42606/0 (socket says 172.21.15.43:42606) 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.200+0000 7f89abe00640 1 -- 172.21.15.43:0/2239004963 learned_addr learned my addr 172.21.15.43:0/2239004963 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.200+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2239004963 >> v2:172.21.15.69:3300/0 conn(0x7f89b415d430 0x7f89b40d3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.200+0000 7f89b9800640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 0x7f89b40d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.200+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481471336 0 0) 0x7f899c001110 con 0x7f89b40cfaa0 2024-01-16T15:07:19.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8994003520 con 0x7f89b40cfaa0 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98542749 0 0) 0x7f8994003520 con 0x7f89b40cfaa0 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 msgr2=0x7f89b40d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 0x7f89b40d1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 >> v2:172.21.15.69:3300/0 conn(0x7f89b415d430 msgr2=0x7f89b40d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 --2- 172.21.15.43:0/2239004963 >> v2:172.21.15.69:3300/0 conn(0x7f89b415d430 0x7f89b40d3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b415d810 con 0x7f89b40cfaa0 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f899c001820 con 0x7f89b40cfaa0 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 >> v1:172.21.15.43:6789/0 conn(0x7f89b40cfaa0 legacy=0x7f89b41615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f8994003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89940047c0 0x7f8994004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 0x7f8994005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b40d09d0 con 0x7f8994003a10 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b40d1030 con 0x7f89940047c0 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899400bab0 con 0x7f89b40d1690 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.201+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f8994003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.202+0000 7f89b8e00640 1 -- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 msgr2=0x7f8994005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.202+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b40d1690 0x7f8994005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.202+0000 7f89b8e00640 1 -- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89940047c0 msgr2=0x7f8994004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.202+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89940047c0 0x7f8994004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.202+0000 7f89b8e00640 1 -- 172.21.15.43:0/2239004963 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994003520 con 0x7f8994003a10 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.203+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f8994003e30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009a90 tx=0x7f89940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecab7f8c2c11e68 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.203+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a402c6b0 con 0x7f8994003a10 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.203+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a402c810 con 0x7f8994003a10 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.203+0000 7f89b3e00640 1 -- 172.21.15.43:0/2239004963 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a402cba0 con 0x7f8994003a10 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 -- 172.21.15.43:0/2239004963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 msgr2=0x7f8994003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 --2- 172.21.15.43:0/2239004963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f8994003e30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009a90 tx=0x7f89940057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89b9800640 1 -- 172.21.15.43:0/2239004963 reap_dead start 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 -- 172.21.15.43:0/2239004963 shutdown_connections 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 -- 172.21.15.43:0/2239004963 >> 172.21.15.43:0/2239004963 conn(0x7f89b408cb00 msgr2=0x7f89b409c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 -- 172.21.15.43:0/2239004963 shutdown_connections 2024-01-16T15:07:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.204+0000 7f89ba200640 1 -- 172.21.15.43:0/2239004963 wait complete. 2024-01-16T15:07:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.206+0000 7f89ba200640 1 Processor -- start 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.206+0000 7f89ba200640 1 -- start start 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b41600f0 0x7f89b4157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 0x7f89b415b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b41f4ba0 con 0x7f8994003a10 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b415b7a0 con 0x7f89b415adb0 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89ba200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b40b5c90 con 0x7f89b41600f0 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89b8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89b8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53454/0 (socket says 172.21.15.43:53454) 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.207+0000 7f89b8e00640 1 -- 172.21.15.43:0/2045870755 learned_addr learned my addr 172.21.15.43:0/2045870755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89abe00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b41600f0 0x7f89b4157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b9800640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 0x7f89b415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 -- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b41600f0 msgr2=0x7f89b4157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b41600f0 0x7f89b4157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 -- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 msgr2=0x7f89b415b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 0x7f89b415b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 -- 172.21.15.43:0/2045870755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b41f3f60 con 0x7f8994003a10 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b9800640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 0x7f89b415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.208+0000 7f89b8e00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f89a402c250 tx=0x7f89a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.209+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4003810 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.209+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a4003970 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.209+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4003ce0 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.209+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b41f4200 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.209+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b41f47c0 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.210+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b4067960 con 0x7f8994003a10 2024-01-16T15:07:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.211+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f89a4040450 con 0x7f8994003a10 2024-01-16T15:07:19.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.212+0000 7f89b1600640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 0x7f898c06e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.212+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f89a40aa2e0 con 0x7f8994003a10 2024-01-16T15:07:19.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.212+0000 7f89abe00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 0x7f898c06e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.213+0000 7f89abe00640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 0x7f898c06e0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f89b40b8fa0 tx=0x7f899c001c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.219+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89a4076d00 con 0x7f8994003a10 2024-01-16T15:07:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.482+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f89b404e840 con 0x7f8994003a10 2024-01-16T15:07:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.483+0000 7f89b1600640 1 -- 172.21.15.43:0/2045870755 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f89a40044e0 con 0x7f8994003a10 2024-01-16T15:07:19.461 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.462 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:07:19.463 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:07:19.464 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:07:19.465 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.467 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:07:19.468 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.470 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:07:19.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:19.473 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:07:19.474 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 msgr2=0x7f898c06e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 0x7f898c06e0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f89b40b8fa0 tx=0x7f899c001c90 comp rx=0 tx=0).stop 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 msgr2=0x7f89b415e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f89a402c250 tx=0x7f89a4004730 comp rx=0 tx=0).stop 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 shutdown_connections 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f898c06bc10 0x7f898c06e0d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b415adb0 0x7f89b415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b41600f0 0x7f89b4157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 --2- 172.21.15.43:0/2045870755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8994003a10 0x7f89b415e840 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.486+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 >> 172.21.15.43:0/2045870755 conn(0x7f89b408cb00 msgr2=0x7f89b406b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:19.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.487+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 shutdown_connections 2024-01-16T15:07:19.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.487+0000 7f89ba200640 1 -- 172.21.15.43:0/2045870755 wait complete. 2024-01-16T15:07:19.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:19.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.730+0000 7f9012600640 1 Processor -- start 2024-01-16T15:07:19.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.730+0000 7f9012600640 1 -- start start 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f9012600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f900c06e4c0 0x7f900c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f9012600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0ad6b0 con 0x7f900c153f10 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f9012600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0b2860 con 0x7f900c15b050 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f9012600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900c0b2b40 con 0x7f900c06e4c0 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f900b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f900c153f10 0x7f900c0b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42608/0 (socket says 172.21.15.43:42608) 2024-01-16T15:07:19.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f900b400640 1 -- 172.21.15.43:0/209953267 learned_addr learned my addr 172.21.15.43:0/209953267 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310885208 0 0) 0x7f900c0ad6b0 con 0x7f900c153f10 2024-01-16T15:07:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.732+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec003520 con 0x7f900c153f10 2024-01-16T15:07:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900be00640 1 --2- 172.21.15.43:0/209953267 >> v2:172.21.15.69:3300/0 conn(0x7f900c06e4c0 0x7f900c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564052434 0 0) 0x7f8fec003520 con 0x7f900c153f10 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 >> v2:172.21.15.69:3300/0 conn(0x7f900c06e4c0 msgr2=0x7f900c15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 --2- 172.21.15.43:0/209953267 >> v2:172.21.15.69:3300/0 conn(0x7f900c06e4c0 0x7f900c15aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 >> v1:172.21.15.69:6789/0 conn(0x7f900c15b050 legacy=0x7f900c06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0b2e20 con 0x7f900c153f10 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8ffc001670 con 0x7f900c153f10 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 >> v1:172.21.15.43:6789/0 conn(0x7f900c153f10 legacy=0x7f900c0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec003970 0x7f8fec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 0x7f8fec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec005180 0x7f8fec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c0b2b40 con 0x7f8fec0047c0 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00e2e0 con 0x7f8fec005180 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00e4a0 con 0x7f8fec003970 2024-01-16T15:07:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.733+0000 7f9002800640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 0x7f8fec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f900b400640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec003970 0x7f8fec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f900be00640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec005180 0x7f8fec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f9002800640 1 -- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec003970 msgr2=0x7f8fec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f9002800640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec003970 0x7f8fec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f9002800640 1 -- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec005180 msgr2=0x7f8fec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f9002800640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec005180 0x7f8fec00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f9002800640 1 -- 172.21.15.43:0/209953267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec003520 con 0x7f8fec0047c0 2024-01-16T15:07:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.734+0000 7f900be00640 1 -- 172.21.15.43:0/209953267 reap_dead start 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.735+0000 7f9002800640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 0x7f8fec004c40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4001750 tx=0x7f8fec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b222b1793edb3d89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.735+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff40052d0 con 0x7f8fec0047c0 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.735+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff402e050 con 0x7f8fec0047c0 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.735+0000 7f900aa00640 1 -- 172.21.15.43:0/209953267 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4003830 con 0x7f8fec0047c0 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 -- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 msgr2=0x7f8fec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 0x7f8fec004c40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4001750 tx=0x7f8fec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 -- 172.21.15.43:0/209953267 shutdown_connections 2024-01-16T15:07:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 --2- 172.21.15.43:0/209953267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fec0047c0 0x7f8fec004c40 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 -- 172.21.15.43:0/209953267 >> 172.21.15.43:0/209953267 conn(0x7f900c0b0d80 msgr2=0x7f900c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 -- 172.21.15.43:0/209953267 shutdown_connections 2024-01-16T15:07:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.736+0000 7f9012600640 1 -- 172.21.15.43:0/209953267 wait complete. 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.738+0000 7f9012600640 1 Processor -- start 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.738+0000 7f9012600640 1 -- start start 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 0x7f900c150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 0x7f900c147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c0b0b80 con 0x7f900c14ace0 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c14b6a0 con 0x7f8fec0047c0 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9012600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900c147690 con 0x7f900c146cd0 2024-01-16T15:07:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53476/0 (socket says 172.21.15.43:53476) 2024-01-16T15:07:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 -- 172.21.15.43:0/2532759882 learned_addr learned my addr 172.21.15.43:0/2532759882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f900be00640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 0x7f900c147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f900b400640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 0x7f900c150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 -- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 msgr2=0x7f900c147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 0x7f900c147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 -- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 msgr2=0x7f900c150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 0x7f900c150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.739+0000 7f9002800640 1 -- 172.21.15.43:0/2532759882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c1516c0 con 0x7f900c14ace0 2024-01-16T15:07:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f900be00640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 0x7f900c147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f900b400640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 0x7f900c150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9002800640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4005040 tx=0x7f8ff4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4007a30 con 0x7f900c14ace0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff40069e0 con 0x7f900c14ace0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4006d70 con 0x7f900c14ace0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c0b0b80 con 0x7f900c14ace0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.740+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c0af980 con 0x7f900c14ace0 2024-01-16T15:07:19.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.741+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c067960 con 0x7f900c14ace0 2024-01-16T15:07:19.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.742+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8ff40035f0 con 0x7f900c14ace0 2024-01-16T15:07:19.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.743+0000 7f9003e00640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 0x7f8fe406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:19.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.743+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f8ff40aa550 con 0x7f900c14ace0 2024-01-16T15:07:19.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.743+0000 7f900b400640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 0x7f8fe406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:19.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.744+0000 7f900b400640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 0x7f8fe406e0b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc001eb0 tx=0x7f8ffc001d60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.748+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ff4076e10 con 0x7f900c14ace0 2024-01-16T15:07:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.949+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f900c0b18f0 con 0x7f900c14ace0 2024-01-16T15:07:19.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.951+0000 7f9003e00640 1 -- 172.21.15.43:0/2532759882 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+12852 (secure 0 0 0) 0x7f8ff404b020 con 0x7f900c14ace0 2024-01-16T15:07:19.929 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:19.929 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":68,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:18.966160+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 msgr2=0x7f8fe406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 0x7f8fe406e0b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc001eb0 tx=0x7f8ffc001d60 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 msgr2=0x7f900c14b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4005040 tx=0x7f8ff4001120 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 shutdown_connections 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8fe406bbf0 0x7f8fe406e0b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f900c146cd0 0x7f900c147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f900c14ace0 0x7f900c14b160 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 --2- 172.21.15.43:0/2532759882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0047c0 0x7f900c150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:19.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 >> 172.21.15.43:0/2532759882 conn(0x7f900c0b0d80 msgr2=0x7f900c0704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:19.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 shutdown_connections 2024-01-16T15:07:19.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:19.954+0000 7f9012600640 1 -- 172.21.15.43:0/2532759882 wait complete. 2024-01-16T15:07:19.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:19.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.189+0000 7f25f7000640 1 Processor -- start 2024-01-16T15:07:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.190+0000 7f25f7000640 1 -- start start 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.191+0000 7f25f7000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f0160bd0 0x7f25f00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.191+0000 7f25f7000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25f00b48d0 0x7f25f00b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.191+0000 7f25f7000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00b23e0 con 0x7f25f00b3040 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.191+0000 7f25f7000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f00cdc70 con 0x7f25f00b48d0 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.191+0000 7f25f7000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00b2870 con 0x7f25f0160bd0 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25f00b48d0 0x7f25f00b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f25f00b48d0 0x7f25f00b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53488/0 (socket says 172.21.15.43:53488) 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 -- 172.21.15.43:0/3093112319 learned_addr learned my addr 172.21.15.43:0/3093112319 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25efe00640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f0160bd0 0x7f25f00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 -- 172.21.15.43:0/3093112319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f0160bd0 msgr2=0x7f25f00b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f0160bd0 0x7f25f00b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 -- 172.21.15.43:0/3093112319 >> v1:172.21.15.69:6789/0 conn(0x7f25f00b3040 legacy=0x7f25f0166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.192+0000 7f25e7e00640 1 -- 172.21.15.43:0/3093112319 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f00d41b0 con 0x7f25f00b48d0 2024-01-16T15:07:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.194+0000 7f25e7e00640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f00b48d0 0x7f25f00b2b00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009870 tx=0x7f25d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fc4d690c8d6f4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.194+0000 7f25ef400640 1 -- 172.21.15.43:0/3093112319 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8031640 con 0x7f25f00b48d0 2024-01-16T15:07:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.194+0000 7f25ef400640 1 -- 172.21.15.43:0/3093112319 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d80317a0 con 0x7f25f00b48d0 2024-01-16T15:07:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.194+0000 7f25ef400640 1 -- 172.21.15.43:0/3093112319 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8031b30 con 0x7f25f00b48d0 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 -- 172.21.15.43:0/3093112319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f00b48d0 msgr2=0x7f25f00b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f00b48d0 0x7f25f00b2b00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009870 tx=0x7f25d802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 -- 172.21.15.43:0/3093112319 shutdown_connections 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f00b48d0 0x7f25f00b2b00 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 --2- 172.21.15.43:0/3093112319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f0160bd0 0x7f25f00b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 -- 172.21.15.43:0/3093112319 >> 172.21.15.43:0/3093112319 conn(0x7f25f009c5a0 msgr2=0x7f25f00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 -- 172.21.15.43:0/3093112319 shutdown_connections 2024-01-16T15:07:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.195+0000 7f25f7000640 1 -- 172.21.15.43:0/3093112319 wait complete. 2024-01-16T15:07:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.197+0000 7f25f7000640 1 Processor -- start 2024-01-16T15:07:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.197+0000 7f25f7000640 1 -- start start 2024-01-16T15:07:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f00b3040 0x7f25f0157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 0x7f25f00b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f0158d90 con 0x7f25f0160bd0 2024-01-16T15:07:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00b5f00 con 0x7f25f00b48d0 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f7000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00b8e10 con 0x7f25f00b3040 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53490/0 (socket says 172.21.15.43:53490) 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 -- 172.21.15.43:0/775241840 learned_addr learned my addr 172.21.15.43:0/775241840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25efe00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f00b3040 0x7f25f0157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25e7e00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 0x7f25f00b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 -- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f00b3040 msgr2=0x7f25f0157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f00b3040 0x7f25f0157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 -- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 msgr2=0x7f25f00b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 0x7f25f00b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25f5000640 1 -- 172.21.15.43:0/775241840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80094d0 con 0x7f25f0160bd0 2024-01-16T15:07:20.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.198+0000 7f25e7e00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 0x7f25f00b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25f5000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f25e800c7a0 tx=0x7f25e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8003b70 con 0x7f25f0160bd0 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8003cf0 con 0x7f25f0160bd0 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e800aa90 con 0x7f25f0160bd0 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f00c7c80 con 0x7f25f0160bd0 2024-01-16T15:07:20.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.199+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f00b61a0 con 0x7f25f0160bd0 2024-01-16T15:07:20.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.200+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f00a1210 con 0x7f25f0160bd0 2024-01-16T15:07:20.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.201+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f25e800abf0 con 0x7f25f0160bd0 2024-01-16T15:07:20.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.203+0000 7f25ecc00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 0x7f25c806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.203+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f25e80882b0 con 0x7f25f0160bd0 2024-01-16T15:07:20.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.203+0000 7f25efe00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 0x7f25c806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.204+0000 7f25efe00640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 0x7f25c806e0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f25e0006fd0 tx=0x7f25e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.207+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25e8054bd0 con 0x7f25f0160bd0 2024-01-16T15:07:20.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.398+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25f00c8c10 con 0x7f25f0160bd0 2024-01-16T15:07:20.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.399+0000 7f25ecc00640 1 -- 172.21.15.43:0/775241840 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25e8003e60 con 0x7f25f0160bd0 2024-01-16T15:07:20.377 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:07:20.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 msgr2=0x7f25c806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 0x7f25c806e0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f25e0006fd0 tx=0x7f25e0008040 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 msgr2=0x7f25f00c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f25e800c7a0 tx=0x7f25e800cc60 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 shutdown_connections 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25c806bbf0 0x7f25c806e0b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25f00b48d0 0x7f25f00b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25f0160bd0 0x7f25f00c8e30 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 --2- 172.21.15.43:0/775241840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25f00b3040 0x7f25f0157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 >> 172.21.15.43:0/775241840 conn(0x7f25f009c5a0 msgr2=0x7f25f006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 shutdown_connections 2024-01-16T15:07:20.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.402+0000 7f25f7000640 1 -- 172.21.15.43:0/775241840 wait complete. 2024-01-16T15:07:20.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:07:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.643+0000 7f4923e00640 1 Processor -- start 2024-01-16T15:07:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.644+0000 7f4923e00640 1 -- start start 2024-01-16T15:07:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.645+0000 7f4923e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4924160b60 0x7f49240d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.645+0000 7f4923e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.645+0000 7f4923e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240b2b30 con 0x7f49240d3cf0 2024-01-16T15:07:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.645+0000 7f4923e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4924160f40 con 0x7f4924160b60 2024-01-16T15:07:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.645+0000 7f4923e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240b2e10 con 0x7f49240d01a0 2024-01-16T15:07:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.646+0000 7f4923400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.646+0000 7f4923400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:52280/0 (socket says 172.21.15.43:52280) 2024-01-16T15:07:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.646+0000 7f4923400640 1 -- 172.21.15.43:0/1626772834 learned_addr learned my addr 172.21.15.43:0/1626772834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.646+0000 7f4922a00640 1 --2- 172.21.15.43:0/1626772834 >> v2:172.21.15.69:3300/0 conn(0x7f4924160b60 0x7f49240d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.647+0000 7f4922a00640 1 -- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 msgr2=0x7f49240b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.647+0000 7f4922a00640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.647+0000 7f4922a00640 1 -- 172.21.15.43:0/1626772834 >> v1:172.21.15.69:6789/0 conn(0x7f49240d3cf0 legacy=0x7f49240cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.647+0000 7f4922a00640 1 -- 172.21.15.43:0/1626772834 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240b30f0 con 0x7f4924160b60 2024-01-16T15:07:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.647+0000 7f4923400640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:20.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.649+0000 7f4922a00640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240d37b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49140097a0 tx=0x7f491402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c266245076277f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.650+0000 7f4921600640 1 -- 172.21.15.43:0/1626772834 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914031780 con 0x7f4924160b60 2024-01-16T15:07:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.650+0000 7f4921600640 1 -- 172.21.15.43:0/1626772834 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49140318e0 con 0x7f4924160b60 2024-01-16T15:07:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.650+0000 7f4921600640 1 -- 172.21.15.43:0/1626772834 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914031c70 con 0x7f4924160b60 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.650+0000 7f4923e00640 1 -- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 msgr2=0x7f49240d37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.650+0000 7f4923e00640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240d37b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f49140097a0 tx=0x7f491402b820 comp rx=0 tx=0).stop 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 -- 172.21.15.43:0/1626772834 shutdown_connections 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f49240b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 --2- 172.21.15.43:0/1626772834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240d37b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 -- 172.21.15.43:0/1626772834 >> 172.21.15.43:0/1626772834 conn(0x7f492409c3d0 msgr2=0x7f49240d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 -- 172.21.15.43:0/1626772834 shutdown_connections 2024-01-16T15:07:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.651+0000 7f4923e00640 1 -- 172.21.15.43:0/1626772834 wait complete. 2024-01-16T15:07:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.653+0000 7f4923e00640 1 Processor -- start 2024-01-16T15:07:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.654+0000 7f4923e00640 1 -- start start 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f492415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 0x7f49240b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492415b820 con 0x7f49240d3cf0 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49240b6670 con 0x7f4924160b60 2024-01-16T15:07:20.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.655+0000 7f4923e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240b6950 con 0x7f49240d01a0 2024-01-16T15:07:20.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f492415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f492415b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:52288/0 (socket says 172.21.15.43:52288) 2024-01-16T15:07:20.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922a00640 1 -- 172.21.15.43:0/3376379895 learned_addr learned my addr 172.21.15.43:0/3376379895 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 0x7f49240b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4923400640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 -- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 msgr2=0x7f492415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f492415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 -- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 msgr2=0x7f49240b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240b7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.656+0000 7f4922000640 1 -- 172.21.15.43:0/3376379895 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c009b30 con 0x7f49240d3cf0 2024-01-16T15:07:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f4922000640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 0x7f49240b5c20 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c390 tx=0x7f490c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c0052d0 con 0x7f49240d3cf0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c005430 con 0x7f49240d3cf0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c003830 con 0x7f49240d3cf0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4914009400 con 0x7f49240d3cf0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.657+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49240b6f10 con 0x7f49240d3cf0 2024-01-16T15:07:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.659+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492404e840 con 0x7f49240d3cf0 2024-01-16T15:07:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.659+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f490c003990 con 0x7f49240d3cf0 2024-01-16T15:07:20.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.660+0000 7f491b400640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 0x7f48fc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.660+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f490c087960 con 0x7f49240d3cf0 2024-01-16T15:07:20.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.661+0000 7f4922a00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 0x7f48fc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.661+0000 7f4922a00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 0x7f48fc06e080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4914000c00 tx=0x7f491403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.667+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f490c0542d0 con 0x7f49240d3cf0 2024-01-16T15:07:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.861+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49240d2950 con 0x7f49240d3cf0 2024-01-16T15:07:20.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.862+0000 7f491b400640 1 -- 172.21.15.43:0/3376379895 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) v1 ==== 84+0+65 (secure 0 0 0) 0x7f490c0578f0 con 0x7f49240d3cf0 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:20.841 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:07:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 msgr2=0x7f48fc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 0x7f48fc06e080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4914000c00 tx=0x7f491403d040 comp rx=0 tx=0).stop 2024-01-16T15:07:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 msgr2=0x7f49240b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 0x7f49240b5c20 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c390 tx=0x7f490c00c850 comp rx=0 tx=0).stop 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 shutdown_connections 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f48fc06bbc0 0x7f48fc06e080 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4924160b60 0x7f49240b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49240d3cf0 0x7f49240b5c20 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 --2- 172.21.15.43:0/3376379895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49240d01a0 0x7f492415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 >> 172.21.15.43:0/3376379895 conn(0x7f492409c3d0 msgr2=0x7f49240b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 shutdown_connections 2024-01-16T15:07:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:20.866+0000 7f4923e00640 1 -- 172.21.15.43:0/3376379895 wait complete. 2024-01-16T15:07:20.863 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:07:20.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:07:21.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.115+0000 7f75b6a00640 1 Processor -- start 2024-01-16T15:07:21.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.115+0000 7f75b6a00640 1 -- start start 2024-01-16T15:07:21.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 0x7f75b00ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75b0070850 0x7f75b0071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b6a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b00ce650 con 0x7f75b0154280 2024-01-16T15:07:21.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b6a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b015e0a0 con 0x7f75b0070850 2024-01-16T15:07:21.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b015e380 con 0x7f75b0153ea0 2024-01-16T15:07:21.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b4c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f75b0154280 0x7f75b015d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42620/0 (socket says 172.21.15.43:42620) 2024-01-16T15:07:21.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.117+0000 7f75b4c00640 1 -- 172.21.15.43:0/3936649593 learned_addr learned my addr 172.21.15.43:0/3936649593 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:21.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b5600640 1 --2- 172.21.15.43:0/3936649593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 0x7f75b00ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b6000640 1 --2- 172.21.15.43:0/3936649593 >> v2:172.21.15.69:3300/0 conn(0x7f75b0070850 0x7f75b0071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213887996 0 0) 0x7f75b00ce650 con 0x7f75b0154280 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7590003520 con 0x7f75b0154280 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b5600640 1 -- 172.21.15.43:0/3936649593 >> v2:172.21.15.69:3300/0 conn(0x7f75b0070850 msgr2=0x7f75b0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b5600640 1 --2- 172.21.15.43:0/3936649593 >> v2:172.21.15.69:3300/0 conn(0x7f75b0070850 0x7f75b0071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b5600640 1 -- 172.21.15.43:0/3936649593 >> v1:172.21.15.43:6789/0 conn(0x7f75b0154280 legacy=0x7f75b015d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3313670046 0 0) 0x7f7590003520 con 0x7f75b0154280 2024-01-16T15:07:21.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.118+0000 7f75b5600640 1 -- 172.21.15.43:0/3936649593 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b015e660 con 0x7f75b0153ea0 2024-01-16T15:07:21.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.120+0000 7f75b5600640 1 --2- 172.21.15.43:0/3936649593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 0x7f75b00ad450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009d90 tx=0x7f75a002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2b844489460718 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.120+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598001670 con 0x7f75b0153ea0 2024-01-16T15:07:21.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.120+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a002c8a0 con 0x7f75b0153ea0 2024-01-16T15:07:21.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.120+0000 7f75afe00640 1 -- 172.21.15.43:0/3936649593 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a002cc30 con 0x7f75b0153ea0 2024-01-16T15:07:21.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 -- 172.21.15.43:0/3936649593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 msgr2=0x7f75b00ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3936649593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 0x7f75b00ad450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009d90 tx=0x7f75a002bd70 comp rx=0 tx=0).stop 2024-01-16T15:07:21.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 -- 172.21.15.43:0/3936649593 shutdown_connections 2024-01-16T15:07:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3936649593 >> v2:172.21.15.69:3300/0 conn(0x7f75b0070850 0x7f75b0071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3936649593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0153ea0 0x7f75b00ad450 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 -- 172.21.15.43:0/3936649593 >> 172.21.15.43:0/3936649593 conn(0x7f75b00cec10 msgr2=0x7f75b00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 -- 172.21.15.43:0/3936649593 shutdown_connections 2024-01-16T15:07:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.121+0000 7f75b6a00640 1 -- 172.21.15.43:0/3936649593 wait complete. 2024-01-16T15:07:21.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.123+0000 7f75b6a00640 1 Processor -- start 2024-01-16T15:07:21.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.123+0000 7f75b6a00640 1 -- start start 2024-01-16T15:07:21.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 0x7f75b00b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75b0154280 0x7f75b0156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00b1630 con 0x7f75b0153ea0 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00b09b0 con 0x7f75b0154280 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00b0c90 con 0x7f75b0070850 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53498/0 (socket says 172.21.15.43:53498) 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.124+0000 7f75b4c00640 1 -- 172.21.15.43:0/3861335676 learned_addr learned my addr 172.21.15.43:0/3861335676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b5600640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 0x7f75b00b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b6000640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75b0154280 0x7f75b0156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 -- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 msgr2=0x7f75b00b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 0x7f75b00b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 -- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75b0154280 msgr2=0x7f75b0156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75b0154280 0x7f75b0156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 -- 172.21.15.43:0/3861335676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0009a70 con 0x7f75b0153ea0 2024-01-16T15:07:21.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b5600640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 0x7f75b00b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b4c00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f759800c780 tx=0x7f759800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598005ad0 con 0x7f75b0153ea0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598005c30 con 0x7f75b0153ea0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598007070 con 0x7f75b0153ea0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.125+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b00b0f70 con 0x7f75b0153ea0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.126+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b00b06a0 con 0x7f75b0153ea0 2024-01-16T15:07:21.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.127+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b00b1770 con 0x7f75b0153ea0 2024-01-16T15:07:21.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.128+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f75980071d0 con 0x7f75b0153ea0 2024-01-16T15:07:21.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.128+0000 7f75ad600640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 0x7f758806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.129+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f7598088170 con 0x7f75b0153ea0 2024-01-16T15:07:21.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.129+0000 7f75b5600640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 0x7f758806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.130+0000 7f75b5600640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 0x7f758806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f75a0036000 tx=0x7f75a0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.133+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7598054a30 con 0x7f75b0153ea0 2024-01-16T15:07:21.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.329+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f75b004e840 con 0x7f75b0153ea0 2024-01-16T15:07:21.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.974+0000 7f75ad600640 1 -- 172.21.15.43:0/3861335676 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v69) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7598058050 con 0x7f75b0153ea0 2024-01-16T15:07:21.953 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.978+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 msgr2=0x7f758806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.978+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 0x7f758806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f75a0036000 tx=0x7f75a0030000 comp rx=0 tx=0).stop 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.978+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 msgr2=0x7f75b0156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.978+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f759800c780 tx=0x7f759800cc40 comp rx=0 tx=0).stop 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 shutdown_connections 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f758806bbc0 0x7f758806e080 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75b0154280 0x7f75b0156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75b0153ea0 0x7f75b0156a30 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 --2- 172.21.15.43:0/3861335676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75b0070850 0x7f75b00b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 >> 172.21.15.43:0/3861335676 conn(0x7f75b00cec10 msgr2=0x7f75b006d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 shutdown_connections 2024-01-16T15:07:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:21.979+0000 7f75b6a00640 1 -- 172.21.15.43:0/3861335676 wait complete. 2024-01-16T15:07:21.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:07:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.214+0000 7fd422800640 1 Processor -- start 2024-01-16T15:07:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.214+0000 7fd422800640 1 -- start start 2024-01-16T15:07:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.215+0000 7fd422800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c15aae0 0x7fd41c06cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd422800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd41c06d2a0 0x7fd41c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd422800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c06d700 con 0x7fd41c0b2d70 2024-01-16T15:07:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd422800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41c15aec0 con 0x7fd41c06d2a0 2024-01-16T15:07:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd422800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c154440 con 0x7fd41c15aae0 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd41aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd41c06d2a0 0x7fd41c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd41be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd41c0b2d70 0x7fd41c15d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42624/0 (socket says 172.21.15.43:42624) 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd41be00640 1 -- 172.21.15.43:0/1266736752 learned_addr learned my addr 172.21.15.43:0/1266736752 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.216+0000 7fd41b400640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c15aae0 0x7fd41c06cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.217+0000 7fd41aa00640 1 -- 172.21.15.43:0/1266736752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c15aae0 msgr2=0x7fd41c06cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.217+0000 7fd41aa00640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c15aae0 0x7fd41c06cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.217+0000 7fd41aa00640 1 -- 172.21.15.43:0/1266736752 >> v1:172.21.15.43:6789/0 conn(0x7fd41c0b2d70 legacy=0x7fd41c15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.217+0000 7fd41aa00640 1 -- 172.21.15.43:0/1266736752 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c157b30 con 0x7fd41c06d2a0 2024-01-16T15:07:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.218+0000 7fd41aa00640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c0b2830 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd4040097a0 tx=0x7fd40402dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a182f110b13fc64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.218+0000 7fd41a000640 1 -- 172.21.15.43:0/1266736752 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40402f880 con 0x7fd41c06d2a0 2024-01-16T15:07:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.218+0000 7fd41a000640 1 -- 172.21.15.43:0/1266736752 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40402f9e0 con 0x7fd41c06d2a0 2024-01-16T15:07:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.218+0000 7fd41a000640 1 -- 172.21.15.43:0/1266736752 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40402fd70 con 0x7fd41c06d2a0 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 -- 172.21.15.43:0/1266736752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 msgr2=0x7fd41c0b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c0b2830 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd4040097a0 tx=0x7fd40402dac0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 -- 172.21.15.43:0/1266736752 shutdown_connections 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c0b2830 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 --2- 172.21.15.43:0/1266736752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c15aae0 0x7fd41c06cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 -- 172.21.15.43:0/1266736752 >> 172.21.15.43:0/1266736752 conn(0x7fd41c0b0ba0 msgr2=0x7fd41c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 -- 172.21.15.43:0/1266736752 shutdown_connections 2024-01-16T15:07:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.219+0000 7fd422800640 1 -- 172.21.15.43:0/1266736752 wait complete. 2024-01-16T15:07:22.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.221+0000 7fd422800640 1 Processor -- start 2024-01-16T15:07:22.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.221+0000 7fd422800640 1 -- start start 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c0b2d70 0x7fd41c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd41c15aae0 0x7fd41c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c09a5f0 con 0x7fd41c06d2a0 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c14bde0 con 0x7fd41c15aae0 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd422800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c099a00 con 0x7fd41c0b2d70 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53504/0 (socket says 172.21.15.43:53504) 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 -- 172.21.15.43:0/2357116639 learned_addr learned my addr 172.21.15.43:0/2357116639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 -- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c0b2d70 msgr2=0x7fd41c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c0b2d70 0x7fd41c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 -- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd41c15aae0 msgr2=0x7fd41c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd41c15aae0 0x7fd41c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.222+0000 7fd41b400640 1 -- 172.21.15.43:0/2357116639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd404009400 con 0x7fd41c06d2a0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd41b400640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00d620 tx=0x7fd40c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00b620 con 0x7fd41c06d2a0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c00b780 con 0x7fd41c06d2a0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41c099c50 con 0x7fd41c06d2a0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00bb30 con 0x7fd41c06d2a0 2024-01-16T15:07:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.223+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c09a210 con 0x7fd41c06d2a0 2024-01-16T15:07:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.224+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41c09a3a0 con 0x7fd41c06d2a0 2024-01-16T15:07:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.227+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd40c00bc90 con 0x7fd41c06d2a0 2024-01-16T15:07:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.227+0000 7fd413400640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 0x7fd3f406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.227+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fd40c089710 con 0x7fd41c06d2a0 2024-01-16T15:07:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.228+0000 7fd41aa00640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 0x7fd3f406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.228+0000 7fd41aa00640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 0x7fd3f406e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd404038000 tx=0x7fd40403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.231+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd40c055d50 con 0x7fd41c06d2a0 2024-01-16T15:07:22.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.428+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd41c09d200 con 0x7fd41c06d2a0 2024-01-16T15:07:22.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.988+0000 7fd413400640 1 -- 172.21.15.43:0/2357116639 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v70) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd40c059370 con 0x7fd41c06d2a0 2024-01-16T15:07:22.966 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.992+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 msgr2=0x7fd3f406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.992+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 0x7fd3f406e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd404038000 tx=0x7fd40403b040 comp rx=0 tx=0).stop 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.992+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 msgr2=0x7fd41c14b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.992+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00d620 tx=0x7fd40c00dae0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 shutdown_connections 2024-01-16T15:07:22.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd3f406bbc0 0x7fd3f406e080 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd41c15aae0 0x7fd41c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd41c0b2d70 0x7fd41c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 --2- 172.21.15.43:0/2357116639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd41c06d2a0 0x7fd41c14b2e0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 >> 172.21.15.43:0/2357116639 conn(0x7fd41c0b0ba0 msgr2=0x7fd41c15bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.993+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 shutdown_connections 2024-01-16T15:07:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:22.994+0000 7fd422800640 1 -- 172.21.15.43:0/2357116639 wait complete. 2024-01-16T15:07:23.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:07:23.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.219+0000 7fda5e000640 1 Processor -- start 2024-01-16T15:07:23.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.220+0000 7fda5e000640 1 -- start start 2024-01-16T15:07:23.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda5e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda580abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda5e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5815b730 con 0x7fda580adb60 2024-01-16T15:07:23.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda5e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580ac740 con 0x7fda5815b350 2024-01-16T15:07:23.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda5e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580ac460 con 0x7fda5809cb30 2024-01-16T15:07:23.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda57e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fda580adb60 0x7fda580adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44872/0 (socket says 172.21.15.43:44872) 2024-01-16T15:07:23.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.221+0000 7fda57e00640 1 -- 172.21.15.43:0/4123135991 learned_addr learned my addr 172.21.15.43:0/4123135991 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda4e800640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda580abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68297953 0 0) 0x7fda50001490 con 0x7fda580adb60 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda38003520 con 0x7fda580adb60 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145803606 0 0) 0x7fda38003520 con 0x7fda580adb60 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 msgr2=0x7fda580abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda580abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 >> v1:172.21.15.69:6789/0 conn(0x7fda5815b350 legacy=0x7fda580b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580b2a50 con 0x7fda580adb60 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fda50001b80 con 0x7fda580adb60 2024-01-16T15:07:23.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.222+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 >> v1:172.21.15.43:6789/0 conn(0x7fda580adb60 legacy=0x7fda580adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380039a0 0x7fda38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580ac460 con 0x7fda380047c0 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3800ba60 con 0x7fda380039a0 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3800bc20 con 0x7fda5809cb30 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda57e00640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda57400640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380039a0 0x7fda38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 -- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 msgr2=0x7fda38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5809cb30 0x7fda38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 -- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380039a0 msgr2=0x7fda38003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380039a0 0x7fda38003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.223+0000 7fda4e800640 1 -- 172.21.15.43:0/4123135991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38003520 con 0x7fda380047c0 2024-01-16T15:07:23.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda4e800640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda38004c40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fda40009a90 tx=0x7fda380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7aeab8cba4d3f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4002c660 con 0x7fda380047c0 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4002c7c0 con 0x7fda380047c0 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda56a00640 1 -- 172.21.15.43:0/4123135991 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4002cb50 con 0x7fda380047c0 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda5e000640 1 -- 172.21.15.43:0/4123135991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 msgr2=0x7fda38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda5e000640 1 --2- 172.21.15.43:0/4123135991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda38004c40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fda40009a90 tx=0x7fda380057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.225+0000 7fda57e00640 1 -- 172.21.15.43:0/4123135991 reap_dead start 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.226+0000 7fda5e000640 1 -- 172.21.15.43:0/4123135991 shutdown_connections 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.226+0000 7fda5e000640 1 -- 172.21.15.43:0/4123135991 >> 172.21.15.43:0/4123135991 conn(0x7fda580b0ba0 msgr2=0x7fda580b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.226+0000 7fda5e000640 1 -- 172.21.15.43:0/4123135991 shutdown_connections 2024-01-16T15:07:23.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.226+0000 7fda5e000640 1 -- 172.21.15.43:0/4123135991 wait complete. 2024-01-16T15:07:23.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.227+0000 7fda5e000640 1 Processor -- start 2024-01-16T15:07:23.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 -- start start 2024-01-16T15:07:23.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 0x7fda58146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda5809aa50 0x7fda5809aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5814b5a0 con 0x7fda380047c0 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5809b410 con 0x7fda5809aa50 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.228+0000 7fda5e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5814ac20 con 0x7fda5814e370 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55026/0 (socket says 172.21.15.43:55026) 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 -- 172.21.15.43:0/1956510800 learned_addr learned my addr 172.21.15.43:0/1956510800 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda4e800640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 0x7fda58146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57e00640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda5809aa50 0x7fda5809aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 -- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 msgr2=0x7fda58146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 0x7fda58146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 -- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda5809aa50 msgr2=0x7fda5809aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda5809aa50 0x7fda5809aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda57400640 1 -- 172.21.15.43:0/1956510800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5814f200 con 0x7fda380047c0 2024-01-16T15:07:23.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.229+0000 7fda4e800640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 0x7fda58146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda57400640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e550 tx=0x7fda4800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda480052e0 con 0x7fda380047c0 2024-01-16T15:07:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48005440 con 0x7fda380047c0 2024-01-16T15:07:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48003810 con 0x7fda380047c0 2024-01-16T15:07:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5814aee0 con 0x7fda380047c0 2024-01-16T15:07:23.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.230+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5814e7d0 con 0x7fda380047c0 2024-01-16T15:07:23.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.231+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5809d200 con 0x7fda380047c0 2024-01-16T15:07:23.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.232+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fda48003970 con 0x7fda380047c0 2024-01-16T15:07:23.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.233+0000 7fda4fe00640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 0x7fda3006e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.233+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fda480882c0 con 0x7fda380047c0 2024-01-16T15:07:23.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.233+0000 7fda4e800640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 0x7fda3006e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.234+0000 7fda4e800640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 0x7fda3006e100 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda40030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.238+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda48054990 con 0x7fda380047c0 2024-01-16T15:07:23.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:23.468+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fda580ac740 con 0x7fda380047c0 2024-01-16T15:07:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.019+0000 7fda4fe00640 1 -- 172.21.15.43:0/1956510800 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7fda580ac740 con 0x7fda380047c0 2024-01-16T15:07:24.000 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 10 and data pool 11 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.028+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 msgr2=0x7fda3006e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.028+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 0x7fda3006e100 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda40030000 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.028+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 msgr2=0x7fda5814eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.028+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e550 tx=0x7fda4800ea10 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.029+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 shutdown_connections 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.029+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda3006bc40 0x7fda3006e100 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.029+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda5809aa50 0x7fda5809aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.029+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda5814e370 0x7fda58146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.029+0000 7fda5e000640 1 --2- 172.21.15.43:0/1956510800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda380047c0 0x7fda5814eb40 secure :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e550 tx=0x7fda4800ea10 comp rx=0 tx=0).stop 2024-01-16T15:07:24.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.030+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 >> 172.21.15.43:0/1956510800 conn(0x7fda580b0ba0 msgr2=0x7fda58070520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.030+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 shutdown_connections 2024-01-16T15:07:24.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.031+0000 7fda5e000640 1 -- 172.21.15.43:0/1956510800 wait complete. 2024-01-16T15:07:24.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.239+0000 7f986d600640 1 Processor -- start 2024-01-16T15:07:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.239+0000 7f986d600640 1 -- start start 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f986d600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 0x7f986806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f986d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f98680abff0 0x7f986815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f986d600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986815c360 con 0x7f98680ad720 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f986d600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98680b41a0 con 0x7f98680abff0 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f986d600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986806de40 con 0x7f986806d520 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f985fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f98680ad720 0x7f986815ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44884/0 (socket says 172.21.15.43:44884) 2024-01-16T15:07:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f985fe00640 1 -- 172.21.15.43:0/1911913592 learned_addr learned my addr 172.21.15.43:0/1911913592 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.241+0000 7f9867400640 1 --2- 172.21.15.43:0/1911913592 >> v2:172.21.15.43:3300/0 conn(0x7f98680abff0 0x7f986815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9866a00640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 0x7f986806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9866000640 1 -- 172.21.15.43:0/1911913592 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177756418 0 0) 0x7f986815c360 con 0x7f98680ad720 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9866000640 1 -- 172.21.15.43:0/1911913592 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9848003520 con 0x7f98680ad720 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9867400640 1 -- 172.21.15.43:0/1911913592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 msgr2=0x7f986806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9867400640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 0x7f986806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9867400640 1 -- 172.21.15.43:0/1911913592 >> v1:172.21.15.43:6789/0 conn(0x7f98680ad720 legacy=0x7f986815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9867400640 1 -- 172.21.15.43:0/1911913592 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986815ba30 con 0x7f98680abff0 2024-01-16T15:07:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.242+0000 7f9866a00640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 0x7f986806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:24.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.243+0000 7f9867400640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98680abff0 0x7f986815f3e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f9860009d90 tx=0x7f986002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00fd44c3e39a7e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.243+0000 7f9866000640 1 -- 172.21.15.43:0/1911913592 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986002cc50 con 0x7f98680abff0 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.243+0000 7f9866000640 1 -- 172.21.15.43:0/1911913592 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986002cdb0 con 0x7f98680abff0 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.243+0000 7f9866000640 1 -- 172.21.15.43:0/1911913592 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860034660 con 0x7f98680abff0 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 -- 172.21.15.43:0/1911913592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98680abff0 msgr2=0x7f986815f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98680abff0 0x7f986815f3e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f9860009d90 tx=0x7f986002bea0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 -- 172.21.15.43:0/1911913592 shutdown_connections 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98680abff0 0x7f986815f3e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 --2- 172.21.15.43:0/1911913592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f986806d520 0x7f986806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 -- 172.21.15.43:0/1911913592 >> 172.21.15.43:0/1911913592 conn(0x7f98680b0d80 msgr2=0x7f98680b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 -- 172.21.15.43:0/1911913592 shutdown_connections 2024-01-16T15:07:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.244+0000 7f986d600640 1 -- 172.21.15.43:0/1911913592 wait complete. 2024-01-16T15:07:24.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.246+0000 7f986d600640 1 Processor -- start 2024-01-16T15:07:24.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.246+0000 7f986d600640 1 -- start start 2024-01-16T15:07:24.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98680abff0 0x7f98681483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98680ad720 0x7f9868153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986815da30 con 0x7f986806d520 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98681559b0 con 0x7f98680ad720 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f986d600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9868071fe0 con 0x7f98680abff0 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f9866a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f9866a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55036/0 (socket says 172.21.15.43:55036) 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.247+0000 7f9866a00640 1 -- 172.21.15.43:0/2391757326 learned_addr learned my addr 172.21.15.43:0/2391757326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 -- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98680abff0 msgr2=0x7f98681483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98680abff0 0x7f98681483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 -- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98680ad720 msgr2=0x7f9868153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98680ad720 0x7f9868153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 -- 172.21.15.43:0/2391757326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9860009a70 con 0x7f986806d520 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f9866a00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f985800e7e0 tx=0x7f985800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858004030 con 0x7f986806d520 2024-01-16T15:07:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9858004190 con 0x7f986806d520 2024-01-16T15:07:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858004520 con 0x7f986806d520 2024-01-16T15:07:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.248+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9868154520 con 0x7f986806d520 2024-01-16T15:07:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.249+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9868072280 con 0x7f986806d520 2024-01-16T15:07:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.250+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9868074770 con 0x7f986806d520 2024-01-16T15:07:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.251+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f985800c6d0 con 0x7f986806d520 2024-01-16T15:07:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.252+0000 7f985ea00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 0x7f984006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.252+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f98580883c0 con 0x7f986806d520 2024-01-16T15:07:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.252+0000 7f985fe00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 0x7f984006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.253+0000 7f985fe00640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 0x7f984006e0b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9868153b10 tx=0x7f9850007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.256+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9858090020 con 0x7f986806d520 2024-01-16T15:07:24.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.449+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98680acf20 con 0x7f986806d520 2024-01-16T15:07:24.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.451+0000 7f985ea00640 1 -- 172.21.15.43:0/2391757326 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16576 (secure 0 0 0) 0x7f9858020020 con 0x7f986806d520 2024-01-16T15:07:24.429 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:24.429 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":71,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:23.470475+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:07:21.386131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:07:22.484863+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 msgr2=0x7f984006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 0x7f984006e0b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9868153b10 tx=0x7f9850007930 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 msgr2=0x7f9868155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f985800e7e0 tx=0x7f985800eca0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 shutdown_connections 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f984006bbf0 0x7f984006e0b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98680ad720 0x7f9868153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98680abff0 0x7f98681483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 --2- 172.21.15.43:0/2391757326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f986806d520 0x7f9868155e70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 >> 172.21.15.43:0/2391757326 conn(0x7f98680b0d80 msgr2=0x7f9868158570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 shutdown_connections 2024-01-16T15:07:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.453+0000 7f986d600640 1 -- 172.21.15.43:0/2391757326 wait complete. 2024-01-16T15:07:24.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:24.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.693+0000 7fc9c7e00640 1 Processor -- start 2024-01-16T15:07:24.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.693+0000 7fc9c7e00640 1 -- start start 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9c81439b0 0x7fc9c80ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc9c80cef70 0x7fc9c815db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c815eab0 con 0x7fc9c80ce800 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c80cdea0 con 0x7fc9c80cef70 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c815e4f0 con 0x7fc9c81439b0 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc9c80ce800 0x7fc9c80712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44886/0 (socket says 172.21.15.43:44886) 2024-01-16T15:07:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.695+0000 7fc9c7400640 1 -- 172.21.15.43:0/3206747695 learned_addr learned my addr 172.21.15.43:0/3206747695 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6000640 1 --2- 172.21.15.43:0/3206747695 >> v2:172.21.15.43:3300/0 conn(0x7fc9c80cef70 0x7fc9c815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6a00640 1 --2- 172.21.15.43:0/3206747695 >> v2:172.21.15.69:3300/0 conn(0x7fc9c81439b0 0x7fc9c80ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c5600640 1 -- 172.21.15.43:0/3206747695 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542782394 0 0) 0x7fc9c815eab0 con 0x7fc9c80ce800 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c5600640 1 -- 172.21.15.43:0/3206747695 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a8003520 con 0x7fc9c80ce800 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6000640 1 -- 172.21.15.43:0/3206747695 >> v2:172.21.15.69:3300/0 conn(0x7fc9c81439b0 msgr2=0x7fc9c80ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6000640 1 --2- 172.21.15.43:0/3206747695 >> v2:172.21.15.69:3300/0 conn(0x7fc9c81439b0 0x7fc9c80ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6000640 1 -- 172.21.15.43:0/3206747695 >> v1:172.21.15.43:6789/0 conn(0x7fc9c80ce800 legacy=0x7fc9c80712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.696+0000 7fc9c6000640 1 -- 172.21.15.43:0/3206747695 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c815e7d0 con 0x7fc9c80cef70 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.697+0000 7fc9c6000640 1 --2- 172.21.15.43:0/3206747695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c80cef70 0x7fc9c815db10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8070ae0 tx=0x7fc9b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea50a504ffc8ddb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.697+0000 7fc9c5600640 1 -- 172.21.15.43:0/3206747695 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b002e070 con 0x7fc9c80cef70 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c5600640 1 -- 172.21.15.43:0/3206747695 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b00315a0 con 0x7fc9c80cef70 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c5600640 1 -- 172.21.15.43:0/3206747695 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0031930 con 0x7fc9c80cef70 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 -- 172.21.15.43:0/3206747695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c80cef70 msgr2=0x7fc9c815db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/3206747695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c80cef70 0x7fc9c815db10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8070ae0 tx=0x7fc9b002b840 comp rx=0 tx=0).stop 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 -- 172.21.15.43:0/3206747695 shutdown_connections 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/3206747695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c80cef70 0x7fc9c815db10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/3206747695 >> v2:172.21.15.69:3300/0 conn(0x7fc9c81439b0 0x7fc9c80ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.698+0000 7fc9c7e00640 1 -- 172.21.15.43:0/3206747695 >> 172.21.15.43:0/3206747695 conn(0x7fc9c80cfdc0 msgr2=0x7fc9c80d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.699+0000 7fc9c7e00640 1 -- 172.21.15.43:0/3206747695 shutdown_connections 2024-01-16T15:07:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.699+0000 7fc9c7e00640 1 -- 172.21.15.43:0/3206747695 wait complete. 2024-01-16T15:07:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.700+0000 7fc9c7e00640 1 Processor -- start 2024-01-16T15:07:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.700+0000 7fc9c7e00640 1 -- start start 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c80cef70 0x7fc9c81595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 0x7fc9c81f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c815bcb0 con 0x7fc9c81439b0 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c815bf00 con 0x7fc9c80cef70 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.701+0000 7fc9c7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c815c1e0 con 0x7fc9c8159bb0 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55054/0 (socket says 172.21.15.43:55054) 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 -- 172.21.15.43:0/4175943679 learned_addr learned my addr 172.21.15.43:0/4175943679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c7400640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 0x7fc9c81f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 -- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 msgr2=0x7fc9c81f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6a00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c80cef70 0x7fc9c81595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 0x7fc9c81f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 -- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c80cef70 msgr2=0x7fc9c81595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c80cef70 0x7fc9c81595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c6000640 1 -- 172.21.15.43:0/4175943679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b0009400 con 0x7fc9c81439b0 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.702+0000 7fc9c7400640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 0x7fc9c81f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9c6000640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009890 tx=0x7fc9b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0032630 con 0x7fc9c81439b0 2024-01-16T15:07:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b0032790 con 0x7fc9c81439b0 2024-01-16T15:07:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c815c4c0 con 0x7fc9c81439b0 2024-01-16T15:07:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0032b40 con 0x7fc9c81439b0 2024-01-16T15:07:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.703+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c81592e0 con 0x7fc9c81439b0 2024-01-16T15:07:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.704+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c815c9e0 con 0x7fc9c81439b0 2024-01-16T15:07:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.705+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc9b003b3f0 con 0x7fc9c81439b0 2024-01-16T15:07:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.707+0000 7fc9bea00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 0x7fc9a006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.707+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc9b00aa550 con 0x7fc9c81439b0 2024-01-16T15:07:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.707+0000 7fc9c6a00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 0x7fc9a006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.708+0000 7fc9c6a00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 0x7fc9a006e080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc9c815bc40 tx=0x7fc9b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.711+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9c815c9e0 con 0x7fc9c81439b0 2024-01-16T15:07:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.912+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9c815c9e0 con 0x7fc9c81439b0 2024-01-16T15:07:24.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.915+0000 7fc9bea00640 1 -- 172.21.15.43:0/4175943679 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16576 (secure 0 0 0) 0x7fc9c815c9e0 con 0x7fc9c81439b0 2024-01-16T15:07:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:24.893 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":71,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:23.470475+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:07:21.386131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:07:22.484863+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:24.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 msgr2=0x7fc9a006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 0x7fc9a006e080 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc9c815bc40 tx=0x7fc9b8008040 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 msgr2=0x7fc9c8158da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009890 tx=0x7fc9b00046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 shutdown_connections 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc9a006bbc0 0x7fc9a006e080 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9c8159bb0 0x7fc9c81f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9c81439b0 0x7fc9c8158da0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 --2- 172.21.15.43:0/4175943679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9c80cef70 0x7fc9c81595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 >> 172.21.15.43:0/4175943679 conn(0x7fc9c80cfdc0 msgr2=0x7fc9c80cb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 shutdown_connections 2024-01-16T15:07:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:24.918+0000 7fc9c7e00640 1 -- 172.21.15.43:0/4175943679 wait complete. 2024-01-16T15:07:24.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:07:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.163+0000 7efe0be00640 1 Processor -- start 2024-01-16T15:07:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.163+0000 7efe0be00640 1 -- start start 2024-01-16T15:07:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe0c15f270 0x7efe0c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c15f930 con 0x7efe0c144770 2024-01-16T15:07:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c0b3b10 con 0x7efe0c06d9e0 2024-01-16T15:07:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c0b43b0 con 0x7efe0c15f270 2024-01-16T15:07:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efe0c144770 0x7efe0c15eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44892/0 (socket says 172.21.15.43:44892) 2024-01-16T15:07:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.165+0000 7efe0aa00640 1 -- 172.21.15.43:0/1894301870 learned_addr learned my addr 172.21.15.43:0/1894301870 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308255233 0 0) 0x7efe0c15f930 con 0x7efe0c144770 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdec003520 con 0x7efe0c144770 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe03e00640 1 --2- 172.21.15.43:0/1894301870 >> v2:172.21.15.69:3300/0 conn(0x7efe0c15f270 0x7efe0c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501633675 0 0) 0x7efdec003520 con 0x7efe0c144770 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 >> v2:172.21.15.69:3300/0 conn(0x7efe0c15f270 msgr2=0x7efe0c0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 --2- 172.21.15.43:0/1894301870 >> v2:172.21.15.69:3300/0 conn(0x7efe0c15f270 0x7efe0c0cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 >> v1:172.21.15.69:6789/0 conn(0x7efe0c06d9e0 legacy=0x7efe0c0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0b3df0 con 0x7efe0c144770 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7efdfc001670 con 0x7efe0c144770 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 >> v1:172.21.15.43:6789/0 conn(0x7efe0c144770 legacy=0x7efe0c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.166+0000 7efe0a000640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec003a30 0x7efdec003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0a000640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdec0047c0 0x7efdec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0a000640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 0x7efdec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c0b43b0 con 0x7efdec0052a0 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec00c1a0 con 0x7efdec003a30 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec00c360 con 0x7efdec0047c0 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0b400640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 0x7efdec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0aa00640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec003a30 0x7efdec003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe03e00640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdec0047c0 0x7efdec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0b400640 1 -- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdec0047c0 msgr2=0x7efdec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0b400640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efdec0047c0 0x7efdec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.167+0000 7efe0b400640 1 -- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec003a30 msgr2=0x7efdec003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.168+0000 7efe0b400640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec003a30 0x7efdec003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.168+0000 7efe0b400640 1 -- 172.21.15.43:0/1894301870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec003520 con 0x7efdec0052a0 2024-01-16T15:07:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.168+0000 7efe0aa00640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec003a30 0x7efdec003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.168+0000 7efe0b400640 1 -- 172.21.15.43:0/1894301870 reap_dead start 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.169+0000 7efe0b400640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 0x7efdec00b980 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efe04001e20 tx=0x7efdec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542dd70aca4c2720 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.169+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04003b70 con 0x7efdec0052a0 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.169+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe04003cd0 con 0x7efdec0052a0 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.169+0000 7efe0a000640 1 -- 172.21.15.43:0/1894301870 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04005310 con 0x7efdec0052a0 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.169+0000 7efe0be00640 1 -- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 msgr2=0x7efdec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 0x7efdec00b980 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efe04001e20 tx=0x7efdec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 -- 172.21.15.43:0/1894301870 shutdown_connections 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 --2- 172.21.15.43:0/1894301870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdec0052a0 0x7efdec00b980 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 -- 172.21.15.43:0/1894301870 >> 172.21.15.43:0/1894301870 conn(0x7efe0c0cec10 msgr2=0x7efe0c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 -- 172.21.15.43:0/1894301870 shutdown_connections 2024-01-16T15:07:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.170+0000 7efe0be00640 1 -- 172.21.15.43:0/1894301870 wait complete. 2024-01-16T15:07:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.172+0000 7efe0be00640 1 Processor -- start 2024-01-16T15:07:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.172+0000 7efe0be00640 1 -- start start 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec0052a0 0x7efe0c1f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0ad480 0x7efe0c15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c0abbb0 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c15e060 con 0x7efdec0052a0 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c15cfc0 con 0x7efe0c0ad480 2024-01-16T15:07:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55080/0 (socket says 172.21.15.43:55080) 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0b400640 1 -- 172.21.15.43:0/2657574242 learned_addr learned my addr 172.21.15.43:0/2657574242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe0aa00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec0052a0 0x7efe0c1f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.173+0000 7efe03e00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0ad480 0x7efe0c15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 -- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0ad480 msgr2=0x7efe0c15c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0ad480 0x7efe0c15c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 -- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec0052a0 msgr2=0x7efe0c1f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec0052a0 0x7efe0c1f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 -- 172.21.15.43:0/2657574242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c15d2a0 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0b400640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7efe04001f40 tx=0x7efe040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04037660 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe040377c0 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04037b70 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.174+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c15d580 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.175+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c15db40 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.176+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c04e840 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.177+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7efe04037cd0 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.178+0000 7efe02a00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 0x7efde406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.178+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efe040aa0f0 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.178+0000 7efe0aa00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 0x7efde406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:25.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.179+0000 7efe0aa00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 0x7efde406e080 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000c00 tx=0x7efdfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:25.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.184+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe04076780 con 0x7efe0c0ab1c0 2024-01-16T15:07:25.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:25.420+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efe0c0b2460 con 0x7efe0c0ab1c0 2024-01-16T15:07:26.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.032+0000 7efe02a00640 1 -- 172.21.15.43:0/2657574242 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7efe040470b0 con 0x7efe0c0ab1c0 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.036+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 msgr2=0x7efde406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.036+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 0x7efde406e080 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000c00 tx=0x7efdfc009040 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.036+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 msgr2=0x7efe0c0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.036+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7efe04001f40 tx=0x7efe040014a0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 shutdown_connections 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efde406bbc0 0x7efde406e080 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe0c0ab1c0 0x7efe0c0ab670 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0ad480 0x7efe0c15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 --2- 172.21.15.43:0/2657574242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdec0052a0 0x7efe0c1f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 >> 172.21.15.43:0/2657574242 conn(0x7efe0c0cec10 msgr2=0x7efe0c0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 shutdown_connections 2024-01-16T15:07:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.037+0000 7efe0be00640 1 -- 172.21.15.43:0/2657574242 wait complete. 2024-01-16T15:07:26.035 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:07:26.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:07:26.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.290+0000 7f90d0c00640 1 Processor -- start 2024-01-16T15:07:26.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.290+0000 7f90d0c00640 1 -- start start 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90d0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f90cc09cb30 0x7f90cc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90d0c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc0adb60 0x7f90cc0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90d0c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc15b730 con 0x7f90cc15b350 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90d0c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc0ac740 con 0x7f90cc09cb30 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90d0c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0ac460 con 0x7f90cc0adb60 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90c3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f90cc09cb30 0x7f90cc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90c3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f90cc09cb30 0x7f90cc0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55090/0 (socket says 172.21.15.43:55090) 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.292+0000 7f90c3e00640 1 -- 172.21.15.43:0/109213519 learned_addr learned my addr 172.21.15.43:0/109213519 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.293+0000 7f90ca800640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc0adb60 0x7f90cc0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.293+0000 7f90c3e00640 1 -- 172.21.15.43:0/109213519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc0adb60 msgr2=0x7f90cc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:26.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.293+0000 7f90c3e00640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc0adb60 0x7f90cc0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.293+0000 7f90c3e00640 1 -- 172.21.15.43:0/109213519 >> v1:172.21.15.69:6789/0 conn(0x7f90cc15b350 legacy=0x7f90cc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:26.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.293+0000 7f90c3e00640 1 -- 172.21.15.43:0/109213519 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0b2a50 con 0x7f90cc09cb30 2024-01-16T15:07:26.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.294+0000 7f90c3e00640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc0abe80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009870 tx=0x7f90b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c593a7e7aebd31 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.295+0000 7f90c9400640 1 -- 172.21.15.43:0/109213519 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4031640 con 0x7f90cc09cb30 2024-01-16T15:07:26.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.295+0000 7f90c9400640 1 -- 172.21.15.43:0/109213519 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b40317a0 con 0x7f90cc09cb30 2024-01-16T15:07:26.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.295+0000 7f90c9400640 1 -- 172.21.15.43:0/109213519 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4031b30 con 0x7f90cc09cb30 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.295+0000 7f90d0c00640 1 -- 172.21.15.43:0/109213519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 msgr2=0x7f90cc0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.295+0000 7f90d0c00640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc0abe80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009870 tx=0x7f90b402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 -- 172.21.15.43:0/109213519 shutdown_connections 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc0abe80 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 --2- 172.21.15.43:0/109213519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc0adb60 0x7f90cc0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 -- 172.21.15.43:0/109213519 >> 172.21.15.43:0/109213519 conn(0x7f90cc0b0ba0 msgr2=0x7f90cc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 -- 172.21.15.43:0/109213519 shutdown_connections 2024-01-16T15:07:26.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.296+0000 7f90d0c00640 1 -- 172.21.15.43:0/109213519 wait complete. 2024-01-16T15:07:26.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.297+0000 7f90d0c00640 1 Processor -- start 2024-01-16T15:07:26.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.298+0000 7f90d0c00640 1 -- start start 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90cc0adb60 0x7f90cc14bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc15b350 0x7f90cc0af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc152e50 con 0x7f90cc09cb30 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc14c1e0 con 0x7f90cc0adb60 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc09bf60 con 0x7f90cc15b350 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55104/0 (socket says 172.21.15.43:55104) 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 -- 172.21.15.43:0/1408738896 learned_addr learned my addr 172.21.15.43:0/1408738896 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:26.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90ca800640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc15b350 0x7f90cc0af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c3e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90cc0adb60 0x7f90cc14bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 -- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc15b350 msgr2=0x7f90cc0af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc15b350 0x7f90cc0af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 -- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90cc0adb60 msgr2=0x7f90cc14bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90cc0adb60 0x7f90cc14bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 -- 172.21.15.43:0/1408738896 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40094d0 con 0x7f90cc09cb30 2024-01-16T15:07:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c9e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c430 tx=0x7f90bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc011790 con 0x7f90cc09cb30 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc0118f0 con 0x7f90cc09cb30 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc011ce0 con 0x7f90cc09cb30 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.300+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc09c240 con 0x7f90cc09cb30 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.301+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc153340 con 0x7f90cc09cb30 2024-01-16T15:07:26.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.302+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc09c430 con 0x7f90cc09cb30 2024-01-16T15:07:26.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.303+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f90bc01c460 con 0x7f90cc09cb30 2024-01-16T15:07:26.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.304+0000 7f90c2000640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 0x7f90a406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.304+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f90bc0883f0 con 0x7f90cc09cb30 2024-01-16T15:07:26.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.304+0000 7f90c3e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 0x7f90a406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.305+0000 7f90c3e00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 0x7f90a406e080 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90b4036000 tx=0x7f90b403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.308+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90cc09c430 con 0x7f90cc09cb30 2024-01-16T15:07:26.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:26.544+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f90cc09c430 con 0x7f90cc09cb30 2024-01-16T15:07:26.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:26.536 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.043+0000 7f90c2000640 1 -- 172.21.15.43:0/1408738896 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f90cc09c430 con 0x7f90cc09cb30 2024-01-16T15:07:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 msgr2=0x7f90a406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 0x7f90a406e080 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90b4036000 tx=0x7f90b403c040 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 msgr2=0x7f90cc1528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c430 tx=0x7f90bc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 shutdown_connections 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90a406bbc0 0x7f90a406e080 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90cc0adb60 0x7f90cc14bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90cc15b350 0x7f90cc0af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 --2- 172.21.15.43:0/1408738896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90cc09cb30 0x7f90cc1528f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 >> 172.21.15.43:0/1408738896 conn(0x7f90cc0b0ba0 msgr2=0x7f90cc14ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 shutdown_connections 2024-01-16T15:07:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.048+0000 7f90d0c00640 1 -- 172.21.15.43:0/1408738896 wait complete. 2024-01-16T15:07:27.050 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:07:27.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:27.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.302+0000 7fb0de400640 1 Processor -- start 2024-01-16T15:07:27.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.302+0000 7fb0de400640 1 -- start start 2024-01-16T15:07:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0de400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0d80abff0 0x7fb0d8071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0de400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d813da30 con 0x7fb0d80b39a0 2024-01-16T15:07:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0de400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80ac560 con 0x7fb0d815a100 2024-01-16T15:07:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0de400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d80ac840 con 0x7fb0d80abff0 2024-01-16T15:07:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0d80abff0 0x7fb0d8071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb0d80abff0 0x7fb0d8071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55120/0 (socket says 172.21.15.43:55120) 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 -- 172.21.15.43:0/3583735590 learned_addr learned my addr 172.21.15.43:0/3583735590 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0ce800640 1 --1- 172.21.15.43:0/3583735590 >> v1:172.21.15.43:6789/0 conn(0x7fb0d80b39a0 0x7fb0d8071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44904/0 (socket says 172.21.15.43:44904) 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 -- 172.21.15.43:0/3583735590 >> v1:172.21.15.69:6789/0 conn(0x7fb0d815a100 legacy=0x7fb0d80b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 -- 172.21.15.43:0/3583735590 >> v1:172.21.15.43:6789/0 conn(0x7fb0d80b39a0 legacy=0x7fb0d8071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.304+0000 7fb0d7e00640 1 -- 172.21.15.43:0/3583735590 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8070650 con 0x7fb0d80abff0 2024-01-16T15:07:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.305+0000 7fb0d7e00640 1 --2- 172.21.15.43:0/3583735590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d8071840 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009e50 tx=0x7fb0d002e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c314238c3206430f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.305+0000 7fb0d6a00640 1 -- 172.21.15.43:0/3583735590 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d002edc0 con 0x7fb0d80abff0 2024-01-16T15:07:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.305+0000 7fb0d6a00640 1 -- 172.21.15.43:0/3583735590 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d00363f0 con 0x7fb0d80abff0 2024-01-16T15:07:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.305+0000 7fb0d6a00640 1 -- 172.21.15.43:0/3583735590 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0036780 con 0x7fb0d80abff0 2024-01-16T15:07:27.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 -- 172.21.15.43:0/3583735590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 msgr2=0x7fb0d8071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 --2- 172.21.15.43:0/3583735590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d8071840 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009e50 tx=0x7fb0d002e220 comp rx=0 tx=0).stop 2024-01-16T15:07:27.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 -- 172.21.15.43:0/3583735590 shutdown_connections 2024-01-16T15:07:27.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 --2- 172.21.15.43:0/3583735590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d8071840 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 -- 172.21.15.43:0/3583735590 >> 172.21.15.43:0/3583735590 conn(0x7fb0d80b0d80 msgr2=0x7fb0d80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 -- 172.21.15.43:0/3583735590 shutdown_connections 2024-01-16T15:07:27.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.306+0000 7fb0de400640 1 -- 172.21.15.43:0/3583735590 wait complete. 2024-01-16T15:07:27.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.308+0000 7fb0de400640 1 Processor -- start 2024-01-16T15:07:27.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.308+0000 7fb0de400640 1 -- start start 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 0x7fb0d80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 0x7fb0d80cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d815d3e0 con 0x7fb0d80abff0 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80c9010 con 0x7fb0d815a100 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0de400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80c8400 con 0x7fb0d80b39a0 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.309+0000 7fb0d7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55122/0 (socket says 172.21.15.43:55122) 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 -- 172.21.15.43:0/3982551900 learned_addr learned my addr 172.21.15.43:0/3982551900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0ce800640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 0x7fb0d80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7e00640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 0x7fb0d80cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 -- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 msgr2=0x7fb0d80cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 0x7fb0d80cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 -- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 msgr2=0x7fb0d80cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 0x7fb0d80cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 -- 172.21.15.43:0/3982551900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0009b30 con 0x7fb0d80abff0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0ce800640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 0x7fb0d80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7e00640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 0x7fb0d80cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.310+0000 7fb0d7400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c450 tx=0x7fb0c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.311+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80052d0 con 0x7fb0d80abff0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.311+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8005430 con 0x7fb0d80abff0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.311+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8012870 con 0x7fb0d80abff0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.311+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d80c86e0 con 0x7fb0d80abff0 2024-01-16T15:07:27.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.311+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d80c8ca0 con 0x7fb0d80abff0 2024-01-16T15:07:27.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.312+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8071e70 con 0x7fb0d80abff0 2024-01-16T15:07:27.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.313+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb0c80129d0 con 0x7fb0d80abff0 2024-01-16T15:07:27.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.314+0000 7fb0cfe00640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 0x7fb0b006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.314+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb0c8087f90 con 0x7fb0d80abff0 2024-01-16T15:07:27.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.315+0000 7fb0ce800640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 0x7fb0b006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.315+0000 7fb0ce800640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 0x7fb0b006e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80b0d50 tx=0x7fb0c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.319+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0c80548b0 con 0x7fb0d80abff0 2024-01-16T15:07:27.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.552+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d8074770 con 0x7fb0d80abff0 2024-01-16T15:07:27.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.553+0000 7fb0cfe00640 1 -- 172.21.15.43:0/3982551900 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fb0c8057ed0 con 0x7fb0d80abff0 2024-01-16T15:07:27.533 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:27.533 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5540,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1054183394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1054183394},{"type":"v1","addr":"172.21.15.69:6835","nonce":1054183394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/3255994502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3255994502},{"type":"v1","addr":"172.21.15.43:6836","nonce":3255994502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5546,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1771464625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":1771464625},{"type":"v1","addr":"172.21.15.43:6837","nonce":1771464625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:07:23.471150+0000","modified":"2024-01-16T15:07:26.545904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6085":{"gid":6085,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/1602463840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1602463840},{"type":"v1","addr":"172.21.15.69:6834","nonce":1602463840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:07:27.533 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 46 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.556+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 msgr2=0x7fb0b006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.556+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 0x7fb0b006e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80b0d50 tx=0x7fb0c0009040 comp rx=0 tx=0).stop 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.556+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 msgr2=0x7fb0d81f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.556+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c450 tx=0x7fb0c800c910 comp rx=0 tx=0).stop 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 shutdown_connections 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb0b006bbc0 0x7fb0b006e080 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d815a100 0x7fb0d80cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0d80b39a0 0x7fb0d80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 --2- 172.21.15.43:0/3982551900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0d80abff0 0x7fb0d81f2c60 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 >> 172.21.15.43:0/3982551900 conn(0x7fb0d80b0d80 msgr2=0x7fb0d80b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 shutdown_connections 2024-01-16T15:07:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.557+0000 7fb0de400640 1 -- 172.21.15.43:0/3982551900 wait complete. 2024-01-16T15:07:27.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:27.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.799+0000 7f3a76200640 1 Processor -- start 2024-01-16T15:07:27.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.800+0000 7f3a76200640 1 -- start start 2024-01-16T15:07:27.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.801+0000 7f3a76200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7013dfd0 0x7f3a7015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a76200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3a700cf050 0x7f3a700cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a76200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a7006c130 0x7f3a7006c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a76200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700cfa10 con 0x7f3a700cf050 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a76200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7015fa70 con 0x7f3a7006c130 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a76200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7015ee60 con 0x7f3a7013dfd0 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a67e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3a700cf050 0x7f3a700cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a67e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3a700cf050 0x7f3a700cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55130/0 (socket says 172.21.15.43:55130) 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a67e00640 1 -- 172.21.15.43:0/394029871 learned_addr learned my addr 172.21.15.43:0/394029871 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a74e00640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7013dfd0 0x7f3a7015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a75800640 1 --2- 172.21.15.43:0/394029871 >> v2:172.21.15.69:3300/0 conn(0x7f3a7006c130 0x7f3a7006c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.802+0000 7f3a67e00640 1 -- 172.21.15.43:0/394029871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7013dfd0 msgr2=0x7f3a7015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.803+0000 7f3a67e00640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7013dfd0 0x7f3a7015be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.803+0000 7f3a67e00640 1 -- 172.21.15.43:0/394029871 >> v2:172.21.15.69:3300/0 conn(0x7f3a7006c130 msgr2=0x7f3a7006c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:27.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.803+0000 7f3a67e00640 1 --2- 172.21.15.43:0/394029871 >> v2:172.21.15.69:3300/0 conn(0x7f3a7006c130 0x7f3a7006c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.803+0000 7f3a67e00640 1 -- 172.21.15.43:0/394029871 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7015f140 con 0x7f3a700cf050 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.804+0000 7f3a67e00640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700cf4d0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3a7006d520 tx=0x7f3a5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a925d1eabf5c67fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.804+0000 7f3a6fe00640 1 -- 172.21.15.43:0/394029871 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58031870 con 0x7f3a700cf050 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.804+0000 7f3a6fe00640 1 -- 172.21.15.43:0/394029871 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a580319d0 con 0x7f3a700cf050 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.804+0000 7f3a6fe00640 1 -- 172.21.15.43:0/394029871 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58031d60 con 0x7f3a700cf050 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 -- 172.21.15.43:0/394029871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 msgr2=0x7f3a700cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700cf4d0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3a7006d520 tx=0x7f3a5802b910 comp rx=0 tx=0).stop 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 -- 172.21.15.43:0/394029871 shutdown_connections 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 --2- 172.21.15.43:0/394029871 >> v2:172.21.15.69:3300/0 conn(0x7f3a7006c130 0x7f3a7006c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700cf4d0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 --2- 172.21.15.43:0/394029871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7013dfd0 0x7f3a7015be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 -- 172.21.15.43:0/394029871 >> 172.21.15.43:0/394029871 conn(0x7f3a7006a860 msgr2=0x7f3a7006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 -- 172.21.15.43:0/394029871 shutdown_connections 2024-01-16T15:07:27.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.805+0000 7f3a76200640 1 -- 172.21.15.43:0/394029871 wait complete. 2024-01-16T15:07:27.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.807+0000 7f3a76200640 1 Processor -- start 2024-01-16T15:07:27.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.807+0000 7f3a76200640 1 -- start start 2024-01-16T15:07:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 0x7f3a70157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 0x7f3a700b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70158030 con 0x7f3a700cf050 2024-01-16T15:07:27.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700cdae0 con 0x7f3a7013dfd0 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a76200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700c8400 con 0x7f3a7006c130 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55136/0 (socket says 172.21.15.43:55136) 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 -- 172.21.15.43:0/3931987760 learned_addr learned my addr 172.21.15.43:0/3931987760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a75800640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 0x7f3a700b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a74e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 0x7f3a70157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 -- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 msgr2=0x7f3a70157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 0x7f3a70157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 -- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 msgr2=0x7f3a700b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.808+0000 7f3a67e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 0x7f3a700b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a67e00640 1 -- 172.21.15.43:0/3931987760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580094d0 con 0x7f3a700cf050 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a74e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 0x7f3a70157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a75800640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 0x7f3a700b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a67e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3a5802b950 tx=0x7f3a580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5803ad30 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a580443f0 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a580447a0 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70158880 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.809+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a700caed0 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.811+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7004e840 con 0x7f3a700cf050 2024-01-16T15:07:27.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.812+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f3a58044900 con 0x7f3a700cf050 2024-01-16T15:07:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.813+0000 7f3a6d600640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 0x7f3a4806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.813+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3a5802d040 con 0x7f3a700cf050 2024-01-16T15:07:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.813+0000 7f3a74e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 0x7f3a4806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.814+0000 7f3a74e00640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 0x7f3a4806e080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006fd0 tx=0x7f3a60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:27.817+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a58077420 con 0x7f3a700cf050 2024-01-16T15:07:27.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.017+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a700b3a30 con 0x7f3a700cf050 2024-01-16T15:07:28.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.022+0000 7f3a6d600640 1 -- 172.21.15.43:0/3931987760 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16563 (secure 0 0 0) 0x7f3a58035020 con 0x7f3a700cf050 2024-01-16T15:07:28.000 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:28.000 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":72,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:07:24.992575+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:07:21.386131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:07:22.484863+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:28.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 msgr2=0x7f3a4806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 0x7f3a4806e080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006fd0 tx=0x7f3a60008040 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 msgr2=0x7f3a700c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3a5802b950 tx=0x7f3a580046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 shutdown_connections 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3a4806bbc0 0x7f3a4806e080 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a7013dfd0 0x7f3a700b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a700cf050 0x7f3a700c7ec0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 --2- 172.21.15.43:0/3931987760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a7006c130 0x7f3a70157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 >> 172.21.15.43:0/3931987760 conn(0x7f3a7006a860 msgr2=0x7f3a7006c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 shutdown_connections 2024-01-16T15:07:28.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.025+0000 7f3a76200640 1 -- 172.21.15.43:0/3931987760 wait complete. 2024-01-16T15:07:28.022 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:07:28.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:07:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.242+0000 7fc41a000640 1 Processor -- start 2024-01-16T15:07:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.243+0000 7fc41a000640 1 -- start start 2024-01-16T15:07:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.244+0000 7fc41a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.244+0000 7fc41a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4140d14a0 0x7fc4140d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.244+0000 7fc41a000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140cf0c0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.244+0000 7fc41a000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4140b01b0 con 0x7fc4140d14a0 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.244+0000 7fc41a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41415d990 con 0x7fc41415d5b0 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.245+0000 7fc418c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.245+0000 7fc418c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:41874/0 (socket says 172.21.15.43:41874) 2024-01-16T15:07:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.245+0000 7fc418c00640 1 -- 172.21.15.43:0/815056437 learned_addr learned my addr 172.21.15.43:0/815056437 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.245+0000 7fc413e00640 1 --2- 172.21.15.43:0/815056437 >> v2:172.21.15.69:3300/0 conn(0x7fc4140d14a0 0x7fc4140d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.246+0000 7fc418c00640 1 -- 172.21.15.43:0/815056437 >> v2:172.21.15.69:3300/0 conn(0x7fc4140d14a0 msgr2=0x7fc4140d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.246+0000 7fc418c00640 1 --2- 172.21.15.43:0/815056437 >> v2:172.21.15.69:3300/0 conn(0x7fc4140d14a0 0x7fc4140d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.246+0000 7fc418c00640 1 -- 172.21.15.43:0/815056437 >> v1:172.21.15.69:6789/0 conn(0x7fc4140cf7a0 legacy=0x7fc4140cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.246+0000 7fc418c00640 1 -- 172.21.15.43:0/815056437 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140d1e40 con 0x7fc41415d5b0 2024-01-16T15:07:28.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.246+0000 7fc413400640 1 -- 172.21.15.43:0/815056437 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474564961 0 0) 0x7fc4140cf0c0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.248+0000 7fc418c00640 1 --2- 172.21.15.43:0/815056437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f2ede27df6615c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.248+0000 7fc413400640 1 -- 172.21.15.43:0/815056437 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404030840 con 0x7fc41415d5b0 2024-01-16T15:07:28.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc413400640 1 -- 172.21.15.43:0/815056437 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4040309a0 con 0x7fc41415d5b0 2024-01-16T15:07:28.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc413400640 1 -- 172.21.15.43:0/815056437 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404030d30 con 0x7fc41415d5b0 2024-01-16T15:07:28.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 -- 172.21.15.43:0/815056437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 msgr2=0x7fc4140d3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 --2- 172.21.15.43:0/815056437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40402b980 comp rx=0 tx=0).stop 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 -- 172.21.15.43:0/815056437 shutdown_connections 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 --2- 172.21.15.43:0/815056437 >> v2:172.21.15.69:3300/0 conn(0x7fc4140d14a0 0x7fc4140d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 --2- 172.21.15.43:0/815056437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41415d5b0 0x7fc4140d3990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 -- 172.21.15.43:0/815056437 >> 172.21.15.43:0/815056437 conn(0x7fc41409c3b0 msgr2=0x7fc4140d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 -- 172.21.15.43:0/815056437 shutdown_connections 2024-01-16T15:07:28.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.249+0000 7fc41a000640 1 -- 172.21.15.43:0/815056437 wait complete. 2024-01-16T15:07:28.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.251+0000 7fc41a000640 1 Processor -- start 2024-01-16T15:07:28.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.251+0000 7fc41a000640 1 -- start start 2024-01-16T15:07:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4140d14a0 0x7fc4140b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41415d5b0 0x7fc4140b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41415acd0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140cbe70 con 0x7fc41415d5b0 2024-01-16T15:07:28.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.252+0000 7fc41a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140cc150 con 0x7fc4140d14a0 2024-01-16T15:07:28.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc413e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4140d14a0 0x7fc4140b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55146/0 (socket says 172.21.15.43:55146) 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 -- 172.21.15.43:0/2125029126 learned_addr learned my addr 172.21.15.43:0/2125029126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc419600640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41415d5b0 0x7fc4140b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 -- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4140d14a0 msgr2=0x7fc4140b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4140d14a0 0x7fc4140b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 -- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41415d5b0 msgr2=0x7fc4140b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41415d5b0 0x7fc4140b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.253+0000 7fc418c00640 1 -- 172.21.15.43:0/2125029126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040094d0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc418c00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097c0 tx=0x7fc404004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40403e040 con 0x7fc4140cf7a0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404039c90 con 0x7fc4140cf7a0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4140cc430 con 0x7fc4140cf7a0 2024-01-16T15:07:28.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040443f0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.254+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4140cb140 con 0x7fc4140cf7a0 2024-01-16T15:07:28.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.256+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41404e840 con 0x7fc4140cf7a0 2024-01-16T15:07:28.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.256+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc40404d3b0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.258+0000 7fc410c00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 0x7fc3ec06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.258+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc4040aabb0 con 0x7fc4140cf7a0 2024-01-16T15:07:28.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.258+0000 7fc413e00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 0x7fc3ec06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.259+0000 7fc413e00640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 0x7fc3ec06e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc41415bd10 tx=0x7fc3fc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.263+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc404077390 con 0x7fc4140cf7a0 2024-01-16T15:07:28.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.454+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4140afe40 con 0x7fc3ec06bbc0 2024-01-16T15:07:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.479+0000 7fc410c00640 1 -- 172.21.15.43:0/2125029126 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286942 (secure 0 0 0) 0x7fc3fc011a30 con 0x7fc3ec06bbc0 2024-01-16T15:07:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:28.463 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.486+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 msgr2=0x7fc3ec06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.486+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 0x7fc3ec06e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc41415bd10 tx=0x7fc3fc009190 comp rx=0 tx=0).stop 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.486+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 msgr2=0x7fc41415a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.486+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097c0 tx=0x7fc404004730 comp rx=0 tx=0).stop 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 shutdown_connections 2024-01-16T15:07:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc3ec06bbc0 0x7fc3ec06e080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41415d5b0 0x7fc4140b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4140d14a0 0x7fc4140b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 --2- 172.21.15.43:0/2125029126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4140cf7a0 0x7fc41415a790 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 >> 172.21.15.43:0/2125029126 conn(0x7fc41409c3b0 msgr2=0x7fc41415de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 shutdown_connections 2024-01-16T15:07:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.487+0000 7fc41a000640 1 -- 172.21.15.43:0/2125029126 wait complete. 2024-01-16T15:07:28.484 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-01-16T15:07:26.994772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405964,"kb_used_data":7956,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568756,"statfs":{"total":773094113280,"available":772678406144,"internally_reserved":0,"allocated":8146944,"data_stored":4089988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":407424556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.016993"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045016+0000","last_change":"2024-01-16T15:07:24.036652+0000","last_active":"2024-01-16T15:07:24.045016+0000","last_peered":"2024-01-16T15:07:24.045016+0000","last_clean":"2024-01-16T15:07:24.045016+0000","last_became_active":"2024-01-16T15:07:24.036288+0000","last_became_peered":"2024-01-16T15:07:24.036288+0000","last_unstale":"2024-01-16T15:07:24.045016+0000","last_undegraded":"2024-01-16T15:07:24.045016+0000","last_fullsized":"2024-01-16T15:07:24.045016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:50.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.035110+0000","last_change":"2024-01-16T15:07:24.017571+0000","last_active":"2024-01-16T15:07:24.035110+0000","last_peered":"2024-01-16T15:07:24.035110+0000","last_clean":"2024-01-16T15:07:24.035110+0000","last_became_active":"2024-01-16T15:07:24.017183+0000","last_became_peered":"2024-01-16T15:07:24.017183+0000","last_unstale":"2024-01-16T15:07:24.035110+0000","last_undegraded":"2024-01-16T15:07:24.035110+0000","last_fullsized":"2024-01-16T15:07:24.035110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:59.209707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.031797+0000","last_change":"2024-01-16T15:07:24.029356+0000","last_active":"2024-01-16T15:07:24.031797+0000","last_peered":"2024-01-16T15:07:24.031797+0000","last_clean":"2024-01-16T15:07:24.031797+0000","last_became_active":"2024-01-16T15:07:24.028977+0000","last_became_peered":"2024-01-16T15:07:24.028977+0000","last_unstale":"2024-01-16T15:07:24.031797+0000","last_undegraded":"2024-01-16T15:07:24.031797+0000","last_fullsized":"2024-01-16T15:07:24.031797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:31.449130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001599+0000","last_change":"2024-01-16T15:07:24.037957+0000","last_active":"2024-01-16T15:07:25.001599+0000","last_peered":"2024-01-16T15:07:25.001599+0000","last_clean":"2024-01-16T15:07:25.001599+0000","last_became_active":"2024-01-16T15:07:24.037297+0000","last_became_peered":"2024-01-16T15:07:24.037297+0000","last_unstale":"2024-01-16T15:07:25.001599+0000","last_undegraded":"2024-01-16T15:07:25.001599+0000","last_fullsized":"2024-01-16T15:07:25.001599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:24.256412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999513+0000","last_change":"2024-01-16T15:07:24.027699+0000","last_active":"2024-01-16T15:07:24.999513+0000","last_peered":"2024-01-16T15:07:24.999513+0000","last_clean":"2024-01-16T15:07:24.999513+0000","last_became_active":"2024-01-16T15:07:24.027434+0000","last_became_peered":"2024-01-16T15:07:24.027434+0000","last_unstale":"2024-01-16T15:07:24.999513+0000","last_undegraded":"2024-01-16T15:07:24.999513+0000","last_fullsized":"2024-01-16T15:07:24.999513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:53.983350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.028767+0000","last_change":"2024-01-16T15:07:24.015524+0000","last_active":"2024-01-16T15:07:24.028767+0000","last_peered":"2024-01-16T15:07:24.028767+0000","last_clean":"2024-01-16T15:07:24.028767+0000","last_became_active":"2024-01-16T15:07:24.015042+0000","last_became_peered":"2024-01-16T15:07:24.015042+0000","last_unstale":"2024-01-16T15:07:24.028767+0000","last_undegraded":"2024-01-16T15:07:24.028767+0000","last_fullsized":"2024-01-16T15:07:24.028767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.655408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.030925+0000","last_change":"2024-01-16T15:07:24.025486+0000","last_active":"2024-01-16T15:07:24.030925+0000","last_peered":"2024-01-16T15:07:24.030925+0000","last_clean":"2024-01-16T15:07:24.030925+0000","last_became_active":"2024-01-16T15:07:24.025201+0000","last_became_peered":"2024-01-16T15:07:24.025201+0000","last_unstale":"2024-01-16T15:07:24.030925+0000","last_undegraded":"2024-01-16T15:07:24.030925+0000","last_fullsized":"2024-01-16T15:07:24.030925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:49.215397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038588+0000","last_change":"2024-01-16T15:07:24.027793+0000","last_active":"2024-01-16T15:07:24.038588+0000","last_peered":"2024-01-16T15:07:24.038588+0000","last_clean":"2024-01-16T15:07:24.038588+0000","last_became_active":"2024-01-16T15:07:24.027502+0000","last_became_peered":"2024-01-16T15:07:24.027502+0000","last_unstale":"2024-01-16T15:07:24.038588+0000","last_undegraded":"2024-01-16T15:07:24.038588+0000","last_fullsized":"2024-01-16T15:07:24.038588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:38.505806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999437+0000","last_change":"2024-01-16T15:07:24.032781+0000","last_active":"2024-01-16T15:07:24.999437+0000","last_peered":"2024-01-16T15:07:24.999437+0000","last_clean":"2024-01-16T15:07:24.999437+0000","last_became_active":"2024-01-16T15:07:24.032460+0000","last_became_peered":"2024-01-16T15:07:24.032460+0000","last_unstale":"2024-01-16T15:07:24.999437+0000","last_undegraded":"2024-01-16T15:07:24.999437+0000","last_fullsized":"2024-01-16T15:07:24.999437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:56.173386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003011+0000","last_change":"2024-01-16T15:07:24.031359+0000","last_active":"2024-01-16T15:07:25.003011+0000","last_peered":"2024-01-16T15:07:25.003011+0000","last_clean":"2024-01-16T15:07:25.003011+0000","last_became_active":"2024-01-16T15:07:24.029570+0000","last_became_peered":"2024-01-16T15:07:24.029570+0000","last_unstale":"2024-01-16T15:07:25.003011+0000","last_undegraded":"2024-01-16T15:07:25.003011+0000","last_fullsized":"2024-01-16T15:07:25.003011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:29.933024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.043484+0000","last_change":"2024-01-16T15:07:24.031503+0000","last_active":"2024-01-16T15:07:24.043484+0000","last_peered":"2024-01-16T15:07:24.043484+0000","last_clean":"2024-01-16T15:07:24.043484+0000","last_became_active":"2024-01-16T15:07:24.030065+0000","last_became_peered":"2024-01-16T15:07:24.030065+0000","last_unstale":"2024-01-16T15:07:24.043484+0000","last_undegraded":"2024-01-16T15:07:24.043484+0000","last_fullsized":"2024-01-16T15:07:24.043484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:16.331763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001685+0000","last_change":"2024-01-16T15:07:24.035513+0000","last_active":"2024-01-16T15:07:25.001685+0000","last_peered":"2024-01-16T15:07:25.001685+0000","last_clean":"2024-01-16T15:07:25.001685+0000","last_became_active":"2024-01-16T15:07:24.033958+0000","last_became_peered":"2024-01-16T15:07:24.033958+0000","last_unstale":"2024-01-16T15:07:25.001685+0000","last_undegraded":"2024-01-16T15:07:25.001685+0000","last_fullsized":"2024-01-16T15:07:25.001685+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:30.398205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.024859+0000","last_change":"2024-01-16T15:07:24.017333+0000","last_active":"2024-01-16T15:07:24.024859+0000","last_peered":"2024-01-16T15:07:24.024859+0000","last_clean":"2024-01-16T15:07:24.024859+0000","last_became_active":"2024-01-16T15:07:24.016711+0000","last_became_peered":"2024-01-16T15:07:24.016711+0000","last_unstale":"2024-01-16T15:07:24.024859+0000","last_undegraded":"2024-01-16T15:07:24.024859+0000","last_fullsized":"2024-01-16T15:07:24.024859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:30.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.000070+0000","last_change":"2024-01-16T15:07:24.035814+0000","last_active":"2024-01-16T15:07:25.000070+0000","last_peered":"2024-01-16T15:07:25.000070+0000","last_clean":"2024-01-16T15:07:25.000070+0000","last_became_active":"2024-01-16T15:07:24.034222+0000","last_became_peered":"2024-01-16T15:07:24.034222+0000","last_unstale":"2024-01-16T15:07:25.000070+0000","last_undegraded":"2024-01-16T15:07:25.000070+0000","last_fullsized":"2024-01-16T15:07:25.000070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:13:39.524640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.028925+0000","last_change":"2024-01-16T15:07:24.026978+0000","last_active":"2024-01-16T15:07:24.028925+0000","last_peered":"2024-01-16T15:07:24.028925+0000","last_clean":"2024-01-16T15:07:24.028925+0000","last_became_active":"2024-01-16T15:07:24.026705+0000","last_became_peered":"2024-01-16T15:07:24.026705+0000","last_unstale":"2024-01-16T15:07:24.028925+0000","last_undegraded":"2024-01-16T15:07:24.028925+0000","last_fullsized":"2024-01-16T15:07:24.028925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:26.803648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001365+0000","last_change":"2024-01-16T15:07:24.017363+0000","last_active":"2024-01-16T15:07:25.001365+0000","last_peered":"2024-01-16T15:07:25.001365+0000","last_clean":"2024-01-16T15:07:25.001365+0000","last_became_active":"2024-01-16T15:07:24.015531+0000","last_became_peered":"2024-01-16T15:07:24.015531+0000","last_unstale":"2024-01-16T15:07:25.001365+0000","last_undegraded":"2024-01-16T15:07:25.001365+0000","last_fullsized":"2024-01-16T15:07:25.001365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:09.853233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999584+0000","last_change":"2024-01-16T15:07:24.035335+0000","last_active":"2024-01-16T15:07:24.999584+0000","last_peered":"2024-01-16T15:07:24.999584+0000","last_clean":"2024-01-16T15:07:24.999584+0000","last_became_active":"2024-01-16T15:07:24.034988+0000","last_became_peered":"2024-01-16T15:07:24.034988+0000","last_unstale":"2024-01-16T15:07:24.999584+0000","last_undegraded":"2024-01-16T15:07:24.999584+0000","last_fullsized":"2024-01-16T15:07:24.999584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:21.644838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002219+0000","last_change":"2024-01-16T15:07:24.033655+0000","last_active":"2024-01-16T15:07:25.002219+0000","last_peered":"2024-01-16T15:07:25.002219+0000","last_clean":"2024-01-16T15:07:25.002219+0000","last_became_active":"2024-01-16T15:07:24.032653+0000","last_became_peered":"2024-01-16T15:07:24.032653+0000","last_unstale":"2024-01-16T15:07:25.002219+0000","last_undegraded":"2024-01-16T15:07:25.002219+0000","last_fullsized":"2024-01-16T15:07:25.002219+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.655408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037656+0000","last_change":"2024-01-16T15:07:24.026668+0000","last_active":"2024-01-16T15:07:24.037656+0000","last_peered":"2024-01-16T15:07:24.037656+0000","last_clean":"2024-01-16T15:07:24.037656+0000","last_became_active":"2024-01-16T15:07:24.026363+0000","last_became_peered":"2024-01-16T15:07:24.026363+0000","last_unstale":"2024-01-16T15:07:24.037656+0000","last_undegraded":"2024-01-16T15:07:24.037656+0000","last_fullsized":"2024-01-16T15:07:24.037656+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:47.962892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.027735+0000","last_change":"2024-01-16T15:07:24.025580+0000","last_active":"2024-01-16T15:07:24.027735+0000","last_peered":"2024-01-16T15:07:24.027735+0000","last_clean":"2024-01-16T15:07:24.027735+0000","last_became_active":"2024-01-16T15:07:24.023057+0000","last_became_peered":"2024-01-16T15:07:24.023057+0000","last_unstale":"2024-01-16T15:07:24.027735+0000","last_undegraded":"2024-01-16T15:07:24.027735+0000","last_fullsized":"2024-01-16T15:07:24.027735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:55.523923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002109+0000","last_change":"2024-01-16T15:07:24.011040+0000","last_active":"2024-01-16T15:07:25.002109+0000","last_peered":"2024-01-16T15:07:25.002109+0000","last_clean":"2024-01-16T15:07:25.002109+0000","last_became_active":"2024-01-16T15:07:24.010259+0000","last_became_peered":"2024-01-16T15:07:24.010259+0000","last_unstale":"2024-01-16T15:07:25.002109+0000","last_undegraded":"2024-01-16T15:07:25.002109+0000","last_fullsized":"2024-01-16T15:07:25.002109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:51.439554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001701+0000","last_change":"2024-01-16T15:07:24.030463+0000","last_active":"2024-01-16T15:07:25.001701+0000","last_peered":"2024-01-16T15:07:25.001701+0000","last_clean":"2024-01-16T15:07:25.001701+0000","last_became_active":"2024-01-16T15:07:24.029645+0000","last_became_peered":"2024-01-16T15:07:24.029645+0000","last_unstale":"2024-01-16T15:07:25.001701+0000","last_undegraded":"2024-01-16T15:07:25.001701+0000","last_fullsized":"2024-01-16T15:07:25.001701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:00.370425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044523+0000","last_change":"2024-01-16T15:07:24.033503+0000","last_active":"2024-01-16T15:07:24.044523+0000","last_peered":"2024-01-16T15:07:24.044523+0000","last_clean":"2024-01-16T15:07:24.044523+0000","last_became_active":"2024-01-16T15:07:24.032603+0000","last_became_peered":"2024-01-16T15:07:24.032603+0000","last_unstale":"2024-01-16T15:07:24.044523+0000","last_undegraded":"2024-01-16T15:07:24.044523+0000","last_fullsized":"2024-01-16T15:07:24.044523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:34.306686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.047829+0000","last_change":"2024-01-16T15:07:24.040015+0000","last_active":"2024-01-16T15:07:24.047829+0000","last_peered":"2024-01-16T15:07:24.047829+0000","last_clean":"2024-01-16T15:07:24.047829+0000","last_became_active":"2024-01-16T15:07:24.039767+0000","last_became_peered":"2024-01-16T15:07:24.039767+0000","last_unstale":"2024-01-16T15:07:24.047829+0000","last_undegraded":"2024-01-16T15:07:24.047829+0000","last_fullsized":"2024-01-16T15:07:24.047829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:40.006495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001758+0000","last_change":"2024-01-16T15:07:24.025703+0000","last_active":"2024-01-16T15:07:25.001758+0000","last_peered":"2024-01-16T15:07:25.001758+0000","last_clean":"2024-01-16T15:07:25.001758+0000","last_became_active":"2024-01-16T15:07:24.025367+0000","last_became_peered":"2024-01-16T15:07:24.025367+0000","last_unstale":"2024-01-16T15:07:25.001758+0000","last_undegraded":"2024-01-16T15:07:25.001758+0000","last_fullsized":"2024-01-16T15:07:25.001758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:30.667083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.040082+0000","last_change":"2024-01-16T15:07:24.038801+0000","last_active":"2024-01-16T15:07:24.040082+0000","last_peered":"2024-01-16T15:07:24.040082+0000","last_clean":"2024-01-16T15:07:24.040082+0000","last_became_active":"2024-01-16T15:07:24.038182+0000","last_became_peered":"2024-01-16T15:07:24.038182+0000","last_unstale":"2024-01-16T15:07:24.040082+0000","last_undegraded":"2024-01-16T15:07:24.040082+0000","last_fullsized":"2024-01-16T15:07:24.040082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:29.933024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.032540+0000","last_change":"2024-01-16T15:07:24.029437+0000","last_active":"2024-01-16T15:07:24.032540+0000","last_peered":"2024-01-16T15:07:24.032540+0000","last_clean":"2024-01-16T15:07:24.032540+0000","last_became_active":"2024-01-16T15:07:24.027019+0000","last_became_peered":"2024-01-16T15:07:24.027019+0000","last_unstale":"2024-01-16T15:07:24.032540+0000","last_undegraded":"2024-01-16T15:07:24.032540+0000","last_fullsized":"2024-01-16T15:07:24.032540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:01.387317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002626+0000","last_change":"2024-01-16T15:07:24.026705+0000","last_active":"2024-01-16T15:07:25.002626+0000","last_peered":"2024-01-16T15:07:25.002626+0000","last_clean":"2024-01-16T15:07:25.002626+0000","last_became_active":"2024-01-16T15:07:24.026434+0000","last_became_peered":"2024-01-16T15:07:24.026434+0000","last_unstale":"2024-01-16T15:07:25.002626+0000","last_undegraded":"2024-01-16T15:07:25.002626+0000","last_fullsized":"2024-01-16T15:07:25.002626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:52.675347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.032575+0000","last_change":"2024-01-16T15:07:24.030383+0000","last_active":"2024-01-16T15:07:24.032575+0000","last_peered":"2024-01-16T15:07:24.032575+0000","last_clean":"2024-01-16T15:07:24.032575+0000","last_became_active":"2024-01-16T15:07:24.030113+0000","last_became_peered":"2024-01-16T15:07:24.030113+0000","last_unstale":"2024-01-16T15:07:24.032575+0000","last_undegraded":"2024-01-16T15:07:24.032575+0000","last_fullsized":"2024-01-16T15:07:24.032575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:52.147402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.039812+0000","last_change":"2024-01-16T15:07:24.038390+0000","last_active":"2024-01-16T15:07:24.039812+0000","last_peered":"2024-01-16T15:07:24.039812+0000","last_clean":"2024-01-16T15:07:24.039812+0000","last_became_active":"2024-01-16T15:07:24.038085+0000","last_became_peered":"2024-01-16T15:07:24.038085+0000","last_unstale":"2024-01-16T15:07:24.039812+0000","last_undegraded":"2024-01-16T15:07:24.039812+0000","last_fullsized":"2024-01-16T15:07:24.039812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:30.667083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.039666+0000","last_change":"2024-01-16T15:07:24.035954+0000","last_active":"2024-01-16T15:07:24.039666+0000","last_peered":"2024-01-16T15:07:24.039666+0000","last_clean":"2024-01-16T15:07:24.039666+0000","last_became_active":"2024-01-16T15:07:24.035648+0000","last_became_peered":"2024-01-16T15:07:24.035648+0000","last_unstale":"2024-01-16T15:07:24.039666+0000","last_undegraded":"2024-01-16T15:07:24.039666+0000","last_fullsized":"2024-01-16T15:07:24.039666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:00.370425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.033712+0000","last_change":"2024-01-16T15:07:24.016680+0000","last_active":"2024-01-16T15:07:24.033712+0000","last_peered":"2024-01-16T15:07:24.033712+0000","last_clean":"2024-01-16T15:07:24.033712+0000","last_became_active":"2024-01-16T15:07:24.015079+0000","last_became_peered":"2024-01-16T15:07:24.015079+0000","last_unstale":"2024-01-16T15:07:24.033712+0000","last_undegraded":"2024-01-16T15:07:24.033712+0000","last_fullsized":"2024-01-16T15:07:24.033712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:11.466328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.047265+0000","last_change":"2024-01-16T15:07:24.039621+0000","last_active":"2024-01-16T15:07:24.047265+0000","last_peered":"2024-01-16T15:07:24.047265+0000","last_clean":"2024-01-16T15:07:24.047265+0000","last_became_active":"2024-01-16T15:07:24.039274+0000","last_became_peered":"2024-01-16T15:07:24.039274+0000","last_unstale":"2024-01-16T15:07:24.047265+0000","last_undegraded":"2024-01-16T15:07:24.047265+0000","last_fullsized":"2024-01-16T15:07:24.047265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:40:43.109312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049135+0000","last_change":"2024-01-16T15:07:23.023387+0000","last_active":"2024-01-16T15:07:24.049135+0000","last_peered":"2024-01-16T15:07:24.049135+0000","last_clean":"2024-01-16T15:07:24.049135+0000","last_became_active":"2024-01-16T15:07:23.023088+0000","last_became_peered":"2024-01-16T15:07:23.023088+0000","last_unstale":"2024-01-16T15:07:24.049135+0000","last_undegraded":"2024-01-16T15:07:24.049135+0000","last_fullsized":"2024-01-16T15:07:24.049135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:34.997413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999737+0000","last_change":"2024-01-16T15:07:24.032440+0000","last_active":"2024-01-16T15:07:24.999737+0000","last_peered":"2024-01-16T15:07:24.999737+0000","last_clean":"2024-01-16T15:07:24.999737+0000","last_became_active":"2024-01-16T15:07:24.029314+0000","last_became_peered":"2024-01-16T15:07:24.029314+0000","last_unstale":"2024-01-16T15:07:24.999737+0000","last_undegraded":"2024-01-16T15:07:24.999737+0000","last_fullsized":"2024-01-16T15:07:24.999737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.125133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.042534+0000","last_change":"2024-01-16T15:07:23.008387+0000","last_active":"2024-01-16T15:07:24.042534+0000","last_peered":"2024-01-16T15:07:24.042534+0000","last_clean":"2024-01-16T15:07:24.042534+0000","last_became_active":"2024-01-16T15:07:23.007916+0000","last_became_peered":"2024-01-16T15:07:23.007916+0000","last_unstale":"2024-01-16T15:07:24.042534+0000","last_undegraded":"2024-01-16T15:07:24.042534+0000","last_fullsized":"2024-01-16T15:07:24.042534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:27.067987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038522+0000","last_change":"2024-01-16T15:07:24.032667+0000","last_active":"2024-01-16T15:07:24.038522+0000","last_peered":"2024-01-16T15:07:24.038522+0000","last_clean":"2024-01-16T15:07:24.038522+0000","last_became_active":"2024-01-16T15:07:24.032097+0000","last_became_peered":"2024-01-16T15:07:24.032097+0000","last_unstale":"2024-01-16T15:07:24.038522+0000","last_undegraded":"2024-01-16T15:07:24.038522+0000","last_fullsized":"2024-01-16T15:07:24.038522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:19.018624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002127+0000","last_change":"2024-01-16T15:07:23.012327+0000","last_active":"2024-01-16T15:07:25.002127+0000","last_peered":"2024-01-16T15:07:25.002127+0000","last_clean":"2024-01-16T15:07:25.002127+0000","last_became_active":"2024-01-16T15:07:23.011997+0000","last_became_peered":"2024-01-16T15:07:23.011997+0000","last_unstale":"2024-01-16T15:07:25.002127+0000","last_undegraded":"2024-01-16T15:07:25.002127+0000","last_fullsized":"2024-01-16T15:07:25.002127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:00.664016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045318+0000","last_change":"2024-01-16T15:07:24.036752+0000","last_active":"2024-01-16T15:07:24.045318+0000","last_peered":"2024-01-16T15:07:24.045318+0000","last_clean":"2024-01-16T15:07:24.045318+0000","last_became_active":"2024-01-16T15:07:24.035229+0000","last_became_peered":"2024-01-16T15:07:24.035229+0000","last_unstale":"2024-01-16T15:07:24.045318+0000","last_undegraded":"2024-01-16T15:07:24.045318+0000","last_fullsized":"2024-01-16T15:07:24.045318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:15.610651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049180+0000","last_change":"2024-01-16T15:07:23.016160+0000","last_active":"2024-01-16T15:07:24.049180+0000","last_peered":"2024-01-16T15:07:24.049180+0000","last_clean":"2024-01-16T15:07:24.049180+0000","last_became_active":"2024-01-16T15:07:23.015624+0000","last_became_peered":"2024-01-16T15:07:23.015624+0000","last_unstale":"2024-01-16T15:07:24.049180+0000","last_undegraded":"2024-01-16T15:07:24.049180+0000","last_fullsized":"2024-01-16T15:07:24.049180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:57.033174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038754+0000","last_change":"2024-01-16T15:07:24.031127+0000","last_active":"2024-01-16T15:07:24.038754+0000","last_peered":"2024-01-16T15:07:24.038754+0000","last_clean":"2024-01-16T15:07:24.038754+0000","last_became_active":"2024-01-16T15:07:24.030845+0000","last_became_peered":"2024-01-16T15:07:24.030845+0000","last_unstale":"2024-01-16T15:07:24.038754+0000","last_undegraded":"2024-01-16T15:07:24.038754+0000","last_fullsized":"2024-01-16T15:07:24.038754+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:09.853233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.043922+0000","last_change":"2024-01-16T15:07:23.011619+0000","last_active":"2024-01-16T15:07:24.043922+0000","last_peered":"2024-01-16T15:07:24.043922+0000","last_clean":"2024-01-16T15:07:24.043922+0000","last_became_active":"2024-01-16T15:07:23.011220+0000","last_became_peered":"2024-01-16T15:07:23.011220+0000","last_unstale":"2024-01-16T15:07:24.043922+0000","last_undegraded":"2024-01-16T15:07:24.043922+0000","last_fullsized":"2024-01-16T15:07:24.043922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:36.489899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002470+0000","last_change":"2024-01-16T15:07:24.025622+0000","last_active":"2024-01-16T15:07:25.002470+0000","last_peered":"2024-01-16T15:07:25.002470+0000","last_clean":"2024-01-16T15:07:25.002470+0000","last_became_active":"2024-01-16T15:07:24.025303+0000","last_became_peered":"2024-01-16T15:07:24.025303+0000","last_unstale":"2024-01-16T15:07:25.002470+0000","last_undegraded":"2024-01-16T15:07:25.002470+0000","last_fullsized":"2024-01-16T15:07:25.002470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:58.049449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048652+0000","last_change":"2024-01-16T15:07:23.014210+0000","last_active":"2024-01-16T15:07:24.048652+0000","last_peered":"2024-01-16T15:07:24.048652+0000","last_clean":"2024-01-16T15:07:24.048652+0000","last_became_active":"2024-01-16T15:07:23.005942+0000","last_became_peered":"2024-01-16T15:07:23.005942+0000","last_unstale":"2024-01-16T15:07:24.048652+0000","last_undegraded":"2024-01-16T15:07:24.048652+0000","last_fullsized":"2024-01-16T15:07:24.048652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:25:32.507386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998727+0000","last_change":"2024-01-16T15:07:24.030303+0000","last_active":"2024-01-16T15:07:24.998727+0000","last_peered":"2024-01-16T15:07:24.998727+0000","last_clean":"2024-01-16T15:07:24.998727+0000","last_became_active":"2024-01-16T15:07:24.027753+0000","last_became_peered":"2024-01-16T15:07:24.027753+0000","last_unstale":"2024-01-16T15:07:24.998727+0000","last_undegraded":"2024-01-16T15:07:24.998727+0000","last_fullsized":"2024-01-16T15:07:24.998727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:44.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001337+0000","last_change":"2024-01-16T15:07:23.017741+0000","last_active":"2024-01-16T15:07:25.001337+0000","last_peered":"2024-01-16T15:07:25.001337+0000","last_clean":"2024-01-16T15:07:25.001337+0000","last_became_active":"2024-01-16T15:07:23.017445+0000","last_became_peered":"2024-01-16T15:07:23.017445+0000","last_unstale":"2024-01-16T15:07:25.001337+0000","last_undegraded":"2024-01-16T15:07:25.001337+0000","last_fullsized":"2024-01-16T15:07:25.001337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:29.911078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.026466+0000","last_change":"2024-01-16T15:07:24.024151+0000","last_active":"2024-01-16T15:07:24.026466+0000","last_peered":"2024-01-16T15:07:24.026466+0000","last_clean":"2024-01-16T15:07:24.026466+0000","last_became_active":"2024-01-16T15:07:24.023824+0000","last_became_peered":"2024-01-16T15:07:24.023824+0000","last_unstale":"2024-01-16T15:07:24.026466+0000","last_undegraded":"2024-01-16T15:07:24.026466+0000","last_fullsized":"2024-01-16T15:07:24.026466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:19.908915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998695+0000","last_change":"2024-01-16T15:07:23.012897+0000","last_active":"2024-01-16T15:07:24.998695+0000","last_peered":"2024-01-16T15:07:24.998695+0000","last_clean":"2024-01-16T15:07:24.998695+0000","last_became_active":"2024-01-16T15:07:23.012628+0000","last_became_peered":"2024-01-16T15:07:23.012628+0000","last_unstale":"2024-01-16T15:07:24.998695+0000","last_undegraded":"2024-01-16T15:07:24.998695+0000","last_fullsized":"2024-01-16T15:07:24.998695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:19.638548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.031542+0000","last_change":"2024-01-16T15:07:24.027138+0000","last_active":"2024-01-16T15:07:24.031542+0000","last_peered":"2024-01-16T15:07:24.031542+0000","last_clean":"2024-01-16T15:07:24.031542+0000","last_became_active":"2024-01-16T15:07:24.026870+0000","last_became_peered":"2024-01-16T15:07:24.026870+0000","last_unstale":"2024-01-16T15:07:24.031542+0000","last_undegraded":"2024-01-16T15:07:24.031542+0000","last_fullsized":"2024-01-16T15:07:24.031542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:51.439554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001948+0000","last_change":"2024-01-16T15:07:23.013637+0000","last_active":"2024-01-16T15:07:25.001948+0000","last_peered":"2024-01-16T15:07:25.001948+0000","last_clean":"2024-01-16T15:07:25.001948+0000","last_became_active":"2024-01-16T15:07:23.013141+0000","last_became_peered":"2024-01-16T15:07:23.013141+0000","last_unstale":"2024-01-16T15:07:25.001948+0000","last_undegraded":"2024-01-16T15:07:25.001948+0000","last_fullsized":"2024-01-16T15:07:25.001948+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:25:32.507386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045033+0000","last_change":"2024-01-16T15:07:24.038413+0000","last_active":"2024-01-16T15:07:24.045033+0000","last_peered":"2024-01-16T15:07:24.045033+0000","last_clean":"2024-01-16T15:07:24.045033+0000","last_became_active":"2024-01-16T15:07:24.038121+0000","last_became_peered":"2024-01-16T15:07:24.038121+0000","last_unstale":"2024-01-16T15:07:24.045033+0000","last_undegraded":"2024-01-16T15:07:24.045033+0000","last_fullsized":"2024-01-16T15:07:24.045033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:46.416261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.011236+0000","last_change":"2024-01-16T15:07:23.018594+0000","last_active":"2024-01-16T15:07:24.011236+0000","last_peered":"2024-01-16T15:07:24.011236+0000","last_clean":"2024-01-16T15:07:24.011236+0000","last_became_active":"2024-01-16T15:07:23.018195+0000","last_became_peered":"2024-01-16T15:07:23.018195+0000","last_unstale":"2024-01-16T15:07:24.011236+0000","last_undegraded":"2024-01-16T15:07:24.011236+0000","last_fullsized":"2024-01-16T15:07:24.011236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:51.659072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.035320+0000","last_change":"2024-01-16T15:07:24.018277+0000","last_active":"2024-01-16T15:07:24.035320+0000","last_peered":"2024-01-16T15:07:24.035320+0000","last_clean":"2024-01-16T15:07:24.035320+0000","last_became_active":"2024-01-16T15:07:24.017965+0000","last_became_peered":"2024-01-16T15:07:24.017965+0000","last_unstale":"2024-01-16T15:07:24.035320+0000","last_undegraded":"2024-01-16T15:07:24.035320+0000","last_fullsized":"2024-01-16T15:07:24.035320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:01.680291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.050254+0000","last_change":"2024-01-16T15:07:23.018628+0000","last_active":"2024-01-16T15:07:24.050254+0000","last_peered":"2024-01-16T15:07:24.050254+0000","last_clean":"2024-01-16T15:07:24.050254+0000","last_became_active":"2024-01-16T15:07:23.018030+0000","last_became_peered":"2024-01-16T15:07:23.018030+0000","last_unstale":"2024-01-16T15:07:24.050254+0000","last_undegraded":"2024-01-16T15:07:24.050254+0000","last_fullsized":"2024-01-16T15:07:24.050254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:29.841768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999853+0000","last_change":"2024-01-16T15:07:24.029472+0000","last_active":"2024-01-16T15:07:24.999853+0000","last_peered":"2024-01-16T15:07:24.999853+0000","last_clean":"2024-01-16T15:07:24.999853+0000","last_became_active":"2024-01-16T15:07:24.029210+0000","last_became_peered":"2024-01-16T15:07:24.029210+0000","last_unstale":"2024-01-16T15:07:24.999853+0000","last_undegraded":"2024-01-16T15:07:24.999853+0000","last_fullsized":"2024-01-16T15:07:24.999853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:30.927352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001974+0000","last_change":"2024-01-16T15:07:23.019932+0000","last_active":"2024-01-16T15:07:25.001974+0000","last_peered":"2024-01-16T15:07:25.001974+0000","last_clean":"2024-01-16T15:07:25.001974+0000","last_became_active":"2024-01-16T15:07:23.019172+0000","last_became_peered":"2024-01-16T15:07:23.019172+0000","last_unstale":"2024-01-16T15:07:25.001974+0000","last_undegraded":"2024-01-16T15:07:25.001974+0000","last_fullsized":"2024-01-16T15:07:25.001974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:11.290476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044929+0000","last_change":"2024-01-16T15:07:24.038072+0000","last_active":"2024-01-16T15:07:24.044929+0000","last_peered":"2024-01-16T15:07:24.044929+0000","last_clean":"2024-01-16T15:07:24.044929+0000","last_became_active":"2024-01-16T15:07:24.037696+0000","last_became_peered":"2024-01-16T15:07:24.037696+0000","last_unstale":"2024-01-16T15:07:24.044929+0000","last_undegraded":"2024-01-16T15:07:24.044929+0000","last_fullsized":"2024-01-16T15:07:24.044929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:29.933339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001673+0000","last_change":"2024-01-16T15:07:23.020474+0000","last_active":"2024-01-16T15:07:25.001673+0000","last_peered":"2024-01-16T15:07:25.001673+0000","last_clean":"2024-01-16T15:07:25.001673+0000","last_became_active":"2024-01-16T15:07:23.019485+0000","last_became_peered":"2024-01-16T15:07:23.019485+0000","last_unstale":"2024-01-16T15:07:25.001673+0000","last_undegraded":"2024-01-16T15:07:25.001673+0000","last_fullsized":"2024-01-16T15:07:25.001673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:44.030793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.029139+0000","last_change":"2024-01-16T15:07:24.021617+0000","last_active":"2024-01-16T15:07:24.029139+0000","last_peered":"2024-01-16T15:07:24.029139+0000","last_clean":"2024-01-16T15:07:24.029139+0000","last_became_active":"2024-01-16T15:07:24.021272+0000","last_became_peered":"2024-01-16T15:07:24.021272+0000","last_unstale":"2024-01-16T15:07:24.029139+0000","last_undegraded":"2024-01-16T15:07:24.029139+0000","last_fullsized":"2024-01-16T15:07:24.029139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:58.049449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.010044+0000","last_change":"2024-01-16T15:07:23.008356+0000","last_active":"2024-01-16T15:07:24.010044+0000","last_peered":"2024-01-16T15:07:24.010044+0000","last_clean":"2024-01-16T15:07:24.010044+0000","last_became_active":"2024-01-16T15:07:23.007780+0000","last_became_peered":"2024-01-16T15:07:23.007780+0000","last_unstale":"2024-01-16T15:07:24.010044+0000","last_undegraded":"2024-01-16T15:07:24.010044+0000","last_fullsized":"2024-01-16T15:07:24.010044+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:08.836959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.025085+0000","last_change":"2024-01-16T15:07:24.015178+0000","last_active":"2024-01-16T15:07:24.025085+0000","last_peered":"2024-01-16T15:07:24.025085+0000","last_clean":"2024-01-16T15:07:24.025085+0000","last_became_active":"2024-01-16T15:07:24.014560+0000","last_became_peered":"2024-01-16T15:07:24.014560+0000","last_unstale":"2024-01-16T15:07:24.025085+0000","last_undegraded":"2024-01-16T15:07:24.025085+0000","last_fullsized":"2024-01-16T15:07:24.025085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:58.571129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.017509+0000","last_change":"2024-01-16T15:07:23.019681+0000","last_active":"2024-01-16T15:07:24.017509+0000","last_peered":"2024-01-16T15:07:24.017509+0000","last_clean":"2024-01-16T15:07:24.017509+0000","last_became_active":"2024-01-16T15:07:23.019392+0000","last_became_peered":"2024-01-16T15:07:23.019392+0000","last_unstale":"2024-01-16T15:07:24.017509+0000","last_undegraded":"2024-01-16T15:07:24.017509+0000","last_fullsized":"2024-01-16T15:07:24.017509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:24.856680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998905+0000","last_change":"2024-01-16T15:07:24.028679+0000","last_active":"2024-01-16T15:07:24.998905+0000","last_peered":"2024-01-16T15:07:24.998905+0000","last_clean":"2024-01-16T15:07:24.998905+0000","last_became_active":"2024-01-16T15:07:24.028384+0000","last_became_peered":"2024-01-16T15:07:24.028384+0000","last_unstale":"2024-01-16T15:07:24.998905+0000","last_undegraded":"2024-01-16T15:07:24.998905+0000","last_fullsized":"2024-01-16T15:07:24.998905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:01.469289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001409+0000","last_change":"2024-01-16T15:07:23.014879+0000","last_active":"2024-01-16T15:07:25.001409+0000","last_peered":"2024-01-16T15:07:25.001409+0000","last_clean":"2024-01-16T15:07:25.001409+0000","last_became_active":"2024-01-16T15:07:23.010432+0000","last_became_peered":"2024-01-16T15:07:23.010432+0000","last_unstale":"2024-01-16T15:07:25.001409+0000","last_undegraded":"2024-01-16T15:07:25.001409+0000","last_fullsized":"2024-01-16T15:07:25.001409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:10.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003183+0000","last_change":"2024-01-16T15:07:24.021543+0000","last_active":"2024-01-16T15:07:25.003183+0000","last_peered":"2024-01-16T15:07:25.003183+0000","last_clean":"2024-01-16T15:07:25.003183+0000","last_became_active":"2024-01-16T15:07:24.021130+0000","last_became_peered":"2024-01-16T15:07:24.021130+0000","last_unstale":"2024-01-16T15:07:25.003183+0000","last_undegraded":"2024-01-16T15:07:25.003183+0000","last_fullsized":"2024-01-16T15:07:25.003183+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:49.215397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049925+0000","last_change":"2024-01-16T15:07:23.014053+0000","last_active":"2024-01-16T15:07:24.049925+0000","last_peered":"2024-01-16T15:07:24.049925+0000","last_clean":"2024-01-16T15:07:24.049925+0000","last_became_active":"2024-01-16T15:07:23.012679+0000","last_became_peered":"2024-01-16T15:07:23.012679+0000","last_unstale":"2024-01-16T15:07:24.049925+0000","last_undegraded":"2024-01-16T15:07:24.049925+0000","last_fullsized":"2024-01-16T15:07:24.049925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:14:59.354150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044991+0000","last_change":"2024-01-16T15:07:24.034377+0000","last_active":"2024-01-16T15:07:24.044991+0000","last_peered":"2024-01-16T15:07:24.044991+0000","last_clean":"2024-01-16T15:07:24.044991+0000","last_became_active":"2024-01-16T15:07:24.034114+0000","last_became_peered":"2024-01-16T15:07:24.034114+0000","last_unstale":"2024-01-16T15:07:24.044991+0000","last_undegraded":"2024-01-16T15:07:24.044991+0000","last_fullsized":"2024-01-16T15:07:24.044991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:49.482278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003233+0000","last_change":"2024-01-16T15:07:23.014544+0000","last_active":"2024-01-16T15:07:25.003233+0000","last_peered":"2024-01-16T15:07:25.003233+0000","last_clean":"2024-01-16T15:07:25.003233+0000","last_became_active":"2024-01-16T15:07:23.014196+0000","last_became_peered":"2024-01-16T15:07:23.014196+0000","last_unstale":"2024-01-16T15:07:25.003233+0000","last_undegraded":"2024-01-16T15:07:25.003233+0000","last_fullsized":"2024-01-16T15:07:25.003233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:37.489532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001802+0000","last_change":"2024-01-16T15:07:24.034662+0000","last_active":"2024-01-16T15:07:25.001802+0000","last_peered":"2024-01-16T15:07:25.001802+0000","last_clean":"2024-01-16T15:07:25.001802+0000","last_became_active":"2024-01-16T15:07:24.030691+0000","last_became_peered":"2024-01-16T15:07:24.030691+0000","last_unstale":"2024-01-16T15:07:25.001802+0000","last_undegraded":"2024-01-16T15:07:25.001802+0000","last_fullsized":"2024-01-16T15:07:25.001802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:01.680291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045599+0000","last_change":"2024-01-16T15:07:23.020613+0000","last_active":"2024-01-16T15:07:24.045599+0000","last_peered":"2024-01-16T15:07:24.045599+0000","last_clean":"2024-01-16T15:07:24.045599+0000","last_became_active":"2024-01-16T15:07:23.013608+0000","last_became_peered":"2024-01-16T15:07:23.013608+0000","last_unstale":"2024-01-16T15:07:24.045599+0000","last_undegraded":"2024-01-16T15:07:24.045599+0000","last_fullsized":"2024-01-16T15:07:24.045599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:28.916750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999881+0000","last_change":"2024-01-16T15:07:24.030061+0000","last_active":"2024-01-16T15:07:24.999881+0000","last_peered":"2024-01-16T15:07:24.999881+0000","last_clean":"2024-01-16T15:07:24.999881+0000","last_became_active":"2024-01-16T15:07:24.027485+0000","last_became_peered":"2024-01-16T15:07:24.027485+0000","last_unstale":"2024-01-16T15:07:24.999881+0000","last_undegraded":"2024-01-16T15:07:24.999881+0000","last_fullsized":"2024-01-16T15:07:24.999881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:38.474326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.022751+0000","last_change":"2024-01-16T15:07:23.024687+0000","last_active":"2024-01-16T15:07:24.022751+0000","last_peered":"2024-01-16T15:07:24.022751+0000","last_clean":"2024-01-16T15:07:24.022751+0000","last_became_active":"2024-01-16T15:07:23.024293+0000","last_became_peered":"2024-01-16T15:07:23.024293+0000","last_unstale":"2024-01-16T15:07:24.022751+0000","last_undegraded":"2024-01-16T15:07:24.022751+0000","last_fullsized":"2024-01-16T15:07:24.022751+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:08.806948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.026560+0000","last_change":"2024-01-16T15:07:24.021885+0000","last_active":"2024-01-16T15:07:24.026560+0000","last_peered":"2024-01-16T15:07:24.026560+0000","last_clean":"2024-01-16T15:07:24.026560+0000","last_became_active":"2024-01-16T15:07:24.021567+0000","last_became_peered":"2024-01-16T15:07:24.021567+0000","last_unstale":"2024-01-16T15:07:24.026560+0000","last_undegraded":"2024-01-16T15:07:24.026560+0000","last_fullsized":"2024-01-16T15:07:24.026560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.858660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.010276+0000","last_change":"2024-01-16T15:07:23.008169+0000","last_active":"2024-01-16T15:07:24.010276+0000","last_peered":"2024-01-16T15:07:24.010276+0000","last_clean":"2024-01-16T15:07:24.010276+0000","last_became_active":"2024-01-16T15:07:23.007390+0000","last_became_peered":"2024-01-16T15:07:23.007390+0000","last_unstale":"2024-01-16T15:07:24.010276+0000","last_undegraded":"2024-01-16T15:07:24.010276+0000","last_fullsized":"2024-01-16T15:07:24.010276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:29.650808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038854+0000","last_change":"2024-01-16T15:07:24.034170+0000","last_active":"2024-01-16T15:07:24.038854+0000","last_peered":"2024-01-16T15:07:24.038854+0000","last_clean":"2024-01-16T15:07:24.038854+0000","last_became_active":"2024-01-16T15:07:24.033845+0000","last_became_peered":"2024-01-16T15:07:24.033845+0000","last_unstale":"2024-01-16T15:07:24.038854+0000","last_undegraded":"2024-01-16T15:07:24.038854+0000","last_fullsized":"2024-01-16T15:07:24.038854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:20.223725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019092+0000","last_change":"2024-01-16T15:07:23.018354+0000","last_active":"2024-01-16T15:07:24.019092+0000","last_peered":"2024-01-16T15:07:24.019092+0000","last_clean":"2024-01-16T15:07:24.019092+0000","last_became_active":"2024-01-16T15:07:23.017990+0000","last_became_peered":"2024-01-16T15:07:23.017990+0000","last_unstale":"2024-01-16T15:07:24.019092+0000","last_undegraded":"2024-01-16T15:07:24.019092+0000","last_fullsized":"2024-01-16T15:07:24.019092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:59.924301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.036425+0000","last_change":"2024-01-16T15:07:24.033827+0000","last_active":"2024-01-16T15:07:24.036425+0000","last_peered":"2024-01-16T15:07:24.036425+0000","last_clean":"2024-01-16T15:07:24.036425+0000","last_became_active":"2024-01-16T15:07:24.033520+0000","last_became_peered":"2024-01-16T15:07:24.033520+0000","last_unstale":"2024-01-16T15:07:24.036425+0000","last_undegraded":"2024-01-16T15:07:24.036425+0000","last_fullsized":"2024-01-16T15:07:24.036425+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:57.020614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:23.998159+0000","last_change":"2024-01-16T15:07:23.013667+0000","last_active":"2024-01-16T15:07:23.998159+0000","last_peered":"2024-01-16T15:07:23.998159+0000","last_clean":"2024-01-16T15:07:23.998159+0000","last_became_active":"2024-01-16T15:07:23.013388+0000","last_became_peered":"2024-01-16T15:07:23.013388+0000","last_unstale":"2024-01-16T15:07:23.998159+0000","last_undegraded":"2024-01-16T15:07:23.998159+0000","last_fullsized":"2024-01-16T15:07:23.998159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:51.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038025+0000","last_change":"2024-01-16T15:07:24.034844+0000","last_active":"2024-01-16T15:07:24.038025+0000","last_peered":"2024-01-16T15:07:24.038025+0000","last_clean":"2024-01-16T15:07:24.038025+0000","last_became_active":"2024-01-16T15:07:24.034414+0000","last_became_peered":"2024-01-16T15:07:24.034414+0000","last_unstale":"2024-01-16T15:07:24.038025+0000","last_undegraded":"2024-01-16T15:07:24.038025+0000","last_fullsized":"2024-01-16T15:07:24.038025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:02.946349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048906+0000","last_change":"2024-01-16T15:07:23.013248+0000","last_active":"2024-01-16T15:07:24.048906+0000","last_peered":"2024-01-16T15:07:24.048906+0000","last_clean":"2024-01-16T15:07:24.048906+0000","last_became_active":"2024-01-16T15:07:23.012273+0000","last_became_peered":"2024-01-16T15:07:23.012273+0000","last_unstale":"2024-01-16T15:07:24.048906+0000","last_undegraded":"2024-01-16T15:07:24.048906+0000","last_fullsized":"2024-01-16T15:07:24.048906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:50.423279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.030721+0000","last_change":"2024-01-16T15:07:24.024871+0000","last_active":"2024-01-16T15:07:24.030721+0000","last_peered":"2024-01-16T15:07:24.030721+0000","last_clean":"2024-01-16T15:07:24.030721+0000","last_became_active":"2024-01-16T15:07:24.022629+0000","last_became_peered":"2024-01-16T15:07:24.022629+0000","last_unstale":"2024-01-16T15:07:24.030721+0000","last_undegraded":"2024-01-16T15:07:24.030721+0000","last_fullsized":"2024-01-16T15:07:24.030721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:52.675347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019251+0000","last_change":"2024-01-16T15:07:23.020610+0000","last_active":"2024-01-16T15:07:24.019251+0000","last_peered":"2024-01-16T15:07:24.019251+0000","last_clean":"2024-01-16T15:07:24.019251+0000","last_became_active":"2024-01-16T15:07:23.020290+0000","last_became_peered":"2024-01-16T15:07:23.020290+0000","last_unstale":"2024-01-16T15:07:24.019251+0000","last_undegraded":"2024-01-16T15:07:24.019251+0000","last_fullsized":"2024-01-16T15:07:24.019251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:35:46.868161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.046833+0000","last_change":"2024-01-16T15:07:24.040330+0000","last_active":"2024-01-16T15:07:24.046833+0000","last_peered":"2024-01-16T15:07:24.046833+0000","last_clean":"2024-01-16T15:07:24.046833+0000","last_became_active":"2024-01-16T15:07:24.040053+0000","last_became_peered":"2024-01-16T15:07:24.040053+0000","last_unstale":"2024-01-16T15:07:24.046833+0000","last_undegraded":"2024-01-16T15:07:24.046833+0000","last_fullsized":"2024-01-16T15:07:24.046833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.224060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999985+0000","last_change":"2024-01-16T15:07:23.008633+0000","last_active":"2024-01-16T15:07:24.999985+0000","last_peered":"2024-01-16T15:07:24.999985+0000","last_clean":"2024-01-16T15:07:24.999985+0000","last_became_active":"2024-01-16T15:07:23.008193+0000","last_became_peered":"2024-01-16T15:07:23.008193+0000","last_unstale":"2024-01-16T15:07:24.999985+0000","last_undegraded":"2024-01-16T15:07:24.999985+0000","last_fullsized":"2024-01-16T15:07:24.999985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:15.276116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048227+0000","last_change":"2024-01-16T15:07:24.043384+0000","last_active":"2024-01-16T15:07:24.048227+0000","last_peered":"2024-01-16T15:07:24.048227+0000","last_clean":"2024-01-16T15:07:24.048227+0000","last_became_active":"2024-01-16T15:07:24.039785+0000","last_became_peered":"2024-01-16T15:07:24.039785+0000","last_unstale":"2024-01-16T15:07:24.048227+0000","last_undegraded":"2024-01-16T15:07:24.048227+0000","last_fullsized":"2024-01-16T15:07:24.048227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:16.076911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:23.999252+0000","last_change":"2024-01-16T15:07:23.014030+0000","last_active":"2024-01-16T15:07:23.999252+0000","last_peered":"2024-01-16T15:07:23.999252+0000","last_clean":"2024-01-16T15:07:23.999252+0000","last_became_active":"2024-01-16T15:07:23.012317+0000","last_became_peered":"2024-01-16T15:07:23.012317+0000","last_unstale":"2024-01-16T15:07:23.999252+0000","last_undegraded":"2024-01-16T15:07:23.999252+0000","last_fullsized":"2024-01-16T15:07:23.999252+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.188168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999311+0000","last_change":"2024-01-16T15:07:24.029017+0000","last_active":"2024-01-16T15:07:24.999311+0000","last_peered":"2024-01-16T15:07:24.999311+0000","last_clean":"2024-01-16T15:07:24.999311+0000","last_became_active":"2024-01-16T15:07:24.028121+0000","last_became_peered":"2024-01-16T15:07:24.028121+0000","last_unstale":"2024-01-16T15:07:24.999311+0000","last_undegraded":"2024-01-16T15:07:24.999311+0000","last_fullsized":"2024-01-16T15:07:24.999311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:14.141767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999630+0000","last_change":"2024-01-16T15:07:23.018693+0000","last_active":"2024-01-16T15:07:24.999630+0000","last_peered":"2024-01-16T15:07:24.999630+0000","last_clean":"2024-01-16T15:07:24.999630+0000","last_became_active":"2024-01-16T15:07:23.018428+0000","last_became_peered":"2024-01-16T15:07:23.018428+0000","last_unstale":"2024-01-16T15:07:24.999630+0000","last_undegraded":"2024-01-16T15:07:24.999630+0000","last_fullsized":"2024-01-16T15:07:24.999630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:03.545845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037775+0000","last_change":"2024-01-16T15:07:24.027194+0000","last_active":"2024-01-16T15:07:24.037775+0000","last_peered":"2024-01-16T15:07:24.037775+0000","last_clean":"2024-01-16T15:07:24.037775+0000","last_became_active":"2024-01-16T15:07:24.026587+0000","last_became_peered":"2024-01-16T15:07:24.026587+0000","last_unstale":"2024-01-16T15:07:24.037775+0000","last_undegraded":"2024-01-16T15:07:24.037775+0000","last_fullsized":"2024-01-16T15:07:24.037775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:30.398205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001272+0000","last_change":"2024-01-16T15:07:23.016781+0000","last_active":"2024-01-16T15:07:25.001272+0000","last_peered":"2024-01-16T15:07:25.001272+0000","last_clean":"2024-01-16T15:07:25.001272+0000","last_became_active":"2024-01-16T15:07:23.016451+0000","last_became_peered":"2024-01-16T15:07:23.016451+0000","last_unstale":"2024-01-16T15:07:25.001272+0000","last_undegraded":"2024-01-16T15:07:25.001272+0000","last_fullsized":"2024-01-16T15:07:25.001272+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:13:38.508365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001363+0000","last_change":"2024-01-16T15:07:24.023693+0000","last_active":"2024-01-16T15:07:25.001363+0000","last_peered":"2024-01-16T15:07:25.001363+0000","last_clean":"2024-01-16T15:07:25.001363+0000","last_became_active":"2024-01-16T15:07:24.023402+0000","last_became_peered":"2024-01-16T15:07:24.023402+0000","last_unstale":"2024-01-16T15:07:25.001363+0000","last_undegraded":"2024-01-16T15:07:25.001363+0000","last_fullsized":"2024-01-16T15:07:25.001363+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:11.466328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001303+0000","last_change":"2024-01-16T15:07:23.015966+0000","last_active":"2024-01-16T15:07:25.001303+0000","last_peered":"2024-01-16T15:07:25.001303+0000","last_clean":"2024-01-16T15:07:25.001303+0000","last_became_active":"2024-01-16T15:07:23.015544+0000","last_became_peered":"2024-01-16T15:07:23.015544+0000","last_unstale":"2024-01-16T15:07:25.001303+0000","last_undegraded":"2024-01-16T15:07:25.001303+0000","last_fullsized":"2024-01-16T15:07:25.001303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:18.002349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037264+0000","last_change":"2024-01-16T15:07:24.025123+0000","last_active":"2024-01-16T15:07:24.037264+0000","last_peered":"2024-01-16T15:07:24.037264+0000","last_clean":"2024-01-16T15:07:24.037264+0000","last_became_active":"2024-01-16T15:07:24.024226+0000","last_became_peered":"2024-01-16T15:07:24.024226+0000","last_unstale":"2024-01-16T15:07:24.037264+0000","last_undegraded":"2024-01-16T15:07:24.037264+0000","last_fullsized":"2024-01-16T15:07:24.037264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:24.256412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.007910+0000","last_change":"2024-01-16T15:07:23.011710+0000","last_active":"2024-01-16T15:07:24.007910+0000","last_peered":"2024-01-16T15:07:24.007910+0000","last_clean":"2024-01-16T15:07:24.007910+0000","last_became_active":"2024-01-16T15:07:23.011444+0000","last_became_peered":"2024-01-16T15:07:23.011444+0000","last_unstale":"2024-01-16T15:07:24.007910+0000","last_undegraded":"2024-01-16T15:07:24.007910+0000","last_fullsized":"2024-01-16T15:07:24.007910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:05.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.034666+0000","last_change":"2024-01-16T15:07:24.033222+0000","last_active":"2024-01-16T15:07:24.034666+0000","last_peered":"2024-01-16T15:07:24.034666+0000","last_clean":"2024-01-16T15:07:24.034666+0000","last_became_active":"2024-01-16T15:07:24.032918+0000","last_became_peered":"2024-01-16T15:07:24.032918+0000","last_unstale":"2024-01-16T15:07:24.034666+0000","last_undegraded":"2024-01-16T15:07:24.034666+0000","last_fullsized":"2024-01-16T15:07:24.034666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:50.964486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049735+0000","last_change":"2024-01-16T15:07:23.016337+0000","last_active":"2024-01-16T15:07:24.049735+0000","last_peered":"2024-01-16T15:07:24.049735+0000","last_clean":"2024-01-16T15:07:24.049735+0000","last_became_active":"2024-01-16T15:07:23.016025+0000","last_became_peered":"2024-01-16T15:07:23.016025+0000","last_unstale":"2024-01-16T15:07:24.049735+0000","last_undegraded":"2024-01-16T15:07:24.049735+0000","last_fullsized":"2024-01-16T15:07:24.049735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:36.334216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":206,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019225+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:07:24.019225+0000","last_peered":"2024-01-16T15:07:24.019225+0000","last_clean":"2024-01-16T15:07:24.019225+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:07:24.019225+0000","last_undegraded":"2024-01-16T15:07:24.019225+0000","last_fullsized":"2024-01-16T15:07:24.019225+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542244,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50656,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1462272,"data_stored":953247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.468,"15min":1.2370000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.6260000000000003,"5min":7.8949999999999996,"15min":7.8949999999999996},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.841,"5min":1.5369999999999999,"15min":1.3149999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":6.5039999999999996,"5min":7.8460000000000001,"15min":7.8460000000000001},"last":1.0489999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.444,"15min":1.2789999999999999},"min":{"1min":0.92300000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":5.3010000000000002,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":2.4929999999999999,"5min":1.55,"15min":1.2929999999999999},"min":{"1min":0.83899999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":15.452,"5min":15.452,"15min":15.452},"last":1.1299999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5369999999999999,"15min":1.365},"min":{"1min":0.83099999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":5.9130000000000003,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.556,"15min":1.3779999999999999},"min":{"1min":0.95599999999999996,"5min":0.504,"15min":0.504},"max":{"1min":3.6379999999999999,"5min":10.661,"15min":10.661},"last":0.97099999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":1.752,"15min":1.4259999999999999},"min":{"1min":0.85199999999999998,"5min":0.61499999999999999,"15min":0.56999999999999995},"max":{"1min":10.654999999999999,"5min":10.654999999999999,"15min":10.654999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.651,"5min":1.716,"15min":1.514},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":5.8029999999999999,"5min":13.193,"15min":13.193},"last":1.3280000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.417,"15min":1.3899999999999999},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.752,"5min":16.376000000000001,"15min":16.376000000000001},"last":1.107},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.476,"15min":1.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.0680000000000001,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.446}]},{"osd":5,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.514,"15min":1.544},"min":{"1min":0.623,"5min":0.497,"15min":0.497},"max":{"1min":5.1849999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":1.415},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.5509999999999999,"15min":1.5600000000000001},"min":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8380000000000001,"5min":7.9219999999999997,"15min":7.9219999999999997},"last":1.169}]},{"osd":6,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.5960000000000001,"15min":1.661},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.8879999999999999,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.242},{"interface":"front","average":{"1min":1.542,"5min":1.7090000000000001,"15min":1.6890000000000001},"min":{"1min":0.65200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.6739999999999999,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.3859999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542243,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45916,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325924,"statfs":{"total":96636764160,"available":96589746176,"internally_reserved":0,"allocated":868352,"data_stored":362355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3769999999999998,"5min":2.2469999999999999,"15min":1.5209999999999999},"min":{"1min":0.60399999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":16.25,"5min":28.327999999999999,"15min":28.327999999999999},"last":1.214},{"interface":"front","average":{"1min":2.4369999999999998,"5min":2.2799999999999998,"15min":1.5249999999999999},"min":{"1min":0.629,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":22.43,"5min":30.280999999999999,"15min":30.280999999999999},"last":1.3109999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.855,"5min":1.9099999999999999,"15min":1.3500000000000001},"min":{"1min":0.78700000000000003,"5min":0.57699999999999996,"15min":0.56399999999999995},"max":{"1min":28.122,"5min":28.122,"15min":28.122},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.365,"15min":1.204},"min":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":5.4880000000000004,"5min":8.9719999999999995,"15min":8.9719999999999995},"last":1.01}]},{"osd":2,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":1.851,"15min":1.4279999999999999},"min":{"1min":0.70799999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"max":{"1min":6.8129999999999997,"5min":17.991,"15min":17.991},"last":1.829},{"interface":"front","average":{"1min":2.177,"5min":1.9890000000000001,"15min":1.601},"min":{"1min":0.749,"5min":0.67100000000000004,"15min":0.65800000000000003},"max":{"1min":6.7720000000000002,"5min":18.02,"15min":18.02},"last":1.639}]},{"osd":3,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":3.1070000000000002,"5min":2.0379999999999998,"15min":1.5049999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.61599999999999999},"max":{"1min":17.75,"5min":17.75,"15min":17.75},"last":1.4810000000000001},{"interface":"front","average":{"1min":2.4359999999999999,"5min":1.8420000000000001,"15min":1.4630000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.56299999999999994},"max":{"1min":17.794,"5min":17.794,"15min":17.794},"last":1.5209999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.5760000000000001,"15min":1.4039999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":3.694,"5min":15.481999999999999,"15min":15.481999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":2.4329999999999998,"5min":1.73,"15min":1.534},"min":{"1min":0.625,"5min":0.58099999999999996,"15min":0.53900000000000003},"max":{"1min":16.303000000000001,"5min":16.303000000000001,"15min":16.303000000000001},"last":1.3680000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.385,"15min":1.327},"min":{"1min":0.58999999999999997,"5min":0.50900000000000001,"15min":0.45200000000000001},"max":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.95,"5min":1.6850000000000001,"15min":1.601},"min":{"1min":0.78800000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":5.7009999999999996,"5min":9.0419999999999998,"15min":9.0419999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.516,"15min":1.472},"min":{"1min":0.69099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.6449999999999996,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":1.6850000000000001},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.738,"15min":1.6399999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":11.788,"5min":11.788,"15min":11.788},"last":1.915}]}]},{"osd":5,"up_from":14,"seq":60129542244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":864256,"data_stored":362353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6299999999999999,"5min":2.02,"15min":1.4319999999999999},"min":{"1min":0.66600000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":26.228999999999999,"5min":26.228999999999999,"15min":26.228999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":3.4279999999999999,"5min":2.766,"15min":1.609},"min":{"1min":0.80200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.867000000000001,"5min":56.011000000000003,"15min":56.011000000000003},"last":0.76500000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.5509999999999999,"15min":1.258},"min":{"1min":0.61599999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":10.926,"5min":16.687999999999999,"15min":16.687999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.3130000000000002,"15min":1.571},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.8380000000000001,"5min":67.930000000000007,"15min":67.930000000000007},"last":1.0880000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.607,"15min":1.351},"min":{"1min":0.53400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4039999999999999,"5min":13.013,"15min":13.013},"last":1.032},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.546,"15min":1.3460000000000001},"min":{"1min":0.60599999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.0720000000000001,"5min":11.09,"15min":11.09},"last":1.069}]},{"osd":3,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.242,"5min":2.0390000000000001,"15min":1.528},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":12.109999999999999,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.421},{"interface":"front","average":{"1min":2.3460000000000001,"5min":2.0249999999999999,"15min":1.611},"min":{"1min":0.75,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":12.135999999999999,"5min":13.356999999999999,"15min":13.356999999999999},"last":1.472}]},{"osd":4,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.444,"15min":1.3120000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.4430000000000001,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":1.004},{"interface":"front","average":{"1min":1.361,"5min":1.669,"15min":1.4390000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5910000000000002,"5min":16.268000000000001,"15min":16.268000000000001},"last":0.94499999999999995}]},{"osd":6,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.583,"15min":1.4550000000000001},"min":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.4950000000000001,"5min":13.263999999999999,"15min":13.263999999999999},"last":1.258},{"interface":"front","average":{"1min":1.53,"5min":1.5509999999999999,"15min":1.522},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":3.52,"5min":4.9299999999999997,"15min":7.335},"last":1.4370000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.865,"15min":1.6359999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.2290000000000001,"5min":14.345000000000001,"15min":14.345000000000001},"last":1.333},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.9450000000000001,"15min":1.698},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.577,"5min":13.491,"15min":13.491},"last":1.3839999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50092,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321748,"statfs":{"total":96636764160,"available":96585469952,"internally_reserved":0,"allocated":876544,"data_stored":368661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.4359999999999999,"15min":1.2050000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":9.5589999999999993,"5min":12.587999999999999,"15min":12.587999999999999},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1230000000000002,"15min":1.3919999999999999},"min":{"1min":0.71099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":4.3380000000000001,"5min":60.106000000000002,"15min":60.106000000000002},"last":1.2150000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.901,"15min":1.4930000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.7919999999999998,"5min":14.19,"15min":14.19},"last":1.466},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.673,"15min":1.369},"min":{"1min":0.68300000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":6.8170000000000002,"5min":9.5589999999999993,"15min":9.5589999999999993},"last":1.9930000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7689999999999999,"5min":1.357,"15min":1.232},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":9.7140000000000004,"5min":9.7140000000000004,"15min":9.7140000000000004},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":2.1789999999999998,"15min":1.617},"min":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":6.516,"5min":35.024999999999999,"15min":35.024999999999999},"last":1.3540000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":3.0819999999999999,"5min":1.829,"15min":1.5429999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":32.133000000000003,"5min":32.133000000000003,"15min":32.133000000000003},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.3819999999999999,"15min":1.3400000000000001},"min":{"1min":0.71899999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":4.2880000000000003,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":1.413}]},{"osd":5,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.5980000000000001,"15min":1.532},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.0289999999999999,"5min":14.568,"15min":14.568},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.4830000000000001,"15min":1.55},"min":{"1min":0.63,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.1479999999999997,"5min":7.1479999999999997,"15min":7.1479999999999997},"last":2.0569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.413,"15min":1.444},"min":{"1min":0.52400000000000002,"5min":0.437,"15min":0.437},"max":{"1min":2.8029999999999999,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":1.516},{"interface":"front","average":{"1min":1.25,"5min":1.6830000000000001,"15min":1.6259999999999999},"min":{"1min":0.498,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.1459999999999999,"5min":14.317,"15min":14.317},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.5509999999999999,"15min":1.655},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":7.0339999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.5209999999999999,"15min":1.595},"min":{"1min":0.66000000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.9390000000000001,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.952}]}]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54240,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317600,"statfs":{"total":96636764160,"available":96581222400,"internally_reserved":0,"allocated":868352,"data_stored":362375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":2.0910000000000002,"15min":1.351},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":50.988999999999997,"5min":50.988999999999997,"15min":50.988999999999997},"last":0.69599999999999995},{"interface":"front","average":{"1min":2.073,"5min":1.409,"15min":1.1539999999999999},"min":{"1min":0.45000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.4309999999999992,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.3839999999999999,"15min":1.1259999999999999},"min":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":11.634,"5min":11.634,"15min":11.634},"last":0.93300000000000005},{"interface":"front","average":{"1min":3.125,"5min":1.7949999999999999,"15min":1.421},"min":{"1min":0.52600000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":20.385999999999999,"5min":20.385999999999999,"15min":20.385999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.587,"15min":1.29},"min":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.42699999999999999},"max":{"1min":11.475,"5min":11.475,"15min":11.475},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.4249999999999998,"5min":1.835,"15min":1.482},"min":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":0.88600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3250000000000002,"5min":1.7130000000000001,"15min":1.458},"min":{"1min":0.62,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":9.5600000000000005,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":1.0720000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":1.6830000000000001,"15min":1.5089999999999999},"min":{"1min":0.73999999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":11.528,"5min":11.528,"15min":11.528},"last":1.2150000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6110000000000002,"5min":2.0569999999999999,"15min":1.698},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":24.349,"5min":24.349,"15min":24.349},"last":1.145},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.9119999999999999,"15min":1.569},"min":{"1min":0.92600000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":8.6489999999999991,"5min":12.616,"15min":12.616},"last":1.3100000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.746,"15min":1.5660000000000001},"min":{"1min":0.98999999999999999,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":11.193,"5min":11.193,"15min":11.193},"last":1.02},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.1200000000000001,"15min":1.774},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":24.058,"5min":24.058,"15min":24.058},"last":1.284}]},{"osd":7,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5499999999999998,"5min":1.859,"15min":1.643},"min":{"1min":0.67800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.5120000000000005,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.4019999999999999},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.8480000000000001,"15min":1.645},"min":{"1min":1.0389999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":11.082000000000001,"5min":11.082000000000001,"15min":11.082000000000001},"last":1.169}]}]},{"osd":2,"up_from":14,"seq":60129542243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":872448,"data_stored":363875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":2.1989999999999998,"15min":1.349},"min":{"1min":0.41899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":64.387,"15min":64.387},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.5660000000000001,"15min":1.175},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":10.151,"5min":11.339,"15min":11.339},"last":0.92700000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.9059999999999999,"15min":1.2410000000000001},"min":{"1min":0.39500000000000002,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":2.5699999999999998,"5min":27.263000000000002,"15min":27.263000000000002},"last":1.2410000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.1669999999999998,"15min":1.5369999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":16.163,"5min":23.547999999999998,"15min":23.547999999999998},"last":1.4059999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":1.9059999999999999,"15min":1.409},"min":{"1min":0.60999999999999999,"5min":0.40100000000000002,"15min":0.33100000000000002},"max":{"1min":8,"5min":27.311,"15min":27.311},"last":1.542},{"interface":"front","average":{"1min":2.3170000000000002,"5min":2.0430000000000001,"15min":1.444},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":13.84,"5min":26.555,"15min":26.555},"last":1.139}]},{"osd":4,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.869,"15min":1.4910000000000001},"min":{"1min":0.88400000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.3010000000000002,"5min":26.503,"15min":26.503},"last":1.302},{"interface":"front","average":{"1min":1.863,"5min":1.726,"15min":1.458},"min":{"1min":0.78400000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":10.542999999999999,"5min":11.188000000000001,"15min":11.188000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.97,"15min":1.571},"min":{"1min":1.022,"5min":0.80800000000000005,"15min":0.71899999999999997},"max":{"1min":2.7490000000000001,"5min":27.149000000000001,"15min":27.149000000000001},"last":1.474},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.966,"15min":1.655},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.895,"5min":26.349,"15min":26.349},"last":1.4279999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.8200000000000001,"15min":1.544},"min":{"1min":0.873,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":10.433,"5min":11.266,"15min":11.266},"last":1.198},{"interface":"front","average":{"1min":1.5960000000000001,"5min":2.2120000000000002,"15min":1.7130000000000001},"min":{"1min":1.141,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":3.1800000000000002,"5min":27.076000000000001,"15min":27.076000000000001},"last":1.3720000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1869999999999998,"5min":1.9930000000000001,"15min":1.7210000000000001},"min":{"1min":0.82999999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":10.617000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":1.351},{"interface":"front","average":{"1min":1.663,"5min":2.137,"15min":1.6839999999999999},"min":{"1min":1.1399999999999999,"5min":0.754,"15min":0.66600000000000004},"max":{"1min":3.2349999999999999,"5min":27.207999999999998,"15min":27.207999999999998},"last":1.607}]}]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54256,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317584,"statfs":{"total":96636764160,"available":96581206016,"internally_reserved":0,"allocated":880640,"data_stored":364491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.603,"15min":1.2050000000000001},"min":{"1min":0.41299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":11.516999999999999,"5min":14.407,"15min":14.407},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.198,"5min":1.4279999999999999,"15min":1.2070000000000001},"min":{"1min":0.68500000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":8.9719999999999995,"5min":8.9719999999999995,"15min":8.9719999999999995},"last":0.69699999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":1.639,"15min":1.2549999999999999},"min":{"1min":0.57399999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":16.488,"5min":16.488,"15min":16.488},"last":1.0449999999999999},{"interface":"front","average":{"1min":2.524,"5min":2.0539999999999998,"15min":1.4850000000000001},"min":{"1min":0.59699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":11.215,"5min":22.550999999999998,"15min":22.550999999999998},"last":0.92200000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.958,"15min":1.5189999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.117,"5min":14.653,"15min":14.653},"last":1.514},{"interface":"front","average":{"1min":2.9039999999999999,"5min":1.8879999999999999,"15min":1.4870000000000001},"min":{"1min":0.65600000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":12.220000000000001,"5min":12.220000000000001,"15min":12.220000000000001},"last":1.635}]},{"osd":4,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7730000000000001,"5min":1.919,"15min":1.575},"min":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.81000000000000005},"max":{"1min":8.3620000000000001,"5min":12.675000000000001,"15min":12.675000000000001},"last":1.381},{"interface":"front","average":{"1min":3.3839999999999999,"5min":2.0339999999999998,"15min":1.6419999999999999},"min":{"1min":0.99199999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":19.236999999999998,"5min":19.236999999999998,"15min":19.236999999999998},"last":1.3129999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.609,"5min":1.8149999999999999,"15min":1.629},"min":{"1min":1.0329999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":8.2859999999999996},"last":1.53},{"interface":"front","average":{"1min":3.6019999999999999,"5min":1.9630000000000001,"15min":1.6299999999999999},"min":{"1min":1.01,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":17.138000000000002,"5min":17.138000000000002,"15min":17.138000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4700000000000002,"5min":2.161,"15min":1.7809999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":11.404,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.4379999999999999},{"interface":"front","average":{"1min":3.0169999999999999,"5min":2.0110000000000001,"15min":1.764},"min":{"1min":1.165,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":13.243,"5min":13.243,"15min":13.243},"last":1.2110000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":3.4470000000000001,"5min":2.0009999999999999,"15min":1.7090000000000001},"min":{"1min":1.135,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":17.010000000000002,"5min":17.010000000000002,"15min":17.010000000000002},"last":1.575},{"interface":"front","average":{"1min":2.5680000000000001,"5min":2.2349999999999999,"15min":1.8540000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":11.457000000000001,"5min":14.884,"15min":14.884},"last":1.2729999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542244,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54812,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317028,"statfs":{"total":96636764160,"available":96580636672,"internally_reserved":0,"allocated":1454080,"data_stored":952631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":54655425},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.6320000000000001,"5min":1.762,"15min":1.1859999999999999},"min":{"1min":0.58499999999999996,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":19.058,"5min":19.058,"15min":19.058},"last":0.377},{"interface":"front","average":{"1min":2.496,"5min":2.0630000000000002,"15min":1.296},"min":{"1min":0.70399999999999996,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.846,"5min":24.414000000000001,"15min":24.414000000000001},"last":0.76800000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.8159999999999998,"5min":1.7230000000000001,"15min":1.248},"min":{"1min":0.40799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":18.861000000000001,"5min":18.861000000000001,"15min":18.861000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.984,"5min":1.944,"15min":1.3069999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":13.936,"5min":38.893000000000001,"15min":38.893000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.3700000000000001,"5min":2.1629999999999998,"15min":1.337},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":10.090999999999999,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.948,"5min":1.9259999999999999,"15min":1.413},"min":{"1min":0.69799999999999995,"5min":0.47799999999999998,"15min":0.46999999999999997},"max":{"1min":18.966999999999999,"5min":18.966999999999999,"15min":18.966999999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.7050000000000001,"15min":1.343},"min":{"1min":0.67900000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":6.0199999999999996,"5min":20.327000000000002,"15min":20.327000000000002},"last":1.145},{"interface":"front","average":{"1min":3.1280000000000001,"5min":2.008,"15min":1.5600000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.71399999999999997},"max":{"1min":19.004999999999999,"5min":19.004999999999999,"15min":19.004999999999999},"last":1.1930000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":2.3460000000000001,"15min":1.633},"min":{"1min":0.80900000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":14.630000000000001,"5min":24.611000000000001,"15min":24.611000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.95,"5min":1.9830000000000001,"15min":1.5109999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":6.0119999999999996,"5min":20.199000000000002,"15min":20.199000000000002},"last":1.2509999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":4.8700000000000001,"5min":2.448,"15min":1.802},"min":{"1min":0.79000000000000004,"5min":0.754,"15min":0.754},"max":{"1min":26.998000000000001,"5min":26.998000000000001,"15min":26.998000000000001},"last":0.62},{"interface":"front","average":{"1min":2.6960000000000002,"5min":2.3900000000000001,"15min":1.712},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":14.326000000000001,"5min":24.478999999999999,"15min":24.478999999999999},"last":1.002}]},{"osd":7,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.214,"15min":1.73},"min":{"1min":0.76100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":14.275,"5min":17.957000000000001,"15min":17.957000000000001},"last":1.4319999999999999},{"interface":"front","average":{"1min":2.1859999999999999,"5min":1.996,"15min":1.609},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":6.3360000000000003,"5min":20.279,"15min":20.279},"last":1.3480000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:07:28.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:07:28.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.779+0000 7f7597200640 1 Processor -- start 2024-01-16T15:07:28.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.780+0000 7f7597200640 1 -- start start 2024-01-16T15:07:28.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.781+0000 7f7597200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f759815d5b0 0x7f75980d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.781+0000 7f7597200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75980cf0c0 con 0x7f75980b1650 2024-01-16T15:07:28.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.781+0000 7f7597200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75980b01b0 con 0x7f75980d14a0 2024-01-16T15:07:28.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.781+0000 7f7597200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75980d05c0 con 0x7f759815d5b0 2024-01-16T15:07:28.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7596800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f75980b1650 0x7f75980cfeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44912/0 (socket says 172.21.15.43:44912) 2024-01-16T15:07:28.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7596800640 1 -- 172.21.15.43:0/1416575473 learned_addr learned my addr 172.21.15.43:0/1416575473 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:28.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7595e00640 1 --2- 172.21.15.43:0/1416575473 >> v2:172.21.15.43:3300/0 conn(0x7f759815d5b0 0x7f75980d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7595e00640 1 -- 172.21.15.43:0/1416575473 >> v1:172.21.15.69:6789/0 conn(0x7f75980d14a0 legacy=0x7f75980cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7595e00640 1 -- 172.21.15.43:0/1416575473 >> v1:172.21.15.43:6789/0 conn(0x7f75980b1650 legacy=0x7f75980cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7594a00640 1 -- 172.21.15.43:0/1416575473 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889712593 0 0) 0x7f75980cf0c0 con 0x7f75980b1650 2024-01-16T15:07:28.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.782+0000 7f7595e00640 1 -- 172.21.15.43:0/1416575473 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759815d990 con 0x7f759815d5b0 2024-01-16T15:07:28.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.784+0000 7f7595e00640 1 --2- 172.21.15.43:0/1416575473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f759815d5b0 0x7f75980d3990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f75880097a0 tx=0x7f758802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b88c9db94f8fa8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.784+0000 7f7594a00640 1 -- 172.21.15.43:0/1416575473 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588030610 con 0x7f759815d5b0 2024-01-16T15:07:28.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.784+0000 7f7594a00640 1 -- 172.21.15.43:0/1416575473 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588030770 con 0x7f759815d5b0 2024-01-16T15:07:28.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.784+0000 7f7594a00640 1 -- 172.21.15.43:0/1416575473 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588030b00 con 0x7f759815d5b0 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 -- 172.21.15.43:0/1416575473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f759815d5b0 msgr2=0x7f75980d3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 --2- 172.21.15.43:0/1416575473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f759815d5b0 0x7f75980d3990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f75880097a0 tx=0x7f758802b960 comp rx=0 tx=0).stop 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 -- 172.21.15.43:0/1416575473 shutdown_connections 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 --2- 172.21.15.43:0/1416575473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f759815d5b0 0x7f75980d3990 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 -- 172.21.15.43:0/1416575473 >> 172.21.15.43:0/1416575473 conn(0x7f759809c3b0 msgr2=0x7f75980d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 -- 172.21.15.43:0/1416575473 shutdown_connections 2024-01-16T15:07:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.785+0000 7f7597200640 1 -- 172.21.15.43:0/1416575473 wait complete. 2024-01-16T15:07:28.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.787+0000 7f7597200640 1 Processor -- start 2024-01-16T15:07:28.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.787+0000 7f7597200640 1 -- start start 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75980d14a0 0x7f75980c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759815d5b0 0x7f75980b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598152a20 con 0x7f75980b1650 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980b5c90 con 0x7f759815d5b0 2024-01-16T15:07:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7597200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980b5f70 con 0x7f75980d14a0 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55150/0 (socket says 172.21.15.43:55150) 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 -- 172.21.15.43:0/4105387409 learned_addr learned my addr 172.21.15.43:0/4105387409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 -- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75980d14a0 msgr2=0x7f75980c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75980d14a0 0x7f75980c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 -- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759815d5b0 msgr2=0x7f75980b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759815d5b0 0x7f75980b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.789+0000 7f7595e00640 1 -- 172.21.15.43:0/4105387409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588009400 con 0x7f75980b1650 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f7595e00640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f75880098c0 tx=0x7f7588004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758803e040 con 0x7f75980b1650 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588039cb0 con 0x7f75980b1650 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588044630 con 0x7f75980b1650 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75980b6250 con 0x7f75980b1650 2024-01-16T15:07:28.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.790+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980b6810 con 0x7f75980b1650 2024-01-16T15:07:28.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.791+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759804e840 con 0x7f75980b1650 2024-01-16T15:07:28.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.792+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7588043390 con 0x7f75980b1650 2024-01-16T15:07:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.794+0000 7f758e000640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 0x7f757006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.794+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f75880aaa30 con 0x7f75980b1650 2024-01-16T15:07:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.794+0000 7f7595400640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 0x7f757006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.795+0000 7f7595400640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 0x7f757006e080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f75800012e0 tx=0x7f7580007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.798+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7588077100 con 0x7f75980b1650 2024-01-16T15:07:28.931 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-01-16T15:07:28.953+0000 7f5921200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:07:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:28.994+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7598066d20 con 0x7f757006bbc0 2024-01-16T15:07:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.009+0000 7f758e000640 1 -- 172.21.15.43:0/4105387409 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286942 (secure 0 0 0) 0x7f7598066d20 con 0x7f757006bbc0 2024-01-16T15:07:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:28.993 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:07:28.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 msgr2=0x7f757006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 0x7f757006e080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f75800012e0 tx=0x7f7580007d30 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 msgr2=0x7f75981524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f75880098c0 tx=0x7f7588004730 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 shutdown_connections 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f757006bbc0 0x7f757006e080 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759815d5b0 0x7f75980b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75980d14a0 0x7f75980c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 --2- 172.21.15.43:0/4105387409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75980b1650 0x7f75981524e0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.016+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 >> 172.21.15.43:0/4105387409 conn(0x7f759809c3b0 msgr2=0x7f759815db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.017+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 shutdown_connections 2024-01-16T15:07:28.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.017+0000 7f7597200640 1 -- 172.21.15.43:0/4105387409 wait complete. 2024-01-16T15:07:29.017 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-01-16T15:07:26.994772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405964,"kb_used_data":7956,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568756,"statfs":{"total":773094113280,"available":772678406144,"internally_reserved":0,"allocated":8146944,"data_stored":4089988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":407424556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.016993"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045016+0000","last_change":"2024-01-16T15:07:24.036652+0000","last_active":"2024-01-16T15:07:24.045016+0000","last_peered":"2024-01-16T15:07:24.045016+0000","last_clean":"2024-01-16T15:07:24.045016+0000","last_became_active":"2024-01-16T15:07:24.036288+0000","last_became_peered":"2024-01-16T15:07:24.036288+0000","last_unstale":"2024-01-16T15:07:24.045016+0000","last_undegraded":"2024-01-16T15:07:24.045016+0000","last_fullsized":"2024-01-16T15:07:24.045016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:50.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.035110+0000","last_change":"2024-01-16T15:07:24.017571+0000","last_active":"2024-01-16T15:07:24.035110+0000","last_peered":"2024-01-16T15:07:24.035110+0000","last_clean":"2024-01-16T15:07:24.035110+0000","last_became_active":"2024-01-16T15:07:24.017183+0000","last_became_peered":"2024-01-16T15:07:24.017183+0000","last_unstale":"2024-01-16T15:07:24.035110+0000","last_undegraded":"2024-01-16T15:07:24.035110+0000","last_fullsized":"2024-01-16T15:07:24.035110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:59.209707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.031797+0000","last_change":"2024-01-16T15:07:24.029356+0000","last_active":"2024-01-16T15:07:24.031797+0000","last_peered":"2024-01-16T15:07:24.031797+0000","last_clean":"2024-01-16T15:07:24.031797+0000","last_became_active":"2024-01-16T15:07:24.028977+0000","last_became_peered":"2024-01-16T15:07:24.028977+0000","last_unstale":"2024-01-16T15:07:24.031797+0000","last_undegraded":"2024-01-16T15:07:24.031797+0000","last_fullsized":"2024-01-16T15:07:24.031797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:31.449130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001599+0000","last_change":"2024-01-16T15:07:24.037957+0000","last_active":"2024-01-16T15:07:25.001599+0000","last_peered":"2024-01-16T15:07:25.001599+0000","last_clean":"2024-01-16T15:07:25.001599+0000","last_became_active":"2024-01-16T15:07:24.037297+0000","last_became_peered":"2024-01-16T15:07:24.037297+0000","last_unstale":"2024-01-16T15:07:25.001599+0000","last_undegraded":"2024-01-16T15:07:25.001599+0000","last_fullsized":"2024-01-16T15:07:25.001599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:24.256412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999513+0000","last_change":"2024-01-16T15:07:24.027699+0000","last_active":"2024-01-16T15:07:24.999513+0000","last_peered":"2024-01-16T15:07:24.999513+0000","last_clean":"2024-01-16T15:07:24.999513+0000","last_became_active":"2024-01-16T15:07:24.027434+0000","last_became_peered":"2024-01-16T15:07:24.027434+0000","last_unstale":"2024-01-16T15:07:24.999513+0000","last_undegraded":"2024-01-16T15:07:24.999513+0000","last_fullsized":"2024-01-16T15:07:24.999513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:53.983350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.028767+0000","last_change":"2024-01-16T15:07:24.015524+0000","last_active":"2024-01-16T15:07:24.028767+0000","last_peered":"2024-01-16T15:07:24.028767+0000","last_clean":"2024-01-16T15:07:24.028767+0000","last_became_active":"2024-01-16T15:07:24.015042+0000","last_became_peered":"2024-01-16T15:07:24.015042+0000","last_unstale":"2024-01-16T15:07:24.028767+0000","last_undegraded":"2024-01-16T15:07:24.028767+0000","last_fullsized":"2024-01-16T15:07:24.028767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.655408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.030925+0000","last_change":"2024-01-16T15:07:24.025486+0000","last_active":"2024-01-16T15:07:24.030925+0000","last_peered":"2024-01-16T15:07:24.030925+0000","last_clean":"2024-01-16T15:07:24.030925+0000","last_became_active":"2024-01-16T15:07:24.025201+0000","last_became_peered":"2024-01-16T15:07:24.025201+0000","last_unstale":"2024-01-16T15:07:24.030925+0000","last_undegraded":"2024-01-16T15:07:24.030925+0000","last_fullsized":"2024-01-16T15:07:24.030925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:49.215397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038588+0000","last_change":"2024-01-16T15:07:24.027793+0000","last_active":"2024-01-16T15:07:24.038588+0000","last_peered":"2024-01-16T15:07:24.038588+0000","last_clean":"2024-01-16T15:07:24.038588+0000","last_became_active":"2024-01-16T15:07:24.027502+0000","last_became_peered":"2024-01-16T15:07:24.027502+0000","last_unstale":"2024-01-16T15:07:24.038588+0000","last_undegraded":"2024-01-16T15:07:24.038588+0000","last_fullsized":"2024-01-16T15:07:24.038588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:38.505806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999437+0000","last_change":"2024-01-16T15:07:24.032781+0000","last_active":"2024-01-16T15:07:24.999437+0000","last_peered":"2024-01-16T15:07:24.999437+0000","last_clean":"2024-01-16T15:07:24.999437+0000","last_became_active":"2024-01-16T15:07:24.032460+0000","last_became_peered":"2024-01-16T15:07:24.032460+0000","last_unstale":"2024-01-16T15:07:24.999437+0000","last_undegraded":"2024-01-16T15:07:24.999437+0000","last_fullsized":"2024-01-16T15:07:24.999437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:56.173386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003011+0000","last_change":"2024-01-16T15:07:24.031359+0000","last_active":"2024-01-16T15:07:25.003011+0000","last_peered":"2024-01-16T15:07:25.003011+0000","last_clean":"2024-01-16T15:07:25.003011+0000","last_became_active":"2024-01-16T15:07:24.029570+0000","last_became_peered":"2024-01-16T15:07:24.029570+0000","last_unstale":"2024-01-16T15:07:25.003011+0000","last_undegraded":"2024-01-16T15:07:25.003011+0000","last_fullsized":"2024-01-16T15:07:25.003011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:29.933024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.043484+0000","last_change":"2024-01-16T15:07:24.031503+0000","last_active":"2024-01-16T15:07:24.043484+0000","last_peered":"2024-01-16T15:07:24.043484+0000","last_clean":"2024-01-16T15:07:24.043484+0000","last_became_active":"2024-01-16T15:07:24.030065+0000","last_became_peered":"2024-01-16T15:07:24.030065+0000","last_unstale":"2024-01-16T15:07:24.043484+0000","last_undegraded":"2024-01-16T15:07:24.043484+0000","last_fullsized":"2024-01-16T15:07:24.043484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:16.331763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001685+0000","last_change":"2024-01-16T15:07:24.035513+0000","last_active":"2024-01-16T15:07:25.001685+0000","last_peered":"2024-01-16T15:07:25.001685+0000","last_clean":"2024-01-16T15:07:25.001685+0000","last_became_active":"2024-01-16T15:07:24.033958+0000","last_became_peered":"2024-01-16T15:07:24.033958+0000","last_unstale":"2024-01-16T15:07:25.001685+0000","last_undegraded":"2024-01-16T15:07:25.001685+0000","last_fullsized":"2024-01-16T15:07:25.001685+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:30.398205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.024859+0000","last_change":"2024-01-16T15:07:24.017333+0000","last_active":"2024-01-16T15:07:24.024859+0000","last_peered":"2024-01-16T15:07:24.024859+0000","last_clean":"2024-01-16T15:07:24.024859+0000","last_became_active":"2024-01-16T15:07:24.016711+0000","last_became_peered":"2024-01-16T15:07:24.016711+0000","last_unstale":"2024-01-16T15:07:24.024859+0000","last_undegraded":"2024-01-16T15:07:24.024859+0000","last_fullsized":"2024-01-16T15:07:24.024859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:30.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.000070+0000","last_change":"2024-01-16T15:07:24.035814+0000","last_active":"2024-01-16T15:07:25.000070+0000","last_peered":"2024-01-16T15:07:25.000070+0000","last_clean":"2024-01-16T15:07:25.000070+0000","last_became_active":"2024-01-16T15:07:24.034222+0000","last_became_peered":"2024-01-16T15:07:24.034222+0000","last_unstale":"2024-01-16T15:07:25.000070+0000","last_undegraded":"2024-01-16T15:07:25.000070+0000","last_fullsized":"2024-01-16T15:07:25.000070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:13:39.524640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.028925+0000","last_change":"2024-01-16T15:07:24.026978+0000","last_active":"2024-01-16T15:07:24.028925+0000","last_peered":"2024-01-16T15:07:24.028925+0000","last_clean":"2024-01-16T15:07:24.028925+0000","last_became_active":"2024-01-16T15:07:24.026705+0000","last_became_peered":"2024-01-16T15:07:24.026705+0000","last_unstale":"2024-01-16T15:07:24.028925+0000","last_undegraded":"2024-01-16T15:07:24.028925+0000","last_fullsized":"2024-01-16T15:07:24.028925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:26.803648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001365+0000","last_change":"2024-01-16T15:07:24.017363+0000","last_active":"2024-01-16T15:07:25.001365+0000","last_peered":"2024-01-16T15:07:25.001365+0000","last_clean":"2024-01-16T15:07:25.001365+0000","last_became_active":"2024-01-16T15:07:24.015531+0000","last_became_peered":"2024-01-16T15:07:24.015531+0000","last_unstale":"2024-01-16T15:07:25.001365+0000","last_undegraded":"2024-01-16T15:07:25.001365+0000","last_fullsized":"2024-01-16T15:07:25.001365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:09.853233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999584+0000","last_change":"2024-01-16T15:07:24.035335+0000","last_active":"2024-01-16T15:07:24.999584+0000","last_peered":"2024-01-16T15:07:24.999584+0000","last_clean":"2024-01-16T15:07:24.999584+0000","last_became_active":"2024-01-16T15:07:24.034988+0000","last_became_peered":"2024-01-16T15:07:24.034988+0000","last_unstale":"2024-01-16T15:07:24.999584+0000","last_undegraded":"2024-01-16T15:07:24.999584+0000","last_fullsized":"2024-01-16T15:07:24.999584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:21.644838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002219+0000","last_change":"2024-01-16T15:07:24.033655+0000","last_active":"2024-01-16T15:07:25.002219+0000","last_peered":"2024-01-16T15:07:25.002219+0000","last_clean":"2024-01-16T15:07:25.002219+0000","last_became_active":"2024-01-16T15:07:24.032653+0000","last_became_peered":"2024-01-16T15:07:24.032653+0000","last_unstale":"2024-01-16T15:07:25.002219+0000","last_undegraded":"2024-01-16T15:07:25.002219+0000","last_fullsized":"2024-01-16T15:07:25.002219+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:59.655408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037656+0000","last_change":"2024-01-16T15:07:24.026668+0000","last_active":"2024-01-16T15:07:24.037656+0000","last_peered":"2024-01-16T15:07:24.037656+0000","last_clean":"2024-01-16T15:07:24.037656+0000","last_became_active":"2024-01-16T15:07:24.026363+0000","last_became_peered":"2024-01-16T15:07:24.026363+0000","last_unstale":"2024-01-16T15:07:24.037656+0000","last_undegraded":"2024-01-16T15:07:24.037656+0000","last_fullsized":"2024-01-16T15:07:24.037656+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:47.962892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.027735+0000","last_change":"2024-01-16T15:07:24.025580+0000","last_active":"2024-01-16T15:07:24.027735+0000","last_peered":"2024-01-16T15:07:24.027735+0000","last_clean":"2024-01-16T15:07:24.027735+0000","last_became_active":"2024-01-16T15:07:24.023057+0000","last_became_peered":"2024-01-16T15:07:24.023057+0000","last_unstale":"2024-01-16T15:07:24.027735+0000","last_undegraded":"2024-01-16T15:07:24.027735+0000","last_fullsized":"2024-01-16T15:07:24.027735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:55.523923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002109+0000","last_change":"2024-01-16T15:07:24.011040+0000","last_active":"2024-01-16T15:07:25.002109+0000","last_peered":"2024-01-16T15:07:25.002109+0000","last_clean":"2024-01-16T15:07:25.002109+0000","last_became_active":"2024-01-16T15:07:24.010259+0000","last_became_peered":"2024-01-16T15:07:24.010259+0000","last_unstale":"2024-01-16T15:07:25.002109+0000","last_undegraded":"2024-01-16T15:07:25.002109+0000","last_fullsized":"2024-01-16T15:07:25.002109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:51.439554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001701+0000","last_change":"2024-01-16T15:07:24.030463+0000","last_active":"2024-01-16T15:07:25.001701+0000","last_peered":"2024-01-16T15:07:25.001701+0000","last_clean":"2024-01-16T15:07:25.001701+0000","last_became_active":"2024-01-16T15:07:24.029645+0000","last_became_peered":"2024-01-16T15:07:24.029645+0000","last_unstale":"2024-01-16T15:07:25.001701+0000","last_undegraded":"2024-01-16T15:07:25.001701+0000","last_fullsized":"2024-01-16T15:07:25.001701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:00.370425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044523+0000","last_change":"2024-01-16T15:07:24.033503+0000","last_active":"2024-01-16T15:07:24.044523+0000","last_peered":"2024-01-16T15:07:24.044523+0000","last_clean":"2024-01-16T15:07:24.044523+0000","last_became_active":"2024-01-16T15:07:24.032603+0000","last_became_peered":"2024-01-16T15:07:24.032603+0000","last_unstale":"2024-01-16T15:07:24.044523+0000","last_undegraded":"2024-01-16T15:07:24.044523+0000","last_fullsized":"2024-01-16T15:07:24.044523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:34.306686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.047829+0000","last_change":"2024-01-16T15:07:24.040015+0000","last_active":"2024-01-16T15:07:24.047829+0000","last_peered":"2024-01-16T15:07:24.047829+0000","last_clean":"2024-01-16T15:07:24.047829+0000","last_became_active":"2024-01-16T15:07:24.039767+0000","last_became_peered":"2024-01-16T15:07:24.039767+0000","last_unstale":"2024-01-16T15:07:24.047829+0000","last_undegraded":"2024-01-16T15:07:24.047829+0000","last_fullsized":"2024-01-16T15:07:24.047829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:40.006495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001758+0000","last_change":"2024-01-16T15:07:24.025703+0000","last_active":"2024-01-16T15:07:25.001758+0000","last_peered":"2024-01-16T15:07:25.001758+0000","last_clean":"2024-01-16T15:07:25.001758+0000","last_became_active":"2024-01-16T15:07:24.025367+0000","last_became_peered":"2024-01-16T15:07:24.025367+0000","last_unstale":"2024-01-16T15:07:25.001758+0000","last_undegraded":"2024-01-16T15:07:25.001758+0000","last_fullsized":"2024-01-16T15:07:25.001758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:30.667083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.040082+0000","last_change":"2024-01-16T15:07:24.038801+0000","last_active":"2024-01-16T15:07:24.040082+0000","last_peered":"2024-01-16T15:07:24.040082+0000","last_clean":"2024-01-16T15:07:24.040082+0000","last_became_active":"2024-01-16T15:07:24.038182+0000","last_became_peered":"2024-01-16T15:07:24.038182+0000","last_unstale":"2024-01-16T15:07:24.040082+0000","last_undegraded":"2024-01-16T15:07:24.040082+0000","last_fullsized":"2024-01-16T15:07:24.040082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:29.933024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.032540+0000","last_change":"2024-01-16T15:07:24.029437+0000","last_active":"2024-01-16T15:07:24.032540+0000","last_peered":"2024-01-16T15:07:24.032540+0000","last_clean":"2024-01-16T15:07:24.032540+0000","last_became_active":"2024-01-16T15:07:24.027019+0000","last_became_peered":"2024-01-16T15:07:24.027019+0000","last_unstale":"2024-01-16T15:07:24.032540+0000","last_undegraded":"2024-01-16T15:07:24.032540+0000","last_fullsized":"2024-01-16T15:07:24.032540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:01.387317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002626+0000","last_change":"2024-01-16T15:07:24.026705+0000","last_active":"2024-01-16T15:07:25.002626+0000","last_peered":"2024-01-16T15:07:25.002626+0000","last_clean":"2024-01-16T15:07:25.002626+0000","last_became_active":"2024-01-16T15:07:24.026434+0000","last_became_peered":"2024-01-16T15:07:24.026434+0000","last_unstale":"2024-01-16T15:07:25.002626+0000","last_undegraded":"2024-01-16T15:07:25.002626+0000","last_fullsized":"2024-01-16T15:07:25.002626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:52.675347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.032575+0000","last_change":"2024-01-16T15:07:24.030383+0000","last_active":"2024-01-16T15:07:24.032575+0000","last_peered":"2024-01-16T15:07:24.032575+0000","last_clean":"2024-01-16T15:07:24.032575+0000","last_became_active":"2024-01-16T15:07:24.030113+0000","last_became_peered":"2024-01-16T15:07:24.030113+0000","last_unstale":"2024-01-16T15:07:24.032575+0000","last_undegraded":"2024-01-16T15:07:24.032575+0000","last_fullsized":"2024-01-16T15:07:24.032575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:52.147402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.039812+0000","last_change":"2024-01-16T15:07:24.038390+0000","last_active":"2024-01-16T15:07:24.039812+0000","last_peered":"2024-01-16T15:07:24.039812+0000","last_clean":"2024-01-16T15:07:24.039812+0000","last_became_active":"2024-01-16T15:07:24.038085+0000","last_became_peered":"2024-01-16T15:07:24.038085+0000","last_unstale":"2024-01-16T15:07:24.039812+0000","last_undegraded":"2024-01-16T15:07:24.039812+0000","last_fullsized":"2024-01-16T15:07:24.039812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:30.667083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.039666+0000","last_change":"2024-01-16T15:07:24.035954+0000","last_active":"2024-01-16T15:07:24.039666+0000","last_peered":"2024-01-16T15:07:24.039666+0000","last_clean":"2024-01-16T15:07:24.039666+0000","last_became_active":"2024-01-16T15:07:24.035648+0000","last_became_peered":"2024-01-16T15:07:24.035648+0000","last_unstale":"2024-01-16T15:07:24.039666+0000","last_undegraded":"2024-01-16T15:07:24.039666+0000","last_fullsized":"2024-01-16T15:07:24.039666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:00.370425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.033712+0000","last_change":"2024-01-16T15:07:24.016680+0000","last_active":"2024-01-16T15:07:24.033712+0000","last_peered":"2024-01-16T15:07:24.033712+0000","last_clean":"2024-01-16T15:07:24.033712+0000","last_became_active":"2024-01-16T15:07:24.015079+0000","last_became_peered":"2024-01-16T15:07:24.015079+0000","last_unstale":"2024-01-16T15:07:24.033712+0000","last_undegraded":"2024-01-16T15:07:24.033712+0000","last_fullsized":"2024-01-16T15:07:24.033712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:11.466328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.047265+0000","last_change":"2024-01-16T15:07:24.039621+0000","last_active":"2024-01-16T15:07:24.047265+0000","last_peered":"2024-01-16T15:07:24.047265+0000","last_clean":"2024-01-16T15:07:24.047265+0000","last_became_active":"2024-01-16T15:07:24.039274+0000","last_became_peered":"2024-01-16T15:07:24.039274+0000","last_unstale":"2024-01-16T15:07:24.047265+0000","last_undegraded":"2024-01-16T15:07:24.047265+0000","last_fullsized":"2024-01-16T15:07:24.047265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:40:43.109312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049135+0000","last_change":"2024-01-16T15:07:23.023387+0000","last_active":"2024-01-16T15:07:24.049135+0000","last_peered":"2024-01-16T15:07:24.049135+0000","last_clean":"2024-01-16T15:07:24.049135+0000","last_became_active":"2024-01-16T15:07:23.023088+0000","last_became_peered":"2024-01-16T15:07:23.023088+0000","last_unstale":"2024-01-16T15:07:24.049135+0000","last_undegraded":"2024-01-16T15:07:24.049135+0000","last_fullsized":"2024-01-16T15:07:24.049135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:34.997413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999737+0000","last_change":"2024-01-16T15:07:24.032440+0000","last_active":"2024-01-16T15:07:24.999737+0000","last_peered":"2024-01-16T15:07:24.999737+0000","last_clean":"2024-01-16T15:07:24.999737+0000","last_became_active":"2024-01-16T15:07:24.029314+0000","last_became_peered":"2024-01-16T15:07:24.029314+0000","last_unstale":"2024-01-16T15:07:24.999737+0000","last_undegraded":"2024-01-16T15:07:24.999737+0000","last_fullsized":"2024-01-16T15:07:24.999737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.125133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.042534+0000","last_change":"2024-01-16T15:07:23.008387+0000","last_active":"2024-01-16T15:07:24.042534+0000","last_peered":"2024-01-16T15:07:24.042534+0000","last_clean":"2024-01-16T15:07:24.042534+0000","last_became_active":"2024-01-16T15:07:23.007916+0000","last_became_peered":"2024-01-16T15:07:23.007916+0000","last_unstale":"2024-01-16T15:07:24.042534+0000","last_undegraded":"2024-01-16T15:07:24.042534+0000","last_fullsized":"2024-01-16T15:07:24.042534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:27.067987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038522+0000","last_change":"2024-01-16T15:07:24.032667+0000","last_active":"2024-01-16T15:07:24.038522+0000","last_peered":"2024-01-16T15:07:24.038522+0000","last_clean":"2024-01-16T15:07:24.038522+0000","last_became_active":"2024-01-16T15:07:24.032097+0000","last_became_peered":"2024-01-16T15:07:24.032097+0000","last_unstale":"2024-01-16T15:07:24.038522+0000","last_undegraded":"2024-01-16T15:07:24.038522+0000","last_fullsized":"2024-01-16T15:07:24.038522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:19.018624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002127+0000","last_change":"2024-01-16T15:07:23.012327+0000","last_active":"2024-01-16T15:07:25.002127+0000","last_peered":"2024-01-16T15:07:25.002127+0000","last_clean":"2024-01-16T15:07:25.002127+0000","last_became_active":"2024-01-16T15:07:23.011997+0000","last_became_peered":"2024-01-16T15:07:23.011997+0000","last_unstale":"2024-01-16T15:07:25.002127+0000","last_undegraded":"2024-01-16T15:07:25.002127+0000","last_fullsized":"2024-01-16T15:07:25.002127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:00.664016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045318+0000","last_change":"2024-01-16T15:07:24.036752+0000","last_active":"2024-01-16T15:07:24.045318+0000","last_peered":"2024-01-16T15:07:24.045318+0000","last_clean":"2024-01-16T15:07:24.045318+0000","last_became_active":"2024-01-16T15:07:24.035229+0000","last_became_peered":"2024-01-16T15:07:24.035229+0000","last_unstale":"2024-01-16T15:07:24.045318+0000","last_undegraded":"2024-01-16T15:07:24.045318+0000","last_fullsized":"2024-01-16T15:07:24.045318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:15.610651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049180+0000","last_change":"2024-01-16T15:07:23.016160+0000","last_active":"2024-01-16T15:07:24.049180+0000","last_peered":"2024-01-16T15:07:24.049180+0000","last_clean":"2024-01-16T15:07:24.049180+0000","last_became_active":"2024-01-16T15:07:23.015624+0000","last_became_peered":"2024-01-16T15:07:23.015624+0000","last_unstale":"2024-01-16T15:07:24.049180+0000","last_undegraded":"2024-01-16T15:07:24.049180+0000","last_fullsized":"2024-01-16T15:07:24.049180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:57.033174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038754+0000","last_change":"2024-01-16T15:07:24.031127+0000","last_active":"2024-01-16T15:07:24.038754+0000","last_peered":"2024-01-16T15:07:24.038754+0000","last_clean":"2024-01-16T15:07:24.038754+0000","last_became_active":"2024-01-16T15:07:24.030845+0000","last_became_peered":"2024-01-16T15:07:24.030845+0000","last_unstale":"2024-01-16T15:07:24.038754+0000","last_undegraded":"2024-01-16T15:07:24.038754+0000","last_fullsized":"2024-01-16T15:07:24.038754+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:09.853233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.043922+0000","last_change":"2024-01-16T15:07:23.011619+0000","last_active":"2024-01-16T15:07:24.043922+0000","last_peered":"2024-01-16T15:07:24.043922+0000","last_clean":"2024-01-16T15:07:24.043922+0000","last_became_active":"2024-01-16T15:07:23.011220+0000","last_became_peered":"2024-01-16T15:07:23.011220+0000","last_unstale":"2024-01-16T15:07:24.043922+0000","last_undegraded":"2024-01-16T15:07:24.043922+0000","last_fullsized":"2024-01-16T15:07:24.043922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:36.489899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.002470+0000","last_change":"2024-01-16T15:07:24.025622+0000","last_active":"2024-01-16T15:07:25.002470+0000","last_peered":"2024-01-16T15:07:25.002470+0000","last_clean":"2024-01-16T15:07:25.002470+0000","last_became_active":"2024-01-16T15:07:24.025303+0000","last_became_peered":"2024-01-16T15:07:24.025303+0000","last_unstale":"2024-01-16T15:07:25.002470+0000","last_undegraded":"2024-01-16T15:07:25.002470+0000","last_fullsized":"2024-01-16T15:07:25.002470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:58.049449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048652+0000","last_change":"2024-01-16T15:07:23.014210+0000","last_active":"2024-01-16T15:07:24.048652+0000","last_peered":"2024-01-16T15:07:24.048652+0000","last_clean":"2024-01-16T15:07:24.048652+0000","last_became_active":"2024-01-16T15:07:23.005942+0000","last_became_peered":"2024-01-16T15:07:23.005942+0000","last_unstale":"2024-01-16T15:07:24.048652+0000","last_undegraded":"2024-01-16T15:07:24.048652+0000","last_fullsized":"2024-01-16T15:07:24.048652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:25:32.507386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998727+0000","last_change":"2024-01-16T15:07:24.030303+0000","last_active":"2024-01-16T15:07:24.998727+0000","last_peered":"2024-01-16T15:07:24.998727+0000","last_clean":"2024-01-16T15:07:24.998727+0000","last_became_active":"2024-01-16T15:07:24.027753+0000","last_became_peered":"2024-01-16T15:07:24.027753+0000","last_unstale":"2024-01-16T15:07:24.998727+0000","last_undegraded":"2024-01-16T15:07:24.998727+0000","last_fullsized":"2024-01-16T15:07:24.998727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:26:44.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001337+0000","last_change":"2024-01-16T15:07:23.017741+0000","last_active":"2024-01-16T15:07:25.001337+0000","last_peered":"2024-01-16T15:07:25.001337+0000","last_clean":"2024-01-16T15:07:25.001337+0000","last_became_active":"2024-01-16T15:07:23.017445+0000","last_became_peered":"2024-01-16T15:07:23.017445+0000","last_unstale":"2024-01-16T15:07:25.001337+0000","last_undegraded":"2024-01-16T15:07:25.001337+0000","last_fullsized":"2024-01-16T15:07:25.001337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:29.911078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.026466+0000","last_change":"2024-01-16T15:07:24.024151+0000","last_active":"2024-01-16T15:07:24.026466+0000","last_peered":"2024-01-16T15:07:24.026466+0000","last_clean":"2024-01-16T15:07:24.026466+0000","last_became_active":"2024-01-16T15:07:24.023824+0000","last_became_peered":"2024-01-16T15:07:24.023824+0000","last_unstale":"2024-01-16T15:07:24.026466+0000","last_undegraded":"2024-01-16T15:07:24.026466+0000","last_fullsized":"2024-01-16T15:07:24.026466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:19.908915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998695+0000","last_change":"2024-01-16T15:07:23.012897+0000","last_active":"2024-01-16T15:07:24.998695+0000","last_peered":"2024-01-16T15:07:24.998695+0000","last_clean":"2024-01-16T15:07:24.998695+0000","last_became_active":"2024-01-16T15:07:23.012628+0000","last_became_peered":"2024-01-16T15:07:23.012628+0000","last_unstale":"2024-01-16T15:07:24.998695+0000","last_undegraded":"2024-01-16T15:07:24.998695+0000","last_fullsized":"2024-01-16T15:07:24.998695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:35:19.638548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.031542+0000","last_change":"2024-01-16T15:07:24.027138+0000","last_active":"2024-01-16T15:07:24.031542+0000","last_peered":"2024-01-16T15:07:24.031542+0000","last_clean":"2024-01-16T15:07:24.031542+0000","last_became_active":"2024-01-16T15:07:24.026870+0000","last_became_peered":"2024-01-16T15:07:24.026870+0000","last_unstale":"2024-01-16T15:07:24.031542+0000","last_undegraded":"2024-01-16T15:07:24.031542+0000","last_fullsized":"2024-01-16T15:07:24.031542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:51.439554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001948+0000","last_change":"2024-01-16T15:07:23.013637+0000","last_active":"2024-01-16T15:07:25.001948+0000","last_peered":"2024-01-16T15:07:25.001948+0000","last_clean":"2024-01-16T15:07:25.001948+0000","last_became_active":"2024-01-16T15:07:23.013141+0000","last_became_peered":"2024-01-16T15:07:23.013141+0000","last_unstale":"2024-01-16T15:07:25.001948+0000","last_undegraded":"2024-01-16T15:07:25.001948+0000","last_fullsized":"2024-01-16T15:07:25.001948+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:25:32.507386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045033+0000","last_change":"2024-01-16T15:07:24.038413+0000","last_active":"2024-01-16T15:07:24.045033+0000","last_peered":"2024-01-16T15:07:24.045033+0000","last_clean":"2024-01-16T15:07:24.045033+0000","last_became_active":"2024-01-16T15:07:24.038121+0000","last_became_peered":"2024-01-16T15:07:24.038121+0000","last_unstale":"2024-01-16T15:07:24.045033+0000","last_undegraded":"2024-01-16T15:07:24.045033+0000","last_fullsized":"2024-01-16T15:07:24.045033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:46.416261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.011236+0000","last_change":"2024-01-16T15:07:23.018594+0000","last_active":"2024-01-16T15:07:24.011236+0000","last_peered":"2024-01-16T15:07:24.011236+0000","last_clean":"2024-01-16T15:07:24.011236+0000","last_became_active":"2024-01-16T15:07:23.018195+0000","last_became_peered":"2024-01-16T15:07:23.018195+0000","last_unstale":"2024-01-16T15:07:24.011236+0000","last_undegraded":"2024-01-16T15:07:24.011236+0000","last_fullsized":"2024-01-16T15:07:24.011236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:51.659072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.035320+0000","last_change":"2024-01-16T15:07:24.018277+0000","last_active":"2024-01-16T15:07:24.035320+0000","last_peered":"2024-01-16T15:07:24.035320+0000","last_clean":"2024-01-16T15:07:24.035320+0000","last_became_active":"2024-01-16T15:07:24.017965+0000","last_became_peered":"2024-01-16T15:07:24.017965+0000","last_unstale":"2024-01-16T15:07:24.035320+0000","last_undegraded":"2024-01-16T15:07:24.035320+0000","last_fullsized":"2024-01-16T15:07:24.035320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:01.680291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.050254+0000","last_change":"2024-01-16T15:07:23.018628+0000","last_active":"2024-01-16T15:07:24.050254+0000","last_peered":"2024-01-16T15:07:24.050254+0000","last_clean":"2024-01-16T15:07:24.050254+0000","last_became_active":"2024-01-16T15:07:23.018030+0000","last_became_peered":"2024-01-16T15:07:23.018030+0000","last_unstale":"2024-01-16T15:07:24.050254+0000","last_undegraded":"2024-01-16T15:07:24.050254+0000","last_fullsized":"2024-01-16T15:07:24.050254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:13:29.841768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999853+0000","last_change":"2024-01-16T15:07:24.029472+0000","last_active":"2024-01-16T15:07:24.999853+0000","last_peered":"2024-01-16T15:07:24.999853+0000","last_clean":"2024-01-16T15:07:24.999853+0000","last_became_active":"2024-01-16T15:07:24.029210+0000","last_became_peered":"2024-01-16T15:07:24.029210+0000","last_unstale":"2024-01-16T15:07:24.999853+0000","last_undegraded":"2024-01-16T15:07:24.999853+0000","last_fullsized":"2024-01-16T15:07:24.999853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:55:30.927352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001974+0000","last_change":"2024-01-16T15:07:23.019932+0000","last_active":"2024-01-16T15:07:25.001974+0000","last_peered":"2024-01-16T15:07:25.001974+0000","last_clean":"2024-01-16T15:07:25.001974+0000","last_became_active":"2024-01-16T15:07:23.019172+0000","last_became_peered":"2024-01-16T15:07:23.019172+0000","last_unstale":"2024-01-16T15:07:25.001974+0000","last_undegraded":"2024-01-16T15:07:25.001974+0000","last_fullsized":"2024-01-16T15:07:25.001974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:11.290476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044929+0000","last_change":"2024-01-16T15:07:24.038072+0000","last_active":"2024-01-16T15:07:24.044929+0000","last_peered":"2024-01-16T15:07:24.044929+0000","last_clean":"2024-01-16T15:07:24.044929+0000","last_became_active":"2024-01-16T15:07:24.037696+0000","last_became_peered":"2024-01-16T15:07:24.037696+0000","last_unstale":"2024-01-16T15:07:24.044929+0000","last_undegraded":"2024-01-16T15:07:24.044929+0000","last_fullsized":"2024-01-16T15:07:24.044929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:29.933339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001673+0000","last_change":"2024-01-16T15:07:23.020474+0000","last_active":"2024-01-16T15:07:25.001673+0000","last_peered":"2024-01-16T15:07:25.001673+0000","last_clean":"2024-01-16T15:07:25.001673+0000","last_became_active":"2024-01-16T15:07:23.019485+0000","last_became_peered":"2024-01-16T15:07:23.019485+0000","last_unstale":"2024-01-16T15:07:25.001673+0000","last_undegraded":"2024-01-16T15:07:25.001673+0000","last_fullsized":"2024-01-16T15:07:25.001673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:44.030793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.029139+0000","last_change":"2024-01-16T15:07:24.021617+0000","last_active":"2024-01-16T15:07:24.029139+0000","last_peered":"2024-01-16T15:07:24.029139+0000","last_clean":"2024-01-16T15:07:24.029139+0000","last_became_active":"2024-01-16T15:07:24.021272+0000","last_became_peered":"2024-01-16T15:07:24.021272+0000","last_unstale":"2024-01-16T15:07:24.029139+0000","last_undegraded":"2024-01-16T15:07:24.029139+0000","last_fullsized":"2024-01-16T15:07:24.029139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:58.049449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.010044+0000","last_change":"2024-01-16T15:07:23.008356+0000","last_active":"2024-01-16T15:07:24.010044+0000","last_peered":"2024-01-16T15:07:24.010044+0000","last_clean":"2024-01-16T15:07:24.010044+0000","last_became_active":"2024-01-16T15:07:23.007780+0000","last_became_peered":"2024-01-16T15:07:23.007780+0000","last_unstale":"2024-01-16T15:07:24.010044+0000","last_undegraded":"2024-01-16T15:07:24.010044+0000","last_fullsized":"2024-01-16T15:07:24.010044+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:08.836959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.025085+0000","last_change":"2024-01-16T15:07:24.015178+0000","last_active":"2024-01-16T15:07:24.025085+0000","last_peered":"2024-01-16T15:07:24.025085+0000","last_clean":"2024-01-16T15:07:24.025085+0000","last_became_active":"2024-01-16T15:07:24.014560+0000","last_became_peered":"2024-01-16T15:07:24.014560+0000","last_unstale":"2024-01-16T15:07:24.025085+0000","last_undegraded":"2024-01-16T15:07:24.025085+0000","last_fullsized":"2024-01-16T15:07:24.025085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:11:58.571129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.017509+0000","last_change":"2024-01-16T15:07:23.019681+0000","last_active":"2024-01-16T15:07:24.017509+0000","last_peered":"2024-01-16T15:07:24.017509+0000","last_clean":"2024-01-16T15:07:24.017509+0000","last_became_active":"2024-01-16T15:07:23.019392+0000","last_became_peered":"2024-01-16T15:07:23.019392+0000","last_unstale":"2024-01-16T15:07:24.017509+0000","last_undegraded":"2024-01-16T15:07:24.017509+0000","last_fullsized":"2024-01-16T15:07:24.017509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:24.856680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.998905+0000","last_change":"2024-01-16T15:07:24.028679+0000","last_active":"2024-01-16T15:07:24.998905+0000","last_peered":"2024-01-16T15:07:24.998905+0000","last_clean":"2024-01-16T15:07:24.998905+0000","last_became_active":"2024-01-16T15:07:24.028384+0000","last_became_peered":"2024-01-16T15:07:24.028384+0000","last_unstale":"2024-01-16T15:07:24.998905+0000","last_undegraded":"2024-01-16T15:07:24.998905+0000","last_fullsized":"2024-01-16T15:07:24.998905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:01.469289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001409+0000","last_change":"2024-01-16T15:07:23.014879+0000","last_active":"2024-01-16T15:07:25.001409+0000","last_peered":"2024-01-16T15:07:25.001409+0000","last_clean":"2024-01-16T15:07:25.001409+0000","last_became_active":"2024-01-16T15:07:23.010432+0000","last_became_peered":"2024-01-16T15:07:23.010432+0000","last_unstale":"2024-01-16T15:07:25.001409+0000","last_undegraded":"2024-01-16T15:07:25.001409+0000","last_fullsized":"2024-01-16T15:07:25.001409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:10.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003183+0000","last_change":"2024-01-16T15:07:24.021543+0000","last_active":"2024-01-16T15:07:25.003183+0000","last_peered":"2024-01-16T15:07:25.003183+0000","last_clean":"2024-01-16T15:07:25.003183+0000","last_became_active":"2024-01-16T15:07:24.021130+0000","last_became_peered":"2024-01-16T15:07:24.021130+0000","last_unstale":"2024-01-16T15:07:25.003183+0000","last_undegraded":"2024-01-16T15:07:25.003183+0000","last_fullsized":"2024-01-16T15:07:25.003183+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:49.215397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049925+0000","last_change":"2024-01-16T15:07:23.014053+0000","last_active":"2024-01-16T15:07:24.049925+0000","last_peered":"2024-01-16T15:07:24.049925+0000","last_clean":"2024-01-16T15:07:24.049925+0000","last_became_active":"2024-01-16T15:07:23.012679+0000","last_became_peered":"2024-01-16T15:07:23.012679+0000","last_unstale":"2024-01-16T15:07:24.049925+0000","last_undegraded":"2024-01-16T15:07:24.049925+0000","last_fullsized":"2024-01-16T15:07:24.049925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:14:59.354150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.044991+0000","last_change":"2024-01-16T15:07:24.034377+0000","last_active":"2024-01-16T15:07:24.044991+0000","last_peered":"2024-01-16T15:07:24.044991+0000","last_clean":"2024-01-16T15:07:24.044991+0000","last_became_active":"2024-01-16T15:07:24.034114+0000","last_became_peered":"2024-01-16T15:07:24.034114+0000","last_unstale":"2024-01-16T15:07:24.044991+0000","last_undegraded":"2024-01-16T15:07:24.044991+0000","last_fullsized":"2024-01-16T15:07:24.044991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:49.482278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.003233+0000","last_change":"2024-01-16T15:07:23.014544+0000","last_active":"2024-01-16T15:07:25.003233+0000","last_peered":"2024-01-16T15:07:25.003233+0000","last_clean":"2024-01-16T15:07:25.003233+0000","last_became_active":"2024-01-16T15:07:23.014196+0000","last_became_peered":"2024-01-16T15:07:23.014196+0000","last_unstale":"2024-01-16T15:07:25.003233+0000","last_undegraded":"2024-01-16T15:07:25.003233+0000","last_fullsized":"2024-01-16T15:07:25.003233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:37.489532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001802+0000","last_change":"2024-01-16T15:07:24.034662+0000","last_active":"2024-01-16T15:07:25.001802+0000","last_peered":"2024-01-16T15:07:25.001802+0000","last_clean":"2024-01-16T15:07:25.001802+0000","last_became_active":"2024-01-16T15:07:24.030691+0000","last_became_peered":"2024-01-16T15:07:24.030691+0000","last_unstale":"2024-01-16T15:07:25.001802+0000","last_undegraded":"2024-01-16T15:07:25.001802+0000","last_fullsized":"2024-01-16T15:07:25.001802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:20:01.680291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.045599+0000","last_change":"2024-01-16T15:07:23.020613+0000","last_active":"2024-01-16T15:07:24.045599+0000","last_peered":"2024-01-16T15:07:24.045599+0000","last_clean":"2024-01-16T15:07:24.045599+0000","last_became_active":"2024-01-16T15:07:23.013608+0000","last_became_peered":"2024-01-16T15:07:23.013608+0000","last_unstale":"2024-01-16T15:07:24.045599+0000","last_undegraded":"2024-01-16T15:07:24.045599+0000","last_fullsized":"2024-01-16T15:07:24.045599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:28.916750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999881+0000","last_change":"2024-01-16T15:07:24.030061+0000","last_active":"2024-01-16T15:07:24.999881+0000","last_peered":"2024-01-16T15:07:24.999881+0000","last_clean":"2024-01-16T15:07:24.999881+0000","last_became_active":"2024-01-16T15:07:24.027485+0000","last_became_peered":"2024-01-16T15:07:24.027485+0000","last_unstale":"2024-01-16T15:07:24.999881+0000","last_undegraded":"2024-01-16T15:07:24.999881+0000","last_fullsized":"2024-01-16T15:07:24.999881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:38.474326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.022751+0000","last_change":"2024-01-16T15:07:23.024687+0000","last_active":"2024-01-16T15:07:24.022751+0000","last_peered":"2024-01-16T15:07:24.022751+0000","last_clean":"2024-01-16T15:07:24.022751+0000","last_became_active":"2024-01-16T15:07:23.024293+0000","last_became_peered":"2024-01-16T15:07:23.024293+0000","last_unstale":"2024-01-16T15:07:24.022751+0000","last_undegraded":"2024-01-16T15:07:24.022751+0000","last_fullsized":"2024-01-16T15:07:24.022751+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:08.806948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.026560+0000","last_change":"2024-01-16T15:07:24.021885+0000","last_active":"2024-01-16T15:07:24.026560+0000","last_peered":"2024-01-16T15:07:24.026560+0000","last_clean":"2024-01-16T15:07:24.026560+0000","last_became_active":"2024-01-16T15:07:24.021567+0000","last_became_peered":"2024-01-16T15:07:24.021567+0000","last_unstale":"2024-01-16T15:07:24.026560+0000","last_undegraded":"2024-01-16T15:07:24.026560+0000","last_fullsized":"2024-01-16T15:07:24.026560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.858660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.010276+0000","last_change":"2024-01-16T15:07:23.008169+0000","last_active":"2024-01-16T15:07:24.010276+0000","last_peered":"2024-01-16T15:07:24.010276+0000","last_clean":"2024-01-16T15:07:24.010276+0000","last_became_active":"2024-01-16T15:07:23.007390+0000","last_became_peered":"2024-01-16T15:07:23.007390+0000","last_unstale":"2024-01-16T15:07:24.010276+0000","last_undegraded":"2024-01-16T15:07:24.010276+0000","last_fullsized":"2024-01-16T15:07:24.010276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:29.650808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038854+0000","last_change":"2024-01-16T15:07:24.034170+0000","last_active":"2024-01-16T15:07:24.038854+0000","last_peered":"2024-01-16T15:07:24.038854+0000","last_clean":"2024-01-16T15:07:24.038854+0000","last_became_active":"2024-01-16T15:07:24.033845+0000","last_became_peered":"2024-01-16T15:07:24.033845+0000","last_unstale":"2024-01-16T15:07:24.038854+0000","last_undegraded":"2024-01-16T15:07:24.038854+0000","last_fullsized":"2024-01-16T15:07:24.038854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:20.223725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019092+0000","last_change":"2024-01-16T15:07:23.018354+0000","last_active":"2024-01-16T15:07:24.019092+0000","last_peered":"2024-01-16T15:07:24.019092+0000","last_clean":"2024-01-16T15:07:24.019092+0000","last_became_active":"2024-01-16T15:07:23.017990+0000","last_became_peered":"2024-01-16T15:07:23.017990+0000","last_unstale":"2024-01-16T15:07:24.019092+0000","last_undegraded":"2024-01-16T15:07:24.019092+0000","last_fullsized":"2024-01-16T15:07:24.019092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:59.924301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.036425+0000","last_change":"2024-01-16T15:07:24.033827+0000","last_active":"2024-01-16T15:07:24.036425+0000","last_peered":"2024-01-16T15:07:24.036425+0000","last_clean":"2024-01-16T15:07:24.036425+0000","last_became_active":"2024-01-16T15:07:24.033520+0000","last_became_peered":"2024-01-16T15:07:24.033520+0000","last_unstale":"2024-01-16T15:07:24.036425+0000","last_undegraded":"2024-01-16T15:07:24.036425+0000","last_fullsized":"2024-01-16T15:07:24.036425+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:59:57.020614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:23.998159+0000","last_change":"2024-01-16T15:07:23.013667+0000","last_active":"2024-01-16T15:07:23.998159+0000","last_peered":"2024-01-16T15:07:23.998159+0000","last_clean":"2024-01-16T15:07:23.998159+0000","last_became_active":"2024-01-16T15:07:23.013388+0000","last_became_peered":"2024-01-16T15:07:23.013388+0000","last_unstale":"2024-01-16T15:07:23.998159+0000","last_undegraded":"2024-01-16T15:07:23.998159+0000","last_fullsized":"2024-01-16T15:07:23.998159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:51.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.038025+0000","last_change":"2024-01-16T15:07:24.034844+0000","last_active":"2024-01-16T15:07:24.038025+0000","last_peered":"2024-01-16T15:07:24.038025+0000","last_clean":"2024-01-16T15:07:24.038025+0000","last_became_active":"2024-01-16T15:07:24.034414+0000","last_became_peered":"2024-01-16T15:07:24.034414+0000","last_unstale":"2024-01-16T15:07:24.038025+0000","last_undegraded":"2024-01-16T15:07:24.038025+0000","last_fullsized":"2024-01-16T15:07:24.038025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:01:02.946349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048906+0000","last_change":"2024-01-16T15:07:23.013248+0000","last_active":"2024-01-16T15:07:24.048906+0000","last_peered":"2024-01-16T15:07:24.048906+0000","last_clean":"2024-01-16T15:07:24.048906+0000","last_became_active":"2024-01-16T15:07:23.012273+0000","last_became_peered":"2024-01-16T15:07:23.012273+0000","last_unstale":"2024-01-16T15:07:24.048906+0000","last_undegraded":"2024-01-16T15:07:24.048906+0000","last_fullsized":"2024-01-16T15:07:24.048906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:31:50.423279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.030721+0000","last_change":"2024-01-16T15:07:24.024871+0000","last_active":"2024-01-16T15:07:24.030721+0000","last_peered":"2024-01-16T15:07:24.030721+0000","last_clean":"2024-01-16T15:07:24.030721+0000","last_became_active":"2024-01-16T15:07:24.022629+0000","last_became_peered":"2024-01-16T15:07:24.022629+0000","last_unstale":"2024-01-16T15:07:24.030721+0000","last_undegraded":"2024-01-16T15:07:24.030721+0000","last_fullsized":"2024-01-16T15:07:24.030721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:52.675347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019251+0000","last_change":"2024-01-16T15:07:23.020610+0000","last_active":"2024-01-16T15:07:24.019251+0000","last_peered":"2024-01-16T15:07:24.019251+0000","last_clean":"2024-01-16T15:07:24.019251+0000","last_became_active":"2024-01-16T15:07:23.020290+0000","last_became_peered":"2024-01-16T15:07:23.020290+0000","last_unstale":"2024-01-16T15:07:24.019251+0000","last_undegraded":"2024-01-16T15:07:24.019251+0000","last_fullsized":"2024-01-16T15:07:24.019251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:35:46.868161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.046833+0000","last_change":"2024-01-16T15:07:24.040330+0000","last_active":"2024-01-16T15:07:24.046833+0000","last_peered":"2024-01-16T15:07:24.046833+0000","last_clean":"2024-01-16T15:07:24.046833+0000","last_became_active":"2024-01-16T15:07:24.040053+0000","last_became_peered":"2024-01-16T15:07:24.040053+0000","last_unstale":"2024-01-16T15:07:24.046833+0000","last_undegraded":"2024-01-16T15:07:24.046833+0000","last_fullsized":"2024-01-16T15:07:24.046833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:39.224060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999985+0000","last_change":"2024-01-16T15:07:23.008633+0000","last_active":"2024-01-16T15:07:24.999985+0000","last_peered":"2024-01-16T15:07:24.999985+0000","last_clean":"2024-01-16T15:07:24.999985+0000","last_became_active":"2024-01-16T15:07:23.008193+0000","last_became_peered":"2024-01-16T15:07:23.008193+0000","last_unstale":"2024-01-16T15:07:24.999985+0000","last_undegraded":"2024-01-16T15:07:24.999985+0000","last_fullsized":"2024-01-16T15:07:24.999985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:15.276116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.048227+0000","last_change":"2024-01-16T15:07:24.043384+0000","last_active":"2024-01-16T15:07:24.048227+0000","last_peered":"2024-01-16T15:07:24.048227+0000","last_clean":"2024-01-16T15:07:24.048227+0000","last_became_active":"2024-01-16T15:07:24.039785+0000","last_became_peered":"2024-01-16T15:07:24.039785+0000","last_unstale":"2024-01-16T15:07:24.048227+0000","last_undegraded":"2024-01-16T15:07:24.048227+0000","last_fullsized":"2024-01-16T15:07:24.048227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:22:16.076911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:23.999252+0000","last_change":"2024-01-16T15:07:23.014030+0000","last_active":"2024-01-16T15:07:23.999252+0000","last_peered":"2024-01-16T15:07:23.999252+0000","last_clean":"2024-01-16T15:07:23.999252+0000","last_became_active":"2024-01-16T15:07:23.012317+0000","last_became_peered":"2024-01-16T15:07:23.012317+0000","last_unstale":"2024-01-16T15:07:23.999252+0000","last_undegraded":"2024-01-16T15:07:23.999252+0000","last_fullsized":"2024-01-16T15:07:23.999252+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:40.188168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999311+0000","last_change":"2024-01-16T15:07:24.029017+0000","last_active":"2024-01-16T15:07:24.999311+0000","last_peered":"2024-01-16T15:07:24.999311+0000","last_clean":"2024-01-16T15:07:24.999311+0000","last_became_active":"2024-01-16T15:07:24.028121+0000","last_became_peered":"2024-01-16T15:07:24.028121+0000","last_unstale":"2024-01-16T15:07:24.999311+0000","last_undegraded":"2024-01-16T15:07:24.999311+0000","last_fullsized":"2024-01-16T15:07:24.999311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:52:14.141767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.999630+0000","last_change":"2024-01-16T15:07:23.018693+0000","last_active":"2024-01-16T15:07:24.999630+0000","last_peered":"2024-01-16T15:07:24.999630+0000","last_clean":"2024-01-16T15:07:24.999630+0000","last_became_active":"2024-01-16T15:07:23.018428+0000","last_became_peered":"2024-01-16T15:07:23.018428+0000","last_unstale":"2024-01-16T15:07:24.999630+0000","last_undegraded":"2024-01-16T15:07:24.999630+0000","last_fullsized":"2024-01-16T15:07:24.999630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:03.545845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037775+0000","last_change":"2024-01-16T15:07:24.027194+0000","last_active":"2024-01-16T15:07:24.037775+0000","last_peered":"2024-01-16T15:07:24.037775+0000","last_clean":"2024-01-16T15:07:24.037775+0000","last_became_active":"2024-01-16T15:07:24.026587+0000","last_became_peered":"2024-01-16T15:07:24.026587+0000","last_unstale":"2024-01-16T15:07:24.037775+0000","last_undegraded":"2024-01-16T15:07:24.037775+0000","last_fullsized":"2024-01-16T15:07:24.037775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:30.398205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001272+0000","last_change":"2024-01-16T15:07:23.016781+0000","last_active":"2024-01-16T15:07:25.001272+0000","last_peered":"2024-01-16T15:07:25.001272+0000","last_clean":"2024-01-16T15:07:25.001272+0000","last_became_active":"2024-01-16T15:07:23.016451+0000","last_became_peered":"2024-01-16T15:07:23.016451+0000","last_unstale":"2024-01-16T15:07:25.001272+0000","last_undegraded":"2024-01-16T15:07:25.001272+0000","last_fullsized":"2024-01-16T15:07:25.001272+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:13:38.508365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001363+0000","last_change":"2024-01-16T15:07:24.023693+0000","last_active":"2024-01-16T15:07:25.001363+0000","last_peered":"2024-01-16T15:07:25.001363+0000","last_clean":"2024-01-16T15:07:25.001363+0000","last_became_active":"2024-01-16T15:07:24.023402+0000","last_became_peered":"2024-01-16T15:07:24.023402+0000","last_unstale":"2024-01-16T15:07:25.001363+0000","last_undegraded":"2024-01-16T15:07:25.001363+0000","last_fullsized":"2024-01-16T15:07:25.001363+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:11.466328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:07:25.001303+0000","last_change":"2024-01-16T15:07:23.015966+0000","last_active":"2024-01-16T15:07:25.001303+0000","last_peered":"2024-01-16T15:07:25.001303+0000","last_clean":"2024-01-16T15:07:25.001303+0000","last_became_active":"2024-01-16T15:07:23.015544+0000","last_became_peered":"2024-01-16T15:07:23.015544+0000","last_unstale":"2024-01-16T15:07:25.001303+0000","last_undegraded":"2024-01-16T15:07:25.001303+0000","last_fullsized":"2024-01-16T15:07:25.001303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:18.002349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.037264+0000","last_change":"2024-01-16T15:07:24.025123+0000","last_active":"2024-01-16T15:07:24.037264+0000","last_peered":"2024-01-16T15:07:24.037264+0000","last_clean":"2024-01-16T15:07:24.037264+0000","last_became_active":"2024-01-16T15:07:24.024226+0000","last_became_peered":"2024-01-16T15:07:24.024226+0000","last_unstale":"2024-01-16T15:07:24.037264+0000","last_undegraded":"2024-01-16T15:07:24.037264+0000","last_fullsized":"2024-01-16T15:07:24.037264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:24.256412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.007910+0000","last_change":"2024-01-16T15:07:23.011710+0000","last_active":"2024-01-16T15:07:24.007910+0000","last_peered":"2024-01-16T15:07:24.007910+0000","last_clean":"2024-01-16T15:07:24.007910+0000","last_became_active":"2024-01-16T15:07:23.011444+0000","last_became_peered":"2024-01-16T15:07:23.011444+0000","last_unstale":"2024-01-16T15:07:24.007910+0000","last_undegraded":"2024-01-16T15:07:24.007910+0000","last_fullsized":"2024-01-16T15:07:24.007910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:05.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.034666+0000","last_change":"2024-01-16T15:07:24.033222+0000","last_active":"2024-01-16T15:07:24.034666+0000","last_peered":"2024-01-16T15:07:24.034666+0000","last_clean":"2024-01-16T15:07:24.034666+0000","last_became_active":"2024-01-16T15:07:24.032918+0000","last_became_peered":"2024-01-16T15:07:24.032918+0000","last_unstale":"2024-01-16T15:07:24.034666+0000","last_undegraded":"2024-01-16T15:07:24.034666+0000","last_fullsized":"2024-01-16T15:07:24.034666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:22.975799+0000","last_clean_scrub_stamp":"2024-01-16T15:07:22.975799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:50.964486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.049735+0000","last_change":"2024-01-16T15:07:23.016337+0000","last_active":"2024-01-16T15:07:24.049735+0000","last_peered":"2024-01-16T15:07:24.049735+0000","last_clean":"2024-01-16T15:07:24.049735+0000","last_became_active":"2024-01-16T15:07:23.016025+0000","last_became_peered":"2024-01-16T15:07:23.016025+0000","last_unstale":"2024-01-16T15:07:24.049735+0000","last_undegraded":"2024-01-16T15:07:24.049735+0000","last_fullsized":"2024-01-16T15:07:24.049735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:07:21.959524+0000","last_clean_scrub_stamp":"2024-01-16T15:07:21.959524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:25:36.334216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":206,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:07:24.019225+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:07:24.019225+0000","last_peered":"2024-01-16T15:07:24.019225+0000","last_clean":"2024-01-16T15:07:24.019225+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:07:24.019225+0000","last_undegraded":"2024-01-16T15:07:24.019225+0000","last_fullsized":"2024-01-16T15:07:24.019225+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542244,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50656,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1462272,"data_stored":953247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.468,"15min":1.2370000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.6260000000000003,"5min":7.8949999999999996,"15min":7.8949999999999996},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.841,"5min":1.5369999999999999,"15min":1.3149999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":6.5039999999999996,"5min":7.8460000000000001,"15min":7.8460000000000001},"last":1.0489999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.444,"15min":1.2789999999999999},"min":{"1min":0.92300000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":5.3010000000000002,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":2.4929999999999999,"5min":1.55,"15min":1.2929999999999999},"min":{"1min":0.83899999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":15.452,"5min":15.452,"15min":15.452},"last":1.1299999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5369999999999999,"15min":1.365},"min":{"1min":0.83099999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":5.9130000000000003,"5min":10.707000000000001,"15min":10.707000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.556,"15min":1.3779999999999999},"min":{"1min":0.95599999999999996,"5min":0.504,"15min":0.504},"max":{"1min":3.6379999999999999,"5min":10.661,"15min":10.661},"last":0.97099999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":1.752,"15min":1.4259999999999999},"min":{"1min":0.85199999999999998,"5min":0.61499999999999999,"15min":0.56999999999999995},"max":{"1min":10.654999999999999,"5min":10.654999999999999,"15min":10.654999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.651,"5min":1.716,"15min":1.514},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":5.8029999999999999,"5min":13.193,"15min":13.193},"last":1.3280000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.417,"15min":1.3899999999999999},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.752,"5min":16.376000000000001,"15min":16.376000000000001},"last":1.107},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.476,"15min":1.4790000000000001},"min":{"1min":0.56899999999999995,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.0680000000000001,"5min":5.9720000000000004,"15min":5.9720000000000004},"last":1.446}]},{"osd":5,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.514,"15min":1.544},"min":{"1min":0.623,"5min":0.497,"15min":0.497},"max":{"1min":5.1849999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":1.415},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.5509999999999999,"15min":1.5600000000000001},"min":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8380000000000001,"5min":7.9219999999999997,"15min":7.9219999999999997},"last":1.169}]},{"osd":6,"last update":"Tue Jan 16 15:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.5960000000000001,"15min":1.661},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.8879999999999999,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.242},{"interface":"front","average":{"1min":1.542,"5min":1.7090000000000001,"15min":1.6890000000000001},"min":{"1min":0.65200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.6739999999999999,"5min":8.4979999999999993,"15min":8.4979999999999993},"last":1.3859999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542243,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45916,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325924,"statfs":{"total":96636764160,"available":96589746176,"internally_reserved":0,"allocated":868352,"data_stored":362355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3769999999999998,"5min":2.2469999999999999,"15min":1.5209999999999999},"min":{"1min":0.60399999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":16.25,"5min":28.327999999999999,"15min":28.327999999999999},"last":1.214},{"interface":"front","average":{"1min":2.4369999999999998,"5min":2.2799999999999998,"15min":1.5249999999999999},"min":{"1min":0.629,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":22.43,"5min":30.280999999999999,"15min":30.280999999999999},"last":1.3109999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.855,"5min":1.9099999999999999,"15min":1.3500000000000001},"min":{"1min":0.78700000000000003,"5min":0.57699999999999996,"15min":0.56399999999999995},"max":{"1min":28.122,"5min":28.122,"15min":28.122},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.365,"15min":1.204},"min":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":5.4880000000000004,"5min":8.9719999999999995,"15min":8.9719999999999995},"last":1.01}]},{"osd":2,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":1.851,"15min":1.4279999999999999},"min":{"1min":0.70799999999999996,"5min":0.63200000000000001,"15min":0.59499999999999997},"max":{"1min":6.8129999999999997,"5min":17.991,"15min":17.991},"last":1.829},{"interface":"front","average":{"1min":2.177,"5min":1.9890000000000001,"15min":1.601},"min":{"1min":0.749,"5min":0.67100000000000004,"15min":0.65800000000000003},"max":{"1min":6.7720000000000002,"5min":18.02,"15min":18.02},"last":1.639}]},{"osd":3,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":3.1070000000000002,"5min":2.0379999999999998,"15min":1.5049999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.61599999999999999},"max":{"1min":17.75,"5min":17.75,"15min":17.75},"last":1.4810000000000001},{"interface":"front","average":{"1min":2.4359999999999999,"5min":1.8420000000000001,"15min":1.4630000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.56299999999999994},"max":{"1min":17.794,"5min":17.794,"15min":17.794},"last":1.5209999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.5760000000000001,"15min":1.4039999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":3.694,"5min":15.481999999999999,"15min":15.481999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":2.4329999999999998,"5min":1.73,"15min":1.534},"min":{"1min":0.625,"5min":0.58099999999999996,"15min":0.53900000000000003},"max":{"1min":16.303000000000001,"5min":16.303000000000001,"15min":16.303000000000001},"last":1.3680000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.385,"15min":1.327},"min":{"1min":0.58999999999999997,"5min":0.50900000000000001,"15min":0.45200000000000001},"max":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":4.7380000000000004},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.95,"5min":1.6850000000000001,"15min":1.601},"min":{"1min":0.78800000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":5.7009999999999996,"5min":9.0419999999999998,"15min":9.0419999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.516,"15min":1.472},"min":{"1min":0.69099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.6449999999999996,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":1.6850000000000001},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.738,"15min":1.6399999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":11.788,"5min":11.788,"15min":11.788},"last":1.915}]}]},{"osd":5,"up_from":14,"seq":60129542244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45912,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325928,"statfs":{"total":96636764160,"available":96589750272,"internally_reserved":0,"allocated":864256,"data_stored":362353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6299999999999999,"5min":2.02,"15min":1.4319999999999999},"min":{"1min":0.66600000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":26.228999999999999,"5min":26.228999999999999,"15min":26.228999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":3.4279999999999999,"5min":2.766,"15min":1.609},"min":{"1min":0.80200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.867000000000001,"5min":56.011000000000003,"15min":56.011000000000003},"last":0.76500000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.5509999999999999,"15min":1.258},"min":{"1min":0.61599999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":10.926,"5min":16.687999999999999,"15min":16.687999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.3130000000000002,"15min":1.571},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.8380000000000001,"5min":67.930000000000007,"15min":67.930000000000007},"last":1.0880000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.607,"15min":1.351},"min":{"1min":0.53400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4039999999999999,"5min":13.013,"15min":13.013},"last":1.032},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.546,"15min":1.3460000000000001},"min":{"1min":0.60599999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.0720000000000001,"5min":11.09,"15min":11.09},"last":1.069}]},{"osd":3,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.242,"5min":2.0390000000000001,"15min":1.528},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":12.109999999999999,"5min":16.193999999999999,"15min":16.193999999999999},"last":1.421},{"interface":"front","average":{"1min":2.3460000000000001,"5min":2.0249999999999999,"15min":1.611},"min":{"1min":0.75,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":12.135999999999999,"5min":13.356999999999999,"15min":13.356999999999999},"last":1.472}]},{"osd":4,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.444,"15min":1.3120000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.4430000000000001,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":1.004},{"interface":"front","average":{"1min":1.361,"5min":1.669,"15min":1.4390000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5910000000000002,"5min":16.268000000000001,"15min":16.268000000000001},"last":0.94499999999999995}]},{"osd":6,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.583,"15min":1.4550000000000001},"min":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.4950000000000001,"5min":13.263999999999999,"15min":13.263999999999999},"last":1.258},{"interface":"front","average":{"1min":1.53,"5min":1.5509999999999999,"15min":1.522},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":3.52,"5min":4.9299999999999997,"15min":7.335},"last":1.4370000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.865,"15min":1.6359999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.2290000000000001,"5min":14.345000000000001,"15min":14.345000000000001},"last":1.333},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.9450000000000001,"15min":1.698},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.577,"5min":13.491,"15min":13.491},"last":1.3839999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50092,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321748,"statfs":{"total":96636764160,"available":96585469952,"internally_reserved":0,"allocated":876544,"data_stored":368661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.4359999999999999,"15min":1.2050000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":9.5589999999999993,"5min":12.587999999999999,"15min":12.587999999999999},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":2.1230000000000002,"15min":1.3919999999999999},"min":{"1min":0.71099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":4.3380000000000001,"5min":60.106000000000002,"15min":60.106000000000002},"last":1.2150000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.901,"15min":1.4930000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.7919999999999998,"5min":14.19,"15min":14.19},"last":1.466},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.673,"15min":1.369},"min":{"1min":0.68300000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":6.8170000000000002,"5min":9.5589999999999993,"15min":9.5589999999999993},"last":1.9930000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7689999999999999,"5min":1.357,"15min":1.232},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":9.7140000000000004,"5min":9.7140000000000004,"15min":9.7140000000000004},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":2.1789999999999998,"15min":1.617},"min":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":6.516,"5min":35.024999999999999,"15min":35.024999999999999},"last":1.3540000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":3.0819999999999999,"5min":1.829,"15min":1.5429999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":32.133000000000003,"5min":32.133000000000003,"15min":32.133000000000003},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.3819999999999999,"15min":1.3400000000000001},"min":{"1min":0.71899999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":4.2880000000000003,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":1.413}]},{"osd":5,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.5980000000000001,"15min":1.532},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.0289999999999999,"5min":14.568,"15min":14.568},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.4830000000000001,"15min":1.55},"min":{"1min":0.63,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":7.1479999999999997,"5min":7.1479999999999997,"15min":7.1479999999999997},"last":2.0569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.413,"15min":1.444},"min":{"1min":0.52400000000000002,"5min":0.437,"15min":0.437},"max":{"1min":2.8029999999999999,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":1.516},{"interface":"front","average":{"1min":1.25,"5min":1.6830000000000001,"15min":1.6259999999999999},"min":{"1min":0.498,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.1459999999999999,"5min":14.317,"15min":14.317},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:06:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.5509999999999999,"15min":1.655},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":7.0339999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.5209999999999999,"15min":1.595},"min":{"1min":0.66000000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.9390000000000001,"5min":5.4610000000000003,"15min":5.4610000000000003},"last":1.952}]}]},{"osd":3,"up_from":14,"seq":60129542245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54240,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317600,"statfs":{"total":96636764160,"available":96581222400,"internally_reserved":0,"allocated":868352,"data_stored":362375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":4.5570000000000004,"5min":2.0910000000000002,"15min":1.351},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":50.988999999999997,"5min":50.988999999999997,"15min":50.988999999999997},"last":0.69599999999999995},{"interface":"front","average":{"1min":2.073,"5min":1.409,"15min":1.1539999999999999},"min":{"1min":0.45000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.4309999999999992,"5min":8.4309999999999992,"15min":8.4309999999999992},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.3839999999999999,"15min":1.1259999999999999},"min":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":11.634,"5min":11.634,"15min":11.634},"last":0.93300000000000005},{"interface":"front","average":{"1min":3.125,"5min":1.7949999999999999,"15min":1.421},"min":{"1min":0.52600000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":20.385999999999999,"5min":20.385999999999999,"15min":20.385999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.587,"15min":1.29},"min":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.42699999999999999},"max":{"1min":11.475,"5min":11.475,"15min":11.475},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.4249999999999998,"5min":1.835,"15min":1.482},"min":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":24.553999999999998,"5min":24.553999999999998,"15min":24.553999999999998},"last":0.88600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3250000000000002,"5min":1.7130000000000001,"15min":1.458},"min":{"1min":0.62,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":9.5600000000000005,"5min":9.5600000000000005,"15min":9.5600000000000005},"last":1.0720000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":1.6830000000000001,"15min":1.5089999999999999},"min":{"1min":0.73999999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":11.528,"5min":11.528,"15min":11.528},"last":1.2150000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6110000000000002,"5min":2.0569999999999999,"15min":1.698},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":24.349,"5min":24.349,"15min":24.349},"last":1.145},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.9119999999999999,"15min":1.569},"min":{"1min":0.92600000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":8.6489999999999991,"5min":12.616,"15min":12.616},"last":1.3100000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.746,"15min":1.5660000000000001},"min":{"1min":0.98999999999999999,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":11.193,"5min":11.193,"15min":11.193},"last":1.02},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.1200000000000001,"15min":1.774},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":24.058,"5min":24.058,"15min":24.058},"last":1.284}]},{"osd":7,"last update":"Tue Jan 16 15:06:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5499999999999998,"5min":1.859,"15min":1.643},"min":{"1min":0.67800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.5120000000000005,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.4019999999999999},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.8480000000000001,"15min":1.645},"min":{"1min":1.0389999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":11.082000000000001,"5min":11.082000000000001,"15min":11.082000000000001},"last":1.169}]}]},{"osd":2,"up_from":14,"seq":60129542243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":872448,"data_stored":363875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":2.1989999999999998,"15min":1.349},"min":{"1min":0.41899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5269999999999999,"5min":64.387,"15min":64.387},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.5660000000000001,"15min":1.175},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":10.151,"5min":11.339,"15min":11.339},"last":0.92700000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.9059999999999999,"15min":1.2410000000000001},"min":{"1min":0.39500000000000002,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":2.5699999999999998,"5min":27.263000000000002,"15min":27.263000000000002},"last":1.2410000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.1669999999999998,"15min":1.5369999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":16.163,"5min":23.547999999999998,"15min":23.547999999999998},"last":1.4059999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":1.9059999999999999,"15min":1.409},"min":{"1min":0.60999999999999999,"5min":0.40100000000000002,"15min":0.33100000000000002},"max":{"1min":8,"5min":27.311,"15min":27.311},"last":1.542},{"interface":"front","average":{"1min":2.3170000000000002,"5min":2.0430000000000001,"15min":1.444},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":13.84,"5min":26.555,"15min":26.555},"last":1.139}]},{"osd":4,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.869,"15min":1.4910000000000001},"min":{"1min":0.88400000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.3010000000000002,"5min":26.503,"15min":26.503},"last":1.302},{"interface":"front","average":{"1min":1.863,"5min":1.726,"15min":1.458},"min":{"1min":0.78400000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":10.542999999999999,"5min":11.188000000000001,"15min":11.188000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.97,"15min":1.571},"min":{"1min":1.022,"5min":0.80800000000000005,"15min":0.71899999999999997},"max":{"1min":2.7490000000000001,"5min":27.149000000000001,"15min":27.149000000000001},"last":1.474},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.966,"15min":1.655},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.895,"5min":26.349,"15min":26.349},"last":1.4279999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.8200000000000001,"15min":1.544},"min":{"1min":0.873,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":10.433,"5min":11.266,"15min":11.266},"last":1.198},{"interface":"front","average":{"1min":1.5960000000000001,"5min":2.2120000000000002,"15min":1.7130000000000001},"min":{"1min":1.141,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":3.1800000000000002,"5min":27.076000000000001,"15min":27.076000000000001},"last":1.3720000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1869999999999998,"5min":1.9930000000000001,"15min":1.7210000000000001},"min":{"1min":0.82999999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":10.617000000000001,"5min":11.303000000000001,"15min":11.303000000000001},"last":1.351},{"interface":"front","average":{"1min":1.663,"5min":2.137,"15min":1.6839999999999999},"min":{"1min":1.1399999999999999,"5min":0.754,"15min":0.66600000000000004},"max":{"1min":3.2349999999999999,"5min":27.207999999999998,"15min":27.207999999999998},"last":1.607}]}]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54256,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317584,"statfs":{"total":96636764160,"available":96581206016,"internally_reserved":0,"allocated":880640,"data_stored":364491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.603,"15min":1.2050000000000001},"min":{"1min":0.41299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":11.516999999999999,"5min":14.407,"15min":14.407},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.198,"5min":1.4279999999999999,"15min":1.2070000000000001},"min":{"1min":0.68500000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":8.9719999999999995,"5min":8.9719999999999995,"15min":8.9719999999999995},"last":0.69699999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":1.639,"15min":1.2549999999999999},"min":{"1min":0.57399999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":16.488,"5min":16.488,"15min":16.488},"last":1.0449999999999999},{"interface":"front","average":{"1min":2.524,"5min":2.0539999999999998,"15min":1.4850000000000001},"min":{"1min":0.59699999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":11.215,"5min":22.550999999999998,"15min":22.550999999999998},"last":0.92200000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.958,"15min":1.5189999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.117,"5min":14.653,"15min":14.653},"last":1.514},{"interface":"front","average":{"1min":2.9039999999999999,"5min":1.8879999999999999,"15min":1.4870000000000001},"min":{"1min":0.65600000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":12.220000000000001,"5min":12.220000000000001,"15min":12.220000000000001},"last":1.635}]},{"osd":4,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7730000000000001,"5min":1.919,"15min":1.575},"min":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.81000000000000005},"max":{"1min":8.3620000000000001,"5min":12.675000000000001,"15min":12.675000000000001},"last":1.381},{"interface":"front","average":{"1min":3.3839999999999999,"5min":2.0339999999999998,"15min":1.6419999999999999},"min":{"1min":0.99199999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":19.236999999999998,"5min":19.236999999999998,"15min":19.236999999999998},"last":1.3129999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.609,"5min":1.8149999999999999,"15min":1.629},"min":{"1min":1.0329999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":8.2859999999999996},"last":1.53},{"interface":"front","average":{"1min":3.6019999999999999,"5min":1.9630000000000001,"15min":1.6299999999999999},"min":{"1min":1.01,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":17.138000000000002,"5min":17.138000000000002,"15min":17.138000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4700000000000002,"5min":2.161,"15min":1.7809999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":11.404,"5min":14.781000000000001,"15min":14.781000000000001},"last":1.4379999999999999},{"interface":"front","average":{"1min":3.0169999999999999,"5min":2.0110000000000001,"15min":1.764},"min":{"1min":1.165,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":13.243,"5min":13.243,"15min":13.243},"last":1.2110000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:06:29 2024","interfaces":[{"interface":"back","average":{"1min":3.4470000000000001,"5min":2.0009999999999999,"15min":1.7090000000000001},"min":{"1min":1.135,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":17.010000000000002,"5min":17.010000000000002,"15min":17.010000000000002},"last":1.575},{"interface":"front","average":{"1min":2.5680000000000001,"5min":2.2349999999999999,"15min":1.8540000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":11.457000000000001,"5min":14.884,"15min":14.884},"last":1.2729999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542244,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54812,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317028,"statfs":{"total":96636764160,"available":96580636672,"internally_reserved":0,"allocated":1454080,"data_stored":952631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":54655425},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.6320000000000001,"5min":1.762,"15min":1.1859999999999999},"min":{"1min":0.58499999999999996,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":19.058,"5min":19.058,"15min":19.058},"last":0.377},{"interface":"front","average":{"1min":2.496,"5min":2.0630000000000002,"15min":1.296},"min":{"1min":0.70399999999999996,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.846,"5min":24.414000000000001,"15min":24.414000000000001},"last":0.76800000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.8159999999999998,"5min":1.7230000000000001,"15min":1.248},"min":{"1min":0.40799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":18.861000000000001,"5min":18.861000000000001,"15min":18.861000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.984,"5min":1.944,"15min":1.3069999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":13.936,"5min":38.893000000000001,"15min":38.893000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.3700000000000001,"5min":2.1629999999999998,"15min":1.337},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":10.090999999999999,"5min":23.015999999999998,"15min":23.015999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.948,"5min":1.9259999999999999,"15min":1.413},"min":{"1min":0.69799999999999995,"5min":0.47799999999999998,"15min":0.46999999999999997},"max":{"1min":18.966999999999999,"5min":18.966999999999999,"15min":18.966999999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.7050000000000001,"15min":1.343},"min":{"1min":0.67900000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":6.0199999999999996,"5min":20.327000000000002,"15min":20.327000000000002},"last":1.145},{"interface":"front","average":{"1min":3.1280000000000001,"5min":2.008,"15min":1.5600000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.71399999999999997},"max":{"1min":19.004999999999999,"5min":19.004999999999999,"15min":19.004999999999999},"last":1.1930000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":2.3460000000000001,"15min":1.633},"min":{"1min":0.80900000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":14.630000000000001,"5min":24.611000000000001,"15min":24.611000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.95,"5min":1.9830000000000001,"15min":1.5109999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":6.0119999999999996,"5min":20.199000000000002,"15min":20.199000000000002},"last":1.2509999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":4.8700000000000001,"5min":2.448,"15min":1.802},"min":{"1min":0.79000000000000004,"5min":0.754,"15min":0.754},"max":{"1min":26.998000000000001,"5min":26.998000000000001,"15min":26.998000000000001},"last":0.62},{"interface":"front","average":{"1min":2.6960000000000002,"5min":2.3900000000000001,"15min":1.712},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":14.326000000000001,"5min":24.478999999999999,"15min":24.478999999999999},"last":1.002}]},{"osd":7,"last update":"Tue Jan 16 15:06:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.214,"15min":1.73},"min":{"1min":0.76100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":14.275,"5min":17.957000000000001,"15min":17.957000000000001},"last":1.4319999999999999},{"interface":"front","average":{"1min":2.1859999999999999,"5min":1.996,"15min":1.609},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":6.3360000000000003,"5min":20.279,"15min":20.279},"last":1.3480000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:07:29.022 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:07:29.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:07:29.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.316+0000 7f24b9e00640 1 Processor -- start 2024-01-16T15:07:29.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.316+0000 7f24b9e00640 1 -- start start 2024-01-16T15:07:29.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f24b40cfa30 0x7f24b40cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b9e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b9e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b40b2980 con 0x7f24b40d1fd0 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b9e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b40cf030 con 0x7f24b40cfa30 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b9e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40d0350 con 0x7f24b40b33d0 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f24b40cfa30 0x7f24b40cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f24b40cfa30 0x7f24b40cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55162/0 (socket says 172.21.15.43:55162) 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.318+0000 7f24b8a00640 1 -- 172.21.15.43:0/946287924 learned_addr learned my addr 172.21.15.43:0/946287924 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.319+0000 7f24b9400640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.319+0000 7f24b8a00640 1 -- 172.21.15.43:0/946287924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 msgr2=0x7f24b40b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.319+0000 7f24b8a00640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.319+0000 7f24b8a00640 1 -- 172.21.15.43:0/946287924 >> v1:172.21.15.69:6789/0 conn(0x7f24b40d1fd0 legacy=0x7f24b40ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.319+0000 7f24b8a00640 1 -- 172.21.15.43:0/946287924 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40cf220 con 0x7f24b40cfa30 2024-01-16T15:07:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.320+0000 7f24b8a00640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b40cfe10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009870 tx=0x7f24a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f5ad14b89729f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.321+0000 7f24b3400640 1 -- 172.21.15.43:0/946287924 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4030840 con 0x7f24b40cfa30 2024-01-16T15:07:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.321+0000 7f24b3400640 1 -- 172.21.15.43:0/946287924 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a40309a0 con 0x7f24b40cfa30 2024-01-16T15:07:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.321+0000 7f24b3400640 1 -- 172.21.15.43:0/946287924 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4030d30 con 0x7f24b40cfa30 2024-01-16T15:07:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.321+0000 7f24b9e00640 1 -- 172.21.15.43:0/946287924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 msgr2=0x7f24b40cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.321+0000 7f24b9e00640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b40cfe10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009870 tx=0x7f24a402b980 comp rx=0 tx=0).stop 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 -- 172.21.15.43:0/946287924 shutdown_connections 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 --2- 172.21.15.43:0/946287924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b40cfe10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 -- 172.21.15.43:0/946287924 >> 172.21.15.43:0/946287924 conn(0x7f24b40d3020 msgr2=0x7f24b40d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 -- 172.21.15.43:0/946287924 shutdown_connections 2024-01-16T15:07:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.322+0000 7f24b9e00640 1 -- 172.21.15.43:0/946287924 wait complete. 2024-01-16T15:07:29.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.323+0000 7f24b9e00640 1 Processor -- start 2024-01-16T15:07:29.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.324+0000 7f24b9e00640 1 -- start start 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.324+0000 7f24b9e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b40d1fd0 0x7f24b415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4151de0 con 0x7f24b40cfa30 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40ca0f0 con 0x7f24b40d1fd0 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b415fbc0 con 0x7f24b40b33d0 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55174/0 (socket says 172.21.15.43:55174) 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b3e00640 1 -- 172.21.15.43:0/949768 learned_addr learned my addr 172.21.15.43:0/949768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b9400640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b40d1fd0 0x7f24b415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.325+0000 7f24b8a00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 -- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 msgr2=0x7f24b40b8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 -- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b40d1fd0 msgr2=0x7f24b415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b40d1fd0 0x7f24b415f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 -- 172.21.15.43:0/949768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40094d0 con 0x7f24b40cfa30 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b3e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f249c00c3c0 tx=0x7f249c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c0052d0 con 0x7f24b40cfa30 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c005450 con 0x7f24b40cfa30 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c007260 con 0x7f24b40cfa30 2024-01-16T15:07:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b415fea0 con 0x7f24b40cfa30 2024-01-16T15:07:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.326+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b40ca620 con 0x7f24b40cfa30 2024-01-16T15:07:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.327+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b40d1bc0 con 0x7f24b40cfa30 2024-01-16T15:07:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.328+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f249c0073c0 con 0x7f24b40cfa30 2024-01-16T15:07:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.329+0000 7f24b0c00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 0x7f248c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.330+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f249c0877a0 con 0x7f24b40cfa30 2024-01-16T15:07:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.330+0000 7f24b8a00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 0x7f248c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.331+0000 7f24b8a00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 0x7f248c06e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f24a4036000 tx=0x7f24a4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.334+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f249c054110 con 0x7f24b40cfa30 2024-01-16T15:07:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.596+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f24b404e840 con 0x7f24b40cfa30 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.600+0000 7f24b0c00640 1 -- 172.21.15.43:0/949768 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f249c057730 con 0x7f24b40cfa30 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:07:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:07:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.602+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 msgr2=0x7f248c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.602+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 0x7f248c06e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f24a4036000 tx=0x7f24a4008030 comp rx=0 tx=0).stop 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.602+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 msgr2=0x7f24b415ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.602+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f249c00c3c0 tx=0x7f249c00c880 comp rx=0 tx=0).stop 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 shutdown_connections 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f248c06bbc0 0x7f248c06e080 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b40d1fd0 0x7f24b415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24b40cfa30 0x7f24b415ec20 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 --2- 172.21.15.43:0/949768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b40b33d0 0x7f24b40b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 >> 172.21.15.43:0/949768 conn(0x7f24b40d3020 msgr2=0x7f24b40b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:29.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 shutdown_connections 2024-01-16T15:07:29.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.603+0000 7f24b9e00640 1 -- 172.21.15.43:0/949768 wait complete. 2024-01-16T15:07:29.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:29.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.834+0000 7facf9200640 1 Processor -- start 2024-01-16T15:07:29.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.835+0000 7facf9200640 1 -- start start 2024-01-16T15:07:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.836+0000 7facf9200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facf4160970 0x7facf40b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.836+0000 7facf9200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.836+0000 7facf9200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf4071cd0 con 0x7facf4070780 2024-01-16T15:07:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.836+0000 7facf9200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf40b2120 con 0x7facf4160970 2024-01-16T15:07:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.836+0000 7facf9200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40b2400 con 0x7facf40b3140 2024-01-16T15:07:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facf4160970 0x7facf40b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf2800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7facf4070780 0x7facf40acbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44928/0 (socket says 172.21.15.43:44928) 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf2800640 1 -- 172.21.15.43:0/329424370 learned_addr learned my addr 172.21.15.43:0/329424370 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1400640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf0a00640 1 -- 172.21.15.43:0/329424370 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221286898 0 0) 0x7facf4071cd0 con 0x7facf4070780 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1e00640 1 -- 172.21.15.43:0/329424370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 msgr2=0x7facf406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1e00640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1e00640 1 -- 172.21.15.43:0/329424370 >> v1:172.21.15.43:6789/0 conn(0x7facf4070780 legacy=0x7facf40acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.837+0000 7facf1e00640 1 -- 172.21.15.43:0/329424370 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40b26e0 con 0x7facf4160970 2024-01-16T15:07:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.839+0000 7facf1e00640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf40b2c00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7face40097a0 tx=0x7face402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5c5af4753ecd83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.839+0000 7facf0a00640 1 -- 172.21.15.43:0/329424370 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face402e070 con 0x7facf4160970 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.839+0000 7facf0a00640 1 -- 172.21.15.43:0/329424370 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face40315a0 con 0x7facf4160970 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.839+0000 7facf0a00640 1 -- 172.21.15.43:0/329424370 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face4031930 con 0x7facf4160970 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 -- 172.21.15.43:0/329424370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 msgr2=0x7facf40b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf40b2c00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7face40097a0 tx=0x7face402b840 comp rx=0 tx=0).stop 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 -- 172.21.15.43:0/329424370 shutdown_connections 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 --2- 172.21.15.43:0/329424370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf40b2c00 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 -- 172.21.15.43:0/329424370 >> 172.21.15.43:0/329424370 conn(0x7facf40b0d80 msgr2=0x7facf40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 -- 172.21.15.43:0/329424370 shutdown_connections 2024-01-16T15:07:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.840+0000 7facf9200640 1 -- 172.21.15.43:0/329424370 wait complete. 2024-01-16T15:07:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.842+0000 7facf9200640 1 Processor -- start 2024-01-16T15:07:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.842+0000 7facf9200640 1 -- start start 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 0x7facf414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf4157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf409af60 con 0x7facf4160970 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf409b1b0 con 0x7facf4070780 2024-01-16T15:07:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.843+0000 7facf9200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf409b490 con 0x7facf40b3140 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55194/0 (socket says 172.21.15.43:55194) 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 0x7facf414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 -- 172.21.15.43:0/1549485044 learned_addr learned my addr 172.21.15.43:0/1549485044 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf1400640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf4157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 -- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 msgr2=0x7facf4157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf4157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 -- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 msgr2=0x7facf414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 0x7facf414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 -- 172.21.15.43:0/1549485044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4009400 con 0x7facf4160970 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf1400640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf4157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf1e00640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 0x7facf414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facf2800640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7facec00c430 tx=0x7facec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.844+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec0074e0 con 0x7facf4160970 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.845+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec007660 con 0x7facf4160970 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.845+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf409b770 con 0x7facf4160970 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.845+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec004a20 con 0x7facf4160970 2024-01-16T15:07:29.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.845+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf409bd30 con 0x7facf4160970 2024-01-16T15:07:29.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.846+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf409bf50 con 0x7facf4160970 2024-01-16T15:07:29.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.847+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7facec004b80 con 0x7facf4160970 2024-01-16T15:07:29.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.848+0000 7facea000640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 0x7faccc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:29.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.848+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7facec0882a0 con 0x7facf4160970 2024-01-16T15:07:29.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.848+0000 7facf1e00640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 0x7faccc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:29.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.849+0000 7facf1e00640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 0x7faccc06e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7face4036000 tx=0x7face403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:29.852+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facec054c80 con 0x7facf4160970 2024-01-16T15:07:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.095+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facf4074770 con 0x7facf4160970 2024-01-16T15:07:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.096+0000 7facea000640 1 -- 172.21.15.43:0/1549485044 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4915 (secure 0 0 0) 0x7facec0582a0 con 0x7facf4160970 2024-01-16T15:07:30.075 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:07:30.075 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5540,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1054183394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1054183394},{"type":"v1","addr":"172.21.15.69:6835","nonce":1054183394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/3255994502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3255994502},{"type":"v1","addr":"172.21.15.43:6836","nonce":3255994502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5546,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1771464625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":1771464625},{"type":"v1","addr":"172.21.15.43:6837","nonce":1771464625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:07:23.471150+0000","modified":"2024-01-16T15:07:26.545904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6085":{"gid":6085,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6834/1602463840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1602463840},{"type":"v1","addr":"172.21.15.69:6834","nonce":1602463840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:07:30.075 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 46 2024-01-16T15:07:30.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 msgr2=0x7faccc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 0x7faccc06e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7face4036000 tx=0x7face403d040 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 msgr2=0x7facf41f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7facec00c430 tx=0x7facec00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 shutdown_connections 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faccc06bbc0 0x7faccc06e080 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facf4160970 0x7facf41f65b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facf40b3140 0x7facf4157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 --2- 172.21.15.43:0/1549485044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facf4070780 0x7facf414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.099+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 >> 172.21.15.43:0/1549485044 conn(0x7facf40b0d80 msgr2=0x7facf41555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.100+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 shutdown_connections 2024-01-16T15:07:30.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.100+0000 7facf9200640 1 -- 172.21.15.43:0/1549485044 wait complete. 2024-01-16T15:07:30.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:07:23.471150+0000', 'modified': '2024-01-16T15:07:26.545904+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6085': {'gid': 6085, 'name': 'd', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6834/1602463840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 1602463840}, {'type': 'v1', 'addr': '172.21.15.69:6834', 'nonce': 1602463840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:07:30.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:07:30.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:07:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.753+0000 7fc3aea00640 1 Processor -- start 2024-01-16T15:07:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.754+0000 7fc3aea00640 1 -- start start 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.755+0000 7fc3aea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc3a806af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3aea00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80b2510 con 0x7fc3a8159830 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3aea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a806ccb0 con 0x7fc3a8159450 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3aea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a806a440 con 0x7fc3a8071680 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3ae000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc3a8159830 0x7fc3a806d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44944/0 (socket says 172.21.15.43:44944) 2024-01-16T15:07:30.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3ae000640 1 -- 172.21.15.43:0/1702699956 learned_addr learned my addr 172.21.15.43:0/1702699956 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3acc00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc3a806af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857783950 0 0) 0x7fc3a80b2510 con 0x7fc3a8159830 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.756+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388003520 con 0x7fc3a8159830 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357331205 0 0) 0x7fc388003520 con 0x7fc3a8159830 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 msgr2=0x7fc3a806af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc3a806af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 >> v1:172.21.15.69:6789/0 conn(0x7fc3a8159450 legacy=0x7fc3a8070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80b2760 con 0x7fc3a8159830 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc3a00019f0 con 0x7fc3a8159830 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 >> v1:172.21.15.43:6789/0 conn(0x7fc3a8159830 legacy=0x7fc3a806d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc388003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3880047c0 0x7fc388004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a806a440 con 0x7fc388003a10 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800ba80 con 0x7fc3880047c0 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800bc40 con 0x7fc3a8071680 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.757+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc388003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3acc00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3880047c0 0x7fc388004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ae000640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ad600640 1 -- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 msgr2=0x7fc388005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc388005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ad600640 1 -- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3880047c0 msgr2=0x7fc388004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3880047c0 0x7fc388004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ad600640 1 -- 172.21.15.43:0/1702699956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388003520 con 0x7fc388003a10 2024-01-16T15:07:30.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.758+0000 7fc3ae000640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8071680 0x7fc388005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.759+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc388003e30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc398001750 tx=0x7fc3880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2632315abb9c59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980052d0 con 0x7fc388003a10 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39802e050 con 0x7fc388003a10 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3a7e00640 1 -- 172.21.15.43:0/1702699956 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980035f0 con 0x7fc388003a10 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 -- 172.21.15.43:0/1702699956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 msgr2=0x7fc388003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 --2- 172.21.15.43:0/1702699956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc388003e30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc398001750 tx=0x7fc3880057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3ae000640 1 -- 172.21.15.43:0/1702699956 reap_dead start 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 -- 172.21.15.43:0/1702699956 shutdown_connections 2024-01-16T15:07:30.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 -- 172.21.15.43:0/1702699956 >> 172.21.15.43:0/1702699956 conn(0x7fc3a80b1630 msgr2=0x7fc3a80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 -- 172.21.15.43:0/1702699956 shutdown_connections 2024-01-16T15:07:30.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.760+0000 7fc3aea00640 1 -- 172.21.15.43:0/1702699956 wait complete. 2024-01-16T15:07:30.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.762+0000 7fc3aea00640 1 Processor -- start 2024-01-16T15:07:30.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.762+0000 7fc3aea00640 1 -- start start 2024-01-16T15:07:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 0x7fc3a8149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 0x7fc3a8149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81428c0 con 0x7fc388003a10 2024-01-16T15:07:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8142290 con 0x7fc3a813de70 2024-01-16T15:07:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.763+0000 7fc3aea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a813b6b0 con 0x7fc3a8149750 2024-01-16T15:07:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55226/0 (socket says 172.21.15.43:55226) 2024-01-16T15:07:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 -- 172.21.15.43:0/3265826708 learned_addr learned my addr 172.21.15.43:0/3265826708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 -- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 msgr2=0x7fc3a8149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ae000640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 0x7fc3a8149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 0x7fc3a8149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 0x7fc3a8149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 -- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 msgr2=0x7fc3a8149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 0x7fc3a8149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ad600640 1 -- 172.21.15.43:0/3265826708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80ce590 con 0x7fc388003a10 2024-01-16T15:07:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 0x7fc3a8149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.764+0000 7fc3ae000640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 0x7fc3a8149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc398005040 tx=0x7fc398001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3a5600640 1 -- 172.21.15.43:0/3265826708 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980076c0 con 0x7fc388003a10 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3a5600640 1 -- 172.21.15.43:0/3265826708 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398007a30 con 0x7fc388003a10 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3a5600640 1 -- 172.21.15.43:0/3265826708 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398006c20 con 0x7fc388003a10 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a80ce870 con 0x7fc388003a10 2024-01-16T15:07:30.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.765+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a813b890 con 0x7fc388003a10 2024-01-16T15:07:30.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.767+0000 7fc3a5600640 1 -- 172.21.15.43:0/3265826708 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc3980035f0 con 0x7fc388003a10 2024-01-16T15:07:30.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.768+0000 7fc3a5600640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 0x7fc38006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.768+0000 7fc3a5600640 1 -- 172.21.15.43:0/3265826708 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc3980aa110 con 0x7fc388003a10 2024-01-16T15:07:30.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.769+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 0x7fc38006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.769+0000 7fc3b2d64740 1 Processor -- start 2024-01-16T15:07:30.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.769+0000 7fc3b2d64740 1 -- start start 2024-01-16T15:07:30.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.769+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 0x7fc38006e0b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc3a81f66f0 tx=0x7fc390008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b24270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b247b0 0x5642c6b68730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a83880 con 0x5642c6b25070 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a92ae0 con 0x5642c6b247b0 2024-01-16T15:07:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a901d0 con 0x5642c6b1fd70 2024-01-16T15:07:30.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55238/0 (socket says 172.21.15.43:55238) 2024-01-16T15:07:30.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.770+0000 7fc3ad600640 1 -- 172.21.15.43:0/4081031532 learned_addr learned my addr 172.21.15.43:0/4081031532 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:30.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3acc00640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b24270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 -- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 msgr2=0x5642c6b24270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b24270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 -- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b247b0 msgr2=0x5642c6b68730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ae000640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b247b0 0x5642c6b68730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b247b0 0x5642c6b68730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 -- 172.21.15.43:0/4081031532 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642c6a53860 con 0x5642c6b25070 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3acc00640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b24270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc3ad600640 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc39807a740 tx=0x7fc39807a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ccde4256f6be63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc39e000640 1 -- 172.21.15.43:0/4081031532 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39807e900 con 0x5642c6b25070 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc39e000640 1 -- 172.21.15.43:0/4081031532 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398082ab0 con 0x5642c6b25070 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.771+0000 7fc39e000640 1 -- 172.21.15.43:0/4081031532 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398001370 con 0x5642c6b25070 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 -- 172.21.15.43:0/4081031532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 msgr2=0x5642c6b25450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc39807a740 tx=0x7fc39807a1d0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 -- 172.21.15.43:0/4081031532 shutdown_connections 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b247b0 0x5642c6b68730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b24270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/4081031532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b25070 0x5642c6b25450 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 -- 172.21.15.43:0/4081031532 >> 172.21.15.43:0/4081031532 conn(0x5642c6ae6c70 msgr2=0x5642c6b6ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 -- 172.21.15.43:0/4081031532 shutdown_connections 2024-01-16T15:07:30.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.772+0000 7fc3b2d64740 1 -- 172.21.15.43:0/4081031532 wait complete. 2024-01-16T15:07:30.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.773+0000 7fc3b2d64740 1 -- start start 2024-01-16T15:07:30.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.773+0000 7fc3b2d64740 1 Processor -- start 2024-01-16T15:07:30.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b6cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b44fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a8bac0 con 0x5642c6b247b0 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a282f0 con 0x5642c6b25070 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a45c70 con 0x5642c6b1fd70 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.774+0000 7fc3b2d64740 10 client.0 _ll_register_callbacks cb 0x5642c6a4a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55248/0 (socket says 172.21.15.43:55248) 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 -- 172.21.15.43:0/1245771325 learned_addr learned my addr 172.21.15.43:0/1245771325 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 -- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 msgr2=0x5642c6b6cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b6cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 -- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 msgr2=0x5642c6b44fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b44fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 -- 172.21.15.43:0/1245771325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a813b890 con 0x5642c6b247b0 2024-01-16T15:07:30.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc3acc00640 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc390012e80 tx=0x7fc390004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b0db878d778053 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.775+0000 7fc396a00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:07:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc390015660 con 0x5642c6b247b0 2024-01-16T15:07:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642c6a84aa0 con 0x5642c6b247b0 2024-01-16T15:07:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc3900157c0 con 0x5642c6b247b0 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc390015980 con 0x5642c6b247b0 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 10 client.7158 fetch_fsmap learned FSMap version 46 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x5642c6a8a140 con 0x5642c6b247b0 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5642c6a84aa0 con 0x5642c6b247b0 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc390015ae0 con 0x5642c6b247b0 2024-01-16T15:07:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 10 client.7158 fetch_fsmap finished waiting for FSMap version 46 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 10 client.7158 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 20 client.7158 populate_metadata read hostname 'smithi043' 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b43cc0 0x5642c6b87c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 4 client.7158 mds_command: new command op to 6085 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.776+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 --> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642c6a847c0 con 0x5642c6b43cc0 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.777+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b43cc0 0x5642c6b87c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.778+0000 7fc3ad600640 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b43cc0 0x5642c6b87c00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d3e182d1a8519b server_cookie=95a411d73690d3d5 in_seq=0 out_seq=0 2024-01-16T15:07:30.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.778+0000 7fc396a00640 10 client.7158 ms_handle_connect on v2:172.21.15.69:6832/1602463840 2024-01-16T15:07:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc396a00640 1 -- 172.21.15.43:0/1245771325 <== mds.0 v2:172.21.15.69:6832/1602463840 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5642c6a847c0 con 0x5642c6b43cc0 2024-01-16T15:07:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc396a00640 10 client.7158 handle_command_reply: tid=0 2024-01-16T15:07:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 client.7158 shutdown 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 msgr2=0x5642c6b44aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc390012e80 tx=0x7fc390004730 comp rx=0 tx=0).stop 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 shutdown_connections 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b43cc0 0x5642c6b87c00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b44fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b247b0 0x5642c6b44aa0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/1245771325 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1fd70 0x5642c6b6cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 >> 172.21.15.43:0/1245771325 conn(0x5642c6ae6c70 msgr2=0x5642c6b1edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.780+0000 7fc396a00640 0 client.7158 ms_handle_reset on v2:172.21.15.69:6832/1602463840 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.781+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 shutdown_connections 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.781+0000 7fc3b2d64740 1 -- 172.21.15.43:0/1245771325 wait complete. 2024-01-16T15:07:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.781+0000 7fc3b2d64740 20 client.7158 trim_cache size 0 max 16384 2024-01-16T15:07:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.798+0000 7fc3b2d64740 1 Processor -- start 2024-01-16T15:07:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.798+0000 7fc3b2d64740 1 -- start start 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b415f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1f500 0x5642c6b41b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c67eef20 con 0x5642c6b1fd70 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6a0f520 con 0x5642c6b25070 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642c6b45230 con 0x5642c6b1f500 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55264/0 (socket says 172.21.15.43:55264) 2024-01-16T15:07:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3ad600640 1 -- 172.21.15.43:0/3421734247 learned_addr learned my addr 172.21.15.43:0/3421734247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.799+0000 7fc3ad600640 1 -- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1f500 msgr2=0x5642c6b41b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1f500 0x5642c6b41b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc3ad600640 1 -- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 msgr2=0x5642c6b415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc3ad600640 1 -- 172.21.15.43:0/3421734247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642c6a27b80 con 0x5642c6b1fd70 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc398092240 tx=0x7fc398048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32882583468dd9a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc39ea00640 1 -- 172.21.15.43:0/3421734247 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398078b50 con 0x5642c6b1fd70 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc39ea00640 1 -- 172.21.15.43:0/3421734247 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398001370 con 0x5642c6b1fd70 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.800+0000 7fc39ea00640 1 -- 172.21.15.43:0/3421734247 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398079ca0 con 0x5642c6b1fd70 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3421734247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 msgr2=0x5642c6b1efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc398092240 tx=0x7fc398048000 comp rx=0 tx=0).stop 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3421734247 shutdown_connections 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b25070 0x5642c6b415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b1f500 0x5642c6b41b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3421734247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1fd70 0x5642c6b1efc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3421734247 >> 172.21.15.43:0/3421734247 conn(0x5642c6ae6c70 msgr2=0x5642c6b4ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3421734247 shutdown_connections 2024-01-16T15:07:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3421734247 wait complete. 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.801+0000 7fc3b2d64740 1 -- start start 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.802+0000 7fc3b2d64740 1 Processor -- start 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b1fd70 0x5642c6b6b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b25070 0x5642c6be1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6b6ba90 con 0x5642c6b1f500 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642c6be1bc0 con 0x5642c6b1fd70 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642c6b531b0 con 0x5642c6b25070 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3b2d64740 10 client.0 _ll_register_callbacks cb 0x7fc39001a040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55268/0 (socket says 172.21.15.43:55268) 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 -- 172.21.15.43:0/3943068982 learned_addr learned my addr 172.21.15.43:0/3943068982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 -- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b25070 msgr2=0x5642c6be1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b25070 0x5642c6be1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 -- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b1fd70 msgr2=0x5642c6b6b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b1fd70 0x5642c6b6b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.803+0000 7fc3ad600640 1 -- 172.21.15.43:0/3943068982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642c6a53860 con 0x5642c6b1f500 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc3ad600640 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc398076cb0 tx=0x7fc39802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213b46b0704f2442 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398076ec0 con 0x5642c6b1f500 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc398077020 con 0x5642c6b1f500 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc398082ab0 con 0x5642c6b1f500 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc398079ca0 con 0x5642c6b1f500 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc3b2d64740 10 client.7164 fetch_fsmap learned FSMap version 46 2024-01-16T15:07:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x5642c6a4d170 con 0x5642c6b1f500 2024-01-16T15:07:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398082ab0 con 0x5642c6b1f500 2024-01-16T15:07:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.804+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc39808a020 con 0x5642c6b1f500 2024-01-16T15:07:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 10 client.7164 fetch_fsmap finished waiting for FSMap version 46 2024-01-16T15:07:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 10 client.7164 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:07:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 20 client.7164 populate_metadata read hostname 'smithi043' 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b412a0 0x5642c6b41680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 4 client.7164 mds_command: new command op to 6085 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 --> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5642c678c6c0 con 0x5642c6b412a0 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.805+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b412a0 0x5642c6b41680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.806+0000 7fc3acc00640 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b412a0 0x5642c6b41680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ed1e3374ffc3b1 server_cookie=2cb8e7090bad14c6 in_seq=0 out_seq=0 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.806+0000 7fc396a00640 10 client.7164 ms_handle_connect on v2:172.21.15.69:6832/1602463840 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc396a00640 1 -- 172.21.15.43:0/3943068982 <== mds.0 v2:172.21.15.69:6832/1602463840 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5642c678c6c0 con 0x5642c6b412a0 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc396a00640 10 client.7164 handle_command_reply: tid=0 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 client.7164 shutdown 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 msgr2=0x5642c6bd7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc398076cb0 tx=0x7fc39802bff0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 shutdown_connections 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:6832/1602463840,v1:172.21.15.69:6834/1602463840] conn(0x5642c6b412a0 0x5642c6b41680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5642c6b1fd70 0x5642c6b6b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5642c6b25070 0x5642c6be1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 --2- 172.21.15.43:0/3943068982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642c6b1f500 0x5642c6bd7620 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc396a00640 0 client.7164 ms_handle_reset on v2:172.21.15.69:6832/1602463840 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 >> 172.21.15.43:0/3943068982 conn(0x5642c6ae6c70 msgr2=0x5642c6b6d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 shutdown_connections 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 1 -- 172.21.15.43:0/3943068982 wait complete. 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.807+0000 7fc3b2d64740 20 client.7164 trim_cache size 0 max 16384 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 6085, 2024-01-16T15:07:30.787 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.7819481079999999, 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 46, 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 72, 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 72, 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 12.186881194 2024-01-16T15:07:30.788 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 msgr2=0x7fc38006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 0x7fc38006e0b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc3a81f66f0 tx=0x7fc390008040 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 msgr2=0x7fc3a813b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc398005040 tx=0x7fc398001120 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 shutdown_connections 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc38006bbf0 0x7fc38006e0b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3a813de70 0x7fc3a8149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3a8149750 0x7fc3a8149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 --2- 172.21.15.43:0/3265826708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc388003a10 0x7fc3a813b170 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 >> 172.21.15.43:0/3265826708 conn(0x7fc3a80b1630 msgr2=0x7fc3a80afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 shutdown_connections 2024-01-16T15:07:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:30.808+0000 7fc3aea00640 1 -- 172.21.15.43:0/3265826708 wait complete. 2024-01-16T15:07:30.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:07:31.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.050+0000 7fddb1000640 1 Processor -- start 2024-01-16T15:07:31.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.050+0000 7fddb1000640 1 -- start start 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.051+0000 7fddb1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac160b60 0x7fddac0d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.051+0000 7fddb1000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fddac0d01a0 0x7fddac0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddb1000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac0b2b30 con 0x7fddac0d3cf0 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddb1000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddac160f40 con 0x7fddac0d01a0 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddb1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0b2e10 con 0x7fddac160b60 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddaaa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fddac0d3cf0 0x7fddac0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44948/0 (socket says 172.21.15.43:44948) 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddaaa00640 1 -- 172.21.15.43:0/3504930498 learned_addr learned my addr 172.21.15.43:0/3504930498 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddabe00640 1 --2- 172.21.15.43:0/3504930498 >> v2:172.21.15.43:3300/0 conn(0x7fddac0d01a0 0x7fddac0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddab400640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac160b60 0x7fddac0d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddabe00640 1 -- 172.21.15.43:0/3504930498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac160b60 msgr2=0x7fddac0d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddabe00640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac160b60 0x7fddac0d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.052+0000 7fddabe00640 1 -- 172.21.15.43:0/3504930498 >> v1:172.21.15.43:6789/0 conn(0x7fddac0d3cf0 legacy=0x7fddac0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.053+0000 7fddabe00640 1 -- 172.21.15.43:0/3504930498 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0b30f0 con 0x7fddac0d01a0 2024-01-16T15:07:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.053+0000 7fddaa000640 1 -- 172.21.15.43:0/3504930498 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669449035 0 0) 0x7fddac0b2b30 con 0x7fddac0d3cf0 2024-01-16T15:07:31.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.054+0000 7fddabe00640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac0d01a0 0x7fddac0b3bd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009d90 tx=0x7fdda402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea977c81d48404b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:31.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.054+0000 7fddaa000640 1 -- 172.21.15.43:0/3504930498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda402cbc0 con 0x7fddac0d01a0 2024-01-16T15:07:31.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.054+0000 7fddaa000640 1 -- 172.21.15.43:0/3504930498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda402cd20 con 0x7fddac0d01a0 2024-01-16T15:07:31.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.054+0000 7fddaa000640 1 -- 172.21.15.43:0/3504930498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4034660 con 0x7fddac0d01a0 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 -- 172.21.15.43:0/3504930498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac0d01a0 msgr2=0x7fddac0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac0d01a0 0x7fddac0b3bd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009d90 tx=0x7fdda402be10 comp rx=0 tx=0).stop 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 -- 172.21.15.43:0/3504930498 shutdown_connections 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac0d01a0 0x7fddac0b3bd0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 --2- 172.21.15.43:0/3504930498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac160b60 0x7fddac0d37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 -- 172.21.15.43:0/3504930498 >> 172.21.15.43:0/3504930498 conn(0x7fddac09c3d0 msgr2=0x7fddac0d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 -- 172.21.15.43:0/3504930498 shutdown_connections 2024-01-16T15:07:31.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.055+0000 7fddb1000640 1 -- 172.21.15.43:0/3504930498 wait complete. 2024-01-16T15:07:31.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.057+0000 7fddb1000640 1 Processor -- start 2024-01-16T15:07:31.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.057+0000 7fddb1000640 1 -- start start 2024-01-16T15:07:31.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddac0d01a0 0x7fddac1441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac0d3cf0 0x7fddac1f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac156d40 con 0x7fddac160b60 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0cd0a0 con 0x7fddac0d01a0 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddb1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0ccdc0 con 0x7fddac0d3cf0 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddabe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddabe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55300/0 (socket says 172.21.15.43:55300) 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.058+0000 7fddabe00640 1 -- 172.21.15.43:0/263717665 learned_addr learned my addr 172.21.15.43:0/263717665 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddab400640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddac0d01a0 0x7fddac1441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddaaa00640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac0d3cf0 0x7fddac1f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 -- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac0d3cf0 msgr2=0x7fddac1f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac0d3cf0 0x7fddac1f5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 -- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddac0d01a0 msgr2=0x7fddac1441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddac0d01a0 0x7fddac1441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 -- 172.21.15.43:0/263717665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda4009a70 con 0x7fddac160b60 2024-01-16T15:07:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.059+0000 7fddabe00640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009eb0 tx=0x7fdda4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.060+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda403f420 con 0x7fddac160b60 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.060+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda402ce90 con 0x7fddac160b60 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.060+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda403f780 con 0x7fddac160b60 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.060+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac1566f0 con 0x7fddac160b60 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.060+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac1f63d0 con 0x7fddac160b60 2024-01-16T15:07:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.061+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddac15e8a0 con 0x7fddac160b60 2024-01-16T15:07:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.062+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fdda403f8e0 con 0x7fddac160b60 2024-01-16T15:07:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.063+0000 7fdda3400640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 0x7fdd8406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.063+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdda40ab200 con 0x7fddac160b60 2024-01-16T15:07:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.063+0000 7fddab400640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 0x7fdd8406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.064+0000 7fddab400640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 0x7fdd8406e0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c006fd0 tx=0x7fdd9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:31.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.067+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddac15e8a0 con 0x7fddac160b60 2024-01-16T15:07:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.302+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fddac15e8a0 con 0x7fddac160b60 2024-01-16T15:07:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.302+0000 7fdda3400640 1 -- 172.21.15.43:0/263717665 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdda404d060 con 0x7fddac160b60 2024-01-16T15:07:31.280 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 msgr2=0x7fdd8406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 0x7fdd8406e0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c006fd0 tx=0x7fdd9c008040 comp rx=0 tx=0).stop 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 msgr2=0x7fddac15eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009eb0 tx=0x7fdda4004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 shutdown_connections 2024-01-16T15:07:31.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd8406bbf0 0x7fdd8406e0b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fddac160b60 0x7fddac15eb80 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddac0d3cf0 0x7fddac1f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 --2- 172.21.15.43:0/263717665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddac0d01a0 0x7fddac1441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 >> 172.21.15.43:0/263717665 conn(0x7fddac09c3d0 msgr2=0x7fddac06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 shutdown_connections 2024-01-16T15:07:31.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:31.305+0000 7fddb1000640 1 -- 172.21.15.43:0/263717665 wait complete. 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:07:31.303 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:07:31.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.304 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.357 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41987sec preferred_lft 41987sec 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:31.358 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:07:31.359 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:31.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.360 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:07:31.453 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:07:31.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.458 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:07:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:07:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:07:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:07:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:07:31.512 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:07:31.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:07:31.513 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:07:31.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.706 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:07:31.759 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:07:31.759 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:07:31.760 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:07:31.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:31.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:07:31.858 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:07:31.858 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:07:31.858 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:31.912 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:07:31.912 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:07:31.912 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:31.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:07:32.068 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:07:32.122 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:07:32.122 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:07:32.122 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:07:32.122 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:07:32.122 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:07:32.123 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:07:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:07:32.125 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:07:32.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:32.126 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:07:32.180 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:07:32.180 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:07:32.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:07:32.264 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:07:32.307 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:07:32.307 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:07:32.307 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:07:32.308 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:07:32.309 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:07:32.310 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:07:32.310 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:07:32.310 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:07:32.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:32.311 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:07:32.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:07:32.401+0000 7f06ac047480 -1 init, newargv = 0x564b3e258910 newargc=15 2024-01-16T15:07:32.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[61375]: starting ceph client 2024-01-16T15:07:32.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[61375]: starting fuse 2024-01-16T15:07:32.398 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:07:32.398 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:07:32.399 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:07:32.400 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:07:32.605 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.61375.asok 2024-01-16T15:07:32.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:07:32.623 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61375.asok 2024-01-16T15:07:32.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:32.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61375.asok status 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "61375", 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:32.874 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 7182, 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 7182 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1921853163 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:32.875 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1921853163 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.7182 192.168.0.1:0/1921853163", 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/1921853163", 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 46, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 72, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:07:32.876 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:07:32.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:32.891 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:07:32.891 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:32.944 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:07:32.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:32.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:32.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:33.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:07:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.275+0000 7f5fdae00640 1 Processor -- start 2024-01-16T15:07:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.275+0000 7f5fdae00640 1 -- start start 2024-01-16T15:07:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fdae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5fd40b39a0 0x7f5fd4071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fdae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd413da30 con 0x7f5fd40abff0 2024-01-16T15:07:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fdae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd40ac560 con 0x7f5fd415a100 2024-01-16T15:07:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fdae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40ac840 con 0x7f5fd40b39a0 2024-01-16T15:07:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fd8e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5fd40abff0 0x7f5fd4071670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44900/0 (socket says 172.21.15.43:44900) 2024-01-16T15:07:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.277+0000 7f5fd8e00640 1 -- 172.21.15.43:0/3600453630 learned_addr learned my addr 172.21.15.43:0/3600453630 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.278+0000 7f5fcbe00640 1 --2- 172.21.15.43:0/3600453630 >> v2:172.21.15.69:3300/0 conn(0x7f5fd40b39a0 0x7f5fd4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.278+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936635159 0 0) 0x7f5fd413da30 con 0x7f5fd40abff0 2024-01-16T15:07:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.278+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb4005730 con 0x7f5fd40abff0 2024-01-16T15:07:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.278+0000 7f5fcbe00640 1 -- 172.21.15.43:0/3600453630 >> v1:172.21.15.69:6789/0 conn(0x7f5fd415a100 legacy=0x7f5fd40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.279+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2749345530 0 0) 0x7f5fb4005730 con 0x7f5fd40abff0 2024-01-16T15:07:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.279+0000 7f5fcbe00640 1 -- 172.21.15.43:0/3600453630 >> v1:172.21.15.43:6789/0 conn(0x7f5fd40abff0 legacy=0x7f5fd4071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.279+0000 7f5fcbe00640 1 -- 172.21.15.43:0/3600453630 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4070650 con 0x7f5fd40b39a0 2024-01-16T15:07:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.280+0000 7f5fcbe00640 1 --2- 172.21.15.43:0/3600453630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd40b39a0 0x7f5fd4071130 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0097a0 tx=0x7f5fbc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3accb1efbe39fefd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.280+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc030610 con 0x7f5fd40b39a0 2024-01-16T15:07:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.280+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc030770 con 0x7f5fd40b39a0 2024-01-16T15:07:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fd3400640 1 -- 172.21.15.43:0/3600453630 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc030b00 con 0x7f5fd40b39a0 2024-01-16T15:07:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 -- 172.21.15.43:0/3600453630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd40b39a0 msgr2=0x7f5fd4071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 --2- 172.21.15.43:0/3600453630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd40b39a0 0x7f5fd4071130 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0097a0 tx=0x7f5fbc02b960 comp rx=0 tx=0).stop 2024-01-16T15:07:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 -- 172.21.15.43:0/3600453630 shutdown_connections 2024-01-16T15:07:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 --2- 172.21.15.43:0/3600453630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd40b39a0 0x7f5fd4071130 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 -- 172.21.15.43:0/3600453630 >> 172.21.15.43:0/3600453630 conn(0x7f5fd40b0d80 msgr2=0x7f5fd40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 -- 172.21.15.43:0/3600453630 shutdown_connections 2024-01-16T15:07:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.281+0000 7f5fdae00640 1 -- 172.21.15.43:0/3600453630 wait complete. 2024-01-16T15:07:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.284+0000 7f5fdae00640 1 Processor -- start 2024-01-16T15:07:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.284+0000 7f5fdae00640 1 -- start start 2024-01-16T15:07:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fd40b39a0 0x7f5fd40ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd415a100 0x7f5fd4155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40ca7f0 con 0x7f5fd40abff0 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40cd0d0 con 0x7f5fd415a100 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fdae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40c9c90 con 0x7f5fd40b39a0 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.285+0000 7f5fd3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52698/0 (socket says 172.21.15.43:52698) 2024-01-16T15:07:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 -- 172.21.15.43:0/117557112 learned_addr learned my addr 172.21.15.43:0/117557112 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd8e00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd415a100 0x7f5fd4155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fcbe00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fd40b39a0 0x7f5fd40ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 -- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fd40b39a0 msgr2=0x7f5fd40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fd40b39a0 0x7f5fd40ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 -- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd415a100 msgr2=0x7f5fd4155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd415a100 0x7f5fd4155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 -- 172.21.15.43:0/117557112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc009400 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.286+0000 7f5fd3e00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400c450 tx=0x7f5fc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.287+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40052d0 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.287+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4005430 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.287+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4012870 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.287+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd40cd390 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.287+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd40c9920 con 0x7f5fd40abff0 2024-01-16T15:07:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.288+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd4074770 con 0x7f5fd40abff0 2024-01-16T15:07:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.289+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5fc40129d0 con 0x7f5fd40abff0 2024-01-16T15:07:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.290+0000 7f5fd0c00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 0x7f5fac06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.290+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5fc4087ff0 con 0x7f5fd40abff0 2024-01-16T15:07:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.291+0000 7f5fcbe00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 0x7f5fac06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.291+0000 7f5fcbe00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 0x7f5fac06e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc036000 tx=0x7f5fbc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.296+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fc40548b0 con 0x7f5fd40abff0 2024-01-16T15:07:33.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.527+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5fd4072580 con 0x7f5fd40abff0 2024-01-16T15:07:33.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.528+0000 7f5fd0c00640 1 -- 172.21.15.43:0/117557112 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5fc4057ed0 con 0x7f5fd40abff0 2024-01-16T15:07:33.506 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:07:33.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 msgr2=0x7f5fac06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 0x7f5fac06e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc036000 tx=0x7f5fbc008030 comp rx=0 tx=0).stop 2024-01-16T15:07:33.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 msgr2=0x7f5fd4098ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:33.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400c450 tx=0x7f5fc400c910 comp rx=0 tx=0).stop 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 shutdown_connections 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fac06bbc0 0x7f5fac06e080 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fd415a100 0x7f5fd4155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fd40b39a0 0x7f5fd40ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 --2- 172.21.15.43:0/117557112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fd40abff0 0x7f5fd4098ae0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 >> 172.21.15.43:0/117557112 conn(0x7f5fd40b0d80 msgr2=0x7f5fd40b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 shutdown_connections 2024-01-16T15:07:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:33.531+0000 7f5fdae00640 1 -- 172.21.15.43:0/117557112 wait complete. 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:07:33.529 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:07:33.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.529 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:33.584 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42146sec preferred_lft 42146sec 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.585 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:07:33.586 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:07:33.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.587 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:07:33.674 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:07:33.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.679 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:07:33.728 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:07:33.728 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:07:33.729 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:07:33.729 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:07:33.729 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:07:33.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:07:33.730 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:07:33.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.919 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:07:33.967 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:07:33.967 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:07:33.968 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:07:33.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:33.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:07:34.054 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:07:34.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:07:34.055 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:34.109 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:07:34.109 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:07:34.110 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:34.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:07:34.244 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:07:34.297 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:07:34.298 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:07:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:07:34.300 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:07:34.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:34.301 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:07:34.347 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:07:34.347 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:07:34.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:07:34.428 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:07:34.470 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:07:34.471 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:07:34.472 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:07:34.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:34.473 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:07:34.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:07:34.557+0000 7fa2f2c60480 -1 init, newargv = 0x5631a9e08c00 newargc=15 2024-01-16T15:07:34.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44850]: starting ceph client 2024-01-16T15:07:34.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44850]: starting fuse 2024-01-16T15:07:34.558 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:07:34.558 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:07:34.559 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:07:34.560 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:07:34.748 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.44850.asok 2024-01-16T15:07:34.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:07:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:07:34.764 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44850.asok 2024-01-16T15:07:34.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:34.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44850.asok status 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "44850", 2024-01-16T15:07:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5624, 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5624 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4206222953 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:07:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4206222953 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5624 192.168.0.1:0/4206222953", 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/4206222953", 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 46, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 72, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:07:34.999 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:07:35.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:35.012 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:07:35.012 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:35.062 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:07:35.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:35.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:35.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:35.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:07:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.406+0000 7efef5600640 1 Processor -- start 2024-01-16T15:07:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.407+0000 7efef5600640 1 -- start start 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.408+0000 7efef5600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efef006dc30 0x7efef00ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.408+0000 7efef5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef0070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efef5600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef006d7a0 con 0x7efef00acd70 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efef5600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef015a5b0 con 0x7efef006dc30 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efef5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00ae9d0 con 0x7efef0070490 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efef006dc30 0x7efef00ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efef006dc30 0x7efef00ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52712/0 (socket says 172.21.15.43:52712) 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 -- 172.21.15.43:0/2066939401 learned_addr learned my addr 172.21.15.43:0/2066939401 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 -- 172.21.15.43:0/2066939401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 msgr2=0x7efef0070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 --2- 172.21.15.43:0/2066939401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef0070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 -- 172.21.15.43:0/2066939401 >> v1:172.21.15.69:6789/0 conn(0x7efef00acd70 legacy=0x7efef00ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.409+0000 7efee5e00640 1 -- 172.21.15.43:0/2066939401 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef006ce70 con 0x7efef006dc30 2024-01-16T15:07:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.411+0000 7efee5e00640 1 --2- 172.21.15.43:0/2066939401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00ae490 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7efed8009870 tx=0x7efed802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830b7f7e4392a662 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.411+0000 7efeee000640 1 -- 172.21.15.43:0/2066939401 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8031640 con 0x7efef006dc30 2024-01-16T15:07:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.411+0000 7efeee000640 1 -- 172.21.15.43:0/2066939401 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed80317a0 con 0x7efef006dc30 2024-01-16T15:07:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.411+0000 7efeee000640 1 -- 172.21.15.43:0/2066939401 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8031b30 con 0x7efef006dc30 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 -- 172.21.15.43:0/2066939401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 msgr2=0x7efef00ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 --2- 172.21.15.43:0/2066939401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00ae490 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7efed8009870 tx=0x7efed802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 -- 172.21.15.43:0/2066939401 shutdown_connections 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 --2- 172.21.15.43:0/2066939401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00ae490 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 --2- 172.21.15.43:0/2066939401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef0070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 -- 172.21.15.43:0/2066939401 >> 172.21.15.43:0/2066939401 conn(0x7efef00b0ba0 msgr2=0x7efef00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 -- 172.21.15.43:0/2066939401 shutdown_connections 2024-01-16T15:07:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.412+0000 7efef5600640 1 -- 172.21.15.43:0/2066939401 wait complete. 2024-01-16T15:07:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.414+0000 7efef5600640 1 Processor -- start 2024-01-16T15:07:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.414+0000 7efef5600640 1 -- start start 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efef00acd70 0x7efef014c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef014d000 con 0x7efef006dc30 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b31c0 con 0x7efef00acd70 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efef5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b2ee0 con 0x7efef0070490 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efeeea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efeeea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52722/0 (socket says 172.21.15.43:52722) 2024-01-16T15:07:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.415+0000 7efeeea00640 1 -- 172.21.15.43:0/3781668738 learned_addr learned my addr 172.21.15.43:0/3781668738 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efee5e00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeef400640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efef00acd70 0x7efef014c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 -- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 msgr2=0x7efef00ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef00ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 -- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efef00acd70 msgr2=0x7efef014c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efef00acd70 0x7efef014c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 -- 172.21.15.43:0/3781668738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80094d0 con 0x7efef006dc30 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efee5e00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef00ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efeeea00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7efee000ba50 tx=0x7efee000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000cc10 con 0x7efef006dc30 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee000cd70 con 0x7efef006dc30 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0014680 con 0x7efef006dc30 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.416+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef014b8f0 con 0x7efef006dc30 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.417+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef014beb0 con 0x7efef006dc30 2024-01-16T15:07:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.418+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef014c0a0 con 0x7efef006dc30 2024-01-16T15:07:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.419+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7efee001d3b0 con 0x7efef006dc30 2024-01-16T15:07:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.420+0000 7efee7400640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 0x7efec806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.420+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efee00886e0 con 0x7efef006dc30 2024-01-16T15:07:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.420+0000 7efee5e00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 0x7efec806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.421+0000 7efee5e00640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 0x7efec806e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7efed8036000 tx=0x7efed803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.424+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efee0054db0 con 0x7efef006dc30 2024-01-16T15:07:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:35.662+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efef009d200 con 0x7efef006dc30 2024-01-16T15:07:36.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.076+0000 7efee7400640 1 -- 172.21.15.43:0/3781668738 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 136+0+0 (secure 0 0 0) 0x7efee00583d0 con 0x7efef006dc30 2024-01-16T15:07:36.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 msgr2=0x7efec806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 0x7efec806e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7efed8036000 tx=0x7efed803c040 comp rx=0 tx=0).stop 2024-01-16T15:07:36.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 msgr2=0x7efef00cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7efee000ba50 tx=0x7efee000bf10 comp rx=0 tx=0).stop 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 shutdown_connections 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efec806bbc0 0x7efec806e080 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efef00acd70 0x7efef014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efef0070490 0x7efef00ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.081+0000 7efef5600640 1 --2- 172.21.15.43:0/3781668738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efef006dc30 0x7efef00cb940 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.082+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 >> 172.21.15.43:0/3781668738 conn(0x7efef00b0ba0 msgr2=0x7efef00b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.082+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 shutdown_connections 2024-01-16T15:07:36.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.082+0000 7efef5600640 1 -- 172.21.15.43:0/3781668738 wait complete. 2024-01-16T15:07:36.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:07:36.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.336+0000 7f5aba800640 1 Processor -- start 2024-01-16T15:07:36.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.337+0000 7f5aba800640 1 -- start start 2024-01-16T15:07:36.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.338+0000 7f5aba800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5ab415b340 0x7f5ab40abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.338+0000 7f5aba800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.338+0000 7f5aba800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b4130 con 0x7f5ab40ce880 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.338+0000 7f5aba800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab40ac530 con 0x7f5ab415b340 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.338+0000 7f5aba800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab40b2080 con 0x7f5ab40cbcb0 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5ab40ce880 0x7f5ab415ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44914/0 (socket says 172.21.15.43:44914) 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3e00640 1 -- 172.21.15.43:0/304487127 learned_addr learned my addr 172.21.15.43:0/304487127 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3400640 1 --2- 172.21.15.43:0/304487127 >> v2:172.21.15.43:3300/0 conn(0x7f5ab415b340 0x7f5ab40abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab2a00640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab2000640 1 -- 172.21.15.43:0/304487127 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015970090 0 0) 0x7f5ab40b4130 con 0x7f5ab40ce880 2024-01-16T15:07:36.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3400640 1 -- 172.21.15.43:0/304487127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 msgr2=0x7f5ab406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3400640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3400640 1 -- 172.21.15.43:0/304487127 >> v1:172.21.15.43:6789/0 conn(0x7f5ab40ce880 legacy=0x7f5ab415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.339+0000 7f5ab3400640 1 -- 172.21.15.43:0/304487127 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab40b2360 con 0x7f5ab415b340 2024-01-16T15:07:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.340+0000 7f5ab3400640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab40abff0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40097a0 tx=0x7f5aa402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b32d6f2c0b293ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5ab2000640 1 -- 172.21.15.43:0/304487127 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa402d040 con 0x7f5ab415b340 2024-01-16T15:07:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5ab2000640 1 -- 172.21.15.43:0/304487127 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa4031810 con 0x7f5ab415b340 2024-01-16T15:07:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5ab2000640 1 -- 172.21.15.43:0/304487127 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4031ba0 con 0x7f5ab415b340 2024-01-16T15:07:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 -- 172.21.15.43:0/304487127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 msgr2=0x7f5ab40abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab40abff0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40097a0 tx=0x7f5aa402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 -- 172.21.15.43:0/304487127 shutdown_connections 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 --2- 172.21.15.43:0/304487127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab40abff0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.341+0000 7f5aba800640 1 -- 172.21.15.43:0/304487127 >> 172.21.15.43:0/304487127 conn(0x7f5ab40b0d80 msgr2=0x7f5ab40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.342+0000 7f5aba800640 1 -- 172.21.15.43:0/304487127 shutdown_connections 2024-01-16T15:07:36.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.342+0000 7f5aba800640 1 -- 172.21.15.43:0/304487127 wait complete. 2024-01-16T15:07:36.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.343+0000 7f5aba800640 1 Processor -- start 2024-01-16T15:07:36.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.344+0000 7f5aba800640 1 -- start start 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.344+0000 7f5aba800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab41595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5aba800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 0x7f5ab414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5aba800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5aba800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415c130 con 0x7f5ab415b340 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5aba800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415c990 con 0x7f5ab40ce880 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5aba800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab414d0a0 con 0x7f5ab40cbcb0 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52746/0 (socket says 172.21.15.43:52746) 2024-01-16T15:07:36.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 -- 172.21.15.43:0/2410766203 learned_addr learned my addr 172.21.15.43:0/2410766203 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:36.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3400640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab41595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab2a00640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 0x7f5ab414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 -- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 msgr2=0x7f5ab41595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab41595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 -- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 msgr2=0x7f5ab414cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 0x7f5ab414cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab3e00640 1 -- 172.21.15.43:0/2410766203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4009400 con 0x7f5ab415b340 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.345+0000 7f5ab2a00640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 0x7f5ab414cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5ab3e00640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c7d0 tx=0x7f5aac00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00a640 con 0x7f5ab415b340 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac00a7a0 con 0x7f5ab415b340 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00ab90 con 0x7f5ab415b340 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab4159bd0 con 0x7f5ab415b340 2024-01-16T15:07:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.346+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab415c620 con 0x7f5ab415b340 2024-01-16T15:07:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.347+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab415cb40 con 0x7f5ab415b340 2024-01-16T15:07:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.349+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5aac00acf0 con 0x7f5ab415b340 2024-01-16T15:07:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.350+0000 7f5aab400640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 0x7f5a8c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.350+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5aac087910 con 0x7f5ab415b340 2024-01-16T15:07:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.350+0000 7f5ab3400640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 0x7f5a8c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.351+0000 7f5ab3400640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 0x7f5a8c06e0b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4036000 tx=0x7f5aa4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.354+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ab415cb40 con 0x7f5ab415b340 2024-01-16T15:07:36.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.572+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab40b2c00 con 0x7f5a8c06bbf0 2024-01-16T15:07:36.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.574+0000 7f5aab400640 1 -- 172.21.15.43:0/2410766203 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5aa40041e0 con 0x7f5a8c06bbf0 2024-01-16T15:07:36.552 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:07:36.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:07:36.553 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:07:36.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:07:36.553 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 msgr2=0x7f5a8c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 0x7f5a8c06e0b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4036000 tx=0x7f5aa4008030 comp rx=0 tx=0).stop 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 msgr2=0x7f5ab414a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c7d0 tx=0x7f5aac00cc90 comp rx=0 tx=0).stop 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 shutdown_connections 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a8c06bbf0 0x7f5a8c06e0b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ab40ce880 0x7f5ab414cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ab415b340 0x7f5ab414a1b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 --2- 172.21.15.43:0/2410766203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ab40cbcb0 0x7f5ab41595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 >> 172.21.15.43:0/2410766203 conn(0x7f5ab40b0d80 msgr2=0x7f5ab41516b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 shutdown_connections 2024-01-16T15:07:36.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.577+0000 7f5aba800640 1 -- 172.21.15.43:0/2410766203 wait complete. 2024-01-16T15:07:36.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:07:36.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.828+0000 7f3123200640 1 Processor -- start 2024-01-16T15:07:36.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.828+0000 7f3123200640 1 -- start start 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3123200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31240aec10 0x7f3124167cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3123200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240af020 con 0x7f312406ab50 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3123200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312415d940 con 0x7f312415d170 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3123200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312415dc20 con 0x7f31240aec10 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3121400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f312406ab50 0x7f31240ae500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44920/0 (socket says 172.21.15.43:44920) 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3121400640 1 -- 172.21.15.43:0/2119721534 learned_addr learned my addr 172.21.15.43:0/2119721534 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:07:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.830+0000 7f3122800640 1 --2- 172.21.15.43:0/2119721534 >> v2:172.21.15.43:3300/0 conn(0x7f31240aec10 0x7f3124167cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.831+0000 7f3120a00640 1 -- 172.21.15.43:0/2119721534 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318500298 0 0) 0x7f31240af020 con 0x7f312406ab50 2024-01-16T15:07:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.831+0000 7f3120a00640 1 -- 172.21.15.43:0/2119721534 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104003520 con 0x7f312406ab50 2024-01-16T15:07:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.831+0000 7f3122800640 1 -- 172.21.15.43:0/2119721534 >> v1:172.21.15.69:6789/0 conn(0x7f312415d170 legacy=0x7f312406a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.831+0000 7f3122800640 1 -- 172.21.15.43:0/2119721534 >> v1:172.21.15.43:6789/0 conn(0x7f312406ab50 legacy=0x7f31240ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.831+0000 7f3122800640 1 -- 172.21.15.43:0/2119721534 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312415df00 con 0x7f31240aec10 2024-01-16T15:07:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.832+0000 7f3122800640 1 --2- 172.21.15.43:0/2119721534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124167cb0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f311c0097a0 tx=0x7f311c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff038d3e888c1f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.832+0000 7f3120a00640 1 -- 172.21.15.43:0/2119721534 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c030610 con 0x7f31240aec10 2024-01-16T15:07:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.832+0000 7f3120a00640 1 -- 172.21.15.43:0/2119721534 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c030770 con 0x7f31240aec10 2024-01-16T15:07:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.832+0000 7f3120a00640 1 -- 172.21.15.43:0/2119721534 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c030b00 con 0x7f31240aec10 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 -- 172.21.15.43:0/2119721534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 msgr2=0x7f3124167cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 --2- 172.21.15.43:0/2119721534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124167cb0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f311c0097a0 tx=0x7f311c02b960 comp rx=0 tx=0).stop 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 -- 172.21.15.43:0/2119721534 shutdown_connections 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 --2- 172.21.15.43:0/2119721534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124167cb0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 -- 172.21.15.43:0/2119721534 >> 172.21.15.43:0/2119721534 conn(0x7f3124071260 msgr2=0x7f3124071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 -- 172.21.15.43:0/2119721534 shutdown_connections 2024-01-16T15:07:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.833+0000 7f3123200640 1 -- 172.21.15.43:0/2119721534 wait complete. 2024-01-16T15:07:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.835+0000 7f3123200640 1 Processor -- start 2024-01-16T15:07:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.835+0000 7f3123200640 1 -- start start 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312406ab50 0x7f31241448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312415d170 0x7f31241477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124157570 con 0x7f31240aec10 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124147e40 con 0x7f312406ab50 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3123200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3124145290 con 0x7f312415d170 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3121400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3121400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52760/0 (socket says 172.21.15.43:52760) 2024-01-16T15:07:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.836+0000 7f3121400640 1 -- 172.21.15.43:0/2851300474 learned_addr learned my addr 172.21.15.43:0/2851300474 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121e00640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312406ab50 0x7f31241448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3122800640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312415d170 0x7f31241477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 -- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312415d170 msgr2=0x7f31241477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312415d170 0x7f31241477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 -- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312406ab50 msgr2=0x7f31241448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312406ab50 0x7f31241448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 -- 172.21.15.43:0/2851300474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c009400 con 0x7f31240aec10 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3121400640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c500 tx=0x7f310c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c005ce0 con 0x7f31240aec10 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c005e40 con 0x7f31240aec10 2024-01-16T15:07:36.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c007330 con 0x7f31240aec10 2024-01-16T15:07:36.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.837+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3124145570 con 0x7f31240aec10 2024-01-16T15:07:36.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.838+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3124148120 con 0x7f31240aec10 2024-01-16T15:07:36.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.839+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31240b5340 con 0x7f31240aec10 2024-01-16T15:07:36.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.840+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f310c007550 con 0x7f31240aec10 2024-01-16T15:07:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.841+0000 7f311a000640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 0x7f30fc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.841+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f310c087b50 con 0x7f31240aec10 2024-01-16T15:07:36.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.842+0000 7f3121e00640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 0x7f30fc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.842+0000 7f3121e00640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 0x7f30fc06e080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f31140012e0 tx=0x7f3114009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:36.846+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f310c0544d0 con 0x7f31240aec10 2024-01-16T15:07:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.039+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f312404e840 con 0x7f31240aec10 2024-01-16T15:07:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.040+0000 7f311a000640 1 -- 172.21.15.43:0/2851300474 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f310c057af0 con 0x7f31240aec10 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 msgr2=0x7f30fc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 0x7f30fc06e080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f31140012e0 tx=0x7f3114009040 comp rx=0 tx=0).stop 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 msgr2=0x7f3124147280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c500 tx=0x7f310c00c9c0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 shutdown_connections 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30fc06bbc0 0x7f30fc06e080 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f312415d170 0x7f31241477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31240aec10 0x7f3124147280 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 --2- 172.21.15.43:0/2851300474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312406ab50 0x7f31241448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 >> 172.21.15.43:0/2851300474 conn(0x7f3124071260 msgr2=0x7f3124071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 shutdown_connections 2024-01-16T15:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.043+0000 7f3123200640 1 -- 172.21.15.43:0/2851300474 wait complete. 2024-01-16T15:07:37.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:07:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.284+0000 7fe4fa400640 1 Processor -- start 2024-01-16T15:07:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.284+0000 7fe4fa400640 1 -- start start 2024-01-16T15:07:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.286+0000 7fe4fa400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4f40acd70 0x7fe4f406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.286+0000 7fe4fa400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4f4070490 0x7fe4f4070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.286+0000 7fe4fa400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f406d7a0 con 0x7fe4f40ae490 2024-01-16T15:07:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.286+0000 7fe4fa400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f415a5b0 con 0x7fe4f4070490 2024-01-16T15:07:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.286+0000 7fe4fa400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f406ce70 con 0x7fe4f40acd70 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4f4070490 0x7fe4f4070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe4f4070490 0x7fe4f4070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52768/0 (socket says 172.21.15.43:52768) 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 -- 172.21.15.43:0/4236749892 learned_addr learned my addr 172.21.15.43:0/4236749892 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 -- 172.21.15.43:0/4236749892 >> v2:172.21.15.69:3300/0 conn(0x7fe4f40acd70 msgr2=0x7fe4f406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 --2- 172.21.15.43:0/4236749892 >> v2:172.21.15.69:3300/0 conn(0x7fe4f40acd70 0x7fe4f406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 -- 172.21.15.43:0/4236749892 >> v1:172.21.15.69:6789/0 conn(0x7fe4f40ae490 legacy=0x7fe4f40ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.287+0000 7fe4f3e00640 1 -- 172.21.15.43:0/4236749892 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f406d150 con 0x7fe4f4070490 2024-01-16T15:07:37.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.288+0000 7fe4f3e00640 1 --2- 172.21.15.43:0/4236749892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f4070490 0x7fe4f4070900 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40700f0 tx=0x7fe4ec02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37b6a7c092e07bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.288+0000 7fe4f2a00640 1 -- 172.21.15.43:0/4236749892 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec02f8e0 con 0x7fe4f4070490 2024-01-16T15:07:37.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4f2a00640 1 -- 172.21.15.43:0/4236749892 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec02fa40 con 0x7fe4f4070490 2024-01-16T15:07:37.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4f2a00640 1 -- 172.21.15.43:0/4236749892 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec02fdd0 con 0x7fe4f4070490 2024-01-16T15:07:37.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 -- 172.21.15.43:0/4236749892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f4070490 msgr2=0x7fe4f4070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 --2- 172.21.15.43:0/4236749892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f4070490 0x7fe4f4070900 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40700f0 tx=0x7fe4ec02db20 comp rx=0 tx=0).stop 2024-01-16T15:07:37.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 -- 172.21.15.43:0/4236749892 shutdown_connections 2024-01-16T15:07:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 --2- 172.21.15.43:0/4236749892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f4070490 0x7fe4f4070900 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 --2- 172.21.15.43:0/4236749892 >> v2:172.21.15.69:3300/0 conn(0x7fe4f40acd70 0x7fe4f406dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.289+0000 7fe4fa400640 1 -- 172.21.15.43:0/4236749892 >> 172.21.15.43:0/4236749892 conn(0x7fe4f40b0ba0 msgr2=0x7fe4f40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.290+0000 7fe4fa400640 1 -- 172.21.15.43:0/4236749892 shutdown_connections 2024-01-16T15:07:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.290+0000 7fe4fa400640 1 -- 172.21.15.43:0/4236749892 wait complete. 2024-01-16T15:07:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.291+0000 7fe4fa400640 1 Processor -- start 2024-01-16T15:07:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.291+0000 7fe4fa400640 1 -- start start 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4f4070490 0x7fe4f4158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f40ae490 0x7fe4f415c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f41596f0 con 0x7fe4f40acd70 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f414c680 con 0x7fe4f40ae490 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.292+0000 7fe4fa400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4159b10 con 0x7fe4f4070490 2024-01-16T15:07:37.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52776/0 (socket says 172.21.15.43:52776) 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 -- 172.21.15.43:0/3588016318 learned_addr learned my addr 172.21.15.43:0/3588016318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4f3e00640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f40ae490 0x7fe4f415c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4f3400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4f4070490 0x7fe4f4158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 -- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4f4070490 msgr2=0x7fe4f4158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4f4070490 0x7fe4f4158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 -- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f40ae490 msgr2=0x7fe4f415c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f40ae490 0x7fe4f415c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.293+0000 7fe4ea800640 1 -- 172.21.15.43:0/3588016318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0094d0 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4ea800640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00c410 tx=0x7fe4dc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0035f0 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc003750 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f4158a70 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc003b40 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.294+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f4159030 con 0x7fe4f40acd70 2024-01-16T15:07:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.295+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f409d200 con 0x7fe4f40acd70 2024-01-16T15:07:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.296+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe4dc003ca0 con 0x7fe4f40acd70 2024-01-16T15:07:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.297+0000 7fe4ebe00640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 0x7fe4cc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.297+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe4dc087bd0 con 0x7fe4f40acd70 2024-01-16T15:07:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.297+0000 7fe4f3400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 0x7fe4cc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.298+0000 7fe4f3400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 0x7fe4cc06e0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4007dc0 tx=0x7fe4e40072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.302+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4dc054400 con 0x7fe4f40acd70 2024-01-16T15:07:37.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.523+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4f40b3d90 con 0x7fe4cc06bbf0 2024-01-16T15:07:37.509 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:07:37.531+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:37.509 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:07:37.531+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:37.509 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:07:37.531+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:37.509 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:07:37.531+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:37.509 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:07:37.531+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.597+0000 7fe4ebe00640 1 -- 172.21.15.43:0/3588016318 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe4e40079b0 con 0x7fe4cc06bbf0 2024-01-16T15:07:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 msgr2=0x7fe4cc06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 0x7fe4cc06e0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4007dc0 tx=0x7fe4e40072f0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 msgr2=0x7fe4f41e37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00c410 tx=0x7fe4dc00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 shutdown_connections 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4cc06bbf0 0x7fe4cc06e0b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4f40ae490 0x7fe4f415c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f40acd70 0x7fe4f41e37c0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 --2- 172.21.15.43:0/3588016318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4f4070490 0x7fe4f4158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 >> 172.21.15.43:0/3588016318 conn(0x7fe4f40b0ba0 msgr2=0x7fe4f40af070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 shutdown_connections 2024-01-16T15:07:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.601+0000 7fe4fa400640 1 -- 172.21.15.43:0/3588016318 wait complete. 2024-01-16T15:07:37.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:07:37.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.841+0000 7f96d7200640 1 Processor -- start 2024-01-16T15:07:37.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.841+0000 7f96d7200640 1 -- start start 2024-01-16T15:07:37.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96d81435d0 0x7f96d8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81439b0 0x7f96d80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 0x7f96d8070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d80b4400 con 0x7f96d81435d0 2024-01-16T15:07:37.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d8070d40 con 0x7f96d80b3f70 2024-01-16T15:07:37.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8071020 con 0x7f96d81439b0 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96d81435d0 0x7f96d8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 0x7f96d8070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81439b0 0x7f96d80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 0x7f96d8070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:59770/0 (socket says 172.21.15.43:59770) 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96d81435d0 0x7f96d8071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52796/0 (socket says 172.21.15.43:52796) 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.843+0000 7f96d6800640 1 -- 172.21.15.43:0/1480814939 learned_addr learned my addr 172.21.15.43:0/1480814939 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.844+0000 7f96d5e00640 1 -- 172.21.15.43:0/1480814939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81439b0 msgr2=0x7f96d80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.844+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1480814939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81439b0 0x7f96d80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.844+0000 7f96d5e00640 1 -- 172.21.15.43:0/1480814939 >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 msgr2=0x7f96d8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:37.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.844+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1480814939 >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 0x7f96d8070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.844+0000 7f96d5e00640 1 -- 172.21.15.43:0/1480814939 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8071300 con 0x7f96d81435d0 2024-01-16T15:07:37.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.846+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1480814939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81435d0 0x7f96d8071de0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f96c80097a0 tx=0x7f96c802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9379a6bfe180d651 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.846+0000 7f96d5400640 1 -- 172.21.15.43:0/1480814939 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8031780 con 0x7f96d81435d0 2024-01-16T15:07:37.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.846+0000 7f96d5400640 1 -- 172.21.15.43:0/1480814939 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c80318e0 con 0x7f96d81435d0 2024-01-16T15:07:37.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.846+0000 7f96d5400640 1 -- 172.21.15.43:0/1480814939 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8031c70 con 0x7f96d81435d0 2024-01-16T15:07:37.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 -- 172.21.15.43:0/1480814939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81435d0 msgr2=0x7f96d8071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 --2- 172.21.15.43:0/1480814939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81435d0 0x7f96d8071de0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f96c80097a0 tx=0x7f96c802b820 comp rx=0 tx=0).stop 2024-01-16T15:07:37.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 -- 172.21.15.43:0/1480814939 shutdown_connections 2024-01-16T15:07:37.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 --2- 172.21.15.43:0/1480814939 >> v2:172.21.15.69:3300/0 conn(0x7f96d80b3f70 0x7f96d8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 --2- 172.21.15.43:0/1480814939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81439b0 0x7f96d80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 --2- 172.21.15.43:0/1480814939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81435d0 0x7f96d8071de0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 -- 172.21.15.43:0/1480814939 >> 172.21.15.43:0/1480814939 conn(0x7f96d80cec10 msgr2=0x7f96d80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:37.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 -- 172.21.15.43:0/1480814939 shutdown_connections 2024-01-16T15:07:37.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.847+0000 7f96d7200640 1 -- 172.21.15.43:0/1480814939 wait complete. 2024-01-16T15:07:37.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.850+0000 7f96d7200640 1 Processor -- start 2024-01-16T15:07:37.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.850+0000 7f96d7200640 1 -- start start 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96d80b3f70 0x7f96d815b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81435d0 0x7f96d80a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d80a97c0 con 0x7f96d81439b0 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8140650 con 0x7f96d80b3f70 2024-01-16T15:07:37.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.851+0000 7f96d7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8140930 con 0x7f96d81435d0 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52802/0 (socket says 172.21.15.43:52802) 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 -- 172.21.15.43:0/1728888029 learned_addr learned my addr 172.21.15.43:0/1728888029 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96d80b3f70 0x7f96d815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96cfe00640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81435d0 0x7f96d80a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 -- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81435d0 msgr2=0x7f96d80a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81435d0 0x7f96d80a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 -- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96d80b3f70 msgr2=0x7f96d815b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96d80b3f70 0x7f96d815b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.852+0000 7f96d6800640 1 -- 172.21.15.43:0/1728888029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c8009400 con 0x7f96d81439b0 2024-01-16T15:07:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96d6800640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009330 tx=0x7f96d000d480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d000dea0 con 0x7f96d81439b0 2024-01-16T15:07:37.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d000b310 con 0x7f96d81439b0 2024-01-16T15:07:37.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d8140c10 con 0x7f96d81439b0 2024-01-16T15:07:37.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0003bb0 con 0x7f96d81439b0 2024-01-16T15:07:37.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.853+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d815bdc0 con 0x7f96d81439b0 2024-01-16T15:07:37.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.854+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8141130 con 0x7f96d81439b0 2024-01-16T15:07:37.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.856+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 20) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f96d0009050 con 0x7f96d81439b0 2024-01-16T15:07:37.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.856+0000 7f96ce000640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 0x7f96b006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:37.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.856+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f96d0088640 con 0x7f96d81439b0 2024-01-16T15:07:37.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.857+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 0x7f96b006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:37.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.858+0000 7f96d5e00640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 0x7f96b006e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f96c8036000 tx=0x7f96c803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:37.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:37.861+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96d0054f00 con 0x7f96d81439b0 2024-01-16T15:07:38.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.088+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f96d81407c0 con 0x7f96b006bbc0 2024-01-16T15:07:38.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.095+0000 7f96ce000640 1 -- 172.21.15.43:0/1728888029 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f96d81407c0 con 0x7f96b006bbc0 2024-01-16T15:07:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:07:38.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.098+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 msgr2=0x7f96b006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:38.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.098+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 0x7f96b006e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f96c8036000 tx=0x7f96c803d040 comp rx=0 tx=0).stop 2024-01-16T15:07:38.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.098+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 msgr2=0x7f96d80ac530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:38.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.098+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009330 tx=0x7f96d000d480 comp rx=0 tx=0).stop 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 shutdown_connections 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b006bbc0 0x7f96b006e080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96d81439b0 0x7f96d80ac530 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96d81435d0 0x7f96d80a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 --2- 172.21.15.43:0/1728888029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96d80b3f70 0x7f96d815b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 >> 172.21.15.43:0/1728888029 conn(0x7f96d80cec10 msgr2=0x7f96d80af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:38.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 shutdown_connections 2024-01-16T15:07:38.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:07:38.099+0000 7f96d7200640 1 -- 172.21.15.43:0/1728888029 wait complete. 2024-01-16T15:07:38.099 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:07:38.099 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:38.099 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:38.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:38.155 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:07:54.718 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:07:54.718 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:07:54.718 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.4762 s, 65.2 MB/s 2024-01-16T15:07:54.720 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:07:54.720 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:54.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:54.777 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:07:56.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:56.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:08.733 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:08:08.733 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:08:08.733 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.8908 s, 77.3 MB/s 2024-01-16T15:08:08.735 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:08:08.735 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:08:08.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:08.794 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:08:23.905 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:08:23.905 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:08:23.906 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.0447 s, 71.4 MB/s 2024-01-16T15:08:23.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-16T15:08:24.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.159+0000 7f1dfc000640 1 Processor -- start 2024-01-16T15:08:24.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.160+0000 7f1dfc000640 1 -- start start 2024-01-16T15:08:24.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.161+0000 7f1dfc000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1df415b350 0x7f1df40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1dfc000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1df40adb60 0x7f1df40adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1dfc000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df415b730 con 0x7f1df409cb30 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1dfc000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40ac740 con 0x7f1df415b350 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1dfc000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40ac460 con 0x7f1df40adb60 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1df415b350 0x7f1df40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1df415b350 0x7f1df40abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35860/0 (socket says 172.21.15.43:35860) 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 -- 172.21.15.43:0/3265719236 learned_addr learned my addr 172.21.15.43:0/3265719236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1dfa000640 1 --2- 172.21.15.43:0/3265719236 >> v2:172.21.15.69:3300/0 conn(0x7f1df40adb60 0x7f1df40adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 -- 172.21.15.43:0/3265719236 >> v2:172.21.15.69:3300/0 conn(0x7f1df40adb60 msgr2=0x7f1df40adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 --2- 172.21.15.43:0/3265719236 >> v2:172.21.15.69:3300/0 conn(0x7f1df40adb60 0x7f1df40adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 -- 172.21.15.43:0/3265719236 >> v1:172.21.15.69:6789/0 conn(0x7f1df409cb30 legacy=0x7f1df40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:24.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.162+0000 7f1df9600640 1 -- 172.21.15.43:0/3265719236 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40b2a50 con 0x7f1df415b350 2024-01-16T15:08:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.164+0000 7f1df9600640 1 --2- 172.21.15.43:0/3265719236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df415b350 0x7f1df40abe80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1df406dac0 tx=0x7f1ddc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eba15d01af460f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.164+0000 7f1df3e00640 1 -- 172.21.15.43:0/3265719236 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc031870 con 0x7f1df415b350 2024-01-16T15:08:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.164+0000 7f1df3e00640 1 -- 172.21.15.43:0/3265719236 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0319d0 con 0x7f1df415b350 2024-01-16T15:08:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.164+0000 7f1df3e00640 1 -- 172.21.15.43:0/3265719236 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc031d60 con 0x7f1df415b350 2024-01-16T15:08:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 -- 172.21.15.43:0/3265719236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df415b350 msgr2=0x7f1df40abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 --2- 172.21.15.43:0/3265719236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df415b350 0x7f1df40abe80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1df406dac0 tx=0x7f1ddc02b910 comp rx=0 tx=0).stop 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 -- 172.21.15.43:0/3265719236 shutdown_connections 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 --2- 172.21.15.43:0/3265719236 >> v2:172.21.15.69:3300/0 conn(0x7f1df40adb60 0x7f1df40adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 --2- 172.21.15.43:0/3265719236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df415b350 0x7f1df40abe80 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 -- 172.21.15.43:0/3265719236 >> 172.21.15.43:0/3265719236 conn(0x7f1df40b0ba0 msgr2=0x7f1df40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 -- 172.21.15.43:0/3265719236 shutdown_connections 2024-01-16T15:08:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.165+0000 7f1dfc000640 1 -- 172.21.15.43:0/3265719236 wait complete. 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.167+0000 7f1dfc000640 1 Processor -- start 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.167+0000 7f1dfc000640 1 -- start start 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df409cb30 0x7f1df414ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 0x7f1df413e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4153860 con 0x7f1df40adb60 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4154100 con 0x7f1df409cb30 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1dfc000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4158650 con 0x7f1df415b350 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1df8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1df8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35870/0 (socket says 172.21.15.43:35870) 2024-01-16T15:08:24.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1df8c00640 1 -- 172.21.15.43:0/2586943014 learned_addr learned my addr 172.21.15.43:0/2586943014 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.168+0000 7f1df9600640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df409cb30 0x7f1df414ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1dfa000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 0x7f1df413e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 -- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 msgr2=0x7f1df413e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 0x7f1df413e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 -- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df409cb30 msgr2=0x7f1df414ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df409cb30 0x7f1df414ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 -- 172.21.15.43:0/2586943014 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc0094d0 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1dfa000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 0x7f1df413e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df8c00640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1de400c7c0 tx=0x7f1de400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400aa50 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400abb0 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4012420 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.169+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df414f3e0 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.170+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df4153d90 con 0x7f1df40adb60 2024-01-16T15:08:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.171+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df406e6d0 con 0x7f1df40adb60 2024-01-16T15:08:24.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.172+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f1de4012580 con 0x7f1df40adb60 2024-01-16T15:08:24.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.172+0000 7f1df1600640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 0x7f1dcc06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.173+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1de4088940 con 0x7f1df40adb60 2024-01-16T15:08:24.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.173+0000 7f1df9600640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 0x7f1dcc06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.173+0000 7f1df9600640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 0x7f1dcc06e160 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc036000 tx=0x7f1ddc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.177+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1de4055000 con 0x7f1df40adb60 2024-01-16T15:08:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.371+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f1df4154270 con 0x7f1df40adb60 2024-01-16T15:08:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.376+0000 7f1df1600640 1 -- 172.21.15.43:0/2586943014 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1de4058620 con 0x7f1df40adb60 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.379+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 msgr2=0x7f1dcc06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.379+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 0x7f1dcc06e160 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc036000 tx=0x7f1ddc03c040 comp rx=0 tx=0).stop 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.379+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 msgr2=0x7f1df4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.379+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1de400c7c0 tx=0x7f1de400cc80 comp rx=0 tx=0).stop 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 shutdown_connections 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1dcc06bca0 0x7f1dcc06e160 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1df415b350 0x7f1df413e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1df40adb60 0x7f1df4158110 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 --2- 172.21.15.43:0/2586943014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1df409cb30 0x7f1df414ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 >> 172.21.15.43:0/2586943014 conn(0x7f1df40b0ba0 msgr2=0x7f1df406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:24.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 shutdown_connections 2024-01-16T15:08:24.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.380+0000 7f1dfc000640 1 -- 172.21.15.43:0/2586943014 wait complete. 2024-01-16T15:08:24.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:08:24.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.628+0000 7f7e2be00640 1 Processor -- start 2024-01-16T15:08:24.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.628+0000 7f7e2be00640 1 -- start start 2024-01-16T15:08:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.629+0000 7f7e2be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e2c15b480 0x7f7e2c0cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.629+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.629+0000 7f7e2be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0adad0 con 0x7f7e2c0cf960 2024-01-16T15:08:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.629+0000 7f7e2be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c0cfdf0 con 0x7f7e2c15b480 2024-01-16T15:08:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.629+0000 7f7e2be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b1170 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e2c15b480 0x7f7e2c0cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e2c15b480 0x7f7e2c0cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35884/0 (socket says 172.21.15.43:35884) 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 -- 172.21.15.43:0/18725155 learned_addr learned my addr 172.21.15.43:0/18725155 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e23e00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 -- 172.21.15.43:0/18725155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0d1ab0 msgr2=0x7f7e2c0cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.630+0000 7f7e2aa00640 1 -- 172.21.15.43:0/18725155 >> v1:172.21.15.69:6789/0 conn(0x7f7e2c0cf960 legacy=0x7f7e2c0cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.631+0000 7f7e2aa00640 1 -- 172.21.15.43:0/18725155 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0ccc90 con 0x7f7e2c15b480 2024-01-16T15:08:24.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.632+0000 7f7e2aa00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c15b480 0x7f7e2c0cc460 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009870 tx=0x7f7e1c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6f2269cabdea2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.632+0000 7f7e2a000640 1 -- 172.21.15.43:0/18725155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c02f840 con 0x7f7e2c15b480 2024-01-16T15:08:24.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.632+0000 7f7e2a000640 1 -- 172.21.15.43:0/18725155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c02f9a0 con 0x7f7e2c15b480 2024-01-16T15:08:24.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.632+0000 7f7e2a000640 1 -- 172.21.15.43:0/18725155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c02fd30 con 0x7f7e2c15b480 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 -- 172.21.15.43:0/18725155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c15b480 msgr2=0x7f7e2c0cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c15b480 0x7f7e2c0cc460 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009870 tx=0x7f7e1c02ba20 comp rx=0 tx=0).stop 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 -- 172.21.15.43:0/18725155 shutdown_connections 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 --2- 172.21.15.43:0/18725155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c15b480 0x7f7e2c0cc460 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 -- 172.21.15.43:0/18725155 >> 172.21.15.43:0/18725155 conn(0x7f7e2c0d0e40 msgr2=0x7f7e2c0d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 -- 172.21.15.43:0/18725155 shutdown_connections 2024-01-16T15:08:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.633+0000 7f7e2be00640 1 -- 172.21.15.43:0/18725155 wait complete. 2024-01-16T15:08:24.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.635+0000 7f7e2be00640 1 Processor -- start 2024-01-16T15:08:24.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.635+0000 7f7e2be00640 1 -- start start 2024-01-16T15:08:24.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c0cf960 0x7f7e2c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c15b480 0x7f7e2c0b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1f3f60 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06d2c0 con 0x7f7e2c0cf960 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.636+0000 7f7e2be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1f41b0 con 0x7f7e2c15b480 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35888/0 (socket says 172.21.15.43:35888) 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 -- 172.21.15.43:0/45787834 learned_addr learned my addr 172.21.15.43:0/45787834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e2aa00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c0cf960 0x7f7e2c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e2b400640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c15b480 0x7f7e2c0b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 -- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c15b480 msgr2=0x7f7e2c0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c15b480 0x7f7e2c0b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 -- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c0cf960 msgr2=0x7f7e2c1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c0cf960 0x7f7e2c1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.637+0000 7f7e23e00640 1 -- 172.21.15.43:0/45787834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0094d0 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e23e00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400d800 tx=0x7f7e1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400b6c0 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1400b820 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400bbb0 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c1f4490 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.638+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c1577f0 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.639+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c04e840 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.640+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f7e1400bd10 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.641+0000 7f7e22a00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 0x7f7e0406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:24.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.641+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7e14088be0 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.641+0000 7f7e2aa00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 0x7f7e0406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:24.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.642+0000 7f7e2aa00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 0x7f7e0406e110 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c036000 tx=0x7f7e1c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:24.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.646+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e14055300 con 0x7f7e2c0d1ab0 2024-01-16T15:08:24.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.876+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e2c0b0e00 con 0x7f7e0406bc50 2024-01-16T15:08:24.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.907+0000 7f7e22a00640 1 -- 172.21.15.43:0/45787834 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e1c02fc10 con 0x7f7e0406bc50 2024-01-16T15:08:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 msgr2=0x7f7e0406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 0x7f7e0406e110 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c036000 tx=0x7f7e1c008030 comp rx=0 tx=0).stop 2024-01-16T15:08:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 msgr2=0x7f7e2c0b5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400d800 tx=0x7f7e1400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 shutdown_connections 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e0406bc50 0x7f7e0406e110 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2c15b480 0x7f7e2c0b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e2c0d1ab0 0x7f7e2c0b5a10 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 --2- 172.21.15.43:0/45787834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2c0cf960 0x7f7e2c1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 >> 172.21.15.43:0/45787834 conn(0x7f7e2c0d0e40 msgr2=0x7f7e2c0ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 shutdown_connections 2024-01-16T15:08:24.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:24.911+0000 7f7e2be00640 1 -- 172.21.15.43:0/45787834 wait complete. 2024-01-16T15:08:24.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:08:25.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.151+0000 7fcb39200640 1 Processor -- start 2024-01-16T15:08:25.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.151+0000 7fcb39200640 1 -- start start 2024-01-16T15:08:25.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb39200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb34153ea0 0x7fcb340ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb39200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb340ce650 con 0x7fcb34154280 2024-01-16T15:08:25.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb39200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb34154710 con 0x7fcb34070850 2024-01-16T15:08:25.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb39200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3415e0a0 con 0x7fcb34153ea0 2024-01-16T15:08:25.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb32a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcb34154280 0x7fcb3415d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57602/0 (socket says 172.21.15.43:57602) 2024-01-16T15:08:25.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.153+0000 7fcb32a00640 1 -- 172.21.15.43:0/1670328752 learned_addr learned my addr 172.21.15.43:0/1670328752 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:25.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb33400640 1 --2- 172.21.15.43:0/1670328752 >> v2:172.21.15.69:3300/0 conn(0x7fcb34153ea0 0x7fcb340ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065960441 0 0) 0x7fcb340ce650 con 0x7fcb34154280 2024-01-16T15:08:25.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb14003520 con 0x7fcb34154280 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 505551420 0 0) 0x7fcb14003520 con 0x7fcb34154280 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 >> v2:172.21.15.69:3300/0 conn(0x7fcb34153ea0 msgr2=0x7fcb340ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 --2- 172.21.15.43:0/1670328752 >> v2:172.21.15.69:3300/0 conn(0x7fcb34153ea0 0x7fcb340ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 >> v1:172.21.15.69:6789/0 conn(0x7fcb34070850 legacy=0x7fcb34071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3415e380 con 0x7fcb34154280 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcb1c001670 con 0x7fcb34154280 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 >> v1:172.21.15.43:6789/0 conn(0x7fcb34154280 legacy=0x7fcb3415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 0x7fcb14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.154+0000 7fcb32000640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140047c0 0x7fcb14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb32000640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb14005180 0x7fcb1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3415e0a0 con 0x7fcb140039e0 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1400c000 con 0x7fcb14005180 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1400c1c0 con 0x7fcb140047c0 2024-01-16T15:08:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 0x7fcb14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb32a00640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140047c0 0x7fcb14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33e00640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb14005180 0x7fcb1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 -- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140047c0 msgr2=0x7fcb14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140047c0 0x7fcb14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 -- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb14005180 msgr2=0x7fcb1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb14005180 0x7fcb1400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33400640 1 -- 172.21.15.43:0/1670328752 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb14003520 con 0x7fcb140039e0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33e00640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb14005180 0x7fcb1400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.155+0000 7fcb33e00640 1 -- 172.21.15.43:0/1670328752 reap_dead start 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.157+0000 7fcb33400640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 0x7fcb14003dc0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009ac0 tx=0x7fcb140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e459b18fe7d52fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.157+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2402c6b0 con 0x7fcb140039e0 2024-01-16T15:08:25.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.157+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2402c810 con 0x7fcb140039e0 2024-01-16T15:08:25.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.157+0000 7fcb32000640 1 -- 172.21.15.43:0/1670328752 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2402cba0 con 0x7fcb140039e0 2024-01-16T15:08:25.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 -- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 msgr2=0x7fcb14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 0x7fcb14003dc0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009ac0 tx=0x7fcb140057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 -- 172.21.15.43:0/1670328752 shutdown_connections 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 --2- 172.21.15.43:0/1670328752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb140039e0 0x7fcb14003dc0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 -- 172.21.15.43:0/1670328752 >> 172.21.15.43:0/1670328752 conn(0x7fcb340cec10 msgr2=0x7fcb340cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 -- 172.21.15.43:0/1670328752 shutdown_connections 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.158+0000 7fcb39200640 1 -- 172.21.15.43:0/1670328752 wait complete. 2024-01-16T15:08:25.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.160+0000 7fcb39200640 1 Processor -- start 2024-01-16T15:08:25.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.160+0000 7fcb39200640 1 -- start start 2024-01-16T15:08:25.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 0x7fcb34142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 0x7fcb34142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34159950 con 0x7fcb341583d0 2024-01-16T15:08:25.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34158cd0 con 0x7fcb34151f50 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb39200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34158fb0 con 0x7fcb140039e0 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35908/0 (socket says 172.21.15.43:35908) 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 -- 172.21.15.43:0/956669503 learned_addr learned my addr 172.21.15.43:0/956669503 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33400640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 0x7fcb34142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb32a00640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 0x7fcb34142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 -- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 msgr2=0x7fcb34142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 0x7fcb34142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 -- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 msgr2=0x7fcb34142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 0x7fcb34142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33e00640 1 -- 172.21.15.43:0/956669503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34159290 con 0x7fcb341583d0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb33400640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 0x7fcb34142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.161+0000 7fcb32a00640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 0x7fcb34142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb33e00640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0012e0 tx=0x7fcb2c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c003b70 con 0x7fcb341583d0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c003cd0 con 0x7fcb341583d0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb34159570 con 0x7fcb341583d0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c0052b0 con 0x7fcb341583d0 2024-01-16T15:08:25.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.162+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb34158850 con 0x7fcb341583d0 2024-01-16T15:08:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.164+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb34159a90 con 0x7fcb341583d0 2024-01-16T15:08:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.164+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fcb2c005410 con 0x7fcb341583d0 2024-01-16T15:08:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.165+0000 7fcb2b400640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 0x7fcb0c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.165+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcb2c087fc0 con 0x7fcb341583d0 2024-01-16T15:08:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.166+0000 7fcb33400640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 0x7fcb0c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.166+0000 7fcb33400640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 0x7fcb0c06e110 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcb24036000 tx=0x7fcb24030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.170+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb34159a90 con 0x7fcb341583d0 2024-01-16T15:08:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.392+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb34066e60 con 0x7fcb0c06bc50 2024-01-16T15:08:25.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.404+0000 7fcb2b400640 1 -- 172.21.15.43:0/956669503 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fcb34066e60 con 0x7fcb0c06bc50 2024-01-16T15:08:25.382 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 msgr2=0x7fcb0c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 0x7fcb0c06e110 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcb24036000 tx=0x7fcb24030000 comp rx=0 tx=0).stop 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 msgr2=0x7fcb3415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0012e0 tx=0x7fcb2c00db80 comp rx=0 tx=0).stop 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 shutdown_connections 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb0c06bc50 0x7fcb0c06e110 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb34151f50 0x7fcb34142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb341583d0 0x7fcb3415ad50 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 --2- 172.21.15.43:0/956669503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb140039e0 0x7fcb34142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:25.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 >> 172.21.15.43:0/956669503 conn(0x7fcb340cec10 msgr2=0x7fcb340ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:25.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 shutdown_connections 2024-01-16T15:08:25.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:25.407+0000 7fcb39200640 1 -- 172.21.15.43:0/956669503 wait complete. 2024-01-16T15:08:25.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:25.403 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759) 2024-01-16T15:08:25.465 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:08:25.465 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:08:25.465 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:08:25.466 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:08:26.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:26.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:27.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:27.467 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759) 2024-01-16T15:08:27.536 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:08:27.536 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:08:27.536 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:08:27.537 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:08:29.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:29.538 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759) 2024-01-16T15:08:29.607 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/5b1a572d-a032-44e9-9215-b28c9cdac759 2024-01-16T15:08:29.607 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:08:29.607 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:08:29.609 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:08:29.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9611 2024-01-16T15:08:29.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.868+0000 7f4601c00640 1 Processor -- start 2024-01-16T15:08:29.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.868+0000 7f4601c00640 1 -- start start 2024-01-16T15:08:29.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc1443e0 0x7f45fc0721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45fc144920 0x7f45fc0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45fc0b56e0 0x7f45fc06a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45fc1447c0 con 0x7f45fc144920 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45fc0b5b70 con 0x7f45fc0b56e0 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f4601c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc072700 con 0x7f45fc1443e0 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f45fb400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45fc144920 0x7f45fc0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f45fb400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f45fc144920 0x7f45fc0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35910/0 (socket says 172.21.15.43:35910) 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.870+0000 7f45fb400640 1 -- 172.21.15.43:0/2184693920 learned_addr learned my addr 172.21.15.43:0/2184693920 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fbe00640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc1443e0 0x7f45fc0721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f4601200640 1 --2- 172.21.15.43:0/2184693920 >> v2:172.21.15.69:3300/0 conn(0x7f45fc0b56e0 0x7f45fc06a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fb400640 1 -- 172.21.15.43:0/2184693920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc1443e0 msgr2=0x7f45fc0721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:29.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fb400640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc1443e0 0x7f45fc0721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fb400640 1 -- 172.21.15.43:0/2184693920 >> v2:172.21.15.69:3300/0 conn(0x7f45fc0b56e0 msgr2=0x7f45fc06a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:29.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fb400640 1 --2- 172.21.15.43:0/2184693920 >> v2:172.21.15.69:3300/0 conn(0x7f45fc0b56e0 0x7f45fc06a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.871+0000 7f45fb400640 1 -- 172.21.15.43:0/2184693920 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc15fa00 con 0x7f45fc144920 2024-01-16T15:08:29.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.872+0000 7f45fb400640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc144920 0x7f45fc0b51a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0ce330 tx=0x7f45e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5250306dd3f7a4d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:29.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f45faa00640 1 -- 172.21.15.43:0/2184693920 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4031640 con 0x7f45fc144920 2024-01-16T15:08:29.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f45faa00640 1 -- 172.21.15.43:0/2184693920 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e40317a0 con 0x7f45fc144920 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f45faa00640 1 -- 172.21.15.43:0/2184693920 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4031b30 con 0x7f45fc144920 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 -- 172.21.15.43:0/2184693920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc144920 msgr2=0x7f45fc0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc144920 0x7f45fc0b51a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0ce330 tx=0x7f45e402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 -- 172.21.15.43:0/2184693920 shutdown_connections 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 --2- 172.21.15.43:0/2184693920 >> v2:172.21.15.69:3300/0 conn(0x7f45fc0b56e0 0x7f45fc06a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc144920 0x7f45fc0b51a0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 --2- 172.21.15.43:0/2184693920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc1443e0 0x7f45fc0721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.873+0000 7f4601c00640 1 -- 172.21.15.43:0/2184693920 >> 172.21.15.43:0/2184693920 conn(0x7f45fc0cfde0 msgr2=0x7f45fc0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.874+0000 7f4601c00640 1 -- 172.21.15.43:0/2184693920 shutdown_connections 2024-01-16T15:08:29.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.874+0000 7f4601c00640 1 -- 172.21.15.43:0/2184693920 wait complete. 2024-01-16T15:08:29.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.875+0000 7f4601c00640 1 Processor -- start 2024-01-16T15:08:29.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 -- start start 2024-01-16T15:08:29.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45fc0b56e0 0x7f45fc0b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 0x7f45fc159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0af360 con 0x7f45fc1443e0 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0b31c0 con 0x7f45fc0b56e0 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.876+0000 7f4601c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0aeb00 con 0x7f45fc144920 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35918/0 (socket says 172.21.15.43:35918) 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 -- 172.21.15.43:0/1972317765 learned_addr learned my addr 172.21.15.43:0/1972317765 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fbe00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45fc0b56e0 0x7f45fc0b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f4601200640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 0x7f45fc159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 -- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 msgr2=0x7f45fc159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 0x7f45fc159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 -- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45fc0b56e0 msgr2=0x7f45fc0b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:29.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45fc0b56e0 0x7f45fc0b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f45fb400640 1 -- 172.21.15.43:0/1972317765 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e40094d0 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.877+0000 7f4601200640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 0x7f45fc159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f45fb400640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e403ac60 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e403adc0 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4044630 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc0aeda0 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.878+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc0b4ba0 con 0x7f45fc1443e0 2024-01-16T15:08:29.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.880+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc0b2f90 con 0x7f45fc1443e0 2024-01-16T15:08:29.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.881+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f45e402f050 con 0x7f45fc1443e0 2024-01-16T15:08:29.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.881+0000 7f45f3e00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 0x7f45d406e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:29.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.881+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f45e40aa5a0 con 0x7f45fc1443e0 2024-01-16T15:08:29.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.882+0000 7f45fbe00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 0x7f45d406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:29.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.882+0000 7f45fbe00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 0x7f45d406e140 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006fd0 tx=0x7f45ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:29.886+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45fc0b2f90 con 0x7f45fc1443e0 2024-01-16T15:08:30.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.122+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9611", "target": ["mon-mgr", ""]}) v1 -- 0x7f45fc15f3b0 con 0x7f45d406bc80 2024-01-16T15:08:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.273+0000 7f45f3e00640 1 -- 172.21.15.43:0/1972317765 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45ec00b600 con 0x7f45d406bc80 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.276+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 msgr2=0x7f45d406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.276+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 0x7f45d406e140 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006fd0 tx=0x7f45ec008040 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.276+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 msgr2=0x7f45fc15b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.276+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e40046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 shutdown_connections 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f45d406bc80 0x7f45d406e140 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45fc144920 0x7f45fc159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45fc1443e0 0x7f45fc15b6f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 --2- 172.21.15.43:0/1972317765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45fc0b56e0 0x7f45fc0b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 >> 172.21.15.43:0/1972317765 conn(0x7f45fc0cfde0 msgr2=0x7f45fc13ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 shutdown_connections 2024-01-16T15:08:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.277+0000 7f4601c00640 1 -- 172.21.15.43:0/1972317765 wait complete. 2024-01-16T15:08:30.258 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.280+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.258 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.281+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.258 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.281+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.258 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.281+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.258 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.281+0000 7fd4ebe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9612 2024-01-16T15:08:30.286 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.309+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.286 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.309+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.286 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.309+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.287 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.309+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.287 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:30.309+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.497+0000 7fb43fe00640 1 Processor -- start 2024-01-16T15:08:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.498+0000 7fb43fe00640 1 -- start start 2024-01-16T15:08:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.499+0000 7fb43fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb44015d3b0 0x7fb4400ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.499+0000 7fb43fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4400b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.499+0000 7fb43fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4400c6bf0 con 0x7fb4400d0fa0 2024-01-16T15:08:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.499+0000 7fb43fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4400d1430 con 0x7fb44015d3b0 2024-01-16T15:08:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.499+0000 7fb43fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400cedc0 con 0x7fb4400d1d90 2024-01-16T15:08:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43e000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb4400d0fa0 0x7fb4400d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57608/0 (socket says 172.21.15.43:57608) 2024-01-16T15:08:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43e000640 1 -- 172.21.15.43:0/3250030236 learned_addr learned my addr 172.21.15.43:0/3250030236 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43ea00640 1 --2- 172.21.15.43:0/3250030236 >> v2:172.21.15.69:3300/0 conn(0x7fb44015d3b0 0x7fb4400ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43f400640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4400b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675670693 0 0) 0x7fb4400c6bf0 con 0x7fb4400d0fa0 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb420003520 con 0x7fb4400d0fa0 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790816892 0 0) 0x7fb420003520 con 0x7fb4400d0fa0 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 msgr2=0x7fb4400b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.500+0000 7fb43d600640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4400b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 >> v2:172.21.15.69:3300/0 conn(0x7fb44015d3b0 msgr2=0x7fb4400ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 --2- 172.21.15.43:0/3250030236 >> v2:172.21.15.69:3300/0 conn(0x7fb44015d3b0 0x7fb4400ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4400c6e40 con 0x7fb4400d0fa0 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb4280019f0 con 0x7fb4400d0fa0 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 >> v1:172.21.15.43:6789/0 conn(0x7fb4400d0fa0 legacy=0x7fb4400d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb420003f60 0x7fb4200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420004d00 0x7fb420005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400d1430 con 0x7fb420003f60 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400cedc0 con 0x7fb420004d00 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42000ba30 con 0x7fb4400d1d90 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43e000640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb420003f60 0x7fb4200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43ea00640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43e000640 1 -- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 msgr2=0x7fb4200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.501+0000 7fb43e000640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4400d1d90 0x7fb4200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.502+0000 7fb43f400640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420004d00 0x7fb420005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.502+0000 7fb43e000640 1 -- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420004d00 msgr2=0x7fb420005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.502+0000 7fb43e000640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420004d00 0x7fb420005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.502+0000 7fb43e000640 1 -- 172.21.15.43:0/3250030236 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420003520 con 0x7fb420003f60 2024-01-16T15:08:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.503+0000 7fb43e000640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb420003f60 0x7fb4200047c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb42800a860 tx=0x7fb4200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5210c9f8519eb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.503+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428034630 con 0x7fb420003f60 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.503+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42802f070 con 0x7fb420003f60 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.503+0000 7fb43d600640 1 -- 172.21.15.43:0/3250030236 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428034a30 con 0x7fb420003f60 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 -- 172.21.15.43:0/3250030236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb420003f60 msgr2=0x7fb4200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 --2- 172.21.15.43:0/3250030236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb420003f60 0x7fb4200047c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb42800a860 tx=0x7fb4200057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43f400640 1 -- 172.21.15.43:0/3250030236 reap_dead start 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 -- 172.21.15.43:0/3250030236 shutdown_connections 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 -- 172.21.15.43:0/3250030236 >> 172.21.15.43:0/3250030236 conn(0x7fb4400d30a0 msgr2=0x7fb4400d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 -- 172.21.15.43:0/3250030236 shutdown_connections 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.504+0000 7fb43fe00640 1 -- 172.21.15.43:0/3250030236 wait complete. 2024-01-16T15:08:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.506+0000 7fb43fe00640 1 Processor -- start 2024-01-16T15:08:30.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.506+0000 7fb43fe00640 1 -- start start 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420003f60 0x7fb4400c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4401f63f0 0x7fb4401f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400b3060 con 0x7fb4401f6fb0 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401f6dc0 con 0x7fb420003f60 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400b21b0 con 0x7fb4401f63f0 2024-01-16T15:08:30.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35980/0 (socket says 172.21.15.43:35980) 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43e000640 1 -- 172.21.15.43:0/2801867729 learned_addr learned my addr 172.21.15.43:0/2801867729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43f400640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4401f63f0 0x7fb4401f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43ea00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420003f60 0x7fb4400c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.507+0000 7fb43e000640 1 -- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4401f63f0 msgr2=0x7fb4401f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43e000640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4401f63f0 0x7fb4401f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43e000640 1 -- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420003f60 msgr2=0x7fb4400c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43e000640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420003f60 0x7fb4400c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43e000640 1 -- 172.21.15.43:0/2801867729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4400b23e0 con 0x7fb4401f6fb0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43e000640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb42800a860 tx=0x7fb428001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428034840 con 0x7fb4401f6fb0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4280349a0 con 0x7fb4401f6fb0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428034d30 con 0x7fb4401f6fb0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4400b26c0 con 0x7fb4401f6fb0 2024-01-16T15:08:30.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.508+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4400b2c80 con 0x7fb4401f6fb0 2024-01-16T15:08:30.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.510+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44004e840 con 0x7fb4401f6fb0 2024-01-16T15:08:30.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.511+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fb428031030 con 0x7fb4401f6fb0 2024-01-16T15:08:30.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.512+0000 7fb437400640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 0x7fb41806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:30.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.512+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb4280aab00 con 0x7fb4401f6fb0 2024-01-16T15:08:30.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.513+0000 7fb43ea00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 0x7fb41806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:30.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.513+0000 7fb43ea00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 0x7fb41806e110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb4300099d0 tx=0x7fb430008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:30.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.517+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb428077220 con 0x7fb4401f6fb0 2024-01-16T15:08:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.751+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9612", "target": ["mon-mgr", ""]}) v1 -- 0x7fb44015d790 con 0x7fb41806bc50 2024-01-16T15:08:30.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.921+0000 7fb437400640 1 -- 172.21.15.43:0/2801867729 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb430009620 con 0x7fb41806bc50 2024-01-16T15:08:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 msgr2=0x7fb41806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 0x7fb41806e110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb4300099d0 tx=0x7fb430008040 comp rx=0 tx=0).stop 2024-01-16T15:08:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 msgr2=0x7fb4400afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb42800a860 tx=0x7fb428001d80 comp rx=0 tx=0).stop 2024-01-16T15:08:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 shutdown_connections 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb41806bc50 0x7fb41806e110 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4401f63f0 0x7fb4401f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4401f6fb0 0x7fb4400afc60 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 --2- 172.21.15.43:0/2801867729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb420003f60 0x7fb4400c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 >> 172.21.15.43:0/2801867729 conn(0x7fb4400d30a0 msgr2=0x7fb4400cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 shutdown_connections 2024-01-16T15:08:30.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:30.924+0000 7fb43fe00640 1 -- 172.21.15.43:0/2801867729 wait complete. 2024-01-16T15:08:30.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9613 2024-01-16T15:08:31.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.217+0000 7f9329200640 1 Processor -- start 2024-01-16T15:08:31.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.218+0000 7f9329200640 1 -- start start 2024-01-16T15:08:31.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.219+0000 7f9329200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93240acd70 0x7f932406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.219+0000 7f9329200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93240ae490 0x7f93240ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.219+0000 7f9329200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932406d7a0 con 0x7f93240ad690 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.219+0000 7f9329200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932415a5b0 con 0x7f93240ae490 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.219+0000 7f9329200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932406ce70 con 0x7f93240acd70 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93240ae490 0x7f93240ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93240ae490 0x7f93240ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35992/0 (socket says 172.21.15.43:35992) 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 -- 172.21.15.43:0/1085349369 learned_addr learned my addr 172.21.15.43:0/1085349369 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 -- 172.21.15.43:0/1085349369 >> v2:172.21.15.69:3300/0 conn(0x7f93240acd70 msgr2=0x7f932406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 --2- 172.21.15.43:0/1085349369 >> v2:172.21.15.69:3300/0 conn(0x7f93240acd70 0x7f932406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 -- 172.21.15.43:0/1085349369 >> v1:172.21.15.69:6789/0 conn(0x7f93240ad690 legacy=0x7f9324070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:31.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.220+0000 7f9321400640 1 -- 172.21.15.43:0/1085349369 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932406d150 con 0x7f93240ae490 2024-01-16T15:08:31.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.221+0000 7f9321400640 1 --2- 172.21.15.43:0/1085349369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240ae490 0x7f93240ad150 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f93240700f0 tx=0x7f930c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86461f94b27073b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:31.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9320a00640 1 -- 172.21.15.43:0/1085349369 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c031870 con 0x7f93240ae490 2024-01-16T15:08:31.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9320a00640 1 -- 172.21.15.43:0/1085349369 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c0319d0 con 0x7f93240ae490 2024-01-16T15:08:31.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9320a00640 1 -- 172.21.15.43:0/1085349369 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c031d60 con 0x7f93240ae490 2024-01-16T15:08:31.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 -- 172.21.15.43:0/1085349369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240ae490 msgr2=0x7f93240ad150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:31.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 --2- 172.21.15.43:0/1085349369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240ae490 0x7f93240ad150 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f93240700f0 tx=0x7f930c02b910 comp rx=0 tx=0).stop 2024-01-16T15:08:31.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 -- 172.21.15.43:0/1085349369 shutdown_connections 2024-01-16T15:08:31.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 --2- 172.21.15.43:0/1085349369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240ae490 0x7f93240ad150 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:31.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 --2- 172.21.15.43:0/1085349369 >> v2:172.21.15.69:3300/0 conn(0x7f93240acd70 0x7f932406dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:31.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.222+0000 7f9329200640 1 -- 172.21.15.43:0/1085349369 >> 172.21.15.43:0/1085349369 conn(0x7f93240b0ba0 msgr2=0x7f93240b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:31.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.223+0000 7f9329200640 1 -- 172.21.15.43:0/1085349369 shutdown_connections 2024-01-16T15:08:31.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.223+0000 7f9329200640 1 -- 172.21.15.43:0/1085349369 wait complete. 2024-01-16T15:08:31.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.224+0000 7f9329200640 1 Processor -- start 2024-01-16T15:08:31.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.225+0000 7f9329200640 1 -- start start 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.225+0000 7f9329200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9329200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 0x7f9324158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9329200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93240ae490 0x7f9324154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9329200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240cc8e0 con 0x7f93240acd70 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9329200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93240cca50 con 0x7f93240ad690 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9329200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932415d500 con 0x7f93240ae490 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35996/0 (socket says 172.21.15.43:35996) 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 -- 172.21.15.43:0/742519243 learned_addr learned my addr 172.21.15.43:0/742519243 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:31.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321400640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 0x7f9324158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 -- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93240ae490 msgr2=0x7f9324154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93240ae490 0x7f9324154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 -- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 msgr2=0x7f9324158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 0x7f9324158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321e00640 1 -- 172.21.15.43:0/742519243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0094d0 con 0x7f93240acd70 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.226+0000 7f9321400640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 0x7f9324158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f9321e00640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f931400b440 tx=0x7f931400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314011890 con 0x7f93240acd70 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93140119f0 con 0x7f93240acd70 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314011da0 con 0x7f93240acd70 2024-01-16T15:08:31.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9324159050 con 0x7f93240acd70 2024-01-16T15:08:31.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.227+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9324159610 con 0x7f93240acd70 2024-01-16T15:08:31.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.229+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932409d200 con 0x7f93240acd70 2024-01-16T15:08:31.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.229+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f931401a400 con 0x7f93240acd70 2024-01-16T15:08:31.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.230+0000 7f931a000640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 0x7f92fc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:31.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.230+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9314088a10 con 0x7f93240acd70 2024-01-16T15:08:31.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.231+0000 7f9321400640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 0x7f92fc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:31.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.231+0000 7f9321400640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 0x7f92fc06e110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f930c036000 tx=0x7f930c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:31.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.235+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93140550d0 con 0x7f93240acd70 2024-01-16T15:08:31.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.423+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f93140554a0 con 0x7f93240acd70 2024-01-16T15:08:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:31.470+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9613", "target": ["mon-mgr", ""]}) v1 -- 0x7f93240b3d90 con 0x7f92fc06bc50 2024-01-16T15:08:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.818+0000 7f931a000640 1 -- 172.21.15.43:0/742519243 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f930c031dd0 con 0x7f92fc06bc50 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 msgr2=0x7f92fc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 0x7f92fc06e110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f930c036000 tx=0x7f930c03c040 comp rx=0 tx=0).stop 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 msgr2=0x7f9324153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f931400b440 tx=0x7f931400b900 comp rx=0 tx=0).stop 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 shutdown_connections 2024-01-16T15:08:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92fc06bc50 0x7f92fc06e110 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93240ad690 0x7f9324158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.822+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93240ae490 0x7f9324154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.823+0000 7f9329200640 1 --2- 172.21.15.43:0/742519243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93240acd70 0x7f9324153d00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.823+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 >> 172.21.15.43:0/742519243 conn(0x7f93240b0ba0 msgr2=0x7f9324155b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.823+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 shutdown_connections 2024-01-16T15:08:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:33.823+0000 7f9329200640 1 -- 172.21.15.43:0/742519243 wait complete. 2024-01-16T15:08:33.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9614 2024-01-16T15:08:34.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.068+0000 7f5bdf000640 1 Processor -- start 2024-01-16T15:08:34.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.068+0000 7f5bdf000640 1 -- start start 2024-01-16T15:08:34.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdf000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5bd8156eb0 0x7f5bd80b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdf000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5bd8157290 0x7f5bd806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdf000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd80b2bf0 con 0x7f5bd815a840 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdf000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd8071000 con 0x7f5bd8156eb0 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdf000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd80b2630 con 0x7f5bd8157290 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdd200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5bd815a840 0x7f5bd80708f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56092/0 (socket says 172.21.15.43:56092) 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bddc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5bd8156eb0 0x7f5bd80b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bdd200640 1 -- 172.21.15.43:0/1552715575 learned_addr learned my addr 172.21.15.43:0/1552715575 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.070+0000 7f5bde600640 1 --2- 172.21.15.43:0/1552715575 >> v2:172.21.15.69:3300/0 conn(0x7f5bd8157290 0x7f5bd806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.071+0000 7f5bddc00640 1 -- 172.21.15.43:0/1552715575 >> v2:172.21.15.69:3300/0 conn(0x7f5bd8157290 msgr2=0x7f5bd806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.071+0000 7f5bddc00640 1 --2- 172.21.15.43:0/1552715575 >> v2:172.21.15.69:3300/0 conn(0x7f5bd8157290 0x7f5bd806cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.071+0000 7f5bddc00640 1 -- 172.21.15.43:0/1552715575 >> v1:172.21.15.43:6789/0 conn(0x7f5bd815a840 legacy=0x7f5bd80708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.071+0000 7f5bddc00640 1 -- 172.21.15.43:0/1552715575 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd80b2910 con 0x7f5bd8156eb0 2024-01-16T15:08:34.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.071+0000 7f5bd7e00640 1 -- 172.21.15.43:0/1552715575 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446983924 0 0) 0x7f5bd80b2bf0 con 0x7f5bd815a840 2024-01-16T15:08:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.072+0000 7f5bddc00640 1 --2- 172.21.15.43:0/1552715575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8156eb0 0x7f5bd80b1cc0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80b4bc0 tx=0x7f5bc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acc969f703785bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.072+0000 7f5bd7e00640 1 -- 172.21.15.43:0/1552715575 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8031830 con 0x7f5bd8156eb0 2024-01-16T15:08:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.072+0000 7f5bd7e00640 1 -- 172.21.15.43:0/1552715575 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8031990 con 0x7f5bd8156eb0 2024-01-16T15:08:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.072+0000 7f5bd7e00640 1 -- 172.21.15.43:0/1552715575 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8031d20 con 0x7f5bd8156eb0 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 -- 172.21.15.43:0/1552715575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8156eb0 msgr2=0x7f5bd80b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 --2- 172.21.15.43:0/1552715575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8156eb0 0x7f5bd80b1cc0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80b4bc0 tx=0x7f5bc802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 -- 172.21.15.43:0/1552715575 shutdown_connections 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 --2- 172.21.15.43:0/1552715575 >> v2:172.21.15.69:3300/0 conn(0x7f5bd8157290 0x7f5bd806cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 --2- 172.21.15.43:0/1552715575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8156eb0 0x7f5bd80b1cc0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 -- 172.21.15.43:0/1552715575 >> 172.21.15.43:0/1552715575 conn(0x7f5bd80cec10 msgr2=0x7f5bd80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 -- 172.21.15.43:0/1552715575 shutdown_connections 2024-01-16T15:08:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.073+0000 7f5bdf000640 1 -- 172.21.15.43:0/1552715575 wait complete. 2024-01-16T15:08:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.075+0000 7f5bdf000640 1 Processor -- start 2024-01-16T15:08:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.075+0000 7f5bdf000640 1 -- start start 2024-01-16T15:08:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd8156eb0 0x7f5bd814c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 0x7f5bd814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd8148c00 con 0x7f5bd8157290 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd814a610 con 0x7f5bd8156eb0 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdf000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd814a8f0 con 0x7f5bd815a840 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42830/0 (socket says 172.21.15.43:42830) 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bdd200640 1 -- 172.21.15.43:0/2176394665 learned_addr learned my addr 172.21.15.43:0/2176394665 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.076+0000 7f5bddc00640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd8156eb0 0x7f5bd814c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bde600640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 0x7f5bd814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 -- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 msgr2=0x7f5bd814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 0x7f5bd814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 -- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd8156eb0 msgr2=0x7f5bd814c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd8156eb0 0x7f5bd814c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 -- 172.21.15.43:0/2176394665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8009400 con 0x7f5bd8157290 2024-01-16T15:08:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bde600640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 0x7f5bd814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdd200640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c430 tx=0x7f5bc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00074e0 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0007660 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0004a20 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.077+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd814abd0 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.078+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd8146a30 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.079+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd814adc0 con 0x7f5bd8157290 2024-01-16T15:08:34.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.080+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f5bc0004b80 con 0x7f5bd8157290 2024-01-16T15:08:34.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.080+0000 7f5bd5600640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 0x7f5bb006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:34.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.081+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5bc0088540 con 0x7f5bd8157290 2024-01-16T15:08:34.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.081+0000 7f5bddc00640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 0x7f5bb006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.082+0000 7f5bddc00640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 0x7f5bb006e230 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:34.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.085+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bc0054c30 con 0x7f5bd8157290 2024-01-16T15:08:34.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:34.317+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bd8146cb0 con 0x7f5bb006bd70 2024-01-16T15:08:38.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.820+0000 7f5bd5600640 1 -- 172.21.15.43:0/2176394665 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5bd8146cb0 con 0x7f5bb006bd70 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 msgr2=0x7f5bb006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 0x7f5bb006e230 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc803c040 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 msgr2=0x7f5bd81464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c430 tx=0x7f5bc000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 shutdown_connections 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5bb006bd70 0x7f5bb006e230 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bd815a840 0x7f5bd814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bd8157290 0x7f5bd81464f0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 --2- 172.21.15.43:0/2176394665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bd8156eb0 0x7f5bd814c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 >> 172.21.15.43:0/2176394665 conn(0x7f5bd80cec10 msgr2=0x7f5bd80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:38.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.824+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 shutdown_connections 2024-01-16T15:08:38.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:38.825+0000 7f5bdf000640 1 -- 172.21.15.43:0/2176394665 wait complete. 2024-01-16T15:08:38.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9615 2024-01-16T15:08:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.072+0000 7f4c4ca00640 1 Processor -- start 2024-01-16T15:08:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.073+0000 7f4c4ca00640 1 -- start start 2024-01-16T15:08:39.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.074+0000 7f4c4ca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c481435d0 0x7f4c48071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.074+0000 7f4c4ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c481439b0 0x7f4c480b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.074+0000 7f4c4ca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c48070d40 con 0x7f4c480b3f70 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.074+0000 7f4c4ca00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c480b43e0 con 0x7f4c481435d0 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.074+0000 7f4c4ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c48071020 con 0x7f4c481439b0 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.075+0000 7f4c46800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c481435d0 0x7f4c48071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.075+0000 7f4c46800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c481435d0 0x7f4c48071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42834/0 (socket says 172.21.15.43:42834) 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.075+0000 7f4c46800640 1 -- 172.21.15.43:0/3552906985 learned_addr learned my addr 172.21.15.43:0/3552906985 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.075+0000 7f4c47200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4c480b3f70 0x7f4c48070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56102/0 (socket says 172.21.15.43:56102) 2024-01-16T15:08:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.075+0000 7f4c3fe00640 1 --2- 172.21.15.43:0/3552906985 >> v2:172.21.15.69:3300/0 conn(0x7f4c481439b0 0x7f4c480b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c45e00640 1 -- 172.21.15.43:0/3552906985 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990976710 0 0) 0x7f4c48070d40 con 0x7f4c480b3f70 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c45e00640 1 -- 172.21.15.43:0/3552906985 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c28003520 con 0x7f4c480b3f70 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c46800640 1 -- 172.21.15.43:0/3552906985 >> v2:172.21.15.69:3300/0 conn(0x7f4c481439b0 msgr2=0x7f4c480b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c46800640 1 --2- 172.21.15.43:0/3552906985 >> v2:172.21.15.69:3300/0 conn(0x7f4c481439b0 0x7f4c480b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c46800640 1 -- 172.21.15.43:0/3552906985 >> v1:172.21.15.43:6789/0 conn(0x7f4c480b3f70 legacy=0x7f4c48070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:39.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.076+0000 7f4c46800640 1 -- 172.21.15.43:0/3552906985 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48071300 con 0x7f4c481435d0 2024-01-16T15:08:39.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.077+0000 7f4c46800640 1 --2- 172.21.15.43:0/3552906985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481435d0 0x7f4c48071de0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4c480cbe80 tx=0x7f4c3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44aaabff2f8f556 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:39.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.078+0000 7f4c45e00640 1 -- 172.21.15.43:0/3552906985 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38031870 con 0x7f4c481435d0 2024-01-16T15:08:39.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.078+0000 7f4c45e00640 1 -- 172.21.15.43:0/3552906985 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c380319d0 con 0x7f4c481435d0 2024-01-16T15:08:39.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.078+0000 7f4c45e00640 1 -- 172.21.15.43:0/3552906985 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38031d60 con 0x7f4c481435d0 2024-01-16T15:08:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 -- 172.21.15.43:0/3552906985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481435d0 msgr2=0x7f4c48071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/3552906985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481435d0 0x7f4c48071de0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4c480cbe80 tx=0x7f4c3802b910 comp rx=0 tx=0).stop 2024-01-16T15:08:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 -- 172.21.15.43:0/3552906985 shutdown_connections 2024-01-16T15:08:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/3552906985 >> v2:172.21.15.69:3300/0 conn(0x7f4c481439b0 0x7f4c480b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/3552906985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481435d0 0x7f4c48071de0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 -- 172.21.15.43:0/3552906985 >> 172.21.15.43:0/3552906985 conn(0x7f4c480cec10 msgr2=0x7f4c480cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 -- 172.21.15.43:0/3552906985 shutdown_connections 2024-01-16T15:08:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.079+0000 7f4c4ca00640 1 -- 172.21.15.43:0/3552906985 wait complete. 2024-01-16T15:08:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.081+0000 7f4c4ca00640 1 Processor -- start 2024-01-16T15:08:39.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.081+0000 7f4c4ca00640 1 -- start start 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c480b3f70 0x7f4c480abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c481435d0 0x7f4c48164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c481652e0 con 0x7f4c481439b0 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c480ade00 con 0x7f4c480b3f70 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c4ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c480ae0e0 con 0x7f4c481435d0 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c47200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c47200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42848/0 (socket says 172.21.15.43:42848) 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.083+0000 7f4c47200640 1 -- 172.21.15.43:0/2554089904 learned_addr learned my addr 172.21.15.43:0/2554089904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c3fe00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c481435d0 0x7f4c48164da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c46800640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c480b3f70 0x7f4c480abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 -- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c481435d0 msgr2=0x7f4c48164da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c481435d0 0x7f4c48164da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 -- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c480b3f70 msgr2=0x7f4c480abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c480b3f70 0x7f4c480abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 -- 172.21.15.43:0/2554089904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380094d0 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c47200640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c7c0 tx=0x7f4c4000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000a640 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4000a7a0 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c480ab0e0 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.084+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000ab30 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.085+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c480ab6a0 con 0x7f4c481439b0 2024-01-16T15:08:39.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.086+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c480ab890 con 0x7f4c481439b0 2024-01-16T15:08:39.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.087+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f4c4000ac90 con 0x7f4c481439b0 2024-01-16T15:08:39.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.088+0000 7f4c3ea00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 0x7f4c2006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:39.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.088+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4c40088090 con 0x7f4c481439b0 2024-01-16T15:08:39.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.088+0000 7f4c46800640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 0x7f4c2006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:39.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.089+0000 7f4c46800640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 0x7f4c2006e230 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4c38036000 tx=0x7f4c3803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:39.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.093+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c40054720 con 0x7f4c481439b0 2024-01-16T15:08:39.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:39.324+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9615", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c480718c0 con 0x7f4c2006bd70 2024-01-16T15:08:43.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.812+0000 7f4c3ea00640 1 -- 172.21.15.43:0/2554089904 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4c3803a420 con 0x7f4c2006bd70 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 msgr2=0x7f4c2006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 0x7f4c2006e230 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4c38036000 tx=0x7f4c3803c040 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 msgr2=0x7f4c480ad120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c7c0 tx=0x7f4c4000cc80 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 shutdown_connections 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c2006bd70 0x7f4c2006e230 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.815+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c481439b0 0x7f4c480ad120 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.816+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c481435d0 0x7f4c48164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.816+0000 7f4c4ca00640 1 --2- 172.21.15.43:0/2554089904 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c480b3f70 0x7f4c480abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.816+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 >> 172.21.15.43:0/2554089904 conn(0x7f4c480cec10 msgr2=0x7f4c480b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.816+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 shutdown_connections 2024-01-16T15:08:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:43.816+0000 7f4c4ca00640 1 -- 172.21.15.43:0/2554089904 wait complete. 2024-01-16T15:08:43.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9616 2024-01-16T15:08:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.076+0000 7fb8b8c00640 1 Processor -- start 2024-01-16T15:08:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.076+0000 7fb8b8c00640 1 -- start start 2024-01-16T15:08:44.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8b4161b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8b8c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b40b3ff0 con 0x7fb8b40b2720 2024-01-16T15:08:44.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8b8c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b40b3270 con 0x7fb8b406bb60 2024-01-16T15:08:44.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8b8c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40b3550 con 0x7fb8b415d3c0 2024-01-16T15:08:44.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8a9200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb8b40b2720 0x7fb8b406b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40630/0 (socket says 172.21.15.43:40630) 2024-01-16T15:08:44.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.078+0000 7fb8a9200640 1 -- 172.21.15.43:0/2912534055 learned_addr learned my addr 172.21.15.43:0/2912534055 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:44.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884196283 0 0) 0x7fb8b40b3ff0 con 0x7fb8b40b2720 2024-01-16T15:08:44.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8b4161b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894003520 con 0x7fb8b40b2720 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495211895 0 0) 0x7fb894003520 con 0x7fb8b40b2720 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 msgr2=0x7fb8b4161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8b4161b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 >> v1:172.21.15.69:6789/0 conn(0x7fb8b406bb60 legacy=0x7fb8b406cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b41610a0 con 0x7fb8b40b2720 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb89c001670 con 0x7fb8b40b2720 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 >> v1:172.21.15.43:6789/0 conn(0x7fb8b40b2720 legacy=0x7fb8b406b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8940039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894003fd0 0x7fb8940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb894004d00 0x7fb894005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40b3550 con 0x7fb894004d00 2024-01-16T15:08:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400bb40 con 0x7fb894003fd0 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.079+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400bd00 con 0x7fb8b415d3c0 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb894004d00 0x7fb894005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8940039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8a9200640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894003fd0 0x7fb8940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 -- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 msgr2=0x7fb8940039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b415d3c0 0x7fb8940039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 -- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894003fd0 msgr2=0x7fb8940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894003fd0 0x7fb8940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.080+0000 7fb8b2800640 1 -- 172.21.15.43:0/2912534055 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894003520 con 0x7fb894004d00 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.081+0000 7fb8b2800640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb894004d00 0x7fb894005180 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac001720 tx=0x7fb8940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e75768569b2cd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.081+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0052d0 con 0x7fb894004d00 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.081+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac02e050 con 0x7fb894004d00 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.081+0000 7fb8b1400640 1 -- 172.21.15.43:0/2912534055 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003830 con 0x7fb894004d00 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 -- 172.21.15.43:0/2912534055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb894004d00 msgr2=0x7fb894005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/2912534055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb894004d00 0x7fb894005180 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac001720 tx=0x7fb8940057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b2800640 1 -- 172.21.15.43:0/2912534055 reap_dead start 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 -- 172.21.15.43:0/2912534055 shutdown_connections 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 -- 172.21.15.43:0/2912534055 >> 172.21.15.43:0/2912534055 conn(0x7fb8b40cfd60 msgr2=0x7fb8b4070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 -- 172.21.15.43:0/2912534055 shutdown_connections 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.082+0000 7fb8b8c00640 1 -- 172.21.15.43:0/2912534055 wait complete. 2024-01-16T15:08:44.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.084+0000 7fb8b8c00640 1 Processor -- start 2024-01-16T15:08:44.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.084+0000 7fb8b8c00640 1 -- start start 2024-01-16T15:08:44.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 0x7fb8b41553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b4144a80 0x7fb8b4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4158de0 con 0x7fb8b406e760 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b406f150 con 0x7fb894004d00 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.085+0000 7fb8b8c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4155930 con 0x7fb8b4144a80 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60190/0 (socket says 172.21.15.43:60190) 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 -- 172.21.15.43:0/3122027017 learned_addr learned my addr 172.21.15.43:0/3122027017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 0x7fb8b41553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8a9200640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b4144a80 0x7fb8b4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 -- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b4144a80 msgr2=0x7fb8b4157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b4144a80 0x7fb8b4157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 -- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 msgr2=0x7fb8b41553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 0x7fb8b41553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 -- 172.21.15.43:0/3122027017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4158160 con 0x7fb8b406e760 2024-01-16T15:08:44.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 0x7fb8b41553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:44.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.086+0000 7fb8b2800640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005040 tx=0x7fb8ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.087+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac007a30 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.087+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac0067d0 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.087+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac006b80 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.087+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b4158440 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.087+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b4158a00 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.088+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b4158f20 con 0x7fb8b406e760 2024-01-16T15:08:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.089+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fb8ac006ce0 con 0x7fb8b406e760 2024-01-16T15:08:44.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.090+0000 7fb8aaa00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 0x7fb88c06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.091+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb8ac0a9f20 con 0x7fb8b406e760 2024-01-16T15:08:44.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.091+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 0x7fb88c06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.091+0000 7fb8b1e00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 0x7fb88c06e260 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4004310 tx=0x7fb8a400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.095+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8ac0767e0 con 0x7fb8b406e760 2024-01-16T15:08:44.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.323+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8b4161660 con 0x7fb88c06bda0 2024-01-16T15:08:44.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.482+0000 7fb8aaa00640 1 -- 172.21.15.43:0/3122027017 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb8a4004890 con 0x7fb88c06bda0 2024-01-16T15:08:44.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 msgr2=0x7fb88c06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 0x7fb88c06e260 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4004310 tx=0x7fb8a400b040 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 msgr2=0x7fb8b406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005040 tx=0x7fb8ac001120 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 shutdown_connections 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb88c06bda0 0x7fb88c06e260 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8b406e760 0x7fb8b406ec10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8b4144a80 0x7fb8b4157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 --2- 172.21.15.43:0/3122027017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb894004d00 0x7fb8b41553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 >> 172.21.15.43:0/3122027017 conn(0x7fb8b40cfd60 msgr2=0x7fb8b415fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 shutdown_connections 2024-01-16T15:08:44.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.486+0000 7fb8b8c00640 1 -- 172.21.15.43:0/3122027017 wait complete. 2024-01-16T15:08:44.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9617 2024-01-16T15:08:44.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.732+0000 7f317ca00640 1 Processor -- start 2024-01-16T15:08:44.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.733+0000 7f317ca00640 1 -- start start 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f317ca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31780d1d90 0x7f31780ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f317ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31780d0fa0 0x7f31780b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f317ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780c6bf0 con 0x7f317815d3b0 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f317ca00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31780d13f0 con 0x7f31780d1d90 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f317ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31780cedc0 con 0x7f31780d0fa0 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.734+0000 7f3176a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31780d1d90 0x7f31780ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f31780d1d90 0x7f31780ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60206/0 (socket says 172.21.15.43:60206) 2024-01-16T15:08:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 -- 172.21.15.43:0/1320414750 learned_addr learned my addr 172.21.15.43:0/1320414750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 -- 172.21.15.43:0/1320414750 >> v2:172.21.15.69:3300/0 conn(0x7f31780d0fa0 msgr2=0x7f31780b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 --2- 172.21.15.43:0/1320414750 >> v2:172.21.15.69:3300/0 conn(0x7f31780d0fa0 0x7f31780b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 -- 172.21.15.43:0/1320414750 >> v1:172.21.15.69:6789/0 conn(0x7f317815d3b0 legacy=0x7f31780d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.735+0000 7f3176a00640 1 -- 172.21.15.43:0/1320414750 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780c6e40 con 0x7f31780d1d90 2024-01-16T15:08:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.736+0000 7f3176a00640 1 --2- 172.21.15.43:0/1320414750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31780d1d90 0x7f31780ce6c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f3178072b10 tx=0x7f316002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5df08b733f1334a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f3176000640 1 -- 172.21.15.43:0/1320414750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600317e0 con 0x7f31780d1d90 2024-01-16T15:08:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f3176000640 1 -- 172.21.15.43:0/1320414750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160031940 con 0x7f31780d1d90 2024-01-16T15:08:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f3176000640 1 -- 172.21.15.43:0/1320414750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160031cd0 con 0x7f31780d1d90 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 -- 172.21.15.43:0/1320414750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31780d1d90 msgr2=0x7f31780ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 --2- 172.21.15.43:0/1320414750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31780d1d90 0x7f31780ce6c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f3178072b10 tx=0x7f316002b880 comp rx=0 tx=0).stop 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 -- 172.21.15.43:0/1320414750 shutdown_connections 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 --2- 172.21.15.43:0/1320414750 >> v2:172.21.15.69:3300/0 conn(0x7f31780d0fa0 0x7f31780b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 --2- 172.21.15.43:0/1320414750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31780d1d90 0x7f31780ce6c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 -- 172.21.15.43:0/1320414750 >> 172.21.15.43:0/1320414750 conn(0x7f31780d30a0 msgr2=0x7f31780d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 -- 172.21.15.43:0/1320414750 shutdown_connections 2024-01-16T15:08:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.737+0000 7f317ca00640 1 -- 172.21.15.43:0/1320414750 wait complete. 2024-01-16T15:08:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.739+0000 7f317ca00640 1 Processor -- start 2024-01-16T15:08:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.739+0000 7f317ca00640 1 -- start start 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 0x7f31781f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 0x7f317809cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317809f8f0 con 0x7f317815d3b0 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780ccba0 con 0x7f31780d0fa0 2024-01-16T15:08:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.740+0000 7f317ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317809ecb0 con 0x7f31780d1d90 2024-01-16T15:08:44.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60222/0 (socket says 172.21.15.43:60222) 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 -- 172.21.15.43:0/581419223 learned_addr learned my addr 172.21.15.43:0/581419223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177400640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 0x7f31781f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3176a00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 0x7f317809cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 -- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 msgr2=0x7f317809cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 0x7f317809cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 -- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 msgr2=0x7f31781f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 0x7f31781f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177e00640 1 -- 172.21.15.43:0/581419223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600094d0 con 0x7f317815d3b0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3176a00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 0x7f317809cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.741+0000 7f3177400640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 0x7f31781f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f3177e00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f317000b450 tx=0x7f317000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170011890 con 0x7f317815d3b0 2024-01-16T15:08:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31700119f0 con 0x7f317815d3b0 2024-01-16T15:08:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170011d80 con 0x7f317815d3b0 2024-01-16T15:08:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317809ef50 con 0x7f317815d3b0 2024-01-16T15:08:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.742+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317809f510 con 0x7f317815d3b0 2024-01-16T15:08:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.743+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31780a0e50 con 0x7f317815d3b0 2024-01-16T15:08:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.744+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f317001a400 con 0x7f317815d3b0 2024-01-16T15:08:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.745+0000 7f316f400640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 0x7f315006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.745+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3170088900 con 0x7f317815d3b0 2024-01-16T15:08:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.745+0000 7f3177400640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 0x7f315006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.746+0000 7f3177400640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 0x7f315006e230 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3168007eb0 tx=0x7f31680073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.750+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3170055270 con 0x7f317815d3b0 2024-01-16T15:08:44.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:44.979+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9617", "target": ["mon-mgr", ""]}) v1 -- 0x7f31780ce6c0 con 0x7f315006bd70 2024-01-16T15:08:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.144+0000 7f316f400640 1 -- 172.21.15.43:0/581419223 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31780ce6c0 con 0x7f315006bd70 2024-01-16T15:08:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.147+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 msgr2=0x7f315006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.147+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 0x7f315006e230 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3168007eb0 tx=0x7f31680073c0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 msgr2=0x7f317809d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f317000b450 tx=0x7f317000b910 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 shutdown_connections 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f315006bd70 0x7f315006e230 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f317815d3b0 0x7f317809d320 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31780d1d90 0x7f317809cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 --2- 172.21.15.43:0/581419223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0fa0 0x7f31781f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 >> 172.21.15.43:0/581419223 conn(0x7f31780d30a0 msgr2=0x7f317815f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 shutdown_connections 2024-01-16T15:08:45.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.148+0000 7f317ca00640 1 -- 172.21.15.43:0/581419223 wait complete. 2024-01-16T15:08:45.147 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:08:45.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:08:45.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.394+0000 7ff107200640 1 Processor -- start 2024-01-16T15:08:45.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.394+0000 7ff107200640 1 -- start start 2024-01-16T15:08:45.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff107200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff1000d1fd0 0x7ff1000ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff107200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1000b2980 con 0x7ff1000cfa30 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff107200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1000cf220 con 0x7ff1000b33d0 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff107200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000cf500 con 0x7ff1000d1fd0 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff105e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff1000cfa30 0x7ff1000cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40634/0 (socket says 172.21.15.43:40634) 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff105e00640 1 -- 172.21.15.43:0/1093607347 learned_addr learned my addr 172.21.15.43:0/1093607347 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.396+0000 7ff105400640 1 --2- 172.21.15.43:0/1093607347 >> v2:172.21.15.43:3300/0 conn(0x7ff1000d1fd0 0x7ff1000ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.397+0000 7ff104a00640 1 -- 172.21.15.43:0/1093607347 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210324423 0 0) 0x7ff1000b2980 con 0x7ff1000cfa30 2024-01-16T15:08:45.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.397+0000 7ff104a00640 1 -- 172.21.15.43:0/1093607347 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e0003520 con 0x7ff1000cfa30 2024-01-16T15:08:45.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.397+0000 7ff105400640 1 -- 172.21.15.43:0/1093607347 >> v1:172.21.15.69:6789/0 conn(0x7ff1000b33d0 legacy=0x7ff1000b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.397+0000 7ff105400640 1 -- 172.21.15.43:0/1093607347 >> v1:172.21.15.43:6789/0 conn(0x7ff1000cfa30 legacy=0x7ff1000cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.397+0000 7ff105400640 1 -- 172.21.15.43:0/1093607347 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000b2bd0 con 0x7ff1000d1fd0 2024-01-16T15:08:45.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.398+0000 7ff105400640 1 --2- 172.21.15.43:0/1093607347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000d1fd0 0x7ff1000ce920 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097a0 tx=0x7ff0e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581ace94af77570b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.398+0000 7ff104a00640 1 -- 172.21.15.43:0/1093607347 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8030610 con 0x7ff1000d1fd0 2024-01-16T15:08:45.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.398+0000 7ff104a00640 1 -- 172.21.15.43:0/1093607347 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e8030770 con 0x7ff1000d1fd0 2024-01-16T15:08:45.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.398+0000 7ff104a00640 1 -- 172.21.15.43:0/1093607347 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8030ae0 con 0x7ff1000d1fd0 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 -- 172.21.15.43:0/1093607347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000d1fd0 msgr2=0x7ff1000ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 --2- 172.21.15.43:0/1093607347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000d1fd0 0x7ff1000ce920 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097a0 tx=0x7ff0e802b960 comp rx=0 tx=0).stop 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 -- 172.21.15.43:0/1093607347 shutdown_connections 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 --2- 172.21.15.43:0/1093607347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000d1fd0 0x7ff1000ce920 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 -- 172.21.15.43:0/1093607347 >> 172.21.15.43:0/1093607347 conn(0x7ff1000d3020 msgr2=0x7ff1000d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 -- 172.21.15.43:0/1093607347 shutdown_connections 2024-01-16T15:08:45.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.399+0000 7ff107200640 1 -- 172.21.15.43:0/1093607347 wait complete. 2024-01-16T15:08:45.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.401+0000 7ff107200640 1 Processor -- start 2024-01-16T15:08:45.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.401+0000 7ff107200640 1 -- start start 2024-01-16T15:08:45.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 0x7ff100159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1000d1fd0 0x7ff1000cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000cb570 con 0x7ff1000cfa30 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000cb850 con 0x7ff1000d1fd0 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff107200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000cbb30 con 0x7ff1000b33d0 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.402+0000 7ff105400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60246/0 (socket says 172.21.15.43:60246) 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 -- 172.21.15.43:0/2829046319 learned_addr learned my addr 172.21.15.43:0/2829046319 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105e00640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 0x7ff100159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff106800640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1000d1fd0 0x7ff1000cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 -- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 msgr2=0x7ff100159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 0x7ff100159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 -- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1000d1fd0 msgr2=0x7ff1000cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1000d1fd0 0x7ff1000cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105400640 1 -- 172.21.15.43:0/2829046319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8009400 con 0x7ff1000cfa30 2024-01-16T15:08:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.403+0000 7ff105e00640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 0x7ff100159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:45.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff105400640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8000c00 tx=0x7ff0e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8039c90 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e803f040 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1000cbe10 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8006e10 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.404+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10015a280 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.405+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1000cc330 con 0x7ff1000cfa30 2024-01-16T15:08:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.406+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7ff0e8004030 con 0x7ff1000cfa30 2024-01-16T15:08:45.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.407+0000 7ff0fe000640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 0x7ff0d806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:45.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.407+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff0e80aa2d0 con 0x7ff1000cfa30 2024-01-16T15:08:45.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.407+0000 7ff105e00640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 0x7ff0d806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:45.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.408+0000 7ff105e00640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 0x7ff0d806e230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00011e0 tx=0x7ff0f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:45.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.412+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1000cc330 con 0x7ff1000cfa30 2024-01-16T15:08:45.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.645+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1000cb9c0 con 0x7ff0d806bd70 2024-01-16T15:08:45.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.658+0000 7ff0fe000640 1 -- 172.21.15.43:0/2829046319 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7ff1000cb9c0 con 0x7ff0d806bd70 2024-01-16T15:08:45.635 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:45.635 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:45.636 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.660+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 msgr2=0x7ff0d806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.660+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 0x7ff0d806e230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00011e0 tx=0x7ff0f0009040 comp rx=0 tx=0).stop 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 msgr2=0x7ff10015db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8000c00 tx=0x7ff0e80046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 shutdown_connections 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff0d806bd70 0x7ff0d806e230 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1000d1fd0 0x7ff1000cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1000cfa30 0x7ff10015db30 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 --2- 172.21.15.43:0/2829046319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1000b33d0 0x7ff100159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:45.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 >> 172.21.15.43:0/2829046319 conn(0x7ff1000d3020 msgr2=0x7ff1000cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:45.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 shutdown_connections 2024-01-16T15:08:45.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:45.661+0000 7ff107200640 1 -- 172.21.15.43:0/2829046319 wait complete. 2024-01-16T15:08:45.657 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:08:46.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:08:46.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.913+0000 7f163ca00640 1 Processor -- start 2024-01-16T15:08:46.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.914+0000 7f163ca00640 1 -- start start 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.915+0000 7f163ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f16380b2eb0 0x7f16381618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.915+0000 7f163ca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16380b3290 con 0x7f16380714e0 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.915+0000 7f163ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163815e350 con 0x7f163815d800 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f163ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163815e630 con 0x7f16380b2eb0 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f1637200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f16380714e0 0x7f163806a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40636/0 (socket says 172.21.15.43:40636) 2024-01-16T15:08:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f1637200640 1 -- 172.21.15.43:0/2542487341 learned_addr learned my addr 172.21.15.43:0/2542487341 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:46.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966534398 0 0) 0x7f16380b3290 con 0x7f16380714e0 2024-01-16T15:08:46.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618003520 con 0x7f16380714e0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.916+0000 7f1636800640 1 --2- 172.21.15.43:0/2542487341 >> v2:172.21.15.69:3300/0 conn(0x7f16380b2eb0 0x7f16381618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1408258473 0 0) 0x7f1618003520 con 0x7f16380714e0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 >> v2:172.21.15.69:3300/0 conn(0x7f16380b2eb0 msgr2=0x7f16381618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 --2- 172.21.15.43:0/2542487341 >> v2:172.21.15.69:3300/0 conn(0x7f16380b2eb0 0x7f16381618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 >> v1:172.21.15.69:6789/0 conn(0x7f163815d800 legacy=0x7f1638070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638160df0 con 0x7f16380714e0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f1630001670 con 0x7f16380714e0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 >> v1:172.21.15.43:6789/0 conn(0x7f16380714e0 legacy=0x7f163806a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1618003a30 0x7f1618003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f161800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163815e630 con 0x7f16180052a0 2024-01-16T15:08:46.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161800c1a0 con 0x7f16180047c0 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161800c360 con 0x7f1618003a30 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.917+0000 7f1637200640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f161800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1636800640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1618003a30 0x7f1618003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1635e00640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1637200640 1 -- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1618003a30 msgr2=0x7f1618003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1637200640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1618003a30 0x7f1618003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1637200640 1 -- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180047c0 msgr2=0x7f1618004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1637200640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1637200640 1 -- 172.21.15.43:0/2542487341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618003520 con 0x7f16180052a0 2024-01-16T15:08:46.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.918+0000 7f1635e00640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16180047c0 0x7f1618004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:46.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.919+0000 7f1637200640 1 -- 172.21.15.43:0/2542487341 reap_dead start 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.919+0000 7f1637200640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f161800b980 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1630001eb0 tx=0x7f16180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2f91c8c95c16be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.919+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630005ad0 con 0x7f16180052a0 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.919+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1630005c30 con 0x7f16180052a0 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.919+0000 7f1635400640 1 -- 172.21.15.43:0/2542487341 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630007070 con 0x7f16180052a0 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 -- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 msgr2=0x7f161800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f161800b980 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1630001eb0 tx=0x7f16180057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 -- 172.21.15.43:0/2542487341 shutdown_connections 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 --2- 172.21.15.43:0/2542487341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f161800b980 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 -- 172.21.15.43:0/2542487341 >> 172.21.15.43:0/2542487341 conn(0x7f16380b15c0 msgr2=0x7f16380b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 -- 172.21.15.43:0/2542487341 shutdown_connections 2024-01-16T15:08:46.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.920+0000 7f163ca00640 1 -- 172.21.15.43:0/2542487341 wait complete. 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.922+0000 7f163ca00640 1 Processor -- start 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.922+0000 7f163ca00640 1 -- start start 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16380cbc90 0x7f16380cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16380cd7e0 0x7f16381f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380cce10 con 0x7f16180052a0 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380cc200 con 0x7f16380cbc90 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f163ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16380cc4e0 con 0x7f16380cd7e0 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f1636800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f1636800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60268/0 (socket says 172.21.15.43:60268) 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f1636800640 1 -- 172.21.15.43:0/2017488695 learned_addr learned my addr 172.21.15.43:0/2017488695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.923+0000 7f1635e00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16380cbc90 0x7f16380cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 -- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16380cd7e0 msgr2=0x7f16381f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16380cd7e0 0x7f16381f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 -- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16380cbc90 msgr2=0x7f16380cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16380cbc90 0x7f16380cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 -- 172.21.15.43:0/2017488695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16380cc7c0 con 0x7f16180052a0 2024-01-16T15:08:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f1636800640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f162800f580 tx=0x7f162800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628013810 con 0x7f16180052a0 2024-01-16T15:08:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628013970 con 0x7f16180052a0 2024-01-16T15:08:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16380ccaa0 con 0x7f16180052a0 2024-01-16T15:08:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628013ce0 con 0x7f16180052a0 2024-01-16T15:08:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.924+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1638144490 con 0x7f16180052a0 2024-01-16T15:08:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.925+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16380b21b0 con 0x7f16180052a0 2024-01-16T15:08:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.926+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f162801f460 con 0x7f16180052a0 2024-01-16T15:08:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.928+0000 7f162ea00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 0x7f161006e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.928+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1628088930 con 0x7f16180052a0 2024-01-16T15:08:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.928+0000 7f1635e00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 0x7f161006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.929+0000 7f1635e00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 0x7f161006e260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1620007fd0 tx=0x7f1620009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:46.932+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1628055170 con 0x7f16180052a0 2024-01-16T15:08:47.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.169+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f16380b26c0 con 0x7f161006bda0 2024-01-16T15:08:47.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.186+0000 7f162ea00640 1 -- 172.21.15.43:0/2017488695 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f162000d040 con 0x7f161006bda0 2024-01-16T15:08:47.163 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:47.163 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:47.163 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:47.164 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 msgr2=0x7f161006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 0x7f161006e260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1620007fd0 tx=0x7f1620009040 comp rx=0 tx=0).stop 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 msgr2=0x7f1638143f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f162800f580 tx=0x7f162800fa40 comp rx=0 tx=0).stop 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 shutdown_connections 2024-01-16T15:08:47.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f161006bda0 0x7f161006e260 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16380cbc90 0x7f16380cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f16380cd7e0 0x7f16381f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 --2- 172.21.15.43:0/2017488695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16180052a0 0x7f1638143f50 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.188+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 >> 172.21.15.43:0/2017488695 conn(0x7f16380b15c0 msgr2=0x7f16380afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.189+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 shutdown_connections 2024-01-16T15:08:47.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:47.189+0000 7f163ca00640 1 -- 172.21.15.43:0/2017488695 wait complete. 2024-01-16T15:08:47.184 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:08:48.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:08:48.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.442+0000 7f7b67200640 1 Processor -- start 2024-01-16T15:08:48.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.442+0000 7f7b67200640 1 -- start start 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.443+0000 7f7b67200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b6815d3b0 0x7f7b680ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.443+0000 7f7b67200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b680c6bf0 con 0x7f7b680d1d90 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.443+0000 7f7b67200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b680d1430 con 0x7f7b680d0fa0 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.443+0000 7f7b67200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b680cedc0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.444+0000 7f7b66800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7b680d1d90 0x7f7b680b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40652/0 (socket says 172.21.15.43:40652) 2024-01-16T15:08:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.444+0000 7f7b66800640 1 -- 172.21.15.43:0/2276178026 learned_addr learned my addr 172.21.15.43:0/2276178026 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:48.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.444+0000 7f7b65e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b6815d3b0 0x7f7b680ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:48.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.444+0000 7f7b64a00640 1 -- 172.21.15.43:0/2276178026 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558087074 0 0) 0x7f7b680c6bf0 con 0x7f7b680d1d90 2024-01-16T15:08:48.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.444+0000 7f7b64a00640 1 -- 172.21.15.43:0/2276178026 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48003520 con 0x7f7b680d1d90 2024-01-16T15:08:48.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.445+0000 7f7b65e00640 1 -- 172.21.15.43:0/2276178026 >> v1:172.21.15.69:6789/0 conn(0x7f7b680d0fa0 legacy=0x7f7b680d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.445+0000 7f7b65e00640 1 -- 172.21.15.43:0/2276178026 >> v1:172.21.15.43:6789/0 conn(0x7f7b680d1d90 legacy=0x7f7b680b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.445+0000 7f7b65e00640 1 -- 172.21.15.43:0/2276178026 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680c6e40 con 0x7f7b6815d3b0 2024-01-16T15:08:48.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.446+0000 7f7b65e00640 1 --2- 172.21.15.43:0/2276178026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b680ce6c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7b580097a0 tx=0x7f7b5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78739a7ef97b6982 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:48.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.446+0000 7f7b64a00640 1 -- 172.21.15.43:0/2276178026 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58031830 con 0x7f7b6815d3b0 2024-01-16T15:08:48.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.446+0000 7f7b64a00640 1 -- 172.21.15.43:0/2276178026 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b58031990 con 0x7f7b6815d3b0 2024-01-16T15:08:48.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.446+0000 7f7b64a00640 1 -- 172.21.15.43:0/2276178026 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58031d20 con 0x7f7b6815d3b0 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 -- 172.21.15.43:0/2276178026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 msgr2=0x7f7b680ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 --2- 172.21.15.43:0/2276178026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b680ce6c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7b580097a0 tx=0x7f7b5802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 -- 172.21.15.43:0/2276178026 shutdown_connections 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 --2- 172.21.15.43:0/2276178026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b680ce6c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 -- 172.21.15.43:0/2276178026 >> 172.21.15.43:0/2276178026 conn(0x7f7b680d30a0 msgr2=0x7f7b680d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 -- 172.21.15.43:0/2276178026 shutdown_connections 2024-01-16T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.447+0000 7f7b67200640 1 -- 172.21.15.43:0/2276178026 wait complete. 2024-01-16T15:08:48.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.449+0000 7f7b67200640 1 Processor -- start 2024-01-16T15:08:48.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.449+0000 7f7b67200640 1 -- start start 2024-01-16T15:08:48.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d0fa0 0x7f7b68148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 0x7f7b6814ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680cb9b0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6815af10 con 0x7f7b680d1d90 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.450+0000 7f7b67200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680cb360 con 0x7f7b680d0fa0 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60294/0 (socket says 172.21.15.43:60294) 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 -- 172.21.15.43:0/949550173 learned_addr learned my addr 172.21.15.43:0/949550173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b65400640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 0x7f7b6814ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 -- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d0fa0 msgr2=0x7f7b68148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b65e00640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d0fa0 0x7f7b68148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d0fa0 0x7f7b68148110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 -- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 msgr2=0x7f7b6814ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 0x7f7b6814ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 -- 172.21.15.43:0/949550173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58009400 con 0x7f7b6815d3b0 2024-01-16T15:08:48.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b65400640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 0x7f7b6814ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:48.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.451+0000 7f7b66800640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000c430 tx=0x7f7b6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:48.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.452+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60005ce0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.452+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b60005e40 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.452+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60007310 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.452+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b680cb640 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.452+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68158450 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.453+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b680cb1c0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.454+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f7b60007470 con 0x7f7b6815d3b0 2024-01-16T15:08:48.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.455+0000 7f7b5e000640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 0x7f7b4006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:48.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.455+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7b60087df0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.455+0000 7f7b65e00640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 0x7f7b4006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:48.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.456+0000 7f7b65e00640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 0x7f7b4006e230 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7b58036000 tx=0x7f7b5803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:48.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.459+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b680cb1c0 con 0x7f7b6815d3b0 2024-01-16T15:08:48.646 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:48.668+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:48.646 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:48.668+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:48.646 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:48.668+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:48.646 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:48.668+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:48.646 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:48.668+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.703+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b680cb4d0 con 0x7f7b4006bd70 2024-01-16T15:08:48.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.737+0000 7f7b5e000640 1 -- 172.21.15.43:0/949550173 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f7b680cb4d0 con 0x7f7b4006bd70 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:48.715 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:48.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.739+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 msgr2=0x7f7b4006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.739+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 0x7f7b4006e230 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7b58036000 tx=0x7f7b5803c040 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 msgr2=0x7f7b68157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000c430 tx=0x7f7b6000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 shutdown_connections 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b4006bd70 0x7f7b4006e230 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d1d90 0x7f7b6814ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b6815d3b0 0x7f7b68157f10 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 --2- 172.21.15.43:0/949550173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d0fa0 0x7f7b68148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 >> 172.21.15.43:0/949550173 conn(0x7f7b680d30a0 msgr2=0x7f7b680d3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 shutdown_connections 2024-01-16T15:08:48.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:48.740+0000 7f7b67200640 1 -- 172.21.15.43:0/949550173 wait complete. 2024-01-16T15:08:48.736 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:08:49.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:08:49.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.995+0000 7f86d5000640 1 Processor -- start 2024-01-16T15:08:49.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.995+0000 7f86d5000640 1 -- start start 2024-01-16T15:08:49.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.996+0000 7f86d5000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f86d006ee30 0x7f86d0168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:49.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.996+0000 7f86d5000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d0142d50 con 0x7f86d0070470 2024-01-16T15:08:49.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.996+0000 7f86d5000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d006f430 con 0x7f86d00cd150 2024-01-16T15:08:49.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86d5000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00cd530 con 0x7f86d006ee30 2024-01-16T15:08:49.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86ce800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f86d006ee30 0x7f86d0168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86cd400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f86d0070470 0x7f86d006e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40668/0 (socket says 172.21.15.43:40668) 2024-01-16T15:08:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86cd400640 1 -- 172.21.15.43:0/1228415588 learned_addr learned my addr 172.21.15.43:0/1228415588 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86ce800640 1 -- 172.21.15.43:0/1228415588 >> v1:172.21.15.69:6789/0 conn(0x7f86d00cd150 legacy=0x7f86d006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86ce800640 1 -- 172.21.15.43:0/1228415588 >> v1:172.21.15.43:6789/0 conn(0x7f86d0070470 legacy=0x7f86d006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.997+0000 7f86ce800640 1 -- 172.21.15.43:0/1228415588 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0142fa0 con 0x7f86d006ee30 2024-01-16T15:08:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86ce800640 1 --2- 172.21.15.43:0/1228415588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d0168120 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f86c80097a0 tx=0x7f86c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17386dce16808104 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86cca00640 1 -- 172.21.15.43:0/1228415588 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8031830 con 0x7f86d006ee30 2024-01-16T15:08:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86cca00640 1 -- 172.21.15.43:0/1228415588 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c8031990 con 0x7f86d006ee30 2024-01-16T15:08:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86cca00640 1 -- 172.21.15.43:0/1228415588 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8031d20 con 0x7f86d006ee30 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86d5000640 1 -- 172.21.15.43:0/1228415588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 msgr2=0x7f86d0168120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:49.999+0000 7f86d5000640 1 --2- 172.21.15.43:0/1228415588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d0168120 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f86c80097a0 tx=0x7f86c802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.000+0000 7f86d5000640 1 -- 172.21.15.43:0/1228415588 shutdown_connections 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.000+0000 7f86d5000640 1 --2- 172.21.15.43:0/1228415588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d0168120 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.000+0000 7f86d5000640 1 -- 172.21.15.43:0/1228415588 >> 172.21.15.43:0/1228415588 conn(0x7f86d00cdb20 msgr2=0x7f86d00cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.000+0000 7f86d5000640 1 -- 172.21.15.43:0/1228415588 shutdown_connections 2024-01-16T15:08:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.000+0000 7f86d5000640 1 -- 172.21.15.43:0/1228415588 wait complete. 2024-01-16T15:08:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.001+0000 7f86d5000640 1 Processor -- start 2024-01-16T15:08:49.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.002+0000 7f86d5000640 1 -- start start 2024-01-16T15:08:49.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:49.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 0x7f86d00c3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00cd150 0x7f86d01f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00c9b80 con 0x7f86d006ee30 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00c3dc0 con 0x7f86d0070470 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86d5000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00c9e20 con 0x7f86d00cd150 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60326/0 (socket says 172.21.15.43:60326) 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 -- 172.21.15.43:0/1249119697 learned_addr learned my addr 172.21.15.43:0/1249119697 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cd400640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 0x7f86d00c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86ce800640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00cd150 0x7f86d01f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 -- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00cd150 msgr2=0x7f86d01f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00cd150 0x7f86d01f3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 -- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 msgr2=0x7f86d00c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 0x7f86d00c3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cde00640 1 -- 172.21.15.43:0/1249119697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8009400 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.003+0000 7f86cd400640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 0x7f86d00c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86cde00640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c450 tx=0x7f86c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00052d0 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0005430 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d0143c00 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0012870 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.004+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d00ca3e0 con 0x7f86d006ee30 2024-01-16T15:08:49.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.005+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d006fb50 con 0x7f86d006ee30 2024-01-16T15:08:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.006+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f86c00129d0 con 0x7f86d006ee30 2024-01-16T15:08:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.008+0000 7f86c6000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 0x7f86a806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.008+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f86c0088080 con 0x7f86d006ee30 2024-01-16T15:08:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.008+0000 7f86cd400640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 0x7f86a806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.009+0000 7f86cd400640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 0x7f86a806e230 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f86d0160d20 tx=0x7f86b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.012+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86c0054940 con 0x7f86d006ee30 2024-01-16T15:08:50.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.122+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 23) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f86c0054d10 con 0x7f86d006ee30 2024-01-16T15:08:50.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:50.247+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f86d00c9f90 con 0x7f86a806bd70 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.651+0000 7f86c6000640 1 -- 172.21.15.43:0/1249119697 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f86d00c9f90 con 0x7f86a806bd70 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:53.629 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:53.630 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:53.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:53.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:53.630 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:53.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.654+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 msgr2=0x7f86a806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:53.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.654+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 0x7f86a806e230 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f86d0160d20 tx=0x7f86b8009040 comp rx=0 tx=0).stop 2024-01-16T15:08:53.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.654+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 msgr2=0x7f86d00cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:53.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.654+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f86c000c450 tx=0x7f86c000c910 comp rx=0 tx=0).stop 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 shutdown_connections 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86a806bd70 0x7f86a806e230 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86d0070470 0x7f86d00c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86d00cd150 0x7f86d01f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 --2- 172.21.15.43:0/1249119697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d006ee30 0x7f86d00cabe0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 >> 172.21.15.43:0/1249119697 conn(0x7f86d00cdb20 msgr2=0x7f86d006bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.655+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 shutdown_connections 2024-01-16T15:08:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:53.656+0000 7f86d5000640 1 -- 172.21.15.43:0/1249119697 wait complete. 2024-01-16T15:08:53.647 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:53.670+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:53.648 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:53.670+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:53.648 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:53.670+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:53.648 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:53.670+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:53.648 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:08:53.670+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:53.657 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:08:54.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:08:55.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.129+0000 7f73a3400640 1 Processor -- start 2024-01-16T15:08:55.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.130+0000 7f73a3400640 1 -- start start 2024-01-16T15:08:55.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.133+0000 7f73a3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f739c15b350 0x7f739c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.135+0000 7f73a3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f739c09cb30 0x7f739c0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.135+0000 7f73a3400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c15b730 con 0x7f739c0adb60 2024-01-16T15:08:55.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.135+0000 7f73a3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c0ac740 con 0x7f739c09cb30 2024-01-16T15:08:55.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.135+0000 7f73a3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c0ac460 con 0x7f739c15b350 2024-01-16T15:08:55.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.135+0000 7f73a1400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f739c0adb60 0x7f739c0adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55896/0 (socket says 172.21.15.43:55896) 2024-01-16T15:08:55.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.136+0000 7f73a1400640 1 -- 172.21.15.43:0/3917962030 learned_addr learned my addr 172.21.15.43:0/3917962030 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:55.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.136+0000 7f73a0a00640 1 --2- 172.21.15.43:0/3917962030 >> v2:172.21.15.69:3300/0 conn(0x7f739c15b350 0x7f739c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.136+0000 7f739be00640 1 --2- 172.21.15.43:0/3917962030 >> v2:172.21.15.43:3300/0 conn(0x7f739c09cb30 0x7f739c0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.138+0000 7f739b400640 1 -- 172.21.15.43:0/3917962030 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710828513 0 0) 0x7f7394001490 con 0x7f739c0adb60 2024-01-16T15:08:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.138+0000 7f739be00640 1 -- 172.21.15.43:0/3917962030 >> v2:172.21.15.69:3300/0 conn(0x7f739c15b350 msgr2=0x7f739c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.138+0000 7f739be00640 1 --2- 172.21.15.43:0/3917962030 >> v2:172.21.15.69:3300/0 conn(0x7f739c15b350 0x7f739c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.138+0000 7f739be00640 1 -- 172.21.15.43:0/3917962030 >> v1:172.21.15.43:6789/0 conn(0x7f739c0adb60 legacy=0x7f739c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.138+0000 7f739be00640 1 -- 172.21.15.43:0/3917962030 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c0b2a50 con 0x7f739c09cb30 2024-01-16T15:08:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.142+0000 7f739be00640 1 --2- 172.21.15.43:0/3917962030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c09cb30 0x7f739c0b3530 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f739c0b5100 tx=0x7f738402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9d27ae2ebe439f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.142+0000 7f739b400640 1 -- 172.21.15.43:0/3917962030 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384031640 con 0x7f739c09cb30 2024-01-16T15:08:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.143+0000 7f739b400640 1 -- 172.21.15.43:0/3917962030 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73840317a0 con 0x7f739c09cb30 2024-01-16T15:08:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.143+0000 7f739b400640 1 -- 172.21.15.43:0/3917962030 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384031b70 con 0x7f739c09cb30 2024-01-16T15:08:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.145+0000 7f73a3400640 1 -- 172.21.15.43:0/3917962030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c09cb30 msgr2=0x7f739c0b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.145+0000 7f73a3400640 1 --2- 172.21.15.43:0/3917962030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c09cb30 0x7f739c0b3530 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f739c0b5100 tx=0x7f738402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.146+0000 7f73a3400640 1 -- 172.21.15.43:0/3917962030 shutdown_connections 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.146+0000 7f73a3400640 1 --2- 172.21.15.43:0/3917962030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c09cb30 0x7f739c0b3530 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.146+0000 7f73a3400640 1 --2- 172.21.15.43:0/3917962030 >> v2:172.21.15.69:3300/0 conn(0x7f739c15b350 0x7f739c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.146+0000 7f73a3400640 1 -- 172.21.15.43:0/3917962030 >> 172.21.15.43:0/3917962030 conn(0x7f739c0b0ba0 msgr2=0x7f739c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.147+0000 7f73a3400640 1 -- 172.21.15.43:0/3917962030 shutdown_connections 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.147+0000 7f73a3400640 1 -- 172.21.15.43:0/3917962030 wait complete. 2024-01-16T15:08:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.151+0000 7f73a3400640 1 Processor -- start 2024-01-16T15:08:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.151+0000 7f73a3400640 1 -- start start 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f739c09cb30 0x7f739c152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f739c0adb60 0x7f739c1ffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1563d0 con 0x7f739c15b350 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c156620 con 0x7f739c0adb60 2024-01-16T15:08:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739c156900 con 0x7f739c09cb30 2024-01-16T15:08:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53908/0 (socket says 172.21.15.43:53908) 2024-01-16T15:08:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.153+0000 7f73a1400640 1 -- 172.21.15.43:0/3681641453 learned_addr learned my addr 172.21.15.43:0/3681641453 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f739be00640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f739c0adb60 0x7f739c1ffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f73a1400640 1 -- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f739c09cb30 msgr2=0x7f739c152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f73a1400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f739c09cb30 0x7f739c152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f73a1400640 1 -- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f739c0adb60 msgr2=0x7f739c1ffcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f73a1400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f739c0adb60 0x7f739c1ffcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.154+0000 7f73a1400640 1 -- 172.21.15.43:0/3681641453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7384009400 con 0x7f739c15b350 2024-01-16T15:08:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.155+0000 7f73a1400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f739400c7f0 tx=0x7f739400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.155+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739400a5e0 con 0x7f739c15b350 2024-01-16T15:08:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.156+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c156be0 con 0x7f739c15b350 2024-01-16T15:08:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.156+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c148100 con 0x7f739c15b350 2024-01-16T15:08:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.156+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739400a740 con 0x7f739c15b350 2024-01-16T15:08:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.157+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739400ab30 con 0x7f739c15b350 2024-01-16T15:08:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.157+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c005250 con 0x7f739c15b350 2024-01-16T15:08:55.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.163+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 23) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f739400ac90 con 0x7f739c15b350 2024-01-16T15:08:55.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.164+0000 7f7398c00640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 0x7f737406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:55.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.165+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73940882d0 con 0x7f739c15b350 2024-01-16T15:08:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.168+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73940549f0 con 0x7f739c15b350 2024-01-16T15:08:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.169+0000 7f73a0a00640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 0x7f737406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:55.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.171+0000 7f73a0a00640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 0x7f737406e2f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f738c006a90 tx=0x7f738c006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:55.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.652+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f736c000bc0 con 0x7f737406be30 2024-01-16T15:08:55.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.751+0000 7f7398c00640 1 -- 172.21.15.43:0/3681641453 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f736c000bc0 con 0x7f737406be30 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "in-progress", 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_report": { 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "percentage completed": 0.0, 2024-01-16T15:08:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout: "regfiles copied": "0/3", 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout: "subdirs copied": "0/1", 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout: "symlinks copied": "0/0" 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:55.734 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.768+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 msgr2=0x7f737406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.768+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 0x7f737406e2f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f738c006a90 tx=0x7f738c006a20 comp rx=0 tx=0).stop 2024-01-16T15:08:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.768+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 msgr2=0x7f739c147bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.768+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f739400c7f0 tx=0x7f739400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 shutdown_connections 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f737406be30 0x7f737406e2f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f739c0adb60 0x7f739c1ffcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f739c15b350 0x7f739c147bc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 --2- 172.21.15.43:0/3681641453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f739c09cb30 0x7f739c152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.775+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 >> 172.21.15.43:0/3681641453 conn(0x7f739c0b0ba0 msgr2=0x7f739c145cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.776+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 shutdown_connections 2024-01-16T15:08:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:55.776+0000 7f73a3400640 1 -- 172.21.15.43:0/3681641453 wait complete. 2024-01-16T15:08:55.803 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:08:55.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9617 2024-01-16T15:08:56.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.263+0000 7f7f9ec00640 1 Processor -- start 2024-01-16T15:08:56.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.263+0000 7f7f9ec00640 1 -- start start 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.265+0000 7f7f9ec00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f9815b350 0x7f7f980abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.265+0000 7f7f9ec00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9815b730 con 0x7f7f9809cb30 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.265+0000 7f7f9ec00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f980ac740 con 0x7f7f980adb60 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.265+0000 7f7f9ec00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f980ac460 con 0x7f7f9815b350 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.266+0000 7f7f97400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7f9809cb30 0x7f7f980b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55910/0 (socket says 172.21.15.43:55910) 2024-01-16T15:08:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.266+0000 7f7f97400640 1 -- 172.21.15.43:0/3884202458 learned_addr learned my addr 172.21.15.43:0/3884202458 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:56.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.266+0000 7f7f97e00640 1 --2- 172.21.15.43:0/3884202458 >> v2:172.21.15.69:3300/0 conn(0x7f7f9815b350 0x7f7f980abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.266+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988132539 0 0) 0x7f7f80001110 con 0x7f7f9809cb30 2024-01-16T15:08:56.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.266+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f78003520 con 0x7f7f9809cb30 2024-01-16T15:08:56.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232806318 0 0) 0x7f7f78003520 con 0x7f7f9809cb30 2024-01-16T15:08:56.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 >> v2:172.21.15.69:3300/0 conn(0x7f7f9815b350 msgr2=0x7f7f980abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 --2- 172.21.15.43:0/3884202458 >> v2:172.21.15.69:3300/0 conn(0x7f7f9815b350 0x7f7f980abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 >> v1:172.21.15.69:6789/0 conn(0x7f7f980adb60 legacy=0x7f7f980adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980b2a50 con 0x7f7f9809cb30 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7f80001800 con 0x7f7f9809cb30 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 >> v1:172.21.15.43:6789/0 conn(0x7f7f9809cb30 legacy=0x7f7f980b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f780039c0 0x7f7f78003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780051d0 0x7f7f7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980ac460 con 0x7f7f780047c0 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7800c100 con 0x7f7f780039c0 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7800c2c0 con 0x7f7f780051d0 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.267+0000 7f7f97400640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97e00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f780039c0 0x7f7f78003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97400640 1 -- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780051d0 msgr2=0x7f7f7800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97400640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780051d0 0x7f7f7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97400640 1 -- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f780039c0 msgr2=0x7f7f78003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97400640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f780039c0 0x7f7f78003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f97400640 1 -- 172.21.15.43:0/3884202458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78003520 con 0x7f7f780047c0 2024-01-16T15:08:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.268+0000 7f7f9cc00640 1 -- 172.21.15.43:0/3884202458 reap_dead start 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.269+0000 7f7f97400640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7f80001eb0 tx=0x7f7f780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf75a9cb2de95af8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.270+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80005ad0 con 0x7f7f780047c0 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.270+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80005c30 con 0x7f7f780047c0 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.270+0000 7f7f96a00640 1 -- 172.21.15.43:0/3884202458 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8002c9c0 con 0x7f7f780047c0 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.270+0000 7f7f9ec00640 1 -- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 msgr2=0x7f7f78004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.270+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7f80001eb0 tx=0x7f7f780057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.271+0000 7f7f9ec00640 1 -- 172.21.15.43:0/3884202458 shutdown_connections 2024-01-16T15:08:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.271+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/3884202458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f780047c0 0x7f7f78004c40 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.271+0000 7f7f9ec00640 1 -- 172.21.15.43:0/3884202458 >> 172.21.15.43:0/3884202458 conn(0x7f7f980b0ba0 msgr2=0x7f7f980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.271+0000 7f7f9ec00640 1 -- 172.21.15.43:0/3884202458 shutdown_connections 2024-01-16T15:08:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.271+0000 7f7f9ec00640 1 -- 172.21.15.43:0/3884202458 wait complete. 2024-01-16T15:08:56.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.273+0000 7f7f9ec00640 1 Processor -- start 2024-01-16T15:08:56.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.274+0000 7f7f9ec00640 1 -- start start 2024-01-16T15:08:56.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780047c0 0x7f7f981e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f9809ab80 0x7f7f9809b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f981e7050 con 0x7f7f98071740 2024-01-16T15:08:56.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9809b540 con 0x7f7f9809ab80 2024-01-16T15:08:56.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.275+0000 7f7f9ec00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980722b0 con 0x7f7f780047c0 2024-01-16T15:08:56.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.277+0000 7f7f97400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.277+0000 7f7f97400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53930/0 (socket says 172.21.15.43:53930) 2024-01-16T15:08:56.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.277+0000 7f7f97400640 1 -- 172.21.15.43:0/984820461 learned_addr learned my addr 172.21.15.43:0/984820461 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 -- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780047c0 msgr2=0x7f7f981e6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780047c0 0x7f7f981e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 -- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f9809ab80 msgr2=0x7f7f9809b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f9809ab80 0x7f7f9809b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 -- 172.21.15.43:0/984820461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f981411f0 con 0x7f7f98071740 2024-01-16T15:08:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.278+0000 7f7f97400640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7f80001eb0 tx=0x7f7f80001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.279+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8002cc70 con 0x7f7f98071740 2024-01-16T15:08:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.279+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8003a070 con 0x7f7f98071740 2024-01-16T15:08:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.279+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80007070 con 0x7f7f98071740 2024-01-16T15:08:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.279+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f981414d0 con 0x7f7f98071740 2024-01-16T15:08:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.279+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f98141a90 con 0x7f7f98071740 2024-01-16T15:08:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.283+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f98141cb0 con 0x7f7f98071740 2024-01-16T15:08:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.285+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7f80032070 con 0x7f7f98071740 2024-01-16T15:08:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.288+0000 7f7f8fe00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 0x7f7f7006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.288+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7f800a9710 con 0x7f7f98071740 2024-01-16T15:08:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.293+0000 7f7f97e00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 0x7f7f7006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.295+0000 7f7f97e00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 0x7f7f7006e380 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7f88006a90 tx=0x7f7f88006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.301+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f80075fd0 con 0x7f7f98071740 2024-01-16T15:08:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.780+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9617", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f9806e720 con 0x7f7f7006bec0 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.824+0000 7f7f8fe00640 1 -- 172.21.15.43:0/984820461 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f7f8800b1c0 con 0x7f7f7006bec0 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:08:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:08:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:08:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:08:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:56.803 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.829+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 msgr2=0x7f7f7006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.829+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 0x7f7f7006e380 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7f88006a90 tx=0x7f7f88006a20 comp rx=0 tx=0).stop 2024-01-16T15:08:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.829+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 msgr2=0x7f7f98071bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.829+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7f80001eb0 tx=0x7f7f80001ac0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 shutdown_connections 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f7006bec0 0x7f7f7006e380 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f9809ab80 0x7f7f9809b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f98071740 0x7f7f98071bc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 --2- 172.21.15.43:0/984820461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f780047c0 0x7f7f981e6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 >> 172.21.15.43:0/984820461 conn(0x7f7f980b0ba0 msgr2=0x7f7f980b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 shutdown_connections 2024-01-16T15:08:56.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:56.831+0000 7f7f9ec00640 1 -- 172.21.15.43:0/984820461 wait complete. 2024-01-16T15:08:56.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:08:56.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:56.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.249+0000 7f5380200640 1 Processor -- start 2024-01-16T15:08:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.250+0000 7f5380200640 1 -- start start 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f5380200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53780b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f5380200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537815c360 con 0x7f537806d520 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f5380200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537815ba30 con 0x7f53780ad720 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f5380200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780b41a0 con 0x7f53780abff0 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f537d800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f537806d520 0x7f537806d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55922/0 (socket says 172.21.15.43:55922) 2024-01-16T15:08:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.251+0000 7f537d800640 1 -- 172.21.15.43:0/3755765378 learned_addr learned my addr 172.21.15.43:0/3755765378 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f537e200640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53780b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347642028 0 0) 0x7f537815c360 con 0x7f537806d520 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358003520 con 0x7f537806d520 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143271351 0 0) 0x7f5358003520 con 0x7f537806d520 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 msgr2=0x7f53780b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53780b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 >> v1:172.21.15.69:6789/0 conn(0x7f53780ad720 legacy=0x7f53780b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537815bd10 con 0x7f537806d520 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f5368001670 con 0x7f537806d520 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 >> v1:172.21.15.43:6789/0 conn(0x7f537806d520 legacy=0x7f537806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5358003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.252+0000 7f5377e00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358004d00 0x7f5358005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5358005950 con 0x7f53580039a0 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5358005ac0 con 0x7f5358004d00 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5358005c30 con 0x7f53780abff0 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5358003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537e200640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358004d00 0x7f5358005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537ce00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 -- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 msgr2=0x7f53580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 -- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358004d00 msgr2=0x7f5358005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358004d00 0x7f5358005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537d800640 1 -- 172.21.15.43:0/3755765378 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358003520 con 0x7f53580039a0 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.253+0000 7f537ce00640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53780abff0 0x7f53580047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.254+0000 7f537d800640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5358003da0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f536800c0e0 tx=0x7f53580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0b454dfdbbc3b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.255+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5368005ad0 con 0x7f53580039a0 2024-01-16T15:08:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.255+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5368005c30 con 0x7f53580039a0 2024-01-16T15:08:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.255+0000 7f5377e00640 1 -- 172.21.15.43:0/3755765378 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536802c810 con 0x7f53580039a0 2024-01-16T15:08:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 -- 172.21.15.43:0/3755765378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 msgr2=0x7f5358003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 --2- 172.21.15.43:0/3755765378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5358003da0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f536800c0e0 tx=0x7f53580057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f537e200640 1 -- 172.21.15.43:0/3755765378 reap_dead start 2024-01-16T15:08:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 -- 172.21.15.43:0/3755765378 shutdown_connections 2024-01-16T15:08:57.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 -- 172.21.15.43:0/3755765378 >> 172.21.15.43:0/3755765378 conn(0x7f53780b0d80 msgr2=0x7f53780b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:57.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 -- 172.21.15.43:0/3755765378 shutdown_connections 2024-01-16T15:08:57.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.256+0000 7f5380200640 1 -- 172.21.15.43:0/3755765378 wait complete. 2024-01-16T15:08:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.260+0000 7f5380200640 1 Processor -- start 2024-01-16T15:08:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.260+0000 7f5380200640 1 -- start start 2024-01-16T15:08:57.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.261+0000 7f5380200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f5380200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 0x7f53781f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f5380200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f537815d1b0 0x7f537815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f5380200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53781f2780 con 0x7f53580039a0 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f5380200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53781459b0 con 0x7f537815d1b0 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f5380200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780cbd40 con 0x7f53781f1dc0 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f537d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f537d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53962/0 (socket says 172.21.15.43:53962) 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f537d800640 1 -- 172.21.15.43:0/3243171629 learned_addr learned my addr 172.21.15.43:0/3243171629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f537e200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f537815d1b0 0x7f537815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.262+0000 7f537ce00640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 0x7f53781f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537d800640 1 -- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 msgr2=0x7f53781f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537d800640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 0x7f53781f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537d800640 1 -- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f537815d1b0 msgr2=0x7f537815d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537d800640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f537815d1b0 0x7f537815d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537d800640 1 -- 172.21.15.43:0/3243171629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780cc5e0 con 0x7f53580039a0 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.263+0000 7f537ce00640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 0x7f53781f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:57.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.264+0000 7f537d800640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f536800c200 tx=0x7f537815c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:57.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.264+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536802cc20 con 0x7f53580039a0 2024-01-16T15:08:57.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.264+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5368001720 con 0x7f53580039a0 2024-01-16T15:08:57.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.264+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5368007070 con 0x7f53580039a0 2024-01-16T15:08:57.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.264+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537815df10 con 0x7f53580039a0 2024-01-16T15:08:57.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.265+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53780cc270 con 0x7f53580039a0 2024-01-16T15:08:57.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.266+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5348005250 con 0x7f53580039a0 2024-01-16T15:08:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.270+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5368032070 con 0x7f53580039a0 2024-01-16T15:08:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.272+0000 7f5375600640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 0x7f535006e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.272+0000 7f537ce00640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 0x7f535006e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.272+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f536803c040 con 0x7f53580039a0 2024-01-16T15:08:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.273+0000 7f537ce00640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 0x7f535006e450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5360007fd0 tx=0x7f5360009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.275+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53680762c0 con 0x7f53580039a0 2024-01-16T15:08:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.530+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f53480050c0 con 0x7f53580039a0 2024-01-16T15:08:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.532+0000 7f5375600640 1 -- 172.21.15.43:0/3243171629 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7f5368045090 con 0x7f53580039a0 2024-01-16T15:08:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:08:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:08:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 589, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:08:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 967, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3906158119, 2024-01-16T15:08:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 7841382400, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 765252730880, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 64283168, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 56971970, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 16, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 21 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:08:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:08:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:08:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 msgr2=0x7f535006e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 0x7f535006e450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5360007fd0 tx=0x7f5360009040 comp rx=0 tx=0).stop 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 msgr2=0x7f5378145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f536800c200 tx=0x7f537815c2d0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 shutdown_connections 2024-01-16T15:08:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f535006bf90 0x7f535006e450 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f537815d1b0 0x7f537815d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53781f1dc0 0x7f53781f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 --2- 172.21.15.43:0/3243171629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580039a0 0x7f5378145470 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 >> 172.21.15.43:0/3243171629 conn(0x7f53780b0d80 msgr2=0x7f53780af580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 shutdown_connections 2024-01-16T15:08:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:57.534+0000 7f5380200640 1 -- 172.21.15.43:0/3243171629 wait complete. 2024-01-16T15:08:58.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:08:58.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.780+0000 7f020be00640 1 Processor -- start 2024-01-16T15:08:58.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.780+0000 7f020be00640 1 -- start start 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143570 0x7f020c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f020c143950 0x7f020c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c071700 con 0x7f020c15e6b0 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020c15b410 con 0x7f020c143950 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c15b130 con 0x7f020c143570 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f020c143950 0x7f020c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f020c143950 0x7f020c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53976/0 (socket says 172.21.15.43:53976) 2024-01-16T15:08:58.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 -- 172.21.15.43:0/223563071 learned_addr learned my addr 172.21.15.43:0/223563071 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:58.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f020aa00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143570 0x7f020c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:58.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 -- 172.21.15.43:0/223563071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143570 msgr2=0x7f020c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:58.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143570 0x7f020c0cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:58.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 -- 172.21.15.43:0/223563071 >> v1:172.21.15.69:6789/0 conn(0x7f020c15e6b0 legacy=0x7f020c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:58.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.782+0000 7f0201e00640 1 -- 172.21.15.43:0/223563071 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c15dbd0 con 0x7f020c143950 2024-01-16T15:08:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f0201e00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c143950 0x7f020c0cf590 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a6ac5cb463917 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f020a000640 1 -- 172.21.15.43:0/223563071 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f40317c0 con 0x7f020c143950 2024-01-16T15:08:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f020a000640 1 -- 172.21.15.43:0/223563071 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f4031920 con 0x7f020c143950 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f020a000640 1 -- 172.21.15.43:0/223563071 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4031cb0 con 0x7f020c143950 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f020be00640 1 -- 172.21.15.43:0/223563071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c143950 msgr2=0x7f020c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.784+0000 7f020be00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c143950 0x7f020c0cf590 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f402b860 comp rx=0 tx=0).stop 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 -- 172.21.15.43:0/223563071 shutdown_connections 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c143950 0x7f020c0cf590 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 --2- 172.21.15.43:0/223563071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143570 0x7f020c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 -- 172.21.15.43:0/223563071 >> 172.21.15.43:0/223563071 conn(0x7f020c06a860 msgr2=0x7f020c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 -- 172.21.15.43:0/223563071 shutdown_connections 2024-01-16T15:08:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.785+0000 7f020be00640 1 -- 172.21.15.43:0/223563071 wait complete. 2024-01-16T15:08:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.786+0000 7f020be00640 1 Processor -- start 2024-01-16T15:08:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.786+0000 7f020be00640 1 -- start start 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 0x7f020c141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143950 0x7f020c159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f6590 con 0x7f020c15e6b0 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f6870 con 0x7f020c143570 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.787+0000 7f020be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1f6b50 con 0x7f020c143950 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53988/0 (socket says 172.21.15.43:53988) 2024-01-16T15:08:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 -- 172.21.15.43:0/3861340015 learned_addr learned my addr 172.21.15.43:0/3861340015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020aa00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 0x7f020c141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 -- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143950 msgr2=0x7f020c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143950 0x7f020c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 -- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 msgr2=0x7f020c141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 0x7f020c141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 -- 172.21.15.43:0/3861340015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f40094d0 con 0x7f020c15e6b0 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020aa00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 0x7f020c141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.788+0000 7f020b400640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f020400c780 tx=0x7f020400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.789+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204004b60 con 0x7f020c15e6b0 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.789+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020400ce90 con 0x7f020c15e6b0 2024-01-16T15:08:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.789+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020400a850 con 0x7f020c15e6b0 2024-01-16T15:08:58.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.789+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c1f6e30 con 0x7f020c15e6b0 2024-01-16T15:08:58.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.789+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c1f6100 con 0x7f020c15e6b0 2024-01-16T15:08:58.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.790+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c1f6350 con 0x7f020c15e6b0 2024-01-16T15:08:58.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.791+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f020400a9b0 con 0x7f020c15e6b0 2024-01-16T15:08:58.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.792+0000 7f0203400640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 0x7f01e406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:58.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.793+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0204088a70 con 0x7f020c15e6b0 2024-01-16T15:08:58.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.793+0000 7f020aa00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 0x7f01e406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:58.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.793+0000 7f020aa00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 0x7f01e406e350 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00b820 tx=0x7f01fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:58.798+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f020c1f6350 con 0x7f020c15e6b0 2024-01-16T15:08:59.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.058+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f020c1f6350 con 0x7f020c15e6b0 2024-01-16T15:08:59.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.059+0000 7f0203400640 1 -- 172.21.15.43:0/3861340015 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7f020c1f6350 con 0x7f020c15e6b0 2024-01-16T15:08:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:08:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 591, 2024-01-16T15:08:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:08:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:59.040 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 967, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3906158119, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 7841382400, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 765252730880, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 64283168, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 56971970, 2024-01-16T15:08:59.041 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 16, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 21 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:08:59.042 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:08:59.043 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:08:59.044 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0, 2024-01-16T15:08:59.045 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:08:59.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:59.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:08:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.061+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 msgr2=0x7f01e406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.061+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 0x7f01e406e350 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00b820 tx=0x7f01fc008040 comp rx=0 tx=0).stop 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.061+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 msgr2=0x7f020c0cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.061+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f020400c780 tx=0x7f020400cc40 comp rx=0 tx=0).stop 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 shutdown_connections 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f01e406be90 0x7f01e406e350 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f020c15e6b0 0x7f020c0cd2f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f020c143950 0x7f020c159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 --2- 172.21.15.43:0/3861340015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f020c143570 0x7f020c141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:59.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 >> 172.21.15.43:0/3861340015 conn(0x7f020c06a860 msgr2=0x7f020c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:59.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 shutdown_connections 2024-01-16T15:08:59.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:08:59.062+0000 7f020be00640 1 -- 172.21.15.43:0/3861340015 wait complete. 2024-01-16T15:09:00.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.310+0000 7f9c42c00640 1 Processor -- start 2024-01-16T15:09:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.311+0000 7f9c42c00640 1 -- start start 2024-01-16T15:09:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.312+0000 7f9c42c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c3c06d520 0x7f9c3c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.312+0000 7f9c42c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c15c360 con 0x7f9c3c0abff0 2024-01-16T15:09:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.312+0000 7f9c42c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c15faf0 con 0x7f9c3c0ad720 2024-01-16T15:09:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.312+0000 7f9c42c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c0b41a0 con 0x7f9c3c06d520 2024-01-16T15:09:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c40c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9c3c0abff0 0x7f9c3c15f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55930/0 (socket says 172.21.15.43:55930) 2024-01-16T15:09:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c40c00640 1 -- 172.21.15.43:0/4265669908 learned_addr learned my addr 172.21.15.43:0/4265669908 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3be00640 1 --2- 172.21.15.43:0/4265669908 >> v2:172.21.15.69:3300/0 conn(0x7f9c3c06d520 0x7f9c3c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750871996 0 0) 0x7f9c3c15c360 con 0x7f9c3c0abff0 2024-01-16T15:09:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c003520 con 0x7f9c3c0abff0 2024-01-16T15:09:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800313920 0 0) 0x7f9c1c003520 con 0x7f9c3c0abff0 2024-01-16T15:09:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 >> v2:172.21.15.69:3300/0 conn(0x7f9c3c06d520 msgr2=0x7f9c3c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.313+0000 7f9c3aa00640 1 --2- 172.21.15.43:0/4265669908 >> v2:172.21.15.69:3300/0 conn(0x7f9c3c06d520 0x7f9c3c06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 >> v1:172.21.15.69:6789/0 conn(0x7f9c3c0ad720 legacy=0x7f9c3c15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c06de40 con 0x7f9c3c0abff0 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9c340019f0 con 0x7f9c3c0abff0 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 >> v1:172.21.15.43:6789/0 conn(0x7f9c3c0abff0 legacy=0x7f9c3c15f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0039c0 0x7f9c1c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 0x7f9c1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c005180 0x7f9c1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00c050 con 0x7f9c1c0047c0 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00c1c0 con 0x7f9c1c005180 2024-01-16T15:09:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00c380 con 0x7f9c1c0039c0 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 0x7f9c1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3be00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0039c0 0x7f9c1c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 -- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0039c0 msgr2=0x7f9c1c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0039c0 0x7f9c1c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 -- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c005180 msgr2=0x7f9c1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c40c00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c005180 0x7f9c1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c1c005180 0x7f9c1c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c3b400640 1 -- 172.21.15.43:0/4265669908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c003520 con 0x7f9c1c0047c0 2024-01-16T15:09:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.314+0000 7f9c40c00640 1 -- 172.21.15.43:0/4265669908 reap_dead start 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.315+0000 7f9c3b400640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 0x7f9c1c004c40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c070290 tx=0x7f9c1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85822e33b54809e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240052d0 con 0x7f9c1c0047c0 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2402e050 con 0x7f9c1c0047c0 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c3aa00640 1 -- 172.21.15.43:0/4265669908 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24003830 con 0x7f9c1c0047c0 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 -- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 msgr2=0x7f9c1c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 0x7f9c1c004c40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c070290 tx=0x7f9c1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 -- 172.21.15.43:0/4265669908 shutdown_connections 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 --2- 172.21.15.43:0/4265669908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c1c0047c0 0x7f9c1c004c40 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 -- 172.21.15.43:0/4265669908 >> 172.21.15.43:0/4265669908 conn(0x7f9c3c0b0d80 msgr2=0x7f9c3c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 -- 172.21.15.43:0/4265669908 shutdown_connections 2024-01-16T15:09:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.316+0000 7f9c42c00640 1 -- 172.21.15.43:0/4265669908 wait complete. 2024-01-16T15:09:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.318+0000 7f9c42c00640 1 Processor -- start 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.318+0000 7f9c42c00640 1 -- start start 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0047c0 0x7f9c3c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c3c0c8400 0x7f9c3c1f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c15f490 con 0x7f9c3c15ead0 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0c8970 con 0x7f9c3c0c8400 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.319+0000 7f9c42c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c1f71e0 con 0x7f9c1c0047c0 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54000/0 (socket says 172.21.15.43:54000) 2024-01-16T15:09:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 -- 172.21.15.43:0/407112347 learned_addr learned my addr 172.21.15.43:0/407112347 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c3be00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0047c0 0x7f9c3c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c3b400640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c3c0c8400 0x7f9c3c1f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 -- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0047c0 msgr2=0x7f9c3c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0047c0 0x7f9c3c0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 -- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c3c0c8400 msgr2=0x7f9c3c1f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c3c0c8400 0x7f9c3c1f6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.320+0000 7f9c40c00640 1 -- 172.21.15.43:0/407112347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c1f3c00 con 0x7f9c3c15ead0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c40c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0acc20 tx=0x7f9c3400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34004d60 con 0x7f9c3c15ead0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3400bb00 con 0x7f9c3c15ead0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c1f2e40 con 0x7f9c3c15ead0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400beb0 con 0x7f9c3c15ead0 2024-01-16T15:09:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.321+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c1f3400 con 0x7f9c3c15ead0 2024-01-16T15:09:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.322+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c074770 con 0x7f9c3c15ead0 2024-01-16T15:09:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.323+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9c34007070 con 0x7f9c3c15ead0 2024-01-16T15:09:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.324+0000 7f9c33e00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 0x7f9c1406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.324+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9c34088930 con 0x7f9c3c15ead0 2024-01-16T15:09:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.324+0000 7f9c3be00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 0x7f9c1406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.325+0000 7f9c3be00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 0x7f9c1406e350 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c15fa90 tx=0x7f9c2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.328+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c340551f0 con 0x7f9c3c15ead0 2024-01-16T15:09:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.596+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9c3c0c8b70 con 0x7f9c3c15ead0 2024-01-16T15:09:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.596+0000 7f9c33e00640 1 -- 172.21.15.43:0/407112347 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2422 (secure 0 0 0) 0x7f9c34058810 con 0x7f9c3c15ead0 2024-01-16T15:09:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:00.574 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 592, 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 967, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3906158119, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 7841382400, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 765252730880, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 64280181, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 56955372, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 16, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 18 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:00.581 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.599+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 msgr2=0x7f9c1406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.599+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 0x7f9c1406e350 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c15fa90 tx=0x7f9c2c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.599+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 msgr2=0x7f9c3c15ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:00.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.599+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0acc20 tx=0x7f9c3400b4a0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 shutdown_connections 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9c1406be90 0x7f9c1406e350 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c3c0c8400 0x7f9c3c1f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c3c15ead0 0x7f9c3c15ef50 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 --2- 172.21.15.43:0/407112347 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c1c0047c0 0x7f9c3c0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 >> 172.21.15.43:0/407112347 conn(0x7f9c3c0b0d80 msgr2=0x7f9c3c0b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 shutdown_connections 2024-01-16T15:09:00.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:00.600+0000 7f9c42c00640 1 -- 172.21.15.43:0/407112347 wait complete. 2024-01-16T15:09:01.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:01.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.842+0000 7f19a3000640 1 Processor -- start 2024-01-16T15:09:01.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.842+0000 7f19a3000640 1 -- start start 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a3000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f199c13dfd0 0x7f199c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a3000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199c0ce7c0 con 0x7f199c0cf050 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a3000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199c140900 con 0x7f199c13dfd0 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c15fa70 con 0x7f199c0cf500 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a1c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f199c13dfd0 0x7f199c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a1c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f199c13dfd0 0x7f199c15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54002/0 (socket says 172.21.15.43:54002) 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.844+0000 7f19a1c00640 1 -- 172.21.15.43:0/4168465991 learned_addr learned my addr 172.21.15.43:0/4168465991 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:01.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.845+0000 7f19a2600640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.845+0000 7f19a1c00640 1 -- 172.21.15.43:0/4168465991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 msgr2=0x7f199c06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:01.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.845+0000 7f19a1c00640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c06c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.845+0000 7f19a1c00640 1 -- 172.21.15.43:0/4168465991 >> v1:172.21.15.43:6789/0 conn(0x7f199c0cf050 legacy=0x7f199c06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.845+0000 7f19a1c00640 1 -- 172.21.15.43:0/4168465991 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c0cf950 con 0x7f199c13dfd0 2024-01-16T15:09:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.846+0000 7f19a1c00640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c13dfd0 0x7f199c15be70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f942d1954030c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.846+0000 7f199be00640 1 -- 172.21.15.43:0/4168465991 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c030840 con 0x7f199c13dfd0 2024-01-16T15:09:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.846+0000 7f199be00640 1 -- 172.21.15.43:0/4168465991 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0309a0 con 0x7f199c13dfd0 2024-01-16T15:09:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.846+0000 7f199be00640 1 -- 172.21.15.43:0/4168465991 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c030d30 con 0x7f199c13dfd0 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 -- 172.21.15.43:0/4168465991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c13dfd0 msgr2=0x7f199c15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c13dfd0 0x7f199c15be70 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c02b980 comp rx=0 tx=0).stop 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 -- 172.21.15.43:0/4168465991 shutdown_connections 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 --2- 172.21.15.43:0/4168465991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c13dfd0 0x7f199c15be70 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 -- 172.21.15.43:0/4168465991 >> 172.21.15.43:0/4168465991 conn(0x7f199c06a860 msgr2=0x7f199c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 -- 172.21.15.43:0/4168465991 shutdown_connections 2024-01-16T15:09:01.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.847+0000 7f19a3000640 1 -- 172.21.15.43:0/4168465991 wait complete. 2024-01-16T15:09:01.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.849+0000 7f19a3000640 1 Processor -- start 2024-01-16T15:09:01.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.849+0000 7f19a3000640 1 -- start start 2024-01-16T15:09:01.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f199c13dfd0 0x7f199c0b1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b20e0 con 0x7f199c0cf050 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b23c0 con 0x7f199c13dfd0 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b26a0 con 0x7f199c0cf500 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54014/0 (socket says 172.21.15.43:54014) 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.850+0000 7f19a1c00640 1 -- 172.21.15.43:0/3050329304 learned_addr learned my addr 172.21.15.43:0/3050329304 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1200640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a2600640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f199c13dfd0 0x7f199c0b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 -- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 msgr2=0x7f199c0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 -- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f199c13dfd0 msgr2=0x7f199c0b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:01.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f199c13dfd0 0x7f199c0b1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:01.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 -- 172.21.15.43:0/3050329304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094d0 con 0x7f199c0cf050 2024-01-16T15:09:01.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.851+0000 7f19a1c00640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.852+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c039c90 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.852+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c039df0 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.852+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c043640 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.852+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c0b2980 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.852+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199c0b36c0 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.853+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c04e840 con 0x7f199c0cf050 2024-01-16T15:09:01.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.854+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f198c0437a0 con 0x7f199c0cf050 2024-01-16T15:09:01.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.855+0000 7f1999600640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 0x7f197406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:01.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.855+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f198c03e070 con 0x7f199c0cf050 2024-01-16T15:09:01.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.855+0000 7f19a1200640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 0x7f197406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:01.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.856+0000 7f19a1200640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 0x7f197406e350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f199c0677b0 tx=0x7f1984009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:01.859+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f198c077fd0 con 0x7f199c0cf050 2024-01-16T15:09:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.122+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f199c1361a0 con 0x7f199c0cf050 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.123+0000 7f1999600640 1 -- 172.21.15.43:0/3050329304 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f198c034070 con 0x7f199c0cf050 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 594, 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1315, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 5369993003, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 11167588352, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 761926524928, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 178871953, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 178885514, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 43, 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 48 2024-01-16T15:09:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:02.107 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.125+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 msgr2=0x7f197406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.125+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 0x7f197406e350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f199c0677b0 tx=0x7f1984009040 comp rx=0 tx=0).stop 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 msgr2=0x7f199c070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c004730 comp rx=0 tx=0).stop 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 shutdown_connections 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f197406be90 0x7f197406e350 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f199c13dfd0 0x7f199c0b1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f199c0cf500 0x7f199c0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 --2- 172.21.15.43:0/3050329304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f199c0cf050 0x7f199c070f90 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 >> 172.21.15.43:0/3050329304 conn(0x7f199c06a860 msgr2=0x7f199c15de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 shutdown_connections 2024-01-16T15:09:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:02.126+0000 7f19a3000640 1 -- 172.21.15.43:0/3050329304 wait complete. 2024-01-16T15:09:03.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:03.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.372+0000 7fbd0ee00640 1 Processor -- start 2024-01-16T15:09:03.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.372+0000 7fbd0ee00640 1 -- start start 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.373+0000 7fbd0ee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbd0813dfd0 0x7fbd0815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.373+0000 7fbd0ee00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd080cf050 0x7fbd080cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0ee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080ce7c0 con 0x7fbd0806c130 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0ee00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd080cfa40 con 0x7fbd080cf050 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0ee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd08140900 con 0x7fbd0813dfd0 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd080cf050 0x7fbd080cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd080cf050 0x7fbd080cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53348/0 (socket says 172.21.15.43:53348) 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 -- 172.21.15.43:0/1073659282 learned_addr learned my addr 172.21.15.43:0/1073659282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:03.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0e400640 1 --1- 172.21.15.43:0/1073659282 >> v1:172.21.15.43:6789/0 conn(0x7fbd0806c130 0x7fbd0806c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60574/0 (socket says 172.21.15.43:60574) 2024-01-16T15:09:03.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0da00640 1 --2- 172.21.15.43:0/1073659282 >> v2:172.21.15.69:3300/0 conn(0x7fbd0813dfd0 0x7fbd0815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 -- 172.21.15.43:0/1073659282 >> v2:172.21.15.69:3300/0 conn(0x7fbd0813dfd0 msgr2=0x7fbd0815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:03.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 --2- 172.21.15.43:0/1073659282 >> v2:172.21.15.69:3300/0 conn(0x7fbd0813dfd0 0x7fbd0815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 -- 172.21.15.43:0/1073659282 >> v1:172.21.15.43:6789/0 conn(0x7fbd0806c130 legacy=0x7fbd0806c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.374+0000 7fbd0d000640 1 -- 172.21.15.43:0/1073659282 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0815fa70 con 0x7fbd080cf050 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd0d000640 1 --2- 172.21.15.43:0/1073659282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd080cf050 0x7fbd080cf500 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbd080cfc10 tx=0x7fbcf002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f0c87de2326e5ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd07e00640 1 -- 172.21.15.43:0/1073659282 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0030880 con 0x7fbd080cf050 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd07e00640 1 -- 172.21.15.43:0/1073659282 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf00309e0 con 0x7fbd080cf050 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd07e00640 1 -- 172.21.15.43:0/1073659282 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0030d70 con 0x7fbd080cf050 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1073659282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd080cf050 msgr2=0x7fbd080cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.376+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1073659282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd080cf050 0x7fbd080cf500 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbd080cfc10 tx=0x7fbcf002b9c0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1073659282 shutdown_connections 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1073659282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd080cf050 0x7fbd080cf500 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1073659282 >> v2:172.21.15.69:3300/0 conn(0x7fbd0813dfd0 0x7fbd0815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1073659282 >> 172.21.15.43:0/1073659282 conn(0x7fbd0806a860 msgr2=0x7fbd0806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1073659282 shutdown_connections 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.377+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1073659282 wait complete. 2024-01-16T15:09:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.378+0000 7fbd0ee00640 1 Processor -- start 2024-01-16T15:09:03.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 -- start start 2024-01-16T15:09:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 0x7fbd08149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd080cf050 0x7fbd0815dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080b2d00 con 0x7fbd0813dfd0 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0814a4d0 con 0x7fbd0806c130 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.379+0000 7fbd0ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080b2f50 con 0x7fbd080cf050 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53350/0 (socket says 172.21.15.43:53350) 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 -- 172.21.15.43:0/1938855920 learned_addr learned my addr 172.21.15.43:0/1938855920 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0da00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 0x7fbd08149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 -- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd080cf050 msgr2=0x7fbd0815dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd080cf050 0x7fbd0815dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 -- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 msgr2=0x7fbd08149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 0x7fbd08149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 -- 172.21.15.43:0/1938855920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00094d0 con 0x7fbd0813dfd0 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0da00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 0x7fbd08149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.380+0000 7fbd0e400640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c780 tx=0x7fbd0000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.381+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0000ce90 con 0x7fbd0813dfd0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.381+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0000a640 con 0x7fbd0813dfd0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.381+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0000a9f0 con 0x7fbd0813dfd0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.381+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd080b3230 con 0x7fbd0813dfd0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.381+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0815bc40 con 0x7fbd0813dfd0 2024-01-16T15:09:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.382+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0804e840 con 0x7fbd0813dfd0 2024-01-16T15:09:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.383+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbd0000ab50 con 0x7fbd0813dfd0 2024-01-16T15:09:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.384+0000 7fbd05600640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 0x7fbce006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.384+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd00087f00 con 0x7fbd0813dfd0 2024-01-16T15:09:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.384+0000 7fbd0da00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 0x7fbce006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.385+0000 7fbd0da00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 0x7fbce006e350 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8006fd0 tx=0x7fbcf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:03.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.389+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd00054610 con 0x7fbd0813dfd0 2024-01-16T15:09:03.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.662+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbd080ce9a0 con 0x7fbd0813dfd0 2024-01-16T15:09:03.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.663+0000 7fbd05600640 1 -- 172.21.15.43:0/1938855920 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fbd00017070 con 0x7fbd0813dfd0 2024-01-16T15:09:03.640 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:03.641 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 595, 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:03.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:03.643 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1315, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 5369993003, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 11167588352, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 761926524928, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 178871953, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 178885514, 2024-01-16T15:09:03.644 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 43, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 48 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:03.645 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:03.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:03.647 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:03.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:03.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:03.648 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:03.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.665+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 msgr2=0x7fbce006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.665+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 0x7fbce006e350 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8006fd0 tx=0x7fbcf8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.665+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 msgr2=0x7fbd08155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.665+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c780 tx=0x7fbd0000cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 shutdown_connections 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbce006be90 0x7fbce006e350 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd0813dfd0 0x7fbd08155cf0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd080cf050 0x7fbd0815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 --2- 172.21.15.43:0/1938855920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0806c130 0x7fbd08149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 >> 172.21.15.43:0/1938855920 conn(0x7fbd0806a860 msgr2=0x7fbd0813e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 shutdown_connections 2024-01-16T15:09:03.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:03.666+0000 7fbd0ee00640 1 -- 172.21.15.43:0/1938855920 wait complete. 2024-01-16T15:09:04.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:05.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.066+0000 7ffa89400640 1 Processor -- start 2024-01-16T15:09:05.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.067+0000 7ffa89400640 1 -- start start 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 0x7ffa840b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffa84070780 0x7ffa8406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa84071cd0 con 0x7ffa84160970 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa840b2120 con 0x7ffa84070780 2024-01-16T15:09:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.070+0000 7ffa89400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840b2400 con 0x7ffa840b3140 2024-01-16T15:09:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.071+0000 7ffa81e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.072+0000 7ffa81e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34202/0 (socket says 172.21.15.43:34202) 2024-01-16T15:09:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.072+0000 7ffa81e00640 1 -- 172.21.15.43:0/2069349788 learned_addr learned my addr 172.21.15.43:0/2069349788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:05.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.074+0000 7ffa82800640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 0x7ffa840b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.074+0000 7ffa81e00640 1 -- 172.21.15.43:0/2069349788 >> v2:172.21.15.69:3300/0 conn(0x7ffa84070780 msgr2=0x7ffa8406da60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:09:05.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.074+0000 7ffa81e00640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.69:3300/0 conn(0x7ffa84070780 0x7ffa8406da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.074+0000 7ffa81e00640 1 -- 172.21.15.43:0/2069349788 >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 msgr2=0x7ffa840b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.075+0000 7ffa81e00640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 0x7ffa840b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.075+0000 7ffa81e00640 1 -- 172.21.15.43:0/2069349788 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa840b26e0 con 0x7ffa840b3140 2024-01-16T15:09:05.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.077+0000 7ffa82800640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 0x7ffa840b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:05.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.079+0000 7ffa81e00640 1 --2- 172.21.15.43:0/2069349788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa840ac080 tx=0x7ffa7402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5755598c6b80284b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.080+0000 7ffa81400640 1 -- 172.21.15.43:0/2069349788 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74031830 con 0x7ffa840b3140 2024-01-16T15:09:05.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.080+0000 7ffa81400640 1 -- 172.21.15.43:0/2069349788 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa74031990 con 0x7ffa840b3140 2024-01-16T15:09:05.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.080+0000 7ffa81400640 1 -- 172.21.15.43:0/2069349788 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74031d60 con 0x7ffa840b3140 2024-01-16T15:09:05.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 -- 172.21.15.43:0/2069349788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 msgr2=0x7ffa8406d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 --2- 172.21.15.43:0/2069349788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa840ac080 tx=0x7ffa7402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 -- 172.21.15.43:0/2069349788 shutdown_connections 2024-01-16T15:09:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.69:3300/0 conn(0x7ffa84070780 0x7ffa8406da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 --2- 172.21.15.43:0/2069349788 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa840b3140 0x7ffa8406d520 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 --2- 172.21.15.43:0/2069349788 >> v2:172.21.15.43:3300/0 conn(0x7ffa84160970 0x7ffa840b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.082+0000 7ffa89400640 1 -- 172.21.15.43:0/2069349788 >> 172.21.15.43:0/2069349788 conn(0x7ffa840b0d80 msgr2=0x7ffa840b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.083+0000 7ffa89400640 1 -- 172.21.15.43:0/2069349788 shutdown_connections 2024-01-16T15:09:05.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.083+0000 7ffa89400640 1 -- 172.21.15.43:0/2069349788 wait complete. 2024-01-16T15:09:05.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.089+0000 7ffa89400640 1 Processor -- start 2024-01-16T15:09:05.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.090+0000 7ffa89400640 1 -- start start 2024-01-16T15:09:05.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.093+0000 7ffa89400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 0x7ffa841fa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.093+0000 7ffa89400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.093+0000 7ffa89400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa8409ca90 0x7ffa8415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa82800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 0x7ffa841fa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa89400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8409cf20 con 0x7ffa84070780 2024-01-16T15:09:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa81e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa83200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa8409ca90 0x7ffa8415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa81e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:52242/0 (socket says 172.21.15.43:52242) 2024-01-16T15:09:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa81e00640 1 -- 172.21.15.43:0/3736429310 learned_addr learned my addr 172.21.15.43:0/3736429310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa89400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84145c10 con 0x7ffa84160970 2024-01-16T15:09:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.094+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8415f410 con 0x7ffa8409ca90 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.095+0000 7ffa81e00640 1 -- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa8409ca90 msgr2=0x7ffa8415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.095+0000 7ffa81e00640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa8409ca90 0x7ffa8415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.095+0000 7ffa81e00640 1 -- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 msgr2=0x7ffa841fa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.096+0000 7ffa81e00640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 0x7ffa841fa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.096+0000 7ffa81e00640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740094d0 con 0x7ffa84160970 2024-01-16T15:09:05.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.096+0000 7ffa82800640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 0x7ffa841fa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:05.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.096+0000 7ffa81e00640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa7402b910 tx=0x7ffa740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.097+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740317f0 con 0x7ffa84160970 2024-01-16T15:09:05.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.097+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7403a850 con 0x7ffa84160970 2024-01-16T15:09:05.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.097+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7403abc0 con 0x7ffa84160970 2024-01-16T15:09:05.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.097+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa841452e0 con 0x7ffa84160970 2024-01-16T15:09:05.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.098+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa841458a0 con 0x7ffa84160970 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.103+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ffa74042070 con 0x7ffa84160970 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.103+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa84145d80 con 0x7ffa84160970 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.106+0000 7ffa7aa00640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 0x7ffa5c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.106+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffa740aa540 con 0x7ffa84160970 2024-01-16T15:09:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.116+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa74076d70 con 0x7ffa84160970 2024-01-16T15:09:05.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.129+0000 7ffa82800640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 0x7ffa5c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.164+0000 7ffa82800640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 0x7ffa5c06e420 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00d7c0 tx=0x7ffa7c007770 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.744+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ffa84074770 con 0x7ffa84160970 2024-01-16T15:09:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.747+0000 7ffa7aa00640 1 -- 172.21.15.43:0/3736429310 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7ffa74000e60 con 0x7ffa84160970 2024-01-16T15:09:05.726 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:05.726 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:05.727 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 598, 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:05.728 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:05.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1315, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 5369993003, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 11167588352, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 761926524928, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 178871190, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 178880999, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 43, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 46 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:05.730 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:05.731 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:05.732 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:05.733 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:05.734 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:05.734 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 msgr2=0x7ffa5c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 0x7ffa5c06e420 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00d7c0 tx=0x7ffa7c007770 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 msgr2=0x7ffa8409c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa7402b910 tx=0x7ffa740046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 shutdown_connections 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffa5c06bf60 0x7ffa5c06e420 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84160970 0x7ffa8409c550 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa8409ca90 0x7ffa8415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 --2- 172.21.15.43:0/3736429310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa84070780 0x7ffa841fa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.753+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 >> 172.21.15.43:0/3736429310 conn(0x7ffa840b0d80 msgr2=0x7ffa840acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.754+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 shutdown_connections 2024-01-16T15:09:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:05.754+0000 7ffa89400640 1 -- 172.21.15.43:0/3736429310 wait complete. 2024-01-16T15:09:06.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.975+0000 7fe285200640 1 Processor -- start 2024-01-16T15:09:06.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.975+0000 7fe285200640 1 -- start start 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe285200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2800cf050 0x7fe2800cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe285200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2800cd7b0 0x7fe2800cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe285200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28015fa70 con 0x7fe28013dfd0 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe285200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800cfa40 con 0x7fe2800cf050 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe285200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28015c580 con 0x7fe2800cd7b0 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe277e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2800cf050 0x7fe2800cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe277e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2800cf050 0x7fe2800cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53380/0 (socket says 172.21.15.43:53380) 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.977+0000 7fe277e00640 1 -- 172.21.15.43:0/4169784592 learned_addr learned my addr 172.21.15.43:0/4169784592 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.978+0000 7fe27fe00640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2800cd7b0 0x7fe2800cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.978+0000 7fe277e00640 1 -- 172.21.15.43:0/4169784592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2800cd7b0 msgr2=0x7fe2800cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.978+0000 7fe277e00640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2800cd7b0 0x7fe2800cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.978+0000 7fe277e00640 1 -- 172.21.15.43:0/4169784592 >> v1:172.21.15.43:6789/0 conn(0x7fe28013dfd0 legacy=0x7fe28015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.978+0000 7fe277e00640 1 -- 172.21.15.43:0/4169784592 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280140900 con 0x7fe2800cf050 2024-01-16T15:09:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.979+0000 7fe277e00640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cf050 0x7fe2800cf500 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe26802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364251901ee8fe94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.979+0000 7fe27ea00640 1 -- 172.21.15.43:0/4169784592 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268030840 con 0x7fe2800cf050 2024-01-16T15:09:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.979+0000 7fe27ea00640 1 -- 172.21.15.43:0/4169784592 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2680309a0 con 0x7fe2800cf050 2024-01-16T15:09:06.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.979+0000 7fe27ea00640 1 -- 172.21.15.43:0/4169784592 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268030d30 con 0x7fe2800cf050 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 -- 172.21.15.43:0/4169784592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cf050 msgr2=0x7fe2800cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cf050 0x7fe2800cf500 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe26802b980 comp rx=0 tx=0).stop 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 -- 172.21.15.43:0/4169784592 shutdown_connections 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2800cd7b0 0x7fe2800cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 --2- 172.21.15.43:0/4169784592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cf050 0x7fe2800cf500 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 -- 172.21.15.43:0/4169784592 >> 172.21.15.43:0/4169784592 conn(0x7fe28006a860 msgr2=0x7fe28006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 -- 172.21.15.43:0/4169784592 shutdown_connections 2024-01-16T15:09:06.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.980+0000 7fe285200640 1 -- 172.21.15.43:0/4169784592 wait complete. 2024-01-16T15:09:06.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.982+0000 7fe285200640 1 Processor -- start 2024-01-16T15:09:06.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.982+0000 7fe285200640 1 -- start start 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2800cf050 0x7fe2800710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe28013dfd0 0x7fe28015a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280071900 con 0x7fe2800cd7b0 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801f6660 con 0x7fe2800cf050 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe285200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28013ba80 con 0x7fe28013dfd0 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53394/0 (socket says 172.21.15.43:53394) 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 -- 172.21.15.43:0/3984074379 learned_addr learned my addr 172.21.15.43:0/3984074379 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27fe00640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe28013dfd0 0x7fe28015a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 -- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe28013dfd0 msgr2=0x7fe28015a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe28013dfd0 0x7fe28015a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 -- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2800cf050 msgr2=0x7fe2800710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2800cf050 0x7fe2800710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.983+0000 7fe27f400640 1 -- 172.21.15.43:0/3984074379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680094d0 con 0x7fe2800cd7b0 2024-01-16T15:09:06.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe27f400640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c390 tx=0x7fe27000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2700052d0 con 0x7fe2800cd7b0 2024-01-16T15:09:06.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe270005430 con 0x7fe2800cd7b0 2024-01-16T15:09:06.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270003810 con 0x7fe2800cd7b0 2024-01-16T15:09:06.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280071630 con 0x7fe2800cd7b0 2024-01-16T15:09:06.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.984+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28013afa0 con 0x7fe2800cd7b0 2024-01-16T15:09:06.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.985+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28004e840 con 0x7fe2800cd7b0 2024-01-16T15:09:06.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.986+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe270003970 con 0x7fe2800cd7b0 2024-01-16T15:09:06.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.987+0000 7fe277400640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 0x7fe25806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.987+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe270087db0 con 0x7fe2800cd7b0 2024-01-16T15:09:06.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.987+0000 7fe277e00640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 0x7fe25806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.988+0000 7fe277e00640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 0x7fe25806e380 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe268036000 tx=0x7fe26803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:06.991+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe270054410 con 0x7fe2800cd7b0 2024-01-16T15:09:07.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.251+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe2800ce9a0 con 0x7fe2800cd7b0 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.252+0000 7fe277400640 1 -- 172.21.15.43:0/3984074379 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fe270017020 con 0x7fe2800cd7b0 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.230 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 599, 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:07.231 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1541, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6317985376, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12350525440, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760743587840, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 265171590, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 257851902, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 63, 2024-01-16T15:09:07.233 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 67 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:07.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:07.235 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.33329999446868896, 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.19045713543891907, 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:07.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:07.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:07.237 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 msgr2=0x7fe25806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 0x7fe25806e380 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe268036000 tx=0x7fe26803b040 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 msgr2=0x7fe2801f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c390 tx=0x7fe27000c850 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 shutdown_connections 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe25806bec0 0x7fe25806e380 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2800cf050 0x7fe2800710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe28013dfd0 0x7fe28015a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 --2- 172.21.15.43:0/3984074379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2800cd7b0 0x7fe2801f6120 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 >> 172.21.15.43:0/3984074379 conn(0x7fe28006a860 msgr2=0x7fe28015e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 shutdown_connections 2024-01-16T15:09:07.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:07.255+0000 7fe285200640 1 -- 172.21.15.43:0/3984074379 wait complete. 2024-01-16T15:09:08.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:08.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.680+0000 7fe7c8c00640 1 Processor -- start 2024-01-16T15:09:08.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.680+0000 7fe7c8c00640 1 -- start start 2024-01-16T15:09:08.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c8c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7c4161930 0x7fe7c40ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c8c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40ad510 con 0x7fe7c40719c0 2024-01-16T15:09:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c8c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40acc50 con 0x7fe7c406d520 2024-01-16T15:09:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c8c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c4071e60 con 0x7fe7c4161930 2024-01-16T15:09:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c1400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe7c40719c0 0x7fe7c40ad9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60590/0 (socket says 172.21.15.43:60590) 2024-01-16T15:09:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.682+0000 7fe7c1400640 1 -- 172.21.15.43:0/2656228858 learned_addr learned my addr 172.21.15.43:0/2656228858 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.684+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/2656228858 >> v2:172.21.15.69:3300/0 conn(0x7fe7c4161930 0x7fe7c40ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.684+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489080260 0 0) 0x7fe7c40ad510 con 0x7fe7c40719c0 2024-01-16T15:09:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.684+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a4003520 con 0x7fe7c40719c0 2024-01-16T15:09:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698849481 0 0) 0x7fe7a4003520 con 0x7fe7c40719c0 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 >> v2:172.21.15.69:3300/0 conn(0x7fe7c4161930 msgr2=0x7fe7c40ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 --2- 172.21.15.43:0/2656228858 >> v2:172.21.15.69:3300/0 conn(0x7fe7c4161930 0x7fe7c40ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 >> v1:172.21.15.69:6789/0 conn(0x7fe7c406d520 legacy=0x7fe7c406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40acec0 con 0x7fe7c40719c0 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fe7ac001670 con 0x7fe7c40719c0 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 >> v1:172.21.15.43:6789/0 conn(0x7fe7c40719c0 legacy=0x7fe7c40ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7a4003970 0x7fe7a4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a40047c0 0x7fe7a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c4071e60 con 0x7fe7a4003970 2024-01-16T15:09:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a400c0b0 con 0x7fe7a40047c0 2024-01-16T15:09:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.685+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a400c270 con 0x7fe7a4005180 2024-01-16T15:09:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.687+0000 7fe7c2800640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.690+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7a4003970 0x7fe7a4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.690+0000 7fe7c1400640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a40047c0 0x7fe7a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c2800640 1 -- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a40047c0 msgr2=0x7fe7a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c2800640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a40047c0 0x7fe7a4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c2800640 1 -- 172.21.15.43:0/2656228858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7a4003970 msgr2=0x7fe7a4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c2800640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7a4003970 0x7fe7a4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c2800640 1 -- 172.21.15.43:0/2656228858 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40acec0 con 0x7fe7a4005180 2024-01-16T15:09:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.691+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7a4003970 0x7fe7a4003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.693+0000 7fe7c2800640 1 -- 172.21.15.43:0/2656228858 reap_dead start 2024-01-16T15:09:08.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.695+0000 7fe7c2800640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7a400b980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc001d20 tx=0x7fe7a40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6133da8da1d5415a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.696+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc034070 con 0x7fe7a4005180 2024-01-16T15:09:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.697+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc003970 con 0x7fe7a4005180 2024-01-16T15:09:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.697+0000 7fe7c0a00640 1 -- 172.21.15.43:0/2656228858 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc003d00 con 0x7fe7a4005180 2024-01-16T15:09:08.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.698+0000 7fe7c8c00640 1 -- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 msgr2=0x7fe7a400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.698+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7a400b980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc001d20 tx=0x7fe7a40057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.704+0000 7fe7c8c00640 1 -- 172.21.15.43:0/2656228858 shutdown_connections 2024-01-16T15:09:08.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.704+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/2656228858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7a400b980 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.704+0000 7fe7c8c00640 1 -- 172.21.15.43:0/2656228858 >> 172.21.15.43:0/2656228858 conn(0x7fe7c40b0d80 msgr2=0x7fe7c40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:08.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.708+0000 7fe7c8c00640 1 -- 172.21.15.43:0/2656228858 shutdown_connections 2024-01-16T15:09:08.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.708+0000 7fe7c8c00640 1 -- 172.21.15.43:0/2656228858 wait complete. 2024-01-16T15:09:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.711+0000 7fe7c8c00640 1 Processor -- start 2024-01-16T15:09:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.712+0000 7fe7c8c00640 1 -- start start 2024-01-16T15:09:08.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.717+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7c40ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.717+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 0x7fe7c415fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.717+0000 7fe7c8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53398/0 (socket says 172.21.15.43:53398) 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c1e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7c40ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c2800640 1 -- 172.21.15.43:0/3782097991 learned_addr learned my addr 172.21.15.43:0/3782097991 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40ce6f0 con 0x7fe7c415f580 2024-01-16T15:09:08.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c4159c40 con 0x7fe7c41f6cf0 2024-01-16T15:09:08.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c415e680 con 0x7fe7a4005180 2024-01-16T15:09:08.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.719+0000 7fe7c1400640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 0x7fe7c415fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.721+0000 7fe7c1e00640 1 -- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 msgr2=0x7fe7c415fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.722+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 0x7fe7c415fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.722+0000 7fe7c1e00640 1 -- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 msgr2=0x7fe7c4160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.722+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.722+0000 7fe7c1e00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c4159ec0 con 0x7fe7a4005180 2024-01-16T15:09:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.723+0000 7fe7c2800640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:08.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.724+0000 7fe7c1400640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 0x7fe7c415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:08.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.724+0000 7fe7c1e00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7c40ce1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006fd0 tx=0x7fe7b400f8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:08.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.734+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4014870 con 0x7fe7a4005180 2024-01-16T15:09:08.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.734+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b40149d0 con 0x7fe7a4005180 2024-01-16T15:09:08.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.734+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c415e920 con 0x7fe7a4005180 2024-01-16T15:09:08.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.735+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c415eee0 con 0x7fe7a4005180 2024-01-16T15:09:08.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.735+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4014d40 con 0x7fe7a4005180 2024-01-16T15:09:08.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.737+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe794005250 con 0x7fe7a4005180 2024-01-16T15:09:08.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.744+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe7b4015410 con 0x7fe7a4005180 2024-01-16T15:09:08.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.745+0000 7fe7ba000640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 0x7fe79c06e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:08.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.746+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe7b4089700 con 0x7fe7a4005180 2024-01-16T15:09:08.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.758+0000 7fe7c1400640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 0x7fe79c06e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:08.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.758+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7b4055d60 con 0x7fe7a4005180 2024-01-16T15:09:08.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:08.760+0000 7fe7c1400640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 0x7fe79c06e450 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4071200 tx=0x7fe7ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.271+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe7940057d0 con 0x7fe7a4005180 2024-01-16T15:09:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.275+0000 7fe7ba000640 1 -- 172.21.15.43:0/3782097991 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fe7b4019090 con 0x7fe7a4005180 2024-01-16T15:09:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 601, 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1541, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6317985376, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12350525440, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760743587840, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 200834663, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 200844043, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 48, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 51 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.33329999446868896, 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.19045713543891907, 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:09.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 msgr2=0x7fe79c06e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 0x7fe79c06e450 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4071200 tx=0x7fe7ac009040 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 msgr2=0x7fe7c40ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7c40ce1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006fd0 tx=0x7fe7b400f8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 shutdown_connections 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe79c06bf90 0x7fe79c06e450 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4071200 tx=0x7fe7ac009040 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7c41f6cf0 0x7fe7c415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7c415f580 0x7fe7c4160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 --2- 172.21.15.43:0/3782097991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a4005180 0x7fe7c40ce1b0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006fd0 tx=0x7fe7b400f8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.285+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 >> 172.21.15.43:0/3782097991 conn(0x7fe7c40b0d80 msgr2=0x7fe7c40b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.289+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 shutdown_connections 2024-01-16T15:09:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:09.291+0000 7fe7c8c00640 1 -- 172.21.15.43:0/3782097991 wait complete. 2024-01-16T15:09:10.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:10.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.547+0000 7f17fe000640 1 Processor -- start 2024-01-16T15:09:10.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.548+0000 7f17fe000640 1 -- start start 2024-01-16T15:09:10.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17f8143570 0x7f17f80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17f8143950 0x7f17f80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f8071700 con 0x7f17f8143950 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f815b410 con 0x7f17f8143570 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17fe000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f815b130 con 0x7f17f815e6b0 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.549+0000 7f17efe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17f8143950 0x7f17f80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17f8143950 0x7f17f80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53414/0 (socket says 172.21.15.43:53414) 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 -- 172.21.15.43:0/3467047789 learned_addr learned my addr 172.21.15.43:0/3467047789 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17fd600640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17fcc00640 1 --2- 172.21.15.43:0/3467047789 >> v2:172.21.15.69:3300/0 conn(0x7f17f8143570 0x7f17f80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 -- 172.21.15.43:0/3467047789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 msgr2=0x7f17f815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f815a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 -- 172.21.15.43:0/3467047789 >> v2:172.21.15.69:3300/0 conn(0x7f17f8143570 msgr2=0x7f17f80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 --2- 172.21.15.43:0/3467047789 >> v2:172.21.15.69:3300/0 conn(0x7f17f8143570 0x7f17f80cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.550+0000 7f17efe00640 1 -- 172.21.15.43:0/3467047789 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f815dbd0 con 0x7f17f8143950 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.551+0000 7f17efe00640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f80cf590 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f17f8070c70 tx=0x7f17e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb2c5dc10f07b23 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.551+0000 7f17f7e00640 1 -- 172.21.15.43:0/3467047789 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e00317e0 con 0x7f17f8143950 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.551+0000 7f17f7e00640 1 -- 172.21.15.43:0/3467047789 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e0031940 con 0x7f17f8143950 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.551+0000 7f17f7e00640 1 -- 172.21.15.43:0/3467047789 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0031cd0 con 0x7f17f8143950 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 -- 172.21.15.43:0/3467047789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 msgr2=0x7f17f80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f80cf590 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f17f8070c70 tx=0x7f17e002b880 comp rx=0 tx=0).stop 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 -- 172.21.15.43:0/3467047789 shutdown_connections 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f815a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 --2- 172.21.15.43:0/3467047789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f80cf590 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 --2- 172.21.15.43:0/3467047789 >> v2:172.21.15.69:3300/0 conn(0x7f17f8143570 0x7f17f80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 -- 172.21.15.43:0/3467047789 >> 172.21.15.43:0/3467047789 conn(0x7f17f806a860 msgr2=0x7f17f806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 -- 172.21.15.43:0/3467047789 shutdown_connections 2024-01-16T15:09:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.552+0000 7f17fe000640 1 -- 172.21.15.43:0/3467047789 wait complete. 2024-01-16T15:09:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.554+0000 7f17fe000640 1 Processor -- start 2024-01-16T15:09:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.554+0000 7f17fe000640 1 -- start start 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8143570 0x7f17f80ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f81e5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f81e4c70 con 0x7f17f8143950 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f813fb10 con 0x7f17f8143570 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fe000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80ca920 con 0x7f17f815e6b0 2024-01-16T15:09:10.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53422/0 (socket says 172.21.15.43:53422) 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17efe00640 1 -- 172.21.15.43:0/2875142902 learned_addr learned my addr 172.21.15.43:0/2875142902 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.555+0000 7f17fcc00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8143570 0x7f17f80ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17fd600640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f81e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 -- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 msgr2=0x7f17f81e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f81e5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 -- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8143570 msgr2=0x7f17f80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8143570 0x7f17f80ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 -- 172.21.15.43:0/2875142902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e00094d0 con 0x7f17f8143950 2024-01-16T15:09:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17efe00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000c00 tx=0x7f17e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0032880 con 0x7f17f8143950 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e0032a20 con 0x7f17f8143950 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0032dd0 con 0x7f17f8143950 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f81e42d0 con 0x7f17f8143950 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.556+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f81e4890 con 0x7f17f8143950 2024-01-16T15:09:10.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.558+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f81e4db0 con 0x7f17f8143950 2024-01-16T15:09:10.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.558+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f17e004e450 con 0x7f17f8143950 2024-01-16T15:09:10.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.559+0000 7f17f5600640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 0x7f17d006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.559+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f17e00aaa90 con 0x7f17f8143950 2024-01-16T15:09:10.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.559+0000 7f17fcc00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 0x7f17d006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.560+0000 7f17fcc00640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 0x7f17d006e350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f17e8006fd0 tx=0x7f17e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.564+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17f81e4db0 con 0x7f17f8143950 2024-01-16T15:09:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.820+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f17f81e4db0 con 0x7f17f8143950 2024-01-16T15:09:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.821+0000 7f17f5600640 1 -- 172.21.15.43:0/2875142902 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f17f81e4db0 con 0x7f17f8143950 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:10.799 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 603, 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:10.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:10.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 1541, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 6317985376, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 12350525440, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 760743587840, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 200834663, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 200844043, 2024-01-16T15:09:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 48, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 51 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:10.803 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:10.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.33329999446868896, 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-16T15:09:10.805 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.19045713543891907, 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 msgr2=0x7f17d006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 0x7f17d006e350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f17e8006fd0 tx=0x7f17e8008040 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 msgr2=0x7f17f813e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000c00 tx=0x7f17e00046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 shutdown_connections 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17d006be90 0x7f17d006e350 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17f815e6b0 0x7f17f81e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17f8143950 0x7f17f813e8e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 --2- 172.21.15.43:0/2875142902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17f8143570 0x7f17f80ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 >> 172.21.15.43:0/2875142902 conn(0x7f17f806a860 msgr2=0x7f17f806c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 shutdown_connections 2024-01-16T15:09:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:10.824+0000 7f17fe000640 1 -- 172.21.15.43:0/2875142902 wait complete. 2024-01-16T15:09:11.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:12.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.067+0000 7f4f6f400640 1 Processor -- start 2024-01-16T15:09:12.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.067+0000 7f4f6f400640 1 -- start start 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 0x7f4f680cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f68143950 0x7f4f680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f68071700 con 0x7f4f6815e6b0 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6f400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6815b410 con 0x7f4f68143950 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6815b130 con 0x7f4f68143570 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.069+0000 7f4f6d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f68143950 0x7f4f680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4f68143950 0x7f4f680cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53432/0 (socket says 172.21.15.43:53432) 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 -- 172.21.15.43:0/3580719469 learned_addr learned my addr 172.21.15.43:0/3580719469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6e000640 1 --2- 172.21.15.43:0/3580719469 >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 0x7f4f680cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 -- 172.21.15.43:0/3580719469 >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 msgr2=0x7f4f680cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 --2- 172.21.15.43:0/3580719469 >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 0x7f4f680cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 -- 172.21.15.43:0/3580719469 >> v1:172.21.15.69:6789/0 conn(0x7f4f6815e6b0 legacy=0x7f4f6815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6d600640 1 -- 172.21.15.43:0/3580719469 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6815dbd0 con 0x7f4f68143950 2024-01-16T15:09:12.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.070+0000 7f4f6e000640 1 --2- 172.21.15.43:0/3580719469 >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 0x7f4f680cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.071+0000 7f4f6d600640 1 --2- 172.21.15.43:0/3580719469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143950 0x7f4f680cf590 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f4f68070c70 tx=0x7f4f5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877a48edbcacf3a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6cc00640 1 -- 172.21.15.43:0/3580719469 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580317e0 con 0x7f4f68143950 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6cc00640 1 -- 172.21.15.43:0/3580719469 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f58031940 con 0x7f4f68143950 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6cc00640 1 -- 172.21.15.43:0/3580719469 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58031cd0 con 0x7f4f68143950 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 -- 172.21.15.43:0/3580719469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143950 msgr2=0x7f4f680cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 --2- 172.21.15.43:0/3580719469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143950 0x7f4f680cf590 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f4f68070c70 tx=0x7f4f5802b880 comp rx=0 tx=0).stop 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 -- 172.21.15.43:0/3580719469 shutdown_connections 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 --2- 172.21.15.43:0/3580719469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143950 0x7f4f680cf590 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 --2- 172.21.15.43:0/3580719469 >> v2:172.21.15.69:3300/0 conn(0x7f4f68143570 0x7f4f680cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 -- 172.21.15.43:0/3580719469 >> 172.21.15.43:0/3580719469 conn(0x7f4f6806a860 msgr2=0x7f4f6806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.072+0000 7f4f6f400640 1 -- 172.21.15.43:0/3580719469 shutdown_connections 2024-01-16T15:09:12.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.073+0000 7f4f6f400640 1 -- 172.21.15.43:0/3580719469 wait complete. 2024-01-16T15:09:12.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.074+0000 7f4f6f400640 1 Processor -- start 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.074+0000 7f4f6f400640 1 -- start start 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 0x7f4f6813c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 0x7f4f6813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f680cd3d0 con 0x7f4f68143570 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681f2f30 con 0x7f4f6815e6b0 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6813d2b0 con 0x7f4f68143950 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53442/0 (socket says 172.21.15.43:53442) 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.075+0000 7f4f6e000640 1 -- 172.21.15.43:0/1567004781 learned_addr learned my addr 172.21.15.43:0/1567004781 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:12.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6d600640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 0x7f4f6813c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 -- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 msgr2=0x7f4f6813c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 0x7f4f6813c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6ea00640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 0x7f4f6813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 -- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 msgr2=0x7f4f6813cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 0x7f4f6813cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 -- 172.21.15.43:0/1567004781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580094d0 con 0x7f4f68143570 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6d600640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 0x7f4f6813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6ea00640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 0x7f4f6813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f6e000640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c7f0 tx=0x7f4f5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.076+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50007820 con 0x7f4f68143570 2024-01-16T15:09:12.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.077+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f50007980 con 0x7f4f68143570 2024-01-16T15:09:12.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.077+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50007cf0 con 0x7f4f68143570 2024-01-16T15:09:12.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.077+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f680cc7c0 con 0x7f4f68143570 2024-01-16T15:09:12.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.077+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f680ccd80 con 0x7f4f68143570 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.078+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6804e840 con 0x7f4f68143570 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.079+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4f500113f0 con 0x7f4f68143570 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.079+0000 7f4f66000640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 0x7f4f4006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.080+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4f500888f0 con 0x7f4f68143570 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.080+0000 7f4f6d600640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 0x7f4f4006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:12.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.080+0000 7f4f6d600640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 0x7f4f4006e380 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4f58036000 tx=0x7f4f5803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:12.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.084+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f50055150 con 0x7f4f68143570 2024-01-16T15:09:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.342+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4f680b3910 con 0x7f4f68143570 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.343+0000 7f4f66000640 1 -- 172.21.15.43:0/1567004781 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f4f5001f980 con 0x7f4f68143570 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:12.321 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:12.322 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 604, 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:12.323 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:12.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2299, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 9497348354, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 18856132608, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 754237980672, 2024-01-16T15:09:12.325 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 465222803, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 465588526, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 111, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 116 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:12.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.328 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (26s)\n [==============..............] (remaining: 26s)", 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.5, 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:12.329 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (26s)\n [========....................] (remaining: 66s)", 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.28571429848670959, 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:12.330 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 msgr2=0x7f4f4006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 0x7f4f4006e380 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4f58036000 tx=0x7f4f5803d040 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 msgr2=0x7f4f6813a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c7f0 tx=0x7f4f5000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 shutdown_connections 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4006bec0 0x7f4f4006e380 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f6815e6b0 0x7f4f6813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f68143950 0x7f4f6813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 --2- 172.21.15.43:0/1567004781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f68143570 0x7f4f6813a9a0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 >> 172.21.15.43:0/1567004781 conn(0x7f4f6806a860 msgr2=0x7f4f6806ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 shutdown_connections 2024-01-16T15:09:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:12.346+0000 7f4f6f400640 1 -- 172.21.15.43:0/1567004781 wait complete. 2024-01-16T15:09:13.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9618 2024-01-16T15:09:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.605+0000 7f1231a00640 1 Processor -- start 2024-01-16T15:09:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.606+0000 7f1231a00640 1 -- start start 2024-01-16T15:09:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f122c144770 0x7f122c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f122c0b35a0 0x7f122c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c15f930 con 0x7f122c06c9d0 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122c0b43b0 con 0x7f122c144770 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122c0b3b10 con 0x7f122c0b35a0 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f122be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f122c144770 0x7f122c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f122c06c9d0 0x7f122c06ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44828/0 (socket says 172.21.15.43:44828) 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.607+0000 7f1231000640 1 -- 172.21.15.43:0/3057437678 learned_addr learned my addr 172.21.15.43:0/3057437678 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122b400640 1 --2- 172.21.15.43:0/3057437678 >> v2:172.21.15.69:3300/0 conn(0x7f122c0b35a0 0x7f122c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122aa00640 1 -- 172.21.15.43:0/3057437678 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768173268 0 0) 0x7f122c15f930 con 0x7f122c06c9d0 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122be00640 1 -- 172.21.15.43:0/3057437678 >> v2:172.21.15.69:3300/0 conn(0x7f122c0b35a0 msgr2=0x7f122c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122be00640 1 --2- 172.21.15.43:0/3057437678 >> v2:172.21.15.69:3300/0 conn(0x7f122c0b35a0 0x7f122c15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122be00640 1 -- 172.21.15.43:0/3057437678 >> v1:172.21.15.43:6789/0 conn(0x7f122c06c9d0 legacy=0x7f122c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.608+0000 7f122be00640 1 -- 172.21.15.43:0/3057437678 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0b3df0 con 0x7f122c144770 2024-01-16T15:09:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.609+0000 7f122be00640 1 --2- 172.21.15.43:0/3057437678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c144770 0x7f122c15eb60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f122c0ae900 tx=0x7f121c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef39b76bafb5466 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.609+0000 7f122aa00640 1 -- 172.21.15.43:0/3057437678 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c031830 con 0x7f122c144770 2024-01-16T15:09:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f122aa00640 1 -- 172.21.15.43:0/3057437678 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c031990 con 0x7f122c144770 2024-01-16T15:09:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f122aa00640 1 -- 172.21.15.43:0/3057437678 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c031d20 con 0x7f122c144770 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 -- 172.21.15.43:0/3057437678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c144770 msgr2=0x7f122c15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 --2- 172.21.15.43:0/3057437678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c144770 0x7f122c15eb60 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f122c0ae900 tx=0x7f121c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 -- 172.21.15.43:0/3057437678 shutdown_connections 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 --2- 172.21.15.43:0/3057437678 >> v2:172.21.15.69:3300/0 conn(0x7f122c0b35a0 0x7f122c15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 --2- 172.21.15.43:0/3057437678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c144770 0x7f122c15eb60 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 -- 172.21.15.43:0/3057437678 >> 172.21.15.43:0/3057437678 conn(0x7f122c0cec10 msgr2=0x7f122c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 -- 172.21.15.43:0/3057437678 shutdown_connections 2024-01-16T15:09:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.610+0000 7f1231a00640 1 -- 172.21.15.43:0/3057437678 wait complete. 2024-01-16T15:09:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.612+0000 7f1231a00640 1 Processor -- start 2024-01-16T15:09:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.612+0000 7f1231a00640 1 -- start start 2024-01-16T15:09:13.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f122c06c9d0 0x7f122c1f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122c144770 0x7f122c158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c1565f0 con 0x7f122c0b35a0 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c156840 con 0x7f122c144770 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.613+0000 7f1231a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c156b20 con 0x7f122c06c9d0 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35522/0 (socket says 172.21.15.43:35522) 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 -- 172.21.15.43:0/949927424 learned_addr learned my addr 172.21.15.43:0/949927424 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f1231000640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122c144770 0x7f122c158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 -- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f122c06c9d0 msgr2=0x7f122c1f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f122c06c9d0 0x7f122c1f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 -- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122c144770 msgr2=0x7f122c158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122c144770 0x7f122c158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.614+0000 7f122b400640 1 -- 172.21.15.43:0/949927424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c009400 con 0x7f122c0b35a0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f122b400640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f121400d800 tx=0x7f121400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121400b6c0 con 0x7f122c0b35a0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121400b820 con 0x7f122c0b35a0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c156e00 con 0x7f122c0b35a0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121400bbb0 con 0x7f122c0b35a0 2024-01-16T15:09:13.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.615+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c1573c0 con 0x7f122c0b35a0 2024-01-16T15:09:13.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.616+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c157550 con 0x7f122c0b35a0 2024-01-16T15:09:13.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.617+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f121400bd10 con 0x7f122c0b35a0 2024-01-16T15:09:13.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.619+0000 7f1223e00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 0x7f120406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:13.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.619+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1214088d70 con 0x7f122c0b35a0 2024-01-16T15:09:13.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.619+0000 7f122be00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 0x7f120406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:13.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.620+0000 7f122be00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 0x7f120406e350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f121c036000 tx=0x7f121c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:13.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.624+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12140553d0 con 0x7f122c0b35a0 2024-01-16T15:09:13.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:13.861+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9618", "target": ["mon-mgr", ""]}) v1 -- 0x7f122c0b48f0 con 0x7f120406be90 2024-01-16T15:09:14.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.047+0000 7f1223e00640 1 -- 172.21.15.43:0/949927424 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f122c0b48f0 con 0x7f120406be90 2024-01-16T15:09:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.050+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 msgr2=0x7f120406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.050+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 0x7f120406e350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f121c036000 tx=0x7f121c030000 comp rx=0 tx=0).stop 2024-01-16T15:09:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.050+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 msgr2=0x7f122c1f3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.050+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f121400d800 tx=0x7f121400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 shutdown_connections 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f120406be90 0x7f120406e350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122c144770 0x7f122c158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f122c0b35a0 0x7f122c1f3350 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 --2- 172.21.15.43:0/949927424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f122c06c9d0 0x7f122c1f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 >> 172.21.15.43:0/949927424 conn(0x7f122c0cec10 msgr2=0x7f122c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 shutdown_connections 2024-01-16T15:09:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.051+0000 7f1231a00640 1 -- 172.21.15.43:0/949927424 wait complete. 2024-01-16T15:09:14.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone9619 2024-01-16T15:09:14.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.301+0000 7fc542c00640 1 Processor -- start 2024-01-16T15:09:14.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.301+0000 7fc542c00640 1 -- start start 2024-01-16T15:09:14.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.302+0000 7fc542c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc542c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0c6bf0 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc542c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0d13d0 con 0x7fc53c15d3b0 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc542c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0cedc0 con 0x7fc53c0d1d90 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc542200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc53c0d0fa0 0x7fc53c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44840/0 (socket says 172.21.15.43:44840) 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc542200640 1 -- 172.21.15.43:0/576260290 learned_addr learned my addr 172.21.15.43:0/576260290 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:14.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.303+0000 7fc540e00640 1 --2- 172.21.15.43:0/576260290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418122535 0 0) 0x7fc53c0c6bf0 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51c003520 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925780621 0 0) 0x7fc53c0d13d0 con 0x7fc53c15d3b0 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c0c6bf0 con 0x7fc53c15d3b0 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc540e00640 1 -- 172.21.15.43:0/576260290 >> v1:172.21.15.69:6789/0 conn(0x7fc53c15d3b0 legacy=0x7fc53c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc540e00640 1 -- 172.21.15.43:0/576260290 >> v1:172.21.15.43:6789/0 conn(0x7fc53c0d0fa0 legacy=0x7fc53c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.304+0000 7fc540e00640 1 -- 172.21.15.43:0/576260290 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0c6e40 con 0x7fc53c0d1d90 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.305+0000 7fc540e00640 1 --2- 172.21.15.43:0/576260290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c0ce6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5240097d0 tx=0x7fc52402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc13ff5c1977faf2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.306+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524031800 con 0x7fc53c0d1d90 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.306+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc524031960 con 0x7fc53c0d1d90 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.306+0000 7fc53be00640 1 -- 172.21.15.43:0/576260290 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524031cf0 con 0x7fc53c0d1d90 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 -- 172.21.15.43:0/576260290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 msgr2=0x7fc53c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 --2- 172.21.15.43:0/576260290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c0ce6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5240097d0 tx=0x7fc52402b8a0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 -- 172.21.15.43:0/576260290 shutdown_connections 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 --2- 172.21.15.43:0/576260290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c0ce6c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 -- 172.21.15.43:0/576260290 >> 172.21.15.43:0/576260290 conn(0x7fc53c0d30a0 msgr2=0x7fc53c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 -- 172.21.15.43:0/576260290 shutdown_connections 2024-01-16T15:09:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.307+0000 7fc542c00640 1 -- 172.21.15.43:0/576260290 wait complete. 2024-01-16T15:09:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.309+0000 7fc542c00640 1 Processor -- start 2024-01-16T15:09:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.309+0000 7fc542c00640 1 -- start start 2024-01-16T15:09:14.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc53c15d3b0 0x7fc53c0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c09df80 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c09d6e0 con 0x7fc53c15d3b0 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c09d9c0 con 0x7fc53c0d1d90 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc541800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc541800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35524/0 (socket says 172.21.15.43:35524) 2024-01-16T15:09:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc541800640 1 -- 172.21.15.43:0/3881923304 learned_addr learned my addr 172.21.15.43:0/3881923304 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc540e00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.310+0000 7fc542200640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc53c15d3b0 0x7fc53c0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 -- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 msgr2=0x7fc53c09cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c09cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 -- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc53c15d3b0 msgr2=0x7fc53c0b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc53c15d3b0 0x7fc53c0b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 -- 172.21.15.43:0/3881923304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524009400 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc541800640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0052d0 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c005450 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c003830 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.311+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c09dca0 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.312+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c152970 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.313+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c0b6040 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.314+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc52c003990 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.315+0000 7fc539600640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 0x7fc51406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.315+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc52c087df0 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.315+0000 7fc540e00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 0x7fc51406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.316+0000 7fc540e00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 0x7fc51406e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc524036000 tx=0x7fc52403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.320+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc53c0b6040 con 0x7fc53c0d0fa0 2024-01-16T15:09:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.550+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone9619", "target": ["mon-mgr", ""]}) v1 -- 0x7fc53c15d790 con 0x7fc51406be90 2024-01-16T15:09:14.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.713+0000 7fc539600640 1 -- 172.21.15.43:0/3881923304 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc524031be0 con 0x7fc51406be90 2024-01-16T15:09:14.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 msgr2=0x7fc51406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 0x7fc51406e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc524036000 tx=0x7fc52403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 msgr2=0x7fc53c152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 shutdown_connections 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc51406be90 0x7fc51406e350 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc53c15d3b0 0x7fc53c0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc53c0d1d90 0x7fc53c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 --2- 172.21.15.43:0/3881923304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc53c0d0fa0 0x7fc53c152430 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.716+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 >> 172.21.15.43:0/3881923304 conn(0x7fc53c0d30a0 msgr2=0x7fc53c15dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.717+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 shutdown_connections 2024-01-16T15:09:14.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.717+0000 7fc542c00640 1 -- 172.21.15.43:0/3881923304 wait complete. 2024-01-16T15:09:14.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96110 2024-01-16T15:09:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.973+0000 7f9bfb200640 1 Processor -- start 2024-01-16T15:09:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.973+0000 7f9bfb200640 1 -- start start 2024-01-16T15:09:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bfb200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9bfc0d1d90 0x7f9bfc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bfb200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0c6bf0 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bfb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0d13f0 con 0x7f9bfc0d0fa0 2024-01-16T15:09:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bfb200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bfc0cedc0 con 0x7f9bfc0d1d90 2024-01-16T15:09:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bf9e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9bfc15d3b0 0x7f9bfc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44852/0 (socket says 172.21.15.43:44852) 2024-01-16T15:09:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.975+0000 7f9bf9e00640 1 -- 172.21.15.43:0/2442260578 learned_addr learned my addr 172.21.15.43:0/2442260578 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2442260578 >> v2:172.21.15.69:3300/0 conn(0x7f9bfc0d1d90 0x7f9bfc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788381993 0 0) 0x7f9bfc0c6bf0 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bdc003520 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133719410 0 0) 0x7f9bdc003520 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 >> v2:172.21.15.69:3300/0 conn(0x7f9bfc0d1d90 msgr2=0x7f9bfc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 --2- 172.21.15.43:0/2442260578 >> v2:172.21.15.69:3300/0 conn(0x7f9bfc0d1d90 0x7f9bfc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 >> v1:172.21.15.69:6789/0 conn(0x7f9bfc0d0fa0 legacy=0x7f9bfc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc0c6e40 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9bec001670 con 0x7f9bfc15d3b0 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 >> v1:172.21.15.43:6789/0 conn(0x7f9bfc15d3b0 legacy=0x7f9bfc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bdc003a30 0x7f9bdc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.976+0000 7f9bf9400640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bdc0052a0 0x7f9bdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc0cedc0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc00c170 con 0x7f9bdc003a30 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc00c330 con 0x7f9bdc0052a0 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 -- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bdc0052a0 msgr2=0x7f9bdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bdc0052a0 0x7f9bdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 -- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bdc003a30 msgr2=0x7f9bdc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf9e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bdc003a30 0x7f9bdc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bdc003a30 0x7f9bdc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bfa800640 1 -- 172.21.15.43:0/2442260578 reap_dead start 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf3e00640 1 -- 172.21.15.43:0/2442260578 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc003520 con 0x7f9bdc0047c0 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.977+0000 7f9bf9e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bdc003a30 0x7f9bdc003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ac0 tx=0x7f9bdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0aef6b01ad327e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402c6d0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be402c830 con 0x7f9bdc0047c0 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bf9400640 1 -- 172.21.15.43:0/2442260578 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402cbc0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bfb200640 1 -- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 msgr2=0x7f9bdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.979+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ac0 tx=0x7f9bdc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.980+0000 7f9bfb200640 1 -- 172.21.15.43:0/2442260578 shutdown_connections 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.980+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2442260578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.980+0000 7f9bfb200640 1 -- 172.21.15.43:0/2442260578 >> 172.21.15.43:0/2442260578 conn(0x7f9bfc0d30a0 msgr2=0x7f9bfc0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.980+0000 7f9bfb200640 1 -- 172.21.15.43:0/2442260578 shutdown_connections 2024-01-16T15:09:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.980+0000 7f9bfb200640 1 -- 172.21.15.43:0/2442260578 wait complete. 2024-01-16T15:09:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.982+0000 7f9bfb200640 1 Processor -- start 2024-01-16T15:09:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.982+0000 7f9bfb200640 1 -- start start 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bfc09cde0 0x7f9bfc0a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bfc0a0980 0x7f9bfc0cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc14fb40 con 0x7f9bdc0047c0 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc09f730 con 0x7f9bfc0a0980 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bfb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc0cbc40 con 0x7f9bfc09cde0 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.983+0000 7f9bf9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35540/0 (socket says 172.21.15.43:35540) 2024-01-16T15:09:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 -- 172.21.15.43:0/2889318073 learned_addr learned my addr 172.21.15.43:0/2889318073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bfa800640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bfc0a0980 0x7f9bfc0cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bfc09cde0 0x7f9bfc0a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 -- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bfc09cde0 msgr2=0x7f9bfc0a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bfc09cde0 0x7f9bfc0a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 -- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bfc0a0980 msgr2=0x7f9bfc0cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bfc0a0980 0x7f9bfc0cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 -- 172.21.15.43:0/2889318073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc15bef0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.984+0000 7f9bf9e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00d770 tx=0x7f9bec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.985+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0072d0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.985+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec007430 con 0x7f9bdc0047c0 2024-01-16T15:09:14.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.985+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0047e0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.985+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc0cb960 con 0x7f9bdc0047c0 2024-01-16T15:09:14.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.985+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc09fc40 con 0x7f9bdc0047c0 2024-01-16T15:09:14.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.986+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc0ce6a0 con 0x7f9bdc0047c0 2024-01-16T15:09:14.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.987+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9bec004940 con 0x7f9bdc0047c0 2024-01-16T15:09:14.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.988+0000 7f9bf2000640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 0x7f9bd406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.988+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9bec088010 con 0x7f9bdc0047c0 2024-01-16T15:09:14.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.988+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 0x7f9bd406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.989+0000 7f9bf3e00640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 0x7f9bd406e380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:14.993+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bfc0ce6a0 con 0x7f9bdc0047c0 2024-01-16T15:09:15.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.240+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96110", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bfc0b4fe0 con 0x7f9bd406bec0 2024-01-16T15:09:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.414+0000 7f9bf2000640 1 -- 172.21.15.43:0/2889318073 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9bfc0b4fe0 con 0x7f9bd406bec0 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.417+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 msgr2=0x7f9bd406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.417+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 0x7f9bd406e380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be4030000 comp rx=0 tx=0).stop 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.417+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 msgr2=0x7f9bfc14f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.417+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00d770 tx=0x7f9bec00dc30 comp rx=0 tx=0).stop 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 shutdown_connections 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9bd406bec0 0x7f9bd406e380 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bfc0a0980 0x7f9bfc0cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bfc09cde0 0x7f9bfc0a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 --2- 172.21.15.43:0/2889318073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bdc0047c0 0x7f9bfc14f600 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 >> 172.21.15.43:0/2889318073 conn(0x7f9bfc0d30a0 msgr2=0x7f9bfc0b7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 shutdown_connections 2024-01-16T15:09:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.418+0000 7f9bfb200640 1 -- 172.21.15.43:0/2889318073 wait complete. 2024-01-16T15:09:15.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96111 2024-01-16T15:09:15.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.670+0000 7ffa0fe00640 1 Processor -- start 2024-01-16T15:09:15.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.670+0000 7ffa0fe00640 1 -- start start 2024-01-16T15:09:15.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ffa080abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0815b730 con 0x7ffa0815b350 2024-01-16T15:09:15.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080ac740 con 0x7ffa080adb60 2024-01-16T15:09:15.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080ac460 con 0x7ffa0809cb30 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0d400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ffa0815b350 0x7ffa080b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44862/0 (socket says 172.21.15.43:44862) 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.672+0000 7ffa0d400640 1 -- 172.21.15.43:0/3731955679 learned_addr learned my addr 172.21.15.43:0/3731955679 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa0ca00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ffa080abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926362596 0 0) 0x7ff9f8001110 con 0x7ffa0815b350 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e8003520 con 0x7ffa0815b350 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2464300468 0 0) 0x7ff9e8003520 con 0x7ffa0815b350 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 msgr2=0x7ffa080abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ffa080abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 >> v1:172.21.15.69:6789/0 conn(0x7ffa080adb60 legacy=0x7ffa080adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa080b2a50 con 0x7ffa0815b350 2024-01-16T15:09:15.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff9f8001800 con 0x7ffa0815b350 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 >> v1:172.21.15.43:6789/0 conn(0x7ffa0815b350 legacy=0x7ffa080b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ff9e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8003f70 0x7ff9e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9e8004d00 0x7ff9e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080ac460 con 0x7ff9e8004d00 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e800bac0 con 0x7ff9e8003f70 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.673+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e800bc80 con 0x7ffa0809cb30 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9e8004d00 0x7ff9e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0d400640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ff9e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0ca00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8003f70 0x7ff9e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 -- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 msgr2=0x7ff9e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa0809cb30 0x7ff9e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 -- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8003f70 msgr2=0x7ff9e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:15.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8003f70 0x7ff9e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.674+0000 7ffa0de00640 1 -- 172.21.15.43:0/3731955679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8003520 con 0x7ff9e8004d00 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.675+0000 7ffa0de00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9e8004d00 0x7ff9e8005180 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ffa00001cd0 tx=0x7ff9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea212f5a28d6804f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00003970 con 0x7ff9e8004d00 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa00003ad0 con 0x7ff9e8004d00 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa07e00640 1 -- 172.21.15.43:0/3731955679 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00003e80 con 0x7ff9e8004d00 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 -- 172.21.15.43:0/3731955679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9e8004d00 msgr2=0x7ff9e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/3731955679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9e8004d00 0x7ff9e8005180 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ffa00001cd0 tx=0x7ff9e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0de00640 1 -- 172.21.15.43:0/3731955679 reap_dead start 2024-01-16T15:09:15.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 -- 172.21.15.43:0/3731955679 shutdown_connections 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 -- 172.21.15.43:0/3731955679 >> 172.21.15.43:0/3731955679 conn(0x7ffa080b0ba0 msgr2=0x7ffa080b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 -- 172.21.15.43:0/3731955679 shutdown_connections 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.676+0000 7ffa0fe00640 1 -- 172.21.15.43:0/3731955679 wait complete. 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.678+0000 7ffa0fe00640 1 Processor -- start 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.678+0000 7ffa0fe00640 1 -- start start 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.679+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8004d00 0x7ffa0809b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.679+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.679+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0814aeb0 0x7ffa080af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0814c770 con 0x7ffa0814e1b0 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0814ca50 con 0x7ffa0814aeb0 2024-01-16T15:09:15.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0814cd30 con 0x7ff9e8004d00 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35566/0 (socket says 172.21.15.43:35566) 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 -- 172.21.15.43:0/2107890572 learned_addr learned my addr 172.21.15.43:0/2107890572 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0de00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0814aeb0 0x7ffa080af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0d400640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8004d00 0x7ffa0809b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 -- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8004d00 msgr2=0x7ffa0809b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8004d00 0x7ffa0809b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 -- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0814aeb0 msgr2=0x7ffa080af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0814aeb0 0x7ffa080af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.680+0000 7ffa0ca00640 1 -- 172.21.15.43:0/2107890572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0814d010 con 0x7ffa0814e1b0 2024-01-16T15:09:15.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa0ca00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0011600 tx=0x7ff9f0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f000f920 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f000fa80 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f000fdf0 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0814d2f0 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.681+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0809b680 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.682+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0809d200 con 0x7ffa0814e1b0 2024-01-16T15:09:15.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.684+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff9f0015460 con 0x7ffa0814e1b0 2024-01-16T15:09:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.685+0000 7ffa05600640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 0x7ff9e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.685+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff9f0088d30 con 0x7ffa0814e1b0 2024-01-16T15:09:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.685+0000 7ffa0d400640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 0x7ff9e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:15.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.686+0000 7ffa0d400640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 0x7ff9e006e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8001270 tx=0x7ff9f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:15.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.689+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9f0055350 con 0x7ffa0814e1b0 2024-01-16T15:09:15.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:15.919+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96111", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa08066ed0 con 0x7ff9e006be90 2024-01-16T15:09:16.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.091+0000 7ffa05600640 1 -- 172.21.15.43:0/2107890572 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffa08066ed0 con 0x7ff9e006be90 2024-01-16T15:09:16.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 msgr2=0x7ff9e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 0x7ff9e006e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8001270 tx=0x7ff9f8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 msgr2=0x7ffa080af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0011600 tx=0x7ff9f0011ac0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 shutdown_connections 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9e006be90 0x7ff9e006e350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa0814aeb0 0x7ffa080af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa0814e1b0 0x7ffa080af240 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.094+0000 7ffa0fe00640 1 --2- 172.21.15.43:0/2107890572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8004d00 0x7ffa0809b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.095+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 >> 172.21.15.43:0/2107890572 conn(0x7ffa080b0ba0 msgr2=0x7ffa080b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:16.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.095+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 shutdown_connections 2024-01-16T15:09:16.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.095+0000 7ffa0fe00640 1 -- 172.21.15.43:0/2107890572 wait complete. 2024-01-16T15:09:16.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96112 2024-01-16T15:09:16.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.344+0000 7f667f400640 1 Processor -- start 2024-01-16T15:09:16.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.344+0000 7f667f400640 1 -- start start 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6678070990 0x7f6678070d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66780c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667f400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780cee40 con 0x7f667806a440 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667815d820 con 0x7f6678070990 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780712b0 con 0x7f667815dfe0 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667ea00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f667806a440 0x7f667806a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44876/0 (socket says 172.21.15.43:44876) 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667ea00640 1 -- 172.21.15.43:0/4168043465 learned_addr learned my addr 172.21.15.43:0/4168043465 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.346+0000 7f667e000640 1 --2- 172.21.15.43:0/4168043465 >> v2:172.21.15.69:3300/0 conn(0x7f6678070990 0x7f6678070d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667d600640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66780c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791450421 0 0) 0x7f66780cee40 con 0x7f667806a440 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6658003520 con 0x7f667806a440 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169926911 0 0) 0x7f6658003520 con 0x7f667806a440 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 msgr2=0x7f66780c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66780c43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 >> v2:172.21.15.69:3300/0 conn(0x7f6678070990 msgr2=0x7f6678070d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 --2- 172.21.15.43:0/4168043465 >> v2:172.21.15.69:3300/0 conn(0x7f6678070990 0x7f6678070d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667806e1e0 con 0x7f667806a440 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f66700019f0 con 0x7f667806a440 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 >> v1:172.21.15.43:6789/0 conn(0x7f667806a440 legacy=0x7f667806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66580039e0 0x7f6658003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6658004d00 0x7f6658005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667815d820 con 0x7f6658004d00 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780712b0 con 0x7f66580039e0 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.347+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665800b980 con 0x7f667815dfe0 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6658004d00 0x7f6658005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667e000640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66580039e0 0x7f6658003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667d600640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 -- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 msgr2=0x7f66580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667815dfe0 0x7f66580047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 -- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66580039e0 msgr2=0x7f6658003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66580039e0 0x7f6658003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.348+0000 7f667ea00640 1 -- 172.21.15.43:0/4168043465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658003520 con 0x7f6658004d00 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.349+0000 7f667ea00640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6658004d00 0x7f6658005180 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f667000a790 tx=0x7f66580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdce3fabe84c07c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670034630 con 0x7f6658004d00 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667002f070 con 0x7f6658004d00 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667cc00640 1 -- 172.21.15.43:0/4168043465 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700349f0 con 0x7f6658004d00 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667f400640 1 -- 172.21.15.43:0/4168043465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6658004d00 msgr2=0x7f6658005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667f400640 1 --2- 172.21.15.43:0/4168043465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6658004d00 0x7f6658005180 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f667000a790 tx=0x7f66580057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667ea00640 1 -- 172.21.15.43:0/4168043465 reap_dead start 2024-01-16T15:09:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667f400640 1 -- 172.21.15.43:0/4168043465 shutdown_connections 2024-01-16T15:09:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.350+0000 7f667f400640 1 -- 172.21.15.43:0/4168043465 >> 172.21.15.43:0/4168043465 conn(0x7f66780cf500 msgr2=0x7f66780cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.351+0000 7f667f400640 1 -- 172.21.15.43:0/4168043465 shutdown_connections 2024-01-16T15:09:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.351+0000 7f667f400640 1 -- 172.21.15.43:0/4168043465 wait complete. 2024-01-16T15:09:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.353+0000 7f667f400640 1 Processor -- start 2024-01-16T15:09:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.353+0000 7f667f400640 1 -- start start 2024-01-16T15:09:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.354+0000 7f667f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 0x7f6678152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.354+0000 7f667f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66781546c0 0x7f66780b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667f400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678151ef0 con 0x7f66781519f0 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66781521d0 con 0x7f66781546c0 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66781524b0 con 0x7f6658004d00 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35578/0 (socket says 172.21.15.43:35578) 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 -- 172.21.15.43:0/2158200544 learned_addr learned my addr 172.21.15.43:0/2158200544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667ea00640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66781546c0 0x7f66780b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667e000640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 0x7f6678152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 -- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 msgr2=0x7f6678152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 0x7f6678152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 -- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66781546c0 msgr2=0x7f66780b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66781546c0 0x7f66780b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667d600640 1 -- 172.21.15.43:0/2158200544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678152790 con 0x7f66781519f0 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.355+0000 7f667e000640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 0x7f6678152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f667d600640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f6660010860 tx=0x7f6660010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660009610 con 0x7f66781519f0 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666000eb60 con 0x7f66781519f0 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666001c400 con 0x7f66781519f0 2024-01-16T15:09:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6678152a70 con 0x7f66781519f0 2024-01-16T15:09:16.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.356+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678154b50 con 0x7f66781519f0 2024-01-16T15:09:16.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.357+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667806bf10 con 0x7f66781519f0 2024-01-16T15:09:16.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.359+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f666001c560 con 0x7f66781519f0 2024-01-16T15:09:16.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.359+0000 7f6676000640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 0x7f665006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:16.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.360+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6660088db0 con 0x7f66781519f0 2024-01-16T15:09:16.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.360+0000 7f667e000640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 0x7f665006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:16.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.361+0000 7f667e000640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 0x7f665006e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6668009a00 tx=0x7f6668008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.364+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66600554c0 con 0x7f66781519f0 2024-01-16T15:09:16.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.594+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96112", "target": ["mon-mgr", ""]}) v1 -- 0x7f6678154dd0 con 0x7f665006be90 2024-01-16T15:09:16.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.766+0000 7f6676000640 1 -- 172.21.15.43:0/2158200544 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6668009670 con 0x7f665006be90 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.769+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 msgr2=0x7f665006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.769+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 0x7f665006e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6668009a00 tx=0x7f6668008040 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.769+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 msgr2=0x7f6678147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.769+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f6660010860 tx=0x7f6660010d20 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 shutdown_connections 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f665006be90 0x7f665006e350 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66781546c0 0x7f66780b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66781519f0 0x7f6678147f20 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 --2- 172.21.15.43:0/2158200544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6658004d00 0x7f6678152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 >> 172.21.15.43:0/2158200544 conn(0x7f66780cf500 msgr2=0x7f667815e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:16.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 shutdown_connections 2024-01-16T15:09:16.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:16.770+0000 7f667f400640 1 -- 172.21.15.43:0/2158200544 wait complete. 2024-01-16T15:09:16.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96113 2024-01-16T15:09:17.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.121+0000 7fba8b000640 1 Processor -- start 2024-01-16T15:09:17.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.121+0000 7fba8b000640 1 -- start start 2024-01-16T15:09:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.127+0000 7fba8b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c0e1060 0x7fba7c038600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.128+0000 7fba8b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba7c03a5a0 0x7fba7c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.128+0000 7fba8b000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c04f510 con 0x7fba7c050da0 2024-01-16T15:09:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.128+0000 7fba8b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7c04f7f0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.128+0000 7fba8b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c04fad0 con 0x7fba7c0e1060 2024-01-16T15:09:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.129+0000 7fba89200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fba7c050da0 0x7fba7c039e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44878/0 (socket says 172.21.15.43:44878) 2024-01-16T15:09:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.129+0000 7fba89200640 1 -- 172.21.15.43:0/2105658528 learned_addr learned my addr 172.21.15.43:0/2105658528 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:17.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.130+0000 7fba89c00640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c0e1060 0x7fba7c038600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.130+0000 7fba8a600640 1 --2- 172.21.15.43:0/2105658528 >> v2:172.21.15.69:3300/0 conn(0x7fba7c03a5a0 0x7fba7c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.131+0000 7fba8a600640 1 -- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c0e1060 msgr2=0x7fba7c038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:17.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.131+0000 7fba8a600640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c0e1060 0x7fba7c038600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:17.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.131+0000 7fba8a600640 1 -- 172.21.15.43:0/2105658528 >> v1:172.21.15.43:6789/0 conn(0x7fba7c050da0 legacy=0x7fba7c039e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:17.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.133+0000 7fba8a600640 1 -- 172.21.15.43:0/2105658528 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c04fdb0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.137+0000 7fba8a600640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c03a5a0 0x7fba7c005fb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fba84063e60 tx=0x7fba840a5e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca96e634476e8753 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.138+0000 7fba83e00640 1 -- 172.21.15.43:0/2105658528 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba840a6c70 con 0x7fba7c03a5a0 2024-01-16T15:09:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.138+0000 7fba83e00640 1 -- 172.21.15.43:0/2105658528 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba840a6dd0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.138+0000 7fba83e00640 1 -- 172.21.15.43:0/2105658528 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba840ae640 con 0x7fba7c03a5a0 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 -- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c03a5a0 msgr2=0x7fba7c005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c03a5a0 0x7fba7c005fb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fba84063e60 tx=0x7fba840a5e00 comp rx=0 tx=0).stop 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 -- 172.21.15.43:0/2105658528 shutdown_connections 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c03a5a0 0x7fba7c005fb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 --2- 172.21.15.43:0/2105658528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c0e1060 0x7fba7c038600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:17.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.141+0000 7fba8b000640 1 -- 172.21.15.43:0/2105658528 >> 172.21.15.43:0/2105658528 conn(0x7fba7c009f80 msgr2=0x7fba7c00a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:17.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.144+0000 7fba8b000640 1 -- 172.21.15.43:0/2105658528 shutdown_connections 2024-01-16T15:09:17.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.144+0000 7fba8b000640 1 -- 172.21.15.43:0/2105658528 wait complete. 2024-01-16T15:09:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.150+0000 7fba8b000640 1 Processor -- start 2024-01-16T15:09:17.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.151+0000 7fba8b000640 1 -- start start 2024-01-16T15:09:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.153+0000 7fba8b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 0x7fba7c04e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.154+0000 7fba8b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 0x7fba7c0ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.154+0000 7fba8b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7c0e1060 0x7fba7c037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.155+0000 7fba8b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0de490 con 0x7fba7c0e1060 2024-01-16T15:09:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.155+0000 7fba8b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c04d630 con 0x7fba7c050da0 2024-01-16T15:09:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.155+0000 7fba8b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c04d910 con 0x7fba7c03a5a0 2024-01-16T15:09:17.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.156+0000 7fba89200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 0x7fba7c0ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.156+0000 7fba89200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 0x7fba7c0ddf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:42180/0 (socket says 172.21.15.43:42180) 2024-01-16T15:09:17.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.156+0000 7fba89200640 1 -- 172.21.15.43:0/1942240508 learned_addr learned my addr 172.21.15.43:0/1942240508 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:17.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.156+0000 7fba89c00640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 0x7fba7c04e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.156+0000 7fba8a600640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7c0e1060 0x7fba7c037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.158+0000 7fba89c00640 1 -- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 msgr2=0x7fba7c0ddf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.158+0000 7fba89c00640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 0x7fba7c0ddf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.158+0000 7fba89c00640 1 -- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7c0e1060 msgr2=0x7fba7c037fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.158+0000 7fba89c00640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7c0e1060 0x7fba7c037fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.158+0000 7fba89c00640 1 -- 172.21.15.43:0/1942240508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8404f730 con 0x7fba7c03a5a0 2024-01-16T15:09:17.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.167+0000 7fba89c00640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 0x7fba7c04e6d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba7400e870 tx=0x7fba7400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:17.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.169+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7400ee60 con 0x7fba7c03a5a0 2024-01-16T15:09:17.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.169+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7400c940 con 0x7fba7c03a5a0 2024-01-16T15:09:17.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.169+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7400ccd0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.170+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c04dbf0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.176+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c04e1b0 con 0x7fba7c03a5a0 2024-01-16T15:09:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.178+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c04e370 con 0x7fba7c03a5a0 2024-01-16T15:09:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.179+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fba74013470 con 0x7fba7c03a5a0 2024-01-16T15:09:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.182+0000 7fba81600640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 0x7fba5c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.191+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fba7400ce30 con 0x7fba7c03a5a0 2024-01-16T15:09:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.191+0000 7fba89200640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 0x7fba5c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:17.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.197+0000 7fba89200640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 0x7fba5c06e3a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0484f0 tx=0x7fba6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:17.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.199+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba74090060 con 0x7fba7c03a5a0 2024-01-16T15:09:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:17.718+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96113", "target": ["mon-mgr", ""]}) v1 -- 0x7fba7c050370 con 0x7fba5c06bee0 2024-01-16T15:09:18.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.133+0000 7fba81600640 1 -- 172.21.15.43:0/1942240508 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba6c00f7a0 con 0x7fba5c06bee0 2024-01-16T15:09:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.139+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 msgr2=0x7fba5c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.139+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 0x7fba5c06e3a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0484f0 tx=0x7fba6c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.139+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 msgr2=0x7fba7c04e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.139+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 0x7fba7c04e6d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba7400e870 tx=0x7fba7400ed30 comp rx=0 tx=0).stop 2024-01-16T15:09:18.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 shutdown_connections 2024-01-16T15:09:18.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fba5c06bee0 0x7fba5c06e3a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba7c050da0 0x7fba7c0ddf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7c0e1060 0x7fba7c037fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 --2- 172.21.15.43:0/1942240508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba7c03a5a0 0x7fba7c04e6d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 >> 172.21.15.43:0/1942240508 conn(0x7fba7c009f80 msgr2=0x7fba7c035580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 shutdown_connections 2024-01-16T15:09:18.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.140+0000 7fba8b000640 1 -- 172.21.15.43:0/1942240508 wait complete. 2024-01-16T15:09:18.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96114 2024-01-16T15:09:18.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.619+0000 7f60c7c00640 1 Processor -- start 2024-01-16T15:09:18.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.620+0000 7f60c7c00640 1 -- start start 2024-01-16T15:09:18.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.623+0000 7f60c7c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f60c006ee30 0x7f60c0168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.623+0000 7f60c7c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c0142d50 con 0x7f60c0070470 2024-01-16T15:09:18.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.623+0000 7f60c7c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c006f430 con 0x7f60c00cd150 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.623+0000 7f60c7c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c00cd530 con 0x7f60c006ee30 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.624+0000 7f60c5c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f60c006ee30 0x7f60c0168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.624+0000 7f60c5200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f60c00cd150 0x7f60c006fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:49414/0 (socket says 172.21.15.43:49414) 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.624+0000 7f60c5200640 1 -- 172.21.15.43:0/2732320376 learned_addr learned my addr 172.21.15.43:0/2732320376 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.625+0000 7f60c5c00640 1 -- 172.21.15.43:0/2732320376 >> v1:172.21.15.69:6789/0 conn(0x7f60c00cd150 legacy=0x7f60c006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.625+0000 7f60c5c00640 1 -- 172.21.15.43:0/2732320376 >> v1:172.21.15.43:6789/0 conn(0x7f60c0070470 legacy=0x7f60c006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.625+0000 7f60c5c00640 1 -- 172.21.15.43:0/2732320376 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0142fa0 con 0x7f60c006ee30 2024-01-16T15:09:18.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.628+0000 7f60c5c00640 1 --2- 172.21.15.43:0/2732320376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c006ee30 0x7f60c0168120 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009d90 tx=0x7f60b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cdefdb27479cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.632+0000 7f60bf400640 1 -- 172.21.15.43:0/2732320376 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b802cc60 con 0x7f60c006ee30 2024-01-16T15:09:18.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.632+0000 7f60bf400640 1 -- 172.21.15.43:0/2732320376 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b802cdc0 con 0x7f60c006ee30 2024-01-16T15:09:18.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.632+0000 7f60bf400640 1 -- 172.21.15.43:0/2732320376 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b8034660 con 0x7f60c006ee30 2024-01-16T15:09:18.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.634+0000 7f60c7c00640 1 -- 172.21.15.43:0/2732320376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c006ee30 msgr2=0x7f60c0168120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.634+0000 7f60c7c00640 1 --2- 172.21.15.43:0/2732320376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c006ee30 0x7f60c0168120 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009d90 tx=0x7f60b802beb0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.635+0000 7f60c7c00640 1 -- 172.21.15.43:0/2732320376 shutdown_connections 2024-01-16T15:09:18.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.635+0000 7f60c7c00640 1 --2- 172.21.15.43:0/2732320376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c006ee30 0x7f60c0168120 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.635+0000 7f60c7c00640 1 -- 172.21.15.43:0/2732320376 >> 172.21.15.43:0/2732320376 conn(0x7f60c00cdb20 msgr2=0x7f60c00cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:18.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.636+0000 7f60c7c00640 1 -- 172.21.15.43:0/2732320376 shutdown_connections 2024-01-16T15:09:18.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.636+0000 7f60c7c00640 1 -- 172.21.15.43:0/2732320376 wait complete. 2024-01-16T15:09:18.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.640+0000 7f60c7c00640 1 Processor -- start 2024-01-16T15:09:18.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.640+0000 7f60c7c00640 1 -- start start 2024-01-16T15:09:18.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c006ee30 0x7f60c00ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c0070470 0x7f60c014f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01441b0 con 0x7f60c00cd150 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c0144490 con 0x7f60c0070470 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.642+0000 7f60c7c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c0144770 con 0x7f60c006ee30 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.643+0000 7f60c5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.643+0000 7f60c5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35600/0 (socket says 172.21.15.43:35600) 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.643+0000 7f60c5c00640 1 -- 172.21.15.43:0/1729402521 learned_addr learned my addr 172.21.15.43:0/1729402521 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.644+0000 7f60c5200640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c006ee30 0x7f60c00ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.645+0000 7f60c5c00640 1 -- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c006ee30 msgr2=0x7f60c00ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.645+0000 7f60c5c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c006ee30 0x7f60c00ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.645+0000 7f60c5c00640 1 -- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c0070470 msgr2=0x7f60c014f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.645+0000 7f60c5c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c0070470 0x7f60c014f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.645+0000 7f60c5c00640 1 -- 172.21.15.43:0/1729402521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b8009a70 con 0x7f60c00cd150 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60c5c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009eb0 tx=0x7f60b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b803e610 con 0x7f60c00cd150 2024-01-16T15:09:18.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b803e770 con 0x7f60c00cd150 2024-01-16T15:09:18.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b803eb20 con 0x7f60c00cd150 2024-01-16T15:09:18.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0143c00 con 0x7f60c00cd150 2024-01-16T15:09:18.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.646+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0144d30 con 0x7f60c00cd150 2024-01-16T15:09:18.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.649+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c0144f50 con 0x7f60c00cd150 2024-01-16T15:09:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.650+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f60b803b050 con 0x7f60c00cd150 2024-01-16T15:09:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.652+0000 7f60bcc00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 0x7f609806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.652+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f60b80aaf80 con 0x7f60c00cd150 2024-01-16T15:09:18.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.655+0000 7f60c5200640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 0x7f609806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:18.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.657+0000 7f60c5200640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 0x7f609806e3a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005880 tx=0x7f60b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:18.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:18.662+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60b80779a0 con 0x7f60c00cd150 2024-01-16T15:09:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.230+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96114", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c0144600 con 0x7f609806bee0 2024-01-16T15:09:19.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.621+0000 7f60bcc00640 1 -- 172.21.15.43:0/1729402521 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f60c0144600 con 0x7f609806bee0 2024-01-16T15:09:19.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.633+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 msgr2=0x7f609806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.633+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 0x7f609806e3a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005880 tx=0x7f60b0009040 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.633+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 msgr2=0x7f60c00b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.633+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009eb0 tx=0x7f60b8004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 shutdown_connections 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f609806bee0 0x7f609806e3a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c0070470 0x7f60c014f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60c00cd150 0x7f60c00b62b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 --2- 172.21.15.43:0/1729402521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c006ee30 0x7f60c00ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 >> 172.21.15.43:0/1729402521 conn(0x7f60c00cdb20 msgr2=0x7f60c006fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.634+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 shutdown_connections 2024-01-16T15:09:19.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:19.635+0000 7f60c7c00640 1 -- 172.21.15.43:0/1729402521 wait complete. 2024-01-16T15:09:19.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96115 2024-01-16T15:09:20.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.031+0000 7f1c91000640 1 Processor -- start 2024-01-16T15:09:20.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.031+0000 7f1c91000640 1 -- start start 2024-01-16T15:09:20.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c91000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c8c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c91000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c15b730 con 0x7f1c8c15b350 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c91000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c0ac740 con 0x7f1c8c09cb30 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c91000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0ac460 con 0x7f1c8c0ae270 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c89e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1c8c15b350 0x7f1c8c0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44890/0 (socket says 172.21.15.43:44890) 2024-01-16T15:09:20.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.032+0000 7f1c89e00640 1 -- 172.21.15.43:0/2741434090 learned_addr learned my addr 172.21.15.43:0/2741434090 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594817826 0 0) 0x7f1c7c001490 con 0x7f1c8c15b350 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c8a800640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c8c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c003520 con 0x7f1c8c15b350 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224763182 0 0) 0x7f1c6c003520 con 0x7f1c8c15b350 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 msgr2=0x7f1c8c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c8c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 >> v1:172.21.15.69:6789/0 conn(0x7f1c8c09cb30 legacy=0x7f1c8c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0b2a50 con 0x7f1c8c15b350 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f1c7c001b80 con 0x7f1c8c15b350 2024-01-16T15:09:20.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 >> v1:172.21.15.43:6789/0 conn(0x7f1c8c15b350 legacy=0x7f1c8c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c6c0039a0 0x7f1c6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c6c004d00 0x7f1c6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0ac460 con 0x7f1c6c0039a0 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c00bb10 con 0x7f1c6c004d00 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.033+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c00bca0 con 0x7f1c8c0ae270 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c6c0039a0 0x7f1c6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c8a800640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c6c004d00 0x7f1c6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89400640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 -- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 msgr2=0x7f1c6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 -- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c6c004d00 msgr2=0x7f1c6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c6c004d00 0x7f1c6c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89e00640 1 -- 172.21.15.43:0/2741434090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c003520 con 0x7f1c6c0039a0 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.034+0000 7f1c89400640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c8c0ae270 0x7f1c6c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.035+0000 7f1c89e00640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c6c0039a0 0x7f1c6c003dc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00a860 tx=0x7f1c6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b101aea70702179 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c034630 con 0x7f1c6c0039a0 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c02f070 con 0x7f1c6c0039a0 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c88a00640 1 -- 172.21.15.43:0/2741434090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0349f0 con 0x7f1c6c0039a0 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 -- 172.21.15.43:0/2741434090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c6c0039a0 msgr2=0x7f1c6c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 --2- 172.21.15.43:0/2741434090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c6c0039a0 0x7f1c6c003dc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00a860 tx=0x7f1c6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c8a800640 1 -- 172.21.15.43:0/2741434090 reap_dead start 2024-01-16T15:09:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 -- 172.21.15.43:0/2741434090 shutdown_connections 2024-01-16T15:09:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 -- 172.21.15.43:0/2741434090 >> 172.21.15.43:0/2741434090 conn(0x7f1c8c0b0ba0 msgr2=0x7f1c8c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 -- 172.21.15.43:0/2741434090 shutdown_connections 2024-01-16T15:09:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.036+0000 7f1c91000640 1 -- 172.21.15.43:0/2741434090 wait complete. 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.038+0000 7f1c91000640 1 Processor -- start 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.038+0000 7f1c91000640 1 -- start start 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c6c0039a0 0x7f1c8c15d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c8c158790 0x7f1c8c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c159d60 con 0x7f1c8c158bf0 2024-01-16T15:09:20.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c159150 con 0x7f1c8c158790 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c91000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c159430 con 0x7f1c6c0039a0 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c8a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c8a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35618/0 (socket says 172.21.15.43:35618) 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.039+0000 7f1c8a800640 1 -- 172.21.15.43:0/3305490969 learned_addr learned my addr 172.21.15.43:0/3305490969 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c89e00640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c6c0039a0 0x7f1c8c15d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 -- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c6c0039a0 msgr2=0x7f1c8c15d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c6c0039a0 0x7f1c8c15d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 -- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c8c158790 msgr2=0x7f1c8c150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c8c158790 0x7f1c8c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 -- 172.21.15.43:0/3305490969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c159710 con 0x7f1c8c158bf0 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c8a800640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c072730 tx=0x7f1c8400f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.040+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84004440 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.041+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c84003610 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.041+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840039a0 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.041+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c1599f0 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.041+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c15a1f0 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.042+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c159ed0 con 0x7f1c8c158bf0 2024-01-16T15:09:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.043+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1c84003b00 con 0x7f1c8c158bf0 2024-01-16T15:09:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.043+0000 7f1c82000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 0x7f1c6406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.044+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1c84088380 con 0x7f1c8c158bf0 2024-01-16T15:09:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.044+0000 7f1c89e00640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 0x7f1c6406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.045+0000 7f1c89e00640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 0x7f1c6406e380 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c034000 tx=0x7f1c7c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.048+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c84054a90 con 0x7f1c8c158bf0 2024-01-16T15:09:20.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.270+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96115", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c8c0b3ea0 con 0x7f1c6406bec0 2024-01-16T15:09:20.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.441+0000 7f1c82000640 1 -- 172.21.15.43:0/3305490969 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1c8c0b3ea0 con 0x7f1c6406bec0 2024-01-16T15:09:20.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 msgr2=0x7f1c6406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 0x7f1c6406e380 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c034000 tx=0x7f1c7c02f000 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 msgr2=0x7f1c8c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c072730 tx=0x7f1c8400f6f0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 shutdown_connections 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c6406bec0 0x7f1c6406e380 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c8c158790 0x7f1c8c150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8c158bf0 0x7f1c8c1509a0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 --2- 172.21.15.43:0/3305490969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c6c0039a0 0x7f1c8c15d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 >> 172.21.15.43:0/3305490969 conn(0x7f1c8c0b0ba0 msgr2=0x7f1c8c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 shutdown_connections 2024-01-16T15:09:20.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.444+0000 7f1c91000640 1 -- 172.21.15.43:0/3305490969 wait complete. 2024-01-16T15:09:20.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96116 2024-01-16T15:09:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.855+0000 7f8b3bc00640 1 Processor -- start 2024-01-16T15:09:20.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.858+0000 7f8b3bc00640 1 -- start start 2024-01-16T15:09:20.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.862+0000 7f8b3bc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b340b48d0 0x7f8b340b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.862+0000 7f8b3bc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 0x7f8b34167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.862+0000 7f8b3bc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340b23e0 con 0x7f8b34160bd0 2024-01-16T15:09:20.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.862+0000 7f8b3bc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340cdc70 con 0x7f8b340b48d0 2024-01-16T15:09:20.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.862+0000 7f8b3bc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340b2870 con 0x7f8b340b3210 2024-01-16T15:09:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.866+0000 7f8b33e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b340b48d0 0x7f8b340b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.866+0000 7f8b33e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b340b48d0 0x7f8b340b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:42232/0 (socket says 172.21.15.43:42232) 2024-01-16T15:09:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.866+0000 7f8b33e00640 1 -- 172.21.15.43:0/3234785867 learned_addr learned my addr 172.21.15.43:0/3234785867 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.867+0000 7f8b39c00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 0x7f8b34167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b33400640 1 -- 172.21.15.43:0/3234785867 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909519036 0 0) 0x7f8b340b23e0 con 0x7f8b34160bd0 2024-01-16T15:09:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b33e00640 1 -- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 msgr2=0x7f8b34167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:20.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b33e00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 0x7f8b34167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b33e00640 1 -- 172.21.15.43:0/3234785867 >> v1:172.21.15.43:6789/0 conn(0x7f8b34160bd0 legacy=0x7f8b340b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b33e00640 1 -- 172.21.15.43:0/3234785867 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340d41b0 con 0x7f8b340b48d0 2024-01-16T15:09:20.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.868+0000 7f8b39c00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 0x7f8b34167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.873+0000 7f8b33e00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b340b1ea0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009870 tx=0x7f8b2402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9419fdb21c57cee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.874+0000 7f8b33400640 1 -- 172.21.15.43:0/3234785867 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24031640 con 0x7f8b340b48d0 2024-01-16T15:09:20.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.874+0000 7f8b33400640 1 -- 172.21.15.43:0/3234785867 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b240317a0 con 0x7f8b340b48d0 2024-01-16T15:09:20.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.874+0000 7f8b33400640 1 -- 172.21.15.43:0/3234785867 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24031b30 con 0x7f8b340b48d0 2024-01-16T15:09:20.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 -- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 msgr2=0x7f8b340b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b340b1ea0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009870 tx=0x7f8b2402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 -- 172.21.15.43:0/3234785867 shutdown_connections 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b340b3210 0x7f8b34167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/3234785867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b340b1ea0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 -- 172.21.15.43:0/3234785867 >> 172.21.15.43:0/3234785867 conn(0x7f8b3409c5a0 msgr2=0x7f8b340d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 -- 172.21.15.43:0/3234785867 shutdown_connections 2024-01-16T15:09:20.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.876+0000 7f8b3bc00640 1 -- 172.21.15.43:0/3234785867 wait complete. 2024-01-16T15:09:20.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.880+0000 7f8b3bc00640 1 Processor -- start 2024-01-16T15:09:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.881+0000 7f8b3bc00640 1 -- start start 2024-01-16T15:09:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.883+0000 7f8b3bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 0x7f8b341491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.883+0000 7f8b3bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b341f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.883+0000 7f8b3bc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.884+0000 7f8b3bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c6d70 con 0x7f8b340b3210 2024-01-16T15:09:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.884+0000 7f8b3bc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341496e0 con 0x7f8b340b48d0 2024-01-16T15:09:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.884+0000 7f8b3bc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341f3f90 con 0x7f8b34160bd0 2024-01-16T15:09:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.884+0000 7f8b39c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.885+0000 7f8b39c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:41464/0 (socket says 172.21.15.43:41464) 2024-01-16T15:09:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.885+0000 7f8b39c00640 1 -- 172.21.15.43:0/749674815 learned_addr learned my addr 172.21.15.43:0/749674815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.885+0000 7f8b39200640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 0x7f8b341491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.886+0000 7f8b33e00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b341f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.889+0000 7f8b39c00640 1 -- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 msgr2=0x7f8b341f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.890+0000 7f8b39c00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b341f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.890+0000 7f8b39c00640 1 -- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 msgr2=0x7f8b341491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.890+0000 7f8b39c00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 0x7f8b341491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.890+0000 7f8b39c00640 1 -- 172.21.15.43:0/749674815 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b240094d0 con 0x7f8b34160bd0 2024-01-16T15:09:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.891+0000 7f8b33e00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b341f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.891+0000 7f8b39200640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 0x7f8b341491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.891+0000 7f8b39c00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c820 tx=0x7f8b1c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.892+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c007a30 con 0x7f8b34160bd0 2024-01-16T15:09:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.892+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c007b90 con 0x7f8b34160bd0 2024-01-16T15:09:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.892+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b341f7400 con 0x7f8b34160bd0 2024-01-16T15:09:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.893+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b340c6fc0 con 0x7f8b34160bd0 2024-01-16T15:09:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.893+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00ce10 con 0x7f8b34160bd0 2024-01-16T15:09:20.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.895+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b340a1210 con 0x7f8b34160bd0 2024-01-16T15:09:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.903+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8b1c024480 con 0x7f8b34160bd0 2024-01-16T15:09:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.906+0000 7f8b30c00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 0x7f8b0c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.907+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8b1c088bc0 con 0x7f8b34160bd0 2024-01-16T15:09:20.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.909+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b1c0554c0 con 0x7f8b34160bd0 2024-01-16T15:09:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.913+0000 7f8b39200640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 0x7f8b0c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:20.915+0000 7f8b39200640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 0x7f8b0c06e3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0012e0 tx=0x7f8b2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.410+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96116", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b340b3760 con 0x7f8b0c06bee0 2024-01-16T15:09:21.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.755+0000 7f8b30c00640 1 -- 172.21.15.43:0/749674815 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8b2c001a90 con 0x7f8b0c06bee0 2024-01-16T15:09:21.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.765+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 msgr2=0x7f8b0c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.765+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 0x7f8b0c06e3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0012e0 tx=0x7f8b2c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:21.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.765+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 msgr2=0x7f8b341f65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:21.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.765+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c820 tx=0x7f8b1c00cce0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 shutdown_connections 2024-01-16T15:09:21.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8b0c06bee0 0x7f8b0c06e3a0 secure :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0012e0 tx=0x7f8b2c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:21.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b340b48d0 0x7f8b341f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b34160bd0 0x7f8b341f65c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 --2- 172.21.15.43:0/749674815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b340b3210 0x7f8b341491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:21.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.767+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 >> 172.21.15.43:0/749674815 conn(0x7f8b3409c5a0 msgr2=0x7f8b340b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:21.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.769+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 shutdown_connections 2024-01-16T15:09:21.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:21.769+0000 7f8b3bc00640 1 -- 172.21.15.43:0/749674815 wait complete. 2024-01-16T15:09:21.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96117 2024-01-16T15:09:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.274+0000 7faa10800640 1 Processor -- start 2024-01-16T15:09:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.275+0000 7faa10800640 1 -- start start 2024-01-16T15:09:22.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.283+0000 7faa10800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faa0806c380 0x7faa08164550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.283+0000 7faa10800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080b1ca0 con 0x7faa080701e0 2024-01-16T15:09:22.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.283+0000 7faa10800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080cc800 con 0x7faa080709b0 2024-01-16T15:09:22.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.283+0000 7faa10800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0806d070 con 0x7faa0806c380 2024-01-16T15:09:22.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.285+0000 7faa0d400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7faa080701e0 0x7faa08160e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44918/0 (socket says 172.21.15.43:44918) 2024-01-16T15:09:22.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.285+0000 7faa0d400640 1 -- 172.21.15.43:0/4236459869 learned_addr learned my addr 172.21.15.43:0/4236459869 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:22.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.285+0000 7faa0e800640 1 --2- 172.21.15.43:0/4236459869 >> v2:172.21.15.43:3300/0 conn(0x7faa0806c380 0x7faa08164550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.287+0000 7faa0e800640 1 -- 172.21.15.43:0/4236459869 >> v1:172.21.15.69:6789/0 conn(0x7faa080709b0 legacy=0x7faa08070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.287+0000 7faa0e800640 1 -- 172.21.15.43:0/4236459869 >> v1:172.21.15.43:6789/0 conn(0x7faa080701e0 legacy=0x7faa08160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.287+0000 7faa0e800640 1 -- 172.21.15.43:0/4236459869 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0806d350 con 0x7faa0806c380 2024-01-16T15:09:22.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.289+0000 7faa0e800640 1 --2- 172.21.15.43:0/4236459869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0806c380 0x7faa08164550 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7faa00009d90 tx=0x7faa0002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ca4b8cdd25714f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.290+0000 7faa0ca00640 1 -- 172.21.15.43:0/4236459869 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0002ccf0 con 0x7faa0806c380 2024-01-16T15:09:22.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.290+0000 7faa0ca00640 1 -- 172.21.15.43:0/4236459869 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0002ce50 con 0x7faa0806c380 2024-01-16T15:09:22.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.290+0000 7faa0ca00640 1 -- 172.21.15.43:0/4236459869 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa00034660 con 0x7faa0806c380 2024-01-16T15:09:22.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.293+0000 7faa10800640 1 -- 172.21.15.43:0/4236459869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0806c380 msgr2=0x7faa08164550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.293+0000 7faa10800640 1 --2- 172.21.15.43:0/4236459869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0806c380 0x7faa08164550 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7faa00009d90 tx=0x7faa0002bf40 comp rx=0 tx=0).stop 2024-01-16T15:09:22.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.294+0000 7faa10800640 1 -- 172.21.15.43:0/4236459869 shutdown_connections 2024-01-16T15:09:22.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.294+0000 7faa10800640 1 --2- 172.21.15.43:0/4236459869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0806c380 0x7faa08164550 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.294+0000 7faa10800640 1 -- 172.21.15.43:0/4236459869 >> 172.21.15.43:0/4236459869 conn(0x7faa080cbd20 msgr2=0x7faa080b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:22.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.296+0000 7faa10800640 1 -- 172.21.15.43:0/4236459869 shutdown_connections 2024-01-16T15:09:22.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.297+0000 7faa10800640 1 -- 172.21.15.43:0/4236459869 wait complete. 2024-01-16T15:09:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.300+0000 7faa10800640 1 Processor -- start 2024-01-16T15:09:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.300+0000 7faa10800640 1 -- start start 2024-01-16T15:09:22.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.304+0000 7faa10800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 0x7faa08145380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.304+0000 7faa10800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 0x7faa08155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.306+0000 7faa10800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 0x7faa08150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.306+0000 7faa0de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 0x7faa08145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.306+0000 7faa0d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 0x7faa08155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.307+0000 7faa0d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 0x7faa08155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:42254/0 (socket says 172.21.15.43:42254) 2024-01-16T15:09:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.307+0000 7faa0d400640 1 -- 172.21.15.43:0/2612088745 learned_addr learned my addr 172.21.15.43:0/2612088745 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.308+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08152a30 con 0x7faa080709b0 2024-01-16T15:09:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.308+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081465f0 con 0x7faa080701e0 2024-01-16T15:09:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.308+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa08145970 con 0x7faa0806c380 2024-01-16T15:09:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.311+0000 7faa0e800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 0x7faa08150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0de00640 1 -- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 msgr2=0x7faa08155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0de00640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 0x7faa08155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0de00640 1 -- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 msgr2=0x7faa08150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0de00640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 0x7faa08150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0de00640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00009a70 con 0x7faa0806c380 2024-01-16T15:09:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.312+0000 7faa0e800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 0x7faa08150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.314+0000 7faa0de00640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 0x7faa08145380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c3c0 tx=0x7fa9f000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.316+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00052d0 con 0x7faa0806c380 2024-01-16T15:09:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.317+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f0005450 con 0x7faa0806c380 2024-01-16T15:09:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.317+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08145c50 con 0x7faa0806c380 2024-01-16T15:09:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.317+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0003830 con 0x7faa0806c380 2024-01-16T15:09:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.318+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08156450 con 0x7faa0806c380 2024-01-16T15:09:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.319+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0804e840 con 0x7faa0806c380 2024-01-16T15:09:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.322+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa9f0003990 con 0x7faa0806c380 2024-01-16T15:09:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.323+0000 7faa06000640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 0x7fa9e006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:22.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.323+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa9f00880e0 con 0x7faa0806c380 2024-01-16T15:09:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.326+0000 7faa0d400640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 0x7fa9e006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.327+0000 7faa0d400640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 0x7fa9e006e3a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faa08155c30 tx=0x7fa9f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.333+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9f0054840 con 0x7faa0806c380 2024-01-16T15:09:22.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:22.864+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96117", "target": ["mon-mgr", ""]}) v1 -- 0x7faa08159620 con 0x7fa9e006bee0 2024-01-16T15:09:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.375+0000 7faa06000640 1 -- 172.21.15.43:0/2612088745 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa9f8001cd0 con 0x7fa9e006bee0 2024-01-16T15:09:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 msgr2=0x7fa9e006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 0x7fa9e006e3a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faa08155c30 tx=0x7fa9f8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 msgr2=0x7faa08145380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 0x7faa08145380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c3c0 tx=0x7fa9f000c880 comp rx=0 tx=0).stop 2024-01-16T15:09:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 shutdown_connections 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e006bee0 0x7fa9e006e3a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa080701e0 0x7faa08155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa080709b0 0x7faa08150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 --2- 172.21.15.43:0/2612088745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0806c380 0x7faa08145380 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.382+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 >> 172.21.15.43:0/2612088745 conn(0x7faa080cbd20 msgr2=0x7faa080b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.383+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 shutdown_connections 2024-01-16T15:09:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.383+0000 7faa10800640 1 -- 172.21.15.43:0/2612088745 wait complete. 2024-01-16T15:09:23.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96118 2024-01-16T15:09:23.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.655+0000 7f306b200640 1 Processor -- start 2024-01-16T15:09:23.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.655+0000 7f306b200640 1 -- start start 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f306415d3b0 0x7f30640ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30640d0fa0 0x7f30640d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f30640b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30640c6bf0 con 0x7f30640d0fa0 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30640d1ff0 con 0x7f306415d3b0 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30640d1430 con 0x7f30640d1bc0 2024-01-16T15:09:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f3069400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30640d0fa0 0x7f30640d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f3069400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30640d0fa0 0x7f30640d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37024/0 (socket says 172.21.15.43:37024) 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f3069400640 1 -- 172.21.15.43:0/158880671 learned_addr learned my addr 172.21.15.43:0/158880671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f3069e00640 1 --2- 172.21.15.43:0/158880671 >> v2:172.21.15.69:3300/0 conn(0x7f306415d3b0 0x7f30640ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.657+0000 7f306a800640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f30640b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.658+0000 7f3069400640 1 -- 172.21.15.43:0/158880671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 msgr2=0x7f30640b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.658+0000 7f3069400640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f30640b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.658+0000 7f3069400640 1 -- 172.21.15.43:0/158880671 >> v2:172.21.15.69:3300/0 conn(0x7f306415d3b0 msgr2=0x7f30640ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.658+0000 7f3069400640 1 --2- 172.21.15.43:0/158880671 >> v2:172.21.15.69:3300/0 conn(0x7f306415d3b0 0x7f30640ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.658+0000 7f3069400640 1 -- 172.21.15.43:0/158880671 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640cedc0 con 0x7f30640d0fa0 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.659+0000 7f3069400640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30640d0fa0 0x7f30640d1680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f306408b2a0 tx=0x7f304c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04a27b0c7c17c20 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.659+0000 7f3068a00640 1 -- 172.21.15.43:0/158880671 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c02c9e0 con 0x7f30640d0fa0 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.659+0000 7f3068a00640 1 -- 172.21.15.43:0/158880671 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c02cb40 con 0x7f30640d0fa0 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.659+0000 7f3068a00640 1 -- 172.21.15.43:0/158880671 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c0343f0 con 0x7f30640d0fa0 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 -- 172.21.15.43:0/158880671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30640d0fa0 msgr2=0x7f30640d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30640d0fa0 0x7f30640d1680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f306408b2a0 tx=0x7f304c02be40 comp rx=0 tx=0).stop 2024-01-16T15:09:23.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 -- 172.21.15.43:0/158880671 shutdown_connections 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f30640b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 --2- 172.21.15.43:0/158880671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30640d0fa0 0x7f30640d1680 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 --2- 172.21.15.43:0/158880671 >> v2:172.21.15.69:3300/0 conn(0x7f306415d3b0 0x7f30640ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 -- 172.21.15.43:0/158880671 >> 172.21.15.43:0/158880671 conn(0x7f30640d30a0 msgr2=0x7f30640d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 -- 172.21.15.43:0/158880671 shutdown_connections 2024-01-16T15:09:23.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.660+0000 7f306b200640 1 -- 172.21.15.43:0/158880671 wait complete. 2024-01-16T15:09:23.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.662+0000 7f306b200640 1 Processor -- start 2024-01-16T15:09:23.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.662+0000 7f306b200640 1 -- start start 2024-01-16T15:09:23.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 0x7f306415e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f306415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306414bc10 con 0x7f306415d3b0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306414b020 con 0x7f30640d0fa0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.663+0000 7f306b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306414b2b0 con 0x7f30640d1bc0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37038/0 (socket says 172.21.15.43:37038) 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 -- 172.21.15.43:0/2245118675 learned_addr learned my addr 172.21.15.43:0/2245118675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f3069400640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f306415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f3069e00640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 0x7f306415e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 -- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 msgr2=0x7f306415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f306415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 -- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 msgr2=0x7f306415e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 0x7f306415e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f306a800640 1 -- 172.21.15.43:0/2245118675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c009b30 con 0x7f306415d3b0 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.664+0000 7f3069e00640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 0x7f306415e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f306a800640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f305400d5d0 tx=0x7f305400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054014620 con 0x7f306415d3b0 2024-01-16T15:09:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30540147a0 con 0x7f306415d3b0 2024-01-16T15:09:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306414e150 con 0x7f306415d3b0 2024-01-16T15:09:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054014b50 con 0x7f306415d3b0 2024-01-16T15:09:23.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.665+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306414b830 con 0x7f306415d3b0 2024-01-16T15:09:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.666+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306414bd50 con 0x7f306415d3b0 2024-01-16T15:09:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.667+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3054014cb0 con 0x7f306415d3b0 2024-01-16T15:09:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.668+0000 7f3062000640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 0x7f303c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.668+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3054088e30 con 0x7f306415d3b0 2024-01-16T15:09:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.669+0000 7f3069e00640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 0x7f303c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.669+0000 7f3069e00640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 0x7f303c06e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.673+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f306414bd50 con 0x7f306415d3b0 2024-01-16T15:09:23.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:23.906+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96118", "target": ["mon-mgr", ""]}) v1 -- 0x7f30640c7400 con 0x7f303c06be90 2024-01-16T15:09:24.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.867+0000 7f3062000640 1 -- 172.21.15.43:0/2245118675 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f305c007680 con 0x7f303c06be90 2024-01-16T15:09:24.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.870+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 msgr2=0x7f303c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.870+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 0x7f303c06e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).stop 2024-01-16T15:09:24.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.870+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 msgr2=0x7f306414d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.870+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f305400d5d0 tx=0x7f305400da90 comp rx=0 tx=0).stop 2024-01-16T15:09:24.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 shutdown_connections 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f303c06be90 0x7f303c06e350 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f306415d3b0 0x7f306414d3f0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30640d1bc0 0x7f306415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 --2- 172.21.15.43:0/2245118675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30640d0fa0 0x7f306415e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 >> 172.21.15.43:0/2245118675 conn(0x7f30640d30a0 msgr2=0x7f306406af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 shutdown_connections 2024-01-16T15:09:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:24.871+0000 7f306b200640 1 -- 172.21.15.43:0/2245118675 wait complete. 2024-01-16T15:09:24.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96119 2024-01-16T15:09:25.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.118+0000 7fb0a2000640 1 Processor -- start 2024-01-16T15:09:25.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.118+0000 7fb0a2000640 1 -- start start 2024-01-16T15:09:25.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb0a2000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb09c162600 0x7fb09c162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb0a2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c06c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb0a2000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c162fa0 con 0x7fb09c162220 2024-01-16T15:09:25.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb0a2000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09c0706a0 con 0x7fb09c162600 2024-01-16T15:09:25.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb0a2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c070c60 con 0x7fb09c06c3c0 2024-01-16T15:09:25.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb09b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb09c162220 0x7fb09c0b1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48048/0 (socket says 172.21.15.43:48048) 2024-01-16T15:09:25.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb09b400640 1 -- 172.21.15.43:0/1520669753 learned_addr learned my addr 172.21.15.43:0/1520669753 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:25.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb09aa00640 1 --2- 172.21.15.43:0/1520669753 >> v2:172.21.15.43:3300/0 conn(0x7fb09c162600 0x7fb09c162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.120+0000 7fb09be00640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c06c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09a000640 1 -- 172.21.15.43:0/1520669753 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040839905 0 0) 0x7fb09c162fa0 con 0x7fb09c162220 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09a000640 1 -- 172.21.15.43:0/1520669753 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c003520 con 0x7fb09c162220 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09aa00640 1 -- 172.21.15.43:0/1520669753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 msgr2=0x7fb09c06c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09aa00640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c06c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09aa00640 1 -- 172.21.15.43:0/1520669753 >> v1:172.21.15.43:6789/0 conn(0x7fb09c162220 legacy=0x7fb09c0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.121+0000 7fb09aa00640 1 -- 172.21.15.43:0/1520669753 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c06cd60 con 0x7fb09c162600 2024-01-16T15:09:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.122+0000 7fb09aa00640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c162a60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb0840097a0 tx=0x7fb08402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b5e54beeeb8c89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.122+0000 7fb09a000640 1 -- 172.21.15.43:0/1520669753 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084031830 con 0x7fb09c162600 2024-01-16T15:09:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.122+0000 7fb09a000640 1 -- 172.21.15.43:0/1520669753 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084031990 con 0x7fb09c162600 2024-01-16T15:09:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.122+0000 7fb09a000640 1 -- 172.21.15.43:0/1520669753 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084031d00 con 0x7fb09c162600 2024-01-16T15:09:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 -- 172.21.15.43:0/1520669753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 msgr2=0x7fb09c162a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c162a60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb0840097a0 tx=0x7fb08402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 -- 172.21.15.43:0/1520669753 shutdown_connections 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c06c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 --2- 172.21.15.43:0/1520669753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c162a60 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 -- 172.21.15.43:0/1520669753 >> 172.21.15.43:0/1520669753 conn(0x7fb09c0b0ba0 msgr2=0x7fb09c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 -- 172.21.15.43:0/1520669753 shutdown_connections 2024-01-16T15:09:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.123+0000 7fb0a2000640 1 -- 172.21.15.43:0/1520669753 wait complete. 2024-01-16T15:09:25.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.125+0000 7fb0a2000640 1 Processor -- start 2024-01-16T15:09:25.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.125+0000 7fb0a2000640 1 -- start start 2024-01-16T15:09:25.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c0b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 0x7fb09c152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c164f20 con 0x7fb09c162600 2024-01-16T15:09:25.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c164650 con 0x7fb09c162220 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb0a2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c09bbf0 con 0x7fb09c06c3c0 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb09be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb09be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37044/0 (socket says 172.21.15.43:37044) 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.126+0000 7fb09be00640 1 -- 172.21.15.43:0/3008604468 learned_addr learned my addr 172.21.15.43:0/3008604468 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09b400640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c0b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09aa00640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 0x7fb09c152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 -- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 msgr2=0x7fb09c0b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c0b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 -- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 msgr2=0x7fb09c152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 0x7fb09c152c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 -- 172.21.15.43:0/3008604468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084009400 con 0x7fb09c162600 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09b400640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c0b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09aa00640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 0x7fb09c152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:25.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.127+0000 7fb09be00640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb09400e7f0 tx=0x7fb09400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.128+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094009510 con 0x7fb09c162600 2024-01-16T15:09:25.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.128+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09400ee40 con 0x7fb09c162600 2024-01-16T15:09:25.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.128+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09400cb80 con 0x7fb09c162600 2024-01-16T15:09:25.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.128+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c1531d0 con 0x7fb09c162600 2024-01-16T15:09:25.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.128+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c09b910 con 0x7fb09c162600 2024-01-16T15:09:25.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.129+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c165060 con 0x7fb09c162600 2024-01-16T15:09:25.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.130+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb09400cce0 con 0x7fb09c162600 2024-01-16T15:09:25.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.131+0000 7fb093400640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 0x7fb07406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.131+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb094088910 con 0x7fb09c162600 2024-01-16T15:09:25.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.132+0000 7fb09b400640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 0x7fb07406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.132+0000 7fb09b400640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 0x7fb07406e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb08c005c00 tx=0x7fb08c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.136+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb09c165060 con 0x7fb09c162600 2024-01-16T15:09:25.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.358+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96119", "target": ["mon-mgr", ""]}) v1 -- 0x7fb09c0708f0 con 0x7fb07406be90 2024-01-16T15:09:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.520+0000 7fb093400640 1 -- 172.21.15.43:0/3008604468 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb08c001de0 con 0x7fb07406be90 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 msgr2=0x7fb07406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 0x7fb07406e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb08c005c00 tx=0x7fb08c009040 comp rx=0 tx=0).stop 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 msgr2=0x7fb09c09b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb09400e7f0 tx=0x7fb09400ebf0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 shutdown_connections 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb07406be90 0x7fb07406e350 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb09c162220 0x7fb09c152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c162600 0x7fb09c09b130 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 --2- 172.21.15.43:0/3008604468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb09c06c3c0 0x7fb09c0b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.523+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 >> 172.21.15.43:0/3008604468 conn(0x7fb09c0b0ba0 msgr2=0x7fb09c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.524+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 shutdown_connections 2024-01-16T15:09:25.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.524+0000 7fb0a2000640 1 -- 172.21.15.43:0/3008604468 wait complete. 2024-01-16T15:09:25.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone96120 2024-01-16T15:09:25.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.775+0000 7f76dca00640 1 Processor -- start 2024-01-16T15:09:25.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.775+0000 7f76dca00640 1 -- start start 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.776+0000 7f76dca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76d4160bd0 0x7f76d40b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76dca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d40b48d0 0x7f76d40b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76dca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f76d40b3040 0x7f76d4166280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76dca00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d40cdc70 con 0x7f76d4160bd0 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76dca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d40b2870 con 0x7f76d40b3040 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76dca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40b23e0 con 0x7f76d40b48d0 2024-01-16T15:09:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76da000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76d4160bd0 0x7f76d40b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76da000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76d4160bd0 0x7f76d40b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37046/0 (socket says 172.21.15.43:37046) 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76da000640 1 -- 172.21.15.43:0/261854242 learned_addr learned my addr 172.21.15.43:0/261854242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76d9600640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d40b48d0 0x7f76d40b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.777+0000 7f76daa00640 1 --2- 172.21.15.43:0/261854242 >> v2:172.21.15.69:3300/0 conn(0x7f76d40b3040 0x7f76d4166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.778+0000 7f76da000640 1 -- 172.21.15.43:0/261854242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d40b48d0 msgr2=0x7f76d40b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.778+0000 7f76da000640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d40b48d0 0x7f76d40b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.778+0000 7f76da000640 1 -- 172.21.15.43:0/261854242 >> v2:172.21.15.69:3300/0 conn(0x7f76d40b3040 msgr2=0x7f76d4166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.778+0000 7f76da000640 1 --2- 172.21.15.43:0/261854242 >> v2:172.21.15.69:3300/0 conn(0x7f76d40b3040 0x7f76d4166280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.778+0000 7f76da000640 1 -- 172.21.15.43:0/261854242 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40d41b0 con 0x7f76d4160bd0 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.779+0000 7f76da000640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d4160bd0 0x7f76d40b1ea0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f76d40d08d0 tx=0x7f76cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec6aa878eb351b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.779+0000 7f76d8c00640 1 -- 172.21.15.43:0/261854242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc02c9e0 con 0x7f76d4160bd0 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.779+0000 7f76d8c00640 1 -- 172.21.15.43:0/261854242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc02cb40 con 0x7f76d4160bd0 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.779+0000 7f76d8c00640 1 -- 172.21.15.43:0/261854242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0343f0 con 0x7f76d4160bd0 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 -- 172.21.15.43:0/261854242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d4160bd0 msgr2=0x7f76d40b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d4160bd0 0x7f76d40b1ea0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f76d40d08d0 tx=0x7f76cc02be40 comp rx=0 tx=0).stop 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 -- 172.21.15.43:0/261854242 shutdown_connections 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 --2- 172.21.15.43:0/261854242 >> v2:172.21.15.69:3300/0 conn(0x7f76d40b3040 0x7f76d4166280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d40b48d0 0x7f76d40b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 --2- 172.21.15.43:0/261854242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d4160bd0 0x7f76d40b1ea0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 -- 172.21.15.43:0/261854242 >> 172.21.15.43:0/261854242 conn(0x7f76d409c5a0 msgr2=0x7f76d40d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 -- 172.21.15.43:0/261854242 shutdown_connections 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.780+0000 7f76dca00640 1 -- 172.21.15.43:0/261854242 wait complete. 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.782+0000 7f76dca00640 1 Processor -- start 2024-01-16T15:09:25.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.782+0000 7f76dca00640 1 -- start start 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 0x7f76d40b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d4160bd0 0x7f76d406c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c6d70 con 0x7f76d40b3040 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c7bd0 con 0x7f76d40b48d0 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76dca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c6fc0 con 0x7f76d4160bd0 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76da000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76da000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37054/0 (socket says 172.21.15.43:37054) 2024-01-16T15:09:25.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.783+0000 7f76da000640 1 -- 172.21.15.43:0/1409299655 learned_addr learned my addr 172.21.15.43:0/1409299655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76d9600640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 0x7f76d40b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76daa00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d4160bd0 0x7f76d406c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 -- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d4160bd0 msgr2=0x7f76d406c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d4160bd0 0x7f76d406c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 -- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 msgr2=0x7f76d40b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 0x7f76d40b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 -- 172.21.15.43:0/1409299655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc009b30 con 0x7f76d40b3040 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76d9600640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 0x7f76d40b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76da000640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009f40 tx=0x7f76cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc009510 con 0x7f76d40b3040 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc03e450 con 0x7f76d40b3040 2024-01-16T15:09:25.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc03e800 con 0x7f76d40b3040 2024-01-16T15:09:25.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.784+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d40c72a0 con 0x7f76d40b3040 2024-01-16T15:09:25.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.785+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d40c7860 con 0x7f76d40b3040 2024-01-16T15:09:25.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.786+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d40a1210 con 0x7f76d40b3040 2024-01-16T15:09:25.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.787+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f76cc03e960 con 0x7f76d40b3040 2024-01-16T15:09:25.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.788+0000 7f76d2000640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 0x7f76ac06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.788+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f76cc0ab3a0 con 0x7f76d40b3040 2024-01-16T15:09:25.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.788+0000 7f76d9600640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 0x7f76ac06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.789+0000 7f76d9600640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 0x7f76ac06e3a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f76d40c8110 tx=0x7f76bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:25.792+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76cc077d10 con 0x7f76d40b3040 2024-01-16T15:09:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.024+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone96120", "target": ["mon-mgr", ""]}) v1 -- 0x7f76d40b3760 con 0x7f76ac06bee0 2024-01-16T15:09:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.213+0000 7f76d2000640 1 -- 172.21.15.43:0/1409299655 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76bc0095d0 con 0x7f76ac06bee0 2024-01-16T15:09:26.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 msgr2=0x7f76ac06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 0x7f76ac06e3a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f76d40c8110 tx=0x7f76bc008040 comp rx=0 tx=0).stop 2024-01-16T15:09:26.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 msgr2=0x7f76d41527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009f40 tx=0x7f76cc004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 shutdown_connections 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f76ac06bee0 0x7f76ac06e3a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d40b48d0 0x7f76d40b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76d4160bd0 0x7f76d406c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 --2- 172.21.15.43:0/1409299655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76d40b3040 0x7f76d41527d0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 >> 172.21.15.43:0/1409299655 conn(0x7f76d409c5a0 msgr2=0x7f76d40ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 shutdown_connections 2024-01-16T15:09:26.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.217+0000 7f76dca00640 1 -- 172.21.15.43:0/1409299655 wait complete. 2024-01-16T15:09:26.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.452+0000 7fbd99000640 1 Processor -- start 2024-01-16T15:09:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.452+0000 7fbd99000640 1 -- start start 2024-01-16T15:09:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd9406d520 0x7fbd9406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbd940ad720 0x7fbd9415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd940abff0 0x7fbd9415f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940b41a0 con 0x7fbd9406d520 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9415c360 con 0x7fbd940ad720 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd99000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9406de40 con 0x7fbd940abff0 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd91e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd9406d520 0x7fbd9406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd91e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbd9406d520 0x7fbd9406d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37062/0 (socket says 172.21.15.43:37062) 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.454+0000 7fbd91e00640 1 -- 172.21.15.43:0/1840052453 learned_addr learned my addr 172.21.15.43:0/1840052453 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd89200640 1 --2- 172.21.15.43:0/1840052453 >> v2:172.21.15.69:3300/0 conn(0x7fbd940ad720 0x7fbd9415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd92800640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd940abff0 0x7fbd9415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd91e00640 1 -- 172.21.15.43:0/1840052453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd940abff0 msgr2=0x7fbd9415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd91e00640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd940abff0 0x7fbd9415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd91e00640 1 -- 172.21.15.43:0/1840052453 >> v2:172.21.15.69:3300/0 conn(0x7fbd940ad720 msgr2=0x7fbd9415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd91e00640 1 --2- 172.21.15.43:0/1840052453 >> v2:172.21.15.69:3300/0 conn(0x7fbd940ad720 0x7fbd9415ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.455+0000 7fbd91e00640 1 -- 172.21.15.43:0/1840052453 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9415ba30 con 0x7fbd9406d520 2024-01-16T15:09:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd91e00640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd9406d520 0x7fbd9406d900 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fbd940adb00 tx=0x7fbd8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e868d988d5b4eba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd91400640 1 -- 172.21.15.43:0/1840052453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84031870 con 0x7fbd9406d520 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd91400640 1 -- 172.21.15.43:0/1840052453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840319d0 con 0x7fbd9406d520 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd91400640 1 -- 172.21.15.43:0/1840052453 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84031d60 con 0x7fbd9406d520 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd99000640 1 -- 172.21.15.43:0/1840052453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd9406d520 msgr2=0x7fbd9406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.457+0000 7fbd99000640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd9406d520 0x7fbd9406d900 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fbd940adb00 tx=0x7fbd8402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 -- 172.21.15.43:0/1840052453 shutdown_connections 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd940abff0 0x7fbd9415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 --2- 172.21.15.43:0/1840052453 >> v2:172.21.15.69:3300/0 conn(0x7fbd940ad720 0x7fbd9415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 --2- 172.21.15.43:0/1840052453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd9406d520 0x7fbd9406d900 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 -- 172.21.15.43:0/1840052453 >> 172.21.15.43:0/1840052453 conn(0x7fbd940b0d80 msgr2=0x7fbd940b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 -- 172.21.15.43:0/1840052453 shutdown_connections 2024-01-16T15:09:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.458+0000 7fbd99000640 1 -- 172.21.15.43:0/1840052453 wait complete. 2024-01-16T15:09:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.459+0000 7fbd99000640 1 Processor -- start 2024-01-16T15:09:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.460+0000 7fbd99000640 1 -- start start 2024-01-16T15:09:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.460+0000 7fbd99000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9406d520 0x7fbd94071770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd99000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd99000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd940ad720 0x7fbd940cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd99000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9409bad0 con 0x7fbd940abff0 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd99000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9409aaa0 con 0x7fbd940ad720 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd99000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940ccbc0 con 0x7fbd9406d520 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37072/0 (socket says 172.21.15.43:37072) 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 -- 172.21.15.43:0/629572822 learned_addr learned my addr 172.21.15.43:0/629572822 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd91e00640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9406d520 0x7fbd94071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd92800640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd940ad720 0x7fbd940cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 -- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9406d520 msgr2=0x7fbd94071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9406d520 0x7fbd94071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 -- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd940ad720 msgr2=0x7fbd940cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd940ad720 0x7fbd940cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.461+0000 7fbd89200640 1 -- 172.21.15.43:0/629572822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840094d0 con 0x7fbd940abff0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd89200640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00d5d0 tx=0x7fbd7c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c004240 con 0x7fbd940abff0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c0043a0 con 0x7fbd940abff0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c003830 con 0x7fbd940abff0 2024-01-16T15:09:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9409ad10 con 0x7fbd940abff0 2024-01-16T15:09:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.462+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9409b2d0 con 0x7fbd940abff0 2024-01-16T15:09:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.463+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd94074770 con 0x7fbd940abff0 2024-01-16T15:09:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.464+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbd7c003990 con 0x7fbd940abff0 2024-01-16T15:09:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.466+0000 7fbd8aa00640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 0x7fbd6c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.466+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd7c088a80 con 0x7fbd940abff0 2024-01-16T15:09:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.466+0000 7fbd91e00640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 0x7fbd6c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.467+0000 7fbd91e00640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 0x7fbd6c06e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbd84036000 tx=0x7fbd8403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.470+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd7c054fb0 con 0x7fbd940abff0 2024-01-16T15:09:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.737+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbd94165c10 con 0x7fbd940abff0 2024-01-16T15:09:26.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.738+0000 7fbd8aa00640 1 -- 172.21.15.43:0/629572822 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fbd7c0585d0 con 0x7fbd940abff0 2024-01-16T15:09:26.715 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:09:26.716 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 619, 2024-01-16T15:09:26.717 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 74, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:09:26.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:26.719 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 2860, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 11812971192, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 23684747264, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 749409366016, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 185866885, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 192889488, 2024-01-16T15:09:26.720 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 44, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 55 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 47, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 5, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 5, 2024-01-16T15:09:26.721 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 6085 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:09:26.722 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:08:29.033645+0000", 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:09:26.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": { 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "44e0c2fa-b792-4928-bc26-8f8f506ffd4c": { 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "4 ongoing cloning jobs; Avg progress = 83.335% (39s)\n [=======================.....] (remaining: 7s)", 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.83335000276565552, 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "de3e53dc-d45b-4dbd-a863-2e0d00712363": { 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "message": "17 ongoing+pending cloning jobs; Avg progress = 19.608% (39s)\n [=====.......................] (remaining: 2m)", 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "progress": 0.19608235359191895, 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: "add_to_ceph_s": true 2024-01-16T15:09:26.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:26.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:09:26.725 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:09:26.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.740+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 msgr2=0x7fbd6c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 0x7fbd6c06e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbd84036000 tx=0x7fbd8403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 msgr2=0x7fbd940cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00d5d0 tx=0x7fbd7c00da90 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 shutdown_connections 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbd6c06be90 0x7fbd6c06e350 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd940ad720 0x7fbd940cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd940abff0 0x7fbd940cd4e0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 --2- 172.21.15.43:0/629572822 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd9406d520 0x7fbd94071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 >> 172.21.15.43:0/629572822 conn(0x7fbd940b0d80 msgr2=0x7fbd9415ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 shutdown_connections 2024-01-16T15:09:26.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.741+0000 7fbd99000640 1 -- 172.21.15.43:0/629572822 wait complete. 2024-01-16T15:09:26.738 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-16T15:09:26.738 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.19608235359191895; onpen_p = 0.2857142984867096 2024-01-16T15:09:26.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9611 2024-01-16T15:09:26.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.986+0000 7f5c7a600640 1 Processor -- start 2024-01-16T15:09:26.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.986+0000 7f5c7a600640 1 -- start start 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.988+0000 7f5c7a600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5c7415b350 0x7f5c740abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.988+0000 7f5c7a600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7415b730 con 0x7f5c740adb60 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.988+0000 7f5c7a600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c740ac740 con 0x7f5c7409cb30 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.988+0000 7f5c7a600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740ac460 con 0x7f5c7415b350 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5c7415b350 0x7f5c740abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5c7415b350 0x7f5c740abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37074/0 (socket says 172.21.15.43:37074) 2024-01-16T15:09:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 -- 172.21.15.43:0/3917750280 learned_addr learned my addr 172.21.15.43:0/3917750280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:26.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 -- 172.21.15.43:0/3917750280 >> v1:172.21.15.69:6789/0 conn(0x7f5c7409cb30 legacy=0x7f5c740b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 -- 172.21.15.43:0/3917750280 >> v1:172.21.15.43:6789/0 conn(0x7f5c740adb60 legacy=0x7f5c740adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.989+0000 7f5c73400640 1 -- 172.21.15.43:0/3917750280 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740b2a50 con 0x7f5c7415b350 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.990+0000 7f5c73400640 1 --2- 172.21.15.43:0/3917750280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c740abe80 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009870 tx=0x7f5c6402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c43ea8f4caa4f29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.991+0000 7f5c72a00640 1 -- 172.21.15.43:0/3917750280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6402f850 con 0x7f5c7415b350 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.991+0000 7f5c72a00640 1 -- 172.21.15.43:0/3917750280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6402f9b0 con 0x7f5c7415b350 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.991+0000 7f5c72a00640 1 -- 172.21.15.43:0/3917750280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6402fd40 con 0x7f5c7415b350 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.991+0000 7f5c7a600640 1 -- 172.21.15.43:0/3917750280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 msgr2=0x7f5c740abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.991+0000 7f5c7a600640 1 --2- 172.21.15.43:0/3917750280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c740abe80 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009870 tx=0x7f5c6402ba30 comp rx=0 tx=0).stop 2024-01-16T15:09:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.992+0000 7f5c7a600640 1 -- 172.21.15.43:0/3917750280 shutdown_connections 2024-01-16T15:09:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.992+0000 7f5c7a600640 1 --2- 172.21.15.43:0/3917750280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c740abe80 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.992+0000 7f5c7a600640 1 -- 172.21.15.43:0/3917750280 >> 172.21.15.43:0/3917750280 conn(0x7f5c740b0ba0 msgr2=0x7f5c740b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.992+0000 7f5c7a600640 1 -- 172.21.15.43:0/3917750280 shutdown_connections 2024-01-16T15:09:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.992+0000 7f5c7a600640 1 -- 172.21.15.43:0/3917750280 wait complete. 2024-01-16T15:09:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.993+0000 7f5c7a600640 1 Processor -- start 2024-01-16T15:09:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.994+0000 7f5c7a600640 1 -- start start 2024-01-16T15:09:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.994+0000 7f5c7a600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c7409cb30 0x7f5c74142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.994+0000 7f5c7a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c740adb60 0x7f5c7414c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.994+0000 7f5c7a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c7414dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c7a600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414d240 con 0x7f5c7415b350 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c7a600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414d520 con 0x7f5c740adb60 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c7a600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7415b970 con 0x7f5c7409cb30 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c7414dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c7414dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37078/0 (socket says 172.21.15.43:37078) 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 -- 172.21.15.43:0/140552186 learned_addr learned my addr 172.21.15.43:0/140552186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73400640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c7409cb30 0x7f5c74142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c6be00640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c740adb60 0x7f5c7414c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 -- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c7409cb30 msgr2=0x7f5c74142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c7409cb30 0x7f5c74142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 -- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c740adb60 msgr2=0x7f5c7414c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c740adb60 0x7f5c7414c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.995+0000 7f5c73e00640 1 -- 172.21.15.43:0/140552186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640094d0 con 0x7f5c7415b350 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c73e00640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c7415b350 0x7f5c7414dff0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00c7a0 tx=0x7f5c6c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c003b70 con 0x7f5c7415b350 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c003cf0 con 0x7f5c7415b350 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c003ea0 con 0x7f5c7415b350 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c7a600640 1 -- 172.21.15.43:0/140552186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7415bc50 con 0x7f5c7415b350 2024-01-16T15:09:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.996+0000 7f5c7a600640 1 -- 172.21.15.43:0/140552186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7414d800 con 0x7f5c7415b350 2024-01-16T15:09:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.997+0000 7f5c7a600640 1 -- 172.21.15.43:0/140552186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7415be70 con 0x7f5c7415b350 2024-01-16T15:09:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:26.999+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5c6c00aa90 con 0x7f5c7415b350 2024-01-16T15:09:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.000+0000 7f5c6b400640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5c4c06be90 0x7f5c4c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.000+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5c6c088ff0 con 0x7f5c7415b350 2024-01-16T15:09:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.000+0000 7f5c73400640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5c4c06be90 0x7f5c4c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.001+0000 7f5c73400640 1 --2- 172.21.15.43:0/140552186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5c4c06be90 0x7f5c4c06e350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5c64036000 tx=0x7f5c64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.003+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c6c055650 con 0x7f5c7415b350 2024-01-16T15:09:27.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.236+0000 7f5c7a600640 1 -- 172.21.15.43:0/140552186 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9611", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c7414d690 con 0x7f5c4c06be90 2024-01-16T15:09:27.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:27.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:27.237 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:27.255+0000 7fd4e5000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:27.255+0000 7f5c6b400640 1 -- 172.21.15.43:0/140552186 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f5c7414d690 con 0x7f5c4c06be90 2024-01-16T15:09:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:09:29.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9612 2024-01-16T15:09:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.502+0000 7fea9ae00640 1 Processor -- start 2024-01-16T15:09:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.502+0000 7fea9ae00640 1 -- start start 2024-01-16T15:09:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.504+0000 7fea9ae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea94144770 0x7fea9415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.504+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea940ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.504+0000 7fea9ae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9415f930 con 0x7fea940b35a0 2024-01-16T15:09:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.504+0000 7fea9ae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940b43b0 con 0x7fea94144770 2024-01-16T15:09:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.504+0000 7fea9ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9415f530 con 0x7fea9415f0a0 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea99a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea94144770 0x7fea9415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea99a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea94144770 0x7fea9415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:40588/0 (socket says 172.21.15.43:40588) 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea99a00640 1 -- 172.21.15.43:0/3119976023 learned_addr learned my addr 172.21.15.43:0/3119976023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea99000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fea940b35a0 0x7fea9406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48072/0 (socket says 172.21.15.43:48072) 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea9a400640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea940ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea93e00640 1 -- 172.21.15.43:0/3119976023 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973228678 0 0) 0x7fea9415f930 con 0x7fea940b35a0 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.505+0000 7fea93e00640 1 -- 172.21.15.43:0/3119976023 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea940b35a0 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.506+0000 7fea99a00640 1 -- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 msgr2=0x7fea940ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.506+0000 7fea99a00640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea940ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.506+0000 7fea99a00640 1 -- 172.21.15.43:0/3119976023 >> v1:172.21.15.43:6789/0 conn(0x7fea940b35a0 legacy=0x7fea9406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.506+0000 7fea99a00640 1 -- 172.21.15.43:0/3119976023 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940ce9f0 con 0x7fea94144770 2024-01-16T15:09:29.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.507+0000 7fea99a00640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea94144770 0x7fea9415eb60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea84009870 tx=0x7fea8402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea7358fccc0897f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.508+0000 7fea93e00640 1 -- 172.21.15.43:0/3119976023 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84030840 con 0x7fea94144770 2024-01-16T15:09:29.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.508+0000 7fea93e00640 1 -- 172.21.15.43:0/3119976023 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea840309a0 con 0x7fea94144770 2024-01-16T15:09:29.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.508+0000 7fea93e00640 1 -- 172.21.15.43:0/3119976023 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84030d10 con 0x7fea94144770 2024-01-16T15:09:29.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 -- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea94144770 msgr2=0x7fea9415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea94144770 0x7fea9415eb60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea84009870 tx=0x7fea8402b980 comp rx=0 tx=0).stop 2024-01-16T15:09:29.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 -- 172.21.15.43:0/3119976023 shutdown_connections 2024-01-16T15:09:29.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea940ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 --2- 172.21.15.43:0/3119976023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea94144770 0x7fea9415eb60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 -- 172.21.15.43:0/3119976023 >> 172.21.15.43:0/3119976023 conn(0x7fea940cec10 msgr2=0x7fea940cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:29.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 -- 172.21.15.43:0/3119976023 shutdown_connections 2024-01-16T15:09:29.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.509+0000 7fea9ae00640 1 -- 172.21.15.43:0/3119976023 wait complete. 2024-01-16T15:09:29.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.511+0000 7fea9ae00640 1 Processor -- start 2024-01-16T15:09:29.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.511+0000 7fea9ae00640 1 -- start start 2024-01-16T15:09:29.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940b35a0 0x7fea94139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea94144770 0x7fea94155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea9413f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9413a4c0 con 0x7fea94144770 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea94142490 con 0x7fea940b35a0 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.512+0000 7fea9ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea94142770 con 0x7fea9415f0a0 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea94144770 0x7fea94155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea94144770 0x7fea94155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37094/0 (socket says 172.21.15.43:37094) 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 -- 172.21.15.43:0/1644142622 learned_addr learned my addr 172.21.15.43:0/1644142622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea9a400640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea9413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99a00640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940b35a0 0x7fea94139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 -- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 msgr2=0x7fea9413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea9415f0a0 0x7fea9413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 -- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940b35a0 msgr2=0x7fea94139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:29.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940b35a0 0x7fea94139f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:29.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.513+0000 7fea99000640 1 -- 172.21.15.43:0/1644142622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840094d0 con 0x7fea94144770 2024-01-16T15:09:29.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea99000640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea94144770 0x7fea94155fc0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00c410 tx=0x7fea7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0074e0 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c007640 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c004a00 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea9ae00640 1 -- 172.21.15.43:0/1644142622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea94142a50 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.514+0000 7fea9ae00640 1 -- 172.21.15.43:0/1644142622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea94143010 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.516+0000 7fea9ae00640 1 -- 172.21.15.43:0/1644142622 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9404e840 con 0x7fea94144770 2024-01-16T15:09:29.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.516+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fea7c004b60 con 0x7fea94144770 2024-01-16T15:09:29.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.517+0000 7fea91600640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea6c06be90 0x7fea6c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:29.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.518+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fea7c0882b0 con 0x7fea94144770 2024-01-16T15:09:29.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.518+0000 7fea99a00640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea6c06be90 0x7fea6c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:29.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.519+0000 7fea99a00640 1 --2- 172.21.15.43:0/1644142622 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea6c06be90 0x7fea6c06e350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fea84036000 tx=0x7fea84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:29.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.522+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea7c0549d0 con 0x7fea94144770 2024-01-16T15:09:29.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.756+0000 7fea9ae00640 1 -- 172.21.15.43:0/1644142622 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9612", "target": ["mon-mgr", ""]}) v1 -- 0x7fea940b40d0 con 0x7fea6c06be90 2024-01-16T15:09:29.748 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:29.770+0000 7fd4e5000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:29.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:29.771+0000 7fea91600640 1 -- 172.21.15.43:0/1644142622 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fea84030b90 con 0x7fea6c06be90 2024-01-16T15:09:29.748 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:09:31.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9613 2024-01-16T15:09:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.223+0000 7f46f0200640 1 Processor -- start 2024-01-16T15:09:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.228+0000 7f46f0200640 1 -- start start 2024-01-16T15:09:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.233+0000 7f46f0200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.233+0000 7f46f0200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46e806c3c0 0x7f46e806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.233+0000 7f46f0200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e8162fa0 con 0x7f46e8162220 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46f0200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e806cd90 con 0x7f46e806c3c0 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46f0200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80706a0 con 0x7f46e8162600 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46ece00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46ece00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:56132/0 (socket says 172.21.15.43:56132) 2024-01-16T15:09:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46ece00640 1 -- 172.21.15.43:0/2812821591 learned_addr learned my addr 172.21.15.43:0/2812821591 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.234+0000 7f46ee200640 1 --2- 172.21.15.43:0/2812821591 >> v2:172.21.15.69:3300/0 conn(0x7f46e806c3c0 0x7f46e806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.235+0000 7f46ece00640 1 -- 172.21.15.43:0/2812821591 >> v2:172.21.15.69:3300/0 conn(0x7f46e806c3c0 msgr2=0x7f46e806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.236+0000 7f46ece00640 1 --2- 172.21.15.43:0/2812821591 >> v2:172.21.15.69:3300/0 conn(0x7f46e806c3c0 0x7f46e806c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.236+0000 7f46ece00640 1 -- 172.21.15.43:0/2812821591 >> v1:172.21.15.69:6789/0 conn(0x7f46e8162220 legacy=0x7f46e80b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.236+0000 7f46ece00640 1 -- 172.21.15.43:0/2812821591 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e8070c60 con 0x7f46e8162600 2024-01-16T15:09:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.236+0000 7f46e7e00640 1 -- 172.21.15.43:0/2812821591 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113770672 0 0) 0x7f46e8162fa0 con 0x7f46e8162220 2024-01-16T15:09:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.238+0000 7f46ece00640 1 --2- 172.21.15.43:0/2812821591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009e50 tx=0x7f46e002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ad3a16c6cdf135 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.240+0000 7f46e7e00640 1 -- 172.21.15.43:0/2812821591 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e002cb00 con 0x7f46e8162600 2024-01-16T15:09:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.240+0000 7f46e7e00640 1 -- 172.21.15.43:0/2812821591 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e002cc60 con 0x7f46e8162600 2024-01-16T15:09:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.240+0000 7f46e7e00640 1 -- 172.21.15.43:0/2812821591 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e00343f0 con 0x7f46e8162600 2024-01-16T15:09:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.244+0000 7f46f0200640 1 -- 172.21.15.43:0/2812821591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 msgr2=0x7f46e8162a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.244+0000 7f46f0200640 1 --2- 172.21.15.43:0/2812821591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009e50 tx=0x7f46e002bf60 comp rx=0 tx=0).stop 2024-01-16T15:09:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.245+0000 7f46f0200640 1 -- 172.21.15.43:0/2812821591 shutdown_connections 2024-01-16T15:09:32.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.245+0000 7f46f0200640 1 --2- 172.21.15.43:0/2812821591 >> v2:172.21.15.69:3300/0 conn(0x7f46e806c3c0 0x7f46e806c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.245+0000 7f46f0200640 1 --2- 172.21.15.43:0/2812821591 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e8162600 0x7f46e8162a60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.245+0000 7f46f0200640 1 -- 172.21.15.43:0/2812821591 >> 172.21.15.43:0/2812821591 conn(0x7f46e80b0ba0 msgr2=0x7f46e80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.246+0000 7f46f0200640 1 -- 172.21.15.43:0/2812821591 shutdown_connections 2024-01-16T15:09:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.246+0000 7f46f0200640 1 -- 172.21.15.43:0/2812821591 wait complete. 2024-01-16T15:09:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.253+0000 7f46f0200640 1 Processor -- start 2024-01-16T15:09:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.253+0000 7f46f0200640 1 -- start start 2024-01-16T15:09:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.255+0000 7f46f0200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e806c3c0 0x7f46e80b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.255+0000 7f46f0200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46e8162220 0x7f46e81ffb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.255+0000 7f46f0200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.256+0000 7f46f0200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80adcd0 con 0x7f46e8162600 2024-01-16T15:09:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.256+0000 7f46f0200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b2100 con 0x7f46e8162220 2024-01-16T15:09:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.256+0000 7f46f0200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b23e0 con 0x7f46e806c3c0 2024-01-16T15:09:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.257+0000 7f46ee200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.257+0000 7f46ee200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37098/0 (socket says 172.21.15.43:37098) 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.257+0000 7f46ee200640 1 -- 172.21.15.43:0/548604169 learned_addr learned my addr 172.21.15.43:0/548604169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.258+0000 7f46ed800640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e806c3c0 0x7f46e80b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.258+0000 7f46ece00640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46e8162220 0x7f46e81ffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.260+0000 7f46ee200640 1 -- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e806c3c0 msgr2=0x7f46e80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.260+0000 7f46ee200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e806c3c0 0x7f46e80b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.260+0000 7f46ee200640 1 -- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46e8162220 msgr2=0x7f46e81ffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.260+0000 7f46ee200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46e8162220 0x7f46e81ffb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.260+0000 7f46ee200640 1 -- 172.21.15.43:0/548604169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e0009b30 con 0x7f46e8162600 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.262+0000 7f46ee200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f46d000d5d0 tx=0x7f46d000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.265+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00125f0 con 0x7f46e8162600 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.265+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0012770 con 0x7f46e8162600 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.265+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0012b20 con 0x7f46e8162600 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.265+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e80b26c0 con 0x7f46e8162600 2024-01-16T15:09:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.266+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e80b2c80 con 0x7f46e8162600 2024-01-16T15:09:32.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.269+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e806d480 con 0x7f46e8162600 2024-01-16T15:09:32.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.276+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f46d0012c80 con 0x7f46e8162600 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.278+0000 7f46e5600640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 0x7f46c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:32.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.279+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f46d0089730 con 0x7f46e8162600 2024-01-16T15:09:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.283+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46d0055d90 con 0x7f46e8162600 2024-01-16T15:09:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.289+0000 7f46ed800640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 0x7f46c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.295+0000 7f46ed800640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 0x7f46c006e420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f46d8000f60 tx=0x7f46d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:32.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:32.823+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9613", "target": ["mon-mgr", ""]}) v1 -- 0x7f46e80708f0 con 0x7f46c006bf60 2024-01-16T15:09:33.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.380+0000 7f46e5600640 1 -- 172.21.15.43:0/548604169 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46d8001440 con 0x7f46c006bf60 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 msgr2=0x7f46c006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 0x7f46c006e420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f46d8000f60 tx=0x7f46d8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 msgr2=0x7f46e82037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f46d000d5d0 tx=0x7f46d000da90 comp rx=0 tx=0).stop 2024-01-16T15:09:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 shutdown_connections 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c006bf60 0x7f46c006e420 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46e8162220 0x7f46e81ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.383+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46e8162600 0x7f46e82037b0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.384+0000 7f46f0200640 1 --2- 172.21.15.43:0/548604169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46e806c3c0 0x7f46e80b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.384+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 >> 172.21.15.43:0/548604169 conn(0x7f46e80b0ba0 msgr2=0x7f46e80710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.384+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 shutdown_connections 2024-01-16T15:09:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:33.384+0000 7f46f0200640 1 -- 172.21.15.43:0/548604169 wait complete. 2024-01-16T15:09:35.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9614 2024-01-16T15:09:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.642+0000 7f8459c00640 1 Processor -- start 2024-01-16T15:09:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.642+0000 7f8459c00640 1 -- start start 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.644+0000 7f8459c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84541435d0 0x7f8454071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8459c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f84541439b0 0x7f84540b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8459c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8454070d40 con 0x7f84540b3f70 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8459c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84540b43e0 con 0x7f84541439b0 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8459c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8454071020 con 0x7f84541435d0 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f84541439b0 0x7f84540b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f84541439b0 0x7f84540b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48724/0 (socket says 172.21.15.43:48724) 2024-01-16T15:09:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 -- 172.21.15.43:0/149697736 learned_addr learned my addr 172.21.15.43:0/149697736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 -- 172.21.15.43:0/149697736 >> v2:172.21.15.69:3300/0 conn(0x7f84541435d0 msgr2=0x7f8454071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 --2- 172.21.15.43:0/149697736 >> v2:172.21.15.69:3300/0 conn(0x7f84541435d0 0x7f8454071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 -- 172.21.15.43:0/149697736 >> v1:172.21.15.69:6789/0 conn(0x7f84540b3f70 legacy=0x7f8454070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.645+0000 7f8453400640 1 -- 172.21.15.43:0/149697736 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454071300 con 0x7f84541439b0 2024-01-16T15:09:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.647+0000 7f8453400640 1 --2- 172.21.15.43:0/149697736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84541439b0 0x7f84540b3a30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84540cfc10 tx=0x7f843c02da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142ff4aee0129df2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.647+0000 7f8452a00640 1 -- 172.21.15.43:0/149697736 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c02f850 con 0x7f84541439b0 2024-01-16T15:09:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.647+0000 7f8452a00640 1 -- 172.21.15.43:0/149697736 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c02f9b0 con 0x7f84541439b0 2024-01-16T15:09:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.647+0000 7f8452a00640 1 -- 172.21.15.43:0/149697736 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c02fd40 con 0x7f84541439b0 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 -- 172.21.15.43:0/149697736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84541439b0 msgr2=0x7f84540b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 --2- 172.21.15.43:0/149697736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84541439b0 0x7f84540b3a30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84540cfc10 tx=0x7f843c02da90 comp rx=0 tx=0).stop 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 -- 172.21.15.43:0/149697736 shutdown_connections 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 --2- 172.21.15.43:0/149697736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84541439b0 0x7f84540b3a30 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 --2- 172.21.15.43:0/149697736 >> v2:172.21.15.69:3300/0 conn(0x7f84541435d0 0x7f8454071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 -- 172.21.15.43:0/149697736 >> 172.21.15.43:0/149697736 conn(0x7f84540cec10 msgr2=0x7f84540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 -- 172.21.15.43:0/149697736 shutdown_connections 2024-01-16T15:09:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.648+0000 7f8459c00640 1 -- 172.21.15.43:0/149697736 wait complete. 2024-01-16T15:09:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.650+0000 7f8459c00640 1 Processor -- start 2024-01-16T15:09:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.651+0000 7f8459c00640 1 -- start start 2024-01-16T15:09:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84540b3f70 0x7f845413cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84541435d0 0x7f8454139630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84541439b0 0x7f84541581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8454139b70 con 0x7f84540b3f70 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845413c740 con 0x7f84541439b0 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8459c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845413bb30 con 0x7f84541435d0 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8453e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84540b3f70 0x7f845413cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8453e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84540b3f70 0x7f845413cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48738/0 (socket says 172.21.15.43:48738) 2024-01-16T15:09:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.652+0000 7f8453e00640 1 -- 172.21.15.43:0/995729155 learned_addr learned my addr 172.21.15.43:0/995729155 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453400640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84541435d0 0x7f8454139630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8459200640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84541439b0 0x7f84541581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 -- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84541435d0 msgr2=0x7f8454139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84541435d0 0x7f8454139630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 -- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84541439b0 msgr2=0x7f84541581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84541439b0 0x7f84541581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 -- 172.21.15.43:0/995729155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0094d0 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8459200640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84541439b0 0x7f84541581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.653+0000 7f8453e00640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84540b3f70 0x7f845413cbd0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f844400b4a0 tx=0x7f844400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.654+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444010610 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.654+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444010770 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.654+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444010b20 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.654+0000 7f8459c00640 1 -- 172.21.15.43:0/995729155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845413be10 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.654+0000 7f8459c00640 1 -- 172.21.15.43:0/995729155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845413c3d0 con 0x7f84540b3f70 2024-01-16T15:09:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.655+0000 7f8459c00640 1 -- 172.21.15.43:0/995729155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845404e840 con 0x7f84540b3f70 2024-01-16T15:09:35.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.656+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8444010c80 con 0x7f84540b3f70 2024-01-16T15:09:35.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.657+0000 7f844be00640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f842c06bee0 0x7f842c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.657+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8444088420 con 0x7f84540b3f70 2024-01-16T15:09:35.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.657+0000 7f8453400640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f842c06bee0 0x7f842c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.658+0000 7f8453400640 1 --2- 172.21.15.43:0/995729155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f842c06bee0 0x7f842c06e3a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f843c038000 tx=0x7f843c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.662+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8444054a80 con 0x7f84540b3f70 2024-01-16T15:09:35.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.896+0000 7f8459c00640 1 -- 172.21.15.43:0/995729155 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7f84540cf590 con 0x7f842c06bee0 2024-01-16T15:09:35.898 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:35.920+0000 7fd4e5000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:35.921+0000 7f844be00640 1 -- 172.21.15.43:0/995729155 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f84540cf590 con 0x7f842c06bee0 2024-01-16T15:09:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:09:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:09:37.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9615 2024-01-16T15:09:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.171+0000 7ff7dfe00640 1 Processor -- start 2024-01-16T15:09:38.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.171+0000 7ff7dfe00640 1 -- start start 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff7e015f270 0x7ff7e00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7dfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e00cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7dfe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e015f930 con 0x7ff7e0144770 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7dfe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e00b43b0 con 0x7ff7e015f270 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7dfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e00b3b10 con 0x7ff7e006d9e0 2024-01-16T15:09:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.173+0000 7ff7de000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff7e015f270 0x7ff7e00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7dea00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff7e0144770 0x7ff7e015eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57836/0 (socket says 172.21.15.43:57836) 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7dea00640 1 -- 172.21.15.43:0/2228434528 learned_addr learned my addr 172.21.15.43:0/2228434528 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7df400640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e00cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7de000640 1 -- 172.21.15.43:0/2228434528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 msgr2=0x7ff7e00cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7de000640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e00cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7de000640 1 -- 172.21.15.43:0/2228434528 >> v1:172.21.15.43:6789/0 conn(0x7ff7e0144770 legacy=0x7ff7e015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.174+0000 7ff7de000640 1 -- 172.21.15.43:0/2228434528 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e00b3df0 con 0x7ff7e015f270 2024-01-16T15:09:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.175+0000 7ff7de000640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e00cd4a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80097a0 tx=0x7ff7c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a9ef576c9d7a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dd600640 1 -- 172.21.15.43:0/2228434528 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8031830 con 0x7ff7e015f270 2024-01-16T15:09:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dd600640 1 -- 172.21.15.43:0/2228434528 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c8031990 con 0x7ff7e015f270 2024-01-16T15:09:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dd600640 1 -- 172.21.15.43:0/2228434528 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8031d20 con 0x7ff7e015f270 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 -- 172.21.15.43:0/2228434528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 msgr2=0x7ff7e00cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e00cd4a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80097a0 tx=0x7ff7c802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 -- 172.21.15.43:0/2228434528 shutdown_connections 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e00cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/2228434528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e00cd4a0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.176+0000 7ff7dfe00640 1 -- 172.21.15.43:0/2228434528 >> 172.21.15.43:0/2228434528 conn(0x7ff7e00cec10 msgr2=0x7ff7e00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.177+0000 7ff7dfe00640 1 -- 172.21.15.43:0/2228434528 shutdown_connections 2024-01-16T15:09:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.177+0000 7ff7dfe00640 1 -- 172.21.15.43:0/2228434528 wait complete. 2024-01-16T15:09:38.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.179+0000 7ff7dfe00640 1 Processor -- start 2024-01-16T15:09:38.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.179+0000 7ff7dfe00640 1 -- start start 2024-01-16T15:09:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e013dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7e0144770 0x7ff7e00ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e00afb90 con 0x7ff7e015f270 2024-01-16T15:09:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e00acdb0 con 0x7ff7e0144770 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.180+0000 7ff7dfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e00aef10 con 0x7ff7e006d9e0 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48752/0 (socket says 172.21.15.43:48752) 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 -- 172.21.15.43:0/1670119373 learned_addr learned my addr 172.21.15.43:0/1670119373 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7dea00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e013dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7de000640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7e0144770 0x7ff7e00ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 -- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 msgr2=0x7ff7e013dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e013dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 -- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7e0144770 msgr2=0x7ff7e00ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7e0144770 0x7ff7e00ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.181+0000 7ff7df400640 1 -- 172.21.15.43:0/1670119373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009400 con 0x7ff7e015f270 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7df400640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800cd70 tx=0x7ff7d800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8003b90 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d8003cf0 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8004240 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e00af1f0 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.182+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e00af7b0 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.183+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e00afcd0 con 0x7ff7e015f270 2024-01-16T15:09:38.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.184+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff7d80043a0 con 0x7ff7e015f270 2024-01-16T15:09:38.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.185+0000 7ff7d6a00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 0x7ff7b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.185+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff7d8012070 con 0x7ff7e015f270 2024-01-16T15:09:38.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.186+0000 7ff7dea00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 0x7ff7b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.186+0000 7ff7dea00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 0x7ff7b806e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0000f60 tx=0x7ff7d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.190+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7e00afcd0 con 0x7ff7e015f270 2024-01-16T15:09:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:38.418+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9615", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7e00acf70 con 0x7ff7b806be90 2024-01-16T15:09:44.414 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:44.436+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:44.414 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:44.436+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:44.414 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:44.436+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:44.414 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:44.436+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:44.414 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:09:44.436+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:45.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.369+0000 7ff7d6a00640 1 -- 172.21.15.43:0/1670119373 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff7e00acf70 con 0x7ff7b806be90 2024-01-16T15:09:45.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 msgr2=0x7ff7b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 0x7ff7b806e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0000f60 tx=0x7ff7d0009040 comp rx=0 tx=0).stop 2024-01-16T15:09:45.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 msgr2=0x7ff7e0152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800cd70 tx=0x7ff7d800a660 comp rx=0 tx=0).stop 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 shutdown_connections 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff7b806be90 0x7ff7b806e350 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7e0144770 0x7ff7e00ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7e015f270 0x7ff7e0152660 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 --2- 172.21.15.43:0/1670119373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7e006d9e0 0x7ff7e013dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.376+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 >> 172.21.15.43:0/1670119373 conn(0x7ff7e00cec10 msgr2=0x7ff7e0141ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.377+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 shutdown_connections 2024-01-16T15:09:45.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:45.377+0000 7ff7dfe00640 1 -- 172.21.15.43:0/1670119373 wait complete. 2024-01-16T15:09:47.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9616 2024-01-16T15:09:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.656+0000 7f807a200640 1 Processor -- start 2024-01-16T15:09:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.656+0000 7f807a200640 1 -- start start 2024-01-16T15:09:47.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.658+0000 7f807a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.658+0000 7f807a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80740d1840 0x7f80740cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.658+0000 7f807a200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80740cf240 con 0x7f807415d350 2024-01-16T15:09:47.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.658+0000 7f807a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80740d1fb0 con 0x7f80740d1840 2024-01-16T15:09:47.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.658+0000 7f807a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80740ce050 con 0x7f807415d730 2024-01-16T15:09:47.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.659+0000 7f8072a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.659+0000 7f8072a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:58786/0 (socket says 172.21.15.43:58786) 2024-01-16T15:09:47.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.659+0000 7f8072a00640 1 -- 172.21.15.43:0/1691664726 learned_addr learned my addr 172.21.15.43:0/1691664726 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:47.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.659+0000 7f8073e00640 1 --2- 172.21.15.43:0/1691664726 >> v2:172.21.15.69:3300/0 conn(0x7f80740d1840 0x7f80740cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.660+0000 7f8072a00640 1 -- 172.21.15.43:0/1691664726 >> v2:172.21.15.69:3300/0 conn(0x7f80740d1840 msgr2=0x7f80740cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:47.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.660+0000 7f8072a00640 1 --2- 172.21.15.43:0/1691664726 >> v2:172.21.15.69:3300/0 conn(0x7f80740d1840 0x7f80740cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.660+0000 7f8072a00640 1 -- 172.21.15.43:0/1691664726 >> v1:172.21.15.69:6789/0 conn(0x7f807415d350 legacy=0x7f80740ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.660+0000 7f8072a00640 1 -- 172.21.15.43:0/1691664726 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740ce330 con 0x7f807415d730 2024-01-16T15:09:47.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.660+0000 7f8072000640 1 -- 172.21.15.43:0/1691664726 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533039869 0 0) 0x7f80740cf240 con 0x7f807415d350 2024-01-16T15:09:47.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.661+0000 7f8072a00640 1 --2- 172.21.15.43:0/1691664726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f805c009870 tx=0x7f805c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59488cfcef1f9a50 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.662+0000 7f8072000640 1 -- 172.21.15.43:0/1691664726 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c030840 con 0x7f807415d730 2024-01-16T15:09:47.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.662+0000 7f8072000640 1 -- 172.21.15.43:0/1691664726 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c0309a0 con 0x7f807415d730 2024-01-16T15:09:47.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.662+0000 7f8072000640 1 -- 172.21.15.43:0/1691664726 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c030d30 con 0x7f807415d730 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.662+0000 7f807a200640 1 -- 172.21.15.43:0/1691664726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 msgr2=0x7f80740d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.662+0000 7f807a200640 1 --2- 172.21.15.43:0/1691664726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f805c009870 tx=0x7f805c02b980 comp rx=0 tx=0).stop 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 -- 172.21.15.43:0/1691664726 shutdown_connections 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 --2- 172.21.15.43:0/1691664726 >> v2:172.21.15.69:3300/0 conn(0x7f80740d1840 0x7f80740cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 --2- 172.21.15.43:0/1691664726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d730 0x7f80740d3d30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 -- 172.21.15.43:0/1691664726 >> 172.21.15.43:0/1691664726 conn(0x7f807409c5c0 msgr2=0x7f80740d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:47.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 -- 172.21.15.43:0/1691664726 shutdown_connections 2024-01-16T15:09:47.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.663+0000 7f807a200640 1 -- 172.21.15.43:0/1691664726 wait complete. 2024-01-16T15:09:47.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.665+0000 7f807a200640 1 Processor -- start 2024-01-16T15:09:47.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.665+0000 7f807a200640 1 -- start start 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d350 0x7f80740b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f807415d730 0x7f807414e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80741f6b70 con 0x7f80740d1840 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80741f6e50 con 0x7f807415d730 2024-01-16T15:09:47.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f807a200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80741f7130 con 0x7f807415d350 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f8073400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f8073400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34764/0 (socket says 172.21.15.43:34764) 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.666+0000 7f8073400640 1 -- 172.21.15.43:0/215462365 learned_addr learned my addr 172.21.15.43:0/215462365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073e00640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f807415d730 0x7f807414e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8072a00640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d350 0x7f80740b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 -- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d350 msgr2=0x7f80740b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d350 0x7f80740b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 -- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f807415d730 msgr2=0x7f807414e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f807415d730 0x7f807414e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 -- 172.21.15.43:0/215462365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0094d0 con 0x7f80740d1840 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.667+0000 7f8073400640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f806400c4b0 tx=0x7f806400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.668+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064005480 con 0x7f80740d1840 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.668+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640035f0 con 0x7f80740d1840 2024-01-16T15:09:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.668+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640039e0 con 0x7f80740d1840 2024-01-16T15:09:47.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.668+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80741f7410 con 0x7f80740d1840 2024-01-16T15:09:47.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.668+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80741584c0 con 0x7f80740d1840 2024-01-16T15:09:47.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.669+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80741f7930 con 0x7f80740d1840 2024-01-16T15:09:47.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.670+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8064003b40 con 0x7f80740d1840 2024-01-16T15:09:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.671+0000 7f806b400640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 0x7f804c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.671+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8064087e50 con 0x7f80740d1840 2024-01-16T15:09:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.671+0000 7f8072a00640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 0x7f804c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:47.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.672+0000 7f8072a00640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 0x7f804c06e3e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f805c036000 tx=0x7f805c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:47.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.676+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8064054490 con 0x7f80740d1840 2024-01-16T15:09:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:47.896+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f80740cdd00 con 0x7f804c06bf20 2024-01-16T15:09:57.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:57.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:57.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.736+0000 7f806b400640 1 -- 172.21.15.43:0/215462365 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f805c030b90 con 0x7f804c06bf20 2024-01-16T15:09:57.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 msgr2=0x7f804c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 0x7f804c06e3e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f805c036000 tx=0x7f805c03b040 comp rx=0 tx=0).stop 2024-01-16T15:09:57.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 msgr2=0x7f8074157f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f806400c4b0 tx=0x7f806400c970 comp rx=0 tx=0).stop 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 shutdown_connections 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f804c06bf20 0x7f804c06e3e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f807415d730 0x7f807414e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f807415d350 0x7f80740b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 --2- 172.21.15.43:0/215462365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80740d1840 0x7f8074157f80 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 >> 172.21.15.43:0/215462365 conn(0x7f807409c5c0 msgr2=0x7f80740b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 shutdown_connections 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:57.739+0000 7f807a200640 1 -- 172.21.15.43:0/215462365 wait complete. 2024-01-16T15:09:59.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9617 2024-01-16T15:09:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.991+0000 7f0f0b600640 1 Processor -- start 2024-01-16T15:09:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.992+0000 7f0f0b600640 1 -- start start 2024-01-16T15:09:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.993+0000 7f0f0b600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0f040d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.993+0000 7f0f0b600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0415d720 con 0x7f0f0415d340 2024-01-16T15:09:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.993+0000 7f0f0b600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0406d080 con 0x7f0f040cda40 2024-01-16T15:09:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.993+0000 7f0f0b600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f040d0d90 con 0x7f0f04070af0 2024-01-16T15:09:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.994+0000 7f0f08c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0f0415d340 0x7f0f040cb680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41530/0 (socket says 172.21.15.43:41530) 2024-01-16T15:09:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.994+0000 7f0f08c00640 1 -- 172.21.15.43:0/769574623 learned_addr learned my addr 172.21.15.43:0/769574623 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.994+0000 7f0f03e00640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0f040d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.994+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304978179 0 0) 0x7f0f0415d720 con 0x7f0f0415d340 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.994+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee4003520 con 0x7f0f0415d340 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595611707 0 0) 0x7f0ee4003520 con 0x7f0f0415d340 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 msgr2=0x7f0f040d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0f040d0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 >> v1:172.21.15.69:6789/0 conn(0x7f0f040cda40 legacy=0x7f0f04161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0406d360 con 0x7f0f0415d340 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0ef4001670 con 0x7f0f0415d340 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 >> v1:172.21.15.43:6789/0 conn(0x7f0f0415d340 legacy=0x7f0f040cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee40039a0 0x7f0ee4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0ee40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0ee4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f040d0d90 con 0x7f0ee4004d00 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee400bb40 con 0x7f0ee40039a0 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee400bd00 con 0x7f0f04070af0 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f09600640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0ee4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.995+0000 7f0f08c00640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee40039a0 0x7f0ee4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f03e00640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0ee40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f09600640 1 -- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 msgr2=0x7f0ee40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:59.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f09600640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f04070af0 0x7f0ee40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f09600640 1 -- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee40039a0 msgr2=0x7f0ee4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:59.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f09600640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee40039a0 0x7f0ee4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f09600640 1 -- 172.21.15.43:0/769574623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4003520 con 0x7f0ee4004d00 2024-01-16T15:09:59.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.996+0000 7f0f08c00640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ee40039a0 0x7f0ee4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.997+0000 7f0f09600640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0ee4005180 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0efc001e50 tx=0x7f0ee40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da50a24e5e60869e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.997+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc003b70 con 0x7f0ee4004d00 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.997+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0efc003cd0 con 0x7f0ee4004d00 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.997+0000 7f0f03400640 1 -- 172.21.15.43:0/769574623 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc005310 con 0x7f0ee4004d00 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 -- 172.21.15.43:0/769574623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 msgr2=0x7f0ee4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 --2- 172.21.15.43:0/769574623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0ee4005180 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0efc001e50 tx=0x7f0ee40057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f09600640 1 -- 172.21.15.43:0/769574623 reap_dead start 2024-01-16T15:09:59.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 -- 172.21.15.43:0/769574623 shutdown_connections 2024-01-16T15:09:59.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 -- 172.21.15.43:0/769574623 >> 172.21.15.43:0/769574623 conn(0x7f0f040b2ba0 msgr2=0x7f0f040b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:59.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 -- 172.21.15.43:0/769574623 shutdown_connections 2024-01-16T15:09:59.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:09:59.998+0000 7f0f0b600640 1 -- 172.21.15.43:0/769574623 wait complete. 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.000+0000 7f0f0b600640 1 Processor -- start 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.000+0000 7f0f0b600640 1 -- start start 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f041f3100 0x7f0f040b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f040afef0 0x7f0f040b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041f3600 con 0x7f0ee4004d00 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041f38e0 con 0x7f0f041f3100 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f0b600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041f3bc0 con 0x7f0f040afef0 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40530/0 (socket says 172.21.15.43:40530) 2024-01-16T15:09:59.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 -- 172.21.15.43:0/3611247599 learned_addr learned my addr 172.21.15.43:0/3611247599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 -- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f040afef0 msgr2=0x7f0f040b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f09600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f040afef0 0x7f0f040b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f040afef0 0x7f0f040b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 -- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f041f3100 msgr2=0x7f0f040b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f041f3100 0x7f0f040b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.001+0000 7f0f08c00640 1 -- 172.21.15.43:0/3611247599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f041f3ea0 con 0x7f0ee4004d00 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f08c00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e780 tx=0x7f0ef400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400ee90 con 0x7f0ee4004d00 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef400c870 con 0x7f0ee4004d00 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400cc00 con 0x7f0ee4004d00 2024-01-16T15:09:59.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f040b08e0 con 0x7f0ee4004d00 2024-01-16T15:09:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.002+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f040c7560 con 0x7f0ee4004d00 2024-01-16T15:09:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.003+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f041f40c0 con 0x7f0ee4004d00 2024-01-16T15:09:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.004+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0ef401d460 con 0x7f0ee4004d00 2024-01-16T15:09:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.005+0000 7f0f00c00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 0x7f0edc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.005+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0ef4088410 con 0x7f0ee4004d00 2024-01-16T15:09:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.005+0000 7f0f03e00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 0x7f0edc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.006+0000 7f0f03e00640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 0x7f0edc06e3e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0f040cadc0 tx=0x7f0eec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.010+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ef40bde90 con 0x7f0ee4004d00 2024-01-16T15:10:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:00.241+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9617", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f040cf340 con 0x7f0edc06bf20 2024-01-16T15:10:08.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.241+0000 7f0f00c00640 1 -- 172.21.15.43:0/3611247599 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0eec0095c0 con 0x7f0edc06bf20 2024-01-16T15:10:08.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.249+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 msgr2=0x7f0edc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.249+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 0x7f0edc06e3e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0f040cadc0 tx=0x7f0eec008040 comp rx=0 tx=0).stop 2024-01-16T15:10:08.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.249+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 msgr2=0x7f0f040c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:08.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.249+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e780 tx=0x7f0ef400ec40 comp rx=0 tx=0).stop 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 shutdown_connections 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0edc06bf20 0x7f0edc06e3e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f041f3100 0x7f0f040b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f040afef0 0x7f0f040b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 --2- 172.21.15.43:0/3611247599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ee4004d00 0x7f0f040c6e40 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 >> 172.21.15.43:0/3611247599 conn(0x7f0f040b2ba0 msgr2=0x7f0f0415e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 shutdown_connections 2024-01-16T15:10:08.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:08.251+0000 7f0f0b600640 1 -- 172.21.15.43:0/3611247599 wait complete. 2024-01-16T15:10:10.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9618 2024-01-16T15:10:10.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.519+0000 7fb5e3e00640 1 Processor -- start 2024-01-16T15:10:10.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.519+0000 7fb5e3e00640 1 -- start start 2024-01-16T15:10:10.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.520+0000 7fb5e3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5e415f270 0x7fb5e40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.520+0000 7fb5e3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 0x7fb5e40cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.520+0000 7fb5e3e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e415f930 con 0x7fb5e4144770 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.520+0000 7fb5e3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e40b43b0 con 0x7fb5e415f270 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.520+0000 7fb5e3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b3b10 con 0x7fb5e406d9e0 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.521+0000 7fb5e2a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb5e4144770 0x7fb5e415eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:59892/0 (socket says 172.21.15.43:59892) 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.521+0000 7fb5e2a00640 1 -- 172.21.15.43:0/2727820286 learned_addr learned my addr 172.21.15.43:0/2727820286 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.521+0000 7fb5e3400640 1 --2- 172.21.15.43:0/2727820286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 0x7fb5e40cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.521+0000 7fb5dbe00640 1 --2- 172.21.15.43:0/2727820286 >> v2:172.21.15.69:3300/0 conn(0x7fb5e415f270 0x7fb5e40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.522+0000 7fb5e3400640 1 -- 172.21.15.43:0/2727820286 >> v2:172.21.15.69:3300/0 conn(0x7fb5e415f270 msgr2=0x7fb5e40cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:10.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.522+0000 7fb5e3400640 1 --2- 172.21.15.43:0/2727820286 >> v2:172.21.15.69:3300/0 conn(0x7fb5e415f270 0x7fb5e40cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.522+0000 7fb5e3400640 1 -- 172.21.15.43:0/2727820286 >> v1:172.21.15.69:6789/0 conn(0x7fb5e4144770 legacy=0x7fb5e415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.522+0000 7fb5e3400640 1 -- 172.21.15.43:0/2727820286 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40b3df0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.523+0000 7fb5e3400640 1 --2- 172.21.15.43:0/2727820286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 0x7fb5e40cd9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009d90 tx=0x7fb5dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cca39a48df5e0f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:10.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e2000640 1 -- 172.21.15.43:0/2727820286 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc02cc50 con 0x7fb5e406d9e0 2024-01-16T15:10:10.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e2000640 1 -- 172.21.15.43:0/2727820286 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc02cdb0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e2000640 1 -- 172.21.15.43:0/2727820286 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc034660 con 0x7fb5e406d9e0 2024-01-16T15:10:10.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2727820286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 msgr2=0x7fb5e40cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2727820286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 0x7fb5e40cd9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009d90 tx=0x7fb5dc02bea0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2727820286 shutdown_connections 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2727820286 >> v2:172.21.15.69:3300/0 conn(0x7fb5e415f270 0x7fb5e40cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2727820286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e406d9e0 0x7fb5e40cd9e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2727820286 >> 172.21.15.43:0/2727820286 conn(0x7fb5e40cec10 msgr2=0x7fb5e40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2727820286 shutdown_connections 2024-01-16T15:10:10.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.524+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2727820286 wait complete. 2024-01-16T15:10:10.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.526+0000 7fb5e3e00640 1 Processor -- start 2024-01-16T15:10:10.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.526+0000 7fb5e3e00640 1 -- start start 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e4144770 0x7fb5e41517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e415f270 0x7fb5e41f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4151ce0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40711d0 con 0x7fb5e4144770 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.527+0000 7fb5e3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40714b0 con 0x7fb5e415f270 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59560/0 (socket says 172.21.15.43:59560) 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 -- 172.21.15.43:0/2203133157 learned_addr learned my addr 172.21.15.43:0/2203133157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:10.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 -- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e415f270 msgr2=0x7fb5e41f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e415f270 0x7fb5e41f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 -- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e4144770 msgr2=0x7fb5e41517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:10.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e4144770 0x7fb5e41517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:10.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 -- 172.21.15.43:0/2203133157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc009a70 con 0x7fb5e406d9e0 2024-01-16T15:10:10.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e2a00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400c400 tx=0x7fb5d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4005ce0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d4005e40 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.528+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e4071790 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.529+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40072d0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.529+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e40a8500 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.530+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e404e840 con 0x7fb5e406d9e0 2024-01-16T15:10:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.530+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb5d40074f0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.531+0000 7fb5daa00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 0x7fb5bc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:10.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.531+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5d4087ec0 con 0x7fb5e406d9e0 2024-01-16T15:10:10.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.532+0000 7fb5dbe00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 0x7fb5bc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:10.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.532+0000 7fb5dbe00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 0x7fb5bc06e3e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc006a90 tx=0x7fb5cc006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:10.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.536+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5d4054500 con 0x7fb5e406d9e0 2024-01-16T15:10:10.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:10.761+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9618", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5e4144b50 con 0x7fb5bc06bf20 2024-01-16T15:10:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.071+0000 7fb5daa00640 1 -- 172.21.15.43:0/2203133157 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb5cc009620 con 0x7fb5bc06bf20 2024-01-16T15:10:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 msgr2=0x7fb5bc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 0x7fb5bc06e3e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc006a90 tx=0x7fb5cc006a20 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 msgr2=0x7fb5e4070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400c400 tx=0x7fb5d400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 shutdown_connections 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5bc06bf20 0x7fb5bc06e3e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e4144770 0x7fb5e41517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e415f270 0x7fb5e41f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 --2- 172.21.15.43:0/2203133157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5e406d9e0 0x7fb5e4070b40 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.075+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 >> 172.21.15.43:0/2203133157 conn(0x7fb5e40cec10 msgr2=0x7fb5e413e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.076+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 shutdown_connections 2024-01-16T15:10:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:21.076+0000 7fb5e3e00640 1 -- 172.21.15.43:0/2203133157 wait complete. 2024-01-16T15:10:23.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone9619 2024-01-16T15:10:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.356+0000 7f870e800640 1 Processor -- start 2024-01-16T15:10:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.356+0000 7f870e800640 1 -- start start 2024-01-16T15:10:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.357+0000 7f870e800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 0x7f870815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.357+0000 7f870e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f870815f0a0 0x7f87080ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.357+0000 7f870e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870815f930 con 0x7f87080b35a0 2024-01-16T15:10:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.357+0000 7f870e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080b43b0 con 0x7f870815f0a0 2024-01-16T15:10:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.357+0000 7f870e800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870815f530 con 0x7f8708144770 2024-01-16T15:10:23.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.358+0000 7f870d400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 0x7f870815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:23.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.358+0000 7f870ca00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f87080b35a0 0x7f870806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:41982/0 (socket says 172.21.15.43:41982) 2024-01-16T15:10:23.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.358+0000 7f870ca00640 1 -- 172.21.15.43:0/1257065141 learned_addr learned my addr 172.21.15.43:0/1257065141 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.358+0000 7f870de00640 1 --2- 172.21.15.43:0/1257065141 >> v2:172.21.15.69:3300/0 conn(0x7f870815f0a0 0x7f87080ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f8707e00640 1 -- 172.21.15.43:0/1257065141 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019986799 0 0) 0x7f870815f930 con 0x7f87080b35a0 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f8707e00640 1 -- 172.21.15.43:0/1257065141 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e8003520 con 0x7f87080b35a0 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f870d400640 1 -- 172.21.15.43:0/1257065141 >> v2:172.21.15.69:3300/0 conn(0x7f870815f0a0 msgr2=0x7f87080ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f870d400640 1 --2- 172.21.15.43:0/1257065141 >> v2:172.21.15.69:3300/0 conn(0x7f870815f0a0 0x7f87080ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f870d400640 1 -- 172.21.15.43:0/1257065141 >> v1:172.21.15.69:6789/0 conn(0x7f87080b35a0 legacy=0x7f870806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:23.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.359+0000 7f870d400640 1 -- 172.21.15.43:0/1257065141 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080ce9f0 con 0x7f8708144770 2024-01-16T15:10:23.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.360+0000 7f870d400640 1 --2- 172.21.15.43:0/1257065141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 0x7f870815eb60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097d0 tx=0x7f86f802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ab03340412d393 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:23.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f8707e00640 1 -- 172.21.15.43:0/1257065141 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8031800 con 0x7f8708144770 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f8707e00640 1 -- 172.21.15.43:0/1257065141 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f8031960 con 0x7f8708144770 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f8707e00640 1 -- 172.21.15.43:0/1257065141 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8031cf0 con 0x7f8708144770 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 -- 172.21.15.43:0/1257065141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 msgr2=0x7f870815eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 --2- 172.21.15.43:0/1257065141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 0x7f870815eb60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097d0 tx=0x7f86f802b8a0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 -- 172.21.15.43:0/1257065141 shutdown_connections 2024-01-16T15:10:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 --2- 172.21.15.43:0/1257065141 >> v2:172.21.15.69:3300/0 conn(0x7f870815f0a0 0x7f87080ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 --2- 172.21.15.43:0/1257065141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8708144770 0x7f870815eb60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.361+0000 7f870e800640 1 -- 172.21.15.43:0/1257065141 >> 172.21.15.43:0/1257065141 conn(0x7f87080cec10 msgr2=0x7f87080cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.362+0000 7f870e800640 1 -- 172.21.15.43:0/1257065141 shutdown_connections 2024-01-16T15:10:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.362+0000 7f870e800640 1 -- 172.21.15.43:0/1257065141 wait complete. 2024-01-16T15:10:23.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.363+0000 7f870e800640 1 Processor -- start 2024-01-16T15:10:23.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.363+0000 7f870e800640 1 -- start start 2024-01-16T15:10:23.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87080b35a0 0x7f8708157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f870815f0a0 0x7f8708071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8708143770 con 0x7f8708144770 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8708142b60 con 0x7f87080b35a0 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.364+0000 7f870e800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8708142e40 con 0x7f870815f0a0 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59848/0 (socket says 172.21.15.43:59848) 2024-01-16T15:10:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 -- 172.21.15.43:0/707373925 learned_addr learned my addr 172.21.15.43:0/707373925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 -- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f870815f0a0 msgr2=0x7f8708071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f870815f0a0 0x7f8708071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 -- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87080b35a0 msgr2=0x7f8708157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87080b35a0 0x7f8708157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 -- 172.21.15.43:0/707373925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8009400 con 0x7f8708144770 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.365+0000 7f870ca00640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c410 tx=0x7f86f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.366+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0005ce0 con 0x7f8708144770 2024-01-16T15:10:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.366+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0005e40 con 0x7f8708144770 2024-01-16T15:10:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.366+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8708143120 con 0x7f8708144770 2024-01-16T15:10:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.366+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00072d0 con 0x7f8708144770 2024-01-16T15:10:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.366+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87080a9bb0 con 0x7f8708144770 2024-01-16T15:10:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.367+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870804e840 con 0x7f8708144770 2024-01-16T15:10:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.368+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f86f0007430 con 0x7f8708144770 2024-01-16T15:10:23.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.369+0000 7f8705600640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 0x7f86e006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:23.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.369+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f86f0087ea0 con 0x7f8708144770 2024-01-16T15:10:23.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.369+0000 7f870d400640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 0x7f86e006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:23.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.370+0000 7f870d400640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 0x7f86e006e410 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f86f8036000 tx=0x7f86f8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:23.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.373+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86f0058030 con 0x7f8708144770 2024-01-16T15:10:23.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:23.613+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone9619", "target": ["mon-mgr", ""]}) v1 -- 0x7f87080b40d0 con 0x7f86e006bf50 2024-01-16T15:10:27.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:27.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:33.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.073+0000 7f8705600640 1 -- 172.21.15.43:0/707373925 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f86f8031ad0 con 0x7f86e006bf50 2024-01-16T15:10:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 msgr2=0x7f86e006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 0x7f86e006e410 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f86f8036000 tx=0x7f86f8030000 comp rx=0 tx=0).stop 2024-01-16T15:10:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 msgr2=0x7f87080a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c410 tx=0x7f86f000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 shutdown_connections 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86e006bf50 0x7f86e006e410 secure :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f86f8036000 tx=0x7f86f8030000 comp rx=0 tx=0).stop 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f870815f0a0 0x7f8708071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8708144770 0x7f87080a9670 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 --2- 172.21.15.43:0/707373925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87080b35a0 0x7f8708157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.079+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 >> 172.21.15.43:0/707373925 conn(0x7f87080cec10 msgr2=0x7f870815d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.080+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 shutdown_connections 2024-01-16T15:10:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:33.080+0000 7f870e800640 1 -- 172.21.15.43:0/707373925 wait complete. 2024-01-16T15:10:35.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96110 2024-01-16T15:10:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.349+0000 7fbf73200640 1 Processor -- start 2024-01-16T15:10:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.349+0000 7fbf73200640 1 -- start start 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf73200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 0x7fbf740cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf73200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7415f930 con 0x7fbf74144770 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf73200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740b3b10 con 0x7fbf7406d9e0 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf73200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740b43b0 con 0x7fbf7415f270 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf71e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbf74144770 0x7fbf7415eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42886/0 (socket says 172.21.15.43:42886) 2024-01-16T15:10:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.351+0000 7fbf71e00640 1 -- 172.21.15.43:0/3261412863 learned_addr learned my addr 172.21.15.43:0/3261412863 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf71400640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 0x7fbf740cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252661760 0 0) 0x7fbf7415f930 con 0x7fbf74144770 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003520 con 0x7fbf74144770 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3349704375 0 0) 0x7fbf54003520 con 0x7fbf74144770 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 msgr2=0x7fbf740cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 0x7fbf740cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 >> v1:172.21.15.69:6789/0 conn(0x7fbf7406d9e0 legacy=0x7fbf740cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740b3df0 con 0x7fbf74144770 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fbf64001670 con 0x7fbf74144770 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.352+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 >> v1:172.21.15.43:6789/0 conn(0x7fbf74144770 legacy=0x7fbf7415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf54003a10 0x7fbf54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf540047c0 0x7fbf54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 0x7fbf540051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740b43b0 con 0x7fbf54003a10 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400bb40 con 0x7fbf540047c0 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400bd00 con 0x7fbf7415f270 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf54003a10 0x7fbf54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 -- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 msgr2=0x7fbf540051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7415f270 0x7fbf540051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 -- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf540047c0 msgr2=0x7fbf54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf540047c0 0x7fbf54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.353+0000 7fbf71e00640 1 -- 172.21.15.43:0/3261412863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54003520 con 0x7fbf54003a10 2024-01-16T15:10:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.354+0000 7fbf71e00640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf54003a10 0x7fbf54003e30 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400c0e0 tx=0x7fbf540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bbcefe9b3861af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64005ad0 con 0x7fbf54003a10 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf64005c50 con 0x7fbf54003a10 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf70a00640 1 -- 172.21.15.43:0/3261412863 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6402c810 con 0x7fbf54003a10 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf73200640 1 -- 172.21.15.43:0/3261412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf54003a10 msgr2=0x7fbf54003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf73200640 1 --2- 172.21.15.43:0/3261412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf54003a10 0x7fbf54003e30 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400c0e0 tx=0x7fbf540057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.355+0000 7fbf72800640 1 -- 172.21.15.43:0/3261412863 reap_dead start 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.356+0000 7fbf73200640 1 -- 172.21.15.43:0/3261412863 shutdown_connections 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.356+0000 7fbf73200640 1 -- 172.21.15.43:0/3261412863 >> 172.21.15.43:0/3261412863 conn(0x7fbf740cec10 msgr2=0x7fbf740cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.356+0000 7fbf73200640 1 -- 172.21.15.43:0/3261412863 shutdown_connections 2024-01-16T15:10:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.356+0000 7fbf73200640 1 -- 172.21.15.43:0/3261412863 wait complete. 2024-01-16T15:10:35.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.357+0000 7fbf73200640 1 Processor -- start 2024-01-16T15:10:35.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.357+0000 7fbf73200640 1 -- start start 2024-01-16T15:10:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf54003a10 0x7fbf7413b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf74157c80 0x7fbf741425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7413e6d0 con 0x7fbf7415da90 2024-01-16T15:10:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7415e480 con 0x7fbf74157c80 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.358+0000 7fbf73200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74142b20 con 0x7fbf54003a10 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48906/0 (socket says 172.21.15.43:48906) 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 -- 172.21.15.43:0/755168786 learned_addr learned my addr 172.21.15.43:0/755168786 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf71400640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf74157c80 0x7fbf741425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf71e00640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf54003a10 0x7fbf7413b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 -- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf54003a10 msgr2=0x7fbf7413b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf54003a10 0x7fbf7413b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 -- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf74157c80 msgr2=0x7fbf741425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf74157c80 0x7fbf741425e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.359+0000 7fbf72800640 1 -- 172.21.15.43:0/755168786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74158100 con 0x7fbf7415da90 2024-01-16T15:10:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf72800640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fbf7406e030 tx=0x7fbf6c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c005d40 con 0x7fbf7415da90 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00ced0 con 0x7fbf7415da90 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00a7f0 con 0x7fbf7415da90 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7413b840 con 0x7fbf7415da90 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.360+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7413be00 con 0x7fbf7415da90 2024-01-16T15:10:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.361+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7404e840 con 0x7fbf7415da90 2024-01-16T15:10:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.363+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fbf6c00a950 con 0x7fbf7415da90 2024-01-16T15:10:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.364+0000 7fbf6a000640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 0x7fbf4c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.364+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbf6c0879c0 con 0x7fbf7415da90 2024-01-16T15:10:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.364+0000 7fbf71e00640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 0x7fbf4c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.365+0000 7fbf71e00640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 0x7fbf4c06e410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbf64034000 tx=0x7fbf64009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.370+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf6c053f40 con 0x7fbf7415da90 2024-01-16T15:10:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:35.593+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96110", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf74144b50 con 0x7fbf4c06bf50 2024-01-16T15:10:44.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.745+0000 7fbf6a000640 1 -- 172.21.15.43:0/755168786 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbf6403a070 con 0x7fbf4c06bf50 2024-01-16T15:10:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 msgr2=0x7fbf4c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 0x7fbf4c06e410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbf64034000 tx=0x7fbf64009030 comp rx=0 tx=0).stop 2024-01-16T15:10:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 msgr2=0x7fbf7415df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fbf7406e030 tx=0x7fbf6c00cb90 comp rx=0 tx=0).stop 2024-01-16T15:10:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 shutdown_connections 2024-01-16T15:10:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbf4c06bf50 0x7fbf4c06e410 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf74157c80 0x7fbf741425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf7415da90 0x7fbf7415df40 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 --2- 172.21.15.43:0/755168786 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf54003a10 0x7fbf7413b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.749+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 >> 172.21.15.43:0/755168786 conn(0x7fbf740cec10 msgr2=0x7fbf740cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.750+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 shutdown_connections 2024-01-16T15:10:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:44.750+0000 7fbf73200640 1 -- 172.21.15.43:0/755168786 wait complete. 2024-01-16T15:10:46.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96111 2024-01-16T15:10:46.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.012+0000 7f6db1800640 1 Processor -- start 2024-01-16T15:10:46.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.012+0000 7f6db1800640 1 -- start start 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db1800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6dac15f270 0x7f6dac0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db1800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac15f930 con 0x7f6dac06d9e0 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db1800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0b3b10 con 0x7f6dac144770 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db1800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac0b43b0 con 0x7f6dac15f270 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db0e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6dac06d9e0 0x7f6dac0cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55548/0 (socket says 172.21.15.43:55548) 2024-01-16T15:10:46.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.014+0000 7f6db0e00640 1 -- 172.21.15.43:0/3577028607 learned_addr learned my addr 172.21.15.43:0/3577028607 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:10:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6dab400640 1 --2- 172.21.15.43:0/3577028607 >> v2:172.21.15.69:3300/0 conn(0x7f6dac15f270 0x7f6dac0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99357114 0 0) 0x7f6dac15f930 con 0x7f6dac06d9e0 2024-01-16T15:10:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c003520 con 0x7f6dac06d9e0 2024-01-16T15:10:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311712710 0 0) 0x7f6d8c003520 con 0x7f6dac06d9e0 2024-01-16T15:10:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 >> v2:172.21.15.69:3300/0 conn(0x7f6dac15f270 msgr2=0x7f6dac0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 --2- 172.21.15.43:0/3577028607 >> v2:172.21.15.69:3300/0 conn(0x7f6dac15f270 0x7f6dac0cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 >> v1:172.21.15.69:6789/0 conn(0x7f6dac144770 legacy=0x7f6dac15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0b3df0 con 0x7f6dac06d9e0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6da40019f0 con 0x7f6dac06d9e0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.015+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 >> v1:172.21.15.43:6789/0 conn(0x7f6dac06d9e0 legacy=0x7f6dac0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c003a30 0x7f6d8c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d8c0047c0 0x7f6d8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 0x7f6d8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0b43b0 con 0x7f6d8c0051d0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c00c100 con 0x7f6d8c003a30 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c00c2c0 con 0x7f6d8c0047c0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 0x7f6d8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6dab400640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d8c0047c0 0x7f6d8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6dabe00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c003a30 0x7f6d8c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 -- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d8c0047c0 msgr2=0x7f6d8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d8c0047c0 0x7f6d8c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 -- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c003a30 msgr2=0x7f6d8c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c003a30 0x7f6d8c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6db0e00640 1 -- 172.21.15.43:0/3577028607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c003520 con 0x7f6d8c0051d0 2024-01-16T15:10:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.016+0000 7f6dabe00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c003a30 0x7f6d8c003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.017+0000 7f6db0e00640 1 -- 172.21.15.43:0/3577028607 reap_dead start 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.017+0000 7f6db0e00640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 0x7f6d8c00b980 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6da400a760 tx=0x7f6d8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192b575d743c74ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4034630 con 0x7f6d8c0051d0 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da402f070 con 0x7f6d8c0051d0 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6daaa00640 1 -- 172.21.15.43:0/3577028607 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4034a10 con 0x7f6d8c0051d0 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6db1800640 1 -- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 msgr2=0x7f6d8c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6db1800640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 0x7f6d8c00b980 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6da400a760 tx=0x7f6d8c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6db1800640 1 -- 172.21.15.43:0/3577028607 shutdown_connections 2024-01-16T15:10:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6db1800640 1 --2- 172.21.15.43:0/3577028607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d8c0051d0 0x7f6d8c00b980 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.018+0000 7f6db1800640 1 -- 172.21.15.43:0/3577028607 >> 172.21.15.43:0/3577028607 conn(0x7f6dac0cec10 msgr2=0x7f6dac0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.019+0000 7f6db1800640 1 -- 172.21.15.43:0/3577028607 shutdown_connections 2024-01-16T15:10:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.019+0000 7f6db1800640 1 -- 172.21.15.43:0/3577028607 wait complete. 2024-01-16T15:10:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.021+0000 7f6db1800640 1 Processor -- start 2024-01-16T15:10:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.021+0000 7f6db1800640 1 -- start start 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c0051d0 0x7f6dac139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6dac152260 0x7f6dac1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac152c00 con 0x7f6dac1565c0 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac13a420 con 0x7f6d8c0051d0 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db1800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac070980 con 0x7f6dac152260 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53636/0 (socket says 172.21.15.43:53636) 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.022+0000 7f6db0e00640 1 -- 172.21.15.43:0/1726251806 learned_addr learned my addr 172.21.15.43:0/1726251806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6dabe00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c0051d0 0x7f6dac139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6dab400640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6dac152260 0x7f6dac1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 -- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6dac152260 msgr2=0x7f6dac1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6dac152260 0x7f6dac1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 -- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c0051d0 msgr2=0x7f6dac139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c0051d0 0x7f6dac139ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 -- 172.21.15.43:0/1726251806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0706a0 con 0x7f6dac1565c0 2024-01-16T15:10:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db0e00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6da4002230 tx=0x7f6da4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4001730 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da4001890 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4004da0 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.023+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac151960 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.024+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac151f20 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.025+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dac04e840 con 0x7f6dac1565c0 2024-01-16T15:10:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.026+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6da4031030 con 0x7f6dac1565c0 2024-01-16T15:10:47.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.027+0000 7f6da3e00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 0x7f6d8406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:47.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.027+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6da40a9c10 con 0x7f6dac1565c0 2024-01-16T15:10:47.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.027+0000 7f6dabe00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 0x7f6d8406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:47.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.028+0000 7f6dabe00640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 0x7f6d8406e350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c000f60 tx=0x7f6d9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:47.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.032+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6da4076240 con 0x7f6dac1565c0 2024-01-16T15:10:47.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:47.264+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96111", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dac0b4760 con 0x7f6d8406be90 2024-01-16T15:10:54.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.871+0000 7f6da3e00640 1 -- 172.21.15.43:0/1726251806 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6dac0b4760 con 0x7f6d8406be90 2024-01-16T15:10:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.874+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 msgr2=0x7f6d8406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.874+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 0x7f6d8406e350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c000f60 tx=0x7f6d9c009040 comp rx=0 tx=0).stop 2024-01-16T15:10:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.874+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 msgr2=0x7f6dac156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.874+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6da4002230 tx=0x7f6da4001cb0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 shutdown_connections 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6d8406be90 0x7f6d8406e350 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dac1565c0 0x7f6dac156a40 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6dac152260 0x7f6dac1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 --2- 172.21.15.43:0/1726251806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d8c0051d0 0x7f6dac139ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 >> 172.21.15.43:0/1726251806 conn(0x7f6dac0cec10 msgr2=0x7f6dac0cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 shutdown_connections 2024-01-16T15:10:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:54.875+0000 7f6db1800640 1 -- 172.21.15.43:0/1726251806 wait complete. 2024-01-16T15:10:56.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96112 2024-01-16T15:10:57.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.319+0000 7f5fe9800640 1 Processor -- start 2024-01-16T15:10:57.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.319+0000 7f5fe9800640 1 -- start start 2024-01-16T15:10:57.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.322+0000 7f5fe9800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5fe415b350 0x7f5fe40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.322+0000 7f5fe9800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe40adb60 0x7f5fe40adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.322+0000 7f5fe9800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe415b730 con 0x7f5fe409cb30 2024-01-16T15:10:57.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe9800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe40ac740 con 0x7f5fe415b350 2024-01-16T15:10:57.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe9800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40ac460 con 0x7f5fe40adb60 2024-01-16T15:10:57.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5fe415b350 0x7f5fe40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5fe415b350 0x7f5fe40abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38986/0 (socket says 172.21.15.43:38986) 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe2a00640 1 -- 172.21.15.43:0/3280710601 learned_addr learned my addr 172.21.15.43:0/3280710601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe2a00640 1 -- 172.21.15.43:0/3280710601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe40adb60 msgr2=0x7f5fe40adfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.323+0000 7f5fe2a00640 1 --2- 172.21.15.43:0/3280710601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe40adb60 0x7f5fe40adfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.324+0000 7f5fe2a00640 1 -- 172.21.15.43:0/3280710601 >> v1:172.21.15.43:6789/0 conn(0x7f5fe409cb30 legacy=0x7f5fe40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.324+0000 7f5fe2a00640 1 -- 172.21.15.43:0/3280710601 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40b2a50 con 0x7f5fe415b350 2024-01-16T15:10:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.326+0000 7f5fe2a00640 1 --2- 172.21.15.43:0/3280710601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe415b350 0x7f5fe40abe80 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009870 tx=0x7f5fd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9814b3b68df4bdcb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.326+0000 7f5fe1600640 1 -- 172.21.15.43:0/3280710601 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4030860 con 0x7f5fe415b350 2024-01-16T15:10:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.326+0000 7f5fe1600640 1 -- 172.21.15.43:0/3280710601 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd40309c0 con 0x7f5fe415b350 2024-01-16T15:10:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.326+0000 7f5fe1600640 1 -- 172.21.15.43:0/3280710601 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4030860 con 0x7f5fe415b350 2024-01-16T15:10:57.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.328+0000 7f5fe9800640 1 -- 172.21.15.43:0/3280710601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe415b350 msgr2=0x7f5fe40abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.328+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3280710601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe415b350 0x7f5fe40abe80 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009870 tx=0x7f5fd402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.329+0000 7f5fe9800640 1 -- 172.21.15.43:0/3280710601 shutdown_connections 2024-01-16T15:10:57.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.329+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3280710601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe40adb60 0x7f5fe40adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.329+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3280710601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe415b350 0x7f5fe40abe80 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.329+0000 7f5fe9800640 1 -- 172.21.15.43:0/3280710601 >> 172.21.15.43:0/3280710601 conn(0x7f5fe40b0ba0 msgr2=0x7f5fe40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.330+0000 7f5fe9800640 1 -- 172.21.15.43:0/3280710601 shutdown_connections 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.330+0000 7f5fe9800640 1 -- 172.21.15.43:0/3280710601 wait complete. 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.332+0000 7f5fe9800640 1 Processor -- start 2024-01-16T15:10:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.333+0000 7f5fe9800640 1 -- start start 2024-01-16T15:10:57.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.334+0000 7f5fe9800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 0x7f5fe4143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.334+0000 7f5fe9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 0x7f5fe41427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.334+0000 7f5fe9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 0x7f5fe414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe9800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4099a00 con 0x7f5fe40adb60 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe9800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe409a580 con 0x7f5fe415b350 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe9800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4144380 con 0x7f5fe409cb30 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe2a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 0x7f5fe4143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 0x7f5fe414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 0x7f5fe414ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:49522/0 (socket says 172.21.15.43:49522) 2024-01-16T15:10:57.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.335+0000 7f5fe3400640 1 -- 172.21.15.43:0/3372205876 learned_addr learned my addr 172.21.15.43:0/3372205876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:10:57.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.336+0000 7f5fe2000640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 0x7f5fe41427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.337+0000 7f5fe2000640 1 -- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 msgr2=0x7f5fe4143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.337+0000 7f5fe2000640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 0x7f5fe4143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.337+0000 7f5fe2000640 1 -- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 msgr2=0x7f5fe414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.337+0000 7f5fe2000640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 0x7f5fe414ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.337+0000 7f5fe2000640 1 -- 172.21.15.43:0/3372205876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd40094d0 con 0x7f5fe40adb60 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.338+0000 7f5fe2000640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 0x7f5fe41427b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00c390 tx=0x7f5fcc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.339+0000 7f5fe2a00640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 0x7f5fe4143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:10:57.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.341+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc005220 con 0x7f5fe40adb60 2024-01-16T15:10:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.342+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc005380 con 0x7f5fe40adb60 2024-01-16T15:10:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.342+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0054f0 con 0x7f5fe40adb60 2024-01-16T15:10:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.342+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe4099c50 con 0x7f5fe40adb60 2024-01-16T15:10:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.342+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe409a210 con 0x7f5fe40adb60 2024-01-16T15:10:57.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.345+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe409a3d0 con 0x7f5fe40adb60 2024-01-16T15:10:57.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.354+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5fcc007230 con 0x7f5fe40adb60 2024-01-16T15:10:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.356+0000 7f5fdaa00640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 0x7f5fbc06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.356+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5fcc087fd0 con 0x7f5fe40adb60 2024-01-16T15:10:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.361+0000 7f5fe2a00640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 0x7f5fbc06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.361+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fcc0545e0 con 0x7f5fe40adb60 2024-01-16T15:10:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.370+0000 7f5fe2a00640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 0x7f5fbc06e3a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4036000 tx=0x7f5fd403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:57.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:57.746+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96112", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe40ae4f0 con 0x7f5fbc06bee0 2024-01-16T15:10:57.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:57.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:58.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.037+0000 7f5fdaa00640 1 -- 172.21.15.43:0/3372205876 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5fd4030e30 con 0x7f5fbc06bee0 2024-01-16T15:10:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 msgr2=0x7f5fbc06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 0x7f5fbc06e3a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4036000 tx=0x7f5fd403b040 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 msgr2=0x7f5fe41427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 0x7f5fe41427b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00c390 tx=0x7f5fcc00c850 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 shutdown_connections 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5fbc06bee0 0x7f5fbc06e3a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe415b350 0x7f5fe414ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe40adb60 0x7f5fe41427b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 --2- 172.21.15.43:0/3372205876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fe409cb30 0x7f5fe4143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 >> 172.21.15.43:0/3372205876 conn(0x7f5fe40b0ba0 msgr2=0x7f5fe4071020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.039+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 shutdown_connections 2024-01-16T15:10:58.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:10:58.040+0000 7f5fe9800640 1 -- 172.21.15.43:0/3372205876 wait complete. 2024-01-16T15:11:00.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96113 2024-01-16T15:11:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.291+0000 7f673b200640 1 Processor -- start 2024-01-16T15:11:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.291+0000 7f673b200640 1 -- start start 2024-01-16T15:11:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.292+0000 7f673b200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f673c0ad450 0x7f673c154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.292+0000 7f673b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f673c070850 0x7f673c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.292+0000 7f673b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0ce650 con 0x7f673c153ea0 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.292+0000 7f673b200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0ad8e0 con 0x7f673c0ad450 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.292+0000 7f673b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c1547c0 con 0x7f673c070850 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6733e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f673c0ad450 0x7f673c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6739e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f673c153ea0 0x7f673c15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34166/0 (socket says 172.21.15.43:34166) 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6739e00640 1 -- 172.21.15.43:0/1924720770 learned_addr learned my addr 172.21.15.43:0/1924720770 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:11:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f673a800640 1 --2- 172.21.15.43:0/1924720770 >> v2:172.21.15.69:3300/0 conn(0x7f673c070850 0x7f673c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6733e00640 1 -- 172.21.15.43:0/1924720770 >> v2:172.21.15.69:3300/0 conn(0x7f673c070850 msgr2=0x7f673c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6733e00640 1 --2- 172.21.15.43:0/1924720770 >> v2:172.21.15.69:3300/0 conn(0x7f673c070850 0x7f673c071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6733e00640 1 -- 172.21.15.43:0/1924720770 >> v1:172.21.15.43:6789/0 conn(0x7f673c153ea0 legacy=0x7f673c15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.293+0000 7f6733e00640 1 -- 172.21.15.43:0/1924720770 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c15e0a0 con 0x7f673c0ad450 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.294+0000 7f6733e00640 1 --2- 172.21.15.43:0/1924720770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c154280 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f673c157e50 tx=0x7f672402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf0a19f3645c21b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.295+0000 7f6739400640 1 -- 172.21.15.43:0/1924720770 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724031830 con 0x7f673c0ad450 2024-01-16T15:11:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.295+0000 7f6739400640 1 -- 172.21.15.43:0/1924720770 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6724031990 con 0x7f673c0ad450 2024-01-16T15:11:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.295+0000 7f6739400640 1 -- 172.21.15.43:0/1924720770 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724031d20 con 0x7f673c0ad450 2024-01-16T15:11:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.295+0000 7f673b200640 1 -- 172.21.15.43:0/1924720770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 msgr2=0x7f673c154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.295+0000 7f673b200640 1 --2- 172.21.15.43:0/1924720770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c154280 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f673c157e50 tx=0x7f672402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 -- 172.21.15.43:0/1924720770 shutdown_connections 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 --2- 172.21.15.43:0/1924720770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c154280 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 --2- 172.21.15.43:0/1924720770 >> v2:172.21.15.69:3300/0 conn(0x7f673c070850 0x7f673c071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 -- 172.21.15.43:0/1924720770 >> 172.21.15.43:0/1924720770 conn(0x7f673c0cec10 msgr2=0x7f673c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 -- 172.21.15.43:0/1924720770 shutdown_connections 2024-01-16T15:11:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.296+0000 7f673b200640 1 -- 172.21.15.43:0/1924720770 wait complete. 2024-01-16T15:11:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.298+0000 7f673b200640 1 Processor -- start 2024-01-16T15:11:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.298+0000 7f673b200640 1 -- start start 2024-01-16T15:11:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f673c070850 0x7f673c13c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673c153ea0 0x7f673c0af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c13cbc0 con 0x7f673c0ad450 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b1820 con 0x7f673c153ea0 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f673b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b0e50 con 0x7f673c070850 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39028/0 (socket says 172.21.15.43:39028) 2024-01-16T15:11:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 -- 172.21.15.43:0/2662157966 learned_addr learned my addr 172.21.15.43:0/2662157966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 -- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f673c070850 msgr2=0x7f673c13c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f673c070850 0x7f673c13c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 -- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673c153ea0 msgr2=0x7f673c0af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673c153ea0 0x7f673c0af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.299+0000 7f6733e00640 1 -- 172.21.15.43:0/2662157966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724009400 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f6733e00640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f672402bee0 tx=0x7f67240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672403ba70 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672403bbf0 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240453f0 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c0b0240 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.300+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c0b0800 con 0x7f673c0ad450 2024-01-16T15:11:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.302+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c04e840 con 0x7f673c0ad450 2024-01-16T15:11:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.302+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f672404e450 con 0x7f673c0ad450 2024-01-16T15:11:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.303+0000 7f6732000640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 0x7f671406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.303+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f67240aaa80 con 0x7f673c0ad450 2024-01-16T15:11:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.304+0000 7f6739e00640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 0x7f671406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.304+0000 7f6739e00640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 0x7f671406e3a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f672c001550 tx=0x7f672c007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.308+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67240770e0 con 0x7f673c0ad450 2024-01-16T15:11:00.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:00.545+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96113", "target": ["mon-mgr", ""]}) v1 -- 0x7f673c0b0fc0 con 0x7f671406bee0 2024-01-16T15:11:07.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.670+0000 7f6732000640 1 -- 172.21.15.43:0/2662157966 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f673c0b0fc0 con 0x7f671406bee0 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 msgr2=0x7f671406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 0x7f671406e3a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f672c001550 tx=0x7f672c007a40 comp rx=0 tx=0).stop 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 msgr2=0x7f673c0b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f672402bee0 tx=0x7f67240046c0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 shutdown_connections 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f671406bee0 0x7f671406e3a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673c153ea0 0x7f673c0af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f673c0ad450 0x7f673c0b12e0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 --2- 172.21.15.43:0/2662157966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f673c070850 0x7f673c13c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 >> 172.21.15.43:0/2662157966 conn(0x7f673c0cec10 msgr2=0x7f673c0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.675+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 shutdown_connections 2024-01-16T15:11:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:07.676+0000 7f673b200640 1 -- 172.21.15.43:0/2662157966 wait complete. 2024-01-16T15:11:09.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96114 2024-01-16T15:11:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.963+0000 7f504f200640 1 Processor -- start 2024-01-16T15:11:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.964+0000 7f504f200640 1 -- start start 2024-01-16T15:11:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.965+0000 7f504f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5050144770 0x7f505015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50500ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505015f930 con 0x7f50500b35a0 2024-01-16T15:11:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50500b43b0 con 0x7f5050144770 2024-01-16T15:11:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505015f530 con 0x7f505015f0a0 2024-01-16T15:11:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5050144770 0x7f505015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f5047e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f50500b35a0 0x7f505006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48246/0 (socket says 172.21.15.43:48246) 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f5047e00640 1 -- 172.21.15.43:0/2825410524 learned_addr learned my addr 172.21.15.43:0/2825410524 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.966+0000 7f504e800640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50500ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504de00640 1 -- 172.21.15.43:0/2825410524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 msgr2=0x7f50500ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504de00640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50500ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504de00640 1 -- 172.21.15.43:0/2825410524 >> v1:172.21.15.43:6789/0 conn(0x7f50500b35a0 legacy=0x7f505006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504de00640 1 -- 172.21.15.43:0/2825410524 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50500ce9f0 con 0x7f5050144770 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504d400640 1 -- 172.21.15.43:0/2825410524 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685018029 0 0) 0x7f505015f930 con 0x7f50500b35a0 2024-01-16T15:11:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.967+0000 7f504e800640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50500ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.968+0000 7f504de00640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f505015eb60 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f50400097a0 tx=0x7f504002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbf4e99b285f925 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.968+0000 7f504d400640 1 -- 172.21.15.43:0/2825410524 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504002d040 con 0x7f5050144770 2024-01-16T15:11:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.968+0000 7f504d400640 1 -- 172.21.15.43:0/2825410524 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040031810 con 0x7f5050144770 2024-01-16T15:11:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.968+0000 7f504d400640 1 -- 172.21.15.43:0/2825410524 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040031ba0 con 0x7f5050144770 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 -- 172.21.15.43:0/2825410524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 msgr2=0x7f505015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f505015eb60 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f50400097a0 tx=0x7f504002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 -- 172.21.15.43:0/2825410524 shutdown_connections 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50500ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 --2- 172.21.15.43:0/2825410524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f505015eb60 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 -- 172.21.15.43:0/2825410524 >> 172.21.15.43:0/2825410524 conn(0x7f50500cec10 msgr2=0x7f50500cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 -- 172.21.15.43:0/2825410524 shutdown_connections 2024-01-16T15:11:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.969+0000 7f504f200640 1 -- 172.21.15.43:0/2825410524 wait complete. 2024-01-16T15:11:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.971+0000 7f504f200640 1 Processor -- start 2024-01-16T15:11:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.971+0000 7f504f200640 1 -- start start 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50500b35a0 0x7f50501eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50501432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505014a470 con 0x7f5050144770 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50501498c0 con 0x7f50500b35a0 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f504f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505014f770 con 0x7f505015f0a0 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.972+0000 7f5047e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60162/0 (socket says 172.21.15.43:60162) 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 -- 172.21.15.43:0/3082156334 learned_addr learned my addr 172.21.15.43:0/3082156334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f504de00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50500b35a0 0x7f50501eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f504e800640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50501432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 -- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 msgr2=0x7f50501432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50501432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 -- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50500b35a0 msgr2=0x7f50501eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:09.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50500b35a0 0x7f50501eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:09.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 -- 172.21.15.43:0/3082156334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040009400 con 0x7f5050144770 2024-01-16T15:11:09.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.973+0000 7f5047e00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f503800c500 tx=0x7f503800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.974+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038007560 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.974+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50380047e0 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.974+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038004b90 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.974+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5050149ad0 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.974+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505014a090 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.975+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505014a5b0 con 0x7f5050144770 2024-01-16T15:11:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.976+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5038004cf0 con 0x7f5050144770 2024-01-16T15:11:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.977+0000 7f5046000640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 0x7f502806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.977+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f50380878f0 con 0x7f5050144770 2024-01-16T15:11:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.977+0000 7f504de00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 0x7f502806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.978+0000 7f504de00640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 0x7f502806e380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5040036000 tx=0x7f504003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:09.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:09.982+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f505014a5b0 con 0x7f5050144770 2024-01-16T15:11:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:10.216+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96114", "target": ["mon-mgr", ""]}) v1 -- 0x7f50500b40d0 con 0x7f502806bec0 2024-01-16T15:11:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.510+0000 7f5046000640 1 -- 172.21.15.43:0/3082156334 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5040031b00 con 0x7f502806bec0 2024-01-16T15:11:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 msgr2=0x7f502806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 0x7f502806e380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5040036000 tx=0x7f504003c040 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 msgr2=0x7f5050142d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f503800c500 tx=0x7f503800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 shutdown_connections 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f502806bec0 0x7f502806e380 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f505015f0a0 0x7f50501432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5050144770 0x7f5050142d90 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.513+0000 7f504f200640 1 --2- 172.21.15.43:0/3082156334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50500b35a0 0x7f50501eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.514+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 >> 172.21.15.43:0/3082156334 conn(0x7f50500cec10 msgr2=0x7f50500af980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.514+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 shutdown_connections 2024-01-16T15:11:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:18.514+0000 7f504f200640 1 -- 172.21.15.43:0/3082156334 wait complete. 2024-01-16T15:11:20.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96115 2024-01-16T15:11:20.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.881+0000 7f5b19200640 1 Processor -- start 2024-01-16T15:11:20.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.882+0000 7f5b19200640 1 -- start start 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b19200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 0x7f5b140b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b19200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 0x7f5b14167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b19200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b140b23e0 con 0x7f5b14160bd0 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b19200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b140cdc70 con 0x7f5b140b3210 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b19200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140b2870 con 0x7f5b140b48d0 2024-01-16T15:11:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b11e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5b14160bd0 0x7f5b140b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41484/0 (socket says 172.21.15.43:41484) 2024-01-16T15:11:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.884+0000 7f5b11e00640 1 -- 172.21.15.43:0/3850587702 learned_addr learned my addr 172.21.15.43:0/3850587702 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:11:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.885+0000 7f5b11400640 1 --2- 172.21.15.43:0/3850587702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 0x7f5b140b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.885+0000 7f5b12800640 1 --2- 172.21.15.43:0/3850587702 >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 0x7f5b14167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b11400640 1 -- 172.21.15.43:0/3850587702 >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 msgr2=0x7f5b14167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b10a00640 1 -- 172.21.15.43:0/3850587702 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007946897 0 0) 0x7f5b140b23e0 con 0x7f5b14160bd0 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b11400640 1 --2- 172.21.15.43:0/3850587702 >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 0x7f5b14167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b11400640 1 -- 172.21.15.43:0/3850587702 >> v1:172.21.15.43:6789/0 conn(0x7f5b14160bd0 legacy=0x7f5b140b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b11400640 1 -- 172.21.15.43:0/3850587702 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b140d41b0 con 0x7f5b140b48d0 2024-01-16T15:11:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.886+0000 7f5b12800640 1 --2- 172.21.15.43:0/3850587702 >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 0x7f5b14167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.889+0000 7f5b11400640 1 --2- 172.21.15.43:0/3850587702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 0x7f5b140b1ea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0097a0 tx=0x7f5afc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc3e7dcde47395d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.891+0000 7f5b10a00640 1 -- 172.21.15.43:0/3850587702 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc031780 con 0x7f5b140b48d0 2024-01-16T15:11:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.891+0000 7f5b10a00640 1 -- 172.21.15.43:0/3850587702 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc0318e0 con 0x7f5b140b48d0 2024-01-16T15:11:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.891+0000 7f5b10a00640 1 -- 172.21.15.43:0/3850587702 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc031c70 con 0x7f5b140b48d0 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.893+0000 7f5b19200640 1 -- 172.21.15.43:0/3850587702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 msgr2=0x7f5b140b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.893+0000 7f5b19200640 1 --2- 172.21.15.43:0/3850587702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 0x7f5b140b1ea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0097a0 tx=0x7f5afc02b820 comp rx=0 tx=0).stop 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 -- 172.21.15.43:0/3850587702 shutdown_connections 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 --2- 172.21.15.43:0/3850587702 >> v2:172.21.15.43:3300/0 conn(0x7f5b140b3210 0x7f5b14167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 --2- 172.21.15.43:0/3850587702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b48d0 0x7f5b140b1ea0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 -- 172.21.15.43:0/3850587702 >> 172.21.15.43:0/3850587702 conn(0x7f5b1409c5a0 msgr2=0x7f5b140d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 -- 172.21.15.43:0/3850587702 shutdown_connections 2024-01-16T15:11:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.895+0000 7f5b19200640 1 -- 172.21.15.43:0/3850587702 wait complete. 2024-01-16T15:11:20.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.897+0000 7f5b19200640 1 Processor -- start 2024-01-16T15:11:20.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.898+0000 7f5b19200640 1 -- start start 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b3210 0x7f5b140b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b140b48d0 0x7f5b14157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140b67c0 con 0x7f5b140b48d0 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140b5e60 con 0x7f5b14160bd0 2024-01-16T15:11:20.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.899+0000 7f5b19200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140b8620 con 0x7f5b140b3210 2024-01-16T15:11:20.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:38570/0 (socket says 172.21.15.43:38570) 2024-01-16T15:11:20.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 -- 172.21.15.43:0/2550182579 learned_addr learned my addr 172.21.15.43:0/2550182579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b11e00640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b3210 0x7f5b140b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b11400640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b140b48d0 0x7f5b14157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 -- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b3210 msgr2=0x7f5b140b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b3210 0x7f5b140b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 -- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b140b48d0 msgr2=0x7f5b14157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b140b48d0 0x7f5b14157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.900+0000 7f5b12800640 1 -- 172.21.15.43:0/2550182579 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc009400 con 0x7f5b14160bd0 2024-01-16T15:11:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.901+0000 7f5b12800640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e810 tx=0x7f5b0400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.901+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04004030 con 0x7f5b14160bd0 2024-01-16T15:11:20.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.901+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04004190 con 0x7f5b14160bd0 2024-01-16T15:11:20.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.901+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b140b6100 con 0x7f5b14160bd0 2024-01-16T15:11:20.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.901+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04004500 con 0x7f5b14160bd0 2024-01-16T15:11:20.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.902+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b14158240 con 0x7f5b14160bd0 2024-01-16T15:11:20.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.904+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b140b6900 con 0x7f5b14160bd0 2024-01-16T15:11:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.906+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5b0400c6d0 con 0x7f5b14160bd0 2024-01-16T15:11:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.907+0000 7f5b0a000640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 0x7f5aec06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.908+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5b040883f0 con 0x7f5b14160bd0 2024-01-16T15:11:20.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.912+0000 7f5b11e00640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 0x7f5aec06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.915+0000 7f5b11e00640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 0x7f5aec06e420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0012e0 tx=0x7f5b0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:20.917+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b140b6900 con 0x7f5b14160bd0 2024-01-16T15:11:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:21.324+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96115", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b140b3760 con 0x7f5aec06bf60 2024-01-16T15:11:28.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:28.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:28.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.409+0000 7f5b0a000640 1 -- 172.21.15.43:0/2550182579 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b0c00d7d0 con 0x7f5aec06bf60 2024-01-16T15:11:28.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.413+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 msgr2=0x7f5aec06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.413+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 0x7f5aec06e420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0012e0 tx=0x7f5b0c009040 comp rx=0 tx=0).stop 2024-01-16T15:11:28.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.413+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 msgr2=0x7f5b140b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.413+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e810 tx=0x7f5b0400ecd0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 shutdown_connections 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5aec06bf60 0x7f5aec06e420 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b14160bd0 0x7f5b140b7c40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b140b48d0 0x7f5b14157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 --2- 172.21.15.43:0/2550182579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b140b3210 0x7f5b140b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 >> 172.21.15.43:0/2550182579 conn(0x7f5b1409c5a0 msgr2=0x7f5b1406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 shutdown_connections 2024-01-16T15:11:28.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:28.414+0000 7f5b19200640 1 -- 172.21.15.43:0/2550182579 wait complete. 2024-01-16T15:11:30.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96116 2024-01-16T15:11:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.804+0000 7fe71f200640 1 Processor -- start 2024-01-16T15:11:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.804+0000 7fe71f200640 1 -- start start 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.806+0000 7fe71f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 0x7fe7180b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.806+0000 7fe71f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 0x7fe718167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.806+0000 7fe71f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7180b23e0 con 0x7fe718160bd0 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.806+0000 7fe71f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7180cdc70 con 0x7fe7180b48d0 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.806+0000 7fe71f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b2870 con 0x7fe7180b3210 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.807+0000 7fe717400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 0x7fe7180b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.807+0000 7fe717400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 0x7fe7180b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47972/0 (socket says 172.21.15.43:47972) 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.807+0000 7fe717400640 1 -- 172.21.15.43:0/3201501475 learned_addr learned my addr 172.21.15.43:0/3201501475 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.807+0000 7fe71d200640 1 --2- 172.21.15.43:0/3201501475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 0x7fe718167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.808+0000 7fe71d200640 1 -- 172.21.15.43:0/3201501475 >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 msgr2=0x7fe7180b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.808+0000 7fe71d200640 1 --2- 172.21.15.43:0/3201501475 >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 0x7fe7180b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.808+0000 7fe71d200640 1 -- 172.21.15.43:0/3201501475 >> v1:172.21.15.69:6789/0 conn(0x7fe718160bd0 legacy=0x7fe7180b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.808+0000 7fe71d200640 1 -- 172.21.15.43:0/3201501475 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180d41b0 con 0x7fe7180b3210 2024-01-16T15:11:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.809+0000 7fe71d200640 1 --2- 172.21.15.43:0/3201501475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 0x7fe718167290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe710009d90 tx=0x7fe71002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564b16abde4ff0d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.810+0000 7fe716a00640 1 -- 172.21.15.43:0/3201501475 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71002cb40 con 0x7fe7180b3210 2024-01-16T15:11:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.810+0000 7fe716a00640 1 -- 172.21.15.43:0/3201501475 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71002cca0 con 0x7fe7180b3210 2024-01-16T15:11:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.810+0000 7fe716a00640 1 -- 172.21.15.43:0/3201501475 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710034660 con 0x7fe7180b3210 2024-01-16T15:11:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 -- 172.21.15.43:0/3201501475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 msgr2=0x7fe718167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 --2- 172.21.15.43:0/3201501475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 0x7fe718167290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe710009d90 tx=0x7fe71002bd90 comp rx=0 tx=0).stop 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 -- 172.21.15.43:0/3201501475 shutdown_connections 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 --2- 172.21.15.43:0/3201501475 >> v2:172.21.15.43:3300/0 conn(0x7fe7180b48d0 0x7fe7180b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 --2- 172.21.15.43:0/3201501475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b3210 0x7fe718167290 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 -- 172.21.15.43:0/3201501475 >> 172.21.15.43:0/3201501475 conn(0x7fe71809c5a0 msgr2=0x7fe7180d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 -- 172.21.15.43:0/3201501475 shutdown_connections 2024-01-16T15:11:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.811+0000 7fe71f200640 1 -- 172.21.15.43:0/3201501475 wait complete. 2024-01-16T15:11:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.814+0000 7fe71f200640 1 Processor -- start 2024-01-16T15:11:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.814+0000 7fe71f200640 1 -- start start 2024-01-16T15:11:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7180b3210 0x7fe718159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718160bd0 0x7fe7181f3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180c6ea0 con 0x7fe7180b3210 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b12f0 con 0x7fe718160bd0 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe718159d70 con 0x7fe7180b48d0 2024-01-16T15:11:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe717400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe717400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:47070/0 (socket says 172.21.15.43:47070) 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe717400640 1 -- 172.21.15.43:0/3320643944 learned_addr learned my addr 172.21.15.43:0/3320643944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.816+0000 7fe71d200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718160bd0 0x7fe7181f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 -- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718160bd0 msgr2=0x7fe7181f3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718160bd0 0x7fe7181f3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 -- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7180b3210 msgr2=0x7fe718159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7180b3210 0x7fe718159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 -- 172.21.15.43:0/3320643944 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710009a70 con 0x7fe7180b48d0 2024-01-16T15:11:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.817+0000 7fe717400640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe700009f70 tx=0x7fe70000e600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.818+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7000133f0 con 0x7fe7180b48d0 2024-01-16T15:11:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.818+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe700017070 con 0x7fe7180b48d0 2024-01-16T15:11:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.818+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700013750 con 0x7fe7180b48d0 2024-01-16T15:11:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.818+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71815a050 con 0x7fe7180b48d0 2024-01-16T15:11:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.819+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71815a610 con 0x7fe7180b48d0 2024-01-16T15:11:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.822+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7180d0980 con 0x7fe7180b48d0 2024-01-16T15:11:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.823+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe700012350 con 0x7fe7180b48d0 2024-01-16T15:11:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.825+0000 7fe70fe00640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 0x7fe6f006e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.825+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe700088020 con 0x7fe7180b48d0 2024-01-16T15:11:30.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.830+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe700054990 con 0x7fe7180b48d0 2024-01-16T15:11:30.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.839+0000 7fe717e00640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 0x7fe6f006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:30.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:30.845+0000 7fe717e00640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 0x7fe6f006e420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:31.133+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96116", "target": ["mon-mgr", ""]}) v1 -- 0x7fe718066750 con 0x7fe6f006bf60 2024-01-16T15:11:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.473+0000 7fe70fe00640 1 -- 172.21.15.43:0/3320643944 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe718066750 con 0x7fe6f006bf60 2024-01-16T15:11:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 msgr2=0x7fe6f006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 0x7fe6f006e420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708006bc0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 msgr2=0x7fe7181f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe700009f70 tx=0x7fe70000e600 comp rx=0 tx=0).stop 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 shutdown_connections 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f006bf60 0x7fe6f006e420 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718160bd0 0x7fe7181f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7180b48d0 0x7fe7181f3030 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 --2- 172.21.15.43:0/3320643944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7180b3210 0x7fe718159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.476+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 >> 172.21.15.43:0/3320643944 conn(0x7fe71809c5a0 msgr2=0x7fe71806c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.477+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 shutdown_connections 2024-01-16T15:11:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:32.477+0000 7fe71f200640 1 -- 172.21.15.43:0/3320643944 wait complete. 2024-01-16T15:11:34.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96117 2024-01-16T15:11:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.733+0000 7f2daea00640 1 Processor -- start 2024-01-16T15:11:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.733+0000 7f2daea00640 1 -- start start 2024-01-16T15:11:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.735+0000 7f2daea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.735+0000 7f2daea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2da80b2f40 0x7f2da8158310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.735+0000 7f2daea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da806d700 con 0x7f2da806d2a0 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.735+0000 7f2daea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da815aec0 con 0x7f2da80b2f40 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.735+0000 7f2daea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8154440 con 0x7f2da815aae0 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.736+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.736+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:58970/0 (socket says 172.21.15.43:58970) 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.736+0000 7f2da7e00640 1 -- 172.21.15.43:0/3891604157 learned_addr learned my addr 172.21.15.43:0/3891604157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.736+0000 7f2daca00640 1 --2- 172.21.15.43:0/3891604157 >> v2:172.21.15.69:3300/0 conn(0x7f2da80b2f40 0x7f2da8158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.736+0000 7f2da6a00640 1 -- 172.21.15.43:0/3891604157 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909482847 0 0) 0x7f2da806d700 con 0x7f2da806d2a0 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.737+0000 7f2da6a00640 1 -- 172.21.15.43:0/3891604157 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d88003520 con 0x7f2da806d2a0 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.737+0000 7f2da7e00640 1 -- 172.21.15.43:0/3891604157 >> v2:172.21.15.69:3300/0 conn(0x7f2da80b2f40 msgr2=0x7f2da8158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.737+0000 7f2da7e00640 1 --2- 172.21.15.43:0/3891604157 >> v2:172.21.15.69:3300/0 conn(0x7f2da80b2f40 0x7f2da8158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.737+0000 7f2da7e00640 1 -- 172.21.15.43:0/3891604157 >> v1:172.21.15.69:6789/0 conn(0x7f2da806d2a0 legacy=0x7f2da80b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.737+0000 7f2da7e00640 1 -- 172.21.15.43:0/3891604157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8157b30 con 0x7f2da815aae0 2024-01-16T15:11:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.738+0000 7f2da7e00640 1 --2- 172.21.15.43:0/3891604157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009870 tx=0x7f2d9002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d3fb595ce6d50f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.738+0000 7f2da6a00640 1 -- 172.21.15.43:0/3891604157 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90030840 con 0x7f2da815aae0 2024-01-16T15:11:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.738+0000 7f2da6a00640 1 -- 172.21.15.43:0/3891604157 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d900309a0 con 0x7f2da815aae0 2024-01-16T15:11:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.738+0000 7f2da6a00640 1 -- 172.21.15.43:0/3891604157 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90030d30 con 0x7f2da815aae0 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 -- 172.21.15.43:0/3891604157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 msgr2=0x7f2da806cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 --2- 172.21.15.43:0/3891604157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009870 tx=0x7f2d9002b980 comp rx=0 tx=0).stop 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 -- 172.21.15.43:0/3891604157 shutdown_connections 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 --2- 172.21.15.43:0/3891604157 >> v2:172.21.15.69:3300/0 conn(0x7f2da80b2f40 0x7f2da8158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 --2- 172.21.15.43:0/3891604157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da815aae0 0x7f2da806cd60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 -- 172.21.15.43:0/3891604157 >> 172.21.15.43:0/3891604157 conn(0x7f2da80b0ba0 msgr2=0x7f2da80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 -- 172.21.15.43:0/3891604157 shutdown_connections 2024-01-16T15:11:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.739+0000 7f2daea00640 1 -- 172.21.15.43:0/3891604157 wait complete. 2024-01-16T15:11:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.741+0000 7f2daea00640 1 Processor -- start 2024-01-16T15:11:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.741+0000 7f2daea00640 1 -- start start 2024-01-16T15:11:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.742+0000 7f2daea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da806d2a0 0x7f2da809ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2da815aae0 0x7f2da8147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da81f62a0 con 0x7f2da80b2f40 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8154cc0 con 0x7f2da815aae0 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8154f60 con 0x7f2da806d2a0 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2da7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2da7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46502/0 (socket says 172.21.15.43:46502) 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2da7400640 1 -- 172.21.15.43:0/1800132059 learned_addr learned my addr 172.21.15.43:0/1800132059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2da7e00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da806d2a0 0x7f2da809ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.743+0000 7f2daca00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2da815aae0 0x7f2da8147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 -- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da806d2a0 msgr2=0x7f2da809ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da806d2a0 0x7f2da809ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 -- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2da815aae0 msgr2=0x7f2da8147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2da815aae0 0x7f2da8147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 -- 172.21.15.43:0/1800132059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d900094d0 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2da7400640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c7a0 tx=0x7f2d9800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98003b70 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98003cf0 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800aa30 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.744+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da8155240 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.745+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da8155800 con 0x7f2da80b2f40 2024-01-16T15:11:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.746+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da809d200 con 0x7f2da80b2f40 2024-01-16T15:11:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.747+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2d9800ab90 con 0x7f2da80b2f40 2024-01-16T15:11:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.748+0000 7f2d9fe00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 0x7f2d8006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.749+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d98088540 con 0x7f2da80b2f40 2024-01-16T15:11:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.749+0000 7f2da7e00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 0x7f2d8006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.749+0000 7f2da7e00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 0x7f2d8006e350 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2d90036000 tx=0x7f2d90008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.754+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d98054c50 con 0x7f2da80b2f40 2024-01-16T15:11:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:34.985+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96117", "target": ["mon-mgr", ""]}) v1 -- 0x7f2da80b3ab0 con 0x7f2d8006be90 2024-01-16T15:11:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.364+0000 7f2d9fe00640 1 -- 172.21.15.43:0/1800132059 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2d90030b90 con 0x7f2d8006be90 2024-01-16T15:11:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 msgr2=0x7f2d8006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 0x7f2d8006e350 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2d90036000 tx=0x7f2d90008030 comp rx=0 tx=0).stop 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 msgr2=0x7f2da809bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c7a0 tx=0x7f2d9800cc60 comp rx=0 tx=0).stop 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 shutdown_connections 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d8006be90 0x7f2d8006e350 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2da815aae0 0x7f2da8147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2da80b2f40 0x7f2da809bfc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 --2- 172.21.15.43:0/1800132059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2da806d2a0 0x7f2da809ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.369+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 >> 172.21.15.43:0/1800132059 conn(0x7f2da80b0ba0 msgr2=0x7f2da8153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.370+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 shutdown_connections 2024-01-16T15:11:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:41.370+0000 7f2daea00640 1 -- 172.21.15.43:0/1800132059 wait complete. 2024-01-16T15:11:43.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96118 2024-01-16T15:11:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.663+0000 7fc7a7800640 1 Processor -- start 2024-01-16T15:11:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.664+0000 7fc7a7800640 1 -- start start 2024-01-16T15:11:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a7800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7a00acd70 0x7fc7a006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a7800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc7a0070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a7800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a006d7a0 con 0x7fc7a00ae490 2024-01-16T15:11:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a7800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a015a5b0 con 0x7fc7a00acd70 2024-01-16T15:11:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a7800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a006ce70 con 0x7fc7a0070490 2024-01-16T15:11:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc797c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc7a00ae490 0x7fc7a00ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:39852/0 (socket says 172.21.15.43:39852) 2024-01-16T15:11:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc797c00640 1 -- 172.21.15.43:0/1288497236 learned_addr learned my addr 172.21.15.43:0/1288497236 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487428882 0 0) 0x7fc7a006d7a0 con 0x7fc7a00ae490 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a4e00640 1 --2- 172.21.15.43:0/1288497236 >> v2:172.21.15.69:3300/0 conn(0x7fc7a00acd70 0x7fc7a006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.665+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc7a0070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc780003520 con 0x7fc7a00ae490 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3365939377 0 0) 0x7fc780003520 con 0x7fc7a00ae490 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 msgr2=0x7fc7a0070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc7a0070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 >> v2:172.21.15.69:3300/0 conn(0x7fc7a00acd70 msgr2=0x7fc7a006dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 --2- 172.21.15.43:0/1288497236 >> v2:172.21.15.69:3300/0 conn(0x7fc7a00acd70 0x7fc7a006dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a006d150 con 0x7fc7a00ae490 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc788001670 con 0x7fc7a00ae490 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 >> v1:172.21.15.43:6789/0 conn(0x7fc7a00ae490 legacy=0x7fc7a00ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc780003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc780003ef0 0x7fc7800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc780005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015a5b0 con 0x7fc780004d00 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a006ce70 con 0x7fc780003ef0 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78000b980 con 0x7fc7a0070490 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.666+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc780005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a4e00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc780003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc797c00640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc780003ef0 0x7fc7800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a5800640 1 -- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 msgr2=0x7fc780003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0070490 0x7fc780003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a5800640 1 -- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc780003ef0 msgr2=0x7fc7800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc780003ef0 0x7fc7800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.667+0000 7fc7a5800640 1 -- 172.21.15.43:0/1288497236 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc780003520 con 0x7fc780004d00 2024-01-16T15:11:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.668+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc780005180 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc798009fc0 tx=0x7fc7800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac503b06dd6af38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.668+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79802cc00 con 0x7fc780004d00 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.668+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800f040 con 0x7fc780004d00 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.668+0000 7fc79fe00640 1 -- 172.21.15.43:0/1288497236 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798034440 con 0x7fc780004d00 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 -- 172.21.15.43:0/1288497236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 msgr2=0x7fc780005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1288497236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc780005180 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc798009fc0 tx=0x7fc7800057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a5800640 1 -- 172.21.15.43:0/1288497236 reap_dead start 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 -- 172.21.15.43:0/1288497236 shutdown_connections 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 -- 172.21.15.43:0/1288497236 >> 172.21.15.43:0/1288497236 conn(0x7fc7a00b0ba0 msgr2=0x7fc7a00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 -- 172.21.15.43:0/1288497236 shutdown_connections 2024-01-16T15:11:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.669+0000 7fc7a7800640 1 -- 172.21.15.43:0/1288497236 wait complete. 2024-01-16T15:11:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.671+0000 7fc7a7800640 1 Processor -- start 2024-01-16T15:11:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.671+0000 7fc7a7800640 1 -- start start 2024-01-16T15:11:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0099e10 0x7fc7a009c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a009c7c0 0x7fc7a00cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00c9a60 con 0x7fc780004d00 2024-01-16T15:11:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a009d040 con 0x7fc7a009c7c0 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a7800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a0151110 con 0x7fc7a0099e10 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38440/0 (socket says 172.21.15.43:38440) 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc7a4e00640 1 -- 172.21.15.43:0/1375103382 learned_addr learned my addr 172.21.15.43:0/1375103382 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.672+0000 7fc797c00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0099e10 0x7fc7a009c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a5800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a009c7c0 0x7fc7a00cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 -- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0099e10 msgr2=0x7fc7a009c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0099e10 0x7fc7a009c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 -- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a009c7c0 msgr2=0x7fc7a00cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a009c7c0 0x7fc7a00cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 -- 172.21.15.43:0/1375103382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a009cd60 con 0x7fc780004d00 2024-01-16T15:11:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a4e00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc790011a70 tx=0x7fc790011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790009680 con 0x7fc780004d00 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79000fb50 con 0x7fc780004d00 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000fee0 con 0x7fc780004d00 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.673+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a00c93a0 con 0x7fc780004d00 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.674+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a01f2f00 con 0x7fc780004d00 2024-01-16T15:11:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.675+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a00718b0 con 0x7fc780004d00 2024-01-16T15:11:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.676+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc790014440 con 0x7fc780004d00 2024-01-16T15:11:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.676+0000 7fc79d600640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 0x7fc77806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.677+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc790089180 con 0x7fc780004d00 2024-01-16T15:11:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.677+0000 7fc797c00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 0x7fc77806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.677+0000 7fc797c00640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 0x7fc77806e350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc7a009a1f0 tx=0x7fc788009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.681+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7900557e0 con 0x7fc780004d00 2024-01-16T15:11:43.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:43.908+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96118", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7a00ad150 con 0x7fc77806be90 2024-01-16T15:11:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:46.201+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fc790055bb0 con 0x7fc780004d00 2024-01-16T15:11:52.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.488+0000 7fc79d600640 1 -- 172.21.15.43:0/1375103382 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc7a00ad150 con 0x7fc77806be90 2024-01-16T15:11:52.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 msgr2=0x7fc77806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 0x7fc77806e350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc7a009a1f0 tx=0x7fc788009040 comp rx=0 tx=0).stop 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 msgr2=0x7fc7a0150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc790011a70 tx=0x7fc790011f30 comp rx=0 tx=0).stop 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 shutdown_connections 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc77806be90 0x7fc77806e350 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7a009c7c0 0x7fc7a00cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7a0099e10 0x7fc7a009c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 --2- 172.21.15.43:0/1375103382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc780004d00 0x7fc7a0150bd0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:52.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 >> 172.21.15.43:0/1375103382 conn(0x7fc7a00b0ba0 msgr2=0x7fc7a00aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:52.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 shutdown_connections 2024-01-16T15:11:52.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:52.492+0000 7fc7a7800640 1 -- 172.21.15.43:0/1375103382 wait complete. 2024-01-16T15:11:54.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96119 2024-01-16T15:11:54.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.876+0000 7f73e7600640 1 Processor -- start 2024-01-16T15:11:54.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.876+0000 7f73e7600640 1 -- start start 2024-01-16T15:11:54.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.879+0000 7f73e7600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73e00b48d0 0x7f73e00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.879+0000 7f73e7600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 0x7f73e0167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.879+0000 7f73e7600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00b23e0 con 0x7f73e0160bd0 2024-01-16T15:11:54.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.879+0000 7f73e7600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00cdc70 con 0x7f73e00b48d0 2024-01-16T15:11:54.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.879+0000 7f73e7600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00b2870 con 0x7f73e00b3210 2024-01-16T15:11:54.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.880+0000 7f73e4c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f73e0160bd0 0x7f73e00b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:53760/0 (socket says 172.21.15.43:53760) 2024-01-16T15:11:54.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.880+0000 7f73e4c00640 1 -- 172.21.15.43:0/2859482421 learned_addr learned my addr 172.21.15.43:0/2859482421 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:11:54.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.880+0000 7f73e5600640 1 --2- 172.21.15.43:0/2859482421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 0x7f73e0167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:54.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73df400640 1 -- 172.21.15.43:0/2859482421 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724479580 0 0) 0x7f73e00b23e0 con 0x7f73e0160bd0 2024-01-16T15:11:54.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73df400640 1 -- 172.21.15.43:0/2859482421 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c0003520 con 0x7f73e0160bd0 2024-01-16T15:11:54.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73e5600640 1 -- 172.21.15.43:0/2859482421 >> v2:172.21.15.69:3300/0 conn(0x7f73e00b48d0 msgr2=0x7f73e00b1ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:11:54.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73e5600640 1 --2- 172.21.15.43:0/2859482421 >> v2:172.21.15.69:3300/0 conn(0x7f73e00b48d0 0x7f73e00b1ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73e5600640 1 -- 172.21.15.43:0/2859482421 >> v1:172.21.15.69:6789/0 conn(0x7f73e0160bd0 legacy=0x7f73e00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.882+0000 7f73e5600640 1 -- 172.21.15.43:0/2859482421 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00d41b0 con 0x7f73e00b3210 2024-01-16T15:11:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.887+0000 7f73e5600640 1 --2- 172.21.15.43:0/2859482421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 0x7f73e0167290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009d90 tx=0x7f73d802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c572888caf03153 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.888+0000 7f73df400640 1 -- 172.21.15.43:0/2859482421 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d802cbf0 con 0x7f73e00b3210 2024-01-16T15:11:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.889+0000 7f73df400640 1 -- 172.21.15.43:0/2859482421 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d802cd50 con 0x7f73e00b3210 2024-01-16T15:11:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.889+0000 7f73df400640 1 -- 172.21.15.43:0/2859482421 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8034660 con 0x7f73e00b3210 2024-01-16T15:11:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 -- 172.21.15.43:0/2859482421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 msgr2=0x7f73e0167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 --2- 172.21.15.43:0/2859482421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 0x7f73e0167290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009d90 tx=0x7f73d802be40 comp rx=0 tx=0).stop 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 -- 172.21.15.43:0/2859482421 shutdown_connections 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 --2- 172.21.15.43:0/2859482421 >> v2:172.21.15.69:3300/0 conn(0x7f73e00b48d0 0x7f73e00b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 --2- 172.21.15.43:0/2859482421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e00b3210 0x7f73e0167290 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 -- 172.21.15.43:0/2859482421 >> 172.21.15.43:0/2859482421 conn(0x7f73e009c5a0 msgr2=0x7f73e00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 -- 172.21.15.43:0/2859482421 shutdown_connections 2024-01-16T15:11:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.890+0000 7f73e7600640 1 -- 172.21.15.43:0/2859482421 wait complete. 2024-01-16T15:11:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.893+0000 7f73e7600640 1 Processor -- start 2024-01-16T15:11:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.894+0000 7f73e7600640 1 -- start start 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.895+0000 7f73e7600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.895+0000 7f73e7600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e00b48d0 0x7f73e00c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e7600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e0160bd0 0x7f73e01586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e7600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e015b450 con 0x7f73e00b3210 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e7600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0158be0 con 0x7f73e00b48d0 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e7600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e015a7d0 con 0x7f73e0160bd0 2024-01-16T15:11:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52930/0 (socket says 172.21.15.43:52930) 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.896+0000 7f73e4c00640 1 -- 172.21.15.43:0/2438383416 learned_addr learned my addr 172.21.15.43:0/2438383416 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 -- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e0160bd0 msgr2=0x7f73e01586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e0160bd0 0x7f73e01586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 -- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e00b48d0 msgr2=0x7f73e00c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e00b48d0 0x7f73e00c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 -- 172.21.15.43:0/2438383416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8009a70 con 0x7f73e00b3210 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73e4c00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c410 tx=0x7f73d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.897+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0005ce0 con 0x7f73e00b3210 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.898+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d0005e40 con 0x7f73e00b3210 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.898+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0007310 con 0x7f73e00b3210 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.898+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e015aab0 con 0x7f73e00b3210 2024-01-16T15:11:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.898+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e015b070 con 0x7f73e00b3210 2024-01-16T15:11:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.900+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0005250 con 0x7f73e00b3210 2024-01-16T15:11:54.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.904+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f73d0007470 con 0x7f73e00b3210 2024-01-16T15:11:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.907+0000 7f73dcc00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 0x7f73b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.907+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73d0087d80 con 0x7f73e00b3210 2024-01-16T15:11:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.910+0000 7f73dfe00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 0x7f73b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:54.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.916+0000 7f73dfe00640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 0x7f73b806e2f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73e015be30 tx=0x7f73c8006c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:54.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:54.917+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73d00543f0 con 0x7f73e00b3210 2024-01-16T15:11:55.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:11:55.311+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96119", "target": ["mon-mgr", ""]}) v1 -- 0x7f73b0000bc0 con 0x7f73b806be30 2024-01-16T15:11:58.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:58.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:01.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.934+0000 7f73dcc00640 1 -- 172.21.15.43:0/2438383416 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f73b0000bc0 con 0x7f73b806be30 2024-01-16T15:12:01.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 msgr2=0x7f73b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:01.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 0x7f73b806e2f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f73e015be30 tx=0x7f73c8006c80 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 msgr2=0x7f73e015a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c410 tx=0x7f73d000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 shutdown_connections 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f73b806be30 0x7f73b806e2f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73e00b48d0 0x7f73e00c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73e0160bd0 0x7f73e01586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 --2- 172.21.15.43:0/2438383416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73e00b3210 0x7f73e015a160 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 >> 172.21.15.43:0/2438383416 conn(0x7f73e009c5a0 msgr2=0x7f73e00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:01.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 shutdown_connections 2024-01-16T15:12:01.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:01.938+0000 7f73e7600640 1 -- 172.21.15.43:0/2438383416 wait complete. 2024-01-16T15:12:03.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone96120 2024-01-16T15:12:04.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.197+0000 7f4818c00640 1 Processor -- start 2024-01-16T15:12:04.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.198+0000 7f4818c00640 1 -- start start 2024-01-16T15:12:04.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.199+0000 7f4818c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814153ea0 0x7f48140ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.199+0000 7f4818c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4814154280 0x7f481415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.199+0000 7f4818c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48140ce650 con 0x7f4814070850 2024-01-16T15:12:04.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.199+0000 7f4818c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481415ddd0 con 0x7f4814154280 2024-01-16T15:12:04.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.199+0000 7f4818c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481415e0a0 con 0x7f4814153ea0 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4812a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4814154280 0x7f481415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4813e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4814070850 0x7f4814071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34648/0 (socket says 172.21.15.43:34648) 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4813e00640 1 -- 172.21.15.43:0/666170826 learned_addr learned my addr 172.21.15.43:0/666170826 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4813400640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814153ea0 0x7f48140ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4812a00640 1 -- 172.21.15.43:0/666170826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814153ea0 msgr2=0x7f48140ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:04.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4812a00640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814153ea0 0x7f48140ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.200+0000 7f4812a00640 1 -- 172.21.15.43:0/666170826 >> v1:172.21.15.43:6789/0 conn(0x7f4814070850 legacy=0x7f4814071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.201+0000 7f4812a00640 1 -- 172.21.15.43:0/666170826 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481415e380 con 0x7f4814154280 2024-01-16T15:12:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.201+0000 7f4812000640 1 -- 172.21.15.43:0/666170826 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906229224 0 0) 0x7f48140ce650 con 0x7f4814070850 2024-01-16T15:12:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.202+0000 7f4812a00640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814154280 0x7f481415d890 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0097a0 tx=0x7f47fc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2d0d4268d24e08 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:04.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.202+0000 7f4812000640 1 -- 172.21.15.43:0/666170826 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc02d040 con 0x7f4814154280 2024-01-16T15:12:04.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.202+0000 7f4812000640 1 -- 172.21.15.43:0/666170826 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc031810 con 0x7f4814154280 2024-01-16T15:12:04.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.202+0000 7f4812000640 1 -- 172.21.15.43:0/666170826 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc031ba0 con 0x7f4814154280 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 -- 172.21.15.43:0/666170826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814154280 msgr2=0x7f481415d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814154280 0x7f481415d890 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0097a0 tx=0x7f47fc02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 -- 172.21.15.43:0/666170826 shutdown_connections 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814154280 0x7f481415d890 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 --2- 172.21.15.43:0/666170826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814153ea0 0x7f48140ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 -- 172.21.15.43:0/666170826 >> 172.21.15.43:0/666170826 conn(0x7f48140cec10 msgr2=0x7f48140cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 -- 172.21.15.43:0/666170826 shutdown_connections 2024-01-16T15:12:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.203+0000 7f4818c00640 1 -- 172.21.15.43:0/666170826 wait complete. 2024-01-16T15:12:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.205+0000 7f4818c00640 1 Processor -- start 2024-01-16T15:12:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.205+0000 7f4818c00640 1 -- start start 2024-01-16T15:12:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4814153ea0 0x7f4814150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 0x7f48141e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48141e83b0 con 0x7f4814070850 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48141e6ac0 con 0x7f4814153ea0 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4818c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48141e80d0 con 0x7f4814154280 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4813400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4813400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32786/0 (socket says 172.21.15.43:32786) 2024-01-16T15:12:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4813400640 1 -- 172.21.15.43:0/82005773 learned_addr learned my addr 172.21.15.43:0/82005773 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.206+0000 7f4813e00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 0x7f48141e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4812a00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4814153ea0 0x7f4814150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 -- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 msgr2=0x7f48141e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 0x7f48141e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 -- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4814153ea0 msgr2=0x7f4814150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4814153ea0 0x7f4814150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 -- 172.21.15.43:0/82005773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc009400 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813e00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 0x7f48141e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4813400640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f480400d830 tx=0x7f480400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400b660 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480400b7c0 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400bb50 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48141e5aa0 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.207+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48141e6060 con 0x7f4814070850 2024-01-16T15:12:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.209+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48141e5900 con 0x7f4814070850 2024-01-16T15:12:04.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.210+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f480401e460 con 0x7f4814070850 2024-01-16T15:12:04.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.211+0000 7f480b400640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 0x7f47ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:04.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.211+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4804088c80 con 0x7f4814070850 2024-01-16T15:12:04.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.211+0000 7f4812a00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 0x7f47ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.212+0000 7f4812a00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 0x7f47ec06e2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f47fc036000 tx=0x7f47fc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.216+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48141e5900 con 0x7f4814070850 2024-01-16T15:12:04.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:04.448+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone96120", "target": ["mon-mgr", ""]}) v1 -- 0x7f4814070330 con 0x7f47ec06be30 2024-01-16T15:12:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.829+0000 7f480b400640 1 -- 172.21.15.43:0/82005773 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f47fc031b00 con 0x7f47ec06be30 2024-01-16T15:12:08.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.832+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 msgr2=0x7f47ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:08.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.832+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 0x7f47ec06e2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f47fc036000 tx=0x7f47fc03c040 comp rx=0 tx=0).stop 2024-01-16T15:12:08.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.832+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 msgr2=0x7f48141e6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:08.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.832+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f480400d830 tx=0x7f480400dcf0 comp rx=0 tx=0).stop 2024-01-16T15:12:08.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 shutdown_connections 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f47ec06be30 0x7f47ec06e2f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4814153ea0 0x7f4814150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4814154280 0x7f48141e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 --2- 172.21.15.43:0/82005773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4814070850 0x7f48141e6580 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 >> 172.21.15.43:0/82005773 conn(0x7f48140cec10 msgr2=0x7f48140b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 shutdown_connections 2024-01-16T15:12:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:08.833+0000 7f4818c00640 1 -- 172.21.15.43:0/82005773 wait complete. 2024-01-16T15:12:10.834 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:10.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9611 2024-01-16T15:12:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.096+0000 7fcd32400640 1 Processor -- start 2024-01-16T15:12:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.096+0000 7fcd32400640 1 -- start start 2024-01-16T15:12:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.097+0000 7fcd32400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcd2c15d3b0 0x7fcd2c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd32400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0c6bf0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd32400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0d1430 con 0x7fcd2c0d1d90 2024-01-16T15:12:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd32400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0cedc0 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd31000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcd2c15d3b0 0x7fcd2c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd2be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcd2c0d0fa0 0x7fcd2c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34656/0 (socket says 172.21.15.43:34656) 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.098+0000 7fcd2be00640 1 -- 172.21.15.43:0/3437664676 learned_addr learned my addr 172.21.15.43:0/3437664676 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.099+0000 7fcd31000640 1 -- 172.21.15.43:0/3437664676 >> v1:172.21.15.69:6789/0 conn(0x7fcd2c0d1d90 legacy=0x7fcd2c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.099+0000 7fcd31000640 1 -- 172.21.15.43:0/3437664676 >> v1:172.21.15.43:6789/0 conn(0x7fcd2c0d0fa0 legacy=0x7fcd2c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.099+0000 7fcd31000640 1 -- 172.21.15.43:0/3437664676 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0c6e40 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.099+0000 7fcd2b400640 1 -- 172.21.15.43:0/3437664676 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292905927 0 0) 0x7fcd2c0c6bf0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.100+0000 7fcd31000640 1 --2- 172.21.15.43:0/3437664676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c15d3b0 0x7fcd2c0ce6c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0097a0 tx=0x7fcd1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807984db521ec7a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.100+0000 7fcd2b400640 1 -- 172.21.15.43:0/3437664676 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c031830 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.100+0000 7fcd2b400640 1 -- 172.21.15.43:0/3437664676 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c031990 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.100+0000 7fcd2b400640 1 -- 172.21.15.43:0/3437664676 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c031d20 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 -- 172.21.15.43:0/3437664676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c15d3b0 msgr2=0x7fcd2c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 --2- 172.21.15.43:0/3437664676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c15d3b0 0x7fcd2c0ce6c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0097a0 tx=0x7fcd1c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 -- 172.21.15.43:0/3437664676 shutdown_connections 2024-01-16T15:12:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 --2- 172.21.15.43:0/3437664676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c15d3b0 0x7fcd2c0ce6c0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 -- 172.21.15.43:0/3437664676 >> 172.21.15.43:0/3437664676 conn(0x7fcd2c0d30a0 msgr2=0x7fcd2c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 -- 172.21.15.43:0/3437664676 shutdown_connections 2024-01-16T15:12:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.101+0000 7fcd32400640 1 -- 172.21.15.43:0/3437664676 wait complete. 2024-01-16T15:12:11.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.103+0000 7fcd32400640 1 Processor -- start 2024-01-16T15:12:11.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.103+0000 7fcd32400640 1 -- start start 2024-01-16T15:12:11.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd2c0d1d90 0x7fcd2c15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd2c15d3b0 0x7fcd2c160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c15aef0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c09f930 con 0x7fcd2c0d1d90 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd32400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c09fc10 con 0x7fcd2c15d3b0 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd31000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd31000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32804/0 (socket says 172.21.15.43:32804) 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.104+0000 7fcd31000640 1 -- 172.21.15.43:0/2281963229 learned_addr learned my addr 172.21.15.43:0/2281963229 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 -- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd2c15d3b0 msgr2=0x7fcd2c160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd2be00640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd2c0d1d90 0x7fcd2c15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31a00640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd2c15d3b0 0x7fcd2c160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd2c15d3b0 0x7fcd2c160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 -- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd2c0d1d90 msgr2=0x7fcd2c15fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd2c0d1d90 0x7fcd2c15fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 -- 172.21.15.43:0/2281963229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c009400 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.105+0000 7fcd31000640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c000c00 tx=0x7fcd1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.106+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c03aa60 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.106+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c03abc0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.106+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0443f0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.106+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c09fef0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.106+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0a04b0 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.107+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c04e840 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.108+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fcd1c04d450 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.109+0000 7fcd28c00640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 0x7fcd0406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.109+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcd1c0aa770 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.109+0000 7fcd2be00640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 0x7fcd0406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.110+0000 7fcd2be00640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 0x7fcd0406e2c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fcd14007fd0 tx=0x7fcd14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.115+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd1c077030 con 0x7fcd2c0d0fa0 2024-01-16T15:12:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.351+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9611", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd2c15d790 con 0x7fcd0406be00 2024-01-16T15:12:11.335 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:11.359+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:11.335 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:11.359+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:11.336 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:11.359+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:11.336 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:11.359+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:11.336 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:11.359+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.389+0000 7fcd28c00640 1 -- 172.21.15.43:0/2281963229 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fcd14001310 con 0x7fcd0406be00 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:11.366 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:11.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 msgr2=0x7fcd0406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 0x7fcd0406e2c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fcd14007fd0 tx=0x7fcd14009040 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 msgr2=0x7fcd2c1514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c000c00 tx=0x7fcd1c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 shutdown_connections 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd0406be00 0x7fcd0406e2c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd2c0d1d90 0x7fcd2c15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd2c15d3b0 0x7fcd2c160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 --2- 172.21.15.43:0/2281963229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd2c0d0fa0 0x7fcd2c1514b0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 >> 172.21.15.43:0/2281963229 conn(0x7fcd2c0d30a0 msgr2=0x7fcd2c15e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 shutdown_connections 2024-01-16T15:12:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.392+0000 7fcd32400640 1 -- 172.21.15.43:0/2281963229 wait complete. 2024-01-16T15:12:11.389 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:11.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9612 2024-01-16T15:12:11.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.635+0000 7fa34a800640 1 Processor -- start 2024-01-16T15:12:11.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.635+0000 7fa34a800640 1 -- start start 2024-01-16T15:12:11.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa34a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa3440b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa34a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa34406cad0 0x7fa344162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa34a800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa344162fa0 con 0x7fa344162600 2024-01-16T15:12:11.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa34a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3440706a0 con 0x7fa34406cad0 2024-01-16T15:12:11.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa34a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa344070c60 con 0x7fa344162220 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa342a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa344162600 0x7fa34406c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34664/0 (socket says 172.21.15.43:34664) 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.637+0000 7fa342a00640 1 -- 172.21.15.43:0/1577832007 learned_addr learned my addr 172.21.15.43:0/1577832007 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa343400640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa3440b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288042291 0 0) 0x7fa344162fa0 con 0x7fa344162600 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003520 con 0x7fa344162600 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa343e00640 1 --2- 172.21.15.43:0/1577832007 >> v2:172.21.15.69:3300/0 conn(0x7fa34406cad0 0x7fa344162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129855722 0 0) 0x7fa324003520 con 0x7fa344162600 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 msgr2=0x7fa3440b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa3440b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 >> v2:172.21.15.69:3300/0 conn(0x7fa34406cad0 msgr2=0x7fa344162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 --2- 172.21.15.43:0/1577832007 >> v2:172.21.15.69:3300/0 conn(0x7fa34406cad0 0x7fa344162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440722b0 con 0x7fa344162600 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa32c001670 con 0x7fa344162600 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 >> v1:172.21.15.43:6789/0 conn(0x7fa344162600 legacy=0x7fa34406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa324003970 0x7fa324003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3240047c0 0x7fa324004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.638+0000 7fa342000640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440706a0 con 0x7fa324003970 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa344070c60 con 0x7fa3240047c0 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32400ba30 con 0x7fa344162220 2024-01-16T15:12:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa324003970 0x7fa324003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343e00640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa342a00640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3240047c0 0x7fa324004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 -- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 msgr2=0x7fa324005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa344162220 0x7fa324005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 -- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3240047c0 msgr2=0x7fa324004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3240047c0 0x7fa324004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.639+0000 7fa343400640 1 -- 172.21.15.43:0/1577832007 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324003520 con 0x7fa324003970 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.640+0000 7fa343400640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa324003970 0x7fa324003d90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa33400ba00 tx=0x7fa3240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6b82af3af4faa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340325f0 con 0x7fa324003970 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334032750 con 0x7fa324003970 2024-01-16T15:12:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa342000640 1 -- 172.21.15.43:0/1577832007 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334032b20 con 0x7fa324003970 2024-01-16T15:12:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa34a800640 1 -- 172.21.15.43:0/1577832007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa324003970 msgr2=0x7fa324003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa34a800640 1 --2- 172.21.15.43:0/1577832007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa324003970 0x7fa324003d90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa33400ba00 tx=0x7fa3240057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa343e00640 1 -- 172.21.15.43:0/1577832007 reap_dead start 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa34a800640 1 -- 172.21.15.43:0/1577832007 shutdown_connections 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.641+0000 7fa34a800640 1 -- 172.21.15.43:0/1577832007 >> 172.21.15.43:0/1577832007 conn(0x7fa3440b0ba0 msgr2=0x7fa3440b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.642+0000 7fa34a800640 1 -- 172.21.15.43:0/1577832007 shutdown_connections 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.642+0000 7fa34a800640 1 -- 172.21.15.43:0/1577832007 wait complete. 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.644+0000 7fa34a800640 1 Processor -- start 2024-01-16T15:12:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.644+0000 7fa34a800640 1 -- start start 2024-01-16T15:12:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 0x7fa34414d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa34409b9a0 0x7fa34409be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34414d8f0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440b3970 con 0x7fa324003970 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa34a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3441ec0d0 con 0x7fa34409b9a0 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.645+0000 7fa343e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32828/0 (socket says 172.21.15.43:32828) 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 -- 172.21.15.43:0/3141377092 learned_addr learned my addr 172.21.15.43:0/3141377092 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa342a00640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa34409b9a0 0x7fa34409be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343400640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 0x7fa34414d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 -- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa34409b9a0 msgr2=0x7fa34409be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa34409b9a0 0x7fa34409be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 -- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 msgr2=0x7fa34414d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 0x7fa34414d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 -- 172.21.15.43:0/3141377092 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3441eb4c0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343400640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 0x7fa34414d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.646+0000 7fa343e00640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa3440ae8d0 tx=0x7fa33c009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.647+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c007cf0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.647+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c007e70 con 0x7fa3440b2fb0 2024-01-16T15:12:11.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.647+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c014620 con 0x7fa3440b2fb0 2024-01-16T15:12:11.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.647+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3441eb7a0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.647+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3441ebd60 con 0x7fa3440b2fb0 2024-01-16T15:12:11.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.648+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34406d480 con 0x7fa3440b2fb0 2024-01-16T15:12:11.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.649+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fa33c01d3b0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.650+0000 7fa33b400640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 0x7fa31c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.650+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa33c0883e0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.651+0000 7fa343400640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 0x7fa31c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.652+0000 7fa343400640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 0x7fa31c06e2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa3240057a0 tx=0x7fa334030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:11.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.655+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa33c054ab0 con 0x7fa3440b2fb0 2024-01-16T15:12:11.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.893+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9612", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3440708f0 con 0x7fa31c06be30 2024-01-16T15:12:11.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.913+0000 7fa33b400640 1 -- 172.21.15.43:0/3141377092 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fa3340325a0 con 0x7fa31c06be30 2024-01-16T15:12:11.889 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:11.890 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:11.890 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:12:11.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:11.890 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:11.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 msgr2=0x7fa31c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 0x7fa31c06e2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa3240057a0 tx=0x7fa334030000 comp rx=0 tx=0).stop 2024-01-16T15:12:11.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 msgr2=0x7fa3440b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa3440ae8d0 tx=0x7fa33c009d40 comp rx=0 tx=0).stop 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 shutdown_connections 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa31c06be30 0x7fa31c06e2f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3440b2fb0 0x7fa3440b3430 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa34409b9a0 0x7fa34409be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 --2- 172.21.15.43:0/3141377092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa324003970 0x7fa34414d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 >> 172.21.15.43:0/3141377092 conn(0x7fa3440b0ba0 msgr2=0x7fa344071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 shutdown_connections 2024-01-16T15:12:11.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:11.916+0000 7fa34a800640 1 -- 172.21.15.43:0/3141377092 wait complete. 2024-01-16T15:12:11.911 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:11.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9613 2024-01-16T15:12:12.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.166+0000 7f8134e00640 1 Processor -- start 2024-01-16T15:12:12.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.166+0000 7f8134e00640 1 -- start start 2024-01-16T15:12:12.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f8134e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f813009cb30 0x7f81300abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f8134e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f81300adb60 0x7f81300adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f8134e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813015b730 con 0x7f813015b350 2024-01-16T15:12:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f8134e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81300ac740 con 0x7f81300adb60 2024-01-16T15:12:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f8134e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81300ac460 con 0x7f813009cb30 2024-01-16T15:12:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f812e800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f81300adb60 0x7f81300adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f812de00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f813015b350 0x7f81300b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34666/0 (socket says 172.21.15.43:34666) 2024-01-16T15:12:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.168+0000 7f812de00640 1 -- 172.21.15.43:0/3953404440 learned_addr learned my addr 172.21.15.43:0/3953404440 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f8127e00640 1 --2- 172.21.15.43:0/3953404440 >> v2:172.21.15.69:3300/0 conn(0x7f813009cb30 0x7f81300abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f812e800640 1 -- 172.21.15.43:0/3953404440 >> v2:172.21.15.69:3300/0 conn(0x7f813009cb30 msgr2=0x7f81300abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f812e800640 1 --2- 172.21.15.43:0/3953404440 >> v2:172.21.15.69:3300/0 conn(0x7f813009cb30 0x7f81300abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f812e800640 1 -- 172.21.15.43:0/3953404440 >> v1:172.21.15.43:6789/0 conn(0x7f813015b350 legacy=0x7f81300b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f812e800640 1 -- 172.21.15.43:0/3953404440 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300b2a50 con 0x7f81300adb60 2024-01-16T15:12:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.169+0000 7f812d400640 1 -- 172.21.15.43:0/3953404440 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619408680 0 0) 0x7f8120001110 con 0x7f813015b350 2024-01-16T15:12:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.170+0000 7f812e800640 1 --2- 172.21.15.43:0/3953404440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81300adb60 0x7f81300adf70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f81300d0060 tx=0x7f812802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99a1bc6a2c5fa29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.170+0000 7f812d400640 1 -- 172.21.15.43:0/3953404440 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812802cc80 con 0x7f81300adb60 2024-01-16T15:12:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.170+0000 7f812d400640 1 -- 172.21.15.43:0/3953404440 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812802cde0 con 0x7f81300adb60 2024-01-16T15:12:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.170+0000 7f812d400640 1 -- 172.21.15.43:0/3953404440 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128034660 con 0x7f81300adb60 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 -- 172.21.15.43:0/3953404440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81300adb60 msgr2=0x7f81300adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 --2- 172.21.15.43:0/3953404440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81300adb60 0x7f81300adf70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f81300d0060 tx=0x7f812802bed0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 -- 172.21.15.43:0/3953404440 shutdown_connections 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 --2- 172.21.15.43:0/3953404440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81300adb60 0x7f81300adf70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 --2- 172.21.15.43:0/3953404440 >> v2:172.21.15.69:3300/0 conn(0x7f813009cb30 0x7f81300abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 -- 172.21.15.43:0/3953404440 >> 172.21.15.43:0/3953404440 conn(0x7f81300b0ba0 msgr2=0x7f81300b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 -- 172.21.15.43:0/3953404440 shutdown_connections 2024-01-16T15:12:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.171+0000 7f8134e00640 1 -- 172.21.15.43:0/3953404440 wait complete. 2024-01-16T15:12:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.173+0000 7f8134e00640 1 Processor -- start 2024-01-16T15:12:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.173+0000 7f8134e00640 1 -- start start 2024-01-16T15:12:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813009cb30 0x7f81301552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81300adb60 0x7f8130152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8130154e20 con 0x7f813015b350 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81301557f0 con 0x7f81300adb60 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f8134e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8130154210 con 0x7f813009cb30 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f812e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f812e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32844/0 (socket says 172.21.15.43:32844) 2024-01-16T15:12:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.174+0000 7f812e800640 1 -- 172.21.15.43:0/2833379348 learned_addr learned my addr 172.21.15.43:0/2833379348 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812de00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813009cb30 0x7f81301552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f8127e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81300adb60 0x7f8130152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 -- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813009cb30 msgr2=0x7f81301552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813009cb30 0x7f81301552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 -- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81300adb60 msgr2=0x7f8130152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81300adb60 0x7f8130152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 -- 172.21.15.43:0/2833379348 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128009a70 con 0x7f813015b350 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f812e800640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f812802c4e0 tx=0x7f8128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812803e610 con 0x7f813015b350 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812803e770 con 0x7f813015b350 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812803eb20 con 0x7f813015b350 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.175+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81301544f0 con 0x7f813015b350 2024-01-16T15:12:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.176+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8130154ab0 con 0x7f813015b350 2024-01-16T15:12:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.177+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8130154fd0 con 0x7f813015b350 2024-01-16T15:12:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.178+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f812803b050 con 0x7f813015b350 2024-01-16T15:12:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.179+0000 7f8126000640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 0x7f810806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.179+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f81280aaf50 con 0x7f813015b350 2024-01-16T15:12:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.179+0000 7f812de00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 0x7f810806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.180+0000 7f812de00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 0x7f810806e2c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8120000f60 tx=0x7f8120007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.183+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8130154fd0 con 0x7f813015b350 2024-01-16T15:12:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.415+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9613", "target": ["mon-mgr", ""]}) v1 -- 0x7f813006e720 con 0x7f810806be00 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.438+0000 7f8126000640 1 -- 172.21.15.43:0/2833379348 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f8120007270 con 0x7f810806be00 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:12.415 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:12.416 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.441+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 msgr2=0x7f810806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.441+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 0x7f810806e2c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8120000f60 tx=0x7f8120007c00 comp rx=0 tx=0).stop 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.441+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 msgr2=0x7f8130158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.441+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f812802c4e0 tx=0x7f8128004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 shutdown_connections 2024-01-16T15:12:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f810806be00 0x7f810806e2c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81300adb60 0x7f8130152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f813015b350 0x7f8130158de0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 --2- 172.21.15.43:0/2833379348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f813009cb30 0x7f81301552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 >> 172.21.15.43:0/2833379348 conn(0x7f81300b0ba0 msgr2=0x7f813006da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 shutdown_connections 2024-01-16T15:12:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.442+0000 7f8134e00640 1 -- 172.21.15.43:0/2833379348 wait complete. 2024-01-16T15:12:12.438 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:12.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9614 2024-01-16T15:12:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.689+0000 7fd7b4a00640 1 Processor -- start 2024-01-16T15:12:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.690+0000 7fd7b4a00640 1 -- start start 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7b015d3b0 0x7fd7b00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7b00d0fa0 0x7fd7b00d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 0x7fd7b00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b00c6bf0 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b00d1ff0 con 0x7fd7b015d3b0 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7b4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b00d1430 con 0x7fd7b00d1bc0 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7ade00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7b00d0fa0 0x7fd7b00d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7ade00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7b00d0fa0 0x7fd7b00d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32850/0 (socket says 172.21.15.43:32850) 2024-01-16T15:12:12.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.691+0000 7fd7ade00640 1 -- 172.21.15.43:0/3402519505 learned_addr learned my addr 172.21.15.43:0/3402519505 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ae800640 1 --2- 172.21.15.43:0/3402519505 >> v2:172.21.15.69:3300/0 conn(0x7fd7b015d3b0 0x7fd7b00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7af200640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 0x7fd7b00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ade00640 1 -- 172.21.15.43:0/3402519505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 msgr2=0x7fd7b00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ade00640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 0x7fd7b00b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ade00640 1 -- 172.21.15.43:0/3402519505 >> v2:172.21.15.69:3300/0 conn(0x7fd7b015d3b0 msgr2=0x7fd7b00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ade00640 1 --2- 172.21.15.43:0/3402519505 >> v2:172.21.15.69:3300/0 conn(0x7fd7b015d3b0 0x7fd7b00ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7ade00640 1 -- 172.21.15.43:0/3402519505 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b00cedc0 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.692+0000 7fd7af200640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 0x7fd7b00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.693+0000 7fd7ade00640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b00d0fa0 0x7fd7b00d1680 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd7b008b2a0 tx=0x7fd79802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627dedcae917094e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.693+0000 7fd7ad400640 1 -- 172.21.15.43:0/3402519505 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79802c9f0 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7ad400640 1 -- 172.21.15.43:0/3402519505 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79802cb50 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7ad400640 1 -- 172.21.15.43:0/3402519505 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7980343f0 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 -- 172.21.15.43:0/3402519505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b00d0fa0 msgr2=0x7fd7b00d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b00d0fa0 0x7fd7b00d1680 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd7b008b2a0 tx=0x7fd79802be50 comp rx=0 tx=0).stop 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 -- 172.21.15.43:0/3402519505 shutdown_connections 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d1bc0 0x7fd7b00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/3402519505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b00d0fa0 0x7fd7b00d1680 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/3402519505 >> v2:172.21.15.69:3300/0 conn(0x7fd7b015d3b0 0x7fd7b00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 -- 172.21.15.43:0/3402519505 >> 172.21.15.43:0/3402519505 conn(0x7fd7b00d30a0 msgr2=0x7fd7b00d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 -- 172.21.15.43:0/3402519505 shutdown_connections 2024-01-16T15:12:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.694+0000 7fd7b4a00640 1 -- 172.21.15.43:0/3402519505 wait complete. 2024-01-16T15:12:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.696+0000 7fd7b4a00640 1 Processor -- start 2024-01-16T15:12:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.696+0000 7fd7b4a00640 1 -- start start 2024-01-16T15:12:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d0fa0 0x7fd7b00cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7b00d1bc0 0x7fd7b015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b00cb280 con 0x7fd7b015d3b0 2024-01-16T15:12:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b01f6440 con 0x7fd7b00d1bc0 2024-01-16T15:12:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7b4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b01f66e0 con 0x7fd7b00d0fa0 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.697+0000 7fd7af200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32866/0 (socket says 172.21.15.43:32866) 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 -- 172.21.15.43:0/680275659 learned_addr learned my addr 172.21.15.43:0/680275659 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7ade00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7b00d1bc0 0x7fd7b015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7ae800640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d0fa0 0x7fd7b00cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 -- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d0fa0 msgr2=0x7fd7b00cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d0fa0 0x7fd7b00cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 -- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7b00d1bc0 msgr2=0x7fd7b015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7b00d1bc0 0x7fd7b015e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 -- 172.21.15.43:0/680275659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798009b30 con 0x7fd7b015d3b0 2024-01-16T15:12:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.698+0000 7fd7af200640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800afd0 tx=0x7fd7a800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.699+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8004440 con 0x7fd7b015d3b0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.699+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a8003610 con 0x7fd7b015d3b0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.699+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a80039c0 con 0x7fd7b015d3b0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.699+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b01f69c0 con 0x7fd7b015d3b0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.699+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b01f6f80 con 0x7fd7b015d3b0 2024-01-16T15:12:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.700+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b01f71a0 con 0x7fd7b015d3b0 2024-01-16T15:12:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.701+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fd7a8003b20 con 0x7fd7b015d3b0 2024-01-16T15:12:12.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.702+0000 7fd7a6a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 0x7fd78806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:12.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.702+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd7a8088620 con 0x7fd7b015d3b0 2024-01-16T15:12:12.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.703+0000 7fd7ae800640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 0x7fd78806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:12.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.703+0000 7fd7ae800640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 0x7fd78806e2c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0006fd0 tx=0x7fd7a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:12.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.707+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7a8054f90 con 0x7fd7b015d3b0 2024-01-16T15:12:12.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.938+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7b00c7400 con 0x7fd78806be00 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.967+0000 7fd7a6a00640 1 -- 172.21.15.43:0/680275659 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd7a00076f0 con 0x7fd78806be00 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete" 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:12.944 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 msgr2=0x7fd78806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 0x7fd78806e2c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0006fd0 tx=0x7fd7a0008040 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 msgr2=0x7fd7b015ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800afd0 tx=0x7fd7a800ca20 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 shutdown_connections 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd78806be00 0x7fd78806e2c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7b00d1bc0 0x7fd7b015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7b015d3b0 0x7fd7b015ee10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 --2- 172.21.15.43:0/680275659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7b00d0fa0 0x7fd7b00cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 >> 172.21.15.43:0/680275659 conn(0x7fd7b00d30a0 msgr2=0x7fd7b00d3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:12.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.970+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 shutdown_connections 2024-01-16T15:12:12.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:12.971+0000 7fd7b4a00640 1 -- 172.21.15.43:0/680275659 wait complete. 2024-01-16T15:12:12.966 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:12.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9615 2024-01-16T15:12:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.220+0000 7f3e61a00640 1 Processor -- start 2024-01-16T15:12:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.220+0000 7f3e61a00640 1 -- start start 2024-01-16T15:12:13.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e61a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e5c144770 0x7f3e5c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e61a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3e5c0b35a0 0x7f3e5c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e61a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c15f930 con 0x7f3e5c06c9d0 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e61a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b43b0 con 0x7f3e5c144770 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e61a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b3b10 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e5be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e5c144770 0x7f3e5c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e5be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e5c144770 0x7f3e5c15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32906/0 (socket says 172.21.15.43:32906) 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.222+0000 7f3e5be00640 1 -- 172.21.15.43:0/1766966225 learned_addr learned my addr 172.21.15.43:0/1766966225 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.223+0000 7f3e5b400640 1 --2- 172.21.15.43:0/1766966225 >> v2:172.21.15.69:3300/0 conn(0x7f3e5c0b35a0 0x7f3e5c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.223+0000 7f3e5be00640 1 -- 172.21.15.43:0/1766966225 >> v2:172.21.15.69:3300/0 conn(0x7f3e5c0b35a0 msgr2=0x7f3e5c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.223+0000 7f3e5be00640 1 --2- 172.21.15.43:0/1766966225 >> v2:172.21.15.69:3300/0 conn(0x7f3e5c0b35a0 0x7f3e5c15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.223+0000 7f3e5be00640 1 -- 172.21.15.43:0/1766966225 >> v1:172.21.15.69:6789/0 conn(0x7f3e5c06c9d0 legacy=0x7f3e5c06ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:13.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.223+0000 7f3e5be00640 1 -- 172.21.15.43:0/1766966225 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0b3df0 con 0x7f3e5c144770 2024-01-16T15:12:13.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.224+0000 7f3e5be00640 1 --2- 172.21.15.43:0/1766966225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c144770 0x7f3e5c15eb60 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0ae900 tx=0x7f3e4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc1be6bf473b4f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e5aa00640 1 -- 172.21.15.43:0/1766966225 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c031870 con 0x7f3e5c144770 2024-01-16T15:12:13.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e5aa00640 1 -- 172.21.15.43:0/1766966225 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c0319d0 con 0x7f3e5c144770 2024-01-16T15:12:13.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e5aa00640 1 -- 172.21.15.43:0/1766966225 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c031d60 con 0x7f3e5c144770 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 -- 172.21.15.43:0/1766966225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c144770 msgr2=0x7f3e5c15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 --2- 172.21.15.43:0/1766966225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c144770 0x7f3e5c15eb60 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0ae900 tx=0x7f3e4c02b910 comp rx=0 tx=0).stop 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 -- 172.21.15.43:0/1766966225 shutdown_connections 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 --2- 172.21.15.43:0/1766966225 >> v2:172.21.15.69:3300/0 conn(0x7f3e5c0b35a0 0x7f3e5c15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 --2- 172.21.15.43:0/1766966225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c144770 0x7f3e5c15eb60 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.225+0000 7f3e61a00640 1 -- 172.21.15.43:0/1766966225 >> 172.21.15.43:0/1766966225 conn(0x7f3e5c0cec10 msgr2=0x7f3e5c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.226+0000 7f3e61a00640 1 -- 172.21.15.43:0/1766966225 shutdown_connections 2024-01-16T15:12:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.226+0000 7f3e61a00640 1 -- 172.21.15.43:0/1766966225 wait complete. 2024-01-16T15:12:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.227+0000 7f3e61a00640 1 Processor -- start 2024-01-16T15:12:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.227+0000 7f3e61a00640 1 -- start start 2024-01-16T15:12:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.228+0000 7f3e61a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e5c06c9d0 0x7f3e5c15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.228+0000 7f3e61a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 0x7f3e5c13a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c7340 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c66c0 con 0x7f3e5c144770 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c69a0 con 0x7f3e5c06c9d0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32920/0 (socket says 172.21.15.43:32920) 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 -- 172.21.15.43:0/510362235 learned_addr learned my addr 172.21.15.43:0/510362235 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5be00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e5c06c9d0 0x7f3e5c15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 -- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e5c06c9d0 msgr2=0x7f3e5c15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e5c06c9d0 0x7f3e5c15ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61000640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 0x7f3e5c13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 -- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 msgr2=0x7f3e5c13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 0x7f3e5c13a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e5b400640 1 -- 172.21.15.43:0/510362235 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c0094d0 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.229+0000 7f3e61000640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 0x7f3e5c13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e5b400640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d800 tx=0x7f3e4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400b6c0 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4400b820 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400bbb0 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c0c6c80 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.230+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c15d360 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.231+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c15d4f0 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.232+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f3e4400bd10 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.233+0000 7f3e53e00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 0x7f3e3406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.233+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3e44088bd0 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.234+0000 7f3e5be00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 0x7f3e3406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.234+0000 7f3e5be00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 0x7f3e3406e2c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c036000 tx=0x7f3e4c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.238+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e44055390 con 0x7f3e5c0b35a0 2024-01-16T15:12:13.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.475+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9615", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e5c0b1de0 con 0x7f3e3406be00 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.501+0000 7f3e53e00640 1 -- 172.21.15.43:0/510362235 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3e5c0b1de0 con 0x7f3e3406be00 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:13.479 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:13.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:13.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:13.479 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.504+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 msgr2=0x7f3e3406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.504+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 0x7f3e3406e2c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c036000 tx=0x7f3e4c03c040 comp rx=0 tx=0).stop 2024-01-16T15:12:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.504+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 msgr2=0x7f3e5c13a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.504+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d800 tx=0x7f3e4400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 shutdown_connections 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e3406be00 0x7f3e3406e2c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e5c144770 0x7f3e5c13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e5c0b35a0 0x7f3e5c13a310 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 --2- 172.21.15.43:0/510362235 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e5c06c9d0 0x7f3e5c15ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 >> 172.21.15.43:0/510362235 conn(0x7f3e5c0cec10 msgr2=0x7f3e5c070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 shutdown_connections 2024-01-16T15:12:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.505+0000 7f3e61a00640 1 -- 172.21.15.43:0/510362235 wait complete. 2024-01-16T15:12:13.502 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:13.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9616 2024-01-16T15:12:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.752+0000 7f0536400640 1 Processor -- start 2024-01-16T15:12:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.753+0000 7f0536400640 1 -- start start 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.754+0000 7f0536400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f053015f270 0x7f05300cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0536400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053015f930 con 0x7f0530144770 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0536400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300b3b10 con 0x7f053006d9e0 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0536400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300b43b0 con 0x7f053015f270 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0535000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0530144770 0x7f053015eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51258/0 (socket says 172.21.15.43:51258) 2024-01-16T15:12:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0535000640 1 -- 172.21.15.43:0/4274965916 learned_addr learned my addr 172.21.15.43:0/4274965916 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0527e00640 1 --2- 172.21.15.43:0/4274965916 >> v2:172.21.15.43:3300/0 conn(0x7f053015f270 0x7f05300cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f052fe00640 1 -- 172.21.15.43:0/4274965916 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923267266 0 0) 0x7f053015f930 con 0x7f0530144770 2024-01-16T15:12:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f052fe00640 1 -- 172.21.15.43:0/4274965916 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510003520 con 0x7f0530144770 2024-01-16T15:12:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.755+0000 7f0527e00640 1 -- 172.21.15.43:0/4274965916 >> v1:172.21.15.69:6789/0 conn(0x7f053006d9e0 legacy=0x7f05300cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.756+0000 7f0527e00640 1 -- 172.21.15.43:0/4274965916 >> v1:172.21.15.43:6789/0 conn(0x7f0530144770 legacy=0x7f053015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.756+0000 7f0527e00640 1 -- 172.21.15.43:0/4274965916 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300b3df0 con 0x7f053015f270 2024-01-16T15:12:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f0527e00640 1 --2- 172.21.15.43:0/4274965916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f053015f270 0x7f05300cd4a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f05180097a0 tx=0x7f051802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981cb8c22adb574c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f052fe00640 1 -- 172.21.15.43:0/4274965916 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518031830 con 0x7f053015f270 2024-01-16T15:12:13.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f052fe00640 1 -- 172.21.15.43:0/4274965916 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0518031990 con 0x7f053015f270 2024-01-16T15:12:13.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f052fe00640 1 -- 172.21.15.43:0/4274965916 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518031d20 con 0x7f053015f270 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f0536400640 1 -- 172.21.15.43:0/4274965916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f053015f270 msgr2=0x7f05300cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.757+0000 7f0536400640 1 --2- 172.21.15.43:0/4274965916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f053015f270 0x7f05300cd4a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f05180097a0 tx=0x7f051802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.758+0000 7f0536400640 1 -- 172.21.15.43:0/4274965916 shutdown_connections 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.758+0000 7f0536400640 1 --2- 172.21.15.43:0/4274965916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f053015f270 0x7f05300cd4a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.758+0000 7f0536400640 1 -- 172.21.15.43:0/4274965916 >> 172.21.15.43:0/4274965916 conn(0x7f05300cec10 msgr2=0x7f05300cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.758+0000 7f0536400640 1 -- 172.21.15.43:0/4274965916 shutdown_connections 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.758+0000 7f0536400640 1 -- 172.21.15.43:0/4274965916 wait complete. 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.759+0000 7f0536400640 1 Processor -- start 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.759+0000 7f0536400640 1 -- start start 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f053006d9e0 0x7f053013a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f053015f270 0x7f053013d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053013ac50 con 0x7f0530144770 2024-01-16T15:12:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053015c030 con 0x7f053006d9e0 2024-01-16T15:12:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.760+0000 7f0536400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053015b700 con 0x7f053015f270 2024-01-16T15:12:13.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.761+0000 7f0527e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.761+0000 7f0527e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32944/0 (socket says 172.21.15.43:32944) 2024-01-16T15:12:13.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.761+0000 7f0527e00640 1 -- 172.21.15.43:0/91757900 learned_addr learned my addr 172.21.15.43:0/91757900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:13.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.761+0000 7f0535000640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f053006d9e0 0x7f053013a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.761+0000 7f0535a00640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f053015f270 0x7f053013d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 -- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f053015f270 msgr2=0x7f053013d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f053015f270 0x7f053013d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 -- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f053006d9e0 msgr2=0x7f053013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f053006d9e0 0x7f053013a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 -- 172.21.15.43:0/91757900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518009400 con 0x7f0530144770 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0527e00640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0518000c00 tx=0x7f05180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051803a8f0 con 0x7f0530144770 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051803aa50 con 0x7f0530144770 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051803ae00 con 0x7f0530144770 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053015b9e0 con 0x7f0530144770 2024-01-16T15:12:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.762+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05301e5cf0 con 0x7f0530144770 2024-01-16T15:12:13.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.764+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053004e840 con 0x7f0530144770 2024-01-16T15:12:13.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.765+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 27) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f051804d450 con 0x7f0530144770 2024-01-16T15:12:13.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.765+0000 7f052d600640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 0x7f050806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.766+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05180aa530 con 0x7f0530144770 2024-01-16T15:12:13.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.766+0000 7f0535000640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 0x7f050806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.766+0000 7f0535000640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 0x7f050806e2c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f05200011e0 tx=0x7f0520007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:13.770+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0518076df0 con 0x7f0530144770 2024-01-16T15:12:13.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.003+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f0530144b50 con 0x7f050806be00 2024-01-16T15:12:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.023+0000 7f052d600640 1 -- 172.21.15.43:0/91757900 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f0520006b20 con 0x7f050806be00 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:14.000 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:14.001 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:14.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:14.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:14.001 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 msgr2=0x7f050806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 0x7f050806e2c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f05200011e0 tx=0x7f0520007d30 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 msgr2=0x7f05301e57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0518000c00 tx=0x7f05180046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 shutdown_connections 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f050806be00 0x7f050806e2c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f053015f270 0x7f053013d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0530144770 0x7f05301e57b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 --2- 172.21.15.43:0/91757900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f053006d9e0 0x7f053013a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.026+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 >> 172.21.15.43:0/91757900 conn(0x7f05300cec10 msgr2=0x7f05300cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.027+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 shutdown_connections 2024-01-16T15:12:14.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.027+0000 7f0536400640 1 -- 172.21.15.43:0/91757900 wait complete. 2024-01-16T15:12:14.025 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:14.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9617 2024-01-16T15:12:14.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.279+0000 7f42d1c00640 1 Processor -- start 2024-01-16T15:12:14.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.279+0000 7f42d1c00640 1 -- start start 2024-01-16T15:12:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.281+0000 7f42d1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.281+0000 7f42d1c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f42cc0b35a0 0x7f42cc15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.281+0000 7f42d1c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc15f930 con 0x7f42cc06c9d0 2024-01-16T15:12:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.281+0000 7f42d1c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc0b43b0 con 0x7f42cc0b35a0 2024-01-16T15:12:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.281+0000 7f42d1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0b3b10 con 0x7f42cc144770 2024-01-16T15:12:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.282+0000 7f42cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.282+0000 7f42cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:57812/0 (socket says 172.21.15.43:57812) 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.282+0000 7f42cbe00640 1 -- 172.21.15.43:0/2891881545 learned_addr learned my addr 172.21.15.43:0/2891881545 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.282+0000 7f42cb400640 1 --2- 172.21.15.43:0/2891881545 >> v2:172.21.15.69:3300/0 conn(0x7f42cc0b35a0 0x7f42cc15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.283+0000 7f42cbe00640 1 -- 172.21.15.43:0/2891881545 >> v2:172.21.15.69:3300/0 conn(0x7f42cc0b35a0 msgr2=0x7f42cc15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.283+0000 7f42cbe00640 1 --2- 172.21.15.43:0/2891881545 >> v2:172.21.15.69:3300/0 conn(0x7f42cc0b35a0 0x7f42cc15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.283+0000 7f42cbe00640 1 -- 172.21.15.43:0/2891881545 >> v1:172.21.15.69:6789/0 conn(0x7f42cc06c9d0 legacy=0x7f42cc06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.283+0000 7f42cbe00640 1 -- 172.21.15.43:0/2891881545 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0b3df0 con 0x7f42cc144770 2024-01-16T15:12:14.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.284+0000 7f42cbe00640 1 --2- 172.21.15.43:0/2891881545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098a0 tx=0x7f42bc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5ab9c69d4c0158 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.284+0000 7f42caa00640 1 -- 172.21.15.43:0/2891881545 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc030840 con 0x7f42cc144770 2024-01-16T15:12:14.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42caa00640 1 -- 172.21.15.43:0/2891881545 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0309a0 con 0x7f42cc144770 2024-01-16T15:12:14.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42caa00640 1 -- 172.21.15.43:0/2891881545 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc030d30 con 0x7f42cc144770 2024-01-16T15:12:14.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 -- 172.21.15.43:0/2891881545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 msgr2=0x7f42cc15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2891881545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098a0 tx=0x7f42bc02b980 comp rx=0 tx=0).stop 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 -- 172.21.15.43:0/2891881545 shutdown_connections 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2891881545 >> v2:172.21.15.69:3300/0 conn(0x7f42cc0b35a0 0x7f42cc15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2891881545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc15eb60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.285+0000 7f42d1c00640 1 -- 172.21.15.43:0/2891881545 >> 172.21.15.43:0/2891881545 conn(0x7f42cc0cec10 msgr2=0x7f42cc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.286+0000 7f42d1c00640 1 -- 172.21.15.43:0/2891881545 shutdown_connections 2024-01-16T15:12:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.286+0000 7f42d1c00640 1 -- 172.21.15.43:0/2891881545 wait complete. 2024-01-16T15:12:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.287+0000 7f42d1c00640 1 Processor -- start 2024-01-16T15:12:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.287+0000 7f42d1c00640 1 -- start start 2024-01-16T15:12:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc0b35a0 0x7f42cc1f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1f07a0 con 0x7f42cc06c9d0 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc143070 con 0x7f42cc0b35a0 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.288+0000 7f42d1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc140aa0 con 0x7f42cc144770 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32950/0 (socket says 172.21.15.43:32950) 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 -- 172.21.15.43:0/2728073517 learned_addr learned my addr 172.21.15.43:0/2728073517 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42d1200640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 -- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 msgr2=0x7f42cc142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 -- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc0b35a0 msgr2=0x7f42cc1f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cb400640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc0b35a0 0x7f42cc1f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc0b35a0 0x7f42cc1f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.289+0000 7f42cbe00640 1 -- 172.21.15.43:0/2728073517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0094d0 con 0x7f42cc06c9d0 2024-01-16T15:12:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42cbe00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0090b0 tx=0x7f42bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc03e070 con 0x7f42cc06c9d0 2024-01-16T15:12:14.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc039a10 con 0x7f42cc06c9d0 2024-01-16T15:12:14.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc140d80 con 0x7f42cc06c9d0 2024-01-16T15:12:14.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc039de0 con 0x7f42cc06c9d0 2024-01-16T15:12:14.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.290+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc141340 con 0x7f42cc06c9d0 2024-01-16T15:12:14.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.291+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc04e840 con 0x7f42cc06c9d0 2024-01-16T15:12:14.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.292+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f42bc04b400 con 0x7f42cc06c9d0 2024-01-16T15:12:14.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.293+0000 7f42c3e00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 0x7f42a406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.293+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f42bc0aad40 con 0x7f42cc06c9d0 2024-01-16T15:12:14.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.294+0000 7f42cb400640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 0x7f42a406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.294+0000 7f42cb400640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 0x7f42a406e350 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f42cc1f6210 tx=0x7f42b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.298+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42bc0773a0 con 0x7f42cc06c9d0 2024-01-16T15:12:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.530+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9617", "target": ["mon-mgr", ""]}) v1 -- 0x7f42cc144b50 con 0x7f42a406be90 2024-01-16T15:12:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.555+0000 7f42c3e00640 1 -- 172.21.15.43:0/2728073517 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f42b4010840 con 0x7f42a406be90 2024-01-16T15:12:14.532 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:14.532 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:14.532 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:14.532 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:14.533 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.558+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 msgr2=0x7f42a406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.558+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 0x7f42a406e350 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f42cc1f6210 tx=0x7f42b4008040 comp rx=0 tx=0).stop 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.558+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 msgr2=0x7f42cc1f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.558+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0090b0 tx=0x7f42bc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 shutdown_connections 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f42a406be90 0x7f42a406e350 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42cc0b35a0 0x7f42cc1f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42cc144770 0x7f42cc142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 --2- 172.21.15.43:0/2728073517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42cc06c9d0 0x7f42cc1f3860 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 >> 172.21.15.43:0/2728073517 conn(0x7f42cc0cec10 msgr2=0x7f42cc13f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 shutdown_connections 2024-01-16T15:12:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.559+0000 7f42d1c00640 1 -- 172.21.15.43:0/2728073517 wait complete. 2024-01-16T15:12:14.555 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:14.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9618 2024-01-16T15:12:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.806+0000 7f684d200640 1 Processor -- start 2024-01-16T15:12:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.806+0000 7f684d200640 1 -- start start 2024-01-16T15:12:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f684d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f68480acd70 0x7f684806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f684d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 0x7f6848070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f684d200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684806d7a0 con 0x7f68480ae490 2024-01-16T15:12:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f684d200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684815a5b0 con 0x7f68480acd70 2024-01-16T15:12:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f684d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684806ce70 con 0x7f6848070490 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f6845e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f68480acd70 0x7f684806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f683d200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f68480ae490 0x7f68480ad150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51272/0 (socket says 172.21.15.43:51272) 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f683d200640 1 -- 172.21.15.43:0/830964378 learned_addr learned my addr 172.21.15.43:0/830964378 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.808+0000 7f6846800640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 0x7f6848070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.809+0000 7f6845e00640 1 -- 172.21.15.43:0/830964378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 msgr2=0x7f6848070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.809+0000 7f6845e00640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 0x7f6848070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.809+0000 7f6845e00640 1 -- 172.21.15.43:0/830964378 >> v1:172.21.15.43:6789/0 conn(0x7f68480ae490 legacy=0x7f68480ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.809+0000 7f6845e00640 1 -- 172.21.15.43:0/830964378 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684806d150 con 0x7f68480acd70 2024-01-16T15:12:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.809+0000 7f6846800640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 0x7f6848070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.810+0000 7f6845e00640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480acd70 0x7f684806dc30 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f68380097a0 tx=0x7f683802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3512fa120da90c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.810+0000 7f6845400640 1 -- 172.21.15.43:0/830964378 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683802d040 con 0x7f68480acd70 2024-01-16T15:12:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.810+0000 7f6845400640 1 -- 172.21.15.43:0/830964378 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838031810 con 0x7f68480acd70 2024-01-16T15:12:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.810+0000 7f6845400640 1 -- 172.21.15.43:0/830964378 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838031ba0 con 0x7f68480acd70 2024-01-16T15:12:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 -- 172.21.15.43:0/830964378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480acd70 msgr2=0x7f684806dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480acd70 0x7f684806dc30 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f68380097a0 tx=0x7f683802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 -- 172.21.15.43:0/830964378 shutdown_connections 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6848070490 0x7f6848070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 --2- 172.21.15.43:0/830964378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480acd70 0x7f684806dc30 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 -- 172.21.15.43:0/830964378 >> 172.21.15.43:0/830964378 conn(0x7f68480b0ba0 msgr2=0x7f68480b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 -- 172.21.15.43:0/830964378 shutdown_connections 2024-01-16T15:12:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.811+0000 7f684d200640 1 -- 172.21.15.43:0/830964378 wait complete. 2024-01-16T15:12:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.813+0000 7f684d200640 1 Processor -- start 2024-01-16T15:12:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.813+0000 7f684d200640 1 -- start start 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6848070490 0x7f68481f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68480acd70 0x7f6848159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6848159300 con 0x7f68480ae490 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6848157410 con 0x7f6848070490 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f684d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6848153070 con 0x7f68480acd70 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f6846800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f6846800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32966/0 (socket says 172.21.15.43:32966) 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.814+0000 7f6846800640 1 -- 172.21.15.43:0/2283790815 learned_addr learned my addr 172.21.15.43:0/2283790815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f683d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68480acd70 0x7f6848159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6845e00640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6848070490 0x7f68481f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 -- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68480acd70 msgr2=0x7f6848159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68480acd70 0x7f6848159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 -- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6848070490 msgr2=0x7f68481f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6848070490 0x7f68481f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 -- 172.21.15.43:0/2283790815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6838009400 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f6846800640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f684000cef0 tx=0x7f684000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684000d840 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684000d9c0 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684000dd70 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.815+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684809a980 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.816+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6848156dc0 con 0x7f68480ae490 2024-01-16T15:12:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.817+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6848157580 con 0x7f68480ae490 2024-01-16T15:12:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.818+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f684001e3b0 con 0x7f68480ae490 2024-01-16T15:12:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.820+0000 7f683ea00640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 0x7f682006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.820+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6840089090 con 0x7f68480ae490 2024-01-16T15:12:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.820+0000 7f6845e00640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 0x7f682006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.821+0000 7f6845e00640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 0x7f682006e350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6838036000 tx=0x7f683803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:14.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:14.824+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6840055a10 con 0x7f68480ae490 2024-01-16T15:12:15.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.056+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9618", "target": ["mon-mgr", ""]}) v1 -- 0x7f68480b3d90 con 0x7f682006be90 2024-01-16T15:12:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.073+0000 7f683ea00640 1 -- 172.21.15.43:0/2283790815 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6838031610 con 0x7f682006be90 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:15.050 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 msgr2=0x7f682006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 0x7f682006e350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6838036000 tx=0x7f683803c040 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 msgr2=0x7f6848145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f684000cef0 tx=0x7f684000a710 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 shutdown_connections 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f682006be90 0x7f682006e350 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68480ae490 0x7f6848145a40 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68480acd70 0x7f6848159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 --2- 172.21.15.43:0/2283790815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6848070490 0x7f68481f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 >> 172.21.15.43:0/2283790815 conn(0x7f68480b0ba0 msgr2=0x7f68480b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 shutdown_connections 2024-01-16T15:12:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.076+0000 7f684d200640 1 -- 172.21.15.43:0/2283790815 wait complete. 2024-01-16T15:12:15.073 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:15.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone9619 2024-01-16T15:12:15.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.328+0000 7fc23f000640 1 Processor -- start 2024-01-16T15:12:15.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.328+0000 7fc23f000640 1 -- start start 2024-01-16T15:12:15.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc238144770 0x7fc23815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2380b35a0 0x7fc23815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2380b43b0 con 0x7fc2380b35a0 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23815f930 con 0x7fc238144770 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.329+0000 7fc23f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2380b3b10 con 0x7fc23806c9d0 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2380b35a0 0x7fc23815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2380b35a0 0x7fc23815f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32970/0 (socket says 172.21.15.43:32970) 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 -- 172.21.15.43:0/300076776 learned_addr learned my addr 172.21.15.43:0/300076776 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23dc00640 1 --2- 172.21.15.43:0/300076776 >> v2:172.21.15.69:3300/0 conn(0x7fc238144770 0x7fc23815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23e600640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 -- 172.21.15.43:0/300076776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 msgr2=0x7fc23806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23806ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 -- 172.21.15.43:0/300076776 >> v2:172.21.15.69:3300/0 conn(0x7fc238144770 msgr2=0x7fc23815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 --2- 172.21.15.43:0/300076776 >> v2:172.21.15.69:3300/0 conn(0x7fc238144770 0x7fc23815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23d200640 1 -- 172.21.15.43:0/300076776 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2380b3df0 con 0x7fc2380b35a0 2024-01-16T15:12:15.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.330+0000 7fc23e600640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.331+0000 7fc23d200640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815f0a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc23806df90 tx=0x7fc22002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7102db4d23f46f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc237e00640 1 -- 172.21.15.43:0/300076776 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220031860 con 0x7fc2380b35a0 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc237e00640 1 -- 172.21.15.43:0/300076776 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2200319c0 con 0x7fc2380b35a0 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc237e00640 1 -- 172.21.15.43:0/300076776 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220031d50 con 0x7fc2380b35a0 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 -- 172.21.15.43:0/300076776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 msgr2=0x7fc23815f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815f0a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc23806df90 tx=0x7fc22002b900 comp rx=0 tx=0).stop 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 -- 172.21.15.43:0/300076776 shutdown_connections 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 --2- 172.21.15.43:0/300076776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815f0a0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 --2- 172.21.15.43:0/300076776 >> v2:172.21.15.69:3300/0 conn(0x7fc238144770 0x7fc23815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 -- 172.21.15.43:0/300076776 >> 172.21.15.43:0/300076776 conn(0x7fc2380cec10 msgr2=0x7fc2380cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:15.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.332+0000 7fc23f000640 1 -- 172.21.15.43:0/300076776 shutdown_connections 2024-01-16T15:12:15.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.333+0000 7fc23f000640 1 -- 172.21.15.43:0/300076776 wait complete. 2024-01-16T15:12:15.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.334+0000 7fc23f000640 1 Processor -- start 2024-01-16T15:12:15.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.334+0000 7fc23f000640 1 -- start start 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23813ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc238144770 0x7fc238143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238143150 con 0x7fc2380b35a0 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23815e420 con 0x7fc238144770 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.335+0000 7fc23f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23813f260 con 0x7fc23806c9d0 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32980/0 (socket says 172.21.15.43:32980) 2024-01-16T15:12:15.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 -- 172.21.15.43:0/3176332073 learned_addr learned my addr 172.21.15.43:0/3176332073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:15.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 -- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 msgr2=0x7fc23813ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23813ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 -- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc238144770 msgr2=0x7fc238143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc238144770 0x7fc238143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 -- 172.21.15.43:0/3176332073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2200094d0 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc23d200640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc22002b940 tx=0x7fc2200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.336+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22003ad30 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.337+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22003ae90 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.337+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220044630 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.337+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23814f5d0 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.337+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381427b0 con 0x7fc2380b35a0 2024-01-16T15:12:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.338+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23804e840 con 0x7fc2380b35a0 2024-01-16T15:12:15.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.339+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc220044790 con 0x7fc2380b35a0 2024-01-16T15:12:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.340+0000 7fc235600640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 0x7fc21006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.340+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc2200aafe0 con 0x7fc2380b35a0 2024-01-16T15:12:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.340+0000 7fc23dc00640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 0x7fc21006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.341+0000 7fc23dc00640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 0x7fc21006e350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc228009dd0 tx=0x7fc228009450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.345+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2200776d0 con 0x7fc2380b35a0 2024-01-16T15:12:15.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.581+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone9619", "target": ["mon-mgr", ""]}) v1 -- 0x7fc238144b50 con 0x7fc21006be90 2024-01-16T15:12:15.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.599+0000 7fc235600640 1 -- 172.21.15.43:0/3176332073 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fc2280095c0 con 0x7fc21006be90 2024-01-16T15:12:15.575 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:15.576 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:15.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 msgr2=0x7fc21006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 0x7fc21006e350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc228009dd0 tx=0x7fc228009450 comp rx=0 tx=0).stop 2024-01-16T15:12:15.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 msgr2=0x7fc23815a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc22002b940 tx=0x7fc2200046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 shutdown_connections 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc21006be90 0x7fc21006e350 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc238144770 0x7fc238143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2380b35a0 0x7fc23815a5c0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 --2- 172.21.15.43:0/3176332073 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc23806c9d0 0x7fc23813ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 >> 172.21.15.43:0/3176332073 conn(0x7fc2380cec10 msgr2=0x7fc2380cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 shutdown_connections 2024-01-16T15:12:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.602+0000 7fc23f000640 1 -- 172.21.15.43:0/3176332073 wait complete. 2024-01-16T15:12:15.599 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:15.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96110 2024-01-16T15:12:15.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.837+0000 7f61aa400640 1 Processor -- start 2024-01-16T15:12:15.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.838+0000 7f61aa400640 1 -- start start 2024-01-16T15:12:15.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61aa400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f61a415d3b0 0x7f61a40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61aa400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f61a40d0fa0 0x7f61a40d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61aa400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a40c6bf0 con 0x7f61a40d1bc0 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61aa400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40d1430 con 0x7f61a415d3b0 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61aa400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40d1fa0 con 0x7f61a40d0fa0 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61a9000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f61a415d3b0 0x7f61a40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61a9000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f61a415d3b0 0x7f61a40ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32988/0 (socket says 172.21.15.43:32988) 2024-01-16T15:12:15.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.839+0000 7f61a9000640 1 -- 172.21.15.43:0/2344679010 learned_addr learned my addr 172.21.15.43:0/2344679010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.840+0000 7f61a9000640 1 -- 172.21.15.43:0/2344679010 >> v2:172.21.15.69:3300/0 conn(0x7f61a40d0fa0 msgr2=0x7f61a40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.840+0000 7f61a9000640 1 --2- 172.21.15.43:0/2344679010 >> v2:172.21.15.69:3300/0 conn(0x7f61a40d0fa0 0x7f61a40d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.840+0000 7f61a9000640 1 -- 172.21.15.43:0/2344679010 >> v1:172.21.15.69:6789/0 conn(0x7f61a40d1bc0 legacy=0x7f61a40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.840+0000 7f61a9000640 1 -- 172.21.15.43:0/2344679010 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40cedc0 con 0x7f61a415d3b0 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.841+0000 7f61a9000640 1 --2- 172.21.15.43:0/2344679010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a415d3b0 0x7f61a40ce6c0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f61a408c4a0 tx=0x7f619402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57b6e78bf38f363 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.841+0000 7f61a3400640 1 -- 172.21.15.43:0/2344679010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619402c9f0 con 0x7f61a415d3b0 2024-01-16T15:12:15.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.841+0000 7f61a3400640 1 -- 172.21.15.43:0/2344679010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619402cb50 con 0x7f61a415d3b0 2024-01-16T15:12:15.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.841+0000 7f61a3400640 1 -- 172.21.15.43:0/2344679010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61940343f0 con 0x7f61a415d3b0 2024-01-16T15:12:15.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 -- 172.21.15.43:0/2344679010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a415d3b0 msgr2=0x7f61a40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 --2- 172.21.15.43:0/2344679010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a415d3b0 0x7f61a40ce6c0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f61a408c4a0 tx=0x7f619402be50 comp rx=0 tx=0).stop 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 -- 172.21.15.43:0/2344679010 shutdown_connections 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 --2- 172.21.15.43:0/2344679010 >> v2:172.21.15.69:3300/0 conn(0x7f61a40d0fa0 0x7f61a40d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 --2- 172.21.15.43:0/2344679010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a415d3b0 0x7f61a40ce6c0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 -- 172.21.15.43:0/2344679010 >> 172.21.15.43:0/2344679010 conn(0x7f61a40d30a0 msgr2=0x7f61a40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 -- 172.21.15.43:0/2344679010 shutdown_connections 2024-01-16T15:12:15.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.842+0000 7f61aa400640 1 -- 172.21.15.43:0/2344679010 wait complete. 2024-01-16T15:12:15.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.844+0000 7f61aa400640 1 Processor -- start 2024-01-16T15:12:15.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.844+0000 7f61aa400640 1 -- start start 2024-01-16T15:12:15.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a40d0fa0 0x7f61a41e6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a415d3b0 0x7f61a41e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40b61a0 con 0x7f61a40d1bc0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40b6470 con 0x7f61a415d3b0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.845+0000 7f61aa400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40b6750 con 0x7f61a40d0fa0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32996/0 (socket says 172.21.15.43:32996) 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 -- 172.21.15.43:0/2756877003 learned_addr learned my addr 172.21.15.43:0/2756877003 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a9000640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a40d0fa0 0x7f61a41e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a9a00640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a415d3b0 0x7f61a41e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 -- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a40d0fa0 msgr2=0x7f61a41e6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a40d0fa0 0x7f61a41e6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 -- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a415d3b0 msgr2=0x7f61a41e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a415d3b0 0x7f61a41e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 -- 172.21.15.43:0/2756877003 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194009b30 con 0x7f61a40d1bc0 2024-01-16T15:12:15.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.846+0000 7f61a3e00640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f618c00b410 tx=0x7f618c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.847+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c011830 con 0x7f61a40d1bc0 2024-01-16T15:12:15.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.847+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c0119b0 con 0x7f61a40d1bc0 2024-01-16T15:12:15.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.847+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c011d60 con 0x7f61a40d1bc0 2024-01-16T15:12:15.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.847+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a40b6a30 con 0x7f61a40d1bc0 2024-01-16T15:12:15.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.847+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a40b6ff0 con 0x7f61a40d1bc0 2024-01-16T15:12:15.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.848+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a404e840 con 0x7f61a40d1bc0 2024-01-16T15:12:15.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.849+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f618c01a400 con 0x7f61a40d1bc0 2024-01-16T15:12:15.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.850+0000 7f61a0c00640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 0x7f617c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:15.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.850+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f618c088a50 con 0x7f61a40d1bc0 2024-01-16T15:12:15.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.851+0000 7f61a9000640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 0x7f617c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:15.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.852+0000 7f61a9000640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 0x7f617c06e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6194035000 tx=0x7f6194030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:15.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:15.855+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f618c054ff0 con 0x7f61a40d1bc0 2024-01-16T15:12:16.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.086+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96110", "target": ["mon-mgr", ""]}) v1 -- 0x7f61a40c7400 con 0x7f617c06be90 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.105+0000 7f61a0c00640 1 -- 172.21.15.43:0/2756877003 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6194034470 con 0x7f617c06be90 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:16.082 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:16.083 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.108+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 msgr2=0x7f617c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.108+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 0x7f617c06e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6194035000 tx=0x7f6194030000 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.108+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 msgr2=0x7f61a40b5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.108+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f618c00b410 tx=0x7f618c00b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 shutdown_connections 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f617c06be90 0x7f617c06e350 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61a415d3b0 0x7f61a41e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61a40d1bc0 0x7f61a40b5c60 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 --2- 172.21.15.43:0/2756877003 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61a40d0fa0 0x7f61a41e6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 >> 172.21.15.43:0/2756877003 conn(0x7f61a40d30a0 msgr2=0x7f61a40c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 shutdown_connections 2024-01-16T15:12:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.109+0000 7f61aa400640 1 -- 172.21.15.43:0/2756877003 wait complete. 2024-01-16T15:12:16.106 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:16.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96111 2024-01-16T15:12:16.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.352+0000 7ff30e400640 1 Processor -- start 2024-01-16T15:12:16.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.352+0000 7ff30e400640 1 -- start start 2024-01-16T15:12:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.354+0000 7ff30e400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff30815b350 0x7ff3080abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.354+0000 7ff30e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 0x7ff3080b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.354+0000 7ff30e400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30815b730 con 0x7ff3080adb60 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.354+0000 7ff30e400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3080ac740 con 0x7ff30815b350 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.354+0000 7ff30e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3080ac460 con 0x7ff30809cb30 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.355+0000 7ff2fe800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 0x7ff3080b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.355+0000 7ff307e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff3080adb60 0x7ff3080adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:49018/0 (socket says 172.21.15.43:49018) 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.355+0000 7ff307e00640 1 -- 172.21.15.43:0/1638897697 learned_addr learned my addr 172.21.15.43:0/1638897697 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.355+0000 7ff307400640 1 --2- 172.21.15.43:0/1638897697 >> v2:172.21.15.69:3300/0 conn(0x7ff30815b350 0x7ff3080abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff306a00640 1 -- 172.21.15.43:0/1638897697 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877571108 0 0) 0x7ff300001110 con 0x7ff3080adb60 2024-01-16T15:12:16.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff306a00640 1 -- 172.21.15.43:0/1638897697 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e8003520 con 0x7ff3080adb60 2024-01-16T15:12:16.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff307400640 1 -- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 msgr2=0x7ff3080b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff307400640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 0x7ff3080b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff307400640 1 -- 172.21.15.43:0/1638897697 >> v1:172.21.15.69:6789/0 conn(0x7ff3080adb60 legacy=0x7ff3080adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff307400640 1 -- 172.21.15.43:0/1638897697 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3080b2a50 con 0x7ff30815b350 2024-01-16T15:12:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.356+0000 7ff2fe800640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 0x7ff3080b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.357+0000 7ff307400640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff3080abe80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b28b9ce88da9190a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.358+0000 7ff306a00640 1 -- 172.21.15.43:0/1638897697 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f802cc60 con 0x7ff30815b350 2024-01-16T15:12:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.358+0000 7ff306a00640 1 -- 172.21.15.43:0/1638897697 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f802cdc0 con 0x7ff30815b350 2024-01-16T15:12:16.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.358+0000 7ff306a00640 1 -- 172.21.15.43:0/1638897697 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8034660 con 0x7ff30815b350 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 -- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 msgr2=0x7ff3080abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff3080abe80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f802beb0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 -- 172.21.15.43:0/1638897697 shutdown_connections 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff30809cb30 0x7ff3080b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 --2- 172.21.15.43:0/1638897697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff3080abe80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 -- 172.21.15.43:0/1638897697 >> 172.21.15.43:0/1638897697 conn(0x7ff3080b0ba0 msgr2=0x7ff3080b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:16.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 -- 172.21.15.43:0/1638897697 shutdown_connections 2024-01-16T15:12:16.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.359+0000 7ff30e400640 1 -- 172.21.15.43:0/1638897697 wait complete. 2024-01-16T15:12:16.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.361+0000 7ff30e400640 1 Processor -- start 2024-01-16T15:12:16.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.361+0000 7ff30e400640 1 -- start start 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3080adb60 0x7ff30813e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff30809b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30814c710 con 0x7ff30809cb30 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30814cf70 con 0x7ff30815b350 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff30e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30809bd60 con 0x7ff3080adb60 2024-01-16T15:12:16.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33002/0 (socket says 172.21.15.43:33002) 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 -- 172.21.15.43:0/3930603155 learned_addr learned my addr 172.21.15.43:0/3930603155 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff2fe800640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3080adb60 0x7ff30813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307e00640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff30809b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 -- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3080adb60 msgr2=0x7ff30813e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3080adb60 0x7ff30813e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.362+0000 7ff307400640 1 -- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 msgr2=0x7ff30809b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff307400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff30809b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff307400640 1 -- 172.21.15.43:0/3930603155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0009400 con 0x7ff30809cb30 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff307e00640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff30809b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:16.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff307400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f803e610 con 0x7ff30809cb30 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f803e790 con 0x7ff30809cb30 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f8009a70 con 0x7ff30809cb30 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.363+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f803eb40 con 0x7ff30809cb30 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.364+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30814cc00 con 0x7ff30809cb30 2024-01-16T15:12:16.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.365+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30814cd90 con 0x7ff30809cb30 2024-01-16T15:12:16.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.366+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff2f803b050 con 0x7ff30809cb30 2024-01-16T15:12:16.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.366+0000 7ff2ffe00640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 0x7ff2e006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.367+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff2f80aaa80 con 0x7ff30809cb30 2024-01-16T15:12:16.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.367+0000 7ff2fe800640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 0x7ff2e006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.367+0000 7ff2fe800640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 0x7ff2e006e3d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff3081f62b0 tx=0x7ff2f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.371+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2f8077340 con 0x7ff30809cb30 2024-01-16T15:12:16.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.605+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96111", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3080b3a70 con 0x7ff2e006bf10 2024-01-16T15:12:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.625+0000 7ff2ffe00640 1 -- 172.21.15.43:0/3930603155 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff2f00109c0 con 0x7ff2e006bf10 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:16.602 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:16.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 msgr2=0x7ff2e006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 0x7ff2e006e3d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff3081f62b0 tx=0x7ff2f0008040 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 msgr2=0x7ff30814d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f8004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 shutdown_connections 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff2e006bf10 0x7ff2e006e3d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff30815b350 0x7ff30809b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3080adb60 0x7ff30813e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 --2- 172.21.15.43:0/3930603155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff30809cb30 0x7ff30814d400 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 >> 172.21.15.43:0/3930603155 conn(0x7ff3080b0ba0 msgr2=0x7ff308157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 shutdown_connections 2024-01-16T15:12:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.628+0000 7ff30e400640 1 -- 172.21.15.43:0/3930603155 wait complete. 2024-01-16T15:12:16.624 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:16.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96112 2024-01-16T15:12:16.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.866+0000 7f0110000640 1 Processor -- start 2024-01-16T15:12:16.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.867+0000 7f0110000640 1 -- start start 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.868+0000 7f0110000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f01080ae270 0x7f01080abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.868+0000 7f0110000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010815b730 con 0x7f010815b350 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.868+0000 7f0110000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01080ac740 con 0x7f010809cb30 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.868+0000 7f0110000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01080ac460 con 0x7f01080ae270 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f01080ae270 0x7f01080abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010d600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f010815b350 0x7f01080b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51282/0 (socket says 172.21.15.43:51282) 2024-01-16T15:12:16.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010d600640 1 -- 172.21.15.43:0/1727006474 learned_addr learned my addr 172.21.15.43:0/1727006474 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:16.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010e000640 1 -- 172.21.15.43:0/1727006474 >> v1:172.21.15.69:6789/0 conn(0x7f010809cb30 legacy=0x7f01080adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:16.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010e000640 1 -- 172.21.15.43:0/1727006474 >> v1:172.21.15.43:6789/0 conn(0x7f010815b350 legacy=0x7f01080b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f010e000640 1 -- 172.21.15.43:0/1727006474 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080b2a50 con 0x7f01080ae270 2024-01-16T15:12:16.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.869+0000 7f0107e00640 1 -- 172.21.15.43:0/1727006474 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404556706 0 0) 0x7f00f8001490 con 0x7f010815b350 2024-01-16T15:12:16.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.870+0000 7f010e000640 1 --2- 172.21.15.43:0/1727006474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f01080abe80 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f01000097a0 tx=0x7f010002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2db8415280e456f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0107e00640 1 -- 172.21.15.43:0/1727006474 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100031830 con 0x7f01080ae270 2024-01-16T15:12:16.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0107e00640 1 -- 172.21.15.43:0/1727006474 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100031990 con 0x7f01080ae270 2024-01-16T15:12:16.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0107e00640 1 -- 172.21.15.43:0/1727006474 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100031d20 con 0x7f01080ae270 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 -- 172.21.15.43:0/1727006474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 msgr2=0x7f01080abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 --2- 172.21.15.43:0/1727006474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f01080abe80 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f01000097a0 tx=0x7f010002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 -- 172.21.15.43:0/1727006474 shutdown_connections 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 --2- 172.21.15.43:0/1727006474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f01080abe80 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 -- 172.21.15.43:0/1727006474 >> 172.21.15.43:0/1727006474 conn(0x7f01080b0ba0 msgr2=0x7f01080b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 -- 172.21.15.43:0/1727006474 shutdown_connections 2024-01-16T15:12:16.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.871+0000 7f0110000640 1 -- 172.21.15.43:0/1727006474 wait complete. 2024-01-16T15:12:16.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.873+0000 7f0110000640 1 Processor -- start 2024-01-16T15:12:16.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.873+0000 7f0110000640 1 -- start start 2024-01-16T15:12:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f010809cb30 0x7f0108141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 0x7f010815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010815c240 con 0x7f01080ae270 2024-01-16T15:12:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010815c520 con 0x7f010815b350 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f0110000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010815c800 con 0x7f010809cb30 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.874+0000 7f010cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33026/0 (socket says 172.21.15.43:33026) 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 -- 172.21.15.43:0/996589077 learned_addr learned my addr 172.21.15.43:0/996589077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010e000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 0x7f010815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 -- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f010809cb30 msgr2=0x7f0108141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010d600640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f010809cb30 0x7f0108141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f010809cb30 0x7f0108141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 -- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 msgr2=0x7f010815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 0x7f010815dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 -- 172.21.15.43:0/996589077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100009400 con 0x7f01080ae270 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010e000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 0x7f010815dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:16.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f010cc00640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c430 tx=0x7f00f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.875+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0011790 con 0x7f01080ae270 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.876+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f00118f0 con 0x7f01080ae270 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.876+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0011ce0 con 0x7f01080ae270 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.876+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010815cae0 con 0x7f01080ae270 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.876+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010815d2e0 con 0x7f01080ae270 2024-01-16T15:12:16.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.877+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010809d200 con 0x7f01080ae270 2024-01-16T15:12:16.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.878+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f00f001c460 con 0x7f01080ae270 2024-01-16T15:12:16.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.879+0000 7f0105600640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 0x7f00e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:16.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.879+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00f00887d0 con 0x7f01080ae270 2024-01-16T15:12:16.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.880+0000 7f010d600640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 0x7f00e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:16.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.880+0000 7f010d600640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 0x7f00e006e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f00f80012e0 tx=0x7f00f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:16.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:16.884+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00f0054e30 con 0x7f01080ae270 2024-01-16T15:12:17.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.107+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96112", "target": ["mon-mgr", ""]}) v1 -- 0x7f010806e720 con 0x7f00e006be90 2024-01-16T15:12:17.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.126+0000 7f0105600640 1 -- 172.21.15.43:0/996589077 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f00f8001900 con 0x7f00e006be90 2024-01-16T15:12:17.102 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:17.103 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 msgr2=0x7f00e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 0x7f00e006e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f00f80012e0 tx=0x7f00f8009040 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 msgr2=0x7f010815b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c430 tx=0x7f00f000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 shutdown_connections 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00e006be90 0x7f00e006e350 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f010815b350 0x7f010815dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01080ae270 0x7f010815b8d0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 --2- 172.21.15.43:0/996589077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f010809cb30 0x7f0108141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 >> 172.21.15.43:0/996589077 conn(0x7f01080b0ba0 msgr2=0x7f010809bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.129+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 shutdown_connections 2024-01-16T15:12:17.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.130+0000 7f0110000640 1 -- 172.21.15.43:0/996589077 wait complete. 2024-01-16T15:12:17.125 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:17.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96113 2024-01-16T15:12:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.374+0000 7ff89ea00640 1 Processor -- start 2024-01-16T15:12:17.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.374+0000 7ff89ea00640 1 -- start start 2024-01-16T15:12:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.375+0000 7ff89ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8980cfa30 0x7ff8980cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.375+0000 7ff89ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8980d1fd0 0x7ff8980ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.375+0000 7ff89ea00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980b2980 con 0x7ff8980d0e00 2024-01-16T15:12:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.375+0000 7ff89ea00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980d0350 con 0x7ff8980d1fd0 2024-01-16T15:12:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.375+0000 7ff89ea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980cf220 con 0x7ff8980cfa30 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8980d1fd0 0x7ff8980ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89e000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff8980d0e00 0x7ff8980d1270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51286/0 (socket says 172.21.15.43:51286) 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89e000640 1 -- 172.21.15.43:0/269004226 learned_addr learned my addr 172.21.15.43:0/269004226 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89d600640 1 --2- 172.21.15.43:0/269004226 >> v2:172.21.15.69:3300/0 conn(0x7ff8980cfa30 0x7ff8980cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89cc00640 1 -- 172.21.15.43:0/269004226 >> v2:172.21.15.69:3300/0 conn(0x7ff8980cfa30 msgr2=0x7ff8980cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89cc00640 1 --2- 172.21.15.43:0/269004226 >> v2:172.21.15.69:3300/0 conn(0x7ff8980cfa30 0x7ff8980cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89cc00640 1 -- 172.21.15.43:0/269004226 >> v1:172.21.15.43:6789/0 conn(0x7ff8980d0e00 legacy=0x7ff8980d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff89cc00640 1 -- 172.21.15.43:0/269004226 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980cf500 con 0x7ff8980d1fd0 2024-01-16T15:12:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.376+0000 7ff897e00640 1 -- 172.21.15.43:0/269004226 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922643398 0 0) 0x7ff8980b2980 con 0x7ff8980d0e00 2024-01-16T15:12:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.377+0000 7ff89cc00640 1 --2- 172.21.15.43:0/269004226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d1fd0 0x7ff8980ce920 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ff89808d630 tx=0x7ff88002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5eb17d68365bc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff897e00640 1 -- 172.21.15.43:0/269004226 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880031830 con 0x7ff8980d1fd0 2024-01-16T15:12:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff897e00640 1 -- 172.21.15.43:0/269004226 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff880031990 con 0x7ff8980d1fd0 2024-01-16T15:12:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff897e00640 1 -- 172.21.15.43:0/269004226 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880031d20 con 0x7ff8980d1fd0 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 -- 172.21.15.43:0/269004226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d1fd0 msgr2=0x7ff8980ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 --2- 172.21.15.43:0/269004226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d1fd0 0x7ff8980ce920 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ff89808d630 tx=0x7ff88002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 -- 172.21.15.43:0/269004226 shutdown_connections 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 --2- 172.21.15.43:0/269004226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d1fd0 0x7ff8980ce920 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 --2- 172.21.15.43:0/269004226 >> v2:172.21.15.69:3300/0 conn(0x7ff8980cfa30 0x7ff8980cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 -- 172.21.15.43:0/269004226 >> 172.21.15.43:0/269004226 conn(0x7ff8980d3020 msgr2=0x7ff8980d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 -- 172.21.15.43:0/269004226 shutdown_connections 2024-01-16T15:12:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.378+0000 7ff89ea00640 1 -- 172.21.15.43:0/269004226 wait complete. 2024-01-16T15:12:17.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.380+0000 7ff89ea00640 1 Processor -- start 2024-01-16T15:12:17.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.380+0000 7ff89ea00640 1 -- start start 2024-01-16T15:12:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980cfa30 0x7ff8981f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 0x7ff8980c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c7d20 con 0x7ff8980d0e00 2024-01-16T15:12:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c8000 con 0x7ff8980d1fd0 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c82e0 con 0x7ff8980cfa30 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.381+0000 7ff89cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33058/0 (socket says 172.21.15.43:33058) 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 -- 172.21.15.43:0/124785210 learned_addr learned my addr 172.21.15.43:0/124785210 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89e000640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 0x7ff8980c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89d600640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980cfa30 0x7ff8981f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 -- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980cfa30 msgr2=0x7ff8981f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980cfa30 0x7ff8981f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 -- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 msgr2=0x7ff8980c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 0x7ff8980c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 -- 172.21.15.43:0/124785210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009400 con 0x7ff8980d0e00 2024-01-16T15:12:17.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89e000640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 0x7ff8980c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.382+0000 7ff89cc00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ff880009890 tx=0x7ff8800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.383+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88003ba70 con 0x7ff8980d0e00 2024-01-16T15:12:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.383+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88003bbf0 con 0x7ff8980d0e00 2024-01-16T15:12:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.383+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880045400 con 0x7ff8980d0e00 2024-01-16T15:12:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.383+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8980c85c0 con 0x7ff8980d0e00 2024-01-16T15:12:17.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.383+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8981f29b0 con 0x7ff8980d0e00 2024-01-16T15:12:17.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.384+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8980d0a40 con 0x7ff8980d0e00 2024-01-16T15:12:17.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.385+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff88004e440 con 0x7ff8980d0e00 2024-01-16T15:12:17.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.386+0000 7ff895600640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 0x7ff87006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.386+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8800aaac0 con 0x7ff8980d0e00 2024-01-16T15:12:17.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.386+0000 7ff89d600640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 0x7ff87006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.387+0000 7ff89d600640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 0x7ff87006e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff888006fd0 tx=0x7ff888008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.390+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff880077120 con 0x7ff8980d0e00 2024-01-16T15:12:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.621+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96113", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8980c8170 con 0x7ff87006be90 2024-01-16T15:12:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.643+0000 7ff895600640 1 -- 172.21.15.43:0/124785210 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff8980c8170 con 0x7ff87006be90 2024-01-16T15:12:17.620 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:17.620 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:17.620 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:17.621 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 msgr2=0x7ff87006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 0x7ff87006e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff888006fd0 tx=0x7ff888008040 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 msgr2=0x7ff8980b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ff880009890 tx=0x7ff8800046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 shutdown_connections 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff87006be90 0x7ff87006e350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8980d1fd0 0x7ff8980c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8980d0e00 0x7ff8980b8820 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 --2- 172.21.15.43:0/124785210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980cfa30 0x7ff8981f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 >> 172.21.15.43:0/124785210 conn(0x7ff8980d3020 msgr2=0x7ff8980d3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 shutdown_connections 2024-01-16T15:12:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.646+0000 7ff89ea00640 1 -- 172.21.15.43:0/124785210 wait complete. 2024-01-16T15:12:17.643 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:17.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96114 2024-01-16T15:12:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.891+0000 7f96e1a00640 1 Processor -- start 2024-01-16T15:12:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.892+0000 7f96e1a00640 1 -- start start 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.893+0000 7f96e1a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96dc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.893+0000 7f96e1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96dc06cad0 0x7f96dc0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.893+0000 7f96e1a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc071780 con 0x7f96dc072280 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96e1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc0b2320 con 0x7f96dc06cad0 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96e1a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc06df30 con 0x7f96dc160b10 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96d3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f96dc072280 0x7f96dc06c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51302/0 (socket says 172.21.15.43:51302) 2024-01-16T15:12:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96d3e00640 1 -- 172.21.15.43:0/2692163417 learned_addr learned my addr 172.21.15.43:0/2692163417 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96db400640 1 --2- 172.21.15.43:0/2692163417 >> v2:172.21.15.69:3300/0 conn(0x7f96dc06cad0 0x7f96dc0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96daa00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96dc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225123649 0 0) 0x7f96dc071780 con 0x7f96dc072280 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.894+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96bc003520 con 0x7f96dc072280 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215366542 0 0) 0x7f96bc003520 con 0x7f96dc072280 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 msgr2=0x7f96dc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96dc06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 >> v2:172.21.15.69:3300/0 conn(0x7f96dc06cad0 msgr2=0x7f96dc0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 --2- 172.21.15.43:0/2692163417 >> v2:172.21.15.69:3300/0 conn(0x7f96dc06cad0 0x7f96dc0b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0711c0 con 0x7f96dc072280 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f96c4001670 con 0x7f96dc072280 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 >> v1:172.21.15.43:6789/0 conn(0x7f96dc072280 legacy=0x7f96dc06c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96bc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96bc003f20 0x7f96bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc004d00 0x7f96bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0b2320 con 0x7f96bc003f20 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc06df30 con 0x7f96bc004d00 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc00ba00 con 0x7f96dc160b10 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.895+0000 7f96d3e00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96bc003f20 0x7f96bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96db400640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc004d00 0x7f96bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96daa00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96bc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96d3e00640 1 -- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 msgr2=0x7f96bc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96d3e00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc160b10 0x7f96bc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96d3e00640 1 -- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc004d00 msgr2=0x7f96bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96d3e00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc004d00 0x7f96bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96d3e00640 1 -- 172.21.15.43:0/2692163417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc003520 con 0x7f96bc003f20 2024-01-16T15:12:17.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.896+0000 7f96db400640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc004d00 0x7f96bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:17.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.897+0000 7f96d3e00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96bc003f20 0x7f96bc0047c0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c240 tx=0x7f96bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a585e53d5e97ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.897+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4005ad0 con 0x7f96bc003f20 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.897+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4005c50 con 0x7f96bc003f20 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.897+0000 7f96da000640 1 -- 172.21.15.43:0/2692163417 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c402c8b0 con 0x7f96bc003f20 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 -- 172.21.15.43:0/2692163417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96bc003f20 msgr2=0x7f96bc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 --2- 172.21.15.43:0/2692163417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96bc003f20 0x7f96bc0047c0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c240 tx=0x7f96bc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96db400640 1 -- 172.21.15.43:0/2692163417 reap_dead start 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 -- 172.21.15.43:0/2692163417 shutdown_connections 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 -- 172.21.15.43:0/2692163417 >> 172.21.15.43:0/2692163417 conn(0x7f96dc0b0ba0 msgr2=0x7f96dc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 -- 172.21.15.43:0/2692163417 shutdown_connections 2024-01-16T15:12:17.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.898+0000 7f96e1a00640 1 -- 172.21.15.43:0/2692163417 wait complete. 2024-01-16T15:12:17.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.900+0000 7f96e1a00640 1 Processor -- start 2024-01-16T15:12:17.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.900+0000 7f96e1a00640 1 -- start start 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc003f20 0x7f96dc0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 0x7f96dc099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0ca7f0 con 0x7f96dc15b630 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0c96c0 con 0x7f96bc003f20 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.901+0000 7f96e1a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0c8ab0 con 0x7f96dc0ca370 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33080/0 (socket says 172.21.15.43:33080) 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 -- 172.21.15.43:0/3324118659 learned_addr learned my addr 172.21.15.43:0/3324118659 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96daa00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc003f20 0x7f96dc0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96db400640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 0x7f96dc099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 -- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 msgr2=0x7f96dc099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 0x7f96dc099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 -- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc003f20 msgr2=0x7f96dc0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc003f20 0x7f96dc0c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96d3e00640 1 -- 172.21.15.43:0/3324118659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0c8d90 con 0x7f96dc15b630 2024-01-16T15:12:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.902+0000 7f96db400640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 0x7f96dc099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96d3e00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c360 tx=0x7f96c4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4039070 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c402ccc0 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4007290 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc0c9070 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.903+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc0c87a0 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.904+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc09d200 con 0x7f96dc15b630 2024-01-16T15:12:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.905+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f96c4032070 con 0x7f96dc15b630 2024-01-16T15:12:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.906+0000 7f96d2a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 0x7f96b406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.906+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f96c403c040 con 0x7f96dc15b630 2024-01-16T15:12:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.907+0000 7f96daa00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 0x7f96b406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.907+0000 7f96daa00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 0x7f96b406e380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009930 tx=0x7f96cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:17.911+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96c4076100 con 0x7f96dc15b630 2024-01-16T15:12:18.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.149+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96114", "target": ["mon-mgr", ""]}) v1 -- 0x7f96dc0b1f40 con 0x7f96b406bec0 2024-01-16T15:12:18.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.172+0000 7f96d2a00640 1 -- 172.21.15.43:0/3324118659 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f96cc0095c0 con 0x7f96b406bec0 2024-01-16T15:12:18.148 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:18.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:18.150 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:18.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 msgr2=0x7f96b406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 0x7f96b406e380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009930 tx=0x7f96cc008040 comp rx=0 tx=0).stop 2024-01-16T15:12:18.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 msgr2=0x7f96dc0c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c360 tx=0x7f96c4001a00 comp rx=0 tx=0).stop 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 shutdown_connections 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f96b406bec0 0x7f96b406e380 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc0ca370 0x7f96dc099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96dc15b630 0x7f96dc0c9e30 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 --2- 172.21.15.43:0/3324118659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96bc003f20 0x7f96dc0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 >> 172.21.15.43:0/3324118659 conn(0x7f96dc0b0ba0 msgr2=0x7f96dc0ad2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 shutdown_connections 2024-01-16T15:12:18.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.175+0000 7f96e1a00640 1 -- 172.21.15.43:0/3324118659 wait complete. 2024-01-16T15:12:18.170 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:18.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96115 2024-01-16T15:12:18.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.418+0000 7fbc80e00640 1 Processor -- start 2024-01-16T15:12:18.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.419+0000 7fbc80e00640 1 -- start start 2024-01-16T15:12:18.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc80e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbc7c15d3b0 0x7fbc7c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc80e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbc7c0d0fa0 0x7fbc7c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc80e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7c0c6bf0 con 0x7fbc7c0d1bc0 2024-01-16T15:12:18.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc80e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7c0d1430 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc80e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7c0d1fa0 con 0x7fbc7c15d3b0 2024-01-16T15:12:18.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc73e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbc7c0d0fa0 0x7fbc7c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc7be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbc7c0d1bc0 0x7fbc7c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51314/0 (socket says 172.21.15.43:51314) 2024-01-16T15:12:18.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.420+0000 7fbc7be00640 1 -- 172.21.15.43:0/359471814 learned_addr learned my addr 172.21.15.43:0/359471814 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:18.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc7b400640 1 --2- 172.21.15.43:0/359471814 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c15d3b0 0x7fbc7c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc73e00640 1 -- 172.21.15.43:0/359471814 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c15d3b0 msgr2=0x7fbc7c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc73e00640 1 --2- 172.21.15.43:0/359471814 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c15d3b0 0x7fbc7c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc73e00640 1 -- 172.21.15.43:0/359471814 >> v1:172.21.15.43:6789/0 conn(0x7fbc7c0d1bc0 legacy=0x7fbc7c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc73e00640 1 -- 172.21.15.43:0/359471814 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0cedc0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.421+0000 7fbc7aa00640 1 -- 172.21.15.43:0/359471814 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566081012 0 0) 0x7fbc7c0c6bf0 con 0x7fbc7c0d1bc0 2024-01-16T15:12:18.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.422+0000 7fbc73e00640 1 --2- 172.21.15.43:0/359471814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c0d1680 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c08b2a0 tx=0x7fbc6402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8ce3f66362d1c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.422+0000 7fbc7aa00640 1 -- 172.21.15.43:0/359471814 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6402e070 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc7aa00640 1 -- 172.21.15.43:0/359471814 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc640315a0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc7aa00640 1 -- 172.21.15.43:0/359471814 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64031930 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 -- 172.21.15.43:0/359471814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 msgr2=0x7fbc7c0d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 --2- 172.21.15.43:0/359471814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c0d1680 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c08b2a0 tx=0x7fbc6402b840 comp rx=0 tx=0).stop 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 -- 172.21.15.43:0/359471814 shutdown_connections 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 --2- 172.21.15.43:0/359471814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c0d1680 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 --2- 172.21.15.43:0/359471814 >> v2:172.21.15.69:3300/0 conn(0x7fbc7c15d3b0 0x7fbc7c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 -- 172.21.15.43:0/359471814 >> 172.21.15.43:0/359471814 conn(0x7fbc7c0d30a0 msgr2=0x7fbc7c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 -- 172.21.15.43:0/359471814 shutdown_connections 2024-01-16T15:12:18.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.423+0000 7fbc80e00640 1 -- 172.21.15.43:0/359471814 wait complete. 2024-01-16T15:12:18.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.425+0000 7fbc80e00640 1 Processor -- start 2024-01-16T15:12:18.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.425+0000 7fbc80e00640 1 -- start start 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7c0d1bc0 0x7fbc7c1f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c15d3b0 0x7fbc7c0ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0a0dc0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c1f3b70 con 0x7fbc7c0d1bc0 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc80e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0a0800 con 0x7fbc7c15d3b0 2024-01-16T15:12:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.426+0000 7fbc7b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33098/0 (socket says 172.21.15.43:33098) 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 -- 172.21.15.43:0/2227058999 learned_addr learned my addr 172.21.15.43:0/2227058999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc73e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7c0d1bc0 0x7fbc7c1f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7be00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c15d3b0 0x7fbc7c0ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 -- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c15d3b0 msgr2=0x7fbc7c0ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c15d3b0 0x7fbc7c0ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 -- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7c0d1bc0 msgr2=0x7fbc7c1f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7c0d1bc0 0x7fbc7c1f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 -- 172.21.15.43:0/2227058999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64009400 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.427+0000 7fbc7b400640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d800 tx=0x7fbc6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.428+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00b6c0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.428+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00b820 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.428+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00bbb0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.428+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7c0a0ae0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.428+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7c09d260 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.429+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7c09cde0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.430+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbc6c00bd10 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.431+0000 7fbc73400640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 0x7fbc5406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.432+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbc6c088db0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.432+0000 7fbc73e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 0x7fbc5406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.432+0000 7fbc73e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 0x7fbc5406e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbc64036000 tx=0x7fbc64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.436+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc7c09cde0 con 0x7fbc7c0d0fa0 2024-01-16T15:12:18.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.675+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96115", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc7c0c7400 con 0x7fbc5406be90 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.693+0000 7fbc73400640 1 -- 172.21.15.43:0/2227058999 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fbc64004340 con 0x7fbc5406be90 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:18.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:18.671 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.696+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 msgr2=0x7fbc5406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.696+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 0x7fbc5406e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbc64036000 tx=0x7fbc64008030 comp rx=0 tx=0).stop 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.696+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 msgr2=0x7fbc7c15c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.696+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d800 tx=0x7fbc6c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 shutdown_connections 2024-01-16T15:12:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbc5406be90 0x7fbc5406e350 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7c0d1bc0 0x7fbc7c1f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc7c15d3b0 0x7fbc7c0ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 --2- 172.21.15.43:0/2227058999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbc7c0d0fa0 0x7fbc7c15c6d0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 >> 172.21.15.43:0/2227058999 conn(0x7fbc7c0d30a0 msgr2=0x7fbc7c15fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 shutdown_connections 2024-01-16T15:12:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.697+0000 7fbc80e00640 1 -- 172.21.15.43:0/2227058999 wait complete. 2024-01-16T15:12:18.692 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:18.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96116 2024-01-16T15:12:18.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.942+0000 7f6711800640 1 Processor -- start 2024-01-16T15:12:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.942+0000 7f6711800640 1 -- start start 2024-01-16T15:12:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6711800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f670c15b680 0x7f670c0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6711800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c06f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6711800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c06e8a0 con 0x7f670c0b1c70 2024-01-16T15:12:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6711800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c15baa0 con 0x7f670c15b680 2024-01-16T15:12:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6711800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0cf660 con 0x7f670c0cee40 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f6701e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f670c15b680 0x7f670c0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.944+0000 7f670aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f670c0b1c70 0x7f670c0b2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51326/0 (socket says 172.21.15.43:51326) 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f670aa00640 1 -- 172.21.15.43:0/4192908776 learned_addr learned my addr 172.21.15.43:0/4192908776 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f670b400640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c06f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f6701e00640 1 -- 172.21.15.43:0/4192908776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 msgr2=0x7f670c06f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f6701e00640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c06f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f6701e00640 1 -- 172.21.15.43:0/4192908776 >> v1:172.21.15.43:6789/0 conn(0x7f670c0b1c70 legacy=0x7f670c0b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f6701e00640 1 -- 172.21.15.43:0/4192908776 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0cf940 con 0x7f670c15b680 2024-01-16T15:12:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.945+0000 7f670a000640 1 -- 172.21.15.43:0/4192908776 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243049744 0 0) 0x7f670c06e8a0 con 0x7f670c0b1c70 2024-01-16T15:12:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.946+0000 7f6701e00640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c0ce900 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f66f40097a0 tx=0x7f66f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98f1dffbfbdb60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.947+0000 7f670a000640 1 -- 172.21.15.43:0/4192908776 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4031830 con 0x7f670c15b680 2024-01-16T15:12:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.947+0000 7f670a000640 1 -- 172.21.15.43:0/4192908776 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f4031990 con 0x7f670c15b680 2024-01-16T15:12:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.947+0000 7f670a000640 1 -- 172.21.15.43:0/4192908776 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4031d20 con 0x7f670c15b680 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.947+0000 7f6711800640 1 -- 172.21.15.43:0/4192908776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 msgr2=0x7f670c0ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.947+0000 7f6711800640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c0ce900 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f66f40097a0 tx=0x7f66f402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 -- 172.21.15.43:0/4192908776 shutdown_connections 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c06f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 --2- 172.21.15.43:0/4192908776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c0ce900 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 -- 172.21.15.43:0/4192908776 >> 172.21.15.43:0/4192908776 conn(0x7f670c06d5a0 msgr2=0x7f670c06d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 -- 172.21.15.43:0/4192908776 shutdown_connections 2024-01-16T15:12:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.948+0000 7f6711800640 1 -- 172.21.15.43:0/4192908776 wait complete. 2024-01-16T15:12:18.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.949+0000 7f6711800640 1 Processor -- start 2024-01-16T15:12:18.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.949+0000 7f6711800640 1 -- start start 2024-01-16T15:12:18.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 0x7f670c1e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c1e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c1e4600 con 0x7f670c15b680 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c140da0 con 0x7f670c0b1c70 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.950+0000 7f6711800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c13ffe0 con 0x7f670c0cee40 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33112/0 (socket says 172.21.15.43:33112) 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 -- 172.21.15.43:0/145619110 learned_addr learned my addr 172.21.15.43:0/145619110 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f6701e00640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c1e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670aa00640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 0x7f670c1e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 -- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 msgr2=0x7f670c1e3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c1e3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 -- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 msgr2=0x7f670c1e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 0x7f670c1e5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:18.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670b400640 1 -- 172.21.15.43:0/145619110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f4009400 con 0x7f670c15b680 2024-01-16T15:12:18.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.951+0000 7f670aa00640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 0x7f670c1e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f670b400640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f670400d800 tx=0x7f670400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400b6c0 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400b840 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400bbf0 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c1402c0 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.952+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c140880 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.953+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c04e920 con 0x7f670c15b680 2024-01-16T15:12:18.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.954+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f670400bd50 con 0x7f670c15b680 2024-01-16T15:12:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.955+0000 7f6703400640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 0x7f66e406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.956+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6704088d40 con 0x7f670c15b680 2024-01-16T15:12:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.956+0000 7f670aa00640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 0x7f66e406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.956+0000 7f670aa00640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 0x7f66e406e380 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0012e0 tx=0x7f66fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:18.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:18.960+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67040553a0 con 0x7f670c15b680 2024-01-16T15:12:19.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.202+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96116", "target": ["mon-mgr", ""]}) v1 -- 0x7f670c0cf2d0 con 0x7f66e406bec0 2024-01-16T15:12:19.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.223+0000 7f6703400640 1 -- 172.21.15.43:0/145619110 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f66fc004b60 con 0x7f66e406bec0 2024-01-16T15:12:19.199 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:19.199 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:19.199 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:19.200 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:19.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.225+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 msgr2=0x7f66e406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.225+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 0x7f66e406e380 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0012e0 tx=0x7f66fc009040 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.225+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 msgr2=0x7f670c1e1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.225+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f670400d800 tx=0x7f670400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 shutdown_connections 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66e406bec0 0x7f66e406e380 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f670c15b680 0x7f670c1e1970 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f670c0cee40 0x7f670c1e3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 --2- 172.21.15.43:0/145619110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c0b1c70 0x7f670c1e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 >> 172.21.15.43:0/145619110 conn(0x7f670c06d5a0 msgr2=0x7f670c06efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 shutdown_connections 2024-01-16T15:12:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.226+0000 7f6711800640 1 -- 172.21.15.43:0/145619110 wait complete. 2024-01-16T15:12:19.223 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:19.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96117 2024-01-16T15:12:19.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.464+0000 7f1000800640 1 Processor -- start 2024-01-16T15:12:19.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.465+0000 7f1000800640 1 -- start start 2024-01-16T15:12:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.466+0000 7f1000800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 0x7f0ff80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.466+0000 7f1000800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 0x7f0ff80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.466+0000 7f1000800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff815b730 con 0x7f0ff815b350 2024-01-16T15:12:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.466+0000 7f1000800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff80ac740 con 0x7f0ff80adb60 2024-01-16T15:12:19.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.466+0000 7f1000800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80ac460 con 0x7f0ff809cb30 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.467+0000 7f0ffd400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 0x7f0ff80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.467+0000 7f0ffe800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 0x7f0ff80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.467+0000 7f0ffde00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0ff815b350 0x7f0ff80b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:49026/0 (socket says 172.21.15.43:49026) 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.467+0000 7f0ffde00640 1 -- 172.21.15.43:0/3351166170 learned_addr learned my addr 172.21.15.43:0/3351166170 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.468+0000 7f0ffd400640 1 -- 172.21.15.43:0/3351166170 >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 msgr2=0x7f0ff80adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.468+0000 7f0ffd400640 1 --2- 172.21.15.43:0/3351166170 >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 0x7f0ff80adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.468+0000 7f0ffd400640 1 -- 172.21.15.43:0/3351166170 >> v1:172.21.15.69:6789/0 conn(0x7f0ff815b350 legacy=0x7f0ff80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.468+0000 7f0ffd400640 1 -- 172.21.15.43:0/3351166170 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80b2a50 con 0x7f0ff809cb30 2024-01-16T15:12:19.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.468+0000 7f0ffe800640 1 --2- 172.21.15.43:0/3351166170 >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 0x7f0ff80adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:19.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.469+0000 7f0ffd400640 1 --2- 172.21.15.43:0/3351166170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 0x7f0ff80abe80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00097a0 tx=0x7f0fe002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae7390b61077b6be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.470+0000 7f0ffca00640 1 -- 172.21.15.43:0/3351166170 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe002d040 con 0x7f0ff809cb30 2024-01-16T15:12:19.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.470+0000 7f0ffca00640 1 -- 172.21.15.43:0/3351166170 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0031810 con 0x7f0ff809cb30 2024-01-16T15:12:19.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.470+0000 7f0ffca00640 1 -- 172.21.15.43:0/3351166170 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0031ba0 con 0x7f0ff809cb30 2024-01-16T15:12:19.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 -- 172.21.15.43:0/3351166170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 msgr2=0x7f0ff80abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 --2- 172.21.15.43:0/3351166170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 0x7f0ff80abe80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00097a0 tx=0x7f0fe002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 -- 172.21.15.43:0/3351166170 shutdown_connections 2024-01-16T15:12:19.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 --2- 172.21.15.43:0/3351166170 >> v2:172.21.15.69:3300/0 conn(0x7f0ff80adb60 0x7f0ff80adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 --2- 172.21.15.43:0/3351166170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff809cb30 0x7f0ff80abe80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 -- 172.21.15.43:0/3351166170 >> 172.21.15.43:0/3351166170 conn(0x7f0ff80b0ba0 msgr2=0x7f0ff80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:19.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 -- 172.21.15.43:0/3351166170 shutdown_connections 2024-01-16T15:12:19.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.471+0000 7f1000800640 1 -- 172.21.15.43:0/3351166170 wait complete. 2024-01-16T15:12:19.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.474+0000 7f1000800640 1 Processor -- start 2024-01-16T15:12:19.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.474+0000 7f1000800640 1 -- start start 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 0x7f0ff81552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff815b350 0x7f0ff8155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff813eff0 con 0x7f0ff809cb30 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8155d60 con 0x7f0ff80adb60 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.475+0000 7f1000800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff81537d0 con 0x7f0ff815b350 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33122/0 (socket says 172.21.15.43:33122) 2024-01-16T15:12:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 -- 172.21.15.43:0/1217844242 learned_addr learned my addr 172.21.15.43:0/1217844242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffe800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff815b350 0x7f0ff8155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffd400640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 0x7f0ff81552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 -- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff815b350 msgr2=0x7f0ff8155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff815b350 0x7f0ff8155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 -- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 msgr2=0x7f0ff81552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 0x7f0ff81552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffde00640 1 -- 172.21.15.43:0/1217844242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0009400 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.476+0000 7f0ffd400640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 0x7f0ff81552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f0ffde00640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800ca70 tx=0x7f0fe800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800ab10 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800ac70 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8012620 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe0009400 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.477+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8150e20 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.478+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff809d200 con 0x7f0ff809cb30 2024-01-16T15:12:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.479+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0fe8007070 con 0x7f0ff809cb30 2024-01-16T15:12:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.481+0000 7f0ff6000640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 0x7f0fd006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.481+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0fe8088580 con 0x7f0ff809cb30 2024-01-16T15:12:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.481+0000 7f0ffd400640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 0x7f0fd006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.482+0000 7f0ffd400640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 0x7f0fd006e350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0036000 tx=0x7f0fe003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.485+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fe8054b50 con 0x7f0ff809cb30 2024-01-16T15:12:19.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.719+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96117", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ff80ae4f0 con 0x7f0fd006be90 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.736+0000 7f0ff6000640 1 -- 172.21.15.43:0/1217844242 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f0fe0031b50 con 0x7f0fd006be90 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:19.712 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:19.713 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.738+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 msgr2=0x7f0fd006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.738+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 0x7f0fd006e350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0036000 tx=0x7f0fe003c040 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.738+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 msgr2=0x7f0ff813eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.738+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800ca70 tx=0x7f0fe800ce70 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.738+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 shutdown_connections 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0fd006be90 0x7f0fd006e350 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ff80adb60 0x7f0ff81552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ff815b350 0x7f0ff8155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 --2- 172.21.15.43:0/1217844242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ff809cb30 0x7f0ff813eab0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 >> 172.21.15.43:0/1217844242 conn(0x7f0ff80b0ba0 msgr2=0x7f0ff80aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 shutdown_connections 2024-01-16T15:12:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.739+0000 7f1000800640 1 -- 172.21.15.43:0/1217844242 wait complete. 2024-01-16T15:12:19.733 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:19.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96118 2024-01-16T15:12:19.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.980+0000 7f1d93e00640 1 Processor -- start 2024-01-16T15:12:19.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.981+0000 7f1d93e00640 1 -- start start 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.982+0000 7f1d93e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d940c8930 0x7f1d940ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.982+0000 7f1d93e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d940c6bf0 con 0x7f1d940d1d90 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.982+0000 7f1d93e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d940cedc0 con 0x7f1d9415d3b0 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.982+0000 7f1d93e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d940c6e40 con 0x7f1d940c8930 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d940c8930 0x7f1d940ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d940c8930 0x7f1d940ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33134/0 (socket says 172.21.15.43:33134) 2024-01-16T15:12:19.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 -- 172.21.15.43:0/1723446143 learned_addr learned my addr 172.21.15.43:0/1723446143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:19.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 -- 172.21.15.43:0/1723446143 >> v1:172.21.15.69:6789/0 conn(0x7f1d9415d3b0 legacy=0x7f1d940d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:19.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 -- 172.21.15.43:0/1723446143 >> v1:172.21.15.43:6789/0 conn(0x7f1d940d1d90 legacy=0x7f1d940b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d93400640 1 -- 172.21.15.43:0/1723446143 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d940c7120 con 0x7f1d940c8930 2024-01-16T15:12:19.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.983+0000 7f1d92000640 1 -- 172.21.15.43:0/1723446143 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293169020 0 0) 0x7f1d940c6bf0 con 0x7f1d940d1d90 2024-01-16T15:12:19.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.984+0000 7f1d93400640 1 --2- 172.21.15.43:0/1723446143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940ce6c0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009e50 tx=0x7f1d8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0862bbd0f2d0d7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d92000640 1 -- 172.21.15.43:0/1723446143 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c02ca80 con 0x7f1d940c8930 2024-01-16T15:12:19.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d92000640 1 -- 172.21.15.43:0/1723446143 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c02cbe0 con 0x7f1d940c8930 2024-01-16T15:12:19.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d92000640 1 -- 172.21.15.43:0/1723446143 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0343f0 con 0x7f1d940c8930 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d93e00640 1 -- 172.21.15.43:0/1723446143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 msgr2=0x7f1d940ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d93e00640 1 --2- 172.21.15.43:0/1723446143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940ce6c0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009e50 tx=0x7f1d8c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d93e00640 1 -- 172.21.15.43:0/1723446143 shutdown_connections 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d93e00640 1 --2- 172.21.15.43:0/1723446143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940ce6c0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.985+0000 7f1d93e00640 1 -- 172.21.15.43:0/1723446143 >> 172.21.15.43:0/1723446143 conn(0x7f1d940d30a0 msgr2=0x7f1d940d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.986+0000 7f1d93e00640 1 -- 172.21.15.43:0/1723446143 shutdown_connections 2024-01-16T15:12:19.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.986+0000 7f1d93e00640 1 -- 172.21.15.43:0/1723446143 wait complete. 2024-01-16T15:12:19.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.987+0000 7f1d93e00640 1 Processor -- start 2024-01-16T15:12:19.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.987+0000 7f1d93e00640 1 -- start start 2024-01-16T15:12:19.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d940d1d90 0x7f1d9415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9415d3b0 0x7f1d9415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940b6710 con 0x7f1d940c8930 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94158160 con 0x7f1d9415d3b0 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.988+0000 7f1d93e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94158440 con 0x7f1d940d1d90 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33136/0 (socket says 172.21.15.43:33136) 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 -- 172.21.15.43:0/787813847 learned_addr learned my addr 172.21.15.43:0/787813847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d8be00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d940d1d90 0x7f1d9415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d93400640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9415d3b0 0x7f1d9415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 -- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d940d1d90 msgr2=0x7f1d9415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d940d1d90 0x7f1d9415f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 -- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9415d3b0 msgr2=0x7f1d9415f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9415d3b0 0x7f1d9415f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 -- 172.21.15.43:0/787813847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c009b30 con 0x7f1d940c8930 2024-01-16T15:12:19.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.989+0000 7f1d92a00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400b3f0 tx=0x7f1d8400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.990+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84011830 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.990+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d84011990 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.990+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84011d40 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.990+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9415ae10 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.990+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94146f70 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.991+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9415b330 con 0x7f1d940c8930 2024-01-16T15:12:19.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.992+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1d8401a400 con 0x7f1d940c8930 2024-01-16T15:12:19.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.993+0000 7f1d8aa00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 0x7f1d6c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:19.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.993+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1d840886c0 con 0x7f1d940c8930 2024-01-16T15:12:19.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.993+0000 7f1d8be00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 0x7f1d6c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:19.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.994+0000 7f1d8be00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 0x7f1d6c06e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1d940b6090 tx=0x7f1d7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:19.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:19.997+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d9415b330 con 0x7f1d940c8930 2024-01-16T15:12:20.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.233+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96118", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d9415d790 con 0x7f1d6c06be90 2024-01-16T15:12:20.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.254+0000 7f1d8aa00640 1 -- 172.21.15.43:0/787813847 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f1d7c0013d0 con 0x7f1d6c06be90 2024-01-16T15:12:20.230 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:20.230 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:20.231 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:20.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 msgr2=0x7f1d6c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 0x7f1d6c06e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1d940b6090 tx=0x7f1d7c009040 comp rx=0 tx=0).stop 2024-01-16T15:12:20.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 msgr2=0x7f1d940b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400b3f0 tx=0x7f1d8400b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 shutdown_connections 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1d6c06be90 0x7f1d6c06e350 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9415d3b0 0x7f1d9415f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1d940d1d90 0x7f1d9415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 --2- 172.21.15.43:0/787813847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d940c8930 0x7f1d940b61d0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 >> 172.21.15.43:0/787813847 conn(0x7f1d940d30a0 msgr2=0x7f1d9406af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 shutdown_connections 2024-01-16T15:12:20.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.257+0000 7f1d93e00640 1 -- 172.21.15.43:0/787813847 wait complete. 2024-01-16T15:12:20.253 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:20.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96119 2024-01-16T15:12:20.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.497+0000 7ff8e0a00640 1 Processor -- start 2024-01-16T15:12:20.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.497+0000 7ff8e0a00640 1 -- start start 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8e0a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8dc0b4140 0x7ff8dc071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8e0a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8dc1439b0 0x7ff8dc070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8e0a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc070d40 con 0x7ff8dc1435d0 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8e0a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc071020 con 0x7ff8dc1439b0 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8e0a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc071300 con 0x7ff8dc0b4140 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8db400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff8dc1435d0 0x7ff8dc0b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51352/0 (socket says 172.21.15.43:51352) 2024-01-16T15:12:20.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8db400640 1 -- 172.21.15.43:0/1033265188 learned_addr learned my addr 172.21.15.43:0/1033265188 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:20.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8dbe00640 1 --2- 172.21.15.43:0/1033265188 >> v2:172.21.15.43:3300/0 conn(0x7ff8dc1439b0 0x7ff8dc070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8daa00640 1 --2- 172.21.15.43:0/1033265188 >> v2:172.21.15.69:3300/0 conn(0x7ff8dc0b4140 0x7ff8dc071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.499+0000 7ff8da000640 1 -- 172.21.15.43:0/1033265188 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859125013 0 0) 0x7ff8dc070d40 con 0x7ff8dc1435d0 2024-01-16T15:12:20.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.500+0000 7ff8da000640 1 -- 172.21.15.43:0/1033265188 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc003520 con 0x7ff8dc1435d0 2024-01-16T15:12:20.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.500+0000 7ff8dbe00640 1 -- 172.21.15.43:0/1033265188 >> v2:172.21.15.69:3300/0 conn(0x7ff8dc0b4140 msgr2=0x7ff8dc071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:20.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.500+0000 7ff8dbe00640 1 --2- 172.21.15.43:0/1033265188 >> v2:172.21.15.69:3300/0 conn(0x7ff8dc0b4140 0x7ff8dc071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.500+0000 7ff8dbe00640 1 -- 172.21.15.43:0/1033265188 >> v1:172.21.15.43:6789/0 conn(0x7ff8dc1435d0 legacy=0x7ff8dc0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.500+0000 7ff8dbe00640 1 -- 172.21.15.43:0/1033265188 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0715e0 con 0x7ff8dc1439b0 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8dbe00640 1 --2- 172.21.15.43:0/1033265188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc070050 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0cbeb0 tx=0x7ff8d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95830bf74738549 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8da000640 1 -- 172.21.15.43:0/1033265188 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d402e070 con 0x7ff8dc1439b0 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8da000640 1 -- 172.21.15.43:0/1033265188 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d40315a0 con 0x7ff8dc1439b0 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8da000640 1 -- 172.21.15.43:0/1033265188 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4031930 con 0x7ff8dc1439b0 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8e0a00640 1 -- 172.21.15.43:0/1033265188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1439b0 msgr2=0x7ff8dc070050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.501+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/1033265188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc070050 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0cbeb0 tx=0x7ff8d402b840 comp rx=0 tx=0).stop 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 -- 172.21.15.43:0/1033265188 shutdown_connections 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/1033265188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc070050 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/1033265188 >> v2:172.21.15.69:3300/0 conn(0x7ff8dc0b4140 0x7ff8dc071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 -- 172.21.15.43:0/1033265188 >> 172.21.15.43:0/1033265188 conn(0x7ff8dc0cec10 msgr2=0x7ff8dc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 -- 172.21.15.43:0/1033265188 shutdown_connections 2024-01-16T15:12:20.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.502+0000 7ff8e0a00640 1 -- 172.21.15.43:0/1033265188 wait complete. 2024-01-16T15:12:20.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.503+0000 7ff8e0a00640 1 Processor -- start 2024-01-16T15:12:20.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.504+0000 7ff8e0a00640 1 -- start start 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.504+0000 7ff8e0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8dc0b4140 0x7ff8dc1394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8e0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8e0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc1427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8e0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0ce670 con 0x7ff8dc1435d0 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8e0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0cdd80 con 0x7ff8dc1439b0 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8e0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0cc880 con 0x7ff8dc0b4140 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33156/0 (socket says 172.21.15.43:33156) 2024-01-16T15:12:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 -- 172.21.15.43:0/2007856750 learned_addr learned my addr 172.21.15.43:0/2007856750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8dbe00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc1427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8db400640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8dc0b4140 0x7ff8dc1394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 -- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8dc0b4140 msgr2=0x7ff8dc1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8dc0b4140 0x7ff8dc1394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 -- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8dc1439b0 msgr2=0x7ff8dc1427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.505+0000 7ff8daa00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc1427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8daa00640 1 -- 172.21.15.43:0/2007856750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4009400 con 0x7ff8dc1435d0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8daa00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4007ef0 tx=0x7ff8c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4014620 con 0x7ff8dc1435d0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4014780 con 0x7ff8dc1435d0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc0cc300 con 0x7ff8dc1435d0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4014b30 con 0x7ff8dc1435d0 2024-01-16T15:12:20.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.506+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc0cdfb0 con 0x7ff8dc1435d0 2024-01-16T15:12:20.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.507+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc04e840 con 0x7ff8dc1435d0 2024-01-16T15:12:20.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.508+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff8c4014c90 con 0x7ff8dc1435d0 2024-01-16T15:12:20.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.509+0000 7ff8d3400640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 0x7ff8b406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.509+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8c4089000 con 0x7ff8dc1435d0 2024-01-16T15:12:20.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.510+0000 7ff8db400640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 0x7ff8b406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.510+0000 7ff8db400640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 0x7ff8b406e350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0011e0 tx=0x7ff8cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.513+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8c40558c0 con 0x7ff8dc1435d0 2024-01-16T15:12:20.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.735+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96119", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8dc0b3500 con 0x7ff8b406be90 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.754+0000 7ff8d3400640 1 -- 172.21.15.43:0/2007856750 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff8dc0b3500 con 0x7ff8b406be90 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:20.730 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:20.731 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.756+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 msgr2=0x7ff8b406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.756+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 0x7ff8b406e350 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0011e0 tx=0x7ff8cc009040 comp rx=0 tx=0).stop 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.756+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 msgr2=0x7ff8dc136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.756+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4007ef0 tx=0x7ff8c400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 shutdown_connections 2024-01-16T15:12:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8b406be90 0x7ff8b406e350 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8dc1439b0 0x7ff8dc1427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8dc1435d0 0x7ff8dc136fd0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 --2- 172.21.15.43:0/2007856750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8dc0b4140 0x7ff8dc1394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 >> 172.21.15.43:0/2007856750 conn(0x7ff8dc0cec10 msgr2=0x7ff8dc06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 shutdown_connections 2024-01-16T15:12:20.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.757+0000 7ff8e0a00640 1 -- 172.21.15.43:0/2007856750 wait complete. 2024-01-16T15:12:20.750 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:20.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone96120 2024-01-16T15:12:20.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.991+0000 7f1384000640 1 Processor -- start 2024-01-16T15:12:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.991+0000 7f1384000640 1 -- start start 2024-01-16T15:12:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.993+0000 7f1384000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f137c158f40 0x7f137c0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.993+0000 7f1384000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f137c0cb9d0 0x7f137c06f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.993+0000 7f1384000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f137c0cd2b0 con 0x7f137c0cec70 2024-01-16T15:12:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.993+0000 7f1384000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137c0d02e0 con 0x7f137c0cb9d0 2024-01-16T15:12:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.993+0000 7f1384000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137c06f670 con 0x7f137c158f40 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1382000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f137c0cec70 0x7f137c15b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51368/0 (socket says 172.21.15.43:51368) 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1380c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f137c0cb9d0 0x7f137c06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1382000640 1 -- 172.21.15.43:0/1754703471 learned_addr learned my addr 172.21.15.43:0/1754703471 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1381600640 1 --2- 172.21.15.43:0/1754703471 >> v2:172.21.15.69:3300/0 conn(0x7f137c158f40 0x7f137c0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f137be00640 1 -- 172.21.15.43:0/1754703471 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753598806 0 0) 0x7f137c0cd2b0 con 0x7f137c0cec70 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f137be00640 1 -- 172.21.15.43:0/1754703471 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f135c003520 con 0x7f137c0cec70 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1380c00640 1 -- 172.21.15.43:0/1754703471 >> v2:172.21.15.69:3300/0 conn(0x7f137c158f40 msgr2=0x7f137c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1380c00640 1 --2- 172.21.15.43:0/1754703471 >> v2:172.21.15.69:3300/0 conn(0x7f137c158f40 0x7f137c0ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1380c00640 1 -- 172.21.15.43:0/1754703471 >> v1:172.21.15.43:6789/0 conn(0x7f137c0cec70 legacy=0x7f137c15b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.994+0000 7f1380c00640 1 -- 172.21.15.43:0/1754703471 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c0cbe30 con 0x7f137c0cb9d0 2024-01-16T15:12:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.996+0000 7f1380c00640 1 --2- 172.21.15.43:0/1754703471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cb9d0 0x7f137c06f130 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f137c0d20e0 tx=0x7f136402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a16b2e6c81c5899 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.996+0000 7f137be00640 1 -- 172.21.15.43:0/1754703471 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364031830 con 0x7f137c0cb9d0 2024-01-16T15:12:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.996+0000 7f137be00640 1 -- 172.21.15.43:0/1754703471 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1364031990 con 0x7f137c0cb9d0 2024-01-16T15:12:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.996+0000 7f137be00640 1 -- 172.21.15.43:0/1754703471 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364031d20 con 0x7f137c0cb9d0 2024-01-16T15:12:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 -- 172.21.15.43:0/1754703471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cb9d0 msgr2=0x7f137c06f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 --2- 172.21.15.43:0/1754703471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cb9d0 0x7f137c06f130 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f137c0d20e0 tx=0x7f136402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 -- 172.21.15.43:0/1754703471 shutdown_connections 2024-01-16T15:12:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 --2- 172.21.15.43:0/1754703471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cb9d0 0x7f137c06f130 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 --2- 172.21.15.43:0/1754703471 >> v2:172.21.15.69:3300/0 conn(0x7f137c158f40 0x7f137c0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 -- 172.21.15.43:0/1754703471 >> 172.21.15.43:0/1754703471 conn(0x7f137c0cdb20 msgr2=0x7f137c0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 -- 172.21.15.43:0/1754703471 shutdown_connections 2024-01-16T15:12:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.997+0000 7f1384000640 1 -- 172.21.15.43:0/1754703471 wait complete. 2024-01-16T15:12:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.999+0000 7f1384000640 1 Processor -- start 2024-01-16T15:12:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:20.999+0000 7f1384000640 1 -- start start 2024-01-16T15:12:20.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f137c0cb9d0 0x7f137c0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 0x7f137c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c152660 con 0x7f137c0cec70 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0c9060 con 0x7f137c0cb9d0 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1384000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0c82a0 con 0x7f137c158f40 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1380c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1380c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33176/0 (socket says 172.21.15.43:33176) 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1380c00640 1 -- 172.21.15.43:0/3830405460 learned_addr learned my addr 172.21.15.43:0/3830405460 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.000+0000 7f1382000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 0x7f137c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 -- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 msgr2=0x7f137c159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1381600640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f137c0cb9d0 0x7f137c0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 0x7f137c159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 -- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f137c0cb9d0 msgr2=0x7f137c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f137c0cb9d0 0x7f137c0cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 -- 172.21.15.43:0/3830405460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364009400 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1382000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 0x7f137c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1380c00640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1364009890 tx=0x7f13640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136403a630 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136403a7b0 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136403ab60 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c0c8580 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.001+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c0c8b40 con 0x7f137c0cec70 2024-01-16T15:12:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.002+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c04e920 con 0x7f137c0cec70 2024-01-16T15:12:20.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.004+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f136403acc0 con 0x7f137c0cec70 2024-01-16T15:12:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.005+0000 7f1379600640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 0x7f135406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.005+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f13640aa390 con 0x7f137c0cec70 2024-01-16T15:12:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.005+0000 7f1381600640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 0x7f135406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.006+0000 7f1381600640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 0x7f135406e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f136c009970 tx=0x7f136c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:20.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.010+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1364076a50 con 0x7f137c0cec70 2024-01-16T15:12:21.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.240+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone96120", "target": ["mon-mgr", ""]}) v1 -- 0x7f137c0ccae0 con 0x7f135406be90 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.261+0000 7f1379600640 1 -- 172.21.15.43:0/3830405460 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f137c0ccae0 con 0x7f135406be90 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:12:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:21.239 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 msgr2=0x7f135406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 0x7f135406e350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f136c009970 tx=0x7f136c008040 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 msgr2=0x7f137c1599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1364009890 tx=0x7f13640046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 shutdown_connections 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f135406be90 0x7f135406e350 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f137c158f40 0x7f137c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f137c0cec70 0x7f137c1599d0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 --2- 172.21.15.43:0/3830405460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f137c0cb9d0 0x7f137c0cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 >> 172.21.15.43:0/3830405460 conn(0x7f137c0cdb20 msgr2=0x7f137c070770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 shutdown_connections 2024-01-16T15:12:21.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.264+0000 7f1384000640 1 -- 172.21.15.43:0/3830405460 wait complete. 2024-01-16T15:12:21.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:12:21.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.501+0000 7f1f99800640 1 Processor -- start 2024-01-16T15:12:21.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.501+0000 7f1f99800640 1 -- start start 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.502+0000 7f1f99800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f940cf710 0x7f1f940cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.502+0000 7f1f99800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f940d1670 0x7f1f940d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.502+0000 7f1f99800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f940d04b0 con 0x7f1f9415d5d0 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.502+0000 7f1f99800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940ce4e0 con 0x7f1f940d1670 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.502+0000 7f1f99800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940cf010 con 0x7f1f940cf710 2024-01-16T15:12:21.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f940d1670 0x7f1f940d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1f940d1670 0x7f1f940d1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33180/0 (socket says 172.21.15.43:33180) 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 -- 172.21.15.43:0/4116377514 learned_addr learned my addr 172.21.15.43:0/4116377514 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f8be00640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f940cf710 0x7f1f940cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 -- 172.21.15.43:0/4116377514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f940cf710 msgr2=0x7f1f940cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f940cf710 0x7f1f940cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 -- 172.21.15.43:0/4116377514 >> v1:172.21.15.69:6789/0 conn(0x7f1f9415d5d0 legacy=0x7f1f940ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.503+0000 7f1f98e00640 1 -- 172.21.15.43:0/4116377514 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940cdb80 con 0x7f1f940d1670 2024-01-16T15:12:21.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f98e00640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940d1670 0x7f1f940d1ae0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009e50 tx=0x7f1f8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b531147cfdaeaa2e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f93400640 1 -- 172.21.15.43:0/4116377514 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c02ca70 con 0x7f1f940d1670 2024-01-16T15:12:21.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f93400640 1 -- 172.21.15.43:0/4116377514 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c02cbd0 con 0x7f1f940d1670 2024-01-16T15:12:21.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f93400640 1 -- 172.21.15.43:0/4116377514 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0343f0 con 0x7f1f940d1670 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f99800640 1 -- 172.21.15.43:0/4116377514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940d1670 msgr2=0x7f1f940d1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.505+0000 7f1f99800640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940d1670 0x7f1f940d1ae0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009e50 tx=0x7f1f8c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 -- 172.21.15.43:0/4116377514 shutdown_connections 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940d1670 0x7f1f940d1ae0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 --2- 172.21.15.43:0/4116377514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f940cf710 0x7f1f940cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 -- 172.21.15.43:0/4116377514 >> 172.21.15.43:0/4116377514 conn(0x7f1f9409c3d0 msgr2=0x7f1f940d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 -- 172.21.15.43:0/4116377514 shutdown_connections 2024-01-16T15:12:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.506+0000 7f1f99800640 1 -- 172.21.15.43:0/4116377514 wait complete. 2024-01-16T15:12:21.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.507+0000 7f1f99800640 1 Processor -- start 2024-01-16T15:12:21.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 -- start start 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f940d1670 0x7f1f940b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f9415d5d0 0x7f1f940cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b7730 con 0x7f1f940cf710 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.508+0000 7f1f99800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b72a0 con 0x7f1f940d1670 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f99800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b6690 con 0x7f1f9415d5d0 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33188/0 (socket says 172.21.15.43:33188) 2024-01-16T15:12:21.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 -- 172.21.15.43:0/3693540053 learned_addr learned my addr 172.21.15.43:0/3693540053 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f8be00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f940d1670 0x7f1f940b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 -- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f9415d5d0 msgr2=0x7f1f940cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f9415d5d0 0x7f1f940cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 -- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f940d1670 msgr2=0x7f1f940b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f940d1670 0x7f1f940b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.509+0000 7f1f93e00640 1 -- 172.21.15.43:0/3693540053 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c009b30 con 0x7f1f940cf710 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f93e00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400c450 tx=0x7f1f8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f840052d0 con 0x7f1f940cf710 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84005430 con 0x7f1f940cf710 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84012870 con 0x7f1f940cf710 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f940b6970 con 0x7f1f940cf710 2024-01-16T15:12:21.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.510+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f940b6f30 con 0x7f1f940cf710 2024-01-16T15:12:21.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.511+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9406cc60 con 0x7f1f940cf710 2024-01-16T15:12:21.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.512+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1f840129d0 con 0x7f1f940cf710 2024-01-16T15:12:21.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.513+0000 7f1f90c00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 0x7f1f6c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.513+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1f84088310 con 0x7f1f940cf710 2024-01-16T15:12:21.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.513+0000 7f1f8be00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 0x7f1f6c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.514+0000 7f1f8be00640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 0x7f1f6c06e350 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c006fd0 tx=0x7f1f7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.517+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f84054970 con 0x7f1f940cf710 2024-01-16T15:12:21.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.724+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f940cde20 con 0x7f1f6c06be90 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.726+0000 7f1f90c00640 1 -- 172.21.15.43:0/3693540053 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1f7c00e4c0 con 0x7f1f6c06be90 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:21.702 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 msgr2=0x7f1f6c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 0x7f1f6c06e350 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c006fd0 tx=0x7f1f7c008040 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 msgr2=0x7f1f9415ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400c450 tx=0x7f1f8400c910 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 shutdown_connections 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1f6c06be90 0x7f1f6c06e350 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f940d1670 0x7f1f940b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f9415d5d0 0x7f1f940cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 --2- 172.21.15.43:0/3693540053 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f940cf710 0x7f1f9415ba70 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.728+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 >> 172.21.15.43:0/3693540053 conn(0x7f1f9409c3d0 msgr2=0x7f1f940b5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.729+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 shutdown_connections 2024-01-16T15:12:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.729+0000 7f1f99800640 1 -- 172.21.15.43:0/3693540053 wait complete. 2024-01-16T15:12:21.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:12:21.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.969+0000 7f9ad3400640 1 Processor -- start 2024-01-16T15:12:21.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.969+0000 7f9ad3400640 1 -- start start 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9acc06d9e0 0x7f9acc165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad3400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc15f930 con 0x7f9acc15f270 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0b3b10 con 0x7f9acc144770 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9acc0b43b0 con 0x7f9acc06d9e0 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad1600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9acc15f270 0x7f9acc0cd4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51376/0 (socket says 172.21.15.43:51376) 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.971+0000 7f9ad2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9acc06d9e0 0x7f9acc165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad1600640 1 -- 172.21.15.43:0/145583503 learned_addr learned my addr 172.21.15.43:0/145583503 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad0c00640 1 -- 172.21.15.43:0/145583503 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131488082 0 0) 0x7f9acc15f930 con 0x7f9acc15f270 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad0c00640 1 -- 172.21.15.43:0/145583503 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac003520 con 0x7f9acc15f270 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad2a00640 1 -- 172.21.15.43:0/145583503 >> v1:172.21.15.69:6789/0 conn(0x7f9acc144770 legacy=0x7f9acc15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad2a00640 1 -- 172.21.15.43:0/145583503 >> v1:172.21.15.43:6789/0 conn(0x7f9acc15f270 legacy=0x7f9acc0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.972+0000 7f9ad2a00640 1 -- 172.21.15.43:0/145583503 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0b3df0 con 0x7f9acc06d9e0 2024-01-16T15:12:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.973+0000 7f9ad2a00640 1 --2- 172.21.15.43:0/145583503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc165ef0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009d90 tx=0x7f9ac402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09ef119635586d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad0c00640 1 -- 172.21.15.43:0/145583503 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac402ccf0 con 0x7f9acc06d9e0 2024-01-16T15:12:21.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad0c00640 1 -- 172.21.15.43:0/145583503 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac402ce50 con 0x7f9acc06d9e0 2024-01-16T15:12:21.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad0c00640 1 -- 172.21.15.43:0/145583503 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4034660 con 0x7f9acc06d9e0 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 -- 172.21.15.43:0/145583503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 msgr2=0x7f9acc165ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 --2- 172.21.15.43:0/145583503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc165ef0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009d90 tx=0x7f9ac402bf40 comp rx=0 tx=0).stop 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 -- 172.21.15.43:0/145583503 shutdown_connections 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 --2- 172.21.15.43:0/145583503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc165ef0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 -- 172.21.15.43:0/145583503 >> 172.21.15.43:0/145583503 conn(0x7f9acc0cec10 msgr2=0x7f9acc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 -- 172.21.15.43:0/145583503 shutdown_connections 2024-01-16T15:12:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.974+0000 7f9ad3400640 1 -- 172.21.15.43:0/145583503 wait complete. 2024-01-16T15:12:21.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.976+0000 7f9ad3400640 1 Processor -- start 2024-01-16T15:12:21.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.976+0000 7f9ad3400640 1 -- start start 2024-01-16T15:12:21.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.977+0000 7f9ad3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.977+0000 7f9ad3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc144770 0x7f9acc15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.977+0000 7f9ad3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc15f270 0x7f9acc071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.977+0000 7f9ad3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc0ae110 con 0x7f9acc06d9e0 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.977+0000 7f9ad3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc0ad280 con 0x7f9acc15f270 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc071410 con 0x7f9acc144770 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33220/0 (socket says 172.21.15.43:33220) 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 -- 172.21.15.43:0/385228379 learned_addr learned my addr 172.21.15.43:0/385228379 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2a00640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc15f270 0x7f9acc071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 -- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc144770 msgr2=0x7f9acc15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc144770 0x7f9acc15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 -- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc15f270 msgr2=0x7f9acc071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc15f270 0x7f9acc071790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 -- 172.21.15.43:0/385228379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4009a70 con 0x7f9acc06d9e0 2024-01-16T15:12:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.978+0000 7f9ad2000640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c450 tx=0x7f9abc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.979+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0052d0 con 0x7f9acc06d9e0 2024-01-16T15:12:21.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.979+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc005430 con 0x7f9acc06d9e0 2024-01-16T15:12:21.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.979+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc012870 con 0x7f9acc06d9e0 2024-01-16T15:12:21.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.979+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc15aa00 con 0x7f9acc06d9e0 2024-01-16T15:12:21.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.979+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc0ad770 con 0x7f9acc06d9e0 2024-01-16T15:12:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.980+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc13c820 con 0x7f9acc06d9e0 2024-01-16T15:12:21.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.981+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9abc0129d0 con 0x7f9acc06d9e0 2024-01-16T15:12:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.982+0000 7f9aca000640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 0x7f9aa406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.982+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9abc088340 con 0x7f9acc06d9e0 2024-01-16T15:12:21.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.982+0000 7f9ad1600640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 0x7f9aa406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:21.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.983+0000 7f9ad1600640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 0x7f9aa406e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9acc1f6910 tx=0x7f9ab4007950 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:21.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:21.986+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9abc0549a0 con 0x7f9acc06d9e0 2024-01-16T15:12:22.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.195+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9acc144b50 con 0x7f9aa406be90 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.234+0000 7f9aca000640 1 -- 172.21.15.43:0/385228379 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1009 (secure 0 0 0) 0x7f9ab40058b0 con 0x7f9aa406be90 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96115" 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96120" 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9616" 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96111" 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96116" 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96113" 2024-01-16T15:12:22.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96114" 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96112" 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9612" 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9619" 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9614" 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9618" 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9615" 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9617" 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96118" 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96117" 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9613" 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone9611" 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96110" 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone96119" 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:22.215 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 msgr2=0x7f9aa406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 0x7f9aa406e350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9acc1f6910 tx=0x7f9ab4007950 comp rx=0 tx=0).stop 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 msgr2=0x7f9acc15a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c450 tx=0x7f9abc00c910 comp rx=0 tx=0).stop 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 shutdown_connections 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9aa406be90 0x7f9aa406e350 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9acc15f270 0x7f9acc071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9acc144770 0x7f9acc15d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 --2- 172.21.15.43:0/385228379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9acc06d9e0 0x7f9acc15a4c0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 >> 172.21.15.43:0/385228379 conn(0x7f9acc0cec10 msgr2=0x7f9acc06c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 shutdown_connections 2024-01-16T15:12:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.237+0000 7f9ad3400640 1 -- 172.21.15.43:0/385228379 wait complete. 2024-01-16T15:12:22.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96115 2024-01-16T15:12:22.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.480+0000 7fd34f200640 1 Processor -- start 2024-01-16T15:12:22.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.480+0000 7fd34f200640 1 -- start start 2024-01-16T15:12:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd34f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd3480abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd34f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd3480adb60 0x7fd3480adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd34f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34815b730 con 0x7fd34815b350 2024-01-16T15:12:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd34f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480ac740 con 0x7fd3480adb60 2024-01-16T15:12:22.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd34f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480ac460 con 0x7fd34809cb30 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd347e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd34815b350 0x7fd3480b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51380/0 (socket says 172.21.15.43:51380) 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.482+0000 7fd347e00640 1 -- 172.21.15.43:0/2306974199 learned_addr learned my addr 172.21.15.43:0/2306974199 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd34d200640 1 --2- 172.21.15.43:0/2306974199 >> v2:172.21.15.43:3300/0 conn(0x7fd3480adb60 0x7fd3480adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd33fe00640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd3480abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd347400640 1 -- 172.21.15.43:0/2306974199 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080175494 0 0) 0x7fd338001110 con 0x7fd34815b350 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd347400640 1 -- 172.21.15.43:0/2306974199 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328003520 con 0x7fd34815b350 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd34d200640 1 -- 172.21.15.43:0/2306974199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 msgr2=0x7fd3480abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd34d200640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd3480abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd34d200640 1 -- 172.21.15.43:0/2306974199 >> v1:172.21.15.43:6789/0 conn(0x7fd34815b350 legacy=0x7fd3480b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.483+0000 7fd34d200640 1 -- 172.21.15.43:0/2306974199 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480b2a50 con 0x7fd3480adb60 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.484+0000 7fd34d200640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3480adb60 0x7fd3480adfb0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd340009d90 tx=0x7fd34002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829481f7bd718987 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd347400640 1 -- 172.21.15.43:0/2306974199 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34002cc60 con 0x7fd3480adb60 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd347400640 1 -- 172.21.15.43:0/2306974199 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34002cdc0 con 0x7fd3480adb60 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd347400640 1 -- 172.21.15.43:0/2306974199 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340034660 con 0x7fd3480adb60 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 -- 172.21.15.43:0/2306974199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3480adb60 msgr2=0x7fd3480adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3480adb60 0x7fd3480adfb0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd340009d90 tx=0x7fd34002beb0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 -- 172.21.15.43:0/2306974199 shutdown_connections 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3480adb60 0x7fd3480adfb0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 --2- 172.21.15.43:0/2306974199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd3480abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.485+0000 7fd34f200640 1 -- 172.21.15.43:0/2306974199 >> 172.21.15.43:0/2306974199 conn(0x7fd3480b0ba0 msgr2=0x7fd3480b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.486+0000 7fd34f200640 1 -- 172.21.15.43:0/2306974199 shutdown_connections 2024-01-16T15:12:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.486+0000 7fd34f200640 1 -- 172.21.15.43:0/2306974199 wait complete. 2024-01-16T15:12:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.487+0000 7fd34f200640 1 Processor -- start 2024-01-16T15:12:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.487+0000 7fd34f200640 1 -- start start 2024-01-16T15:12:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd34813e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480adb60 0x7fd348140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348159370 con 0x7fd34815b350 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348152f70 con 0x7fd3480adb60 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.488+0000 7fd34f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd348159090 con 0x7fd34809cb30 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33240/0 (socket says 172.21.15.43:33240) 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 -- 172.21.15.43:0/4127339691 learned_addr learned my addr 172.21.15.43:0/4127339691 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd33fe00640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480adb60 0x7fd348140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd347e00640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd34813e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 -- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 msgr2=0x7fd34813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd34813e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 -- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480adb60 msgr2=0x7fd348140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480adb60 0x7fd348140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.489+0000 7fd34d200640 1 -- 172.21.15.43:0/4127339691 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009a70 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd34d200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd340009d90 tx=0x7fd340004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340040650 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3400407b0 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34813ee40 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340040b60 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.490+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34813f400 con 0x7fd34815b350 2024-01-16T15:12:22.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.491+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34813eca0 con 0x7fd34815b350 2024-01-16T15:12:22.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.492+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd340040cc0 con 0x7fd34815b350 2024-01-16T15:12:22.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.493+0000 7fd344c00640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 0x7fd32006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.493+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd3400ab2b0 con 0x7fd34815b350 2024-01-16T15:12:22.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.494+0000 7fd347e00640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 0x7fd32006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.494+0000 7fd347e00640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 0x7fd32006e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd338001370 tx=0x7fd338009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.498+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd34813eca0 con 0x7fd34815b350 2024-01-16T15:12:22.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.720+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96115", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3480ae4f0 con 0x7fd32006be90 2024-01-16T15:12:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.772+0000 7fd344c00640 1 -- 172.21.15.43:0/4127339691 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd338001c50 con 0x7fd32006be90 2024-01-16T15:12:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 msgr2=0x7fd32006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 0x7fd32006e350 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd338001370 tx=0x7fd338009040 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 msgr2=0x7fd348141230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd340009d90 tx=0x7fd340004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 shutdown_connections 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd32006be90 0x7fd32006e350 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3480adb60 0x7fd348140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd34815b350 0x7fd348141230 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.775+0000 7fd34f200640 1 --2- 172.21.15.43:0/4127339691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd34809cb30 0x7fd34813e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.776+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 >> 172.21.15.43:0/4127339691 conn(0x7fd3480b0ba0 msgr2=0x7fd34809ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.776+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 shutdown_connections 2024-01-16T15:12:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.776+0000 7fd34f200640 1 -- 172.21.15.43:0/4127339691 wait complete. 2024-01-16T15:12:22.756 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.780+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.756 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.780+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.756 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.780+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.756 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.780+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.756 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.780+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96120 2024-01-16T15:12:22.788 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.812+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.788 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.812+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.788 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.812+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.788 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.812+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.789 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:12:22.812+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:22.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.994+0000 7fb02ee00640 1 Processor -- start 2024-01-16T15:12:22.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.994+0000 7fb02ee00640 1 -- start start 2024-01-16T15:12:22.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02ee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0280d1fd0 0x7fb0280ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02ee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0280b2980 con 0x7fb0280cfa30 2024-01-16T15:12:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02ee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0280cf220 con 0x7fb0280b33d0 2024-01-16T15:12:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02ee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0280cf500 con 0x7fb0280d1fd0 2024-01-16T15:12:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02da00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb0280cfa30 0x7fb0280cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37520/0 (socket says 172.21.15.43:37520) 2024-01-16T15:12:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.996+0000 7fb02da00640 1 -- 172.21.15.43:0/2468428282 learned_addr learned my addr 172.21.15.43:0/2468428282 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb02d000640 1 --2- 172.21.15.43:0/2468428282 >> v2:172.21.15.69:3300/0 conn(0x7fb0280d1fd0 0x7fb0280ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254634184 0 0) 0x7fb0280b2980 con 0x7fb0280cfa30 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008003520 con 0x7fb0280cfa30 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133887751 0 0) 0x7fb008003520 con 0x7fb0280cfa30 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 >> v2:172.21.15.69:3300/0 conn(0x7fb0280d1fd0 msgr2=0x7fb0280ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 --2- 172.21.15.43:0/2468428282 >> v2:172.21.15.69:3300/0 conn(0x7fb0280d1fd0 0x7fb0280ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 >> v1:172.21.15.69:6789/0 conn(0x7fb0280b33d0 legacy=0x7fb0280b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0280b2bd0 con 0x7fb0280cfa30 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb018001670 con 0x7fb0280cfa30 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 >> v1:172.21.15.43:6789/0 conn(0x7fb0280cfa30 legacy=0x7fb0280cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0080039e0 0x7fb008003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0080047c0 0x7fb008004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 0x7fb00800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280cf500 con 0x7fb008005180 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00800c0b0 con 0x7fb0080039e0 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.997+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00800c270 con 0x7fb0080047c0 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 0x7fb00800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02d000640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0080047c0 0x7fb008004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02da00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0080039e0 0x7fb008003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 -- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0080047c0 msgr2=0x7fb008004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0080047c0 0x7fb008004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 -- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0080039e0 msgr2=0x7fb008003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0080039e0 0x7fb008003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.998+0000 7fb02e400640 1 -- 172.21.15.43:0/2468428282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008003520 con 0x7fb008005180 2024-01-16T15:12:22.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.999+0000 7fb02e400640 1 -- 172.21.15.43:0/2468428282 reap_dead start 2024-01-16T15:12:22.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:22.999+0000 7fb02e400640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 0x7fb00800b980 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb020001e50 tx=0x7fb0080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61998240c4085480 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020003b70 con 0x7fb008005180 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb020003d10 con 0x7fb008005180 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb027e00640 1 -- 172.21.15.43:0/2468428282 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020005310 con 0x7fb008005180 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 -- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 msgr2=0x7fb00800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 0x7fb00800b980 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb020001e50 tx=0x7fb0080057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 -- 172.21.15.43:0/2468428282 shutdown_connections 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 --2- 172.21.15.43:0/2468428282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb008005180 0x7fb00800b980 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 -- 172.21.15.43:0/2468428282 >> 172.21.15.43:0/2468428282 conn(0x7fb0280d3020 msgr2=0x7fb0280d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 -- 172.21.15.43:0/2468428282 shutdown_connections 2024-01-16T15:12:22.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.000+0000 7fb02ee00640 1 -- 172.21.15.43:0/2468428282 wait complete. 2024-01-16T15:12:22.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.002+0000 7fb02ee00640 1 Processor -- start 2024-01-16T15:12:22.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.002+0000 7fb02ee00640 1 -- start start 2024-01-16T15:12:22.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008005180 0x7fb0280b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb02815f640 0x7fb0281f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280b7440 con 0x7fb02806bd30 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02806c6f0 con 0x7fb008005180 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.003+0000 7fb02ee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280b8260 con 0x7fb02815f640 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53804/0 (socket says 172.21.15.43:53804) 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 -- 172.21.15.43:0/3630895315 learned_addr learned my addr 172.21.15.43:0/3630895315 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02da00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008005180 0x7fb0280b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02d000640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb02815f640 0x7fb0281f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 -- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb02815f640 msgr2=0x7fb0281f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb02815f640 0x7fb0281f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 -- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008005180 msgr2=0x7fb0280b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008005180 0x7fb0280b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 -- 172.21.15.43:0/3630895315 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0280b7650 con 0x7fb02806bd30 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.004+0000 7fb02e400640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb020001f70 tx=0x7fb0200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.005+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020039680 con 0x7fb02806bd30 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.005+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0200397e0 con 0x7fb02806bd30 2024-01-16T15:12:22.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.005+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020039b50 con 0x7fb02806bd30 2024-01-16T15:12:22.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.005+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0280b7930 con 0x7fb02806bd30 2024-01-16T15:12:22.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.005+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0280b7ef0 con 0x7fb02806bd30 2024-01-16T15:12:22.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.006+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0280d0a40 con 0x7fb02806bd30 2024-01-16T15:12:22.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.007+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb020039cb0 con 0x7fb02806bd30 2024-01-16T15:12:22.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.008+0000 7fb025600640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 0x7fb00006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.008+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb0200aa4a0 con 0x7fb02806bd30 2024-01-16T15:12:22.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.008+0000 7fb02da00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 0x7fb00006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.009+0000 7fb02da00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 0x7fb00006e380 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb018001050 tx=0x7fb018009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.012+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb020076b00 con 0x7fb02806bd30 2024-01-16T15:12:23.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.235+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96120", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0280b7440 con 0x7fb00006bec0 2024-01-16T15:12:23.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.276+0000 7fb025600640 1 -- 172.21.15.43:0/3630895315 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb0280b7440 con 0x7fb00006bec0 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 msgr2=0x7fb00006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 0x7fb00006e380 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb018001050 tx=0x7fb018009040 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 msgr2=0x7fb02806c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb020001f70 tx=0x7fb0200014a0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 shutdown_connections 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb00006bec0 0x7fb00006e380 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb02806bd30 0x7fb02806c1b0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb02815f640 0x7fb0281f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 --2- 172.21.15.43:0/3630895315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008005180 0x7fb0280b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.279+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 >> 172.21.15.43:0/3630895315 conn(0x7fb0280d3020 msgr2=0x7fb02815b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.280+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 shutdown_connections 2024-01-16T15:12:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.280+0000 7fb02ee00640 1 -- 172.21.15.43:0/3630895315 wait complete. 2024-01-16T15:12:23.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9616 2024-01-16T15:12:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.573+0000 7f3c19e00640 1 Processor -- start 2024-01-16T15:12:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.573+0000 7f3c19e00640 1 -- start start 2024-01-16T15:12:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c19e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c14144770 0x7f3c1415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c19e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3c1415f0a0 0x7f3c140ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c19e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1415f930 con 0x7f3c140b35a0 2024-01-16T15:12:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c19e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c140b43b0 con 0x7f3c14144770 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c19e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1415f510 con 0x7f3c1415f0a0 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c18a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c14144770 0x7f3c1415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c18a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3c14144770 0x7f3c1415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53810/0 (socket says 172.21.15.43:53810) 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.575+0000 7f3c18a00640 1 -- 172.21.15.43:0/257763469 learned_addr learned my addr 172.21.15.43:0/257763469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.576+0000 7f3c19400640 1 --2- 172.21.15.43:0/257763469 >> v2:172.21.15.69:3300/0 conn(0x7f3c1415f0a0 0x7f3c140ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.576+0000 7f3c18a00640 1 -- 172.21.15.43:0/257763469 >> v2:172.21.15.69:3300/0 conn(0x7f3c1415f0a0 msgr2=0x7f3c140ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.576+0000 7f3c18a00640 1 --2- 172.21.15.43:0/257763469 >> v2:172.21.15.69:3300/0 conn(0x7f3c1415f0a0 0x7f3c140ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.576+0000 7f3c18a00640 1 -- 172.21.15.43:0/257763469 >> v1:172.21.15.69:6789/0 conn(0x7f3c140b35a0 legacy=0x7f3c1406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.576+0000 7f3c18a00640 1 -- 172.21.15.43:0/257763469 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c140ce9f0 con 0x7f3c14144770 2024-01-16T15:12:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.577+0000 7f3c18a00640 1 --2- 172.21.15.43:0/257763469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c1415eb60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3c140700e0 tx=0x7f3c0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903ed91b5b4633f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.577+0000 7f3c13e00640 1 -- 172.21.15.43:0/257763469 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04031870 con 0x7f3c14144770 2024-01-16T15:12:23.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.577+0000 7f3c13e00640 1 -- 172.21.15.43:0/257763469 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c040319d0 con 0x7f3c14144770 2024-01-16T15:12:23.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.577+0000 7f3c13e00640 1 -- 172.21.15.43:0/257763469 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04031d60 con 0x7f3c14144770 2024-01-16T15:12:23.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 -- 172.21.15.43:0/257763469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 msgr2=0x7f3c1415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 --2- 172.21.15.43:0/257763469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c1415eb60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3c140700e0 tx=0x7f3c0402b910 comp rx=0 tx=0).stop 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 -- 172.21.15.43:0/257763469 shutdown_connections 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 --2- 172.21.15.43:0/257763469 >> v2:172.21.15.69:3300/0 conn(0x7f3c1415f0a0 0x7f3c140ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 --2- 172.21.15.43:0/257763469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c1415eb60 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.578+0000 7f3c19e00640 1 -- 172.21.15.43:0/257763469 >> 172.21.15.43:0/257763469 conn(0x7f3c140cec10 msgr2=0x7f3c140cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.579+0000 7f3c19e00640 1 -- 172.21.15.43:0/257763469 shutdown_connections 2024-01-16T15:12:23.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.579+0000 7f3c19e00640 1 -- 172.21.15.43:0/257763469 wait complete. 2024-01-16T15:12:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.581+0000 7f3c19e00640 1 Processor -- start 2024-01-16T15:12:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.581+0000 7f3c19e00640 1 -- start start 2024-01-16T15:12:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 0x7f3c14156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c1415f0a0 0x7f3c14141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c14159380 con 0x7f3c14144770 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1413fb80 con 0x7f3c140b35a0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c14156e40 con 0x7f3c1415f0a0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53818/0 (socket says 172.21.15.43:53818) 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 -- 172.21.15.43:0/2622045916 learned_addr learned my addr 172.21.15.43:0/2622045916 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c19400640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c1415f0a0 0x7f3c14141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c18a00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 0x7f3c14156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 -- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c1415f0a0 msgr2=0x7f3c14141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c1415f0a0 0x7f3c14141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 -- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 msgr2=0x7f3c14156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 0x7f3c14156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.582+0000 7f3c0be00640 1 -- 172.21.15.43:0/2622045916 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040094d0 con 0x7f3c14144770 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c18a00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 0x7f3c14156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c0be00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00c430 tx=0x7f3bfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc011790 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0118f0 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc011ce0 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c14158dc0 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.583+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1413f810 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.584+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1413f670 con 0x7f3c14144770 2024-01-16T15:12:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.585+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3bfc01c460 con 0x7f3c14144770 2024-01-16T15:12:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.586+0000 7f3c11600640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 0x7f3bec06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.586+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3bfc088750 con 0x7f3c14144770 2024-01-16T15:12:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.587+0000 7f3c18a00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 0x7f3bec06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.587+0000 7f3c18a00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 0x7f3bec06e380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:23.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.591+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c1413f670 con 0x7f3c14144770 2024-01-16T15:12:23.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.816+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9616", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c140b40d0 con 0x7f3bec06bec0 2024-01-16T15:12:23.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.856+0000 7f3c11600640 1 -- 172.21.15.43:0/2622045916 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c04031c40 con 0x7f3bec06bec0 2024-01-16T15:12:23.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.858+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 msgr2=0x7f3bec06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.858+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 0x7f3bec06e380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403c040 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.858+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 msgr2=0x7f3c14158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.858+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00c430 tx=0x7f3bfc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 shutdown_connections 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3bec06bec0 0x7f3bec06e380 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c1415f0a0 0x7f3c14141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c14144770 0x7f3c14158300 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 --2- 172.21.15.43:0/2622045916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c140b35a0 0x7f3c14156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 >> 172.21.15.43:0/2622045916 conn(0x7f3c140cec10 msgr2=0x7f3c140b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 shutdown_connections 2024-01-16T15:12:23.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:23.859+0000 7f3c19e00640 1 -- 172.21.15.43:0/2622045916 wait complete. 2024-01-16T15:12:23.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96111 2024-01-16T15:12:24.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.114+0000 7ff43fc00640 1 Processor -- start 2024-01-16T15:12:24.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.114+0000 7ff43fc00640 1 -- start start 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43fc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff438162600 0x7ff438162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43fc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438162fa0 con 0x7ff43806c3c0 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43fc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4380706a0 con 0x7ff438162220 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43fc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff438070c60 con 0x7ff438162600 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43dc00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff43806c3c0 0x7ff4380710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37526/0 (socket says 172.21.15.43:37526) 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff43dc00640 1 -- 172.21.15.43:0/1317680995 learned_addr learned my addr 172.21.15.43:0/1317680995 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.116+0000 7ff437e00640 1 --2- 172.21.15.43:0/1317680995 >> v2:172.21.15.43:3300/0 conn(0x7ff438162600 0x7ff438162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.117+0000 7ff437e00640 1 -- 172.21.15.43:0/1317680995 >> v1:172.21.15.69:6789/0 conn(0x7ff438162220 legacy=0x7ff4380b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:24.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.117+0000 7ff437e00640 1 -- 172.21.15.43:0/1317680995 >> v1:172.21.15.43:6789/0 conn(0x7ff43806c3c0 legacy=0x7ff4380710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.117+0000 7ff437e00640 1 -- 172.21.15.43:0/1317680995 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380722b0 con 0x7ff438162600 2024-01-16T15:12:24.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.117+0000 7ff437400640 1 -- 172.21.15.43:0/1317680995 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954441656 0 0) 0x7ff438162fa0 con 0x7ff43806c3c0 2024-01-16T15:12:24.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.118+0000 7ff437e00640 1 --2- 172.21.15.43:0/1317680995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162600 0x7ff438162a60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff4200097a0 tx=0x7ff42002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42d179728206d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.118+0000 7ff437400640 1 -- 172.21.15.43:0/1317680995 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420030610 con 0x7ff438162600 2024-01-16T15:12:24.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.118+0000 7ff437400640 1 -- 172.21.15.43:0/1317680995 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420030770 con 0x7ff438162600 2024-01-16T15:12:24.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.118+0000 7ff437400640 1 -- 172.21.15.43:0/1317680995 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420030b00 con 0x7ff438162600 2024-01-16T15:12:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 -- 172.21.15.43:0/1317680995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162600 msgr2=0x7ff438162a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 --2- 172.21.15.43:0/1317680995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162600 0x7ff438162a60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff4200097a0 tx=0x7ff42002b960 comp rx=0 tx=0).stop 2024-01-16T15:12:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 -- 172.21.15.43:0/1317680995 shutdown_connections 2024-01-16T15:12:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 --2- 172.21.15.43:0/1317680995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162600 0x7ff438162a60 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 -- 172.21.15.43:0/1317680995 >> 172.21.15.43:0/1317680995 conn(0x7ff4380b0ba0 msgr2=0x7ff4380b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:24.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 -- 172.21.15.43:0/1317680995 shutdown_connections 2024-01-16T15:12:24.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.119+0000 7ff43fc00640 1 -- 172.21.15.43:0/1317680995 wait complete. 2024-01-16T15:12:24.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.121+0000 7ff43fc00640 1 Processor -- start 2024-01-16T15:12:24.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.121+0000 7ff43fc00640 1 -- start start 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 0x7ff438149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438162600 0x7ff4380b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4381662d0 con 0x7ff438162220 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4381578c0 con 0x7ff43806c3c0 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43fc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43814a420 con 0x7ff438162600 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff437e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff437e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53834/0 (socket says 172.21.15.43:53834) 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff437e00640 1 -- 172.21.15.43:0/4245143674 learned_addr learned my addr 172.21.15.43:0/4245143674 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43dc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438162600 0x7ff4380b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff43d200640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 0x7ff438149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.122+0000 7ff437e00640 1 -- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438162600 msgr2=0x7ff4380b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff437e00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438162600 0x7ff4380b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff437e00640 1 -- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 msgr2=0x7ff438149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff437e00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 0x7ff438149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff437e00640 1 -- 172.21.15.43:0/4245143674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420009400 con 0x7ff438162220 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff43d200640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 0x7ff438149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff437e00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff42002bf70 tx=0x7ff420004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42003e040 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420039cb0 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420044630 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.123+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff438163550 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.124+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff438165750 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.124+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43809d200 con 0x7ff438162220 2024-01-16T15:12:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.125+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff420043390 con 0x7ff438162220 2024-01-16T15:12:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.126+0000 7ff434c00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 0x7ff41006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.126+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4200aa950 con 0x7ff438162220 2024-01-16T15:12:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.127+0000 7ff43d200640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 0x7ff41006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.127+0000 7ff43d200640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 0x7ff41006e470 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff438160c90 tx=0x7ff428009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.131+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff420077270 con 0x7ff438162220 2024-01-16T15:12:24.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.362+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96111", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4380708f0 con 0x7ff41006bfb0 2024-01-16T15:12:24.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.396+0000 7ff434c00640 1 -- 172.21.15.43:0/4245143674 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff428001340 con 0x7ff41006bfb0 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 msgr2=0x7ff41006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 0x7ff41006e470 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff438160c90 tx=0x7ff428009040 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 msgr2=0x7ff438146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff42002bf70 tx=0x7ff420004730 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 shutdown_connections 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff41006bfb0 0x7ff41006e470 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff438162600 0x7ff4380b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff438162220 0x7ff438146bd0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 --2- 172.21.15.43:0/4245143674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff43806c3c0 0x7ff438149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 >> 172.21.15.43:0/4245143674 conn(0x7ff4380b0ba0 msgr2=0x7ff438161030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 shutdown_connections 2024-01-16T15:12:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.400+0000 7ff43fc00640 1 -- 172.21.15.43:0/4245143674 wait complete. 2024-01-16T15:12:24.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:12:24.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.653+0000 7f5d46c00640 1 Processor -- start 2024-01-16T15:12:24.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.653+0000 7f5d46c00640 1 -- start start 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.654+0000 7f5d46c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d4006e550 0x7f5d400706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.654+0000 7f5d46c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d400b2860 con 0x7f5d4006ce00 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.654+0000 7f5d46c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d400b2b40 con 0x7f5d40070be0 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.654+0000 7f5d46c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d40071090 con 0x7f5d4006e550 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.655+0000 7f5d44c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5d4006ce00 0x7f5d40164030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37528/0 (socket says 172.21.15.43:37528) 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.655+0000 7f5d44c00640 1 -- 172.21.15.43:0/4049409121 learned_addr learned my addr 172.21.15.43:0/4049409121 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:24.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.655+0000 7f5d3fe00640 1 --2- 172.21.15.43:0/4049409121 >> v2:172.21.15.69:3300/0 conn(0x7f5d4006e550 0x7f5d400706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.655+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416264648 0 0) 0x7f5d400b2860 con 0x7f5d4006ce00 2024-01-16T15:12:24.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.655+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d20003520 con 0x7f5d4006ce00 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425906432 0 0) 0x7f5d20003520 con 0x7f5d4006ce00 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 >> v2:172.21.15.69:3300/0 conn(0x7f5d4006e550 msgr2=0x7f5d400706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 --2- 172.21.15.43:0/4049409121 >> v2:172.21.15.69:3300/0 conn(0x7f5d4006e550 0x7f5d400706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 >> v1:172.21.15.69:6789/0 conn(0x7f5d40070be0 legacy=0x7f5d4006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d400b2e20 con 0x7f5d4006ce00 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f5d380019f0 con 0x7f5d4006ce00 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 >> v1:172.21.15.43:6789/0 conn(0x7f5d4006ce00 legacy=0x7f5d40164030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d20003b40 0x7f5d20003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d2000ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2000b5d0 0x7f5d2000da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40071090 con 0x7f5d200088c0 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2000e370 con 0x7f5d20003b40 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2000e530 con 0x7f5d2000b5d0 2024-01-16T15:12:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d37e00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d2000ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.656+0000 7f5d3fe00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d20003b40 0x7f5d20003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d44c00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2000b5d0 0x7f5d2000da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d37e00640 1 -- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2000b5d0 msgr2=0x7f5d2000da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d37e00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2000b5d0 0x7f5d2000da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d37e00640 1 -- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d20003b40 msgr2=0x7f5d20003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d37e00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d20003b40 0x7f5d20003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d37e00640 1 -- 172.21.15.43:0/4049409121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20003520 con 0x7f5d200088c0 2024-01-16T15:12:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.657+0000 7f5d44c00640 1 -- 172.21.15.43:0/4049409121 reap_dead start 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.658+0000 7f5d37e00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d2000ad30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f5d28001750 tx=0x7f5d200037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4323814f1370194d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.658+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d280052d0 con 0x7f5d200088c0 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.658+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2802e050 con 0x7f5d200088c0 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.658+0000 7f5d3f400640 1 -- 172.21.15.43:0/4049409121 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28003830 con 0x7f5d200088c0 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 -- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 msgr2=0x7f5d2000ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d2000ad30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f5d28001750 tx=0x7f5d200037d0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 -- 172.21.15.43:0/4049409121 shutdown_connections 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 --2- 172.21.15.43:0/4049409121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d2000ad30 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 -- 172.21.15.43:0/4049409121 >> 172.21.15.43:0/4049409121 conn(0x7f5d400b0ba0 msgr2=0x7f5d400b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 -- 172.21.15.43:0/4049409121 shutdown_connections 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.659+0000 7f5d46c00640 1 -- 172.21.15.43:0/4049409121 wait complete. 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.661+0000 7f5d46c00640 1 Processor -- start 2024-01-16T15:12:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.661+0000 7f5d46c00640 1 -- start start 2024-01-16T15:12:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d401500f0 0x7f5d4006d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4006daf0 0x7f5d400adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4006df70 con 0x7f5d200088c0 2024-01-16T15:12:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40151150 con 0x7f5d401500f0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d46c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d400abf90 con 0x7f5d4006daf0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53846/0 (socket says 172.21.15.43:53846) 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.662+0000 7f5d3fe00640 1 -- 172.21.15.43:0/608755865 learned_addr learned my addr 172.21.15.43:0/608755865 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d37e00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d401500f0 0x7f5d4006d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d44c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4006daf0 0x7f5d400adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 -- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4006daf0 msgr2=0x7f5d400adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4006daf0 0x7f5d400adcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 -- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d401500f0 msgr2=0x7f5d4006d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d401500f0 0x7f5d4006d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 -- 172.21.15.43:0/608755865 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d400b3a80 con 0x7f5d200088c0 2024-01-16T15:12:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3fe00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000b750 tx=0x7f5d3000bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30010830 con 0x7f5d200088c0 2024-01-16T15:12:24.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d30010990 con 0x7f5d200088c0 2024-01-16T15:12:24.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30010d40 con 0x7f5d200088c0 2024-01-16T15:12:24.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.663+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d400ac270 con 0x7f5d200088c0 2024-01-16T15:12:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.664+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d400ac830 con 0x7f5d200088c0 2024-01-16T15:12:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.665+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4009d200 con 0x7f5d200088c0 2024-01-16T15:12:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.666+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f5d30014430 con 0x7f5d200088c0 2024-01-16T15:12:24.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.667+0000 7f5d3cc00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 0x7f5d1806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:24.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.667+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5d30088b70 con 0x7f5d200088c0 2024-01-16T15:12:24.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.667+0000 7f5d37e00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 0x7f5d1806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:24.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.668+0000 7f5d37e00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 0x7f5d1806e430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d28033000 tx=0x7f5d28009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:24.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.671+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d30055430 con 0x7f5d200088c0 2024-01-16T15:12:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.897+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d400ac100 con 0x7f5d1806bf70 2024-01-16T15:12:24.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.928+0000 7f5d3cc00640 1 -- 172.21.15.43:0/608755865 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5d400ac100 con 0x7f5d1806bf70 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:12:24.905 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:12:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 msgr2=0x7f5d1806e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 0x7f5d1806e430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d28033000 tx=0x7f5d28009030 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 msgr2=0x7f5d400acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000b750 tx=0x7f5d3000bc10 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 shutdown_connections 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d1806bf70 0x7f5d1806e430 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d401500f0 0x7f5d4006d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4006daf0 0x7f5d400adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 --2- 172.21.15.43:0/608755865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d200088c0 0x7f5d400acd50 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 >> 172.21.15.43:0/608755865 conn(0x7f5d400b0ba0 msgr2=0x7f5d4015c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 shutdown_connections 2024-01-16T15:12:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:24.931+0000 7f5d46c00640 1 -- 172.21.15.43:0/608755865 wait complete. 2024-01-16T15:12:24.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:12:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.165+0000 7f0b3f200640 1 Processor -- start 2024-01-16T15:12:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.165+0000 7f0b3f200640 1 -- start start 2024-01-16T15:12:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.166+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b380abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.166+0000 7f0b3f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3815b730 con 0x7f0b3809cb30 2024-01-16T15:12:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b3f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b380ac740 con 0x7f0b3815b350 2024-01-16T15:12:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b3f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b380ac460 con 0x7f0b380ae270 2024-01-16T15:12:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b2fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0b3809cb30 0x7f0b380adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37542/0 (socket says 172.21.15.43:37542) 2024-01-16T15:12:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b2fe00640 1 -- 172.21.15.43:0/3105149612 learned_addr learned my addr 172.21.15.43:0/3105149612 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:25.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b3d200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b380abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.167+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815722198 0 0) 0x7f0b20001110 con 0x7f0b3809cb30 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18003520 con 0x7f0b3809cb30 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081568654 0 0) 0x7f0b28001110 con 0x7f0b3815b350 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b20001110 con 0x7f0b3815b350 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423423043 0 0) 0x7f0b18003520 con 0x7f0b3809cb30 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 msgr2=0x7f0b380abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b380abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 >> v1:172.21.15.69:6789/0 conn(0x7f0b3815b350 legacy=0x7f0b380b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b380b2a50 con 0x7f0b3809cb30 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0b20001780 con 0x7f0b3809cb30 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 >> v1:172.21.15.43:6789/0 conn(0x7f0b3809cb30 legacy=0x7f0b380adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b18003b30 0x7f0b18003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b18004d00 0x7f0b18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b380ac460 con 0x7f0b18004d00 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1800ba30 con 0x7f0b18003b30 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.168+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1800bbf0 con 0x7f0b380ae270 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b18004d00 0x7f0b18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b37e00640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b18003b30 0x7f0b18003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b2fe00640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 -- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 msgr2=0x7f0b180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 -- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b18003b30 msgr2=0x7f0b18003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b18003b30 0x7f0b18003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b3d200640 1 -- 172.21.15.43:0/3105149612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18003520 con 0x7f0b18004d00 2024-01-16T15:12:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.169+0000 7f0b2fe00640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b380ae270 0x7f0b180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.170+0000 7f0b3d200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b18004d00 0x7f0b18005180 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009fc0 tx=0x7f0b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c4f0fef04081c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.170+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3002cc20 con 0x7f0b18004d00 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3000f040 con 0x7f0b18004d00 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b37400640 1 -- 172.21.15.43:0/3105149612 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30034440 con 0x7f0b18004d00 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 -- 172.21.15.43:0/3105149612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b18004d00 msgr2=0x7f0b18005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 --2- 172.21.15.43:0/3105149612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b18004d00 0x7f0b18005180 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009fc0 tx=0x7f0b180057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3d200640 1 -- 172.21.15.43:0/3105149612 reap_dead start 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 -- 172.21.15.43:0/3105149612 shutdown_connections 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 -- 172.21.15.43:0/3105149612 >> 172.21.15.43:0/3105149612 conn(0x7f0b380b0ba0 msgr2=0x7f0b380b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 -- 172.21.15.43:0/3105149612 shutdown_connections 2024-01-16T15:12:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.171+0000 7f0b3f200640 1 -- 172.21.15.43:0/3105149612 wait complete. 2024-01-16T15:12:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.173+0000 7f0b3f200640 1 Processor -- start 2024-01-16T15:12:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.173+0000 7f0b3f200640 1 -- start start 2024-01-16T15:12:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b18004d00 0x7f0b381f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b3813dac0 0x7f0b3813b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38157c30 con 0x7f0b38157260 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3815c920 con 0x7f0b3813dac0 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.174+0000 7f0b3f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38148c30 con 0x7f0b18004d00 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53862/0 (socket says 172.21.15.43:53862) 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 -- 172.21.15.43:0/1133510103 learned_addr learned my addr 172.21.15.43:0/1133510103 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b37e00640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b18004d00 0x7f0b381f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b2fe00640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b3813dac0 0x7f0b3813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 -- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b18004d00 msgr2=0x7f0b381f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b18004d00 0x7f0b381f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 -- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b3813dac0 msgr2=0x7f0b3813b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b3813dac0 0x7f0b3813b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 -- 172.21.15.43:0/1133510103 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3813dfa0 con 0x7f0b38157260 2024-01-16T15:12:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.175+0000 7f0b3d200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f0b30002230 tx=0x7f0b30009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.176+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30034ad0 con 0x7f0b38157260 2024-01-16T15:12:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.176+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b30034c30 con 0x7f0b38157260 2024-01-16T15:12:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.176+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3813e280 con 0x7f0b38157260 2024-01-16T15:12:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.176+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30003990 con 0x7f0b38157260 2024-01-16T15:12:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.176+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3813e840 con 0x7f0b38157260 2024-01-16T15:12:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.177+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38066ed0 con 0x7f0b38157260 2024-01-16T15:12:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.178+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0b3003e440 con 0x7f0b38157260 2024-01-16T15:12:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.179+0000 7f0b34c00640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 0x7f0b1006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.179+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0b300aa640 con 0x7f0b38157260 2024-01-16T15:12:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.179+0000 7f0b37e00640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 0x7f0b1006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.180+0000 7f0b37e00640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 0x7f0b1006e470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0b28007ae0 tx=0x7f0b28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.183+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b30076c70 con 0x7f0b38157260 2024-01-16T15:12:25.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.419+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b3806e720 con 0x7f0b1006bfb0 2024-01-16T15:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.489+0000 7f0b34c00640 1 -- 172.21.15.43:0/1133510103 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0b2800d050 con 0x7f0b1006bfb0 2024-01-16T15:12:25.466 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 msgr2=0x7f0b1006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 0x7f0b1006e470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0b28007ae0 tx=0x7f0b28009040 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 msgr2=0x7f0b381576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f0b30002230 tx=0x7f0b30009750 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 shutdown_connections 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b1006bfb0 0x7f0b1006e470 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b3813dac0 0x7f0b3813b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b38157260 0x7f0b381576f0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 --2- 172.21.15.43:0/1133510103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b18004d00 0x7f0b381f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 >> 172.21.15.43:0/1133510103 conn(0x7f0b380b0ba0 msgr2=0x7f0b380b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.493+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 shutdown_connections 2024-01-16T15:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.494+0000 7f0b3f200640 1 -- 172.21.15.43:0/1133510103 wait complete. 2024-01-16T15:12:25.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:12:25.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.810+0000 7f24c1600640 1 Processor -- start 2024-01-16T15:12:25.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.811+0000 7f24c1600640 1 -- start start 2024-01-16T15:12:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24c1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc06e550 0x7f24bc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24c1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f24bc070db0 0x7f24bc165230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24c1600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24bc06d700 con 0x7f24bc0b23e0 2024-01-16T15:12:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24c1600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0b2860 con 0x7f24bc070db0 2024-01-16T15:12:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24c1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0b2b40 con 0x7f24bc06e550 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24ba000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f24bc0b23e0 0x7f24bc0706a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37548/0 (socket says 172.21.15.43:37548) 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24baa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc06e550 0x7f24bc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24ba000640 1 -- 172.21.15.43:0/3376355481 learned_addr learned my addr 172.21.15.43:0/3376355481 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.813+0000 7f24bb400640 1 --2- 172.21.15.43:0/3376355481 >> v2:172.21.15.43:3300/0 conn(0x7f24bc070db0 0x7f24bc165230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24b9600640 1 -- 172.21.15.43:0/3376355481 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000839120 0 0) 0x7f24bc06d700 con 0x7f24bc0b23e0 2024-01-16T15:12:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24b9600640 1 -- 172.21.15.43:0/3376355481 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c003520 con 0x7f24bc0b23e0 2024-01-16T15:12:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24bb400640 1 -- 172.21.15.43:0/3376355481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc06e550 msgr2=0x7f24bc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24bb400640 1 --2- 172.21.15.43:0/3376355481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc06e550 0x7f24bc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24bb400640 1 -- 172.21.15.43:0/3376355481 >> v1:172.21.15.43:6789/0 conn(0x7f24bc0b23e0 legacy=0x7f24bc0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.814+0000 7f24bb400640 1 -- 172.21.15.43:0/3376355481 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0b2e20 con 0x7f24bc070db0 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.816+0000 7f24bb400640 1 --2- 172.21.15.43:0/3376355481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc070db0 0x7f24bc165230 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f24a40097a0 tx=0x7f24a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54a83ba48f5eeaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.817+0000 7f24b9600640 1 -- 172.21.15.43:0/3376355481 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4031830 con 0x7f24bc070db0 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.817+0000 7f24b9600640 1 -- 172.21.15.43:0/3376355481 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a4031990 con 0x7f24bc070db0 2024-01-16T15:12:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.817+0000 7f24b9600640 1 -- 172.21.15.43:0/3376355481 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4031d00 con 0x7f24bc070db0 2024-01-16T15:12:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 -- 172.21.15.43:0/3376355481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc070db0 msgr2=0x7f24bc165230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 --2- 172.21.15.43:0/3376355481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc070db0 0x7f24bc165230 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f24a40097a0 tx=0x7f24a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 -- 172.21.15.43:0/3376355481 shutdown_connections 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 --2- 172.21.15.43:0/3376355481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc070db0 0x7f24bc165230 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 --2- 172.21.15.43:0/3376355481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc06e550 0x7f24bc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.818+0000 7f24c1600640 1 -- 172.21.15.43:0/3376355481 >> 172.21.15.43:0/3376355481 conn(0x7f24bc0b0ba0 msgr2=0x7f24bc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.819+0000 7f24c1600640 1 -- 172.21.15.43:0/3376355481 shutdown_connections 2024-01-16T15:12:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.819+0000 7f24c1600640 1 -- 172.21.15.43:0/3376355481 wait complete. 2024-01-16T15:12:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.821+0000 7f24c1600640 1 Processor -- start 2024-01-16T15:12:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.822+0000 7f24c1600640 1 -- start start 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc06e550 0x7f24bc099c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24bc0b23e0 0x7f24bc14a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc09a780 con 0x7f24bc06e550 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc09aa60 con 0x7f24bc0b23e0 2024-01-16T15:12:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24c1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc09ad40 con 0x7f24bc070db0 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24ba000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24ba000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46736/0 (socket says 172.21.15.43:46736) 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.823+0000 7f24ba000640 1 -- 172.21.15.43:0/1046891582 learned_addr learned my addr 172.21.15.43:0/1046891582 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24baa00640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc06e550 0x7f24bc099c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 -- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24bc0b23e0 msgr2=0x7f24bc14a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24bc0b23e0 0x7f24bc14a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 -- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc06e550 msgr2=0x7f24bc099c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc06e550 0x7f24bc099c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 -- 172.21.15.43:0/1046891582 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4009400 con 0x7f24bc070db0 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.824+0000 7f24ba000640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00c390 tx=0x7f24ac00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.825+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac005ce0 con 0x7f24bc070db0 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.825+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac005e40 con 0x7f24bc070db0 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.825+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0072d0 con 0x7f24bc070db0 2024-01-16T15:12:25.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.825+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc0b3a80 con 0x7f24bc070db0 2024-01-16T15:12:25.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.825+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc142bf0 con 0x7f24bc070db0 2024-01-16T15:12:25.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.828+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc1473f0 con 0x7f24bc070db0 2024-01-16T15:12:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.830+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f24ac007430 con 0x7f24bc070db0 2024-01-16T15:12:25.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.831+0000 7f24b2a00640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 0x7f249406e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:25.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.831+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f24ac087ae0 con 0x7f24bc070db0 2024-01-16T15:12:25.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.833+0000 7f24baa00640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 0x7f249406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.834+0000 7f24baa00640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 0x7f249406e540 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f24bc09ca00 tx=0x7f24b40039b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:25.838+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24bc1473f0 con 0x7f24bc070db0 2024-01-16T15:12:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.065+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f24bc0b3900 con 0x7f249406c080 2024-01-16T15:12:26.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.118+0000 7f24b2a00640 1 -- 172.21.15.43:0/1046891582 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24b4004780 con 0x7f249406c080 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 msgr2=0x7f249406e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 0x7f249406e540 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f24bc09ca00 tx=0x7f24b40039b0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 msgr2=0x7f24bc14a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00c390 tx=0x7f24ac00c850 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 shutdown_connections 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f249406c080 0x7f249406e540 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24bc0b23e0 0x7f24bc14a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24bc070db0 0x7f24bc14a0f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 --2- 172.21.15.43:0/1046891582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24bc06e550 0x7f24bc099c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 >> 172.21.15.43:0/1046891582 conn(0x7f24bc0b0ba0 msgr2=0x7f24bc151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 shutdown_connections 2024-01-16T15:12:26.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.121+0000 7f24c1600640 1 -- 172.21.15.43:0/1046891582 wait complete. 2024-01-16T15:12:26.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96116 2024-01-16T15:12:26.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.361+0000 7f278aa00640 1 Processor -- start 2024-01-16T15:12:26.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.362+0000 7f278aa00640 1 -- start start 2024-01-16T15:12:26.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f278aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f27840cf380 0x7f27840ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f278aa00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840cd2b0 con 0x7f27840cb9d0 2024-01-16T15:12:26.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f278aa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840d02e0 con 0x7f2784158f40 2024-01-16T15:12:26.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f278aa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27840cbe60 con 0x7f27840cf380 2024-01-16T15:12:26.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f2788a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f27840cb9d0 0x7f278415b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37554/0 (socket says 172.21.15.43:37554) 2024-01-16T15:12:26.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.363+0000 7f2788a00640 1 -- 172.21.15.43:0/4253666756 learned_addr learned my addr 172.21.15.43:0/4253666756 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> v2:172.21.15.69:3300/0 conn(0x7f27840cf380 0x7f27840ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370906171 0 0) 0x7f27840cd2b0 con 0x7f27840cb9d0 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2764003520 con 0x7f27840cb9d0 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981036913 0 0) 0x7f2764003520 con 0x7f27840cb9d0 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 >> v2:172.21.15.69:3300/0 conn(0x7f27840cf380 msgr2=0x7f27840ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:26.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 --2- 172.21.15.43:0/4253666756 >> v2:172.21.15.69:3300/0 conn(0x7f27840cf380 0x7f27840ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 >> v1:172.21.15.69:6789/0 conn(0x7f2784158f40 legacy=0x7f27840cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840cd4e0 con 0x7f27840cb9d0 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f277c0019f0 con 0x7f27840cb9d0 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.364+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> v2:172.21.15.69:3300/0 conn(0x7f27840cf380 0x7f27840ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 >> v1:172.21.15.43:6789/0 conn(0x7f27840cb9d0 legacy=0x7f278415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2764003970 0x7f2764003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 0x7f2764004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2764005180 0x7f276400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27840cbe60 con 0x7f27640047c0 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276400c0d0 con 0x7f2764005180 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276400c290 con 0x7f2764003970 2024-01-16T15:12:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 0x7f2764004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783e00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2764003970 0x7f2764003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 -- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2764003970 msgr2=0x7f2764003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:26.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2764003970 0x7f2764003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 -- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2764005180 msgr2=0x7f276400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2764005180 0x7f276400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2783400640 1 -- 172.21.15.43:0/4253666756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764003520 con 0x7f27640047c0 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.365+0000 7f2788a00640 1 -- 172.21.15.43:0/4253666756 reap_dead start 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.366+0000 7f2783e00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2764003970 0x7f2764003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.367+0000 7f2783400640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 0x7f2764004c40 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f276c0098a0 tx=0x7f27640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd4602182954d68 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.367+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c004230 con 0x7f27640047c0 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.367+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c004390 con 0x7f27640047c0 2024-01-16T15:12:26.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.367+0000 7f2782a00640 1 -- 172.21.15.43:0/4253666756 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c003830 con 0x7f27640047c0 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 -- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 msgr2=0x7f2764004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 0x7f2764004c40 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f276c0098a0 tx=0x7f27640057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 -- 172.21.15.43:0/4253666756 shutdown_connections 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 --2- 172.21.15.43:0/4253666756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27640047c0 0x7f2764004c40 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 -- 172.21.15.43:0/4253666756 >> 172.21.15.43:0/4253666756 conn(0x7f27840cdb20 msgr2=0x7f27840cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 -- 172.21.15.43:0/4253666756 shutdown_connections 2024-01-16T15:12:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.368+0000 7f278aa00640 1 -- 172.21.15.43:0/4253666756 wait complete. 2024-01-16T15:12:26.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.371+0000 7f278aa00640 1 Processor -- start 2024-01-16T15:12:26.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.371+0000 7f278aa00640 1 -- start start 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27640047c0 0x7f2784155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27841574b0 0x7f2784157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2784157e70 con 0x7f27840c4d90 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27840b3ce0 con 0x7f27640047c0 2024-01-16T15:12:26.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f278aa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27840c8170 con 0x7f27841574b0 2024-01-16T15:12:26.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53896/0 (socket says 172.21.15.43:53896) 2024-01-16T15:12:26.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 -- 172.21.15.43:0/2866965269 learned_addr learned my addr 172.21.15.43:0/2866965269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 -- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27841574b0 msgr2=0x7f2784157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27841574b0 0x7f2784157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 -- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27640047c0 msgr2=0x7f2784155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27640047c0 0x7f2784155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.373+0000 7f2788a00640 1 -- 172.21.15.43:0/2866965269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840c83c0 con 0x7f27840c4d90 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f2788a00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f27840d20b0 tx=0x7f277c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c004d60 con 0x7f27840c4d90 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277c00a610 con 0x7f27840c4d90 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c00a9c0 con 0x7f27840c4d90 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27840c86a0 con 0x7f27840c4d90 2024-01-16T15:12:26.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.374+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27840c8c60 con 0x7f27840c4d90 2024-01-16T15:12:26.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.375+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278406fb50 con 0x7f27840c4d90 2024-01-16T15:12:26.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.376+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f277c00ab20 con 0x7f27840c4d90 2024-01-16T15:12:26.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.377+0000 7f277be00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 0x7f275c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.378+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f277c0879c0 con 0x7f27840c4d90 2024-01-16T15:12:26.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.378+0000 7f2783e00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 0x7f275c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.379+0000 7f2783e00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 0x7f275c06e470 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2784068d90 tx=0x7f2774009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.381+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f277c0542e0 con 0x7f27840c4d90 2024-01-16T15:12:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.658+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96116", "target": ["mon-mgr", ""]}) v1 -- 0x7f278415ec00 con 0x7f275c06bfb0 2024-01-16T15:12:26.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.696+0000 7f277be00640 1 -- 172.21.15.43:0/2866965269 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f278415ec00 con 0x7f275c06bfb0 2024-01-16T15:12:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 msgr2=0x7f275c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 0x7f275c06e470 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2784068d90 tx=0x7f2774009040 comp rx=0 tx=0).stop 2024-01-16T15:12:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 msgr2=0x7f27840c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f27840d20b0 tx=0x7f277c00cb80 comp rx=0 tx=0).stop 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 shutdown_connections 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f275c06bfb0 0x7f275c06e470 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27840c4d90 0x7f27840c5210 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27841574b0 0x7f2784157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 --2- 172.21.15.43:0/2866965269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27640047c0 0x7f2784155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 >> 172.21.15.43:0/2866965269 conn(0x7f27840cdb20 msgr2=0x7f27840707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 shutdown_connections 2024-01-16T15:12:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:26.700+0000 7f278aa00640 1 -- 172.21.15.43:0/2866965269 wait complete. 2024-01-16T15:12:26.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96113 2024-01-16T15:12:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.004+0000 7f7c56e00640 1 Processor -- start 2024-01-16T15:12:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.004+0000 7f7c56e00640 1 -- start start 2024-01-16T15:12:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.006+0000 7f7c56e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c500cd150 0x7f7c5006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.006+0000 7f7c56e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c50142d50 con 0x7f7c500702a0 2024-01-16T15:12:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.006+0000 7f7c56e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c500706c0 con 0x7f7c5006ee30 2024-01-16T15:12:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.006+0000 7f7c56e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c500cd530 con 0x7f7c500cd150 2024-01-16T15:12:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.007+0000 7f7c4f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7c500702a0 0x7f7c5006e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37566/0 (socket says 172.21.15.43:37566) 2024-01-16T15:12:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.007+0000 7f7c4f400640 1 -- 172.21.15.43:0/1109861630 learned_addr learned my addr 172.21.15.43:0/1109861630 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.007+0000 7f7c4fe00640 1 --2- 172.21.15.43:0/1109861630 >> v2:172.21.15.69:3300/0 conn(0x7f7c500cd150 0x7f7c5006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577929870 0 0) 0x7f7c50142d50 con 0x7f7c500702a0 2024-01-16T15:12:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c30003520 con 0x7f7c500702a0 2024-01-16T15:12:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830647981 0 0) 0x7f7c500706c0 con 0x7f7c5006ee30 2024-01-16T15:12:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c50142d50 con 0x7f7c5006ee30 2024-01-16T15:12:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4fe00640 1 -- 172.21.15.43:0/1109861630 >> v1:172.21.15.69:6789/0 conn(0x7f7c5006ee30 legacy=0x7f7c500cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4fe00640 1 -- 172.21.15.43:0/1109861630 >> v1:172.21.15.43:6789/0 conn(0x7f7c500702a0 legacy=0x7f7c5006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.008+0000 7f7c4fe00640 1 -- 172.21.15.43:0/1109861630 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5006f430 con 0x7f7c500cd150 2024-01-16T15:12:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.010+0000 7f7c4fe00640 1 --2- 172.21.15.43:0/1109861630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c5006fd60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097a0 tx=0x7f7c4002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61936561217d9d5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.011+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40030710 con 0x7f7c500cd150 2024-01-16T15:12:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.011+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4002e050 con 0x7f7c500cd150 2024-01-16T15:12:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.011+0000 7f7c4ea00640 1 -- 172.21.15.43:0/1109861630 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40030a90 con 0x7f7c500cd150 2024-01-16T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 -- 172.21.15.43:0/1109861630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 msgr2=0x7f7c5006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 --2- 172.21.15.43:0/1109861630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c5006fd60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097a0 tx=0x7f7c4002b790 comp rx=0 tx=0).stop 2024-01-16T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 -- 172.21.15.43:0/1109861630 shutdown_connections 2024-01-16T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 --2- 172.21.15.43:0/1109861630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c5006fd60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 -- 172.21.15.43:0/1109861630 >> 172.21.15.43:0/1109861630 conn(0x7f7c500cdb20 msgr2=0x7f7c500cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.012+0000 7f7c56e00640 1 -- 172.21.15.43:0/1109861630 shutdown_connections 2024-01-16T15:12:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.013+0000 7f7c56e00640 1 -- 172.21.15.43:0/1109861630 wait complete. 2024-01-16T15:12:26.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.016+0000 7f7c56e00640 1 Processor -- start 2024-01-16T15:12:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.016+0000 7f7c56e00640 1 -- start start 2024-01-16T15:12:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5006ee30 0x7f7c501f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c500b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c501f78c0 con 0x7f7c500702a0 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c500c9260 con 0x7f7c500cd150 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.018+0000 7f7c56e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c500c94d0 con 0x7f7c5006ee30 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5006ee30 0x7f7c501f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53906/0 (socket says 172.21.15.43:53906) 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 -- 172.21.15.43:0/4282990399 learned_addr learned my addr 172.21.15.43:0/4282990399 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 -- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5006ee30 msgr2=0x7f7c501f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5006ee30 0x7f7c501f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 -- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 msgr2=0x7f7c500b4f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.019+0000 7f7c4f400640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c500b4f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.020+0000 7f7c4f400640 1 -- 172.21.15.43:0/4282990399 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40009400 con 0x7f7c500702a0 2024-01-16T15:12:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.020+0000 7f7c4f400640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c780 tx=0x7f7c3800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.021+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38005ab0 con 0x7f7c500702a0 2024-01-16T15:12:26.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.021+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38005c10 con 0x7f7c500702a0 2024-01-16T15:12:26.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.021+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38007070 con 0x7f7c500702a0 2024-01-16T15:12:26.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.021+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c500c97b0 con 0x7f7c500702a0 2024-01-16T15:12:26.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.021+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c500cae30 con 0x7f7c500702a0 2024-01-16T15:12:27.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.024+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c50143240 con 0x7f7c500702a0 2024-01-16T15:12:27.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.026+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7c380071d0 con 0x7f7c500702a0 2024-01-16T15:12:27.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.028+0000 7f7c47e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 0x7f7c2806e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.028+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7c38088410 con 0x7f7c500702a0 2024-01-16T15:12:27.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.032+0000 7f7c4fe00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 0x7f7c2806e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:27.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.034+0000 7f7c4fe00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 0x7f7c2806e540 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7c40034000 tx=0x7f7c4002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:27.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.036+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c380bde90 con 0x7f7c500702a0 2024-01-16T15:12:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.259+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96113", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c500cb0b0 con 0x7f7c2806c080 2024-01-16T15:12:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.301+0000 7f7c47e00640 1 -- 172.21.15.43:0/4282990399 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7c40041070 con 0x7f7c2806c080 2024-01-16T15:12:27.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 msgr2=0x7f7c2806e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 0x7f7c2806e540 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7c40034000 tx=0x7f7c4002f000 comp rx=0 tx=0).stop 2024-01-16T15:12:27.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 msgr2=0x7f7c500ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c780 tx=0x7f7c3800cb80 comp rx=0 tx=0).stop 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 shutdown_connections 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c2806c080 0x7f7c2806e540 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c500cd150 0x7f7c500b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c500702a0 0x7f7c500ca8f0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 --2- 172.21.15.43:0/4282990399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5006ee30 0x7f7c501f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.304+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 >> 172.21.15.43:0/4282990399 conn(0x7f7c500cdb20 msgr2=0x7f7c5015dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.305+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 shutdown_connections 2024-01-16T15:12:27.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.305+0000 7f7c56e00640 1 -- 172.21.15.43:0/4282990399 wait complete. 2024-01-16T15:12:27.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96114 2024-01-16T15:12:27.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.531+0000 7fadc1000640 1 Processor -- start 2024-01-16T15:12:27.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.532+0000 7fadc1000640 1 -- start start 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.533+0000 7fadc1000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fadbc06d9e0 0x7fadbc165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.533+0000 7fadc1000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc15f930 con 0x7fadbc144770 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.533+0000 7fadc1000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc0b3b10 con 0x7fadbc15f270 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.533+0000 7fadc1000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadbc0b43b0 con 0x7fadbc06d9e0 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbb400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fadbc144770 0x7fadbc15eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37580/0 (socket says 172.21.15.43:37580) 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbb400640 1 -- 172.21.15.43:0/3312540719 learned_addr learned my addr 172.21.15.43:0/3312540719 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:27.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbbe00640 1 --2- 172.21.15.43:0/3312540719 >> v2:172.21.15.43:3300/0 conn(0x7fadbc06d9e0 0x7fadbc165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:27.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadba000640 1 -- 172.21.15.43:0/3312540719 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277846189 0 0) 0x7fadbc15f930 con 0x7fadbc144770 2024-01-16T15:12:27.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadba000640 1 -- 172.21.15.43:0/3312540719 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9c003520 con 0x7fadbc144770 2024-01-16T15:12:27.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbbe00640 1 -- 172.21.15.43:0/3312540719 >> v1:172.21.15.69:6789/0 conn(0x7fadbc15f270 legacy=0x7fadbc0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbbe00640 1 -- 172.21.15.43:0/3312540719 >> v1:172.21.15.43:6789/0 conn(0x7fadbc144770 legacy=0x7fadbc15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.534+0000 7fadbbe00640 1 -- 172.21.15.43:0/3312540719 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0b3df0 con 0x7fadbc06d9e0 2024-01-16T15:12:27.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.535+0000 7fadbbe00640 1 --2- 172.21.15.43:0/3312540719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc06d9e0 0x7fadbc165ef0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009d90 tx=0x7fadb402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a9e9baffbaa724 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:27.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadba000640 1 -- 172.21.15.43:0/3312540719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb402cd00 con 0x7fadbc06d9e0 2024-01-16T15:12:27.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadba000640 1 -- 172.21.15.43:0/3312540719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb402ce60 con 0x7fadbc06d9e0 2024-01-16T15:12:27.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadba000640 1 -- 172.21.15.43:0/3312540719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4034640 con 0x7fadbc06d9e0 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 -- 172.21.15.43:0/3312540719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc06d9e0 msgr2=0x7fadbc165ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 --2- 172.21.15.43:0/3312540719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc06d9e0 0x7fadbc165ef0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009d90 tx=0x7fadb402bf50 comp rx=0 tx=0).stop 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 -- 172.21.15.43:0/3312540719 shutdown_connections 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 --2- 172.21.15.43:0/3312540719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc06d9e0 0x7fadbc165ef0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 -- 172.21.15.43:0/3312540719 >> 172.21.15.43:0/3312540719 conn(0x7fadbc0cec10 msgr2=0x7fadbc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 -- 172.21.15.43:0/3312540719 shutdown_connections 2024-01-16T15:12:27.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.536+0000 7fadc1000640 1 -- 172.21.15.43:0/3312540719 wait complete. 2024-01-16T15:12:27.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.538+0000 7fadc1000640 1 Processor -- start 2024-01-16T15:12:27.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.538+0000 7fadc1000640 1 -- start start 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadbc06d9e0 0x7fadbc13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadbc15f270 0x7fadbc13bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc13e200 con 0x7fadbc144770 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc13ccb0 con 0x7fadbc15f270 2024-01-16T15:12:27.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadc1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc13cf00 con 0x7fadbc06d9e0 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadbaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadbaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53934/0 (socket says 172.21.15.43:53934) 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.539+0000 7fadbaa00640 1 -- 172.21.15.43:0/1997336303 learned_addr learned my addr 172.21.15.43:0/1997336303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 -- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadbc06d9e0 msgr2=0x7fadbc13dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbb400640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadbc06d9e0 0x7fadbc13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadbc06d9e0 0x7fadbc13dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 -- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadbc15f270 msgr2=0x7fadbc13bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadbc15f270 0x7fadbc13bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 -- 172.21.15.43:0/1997336303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4009a70 con 0x7fadbc144770 2024-01-16T15:12:27.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadbaa00640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fada400c3f0 tx=0x7fada400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40052d0 con 0x7fadbc144770 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada4005450 con 0x7fadbc144770 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4007260 con 0x7fadbc144770 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.540+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc13d1e0 con 0x7fadbc144770 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.541+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc13d7a0 con 0x7fadbc144770 2024-01-16T15:12:27.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.542+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadbc04e840 con 0x7fadbc144770 2024-01-16T15:12:27.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.543+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fada40073c0 con 0x7fadbc144770 2024-01-16T15:12:27.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.544+0000 7fadb3400640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 0x7fad9406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:27.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.544+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fada4087970 con 0x7fadbc144770 2024-01-16T15:12:27.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.545+0000 7fadbb400640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 0x7fad9406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:27.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.545+0000 7fadbb400640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 0x7fad9406e470 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fadac005880 tx=0x7fadac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:27.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.549+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fada4054230 con 0x7fadbc144770 2024-01-16T15:12:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.763+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96114", "target": ["mon-mgr", ""]}) v1 -- 0x7fadbc0b2160 con 0x7fad9406bfb0 2024-01-16T15:12:27.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.801+0000 7fadb3400640 1 -- 172.21.15.43:0/1997336303 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fadbc0b2160 con 0x7fad9406bfb0 2024-01-16T15:12:27.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.803+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 msgr2=0x7fad9406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.803+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 0x7fad9406e470 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fadac005880 tx=0x7fadac009040 comp rx=0 tx=0).stop 2024-01-16T15:12:27.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.803+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 msgr2=0x7fadbc1592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:27.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.803+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fada400c3f0 tx=0x7fada400c8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 shutdown_connections 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad9406bfb0 0x7fad9406e470 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadbc15f270 0x7fadbc13bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadbc144770 0x7fadbc1592b0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 --2- 172.21.15.43:0/1997336303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fadbc06d9e0 0x7fadbc13dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 >> 172.21.15.43:0/1997336303 conn(0x7fadbc0cec10 msgr2=0x7fadbc070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 shutdown_connections 2024-01-16T15:12:27.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:27.804+0000 7fadc1000640 1 -- 172.21.15.43:0/1997336303 wait complete. 2024-01-16T15:12:27.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96112 2024-01-16T15:12:28.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.034+0000 7f6039400640 1 Processor -- start 2024-01-16T15:12:28.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.034+0000 7f6039400640 1 -- start start 2024-01-16T15:12:28.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6039400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60340ae490 0x7f6034070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6039400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603406d7a0 con 0x7f603406dc30 2024-01-16T15:12:28.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6039400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60340ae900 con 0x7f60340acd70 2024-01-16T15:12:28.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6039400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603415a5b0 con 0x7f60340ae490 2024-01-16T15:12:28.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6031e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f603406dc30 0x7f6034070490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37586/0 (socket says 172.21.15.43:37586) 2024-01-16T15:12:28.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.036+0000 7f6031e00640 1 -- 172.21.15.43:0/2514807376 learned_addr learned my addr 172.21.15.43:0/2514807376 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6033200640 1 --2- 172.21.15.43:0/2514807376 >> v2:172.21.15.69:3300/0 conn(0x7f60340ae490 0x7f6034070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640335536 0 0) 0x7f603406d7a0 con 0x7f603406dc30 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6014003520 con 0x7f603406dc30 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701289524 0 0) 0x7f6014003520 con 0x7f603406dc30 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 >> v2:172.21.15.69:3300/0 conn(0x7f60340ae490 msgr2=0x7f6034070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:28.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 --2- 172.21.15.43:0/2514807376 >> v2:172.21.15.69:3300/0 conn(0x7f60340ae490 0x7f6034070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 >> v1:172.21.15.69:6789/0 conn(0x7f60340acd70 legacy=0x7f60340ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603406ce70 con 0x7f603406dc30 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f601c001670 con 0x7f603406dc30 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 >> v1:172.21.15.43:6789/0 conn(0x7f603406dc30 legacy=0x7f6034070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140039c0 0x7f6014003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60140047c0 0x7f6014004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 0x7f601400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603415a5b0 con 0x7f60140051d0 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601400c120 con 0x7f60140039c0 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601400c2e0 con 0x7f60140047c0 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.037+0000 7f6033200640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 0x7f601400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6032800640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140039c0 0x7f6014003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6033200640 1 -- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60140047c0 msgr2=0x7f6014004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6033200640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60140047c0 0x7f6014004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6033200640 1 -- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140039c0 msgr2=0x7f6014003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:28.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6033200640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140039c0 0x7f6014003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.038+0000 7f6033200640 1 -- 172.21.15.43:0/2514807376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014003520 con 0x7f60140051d0 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.039+0000 7f6033200640 1 -- 172.21.15.43:0/2514807376 reap_dead start 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.039+0000 7f6033200640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 0x7f601400b980 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f602c007360 tx=0x7f60140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c572ba00c9102b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.039+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c02ccf0 con 0x7f60140051d0 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.039+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c02ce50 con 0x7f60140051d0 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.039+0000 7f6031400640 1 -- 172.21.15.43:0/2514807376 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c003bd0 con 0x7f60140051d0 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 -- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 msgr2=0x7f601400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 0x7f601400b980 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f602c007360 tx=0x7f60140057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 -- 172.21.15.43:0/2514807376 shutdown_connections 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 --2- 172.21.15.43:0/2514807376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60140051d0 0x7f601400b980 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 -- 172.21.15.43:0/2514807376 >> 172.21.15.43:0/2514807376 conn(0x7f60340b0ba0 msgr2=0x7f60340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 -- 172.21.15.43:0/2514807376 shutdown_connections 2024-01-16T15:12:28.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.040+0000 7f6039400640 1 -- 172.21.15.43:0/2514807376 wait complete. 2024-01-16T15:12:28.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.042+0000 7f6039400640 1 Processor -- start 2024-01-16T15:12:28.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.042+0000 7f6039400640 1 -- start start 2024-01-16T15:12:28.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140051d0 0x7f6034155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60340cd6d0 0x7f60340cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034156d70 con 0x7f603409c9b0 2024-01-16T15:12:28.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60340ce090 con 0x7f60140051d0 2024-01-16T15:12:28.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6039400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6034099a00 con 0x7f60340cd6d0 2024-01-16T15:12:28.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6031e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6031e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53954/0 (socket says 172.21.15.43:53954) 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.043+0000 7f6031e00640 1 -- 172.21.15.43:0/48214363 learned_addr learned my addr 172.21.15.43:0/48214363 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6032800640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140051d0 0x7f6034155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6033200640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60340cd6d0 0x7f60340cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 -- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60340cd6d0 msgr2=0x7f60340cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60340cd6d0 0x7f60340cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 -- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140051d0 msgr2=0x7f6034155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140051d0 0x7f6034155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 -- 172.21.15.43:0/48214363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034099ca0 con 0x7f603409c9b0 2024-01-16T15:12:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.044+0000 7f6031e00640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f601c00c730 tx=0x7f601c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.045+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0072d0 con 0x7f603409c9b0 2024-01-16T15:12:28.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.045+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c007430 con 0x7f603409c9b0 2024-01-16T15:12:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.045+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6034099f80 con 0x7f603409c9b0 2024-01-16T15:12:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.045+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0047e0 con 0x7f603409c9b0 2024-01-16T15:12:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.045+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6034156990 con 0x7f603409c9b0 2024-01-16T15:12:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.046+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6034156eb0 con 0x7f603409c9b0 2024-01-16T15:12:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.047+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f601c004940 con 0x7f603409c9b0 2024-01-16T15:12:28.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.049+0000 7f602aa00640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 0x7f600c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.049+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f601c087c00 con 0x7f603409c9b0 2024-01-16T15:12:28.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.049+0000 7f6032800640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 0x7f600c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.050+0000 7f6032800640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 0x7f600c06e460 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6024001710 tx=0x7f6024009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.054+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6034156eb0 con 0x7f603409c9b0 2024-01-16T15:12:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.266+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96112", "target": ["mon-mgr", ""]}) v1 -- 0x7f60340b3d90 con 0x7f600c06bfa0 2024-01-16T15:12:28.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.315+0000 7f602aa00640 1 -- 172.21.15.43:0/48214363 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f602400d050 con 0x7f600c06bfa0 2024-01-16T15:12:28.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 msgr2=0x7f600c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 0x7f600c06e460 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6024001710 tx=0x7f6024009040 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 msgr2=0x7f6034157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f601c00c730 tx=0x7f601c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 shutdown_connections 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f600c06bfa0 0x7f600c06e460 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60340cd6d0 0x7f60340cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f603409c9b0 0x7f6034157fd0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 --2- 172.21.15.43:0/48214363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60140051d0 0x7f6034155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 >> 172.21.15.43:0/48214363 conn(0x7f60340b0ba0 msgr2=0x7f60340afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 shutdown_connections 2024-01-16T15:12:28.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.319+0000 7f6039400640 1 -- 172.21.15.43:0/48214363 wait complete. 2024-01-16T15:12:28.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9612 2024-01-16T15:12:28.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:28.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:28.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.540+0000 7ffb53e00640 1 Processor -- start 2024-01-16T15:12:28.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.541+0000 7ffb53e00640 1 -- start start 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb53e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb541435d0 0x7ffb54071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb53e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb541439b0 0x7ffb540b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb53e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb54070d40 con 0x7ffb540b3f70 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb53e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb540b43e0 con 0x7ffb541439b0 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb53e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb54071020 con 0x7ffb541435d0 2024-01-16T15:12:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb52000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb541439b0 0x7ffb540b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.542+0000 7ffb52000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb541439b0 0x7ffb540b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53966/0 (socket says 172.21.15.43:53966) 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.543+0000 7ffb52000640 1 -- 172.21.15.43:0/3369175094 learned_addr learned my addr 172.21.15.43:0/3369175094 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.543+0000 7ffb52000640 1 -- 172.21.15.43:0/3369175094 >> v2:172.21.15.69:3300/0 conn(0x7ffb541435d0 msgr2=0x7ffb54071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.543+0000 7ffb52000640 1 --2- 172.21.15.43:0/3369175094 >> v2:172.21.15.69:3300/0 conn(0x7ffb541435d0 0x7ffb54071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.543+0000 7ffb52000640 1 -- 172.21.15.43:0/3369175094 >> v1:172.21.15.69:6789/0 conn(0x7ffb540b3f70 legacy=0x7ffb54070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.543+0000 7ffb52000640 1 -- 172.21.15.43:0/3369175094 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54071300 con 0x7ffb541439b0 2024-01-16T15:12:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.544+0000 7ffb52000640 1 --2- 172.21.15.43:0/3369175094 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb540b3a30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffb540cfc10 tx=0x7ffb3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da8e79221535856 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.544+0000 7ffb51600640 1 -- 172.21.15.43:0/3369175094 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c0317e0 con 0x7ffb541439b0 2024-01-16T15:12:28.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.544+0000 7ffb51600640 1 -- 172.21.15.43:0/3369175094 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c031940 con 0x7ffb541439b0 2024-01-16T15:12:28.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.544+0000 7ffb51600640 1 -- 172.21.15.43:0/3369175094 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c031cd0 con 0x7ffb541439b0 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.545+0000 7ffb53e00640 1 -- 172.21.15.43:0/3369175094 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 msgr2=0x7ffb540b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.545+0000 7ffb53e00640 1 --2- 172.21.15.43:0/3369175094 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb540b3a30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffb540cfc10 tx=0x7ffb3c02b880 comp rx=0 tx=0).stop 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.546+0000 7ffb53e00640 1 -- 172.21.15.43:0/3369175094 shutdown_connections 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.546+0000 7ffb53e00640 1 --2- 172.21.15.43:0/3369175094 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb540b3a30 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.546+0000 7ffb53e00640 1 --2- 172.21.15.43:0/3369175094 >> v2:172.21.15.69:3300/0 conn(0x7ffb541435d0 0x7ffb54071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.546+0000 7ffb53e00640 1 -- 172.21.15.43:0/3369175094 >> 172.21.15.43:0/3369175094 conn(0x7ffb540cec10 msgr2=0x7ffb540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.547+0000 7ffb53e00640 1 -- 172.21.15.43:0/3369175094 shutdown_connections 2024-01-16T15:12:28.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.548+0000 7ffb53e00640 1 -- 172.21.15.43:0/3369175094 wait complete. 2024-01-16T15:12:28.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.550+0000 7ffb53e00640 1 Processor -- start 2024-01-16T15:12:28.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.550+0000 7ffb53e00640 1 -- start start 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb540b3f70 0x7ffb5413cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb541435d0 0x7ffb540b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5413a1d0 con 0x7ffb541439b0 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54157220 con 0x7ffb540b3f70 2024-01-16T15:12:28.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.551+0000 7ffb53e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb541574e0 con 0x7ffb541435d0 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53978/0 (socket says 172.21.15.43:53978) 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 -- 172.21.15.43:0/2408816988 learned_addr learned my addr 172.21.15.43:0/2408816988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 -- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb541435d0 msgr2=0x7ffb540b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb541435d0 0x7ffb540b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 -- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb540b3f70 msgr2=0x7ffb5413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb540b3f70 0x7ffb5413cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.552+0000 7ffb53400640 1 -- 172.21.15.43:0/2408816988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c0094d0 con 0x7ffb541439b0 2024-01-16T15:12:28.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.553+0000 7ffb53400640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c390 tx=0x7ffb4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.553+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0035f0 con 0x7ffb541439b0 2024-01-16T15:12:28.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.553+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c003750 con 0x7ffb541439b0 2024-01-16T15:12:28.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.553+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c003b00 con 0x7ffb541439b0 2024-01-16T15:12:28.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.553+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb541577c0 con 0x7ffb541439b0 2024-01-16T15:12:28.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.554+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb54157d80 con 0x7ffb541439b0 2024-01-16T15:12:28.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.555+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5413f890 con 0x7ffb541439b0 2024-01-16T15:12:28.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.556+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ffb4c003c60 con 0x7ffb541439b0 2024-01-16T15:12:28.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.557+0000 7ffb4aa00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 0x7ffb2c06e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:28.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.557+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffb4c087cf0 con 0x7ffb541439b0 2024-01-16T15:12:28.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.557+0000 7ffb52a00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 0x7ffb2c06e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:28.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.558+0000 7ffb52a00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 0x7ffb2c06e4c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400ab90 tx=0x7ffb44009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.561+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb4c054520 con 0x7ffb541439b0 2024-01-16T15:12:28.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.917+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9612", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb540718c0 con 0x7ffb2c06c000 2024-01-16T15:12:28.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.947+0000 7ffb4aa00640 1 -- 172.21.15.43:0/2408816988 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffb440163f0 con 0x7ffb2c06c000 2024-01-16T15:12:28.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 msgr2=0x7ffb2c06e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 0x7ffb2c06e4c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400ab90 tx=0x7ffb44009250 comp rx=0 tx=0).stop 2024-01-16T15:12:28.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 msgr2=0x7ffb5415b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:28.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c390 tx=0x7ffb4c00c850 comp rx=0 tx=0).stop 2024-01-16T15:12:28.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 shutdown_connections 2024-01-16T15:12:28.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2c06c000 0x7ffb2c06e4c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb541439b0 0x7ffb5415b780 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb541435d0 0x7ffb540b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 --2- 172.21.15.43:0/2408816988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb540b3f70 0x7ffb5413cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:28.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.951+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 >> 172.21.15.43:0/2408816988 conn(0x7ffb540cec10 msgr2=0x7ffb540ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:28.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.952+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 shutdown_connections 2024-01-16T15:12:28.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:28.952+0000 7ffb53e00640 1 -- 172.21.15.43:0/2408816988 wait complete. 2024-01-16T15:12:28.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9619 2024-01-16T15:12:29.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.237+0000 7f8d08600640 1 Processor -- start 2024-01-16T15:12:29.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.237+0000 7f8d08600640 1 -- start start 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.238+0000 7f8d08600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d0015b350 0x7f8d000abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.238+0000 7f8d08600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d000adb60 0x7f8d000adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.238+0000 7f8d08600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0015b730 con 0x7f8d0009cb30 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.238+0000 7f8d08600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d000ac740 con 0x7f8d0015b350 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.238+0000 7f8d08600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000ac460 con 0x7f8d000adb60 2024-01-16T15:12:29.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8d05200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8d0009cb30 0x7f8d000b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37596/0 (socket says 172.21.15.43:37596) 2024-01-16T15:12:29.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8d05200640 1 -- 172.21.15.43:0/2502431124 learned_addr learned my addr 172.21.15.43:0/2502431124 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:29.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2502431124 >> v2:172.21.15.69:3300/0 conn(0x7f8d0015b350 0x7f8d000abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8d06600640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d000adb60 0x7f8d000adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564422212 0 0) 0x7f8ce8001110 con 0x7f8d0009cb30 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.239+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce0003520 con 0x7f8d0009cb30 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.240+0000 7f8d05c00640 1 -- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d000adb60 msgr2=0x7f8d000adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.240+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d000adb60 0x7f8d000adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.240+0000 7f8d05c00640 1 -- 172.21.15.43:0/2502431124 >> v1:172.21.15.43:6789/0 conn(0x7f8d0009cb30 legacy=0x7f8d000b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.240+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575674377 0 0) 0x7f8ce0003520 con 0x7f8d0009cb30 2024-01-16T15:12:29.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.240+0000 7f8d05c00640 1 -- 172.21.15.43:0/2502431124 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d000b2a50 con 0x7f8d0015b350 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0015b350 0x7f8d000abe80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00097a0 tx=0x7f8cf002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5941031a118b9d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8001800 con 0x7f8d0015b350 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf002e050 con 0x7f8d0015b350 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8cffe00640 1 -- 172.21.15.43:0/2502431124 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0030710 con 0x7f8d0015b350 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8d08600640 1 -- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0015b350 msgr2=0x7f8d000abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.241+0000 7f8d08600640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0015b350 0x7f8d000abe80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00097a0 tx=0x7f8cf002b790 comp rx=0 tx=0).stop 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 -- 172.21.15.43:0/2502431124 shutdown_connections 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d000adb60 0x7f8d000adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 --2- 172.21.15.43:0/2502431124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0015b350 0x7f8d000abe80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 -- 172.21.15.43:0/2502431124 >> 172.21.15.43:0/2502431124 conn(0x7f8d000b0ba0 msgr2=0x7f8d000b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 -- 172.21.15.43:0/2502431124 shutdown_connections 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.242+0000 7f8d08600640 1 -- 172.21.15.43:0/2502431124 wait complete. 2024-01-16T15:12:29.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.243+0000 7f8d08600640 1 Processor -- start 2024-01-16T15:12:29.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.243+0000 7f8d08600640 1 -- start start 2024-01-16T15:12:29.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d0009cb30 0x7f8d000af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d000adb60 0x7f8d00142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00154030 con 0x7f8d0015b350 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00142ff0 con 0x7f8d000adb60 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.244+0000 7f8d08600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001f1c10 con 0x7f8d0009cb30 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53994/0 (socket says 172.21.15.43:53994) 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 -- 172.21.15.43:0/2453338225 learned_addr learned my addr 172.21.15.43:0/2453338225 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d0009cb30 0x7f8d000af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d05200640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d000adb60 0x7f8d00142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 -- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d0009cb30 msgr2=0x7f8d000af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d0009cb30 0x7f8d000af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 -- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d000adb60 msgr2=0x7f8d00142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d000adb60 0x7f8d00142570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 -- 172.21.15.43:0/2453338225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf0009400 con 0x7f8d0015b350 2024-01-16T15:12:29.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.245+0000 7f8d06600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f8cf800efc0 tx=0x7f8cf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.246+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8014480 con 0x7f8d0015b350 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.246+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf800f040 con 0x7f8d0015b350 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.246+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8014820 con 0x7f8d0015b350 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.246+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d00154240 con 0x7f8d0015b350 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.246+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d00154800 con 0x7f8d0015b350 2024-01-16T15:12:29.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.247+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0009d200 con 0x7f8d0015b350 2024-01-16T15:12:29.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.248+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8cf8014980 con 0x7f8d0015b350 2024-01-16T15:12:29.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.249+0000 7f8cfd600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 0x7f8cd806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.249+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8cf8089470 con 0x7f8d0015b350 2024-01-16T15:12:29.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.249+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 0x7f8cd806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.250+0000 7f8d05c00640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 0x7f8cd806e470 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0034000 tx=0x7f8cf002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.253+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cf8055b00 con 0x7f8d0015b350 2024-01-16T15:12:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.492+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9619", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d000ae4f0 con 0x7f8cd806bfb0 2024-01-16T15:12:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.545+0000 7f8cfd600640 1 -- 172.21.15.43:0/2453338225 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8cf0030890 con 0x7f8cd806bfb0 2024-01-16T15:12:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.547+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 msgr2=0x7f8cd806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.547+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 0x7f8cd806e470 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0034000 tx=0x7f8cf002f000 comp rx=0 tx=0).stop 2024-01-16T15:12:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.547+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 msgr2=0x7f8d00142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.547+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f8cf800efc0 tx=0x7f8cf800c490 comp rx=0 tx=0).stop 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 shutdown_connections 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cd806bfb0 0x7f8cd806e470 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d000adb60 0x7f8d00142570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d0015b350 0x7f8d00142ab0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 --2- 172.21.15.43:0/2453338225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d0009cb30 0x7f8d000af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 >> 172.21.15.43:0/2453338225 conn(0x7f8d000b0ba0 msgr2=0x7f8d0006daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 shutdown_connections 2024-01-16T15:12:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.548+0000 7f8d08600640 1 -- 172.21.15.43:0/2453338225 wait complete. 2024-01-16T15:12:29.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9614 2024-01-16T15:12:29.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.805+0000 7fa95b800640 1 Processor -- start 2024-01-16T15:12:29.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.805+0000 7fa95b800640 1 -- start start 2024-01-16T15:12:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa95b800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa95415b350 0x7fa9540abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa95b800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95415b730 con 0x7fa9540adb60 2024-01-16T15:12:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa95b800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540ac740 con 0x7fa95409cb30 2024-01-16T15:12:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa95b800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9540ac460 con 0x7fa95415b350 2024-01-16T15:12:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa958e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa95415b350 0x7fa9540abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.806+0000 7fa959800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa9540adb60 0x7fa9540adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37610/0 (socket says 172.21.15.43:37610) 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.807+0000 7fa959800640 1 -- 172.21.15.43:0/3545935449 learned_addr learned my addr 172.21.15.43:0/3545935449 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.807+0000 7fa958e00640 1 -- 172.21.15.43:0/3545935449 >> v1:172.21.15.69:6789/0 conn(0x7fa95409cb30 legacy=0x7fa9540b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.807+0000 7fa958e00640 1 -- 172.21.15.43:0/3545935449 >> v1:172.21.15.43:6789/0 conn(0x7fa9540adb60 legacy=0x7fa9540adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.807+0000 7fa958e00640 1 -- 172.21.15.43:0/3545935449 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540b2a50 con 0x7fa95415b350 2024-01-16T15:12:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.807+0000 7fa953400640 1 -- 172.21.15.43:0/3545935449 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591710792 0 0) 0x7fa94c001490 con 0x7fa9540adb60 2024-01-16T15:12:29.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.809+0000 7fa958e00640 1 --2- 172.21.15.43:0/3545935449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95415b350 0x7fa9540abe80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa9440097a0 tx=0x7fa94402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70f28900c2728c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.809+0000 7fa953400640 1 -- 172.21.15.43:0/3545935449 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944030610 con 0x7fa95415b350 2024-01-16T15:12:29.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.809+0000 7fa953400640 1 -- 172.21.15.43:0/3545935449 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944030770 con 0x7fa95415b350 2024-01-16T15:12:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.809+0000 7fa953400640 1 -- 172.21.15.43:0/3545935449 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944030b00 con 0x7fa95415b350 2024-01-16T15:12:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 -- 172.21.15.43:0/3545935449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95415b350 msgr2=0x7fa9540abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 --2- 172.21.15.43:0/3545935449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95415b350 0x7fa9540abe80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa9440097a0 tx=0x7fa94402b960 comp rx=0 tx=0).stop 2024-01-16T15:12:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 -- 172.21.15.43:0/3545935449 shutdown_connections 2024-01-16T15:12:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 --2- 172.21.15.43:0/3545935449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95415b350 0x7fa9540abe80 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 -- 172.21.15.43:0/3545935449 >> 172.21.15.43:0/3545935449 conn(0x7fa9540b0ba0 msgr2=0x7fa9540b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 -- 172.21.15.43:0/3545935449 shutdown_connections 2024-01-16T15:12:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.810+0000 7fa95b800640 1 -- 172.21.15.43:0/3545935449 wait complete. 2024-01-16T15:12:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.812+0000 7fa95b800640 1 Processor -- start 2024-01-16T15:12:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.812+0000 7fa95b800640 1 -- start start 2024-01-16T15:12:29.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9540adb60 0x7fa954159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa95415b350 0x7fa954159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa954157bc0 con 0x7fa95409cb30 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa954157260 con 0x7fa95415b350 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa95b800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95415d0f0 con 0x7fa9540adb60 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54000/0 (socket says 172.21.15.43:54000) 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 -- 172.21.15.43:0/863317469 learned_addr learned my addr 172.21.15.43:0/863317469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 -- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9540adb60 msgr2=0x7fa954159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9540adb60 0x7fa954159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 -- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa95415b350 msgr2=0x7fa954159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa95415b350 0x7fa954159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.813+0000 7fa958e00640 1 -- 172.21.15.43:0/863317469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944009400 con 0x7fa95409cb30 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa958e00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa9440098c0 tx=0x7fa9440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94403e070 con 0x7fa95409cb30 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944039a10 con 0x7fa95409cb30 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944039da0 con 0x7fa95409cb30 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa954157500 con 0x7fa95409cb30 2024-01-16T15:12:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.814+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9541482b0 con 0x7fa95409cb30 2024-01-16T15:12:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.815+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa954157d00 con 0x7fa95409cb30 2024-01-16T15:12:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.816+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa94404b400 con 0x7fa95409cb30 2024-01-16T15:12:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.817+0000 7fa950c00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 0x7fa92c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.817+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fa9440aab20 con 0x7fa95409cb30 2024-01-16T15:12:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.817+0000 7fa953e00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 0x7fa92c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.818+0000 7fa953e00640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 0x7fa92c06e470 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f60 tx=0x7fa93c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:29.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:29.821+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa954157d00 con 0x7fa95409cb30 2024-01-16T15:12:30.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.030+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9614", "target": ["mon-mgr", ""]}) v1 -- 0x7fa95406e720 con 0x7fa92c06bfb0 2024-01-16T15:12:30.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.060+0000 7fa950c00640 1 -- 172.21.15.43:0/863317469 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa93c005080 con 0x7fa92c06bfb0 2024-01-16T15:12:30.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 msgr2=0x7fa92c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 0x7fa92c06e470 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f60 tx=0x7fa93c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 msgr2=0x7fa954147d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fa9440098c0 tx=0x7fa9440046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 shutdown_connections 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa92c06bfb0 0x7fa92c06e470 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa95415b350 0x7fa954159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9540adb60 0x7fa954159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 --2- 172.21.15.43:0/863317469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa95409cb30 0x7fa954147d70 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.063+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 >> 172.21.15.43:0/863317469 conn(0x7fa9540b0ba0 msgr2=0x7fa95406daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.064+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 shutdown_connections 2024-01-16T15:12:30.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.064+0000 7fa95b800640 1 -- 172.21.15.43:0/863317469 wait complete. 2024-01-16T15:12:30.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9618 2024-01-16T15:12:30.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.289+0000 7f132de00640 1 Processor -- start 2024-01-16T15:12:30.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.290+0000 7f132de00640 1 -- start start 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f132815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 0x7f132815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f132806c9d0 0x7f132806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13280b43b0 con 0x7f132806c9d0 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132815f930 con 0x7f13280b35a0 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.291+0000 7f132de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13280b3b10 con 0x7f1328144770 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f132806c9d0 0x7f132806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f132806c9d0 0x7f132806ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54008/0 (socket says 172.21.15.43:54008) 2024-01-16T15:12:30.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 -- 172.21.15.43:0/2280109919 learned_addr learned my addr 172.21.15.43:0/2280109919 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f131fe00640 1 --2- 172.21.15.43:0/2280109919 >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 0x7f132815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132ca00640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f132815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 -- 172.21.15.43:0/2280109919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 msgr2=0x7f132815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f132815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 -- 172.21.15.43:0/2280109919 >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 msgr2=0x7f132815f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 --2- 172.21.15.43:0/2280109919 >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 0x7f132815f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132d400640 1 -- 172.21.15.43:0/2280109919 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13280b3df0 con 0x7f132806c9d0 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f131fe00640 1 --2- 172.21.15.43:0/2280109919 >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 0x7f132815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:30.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.292+0000 7f132ca00640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f132815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.293+0000 7f132d400640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f132806c9d0 0x7f132806ce60 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f1320009e50 tx=0x7f132002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dab7b6c93bc3da9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.294+0000 7f1327e00640 1 -- 172.21.15.43:0/2280109919 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132002ca80 con 0x7f132806c9d0 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.294+0000 7f1327e00640 1 -- 172.21.15.43:0/2280109919 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132002cbe0 con 0x7f132806c9d0 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.294+0000 7f1327e00640 1 -- 172.21.15.43:0/2280109919 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13200343f0 con 0x7f132806c9d0 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.294+0000 7f132de00640 1 -- 172.21.15.43:0/2280109919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f132806c9d0 msgr2=0x7f132806ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.294+0000 7f132de00640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f132806c9d0 0x7f132806ce60 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f1320009e50 tx=0x7f132002bee0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 -- 172.21.15.43:0/2280109919 shutdown_connections 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f132806c9d0 0x7f132806ce60 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 --2- 172.21.15.43:0/2280109919 >> v2:172.21.15.69:3300/0 conn(0x7f13280b35a0 0x7f132815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 --2- 172.21.15.43:0/2280109919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f132815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 -- 172.21.15.43:0/2280109919 >> 172.21.15.43:0/2280109919 conn(0x7f13280cec10 msgr2=0x7f13280cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 -- 172.21.15.43:0/2280109919 shutdown_connections 2024-01-16T15:12:30.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.295+0000 7f132de00640 1 -- 172.21.15.43:0/2280109919 wait complete. 2024-01-16T15:12:30.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.297+0000 7f132de00640 1 Processor -- start 2024-01-16T15:12:30.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.297+0000 7f132de00640 1 -- start start 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f132806c9d0 0x7f1328140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f13281eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328141180 con 0x7f13280b35a0 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328141460 con 0x7f132806c9d0 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1328141740 con 0x7f1328144770 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54014/0 (socket says 172.21.15.43:54014) 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 -- 172.21.15.43:0/3610511270 learned_addr learned my addr 172.21.15.43:0/3610511270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132ca00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f132806c9d0 0x7f1328140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f132d400640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f13281eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 -- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 msgr2=0x7f13281eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f13281eff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 -- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f132806c9d0 msgr2=0x7f1328140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f132806c9d0 0x7f1328140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.299+0000 7f131fe00640 1 -- 172.21.15.43:0/3610511270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1320009b30 con 0x7f13280b35a0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f131fe00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f131000c7f0 tx=0x7f131000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310007a30 con 0x7f13280b35a0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310007b90 con 0x7f13280b35a0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131000ce40 con 0x7f13280b35a0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1328141a20 con 0x7f13280b35a0 2024-01-16T15:12:30.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.300+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132813ee00 con 0x7f13280b35a0 2024-01-16T15:12:30.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.301+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1328149750 con 0x7f13280b35a0 2024-01-16T15:12:30.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.302+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f13100113f0 con 0x7f13280b35a0 2024-01-16T15:12:30.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.303+0000 7f1325600640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 0x7f130006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.303+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f13100888b0 con 0x7f13280b35a0 2024-01-16T15:12:30.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.303+0000 7f132ca00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 0x7f130006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.304+0000 7f132ca00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 0x7f130006e470 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f131800a8d0 tx=0x7f1318008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.309+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1328149750 con 0x7f13280b35a0 2024-01-16T15:12:30.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.522+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9618", "target": ["mon-mgr", ""]}) v1 -- 0x7f13281415d0 con 0x7f130006bfb0 2024-01-16T15:12:30.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.573+0000 7f1325600640 1 -- 172.21.15.43:0/3610511270 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13281415d0 con 0x7f130006bfb0 2024-01-16T15:12:30.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.576+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 msgr2=0x7f130006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.576+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 0x7f130006e470 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f131800a8d0 tx=0x7f1318008040 comp rx=0 tx=0).stop 2024-01-16T15:12:30.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.576+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 msgr2=0x7f132813e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.576+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f131000c7f0 tx=0x7f131000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 shutdown_connections 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f130006bfb0 0x7f130006e470 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1328144770 0x7f13281eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13280b35a0 0x7f132813e8c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 --2- 172.21.15.43:0/3610511270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f132806c9d0 0x7f1328140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 >> 172.21.15.43:0/3610511270 conn(0x7f13280cec10 msgr2=0x7f13280ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.577+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 shutdown_connections 2024-01-16T15:12:30.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.578+0000 7f132de00640 1 -- 172.21.15.43:0/3610511270 wait complete. 2024-01-16T15:12:30.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9615 2024-01-16T15:12:30.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.878+0000 7f341ae00640 1 Processor -- start 2024-01-16T15:12:30.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.878+0000 7f341ae00640 1 -- start start 2024-01-16T15:12:30.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.879+0000 7f341ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.879+0000 7f341ae00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3414142d50 con 0x7f34140cd150 2024-01-16T15:12:30.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.879+0000 7f341ae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341406f430 con 0x7f3414070470 2024-01-16T15:12:30.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.879+0000 7f341ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34140cd530 con 0x7f341406ee30 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3413e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f34140cd150 0x7f341406fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37626/0 (socket says 172.21.15.43:37626) 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3413e00640 1 -- 172.21.15.43:0/1660178771 learned_addr learned my addr 172.21.15.43:0/1660178771 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3418e00640 1 --2- 172.21.15.43:0/1660178771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3412a00640 1 -- 172.21.15.43:0/1660178771 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651701068 0 0) 0x7f3414142d50 con 0x7f34140cd150 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3412a00640 1 -- 172.21.15.43:0/1660178771 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f4003520 con 0x7f34140cd150 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3418e00640 1 -- 172.21.15.43:0/1660178771 >> v1:172.21.15.69:6789/0 conn(0x7f3414070470 legacy=0x7f341406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3418e00640 1 -- 172.21.15.43:0/1660178771 >> v1:172.21.15.43:6789/0 conn(0x7f34140cd150 legacy=0x7f341406fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.880+0000 7f3418e00640 1 -- 172.21.15.43:0/1660178771 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414142fa0 con 0x7f341406ee30 2024-01-16T15:12:30.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.883+0000 7f3418e00640 1 --2- 172.21.15.43:0/1660178771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414168120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f340c009dc0 tx=0x7f340c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e690301270f229 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.883+0000 7f3412a00640 1 -- 172.21.15.43:0/1660178771 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c02cc40 con 0x7f341406ee30 2024-01-16T15:12:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.883+0000 7f3412a00640 1 -- 172.21.15.43:0/1660178771 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c02cda0 con 0x7f341406ee30 2024-01-16T15:12:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.883+0000 7f3412a00640 1 -- 172.21.15.43:0/1660178771 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c034660 con 0x7f341406ee30 2024-01-16T15:12:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 -- 172.21.15.43:0/1660178771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 msgr2=0x7f3414168120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 --2- 172.21.15.43:0/1660178771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414168120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f340c009dc0 tx=0x7f340c02be90 comp rx=0 tx=0).stop 2024-01-16T15:12:30.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 -- 172.21.15.43:0/1660178771 shutdown_connections 2024-01-16T15:12:30.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 --2- 172.21.15.43:0/1660178771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414168120 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 -- 172.21.15.43:0/1660178771 >> 172.21.15.43:0/1660178771 conn(0x7f34140cdb20 msgr2=0x7f34140cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:30.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.884+0000 7f341ae00640 1 -- 172.21.15.43:0/1660178771 shutdown_connections 2024-01-16T15:12:30.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.885+0000 7f341ae00640 1 -- 172.21.15.43:0/1660178771 wait complete. 2024-01-16T15:12:30.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.887+0000 7f341ae00640 1 Processor -- start 2024-01-16T15:12:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.887+0000 7f341ae00640 1 -- start start 2024-01-16T15:12:30.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.888+0000 7f341ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.888+0000 7f341ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.889+0000 7f341ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 0x7f34141564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.889+0000 7f341ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34140c9470 con 0x7f34140cd150 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.889+0000 7f341ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3414156fb0 con 0x7f3414070470 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.889+0000 7f341ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34140c8860 con 0x7f341406ee30 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.890+0000 7f3413400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.890+0000 7f3413400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:53256/0 (socket says 172.21.15.43:53256) 2024-01-16T15:12:30.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.890+0000 7f3413400640 1 -- 172.21.15.43:0/1827832307 learned_addr learned my addr 172.21.15.43:0/1827832307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.890+0000 7f3418e00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 0x7f34141564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.890+0000 7f3413e00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 -- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 msgr2=0x7f3414141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414141550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 -- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 msgr2=0x7f34141564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 0x7f34141564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:30.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 -- 172.21.15.43:0/1827832307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c009a70 con 0x7f3414070470 2024-01-16T15:12:30.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3418e00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 0x7f34141564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.891+0000 7f3413400640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00c440 tx=0x7f33fc001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.892+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc0052d0 con 0x7f3414070470 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.892+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc005430 con 0x7f3414070470 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.892+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc007260 con 0x7f3414070470 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.892+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3414143c00 con 0x7f3414070470 2024-01-16T15:12:30.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.892+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34140c8e20 con 0x7f3414070470 2024-01-16T15:12:30.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.894+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341406fb50 con 0x7f3414070470 2024-01-16T15:12:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.897+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f33fc0073c0 con 0x7f3414070470 2024-01-16T15:12:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.898+0000 7f340be00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 0x7f33ec06e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.898+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f33fc087a10 con 0x7f3414070470 2024-01-16T15:12:30.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.900+0000 7f3413e00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 0x7f33ec06e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:30.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.902+0000 7f3413e00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 0x7f33ec06e540 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f34040011e0 tx=0x7f3404009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:30.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:30.905+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33fc0542d0 con 0x7f3414070470 2024-01-16T15:12:31.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:31.175+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9615", "target": ["mon-mgr", ""]}) v1 -- 0x7f34140cec50 con 0x7f33ec06c080 2024-01-16T15:12:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.657+0000 7f340be00640 1 -- 172.21.15.43:0/1827832307 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34140cec50 con 0x7f33ec06c080 2024-01-16T15:12:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 msgr2=0x7f33ec06e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 0x7f33ec06e540 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f34040011e0 tx=0x7f3404009040 comp rx=0 tx=0).stop 2024-01-16T15:12:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 msgr2=0x7f3414155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f33fc00c440 tx=0x7f33fc001e70 comp rx=0 tx=0).stop 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 shutdown_connections 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f33ec06c080 0x7f33ec06e540 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3414070470 0x7f3414155f60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34140cd150 0x7f34141564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 --2- 172.21.15.43:0/1827832307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f341406ee30 0x7f3414141550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 >> 172.21.15.43:0/1827832307 conn(0x7f34140cdb20 msgr2=0x7f341406c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.661+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 shutdown_connections 2024-01-16T15:12:38.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.662+0000 7f341ae00640 1 -- 172.21.15.43:0/1827832307 wait complete. 2024-01-16T15:12:38.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9617 2024-01-16T15:12:38.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.906+0000 7f2507200640 1 Processor -- start 2024-01-16T15:12:38.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.906+0000 7f2507200640 1 -- start start 2024-01-16T15:12:38.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2507200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2508144770 0x7f250815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2507200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f250815f0a0 0x7f25080ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2507200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250815f930 con 0x7f25080b35a0 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2507200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080b43b0 con 0x7f250815f0a0 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2507200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250815f530 con 0x7f2508144770 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2506800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f250815f0a0 0x7f25080ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2506800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f250815f0a0 0x7f25080ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56224/0 (socket says 172.21.15.43:56224) 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2506800640 1 -- 172.21.15.43:0/2595829153 learned_addr learned my addr 172.21.15.43:0/2595829153 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2505e00640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2508144770 0x7f250815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2506800640 1 -- 172.21.15.43:0/2595829153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2508144770 msgr2=0x7f250815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.908+0000 7f2506800640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2508144770 0x7f250815eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.909+0000 7f2506800640 1 -- 172.21.15.43:0/2595829153 >> v1:172.21.15.69:6789/0 conn(0x7f25080b35a0 legacy=0x7f250806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:38.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.909+0000 7f2506800640 1 -- 172.21.15.43:0/2595829153 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080ce9f0 con 0x7f250815f0a0 2024-01-16T15:12:38.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.910+0000 7f2506800640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f250815f0a0 0x7f25080ce4b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2500009e80 tx=0x7f250002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08f206466f5dee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:38.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.910+0000 7f2504a00640 1 -- 172.21.15.43:0/2595829153 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250002ca50 con 0x7f250815f0a0 2024-01-16T15:12:38.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.910+0000 7f2504a00640 1 -- 172.21.15.43:0/2595829153 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250002cbb0 con 0x7f250815f0a0 2024-01-16T15:12:38.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.910+0000 7f2504a00640 1 -- 172.21.15.43:0/2595829153 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000343f0 con 0x7f250815f0a0 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 -- 172.21.15.43:0/2595829153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f250815f0a0 msgr2=0x7f25080ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f250815f0a0 0x7f25080ce4b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2500009e80 tx=0x7f250002beb0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 -- 172.21.15.43:0/2595829153 shutdown_connections 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f250815f0a0 0x7f25080ce4b0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 --2- 172.21.15.43:0/2595829153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2508144770 0x7f250815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 -- 172.21.15.43:0/2595829153 >> 172.21.15.43:0/2595829153 conn(0x7f25080cec10 msgr2=0x7f25080cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 -- 172.21.15.43:0/2595829153 shutdown_connections 2024-01-16T15:12:38.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.911+0000 7f2507200640 1 -- 172.21.15.43:0/2595829153 wait complete. 2024-01-16T15:12:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.913+0000 7f2507200640 1 Processor -- start 2024-01-16T15:12:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.913+0000 7f2507200640 1 -- start start 2024-01-16T15:12:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 0x7f250815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f250815f0a0 0x7f2508140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250815bd40 con 0x7f2508144770 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250815c020 con 0x7f25080b35a0 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2507200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250815c300 con 0x7f250815f0a0 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2505400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2505400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56228/0 (socket says 172.21.15.43:56228) 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2505400640 1 -- 172.21.15.43:0/3062612433 learned_addr learned my addr 172.21.15.43:0/3062612433 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.914+0000 7f2505e00640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 0x7f250815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 -- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f250815f0a0 msgr2=0x7f2508140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2506800640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f250815f0a0 0x7f2508140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f250815f0a0 0x7f2508140400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 -- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 msgr2=0x7f250815b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 0x7f250815b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 -- 172.21.15.43:0/3062612433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009b30 con 0x7f2508144770 2024-01-16T15:12:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505e00640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 0x7f250815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2505400640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f24f000b3f0 tx=0x7f24f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0011830 con 0x7f2508144770 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0011990 con 0x7f2508144770 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0011d40 con 0x7f2508144770 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.915+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250815c5e0 con 0x7f2508144770 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.916+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508136f10 con 0x7f2508144770 2024-01-16T15:12:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.917+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250815cb00 con 0x7f2508144770 2024-01-16T15:12:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.918+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f24f001a400 con 0x7f2508144770 2024-01-16T15:12:38.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.919+0000 7f24fe000640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 0x7f24e006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:38.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.919+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f24f0088410 con 0x7f2508144770 2024-01-16T15:12:38.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.919+0000 7f2505e00640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 0x7f24e006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:38.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.920+0000 7f2505e00640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 0x7f24e006e470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f24f800b820 tx=0x7f24f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:38.924+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24f0054cd0 con 0x7f2508144770 2024-01-16T15:12:39.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.153+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9617", "target": ["mon-mgr", ""]}) v1 -- 0x7f25080b40d0 con 0x7f24e006bfb0 2024-01-16T15:12:39.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.195+0000 7f24fe000640 1 -- 172.21.15.43:0/3062612433 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24f800c040 con 0x7f24e006bfb0 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.198+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 msgr2=0x7f24e006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.198+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 0x7f24e006e470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f24f800b820 tx=0x7f24f8008040 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.198+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 msgr2=0x7f25081369d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.198+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f24f000b3f0 tx=0x7f24f000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 shutdown_connections 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f24e006bfb0 0x7f24e006e470 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f250815f0a0 0x7f2508140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2508144770 0x7f25081369d0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 --2- 172.21.15.43:0/3062612433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25080b35a0 0x7f250815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 >> 172.21.15.43:0/3062612433 conn(0x7f25080cec10 msgr2=0x7f2508142e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 shutdown_connections 2024-01-16T15:12:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.199+0000 7f2507200640 1 -- 172.21.15.43:0/3062612433 wait complete. 2024-01-16T15:12:39.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96118 2024-01-16T15:12:39.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.442+0000 7f7852000640 1 Processor -- start 2024-01-16T15:12:39.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.442+0000 7f7852000640 1 -- start start 2024-01-16T15:12:39.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7852000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f784c15d3b0 0x7f784c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7852000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f784c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7852000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c0d2010 con 0x7f784c0d1bc0 2024-01-16T15:12:39.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7852000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784c0c6bf0 con 0x7f784c15d3b0 2024-01-16T15:12:39.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7852000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0d1400 con 0x7f784c0d0fa0 2024-01-16T15:12:39.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7851600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f784c0d1bc0 0x7f784c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41280/0 (socket says 172.21.15.43:41280) 2024-01-16T15:12:39.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.444+0000 7f7851600640 1 -- 172.21.15.43:0/766836512 learned_addr learned my addr 172.21.15.43:0/766836512 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:39.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042551440 0 0) 0x7f7844001110 con 0x7f784c0d1bc0 2024-01-16T15:12:39.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f7850c00640 1 --2- 172.21.15.43:0/766836512 >> v2:172.21.15.69:3300/0 conn(0x7f784c15d3b0 0x7f784c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782c003520 con 0x7f784c0d1bc0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f7843c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f784c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158789382 0 0) 0x7f782c003520 con 0x7f784c0d1bc0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 msgr2=0x7f784c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f784c0d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 >> v2:172.21.15.69:3300/0 conn(0x7f784c15d3b0 msgr2=0x7f784c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 --2- 172.21.15.43:0/766836512 >> v2:172.21.15.69:3300/0 conn(0x7f784c15d3b0 0x7f784c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0cedc0 con 0x7f784c0d1bc0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7844001800 con 0x7f784c0d1bc0 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 >> v1:172.21.15.43:6789/0 conn(0x7f784c0d1bc0 legacy=0x7f784c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0039e0 0x7f782c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f782c0047c0 0x7f782c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f782c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0c6bf0 con 0x7f782c0047c0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0d1400 con 0x7f782c0039e0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.445+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c00bbc0 con 0x7f784c0d0fa0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f782c0047c0 0x7f782c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7850c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0039e0 0x7f782c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7851600640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f782c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 -- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 msgr2=0x7f782c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c0d0fa0 0x7f782c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 -- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0039e0 msgr2=0x7f782c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0039e0 0x7f782c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7843c00640 1 -- 172.21.15.43:0/766836512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c003520 con 0x7f782c0047c0 2024-01-16T15:12:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.446+0000 7f7850c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0039e0 0x7f782c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:39.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.447+0000 7f7843c00640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f782c0047c0 0x7f782c004c40 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f783400ba30 tx=0x7f782c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f892bf8bf273d394 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7834032640 con 0x7f782c0047c0 2024-01-16T15:12:39.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78340327a0 con 0x7f782c0047c0 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f784be00640 1 -- 172.21.15.43:0/766836512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7834032b30 con 0x7f782c0047c0 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f7852000640 1 -- 172.21.15.43:0/766836512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f782c0047c0 msgr2=0x7f782c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f7852000640 1 --2- 172.21.15.43:0/766836512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f782c0047c0 0x7f782c004c40 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f783400ba30 tx=0x7f782c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f7851600640 1 -- 172.21.15.43:0/766836512 reap_dead start 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f7852000640 1 -- 172.21.15.43:0/766836512 shutdown_connections 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.448+0000 7f7852000640 1 -- 172.21.15.43:0/766836512 >> 172.21.15.43:0/766836512 conn(0x7f784c0d30a0 msgr2=0x7f784c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.449+0000 7f7852000640 1 -- 172.21.15.43:0/766836512 shutdown_connections 2024-01-16T15:12:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.449+0000 7f7852000640 1 -- 172.21.15.43:0/766836512 wait complete. 2024-01-16T15:12:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.450+0000 7f7852000640 1 Processor -- start 2024-01-16T15:12:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.451+0000 7f7852000640 1 -- start start 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.451+0000 7f7852000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 0x7f784c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.451+0000 7f7852000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c157fe0 0x7f784c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.451+0000 7f7852000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7852000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0b7b60 con 0x7f784c0b7ff0 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7852000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c1f2ef0 con 0x7f782c0047c0 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7852000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0cd360 con 0x7f784c157fe0 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56232/0 (socket says 172.21.15.43:56232) 2024-01-16T15:12:39.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 -- 172.21.15.43:0/353523586 learned_addr learned my addr 172.21.15.43:0/353523586 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:39.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7850c00640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 0x7f784c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 -- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c157fe0 msgr2=0x7f784c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7843c00640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c157fe0 0x7f784c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c157fe0 0x7f784c0cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 -- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 msgr2=0x7f784c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 0x7f784c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7851600640 1 -- 172.21.15.43:0/353523586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0b6cb0 con 0x7f784c0b7ff0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.452+0000 7f7850c00640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 0x7f784c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7851600640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f784c08c9e0 tx=0x7f784400d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844007440 con 0x7f784c0b7ff0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78440075a0 con 0x7f784c0b7ff0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844004a20 con 0x7f784c0b7ff0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0b6f50 con 0x7f784c0b7ff0 2024-01-16T15:12:39.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.453+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c0b7510 con 0x7f784c0b7ff0 2024-01-16T15:12:39.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.454+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784c04e840 con 0x7f784c0b7ff0 2024-01-16T15:12:39.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.455+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7844004b80 con 0x7f784c0b7ff0 2024-01-16T15:12:39.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.456+0000 7f7849600640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 0x7f782406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.456+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f7844087ba0 con 0x7f784c0b7ff0 2024-01-16T15:12:39.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.456+0000 7f7850c00640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 0x7f782406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.457+0000 7f7850c00640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 0x7f782406e470 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f784c1f6520 tx=0x7f783c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.460+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7844054460 con 0x7f784c0b7ff0 2024-01-16T15:12:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.681+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96118", "target": ["mon-mgr", ""]}) v1 -- 0x7f784c0cd4d0 con 0x7f782406bfb0 2024-01-16T15:12:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.735+0000 7f7849600640 1 -- 172.21.15.43:0/353523586 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f784c0cd4d0 con 0x7f782406bfb0 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 msgr2=0x7f782406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 0x7f782406e470 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f784c1f6520 tx=0x7f783c00b040 comp rx=0 tx=0).stop 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 msgr2=0x7f784c1464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f784c08c9e0 tx=0x7f784400d900 comp rx=0 tx=0).stop 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 shutdown_connections 2024-01-16T15:12:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f782406bfb0 0x7f782406e470 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f784c0b7ff0 0x7f784c1464a0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f784c157fe0 0x7f784c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 --2- 172.21.15.43:0/353523586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f782c0047c0 0x7f784c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 >> 172.21.15.43:0/353523586 conn(0x7f784c0d30a0 msgr2=0x7f784c06c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.738+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 shutdown_connections 2024-01-16T15:12:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.739+0000 7f7852000640 1 -- 172.21.15.43:0/353523586 wait complete. 2024-01-16T15:12:39.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96117 2024-01-16T15:12:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.990+0000 7fb1d1a00640 1 Processor -- start 2024-01-16T15:12:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.990+0000 7fb1d1a00640 1 -- start start 2024-01-16T15:12:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1d1a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1cc15efa0 0x7fb1cc161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1d1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 0x7fb1cc06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1d1a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1cc06c9a0 con 0x7fb1cc06da40 2024-01-16T15:12:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1d1a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc06cc80 con 0x7fb1cc15efa0 2024-01-16T15:12:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1d1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc06cf60 con 0x7fb1cc0b3080 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1c3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb1cc06da40 0x7fb1cc0b2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41282/0 (socket says 172.21.15.43:41282) 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1c3e00640 1 -- 172.21.15.43:0/793116066 learned_addr learned my addr 172.21.15.43:0/793116066 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.992+0000 7fb1cbe00640 1 --2- 172.21.15.43:0/793116066 >> v2:172.21.15.43:3300/0 conn(0x7fb1cc15efa0 0x7fb1cc161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1d1000640 1 --2- 172.21.15.43:0/793116066 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 0x7fb1cc06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1cb400640 1 -- 172.21.15.43:0/793116066 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858376018 0 0) 0x7fb1cc06c9a0 con 0x7fb1cc06da40 2024-01-16T15:12:39.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1cbe00640 1 -- 172.21.15.43:0/793116066 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 msgr2=0x7fb1cc06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1cbe00640 1 --2- 172.21.15.43:0/793116066 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 0x7fb1cc06c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1cbe00640 1 -- 172.21.15.43:0/793116066 >> v1:172.21.15.43:6789/0 conn(0x7fb1cc06da40 legacy=0x7fb1cc0b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1cbe00640 1 -- 172.21.15.43:0/793116066 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc06d240 con 0x7fb1cc15efa0 2024-01-16T15:12:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.993+0000 7fb1d1000640 1 --2- 172.21.15.43:0/793116066 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 0x7fb1cc06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:39.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1cbe00640 1 --2- 172.21.15.43:0/793116066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc161ac0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0b4390 tx=0x7fb1bc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31838e70a215c42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1cb400640 1 -- 172.21.15.43:0/793116066 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0317c0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1cb400640 1 -- 172.21.15.43:0/793116066 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc031920 con 0x7fb1cc15efa0 2024-01-16T15:12:39.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1cb400640 1 -- 172.21.15.43:0/793116066 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc031cb0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1d1a00640 1 -- 172.21.15.43:0/793116066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 msgr2=0x7fb1cc161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.995+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/793116066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc161ac0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0b4390 tx=0x7fb1bc02b860 comp rx=0 tx=0).stop 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 -- 172.21.15.43:0/793116066 shutdown_connections 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/793116066 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc0b3080 0x7fb1cc06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/793116066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc161ac0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 -- 172.21.15.43:0/793116066 >> 172.21.15.43:0/793116066 conn(0x7fb1cc0cec10 msgr2=0x7fb1cc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 -- 172.21.15.43:0/793116066 shutdown_connections 2024-01-16T15:12:39.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.996+0000 7fb1d1a00640 1 -- 172.21.15.43:0/793116066 wait complete. 2024-01-16T15:12:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.998+0000 7fb1d1a00640 1 Processor -- start 2024-01-16T15:12:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.998+0000 7fb1d1a00640 1 -- start start 2024-01-16T15:12:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc06da40 0x7fb1cc0a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc0b3080 0x7fb1cc1fa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc14b3a0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc14b680 con 0x7fb1cc06da40 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0a9770 con 0x7fb1cc0b3080 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56248/0 (socket says 172.21.15.43:56248) 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 -- 172.21.15.43:0/2312360130 learned_addr learned my addr 172.21.15.43:0/2312360130 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 -- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc0b3080 msgr2=0x7fb1cc1fa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc0b3080 0x7fb1cc1fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 -- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc06da40 msgr2=0x7fb1cc0a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc06da40 0x7fb1cc0a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:39.999+0000 7fb1d1000640 1 -- 172.21.15.43:0/2312360130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc009400 con 0x7fb1cc15efa0 2024-01-16T15:12:39.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1d1000640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400cf20 tx=0x7fb1c400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c400d840 con 0x7fb1cc15efa0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c400d9a0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c400dd50 con 0x7fb1cc15efa0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc0a9a50 con 0x7fb1cc15efa0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.000+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc0a93b0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.001+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc04e840 con 0x7fb1cc15efa0 2024-01-16T15:12:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.002+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb1c401e3b0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.003+0000 7fb1c8c00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 0x7fb1a406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.003+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb1c40890e0 con 0x7fb1cc15efa0 2024-01-16T15:12:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.003+0000 7fb1cbe00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 0x7fb1a406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:39.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.004+0000 7fb1cbe00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 0x7fb1a406e470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc036000 tx=0x7fb1bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.007+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1c40559a0 con 0x7fb1cc15efa0 2024-01-16T15:12:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.230+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96117", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1cc0b4d20 con 0x7fb1a406bfb0 2024-01-16T15:12:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.268+0000 7fb1c8c00640 1 -- 172.21.15.43:0/2312360130 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1bc031b80 con 0x7fb1a406bfb0 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.271+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 msgr2=0x7fb1a406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.271+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 0x7fb1a406e470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc036000 tx=0x7fb1bc030000 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.271+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 msgr2=0x7fb1cc1fac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.271+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400cf20 tx=0x7fb1c400a7d0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 shutdown_connections 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb1a406bfb0 0x7fb1a406e470 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1cc15efa0 0x7fb1cc1fac60 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc0b3080 0x7fb1cc1fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 --2- 172.21.15.43:0/2312360130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc06da40 0x7fb1cc0a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 >> 172.21.15.43:0/2312360130 conn(0x7fb1cc0cec10 msgr2=0x7fb1cc15d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 shutdown_connections 2024-01-16T15:12:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.272+0000 7fb1d1a00640 1 -- 172.21.15.43:0/2312360130 wait complete. 2024-01-16T15:12:40.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9613 2024-01-16T15:12:40.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.503+0000 7fc363e00640 1 Processor -- start 2024-01-16T15:12:40.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.503+0000 7fc363e00640 1 -- start start 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.504+0000 7fc363e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc35c15b350 0x7fc35c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.504+0000 7fc363e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc35c0adb60 0x7fc35c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc363e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c15b730 con 0x7fc35c09cb30 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc363e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c0ac740 con 0x7fc35c0adb60 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc363e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c0ac460 con 0x7fc35c15b350 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc360a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc35c09cb30 0x7fc35c0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41296/0 (socket says 172.21.15.43:41296) 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc360a00640 1 -- 172.21.15.43:0/2524028245 learned_addr learned my addr 172.21.15.43:0/2524028245 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:40.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc361e00640 1 --2- 172.21.15.43:0/2524028245 >> v2:172.21.15.43:3300/0 conn(0x7fc35c0adb60 0x7fc35c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.505+0000 7fc361400640 1 --2- 172.21.15.43:0/2524028245 >> v2:172.21.15.69:3300/0 conn(0x7fc35c15b350 0x7fc35c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.506+0000 7fc361e00640 1 -- 172.21.15.43:0/2524028245 >> v2:172.21.15.69:3300/0 conn(0x7fc35c15b350 msgr2=0x7fc35c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.506+0000 7fc361e00640 1 --2- 172.21.15.43:0/2524028245 >> v2:172.21.15.69:3300/0 conn(0x7fc35c15b350 0x7fc35c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.506+0000 7fc361e00640 1 -- 172.21.15.43:0/2524028245 >> v1:172.21.15.43:6789/0 conn(0x7fc35c09cb30 legacy=0x7fc35c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.506+0000 7fc361e00640 1 -- 172.21.15.43:0/2524028245 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0b2a50 con 0x7fc35c0adb60 2024-01-16T15:12:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.506+0000 7fc35be00640 1 -- 172.21.15.43:0/2524028245 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940083306 0 0) 0x7fc344001110 con 0x7fc35c09cb30 2024-01-16T15:12:40.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.507+0000 7fc361e00640 1 --2- 172.21.15.43:0/2524028245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c0adf70 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0b5100 tx=0x7fc35402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23569b9b88aa34ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.507+0000 7fc35be00640 1 -- 172.21.15.43:0/2524028245 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35402cc80 con 0x7fc35c0adb60 2024-01-16T15:12:40.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.507+0000 7fc35be00640 1 -- 172.21.15.43:0/2524028245 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35402cde0 con 0x7fc35c0adb60 2024-01-16T15:12:40.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.507+0000 7fc35be00640 1 -- 172.21.15.43:0/2524028245 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354034660 con 0x7fc35c0adb60 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 -- 172.21.15.43:0/2524028245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 msgr2=0x7fc35c0adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 --2- 172.21.15.43:0/2524028245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c0adf70 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0b5100 tx=0x7fc35402bed0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 -- 172.21.15.43:0/2524028245 shutdown_connections 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 --2- 172.21.15.43:0/2524028245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c0adf70 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 --2- 172.21.15.43:0/2524028245 >> v2:172.21.15.69:3300/0 conn(0x7fc35c15b350 0x7fc35c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 -- 172.21.15.43:0/2524028245 >> 172.21.15.43:0/2524028245 conn(0x7fc35c0b0ba0 msgr2=0x7fc35c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 -- 172.21.15.43:0/2524028245 shutdown_connections 2024-01-16T15:12:40.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.508+0000 7fc363e00640 1 -- 172.21.15.43:0/2524028245 wait complete. 2024-01-16T15:12:40.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.510+0000 7fc363e00640 1 Processor -- start 2024-01-16T15:12:40.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.510+0000 7fc363e00640 1 -- start start 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc35c09cb30 0x7fc35c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35c15b350 0x7fc35c06dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c159470 con 0x7fc35c0adb60 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c150900 con 0x7fc35c15b350 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc363e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c152310 con 0x7fc35c09cb30 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc360a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc360a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56264/0 (socket says 172.21.15.43:56264) 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc360a00640 1 -- 172.21.15.43:0/55347741 learned_addr learned my addr 172.21.15.43:0/55347741 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:40.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.511+0000 7fc361e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35c15b350 0x7fc35c06dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc361400640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc35c09cb30 0x7fc35c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 -- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc35c09cb30 msgr2=0x7fc35c1502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc35c09cb30 0x7fc35c1502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 -- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35c15b350 msgr2=0x7fc35c06dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35c15b350 0x7fc35c06dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 -- 172.21.15.43:0/55347741 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354009a70 con 0x7fc35c0adb60 2024-01-16T15:12:40.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc360a00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c780 tx=0x7fc34400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440072d0 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344007450 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c1525f0 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440047e0 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.512+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c157960 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.513+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c152b10 con 0x7fc35c0adb60 2024-01-16T15:12:40.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.515+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc344004960 con 0x7fc35c0adb60 2024-01-16T15:12:40.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.515+0000 7fc359600640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 0x7fc33406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.516+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fc344087980 con 0x7fc35c0adb60 2024-01-16T15:12:40.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.516+0000 7fc361400640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 0x7fc33406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.516+0000 7fc361400640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 0x7fc33406e470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc34c006fd0 tx=0x7fc34c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.520+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc35c152b10 con 0x7fc35c0adb60 2024-01-16T15:12:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.738+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9613", "target": ["mon-mgr", ""]}) v1 -- 0x7fc35c06e720 con 0x7fc33406bfb0 2024-01-16T15:12:40.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.775+0000 7fc359600640 1 -- 172.21.15.43:0/55347741 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc34c00f860 con 0x7fc33406bfb0 2024-01-16T15:12:40.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 msgr2=0x7fc33406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 0x7fc33406e470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc34c006fd0 tx=0x7fc34c008040 comp rx=0 tx=0).stop 2024-01-16T15:12:40.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 msgr2=0x7fc35c157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c780 tx=0x7fc34400cc40 comp rx=0 tx=0).stop 2024-01-16T15:12:40.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 shutdown_connections 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc33406bfb0 0x7fc33406e470 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35c15b350 0x7fc35c06dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35c0adb60 0x7fc35c157420 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 --2- 172.21.15.43:0/55347741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc35c09cb30 0x7fc35c1502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 >> 172.21.15.43:0/55347741 conn(0x7fc35c0b0ba0 msgr2=0x7fc35c15a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 shutdown_connections 2024-01-16T15:12:40.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:40.778+0000 7fc363e00640 1 -- 172.21.15.43:0/55347741 wait complete. 2024-01-16T15:12:40.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone9611 2024-01-16T15:12:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.007+0000 7efd43800640 1 Processor -- start 2024-01-16T15:12:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.007+0000 7efd43800640 1 -- start start 2024-01-16T15:12:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.008+0000 7efd43800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 0x7efd3c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.008+0000 7efd43800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0b23e0 con 0x7efd3c0b3210 2024-01-16T15:12:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.008+0000 7efd43800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0d41b0 con 0x7efd3c160bd0 2024-01-16T15:12:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.008+0000 7efd43800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0cdc70 con 0x7efd3c0b48d0 2024-01-16T15:12:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.009+0000 7efd41800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efd3c0b3210 0x7efd3c167290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41300/0 (socket says 172.21.15.43:41300) 2024-01-16T15:12:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.009+0000 7efd41800640 1 -- 172.21.15.43:0/2601539594 learned_addr learned my addr 172.21.15.43:0/2601539594 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.009+0000 7efd33c00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 0x7efd3c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.009+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918452319 0 0) 0x7efd3c0b23e0 con 0x7efd3c0b3210 2024-01-16T15:12:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.009+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1c003520 con 0x7efd3c0b3210 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714989777 0 0) 0x7efd1c003520 con 0x7efd3c0b3210 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 msgr2=0x7efd3c0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 0x7efd3c0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 >> v1:172.21.15.69:6789/0 conn(0x7efd3c160bd0 legacy=0x7efd3c0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c160fb0 con 0x7efd3c0b3210 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7efd340019f0 con 0x7efd3c0b3210 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 >> v1:172.21.15.43:6789/0 conn(0x7efd3c0b3210 legacy=0x7efd3c167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd1c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1c0047c0 0x7efd1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 0x7efd1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0cdc70 con 0x7efd1c0039a0 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0054e0 con 0x7efd1c0047c0 2024-01-16T15:12:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c00bec0 con 0x7efd3c0b48d0 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd1c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 -- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 msgr2=0x7efd1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c0b48d0 0x7efd1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 -- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1c0047c0 msgr2=0x7efd1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd1c0047c0 0x7efd1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.010+0000 7efd40e00640 1 -- 172.21.15.43:0/2601539594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c003520 con 0x7efd1c0039a0 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd40e00640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd1c003de0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efd2c001750 tx=0x7efd1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e6a6fba82250a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0052d0 con 0x7efd1c0039a0 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c02e050 con 0x7efd1c0039a0 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd3be00640 1 -- 172.21.15.43:0/2601539594 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0035f0 con 0x7efd1c0039a0 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd43800640 1 -- 172.21.15.43:0/2601539594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 msgr2=0x7efd1c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd43800640 1 --2- 172.21.15.43:0/2601539594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd1c003de0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efd2c001750 tx=0x7efd1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd41800640 1 -- 172.21.15.43:0/2601539594 reap_dead start 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd43800640 1 -- 172.21.15.43:0/2601539594 shutdown_connections 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.012+0000 7efd43800640 1 -- 172.21.15.43:0/2601539594 >> 172.21.15.43:0/2601539594 conn(0x7efd3c09c5a0 msgr2=0x7efd3c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.013+0000 7efd43800640 1 -- 172.21.15.43:0/2601539594 shutdown_connections 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.013+0000 7efd43800640 1 -- 172.21.15.43:0/2601539594 wait complete. 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.014+0000 7efd43800640 1 Processor -- start 2024-01-16T15:12:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.014+0000 7efd43800640 1 -- start start 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd3c06b470 0x7efd3c14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c06b8f0 0x7efd3c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0cb300 con 0x7efd1c0039a0 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c15ad60 con 0x7efd3c06b470 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd43800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0ca680 con 0x7efd3c06b8f0 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd40e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd40e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56290/0 (socket says 172.21.15.43:56290) 2024-01-16T15:12:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.015+0000 7efd40e00640 1 -- 172.21.15.43:0/2406361690 learned_addr learned my addr 172.21.15.43:0/2406361690 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 -- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c06b8f0 msgr2=0x7efd3c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c06b8f0 0x7efd3c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 -- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd3c06b470 msgr2=0x7efd3c14e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd3c06b470 0x7efd3c14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 -- 172.21.15.43:0/2406361690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0ca960 con 0x7efd1c0039a0 2024-01-16T15:12:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd40e00640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efd2c008fd0 tx=0x7efd2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0076c0 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c007a30 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c0cac40 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.016+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c006c00 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.017+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c0ca310 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.017+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c0a1210 con 0x7efd1c0039a0 2024-01-16T15:12:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.019+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7efd2c0035f0 con 0x7efd1c0039a0 2024-01-16T15:12:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.020+0000 7efd39600640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 0x7efd1406e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.020+0000 7efd33c00640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 0x7efd1406e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.020+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7efd2c0aa680 con 0x7efd1c0039a0 2024-01-16T15:12:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.021+0000 7efd33c00640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 0x7efd1406e4c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd24006ce0 tx=0x7efd24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.024+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd2c076f40 con 0x7efd1c0039a0 2024-01-16T15:12:41.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.246+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone9611", "target": ["mon-mgr", ""]}) v1 -- 0x7efd3c15ec60 con 0x7efd1406c000 2024-01-16T15:12:41.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.289+0000 7efd39600640 1 -- 172.21.15.43:0/2406361690 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efd3c15ec60 con 0x7efd1406c000 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 msgr2=0x7efd1406e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 0x7efd1406e4c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd24006ce0 tx=0x7efd24008040 comp rx=0 tx=0).stop 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 msgr2=0x7efd3c0c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efd2c008fd0 tx=0x7efd2c001120 comp rx=0 tx=0).stop 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 shutdown_connections 2024-01-16T15:12:41.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efd1406c000 0x7efd1406e4c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd3c06b470 0x7efd3c14e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd3c06b8f0 0x7efd3c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 --2- 172.21.15.43:0/2406361690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd1c0039a0 0x7efd3c0c6d70 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.292+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 >> 172.21.15.43:0/2406361690 conn(0x7efd3c09c5a0 msgr2=0x7efd3c0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.293+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 shutdown_connections 2024-01-16T15:12:41.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.293+0000 7efd43800640 1 -- 172.21.15.43:0/2406361690 wait complete. 2024-01-16T15:12:41.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96110 2024-01-16T15:12:41.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.536+0000 7f7eb5c00640 1 Processor -- start 2024-01-16T15:12:41.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.537+0000 7f7eb5c00640 1 -- start start 2024-01-16T15:12:41.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.538+0000 7f7eb5c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7eb0144770 0x7f7eb015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.538+0000 7f7eb5c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7eb015f0a0 0x7f7eb00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.538+0000 7f7eb5c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb015f930 con 0x7f7eb00b35a0 2024-01-16T15:12:41.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.538+0000 7f7eb5c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb00b43b0 con 0x7f7eb0144770 2024-01-16T15:12:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.538+0000 7f7eb5c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb015f510 con 0x7f7eb015f0a0 2024-01-16T15:12:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eafe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7eb0144770 0x7f7eb015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7ea7200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7eb00b35a0 0x7f7eb006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41314/0 (socket says 172.21.15.43:41314) 2024-01-16T15:12:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7ea7200640 1 -- 172.21.15.43:0/1153635580 learned_addr learned my addr 172.21.15.43:0/1153635580 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eb5200640 1 --2- 172.21.15.43:0/1153635580 >> v2:172.21.15.69:3300/0 conn(0x7f7eb015f0a0 0x7f7eb00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eafe00640 1 -- 172.21.15.43:0/1153635580 >> v2:172.21.15.69:3300/0 conn(0x7f7eb015f0a0 msgr2=0x7f7eb00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eafe00640 1 --2- 172.21.15.43:0/1153635580 >> v2:172.21.15.69:3300/0 conn(0x7f7eb015f0a0 0x7f7eb00ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eafe00640 1 -- 172.21.15.43:0/1153635580 >> v1:172.21.15.43:6789/0 conn(0x7f7eb00b35a0 legacy=0x7f7eb006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eafe00640 1 -- 172.21.15.43:0/1153635580 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb00ce9f0 con 0x7f7eb0144770 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.539+0000 7f7eaf400640 1 -- 172.21.15.43:0/1153635580 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908160406 0 0) 0x7f7eb015f930 con 0x7f7eb00b35a0 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.540+0000 7f7eafe00640 1 --2- 172.21.15.43:0/1153635580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015eb60 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00700e0 tx=0x7f7ea002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161fd04b2cc930ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eaf400640 1 -- 172.21.15.43:0/1153635580 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0031830 con 0x7f7eb0144770 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eaf400640 1 -- 172.21.15.43:0/1153635580 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea0031990 con 0x7f7eb0144770 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eaf400640 1 -- 172.21.15.43:0/1153635580 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0031d20 con 0x7f7eb0144770 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 -- 172.21.15.43:0/1153635580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 msgr2=0x7f7eb015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/1153635580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015eb60 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00700e0 tx=0x7f7ea002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 -- 172.21.15.43:0/1153635580 shutdown_connections 2024-01-16T15:12:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/1153635580 >> v2:172.21.15.69:3300/0 conn(0x7f7eb015f0a0 0x7f7eb00ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/1153635580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015eb60 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.541+0000 7f7eb5c00640 1 -- 172.21.15.43:0/1153635580 >> 172.21.15.43:0/1153635580 conn(0x7f7eb00cec10 msgr2=0x7f7eb00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.542+0000 7f7eb5c00640 1 -- 172.21.15.43:0/1153635580 shutdown_connections 2024-01-16T15:12:41.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.542+0000 7f7eb5c00640 1 -- 172.21.15.43:0/1153635580 wait complete. 2024-01-16T15:12:41.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.543+0000 7f7eb5c00640 1 Processor -- start 2024-01-16T15:12:41.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.543+0000 7f7eb5c00640 1 -- start start 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00b35a0 0x7f7eb01f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 0x7f7eb0157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb0141da0 con 0x7f7eb0144770 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb015d2e0 con 0x7f7eb015f0a0 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.544+0000 7f7eb5c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb0141fb0 con 0x7f7eb00b35a0 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56316/0 (socket says 172.21.15.43:56316) 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 -- 172.21.15.43:0/689570166 learned_addr learned my addr 172.21.15.43:0/689570166 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7eafe00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00b35a0 0x7f7eb01f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7eb5200640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 0x7f7eb0157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 -- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00b35a0 msgr2=0x7f7eb01f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00b35a0 0x7f7eb01f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 -- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 msgr2=0x7f7eb0157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 0x7f7eb0157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7ea7200640 1 -- 172.21.15.43:0/689570166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea0009400 con 0x7f7eb0144770 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.545+0000 7f7eb5200640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 0x7f7eb0157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:41.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7ea7200640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800c4d0 tx=0x7f7e9800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e980074e0 con 0x7f7eb0144770 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e98007660 con 0x7f7eb0144770 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98004a20 con 0x7f7eb0144770 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb0142290 con 0x7f7eb0144770 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.546+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb0142850 con 0x7f7eb0144770 2024-01-16T15:12:41.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.547+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb0142a60 con 0x7f7eb0144770 2024-01-16T15:12:41.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.548+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7e98004b80 con 0x7f7eb0144770 2024-01-16T15:12:41.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.549+0000 7f7eacc00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 0x7f7e8806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:41.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.549+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f7e98087870 con 0x7f7eb0144770 2024-01-16T15:12:41.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.550+0000 7f7eafe00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 0x7f7e8806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:41.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.550+0000 7f7eafe00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 0x7f7e8806e470 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0036000 tx=0x7f7ea003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:41.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.554+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e98054070 con 0x7f7eb0144770 2024-01-16T15:12:41.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.850+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96110", "target": ["mon-mgr", ""]}) v1 -- 0x7f7eb0142120 con 0x7f7e8806bfb0 2024-01-16T15:12:41.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.893+0000 7f7eacc00640 1 -- 172.21.15.43:0/689570166 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7eb0142120 con 0x7f7e8806bfb0 2024-01-16T15:12:41.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.896+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 msgr2=0x7f7e8806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.896+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 0x7f7e8806e470 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0036000 tx=0x7f7ea003c040 comp rx=0 tx=0).stop 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.896+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 msgr2=0x7f7eb015c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.896+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800c4d0 tx=0x7f7e9800c990 comp rx=0 tx=0).stop 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 shutdown_connections 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e8806bfb0 0x7f7e8806e470 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb015f0a0 0x7f7eb0157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7eb0144770 0x7f7eb015c7e0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 --2- 172.21.15.43:0/689570166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00b35a0 0x7f7eb01f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:41.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 >> 172.21.15.43:0/689570166 conn(0x7f7eb00cec10 msgr2=0x7f7eb00ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:41.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 shutdown_connections 2024-01-16T15:12:41.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:41.897+0000 7f7eb5c00640 1 -- 172.21.15.43:0/689570166 wait complete. 2024-01-16T15:12:41.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone96119 2024-01-16T15:12:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.128+0000 7f3836e00640 1 Processor -- start 2024-01-16T15:12:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.128+0000 7f3836e00640 1 -- start start 2024-01-16T15:12:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.130+0000 7f3836e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 0x7f38300b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.130+0000 7f3836e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 0x7f38300cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.130+0000 7f3836e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300c9630 con 0x7f383015f7e0 2024-01-16T15:12:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.130+0000 7f3836e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38300cbff0 con 0x7f3830144790 2024-01-16T15:12:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.130+0000 7f3836e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300c9300 con 0x7f38300cb620 2024-01-16T15:12:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.131+0000 7f3836400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 0x7f38300cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.131+0000 7f3835000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f383015f7e0 0x7f3830071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:35580/0 (socket says 172.21.15.43:35580) 2024-01-16T15:12:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.131+0000 7f3835a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 0x7f38300b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.131+0000 7f3835000640 1 -- 172.21.15.43:0/2143554632 learned_addr learned my addr 172.21.15.43:0/2143554632 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f382fe00640 1 -- 172.21.15.43:0/2143554632 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112909639 0 0) 0x7f38300c9630 con 0x7f383015f7e0 2024-01-16T15:12:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f382fe00640 1 -- 172.21.15.43:0/2143554632 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3810003520 con 0x7f383015f7e0 2024-01-16T15:12:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f3836400640 1 -- 172.21.15.43:0/2143554632 >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 msgr2=0x7f38300b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f3836400640 1 --2- 172.21.15.43:0/2143554632 >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 0x7f38300b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f3836400640 1 -- 172.21.15.43:0/2143554632 >> v1:172.21.15.69:6789/0 conn(0x7f383015f7e0 legacy=0x7f3830071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f3836400640 1 -- 172.21.15.43:0/2143554632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830070140 con 0x7f38300cb620 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.132+0000 7f3835a00640 1 --2- 172.21.15.43:0/2143554632 >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 0x7f38300b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.133+0000 7f3836400640 1 --2- 172.21.15.43:0/2143554632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 0x7f38300cbab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38280097a0 tx=0x7f382802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a451953f3ef716f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f382fe00640 1 -- 172.21.15.43:0/2143554632 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828031780 con 0x7f38300cb620 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f382fe00640 1 -- 172.21.15.43:0/2143554632 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38280318e0 con 0x7f38300cb620 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f382fe00640 1 -- 172.21.15.43:0/2143554632 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828031c70 con 0x7f38300cb620 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 -- 172.21.15.43:0/2143554632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 msgr2=0x7f38300cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 --2- 172.21.15.43:0/2143554632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 0x7f38300cbab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38280097a0 tx=0x7f382802b820 comp rx=0 tx=0).stop 2024-01-16T15:12:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 -- 172.21.15.43:0/2143554632 shutdown_connections 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 --2- 172.21.15.43:0/2143554632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38300cb620 0x7f38300cbab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 --2- 172.21.15.43:0/2143554632 >> v2:172.21.15.69:3300/0 conn(0x7f3830144790 0x7f38300b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.134+0000 7f3836e00640 1 -- 172.21.15.43:0/2143554632 >> 172.21.15.43:0/2143554632 conn(0x7f383006aeb0 msgr2=0x7f38300b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.135+0000 7f3836e00640 1 -- 172.21.15.43:0/2143554632 shutdown_connections 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.135+0000 7f3836e00640 1 -- 172.21.15.43:0/2143554632 wait complete. 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.137+0000 7f3836e00640 1 Processor -- start 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.137+0000 7f3836e00640 1 -- start start 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 0x7f38301526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383015f7e0 0x7f383015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383015aca0 con 0x7f3830144790 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383015e5d0 con 0x7f38300cb620 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383015e8b0 con 0x7f383015f7e0 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3835000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3835000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56322/0 (socket says 172.21.15.43:56322) 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3835000640 1 -- 172.21.15.43:0/158822958 learned_addr learned my addr 172.21.15.43:0/158822958 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.138+0000 7f3836400640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383015f7e0 0x7f383015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835a00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 0x7f38301526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 -- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383015f7e0 msgr2=0x7f383015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383015f7e0 0x7f383015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 -- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 msgr2=0x7f38301526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 0x7f38301526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 -- 172.21.15.43:0/158822958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820009a70 con 0x7f3830144790 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835a00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 0x7f38301526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f3835000640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f381800c500 tx=0x7f381800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.139+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818005ce0 con 0x7f3830144790 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.140+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818005e40 con 0x7f3830144790 2024-01-16T15:12:42.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.140+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818007330 con 0x7f3830144790 2024-01-16T15:12:42.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.140+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3828009400 con 0x7f3830144790 2024-01-16T15:12:42.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.140+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383015ee70 con 0x7f3830144790 2024-01-16T15:12:42.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.141+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383013f040 con 0x7f3830144790 2024-01-16T15:12:42.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.142+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f3818007490 con 0x7f3830144790 2024-01-16T15:12:42.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.143+0000 7f382d600640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 0x7f380806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.143+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3818087c80 con 0x7f3830144790 2024-01-16T15:12:42.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.143+0000 7f3835a00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 0x7f380806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.144+0000 7f3835a00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 0x7f380806e4a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fd0 tx=0x7f382000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.147+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f383013f040 con 0x7f3830144790 2024-01-16T15:12:42.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.373+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone96119", "target": ["mon-mgr", ""]}) v1 -- 0x7f3830144b70 con 0x7f380806bfe0 2024-01-16T15:12:42.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.406+0000 7f382d600640 1 -- 172.21.15.43:0/158822958 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3820016420 con 0x7f380806bfe0 2024-01-16T15:12:42.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.408+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 msgr2=0x7f380806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.408+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 0x7f380806e4a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fd0 tx=0x7f382000b040 comp rx=0 tx=0).stop 2024-01-16T15:12:42.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.408+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 msgr2=0x7f383015bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.408+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f381800c500 tx=0x7f381800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 shutdown_connections 2024-01-16T15:12:42.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f380806bfe0 0x7f380806e4a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383015f7e0 0x7f383015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3830144790 0x7f383015bdb0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 --2- 172.21.15.43:0/158822958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38300cb620 0x7f38301526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 >> 172.21.15.43:0/158822958 conn(0x7f383006aeb0 msgr2=0x7f38300ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:42.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 shutdown_connections 2024-01-16T15:12:42.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.409+0000 7f3836e00640 1 -- 172.21.15.43:0/158822958 wait complete. 2024-01-16T15:12:42.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:42.409 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:12:42.479 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:12:42.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:12:42.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.793+0000 7f93ee800640 1 Processor -- start 2024-01-16T15:12:42.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.794+0000 7f93ee800640 1 -- start start 2024-01-16T15:12:42.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93ee800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 0x7f93e8164f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93ee800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93e806e040 0x7f93e81674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93ee800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e80b1ea0 con 0x7f93e8070490 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93ee800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e80b2740 con 0x7f93e806e040 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93ee800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e806d520 con 0x7f93e80abe80 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93e6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 0x7f93e8164f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93e7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f93e8070490 0x7f93e8070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:35592/0 (socket says 172.21.15.43:35592) 2024-01-16T15:12:42.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.799+0000 7f93e7400640 1 -- 172.21.15.43:0/3943615292 learned_addr learned my addr 172.21.15.43:0/3943615292 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:42.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.800+0000 7f93e6a00640 1 -- 172.21.15.43:0/3943615292 >> v2:172.21.15.43:3300/0 conn(0x7f93e806e040 msgr2=0x7f93e81674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:42.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.800+0000 7f93e6a00640 1 --2- 172.21.15.43:0/3943615292 >> v2:172.21.15.43:3300/0 conn(0x7f93e806e040 0x7f93e81674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.800+0000 7f93e6a00640 1 -- 172.21.15.43:0/3943615292 >> v1:172.21.15.69:6789/0 conn(0x7f93e8070490 legacy=0x7f93e8070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.800+0000 7f93e6a00640 1 -- 172.21.15.43:0/3943615292 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80b20f0 con 0x7f93e80abe80 2024-01-16T15:12:42.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.802+0000 7f93e6a00640 1 --2- 172.21.15.43:0/3943615292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 0x7f93e8164f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f93d00097a0 tx=0x7f93d002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92477df0de584be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.802+0000 7f93e6000640 1 -- 172.21.15.43:0/3943615292 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d002d040 con 0x7f93e80abe80 2024-01-16T15:12:42.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.802+0000 7f93e6000640 1 -- 172.21.15.43:0/3943615292 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d0031810 con 0x7f93e80abe80 2024-01-16T15:12:42.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.802+0000 7f93e6000640 1 -- 172.21.15.43:0/3943615292 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0031ba0 con 0x7f93e80abe80 2024-01-16T15:12:42.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.804+0000 7f93ee800640 1 -- 172.21.15.43:0/3943615292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 msgr2=0x7f93e8164f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.804+0000 7f93ee800640 1 --2- 172.21.15.43:0/3943615292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 0x7f93e8164f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f93d00097a0 tx=0x7f93d002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.804+0000 7f93ee800640 1 -- 172.21.15.43:0/3943615292 shutdown_connections 2024-01-16T15:12:42.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.804+0000 7f93ee800640 1 --2- 172.21.15.43:0/3943615292 >> v2:172.21.15.43:3300/0 conn(0x7f93e806e040 0x7f93e81674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.804+0000 7f93ee800640 1 --2- 172.21.15.43:0/3943615292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e80abe80 0x7f93e8164f70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.805+0000 7f93ee800640 1 -- 172.21.15.43:0/3943615292 >> 172.21.15.43:0/3943615292 conn(0x7f93e80b0ba0 msgr2=0x7f93e80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:42.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.805+0000 7f93ee800640 1 -- 172.21.15.43:0/3943615292 shutdown_connections 2024-01-16T15:12:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.806+0000 7f93ee800640 1 -- 172.21.15.43:0/3943615292 wait complete. 2024-01-16T15:12:42.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.808+0000 7f93ee800640 1 Processor -- start 2024-01-16T15:12:42.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.808+0000 7f93ee800640 1 -- start start 2024-01-16T15:12:42.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 0x7f93e8144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 0x7f93e8155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e815def0 con 0x7f93e80abe80 2024-01-16T15:12:42.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e809b5e0 con 0x7f93e806e040 2024-01-16T15:12:42.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.810+0000 7f93ee800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8156880 con 0x7f93e8070490 2024-01-16T15:12:42.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.811+0000 7f93e6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 0x7f93e8155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.811+0000 7f93e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 0x7f93e8144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.811+0000 7f93e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 0x7f93e8144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:58144/0 (socket says 172.21.15.43:58144) 2024-01-16T15:12:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56334/0 (socket says 172.21.15.43:56334) 2024-01-16T15:12:42.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.811+0000 7f93e6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 0x7f93e8155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:60092/0 (socket says 172.21.15.43:60092) 2024-01-16T15:12:42.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 -- 172.21.15.43:0/2484648844 learned_addr learned my addr 172.21.15.43:0/2484648844 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 -- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 msgr2=0x7f93e8155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 0x7f93e8155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 -- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 msgr2=0x7f93e8144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 0x7f93e8144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.812+0000 7f93e7e00640 1 -- 172.21.15.43:0/2484648844 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d0009400 con 0x7f93e80abe80 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.813+0000 7f93e7e00640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b970 tx=0x7f93e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.813+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e000cb40 con 0x7f93e80abe80 2024-01-16T15:12:42.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.814+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e000ccc0 con 0x7f93e80abe80 2024-01-16T15:12:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.814+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0014680 con 0x7f93e80abe80 2024-01-16T15:12:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.814+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e809b830 con 0x7f93e80abe80 2024-01-16T15:12:42.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.814+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e809bdf0 con 0x7f93e80abe80 2024-01-16T15:12:42.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.816+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e815e060 con 0x7f93e80abe80 2024-01-16T15:12:42.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.817+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 29) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f93e001d3b0 con 0x7f93e80abe80 2024-01-16T15:12:42.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.819+0000 7f93df400640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 0x7f93c006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:42.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.819+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f93e00884e0 con 0x7f93e80abe80 2024-01-16T15:12:42.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.820+0000 7f93e7400640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 0x7f93c006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:42.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.822+0000 7f93e7400640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 0x7f93c006e4c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007fd0 tx=0x7f93d8007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:42.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:42.827+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93e815e060 con 0x7f93e80abe80 2024-01-16T15:12:43.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:43.078+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f93e806d1b0 con 0x7f93c006c000 2024-01-16T15:12:43.399 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:12:43.419+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:12:43.502 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:12:46.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.252+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f93e0055260 con 0x7f93e80abe80 2024-01-16T15:12:46.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.471+0000 7f93df400640 1 -- 172.21.15.43:0/2484648844 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f93d8001460 con 0x7f93c006c000 2024-01-16T15:12:46.451 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:12:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.476+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 msgr2=0x7f93c006e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.476+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 0x7f93c006e4c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007fd0 tx=0x7f93d8007e80 comp rx=0 tx=0).stop 2024-01-16T15:12:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.476+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 msgr2=0x7f93e8155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.476+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f93e000b970 tx=0x7f93e000be30 comp rx=0 tx=0).stop 2024-01-16T15:12:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 shutdown_connections 2024-01-16T15:12:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f93c006c000 0x7f93c006e4c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e80abe80 0x7f93e8155fe0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93e8070490 0x7f93e8155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 --2- 172.21.15.43:0/2484648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93e806e040 0x7f93e8144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.477+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 >> 172.21.15.43:0/2484648844 conn(0x7f93e80b0ba0 msgr2=0x7f93e806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.479+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 shutdown_connections 2024-01-16T15:12:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:46.479+0000 7f93ee800640 1 -- 172.21.15.43:0/2484648844 wait complete. 2024-01-16T15:12:46.486 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:12:46.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:12:46.647 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:12:46.775 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:12:46.776 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:12:46.776 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:46.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:12:47.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.099+0000 7f9eb8a00640 1 Processor -- start 2024-01-16T15:12:47.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.099+0000 7f9eb8a00640 1 -- start start 2024-01-16T15:12:47.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9eb00b39a0 0x7f9eb0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9eb00abff0 0x7f9eb0071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb8a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb013da30 con 0x7f9eb015a100 2024-01-16T15:12:47.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb8a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00ac840 con 0x7f9eb00b39a0 2024-01-16T15:12:47.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00ac560 con 0x7f9eb00abff0 2024-01-16T15:12:47.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb5600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9eb00b39a0 0x7f9eb0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb5600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9eb00b39a0 0x7f9eb0071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34080/0 (socket says 172.21.15.43:34080) 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.101+0000 7f9eb5600640 1 -- 172.21.15.43:0/3835476975 learned_addr learned my addr 172.21.15.43:0/3835476975 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.102+0000 7f9eb6a00640 1 --2- 172.21.15.43:0/3835476975 >> v2:172.21.15.69:3300/0 conn(0x7f9eb00abff0 0x7f9eb0071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.102+0000 7f9eb5600640 1 -- 172.21.15.43:0/3835476975 >> v2:172.21.15.69:3300/0 conn(0x7f9eb00abff0 msgr2=0x7f9eb0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.102+0000 7f9eb5600640 1 --2- 172.21.15.43:0/3835476975 >> v2:172.21.15.69:3300/0 conn(0x7f9eb00abff0 0x7f9eb0071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.102+0000 7f9eb5600640 1 -- 172.21.15.43:0/3835476975 >> v1:172.21.15.69:6789/0 conn(0x7f9eb015a100 legacy=0x7f9eb00b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.102+0000 7f9eb5600640 1 -- 172.21.15.43:0/3835476975 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb0070650 con 0x7f9eb00b39a0 2024-01-16T15:12:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.103+0000 7f9eb5600640 1 --2- 172.21.15.43:0/3835476975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb00b39a0 0x7f9eb0071130 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00b3e50 tx=0x7f9e9802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afc3025b98a6eda server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.104+0000 7f9eb4c00640 1 -- 172.21.15.43:0/3835476975 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98030860 con 0x7f9eb00b39a0 2024-01-16T15:12:47.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.104+0000 7f9eb4c00640 1 -- 172.21.15.43:0/3835476975 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e980309c0 con 0x7f9eb00b39a0 2024-01-16T15:12:47.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.104+0000 7f9eb4c00640 1 -- 172.21.15.43:0/3835476975 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98030d50 con 0x7f9eb00b39a0 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.104+0000 7f9eb8a00640 1 -- 172.21.15.43:0/3835476975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb00b39a0 msgr2=0x7f9eb0071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.104+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/3835476975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb00b39a0 0x7f9eb0071130 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00b3e50 tx=0x7f9e9802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 -- 172.21.15.43:0/3835476975 shutdown_connections 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/3835476975 >> v2:172.21.15.69:3300/0 conn(0x7f9eb00abff0 0x7f9eb0071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/3835476975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb00b39a0 0x7f9eb0071130 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 -- 172.21.15.43:0/3835476975 >> 172.21.15.43:0/3835476975 conn(0x7f9eb00b0d80 msgr2=0x7f9eb00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 -- 172.21.15.43:0/3835476975 shutdown_connections 2024-01-16T15:12:47.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.105+0000 7f9eb8a00640 1 -- 172.21.15.43:0/3835476975 wait complete. 2024-01-16T15:12:47.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.107+0000 7f9eb8a00640 1 Processor -- start 2024-01-16T15:12:47.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.107+0000 7f9eb8a00640 1 -- start start 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00abff0 0x7f9eb0097810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b39a0 0x7f9eb00af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb015ad20 con 0x7f9eb015a100 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb014e450 con 0x7f9eb00abff0 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.108+0000 7f9eb8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00982b0 con 0x7f9eb00b39a0 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34094/0 (socket says 172.21.15.43:34094) 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 -- 172.21.15.43:0/953665528 learned_addr learned my addr 172.21.15.43:0/953665528 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6000640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00abff0 0x7f9eb0097810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb5600640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b39a0 0x7f9eb00af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 -- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b39a0 msgr2=0x7f9eb00af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b39a0 0x7f9eb00af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 -- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00abff0 msgr2=0x7f9eb0097810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00abff0 0x7f9eb0097810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.109+0000 7f9eb6a00640 1 -- 172.21.15.43:0/953665528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e980094d0 con 0x7f9eb015a100 2024-01-16T15:12:47.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eb6a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8007e40 tx=0x7f9ea800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8014620 con 0x7f9eb015a100 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea800ee90 con 0x7f9eb015a100 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea80149e0 con 0x7f9eb015a100 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb015d290 con 0x7f9eb015a100 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.110+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb015a660 con 0x7f9eb015a100 2024-01-16T15:12:47.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.111+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb0074770 con 0x7f9eb015a100 2024-01-16T15:12:47.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.113+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9ea8014b40 con 0x7f9eb015a100 2024-01-16T15:12:47.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.114+0000 7f9eae000640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 0x7f9e8806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.114+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9ea8088fe0 con 0x7f9eb015a100 2024-01-16T15:12:47.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.114+0000 7f9eb6000640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 0x7f9e8806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.115+0000 7f9eb6000640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 0x7f9e8806e3e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0098f00 tx=0x7f9ea0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.118+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ea80558a0 con 0x7f9eb015a100 2024-01-16T15:12:47.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.313+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9eb00b1b10 con 0x7f9eb015a100 2024-01-16T15:12:47.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.315+0000 7f9eae000640 1 -- 172.21.15.43:0/953665528 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12982 (secure 0 0 0) 0x7f9ea8058ec0 con 0x7f9eb015a100 2024-01-16T15:12:47.292 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:47.292 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:12:46.443843+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/1602463840":"2024-01-17T15:12:43.419513+0000","172.21.15.69:6832/1602463840":"2024-01-17T15:12:43.419513+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 msgr2=0x7f9e8806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 0x7f9e8806e3e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0098f00 tx=0x7f9ea0009040 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 msgr2=0x7f9eb00afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8007e40 tx=0x7f9ea800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 shutdown_connections 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e8806bf20 0x7f9e8806e3e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9eb015a100 0x7f9eb00afd70 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b39a0 0x7f9eb00af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 --2- 172.21.15.43:0/953665528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00abff0 0x7f9eb0097810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.317+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 >> 172.21.15.43:0/953665528 conn(0x7f9eb00b0d80 msgr2=0x7f9eb0072120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:47.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.318+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 shutdown_connections 2024-01-16T15:12:47.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.318+0000 7f9eb8a00640 1 -- 172.21.15.43:0/953665528 wait complete. 2024-01-16T15:12:47.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:12:47.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:12:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.669+0000 7f26a6e00640 1 Processor -- start 2024-01-16T15:12:47.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.669+0000 7f26a6e00640 1 -- start start 2024-01-16T15:12:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a6e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a006ef30 con 0x7f26a00b4940 2024-01-16T15:12:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00b2e90 con 0x7f26a00b3e70 2024-01-16T15:12:47.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b54b0 con 0x7f26a0143ad0 2024-01-16T15:12:47.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a5a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a5a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:51034/0 (socket says 172.21.15.43:51034) 2024-01-16T15:12:47.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.671+0000 7f26a5a00640 1 -- 172.21.15.43:0/2798894176 learned_addr learned my addr 172.21.15.43:0/2798894176 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:47.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.672+0000 7f26a5a00640 1 -- 172.21.15.43:0/2798894176 >> v1:172.21.15.69:6789/0 conn(0x7f26a00b3e70 legacy=0x7f26a015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.672+0000 7f26a5a00640 1 -- 172.21.15.43:0/2798894176 >> v1:172.21.15.43:6789/0 conn(0x7f26a00b4940 legacy=0x7f26a00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.672+0000 7f269fe00640 1 -- 172.21.15.43:0/2798894176 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622630859 0 0) 0x7f26a00b2e90 con 0x7f26a00b3e70 2024-01-16T15:12:47.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.672+0000 7f26a5a00640 1 -- 172.21.15.43:0/2798894176 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a00b5790 con 0x7f26a0143ad0 2024-01-16T15:12:47.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.674+0000 7f26a5a00640 1 --2- 172.21.15.43:0/2798894176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f269000bb40 tx=0x7f269002e180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2506f74f44c7f0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.676+0000 7f269fe00640 1 -- 172.21.15.43:0/2798894176 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690036440 con 0x7f26a0143ad0 2024-01-16T15:12:47.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.676+0000 7f269fe00640 1 -- 172.21.15.43:0/2798894176 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26900365a0 con 0x7f26a0143ad0 2024-01-16T15:12:47.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.676+0000 7f269fe00640 1 -- 172.21.15.43:0/2798894176 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690036990 con 0x7f26a0143ad0 2024-01-16T15:12:47.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.677+0000 7f26a6e00640 1 -- 172.21.15.43:0/2798894176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 msgr2=0x7f26a00b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.677+0000 7f26a6e00640 1 --2- 172.21.15.43:0/2798894176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f269000bb40 tx=0x7f269002e180 comp rx=0 tx=0).stop 2024-01-16T15:12:47.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.678+0000 7f26a6e00640 1 -- 172.21.15.43:0/2798894176 shutdown_connections 2024-01-16T15:12:47.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.678+0000 7f26a6e00640 1 --2- 172.21.15.43:0/2798894176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a0143ad0 0x7f26a00b3930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.678+0000 7f26a6e00640 1 -- 172.21.15.43:0/2798894176 >> 172.21.15.43:0/2798894176 conn(0x7f26a00cfde0 msgr2=0x7f26a00d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:47.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.678+0000 7f26a6e00640 1 -- 172.21.15.43:0/2798894176 shutdown_connections 2024-01-16T15:12:47.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.678+0000 7f26a6e00640 1 -- 172.21.15.43:0/2798894176 wait complete. 2024-01-16T15:12:47.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.680+0000 7f26a6e00640 1 Processor -- start 2024-01-16T15:12:47.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.680+0000 7f26a6e00640 1 -- start start 2024-01-16T15:12:47.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 0x7f26a0153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 0x7f26a0155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a0143ad0 0x7f26a0157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0153f60 con 0x7f26a00b3e70 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0156380 con 0x7f26a0143ad0 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.682+0000 7f26a6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0156660 con 0x7f26a00b4940 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 0x7f26a0155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 0x7f26a0155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:51042/0 (socket says 172.21.15.43:51042) 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5000640 1 -- 172.21.15.43:0/769288848 learned_addr learned my addr 172.21.15.43:0/769288848 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5a00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 0x7f26a0153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a6400640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a0143ad0 0x7f26a0157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5a00640 1 -- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 msgr2=0x7f26a0155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5a00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 0x7f26a0155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5a00640 1 -- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a0143ad0 msgr2=0x7f26a0157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.683+0000 7f26a5a00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a0143ad0 0x7f26a0157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.684+0000 7f26a5a00640 1 -- 172.21.15.43:0/769288848 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269000b770 con 0x7f26a00b3e70 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.684+0000 7f26a5a00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 0x7f26a0153a20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f2690035000 tx=0x7f26900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.684+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690043070 con 0x7f26a00b3e70 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.684+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690034900 con 0x7f26a00b3e70 2024-01-16T15:12:47.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.684+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690034c90 con 0x7f26a00b3e70 2024-01-16T15:12:47.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.685+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a0156940 con 0x7f26a00b3e70 2024-01-16T15:12:47.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.685+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a0156f00 con 0x7f26a00b3e70 2024-01-16T15:12:47.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.687+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a015c4c0 con 0x7f26a00b3e70 2024-01-16T15:12:47.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.688+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f269003d070 con 0x7f26a00b3e70 2024-01-16T15:12:47.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.689+0000 7f269d600640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 0x7f267806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:47.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.690+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f26900aa6c0 con 0x7f26a00b3e70 2024-01-16T15:12:47.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.691+0000 7f26a5000640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 0x7f267806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:47.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.692+0000 7f26a5000640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 0x7f267806e3e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f26a00cbeb0 tx=0x7f2688009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:47.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.698+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2690076fe0 con 0x7f26a00b3e70 2024-01-16T15:12:47.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.891+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26a004e840 con 0x7f26a00b3e70 2024-01-16T15:12:47.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.892+0000 7f269d600640 1 -- 172.21.15.43:0/769288848 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f269003b0a0 con 0x7f26a00b3e70 2024-01-16T15:12:47.869 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:12:47.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.894+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 msgr2=0x7f267806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 0x7f267806e3e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f26a00cbeb0 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-01-16T15:12:47.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 msgr2=0x7f26a0153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:47.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 0x7f26a0153a20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f2690035000 tx=0x7f26900046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 shutdown_connections 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f267806bf20 0x7f267806e3e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a0143ad0 0x7f26a0157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b4940 0x7f26a0155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 --2- 172.21.15.43:0/769288848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26a00b3e70 0x7f26a0153a20 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 >> 172.21.15.43:0/769288848 conn(0x7f26a00cfde0 msgr2=0x7f26a00d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 shutdown_connections 2024-01-16T15:12:47.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:47.895+0000 7f26a6e00640 1 -- 172.21.15.43:0/769288848 wait complete. 2024-01-16T15:12:47.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:12:48.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.124+0000 7fe621400640 1 Processor -- start 2024-01-16T15:12:48.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.124+0000 7fe621400640 1 -- start start 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe621400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe61c1416f0 0x7fe61c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe621400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 0x7fe61c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe621400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c070bf0 con 0x7fe61c141e80 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe621400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61c06db30 con 0x7fe61c1416f0 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe621400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c06d660 con 0x7fe61c06d230 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe61be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe61c1416f0 0x7fe61c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe61be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe61c1416f0 0x7fe61c15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34114/0 (socket says 172.21.15.43:34114) 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.125+0000 7fe61be00640 1 -- 172.21.15.43:0/4167696829 learned_addr learned my addr 172.21.15.43:0/4167696829 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe620a00640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 0x7fe61c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe61be00640 1 -- 172.21.15.43:0/4167696829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 msgr2=0x7fe61c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe61be00640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 0x7fe61c15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe61be00640 1 -- 172.21.15.43:0/4167696829 >> v1:172.21.15.69:6789/0 conn(0x7fe61c141e80 legacy=0x7fe61c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe61be00640 1 -- 172.21.15.43:0/4167696829 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c06d850 con 0x7fe61c1416f0 2024-01-16T15:12:48.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.126+0000 7fe620a00640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 0x7fe61c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:48.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.127+0000 7fe61be00640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c1416f0 0x7fe61c15ed50 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d4889d1f58274c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.127+0000 7fe61aa00640 1 -- 172.21.15.43:0/4167696829 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c031640 con 0x7fe61c1416f0 2024-01-16T15:12:48.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.127+0000 7fe61aa00640 1 -- 172.21.15.43:0/4167696829 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c0317a0 con 0x7fe61c1416f0 2024-01-16T15:12:48.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.127+0000 7fe61aa00640 1 -- 172.21.15.43:0/4167696829 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c031b30 con 0x7fe61c1416f0 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 -- 172.21.15.43:0/4167696829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c1416f0 msgr2=0x7fe61c15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c1416f0 0x7fe61c15ed50 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 -- 172.21.15.43:0/4167696829 shutdown_connections 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c06d230 0x7fe61c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 --2- 172.21.15.43:0/4167696829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c1416f0 0x7fe61c15ed50 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 -- 172.21.15.43:0/4167696829 >> 172.21.15.43:0/4167696829 conn(0x7fe61c0b0840 msgr2=0x7fe61c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 -- 172.21.15.43:0/4167696829 shutdown_connections 2024-01-16T15:12:48.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.128+0000 7fe621400640 1 -- 172.21.15.43:0/4167696829 wait complete. 2024-01-16T15:12:48.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.130+0000 7fe621400640 1 Processor -- start 2024-01-16T15:12:48.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.130+0000 7fe621400640 1 -- start start 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.130+0000 7fe621400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.130+0000 7fe621400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c1416f0 0x7fe61c1e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe621400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe61c141e80 0x7fe61c0b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe621400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1e5670 con 0x7fe61c06d230 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe621400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1e5950 con 0x7fe61c141e80 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe621400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1e5c30 con 0x7fe61c1416f0 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34116/0 (socket says 172.21.15.43:34116) 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 -- 172.21.15.43:0/3641380494 learned_addr learned my addr 172.21.15.43:0/3641380494 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 -- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c1416f0 msgr2=0x7fe61c1e4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c1416f0 0x7fe61c1e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 -- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe61c141e80 msgr2=0x7fe61c0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe61c141e80 0x7fe61c0b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.131+0000 7fe61be00640 1 -- 172.21.15.43:0/3641380494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c0094d0 con 0x7fe61c06d230 2024-01-16T15:12:48.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe61be00640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c03ac60 con 0x7fe61c06d230 2024-01-16T15:12:48.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c03ade0 con 0x7fe61c06d230 2024-01-16T15:12:48.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c044630 con 0x7fe61c06d230 2024-01-16T15:12:48.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c1e5f10 con 0x7fe61c06d230 2024-01-16T15:12:48.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.132+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c1409f0 con 0x7fe61c06d230 2024-01-16T15:12:48.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.133+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61c0cca10 con 0x7fe61c06d230 2024-01-16T15:12:48.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.134+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe60c02f050 con 0x7fe61c06d230 2024-01-16T15:12:48.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.135+0000 7fe613e00640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 0x7fe5f406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.135+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe60c0aa9e0 con 0x7fe61c06d230 2024-01-16T15:12:48.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.135+0000 7fe61b400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 0x7fe5f406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.136+0000 7fe61b400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 0x7fe5f406e3e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.139+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe60c077020 con 0x7fe61c06d230 2024-01-16T15:12:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.341+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe61c1e6430 con 0x7fe61c06d230 2024-01-16T15:12:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.342+0000 7fe613e00640 1 -- 172.21.15.43:0/3641380494 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v79) v1 ==== 80+0+122 (secure 0 0 0) 0x7fe60c035090 con 0x7fe61c06d230 2024-01-16T15:12:48.320 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6834/1602463840 2024-01-17T15:12:43.419513+0000 2024-01-16T15:12:48.320 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/1602463840 2024-01-17T15:12:43.419513+0000 2024-01-16T15:12:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:12:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.344+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 msgr2=0x7fe5f406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.344+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 0x7fe5f406e3e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604006bc0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.344+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 msgr2=0x7fe61c13f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.344+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 shutdown_connections 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe5f406bf20 0x7fe5f406e3e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe61c141e80 0x7fe61c0b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe61c1416f0 0x7fe61c1e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 --2- 172.21.15.43:0/3641380494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe61c06d230 0x7fe61c13f9f0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 >> 172.21.15.43:0/3641380494 conn(0x7fe61c0b0840 msgr2=0x7fe61c144330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 shutdown_connections 2024-01-16T15:12:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.345+0000 7fe621400640 1 -- 172.21.15.43:0/3641380494 wait complete. 2024-01-16T15:12:48.340 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:12:48.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:48.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:12:48.425 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:12:48.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:48.426 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:12:48.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[61375]: fuse finished with error 0 and tester_r 0 2024-01-16T15:12:48.625 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:12:48.625 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:48.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:12:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.933+0000 7fcf54a00640 1 Processor -- start 2024-01-16T15:12:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.933+0000 7fcf54a00640 1 -- start start 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf500cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf54a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcf5015b3d0 0x7fcf500cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf54a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500cc550 con 0x7fcf5015aff0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf54a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500ce7c0 con 0x7fcf5015b3d0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf54a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500cf140 con 0x7fcf500cfb20 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf4e800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcf5015aff0 0x7fcf500cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45598/0 (socket says 172.21.15.43:45598) 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.935+0000 7fcf4e800640 1 -- 172.21.15.43:0/3561428672 learned_addr learned my addr 172.21.15.43:0/3561428672 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf47e00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf500cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4f200640 1 --2- 172.21.15.43:0/3561428672 >> v2:172.21.15.69:3300/0 conn(0x7fcf5015b3d0 0x7fcf500cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176200267 0 0) 0x7fcf500cc550 con 0x7fcf5015aff0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf30003520 con 0x7fcf5015aff0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1373303727 0 0) 0x7fcf30003520 con 0x7fcf5015aff0 2024-01-16T15:12:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 msgr2=0x7fcf500cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf500cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 >> v2:172.21.15.69:3300/0 conn(0x7fcf5015b3d0 msgr2=0x7fcf500cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 --2- 172.21.15.43:0/3561428672 >> v2:172.21.15.69:3300/0 conn(0x7fcf5015b3d0 0x7fcf500cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500cbc20 con 0x7fcf5015aff0 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcf40001670 con 0x7fcf5015aff0 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 >> v1:172.21.15.43:6789/0 conn(0x7fcf5015aff0 legacy=0x7fcf500cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf300039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf30003f60 0x7fcf300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.936+0000 7fcf4de00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30004d00 0x7fcf30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500ce7c0 con 0x7fcf30003f60 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500cf140 con 0x7fcf30004d00 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3000b9b0 con 0x7fcf500cfb20 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf30003f60 0x7fcf300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf4e800640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf300039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf4f200640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30004d00 0x7fcf30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 -- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 msgr2=0x7fcf300039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf500cfb20 0x7fcf300039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 -- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30004d00 msgr2=0x7fcf30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30004d00 0x7fcf30005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.937+0000 7fcf47e00640 1 -- 172.21.15.43:0/3561428672 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30003520 con 0x7fcf30003f60 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.938+0000 7fcf47e00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf30003f60 0x7fcf300047c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009ac0 tx=0x7fcf300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4abc5f3602c596 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3802c6b0 con 0x7fcf30003f60 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3802c810 con 0x7fcf30003f60 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf4de00640 1 -- 172.21.15.43:0/3561428672 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3802cba0 con 0x7fcf30003f60 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf54a00640 1 -- 172.21.15.43:0/3561428672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf30003f60 msgr2=0x7fcf300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf54a00640 1 --2- 172.21.15.43:0/3561428672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf30003f60 0x7fcf300047c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009ac0 tx=0x7fcf300057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf4f200640 1 -- 172.21.15.43:0/3561428672 reap_dead start 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf54a00640 1 -- 172.21.15.43:0/3561428672 shutdown_connections 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf54a00640 1 -- 172.21.15.43:0/3561428672 >> 172.21.15.43:0/3561428672 conn(0x7fcf500d0e70 msgr2=0x7fcf500d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.939+0000 7fcf54a00640 1 -- 172.21.15.43:0/3561428672 shutdown_connections 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.940+0000 7fcf54a00640 1 -- 172.21.15.43:0/3561428672 wait complete. 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.941+0000 7fcf54a00640 1 Processor -- start 2024-01-16T15:12:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.942+0000 7fcf54a00640 1 -- start start 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30003f60 0x7fcf50147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 0x7fcf50148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50147710 con 0x7fcf50148eb0 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50148bb0 con 0x7fcf30003f60 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf54a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50146a90 con 0x7fcf501481f0 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34122/0 (socket says 172.21.15.43:34122) 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 -- 172.21.15.43:0/198770834 learned_addr learned my addr 172.21.15.43:0/198770834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf4f200640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 0x7fcf50148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 -- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 msgr2=0x7fcf50148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 0x7fcf50148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf4e800640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30003f60 0x7fcf50147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 -- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30003f60 msgr2=0x7fcf50147b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30003f60 0x7fcf50147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf47e00640 1 -- 172.21.15.43:0/198770834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50146d70 con 0x7fcf50148eb0 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.943+0000 7fcf4f200640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 0x7fcf50148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf47e00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fcf3802c250 tx=0x7fcf38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38003810 con 0x7fcf50148eb0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38003970 con 0x7fcf50148eb0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38003ce0 con 0x7fcf50148eb0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf50147050 con 0x7fcf50148eb0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.944+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501466a0 con 0x7fcf50148eb0 2024-01-16T15:12:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.946+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf500c7d40 con 0x7fcf50148eb0 2024-01-16T15:12:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.946+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fcf38047450 con 0x7fcf50148eb0 2024-01-16T15:12:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.947+0000 7fcf46a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 0x7fcf2806e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.947+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcf380aa750 con 0x7fcf50148eb0 2024-01-16T15:12:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.948+0000 7fcf4e800640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 0x7fcf2806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.948+0000 7fcf4e800640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 0x7fcf2806e460 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcf40001050 tx=0x7fcf40000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:48.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:48.952+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf380770c0 con 0x7fcf50148eb0 2024-01-16T15:12:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.143+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf5004e840 con 0x7fcf50148eb0 2024-01-16T15:12:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.144+0000 7fcf46a00640 1 -- 172.21.15.43:0/198770834 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12982 (secure 0 0 0) 0x7fcf380044e0 con 0x7fcf50148eb0 2024-01-16T15:12:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:49.122 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:12:46.443843+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/1602463840":"2024-01-17T15:12:43.419513+0000","172.21.15.69:6832/1602463840":"2024-01-17T15:12:43.419513+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:12:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 msgr2=0x7fcf2806e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 0x7fcf2806e460 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcf40001050 tx=0x7fcf40000ed0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 msgr2=0x7fcf50146160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fcf3802c250 tx=0x7fcf38004730 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 shutdown_connections 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcf2806bfa0 0x7fcf2806e460 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf501481f0 0x7fcf50148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf50148eb0 0x7fcf50146160 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 --2- 172.21.15.43:0/198770834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf30003f60 0x7fcf50147b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 >> 172.21.15.43:0/198770834 conn(0x7fcf500d0e70 msgr2=0x7fcf500b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 shutdown_connections 2024-01-16T15:12:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.147+0000 7fcf54a00640 1 -- 172.21.15.43:0/198770834 wait complete. 2024-01-16T15:12:49.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:12:49.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:12:49.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.385+0000 7f889a000640 1 Processor -- start 2024-01-16T15:12:49.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.386+0000 7f889a000640 1 -- start start 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.387+0000 7f889a000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f889415b350 0x7f88940abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.387+0000 7f889a000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889415b730 con 0x7f889409cb30 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.387+0000 7f889a000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940ac740 con 0x7f88940adb60 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.387+0000 7f889a000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88940ac460 con 0x7f889415b350 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f889415b350 0x7f88940abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f889415b350 0x7f88940abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34124/0 (socket says 172.21.15.43:34124) 2024-01-16T15:12:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 -- 172.21.15.43:0/3499410083 learned_addr learned my addr 172.21.15.43:0/3499410083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 -- 172.21.15.43:0/3499410083 >> v1:172.21.15.69:6789/0 conn(0x7f88940adb60 legacy=0x7f88940adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 -- 172.21.15.43:0/3499410083 >> v1:172.21.15.43:6789/0 conn(0x7f889409cb30 legacy=0x7f88940b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.388+0000 7f8893400640 1 -- 172.21.15.43:0/3499410083 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88940b2a50 con 0x7f889415b350 2024-01-16T15:12:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.389+0000 7f8893400640 1 --2- 172.21.15.43:0/3499410083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88940abe80 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8884009870 tx=0x7f888402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c26e7b10bf9be0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f8892a00640 1 -- 172.21.15.43:0/3499410083 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884030860 con 0x7f889415b350 2024-01-16T15:12:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f8892a00640 1 -- 172.21.15.43:0/3499410083 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88840309c0 con 0x7f889415b350 2024-01-16T15:12:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f8892a00640 1 -- 172.21.15.43:0/3499410083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884030d50 con 0x7f889415b350 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f889a000640 1 -- 172.21.15.43:0/3499410083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 msgr2=0x7f88940abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f889a000640 1 --2- 172.21.15.43:0/3499410083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88940abe80 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8884009870 tx=0x7f888402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.390+0000 7f889a000640 1 -- 172.21.15.43:0/3499410083 shutdown_connections 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.391+0000 7f889a000640 1 --2- 172.21.15.43:0/3499410083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88940abe80 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.391+0000 7f889a000640 1 -- 172.21.15.43:0/3499410083 >> 172.21.15.43:0/3499410083 conn(0x7f88940b0ba0 msgr2=0x7f88940b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.391+0000 7f889a000640 1 -- 172.21.15.43:0/3499410083 shutdown_connections 2024-01-16T15:12:49.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.391+0000 7f889a000640 1 -- 172.21.15.43:0/3499410083 wait complete. 2024-01-16T15:12:49.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.393+0000 7f889a000640 1 Processor -- start 2024-01-16T15:12:49.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.393+0000 7f889a000640 1 -- start start 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.394+0000 7f889a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f889409cb30 0x7f88941500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.394+0000 7f889a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88940adb60 0x7f8894146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.394+0000 7f889a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f889a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88941583c0 con 0x7f889415b350 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f889a000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889414cc90 con 0x7f889409cb30 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f889a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889414cf70 con 0x7f88940adb60 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34134/0 (socket says 172.21.15.43:34134) 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 -- 172.21.15.43:0/1075226252 learned_addr learned my addr 172.21.15.43:0/1075226252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893400640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f889409cb30 0x7f88941500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f888be00640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88940adb60 0x7f8894146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 -- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88940adb60 msgr2=0x7f8894146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88940adb60 0x7f8894146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 -- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f889409cb30 msgr2=0x7f88941500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f889409cb30 0x7f88941500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.395+0000 7f8893e00640 1 -- 172.21.15.43:0/1075226252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840094d0 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f8893e00640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f888c00c780 tx=0x7f888c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c004b60 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c00ce90 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c00a850 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889414d250 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.396+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889414d810 con 0x7f889415b350 2024-01-16T15:12:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.398+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889414da30 con 0x7f889415b350 2024-01-16T15:12:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.399+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f888c00a9b0 con 0x7f889415b350 2024-01-16T15:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.400+0000 7f888b400640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 0x7f886c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.400+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f888c088760 con 0x7f889415b350 2024-01-16T15:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.400+0000 7f8893400640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 0x7f886c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.401+0000 7f8893400640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 0x7f886c06e3e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8884036000 tx=0x7f888403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.405+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f888c0bde90 con 0x7f889415b350 2024-01-16T15:12:49.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.598+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f889409d200 con 0x7f889415b350 2024-01-16T15:12:49.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.599+0000 7f888b400640 1 -- 172.21.15.43:0/1075226252 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f888c055080 con 0x7f889415b350 2024-01-16T15:12:49.577 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 msgr2=0x7f886c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 0x7f886c06e3e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8884036000 tx=0x7f888403b040 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 msgr2=0x7f88941493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f888c00c780 tx=0x7f888c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 shutdown_connections 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f886c06bf20 0x7f886c06e3e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889415b350 0x7f88941493d0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88940adb60 0x7f8894146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 --2- 172.21.15.43:0/1075226252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f889409cb30 0x7f88941500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 >> 172.21.15.43:0/1075226252 conn(0x7f88940b0ba0 msgr2=0x7f88940b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:49.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 shutdown_connections 2024-01-16T15:12:49.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.602+0000 7f889a000640 1 -- 172.21.15.43:0/1075226252 wait complete. 2024-01-16T15:12:49.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:12:49.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.843+0000 7f4883800640 1 Processor -- start 2024-01-16T15:12:49.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.843+0000 7f4883800640 1 -- start start 2024-01-16T15:12:49.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.844+0000 7f4883800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f487c072150 0x7f487c0abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.844+0000 7f4883800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c0ac530 con 0x7f487c15a090 2024-01-16T15:12:49.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.844+0000 7f4883800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c0725c0 con 0x7f487c0b39a0 2024-01-16T15:12:49.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.844+0000 7f4883800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487c0ac760 con 0x7f487c072150 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f487c072150 0x7f487c0abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f487c072150 0x7f487c0abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34150/0 (socket says 172.21.15.43:34150) 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 -- 172.21.15.43:0/3744112146 learned_addr learned my addr 172.21.15.43:0/3744112146 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 -- 172.21.15.43:0/3744112146 >> v1:172.21.15.69:6789/0 conn(0x7f487c0b39a0 legacy=0x7f487c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 -- 172.21.15.43:0/3744112146 >> v1:172.21.15.43:6789/0 conn(0x7f487c15a090 legacy=0x7f487c0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.845+0000 7f4881800640 1 -- 172.21.15.43:0/3744112146 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c15a7f0 con 0x7f487c072150 2024-01-16T15:12:49.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f4881800640 1 --2- 172.21.15.43:0/3744112146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c072150 0x7f487c0abff0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb75dfc601cc60b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f487b400640 1 -- 172.21.15.43:0/3744112146 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874030850 con 0x7f487c072150 2024-01-16T15:12:49.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f487b400640 1 -- 172.21.15.43:0/3744112146 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48740309b0 con 0x7f487c072150 2024-01-16T15:12:49.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f487b400640 1 -- 172.21.15.43:0/3744112146 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874030d40 con 0x7f487c072150 2024-01-16T15:12:49.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f4883800640 1 -- 172.21.15.43:0/3744112146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c072150 msgr2=0x7f487c0abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.847+0000 7f4883800640 1 --2- 172.21.15.43:0/3744112146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c072150 0x7f487c0abff0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402b990 comp rx=0 tx=0).stop 2024-01-16T15:12:49.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.848+0000 7f4883800640 1 -- 172.21.15.43:0/3744112146 shutdown_connections 2024-01-16T15:12:49.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.848+0000 7f4883800640 1 --2- 172.21.15.43:0/3744112146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c072150 0x7f487c0abff0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.848+0000 7f4883800640 1 -- 172.21.15.43:0/3744112146 >> 172.21.15.43:0/3744112146 conn(0x7f487c0b0d80 msgr2=0x7f487c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:49.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.848+0000 7f4883800640 1 -- 172.21.15.43:0/3744112146 shutdown_connections 2024-01-16T15:12:49.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.848+0000 7f4883800640 1 -- 172.21.15.43:0/3744112146 wait complete. 2024-01-16T15:12:49.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.850+0000 7f4883800640 1 Processor -- start 2024-01-16T15:12:49.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.850+0000 7f4883800640 1 -- start start 2024-01-16T15:12:49.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 0x7f487c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 0x7f487c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c1511f0 con 0x7f487c15a090 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c1f3910 con 0x7f487c0b39a0 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4883800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c1f3bf0 con 0x7f487c072150 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.851+0000 7f4881800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34164/0 (socket says 172.21.15.43:34164) 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 -- 172.21.15.43:0/265063980 learned_addr learned my addr 172.21.15.43:0/265063980 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f487be00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 0x7f487c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 -- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 msgr2=0x7f487c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4880e00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 0x7f487c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 0x7f487c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 -- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 msgr2=0x7f487c150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 0x7f487c150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 -- 172.21.15.43:0/265063980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48740094d0 con 0x7f487c15a090 2024-01-16T15:12:49.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4880e00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 0x7f487c15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:49.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f487be00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 0x7f487c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:49.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.852+0000 7f4881800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f487402b9d0 tx=0x7f4874004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.853+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874043400 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.853+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874043580 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.853+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874043930 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.853+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c1f3ed0 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.853+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c1f4490 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.854+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c0cc2c0 con 0x7f487c15a090 2024-01-16T15:12:49.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.855+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f487402e070 con 0x7f487c15a090 2024-01-16T15:12:49.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.856+0000 7f4878c00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 0x7f485406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:49.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.856+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f487403e070 con 0x7f487c15a090 2024-01-16T15:12:49.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.856+0000 7f4880e00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 0x7f485406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:49.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.857+0000 7f4880e00640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 0x7f485406e3e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f487c0b07e0 tx=0x7f4864009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:49.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:49.860+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f487c0cc2c0 con 0x7f487c15a090 2024-01-16T15:12:50.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.069+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f487c0cc2c0 con 0x7f487c15a090 2024-01-16T15:12:50.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.070+0000 7f4878c00640 1 -- 172.21.15.43:0/265063980 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v79) v1 ==== 80+0+122 (secure 0 0 0) 0x7f487c0cc2c0 con 0x7f487c15a090 2024-01-16T15:12:50.047 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6834/1602463840 2024-01-17T15:12:43.419513+0000 2024-01-16T15:12:50.047 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/1602463840 2024-01-17T15:12:43.419513+0000 2024-01-16T15:12:50.048 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 msgr2=0x7f485406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 0x7f485406e3e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f487c0b07e0 tx=0x7f4864009040 comp rx=0 tx=0).stop 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 msgr2=0x7f487c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f487402b9d0 tx=0x7f4874004730 comp rx=0 tx=0).stop 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 shutdown_connections 2024-01-16T15:12:50.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f485406bf20 0x7f485406e3e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f487c0b39a0 0x7f487c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f487c15a090 0x7f487c0ca540 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 --2- 172.21.15.43:0/265063980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f487c072150 0x7f487c15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.072+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 >> 172.21.15.43:0/265063980 conn(0x7f487c0b0d80 msgr2=0x7f487c06c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.073+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 shutdown_connections 2024-01-16T15:12:50.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.073+0000 7f4883800640 1 -- 172.21.15.43:0/265063980 wait complete. 2024-01-16T15:12:50.070 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:12:50.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:50.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:12:50.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44850]: fuse finished with error 0 and tester_r 0 2024-01-16T15:12:50.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:12:50.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:50.169 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:12:50.348 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:12:50.348 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:50.410 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:12:50.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:12:50.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:50.410 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:12:50.465 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:12:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:12:50.466 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:12:50.466 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:50.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:12:50.523 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:12:50.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:12:50.523 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:12:50.575 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:12:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:12:50.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:12:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.827+0000 7ff92be00640 1 Processor -- start 2024-01-16T15:12:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.827+0000 7ff92be00640 1 -- start start 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff92c143570 0x7ff92c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff92c143950 0x7ff92c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c071700 con 0x7ff92c15e6b0 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c15b410 con 0x7ff92c143570 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c15b130 con 0x7ff92c143950 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff92c143570 0x7ff92c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff92c143570 0x7ff92c0cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34178/0 (socket says 172.21.15.43:34178) 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff92c15e6b0 0x7ff92c15a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45614/0 (socket says 172.21.15.43:45614) 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.829+0000 7ff92aa00640 1 -- 172.21.15.43:0/2188155437 learned_addr learned my addr 172.21.15.43:0/2188155437 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.830+0000 7ff923e00640 1 --2- 172.21.15.43:0/2188155437 >> v2:172.21.15.69:3300/0 conn(0x7ff92c143950 0x7ff92c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.830+0000 7ff92aa00640 1 -- 172.21.15.43:0/2188155437 >> v2:172.21.15.69:3300/0 conn(0x7ff92c143950 msgr2=0x7ff92c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.830+0000 7ff92aa00640 1 --2- 172.21.15.43:0/2188155437 >> v2:172.21.15.69:3300/0 conn(0x7ff92c143950 0x7ff92c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.830+0000 7ff92aa00640 1 -- 172.21.15.43:0/2188155437 >> v1:172.21.15.43:6789/0 conn(0x7ff92c15e6b0 legacy=0x7ff92c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.830+0000 7ff92aa00640 1 -- 172.21.15.43:0/2188155437 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c15dbd0 con 0x7ff92c143570 2024-01-16T15:12:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.831+0000 7ff92aa00640 1 --2- 172.21.15.43:0/2188155437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143570 0x7ff92c0cf050 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ff92c070f10 tx=0x7ff91c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dced53af4e73448 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.831+0000 7ff92a000640 1 -- 172.21.15.43:0/2188155437 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c031870 con 0x7ff92c143570 2024-01-16T15:12:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.831+0000 7ff92a000640 1 -- 172.21.15.43:0/2188155437 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c0319d0 con 0x7ff92c143570 2024-01-16T15:12:50.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.831+0000 7ff92a000640 1 -- 172.21.15.43:0/2188155437 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c031d60 con 0x7ff92c143570 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 -- 172.21.15.43:0/2188155437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143570 msgr2=0x7ff92c0cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 --2- 172.21.15.43:0/2188155437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143570 0x7ff92c0cf050 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ff92c070f10 tx=0x7ff91c02b910 comp rx=0 tx=0).stop 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 -- 172.21.15.43:0/2188155437 shutdown_connections 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 --2- 172.21.15.43:0/2188155437 >> v2:172.21.15.69:3300/0 conn(0x7ff92c143950 0x7ff92c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 --2- 172.21.15.43:0/2188155437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143570 0x7ff92c0cf050 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 -- 172.21.15.43:0/2188155437 >> 172.21.15.43:0/2188155437 conn(0x7ff92c06a860 msgr2=0x7ff92c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 -- 172.21.15.43:0/2188155437 shutdown_connections 2024-01-16T15:12:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.832+0000 7ff92be00640 1 -- 172.21.15.43:0/2188155437 wait complete. 2024-01-16T15:12:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.834+0000 7ff92be00640 1 Processor -- start 2024-01-16T15:12:50.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.834+0000 7ff92be00640 1 -- start start 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c143570 0x7ff92c1e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92c15e6b0 0x7ff92c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c1412b0 con 0x7ff92c143950 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c141540 con 0x7ff92c143570 2024-01-16T15:12:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.835+0000 7ff92be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c141820 con 0x7ff92c15e6b0 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34194/0 (socket says 172.21.15.43:34194) 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 -- 172.21.15.43:0/3550045810 learned_addr learned my addr 172.21.15.43:0/3550045810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff92aa00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c143570 0x7ff92c1e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff92b400640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92c15e6b0 0x7ff92c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 -- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92c15e6b0 msgr2=0x7ff92c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92c15e6b0 0x7ff92c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 -- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c143570 msgr2=0x7ff92c1e48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c143570 0x7ff92c1e48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.836+0000 7ff923e00640 1 -- 172.21.15.43:0/3550045810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0094d0 con 0x7ff92c143950 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff923e00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d5d0 tx=0x7ff91400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914004240 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140043a0 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914003830 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c141b00 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.837+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c1420c0 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.838+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c04e840 con 0x7ff92c143950 2024-01-16T15:12:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.839+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff914003990 con 0x7ff92c143950 2024-01-16T15:12:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.840+0000 7ff923400640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 0x7ff90406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.840+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff9140887a0 con 0x7ff92c143950 2024-01-16T15:12:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.840+0000 7ff92aa00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 0x7ff90406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.841+0000 7ff92aa00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 0x7ff90406e3e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff91c036000 tx=0x7ff91c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:50.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:50.845+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff914055060 con 0x7ff92c143950 2024-01-16T15:12:51.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.077+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff92c0b3910 con 0x7ff92c143950 2024-01-16T15:12:51.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.078+0000 7ff923400640 1 -- 172.21.15.43:0/3550045810 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3699 (secure 0 0 0) 0x7ff914058680 con 0x7ff92c143950 2024-01-16T15:12:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:51.056 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5540,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1054183394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1054183394},{"type":"v1","addr":"172.21.15.69:6835","nonce":1054183394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/3255994502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3255994502},{"type":"v1","addr":"172.21.15.43:6836","nonce":3255994502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5546,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1771464625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":1771464625},{"type":"v1","addr":"172.21.15.43:6837","nonce":1771464625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2360943131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2360943131},{"type":"v1","addr":"172.21.15.69:6834","nonce":2360943131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-16T15:12:51.056 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 49 2024-01-16T15:12:51.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 msgr2=0x7ff90406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 0x7ff90406e3e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff91c036000 tx=0x7ff91c03c040 comp rx=0 tx=0).stop 2024-01-16T15:12:51.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 msgr2=0x7ff92c140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d5d0 tx=0x7ff91400da90 comp rx=0 tx=0).stop 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 shutdown_connections 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff90406bf20 0x7ff90406e3e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff92c15e6b0 0x7ff92c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff92c143950 0x7ff92c140d70 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.080+0000 7ff92be00640 1 --2- 172.21.15.43:0/3550045810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c143570 0x7ff92c1e48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.081+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 >> 172.21.15.43:0/3550045810 conn(0x7ff92c06a860 msgr2=0x7ff92c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.081+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 shutdown_connections 2024-01-16T15:12:51.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.081+0000 7ff92be00640 1 -- 172.21.15.43:0/3550045810 wait complete. 2024-01-16T15:12:51.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:12:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.317+0000 7f9100a00640 1 Processor -- start 2024-01-16T15:12:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.317+0000 7f9100a00640 1 -- start start 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f9100a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90fc0ad450 0x7f90fc154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f9100a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc0ce650 con 0x7f90fc153ea0 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f9100a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc15e0a0 con 0x7f90fc070850 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f9100a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc0ad8e0 con 0x7f90fc0ad450 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f90fb400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f90fc153ea0 0x7f90fc15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45626/0 (socket says 172.21.15.43:45626) 2024-01-16T15:12:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.319+0000 7f90fb400640 1 -- 172.21.15.43:0/3338299590 learned_addr learned my addr 172.21.15.43:0/3338299590 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90faa00640 1 --2- 172.21.15.43:0/3338299590 >> v2:172.21.15.69:3300/0 conn(0x7f90fc0ad450 0x7f90fc154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840296563 0 0) 0x7f90fc0ce650 con 0x7f90fc153ea0 2024-01-16T15:12:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90dc003520 con 0x7f90fc153ea0 2024-01-16T15:12:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028311625 0 0) 0x7f90dc003520 con 0x7f90fc153ea0 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 >> v2:172.21.15.69:3300/0 conn(0x7f90fc0ad450 msgr2=0x7f90fc154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 --2- 172.21.15.43:0/3338299590 >> v2:172.21.15.69:3300/0 conn(0x7f90fc0ad450 0x7f90fc154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 >> v1:172.21.15.69:6789/0 conn(0x7f90fc070850 legacy=0x7f90fc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc15e380 con 0x7f90fc153ea0 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f90ec001670 con 0x7f90fc153ea0 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.320+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 >> v1:172.21.15.43:6789/0 conn(0x7f90fc153ea0 legacy=0x7f90fc15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90dc0039e0 0x7f90dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc0047c0 0x7f90dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 0x7f90dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00c050 con 0x7f90dc005180 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00c190 con 0x7f90dc0039e0 2024-01-16T15:12:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00c350 con 0x7f90dc0047c0 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fbe00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 0x7f90dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fb400640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90dc0039e0 0x7f90dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90faa00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc0047c0 0x7f90dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fbe00640 1 -- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc0047c0 msgr2=0x7f90dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.321+0000 7f90fbe00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc0047c0 0x7f90dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.322+0000 7f90fbe00640 1 -- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90dc0039e0 msgr2=0x7f90dc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.322+0000 7f90fbe00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90dc0039e0 0x7f90dc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.322+0000 7f90fbe00640 1 -- 172.21.15.43:0/3338299590 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc003520 con 0x7f90dc005180 2024-01-16T15:12:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.322+0000 7f90fb400640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90dc0039e0 0x7f90dc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:51.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.322+0000 7f90fbe00640 1 -- 172.21.15.43:0/3338299590 reap_dead start 2024-01-16T15:12:51.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f90fbe00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 0x7f90dc00b980 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f90f4001f60 tx=0x7f90dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c3211f0c5d3bb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4003b70 con 0x7f90dc005180 2024-01-16T15:12:51.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4003cf0 con 0x7f90dc005180 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f90fa000640 1 -- 172.21.15.43:0/3338299590 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f40052b0 con 0x7f90dc005180 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f9100a00640 1 -- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 msgr2=0x7f90dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.323+0000 7f9100a00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 0x7f90dc00b980 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f90f4001f60 tx=0x7f90dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.324+0000 7f9100a00640 1 -- 172.21.15.43:0/3338299590 shutdown_connections 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.324+0000 7f9100a00640 1 --2- 172.21.15.43:0/3338299590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90dc005180 0x7f90dc00b980 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.324+0000 7f9100a00640 1 -- 172.21.15.43:0/3338299590 >> 172.21.15.43:0/3338299590 conn(0x7f90fc0cec10 msgr2=0x7f90fc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.324+0000 7f9100a00640 1 -- 172.21.15.43:0/3338299590 shutdown_connections 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.324+0000 7f9100a00640 1 -- 172.21.15.43:0/3338299590 wait complete. 2024-01-16T15:12:51.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.326+0000 7f9100a00640 1 Processor -- start 2024-01-16T15:12:51.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.326+0000 7f9100a00640 1 -- start start 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 0x7f90fc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90fc154280 0x7f90fc1f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc1551b0 con 0x7f90fc154700 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc154bf0 con 0x7f90fc154280 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f9100a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc154ed0 con 0x7f90dc005180 2024-01-16T15:12:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34202/0 (socket says 172.21.15.43:34202) 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 -- 172.21.15.43:0/2139452204 learned_addr learned my addr 172.21.15.43:0/2139452204 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fb400640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 0x7f90fc06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 -- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 msgr2=0x7f90fc06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90faa00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90fc154280 0x7f90fc1f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 0x7f90fc06d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 -- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90fc154280 msgr2=0x7f90fc1f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90fc154280 0x7f90fc1f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fbe00640 1 -- 172.21.15.43:0/2139452204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc1559e0 con 0x7f90fc154700 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.327+0000 7f90fb400640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 0x7f90fc06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f90fbe00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f80 tx=0x7f90f40015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4006d50 con 0x7f90fc154700 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4003b80 con 0x7f90fc154700 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f402d6f0 con 0x7f90fc154700 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc155cc0 con 0x7f90fc154700 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.328+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc06db10 con 0x7f90fc154700 2024-01-16T15:12:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.329+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc06dd00 con 0x7f90fc154700 2024-01-16T15:12:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.330+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f90f402d850 con 0x7f90fc154700 2024-01-16T15:12:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.331+0000 7f90f3400640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 0x7f90d406e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.332+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f90f4031030 con 0x7f90fc154700 2024-01-16T15:12:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.332+0000 7f90fb400640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 0x7f90d406e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.332+0000 7f90fb400640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 0x7f90d406e430 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f90ec001200 tx=0x7f90ec000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.336+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90f4076ab0 con 0x7f90fc154700 2024-01-16T15:12:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.531+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f90fc0b1900 con 0x7f90fc154700 2024-01-16T15:12:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.537+0000 7f90f3400640 1 -- 172.21.15.43:0/2139452204 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7f90f407a0d0 con 0x7f90fc154700 2024-01-16T15:12:51.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.540+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 msgr2=0x7f90d406e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.540+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 0x7f90d406e430 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f90ec001200 tx=0x7f90ec000ed0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.540+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 msgr2=0x7f90fc1f6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.540+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f80 tx=0x7f90f40015b0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 shutdown_connections 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f90d406bf70 0x7f90d406e430 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90fc154280 0x7f90fc1f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90fc154700 0x7f90fc1f6250 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 --2- 172.21.15.43:0/2139452204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90dc005180 0x7f90fc06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 >> 172.21.15.43:0/2139452204 conn(0x7f90fc0cec10 msgr2=0x7f90fc153700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 shutdown_connections 2024-01-16T15:12:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.541+0000 7f9100a00640 1 -- 172.21.15.43:0/2139452204 wait complete. 2024-01-16T15:12:51.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:12:51.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.779+0000 7fc905600640 1 Processor -- start 2024-01-16T15:12:51.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.779+0000 7fc905600640 1 -- start start 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc90015d3b0 0x7fc9000ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9000d0fa0 0x7fc9000d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9000b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000c6bf0 con 0x7fc90015d3b0 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000d1ff0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc905600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000d1430 con 0x7fc9000d1bc0 2024-01-16T15:12:51.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.781+0000 7fc8ffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc90015d3b0 0x7fc9000ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ff400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9000d0fa0 0x7fc9000d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc90015d3b0 0x7fc9000ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34208/0 (socket says 172.21.15.43:34208) 2024-01-16T15:12:51.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 -- 172.21.15.43:0/461947841 learned_addr learned my addr 172.21.15.43:0/461947841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:51.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc904c00640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9000b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 -- 172.21.15.43:0/461947841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 msgr2=0x7fc9000b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9000b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 -- 172.21.15.43:0/461947841 >> v2:172.21.15.69:3300/0 conn(0x7fc9000d0fa0 msgr2=0x7fc9000d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/461947841 >> v2:172.21.15.69:3300/0 conn(0x7fc9000d0fa0 0x7fc9000d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.782+0000 7fc8ffe00640 1 -- 172.21.15.43:0/461947841 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000cedc0 con 0x7fc90015d3b0 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.783+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90015d3b0 0x7fc9000ce6c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc90008c4a0 tx=0x7fc8f002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb4dc5aed0aa474 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc8fea00640 1 -- 172.21.15.43:0/461947841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f00317d0 con 0x7fc90015d3b0 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc8fea00640 1 -- 172.21.15.43:0/461947841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0031930 con 0x7fc90015d3b0 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc8fea00640 1 -- 172.21.15.43:0/461947841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0031cc0 con 0x7fc90015d3b0 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 -- 172.21.15.43:0/461947841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90015d3b0 msgr2=0x7fc9000ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90015d3b0 0x7fc9000ce6c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc90008c4a0 tx=0x7fc8f002b870 comp rx=0 tx=0).stop 2024-01-16T15:12:51.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 -- 172.21.15.43:0/461947841 shutdown_connections 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9000b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 --2- 172.21.15.43:0/461947841 >> v2:172.21.15.69:3300/0 conn(0x7fc9000d0fa0 0x7fc9000d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.784+0000 7fc905600640 1 --2- 172.21.15.43:0/461947841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90015d3b0 0x7fc9000ce6c0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.785+0000 7fc905600640 1 -- 172.21.15.43:0/461947841 >> 172.21.15.43:0/461947841 conn(0x7fc9000d30a0 msgr2=0x7fc9000d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.785+0000 7fc905600640 1 -- 172.21.15.43:0/461947841 shutdown_connections 2024-01-16T15:12:51.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.785+0000 7fc905600640 1 -- 172.21.15.43:0/461947841 wait complete. 2024-01-16T15:12:51.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.786+0000 7fc905600640 1 Processor -- start 2024-01-16T15:12:51.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.787+0000 7fc905600640 1 -- start start 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9001485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc90015d3b0 0x7fc90015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900148350 con 0x7fc9000d0fa0 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90015b1a0 con 0x7fc90015d3b0 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc905600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000b6e50 con 0x7fc9000d1bc0 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34220/0 (socket says 172.21.15.43:34220) 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 -- 172.21.15.43:0/935566458 learned_addr learned my addr 172.21.15.43:0/935566458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:51.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ff400640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9001485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 -- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 msgr2=0x7fc9001485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9001485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 -- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc90015d3b0 msgr2=0x7fc90015a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc90015d3b0 0x7fc90015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ffe00640 1 -- 172.21.15.43:0/935566458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00094d0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.788+0000 7fc8ff400640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9001485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc8ffe00640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc8f002b8b0 tx=0x7fc8f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0032660 con 0x7fc9000d0fa0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f00327c0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0032b70 con 0x7fc9000d0fa0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000b70f0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.789+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9000b76b0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.791+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90004e840 con 0x7fc9000d0fa0 2024-01-16T15:12:51.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.792+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc8f0030070 con 0x7fc9000d0fa0 2024-01-16T15:12:51.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.793+0000 7fc8f7e00640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 0x7fc8d806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.793+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc8f00aa7a0 con 0x7fc9000d0fa0 2024-01-16T15:12:51.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.793+0000 7fc8ff400640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 0x7fc8d806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.794+0000 7fc8ff400640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 0x7fc8d806e3e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc9000d1b00 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.798+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8f0076e60 con 0x7fc9000d0fa0 2024-01-16T15:12:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.992+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc900160600 con 0x7fc9000d0fa0 2024-01-16T15:12:51.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:51.999+0000 7fc8f7e00640 1 -- 172.21.15.43:0/935566458 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc8f007a480 con 0x7fc9000d0fa0 2024-01-16T15:12:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 msgr2=0x7fc8d806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 0x7fc8d806e3e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc9000d1b00 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2024-01-16T15:12:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 msgr2=0x7fc900147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc8f002b8b0 tx=0x7fc8f00046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 shutdown_connections 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc8d806bf20 0x7fc8d806e3e0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc90015d3b0 0x7fc90015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9000d1bc0 0x7fc9001485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 --2- 172.21.15.43:0/935566458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9000d0fa0 0x7fc900147e10 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 >> 172.21.15.43:0/935566458 conn(0x7fc9000d30a0 msgr2=0x7fc9000c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 shutdown_connections 2024-01-16T15:12:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.002+0000 7fc905600640 1 -- 172.21.15.43:0/935566458 wait complete. 2024-01-16T15:12:52.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:12:52.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.236+0000 7fb4e3400640 1 Processor -- start 2024-01-16T15:12:52.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.236+0000 7fb4e3400640 1 -- start start 2024-01-16T15:12:52.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb4dc06d520 0x7fb4dc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0abff0 0x7fb4dc15f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc0b41a0 con 0x7fb4dc0ad720 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc15c360 con 0x7fb4dc06d520 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc06de40 con 0x7fb4dc0abff0 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4dbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4dbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34236/0 (socket says 172.21.15.43:34236) 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1213687698 learned_addr learned my addr 172.21.15.43:0/1213687698 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e0a00640 1 --2- 172.21.15.43:0/1213687698 >> v2:172.21.15.69:3300/0 conn(0x7fb4dc06d520 0x7fb4dc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.238+0000 7fb4e1400640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0abff0 0x7fb4dc15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.239+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1213687698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0abff0 msgr2=0x7fb4dc15f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.239+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0abff0 0x7fb4dc15f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.239+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1213687698 >> v2:172.21.15.69:3300/0 conn(0x7fb4dc06d520 msgr2=0x7fb4dc06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:52.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.239+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1213687698 >> v2:172.21.15.69:3300/0 conn(0x7fb4dc06d520 0x7fb4dc06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.239+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1213687698 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc15ba30 con 0x7fb4dc0ad720 2024-01-16T15:12:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.240+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc06c070 tx=0x7fb4c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931873b5084973bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4db400640 1 -- 172.21.15.43:0/1213687698 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4031870 con 0x7fb4dc0ad720 2024-01-16T15:12:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4db400640 1 -- 172.21.15.43:0/1213687698 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c40319d0 con 0x7fb4dc0ad720 2024-01-16T15:12:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4db400640 1 -- 172.21.15.43:0/1213687698 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4031d60 con 0x7fb4dc0ad720 2024-01-16T15:12:52.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 -- 172.21.15.43:0/1213687698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0ad720 msgr2=0x7fb4dc15ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc06c070 tx=0x7fb4c402b910 comp rx=0 tx=0).stop 2024-01-16T15:12:52.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 -- 172.21.15.43:0/1213687698 shutdown_connections 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0abff0 0x7fb4dc15f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1213687698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0ad720 0x7fb4dc15ecd0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1213687698 >> v2:172.21.15.69:3300/0 conn(0x7fb4dc06d520 0x7fb4dc06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.241+0000 7fb4e3400640 1 -- 172.21.15.43:0/1213687698 >> 172.21.15.43:0/1213687698 conn(0x7fb4dc0b0d80 msgr2=0x7fb4dc0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.242+0000 7fb4e3400640 1 -- 172.21.15.43:0/1213687698 shutdown_connections 2024-01-16T15:12:52.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.242+0000 7fb4e3400640 1 -- 172.21.15.43:0/1213687698 wait complete. 2024-01-16T15:12:52.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.243+0000 7fb4e3400640 1 Processor -- start 2024-01-16T15:12:52.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.244+0000 7fb4e3400640 1 -- start start 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc06d520 0x7fb4dc141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0ad720 0x7fb4dc0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc13fea0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc140740 con 0x7fb4dc06d520 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc143390 con 0x7fb4dc0ad720 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34238/0 (socket says 172.21.15.43:34238) 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1043363127 learned_addr learned my addr 172.21.15.43:0/1043363127 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:52.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0ad720 msgr2=0x7fb4dc0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0ad720 0x7fb4dc0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc06d520 msgr2=0x7fb4dc141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc06d520 0x7fb4dc141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.245+0000 7fb4dbe00640 1 -- 172.21.15.43:0/1043363127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c40094d0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4dbe00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b950 tx=0x7fb4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c403ad30 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c40443f0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40447a0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0cd690 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.246+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc1403d0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.247+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc074770 con 0x7fb4dc0abff0 2024-01-16T15:12:52.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.248+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb4c404d450 con 0x7fb4dc0abff0 2024-01-16T15:12:52.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.250+0000 7fb4d8c00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 0x7fb4b406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.250+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb4c402d040 con 0x7fb4dc0abff0 2024-01-16T15:12:52.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.250+0000 7fb4e0a00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 0x7fb4b406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.251+0000 7fb4e0a00640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 0x7fb4b406e410 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009540 tx=0x7fb4cc009450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.254+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4c40773d0 con 0x7fb4dc0abff0 2024-01-16T15:12:52.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:52.517+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7fb4dc0b3bb0 con 0x7fb4dc0abff0 2024-01-16T15:12:53.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.505+0000 7fb4d8c00640 1 -- 172.21.15.43:0/1043363127 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v589) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb4c4035070 con 0x7fb4dc0abff0 2024-01-16T15:12:53.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.508+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 msgr2=0x7fb4b406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.508+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 0x7fb4b406e410 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009540 tx=0x7fb4cc009450 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.508+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 msgr2=0x7fb4dc142470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.508+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b950 tx=0x7fb4c40046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 shutdown_connections 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb4b406bf50 0x7fb4b406e410 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4dc0ad720 0x7fb4dc0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4dc0abff0 0x7fb4dc142470 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 --2- 172.21.15.43:0/1043363127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4dc06d520 0x7fb4dc141f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 >> 172.21.15.43:0/1043363127 conn(0x7fb4dc0b0d80 msgr2=0x7fb4dc0acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 shutdown_connections 2024-01-16T15:12:53.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.509+0000 7fb4e3400640 1 -- 172.21.15.43:0/1043363127 wait complete. 2024-01-16T15:12:53.509 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:12:53.509 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:12:53.509 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:12:53.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:12:53.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.752+0000 7fb52fe00640 1 Processor -- start 2024-01-16T15:12:53.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.753+0000 7fb52fe00640 1 -- start start 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb52fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5300ce010 0x7fb53015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb52fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5300cfbc0 con 0x7fb53006b550 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb52fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5300b0450 con 0x7fb5300cdc30 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb52fe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5300cfe10 con 0x7fb5300ce010 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb527e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5300ce010 0x7fb53015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb527e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb5300ce010 0x7fb53015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59108/0 (socket says 172.21.15.43:59108) 2024-01-16T15:12:53.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.754+0000 7fb527e00640 1 -- 172.21.15.43:0/1028612793 learned_addr learned my addr 172.21.15.43:0/1028612793 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:53.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.755+0000 7fb527e00640 1 -- 172.21.15.43:0/1028612793 >> v1:172.21.15.69:6789/0 conn(0x7fb5300cdc30 legacy=0x7fb53015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:53.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.755+0000 7fb527e00640 1 -- 172.21.15.43:0/1028612793 >> v1:172.21.15.43:6789/0 conn(0x7fb53006b550 legacy=0x7fb53006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.755+0000 7fb527e00640 1 -- 172.21.15.43:0/1028612793 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300d00f0 con 0x7fb5300ce010 2024-01-16T15:12:53.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.756+0000 7fb527e00640 1 --2- 172.21.15.43:0/1028612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300ce010 0x7fb53015d1f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb518009870 tx=0x7fb51802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6020d6a29754dd52 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.756+0000 7fb52e000640 1 -- 172.21.15.43:0/1028612793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518031870 con 0x7fb5300ce010 2024-01-16T15:12:53.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.756+0000 7fb52e000640 1 -- 172.21.15.43:0/1028612793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5180319d0 con 0x7fb5300ce010 2024-01-16T15:12:53.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.756+0000 7fb52e000640 1 -- 172.21.15.43:0/1028612793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518031d60 con 0x7fb5300ce010 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 -- 172.21.15.43:0/1028612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300ce010 msgr2=0x7fb53015d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1028612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300ce010 0x7fb53015d1f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb518009870 tx=0x7fb51802b910 comp rx=0 tx=0).stop 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 -- 172.21.15.43:0/1028612793 shutdown_connections 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1028612793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300ce010 0x7fb53015d1f0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 -- 172.21.15.43:0/1028612793 >> 172.21.15.43:0/1028612793 conn(0x7fb5300d0e70 msgr2=0x7fb5300d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 -- 172.21.15.43:0/1028612793 shutdown_connections 2024-01-16T15:12:53.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.757+0000 7fb52fe00640 1 -- 172.21.15.43:0/1028612793 wait complete. 2024-01-16T15:12:53.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.759+0000 7fb52fe00640 1 Processor -- start 2024-01-16T15:12:53.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.759+0000 7fb52fe00640 1 -- start start 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53006b550 0x7fb5300af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 0x7fb5300c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301581d0 con 0x7fb5300cdc30 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530158790 con 0x7fb53006b550 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb52fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb530157890 con 0x7fb5300ce010 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.760+0000 7fb527e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59110/0 (socket says 172.21.15.43:59110) 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 -- 172.21.15.43:0/1063270195 learned_addr learned my addr 172.21.15.43:0/1063270195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:53.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb52ea00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53006b550 0x7fb5300af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb52f400640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 0x7fb5300c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 -- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 msgr2=0x7fb5300c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 0x7fb5300c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 -- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53006b550 msgr2=0x7fb5300af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53006b550 0x7fb5300af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 -- 172.21.15.43:0/1063270195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5180094d0 con 0x7fb5300cdc30 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb52f400640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 0x7fb5300c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:53.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.761+0000 7fb527e00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb518000c00 tx=0x7fb5180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.762+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518031820 con 0x7fb5300cdc30 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.762+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5180443f0 con 0x7fb5300cdc30 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.762+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180447a0 con 0x7fb5300cdc30 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.762+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5300afcf0 con 0x7fb5300cdc30 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.762+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb530158420 con 0x7fb5300cdc30 2024-01-16T15:12:53.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.763+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb530158920 con 0x7fb5300cdc30 2024-01-16T15:12:53.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.764+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb51804d450 con 0x7fb5300cdc30 2024-01-16T15:12:53.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.765+0000 7fb526a00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 0x7fb50806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.765+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb51802d040 con 0x7fb5300cdc30 2024-01-16T15:12:53.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.765+0000 7fb52ea00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 0x7fb50806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.766+0000 7fb52ea00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 0x7fb50806e3e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb520004ea0 tx=0x7fb520009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.770+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5180775c0 con 0x7fb5300cdc30 2024-01-16T15:12:53.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.968+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb53015d6b0 con 0x7fb5300cdc30 2024-01-16T15:12:53.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.969+0000 7fb526a00640 1 -- 172.21.15.43:0/1063270195 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12982 (secure 0 0 0) 0x7fb518035070 con 0x7fb5300cdc30 2024-01-16T15:12:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:53.947 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:12:46.443843+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/1602463840":"2024-01-17T15:12:43.419513+0000","172.21.15.69:6832/1602463840":"2024-01-17T15:12:43.419513+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:12:53.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 msgr2=0x7fb50806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 0x7fb50806e3e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb520004ea0 tx=0x7fb520009040 comp rx=0 tx=0).stop 2024-01-16T15:12:53.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 msgr2=0x7fb530157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb518000c00 tx=0x7fb5180046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 shutdown_connections 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb50806bf20 0x7fb50806e3e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5300ce010 0x7fb5300c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5300cdc30 0x7fb530157350 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 --2- 172.21.15.43:0/1063270195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb53006b550 0x7fb5300af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.972+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 >> 172.21.15.43:0/1063270195 conn(0x7fb5300d0e70 msgr2=0x7fb5300d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.973+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 shutdown_connections 2024-01-16T15:12:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:53.973+0000 7fb52fe00640 1 -- 172.21.15.43:0/1063270195 wait complete. 2024-01-16T15:12:53.969 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:12:53.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:12:54.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.209+0000 7fd687000640 1 Processor -- start 2024-01-16T15:12:54.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.210+0000 7fd687000640 1 -- start start 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.211+0000 7fd687000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.211+0000 7fd687000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6800d0fa0 0x7fd6800d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.211+0000 7fd687000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800d2010 con 0x7fd6800d1bc0 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.211+0000 7fd687000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6800c6bf0 con 0x7fd6800d0fa0 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.211+0000 7fd687000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800d1400 con 0x7fd68015d3b0 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd686600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd6800d1bc0 0x7fd6800b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54628/0 (socket says 172.21.15.43:54628) 2024-01-16T15:12:54.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd686600640 1 -- 172.21.15.43:0/1757633908 learned_addr learned my addr 172.21.15.43:0/1757633908 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:54.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd685c00640 1 --2- 172.21.15.43:0/1757633908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd685200640 1 --2- 172.21.15.43:0/1757633908 >> v2:172.21.15.69:3300/0 conn(0x7fd6800d0fa0 0x7fd6800d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112728322 0 0) 0x7fd678001490 con 0x7fd6800d1bc0 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.212+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd660003520 con 0x7fd6800d1bc0 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.213+0000 7fd685c00640 1 -- 172.21.15.43:0/1757633908 >> v2:172.21.15.69:3300/0 conn(0x7fd6800d0fa0 msgr2=0x7fd6800d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.213+0000 7fd685c00640 1 --2- 172.21.15.43:0/1757633908 >> v2:172.21.15.69:3300/0 conn(0x7fd6800d0fa0 0x7fd6800d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.213+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18036975 0 0) 0x7fd660003520 con 0x7fd6800d1bc0 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.213+0000 7fd685c00640 1 -- 172.21.15.43:0/1757633908 >> v1:172.21.15.43:6789/0 conn(0x7fd6800d1bc0 legacy=0x7fd6800b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.213+0000 7fd685c00640 1 -- 172.21.15.43:0/1757633908 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800cedc0 con 0x7fd68015d3b0 2024-01-16T15:12:54.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.214+0000 7fd685c00640 1 --2- 172.21.15.43:0/1757633908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800ce6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4078f38888e5b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678001b80 con 0x7fd68015d3b0 2024-01-16T15:12:54.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67002e050 con 0x7fd68015d3b0 2024-01-16T15:12:54.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd67fe00640 1 -- 172.21.15.43:0/1757633908 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670030730 con 0x7fd68015d3b0 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 -- 172.21.15.43:0/1757633908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 msgr2=0x7fd6800ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 --2- 172.21.15.43:0/1757633908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800ce6c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002b710 comp rx=0 tx=0).stop 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 -- 172.21.15.43:0/1757633908 shutdown_connections 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 --2- 172.21.15.43:0/1757633908 >> v2:172.21.15.69:3300/0 conn(0x7fd6800d0fa0 0x7fd6800d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 --2- 172.21.15.43:0/1757633908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800ce6c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 -- 172.21.15.43:0/1757633908 >> 172.21.15.43:0/1757633908 conn(0x7fd6800d30a0 msgr2=0x7fd6800d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.215+0000 7fd687000640 1 -- 172.21.15.43:0/1757633908 shutdown_connections 2024-01-16T15:12:54.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.216+0000 7fd687000640 1 -- 172.21.15.43:0/1757633908 wait complete. 2024-01-16T15:12:54.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.217+0000 7fd687000640 1 Processor -- start 2024-01-16T15:12:54.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.217+0000 7fd687000640 1 -- start start 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 0x7fd6801534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800a0b10 con 0x7fd6800d1bc0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b8c20 con 0x7fd6800d0fa0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.218+0000 7fd687000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b8940 con 0x7fd68015d3b0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59120/0 (socket says 172.21.15.43:59120) 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 -- 172.21.15.43:0/2900201088 learned_addr learned my addr 172.21.15.43:0/2900201088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685c00640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 0x7fd6801534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd686600640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 -- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 msgr2=0x7fd6800cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 -- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 msgr2=0x7fd6801534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 0x7fd6801534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685200640 1 -- 172.21.15.43:0/2900201088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd670009400 con 0x7fd6800d1bc0 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.219+0000 7fd685c00640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 0x7fd6801534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd685200640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd66800e870 tx=0x7fd66800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680094f0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66800c940 con 0x7fd6800d1bc0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66800ccf0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68009cf10 con 0x7fd6800d1bc0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.220+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68009d4d0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.221+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68004e840 con 0x7fd6800d1bc0 2024-01-16T15:12:54.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.222+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fd668013470 con 0x7fd6800d1bc0 2024-01-16T15:12:54.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.223+0000 7fd67d600640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 0x7fd65806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.223+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd6680889e0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.224+0000 7fd685c00640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 0x7fd65806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.224+0000 7fd685c00640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 0x7fd65806e3e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd670008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.228+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd668054fc0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.420+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6800b8f00 con 0x7fd6800d1bc0 2024-01-16T15:12:54.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.421+0000 7fd67d600640 1 -- 172.21.15.43:0/2900201088 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6680585e0 con 0x7fd6800d1bc0 2024-01-16T15:12:54.399 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:12:54.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 msgr2=0x7fd65806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 0x7fd65806e3e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd670008030 comp rx=0 tx=0).stop 2024-01-16T15:12:54.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 msgr2=0x7fd6800cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd66800e870 tx=0x7fd66800ed30 comp rx=0 tx=0).stop 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 shutdown_connections 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd65806bf20 0x7fd65806e3e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd68015d3b0 0x7fd6800cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6800d1bc0 0x7fd6800cc4d0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 --2- 172.21.15.43:0/2900201088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6800d0fa0 0x7fd6801534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.423+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 >> 172.21.15.43:0/2900201088 conn(0x7fd6800d30a0 msgr2=0x7fd6800c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.424+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 shutdown_connections 2024-01-16T15:12:54.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.424+0000 7fd687000640 1 -- 172.21.15.43:0/2900201088 wait complete. 2024-01-16T15:12:54.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:12:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.659+0000 7f2d31600640 1 Processor -- start 2024-01-16T15:12:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.659+0000 7f2d31600640 1 -- start start 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.661+0000 7f2d31600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d2c06d520 0x7f2d2c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.661+0000 7f2d31600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c15c360 con 0x7f2d2c0abff0 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.661+0000 7f2d31600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c15faf0 con 0x7f2d2c0ad720 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.661+0000 7f2d31600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b41a0 con 0x7f2d2c06d520 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.661+0000 7f2d2b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2d2c0abff0 0x7f2d2c15f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54642/0 (socket says 172.21.15.43:54642) 2024-01-16T15:12:54.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d2b400640 1 -- 172.21.15.43:0/4279990720 learned_addr learned my addr 172.21.15.43:0/4279990720 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/4279990720 >> v2:172.21.15.69:3300/0 conn(0x7f2d2c06d520 0x7f2d2c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091405645 0 0) 0x7f2d2c15c360 con 0x7f2d2c0abff0 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c003520 con 0x7f2d2c0abff0 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2177609130 0 0) 0x7f2d0c003520 con 0x7f2d2c0abff0 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 >> v2:172.21.15.69:3300/0 conn(0x7f2d2c06d520 msgr2=0x7f2d2c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 --2- 172.21.15.43:0/4279990720 >> v2:172.21.15.69:3300/0 conn(0x7f2d2c06d520 0x7f2d2c06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 >> v1:172.21.15.69:6789/0 conn(0x7f2d2c0ad720 legacy=0x7f2d2c15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c06de40 con 0x7f2d2c0abff0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f2d240019f0 con 0x7f2d2c0abff0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.662+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 >> v1:172.21.15.43:6789/0 conn(0x7f2d2c0abff0 legacy=0x7f2d2c15f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d0c0039c0 0x7f2d0c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 0x7f2d0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c005180 0x7f2d0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c00c050 con 0x7f2d0c0047c0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c00c1c0 con 0x7f2d0c005180 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c00c380 con 0x7f2d0c0039c0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 0x7f2d0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d0c0039c0 0x7f2d0c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2b400640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c005180 0x7f2d0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 -- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d0c0039c0 msgr2=0x7f2d0c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d0c0039c0 0x7f2d0c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 -- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c005180 msgr2=0x7f2d0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c005180 0x7f2d0c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2a000640 1 -- 172.21.15.43:0/4279990720 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c003520 con 0x7f2d0c0047c0 2024-01-16T15:12:54.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d0c0039c0 0x7f2d0c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:54.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2b400640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c005180 0x7f2d0c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:54.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.663+0000 7f2d2b400640 1 -- 172.21.15.43:0/4279990720 reap_dead start 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d2a000640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 0x7f2d0c004c40 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c070290 tx=0x7f2d0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e52d6bf87dc068 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d140052d0 con 0x7f2d0c0047c0 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1402e050 con 0x7f2d0c0047c0 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d29600640 1 -- 172.21.15.43:0/4279990720 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14003830 con 0x7f2d0c0047c0 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d31600640 1 -- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 msgr2=0x7f2d0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.665+0000 7f2d31600640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 0x7f2d0c004c40 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c070290 tx=0x7f2d0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.666+0000 7f2d31600640 1 -- 172.21.15.43:0/4279990720 shutdown_connections 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.666+0000 7f2d31600640 1 --2- 172.21.15.43:0/4279990720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d0c0047c0 0x7f2d0c004c40 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.666+0000 7f2d31600640 1 -- 172.21.15.43:0/4279990720 >> 172.21.15.43:0/4279990720 conn(0x7f2d2c0b0d80 msgr2=0x7f2d2c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.666+0000 7f2d31600640 1 -- 172.21.15.43:0/4279990720 shutdown_connections 2024-01-16T15:12:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.666+0000 7f2d31600640 1 -- 172.21.15.43:0/4279990720 wait complete. 2024-01-16T15:12:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.667+0000 7f2d31600640 1 Processor -- start 2024-01-16T15:12:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.667+0000 7f2d31600640 1 -- start start 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0047c0 0x7f2d2c0cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 0x7f2d2c1492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0ceb80 con 0x7f2d2c0ce190 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0cdaf0 con 0x7f2d0c0047c0 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.668+0000 7f2d31600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c149810 con 0x7f2d2c15dcc0 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59136/0 (socket says 172.21.15.43:59136) 2024-01-16T15:12:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 -- 172.21.15.43:0/2002549266 learned_addr learned my addr 172.21.15.43:0/2002549266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0047c0 0x7f2d2c0cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2a000640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 0x7f2d2c1492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 -- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 msgr2=0x7f2d2c1492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 0x7f2d2c1492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 -- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0047c0 msgr2=0x7f2d2c0cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0047c0 0x7f2d2c0cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2b400640 1 -- 172.21.15.43:0/2002549266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c15e1c0 con 0x7f2d2c0ce190 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.669+0000 7f2d2a000640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 0x7f2d2c1492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d2b400640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0ac420 tx=0x7f2d2400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400de70 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24004b60 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c0cc7f0 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24007110 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.670+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c0ccdb0 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.671+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c074770 con 0x7f2d2c0ce190 2024-01-16T15:12:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.672+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2d24007270 con 0x7f2d2c0ce190 2024-01-16T15:12:54.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.673+0000 7f2d22a00640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 0x7f2d0406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.673+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2d24088d40 con 0x7f2d2c0ce190 2024-01-16T15:12:54.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.673+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 0x7f2d0406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.674+0000 7f2d2aa00640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 0x7f2d0406e3e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c068d90 tx=0x7f2d1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.677+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d240553e0 con 0x7f2d2c0ce190 2024-01-16T15:12:54.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:54.949+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f2d2c0b38e0 con 0x7f2d2c0ce190 2024-01-16T15:12:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.518+0000 7f2d22a00640 1 -- 172.21.15.43:0/2002549266 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v591) v1 ==== 156+0+0 (secure 0 0 0) 0x7f2d24058a00 con 0x7f2d2c0ce190 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.521+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 msgr2=0x7f2d0406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.521+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 0x7f2d0406e3e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c068d90 tx=0x7f2d1c008040 comp rx=0 tx=0).stop 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.521+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 msgr2=0x7f2d2c0ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.521+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0ac420 tx=0x7f2d2400b4a0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 shutdown_connections 2024-01-16T15:12:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d0406bf20 0x7f2d0406e3e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d2c0ce190 0x7f2d2c0ce640 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2c15dcc0 0x7f2d2c1492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 --2- 172.21.15.43:0/2002549266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0047c0 0x7f2d2c0cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 >> 172.21.15.43:0/2002549266 conn(0x7f2d2c0b0d80 msgr2=0x7f2d2c0affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 shutdown_connections 2024-01-16T15:12:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.522+0000 7f2d31600640 1 -- 172.21.15.43:0/2002549266 wait complete. 2024-01-16T15:12:55.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:12:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.768+0000 7f154e000640 1 Processor -- start 2024-01-16T15:12:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.769+0000 7f154e000640 1 -- start start 2024-01-16T15:12:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.770+0000 7f154e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15480d1d90 0x7f15480d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.770+0000 7f154e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15480cef30 con 0x7f154815d340 2024-01-16T15:12:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.770+0000 7f154e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15480d11a0 con 0x7f15480cdbb0 2024-01-16T15:12:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.770+0000 7f154e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15480ce320 con 0x7f15480d1d90 2024-01-16T15:12:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f154cc00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f154815d340 0x7f15480d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54654/0 (socket says 172.21.15.43:54654) 2024-01-16T15:12:55.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f154cc00640 1 -- 172.21.15.43:0/928135162 learned_addr learned my addr 172.21.15.43:0/928135162 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945524803 0 0) 0x7f15480cef30 con 0x7f154815d340 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1528003520 con 0x7f154815d340 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f153fe00640 1 --2- 172.21.15.43:0/928135162 >> v2:172.21.15.69:3300/0 conn(0x7f15480d1d90 0x7f15480d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862211889 0 0) 0x7f1528003520 con 0x7f154815d340 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 >> v2:172.21.15.69:3300/0 conn(0x7f15480d1d90 msgr2=0x7f15480d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.771+0000 7f1547e00640 1 --2- 172.21.15.43:0/928135162 >> v2:172.21.15.69:3300/0 conn(0x7f15480d1d90 0x7f15480d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 >> v1:172.21.15.69:6789/0 conn(0x7f15480cdbb0 legacy=0x7f15480b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480ce600 con 0x7f154815d340 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f1538001670 con 0x7f154815d340 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 >> v1:172.21.15.43:6789/0 conn(0x7f154815d340 legacy=0x7f15480d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1528003a30 0x7f1528003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 0x7f1528004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280052a0 0x7f152800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480ce320 con 0x7f15280047c0 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152800c1a0 con 0x7f1528003a30 2024-01-16T15:12:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152800c360 con 0x7f15280052a0 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f153fe00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 0x7f1528004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f154cc00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1528003a30 0x7f1528003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f154d600640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280052a0 0x7f152800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.772+0000 7f153fe00640 1 -- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280052a0 msgr2=0x7f152800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f153fe00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280052a0 0x7f152800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f153fe00640 1 -- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1528003a30 msgr2=0x7f1528003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f153fe00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1528003a30 0x7f1528003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f153fe00640 1 -- 172.21.15.43:0/928135162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1528003520 con 0x7f15280047c0 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f154d600640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280052a0 0x7f152800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:55.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.773+0000 7f154d600640 1 -- 172.21.15.43:0/928135162 reap_dead start 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.774+0000 7f153fe00640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 0x7f1528004c40 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f153000ba00 tx=0x7f15280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fe3a493b53f641 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.774+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15300325f0 con 0x7f15280047c0 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.774+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1530032750 con 0x7f15280047c0 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.774+0000 7f1547e00640 1 -- 172.21.15.43:0/928135162 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530032ae0 con 0x7f15280047c0 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 -- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 msgr2=0x7f1528004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 0x7f1528004c40 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f153000ba00 tx=0x7f15280057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 -- 172.21.15.43:0/928135162 shutdown_connections 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 --2- 172.21.15.43:0/928135162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15280047c0 0x7f1528004c40 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 -- 172.21.15.43:0/928135162 >> 172.21.15.43:0/928135162 conn(0x7f154809c370 msgr2=0x7f15480d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 -- 172.21.15.43:0/928135162 shutdown_connections 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.775+0000 7f154e000640 1 -- 172.21.15.43:0/928135162 wait complete. 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.777+0000 7f154e000640 1 Processor -- start 2024-01-16T15:12:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.777+0000 7f154e000640 1 -- start start 2024-01-16T15:12:55.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280047c0 0x7f1548159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15480b48b0 0x7f15480b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1548159c20 con 0x7f15480b26c0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480b3060 con 0x7f15480b48b0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f154e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1548159690 con 0x7f15280047c0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f153fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f153fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59146/0 (socket says 172.21.15.43:59146) 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.778+0000 7f153fe00640 1 -- 172.21.15.43:0/3366172327 learned_addr learned my addr 172.21.15.43:0/3366172327 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f154cc00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280047c0 0x7f1548159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f154d600640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15480b48b0 0x7f15480b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 -- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280047c0 msgr2=0x7f1548159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280047c0 0x7f1548159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 -- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15480b48b0 msgr2=0x7f15480b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15480b48b0 0x7f15480b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 -- 172.21.15.43:0/3366172327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480b5630 con 0x7f15480b26c0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f153fe00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f153002be70 tx=0x7f1530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.779+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530004220 con 0x7f15480b26c0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.780+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15300043a0 con 0x7f15480b26c0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.780+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153003caa0 con 0x7f15480b26c0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.780+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15480b65a0 con 0x7f15480b26c0 2024-01-16T15:12:55.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.780+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15480b6b60 con 0x7f15480b26c0 2024-01-16T15:12:55.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.781+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1548067d70 con 0x7f15480b26c0 2024-01-16T15:12:55.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.782+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f153003cc00 con 0x7f15480b26c0 2024-01-16T15:12:55.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.783+0000 7f1545600640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 0x7f152006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.783+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f15300ab0f0 con 0x7f15480b26c0 2024-01-16T15:12:55.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.783+0000 7f154cc00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 0x7f152006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.784+0000 7f154cc00640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 0x7f152006e3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1538001050 tx=0x7f1538009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.787+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1530077700 con 0x7f15480b26c0 2024-01-16T15:12:55.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.981+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1548067d70 con 0x7f15480b26c0 2024-01-16T15:12:55.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.988+0000 7f1545600640 1 -- 172.21.15.43:0/3366172327 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f153007ad20 con 0x7f15480b26c0 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 msgr2=0x7f152006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 0x7f152006e3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1538001050 tx=0x7f1538009040 comp rx=0 tx=0).stop 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 msgr2=0x7f15480b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f153002be70 tx=0x7f1530004730 comp rx=0 tx=0).stop 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 shutdown_connections 2024-01-16T15:12:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f152006bf20 0x7f152006e3e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15480b48b0 0x7f15480b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15480b26c0 0x7f15480b2b20 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 --2- 172.21.15.43:0/3366172327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15280047c0 0x7f1548159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 >> 172.21.15.43:0/3366172327 conn(0x7f154809c370 msgr2=0x7f15480ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 shutdown_connections 2024-01-16T15:12:55.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:55.991+0000 7f154e000640 1 -- 172.21.15.43:0/3366172327 wait complete. 2024-01-16T15:12:55.987 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:12:55.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:12:56.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:12:56.206 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:12:56.206 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:12:56.206 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:56.265 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:12:56.265 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:12:56.266 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:56.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:12:56.323 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:12:56.323 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:56.381 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:12:56.381 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:12:56.381 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:12:56.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:12:56.437 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:12:56.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:12:56.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.688+0000 7fc586200640 1 Processor -- start 2024-01-16T15:12:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.688+0000 7fc586200640 1 -- start start 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc586200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc580071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc586200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58013da30 con 0x7fc58015a100 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc586200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800ac560 con 0x7fc5800b39a0 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc586200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800ac840 con 0x7fc5800abff0 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc57f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc58015a100 0x7fc5800b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54670/0 (socket says 172.21.15.43:54670) 2024-01-16T15:12:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.690+0000 7fc57f400640 1 -- 172.21.15.43:0/263549009 learned_addr learned my addr 172.21.15.43:0/263549009 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57fe00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc580071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165071459 0 0) 0x7fc58013da30 con 0x7fc58015a100 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc560003520 con 0x7fc58015a100 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2491715174 0 0) 0x7fc560003520 con 0x7fc58015a100 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 msgr2=0x7fc580071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc580071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 >> v1:172.21.15.69:6789/0 conn(0x7fc5800b39a0 legacy=0x7fc580071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580070650 con 0x7fc58015a100 2024-01-16T15:12:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc5700019f0 con 0x7fc58015a100 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 >> v1:172.21.15.43:6789/0 conn(0x7fc58015a100 legacy=0x7fc5800b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5600039a0 0x7fc560003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc5600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc560004d00 0x7fc560005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800ac840 con 0x7fc560004d00 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.691+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56000ba30 con 0x7fc5600039a0 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56000de00 con 0x7fc5800abff0 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc560004d00 0x7fc560005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57f400640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5600039a0 0x7fc560003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc576800640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc5600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 -- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 msgr2=0x7fc5600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5800abff0 0x7fc5600047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 -- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5600039a0 msgr2=0x7fc560003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5600039a0 0x7fc560003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.692+0000 7fc57fe00640 1 -- 172.21.15.43:0/263549009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560003520 con 0x7fc560004d00 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.693+0000 7fc57fe00640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc560004d00 0x7fc560005180 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc578009a90 tx=0x7fc5600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebfabdf785d7b7c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.693+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57802c680 con 0x7fc560004d00 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.693+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57802c7e0 con 0x7fc560004d00 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.693+0000 7fc57ea00640 1 -- 172.21.15.43:0/263549009 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57802cb70 con 0x7fc560004d00 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 -- 172.21.15.43:0/263549009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc560004d00 msgr2=0x7fc560005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 --2- 172.21.15.43:0/263549009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc560004d00 0x7fc560005180 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc578009a90 tx=0x7fc5600057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc57fe00640 1 -- 172.21.15.43:0/263549009 reap_dead start 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 -- 172.21.15.43:0/263549009 shutdown_connections 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 -- 172.21.15.43:0/263549009 >> 172.21.15.43:0/263549009 conn(0x7fc5800b0d80 msgr2=0x7fc5800b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 -- 172.21.15.43:0/263549009 shutdown_connections 2024-01-16T15:12:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.694+0000 7fc586200640 1 -- 172.21.15.43:0/263549009 wait complete. 2024-01-16T15:12:56.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.696+0000 7fc586200640 1 Processor -- start 2024-01-16T15:12:56.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.696+0000 7fc586200640 1 -- start start 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 0x7fc5800cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58015a660 0x7fc580098060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58015d030 con 0x7fc58015c670 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58015add0 con 0x7fc560004d00 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc586200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58015b0b0 con 0x7fc58015a660 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc57fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc57fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59152/0 (socket says 172.21.15.43:59152) 2024-01-16T15:12:56.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.697+0000 7fc57fe00640 1 -- 172.21.15.43:0/223384243 learned_addr learned my addr 172.21.15.43:0/223384243 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc576800640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58015a660 0x7fc580098060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57f400640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 0x7fc5800cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 -- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58015a660 msgr2=0x7fc580098060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58015a660 0x7fc580098060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 -- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 msgr2=0x7fc5800cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 0x7fc5800cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 -- 172.21.15.43:0/223384243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58015b390 con 0x7fc58015c670 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57f400640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 0x7fc5800cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc57fe00640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc578009a90 tx=0x7fc578004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57802c680 con 0x7fc58015c670 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578003810 con 0x7fc58015c670 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578003bc0 con 0x7fc58015c670 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.698+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58015b670 con 0x7fc58015c670 2024-01-16T15:12:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.699+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58015aae0 con 0x7fc58015c670 2024-01-16T15:12:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.700+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58015bb90 con 0x7fc58015c670 2024-01-16T15:12:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.701+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc578003d20 con 0x7fc58015c670 2024-01-16T15:12:56.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.701+0000 7fc577e00640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 0x7fc55806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.702+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc5780a97b0 con 0x7fc58015c670 2024-01-16T15:12:56.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.702+0000 7fc57f400640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 0x7fc55806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.703+0000 7fc57f400640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 0x7fc55806e3e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc570002010 tx=0x7fc570001880 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.706+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc58015bb90 con 0x7fc58015c670 2024-01-16T15:12:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.935+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58015bb90 con 0x7fc58015c670 2024-01-16T15:12:56.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.936+0000 7fc577e00640 1 -- 172.21.15.43:0/223384243 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fc58015bb90 con 0x7fc58015c670 2024-01-16T15:12:56.914 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:12:56.914 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5540,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1054183394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":1054183394},{"type":"v1","addr":"172.21.15.69:6835","nonce":1054183394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/3255994502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3255994502},{"type":"v1","addr":"172.21.15.43:6836","nonce":3255994502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5546,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1771464625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":1771464625},{"type":"v1","addr":"172.21.15.43:6837","nonce":1771464625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2360943131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2360943131},{"type":"v1","addr":"172.21.15.69:6834","nonce":2360943131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-16T15:12:56.914 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 49 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 msgr2=0x7fc55806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 0x7fc55806e3e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc570002010 tx=0x7fc570001880 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 msgr2=0x7fc58015caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc578009a90 tx=0x7fc578004730 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 shutdown_connections 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc55806bf20 0x7fc55806e3e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc58015c670 0x7fc58015caf0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58015a660 0x7fc580098060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 --2- 172.21.15.43:0/223384243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc560004d00 0x7fc5800cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 >> 172.21.15.43:0/223384243 conn(0x7fc5800b0d80 msgr2=0x7fc58006d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 shutdown_connections 2024-01-16T15:12:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:12:56.939+0000 7fc586200640 1 -- 172.21.15.43:0/223384243 wait complete. 2024-01-16T15:12:56.937 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:12:56.937 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:12:56.937 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:12:56.937 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:12:56.937 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:12:56.938 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:12:56.938 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:12:56.938 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:12:56.938 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:12:56.938 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:12:56.938 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:12:56.938 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:12:56.938 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:12:56.939 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:12:56.939 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:12:56.939 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:12:56.960 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:12:56.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:12:56.961 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:12:56.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:12:56.962 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:12:56.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:12:57.008 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:12:57.008 DEBUG:teuthology.parallel:result is None 2024-01-16T15:12:57.009 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:12:57.009 DEBUG:teuthology.parallel:result is None 2024-01-16T15:12:57.015 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:12:57.015 DEBUG:teuthology.parallel:result is None 2024-01-16T15:12:57.173 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:12:57.173 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:12:57.200 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:12:58.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:58.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:59.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-16T15:13:02.960 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:13:02.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:13:02.969 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:13:02.969 DEBUG:teuthology.parallel:result is None 2024-01-16T15:13:02.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:13:03.139 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:13:03.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.193+0000 7fc47ae00640 1 Processor -- start 2024-01-16T15:13:03.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.193+0000 7fc47ae00640 1 -- start start 2024-01-16T15:13:03.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.194+0000 7fc47ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc47406d520 0x7fc47406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc47ae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 0x7fc4740719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc47ae00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc474165430 0x7fc4741658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc47ae00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc474070dc0 con 0x7fc474165430 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc47ae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc474161d00 con 0x7fc4740acb50 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc47ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47406de40 con 0x7fc47406d520 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc474165430 0x7fc4741658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc474165430 0x7fc4741658e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42898/0 (socket says 172.21.15.43:42898) 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 -- 172.21.15.43:0/2436248645 learned_addr learned my addr 172.21.15.43:0/2436248645 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc46b200640 1 --2- 172.21.15.43:0/2436248645 >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 0x7fc4740719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc473e00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc47406d520 0x7fc47406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 -- 172.21.15.43:0/2436248645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc47406d520 msgr2=0x7fc47406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc47406d520 0x7fc47406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 -- 172.21.15.43:0/2436248645 >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 msgr2=0x7fc4740719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 --2- 172.21.15.43:0/2436248645 >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 0x7fc4740719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc478e00640 1 -- 172.21.15.43:0/2436248645 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740cedb0 con 0x7fc474165430 2024-01-16T15:13:03.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.195+0000 7fc46b200640 1 --2- 172.21.15.43:0/2436248645 >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 0x7fc4740719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc478e00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4741658e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc47406fd00 tx=0x7fc46c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e2a8f747fce90c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc473400640 1 -- 172.21.15.43:0/2436248645 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c02ca70 con 0x7fc474165430 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc473400640 1 -- 172.21.15.43:0/2436248645 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c02cbd0 con 0x7fc474165430 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc473400640 1 -- 172.21.15.43:0/2436248645 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c0343f0 con 0x7fc474165430 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc47ae00640 1 -- 172.21.15.43:0/2436248645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 msgr2=0x7fc4741658e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.197+0000 7fc47ae00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4741658e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc47406fd00 tx=0x7fc46c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 -- 172.21.15.43:0/2436248645 shutdown_connections 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4741658e0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 --2- 172.21.15.43:0/2436248645 >> v2:172.21.15.69:3300/0 conn(0x7fc4740acb50 0x7fc4740719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 --2- 172.21.15.43:0/2436248645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc47406d520 0x7fc47406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 -- 172.21.15.43:0/2436248645 >> 172.21.15.43:0/2436248645 conn(0x7fc4740b0d80 msgr2=0x7fc4740b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 -- 172.21.15.43:0/2436248645 shutdown_connections 2024-01-16T15:13:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.198+0000 7fc47ae00640 1 -- 172.21.15.43:0/2436248645 wait complete. 2024-01-16T15:13:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.199+0000 7fc47ae00640 1 Processor -- start 2024-01-16T15:13:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.199+0000 7fc47ae00640 1 -- start start 2024-01-16T15:13:03.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 0x7fc47409af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4740acb50 0x7fc474152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47409aad0 con 0x7fc474165430 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc474099ec0 con 0x7fc47406d520 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.200+0000 7fc47ae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47409a1a0 con 0x7fc4740acb50 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42906/0 (socket says 172.21.15.43:42906) 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 -- 172.21.15.43:0/3837659048 learned_addr learned my addr 172.21.15.43:0/3837659048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc46b200640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4740acb50 0x7fc474152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc473e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 0x7fc47409af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 -- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4740acb50 msgr2=0x7fc474152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4740acb50 0x7fc474152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 -- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 msgr2=0x7fc47409af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 0x7fc47409af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc478e00640 1 -- 172.21.15.43:0/3837659048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c009b30 con 0x7fc474165430 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.201+0000 7fc473e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 0x7fc47409af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc478e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc46c036040 tx=0x7fc46c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c040670 con 0x7fc474165430 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c0407f0 con 0x7fc474165430 2024-01-16T15:13:03.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47409a480 con 0x7fc474165430 2024-01-16T15:13:03.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c040ba0 con 0x7fc474165430 2024-01-16T15:13:03.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.202+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47415c860 con 0x7fc474165430 2024-01-16T15:13:03.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.203+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47409ac80 con 0x7fc474165430 2024-01-16T15:13:03.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.204+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc46c040d00 con 0x7fc474165430 2024-01-16T15:13:03.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.205+0000 7fc470c00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 0x7fc44c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.205+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc46c0ab830 con 0x7fc474165430 2024-01-16T15:13:03.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.205+0000 7fc473e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 0x7fc44c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.206+0000 7fc473e00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 0x7fc44c06e460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.210+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc47409ac80 con 0x7fc474165430 2024-01-16T15:13:03.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:03.417+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc474074770 con 0x7fc474165430 2024-01-16T15:13:05.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.064+0000 7fc470c00640 1 -- 172.21.15.43:0/3837659048 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v84) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc46c077f90 con 0x7fc474165430 2024-01-16T15:13:05.043 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:13:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 msgr2=0x7fc44c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 0x7fc44c06e460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464008040 comp rx=0 tx=0).stop 2024-01-16T15:13:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 msgr2=0x7fc4740993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc46c036040 tx=0x7fc46c004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 shutdown_connections 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc44c06bfa0 0x7fc44c06e460 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc474165430 0x7fc4740993c0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4740acb50 0x7fc474152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 --2- 172.21.15.43:0/3837659048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc47406d520 0x7fc47409af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 >> 172.21.15.43:0/3837659048 conn(0x7fc4740b0d80 msgr2=0x7fc4741613d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 shutdown_connections 2024-01-16T15:13:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.068+0000 7fc47ae00640 1 -- 172.21.15.43:0/3837659048 wait complete. 2024-01-16T15:13:05.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:13:05.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.305+0000 7f2a97c00640 1 Processor -- start 2024-01-16T15:13:05.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.305+0000 7f2a97c00640 1 -- start start 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.306+0000 7f2a97c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2a900ce260 0x7f2a900d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.306+0000 7f2a97c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9015d8b0 con 0x7f2a9015d4d0 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a97c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a900b2400 con 0x7f2a900ce6a0 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a97c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a900b91b0 con 0x7f2a900ce260 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a95200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2a9015d4d0 0x7f2a900cda70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56098/0 (socket says 172.21.15.43:56098) 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a95200640 1 -- 172.21.15.43:0/971799947 learned_addr learned my addr 172.21.15.43:0/971799947 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:05.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a87e00640 1 --2- 172.21.15.43:0/971799947 >> v2:172.21.15.43:3300/0 conn(0x7f2a900ce260 0x7f2a900d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a8fe00640 1 -- 172.21.15.43:0/971799947 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269916948 0 0) 0x7f2a9015d8b0 con 0x7f2a9015d4d0 2024-01-16T15:13:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a8fe00640 1 -- 172.21.15.43:0/971799947 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a70003520 con 0x7f2a9015d4d0 2024-01-16T15:13:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a87e00640 1 -- 172.21.15.43:0/971799947 >> v1:172.21.15.69:6789/0 conn(0x7f2a900ce6a0 legacy=0x7f2a900b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a87e00640 1 -- 172.21.15.43:0/971799947 >> v1:172.21.15.43:6789/0 conn(0x7f2a9015d4d0 legacy=0x7f2a900cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.307+0000 7f2a87e00640 1 -- 172.21.15.43:0/971799947 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a900b26e0 con 0x7f2a900ce260 2024-01-16T15:13:05.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a87e00640 1 --2- 172.21.15.43:0/971799947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce260 0x7f2a900d1850 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2a780097a0 tx=0x7f2a7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c816bab67820b9fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a8fe00640 1 -- 172.21.15.43:0/971799947 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78031830 con 0x7f2a900ce260 2024-01-16T15:13:05.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a8fe00640 1 -- 172.21.15.43:0/971799947 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78031990 con 0x7f2a900ce260 2024-01-16T15:13:05.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a8fe00640 1 -- 172.21.15.43:0/971799947 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78031d20 con 0x7f2a900ce260 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a97c00640 1 -- 172.21.15.43:0/971799947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce260 msgr2=0x7f2a900d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.309+0000 7f2a97c00640 1 --2- 172.21.15.43:0/971799947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce260 0x7f2a900d1850 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2a780097a0 tx=0x7f2a7802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.310+0000 7f2a97c00640 1 -- 172.21.15.43:0/971799947 shutdown_connections 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.310+0000 7f2a97c00640 1 --2- 172.21.15.43:0/971799947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce260 0x7f2a900d1850 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.310+0000 7f2a97c00640 1 -- 172.21.15.43:0/971799947 >> 172.21.15.43:0/971799947 conn(0x7f2a90071f90 msgr2=0x7f2a900723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.310+0000 7f2a97c00640 1 -- 172.21.15.43:0/971799947 shutdown_connections 2024-01-16T15:13:05.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.310+0000 7f2a97c00640 1 -- 172.21.15.43:0/971799947 wait complete. 2024-01-16T15:13:05.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.311+0000 7f2a97c00640 1 Processor -- start 2024-01-16T15:13:05.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.312+0000 7f2a97c00640 1 -- start start 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a900ce260 0x7f2a900b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 0x7f2a9015fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a901608f0 con 0x7f2a900ce6a0 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a900cbb00 con 0x7f2a9015d4d0 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a97c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a900cb4f0 con 0x7f2a900ce260 2024-01-16T15:13:05.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42918/0 (socket says 172.21.15.43:42918) 2024-01-16T15:13:05.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 -- 172.21.15.43:0/513639501 learned_addr learned my addr 172.21.15.43:0/513639501 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 -- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a900ce260 msgr2=0x7f2a900b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a95c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 0x7f2a9015fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a900ce260 0x7f2a900b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.313+0000 7f2a87e00640 1 -- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 msgr2=0x7f2a9015fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a87e00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 0x7f2a9015fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a87e00640 1 -- 172.21.15.43:0/513639501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a78009400 con 0x7f2a900ce6a0 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a95c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 0x7f2a9015fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a87e00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2a780098c0 tx=0x7f2a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7803a6f0 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7803a870 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7803ac20 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.314+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a90160610 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.315+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a900b4400 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.316+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a900d0500 con 0x7f2a900ce6a0 2024-01-16T15:13:05.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.317+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2a7804c3b0 con 0x7f2a900ce6a0 2024-01-16T15:13:05.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.317+0000 7f2a8d600640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 0x7f2a6806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.318+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f2a780aa690 con 0x7f2a900ce6a0 2024-01-16T15:13:05.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.318+0000 7f2a95200640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 0x7f2a6806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.318+0000 7f2a95200640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 0x7f2a6806e430 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a80001570 tx=0x7f2a80007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.322+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a78076df0 con 0x7f2a900ce6a0 2024-01-16T15:13:05.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.583+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2a900cbc70 con 0x7f2a900ce6a0 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.585+0000 7f2a8d600640 1 -- 172.21.15.43:0/513639501 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2a78035020 con 0x7f2a900ce6a0 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:13:05.563 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:13:05.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:13:05.565 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.567 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.568 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:13:05.570 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:13:05.571 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.573 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:05.574 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:13:05.575 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.587+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 msgr2=0x7f2a6806e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.587+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 0x7f2a6806e430 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a80001570 tx=0x7f2a80007930 comp rx=0 tx=0).stop 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 msgr2=0x7f2a900cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2a780098c0 tx=0x7f2a780046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 shutdown_connections 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a6806bf70 0x7f2a6806e430 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9015d4d0 0x7f2a9015fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a900ce6a0 0x7f2a900cbf90 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 --2- 172.21.15.43:0/513639501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a900ce260 0x7f2a900b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 >> 172.21.15.43:0/513639501 conn(0x7f2a90071f90 msgr2=0x7f2a9015bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 shutdown_connections 2024-01-16T15:13:05.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.588+0000 7f2a97c00640 1 -- 172.21.15.43:0/513639501 wait complete. 2024-01-16T15:13:05.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:05.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.824+0000 7efe7be00640 1 Processor -- start 2024-01-16T15:13:05.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.824+0000 7efe7be00640 1 -- start start 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efe7c157770 0x7efe7c06cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe7c15df80 0x7efe7c070720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c161670 con 0x7efe7c06d080 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7be00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c15e410 con 0x7efe7c157770 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c06d500 con 0x7efe7c15df80 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7efe7c157770 0x7efe7c06cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe73e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efe7c06d080 0x7efe7c15d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56112/0 (socket says 172.21.15.43:56112) 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe73e00640 1 -- 172.21.15.43:0/84340189 learned_addr learned my addr 172.21.15.43:0/84340189 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7aa00640 1 --2- 172.21.15.43:0/84340189 >> v2:172.21.15.43:3300/0 conn(0x7efe7c157770 0x7efe7c06cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42924/0 (socket says 172.21.15.43:42924) 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.826+0000 7efe7b400640 1 --2- 172.21.15.43:0/84340189 >> v2:172.21.15.69:3300/0 conn(0x7efe7c15df80 0x7efe7c070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.827+0000 7efe7aa00640 1 -- 172.21.15.43:0/84340189 >> v2:172.21.15.69:3300/0 conn(0x7efe7c15df80 msgr2=0x7efe7c070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.827+0000 7efe7aa00640 1 --2- 172.21.15.43:0/84340189 >> v2:172.21.15.69:3300/0 conn(0x7efe7c15df80 0x7efe7c070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.827+0000 7efe7aa00640 1 -- 172.21.15.43:0/84340189 >> v1:172.21.15.43:6789/0 conn(0x7efe7c06d080 legacy=0x7efe7c15d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.827+0000 7efe7aa00640 1 -- 172.21.15.43:0/84340189 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c15e610 con 0x7efe7c157770 2024-01-16T15:13:05.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.828+0000 7efe7aa00640 1 --2- 172.21.15.43:0/84340189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c157770 0x7efe7c06cb40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7efe7c06ca90 tx=0x7efe6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9718f17f70cc73e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.828+0000 7efe7a000640 1 -- 172.21.15.43:0/84340189 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c031830 con 0x7efe7c157770 2024-01-16T15:13:05.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.828+0000 7efe7a000640 1 -- 172.21.15.43:0/84340189 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c031990 con 0x7efe7c157770 2024-01-16T15:13:05.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.828+0000 7efe7a000640 1 -- 172.21.15.43:0/84340189 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c031d20 con 0x7efe7c157770 2024-01-16T15:13:05.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 -- 172.21.15.43:0/84340189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c157770 msgr2=0x7efe7c06cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 --2- 172.21.15.43:0/84340189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c157770 0x7efe7c06cb40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7efe7c06ca90 tx=0x7efe6c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 -- 172.21.15.43:0/84340189 shutdown_connections 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 --2- 172.21.15.43:0/84340189 >> v2:172.21.15.69:3300/0 conn(0x7efe7c15df80 0x7efe7c070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 --2- 172.21.15.43:0/84340189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c157770 0x7efe7c06cb40 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 -- 172.21.15.43:0/84340189 >> 172.21.15.43:0/84340189 conn(0x7efe7c06a860 msgr2=0x7efe7c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 -- 172.21.15.43:0/84340189 shutdown_connections 2024-01-16T15:13:05.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.829+0000 7efe7be00640 1 -- 172.21.15.43:0/84340189 wait complete. 2024-01-16T15:13:05.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.831+0000 7efe7be00640 1 Processor -- start 2024-01-16T15:13:05.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.832+0000 7efe7be00640 1 -- start start 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe7c157770 0x7efe7c0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 0x7efe7c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c15af90 con 0x7efe7c06d080 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c15b270 con 0x7efe7c15df80 2024-01-16T15:13:05.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c15b550 con 0x7efe7c157770 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42936/0 (socket says 172.21.15.43:42936) 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.833+0000 7efe7aa00640 1 -- 172.21.15.43:0/2517117887 learned_addr learned my addr 172.21.15.43:0/2517117887 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe73e00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe7c157770 0x7efe7c0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7b400640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 0x7efe7c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 -- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe7c157770 msgr2=0x7efe7c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe7c157770 0x7efe7c0cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 -- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 msgr2=0x7efe7c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 0x7efe7c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 -- 172.21.15.43:0/2517117887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c009400 con 0x7efe7c06d080 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7b400640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 0x7efe7c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:05.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7aa00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009890 tx=0x7efe6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c03ba70 con 0x7efe7c06d080 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c03bbf0 con 0x7efe7c06d080 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.834+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c15b830 con 0x7efe7c06d080 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.835+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c045400 con 0x7efe7c06d080 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.835+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c0cda70 con 0x7efe7c06d080 2024-01-16T15:13:05.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.836+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c04e840 con 0x7efe7c06d080 2024-01-16T15:13:05.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.837+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7efe6c04e440 con 0x7efe7c06d080 2024-01-16T15:13:05.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.838+0000 7efe72a00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 0x7efe5406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.838+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4885+0+0 (secure 0 0 0) 0x7efe6c0aaa50 con 0x7efe7c06d080 2024-01-16T15:13:05.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.838+0000 7efe73e00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 0x7efe5406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.839+0000 7efe73e00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 0x7efe5406e3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efe7c15c330 tx=0x7efe64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:05.843+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe6c0771b0 con 0x7efe7c06d080 2024-01-16T15:13:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.115+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe7c070500 con 0x7efe7c06d080 2024-01-16T15:13:06.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.117+0000 7efe72a00640 1 -- 172.21.15.43:0/2517117887 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+12853 (secure 0 0 0) 0x7efe6c07a7d0 con 0x7efe7c06d080 2024-01-16T15:13:06.096 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:06.096 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":84,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:13:05.061269+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 msgr2=0x7efe5406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 0x7efe5406e3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efe7c15c330 tx=0x7efe64009040 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 msgr2=0x7efe7c1545f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009890 tx=0x7efe6c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 shutdown_connections 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efe5406bf20 0x7efe5406e3e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe7c15df80 0x7efe7c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe7c157770 0x7efe7c0cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 --2- 172.21.15.43:0/2517117887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe7c06d080 0x7efe7c1545f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.120+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 >> 172.21.15.43:0/2517117887 conn(0x7efe7c06a860 msgr2=0x7efe7c0ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.121+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 shutdown_connections 2024-01-16T15:13:06.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.121+0000 7efe7be00640 1 -- 172.21.15.43:0/2517117887 wait complete. 2024-01-16T15:13:06.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:13:06.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:13:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.362+0000 7f5463400640 1 Processor -- start 2024-01-16T15:13:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.362+0000 7f5463400640 1 -- start start 2024-01-16T15:13:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.363+0000 7f5463400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c144b80 0x7f545c0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.363+0000 7f5463400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f545c06ab50 0x7f545c15bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.363+0000 7f5463400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c0ce4d0 con 0x7f545c06eb90 2024-01-16T15:13:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.363+0000 7f5463400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545c15b620 con 0x7f545c06ab50 2024-01-16T15:13:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.363+0000 7f5463400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545c06fbf0 con 0x7f545c144b80 2024-01-16T15:13:06.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.364+0000 7f5461600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f545c06eb90 0x7f545c06a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56124/0 (socket says 172.21.15.43:56124) 2024-01-16T15:13:06.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.364+0000 7f5462a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f545c06ab50 0x7f545c15bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.364+0000 7f5461600640 1 -- 172.21.15.43:0/3267911337 learned_addr learned my addr 172.21.15.43:0/3267911337 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:06.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.364+0000 7f5462000640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c144b80 0x7f545c0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5460c00640 1 -- 172.21.15.43:0/3267911337 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174003150 0 0) 0x7f5444001110 con 0x7f545c06eb90 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5460c00640 1 -- 172.21.15.43:0/3267911337 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c003520 con 0x7f545c06eb90 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5462a00640 1 -- 172.21.15.43:0/3267911337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c144b80 msgr2=0x7f545c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5462a00640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c144b80 0x7f545c0cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5462a00640 1 -- 172.21.15.43:0/3267911337 >> v1:172.21.15.43:6789/0 conn(0x7f545c06eb90 legacy=0x7f545c06a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.365+0000 7f5462a00640 1 -- 172.21.15.43:0/3267911337 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c06f350 con 0x7f545c06ab50 2024-01-16T15:13:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.366+0000 7f5462a00640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c06ab50 0x7f545c15bd30 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5454009d90 tx=0x7f545402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af318d60ffae41ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5460c00640 1 -- 172.21.15.43:0/3267911337 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545402cc50 con 0x7f545c06ab50 2024-01-16T15:13:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5460c00640 1 -- 172.21.15.43:0/3267911337 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545402cdb0 con 0x7f545c06ab50 2024-01-16T15:13:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5460c00640 1 -- 172.21.15.43:0/3267911337 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454034640 con 0x7f545c06ab50 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 -- 172.21.15.43:0/3267911337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c06ab50 msgr2=0x7f545c15bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c06ab50 0x7f545c15bd30 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5454009d90 tx=0x7f545402bea0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 -- 172.21.15.43:0/3267911337 shutdown_connections 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c06ab50 0x7f545c15bd30 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 --2- 172.21.15.43:0/3267911337 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c144b80 0x7f545c0cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 -- 172.21.15.43:0/3267911337 >> 172.21.15.43:0/3267911337 conn(0x7f545c06bbb0 msgr2=0x7f545c0b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 -- 172.21.15.43:0/3267911337 shutdown_connections 2024-01-16T15:13:06.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.367+0000 7f5463400640 1 -- 172.21.15.43:0/3267911337 wait complete. 2024-01-16T15:13:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.369+0000 7f5463400640 1 Processor -- start 2024-01-16T15:13:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.369+0000 7f5463400640 1 -- start start 2024-01-16T15:13:06.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f545c06ab50 0x7f545c13b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c06eb90 0x7f545c0b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c0cb7d0 con 0x7f545c144b80 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c0b3180 con 0x7f545c06ab50 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.370+0000 7f5463400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545c0b33f0 con 0x7f545c06eb90 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42952/0 (socket says 172.21.15.43:42952) 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 -- 172.21.15.43:0/927435585 learned_addr learned my addr 172.21.15.43:0/927435585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462000640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f545c06ab50 0x7f545c13b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5461600640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c06eb90 0x7f545c0b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 -- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c06eb90 msgr2=0x7f545c0b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c06eb90 0x7f545c0b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 -- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f545c06ab50 msgr2=0x7f545c13b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f545c06ab50 0x7f545c13b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.371+0000 7f5462a00640 1 -- 172.21.15.43:0/927435585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454009a70 con 0x7f545c144b80 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f5462a00640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f545402c4b0 tx=0x7f5454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545403e610 con 0x7f545c144b80 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545403e770 con 0x7f545c144b80 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545403eb60 con 0x7f545c144b80 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c0b36d0 con 0x7f545c144b80 2024-01-16T15:13:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.372+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c13b940 con 0x7f545c144b80 2024-01-16T15:13:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.373+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545c0b3bf0 con 0x7f545c144b80 2024-01-16T15:13:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.374+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f545403b050 con 0x7f545c144b80 2024-01-16T15:13:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.375+0000 7f545a000640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 0x7f543406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.375+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f54540aa9b0 con 0x7f545c144b80 2024-01-16T15:13:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.376+0000 7f5462000640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 0x7f543406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.376+0000 7f5462000640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 0x7f543406e3e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f544c009a30 tx=0x7f544c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.380+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f545c0b3bf0 con 0x7f545c144b80 2024-01-16T15:13:06.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-01-16T15:13:06.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.566+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f545c0b3bf0 con 0x7f545c144b80 2024-01-16T15:13:06.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.567+0000 7f545a000640 1 -- 172.21.15.43:0/927435585 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f545c0b3bf0 con 0x7f545c144b80 2024-01-16T15:13:06.546 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 msgr2=0x7f543406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 0x7f543406e3e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f544c009a30 tx=0x7f544c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 msgr2=0x7f545c0cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f545402c4b0 tx=0x7f5454004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 shutdown_connections 2024-01-16T15:13:06.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f543406bf20 0x7f543406e3e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f545c144b80 0x7f545c0cb290 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f545c06eb90 0x7f545c0b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 --2- 172.21.15.43:0/927435585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f545c06ab50 0x7f545c13b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 >> 172.21.15.43:0/927435585 conn(0x7f545c06bbb0 msgr2=0x7f545c15ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 shutdown_connections 2024-01-16T15:13:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.570+0000 7f5463400640 1 -- 172.21.15.43:0/927435585 wait complete. 2024-01-16T15:13:06.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:13:06.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.811+0000 7f3041200640 1 Processor -- start 2024-01-16T15:13:06.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.811+0000 7f3041200640 1 -- start start 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3041200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f303c0cf540 0x7f303c0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3041200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f303c06cff0 0x7f303c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3041200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303c06d560 con 0x7f303c155850 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3041200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c06d9a0 con 0x7f303c06cff0 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3041200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c06dc80 con 0x7f303c0cf540 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f303c06cff0 0x7f303c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f303c06cff0 0x7f303c0cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42962/0 (socket says 172.21.15.43:42962) 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 -- 172.21.15.43:0/1443079744 learned_addr learned my addr 172.21.15.43:0/1443079744 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f3039400640 1 --2- 172.21.15.43:0/1443079744 >> v2:172.21.15.69:3300/0 conn(0x7f303c0cf540 0x7f303c0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 -- 172.21.15.43:0/1443079744 >> v2:172.21.15.69:3300/0 conn(0x7f303c0cf540 msgr2=0x7f303c0ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:06.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 --2- 172.21.15.43:0/1443079744 >> v2:172.21.15.69:3300/0 conn(0x7f303c0cf540 0x7f303c0ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 -- 172.21.15.43:0/1443079744 >> v1:172.21.15.69:6789/0 conn(0x7f303c155850 legacy=0x7f303c15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:06.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.813+0000 7f303a800640 1 -- 172.21.15.43:0/1443079744 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c06df60 con 0x7f303c06cff0 2024-01-16T15:13:06.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.814+0000 7f303a800640 1 --2- 172.21.15.43:0/1443079744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c0cd140 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f303c06f820 tx=0x7f303402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5216baf2dc3dbe0b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.815+0000 7f3038a00640 1 -- 172.21.15.43:0/1443079744 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034031870 con 0x7f303c06cff0 2024-01-16T15:13:06.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.815+0000 7f3038a00640 1 -- 172.21.15.43:0/1443079744 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30340319d0 con 0x7f303c06cff0 2024-01-16T15:13:06.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.815+0000 7f3038a00640 1 -- 172.21.15.43:0/1443079744 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034031d60 con 0x7f303c06cff0 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.815+0000 7f3041200640 1 -- 172.21.15.43:0/1443079744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 msgr2=0x7f303c0cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.815+0000 7f3041200640 1 --2- 172.21.15.43:0/1443079744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c0cd140 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f303c06f820 tx=0x7f303402b910 comp rx=0 tx=0).stop 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 -- 172.21.15.43:0/1443079744 shutdown_connections 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 --2- 172.21.15.43:0/1443079744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c0cd140 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 --2- 172.21.15.43:0/1443079744 >> v2:172.21.15.69:3300/0 conn(0x7f303c0cf540 0x7f303c0ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 -- 172.21.15.43:0/1443079744 >> 172.21.15.43:0/1443079744 conn(0x7f303c0b23a0 msgr2=0x7f303c0cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 -- 172.21.15.43:0/1443079744 shutdown_connections 2024-01-16T15:13:06.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.816+0000 7f3041200640 1 -- 172.21.15.43:0/1443079744 wait complete. 2024-01-16T15:13:06.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.817+0000 7f3041200640 1 Processor -- start 2024-01-16T15:13:06.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 -- start start 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f303c0cf540 0x7f303c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f303c155850 0x7f303c153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c149d70 con 0x7f303c06cff0 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0ae750 con 0x7f303c155850 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.818+0000 7f3041200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c15c660 con 0x7f303c0cf540 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42966/0 (socket says 172.21.15.43:42966) 2024-01-16T15:13:06.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 -- 172.21.15.43:0/777803215 learned_addr learned my addr 172.21.15.43:0/777803215 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:06.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 -- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f303c0cf540 msgr2=0x7f303c15dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f303c0cf540 0x7f303c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 -- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f303c155850 msgr2=0x7f303c153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f303c155850 0x7f303c153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.819+0000 7f3039e00640 1 -- 172.21.15.43:0/777803215 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30340094d0 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3039e00640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b3f0 tx=0x7f302c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011830 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302c011990 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011d40 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c15c940 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.820+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c15cf00 con 0x7f303c06cff0 2024-01-16T15:13:06.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.821+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c0ae8c0 con 0x7f303c06cff0 2024-01-16T15:13:06.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.822+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f302c01a400 con 0x7f303c06cff0 2024-01-16T15:13:06.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.823+0000 7f3032000640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 0x7f301406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.823+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f302c088630 con 0x7f303c06cff0 2024-01-16T15:13:06.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.823+0000 7f3039400640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 0x7f301406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.824+0000 7f3039400640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 0x7f301406e3e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f30240070b0 tx=0x7f3024007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:06.827+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f303c0ae8c0 con 0x7f303c06cff0 2024-01-16T15:13:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.022+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f303c0ae8c0 con 0x7f303c06cff0 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.023+0000 7f3032000640 1 -- 172.21.15.43:0/777803215 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) v1 ==== 84+0+65 (secure 0 0 0) 0x7f302c015060 con 0x7f303c06cff0 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:07.001 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:13:07.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 msgr2=0x7f301406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 0x7f301406e3e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f30240070b0 tx=0x7f3024007040 comp rx=0 tx=0).stop 2024-01-16T15:13:07.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 msgr2=0x7f303c149830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b3f0 tx=0x7f302c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 shutdown_connections 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f301406bf20 0x7f301406e3e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f303c155850 0x7f303c153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f303c0cf540 0x7f303c15dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 --2- 172.21.15.43:0/777803215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f303c06cff0 0x7f303c149830 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 >> 172.21.15.43:0/777803215 conn(0x7f303c0b23a0 msgr2=0x7f303c151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 shutdown_connections 2024-01-16T15:13:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.025+0000 7f3041200640 1 -- 172.21.15.43:0/777803215 wait complete. 2024-01-16T15:13:07.022 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:13:07.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:13:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.267+0000 7fd843000640 1 Processor -- start 2024-01-16T15:13:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.267+0000 7fd843000640 1 -- start start 2024-01-16T15:13:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd843000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd83c15f270 0x7fd83c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd843000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd843000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c15f930 con 0x7fd83c144770 2024-01-16T15:13:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd843000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c0b43b0 con 0x7fd83c15f270 2024-01-16T15:13:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd843000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0b3b10 con 0x7fd83c06d9e0 2024-01-16T15:13:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd83c15f270 0x7fd83c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd842600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd842600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:45960/0 (socket says 172.21.15.43:45960) 2024-01-16T15:13:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd83c15f270 0x7fd83c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:46966/0 (socket says 172.21.15.43:46966) 2024-01-16T15:13:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd842600640 1 -- 172.21.15.43:0/2846408934 learned_addr learned my addr 172.21.15.43:0/2846408934 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 -- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 msgr2=0x7fd83c0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 -- 172.21.15.43:0/2846408934 >> v1:172.21.15.69:6789/0 conn(0x7fd83c144770 legacy=0x7fd83c15eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd841200640 1 -- 172.21.15.43:0/2846408934 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0b3df0 con 0x7fd83c15f270 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.270+0000 7fd842600640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.272+0000 7fd841200640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c0cd4a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097a0 tx=0x7fd82402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa0cfcb3cd3fff3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.272+0000 7fd83be00640 1 -- 172.21.15.43:0/2846408934 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82402d040 con 0x7fd83c15f270 2024-01-16T15:13:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.272+0000 7fd83be00640 1 -- 172.21.15.43:0/2846408934 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824031810 con 0x7fd83c15f270 2024-01-16T15:13:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.272+0000 7fd83be00640 1 -- 172.21.15.43:0/2846408934 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824031ba0 con 0x7fd83c15f270 2024-01-16T15:13:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 -- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 msgr2=0x7fd83c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c0cd4a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097a0 tx=0x7fd82402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 -- 172.21.15.43:0/2846408934 shutdown_connections 2024-01-16T15:13:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c0cd4a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 --2- 172.21.15.43:0/2846408934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c06d9e0 0x7fd83c0cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 -- 172.21.15.43:0/2846408934 >> 172.21.15.43:0/2846408934 conn(0x7fd83c0cec10 msgr2=0x7fd83c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:07.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 -- 172.21.15.43:0/2846408934 shutdown_connections 2024-01-16T15:13:07.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.273+0000 7fd843000640 1 -- 172.21.15.43:0/2846408934 wait complete. 2024-01-16T15:13:07.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.275+0000 7fd843000640 1 Processor -- start 2024-01-16T15:13:07.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.275+0000 7fd843000640 1 -- start start 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c144770 0x7fd83c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c13f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c06d2d0 con 0x7fd83c06d9e0 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0716c0 con 0x7fd83c15f270 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd843000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0718f0 con 0x7fd83c144770 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd841c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd841c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42968/0 (socket says 172.21.15.43:42968) 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd841c00640 1 -- 172.21.15.43:0/2352389160 learned_addr learned my addr 172.21.15.43:0/2352389160 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.276+0000 7fd842600640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c13f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841200640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c144770 0x7fd83c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 -- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c144770 msgr2=0x7fd83c071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c144770 0x7fd83c071180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 -- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 msgr2=0x7fd83c13f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c13f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 -- 172.21.15.43:0/2352389160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834009b30 con 0x7fd83c06d9e0 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd842600640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:07.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd841c00640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c400 tx=0x7fd82c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0052d0 con 0x7fd83c06d9e0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c005450 con 0x7fd83c06d9e0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd824009400 con 0x7fd83c06d9e0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c007260 con 0x7fd83c06d9e0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.277+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c071eb0 con 0x7fd83c06d9e0 2024-01-16T15:13:07.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.278+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c04e840 con 0x7fd83c06d9e0 2024-01-16T15:13:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.280+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fd82c0073c0 con 0x7fd83c06d9e0 2024-01-16T15:13:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.282+0000 7fd839600640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 0x7fd81406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.282+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd82c087c90 con 0x7fd83c06d9e0 2024-01-16T15:13:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.282+0000 7fd841200640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 0x7fd81406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.282+0000 7fd841200640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 0x7fd81406e3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd824036000 tx=0x7fd82403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.286+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd82c0542d0 con 0x7fd83c06d9e0 2024-01-16T15:13:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:07.474+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd83c15cc10 con 0x7fd83c06d9e0 2024-01-16T15:13:08.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.099+0000 7fd839600640 1 -- 172.21.15.43:0/2352389160 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd82c0578f0 con 0x7fd83c06d9e0 2024-01-16T15:13:08.078 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:13:08.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.103+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 msgr2=0x7fd81406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.103+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 0x7fd81406e3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd824036000 tx=0x7fd82403c040 comp rx=0 tx=0).stop 2024-01-16T15:13:08.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.103+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 msgr2=0x7fd83c06cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.103+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c400 tx=0x7fd82c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 shutdown_connections 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd81406bf20 0x7fd81406e3e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd83c15f270 0x7fd83c13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd83c144770 0x7fd83c071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 --2- 172.21.15.43:0/2352389160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c06d9e0 0x7fd83c06cd90 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 >> 172.21.15.43:0/2352389160 conn(0x7fd83c0cec10 msgr2=0x7fd83c0ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 shutdown_connections 2024-01-16T15:13:08.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.104+0000 7fd843000640 1 -- 172.21.15.43:0/2352389160 wait complete. 2024-01-16T15:13:08.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:13:08.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.348+0000 7fc198a00640 1 Processor -- start 2024-01-16T15:13:08.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.348+0000 7fc198a00640 1 -- start start 2024-01-16T15:13:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.349+0000 7fc198a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc19415d3b0 0x7fc1940ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.349+0000 7fc198a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1940c6bf0 con 0x7fc1940d1d90 2024-01-16T15:13:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.349+0000 7fc198a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1940d1430 con 0x7fc1940d0fa0 2024-01-16T15:13:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.349+0000 7fc198a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1940cedc0 con 0x7fc19415d3b0 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc19415d3b0 0x7fc1940ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc1940d1d90 0x7fc1940b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56136/0 (socket says 172.21.15.43:56136) 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193e00640 1 -- 172.21.15.43:0/4145490424 learned_addr learned my addr 172.21.15.43:0/4145490424 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193400640 1 -- 172.21.15.43:0/4145490424 >> v1:172.21.15.69:6789/0 conn(0x7fc1940d0fa0 legacy=0x7fc1940d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193400640 1 -- 172.21.15.43:0/4145490424 >> v1:172.21.15.43:6789/0 conn(0x7fc1940d1d90 legacy=0x7fc1940b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc192000640 1 -- 172.21.15.43:0/4145490424 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967665307 0 0) 0x7fc1940c6bf0 con 0x7fc1940d1d90 2024-01-16T15:13:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.350+0000 7fc193400640 1 -- 172.21.15.43:0/4145490424 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940c6e40 con 0x7fc19415d3b0 2024-01-16T15:13:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.352+0000 7fc193400640 1 --2- 172.21.15.43:0/4145490424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc19415d3b0 0x7fc1940ce6c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fc1840097a0 tx=0x7fc18402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9878b0ba5554ed5a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.352+0000 7fc192000640 1 -- 172.21.15.43:0/4145490424 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184031830 con 0x7fc19415d3b0 2024-01-16T15:13:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.352+0000 7fc192000640 1 -- 172.21.15.43:0/4145490424 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc184031990 con 0x7fc19415d3b0 2024-01-16T15:13:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.352+0000 7fc192000640 1 -- 172.21.15.43:0/4145490424 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184031d20 con 0x7fc19415d3b0 2024-01-16T15:13:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 -- 172.21.15.43:0/4145490424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc19415d3b0 msgr2=0x7fc1940ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 --2- 172.21.15.43:0/4145490424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc19415d3b0 0x7fc1940ce6c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fc1840097a0 tx=0x7fc18402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 -- 172.21.15.43:0/4145490424 shutdown_connections 2024-01-16T15:13:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 --2- 172.21.15.43:0/4145490424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc19415d3b0 0x7fc1940ce6c0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 -- 172.21.15.43:0/4145490424 >> 172.21.15.43:0/4145490424 conn(0x7fc1940d30a0 msgr2=0x7fc1940d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 -- 172.21.15.43:0/4145490424 shutdown_connections 2024-01-16T15:13:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.353+0000 7fc198a00640 1 -- 172.21.15.43:0/4145490424 wait complete. 2024-01-16T15:13:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.356+0000 7fc198a00640 1 Processor -- start 2024-01-16T15:13:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.356+0000 7fc198a00640 1 -- start start 2024-01-16T15:13:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1940d0fa0 0x7fc19415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc19415d3b0 0x7fc1941f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19409dd00 con 0x7fc1940d1d90 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19409cde0 con 0x7fc19415d3b0 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc198a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940b8000 con 0x7fc1940d0fa0 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.357+0000 7fc192a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42982/0 (socket says 172.21.15.43:42982) 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 -- 172.21.15.43:0/2515318314 learned_addr learned my addr 172.21.15.43:0/2515318314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc193400640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1940d0fa0 0x7fc19415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc193e00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc19415d3b0 0x7fc1941f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 -- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1940d0fa0 msgr2=0x7fc19415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1940d0fa0 0x7fc19415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 -- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc19415d3b0 msgr2=0x7fc1941f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc19415d3b0 0x7fc1941f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 -- 172.21.15.43:0/2515318314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184009400 con 0x7fc1940d1d90 2024-01-16T15:13:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.358+0000 7fc192a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c450 tx=0x7fc17c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.359+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c0052d0 con 0x7fc1940d1d90 2024-01-16T15:13:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.359+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c005430 con 0x7fc1940d1d90 2024-01-16T15:13:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.359+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c012870 con 0x7fc1940d1d90 2024-01-16T15:13:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.359+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19409d080 con 0x7fc1940d1d90 2024-01-16T15:13:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.359+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19409d640 con 0x7fc1940d1d90 2024-01-16T15:13:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.360+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19409de40 con 0x7fc1940d1d90 2024-01-16T15:13:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.361+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc17c0129d0 con 0x7fc1940d1d90 2024-01-16T15:13:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.362+0000 7fc18b400640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 0x7fc16c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.362+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fc17c0880b0 con 0x7fc1940d1d90 2024-01-16T15:13:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.363+0000 7fc193400640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 0x7fc16c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.364+0000 7fc193400640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 0x7fc16c06e3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc184036000 tx=0x7fc18403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.367+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc19409de40 con 0x7fc1940d1d90 2024-01-16T15:13:08.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:08.565+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc19409de40 con 0x7fc1940d1d90 2024-01-16T15:13:09.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.117+0000 7fc18b400640 1 -- 172.21.15.43:0/2515318314 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc19409de40 con 0x7fc1940d1d90 2024-01-16T15:13:09.096 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:13:09.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 msgr2=0x7fc16c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 0x7fc16c06e3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc184036000 tx=0x7fc18403c040 comp rx=0 tx=0).stop 2024-01-16T15:13:09.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 msgr2=0x7fc1940b86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c450 tx=0x7fc17c00c910 comp rx=0 tx=0).stop 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 shutdown_connections 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc16c06bf20 0x7fc16c06e3e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc19415d3b0 0x7fc1941f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1940d1d90 0x7fc1940b86b0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 --2- 172.21.15.43:0/2515318314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc1940d0fa0 0x7fc19415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.121+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 >> 172.21.15.43:0/2515318314 conn(0x7fc1940d30a0 msgr2=0x7fc19406b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:09.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.122+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 shutdown_connections 2024-01-16T15:13:09.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.122+0000 7fc198a00640 1 -- 172.21.15.43:0/2515318314 wait complete. 2024-01-16T15:13:09.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:13:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.342+0000 7f19efc00640 1 Processor -- start 2024-01-16T15:13:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.342+0000 7f19efc00640 1 -- start start 2024-01-16T15:13:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19efc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19e815b350 0x7f19e80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19efc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e80adb60 0x7f19e80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19efc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e815b730 con 0x7f19e809cb30 2024-01-16T15:13:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19efc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e80ac740 con 0x7f19e815b350 2024-01-16T15:13:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19efc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80ac460 con 0x7f19e80adb60 2024-01-16T15:13:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19ed200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19e815b350 0x7f19e80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19e7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f19e809cb30 0x7f19e80b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56146/0 (socket says 172.21.15.43:56146) 2024-01-16T15:13:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19e7e00640 1 -- 172.21.15.43:0/1971697136 learned_addr learned my addr 172.21.15.43:0/1971697136 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.344+0000 7f19edc00640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e80adb60 0x7f19e80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.345+0000 7f19e7400640 1 -- 172.21.15.43:0/1971697136 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847451057 0 0) 0x7f19d0001110 con 0x7f19e809cb30 2024-01-16T15:13:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.345+0000 7f19ed200640 1 -- 172.21.15.43:0/1971697136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e80adb60 msgr2=0x7f19e80adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.345+0000 7f19ed200640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e80adb60 0x7f19e80adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.345+0000 7f19ed200640 1 -- 172.21.15.43:0/1971697136 >> v1:172.21.15.43:6789/0 conn(0x7f19e809cb30 legacy=0x7f19e80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.345+0000 7f19ed200640 1 -- 172.21.15.43:0/1971697136 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80b2a50 con 0x7f19e815b350 2024-01-16T15:13:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.347+0000 7f19ed200640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e815b350 0x7f19e80abe80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f19d80097a0 tx=0x7f19d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2e7679a20f0acd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.347+0000 7f19e7400640 1 -- 172.21.15.43:0/1971697136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8031830 con 0x7f19e815b350 2024-01-16T15:13:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.347+0000 7f19e7400640 1 -- 172.21.15.43:0/1971697136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8031990 con 0x7f19e815b350 2024-01-16T15:13:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.347+0000 7f19e7400640 1 -- 172.21.15.43:0/1971697136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8031d20 con 0x7f19e815b350 2024-01-16T15:13:09.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 -- 172.21.15.43:0/1971697136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e815b350 msgr2=0x7f19e80abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e815b350 0x7f19e80abe80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f19d80097a0 tx=0x7f19d802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 -- 172.21.15.43:0/1971697136 shutdown_connections 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e80adb60 0x7f19e80adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 --2- 172.21.15.43:0/1971697136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e815b350 0x7f19e80abe80 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 -- 172.21.15.43:0/1971697136 >> 172.21.15.43:0/1971697136 conn(0x7f19e80b0ba0 msgr2=0x7f19e80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 -- 172.21.15.43:0/1971697136 shutdown_connections 2024-01-16T15:13:09.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.348+0000 7f19efc00640 1 -- 172.21.15.43:0/1971697136 wait complete. 2024-01-16T15:13:09.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.350+0000 7f19efc00640 1 Processor -- start 2024-01-16T15:13:09.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.350+0000 7f19efc00640 1 -- start start 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e809cb30 0x7f19e81e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 0x7f19e813cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e81e0ca0 con 0x7f19e80adb60 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e813bf30 con 0x7f19e815b350 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19efc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e813bc00 con 0x7f19e809cb30 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43002/0 (socket says 172.21.15.43:43002) 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19e7e00640 1 -- 172.21.15.43:0/3020748660 learned_addr learned my addr 172.21.15.43:0/3020748660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19ed200640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e809cb30 0x7f19e81e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.351+0000 7f19edc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 0x7f19e813cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 -- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e809cb30 msgr2=0x7f19e81e05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e809cb30 0x7f19e81e05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 -- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 msgr2=0x7f19e813cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 0x7f19e813cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 -- 172.21.15.43:0/3020748660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8009400 con 0x7f19e80adb60 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19edc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 0x7f19e813cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e7e00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c780 tx=0x7f19d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00072d0 con 0x7f19e80adb60 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0007450 con 0x7f19e80adb60 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00047e0 con 0x7f19e80adb60 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e813c180 con 0x7f19e80adb60 2024-01-16T15:13:09.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.352+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e813c740 con 0x7f19e80adb60 2024-01-16T15:13:09.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.354+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e809d200 con 0x7f19e80adb60 2024-01-16T15:13:09.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.355+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f19d0004960 con 0x7f19e80adb60 2024-01-16T15:13:09.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.356+0000 7f19e4c00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 0x7f19c006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.357+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f19d0087a10 con 0x7f19e80adb60 2024-01-16T15:13:09.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.357+0000 7f19ed200640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 0x7f19c006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.358+0000 7f19ed200640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 0x7f19c006e3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f19d8036000 tx=0x7f19d803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.363+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19d0053fc0 con 0x7f19e80adb60 2024-01-16T15:13:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:09.592+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f19e815b730 con 0x7f19e80adb60 2024-01-16T15:13:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.141+0000 7f19e4c00640 1 -- 172.21.15.43:0/3020748660 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7f19e815b730 con 0x7f19e80adb60 2024-01-16T15:13:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 12 and data pool 13 2024-01-16T15:13:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.155+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 msgr2=0x7f19c006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.155+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 0x7f19c006e3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f19d8036000 tx=0x7f19d803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.155+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 msgr2=0x7f19e813b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.155+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c780 tx=0x7f19d000cc40 comp rx=0 tx=0).stop 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 shutdown_connections 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19c006bf20 0x7f19c006e3e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19e815b350 0x7f19e813cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19e80adb60 0x7f19e813b6c0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 --2- 172.21.15.43:0/3020748660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19e809cb30 0x7f19e81e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.157+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 >> 172.21.15.43:0/3020748660 conn(0x7f19e80b0ba0 msgr2=0x7f19e809ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.158+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 shutdown_connections 2024-01-16T15:13:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.158+0000 7f19efc00640 1 -- 172.21.15.43:0/3020748660 wait complete. 2024-01-16T15:13:10.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:10.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.367+0000 7fcfd9000640 1 Processor -- start 2024-01-16T15:13:10.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.367+0000 7fcfd9000640 1 -- start start 2024-01-16T15:13:10.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd9000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcfd4160970 0x7fcfd40b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd9000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcfd406dc30 0x7fcfd40acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd9000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd4071cd0 con 0x7fcfd40b3140 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd9000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40b2120 con 0x7fcfd4160970 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd9000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40b2400 con 0x7fcfd406dc30 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.368+0000 7fcfd1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcfd4160970 0x7fcfd40b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcfd4160970 0x7fcfd40b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43010/0 (socket says 172.21.15.43:43010) 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 -- 172.21.15.43:0/2120357563 learned_addr learned my addr 172.21.15.43:0/2120357563 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd2800640 1 --2- 172.21.15.43:0/2120357563 >> v2:172.21.15.69:3300/0 conn(0x7fcfd406dc30 0x7fcfd40acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 -- 172.21.15.43:0/2120357563 >> v2:172.21.15.69:3300/0 conn(0x7fcfd406dc30 msgr2=0x7fcfd40acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 --2- 172.21.15.43:0/2120357563 >> v2:172.21.15.69:3300/0 conn(0x7fcfd406dc30 0x7fcfd40acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 -- 172.21.15.43:0/2120357563 >> v1:172.21.15.69:6789/0 conn(0x7fcfd40b3140 legacy=0x7fcfd406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.369+0000 7fcfd1e00640 1 -- 172.21.15.43:0/2120357563 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40b26e0 con 0x7fcfd4160970 2024-01-16T15:13:10.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.370+0000 7fcfd1e00640 1 --2- 172.21.15.43:0/2120357563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd4160970 0x7fcfd40b2c00 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4072910 tx=0x7fcfc402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e545163ea071a43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd0a00640 1 -- 172.21.15.43:0/2120357563 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40317e0 con 0x7fcfd4160970 2024-01-16T15:13:10.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd0a00640 1 -- 172.21.15.43:0/2120357563 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc4031940 con 0x7fcfd4160970 2024-01-16T15:13:10.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd0a00640 1 -- 172.21.15.43:0/2120357563 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4031cd0 con 0x7fcfd4160970 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 -- 172.21.15.43:0/2120357563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd4160970 msgr2=0x7fcfd40b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 --2- 172.21.15.43:0/2120357563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd4160970 0x7fcfd40b2c00 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4072910 tx=0x7fcfc402b880 comp rx=0 tx=0).stop 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 -- 172.21.15.43:0/2120357563 shutdown_connections 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 --2- 172.21.15.43:0/2120357563 >> v2:172.21.15.69:3300/0 conn(0x7fcfd406dc30 0x7fcfd40acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 --2- 172.21.15.43:0/2120357563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd4160970 0x7fcfd40b2c00 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 -- 172.21.15.43:0/2120357563 >> 172.21.15.43:0/2120357563 conn(0x7fcfd40b0d80 msgr2=0x7fcfd40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 -- 172.21.15.43:0/2120357563 shutdown_connections 2024-01-16T15:13:10.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.371+0000 7fcfd9000640 1 -- 172.21.15.43:0/2120357563 wait complete. 2024-01-16T15:13:10.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.373+0000 7fcfd9000640 1 Processor -- start 2024-01-16T15:13:10.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.373+0000 7fcfd9000640 1 -- start start 2024-01-16T15:13:10.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd40b3140 0x7fcfd40cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd4160970 0x7fcfd409b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd414a2a0 con 0x7fcfd406dc30 2024-01-16T15:13:10.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd4149b80 con 0x7fcfd4160970 2024-01-16T15:13:10.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd9000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd409bdb0 con 0x7fcfd40b3140 2024-01-16T15:13:10.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43024/0 (socket says 172.21.15.43:43024) 2024-01-16T15:13:10.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 -- 172.21.15.43:0/579934864 learned_addr learned my addr 172.21.15.43:0/579934864 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 -- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd40b3140 msgr2=0x7fcfd40cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd40b3140 0x7fcfd40cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 -- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd4160970 msgr2=0x7fcfd409b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd2800640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd4160970 0x7fcfd409b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd4160970 0x7fcfd409b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.375+0000 7fcfd1e00640 1 -- 172.21.15.43:0/579934864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc40094d0 con 0x7fcfd406dc30 2024-01-16T15:13:10.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfd1e00640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4000c00 tx=0x7fcfc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4032880 con 0x7fcfd406dc30 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc4032a00 con 0x7fcfd406dc30 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4032db0 con 0x7fcfd406dc30 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd409c090 con 0x7fcfd406dc30 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.376+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd409ba70 con 0x7fcfd406dc30 2024-01-16T15:13:10.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.378+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd409c5b0 con 0x7fcfd406dc30 2024-01-16T15:13:10.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.379+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fcfc404d3f0 con 0x7fcfd406dc30 2024-01-16T15:13:10.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.380+0000 7fcfca000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 0x7fcfac06e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.380+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcfc40aa6d0 con 0x7fcfd406dc30 2024-01-16T15:13:10.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.380+0000 7fcfd1400640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 0x7fcfac06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.381+0000 7fcfd1400640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 0x7fcfac06e430 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcfd41f7580 tx=0x7fcfbc0079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.384+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfd409c5b0 con 0x7fcfd406dc30 2024-01-16T15:13:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.578+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfd414a070 con 0x7fcfd406dc30 2024-01-16T15:13:10.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.580+0000 7fcfca000640 1 -- 172.21.15.43:0/579934864 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16568 (secure 0 0 0) 0x7fcfd414a070 con 0x7fcfd406dc30 2024-01-16T15:13:10.559 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:10.560 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":87,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:13:09.594220+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:07.528802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:08.622339+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 msgr2=0x7fcfac06e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 0x7fcfac06e430 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcfd41f7580 tx=0x7fcfbc0079d0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 msgr2=0x7fcfd414cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4000c00 tx=0x7fcfc40046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 shutdown_connections 2024-01-16T15:13:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcfac06bf70 0x7fcfac06e430 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd4160970 0x7fcfd409b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcfd40b3140 0x7fcfd40cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 --2- 172.21.15.43:0/579934864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcfd406dc30 0x7fcfd414cca0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.583+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 >> 172.21.15.43:0/579934864 conn(0x7fcfd40b0d80 msgr2=0x7fcfd40710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.584+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 shutdown_connections 2024-01-16T15:13:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.584+0000 7fcfd9000640 1 -- 172.21.15.43:0/579934864 wait complete. 2024-01-16T15:13:10.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:10.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.826+0000 7f548b200640 1 Processor -- start 2024-01-16T15:13:10.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.826+0000 7f548b200640 1 -- start start 2024-01-16T15:13:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.827+0000 7f548b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 0x7f54840b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.827+0000 7f548b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548415c360 con 0x7f548406d520 2024-01-16T15:13:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f548b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548415ba30 con 0x7f54840abff0 2024-01-16T15:13:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f548b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54840b41a0 con 0x7f54840ad720 2024-01-16T15:13:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f5483e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f548406d520 0x7f548406d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56158/0 (socket says 172.21.15.43:56158) 2024-01-16T15:13:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f5483e00640 1 -- 172.21.15.43:0/1926200134 learned_addr learned my addr 172.21.15.43:0/1926200134 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f547b200640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 0x7f54840b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.828+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171095286 0 0) 0x7f548415c360 con 0x7f548406d520 2024-01-16T15:13:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5464003520 con 0x7f548406d520 2024-01-16T15:13:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188203625 0 0) 0x7f5464003520 con 0x7f548406d520 2024-01-16T15:13:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 msgr2=0x7f54840b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 0x7f54840b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 >> v1:172.21.15.69:6789/0 conn(0x7f54840abff0 legacy=0x7f54840b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548415bd10 con 0x7f548406d520 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f5474001670 con 0x7f548406d520 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 >> v1:172.21.15.43:6789/0 conn(0x7f548406d520 legacy=0x7f548406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54640039a0 0x7f5464003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5464004350 0x7f5464004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 0x7f5464005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546400b980 con 0x7f54640039a0 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546400baf0 con 0x7f5464004350 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546400bcb0 con 0x7f54840ad720 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.829+0000 7f5483e00640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54640039a0 0x7f5464003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f547b200640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5464004350 0x7f5464004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f5483e00640 1 -- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 msgr2=0x7f5464005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f5483e00640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54840ad720 0x7f5464005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f5483e00640 1 -- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5464004350 msgr2=0x7f5464004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f5483e00640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5464004350 0x7f5464004820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.830+0000 7f5483e00640 1 -- 172.21.15.43:0/1926200134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464003520 con 0x7f54640039a0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.831+0000 7f5483e00640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54640039a0 0x7f5464003d80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f547400c240 tx=0x7f54640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cec273bd46ed12 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.831+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474005ad0 con 0x7f54640039a0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.831+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474005c30 con 0x7f54640039a0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.831+0000 7f5483400640 1 -- 172.21.15.43:0/1926200134 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547402cb10 con 0x7f54640039a0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 -- 172.21.15.43:0/1926200134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54640039a0 msgr2=0x7f5464003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 --2- 172.21.15.43:0/1926200134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54640039a0 0x7f5464003d80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f547400c240 tx=0x7f54640057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f5489200640 1 -- 172.21.15.43:0/1926200134 reap_dead start 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 -- 172.21.15.43:0/1926200134 shutdown_connections 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 -- 172.21.15.43:0/1926200134 >> 172.21.15.43:0/1926200134 conn(0x7f54840b0d80 msgr2=0x7f54840b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 -- 172.21.15.43:0/1926200134 shutdown_connections 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.832+0000 7f548b200640 1 -- 172.21.15.43:0/1926200134 wait complete. 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.834+0000 7f548b200640 1 Processor -- start 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.834+0000 7f548b200640 1 -- start start 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 0x7f54841f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 0x7f5484071590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54840c8400 con 0x7f54841f2ce0 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54841f34f0 con 0x7f54640039a0 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f548b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54841f37d0 con 0x7f5484149af0 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f547b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f547b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43044/0 (socket says 172.21.15.43:43044) 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f547b200640 1 -- 172.21.15.43:0/3708724527 learned_addr learned my addr 172.21.15.43:0/3708724527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:10.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f5483e00640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 0x7f54841f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.835+0000 7f5489200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 0x7f5484071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 -- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 msgr2=0x7f5484071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 0x7f5484071590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 -- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 msgr2=0x7f54841f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 0x7f54841f6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 -- 172.21.15.43:0/3708724527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54841f3ab0 con 0x7f54841f2ce0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f5483e00640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 0x7f54841f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f5489200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 0x7f5484071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f547b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f546c010890 tx=0x7f546c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00eb60 con 0x7f54841f2ce0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00ece0 con 0x7f54841f2ce0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c01c640 con 0x7f54841f2ce0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54840c8650 con 0x7f54841f2ce0 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.836+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54840cac30 con 0x7f54841f2ce0 2024-01-16T15:13:10.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.838+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54841f3cf0 con 0x7f54841f2ce0 2024-01-16T15:13:10.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.839+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f546c0143f0 con 0x7f54841f2ce0 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.840+0000 7f5480c00640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 0x7f545c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.840+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f546c0885d0 con 0x7f54841f2ce0 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.840+0000 7f5483e00640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 0x7f545c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.841+0000 7f5483e00640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 0x7f545c06e3e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5474034000 tx=0x7f5474009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:10.845+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f546c054ef0 con 0x7f54841f2ce0 2024-01-16T15:13:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.041+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54840caeb0 con 0x7f54841f2ce0 2024-01-16T15:13:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.043+0000 7f5480c00640 1 -- 172.21.15.43:0/3708724527 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f546c058510 con 0x7f54841f2ce0 2024-01-16T15:13:11.022 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:11.022 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":87,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:13:09.594220+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:07.528802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:08.622339+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 msgr2=0x7f545c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 0x7f545c06e3e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5474034000 tx=0x7f5474009030 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 msgr2=0x7f54840ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f546c010890 tx=0x7f546c010d50 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 shutdown_connections 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f545c06bf20 0x7f545c06e3e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5484149af0 0x7f5484071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54841f2ce0 0x7f54840ca6f0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 --2- 172.21.15.43:0/3708724527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640039a0 0x7f54841f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 >> 172.21.15.43:0/3708724527 conn(0x7f54840b0d80 msgr2=0x7f54840acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.046+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 shutdown_connections 2024-01-16T15:13:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.047+0000 7f548b200640 1 -- 172.21.15.43:0/3708724527 wait complete. 2024-01-16T15:13:11.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:13:11.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.290+0000 7f07aee00640 1 Processor -- start 2024-01-16T15:13:11.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.291+0000 7f07aee00640 1 -- start start 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.292+0000 7f07aee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f07a806ee30 0x7f07a8168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.292+0000 7f07aee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a8142d50 con 0x7f07a80cd150 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.292+0000 7f07aee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a806f430 con 0x7f07a8070470 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.292+0000 7f07aee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a80cd530 con 0x7f07a806ee30 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07a7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f07a80cd150 0x7f07a806fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56162/0 (socket says 172.21.15.43:56162) 2024-01-16T15:13:11.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07a7e00640 1 -- 172.21.15.43:0/295885791 learned_addr learned my addr 172.21.15.43:0/295885791 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07ace00640 1 --2- 172.21.15.43:0/295885791 >> v2:172.21.15.69:3300/0 conn(0x7f07a806ee30 0x7f07a8168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498355370 0 0) 0x7f07a8142d50 con 0x7f07a80cd150 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0788003520 con 0x7f07a80cd150 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.293+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166984393 0 0) 0x7f0788003520 con 0x7f07a80cd150 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 >> v2:172.21.15.69:3300/0 conn(0x7f07a806ee30 msgr2=0x7f07a8168120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 --2- 172.21.15.43:0/295885791 >> v2:172.21.15.69:3300/0 conn(0x7f07a806ee30 0x7f07a8168120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 >> v1:172.21.15.69:6789/0 conn(0x7f07a8070470 legacy=0x7f07a806e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8142fa0 con 0x7f07a80cd150 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0798001670 con 0x7f07a80cd150 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 >> v1:172.21.15.43:6789/0 conn(0x7f07a80cd150 legacy=0x7f07a806fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0788003970 0x7f0788003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07880047c0 0x7f0788004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f078800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80cd530 con 0x7f07880052a0 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078800c150 con 0x7f07880047c0 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078800c310 con 0x7f0788003970 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f078800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07a7400640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07880047c0 0x7f0788004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 -- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0788003970 msgr2=0x7f0788003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0788003970 0x7f0788003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 -- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07880047c0 msgr2=0x7f0788004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:11.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07880047c0 0x7f0788004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.294+0000 7f07ace00640 1 -- 172.21.15.43:0/295885791 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788003520 con 0x7f07880052a0 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.295+0000 7f07ace00640 1 -- 172.21.15.43:0/295885791 reap_dead start 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07ace00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f078800b980 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f07a000ba00 tx=0x7f07880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b676773edd467d90 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0004020 con 0x7f07880052a0 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0004180 con 0x7f07880052a0 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07a6a00640 1 -- 172.21.15.43:0/295885791 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0003610 con 0x7f07880052a0 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07aee00640 1 -- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 msgr2=0x7f078800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.296+0000 7f07aee00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f078800b980 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f07a000ba00 tx=0x7f07880057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.297+0000 7f07aee00640 1 -- 172.21.15.43:0/295885791 shutdown_connections 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.297+0000 7f07aee00640 1 --2- 172.21.15.43:0/295885791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f078800b980 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.297+0000 7f07aee00640 1 -- 172.21.15.43:0/295885791 >> 172.21.15.43:0/295885791 conn(0x7f07a80cdb20 msgr2=0x7f07a80cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:11.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.297+0000 7f07aee00640 1 -- 172.21.15.43:0/295885791 shutdown_connections 2024-01-16T15:13:11.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.297+0000 7f07aee00640 1 -- 172.21.15.43:0/295885791 wait complete. 2024-01-16T15:13:11.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.299+0000 7f07aee00640 1 Processor -- start 2024-01-16T15:13:11.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.299+0000 7f07aee00640 1 -- start start 2024-01-16T15:13:11.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 0x7f07a80c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07a806b850 0x7f07a81582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c6580 con 0x7f07880052a0 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c6860 con 0x7f07a80cad10 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07aee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c6b40 con 0x7f07a806b850 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43070/0 (socket says 172.21.15.43:43070) 2024-01-16T15:13:11.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07a7e00640 1 -- 172.21.15.43:0/1148067098 learned_addr learned my addr 172.21.15.43:0/1148067098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:11.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.300+0000 7f07ace00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07a806b850 0x7f07a81582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7400640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 0x7f07a80c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 -- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07a806b850 msgr2=0x7f07a81582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07a806b850 0x7f07a81582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 -- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 msgr2=0x7f07a80c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 0x7f07a80c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 -- 172.21.15.43:0/1148067098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8143c00 con 0x7f07880052a0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7400640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 0x7f07a80c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07a7e00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f079800c700 tx=0x7f079800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980072d0 con 0x7f07880052a0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798007430 con 0x7f07880052a0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980047e0 con 0x7f07880052a0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a80c6e20 con 0x7f07880052a0 2024-01-16T15:13:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.301+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a80c6240 con 0x7f07880052a0 2024-01-16T15:13:11.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.303+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a80c7340 con 0x7f07880052a0 2024-01-16T15:13:11.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.303+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0798004940 con 0x7f07880052a0 2024-01-16T15:13:11.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.304+0000 7f079fe00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 0x7f078006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.305+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0798087b10 con 0x7f07880052a0 2024-01-16T15:13:11.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.305+0000 7f07a7400640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 0x7f078006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.305+0000 7f07a7400640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 0x7f078006e3e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f07a80d20e0 tx=0x7f0790009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.309+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07a80c7340 con 0x7f07880052a0 2024-01-16T15:13:11.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:11.537+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f07a80c7340 con 0x7f07880052a0 2024-01-16T15:13:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.178+0000 7f079fe00640 1 -- 172.21.15.43:0/1148067098 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f07a80c7340 con 0x7f07880052a0 2024-01-16T15:13:12.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 msgr2=0x7f078006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 0x7f078006e3e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f07a80d20e0 tx=0x7f0790009040 comp rx=0 tx=0).stop 2024-01-16T15:13:12.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 msgr2=0x7f07a80c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f079800c700 tx=0x7f079800cbc0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 shutdown_connections 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f078006bf20 0x7f078006e3e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07a80cad10 0x7f07a80c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07a806b850 0x7f07a81582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 --2- 172.21.15.43:0/1148067098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07880052a0 0x7f07a80c7620 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.181+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 >> 172.21.15.43:0/1148067098 conn(0x7f07a80cdb20 msgr2=0x7f07a815ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.182+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 shutdown_connections 2024-01-16T15:13:12.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.182+0000 7f07aee00640 1 -- 172.21.15.43:0/1148067098 wait complete. 2024-01-16T15:13:12.181 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:13:12.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:13:12.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.425+0000 7f4548a00640 1 Processor -- start 2024-01-16T15:13:12.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.425+0000 7f4548a00640 1 -- start start 2024-01-16T15:13:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4548a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f454415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4548a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45440b35a0 0x7f454415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4548a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454415f930 con 0x7f454406c9d0 2024-01-16T15:13:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4548a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45440b43b0 con 0x7f45440b35a0 2024-01-16T15:13:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4548a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45440b3b10 con 0x7f4544144770 2024-01-16T15:13:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4543200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f454406c9d0 0x7f454406ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56170/0 (socket says 172.21.15.43:56170) 2024-01-16T15:13:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4543200640 1 -- 172.21.15.43:0/1845997407 learned_addr learned my addr 172.21.15.43:0/1845997407 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:12.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f4542800640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f454415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.427+0000 7f453be00640 1 --2- 172.21.15.43:0/1845997407 >> v2:172.21.15.69:3300/0 conn(0x7f45440b35a0 0x7f454415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530111955 0 0) 0x7f454415f930 con 0x7f454406c9d0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4524003520 con 0x7f454406c9d0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3459238377 0 0) 0x7f4524003520 con 0x7f454406c9d0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 msgr2=0x7f454415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f454415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 >> v2:172.21.15.69:3300/0 conn(0x7f45440b35a0 msgr2=0x7f454415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 --2- 172.21.15.43:0/1845997407 >> v2:172.21.15.69:3300/0 conn(0x7f45440b35a0 0x7f454415f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440b3df0 con 0x7f454406c9d0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f453c0019f0 con 0x7f454406c9d0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 >> v1:172.21.15.43:6789/0 conn(0x7f454406c9d0 legacy=0x7f454406ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45240039e0 0x7f4524003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4524004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f4524005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440b43b0 con 0x7f45240047c0 2024-01-16T15:13:12.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440b3b10 con 0x7f45240039e0 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452400b980 con 0x7f4544144770 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.428+0000 7f453be00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4524004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f4542800640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45240039e0 0x7f4524003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f4543200640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f4524005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f453be00640 1 -- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 msgr2=0x7f4524005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f453be00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f4524005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f453be00640 1 -- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45240039e0 msgr2=0x7f4524003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f453be00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45240039e0 0x7f4524003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f453be00640 1 -- 172.21.15.43:0/1845997407 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524003520 con 0x7f45240047c0 2024-01-16T15:13:12.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.429+0000 7f4543200640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4544144770 0x7f4524005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.430+0000 7f453be00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4524004c40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f452c009af0 tx=0x7f45240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c252a68a7725be0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.430+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c02c710 con 0x7f45240047c0 2024-01-16T15:13:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.430+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452c02c870 con 0x7f45240047c0 2024-01-16T15:13:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.430+0000 7f4541e00640 1 -- 172.21.15.43:0/1845997407 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c02cc00 con 0x7f45240047c0 2024-01-16T15:13:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 -- 172.21.15.43:0/1845997407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 msgr2=0x7f4524004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 --2- 172.21.15.43:0/1845997407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4524004c40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f452c009af0 tx=0x7f45240057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4543200640 1 -- 172.21.15.43:0/1845997407 reap_dead start 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 -- 172.21.15.43:0/1845997407 shutdown_connections 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 -- 172.21.15.43:0/1845997407 >> 172.21.15.43:0/1845997407 conn(0x7f45440cec10 msgr2=0x7f45440cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 -- 172.21.15.43:0/1845997407 shutdown_connections 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.431+0000 7f4548a00640 1 -- 172.21.15.43:0/1845997407 wait complete. 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.433+0000 7f4548a00640 1 Processor -- start 2024-01-16T15:13:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.433+0000 7f4548a00640 1 -- start start 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 0x7f454415a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 0x7f454415d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454415d760 con 0x7f45240047c0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544141980 con 0x7f454413f6e0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4548a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441410e0 con 0x7f454415cda0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4542800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4542800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43088/0 (socket says 172.21.15.43:43088) 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.434+0000 7f4542800640 1 -- 172.21.15.43:0/2096417015 learned_addr learned my addr 172.21.15.43:0/2096417015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f453be00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 0x7f454415a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4543200640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 0x7f454415d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 -- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 msgr2=0x7f454415d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 0x7f454415d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 -- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 msgr2=0x7f454415a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 0x7f454415a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 -- 172.21.15.43:0/2096417015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45441413c0 con 0x7f45240047c0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f453be00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 0x7f454415a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4543200640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 0x7f454415d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4542800640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f4534011600 tx=0x7f4534011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453400f920 con 0x7f45240047c0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453400fa80 con 0x7f45240047c0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453400fdf0 con 0x7f45240047c0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.435+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441416a0 con 0x7f45240047c0 2024-01-16T15:13:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.436+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4544140dd0 con 0x7f45240047c0 2024-01-16T15:13:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.437+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454404e840 con 0x7f45240047c0 2024-01-16T15:13:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.438+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4534015460 con 0x7f45240047c0 2024-01-16T15:13:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.439+0000 7f453aa00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 0x7f451c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.439+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f45340889e0 con 0x7f45240047c0 2024-01-16T15:13:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.440+0000 7f453be00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 0x7f451c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.440+0000 7f453be00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 0x7f451c06e3e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f452c036000 tx=0x7f452c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.444+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45340552a0 con 0x7f45240047c0 2024-01-16T15:13:12.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:12.676+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f45440b2460 con 0x7f45240047c0 2024-01-16T15:13:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.181+0000 7f453aa00640 1 -- 172.21.15.43:0/2096417015 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f45340588c0 con 0x7f45240047c0 2024-01-16T15:13:13.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 msgr2=0x7f451c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 0x7f451c06e3e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f452c036000 tx=0x7f452c030000 comp rx=0 tx=0).stop 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 msgr2=0x7f4544140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f4534011600 tx=0x7f4534011ac0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 shutdown_connections 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f451c06bf20 0x7f451c06e3e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f454413f6e0 0x7f454415a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f454415cda0 0x7f454415d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 --2- 172.21.15.43:0/2096417015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45240047c0 0x7f4544140890 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 >> 172.21.15.43:0/2096417015 conn(0x7f45440cec10 msgr2=0x7f45440ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.185+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 shutdown_connections 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.186+0000 7f4548a00640 1 -- 172.21.15.43:0/2096417015 wait complete. 2024-01-16T15:13:13.189 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:13:13.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:13.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.430+0000 7f1726e00640 1 Processor -- start 2024-01-16T15:13:13.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.431+0000 7f1726e00640 1 -- start start 2024-01-16T15:13:13.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.432+0000 7f1726e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17200abff0 0x7f17200b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.432+0000 7f1726e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172015c360 con 0x7f17200ad720 2024-01-16T15:13:13.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.432+0000 7f1726e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172015ba30 con 0x7f172006d520 2024-01-16T15:13:13.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.432+0000 7f1726e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17200b41a0 con 0x7f17200abff0 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f17200ad720 0x7f17200b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33054/0 (socket says 172.21.15.43:33054) 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171f400640 1 -- 172.21.15.43:0/3915735021 learned_addr learned my addr 172.21.15.43:0/3915735021 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f1724e00640 1 --2- 172.21.15.43:0/3915735021 >> v2:172.21.15.69:3300/0 conn(0x7f17200abff0 0x7f17200b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191581067 0 0) 0x7f172015c360 con 0x7f17200ad720 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1700003520 con 0x7f17200ad720 2024-01-16T15:13:13.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631786800 0 0) 0x7f1700003520 con 0x7f17200ad720 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 >> v2:172.21.15.69:3300/0 conn(0x7f17200abff0 msgr2=0x7f17200b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 --2- 172.21.15.43:0/3915735021 >> v2:172.21.15.69:3300/0 conn(0x7f17200abff0 0x7f17200b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 >> v1:172.21.15.69:6789/0 conn(0x7f172006d520 legacy=0x7f172006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.433+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172015bd10 con 0x7f17200ad720 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f17080019f0 con 0x7f17200ad720 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 >> v1:172.21.15.43:6789/0 conn(0x7f17200ad720 legacy=0x7f17200b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1700003970 0x7f1700003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 0x7f1700004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1700005180 0x7f170000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170000bfd0 con 0x7f17000047c0 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170000c140 con 0x7f1700005180 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170000c300 con 0x7f1700003970 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f1724e00640 1 --2- 172.21.15.43:0/3915735021 >> v2:172.21.15.69:3300/0 conn(0x7f17200abff0 0x7f17200b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:13.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 0x7f1700004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f1724e00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1700005180 0x7f170000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171fe00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1700003970 0x7f1700003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 -- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1700003970 msgr2=0x7f1700003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1700003970 0x7f1700003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 -- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1700005180 msgr2=0x7f170000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1700005180 0x7f170000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f171f400640 1 -- 172.21.15.43:0/3915735021 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1700003520 con 0x7f17000047c0 2024-01-16T15:13:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.434+0000 7f1724e00640 1 -- 172.21.15.43:0/3915735021 reap_dead start 2024-01-16T15:13:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f171f400640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 0x7f1700004c40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f172006fdc0 tx=0x7f17000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4e7137fc9fc545 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708034630 con 0x7f17000047c0 2024-01-16T15:13:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170802f070 con 0x7f17000047c0 2024-01-16T15:13:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f171ea00640 1 -- 172.21.15.43:0/3915735021 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17080349d0 con 0x7f17000047c0 2024-01-16T15:13:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f1726e00640 1 -- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 msgr2=0x7f1700004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.436+0000 7f1726e00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 0x7f1700004c40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f172006fdc0 tx=0x7f17000057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.437+0000 7f1726e00640 1 -- 172.21.15.43:0/3915735021 shutdown_connections 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.437+0000 7f1726e00640 1 --2- 172.21.15.43:0/3915735021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17000047c0 0x7f1700004c40 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.437+0000 7f1726e00640 1 -- 172.21.15.43:0/3915735021 >> 172.21.15.43:0/3915735021 conn(0x7f17200b0d80 msgr2=0x7f17200b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.437+0000 7f1726e00640 1 -- 172.21.15.43:0/3915735021 shutdown_connections 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.437+0000 7f1726e00640 1 -- 172.21.15.43:0/3915735021 wait complete. 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.438+0000 7f1726e00640 1 Processor -- start 2024-01-16T15:13:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.439+0000 7f1726e00640 1 -- start start 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17000047c0 0x7f17201f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 0x7f172015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201f6f30 con 0x7f17200c8670 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201f3000 con 0x7f17000047c0 2024-01-16T15:13:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1726e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17201f2a10 con 0x7f172015abb0 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f171f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f171f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42442/0 (socket says 172.21.15.43:42442) 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f171f400640 1 -- 172.21.15.43:0/3997372577 learned_addr learned my addr 172.21.15.43:0/3997372577 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f171fe00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17000047c0 0x7f17201f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.440+0000 7f1724e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 0x7f172015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 -- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 msgr2=0x7f172015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 0x7f172015ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 -- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17000047c0 msgr2=0x7f17201f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17000047c0 0x7f17201f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 -- 172.21.15.43:0/3997372577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17201f3960 con 0x7f17200c8670 2024-01-16T15:13:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f1724e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 0x7f172015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:13.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f171f400640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f170802cb30 tx=0x7f1708001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708034430 con 0x7f17200c8670 2024-01-16T15:13:13.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1708034590 con 0x7f17200c8670 2024-01-16T15:13:13.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708034920 con 0x7f17200c8670 2024-01-16T15:13:13.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.441+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17201f6970 con 0x7f17200c8670 2024-01-16T15:13:13.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.442+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17201f2c20 con 0x7f17200c8670 2024-01-16T15:13:13.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.443+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17201f6e90 con 0x7f17200c8670 2024-01-16T15:13:13.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.444+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f1708031030 con 0x7f17200c8670 2024-01-16T15:13:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.445+0000 7f1717e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 0x7f16f806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.445+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f17080aa510 con 0x7f17200c8670 2024-01-16T15:13:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.445+0000 7f171fe00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 0x7f16f806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.446+0000 7f171fe00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 0x7f16f806e3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1710000c00 tx=0x7f1710009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.450+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1708076ef0 con 0x7f17200c8670 2024-01-16T15:13:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.685+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17201f6e90 con 0x7f17200c8670 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.685+0000 7f1717e00640 1 -- 172.21.15.43:0/3997372577 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f170804b090 con 0x7f17200c8670 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/206348255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":206348255},{"type":"v1","addr":"172.21.15.69:6834","nonce":206348255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3389933703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3389933703},{"type":"v1","addr":"172.21.15.69:6835","nonce":3389933703}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/1040529098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1040529098},{"type":"v1","addr":"172.21.15.43:6835","nonce":1040529098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:13:09.595061+0000","modified":"2024-01-16T15:13:12.678462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.43:6837/3379600529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3379600529},{"type":"v1","addr":"172.21.15.43:6837","nonce":3379600529}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:13:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 57 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 msgr2=0x7f16f806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 0x7f16f806e3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1710000c00 tx=0x7f1710009040 comp rx=0 tx=0).stop 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 msgr2=0x7f17201f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f170802cb30 tx=0x7f1708001d80 comp rx=0 tx=0).stop 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 shutdown_connections 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f16f806bf20 0x7f16f806e3e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f172015abb0 0x7f172015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17200c8670 0x7f17201f6000 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 --2- 172.21.15.43:0/3997372577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17000047c0 0x7f17201f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.688+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 >> 172.21.15.43:0/3997372577 conn(0x7f17200b0d80 msgr2=0x7f172006d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.689+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 shutdown_connections 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.689+0000 7f1726e00640 1 -- 172.21.15.43:0/3997372577 wait complete. 2024-01-16T15:13:13.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.925+0000 7f6261a00640 1 Processor -- start 2024-01-16T15:13:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.925+0000 7f6261a00640 1 -- start start 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f6261a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f625c06d520 0x7f625c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f6261a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c0ad720 0x7f625c15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f6261a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c15c360 con 0x7f625c0abff0 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f6261a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625c0b41a0 con 0x7f625c06d520 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f6261a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c06de40 con 0x7f625c0ad720 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f625aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f625c06d520 0x7f625c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f625aa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f625c06d520 0x7f625c06d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42444/0 (socket says 172.21.15.43:42444) 2024-01-16T15:13:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.927+0000 7f625aa00640 1 -- 172.21.15.43:0/369398034 learned_addr learned my addr 172.21.15.43:0/369398034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.928+0000 7f625a000640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c0ad720 0x7f625c15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.928+0000 7f625aa00640 1 -- 172.21.15.43:0/369398034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c0ad720 msgr2=0x7f625c15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.928+0000 7f625aa00640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c0ad720 0x7f625c15ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.928+0000 7f625aa00640 1 -- 172.21.15.43:0/369398034 >> v1:172.21.15.69:6789/0 conn(0x7f625c0abff0 legacy=0x7f625c15f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.928+0000 7f625aa00640 1 -- 172.21.15.43:0/369398034 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c15ba30 con 0x7f625c06d520 2024-01-16T15:13:13.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.930+0000 7f625aa00640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c06d520 0x7f625c06d900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6264f28f994f3927 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.930+0000 7f6259600640 1 -- 172.21.15.43:0/369398034 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c030840 con 0x7f625c06d520 2024-01-16T15:13:13.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.930+0000 7f6259600640 1 -- 172.21.15.43:0/369398034 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c0309a0 con 0x7f625c06d520 2024-01-16T15:13:13.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.930+0000 7f6259600640 1 -- 172.21.15.43:0/369398034 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c030d30 con 0x7f625c06d520 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 -- 172.21.15.43:0/369398034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c06d520 msgr2=0x7f625c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c06d520 0x7f625c06d900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b980 comp rx=0 tx=0).stop 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 -- 172.21.15.43:0/369398034 shutdown_connections 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c0ad720 0x7f625c15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 --2- 172.21.15.43:0/369398034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c06d520 0x7f625c06d900 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 -- 172.21.15.43:0/369398034 >> 172.21.15.43:0/369398034 conn(0x7f625c0b0d80 msgr2=0x7f625c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 -- 172.21.15.43:0/369398034 shutdown_connections 2024-01-16T15:13:13.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.931+0000 7f6261a00640 1 -- 172.21.15.43:0/369398034 wait complete. 2024-01-16T15:13:13.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.933+0000 7f6261a00640 1 Processor -- start 2024-01-16T15:13:13.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.933+0000 7f6261a00640 1 -- start start 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 0x7f625c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0ad720 0x7f625c0cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0cd900 con 0x7f625c0abff0 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0cccf0 con 0x7f625c0ad720 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f6261a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0ccfd0 con 0x7f625c06d520 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f625a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f625a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42454/0 (socket says 172.21.15.43:42454) 2024-01-16T15:13:13.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.934+0000 7f625a000640 1 -- 172.21.15.43:0/4123828588 learned_addr learned my addr 172.21.15.43:0/4123828588 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:13.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625b400640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0ad720 0x7f625c0cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625aa00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 0x7f625c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 -- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 msgr2=0x7f625c0cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 0x7f625c0cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 -- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0ad720 msgr2=0x7f625c0cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0ad720 0x7f625c0cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 -- 172.21.15.43:0/4123828588 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c0094d0 con 0x7f625c0abff0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625aa00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 0x7f625c0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f625a000640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f624400d5c0 tx=0x7f624400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62440125f0 con 0x7f625c0abff0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6244012750 con 0x7f625c0abff0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.935+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244012b00 con 0x7f625c0abff0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.936+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c0cd2b0 con 0x7f625c0abff0 2024-01-16T15:13:13.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.936+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c0c96c0 con 0x7f625c0abff0 2024-01-16T15:13:13.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.937+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c0cdab0 con 0x7f625c0abff0 2024-01-16T15:13:13.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.938+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6244012c60 con 0x7f625c0abff0 2024-01-16T15:13:13.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.939+0000 7f6252a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 0x7f623406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.939+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6244088a40 con 0x7f625c0abff0 2024-01-16T15:13:13.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.939+0000 7f625aa00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 0x7f623406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.940+0000 7f625aa00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 0x7f623406e3e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f624c036000 tx=0x7f624c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:13.944+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f625c0cdab0 con 0x7f625c0abff0 2024-01-16T15:13:14.063 INFO:tasks.ceph.mds.a.smithi043.stderr:2024-01-16T15:13:14.085+0000 7f0def000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:13:14.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.139+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c074770 con 0x7f625c0abff0 2024-01-16T15:13:14.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.142+0000 7f6252a00640 1 -- 172.21.15.43:0/4123828588 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16555 (secure 0 0 0) 0x7f6244055300 con 0x7f625c0abff0 2024-01-16T15:13:14.121 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:14.122 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":88,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:13:11.141867+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:07.528802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:08.622339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 msgr2=0x7f623406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 0x7f623406e3e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f624c036000 tx=0x7f624c03b040 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 msgr2=0x7f625c14c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f624400d5c0 tx=0x7f624400da80 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 shutdown_connections 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f623406bf20 0x7f623406e3e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0ad720 0x7f625c0cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f625c0abff0 0x7f625c14c290 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 --2- 172.21.15.43:0/4123828588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f625c06d520 0x7f625c0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 >> 172.21.15.43:0/4123828588 conn(0x7f625c0b0d80 msgr2=0x7f625c15ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 shutdown_connections 2024-01-16T15:13:14.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.145+0000 7f6261a00640 1 -- 172.21.15.43:0/4123828588 wait complete. 2024-01-16T15:13:14.142 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:13:14.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:13:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.384+0000 7fe64f800640 1 Processor -- start 2024-01-16T15:13:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.384+0000 7fe64f800640 1 -- start start 2024-01-16T15:13:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.386+0000 7fe64f800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 0x7fe64806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.386+0000 7fe64f800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe6480ad720 0x7fe64815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.386+0000 7fe64f800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64815c360 con 0x7fe6480abff0 2024-01-16T15:13:14.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.386+0000 7fe64f800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480b41a0 con 0x7fe6480ad720 2024-01-16T15:13:14.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.386+0000 7fe64f800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64806de40 con 0x7fe64806d520 2024-01-16T15:13:14.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe64ce00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 0x7fe64806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe6480ad720 0x7fe64815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe6480ad720 0x7fe64815ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:53496/0 (socket says 172.21.15.43:53496) 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 -- 172.21.15.43:0/1317493410 learned_addr learned my addr 172.21.15.43:0/1317493410 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 -- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 msgr2=0x7fe64806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 0x7fe64806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.387+0000 7fe63fe00640 1 -- 172.21.15.43:0/1317493410 >> v1:172.21.15.69:6789/0 conn(0x7fe6480abff0 legacy=0x7fe64815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.388+0000 7fe63fe00640 1 -- 172.21.15.43:0/1317493410 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64815ba30 con 0x7fe6480ad720 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.388+0000 7fe64ce00640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 0x7fe64806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.389+0000 7fe63fe00640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480ad720 0x7fe64815ecd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe630009870 tx=0x7fe63002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=373cabc70371387b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.389+0000 7fe647e00640 1 -- 172.21.15.43:0/1317493410 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630030840 con 0x7fe6480ad720 2024-01-16T15:13:14.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.389+0000 7fe647e00640 1 -- 172.21.15.43:0/1317493410 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6300309a0 con 0x7fe6480ad720 2024-01-16T15:13:14.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.389+0000 7fe647e00640 1 -- 172.21.15.43:0/1317493410 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630030d30 con 0x7fe6480ad720 2024-01-16T15:13:14.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 -- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480ad720 msgr2=0x7fe64815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480ad720 0x7fe64815ecd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe630009870 tx=0x7fe63002b980 comp rx=0 tx=0).stop 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 -- 172.21.15.43:0/1317493410 shutdown_connections 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480ad720 0x7fe64815ecd0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 --2- 172.21.15.43:0/1317493410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64806d520 0x7fe64806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 -- 172.21.15.43:0/1317493410 >> 172.21.15.43:0/1317493410 conn(0x7fe6480b0d80 msgr2=0x7fe6480b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 -- 172.21.15.43:0/1317493410 shutdown_connections 2024-01-16T15:13:14.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.390+0000 7fe64f800640 1 -- 172.21.15.43:0/1317493410 wait complete. 2024-01-16T15:13:14.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.392+0000 7fe64f800640 1 Processor -- start 2024-01-16T15:13:14.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.393+0000 7fe64f800640 1 -- start start 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 0x7fe6480c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 0x7fe648071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480c84a0 con 0x7fe64806d520 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480c8780 con 0x7fe6480abff0 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64f800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480c8a60 con 0x7fe6480ad720 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42456/0 (socket says 172.21.15.43:42456) 2024-01-16T15:13:14.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.394+0000 7fe64ce00640 1 -- 172.21.15.43:0/2236974388 learned_addr learned my addr 172.21.15.43:0/2236974388 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:14.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe63fe00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 0x7fe6480c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64d800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 0x7fe648071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 -- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 msgr2=0x7fe648071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 0x7fe648071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 -- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 msgr2=0x7fe6480c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 0x7fe6480c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 -- 172.21.15.43:0/2236974388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe638009400 con 0x7fe64806d520 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe63fe00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 0x7fe6480c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64d800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 0x7fe648071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64ce00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fe63800ebe0 tx=0x7fe63800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638009760 con 0x7fe64806d520 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63800cc30 con 0x7fe64806d520 2024-01-16T15:13:14.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63800eee0 con 0x7fe64806d520 2024-01-16T15:13:14.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300094d0 con 0x7fe64806d520 2024-01-16T15:13:14.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.395+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6480c9020 con 0x7fe64806d520 2024-01-16T15:13:14.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.397+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648074770 con 0x7fe64806d520 2024-01-16T15:13:14.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.398+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe638013470 con 0x7fe64806d520 2024-01-16T15:13:14.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.399+0000 7fe645600640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 0x7fe62006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.399+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fe638088af0 con 0x7fe64806d520 2024-01-16T15:13:14.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.399+0000 7fe63fe00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 0x7fe62006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.400+0000 7fe63fe00640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 0x7fe62006e3e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe630036000 tx=0x7fe63003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.403+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe638055190 con 0x7fe64806d520 2024-01-16T15:13:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.594+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6480c88f0 con 0x7fe62006bf20 2024-01-16T15:13:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.623+0000 7fe645600640 1 -- 172.21.15.43:0/2236974388 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286840 (secure 0 0 0) 0x7fe6480c88f0 con 0x7fe62006bf20 2024-01-16T15:13:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:14.608 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 msgr2=0x7fe62006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 0x7fe62006e3e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe630036000 tx=0x7fe63003b040 comp rx=0 tx=0).stop 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 msgr2=0x7fe648149460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fe63800ebe0 tx=0x7fe63800c520 comp rx=0 tx=0).stop 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 shutdown_connections 2024-01-16T15:13:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe62006bf20 0x7fe62006e3e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480abff0 0x7fe6480c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480ad720 0x7fe648071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 --2- 172.21.15.43:0/2236974388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe64806d520 0x7fe648149460 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 >> 172.21.15.43:0/2236974388 conn(0x7fe6480b0d80 msgr2=0x7fe6480af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 shutdown_connections 2024-01-16T15:13:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.630+0000 7fe64f800640 1 -- 172.21.15.43:0/2236974388 wait complete. 2024-01-16T15:13:14.630 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-01-16T15:13:14.186724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":572124,"kb_used_data":9440,"kb_used_omap":12,"kb_used_meta":562547,"kb_avail":754402596,"statfs":{"total":773094113280,"available":772508258304,"internally_reserved":0,"allocated":9666560,"data_stored":4793811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":576048684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.061901"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156147+0000","last_change":"2024-01-16T15:13:10.151745+0000","last_active":"2024-01-16T15:13:10.156147+0000","last_peered":"2024-01-16T15:13:10.156147+0000","last_clean":"2024-01-16T15:13:10.156147+0000","last_became_active":"2024-01-16T15:13:10.151455+0000","last_became_peered":"2024-01-16T15:13:10.151455+0000","last_unstale":"2024-01-16T15:13:10.156147+0000","last_undegraded":"2024-01-16T15:13:10.156147+0000","last_fullsized":"2024-01-16T15:13:10.156147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:04.893705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167414+0000","last_change":"2024-01-16T15:13:10.163440+0000","last_active":"2024-01-16T15:13:10.167414+0000","last_peered":"2024-01-16T15:13:10.167414+0000","last_clean":"2024-01-16T15:13:10.167414+0000","last_became_active":"2024-01-16T15:13:10.163147+0000","last_became_peered":"2024-01-16T15:13:10.163147+0000","last_unstale":"2024-01-16T15:13:10.167414+0000","last_undegraded":"2024-01-16T15:13:10.167414+0000","last_fullsized":"2024-01-16T15:13:10.167414+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:59.442824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158876+0000","last_change":"2024-01-16T15:13:10.158057+0000","last_active":"2024-01-16T15:13:10.158876+0000","last_peered":"2024-01-16T15:13:10.158876+0000","last_clean":"2024-01-16T15:13:10.158876+0000","last_became_active":"2024-01-16T15:13:10.157736+0000","last_became_peered":"2024-01-16T15:13:10.157736+0000","last_unstale":"2024-01-16T15:13:10.158876+0000","last_undegraded":"2024-01-16T15:13:10.158876+0000","last_fullsized":"2024-01-16T15:13:10.158876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:59.878158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180419+0000","last_change":"2024-01-16T15:13:10.177628+0000","last_active":"2024-01-16T15:13:10.180419+0000","last_peered":"2024-01-16T15:13:10.180419+0000","last_clean":"2024-01-16T15:13:10.180419+0000","last_became_active":"2024-01-16T15:13:10.177334+0000","last_became_peered":"2024-01-16T15:13:10.177334+0000","last_unstale":"2024-01-16T15:13:10.180419+0000","last_undegraded":"2024-01-16T15:13:10.180419+0000","last_fullsized":"2024-01-16T15:13:10.180419+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:55.522718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175993+0000","last_change":"2024-01-16T15:13:10.174080+0000","last_active":"2024-01-16T15:13:10.175993+0000","last_peered":"2024-01-16T15:13:10.175993+0000","last_clean":"2024-01-16T15:13:10.175993+0000","last_became_active":"2024-01-16T15:13:10.173718+0000","last_became_peered":"2024-01-16T15:13:10.173718+0000","last_unstale":"2024-01-16T15:13:10.175993+0000","last_undegraded":"2024-01-16T15:13:10.175993+0000","last_fullsized":"2024-01-16T15:13:10.175993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.661926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.161953+0000","last_change":"2024-01-16T15:13:10.157470+0000","last_active":"2024-01-16T15:13:10.161953+0000","last_peered":"2024-01-16T15:13:10.161953+0000","last_clean":"2024-01-16T15:13:10.161953+0000","last_became_active":"2024-01-16T15:13:10.157023+0000","last_became_peered":"2024-01-16T15:13:10.157023+0000","last_unstale":"2024-01-16T15:13:10.161953+0000","last_undegraded":"2024-01-16T15:13:10.161953+0000","last_fullsized":"2024-01-16T15:13:10.161953+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:07.126646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.157369+0000","last_change":"2024-01-16T15:13:10.151185+0000","last_active":"2024-01-16T15:13:10.157369+0000","last_peered":"2024-01-16T15:13:10.157369+0000","last_clean":"2024-01-16T15:13:10.157369+0000","last_became_active":"2024-01-16T15:13:10.150867+0000","last_became_peered":"2024-01-16T15:13:10.150867+0000","last_unstale":"2024-01-16T15:13:10.157369+0000","last_undegraded":"2024-01-16T15:13:10.157369+0000","last_fullsized":"2024-01-16T15:13:10.157369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:50:02.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171606+0000","last_change":"2024-01-16T15:13:10.170420+0000","last_active":"2024-01-16T15:13:10.171606+0000","last_peered":"2024-01-16T15:13:10.171606+0000","last_clean":"2024-01-16T15:13:10.171606+0000","last_became_active":"2024-01-16T15:13:10.169529+0000","last_became_peered":"2024-01-16T15:13:10.169529+0000","last_unstale":"2024-01-16T15:13:10.171606+0000","last_undegraded":"2024-01-16T15:13:10.171606+0000","last_fullsized":"2024-01-16T15:13:10.171606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.752436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162052+0000","last_change":"2024-01-16T15:13:10.160732+0000","last_active":"2024-01-16T15:13:10.162052+0000","last_peered":"2024-01-16T15:13:10.162052+0000","last_clean":"2024-01-16T15:13:10.162052+0000","last_became_active":"2024-01-16T15:13:10.160402+0000","last_became_peered":"2024-01-16T15:13:10.160402+0000","last_unstale":"2024-01-16T15:13:10.162052+0000","last_undegraded":"2024-01-16T15:13:10.162052+0000","last_fullsized":"2024-01-16T15:13:10.162052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:58.566297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167640+0000","last_change":"2024-01-16T15:13:10.165155+0000","last_active":"2024-01-16T15:13:10.167640+0000","last_peered":"2024-01-16T15:13:10.167640+0000","last_clean":"2024-01-16T15:13:10.167640+0000","last_became_active":"2024-01-16T15:13:10.164870+0000","last_became_peered":"2024-01-16T15:13:10.164870+0000","last_unstale":"2024-01-16T15:13:10.167640+0000","last_undegraded":"2024-01-16T15:13:10.167640+0000","last_fullsized":"2024-01-16T15:13:10.167640+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:58.566297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176317+0000","last_change":"2024-01-16T15:13:10.169151+0000","last_active":"2024-01-16T15:13:10.176317+0000","last_peered":"2024-01-16T15:13:10.176317+0000","last_clean":"2024-01-16T15:13:10.176317+0000","last_became_active":"2024-01-16T15:13:10.163706+0000","last_became_peered":"2024-01-16T15:13:10.163706+0000","last_unstale":"2024-01-16T15:13:10.176317+0000","last_undegraded":"2024-01-16T15:13:10.176317+0000","last_fullsized":"2024-01-16T15:13:10.176317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:19.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164989+0000","last_change":"2024-01-16T15:13:10.162953+0000","last_active":"2024-01-16T15:13:10.164989+0000","last_peered":"2024-01-16T15:13:10.164989+0000","last_clean":"2024-01-16T15:13:10.164989+0000","last_became_active":"2024-01-16T15:13:10.162668+0000","last_became_peered":"2024-01-16T15:13:10.162668+0000","last_unstale":"2024-01-16T15:13:10.164989+0000","last_undegraded":"2024-01-16T15:13:10.164989+0000","last_fullsized":"2024-01-16T15:13:10.164989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:28.363739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154417+0000","last_change":"2024-01-16T15:13:10.151668+0000","last_active":"2024-01-16T15:13:10.154417+0000","last_peered":"2024-01-16T15:13:10.154417+0000","last_clean":"2024-01-16T15:13:10.154417+0000","last_became_active":"2024-01-16T15:13:10.151338+0000","last_became_peered":"2024-01-16T15:13:10.151338+0000","last_unstale":"2024-01-16T15:13:10.154417+0000","last_undegraded":"2024-01-16T15:13:10.154417+0000","last_fullsized":"2024-01-16T15:13:10.154417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:47.062688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.181138+0000","last_change":"2024-01-16T15:13:10.179253+0000","last_active":"2024-01-16T15:13:10.181138+0000","last_peered":"2024-01-16T15:13:10.181138+0000","last_clean":"2024-01-16T15:13:10.181138+0000","last_became_active":"2024-01-16T15:13:10.178510+0000","last_became_peered":"2024-01-16T15:13:10.178510+0000","last_unstale":"2024-01-16T15:13:10.181138+0000","last_undegraded":"2024-01-16T15:13:10.181138+0000","last_fullsized":"2024-01-16T15:13:10.181138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:07.967072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168368+0000","last_change":"2024-01-16T15:13:10.165586+0000","last_active":"2024-01-16T15:13:10.168368+0000","last_peered":"2024-01-16T15:13:10.168368+0000","last_clean":"2024-01-16T15:13:10.168368+0000","last_became_active":"2024-01-16T15:13:10.165264+0000","last_became_peered":"2024-01-16T15:13:10.165264+0000","last_unstale":"2024-01-16T15:13:10.168368+0000","last_undegraded":"2024-01-16T15:13:10.168368+0000","last_fullsized":"2024-01-16T15:13:10.168368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:54.237403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150762+0000","last_change":"2024-01-16T15:13:10.148246+0000","last_active":"2024-01-16T15:13:10.150762+0000","last_peered":"2024-01-16T15:13:10.150762+0000","last_clean":"2024-01-16T15:13:10.150762+0000","last_became_active":"2024-01-16T15:13:10.147957+0000","last_became_peered":"2024-01-16T15:13:10.147957+0000","last_unstale":"2024-01-16T15:13:10.150762+0000","last_undegraded":"2024-01-16T15:13:10.150762+0000","last_fullsized":"2024-01-16T15:13:10.150762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:42.927055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150686+0000","last_change":"2024-01-16T15:13:10.145984+0000","last_active":"2024-01-16T15:13:10.150686+0000","last_peered":"2024-01-16T15:13:10.150686+0000","last_clean":"2024-01-16T15:13:10.150686+0000","last_became_active":"2024-01-16T15:13:10.145569+0000","last_became_peered":"2024-01-16T15:13:10.145569+0000","last_unstale":"2024-01-16T15:13:10.150686+0000","last_undegraded":"2024-01-16T15:13:10.150686+0000","last_fullsized":"2024-01-16T15:13:10.150686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:27.675509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154258+0000","last_change":"2024-01-16T15:13:10.147373+0000","last_active":"2024-01-16T15:13:10.154258+0000","last_peered":"2024-01-16T15:13:10.154258+0000","last_clean":"2024-01-16T15:13:10.154258+0000","last_became_active":"2024-01-16T15:13:10.145161+0000","last_became_peered":"2024-01-16T15:13:10.145161+0000","last_unstale":"2024-01-16T15:13:10.154258+0000","last_undegraded":"2024-01-16T15:13:10.154258+0000","last_fullsized":"2024-01-16T15:13:10.154258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:44:52.704937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176567+0000","last_change":"2024-01-16T15:13:10.165214+0000","last_active":"2024-01-16T15:13:10.176567+0000","last_peered":"2024-01-16T15:13:10.176567+0000","last_clean":"2024-01-16T15:13:10.176567+0000","last_became_active":"2024-01-16T15:13:10.164899+0000","last_became_peered":"2024-01-16T15:13:10.164899+0000","last_unstale":"2024-01-16T15:13:10.176567+0000","last_undegraded":"2024-01-16T15:13:10.176567+0000","last_fullsized":"2024-01-16T15:13:10.176567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:09.698637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.155005+0000","last_change":"2024-01-16T15:13:10.149948+0000","last_active":"2024-01-16T15:13:10.155005+0000","last_peered":"2024-01-16T15:13:10.155005+0000","last_clean":"2024-01-16T15:13:10.155005+0000","last_became_active":"2024-01-16T15:13:10.149671+0000","last_became_peered":"2024-01-16T15:13:10.149671+0000","last_unstale":"2024-01-16T15:13:10.155005+0000","last_undegraded":"2024-01-16T15:13:10.155005+0000","last_fullsized":"2024-01-16T15:13:10.155005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:00.540556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154311+0000","last_change":"2024-01-16T15:13:10.146925+0000","last_active":"2024-01-16T15:13:10.154311+0000","last_peered":"2024-01-16T15:13:10.154311+0000","last_clean":"2024-01-16T15:13:10.154311+0000","last_became_active":"2024-01-16T15:13:10.146575+0000","last_became_peered":"2024-01-16T15:13:10.146575+0000","last_unstale":"2024-01-16T15:13:10.154311+0000","last_undegraded":"2024-01-16T15:13:10.154311+0000","last_fullsized":"2024-01-16T15:13:10.154311+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:34.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176988+0000","last_change":"2024-01-16T15:13:10.170105+0000","last_active":"2024-01-16T15:13:10.176988+0000","last_peered":"2024-01-16T15:13:10.176988+0000","last_clean":"2024-01-16T15:13:10.176988+0000","last_became_active":"2024-01-16T15:13:10.169574+0000","last_became_peered":"2024-01-16T15:13:10.169574+0000","last_unstale":"2024-01-16T15:13:10.176988+0000","last_undegraded":"2024-01-16T15:13:10.176988+0000","last_fullsized":"2024-01-16T15:13:10.176988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:51.431529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158505+0000","last_change":"2024-01-16T15:13:10.155229+0000","last_active":"2024-01-16T15:13:10.158505+0000","last_peered":"2024-01-16T15:13:10.158505+0000","last_clean":"2024-01-16T15:13:10.158505+0000","last_became_active":"2024-01-16T15:13:10.154347+0000","last_became_peered":"2024-01-16T15:13:10.154347+0000","last_unstale":"2024-01-16T15:13:10.158505+0000","last_undegraded":"2024-01-16T15:13:10.158505+0000","last_fullsized":"2024-01-16T15:13:10.158505+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:55.691177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162766+0000","last_change":"2024-01-16T15:13:10.153721+0000","last_active":"2024-01-16T15:13:10.162766+0000","last_peered":"2024-01-16T15:13:10.162766+0000","last_clean":"2024-01-16T15:13:10.162766+0000","last_became_active":"2024-01-16T15:13:10.153430+0000","last_became_peered":"2024-01-16T15:13:10.153430+0000","last_unstale":"2024-01-16T15:13:10.162766+0000","last_undegraded":"2024-01-16T15:13:10.162766+0000","last_fullsized":"2024-01-16T15:13:10.162766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:18.037258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.173712+0000","last_change":"2024-01-16T15:13:10.166616+0000","last_active":"2024-01-16T15:13:10.173712+0000","last_peered":"2024-01-16T15:13:10.173712+0000","last_clean":"2024-01-16T15:13:10.173712+0000","last_became_active":"2024-01-16T15:13:10.166340+0000","last_became_peered":"2024-01-16T15:13:10.166340+0000","last_unstale":"2024-01-16T15:13:10.173712+0000","last_undegraded":"2024-01-16T15:13:10.173712+0000","last_fullsized":"2024-01-16T15:13:10.173712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:43.176639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170145+0000","last_change":"2024-01-16T15:13:10.167640+0000","last_active":"2024-01-16T15:13:10.170145+0000","last_peered":"2024-01-16T15:13:10.170145+0000","last_clean":"2024-01-16T15:13:10.170145+0000","last_became_active":"2024-01-16T15:13:10.166757+0000","last_became_peered":"2024-01-16T15:13:10.166757+0000","last_unstale":"2024-01-16T15:13:10.170145+0000","last_undegraded":"2024-01-16T15:13:10.170145+0000","last_fullsized":"2024-01-16T15:13:10.170145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:18.666895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164440+0000","last_change":"2024-01-16T15:13:10.163097+0000","last_active":"2024-01-16T15:13:10.164440+0000","last_peered":"2024-01-16T15:13:10.164440+0000","last_clean":"2024-01-16T15:13:10.164440+0000","last_became_active":"2024-01-16T15:13:10.162735+0000","last_became_peered":"2024-01-16T15:13:10.162735+0000","last_unstale":"2024-01-16T15:13:10.164440+0000","last_undegraded":"2024-01-16T15:13:10.164440+0000","last_fullsized":"2024-01-16T15:13:10.164440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:53.396086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156483+0000","last_change":"2024-01-16T15:13:10.153644+0000","last_active":"2024-01-16T15:13:10.156483+0000","last_peered":"2024-01-16T15:13:10.156483+0000","last_clean":"2024-01-16T15:13:10.156483+0000","last_became_active":"2024-01-16T15:13:10.153286+0000","last_became_peered":"2024-01-16T15:13:10.153286+0000","last_unstale":"2024-01-16T15:13:10.156483+0000","last_undegraded":"2024-01-16T15:13:10.156483+0000","last_fullsized":"2024-01-16T15:13:10.156483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:33.303213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.179561+0000","last_change":"2024-01-16T15:13:10.175640+0000","last_active":"2024-01-16T15:13:10.179561+0000","last_peered":"2024-01-16T15:13:10.179561+0000","last_clean":"2024-01-16T15:13:10.179561+0000","last_became_active":"2024-01-16T15:13:10.174512+0000","last_became_peered":"2024-01-16T15:13:10.174512+0000","last_unstale":"2024-01-16T15:13:10.179561+0000","last_undegraded":"2024-01-16T15:13:10.179561+0000","last_fullsized":"2024-01-16T15:13:10.179561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:56.824444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.178450+0000","last_change":"2024-01-16T15:13:10.174242+0000","last_active":"2024-01-16T15:13:10.178450+0000","last_peered":"2024-01-16T15:13:10.178450+0000","last_clean":"2024-01-16T15:13:10.178450+0000","last_became_active":"2024-01-16T15:13:10.169627+0000","last_became_peered":"2024-01-16T15:13:10.169627+0000","last_unstale":"2024-01-16T15:13:10.178450+0000","last_undegraded":"2024-01-16T15:13:10.178450+0000","last_fullsized":"2024-01-16T15:13:10.178450+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:51.074211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170866+0000","last_change":"2024-01-16T15:13:10.166842+0000","last_active":"2024-01-16T15:13:10.170866+0000","last_peered":"2024-01-16T15:13:10.170866+0000","last_clean":"2024-01-16T15:13:10.170866+0000","last_became_active":"2024-01-16T15:13:10.166048+0000","last_became_peered":"2024-01-16T15:13:10.166048+0000","last_unstale":"2024-01-16T15:13:10.170866+0000","last_undegraded":"2024-01-16T15:13:10.170866+0000","last_fullsized":"2024-01-16T15:13:10.170866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.752436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176104+0000","last_change":"2024-01-16T15:13:10.175048+0000","last_active":"2024-01-16T15:13:10.176104+0000","last_peered":"2024-01-16T15:13:10.176104+0000","last_clean":"2024-01-16T15:13:10.176104+0000","last_became_active":"2024-01-16T15:13:10.174410+0000","last_became_peered":"2024-01-16T15:13:10.174410+0000","last_unstale":"2024-01-16T15:13:10.176104+0000","last_undegraded":"2024-01-16T15:13:10.176104+0000","last_fullsized":"2024-01-16T15:13:10.176104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:33.445285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156020+0000","last_change":"2024-01-16T15:13:10.154000+0000","last_active":"2024-01-16T15:13:10.156020+0000","last_peered":"2024-01-16T15:13:10.156020+0000","last_clean":"2024-01-16T15:13:10.156020+0000","last_became_active":"2024-01-16T15:13:10.153439+0000","last_became_peered":"2024-01-16T15:13:10.153439+0000","last_unstale":"2024-01-16T15:13:10.156020+0000","last_undegraded":"2024-01-16T15:13:10.156020+0000","last_fullsized":"2024-01-16T15:13:10.156020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:07.126646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"87'4","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.174915+0000","last_change":"2024-01-16T15:13:09.151686+0000","last_active":"2024-01-16T15:13:10.174915+0000","last_peered":"2024-01-16T15:13:10.174915+0000","last_clean":"2024-01-16T15:13:10.174915+0000","last_became_active":"2024-01-16T15:13:09.150673+0000","last_became_peered":"2024-01-16T15:13:09.150673+0000","last_unstale":"2024-01-16T15:13:10.174915+0000","last_undegraded":"2024-01-16T15:13:10.174915+0000","last_fullsized":"2024-01-16T15:13:10.174915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:08.190150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164670+0000","last_change":"2024-01-16T15:13:10.160577+0000","last_active":"2024-01-16T15:13:10.164670+0000","last_peered":"2024-01-16T15:13:10.164670+0000","last_clean":"2024-01-16T15:13:10.164670+0000","last_became_active":"2024-01-16T15:13:10.160238+0000","last_became_peered":"2024-01-16T15:13:10.160238+0000","last_unstale":"2024-01-16T15:13:10.164670+0000","last_undegraded":"2024-01-16T15:13:10.164670+0000","last_fullsized":"2024-01-16T15:13:10.164670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:53.320778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.172822+0000","last_change":"2024-01-16T15:13:09.151756+0000","last_active":"2024-01-16T15:13:10.172822+0000","last_peered":"2024-01-16T15:13:10.172822+0000","last_clean":"2024-01-16T15:13:10.172822+0000","last_became_active":"2024-01-16T15:13:09.150392+0000","last_became_peered":"2024-01-16T15:13:09.150392+0000","last_unstale":"2024-01-16T15:13:10.172822+0000","last_undegraded":"2024-01-16T15:13:10.172822+0000","last_fullsized":"2024-01-16T15:13:10.172822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:32.435368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175823+0000","last_change":"2024-01-16T15:13:10.174551+0000","last_active":"2024-01-16T15:13:10.175823+0000","last_peered":"2024-01-16T15:13:10.175823+0000","last_clean":"2024-01-16T15:13:10.175823+0000","last_became_active":"2024-01-16T15:13:10.174229+0000","last_became_peered":"2024-01-16T15:13:10.174229+0000","last_unstale":"2024-01-16T15:13:10.175823+0000","last_undegraded":"2024-01-16T15:13:10.175823+0000","last_fullsized":"2024-01-16T15:13:10.175823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:17.170146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182663+0000","last_change":"2024-01-16T15:13:09.147173+0000","last_active":"2024-01-16T15:13:10.182663+0000","last_peered":"2024-01-16T15:13:10.182663+0000","last_clean":"2024-01-16T15:13:10.182663+0000","last_became_active":"2024-01-16T15:13:09.146895+0000","last_became_peered":"2024-01-16T15:13:09.146895+0000","last_unstale":"2024-01-16T15:13:10.182663+0000","last_undegraded":"2024-01-16T15:13:10.182663+0000","last_fullsized":"2024-01-16T15:13:10.182663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:28.989720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158249+0000","last_change":"2024-01-16T15:13:10.157632+0000","last_active":"2024-01-16T15:13:10.158249+0000","last_peered":"2024-01-16T15:13:10.158249+0000","last_clean":"2024-01-16T15:13:10.158249+0000","last_became_active":"2024-01-16T15:13:10.157269+0000","last_became_peered":"2024-01-16T15:13:10.157269+0000","last_unstale":"2024-01-16T15:13:10.158249+0000","last_undegraded":"2024-01-16T15:13:10.158249+0000","last_fullsized":"2024-01-16T15:13:10.158249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:55.704133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182277+0000","last_change":"2024-01-16T15:13:09.140816+0000","last_active":"2024-01-16T15:13:10.182277+0000","last_peered":"2024-01-16T15:13:10.182277+0000","last_clean":"2024-01-16T15:13:10.182277+0000","last_became_active":"2024-01-16T15:13:09.140552+0000","last_became_peered":"2024-01-16T15:13:09.140552+0000","last_unstale":"2024-01-16T15:13:10.182277+0000","last_undegraded":"2024-01-16T15:13:10.182277+0000","last_fullsized":"2024-01-16T15:13:10.182277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:40.673100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.169816+0000","last_change":"2024-01-16T15:13:10.167193+0000","last_active":"2024-01-16T15:13:10.169816+0000","last_peered":"2024-01-16T15:13:10.169816+0000","last_clean":"2024-01-16T15:13:10.169816+0000","last_became_active":"2024-01-16T15:13:10.163090+0000","last_became_peered":"2024-01-16T15:13:10.163090+0000","last_unstale":"2024-01-16T15:13:10.169816+0000","last_undegraded":"2024-01-16T15:13:10.169816+0000","last_fullsized":"2024-01-16T15:13:10.169816+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:43.267405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180200+0000","last_change":"2024-01-16T15:13:09.148297+0000","last_active":"2024-01-16T15:13:10.180200+0000","last_peered":"2024-01-16T15:13:10.180200+0000","last_clean":"2024-01-16T15:13:10.180200+0000","last_became_active":"2024-01-16T15:13:09.147302+0000","last_became_peered":"2024-01-16T15:13:09.147302+0000","last_unstale":"2024-01-16T15:13:10.180200+0000","last_undegraded":"2024-01-16T15:13:10.180200+0000","last_fullsized":"2024-01-16T15:13:10.180200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:57.556380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.161949+0000","last_change":"2024-01-16T15:13:10.155895+0000","last_active":"2024-01-16T15:13:10.161949+0000","last_peered":"2024-01-16T15:13:10.161949+0000","last_clean":"2024-01-16T15:13:10.161949+0000","last_became_active":"2024-01-16T15:13:10.155529+0000","last_became_peered":"2024-01-16T15:13:10.155529+0000","last_unstale":"2024-01-16T15:13:10.161949+0000","last_undegraded":"2024-01-16T15:13:10.161949+0000","last_fullsized":"2024-01-16T15:13:10.161949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:19.057854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.172812+0000","last_change":"2024-01-16T15:13:09.150066+0000","last_active":"2024-01-16T15:13:10.172812+0000","last_peered":"2024-01-16T15:13:10.172812+0000","last_clean":"2024-01-16T15:13:10.172812+0000","last_became_active":"2024-01-16T15:13:09.149367+0000","last_became_peered":"2024-01-16T15:13:09.149367+0000","last_unstale":"2024-01-16T15:13:10.172812+0000","last_undegraded":"2024-01-16T15:13:10.172812+0000","last_fullsized":"2024-01-16T15:13:10.172812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:57.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170986+0000","last_change":"2024-01-16T15:13:10.168840+0000","last_active":"2024-01-16T15:13:10.170986+0000","last_peered":"2024-01-16T15:13:10.170986+0000","last_clean":"2024-01-16T15:13:10.170986+0000","last_became_active":"2024-01-16T15:13:10.168554+0000","last_became_peered":"2024-01-16T15:13:10.168554+0000","last_unstale":"2024-01-16T15:13:10.170986+0000","last_undegraded":"2024-01-16T15:13:10.170986+0000","last_fullsized":"2024-01-16T15:13:10.170986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:39.593178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150518+0000","last_change":"2024-01-16T15:13:09.145669+0000","last_active":"2024-01-16T15:13:10.150518+0000","last_peered":"2024-01-16T15:13:10.150518+0000","last_clean":"2024-01-16T15:13:10.150518+0000","last_became_active":"2024-01-16T15:13:09.145356+0000","last_became_peered":"2024-01-16T15:13:09.145356+0000","last_unstale":"2024-01-16T15:13:10.150518+0000","last_undegraded":"2024-01-16T15:13:10.150518+0000","last_fullsized":"2024-01-16T15:13:10.150518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:08.190150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168350+0000","last_change":"2024-01-16T15:13:10.165520+0000","last_active":"2024-01-16T15:13:10.168350+0000","last_peered":"2024-01-16T15:13:10.168350+0000","last_clean":"2024-01-16T15:13:10.168350+0000","last_became_active":"2024-01-16T15:13:10.165242+0000","last_became_peered":"2024-01-16T15:13:10.165242+0000","last_unstale":"2024-01-16T15:13:10.168350+0000","last_undegraded":"2024-01-16T15:13:10.168350+0000","last_fullsized":"2024-01-16T15:13:10.168350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:55.704133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.160290+0000","last_change":"2024-01-16T15:13:09.146887+0000","last_active":"2024-01-16T15:13:10.160290+0000","last_peered":"2024-01-16T15:13:10.160290+0000","last_clean":"2024-01-16T15:13:10.160290+0000","last_became_active":"2024-01-16T15:13:09.146588+0000","last_became_peered":"2024-01-16T15:13:09.146588+0000","last_unstale":"2024-01-16T15:13:10.160290+0000","last_undegraded":"2024-01-16T15:13:10.160290+0000","last_fullsized":"2024-01-16T15:13:10.160290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:44.486393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.179224+0000","last_change":"2024-01-16T15:13:10.174999+0000","last_active":"2024-01-16T15:13:10.179224+0000","last_peered":"2024-01-16T15:13:10.179224+0000","last_clean":"2024-01-16T15:13:10.179224+0000","last_became_active":"2024-01-16T15:13:10.174699+0000","last_became_peered":"2024-01-16T15:13:10.174699+0000","last_unstale":"2024-01-16T15:13:10.179224+0000","last_undegraded":"2024-01-16T15:13:10.179224+0000","last_fullsized":"2024-01-16T15:13:10.179224+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:02.049659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.140960+0000","last_change":"2024-01-16T15:13:09.152842+0000","last_active":"2024-01-16T15:13:10.140960+0000","last_peered":"2024-01-16T15:13:10.140960+0000","last_clean":"2024-01-16T15:13:10.140960+0000","last_became_active":"2024-01-16T15:13:09.152384+0000","last_became_peered":"2024-01-16T15:13:09.152384+0000","last_unstale":"2024-01-16T15:13:10.140960+0000","last_undegraded":"2024-01-16T15:13:10.140960+0000","last_fullsized":"2024-01-16T15:13:10.140960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:08.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158438+0000","last_change":"2024-01-16T15:13:10.155407+0000","last_active":"2024-01-16T15:13:10.158438+0000","last_peered":"2024-01-16T15:13:10.158438+0000","last_clean":"2024-01-16T15:13:10.158438+0000","last_became_active":"2024-01-16T15:13:10.155124+0000","last_became_peered":"2024-01-16T15:13:10.155124+0000","last_unstale":"2024-01-16T15:13:10.158438+0000","last_undegraded":"2024-01-16T15:13:10.158438+0000","last_fullsized":"2024-01-16T15:13:10.158438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:33.445285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.136568+0000","last_change":"2024-01-16T15:13:09.151431+0000","last_active":"2024-01-16T15:13:10.136568+0000","last_peered":"2024-01-16T15:13:10.136568+0000","last_clean":"2024-01-16T15:13:10.136568+0000","last_became_active":"2024-01-16T15:13:09.151009+0000","last_became_peered":"2024-01-16T15:13:09.151009+0000","last_unstale":"2024-01-16T15:13:10.136568+0000","last_undegraded":"2024-01-16T15:13:10.136568+0000","last_fullsized":"2024-01-16T15:13:10.136568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:14.148344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162208+0000","last_change":"2024-01-16T15:13:10.157277+0000","last_active":"2024-01-16T15:13:10.162208+0000","last_peered":"2024-01-16T15:13:10.162208+0000","last_clean":"2024-01-16T15:13:10.162208+0000","last_became_active":"2024-01-16T15:13:10.156819+0000","last_became_peered":"2024-01-16T15:13:10.156819+0000","last_unstale":"2024-01-16T15:13:10.162208+0000","last_undegraded":"2024-01-16T15:13:10.162208+0000","last_fullsized":"2024-01-16T15:13:10.162208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:53.320778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"87'3","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180544+0000","last_change":"2024-01-16T15:13:09.152324+0000","last_active":"2024-01-16T15:13:10.180544+0000","last_peered":"2024-01-16T15:13:10.180544+0000","last_clean":"2024-01-16T15:13:10.180544+0000","last_became_active":"2024-01-16T15:13:09.150771+0000","last_became_peered":"2024-01-16T15:13:09.150771+0000","last_unstale":"2024-01-16T15:13:10.180544+0000","last_undegraded":"2024-01-16T15:13:10.180544+0000","last_fullsized":"2024-01-16T15:13:10.180544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:16.244236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170997+0000","last_change":"2024-01-16T15:13:10.166424+0000","last_active":"2024-01-16T15:13:10.170997+0000","last_peered":"2024-01-16T15:13:10.170997+0000","last_clean":"2024-01-16T15:13:10.170997+0000","last_became_active":"2024-01-16T15:13:10.166103+0000","last_became_peered":"2024-01-16T15:13:10.166103+0000","last_unstale":"2024-01-16T15:13:10.170997+0000","last_undegraded":"2024-01-16T15:13:10.170997+0000","last_fullsized":"2024-01-16T15:13:10.170997+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:14.228009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"87'6","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.184317+0000","last_change":"2024-01-16T15:13:09.152546+0000","last_active":"2024-01-16T15:13:10.184317+0000","last_peered":"2024-01-16T15:13:10.184317+0000","last_clean":"2024-01-16T15:13:10.184317+0000","last_became_active":"2024-01-16T15:13:09.152268+0000","last_became_peered":"2024-01-16T15:13:09.152268+0000","last_unstale":"2024-01-16T15:13:10.184317+0000","last_undegraded":"2024-01-16T15:13:10.184317+0000","last_fullsized":"2024-01-16T15:13:10.184317+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:42.077697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162634+0000","last_change":"2024-01-16T15:13:10.159408+0000","last_active":"2024-01-16T15:13:10.162634+0000","last_peered":"2024-01-16T15:13:10.162634+0000","last_clean":"2024-01-16T15:13:10.162634+0000","last_became_active":"2024-01-16T15:13:10.154989+0000","last_became_peered":"2024-01-16T15:13:10.154989+0000","last_unstale":"2024-01-16T15:13:10.162634+0000","last_undegraded":"2024-01-16T15:13:10.162634+0000","last_fullsized":"2024-01-16T15:13:10.162634+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:43:07.856847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.184583+0000","last_change":"2024-01-16T15:13:09.142988+0000","last_active":"2024-01-16T15:13:10.184583+0000","last_peered":"2024-01-16T15:13:10.184583+0000","last_clean":"2024-01-16T15:13:10.184583+0000","last_became_active":"2024-01-16T15:13:09.139882+0000","last_became_peered":"2024-01-16T15:13:09.139882+0000","last_unstale":"2024-01-16T15:13:10.184583+0000","last_undegraded":"2024-01-16T15:13:10.184583+0000","last_fullsized":"2024-01-16T15:13:10.184583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:33.048007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171037+0000","last_change":"2024-01-16T15:13:10.169557+0000","last_active":"2024-01-16T15:13:10.171037+0000","last_peered":"2024-01-16T15:13:10.171037+0000","last_clean":"2024-01-16T15:13:10.171037+0000","last_became_active":"2024-01-16T15:13:10.169265+0000","last_became_peered":"2024-01-16T15:13:10.169265+0000","last_unstale":"2024-01-16T15:13:10.171037+0000","last_undegraded":"2024-01-16T15:13:10.171037+0000","last_fullsized":"2024-01-16T15:13:10.171037+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:11.855352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158436+0000","last_change":"2024-01-16T15:13:09.147378+0000","last_active":"2024-01-16T15:13:10.158436+0000","last_peered":"2024-01-16T15:13:10.158436+0000","last_clean":"2024-01-16T15:13:10.158436+0000","last_became_active":"2024-01-16T15:13:09.147101+0000","last_became_peered":"2024-01-16T15:13:09.147101+0000","last_unstale":"2024-01-16T15:13:10.158436+0000","last_undegraded":"2024-01-16T15:13:10.158436+0000","last_fullsized":"2024-01-16T15:13:10.158436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:19.238253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167897+0000","last_change":"2024-01-16T15:13:10.160889+0000","last_active":"2024-01-16T15:13:10.167897+0000","last_peered":"2024-01-16T15:13:10.167897+0000","last_clean":"2024-01-16T15:13:10.167897+0000","last_became_active":"2024-01-16T15:13:10.160613+0000","last_became_peered":"2024-01-16T15:13:10.160613+0000","last_unstale":"2024-01-16T15:13:10.167897+0000","last_undegraded":"2024-01-16T15:13:10.167897+0000","last_fullsized":"2024-01-16T15:13:10.167897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:27.704754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.144379+0000","last_change":"2024-01-16T15:13:09.147474+0000","last_active":"2024-01-16T15:13:10.144379+0000","last_peered":"2024-01-16T15:13:10.144379+0000","last_clean":"2024-01-16T15:13:10.144379+0000","last_became_active":"2024-01-16T15:13:09.147126+0000","last_became_peered":"2024-01-16T15:13:09.147126+0000","last_unstale":"2024-01-16T15:13:10.144379+0000","last_undegraded":"2024-01-16T15:13:10.144379+0000","last_fullsized":"2024-01-16T15:13:10.144379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:57.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175383+0000","last_change":"2024-01-16T15:13:10.165981+0000","last_active":"2024-01-16T15:13:10.175383+0000","last_peered":"2024-01-16T15:13:10.175383+0000","last_clean":"2024-01-16T15:13:10.175383+0000","last_became_active":"2024-01-16T15:13:10.165711+0000","last_became_peered":"2024-01-16T15:13:10.165711+0000","last_unstale":"2024-01-16T15:13:10.175383+0000","last_undegraded":"2024-01-16T15:13:10.175383+0000","last_fullsized":"2024-01-16T15:13:10.175383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:29.690714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180929+0000","last_change":"2024-01-16T15:13:09.142813+0000","last_active":"2024-01-16T15:13:10.180929+0000","last_peered":"2024-01-16T15:13:10.180929+0000","last_clean":"2024-01-16T15:13:10.180929+0000","last_became_active":"2024-01-16T15:13:09.142527+0000","last_became_peered":"2024-01-16T15:13:09.142527+0000","last_unstale":"2024-01-16T15:13:10.180929+0000","last_undegraded":"2024-01-16T15:13:10.180929+0000","last_fullsized":"2024-01-16T15:13:10.180929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:43:03.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156877+0000","last_change":"2024-01-16T15:13:10.154560+0000","last_active":"2024-01-16T15:13:10.156877+0000","last_peered":"2024-01-16T15:13:10.156877+0000","last_clean":"2024-01-16T15:13:10.156877+0000","last_became_active":"2024-01-16T15:13:10.154256+0000","last_became_peered":"2024-01-16T15:13:10.154256+0000","last_unstale":"2024-01-16T15:13:10.156877+0000","last_undegraded":"2024-01-16T15:13:10.156877+0000","last_fullsized":"2024-01-16T15:13:10.156877+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:29.484144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171893+0000","last_change":"2024-01-16T15:13:09.150981+0000","last_active":"2024-01-16T15:13:10.171893+0000","last_peered":"2024-01-16T15:13:10.171893+0000","last_clean":"2024-01-16T15:13:10.171893+0000","last_became_active":"2024-01-16T15:13:09.149150+0000","last_became_peered":"2024-01-16T15:13:09.149150+0000","last_unstale":"2024-01-16T15:13:10.171893+0000","last_undegraded":"2024-01-16T15:13:10.171893+0000","last_fullsized":"2024-01-16T15:13:10.171893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:43:06.846930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156792+0000","last_change":"2024-01-16T15:13:10.153848+0000","last_active":"2024-01-16T15:13:10.156792+0000","last_peered":"2024-01-16T15:13:10.156792+0000","last_clean":"2024-01-16T15:13:10.156792+0000","last_became_active":"2024-01-16T15:13:10.153019+0000","last_became_peered":"2024-01-16T15:13:10.153019+0000","last_unstale":"2024-01-16T15:13:10.156792+0000","last_undegraded":"2024-01-16T15:13:10.156792+0000","last_fullsized":"2024-01-16T15:13:10.156792+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:43.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.178490+0000","last_change":"2024-01-16T15:13:09.146055+0000","last_active":"2024-01-16T15:13:10.178490+0000","last_peered":"2024-01-16T15:13:10.178490+0000","last_clean":"2024-01-16T15:13:10.178490+0000","last_became_active":"2024-01-16T15:13:09.145773+0000","last_became_peered":"2024-01-16T15:13:09.145773+0000","last_unstale":"2024-01-16T15:13:10.178490+0000","last_undegraded":"2024-01-16T15:13:10.178490+0000","last_fullsized":"2024-01-16T15:13:10.178490+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:59.247724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175948+0000","last_change":"2024-01-16T15:13:10.174037+0000","last_active":"2024-01-16T15:13:10.175948+0000","last_peered":"2024-01-16T15:13:10.175948+0000","last_clean":"2024-01-16T15:13:10.175948+0000","last_became_active":"2024-01-16T15:13:10.173730+0000","last_became_peered":"2024-01-16T15:13:10.173730+0000","last_unstale":"2024-01-16T15:13:10.175948+0000","last_undegraded":"2024-01-16T15:13:10.175948+0000","last_fullsized":"2024-01-16T15:13:10.175948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:15.840801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.151254+0000","last_change":"2024-01-16T15:13:09.147750+0000","last_active":"2024-01-16T15:13:10.151254+0000","last_peered":"2024-01-16T15:13:10.151254+0000","last_clean":"2024-01-16T15:13:10.151254+0000","last_became_active":"2024-01-16T15:13:09.147460+0000","last_became_peered":"2024-01-16T15:13:09.147460+0000","last_unstale":"2024-01-16T15:13:10.151254+0000","last_undegraded":"2024-01-16T15:13:10.151254+0000","last_fullsized":"2024-01-16T15:13:10.151254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:10.689821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.181119+0000","last_change":"2024-01-16T15:13:10.179121+0000","last_active":"2024-01-16T15:13:10.181119+0000","last_peered":"2024-01-16T15:13:10.181119+0000","last_clean":"2024-01-16T15:13:10.181119+0000","last_became_active":"2024-01-16T15:13:10.177505+0000","last_became_peered":"2024-01-16T15:13:10.177505+0000","last_unstale":"2024-01-16T15:13:10.181119+0000","last_undegraded":"2024-01-16T15:13:10.181119+0000","last_fullsized":"2024-01-16T15:13:10.181119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:07.488113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180396+0000","last_change":"2024-01-16T15:13:09.146726+0000","last_active":"2024-01-16T15:13:10.180396+0000","last_peered":"2024-01-16T15:13:10.180396+0000","last_clean":"2024-01-16T15:13:10.180396+0000","last_became_active":"2024-01-16T15:13:09.146300+0000","last_became_peered":"2024-01-16T15:13:09.146300+0000","last_unstale":"2024-01-16T15:13:10.180396+0000","last_undegraded":"2024-01-16T15:13:10.180396+0000","last_fullsized":"2024-01-16T15:13:10.180396+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:09.955603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171072+0000","last_change":"2024-01-16T15:13:10.169454+0000","last_active":"2024-01-16T15:13:10.171072+0000","last_peered":"2024-01-16T15:13:10.171072+0000","last_clean":"2024-01-16T15:13:10.171072+0000","last_became_active":"2024-01-16T15:13:10.168762+0000","last_became_peered":"2024-01-16T15:13:10.168762+0000","last_unstale":"2024-01-16T15:13:10.171072+0000","last_undegraded":"2024-01-16T15:13:10.171072+0000","last_fullsized":"2024-01-16T15:13:10.171072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.661926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.129678+0000","last_change":"2024-01-16T15:13:09.140730+0000","last_active":"2024-01-16T15:13:10.129678+0000","last_peered":"2024-01-16T15:13:10.129678+0000","last_clean":"2024-01-16T15:13:10.129678+0000","last_became_active":"2024-01-16T15:13:09.139479+0000","last_became_peered":"2024-01-16T15:13:09.139479+0000","last_unstale":"2024-01-16T15:13:10.129678+0000","last_undegraded":"2024-01-16T15:13:10.129678+0000","last_fullsized":"2024-01-16T15:13:10.129678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:12.555688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154880+0000","last_change":"2024-01-16T15:13:10.148419+0000","last_active":"2024-01-16T15:13:10.154880+0000","last_peered":"2024-01-16T15:13:10.154880+0000","last_clean":"2024-01-16T15:13:10.154880+0000","last_became_active":"2024-01-16T15:13:10.148048+0000","last_became_peered":"2024-01-16T15:13:10.148048+0000","last_unstale":"2024-01-16T15:13:10.154880+0000","last_undegraded":"2024-01-16T15:13:10.154880+0000","last_fullsized":"2024-01-16T15:13:10.154880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:45.625292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.128551+0000","last_change":"2024-01-16T15:13:09.149260+0000","last_active":"2024-01-16T15:13:10.128551+0000","last_peered":"2024-01-16T15:13:10.128551+0000","last_clean":"2024-01-16T15:13:10.128551+0000","last_became_active":"2024-01-16T15:13:09.147795+0000","last_became_peered":"2024-01-16T15:13:09.147795+0000","last_unstale":"2024-01-16T15:13:10.128551+0000","last_undegraded":"2024-01-16T15:13:10.128551+0000","last_fullsized":"2024-01-16T15:13:10.128551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:57.248367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164385+0000","last_change":"2024-01-16T15:13:10.161847+0000","last_active":"2024-01-16T15:13:10.164385+0000","last_peered":"2024-01-16T15:13:10.164385+0000","last_clean":"2024-01-16T15:13:10.164385+0000","last_became_active":"2024-01-16T15:13:10.161523+0000","last_became_peered":"2024-01-16T15:13:10.161523+0000","last_unstale":"2024-01-16T15:13:10.164385+0000","last_undegraded":"2024-01-16T15:13:10.164385+0000","last_fullsized":"2024-01-16T15:13:10.164385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:49.618603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158270+0000","last_change":"2024-01-16T15:13:09.143825+0000","last_active":"2024-01-16T15:13:10.158270+0000","last_peered":"2024-01-16T15:13:10.158270+0000","last_clean":"2024-01-16T15:13:10.158270+0000","last_became_active":"2024-01-16T15:13:09.143491+0000","last_became_peered":"2024-01-16T15:13:09.143491+0000","last_unstale":"2024-01-16T15:13:10.158270+0000","last_undegraded":"2024-01-16T15:13:10.158270+0000","last_fullsized":"2024-01-16T15:13:10.158270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:28.474227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.152836+0000","last_change":"2024-01-16T15:13:10.149842+0000","last_active":"2024-01-16T15:13:10.152836+0000","last_peered":"2024-01-16T15:13:10.152836+0000","last_clean":"2024-01-16T15:13:10.152836+0000","last_became_active":"2024-01-16T15:13:10.149557+0000","last_became_peered":"2024-01-16T15:13:10.149557+0000","last_unstale":"2024-01-16T15:13:10.152836+0000","last_undegraded":"2024-01-16T15:13:10.152836+0000","last_fullsized":"2024-01-16T15:13:10.152836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:39.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182858+0000","last_change":"2024-01-16T15:13:09.146510+0000","last_active":"2024-01-16T15:13:10.182858+0000","last_peered":"2024-01-16T15:13:10.182858+0000","last_clean":"2024-01-16T15:13:10.182858+0000","last_became_active":"2024-01-16T15:13:09.143347+0000","last_became_peered":"2024-01-16T15:13:09.143347+0000","last_unstale":"2024-01-16T15:13:10.182858+0000","last_undegraded":"2024-01-16T15:13:10.182858+0000","last_fullsized":"2024-01-16T15:13:10.182858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:23.970257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.159710+0000","last_change":"2024-01-16T15:13:10.158291+0000","last_active":"2024-01-16T15:13:10.159710+0000","last_peered":"2024-01-16T15:13:10.159710+0000","last_clean":"2024-01-16T15:13:10.159710+0000","last_became_active":"2024-01-16T15:13:10.157922+0000","last_became_peered":"2024-01-16T15:13:10.157922+0000","last_unstale":"2024-01-16T15:13:10.159710+0000","last_undegraded":"2024-01-16T15:13:10.159710+0000","last_fullsized":"2024-01-16T15:13:10.159710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:02.049659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162946+0000","last_change":"2024-01-16T15:13:09.139809+0000","last_active":"2024-01-16T15:13:10.162946+0000","last_peered":"2024-01-16T15:13:10.162946+0000","last_clean":"2024-01-16T15:13:10.162946+0000","last_became_active":"2024-01-16T15:13:09.139512+0000","last_became_peered":"2024-01-16T15:13:09.139512+0000","last_unstale":"2024-01-16T15:13:10.162946+0000","last_undegraded":"2024-01-16T15:13:10.162946+0000","last_fullsized":"2024-01-16T15:13:10.162946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:40.199746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.155149+0000","last_change":"2024-01-16T15:13:10.151299+0000","last_active":"2024-01-16T15:13:10.155149+0000","last_peered":"2024-01-16T15:13:10.155149+0000","last_clean":"2024-01-16T15:13:10.155149+0000","last_became_active":"2024-01-16T15:13:10.142350+0000","last_became_peered":"2024-01-16T15:13:10.142350+0000","last_unstale":"2024-01-16T15:13:10.155149+0000","last_undegraded":"2024-01-16T15:13:10.155149+0000","last_fullsized":"2024-01-16T15:13:10.155149+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:58.554996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.135771+0000","last_change":"2024-01-16T15:13:09.153583+0000","last_active":"2024-01-16T15:13:10.135771+0000","last_peered":"2024-01-16T15:13:10.135771+0000","last_clean":"2024-01-16T15:13:10.135771+0000","last_became_active":"2024-01-16T15:13:09.153175+0000","last_became_peered":"2024-01-16T15:13:09.153175+0000","last_unstale":"2024-01-16T15:13:10.135771+0000","last_undegraded":"2024-01-16T15:13:10.135771+0000","last_fullsized":"2024-01-16T15:13:10.135771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:37.011904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170136+0000","last_change":"2024-01-16T15:13:10.166069+0000","last_active":"2024-01-16T15:13:10.170136+0000","last_peered":"2024-01-16T15:13:10.170136+0000","last_clean":"2024-01-16T15:13:10.170136+0000","last_became_active":"2024-01-16T15:13:10.165744+0000","last_became_peered":"2024-01-16T15:13:10.165744+0000","last_unstale":"2024-01-16T15:13:10.170136+0000","last_undegraded":"2024-01-16T15:13:10.170136+0000","last_fullsized":"2024-01-16T15:13:10.170136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:29.900475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.138906+0000","last_change":"2024-01-16T15:13:09.152498+0000","last_active":"2024-01-16T15:13:10.138906+0000","last_peered":"2024-01-16T15:13:10.138906+0000","last_clean":"2024-01-16T15:13:10.138906+0000","last_became_active":"2024-01-16T15:13:09.152008+0000","last_became_peered":"2024-01-16T15:13:09.152008+0000","last_unstale":"2024-01-16T15:13:10.138906+0000","last_undegraded":"2024-01-16T15:13:10.138906+0000","last_fullsized":"2024-01-16T15:13:10.138906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:34.738173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164733+0000","last_change":"2024-01-16T15:13:10.163069+0000","last_active":"2024-01-16T15:13:10.164733+0000","last_peered":"2024-01-16T15:13:10.164733+0000","last_clean":"2024-01-16T15:13:10.164733+0000","last_became_active":"2024-01-16T15:13:10.162720+0000","last_became_peered":"2024-01-16T15:13:10.162720+0000","last_unstale":"2024-01-16T15:13:10.164733+0000","last_undegraded":"2024-01-16T15:13:10.164733+0000","last_fullsized":"2024-01-16T15:13:10.164733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:39.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154513+0000","last_change":"2024-01-16T15:13:09.142520+0000","last_active":"2024-01-16T15:13:10.154513+0000","last_peered":"2024-01-16T15:13:10.154513+0000","last_clean":"2024-01-16T15:13:10.154513+0000","last_became_active":"2024-01-16T15:13:09.142231+0000","last_became_peered":"2024-01-16T15:13:09.142231+0000","last_unstale":"2024-01-16T15:13:10.154513+0000","last_undegraded":"2024-01-16T15:13:10.154513+0000","last_fullsized":"2024-01-16T15:13:10.154513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:38.946708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.166479+0000","last_change":"2024-01-16T15:13:10.155976+0000","last_active":"2024-01-16T15:13:10.166479+0000","last_peered":"2024-01-16T15:13:10.166479+0000","last_clean":"2024-01-16T15:13:10.166479+0000","last_became_active":"2024-01-16T15:13:10.155649+0000","last_became_peered":"2024-01-16T15:13:10.155649+0000","last_unstale":"2024-01-16T15:13:10.166479+0000","last_undegraded":"2024-01-16T15:13:10.166479+0000","last_fullsized":"2024-01-16T15:13:10.166479+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:35.748090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.138828+0000","last_change":"2024-01-16T15:13:09.149896+0000","last_active":"2024-01-16T15:13:10.138828+0000","last_peered":"2024-01-16T15:13:10.138828+0000","last_clean":"2024-01-16T15:13:10.138828+0000","last_became_active":"2024-01-16T15:13:09.148929+0000","last_became_peered":"2024-01-16T15:13:09.148929+0000","last_unstale":"2024-01-16T15:13:10.138828+0000","last_undegraded":"2024-01-16T15:13:10.138828+0000","last_fullsized":"2024-01-16T15:13:10.138828+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:42.077697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171005+0000","last_change":"2024-01-16T15:13:10.169837+0000","last_active":"2024-01-16T15:13:10.171005+0000","last_peered":"2024-01-16T15:13:10.171005+0000","last_clean":"2024-01-16T15:13:10.171005+0000","last_became_active":"2024-01-16T15:13:10.169049+0000","last_became_peered":"2024-01-16T15:13:10.169049+0000","last_unstale":"2024-01-16T15:13:10.171005+0000","last_undegraded":"2024-01-16T15:13:10.171005+0000","last_fullsized":"2024-01-16T15:13:10.171005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:58.374204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.136554+0000","last_change":"2024-01-16T15:13:09.153140+0000","last_active":"2024-01-16T15:13:10.136554+0000","last_peered":"2024-01-16T15:13:10.136554+0000","last_clean":"2024-01-16T15:13:10.136554+0000","last_became_active":"2024-01-16T15:13:09.151786+0000","last_became_peered":"2024-01-16T15:13:09.151786+0000","last_unstale":"2024-01-16T15:13:10.136554+0000","last_undegraded":"2024-01-16T15:13:10.136554+0000","last_fullsized":"2024-01-16T15:13:10.136554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:58.217563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.152530+0000","last_change":"2024-01-16T15:13:10.140971+0000","last_active":"2024-01-16T15:13:10.152530+0000","last_peered":"2024-01-16T15:13:10.152530+0000","last_clean":"2024-01-16T15:13:10.152530+0000","last_became_active":"2024-01-16T15:13:10.139718+0000","last_became_peered":"2024-01-16T15:13:10.139718+0000","last_unstale":"2024-01-16T15:13:10.152530+0000","last_undegraded":"2024-01-16T15:13:10.152530+0000","last_fullsized":"2024-01-16T15:13:10.152530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:49.618603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.137527+0000","last_change":"2024-01-16T15:13:09.149662+0000","last_active":"2024-01-16T15:13:10.137527+0000","last_peered":"2024-01-16T15:13:10.137527+0000","last_clean":"2024-01-16T15:13:10.137527+0000","last_became_active":"2024-01-16T15:13:09.147255+0000","last_became_peered":"2024-01-16T15:13:09.147255+0000","last_unstale":"2024-01-16T15:13:10.137527+0000","last_undegraded":"2024-01-16T15:13:10.137527+0000","last_fullsized":"2024-01-16T15:13:10.137527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:48.608686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168426+0000","last_change":"2024-01-16T15:13:10.160037+0000","last_active":"2024-01-16T15:13:10.168426+0000","last_peered":"2024-01-16T15:13:10.168426+0000","last_clean":"2024-01-16T15:13:10.168426+0000","last_became_active":"2024-01-16T15:13:10.159336+0000","last_became_peered":"2024-01-16T15:13:10.159336+0000","last_unstale":"2024-01-16T15:13:10.168426+0000","last_undegraded":"2024-01-16T15:13:10.168426+0000","last_fullsized":"2024-01-16T15:13:10.168426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:06.237086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.177383+0000","last_change":"2024-01-16T15:13:09.152363+0000","last_active":"2024-01-16T15:13:10.177383+0000","last_peered":"2024-01-16T15:13:10.177383+0000","last_clean":"2024-01-16T15:13:10.177383+0000","last_became_active":"2024-01-16T15:13:09.152050+0000","last_became_peered":"2024-01-16T15:13:09.152050+0000","last_unstale":"2024-01-16T15:13:10.177383+0000","last_undegraded":"2024-01-16T15:13:10.177383+0000","last_fullsized":"2024-01-16T15:13:10.177383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:53.227486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":223,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:09.142854+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:13:09.142854+0000","last_peered":"2024-01-16T15:13:09.142854+0000","last_clean":"2024-01-16T15:13:09.142854+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:13:09.142854+0000","last_undegraded":"2024-01-16T15:13:09.142854+0000","last_fullsized":"2024-01-16T15:13:09.142854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68844,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302996,"statfs":{"total":96636764160,"available":96566267904,"internally_reserved":0,"allocated":1662976,"data_stored":1046287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.8160000000000001,"15min":1.4870000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":4.1020000000000003,"5min":27.469000000000001,"15min":27.469000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.4740000000000002,"5min":2.0720000000000001,"15min":1.625},"min":{"1min":0.69399999999999995,"5min":0.60299999999999998,"15min":0.47099999999999997},"max":{"1min":25.503,"5min":25.503,"15min":25.503},"last":1.0669999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2689999999999999,"15min":1.2969999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.5010000000000003,"5min":6.718,"15min":6.718},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.274,"15min":1.3440000000000001},"min":{"1min":0.68100000000000005,"5min":0.64700000000000002,"15min":0.373},"max":{"1min":4.3369999999999997,"5min":4.3369999999999997,"15min":15.452},"last":1.1759999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4910000000000001,"15min":1.4450000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.48299999999999998},"max":{"1min":4.2859999999999996,"5min":8.6349999999999998,"15min":10.707000000000001},"last":1.3},{"interface":"front","average":{"1min":2.5289999999999999,"5min":1.7509999999999999,"15min":1.532},"min":{"1min":0.65700000000000003,"5min":0.64400000000000002,"15min":0.504},"max":{"1min":25.478000000000002,"5min":25.478000000000002,"15min":25.478000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.5760000000000001,"15min":1.5720000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.6369999999999996,"5min":13.907999999999999,"15min":13.907999999999999},"last":1.123},{"interface":"front","average":{"1min":1.627,"5min":1.5569999999999999,"15min":1.5780000000000001},"min":{"1min":0.67800000000000005,"5min":0.63,"15min":0.62},"max":{"1min":6.351,"5min":8.6999999999999993,"15min":13.193},"last":1.444}]},{"osd":4,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":2.6619999999999999,"5min":1.6599999999999999,"15min":1.4810000000000001},"min":{"1min":0.60599999999999998,"5min":0.41599999999999998,"15min":0.376},"max":{"1min":25.401,"5min":25.401,"15min":25.401},"last":1.21},{"interface":"front","average":{"1min":1.587,"5min":1.677,"15min":1.5249999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.7859999999999996,"5min":10.134,"15min":10.134},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.603,"15min":1.538},"min":{"1min":0.63800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":4.4539999999999997,"5min":12.528,"15min":12.528},"last":1.5660000000000001},{"interface":"front","average":{"1min":2.6680000000000001,"5min":1.819,"15min":1.6399999999999999},"min":{"1min":0.61399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":26.327000000000002,"5min":26.327000000000002,"15min":26.327000000000002},"last":1.5009999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":2.8149999999999999,"5min":1.97,"15min":1.748},"min":{"1min":0.77400000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":25.451000000000001,"5min":25.451000000000001,"15min":25.451000000000001},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.7090000000000001,"15min":1.6819999999999999},"min":{"1min":0.76600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":5.8250000000000002,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.5349999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542312,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68244,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303596,"statfs":{"total":96636764160,"available":96566882304,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.232,"15min":1.6439999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.518,"5min":4.6600000000000001,"15min":30.164999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.365,"15min":1.712},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6899999999999999,"5min":11.52,"15min":30.280999999999999},"last":1.038}]},{"osd":1,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.4059999999999999,"15min":1.4690000000000001},"min":{"1min":0.61099999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.766,"5min":8.3439999999999994,"15min":28.122},"last":1.536},{"interface":"front","average":{"1min":1.097,"5min":1.2689999999999999,"15min":1.2589999999999999},"min":{"1min":0.70899999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4530000000000001,"5min":7.7489999999999997,"15min":8.9719999999999995},"last":0.71599999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.9750000000000001,"15min":1.6870000000000001},"min":{"1min":0.89700000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.6259999999999999,"5min":26.109000000000002,"15min":26.109000000000002},"last":1.325},{"interface":"front","average":{"1min":1.4359999999999999,"5min":2.117,"15min":1.841},"min":{"1min":0.99099999999999999,"5min":0.67500000000000004,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":26.161000000000001,"15min":26.161000000000001},"last":1.429}]},{"osd":3,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.524,"15min":1.6040000000000001},"min":{"1min":0.77900000000000003,"5min":0.50900000000000001,"15min":0.44700000000000001},"max":{"1min":1.663,"5min":11.289,"15min":17.75},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.6519999999999999,"15min":1.5960000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.55700000000000005},"max":{"1min":1.629,"5min":11.433,"15min":17.794},"last":1.151}]},{"osd":4,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.5840000000000001,"15min":1.5449999999999999},"min":{"1min":0.87,"5min":0.45500000000000002,"15min":0.45000000000000001},"max":{"1min":1.954,"5min":8.4890000000000008,"15min":15.571},"last":1.486},{"interface":"front","average":{"1min":1.228,"5min":1.5840000000000001,"15min":1.583},"min":{"1min":0.74199999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5680000000000001,"5min":23.667999999999999,"15min":23.667999999999999},"last":1.095}]},{"osd":5,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.661,"15min":1.456},"min":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4950000000000001,"5min":23.137,"15min":23.137},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.8009999999999999,"15min":1.673},"min":{"1min":0.82199999999999995,"5min":0.48099999999999998,"15min":0.47099999999999997},"max":{"1min":1.758,"5min":25.684000000000001,"15min":25.684000000000001},"last":1.385}]},{"osd":7,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.5449999999999999,"15min":1.5109999999999999},"min":{"1min":0.57099999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7,"5min":11.597,"15min":11.597},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.768,"15min":1.6899999999999999},"min":{"1min":1.01,"5min":0.499,"15min":0.499},"max":{"1min":1.859,"5min":8.6210000000000004,"15min":11.788},"last":1.5840000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76564,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295276,"statfs":{"total":96636764160,"available":96558362624,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.393,"15min":1.591},"min":{"1min":0.52500000000000002,"5min":0.41299999999999998,"15min":0.39000000000000001},"max":{"1min":1.6739999999999999,"5min":22.289000000000001,"15min":26.228999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.03,"5min":1.5049999999999999,"15min":1.833},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.7689999999999999,"5min":17.131,"15min":56.011000000000003},"last":1.006}]},{"osd":1,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.2470000000000001,"15min":1.373},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.698,"5min":6.0940000000000003,"15min":16.687999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.8340000000000001,"15min":1.9139999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40999999999999998},"max":{"1min":4.5960000000000001,"5min":22.451000000000001,"15min":67.930000000000007},"last":1.292}]},{"osd":2,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.577,"15min":1.5469999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.306,"5min":11.329000000000001,"15min":13.193},"last":1.246},{"interface":"front","average":{"1min":1.899,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.52200000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":16.721,"5min":16.721,"15min":16.721},"last":0.79000000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.8160000000000001,"15min":1.7769999999999999},"min":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.923,"5min":22.382000000000001,"15min":22.382000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.837,"15min":1.8049999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.2400000000000002,"5min":14.685,"15min":14.685},"last":1.157}]},{"osd":4,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":1.4379999999999999,"15min":1.4450000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.2},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.625,"15min":1.577},"min":{"1min":0.51500000000000001,"5min":0.44700000000000001,"15min":0.379},"max":{"1min":1.9790000000000001,"5min":27.006,"15min":27.006},"last":1.0880000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6719999999999999,"15min":1.5780000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.3380000000000001,"5min":10.106,"15min":13.263999999999999},"last":1.403},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.667,"15min":1.6379999999999999},"min":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.505},"max":{"1min":1.7969999999999999,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":1.1140000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.9039999999999999,"15min":1.8009999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.0249999999999999,"5min":35.127000000000002,"15min":35.127000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.7829999999999999,"15min":1.786},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.42999999999999999},"max":{"1min":1.885,"5min":10.016,"15min":13.491},"last":1.3480000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542312,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64084,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307756,"statfs":{"total":96636764160,"available":96571142144,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.103,"15min":1.2270000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.502},"max":{"1min":1.4950000000000001,"5min":6.3339999999999996,"15min":12.587999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.224,"15min":1.47},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.7729999999999999,"5min":4.976,"15min":60.106000000000002},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.595,"15min":1.6830000000000001},"min":{"1min":0.64500000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.673,"5min":10.124000000000001,"15min":22.143000000000001},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.369,"15min":1.51},"min":{"1min":0.76000000000000001,"5min":0.628,"15min":0.57099999999999995},"max":{"1min":1.748,"5min":5.1550000000000002,"15min":22.181999999999999},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.367,"15min":1.3129999999999999},"min":{"1min":0.875,"5min":0.58099999999999996,"15min":0.52800000000000002},"max":{"1min":2.3719999999999999,"5min":7.2110000000000003,"15min":9.7140000000000004},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.49,"15min":1.728},"min":{"1min":0.81699999999999995,"5min":0.63100000000000001,"15min":0.52400000000000002},"max":{"1min":1.696,"5min":13.316000000000001,"15min":35.024999999999999},"last":1.2849999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.546,"15min":1.5780000000000001},"min":{"1min":0.69399999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.8089999999999999,"5min":24.77,"15min":32.133000000000003},"last":1.238},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3879999999999999},"min":{"1min":0.71899999999999997,"5min":0.623,"15min":0.57999999999999996},"max":{"1min":2.2240000000000002,"5min":5.4690000000000003,"15min":7.8140000000000001},"last":1.341}]},{"osd":5,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.4830000000000001},"min":{"1min":0.55200000000000005,"5min":0.47199999999999998,"15min":0.38900000000000001},"max":{"1min":2.0459999999999998,"5min":5.2149999999999999,"15min":14.568},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.3979999999999999,"15min":1.464},"min":{"1min":0.73599999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.141,"5min":3.5129999999999999,"15min":7.1479999999999997},"last":1.6279999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.379,"15min":1.4059999999999999},"min":{"1min":0.67500000000000004,"5min":0.436,"15min":0.436},"max":{"1min":2.2959999999999998,"5min":4.9950000000000001,"15min":9.6630000000000003},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.7909999999999999,"15min":1.679},"min":{"1min":0.94799999999999995,"5min":0.51700000000000002,"15min":0.45100000000000001},"max":{"1min":1.915,"5min":10.054,"15min":14.317},"last":1.907}]},{"osd":7,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5720000000000001,"15min":1.5820000000000001},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.52100000000000002},"max":{"1min":2.0750000000000002,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.625,"15min":1.589},"min":{"1min":0.755,"5min":0.443,"15min":0.443},"max":{"1min":2.6379999999999999,"5min":9.2270000000000003,"15min":9.2270000000000003},"last":1.8500000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76564,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295276,"statfs":{"total":96636764160,"available":96558362624,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.383,"15min":1.514},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.26700000000000002},"max":{"1min":1.5720000000000001,"5min":14.111000000000001,"15min":50.988999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.748,"15min":1.4710000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.29699999999999999},"max":{"1min":1.8109999999999999,"5min":20.274000000000001,"15min":20.274000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.6899999999999999,"15min":1.381},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.274,"5min":18.192,"15min":18.192},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.9870000000000001,"15min":1.738},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":1.5509999999999999,"5min":18.34,"15min":20.385999999999999},"last":0.73799999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":2.3050000000000002,"15min":1.6950000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.5429999999999999,"5min":42.840000000000003,"15min":42.840000000000003},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.137,"5min":2.194,"15min":1.8089999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.35199999999999998},"max":{"1min":1.746,"5min":38.707000000000001,"15min":38.707000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.9299999999999999,"15min":1.76},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.738,"5min":14.148,"15min":18.952000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":2.2120000000000002,"15min":1.778},"min":{"1min":0.755,"5min":0.61599999999999999,"15min":0.54800000000000004},"max":{"1min":1.7669999999999999,"5min":18.088999999999999,"15min":18.088999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.9830000000000001,"15min":1.875},"min":{"1min":0.748,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.9019999999999999,"5min":18.224,"15min":24.349},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.2789999999999999,"5min":2.1850000000000001,"15min":1.913},"min":{"1min":0.66500000000000004,"5min":0.66100000000000003,"15min":0.61899999999999999},"max":{"1min":2.0249999999999999,"5min":14.192,"15min":14.192},"last":1.046}]},{"osd":6,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":2.2509999999999999,"15min":1.837},"min":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.52200000000000002},"max":{"1min":1.762,"5min":22.687000000000001,"15min":22.687000000000001},"last":1.03},{"interface":"front","average":{"1min":1.419,"5min":2.0790000000000002,"15min":1.9510000000000001},"min":{"1min":0.81100000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.044,"5min":18.300000000000001,"15min":24.058},"last":1.2549999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":2.6299999999999999,"15min":2.073},"min":{"1min":0.86099999999999999,"5min":0.76900000000000002,"15min":0.55800000000000005},"max":{"1min":2.1800000000000002,"5min":35.570999999999998,"15min":35.570999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":2.6909999999999998,"15min":2.0369999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.58899999999999997},"max":{"1min":1.857,"5min":37.607999999999997,"15min":37.607999999999997},"last":1.2869999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542312,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.212,"15min":1.4910000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.24399999999999999},"max":{"1min":1.4750000000000001,"5min":11.112,"15min":64.387},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.84099999999999997,"5min":1.581,"15min":1.391},"min":{"1min":0.50700000000000001,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.143,"5min":24.308,"15min":24.308},"last":0.36299999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.3069999999999999,"15min":1.4379999999999999},"min":{"1min":0.65100000000000002,"5min":0.42299999999999999,"15min":0.28299999999999997},"max":{"1min":1.966,"5min":13.497,"15min":27.263000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.994,"15min":1.7969999999999999},"min":{"1min":0.56499999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.716,"5min":23.602,"15min":23.602},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.4590000000000001,"15min":1.5700000000000001},"min":{"1min":0.54500000000000004,"5min":0.497,"15min":0.33100000000000002},"max":{"1min":3.1080000000000001,"5min":13.007,"15min":27.311},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.6060000000000001,"15min":1.6200000000000001},"min":{"1min":0.66700000000000004,"5min":0.49099999999999999,"15min":0.47099999999999997},"max":{"1min":1.8100000000000001,"5min":12.833,"15min":26.555},"last":0.71999999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.7929999999999999,"15min":1.6559999999999999},"min":{"1min":0.86799999999999999,"5min":0.54000000000000004,"15min":0.38900000000000001},"max":{"1min":1.879,"5min":17.146999999999998,"15min":26.503},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.413,"5min":2.097,"15min":1.7390000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.9079999999999999,"5min":25.661000000000001,"15min":25.661000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.6319999999999999,"15min":1.7170000000000001},"min":{"1min":0.99199999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.3029999999999999,"5min":11.005000000000001,"15min":27.149000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.7809999999999999,"15min":1.756},"min":{"1min":0.91500000000000004,"5min":0.66900000000000004,"15min":0.43099999999999999},"max":{"1min":2.081,"5min":10.984,"15min":26.349},"last":0.59799999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":2.577,"15min":1.9319999999999999},"min":{"1min":1.0760000000000001,"5min":0.504,"15min":0.504},"max":{"1min":13.629,"5min":25.603999999999999,"15min":25.603999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.794,"15min":1.8660000000000001},"min":{"1min":1.02,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.2010000000000001,"5min":19.298999999999999,"15min":27.076000000000001},"last":0.629}]},{"osd":7,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.3090000000000002,"15min":1.96},"min":{"1min":0.96499999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"max":{"1min":2.9940000000000002,"5min":24.509,"15min":24.509},"last":0.61699999999999999},{"interface":"front","average":{"1min":2.323,"5min":1.9419999999999999,"15min":1.8959999999999999},"min":{"1min":1.2110000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":10.455,"5min":10.455,"15min":27.207999999999998},"last":0.68600000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542312,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72428,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299412,"statfs":{"total":96636764160,"available":96562597888,"internally_reserved":0,"allocated":1077248,"data_stored":456043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.123,"15min":1.28},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.266,"5min":6.2249999999999996,"15min":14.407},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3169999999999999,"15min":1.3180000000000001},"min":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.3620000000000001,"5min":11.074999999999999,"15min":13.930999999999999},"last":1.238}]},{"osd":2,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.3660000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":5.9130000000000003,"5min":5.9130000000000003,"15min":16.488},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.6080000000000001,"15min":1.71},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.1559999999999997,"5min":21.446000000000002,"15min":22.550999999999998},"last":1.4630000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.831,"15min":1.7509999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.1980000000000004,"5min":22.471,"15min":22.471},"last":1.649},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.6719999999999999,"15min":1.6439999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.35699999999999998},"max":{"1min":2.2570000000000001,"5min":22.550999999999998,"15min":22.550999999999998},"last":1.837}]},{"osd":4,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5940000000000001,"15min":1.663},"min":{"1min":0.69899999999999995,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3159999999999998,"5min":5.3789999999999996,"15min":12.675000000000001},"last":1.526},{"interface":"front","average":{"1min":1.448,"5min":1.649,"15min":1.724},"min":{"1min":0.87,"5min":0.433,"15min":0.433},"max":{"1min":3.6269999999999998,"5min":5.6849999999999996,"15min":19.236999999999998},"last":1.5880000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.851,"15min":1.778},"min":{"1min":0.80900000000000005,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.3719999999999999,"5min":21.117000000000001,"15min":21.117000000000001},"last":1.772},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6799999999999999,"15min":1.7370000000000001},"min":{"1min":0.82999999999999996,"5min":0.65800000000000003,"15min":0.64800000000000002},"max":{"1min":3.5470000000000002,"5min":5.9009999999999998,"15min":17.138000000000002},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.7410000000000001,"15min":1.863},"min":{"1min":0.85499999999999998,"5min":0.77500000000000002,"15min":0.72199999999999998},"max":{"1min":4.2350000000000003,"5min":9.3759999999999994,"15min":14.781000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.7190000000000001,"15min":1.861},"min":{"1min":0.754,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.407,"5min":5.4630000000000001,"15min":23.420999999999999},"last":1.6799999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.76,"15min":1.823},"min":{"1min":0.81999999999999995,"5min":0.71299999999999997,"15min":0.60699999999999998},"max":{"1min":3.7690000000000001,"5min":5.8300000000000001,"15min":17.010000000000002},"last":1.397},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.97,"15min":1.988},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.75700000000000001},"max":{"1min":4.9640000000000004,"5min":10.316000000000001,"15min":14.884},"last":1.333}]}]},{"osd":0,"up_from":14,"seq":60129542313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68836,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303004,"statfs":{"total":96636764160,"available":96566276096,"internally_reserved":0,"allocated":1662976,"data_stored":1046251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":68811201},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.603,"15min":1.456},"min":{"1min":0.51700000000000002,"5min":0.48199999999999998,"15min":0.36499999999999999},"max":{"1min":1.748,"5min":43.459000000000003,"15min":43.459000000000003},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.726,"15min":1.5940000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.28999999999999998},"max":{"1min":2.1339999999999999,"5min":43.627000000000002,"15min":43.627000000000002},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3999999999999999,"15min":1.4099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":18.861000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.028,"5min":1.3120000000000001,"15min":1.452},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.378},"max":{"1min":2.2530000000000001,"5min":8.9979999999999993,"15min":38.893000000000001},"last":0.98499999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.5009999999999999,"15min":1.5449999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.31900000000000001},"max":{"1min":2.0459999999999998,"5min":23.687999999999999,"15min":23.687999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.28,"5min":1.6839999999999999,"15min":1.631},"min":{"1min":0.629,"5min":0.59999999999999998,"15min":0.46800000000000003},"max":{"1min":1.788,"5min":16.978000000000002,"15min":18.966999999999999},"last":1.1000000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.5309999999999999,"15min":1.4830000000000001},"min":{"1min":0.69099999999999995,"5min":0.63400000000000001,"15min":0.58899999999999997},"max":{"1min":2.105,"5min":10.18,"15min":20.327000000000002},"last":1.375},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.581,"15min":1.6879999999999999},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8460000000000001,"5min":4.665,"15min":19.004999999999999},"last":1.2889999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.7410000000000001,"15min":1.8069999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.54600000000000004},"max":{"1min":2.1800000000000002,"5min":13.016,"15min":24.611000000000001},"last":1.329},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":1.6659999999999999},"min":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":20.199000000000002},"last":1.5229999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":2.0150000000000001,"15min":2.0249999999999999},"min":{"1min":0.72999999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":3.9569999999999999,"5min":16.832999999999998,"15min":26.998000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.95,"15min":1.929},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":4.0419999999999998,"5min":16.093,"15min":24.478999999999999},"last":1.4239999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.7649999999999999,"15min":1.8480000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.58399999999999996},"max":{"1min":3.0910000000000002,"5min":7.0759999999999996,"15min":17.957000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.663,"15min":1.71},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.52800000000000002},"max":{"1min":2.423,"5min":5.6109999999999998,"15min":20.279},"last":1.617}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:13:14.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:13:14.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.921+0000 7f52cb200640 1 Processor -- start 2024-01-16T15:13:14.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.921+0000 7f52cb200640 1 -- start start 2024-01-16T15:13:14.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.922+0000 7f52cb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc13dfd0 0x7f52cc15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.922+0000 7f52cb200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52cc0cf4d0 0x7f52cc06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.922+0000 7f52cb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52cc0ce7c0 con 0x7f52cc0cf050 2024-01-16T15:13:14.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.922+0000 7f52cb200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc140900 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.922+0000 7f52cb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc15fa70 con 0x7f52cc13dfd0 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.923+0000 7f52ca800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52cc0cf4d0 0x7f52cc06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.923+0000 7f52c9e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc13dfd0 0x7f52cc15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.923+0000 7f52ca800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52cc0cf4d0 0x7f52cc06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:53514/0 (socket says 172.21.15.43:53514) 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.923+0000 7f52ca800640 1 -- 172.21.15.43:0/3607398168 learned_addr learned my addr 172.21.15.43:0/3607398168 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.923+0000 7f52c3e00640 1 --1- 172.21.15.43:0/3607398168 >> v1:172.21.15.69:6789/0 conn(0x7f52cc0cf050 0x7f52cc06c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:43934/0 (socket says 172.21.15.43:43934) 2024-01-16T15:13:14.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.924+0000 7f52ca800640 1 -- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc13dfd0 msgr2=0x7f52cc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:14.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.924+0000 7f52ca800640 1 --2- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc13dfd0 0x7f52cc15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.924+0000 7f52ca800640 1 -- 172.21.15.43:0/3607398168 >> v1:172.21.15.69:6789/0 conn(0x7f52cc0cf050 legacy=0x7f52cc06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.924+0000 7f52ca800640 1 -- 172.21.15.43:0/3607398168 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc15ee60 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.925+0000 7f52ca800640 1 --2- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc06c840 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009e50 tx=0x7f52c402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4f63e301103f4cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.926+0000 7f52c9400640 1 -- 172.21.15.43:0/3607398168 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c402cb00 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.926+0000 7f52c9400640 1 -- 172.21.15.43:0/3607398168 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c402cc60 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.926+0000 7f52c9400640 1 -- 172.21.15.43:0/3607398168 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c40343f0 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 -- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc0cf4d0 msgr2=0x7f52cc06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 --2- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc06c840 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009e50 tx=0x7f52c402bf60 comp rx=0 tx=0).stop 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 -- 172.21.15.43:0/3607398168 shutdown_connections 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 --2- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc06c840 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 --2- 172.21.15.43:0/3607398168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc13dfd0 0x7f52cc15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 -- 172.21.15.43:0/3607398168 >> 172.21.15.43:0/3607398168 conn(0x7f52cc06a860 msgr2=0x7f52cc06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 -- 172.21.15.43:0/3607398168 shutdown_connections 2024-01-16T15:13:14.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.927+0000 7f52cb200640 1 -- 172.21.15.43:0/3607398168 wait complete. 2024-01-16T15:13:14.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.929+0000 7f52cb200640 1 Processor -- start 2024-01-16T15:13:14.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.929+0000 7f52cb200640 1 -- start start 2024-01-16T15:13:14.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0cf050 0x7f52cc155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc13dfd0 0x7f52cc070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0b26b0 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0cb330 con 0x7f52cc13dfd0 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52cb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0cb610 con 0x7f52cc0cf050 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42472/0 (socket says 172.21.15.43:42472) 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.930+0000 7f52c3e00640 1 -- 172.21.15.43:0/1467445949 learned_addr learned my addr 172.21.15.43:0/1467445949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c9e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0cf050 0x7f52cc155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52ca800640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc13dfd0 0x7f52cc070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 -- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0cf050 msgr2=0x7f52cc155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0cf050 0x7f52cc155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 -- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc13dfd0 msgr2=0x7f52cc070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc13dfd0 0x7f52cc070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 -- 172.21.15.43:0/1467445949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4009b30 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.931+0000 7f52c3e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c3e0 tx=0x7f52b400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.932+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b40074e0 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.932+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4007640 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.932+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b40049e0 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.932+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc0cb8f0 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.932+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc155960 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.933+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc0b2820 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.934+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f52b4004b40 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.935+0000 7f52c2000640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 0x7f52a406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.935+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f52b4088230 con 0x7f52cc0cf4d0 2024-01-16T15:13:14.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.936+0000 7f52c9e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 0x7f52a406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.936+0000 7f52c9e00640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 0x7f52a406e3e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ad0 tx=0x7f52bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:14.940+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52b4054870 con 0x7f52cc0cf4d0 2024-01-16T15:13:15.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.130+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52cc15f700 con 0x7f52a406bf20 2024-01-16T15:13:15.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.152+0000 7f52c2000640 1 -- 172.21.15.43:0/1467445949 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286840 (secure 0 0 0) 0x7f52bc0119a0 con 0x7f52a406bf20 2024-01-16T15:13:15.134 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:15.136 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:13:15.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 msgr2=0x7f52a406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 0x7f52a406e3e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009ad0 tx=0x7f52bc008040 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 msgr2=0x7f52cc0b2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c3e0 tx=0x7f52b400c8a0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 shutdown_connections 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f52a406bf20 0x7f52a406e3e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc13dfd0 0x7f52cc070ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52cc0cf4d0 0x7f52cc0b2b40 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 --2- 172.21.15.43:0/1467445949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0cf050 0x7f52cc155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 >> 172.21.15.43:0/1467445949 conn(0x7f52cc06a860 msgr2=0x7f52cc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.157+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 shutdown_connections 2024-01-16T15:13:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.158+0000 7f52cb200640 1 -- 172.21.15.43:0/1467445949 wait complete. 2024-01-16T15:13:15.158 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-01-16T15:13:14.186724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":572124,"kb_used_data":9440,"kb_used_omap":12,"kb_used_meta":562547,"kb_avail":754402596,"statfs":{"total":773094113280,"available":772508258304,"internally_reserved":0,"allocated":9666560,"data_stored":4793811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":576048684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.061901"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156147+0000","last_change":"2024-01-16T15:13:10.151745+0000","last_active":"2024-01-16T15:13:10.156147+0000","last_peered":"2024-01-16T15:13:10.156147+0000","last_clean":"2024-01-16T15:13:10.156147+0000","last_became_active":"2024-01-16T15:13:10.151455+0000","last_became_peered":"2024-01-16T15:13:10.151455+0000","last_unstale":"2024-01-16T15:13:10.156147+0000","last_undegraded":"2024-01-16T15:13:10.156147+0000","last_fullsized":"2024-01-16T15:13:10.156147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:11:04.893705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167414+0000","last_change":"2024-01-16T15:13:10.163440+0000","last_active":"2024-01-16T15:13:10.167414+0000","last_peered":"2024-01-16T15:13:10.167414+0000","last_clean":"2024-01-16T15:13:10.167414+0000","last_became_active":"2024-01-16T15:13:10.163147+0000","last_became_peered":"2024-01-16T15:13:10.163147+0000","last_unstale":"2024-01-16T15:13:10.167414+0000","last_undegraded":"2024-01-16T15:13:10.167414+0000","last_fullsized":"2024-01-16T15:13:10.167414+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:59.442824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158876+0000","last_change":"2024-01-16T15:13:10.158057+0000","last_active":"2024-01-16T15:13:10.158876+0000","last_peered":"2024-01-16T15:13:10.158876+0000","last_clean":"2024-01-16T15:13:10.158876+0000","last_became_active":"2024-01-16T15:13:10.157736+0000","last_became_peered":"2024-01-16T15:13:10.157736+0000","last_unstale":"2024-01-16T15:13:10.158876+0000","last_undegraded":"2024-01-16T15:13:10.158876+0000","last_fullsized":"2024-01-16T15:13:10.158876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:59.878158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180419+0000","last_change":"2024-01-16T15:13:10.177628+0000","last_active":"2024-01-16T15:13:10.180419+0000","last_peered":"2024-01-16T15:13:10.180419+0000","last_clean":"2024-01-16T15:13:10.180419+0000","last_became_active":"2024-01-16T15:13:10.177334+0000","last_became_peered":"2024-01-16T15:13:10.177334+0000","last_unstale":"2024-01-16T15:13:10.180419+0000","last_undegraded":"2024-01-16T15:13:10.180419+0000","last_fullsized":"2024-01-16T15:13:10.180419+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:55.522718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175993+0000","last_change":"2024-01-16T15:13:10.174080+0000","last_active":"2024-01-16T15:13:10.175993+0000","last_peered":"2024-01-16T15:13:10.175993+0000","last_clean":"2024-01-16T15:13:10.175993+0000","last_became_active":"2024-01-16T15:13:10.173718+0000","last_became_peered":"2024-01-16T15:13:10.173718+0000","last_unstale":"2024-01-16T15:13:10.175993+0000","last_undegraded":"2024-01-16T15:13:10.175993+0000","last_fullsized":"2024-01-16T15:13:10.175993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.661926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.161953+0000","last_change":"2024-01-16T15:13:10.157470+0000","last_active":"2024-01-16T15:13:10.161953+0000","last_peered":"2024-01-16T15:13:10.161953+0000","last_clean":"2024-01-16T15:13:10.161953+0000","last_became_active":"2024-01-16T15:13:10.157023+0000","last_became_peered":"2024-01-16T15:13:10.157023+0000","last_unstale":"2024-01-16T15:13:10.161953+0000","last_undegraded":"2024-01-16T15:13:10.161953+0000","last_fullsized":"2024-01-16T15:13:10.161953+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:07.126646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.157369+0000","last_change":"2024-01-16T15:13:10.151185+0000","last_active":"2024-01-16T15:13:10.157369+0000","last_peered":"2024-01-16T15:13:10.157369+0000","last_clean":"2024-01-16T15:13:10.157369+0000","last_became_active":"2024-01-16T15:13:10.150867+0000","last_became_peered":"2024-01-16T15:13:10.150867+0000","last_unstale":"2024-01-16T15:13:10.157369+0000","last_undegraded":"2024-01-16T15:13:10.157369+0000","last_fullsized":"2024-01-16T15:13:10.157369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:50:02.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171606+0000","last_change":"2024-01-16T15:13:10.170420+0000","last_active":"2024-01-16T15:13:10.171606+0000","last_peered":"2024-01-16T15:13:10.171606+0000","last_clean":"2024-01-16T15:13:10.171606+0000","last_became_active":"2024-01-16T15:13:10.169529+0000","last_became_peered":"2024-01-16T15:13:10.169529+0000","last_unstale":"2024-01-16T15:13:10.171606+0000","last_undegraded":"2024-01-16T15:13:10.171606+0000","last_fullsized":"2024-01-16T15:13:10.171606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.752436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162052+0000","last_change":"2024-01-16T15:13:10.160732+0000","last_active":"2024-01-16T15:13:10.162052+0000","last_peered":"2024-01-16T15:13:10.162052+0000","last_clean":"2024-01-16T15:13:10.162052+0000","last_became_active":"2024-01-16T15:13:10.160402+0000","last_became_peered":"2024-01-16T15:13:10.160402+0000","last_unstale":"2024-01-16T15:13:10.162052+0000","last_undegraded":"2024-01-16T15:13:10.162052+0000","last_fullsized":"2024-01-16T15:13:10.162052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:58.566297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167640+0000","last_change":"2024-01-16T15:13:10.165155+0000","last_active":"2024-01-16T15:13:10.167640+0000","last_peered":"2024-01-16T15:13:10.167640+0000","last_clean":"2024-01-16T15:13:10.167640+0000","last_became_active":"2024-01-16T15:13:10.164870+0000","last_became_peered":"2024-01-16T15:13:10.164870+0000","last_unstale":"2024-01-16T15:13:10.167640+0000","last_undegraded":"2024-01-16T15:13:10.167640+0000","last_fullsized":"2024-01-16T15:13:10.167640+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:58.566297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176317+0000","last_change":"2024-01-16T15:13:10.169151+0000","last_active":"2024-01-16T15:13:10.176317+0000","last_peered":"2024-01-16T15:13:10.176317+0000","last_clean":"2024-01-16T15:13:10.176317+0000","last_became_active":"2024-01-16T15:13:10.163706+0000","last_became_peered":"2024-01-16T15:13:10.163706+0000","last_unstale":"2024-01-16T15:13:10.176317+0000","last_undegraded":"2024-01-16T15:13:10.176317+0000","last_fullsized":"2024-01-16T15:13:10.176317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:19.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164989+0000","last_change":"2024-01-16T15:13:10.162953+0000","last_active":"2024-01-16T15:13:10.164989+0000","last_peered":"2024-01-16T15:13:10.164989+0000","last_clean":"2024-01-16T15:13:10.164989+0000","last_became_active":"2024-01-16T15:13:10.162668+0000","last_became_peered":"2024-01-16T15:13:10.162668+0000","last_unstale":"2024-01-16T15:13:10.164989+0000","last_undegraded":"2024-01-16T15:13:10.164989+0000","last_fullsized":"2024-01-16T15:13:10.164989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:28.363739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154417+0000","last_change":"2024-01-16T15:13:10.151668+0000","last_active":"2024-01-16T15:13:10.154417+0000","last_peered":"2024-01-16T15:13:10.154417+0000","last_clean":"2024-01-16T15:13:10.154417+0000","last_became_active":"2024-01-16T15:13:10.151338+0000","last_became_peered":"2024-01-16T15:13:10.151338+0000","last_unstale":"2024-01-16T15:13:10.154417+0000","last_undegraded":"2024-01-16T15:13:10.154417+0000","last_fullsized":"2024-01-16T15:13:10.154417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:02:47.062688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.181138+0000","last_change":"2024-01-16T15:13:10.179253+0000","last_active":"2024-01-16T15:13:10.181138+0000","last_peered":"2024-01-16T15:13:10.181138+0000","last_clean":"2024-01-16T15:13:10.181138+0000","last_became_active":"2024-01-16T15:13:10.178510+0000","last_became_peered":"2024-01-16T15:13:10.178510+0000","last_unstale":"2024-01-16T15:13:10.181138+0000","last_undegraded":"2024-01-16T15:13:10.181138+0000","last_fullsized":"2024-01-16T15:13:10.181138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:45:07.967072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168368+0000","last_change":"2024-01-16T15:13:10.165586+0000","last_active":"2024-01-16T15:13:10.168368+0000","last_peered":"2024-01-16T15:13:10.168368+0000","last_clean":"2024-01-16T15:13:10.168368+0000","last_became_active":"2024-01-16T15:13:10.165264+0000","last_became_peered":"2024-01-16T15:13:10.165264+0000","last_unstale":"2024-01-16T15:13:10.168368+0000","last_undegraded":"2024-01-16T15:13:10.168368+0000","last_fullsized":"2024-01-16T15:13:10.168368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:54.237403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150762+0000","last_change":"2024-01-16T15:13:10.148246+0000","last_active":"2024-01-16T15:13:10.150762+0000","last_peered":"2024-01-16T15:13:10.150762+0000","last_clean":"2024-01-16T15:13:10.150762+0000","last_became_active":"2024-01-16T15:13:10.147957+0000","last_became_peered":"2024-01-16T15:13:10.147957+0000","last_unstale":"2024-01-16T15:13:10.150762+0000","last_undegraded":"2024-01-16T15:13:10.150762+0000","last_fullsized":"2024-01-16T15:13:10.150762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:42.927055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150686+0000","last_change":"2024-01-16T15:13:10.145984+0000","last_active":"2024-01-16T15:13:10.150686+0000","last_peered":"2024-01-16T15:13:10.150686+0000","last_clean":"2024-01-16T15:13:10.150686+0000","last_became_active":"2024-01-16T15:13:10.145569+0000","last_became_peered":"2024-01-16T15:13:10.145569+0000","last_unstale":"2024-01-16T15:13:10.150686+0000","last_undegraded":"2024-01-16T15:13:10.150686+0000","last_fullsized":"2024-01-16T15:13:10.150686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:27.675509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154258+0000","last_change":"2024-01-16T15:13:10.147373+0000","last_active":"2024-01-16T15:13:10.154258+0000","last_peered":"2024-01-16T15:13:10.154258+0000","last_clean":"2024-01-16T15:13:10.154258+0000","last_became_active":"2024-01-16T15:13:10.145161+0000","last_became_peered":"2024-01-16T15:13:10.145161+0000","last_unstale":"2024-01-16T15:13:10.154258+0000","last_undegraded":"2024-01-16T15:13:10.154258+0000","last_fullsized":"2024-01-16T15:13:10.154258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:44:52.704937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176567+0000","last_change":"2024-01-16T15:13:10.165214+0000","last_active":"2024-01-16T15:13:10.176567+0000","last_peered":"2024-01-16T15:13:10.176567+0000","last_clean":"2024-01-16T15:13:10.176567+0000","last_became_active":"2024-01-16T15:13:10.164899+0000","last_became_peered":"2024-01-16T15:13:10.164899+0000","last_unstale":"2024-01-16T15:13:10.176567+0000","last_undegraded":"2024-01-16T15:13:10.176567+0000","last_fullsized":"2024-01-16T15:13:10.176567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:09.698637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.155005+0000","last_change":"2024-01-16T15:13:10.149948+0000","last_active":"2024-01-16T15:13:10.155005+0000","last_peered":"2024-01-16T15:13:10.155005+0000","last_clean":"2024-01-16T15:13:10.155005+0000","last_became_active":"2024-01-16T15:13:10.149671+0000","last_became_peered":"2024-01-16T15:13:10.149671+0000","last_unstale":"2024-01-16T15:13:10.155005+0000","last_undegraded":"2024-01-16T15:13:10.155005+0000","last_fullsized":"2024-01-16T15:13:10.155005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:00.540556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154311+0000","last_change":"2024-01-16T15:13:10.146925+0000","last_active":"2024-01-16T15:13:10.154311+0000","last_peered":"2024-01-16T15:13:10.154311+0000","last_clean":"2024-01-16T15:13:10.154311+0000","last_became_active":"2024-01-16T15:13:10.146575+0000","last_became_peered":"2024-01-16T15:13:10.146575+0000","last_unstale":"2024-01-16T15:13:10.154311+0000","last_undegraded":"2024-01-16T15:13:10.154311+0000","last_fullsized":"2024-01-16T15:13:10.154311+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:34.821462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176988+0000","last_change":"2024-01-16T15:13:10.170105+0000","last_active":"2024-01-16T15:13:10.176988+0000","last_peered":"2024-01-16T15:13:10.176988+0000","last_clean":"2024-01-16T15:13:10.176988+0000","last_became_active":"2024-01-16T15:13:10.169574+0000","last_became_peered":"2024-01-16T15:13:10.169574+0000","last_unstale":"2024-01-16T15:13:10.176988+0000","last_undegraded":"2024-01-16T15:13:10.176988+0000","last_fullsized":"2024-01-16T15:13:10.176988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:51.431529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158505+0000","last_change":"2024-01-16T15:13:10.155229+0000","last_active":"2024-01-16T15:13:10.158505+0000","last_peered":"2024-01-16T15:13:10.158505+0000","last_clean":"2024-01-16T15:13:10.158505+0000","last_became_active":"2024-01-16T15:13:10.154347+0000","last_became_peered":"2024-01-16T15:13:10.154347+0000","last_unstale":"2024-01-16T15:13:10.158505+0000","last_undegraded":"2024-01-16T15:13:10.158505+0000","last_fullsized":"2024-01-16T15:13:10.158505+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:55.691177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162766+0000","last_change":"2024-01-16T15:13:10.153721+0000","last_active":"2024-01-16T15:13:10.162766+0000","last_peered":"2024-01-16T15:13:10.162766+0000","last_clean":"2024-01-16T15:13:10.162766+0000","last_became_active":"2024-01-16T15:13:10.153430+0000","last_became_peered":"2024-01-16T15:13:10.153430+0000","last_unstale":"2024-01-16T15:13:10.162766+0000","last_undegraded":"2024-01-16T15:13:10.162766+0000","last_fullsized":"2024-01-16T15:13:10.162766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:18.037258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.173712+0000","last_change":"2024-01-16T15:13:10.166616+0000","last_active":"2024-01-16T15:13:10.173712+0000","last_peered":"2024-01-16T15:13:10.173712+0000","last_clean":"2024-01-16T15:13:10.173712+0000","last_became_active":"2024-01-16T15:13:10.166340+0000","last_became_peered":"2024-01-16T15:13:10.166340+0000","last_unstale":"2024-01-16T15:13:10.173712+0000","last_undegraded":"2024-01-16T15:13:10.173712+0000","last_fullsized":"2024-01-16T15:13:10.173712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:43.176639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170145+0000","last_change":"2024-01-16T15:13:10.167640+0000","last_active":"2024-01-16T15:13:10.170145+0000","last_peered":"2024-01-16T15:13:10.170145+0000","last_clean":"2024-01-16T15:13:10.170145+0000","last_became_active":"2024-01-16T15:13:10.166757+0000","last_became_peered":"2024-01-16T15:13:10.166757+0000","last_unstale":"2024-01-16T15:13:10.170145+0000","last_undegraded":"2024-01-16T15:13:10.170145+0000","last_fullsized":"2024-01-16T15:13:10.170145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:18.666895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164440+0000","last_change":"2024-01-16T15:13:10.163097+0000","last_active":"2024-01-16T15:13:10.164440+0000","last_peered":"2024-01-16T15:13:10.164440+0000","last_clean":"2024-01-16T15:13:10.164440+0000","last_became_active":"2024-01-16T15:13:10.162735+0000","last_became_peered":"2024-01-16T15:13:10.162735+0000","last_unstale":"2024-01-16T15:13:10.164440+0000","last_undegraded":"2024-01-16T15:13:10.164440+0000","last_fullsized":"2024-01-16T15:13:10.164440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:53.396086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156483+0000","last_change":"2024-01-16T15:13:10.153644+0000","last_active":"2024-01-16T15:13:10.156483+0000","last_peered":"2024-01-16T15:13:10.156483+0000","last_clean":"2024-01-16T15:13:10.156483+0000","last_became_active":"2024-01-16T15:13:10.153286+0000","last_became_peered":"2024-01-16T15:13:10.153286+0000","last_unstale":"2024-01-16T15:13:10.156483+0000","last_undegraded":"2024-01-16T15:13:10.156483+0000","last_fullsized":"2024-01-16T15:13:10.156483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:33.303213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.179561+0000","last_change":"2024-01-16T15:13:10.175640+0000","last_active":"2024-01-16T15:13:10.179561+0000","last_peered":"2024-01-16T15:13:10.179561+0000","last_clean":"2024-01-16T15:13:10.179561+0000","last_became_active":"2024-01-16T15:13:10.174512+0000","last_became_peered":"2024-01-16T15:13:10.174512+0000","last_unstale":"2024-01-16T15:13:10.179561+0000","last_undegraded":"2024-01-16T15:13:10.179561+0000","last_fullsized":"2024-01-16T15:13:10.179561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:56.824444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.178450+0000","last_change":"2024-01-16T15:13:10.174242+0000","last_active":"2024-01-16T15:13:10.178450+0000","last_peered":"2024-01-16T15:13:10.178450+0000","last_clean":"2024-01-16T15:13:10.178450+0000","last_became_active":"2024-01-16T15:13:10.169627+0000","last_became_peered":"2024-01-16T15:13:10.169627+0000","last_unstale":"2024-01-16T15:13:10.178450+0000","last_undegraded":"2024-01-16T15:13:10.178450+0000","last_fullsized":"2024-01-16T15:13:10.178450+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:51.074211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170866+0000","last_change":"2024-01-16T15:13:10.166842+0000","last_active":"2024-01-16T15:13:10.170866+0000","last_peered":"2024-01-16T15:13:10.170866+0000","last_clean":"2024-01-16T15:13:10.170866+0000","last_became_active":"2024-01-16T15:13:10.166048+0000","last_became_peered":"2024-01-16T15:13:10.166048+0000","last_unstale":"2024-01-16T15:13:10.170866+0000","last_undegraded":"2024-01-16T15:13:10.170866+0000","last_fullsized":"2024-01-16T15:13:10.170866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:50.752436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.176104+0000","last_change":"2024-01-16T15:13:10.175048+0000","last_active":"2024-01-16T15:13:10.176104+0000","last_peered":"2024-01-16T15:13:10.176104+0000","last_clean":"2024-01-16T15:13:10.176104+0000","last_became_active":"2024-01-16T15:13:10.174410+0000","last_became_peered":"2024-01-16T15:13:10.174410+0000","last_unstale":"2024-01-16T15:13:10.176104+0000","last_undegraded":"2024-01-16T15:13:10.176104+0000","last_fullsized":"2024-01-16T15:13:10.176104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:33.445285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156020+0000","last_change":"2024-01-16T15:13:10.154000+0000","last_active":"2024-01-16T15:13:10.156020+0000","last_peered":"2024-01-16T15:13:10.156020+0000","last_clean":"2024-01-16T15:13:10.156020+0000","last_became_active":"2024-01-16T15:13:10.153439+0000","last_became_peered":"2024-01-16T15:13:10.153439+0000","last_unstale":"2024-01-16T15:13:10.156020+0000","last_undegraded":"2024-01-16T15:13:10.156020+0000","last_fullsized":"2024-01-16T15:13:10.156020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:07.126646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"87'4","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.174915+0000","last_change":"2024-01-16T15:13:09.151686+0000","last_active":"2024-01-16T15:13:10.174915+0000","last_peered":"2024-01-16T15:13:10.174915+0000","last_clean":"2024-01-16T15:13:10.174915+0000","last_became_active":"2024-01-16T15:13:09.150673+0000","last_became_peered":"2024-01-16T15:13:09.150673+0000","last_unstale":"2024-01-16T15:13:10.174915+0000","last_undegraded":"2024-01-16T15:13:10.174915+0000","last_fullsized":"2024-01-16T15:13:10.174915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:08.190150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164670+0000","last_change":"2024-01-16T15:13:10.160577+0000","last_active":"2024-01-16T15:13:10.164670+0000","last_peered":"2024-01-16T15:13:10.164670+0000","last_clean":"2024-01-16T15:13:10.164670+0000","last_became_active":"2024-01-16T15:13:10.160238+0000","last_became_peered":"2024-01-16T15:13:10.160238+0000","last_unstale":"2024-01-16T15:13:10.164670+0000","last_undegraded":"2024-01-16T15:13:10.164670+0000","last_fullsized":"2024-01-16T15:13:10.164670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:53.320778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.172822+0000","last_change":"2024-01-16T15:13:09.151756+0000","last_active":"2024-01-16T15:13:10.172822+0000","last_peered":"2024-01-16T15:13:10.172822+0000","last_clean":"2024-01-16T15:13:10.172822+0000","last_became_active":"2024-01-16T15:13:09.150392+0000","last_became_peered":"2024-01-16T15:13:09.150392+0000","last_unstale":"2024-01-16T15:13:10.172822+0000","last_undegraded":"2024-01-16T15:13:10.172822+0000","last_fullsized":"2024-01-16T15:13:10.172822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:32.435368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175823+0000","last_change":"2024-01-16T15:13:10.174551+0000","last_active":"2024-01-16T15:13:10.175823+0000","last_peered":"2024-01-16T15:13:10.175823+0000","last_clean":"2024-01-16T15:13:10.175823+0000","last_became_active":"2024-01-16T15:13:10.174229+0000","last_became_peered":"2024-01-16T15:13:10.174229+0000","last_unstale":"2024-01-16T15:13:10.175823+0000","last_undegraded":"2024-01-16T15:13:10.175823+0000","last_fullsized":"2024-01-16T15:13:10.175823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:17.170146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182663+0000","last_change":"2024-01-16T15:13:09.147173+0000","last_active":"2024-01-16T15:13:10.182663+0000","last_peered":"2024-01-16T15:13:10.182663+0000","last_clean":"2024-01-16T15:13:10.182663+0000","last_became_active":"2024-01-16T15:13:09.146895+0000","last_became_peered":"2024-01-16T15:13:09.146895+0000","last_unstale":"2024-01-16T15:13:10.182663+0000","last_undegraded":"2024-01-16T15:13:10.182663+0000","last_fullsized":"2024-01-16T15:13:10.182663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:28.989720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158249+0000","last_change":"2024-01-16T15:13:10.157632+0000","last_active":"2024-01-16T15:13:10.158249+0000","last_peered":"2024-01-16T15:13:10.158249+0000","last_clean":"2024-01-16T15:13:10.158249+0000","last_became_active":"2024-01-16T15:13:10.157269+0000","last_became_peered":"2024-01-16T15:13:10.157269+0000","last_unstale":"2024-01-16T15:13:10.158249+0000","last_undegraded":"2024-01-16T15:13:10.158249+0000","last_fullsized":"2024-01-16T15:13:10.158249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:55.704133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182277+0000","last_change":"2024-01-16T15:13:09.140816+0000","last_active":"2024-01-16T15:13:10.182277+0000","last_peered":"2024-01-16T15:13:10.182277+0000","last_clean":"2024-01-16T15:13:10.182277+0000","last_became_active":"2024-01-16T15:13:09.140552+0000","last_became_peered":"2024-01-16T15:13:09.140552+0000","last_unstale":"2024-01-16T15:13:10.182277+0000","last_undegraded":"2024-01-16T15:13:10.182277+0000","last_fullsized":"2024-01-16T15:13:10.182277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:33:40.673100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.169816+0000","last_change":"2024-01-16T15:13:10.167193+0000","last_active":"2024-01-16T15:13:10.169816+0000","last_peered":"2024-01-16T15:13:10.169816+0000","last_clean":"2024-01-16T15:13:10.169816+0000","last_became_active":"2024-01-16T15:13:10.163090+0000","last_became_peered":"2024-01-16T15:13:10.163090+0000","last_unstale":"2024-01-16T15:13:10.169816+0000","last_undegraded":"2024-01-16T15:13:10.169816+0000","last_fullsized":"2024-01-16T15:13:10.169816+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:43.267405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180200+0000","last_change":"2024-01-16T15:13:09.148297+0000","last_active":"2024-01-16T15:13:10.180200+0000","last_peered":"2024-01-16T15:13:10.180200+0000","last_clean":"2024-01-16T15:13:10.180200+0000","last_became_active":"2024-01-16T15:13:09.147302+0000","last_became_peered":"2024-01-16T15:13:09.147302+0000","last_unstale":"2024-01-16T15:13:10.180200+0000","last_undegraded":"2024-01-16T15:13:10.180200+0000","last_fullsized":"2024-01-16T15:13:10.180200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:57.556380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.161949+0000","last_change":"2024-01-16T15:13:10.155895+0000","last_active":"2024-01-16T15:13:10.161949+0000","last_peered":"2024-01-16T15:13:10.161949+0000","last_clean":"2024-01-16T15:13:10.161949+0000","last_became_active":"2024-01-16T15:13:10.155529+0000","last_became_peered":"2024-01-16T15:13:10.155529+0000","last_unstale":"2024-01-16T15:13:10.161949+0000","last_undegraded":"2024-01-16T15:13:10.161949+0000","last_fullsized":"2024-01-16T15:13:10.161949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:19.057854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.172812+0000","last_change":"2024-01-16T15:13:09.150066+0000","last_active":"2024-01-16T15:13:10.172812+0000","last_peered":"2024-01-16T15:13:10.172812+0000","last_clean":"2024-01-16T15:13:10.172812+0000","last_became_active":"2024-01-16T15:13:09.149367+0000","last_became_peered":"2024-01-16T15:13:09.149367+0000","last_unstale":"2024-01-16T15:13:10.172812+0000","last_undegraded":"2024-01-16T15:13:10.172812+0000","last_fullsized":"2024-01-16T15:13:10.172812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:57.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170986+0000","last_change":"2024-01-16T15:13:10.168840+0000","last_active":"2024-01-16T15:13:10.170986+0000","last_peered":"2024-01-16T15:13:10.170986+0000","last_clean":"2024-01-16T15:13:10.170986+0000","last_became_active":"2024-01-16T15:13:10.168554+0000","last_became_peered":"2024-01-16T15:13:10.168554+0000","last_unstale":"2024-01-16T15:13:10.170986+0000","last_undegraded":"2024-01-16T15:13:10.170986+0000","last_fullsized":"2024-01-16T15:13:10.170986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:39.593178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.150518+0000","last_change":"2024-01-16T15:13:09.145669+0000","last_active":"2024-01-16T15:13:10.150518+0000","last_peered":"2024-01-16T15:13:10.150518+0000","last_clean":"2024-01-16T15:13:10.150518+0000","last_became_active":"2024-01-16T15:13:09.145356+0000","last_became_peered":"2024-01-16T15:13:09.145356+0000","last_unstale":"2024-01-16T15:13:10.150518+0000","last_undegraded":"2024-01-16T15:13:10.150518+0000","last_fullsized":"2024-01-16T15:13:10.150518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:08.190150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168350+0000","last_change":"2024-01-16T15:13:10.165520+0000","last_active":"2024-01-16T15:13:10.168350+0000","last_peered":"2024-01-16T15:13:10.168350+0000","last_clean":"2024-01-16T15:13:10.168350+0000","last_became_active":"2024-01-16T15:13:10.165242+0000","last_became_peered":"2024-01-16T15:13:10.165242+0000","last_unstale":"2024-01-16T15:13:10.168350+0000","last_undegraded":"2024-01-16T15:13:10.168350+0000","last_fullsized":"2024-01-16T15:13:10.168350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:55.704133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.160290+0000","last_change":"2024-01-16T15:13:09.146887+0000","last_active":"2024-01-16T15:13:10.160290+0000","last_peered":"2024-01-16T15:13:10.160290+0000","last_clean":"2024-01-16T15:13:10.160290+0000","last_became_active":"2024-01-16T15:13:09.146588+0000","last_became_peered":"2024-01-16T15:13:09.146588+0000","last_unstale":"2024-01-16T15:13:10.160290+0000","last_undegraded":"2024-01-16T15:13:10.160290+0000","last_fullsized":"2024-01-16T15:13:10.160290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:44.486393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.179224+0000","last_change":"2024-01-16T15:13:10.174999+0000","last_active":"2024-01-16T15:13:10.179224+0000","last_peered":"2024-01-16T15:13:10.179224+0000","last_clean":"2024-01-16T15:13:10.179224+0000","last_became_active":"2024-01-16T15:13:10.174699+0000","last_became_peered":"2024-01-16T15:13:10.174699+0000","last_unstale":"2024-01-16T15:13:10.179224+0000","last_undegraded":"2024-01-16T15:13:10.179224+0000","last_fullsized":"2024-01-16T15:13:10.179224+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:02.049659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.140960+0000","last_change":"2024-01-16T15:13:09.152842+0000","last_active":"2024-01-16T15:13:10.140960+0000","last_peered":"2024-01-16T15:13:10.140960+0000","last_clean":"2024-01-16T15:13:10.140960+0000","last_became_active":"2024-01-16T15:13:09.152384+0000","last_became_peered":"2024-01-16T15:13:09.152384+0000","last_unstale":"2024-01-16T15:13:10.140960+0000","last_undegraded":"2024-01-16T15:13:10.140960+0000","last_fullsized":"2024-01-16T15:13:10.140960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:08.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158438+0000","last_change":"2024-01-16T15:13:10.155407+0000","last_active":"2024-01-16T15:13:10.158438+0000","last_peered":"2024-01-16T15:13:10.158438+0000","last_clean":"2024-01-16T15:13:10.158438+0000","last_became_active":"2024-01-16T15:13:10.155124+0000","last_became_peered":"2024-01-16T15:13:10.155124+0000","last_unstale":"2024-01-16T15:13:10.158438+0000","last_undegraded":"2024-01-16T15:13:10.158438+0000","last_fullsized":"2024-01-16T15:13:10.158438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:33.445285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.136568+0000","last_change":"2024-01-16T15:13:09.151431+0000","last_active":"2024-01-16T15:13:10.136568+0000","last_peered":"2024-01-16T15:13:10.136568+0000","last_clean":"2024-01-16T15:13:10.136568+0000","last_became_active":"2024-01-16T15:13:09.151009+0000","last_became_peered":"2024-01-16T15:13:09.151009+0000","last_unstale":"2024-01-16T15:13:10.136568+0000","last_undegraded":"2024-01-16T15:13:10.136568+0000","last_fullsized":"2024-01-16T15:13:10.136568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:14.148344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162208+0000","last_change":"2024-01-16T15:13:10.157277+0000","last_active":"2024-01-16T15:13:10.162208+0000","last_peered":"2024-01-16T15:13:10.162208+0000","last_clean":"2024-01-16T15:13:10.162208+0000","last_became_active":"2024-01-16T15:13:10.156819+0000","last_became_peered":"2024-01-16T15:13:10.156819+0000","last_unstale":"2024-01-16T15:13:10.162208+0000","last_undegraded":"2024-01-16T15:13:10.162208+0000","last_fullsized":"2024-01-16T15:13:10.162208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:53.320778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"87'3","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180544+0000","last_change":"2024-01-16T15:13:09.152324+0000","last_active":"2024-01-16T15:13:10.180544+0000","last_peered":"2024-01-16T15:13:10.180544+0000","last_clean":"2024-01-16T15:13:10.180544+0000","last_became_active":"2024-01-16T15:13:09.150771+0000","last_became_peered":"2024-01-16T15:13:09.150771+0000","last_unstale":"2024-01-16T15:13:10.180544+0000","last_undegraded":"2024-01-16T15:13:10.180544+0000","last_fullsized":"2024-01-16T15:13:10.180544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:22:16.244236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170997+0000","last_change":"2024-01-16T15:13:10.166424+0000","last_active":"2024-01-16T15:13:10.170997+0000","last_peered":"2024-01-16T15:13:10.170997+0000","last_clean":"2024-01-16T15:13:10.170997+0000","last_became_active":"2024-01-16T15:13:10.166103+0000","last_became_peered":"2024-01-16T15:13:10.166103+0000","last_unstale":"2024-01-16T15:13:10.170997+0000","last_undegraded":"2024-01-16T15:13:10.170997+0000","last_fullsized":"2024-01-16T15:13:10.170997+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:14.228009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"87'6","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.184317+0000","last_change":"2024-01-16T15:13:09.152546+0000","last_active":"2024-01-16T15:13:10.184317+0000","last_peered":"2024-01-16T15:13:10.184317+0000","last_clean":"2024-01-16T15:13:10.184317+0000","last_became_active":"2024-01-16T15:13:09.152268+0000","last_became_peered":"2024-01-16T15:13:09.152268+0000","last_unstale":"2024-01-16T15:13:10.184317+0000","last_undegraded":"2024-01-16T15:13:10.184317+0000","last_fullsized":"2024-01-16T15:13:10.184317+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:42.077697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162634+0000","last_change":"2024-01-16T15:13:10.159408+0000","last_active":"2024-01-16T15:13:10.162634+0000","last_peered":"2024-01-16T15:13:10.162634+0000","last_clean":"2024-01-16T15:13:10.162634+0000","last_became_active":"2024-01-16T15:13:10.154989+0000","last_became_peered":"2024-01-16T15:13:10.154989+0000","last_unstale":"2024-01-16T15:13:10.162634+0000","last_undegraded":"2024-01-16T15:13:10.162634+0000","last_fullsized":"2024-01-16T15:13:10.162634+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:43:07.856847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.184583+0000","last_change":"2024-01-16T15:13:09.142988+0000","last_active":"2024-01-16T15:13:10.184583+0000","last_peered":"2024-01-16T15:13:10.184583+0000","last_clean":"2024-01-16T15:13:10.184583+0000","last_became_active":"2024-01-16T15:13:09.139882+0000","last_became_peered":"2024-01-16T15:13:09.139882+0000","last_unstale":"2024-01-16T15:13:10.184583+0000","last_undegraded":"2024-01-16T15:13:10.184583+0000","last_fullsized":"2024-01-16T15:13:10.184583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:33.048007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171037+0000","last_change":"2024-01-16T15:13:10.169557+0000","last_active":"2024-01-16T15:13:10.171037+0000","last_peered":"2024-01-16T15:13:10.171037+0000","last_clean":"2024-01-16T15:13:10.171037+0000","last_became_active":"2024-01-16T15:13:10.169265+0000","last_became_peered":"2024-01-16T15:13:10.169265+0000","last_unstale":"2024-01-16T15:13:10.171037+0000","last_undegraded":"2024-01-16T15:13:10.171037+0000","last_fullsized":"2024-01-16T15:13:10.171037+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:34:11.855352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158436+0000","last_change":"2024-01-16T15:13:09.147378+0000","last_active":"2024-01-16T15:13:10.158436+0000","last_peered":"2024-01-16T15:13:10.158436+0000","last_clean":"2024-01-16T15:13:10.158436+0000","last_became_active":"2024-01-16T15:13:09.147101+0000","last_became_peered":"2024-01-16T15:13:09.147101+0000","last_unstale":"2024-01-16T15:13:10.158436+0000","last_undegraded":"2024-01-16T15:13:10.158436+0000","last_fullsized":"2024-01-16T15:13:10.158436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:36:19.238253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.167897+0000","last_change":"2024-01-16T15:13:10.160889+0000","last_active":"2024-01-16T15:13:10.167897+0000","last_peered":"2024-01-16T15:13:10.167897+0000","last_clean":"2024-01-16T15:13:10.167897+0000","last_became_active":"2024-01-16T15:13:10.160613+0000","last_became_peered":"2024-01-16T15:13:10.160613+0000","last_unstale":"2024-01-16T15:13:10.167897+0000","last_undegraded":"2024-01-16T15:13:10.167897+0000","last_fullsized":"2024-01-16T15:13:10.167897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:27.704754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.144379+0000","last_change":"2024-01-16T15:13:09.147474+0000","last_active":"2024-01-16T15:13:10.144379+0000","last_peered":"2024-01-16T15:13:10.144379+0000","last_clean":"2024-01-16T15:13:10.144379+0000","last_became_active":"2024-01-16T15:13:09.147126+0000","last_became_peered":"2024-01-16T15:13:09.147126+0000","last_unstale":"2024-01-16T15:13:10.144379+0000","last_undegraded":"2024-01-16T15:13:10.144379+0000","last_fullsized":"2024-01-16T15:13:10.144379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:57.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175383+0000","last_change":"2024-01-16T15:13:10.165981+0000","last_active":"2024-01-16T15:13:10.175383+0000","last_peered":"2024-01-16T15:13:10.175383+0000","last_clean":"2024-01-16T15:13:10.175383+0000","last_became_active":"2024-01-16T15:13:10.165711+0000","last_became_peered":"2024-01-16T15:13:10.165711+0000","last_unstale":"2024-01-16T15:13:10.175383+0000","last_undegraded":"2024-01-16T15:13:10.175383+0000","last_fullsized":"2024-01-16T15:13:10.175383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:29.690714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180929+0000","last_change":"2024-01-16T15:13:09.142813+0000","last_active":"2024-01-16T15:13:10.180929+0000","last_peered":"2024-01-16T15:13:10.180929+0000","last_clean":"2024-01-16T15:13:10.180929+0000","last_became_active":"2024-01-16T15:13:09.142527+0000","last_became_peered":"2024-01-16T15:13:09.142527+0000","last_unstale":"2024-01-16T15:13:10.180929+0000","last_undegraded":"2024-01-16T15:13:10.180929+0000","last_fullsized":"2024-01-16T15:13:10.180929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:43:03.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156877+0000","last_change":"2024-01-16T15:13:10.154560+0000","last_active":"2024-01-16T15:13:10.156877+0000","last_peered":"2024-01-16T15:13:10.156877+0000","last_clean":"2024-01-16T15:13:10.156877+0000","last_became_active":"2024-01-16T15:13:10.154256+0000","last_became_peered":"2024-01-16T15:13:10.154256+0000","last_unstale":"2024-01-16T15:13:10.156877+0000","last_undegraded":"2024-01-16T15:13:10.156877+0000","last_fullsized":"2024-01-16T15:13:10.156877+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:29.484144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171893+0000","last_change":"2024-01-16T15:13:09.150981+0000","last_active":"2024-01-16T15:13:10.171893+0000","last_peered":"2024-01-16T15:13:10.171893+0000","last_clean":"2024-01-16T15:13:10.171893+0000","last_became_active":"2024-01-16T15:13:09.149150+0000","last_became_peered":"2024-01-16T15:13:09.149150+0000","last_unstale":"2024-01-16T15:13:10.171893+0000","last_undegraded":"2024-01-16T15:13:10.171893+0000","last_fullsized":"2024-01-16T15:13:10.171893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:43:06.846930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.156792+0000","last_change":"2024-01-16T15:13:10.153848+0000","last_active":"2024-01-16T15:13:10.156792+0000","last_peered":"2024-01-16T15:13:10.156792+0000","last_clean":"2024-01-16T15:13:10.156792+0000","last_became_active":"2024-01-16T15:13:10.153019+0000","last_became_peered":"2024-01-16T15:13:10.153019+0000","last_unstale":"2024-01-16T15:13:10.156792+0000","last_undegraded":"2024-01-16T15:13:10.156792+0000","last_fullsized":"2024-01-16T15:13:10.156792+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:43.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.178490+0000","last_change":"2024-01-16T15:13:09.146055+0000","last_active":"2024-01-16T15:13:10.178490+0000","last_peered":"2024-01-16T15:13:10.178490+0000","last_clean":"2024-01-16T15:13:10.178490+0000","last_became_active":"2024-01-16T15:13:09.145773+0000","last_became_peered":"2024-01-16T15:13:09.145773+0000","last_unstale":"2024-01-16T15:13:10.178490+0000","last_undegraded":"2024-01-16T15:13:10.178490+0000","last_fullsized":"2024-01-16T15:13:10.178490+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:59.247724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.175948+0000","last_change":"2024-01-16T15:13:10.174037+0000","last_active":"2024-01-16T15:13:10.175948+0000","last_peered":"2024-01-16T15:13:10.175948+0000","last_clean":"2024-01-16T15:13:10.175948+0000","last_became_active":"2024-01-16T15:13:10.173730+0000","last_became_peered":"2024-01-16T15:13:10.173730+0000","last_unstale":"2024-01-16T15:13:10.175948+0000","last_undegraded":"2024-01-16T15:13:10.175948+0000","last_fullsized":"2024-01-16T15:13:10.175948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:52:15.840801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.151254+0000","last_change":"2024-01-16T15:13:09.147750+0000","last_active":"2024-01-16T15:13:10.151254+0000","last_peered":"2024-01-16T15:13:10.151254+0000","last_clean":"2024-01-16T15:13:10.151254+0000","last_became_active":"2024-01-16T15:13:09.147460+0000","last_became_peered":"2024-01-16T15:13:09.147460+0000","last_unstale":"2024-01-16T15:13:10.151254+0000","last_undegraded":"2024-01-16T15:13:10.151254+0000","last_fullsized":"2024-01-16T15:13:10.151254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:10.689821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.181119+0000","last_change":"2024-01-16T15:13:10.179121+0000","last_active":"2024-01-16T15:13:10.181119+0000","last_peered":"2024-01-16T15:13:10.181119+0000","last_clean":"2024-01-16T15:13:10.181119+0000","last_became_active":"2024-01-16T15:13:10.177505+0000","last_became_peered":"2024-01-16T15:13:10.177505+0000","last_unstale":"2024-01-16T15:13:10.181119+0000","last_undegraded":"2024-01-16T15:13:10.181119+0000","last_fullsized":"2024-01-16T15:13:10.181119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:55:07.488113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.180396+0000","last_change":"2024-01-16T15:13:09.146726+0000","last_active":"2024-01-16T15:13:10.180396+0000","last_peered":"2024-01-16T15:13:10.180396+0000","last_clean":"2024-01-16T15:13:10.180396+0000","last_became_active":"2024-01-16T15:13:09.146300+0000","last_became_peered":"2024-01-16T15:13:09.146300+0000","last_unstale":"2024-01-16T15:13:10.180396+0000","last_undegraded":"2024-01-16T15:13:10.180396+0000","last_fullsized":"2024-01-16T15:13:10.180396+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:09.955603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171072+0000","last_change":"2024-01-16T15:13:10.169454+0000","last_active":"2024-01-16T15:13:10.171072+0000","last_peered":"2024-01-16T15:13:10.171072+0000","last_clean":"2024-01-16T15:13:10.171072+0000","last_became_active":"2024-01-16T15:13:10.168762+0000","last_became_peered":"2024-01-16T15:13:10.168762+0000","last_unstale":"2024-01-16T15:13:10.171072+0000","last_undegraded":"2024-01-16T15:13:10.171072+0000","last_fullsized":"2024-01-16T15:13:10.171072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.661926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.129678+0000","last_change":"2024-01-16T15:13:09.140730+0000","last_active":"2024-01-16T15:13:10.129678+0000","last_peered":"2024-01-16T15:13:10.129678+0000","last_clean":"2024-01-16T15:13:10.129678+0000","last_became_active":"2024-01-16T15:13:09.139479+0000","last_became_peered":"2024-01-16T15:13:09.139479+0000","last_unstale":"2024-01-16T15:13:10.129678+0000","last_undegraded":"2024-01-16T15:13:10.129678+0000","last_fullsized":"2024-01-16T15:13:10.129678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:12.555688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154880+0000","last_change":"2024-01-16T15:13:10.148419+0000","last_active":"2024-01-16T15:13:10.154880+0000","last_peered":"2024-01-16T15:13:10.154880+0000","last_clean":"2024-01-16T15:13:10.154880+0000","last_became_active":"2024-01-16T15:13:10.148048+0000","last_became_peered":"2024-01-16T15:13:10.148048+0000","last_unstale":"2024-01-16T15:13:10.154880+0000","last_undegraded":"2024-01-16T15:13:10.154880+0000","last_fullsized":"2024-01-16T15:13:10.154880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:45.625292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.128551+0000","last_change":"2024-01-16T15:13:09.149260+0000","last_active":"2024-01-16T15:13:10.128551+0000","last_peered":"2024-01-16T15:13:10.128551+0000","last_clean":"2024-01-16T15:13:10.128551+0000","last_became_active":"2024-01-16T15:13:09.147795+0000","last_became_peered":"2024-01-16T15:13:09.147795+0000","last_unstale":"2024-01-16T15:13:10.128551+0000","last_undegraded":"2024-01-16T15:13:10.128551+0000","last_fullsized":"2024-01-16T15:13:10.128551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:57.248367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164385+0000","last_change":"2024-01-16T15:13:10.161847+0000","last_active":"2024-01-16T15:13:10.164385+0000","last_peered":"2024-01-16T15:13:10.164385+0000","last_clean":"2024-01-16T15:13:10.164385+0000","last_became_active":"2024-01-16T15:13:10.161523+0000","last_became_peered":"2024-01-16T15:13:10.161523+0000","last_unstale":"2024-01-16T15:13:10.164385+0000","last_undegraded":"2024-01-16T15:13:10.164385+0000","last_fullsized":"2024-01-16T15:13:10.164385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:49.618603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.158270+0000","last_change":"2024-01-16T15:13:09.143825+0000","last_active":"2024-01-16T15:13:10.158270+0000","last_peered":"2024-01-16T15:13:10.158270+0000","last_clean":"2024-01-16T15:13:10.158270+0000","last_became_active":"2024-01-16T15:13:09.143491+0000","last_became_peered":"2024-01-16T15:13:09.143491+0000","last_unstale":"2024-01-16T15:13:10.158270+0000","last_undegraded":"2024-01-16T15:13:10.158270+0000","last_fullsized":"2024-01-16T15:13:10.158270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:28.474227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.152836+0000","last_change":"2024-01-16T15:13:10.149842+0000","last_active":"2024-01-16T15:13:10.152836+0000","last_peered":"2024-01-16T15:13:10.152836+0000","last_clean":"2024-01-16T15:13:10.152836+0000","last_became_active":"2024-01-16T15:13:10.149557+0000","last_became_peered":"2024-01-16T15:13:10.149557+0000","last_unstale":"2024-01-16T15:13:10.152836+0000","last_undegraded":"2024-01-16T15:13:10.152836+0000","last_fullsized":"2024-01-16T15:13:10.152836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:39.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"87'2","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.182858+0000","last_change":"2024-01-16T15:13:09.146510+0000","last_active":"2024-01-16T15:13:10.182858+0000","last_peered":"2024-01-16T15:13:10.182858+0000","last_clean":"2024-01-16T15:13:10.182858+0000","last_became_active":"2024-01-16T15:13:09.143347+0000","last_became_peered":"2024-01-16T15:13:09.143347+0000","last_unstale":"2024-01-16T15:13:10.182858+0000","last_undegraded":"2024-01-16T15:13:10.182858+0000","last_fullsized":"2024-01-16T15:13:10.182858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:23.970257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.159710+0000","last_change":"2024-01-16T15:13:10.158291+0000","last_active":"2024-01-16T15:13:10.159710+0000","last_peered":"2024-01-16T15:13:10.159710+0000","last_clean":"2024-01-16T15:13:10.159710+0000","last_became_active":"2024-01-16T15:13:10.157922+0000","last_became_peered":"2024-01-16T15:13:10.157922+0000","last_unstale":"2024-01-16T15:13:10.159710+0000","last_undegraded":"2024-01-16T15:13:10.159710+0000","last_fullsized":"2024-01-16T15:13:10.159710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:02.049659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.162946+0000","last_change":"2024-01-16T15:13:09.139809+0000","last_active":"2024-01-16T15:13:10.162946+0000","last_peered":"2024-01-16T15:13:10.162946+0000","last_clean":"2024-01-16T15:13:10.162946+0000","last_became_active":"2024-01-16T15:13:09.139512+0000","last_became_peered":"2024-01-16T15:13:09.139512+0000","last_unstale":"2024-01-16T15:13:10.162946+0000","last_undegraded":"2024-01-16T15:13:10.162946+0000","last_fullsized":"2024-01-16T15:13:10.162946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:40.199746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.155149+0000","last_change":"2024-01-16T15:13:10.151299+0000","last_active":"2024-01-16T15:13:10.155149+0000","last_peered":"2024-01-16T15:13:10.155149+0000","last_clean":"2024-01-16T15:13:10.155149+0000","last_became_active":"2024-01-16T15:13:10.142350+0000","last_became_peered":"2024-01-16T15:13:10.142350+0000","last_unstale":"2024-01-16T15:13:10.155149+0000","last_undegraded":"2024-01-16T15:13:10.155149+0000","last_fullsized":"2024-01-16T15:13:10.155149+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:58.554996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.135771+0000","last_change":"2024-01-16T15:13:09.153583+0000","last_active":"2024-01-16T15:13:10.135771+0000","last_peered":"2024-01-16T15:13:10.135771+0000","last_clean":"2024-01-16T15:13:10.135771+0000","last_became_active":"2024-01-16T15:13:09.153175+0000","last_became_peered":"2024-01-16T15:13:09.153175+0000","last_unstale":"2024-01-16T15:13:10.135771+0000","last_undegraded":"2024-01-16T15:13:10.135771+0000","last_fullsized":"2024-01-16T15:13:10.135771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:37.011904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.170136+0000","last_change":"2024-01-16T15:13:10.166069+0000","last_active":"2024-01-16T15:13:10.170136+0000","last_peered":"2024-01-16T15:13:10.170136+0000","last_clean":"2024-01-16T15:13:10.170136+0000","last_became_active":"2024-01-16T15:13:10.165744+0000","last_became_peered":"2024-01-16T15:13:10.165744+0000","last_unstale":"2024-01-16T15:13:10.170136+0000","last_undegraded":"2024-01-16T15:13:10.170136+0000","last_fullsized":"2024-01-16T15:13:10.170136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:29.900475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.138906+0000","last_change":"2024-01-16T15:13:09.152498+0000","last_active":"2024-01-16T15:13:10.138906+0000","last_peered":"2024-01-16T15:13:10.138906+0000","last_clean":"2024-01-16T15:13:10.138906+0000","last_became_active":"2024-01-16T15:13:09.152008+0000","last_became_peered":"2024-01-16T15:13:09.152008+0000","last_unstale":"2024-01-16T15:13:10.138906+0000","last_undegraded":"2024-01-16T15:13:10.138906+0000","last_fullsized":"2024-01-16T15:13:10.138906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:34.738173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.164733+0000","last_change":"2024-01-16T15:13:10.163069+0000","last_active":"2024-01-16T15:13:10.164733+0000","last_peered":"2024-01-16T15:13:10.164733+0000","last_clean":"2024-01-16T15:13:10.164733+0000","last_became_active":"2024-01-16T15:13:10.162720+0000","last_became_peered":"2024-01-16T15:13:10.162720+0000","last_unstale":"2024-01-16T15:13:10.164733+0000","last_undegraded":"2024-01-16T15:13:10.164733+0000","last_fullsized":"2024-01-16T15:13:10.164733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:39.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.154513+0000","last_change":"2024-01-16T15:13:09.142520+0000","last_active":"2024-01-16T15:13:10.154513+0000","last_peered":"2024-01-16T15:13:10.154513+0000","last_clean":"2024-01-16T15:13:10.154513+0000","last_became_active":"2024-01-16T15:13:09.142231+0000","last_became_peered":"2024-01-16T15:13:09.142231+0000","last_unstale":"2024-01-16T15:13:10.154513+0000","last_undegraded":"2024-01-16T15:13:10.154513+0000","last_fullsized":"2024-01-16T15:13:10.154513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:38.946708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.166479+0000","last_change":"2024-01-16T15:13:10.155976+0000","last_active":"2024-01-16T15:13:10.166479+0000","last_peered":"2024-01-16T15:13:10.166479+0000","last_clean":"2024-01-16T15:13:10.166479+0000","last_became_active":"2024-01-16T15:13:10.155649+0000","last_became_peered":"2024-01-16T15:13:10.155649+0000","last_unstale":"2024-01-16T15:13:10.166479+0000","last_undegraded":"2024-01-16T15:13:10.166479+0000","last_fullsized":"2024-01-16T15:13:10.166479+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:35.748090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.138828+0000","last_change":"2024-01-16T15:13:09.149896+0000","last_active":"2024-01-16T15:13:10.138828+0000","last_peered":"2024-01-16T15:13:10.138828+0000","last_clean":"2024-01-16T15:13:10.138828+0000","last_became_active":"2024-01-16T15:13:09.148929+0000","last_became_peered":"2024-01-16T15:13:09.148929+0000","last_unstale":"2024-01-16T15:13:10.138828+0000","last_undegraded":"2024-01-16T15:13:10.138828+0000","last_fullsized":"2024-01-16T15:13:10.138828+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:42.077697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.171005+0000","last_change":"2024-01-16T15:13:10.169837+0000","last_active":"2024-01-16T15:13:10.171005+0000","last_peered":"2024-01-16T15:13:10.171005+0000","last_clean":"2024-01-16T15:13:10.171005+0000","last_became_active":"2024-01-16T15:13:10.169049+0000","last_became_peered":"2024-01-16T15:13:10.169049+0000","last_unstale":"2024-01-16T15:13:10.171005+0000","last_undegraded":"2024-01-16T15:13:10.171005+0000","last_fullsized":"2024-01-16T15:13:10.171005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:58.374204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.136554+0000","last_change":"2024-01-16T15:13:09.153140+0000","last_active":"2024-01-16T15:13:10.136554+0000","last_peered":"2024-01-16T15:13:10.136554+0000","last_clean":"2024-01-16T15:13:10.136554+0000","last_became_active":"2024-01-16T15:13:09.151786+0000","last_became_peered":"2024-01-16T15:13:09.151786+0000","last_unstale":"2024-01-16T15:13:10.136554+0000","last_undegraded":"2024-01-16T15:13:10.136554+0000","last_fullsized":"2024-01-16T15:13:10.136554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:58.217563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.152530+0000","last_change":"2024-01-16T15:13:10.140971+0000","last_active":"2024-01-16T15:13:10.152530+0000","last_peered":"2024-01-16T15:13:10.152530+0000","last_clean":"2024-01-16T15:13:10.152530+0000","last_became_active":"2024-01-16T15:13:10.139718+0000","last_became_peered":"2024-01-16T15:13:10.139718+0000","last_unstale":"2024-01-16T15:13:10.152530+0000","last_undegraded":"2024-01-16T15:13:10.152530+0000","last_fullsized":"2024-01-16T15:13:10.152530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:49.618603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.137527+0000","last_change":"2024-01-16T15:13:09.149662+0000","last_active":"2024-01-16T15:13:10.137527+0000","last_peered":"2024-01-16T15:13:10.137527+0000","last_clean":"2024-01-16T15:13:10.137527+0000","last_became_active":"2024-01-16T15:13:09.147255+0000","last_became_peered":"2024-01-16T15:13:09.147255+0000","last_unstale":"2024-01-16T15:13:10.137527+0000","last_undegraded":"2024-01-16T15:13:10.137527+0000","last_fullsized":"2024-01-16T15:13:10.137527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:48.608686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.168426+0000","last_change":"2024-01-16T15:13:10.160037+0000","last_active":"2024-01-16T15:13:10.168426+0000","last_peered":"2024-01-16T15:13:10.168426+0000","last_clean":"2024-01-16T15:13:10.168426+0000","last_became_active":"2024-01-16T15:13:10.159336+0000","last_became_peered":"2024-01-16T15:13:10.159336+0000","last_unstale":"2024-01-16T15:13:10.168426+0000","last_undegraded":"2024-01-16T15:13:10.168426+0000","last_fullsized":"2024-01-16T15:13:10.168426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:09.100885+0000","last_clean_scrub_stamp":"2024-01-16T15:13:09.100885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:01:06.237086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"87'1","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-16T15:13:10.177383+0000","last_change":"2024-01-16T15:13:09.152363+0000","last_active":"2024-01-16T15:13:10.177383+0000","last_peered":"2024-01-16T15:13:10.177383+0000","last_clean":"2024-01-16T15:13:10.177383+0000","last_became_active":"2024-01-16T15:13:09.152050+0000","last_became_peered":"2024-01-16T15:13:09.152050+0000","last_unstale":"2024-01-16T15:13:10.177383+0000","last_undegraded":"2024-01-16T15:13:10.177383+0000","last_fullsized":"2024-01-16T15:13:10.177383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:08.090968+0000","last_clean_scrub_stamp":"2024-01-16T15:13:08.090968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:53.227486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":223,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:09.142854+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:13:09.142854+0000","last_peered":"2024-01-16T15:13:09.142854+0000","last_clean":"2024-01-16T15:13:09.142854+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:13:09.142854+0000","last_undegraded":"2024-01-16T15:13:09.142854+0000","last_fullsized":"2024-01-16T15:13:09.142854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68844,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302996,"statfs":{"total":96636764160,"available":96566267904,"internally_reserved":0,"allocated":1662976,"data_stored":1046287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.8160000000000001,"15min":1.4870000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":4.1020000000000003,"5min":27.469000000000001,"15min":27.469000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.4740000000000002,"5min":2.0720000000000001,"15min":1.625},"min":{"1min":0.69399999999999995,"5min":0.60299999999999998,"15min":0.47099999999999997},"max":{"1min":25.503,"5min":25.503,"15min":25.503},"last":1.0669999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2689999999999999,"15min":1.2969999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.5010000000000003,"5min":6.718,"15min":6.718},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.274,"15min":1.3440000000000001},"min":{"1min":0.68100000000000005,"5min":0.64700000000000002,"15min":0.373},"max":{"1min":4.3369999999999997,"5min":4.3369999999999997,"15min":15.452},"last":1.1759999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4910000000000001,"15min":1.4450000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.48299999999999998},"max":{"1min":4.2859999999999996,"5min":8.6349999999999998,"15min":10.707000000000001},"last":1.3},{"interface":"front","average":{"1min":2.5289999999999999,"5min":1.7509999999999999,"15min":1.532},"min":{"1min":0.65700000000000003,"5min":0.64400000000000002,"15min":0.504},"max":{"1min":25.478000000000002,"5min":25.478000000000002,"15min":25.478000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.5760000000000001,"15min":1.5720000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.6369999999999996,"5min":13.907999999999999,"15min":13.907999999999999},"last":1.123},{"interface":"front","average":{"1min":1.627,"5min":1.5569999999999999,"15min":1.5780000000000001},"min":{"1min":0.67800000000000005,"5min":0.63,"15min":0.62},"max":{"1min":6.351,"5min":8.6999999999999993,"15min":13.193},"last":1.444}]},{"osd":4,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":2.6619999999999999,"5min":1.6599999999999999,"15min":1.4810000000000001},"min":{"1min":0.60599999999999998,"5min":0.41599999999999998,"15min":0.376},"max":{"1min":25.401,"5min":25.401,"15min":25.401},"last":1.21},{"interface":"front","average":{"1min":1.587,"5min":1.677,"15min":1.5249999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.7859999999999996,"5min":10.134,"15min":10.134},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.603,"15min":1.538},"min":{"1min":0.63800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":4.4539999999999997,"5min":12.528,"15min":12.528},"last":1.5660000000000001},{"interface":"front","average":{"1min":2.6680000000000001,"5min":1.819,"15min":1.6399999999999999},"min":{"1min":0.61399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":26.327000000000002,"5min":26.327000000000002,"15min":26.327000000000002},"last":1.5009999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:36 2024","interfaces":[{"interface":"back","average":{"1min":2.8149999999999999,"5min":1.97,"15min":1.748},"min":{"1min":0.77400000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":25.451000000000001,"5min":25.451000000000001,"15min":25.451000000000001},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.7090000000000001,"15min":1.6819999999999999},"min":{"1min":0.76600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":5.8250000000000002,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.5349999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542312,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68244,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303596,"statfs":{"total":96636764160,"available":96566882304,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.232,"15min":1.6439999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.518,"5min":4.6600000000000001,"15min":30.164999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.365,"15min":1.712},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6899999999999999,"5min":11.52,"15min":30.280999999999999},"last":1.038}]},{"osd":1,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.4059999999999999,"15min":1.4690000000000001},"min":{"1min":0.61099999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.766,"5min":8.3439999999999994,"15min":28.122},"last":1.536},{"interface":"front","average":{"1min":1.097,"5min":1.2689999999999999,"15min":1.2589999999999999},"min":{"1min":0.70899999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4530000000000001,"5min":7.7489999999999997,"15min":8.9719999999999995},"last":0.71599999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.9750000000000001,"15min":1.6870000000000001},"min":{"1min":0.89700000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.6259999999999999,"5min":26.109000000000002,"15min":26.109000000000002},"last":1.325},{"interface":"front","average":{"1min":1.4359999999999999,"5min":2.117,"15min":1.841},"min":{"1min":0.99099999999999999,"5min":0.67500000000000004,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":26.161000000000001,"15min":26.161000000000001},"last":1.429}]},{"osd":3,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.524,"15min":1.6040000000000001},"min":{"1min":0.77900000000000003,"5min":0.50900000000000001,"15min":0.44700000000000001},"max":{"1min":1.663,"5min":11.289,"15min":17.75},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.6519999999999999,"15min":1.5960000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.55700000000000005},"max":{"1min":1.629,"5min":11.433,"15min":17.794},"last":1.151}]},{"osd":4,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.5840000000000001,"15min":1.5449999999999999},"min":{"1min":0.87,"5min":0.45500000000000002,"15min":0.45000000000000001},"max":{"1min":1.954,"5min":8.4890000000000008,"15min":15.571},"last":1.486},{"interface":"front","average":{"1min":1.228,"5min":1.5840000000000001,"15min":1.583},"min":{"1min":0.74199999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5680000000000001,"5min":23.667999999999999,"15min":23.667999999999999},"last":1.095}]},{"osd":5,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.661,"15min":1.456},"min":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4950000000000001,"5min":23.137,"15min":23.137},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.8009999999999999,"15min":1.673},"min":{"1min":0.82199999999999995,"5min":0.48099999999999998,"15min":0.47099999999999997},"max":{"1min":1.758,"5min":25.684000000000001,"15min":25.684000000000001},"last":1.385}]},{"osd":7,"last update":"Tue Jan 16 15:12:39 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.5449999999999999,"15min":1.5109999999999999},"min":{"1min":0.57099999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7,"5min":11.597,"15min":11.597},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.768,"15min":1.6899999999999999},"min":{"1min":1.01,"5min":0.499,"15min":0.499},"max":{"1min":1.859,"5min":8.6210000000000004,"15min":11.788},"last":1.5840000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76564,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295276,"statfs":{"total":96636764160,"available":96558362624,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.393,"15min":1.591},"min":{"1min":0.52500000000000002,"5min":0.41299999999999998,"15min":0.39000000000000001},"max":{"1min":1.6739999999999999,"5min":22.289000000000001,"15min":26.228999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.03,"5min":1.5049999999999999,"15min":1.833},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.7689999999999999,"5min":17.131,"15min":56.011000000000003},"last":1.006}]},{"osd":1,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.2470000000000001,"15min":1.373},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.698,"5min":6.0940000000000003,"15min":16.687999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.8340000000000001,"15min":1.9139999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40999999999999998},"max":{"1min":4.5960000000000001,"5min":22.451000000000001,"15min":67.930000000000007},"last":1.292}]},{"osd":2,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.577,"15min":1.5469999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.306,"5min":11.329000000000001,"15min":13.193},"last":1.246},{"interface":"front","average":{"1min":1.899,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.52200000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":16.721,"5min":16.721,"15min":16.721},"last":0.79000000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.8160000000000001,"15min":1.7769999999999999},"min":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.923,"5min":22.382000000000001,"15min":22.382000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.837,"15min":1.8049999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.2400000000000002,"5min":14.685,"15min":14.685},"last":1.157}]},{"osd":4,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":1.4379999999999999,"15min":1.4450000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.2},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.625,"15min":1.577},"min":{"1min":0.51500000000000001,"5min":0.44700000000000001,"15min":0.379},"max":{"1min":1.9790000000000001,"5min":27.006,"15min":27.006},"last":1.0880000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6719999999999999,"15min":1.5780000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.3380000000000001,"5min":10.106,"15min":13.263999999999999},"last":1.403},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.667,"15min":1.6379999999999999},"min":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.505},"max":{"1min":1.7969999999999999,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":1.1140000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.9039999999999999,"15min":1.8009999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.0249999999999999,"5min":35.127000000000002,"15min":35.127000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.7829999999999999,"15min":1.786},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.42999999999999999},"max":{"1min":1.885,"5min":10.016,"15min":13.491},"last":1.3480000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542312,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64084,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307756,"statfs":{"total":96636764160,"available":96571142144,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.103,"15min":1.2270000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.502},"max":{"1min":1.4950000000000001,"5min":6.3339999999999996,"15min":12.587999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.224,"15min":1.47},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.7729999999999999,"5min":4.976,"15min":60.106000000000002},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.595,"15min":1.6830000000000001},"min":{"1min":0.64500000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.673,"5min":10.124000000000001,"15min":22.143000000000001},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.369,"15min":1.51},"min":{"1min":0.76000000000000001,"5min":0.628,"15min":0.57099999999999995},"max":{"1min":1.748,"5min":5.1550000000000002,"15min":22.181999999999999},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.367,"15min":1.3129999999999999},"min":{"1min":0.875,"5min":0.58099999999999996,"15min":0.52800000000000002},"max":{"1min":2.3719999999999999,"5min":7.2110000000000003,"15min":9.7140000000000004},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.49,"15min":1.728},"min":{"1min":0.81699999999999995,"5min":0.63100000000000001,"15min":0.52400000000000002},"max":{"1min":1.696,"5min":13.316000000000001,"15min":35.024999999999999},"last":1.2849999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.546,"15min":1.5780000000000001},"min":{"1min":0.69399999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.8089999999999999,"5min":24.77,"15min":32.133000000000003},"last":1.238},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3879999999999999},"min":{"1min":0.71899999999999997,"5min":0.623,"15min":0.57999999999999996},"max":{"1min":2.2240000000000002,"5min":5.4690000000000003,"15min":7.8140000000000001},"last":1.341}]},{"osd":5,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.4830000000000001},"min":{"1min":0.55200000000000005,"5min":0.47199999999999998,"15min":0.38900000000000001},"max":{"1min":2.0459999999999998,"5min":5.2149999999999999,"15min":14.568},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.3979999999999999,"15min":1.464},"min":{"1min":0.73599999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.141,"5min":3.5129999999999999,"15min":7.1479999999999997},"last":1.6279999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.379,"15min":1.4059999999999999},"min":{"1min":0.67500000000000004,"5min":0.436,"15min":0.436},"max":{"1min":2.2959999999999998,"5min":4.9950000000000001,"15min":9.6630000000000003},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.7909999999999999,"15min":1.679},"min":{"1min":0.94799999999999995,"5min":0.51700000000000002,"15min":0.45100000000000001},"max":{"1min":1.915,"5min":10.054,"15min":14.317},"last":1.907}]},{"osd":7,"last update":"Tue Jan 16 15:12:53 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5720000000000001,"15min":1.5820000000000001},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.52100000000000002},"max":{"1min":2.0750000000000002,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.625,"15min":1.589},"min":{"1min":0.755,"5min":0.443,"15min":0.443},"max":{"1min":2.6379999999999999,"5min":9.2270000000000003,"15min":9.2270000000000003},"last":1.8500000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76564,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295276,"statfs":{"total":96636764160,"available":96558362624,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.383,"15min":1.514},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.26700000000000002},"max":{"1min":1.5720000000000001,"5min":14.111000000000001,"15min":50.988999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.748,"15min":1.4710000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.29699999999999999},"max":{"1min":1.8109999999999999,"5min":20.274000000000001,"15min":20.274000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.6899999999999999,"15min":1.381},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.274,"5min":18.192,"15min":18.192},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.9870000000000001,"15min":1.738},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":1.5509999999999999,"5min":18.34,"15min":20.385999999999999},"last":0.73799999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":2.3050000000000002,"15min":1.6950000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.5429999999999999,"5min":42.840000000000003,"15min":42.840000000000003},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.137,"5min":2.194,"15min":1.8089999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.35199999999999998},"max":{"1min":1.746,"5min":38.707000000000001,"15min":38.707000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.9299999999999999,"15min":1.76},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.738,"5min":14.148,"15min":18.952000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":2.2120000000000002,"15min":1.778},"min":{"1min":0.755,"5min":0.61599999999999999,"15min":0.54800000000000004},"max":{"1min":1.7669999999999999,"5min":18.088999999999999,"15min":18.088999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.9830000000000001,"15min":1.875},"min":{"1min":0.748,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.9019999999999999,"5min":18.224,"15min":24.349},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.2789999999999999,"5min":2.1850000000000001,"15min":1.913},"min":{"1min":0.66500000000000004,"5min":0.66100000000000003,"15min":0.61899999999999999},"max":{"1min":2.0249999999999999,"5min":14.192,"15min":14.192},"last":1.046}]},{"osd":6,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":2.2509999999999999,"15min":1.837},"min":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.52200000000000002},"max":{"1min":1.762,"5min":22.687000000000001,"15min":22.687000000000001},"last":1.03},{"interface":"front","average":{"1min":1.419,"5min":2.0790000000000002,"15min":1.9510000000000001},"min":{"1min":0.81100000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.044,"5min":18.300000000000001,"15min":24.058},"last":1.2549999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":2.6299999999999999,"15min":2.073},"min":{"1min":0.86099999999999999,"5min":0.76900000000000002,"15min":0.55800000000000005},"max":{"1min":2.1800000000000002,"5min":35.570999999999998,"15min":35.570999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":2.6909999999999998,"15min":2.0369999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.58899999999999997},"max":{"1min":1.857,"5min":37.607999999999997,"15min":37.607999999999997},"last":1.2869999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542312,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1052672,"data_stored":449046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.212,"15min":1.4910000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.24399999999999999},"max":{"1min":1.4750000000000001,"5min":11.112,"15min":64.387},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.84099999999999997,"5min":1.581,"15min":1.391},"min":{"1min":0.50700000000000001,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.143,"5min":24.308,"15min":24.308},"last":0.36299999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.3069999999999999,"15min":1.4379999999999999},"min":{"1min":0.65100000000000002,"5min":0.42299999999999999,"15min":0.28299999999999997},"max":{"1min":1.966,"5min":13.497,"15min":27.263000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.994,"15min":1.7969999999999999},"min":{"1min":0.56499999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.716,"5min":23.602,"15min":23.602},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.4590000000000001,"15min":1.5700000000000001},"min":{"1min":0.54500000000000004,"5min":0.497,"15min":0.33100000000000002},"max":{"1min":3.1080000000000001,"5min":13.007,"15min":27.311},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.6060000000000001,"15min":1.6200000000000001},"min":{"1min":0.66700000000000004,"5min":0.49099999999999999,"15min":0.47099999999999997},"max":{"1min":1.8100000000000001,"5min":12.833,"15min":26.555},"last":0.71999999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.7929999999999999,"15min":1.6559999999999999},"min":{"1min":0.86799999999999999,"5min":0.54000000000000004,"15min":0.38900000000000001},"max":{"1min":1.879,"5min":17.146999999999998,"15min":26.503},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.413,"5min":2.097,"15min":1.7390000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.9079999999999999,"5min":25.661000000000001,"15min":25.661000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.6319999999999999,"15min":1.7170000000000001},"min":{"1min":0.99199999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.3029999999999999,"5min":11.005000000000001,"15min":27.149000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.7809999999999999,"15min":1.756},"min":{"1min":0.91500000000000004,"5min":0.66900000000000004,"15min":0.43099999999999999},"max":{"1min":2.081,"5min":10.984,"15min":26.349},"last":0.59799999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":2.577,"15min":1.9319999999999999},"min":{"1min":1.0760000000000001,"5min":0.504,"15min":0.504},"max":{"1min":13.629,"5min":25.603999999999999,"15min":25.603999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.794,"15min":1.8660000000000001},"min":{"1min":1.02,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.2010000000000001,"5min":19.298999999999999,"15min":27.076000000000001},"last":0.629}]},{"osd":7,"last update":"Tue Jan 16 15:12:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.3090000000000002,"15min":1.96},"min":{"1min":0.96499999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"max":{"1min":2.9940000000000002,"5min":24.509,"15min":24.509},"last":0.61699999999999999},{"interface":"front","average":{"1min":2.323,"5min":1.9419999999999999,"15min":1.8959999999999999},"min":{"1min":1.2110000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":10.455,"5min":10.455,"15min":27.207999999999998},"last":0.68600000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542312,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72428,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299412,"statfs":{"total":96636764160,"available":96562597888,"internally_reserved":0,"allocated":1077248,"data_stored":456043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.123,"15min":1.28},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.266,"5min":6.2249999999999996,"15min":14.407},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3169999999999999,"15min":1.3180000000000001},"min":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.3620000000000001,"5min":11.074999999999999,"15min":13.930999999999999},"last":1.238}]},{"osd":2,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.3660000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":5.9130000000000003,"5min":5.9130000000000003,"15min":16.488},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.6080000000000001,"15min":1.71},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.1559999999999997,"5min":21.446000000000002,"15min":22.550999999999998},"last":1.4630000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.831,"15min":1.7509999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.1980000000000004,"5min":22.471,"15min":22.471},"last":1.649},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.6719999999999999,"15min":1.6439999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.35699999999999998},"max":{"1min":2.2570000000000001,"5min":22.550999999999998,"15min":22.550999999999998},"last":1.837}]},{"osd":4,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5940000000000001,"15min":1.663},"min":{"1min":0.69899999999999995,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3159999999999998,"5min":5.3789999999999996,"15min":12.675000000000001},"last":1.526},{"interface":"front","average":{"1min":1.448,"5min":1.649,"15min":1.724},"min":{"1min":0.87,"5min":0.433,"15min":0.433},"max":{"1min":3.6269999999999998,"5min":5.6849999999999996,"15min":19.236999999999998},"last":1.5880000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.851,"15min":1.778},"min":{"1min":0.80900000000000005,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":3.3719999999999999,"5min":21.117000000000001,"15min":21.117000000000001},"last":1.772},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6799999999999999,"15min":1.7370000000000001},"min":{"1min":0.82999999999999996,"5min":0.65800000000000003,"15min":0.64800000000000002},"max":{"1min":3.5470000000000002,"5min":5.9009999999999998,"15min":17.138000000000002},"last":1.1879999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.7410000000000001,"15min":1.863},"min":{"1min":0.85499999999999998,"5min":0.77500000000000002,"15min":0.72199999999999998},"max":{"1min":4.2350000000000003,"5min":9.3759999999999994,"15min":14.781000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.7190000000000001,"15min":1.861},"min":{"1min":0.754,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.407,"5min":5.4630000000000001,"15min":23.420999999999999},"last":1.6799999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.76,"15min":1.823},"min":{"1min":0.81999999999999995,"5min":0.71299999999999997,"15min":0.60699999999999998},"max":{"1min":3.7690000000000001,"5min":5.8300000000000001,"15min":17.010000000000002},"last":1.397},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.97,"15min":1.988},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.75700000000000001},"max":{"1min":4.9640000000000004,"5min":10.316000000000001,"15min":14.884},"last":1.333}]}]},{"osd":0,"up_from":14,"seq":60129542313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68836,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303004,"statfs":{"total":96636764160,"available":96566276096,"internally_reserved":0,"allocated":1662976,"data_stored":1046251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":68811201},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.603,"15min":1.456},"min":{"1min":0.51700000000000002,"5min":0.48199999999999998,"15min":0.36499999999999999},"max":{"1min":1.748,"5min":43.459000000000003,"15min":43.459000000000003},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.726,"15min":1.5940000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.28999999999999998},"max":{"1min":2.1339999999999999,"5min":43.627000000000002,"15min":43.627000000000002},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.3999999999999999,"15min":1.4099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":18.861000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.028,"5min":1.3120000000000001,"15min":1.452},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.378},"max":{"1min":2.2530000000000001,"5min":8.9979999999999993,"15min":38.893000000000001},"last":0.98499999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.5009999999999999,"15min":1.5449999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.31900000000000001},"max":{"1min":2.0459999999999998,"5min":23.687999999999999,"15min":23.687999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.28,"5min":1.6839999999999999,"15min":1.631},"min":{"1min":0.629,"5min":0.59999999999999998,"15min":0.46800000000000003},"max":{"1min":1.788,"5min":16.978000000000002,"15min":18.966999999999999},"last":1.1000000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.5309999999999999,"15min":1.4830000000000001},"min":{"1min":0.69099999999999995,"5min":0.63400000000000001,"15min":0.58899999999999997},"max":{"1min":2.105,"5min":10.18,"15min":20.327000000000002},"last":1.375},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.581,"15min":1.6879999999999999},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8460000000000001,"5min":4.665,"15min":19.004999999999999},"last":1.2889999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.7410000000000001,"15min":1.8069999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.54600000000000004},"max":{"1min":2.1800000000000002,"5min":13.016,"15min":24.611000000000001},"last":1.329},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":1.6659999999999999},"min":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":20.199000000000002},"last":1.5229999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":2.0150000000000001,"15min":2.0249999999999999},"min":{"1min":0.72999999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":3.9569999999999999,"5min":16.832999999999998,"15min":26.998000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.95,"15min":1.929},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":4.0419999999999998,"5min":16.093,"15min":24.478999999999999},"last":1.4239999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:12:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.7649999999999999,"15min":1.8480000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.58399999999999996},"max":{"1min":3.0910000000000002,"5min":7.0759999999999996,"15min":17.957000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.663,"15min":1.71},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.52800000000000002},"max":{"1min":2.423,"5min":5.6109999999999998,"15min":20.279},"last":1.617}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:13:15.162 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:13:15.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:13:15.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.446+0000 7f29ed600640 1 Processor -- start 2024-01-16T15:13:15.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.446+0000 7f29ed600640 1 -- start start 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e815d340 0x7f29e80d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29e80cdbb0 0x7f29e80d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29e80d1bc0 0x7f29e80b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80d2050 con 0x7f29e80cdbb0 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80cef30 con 0x7f29e80d1bc0 2024-01-16T15:13:15.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.448+0000 7f29ed600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80d11a0 con 0x7f29e815d340 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29e80cdbb0 0x7f29e80d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29e80cdbb0 0x7f29e80d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42478/0 (socket says 172.21.15.43:42478) 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 -- 172.21.15.43:0/849297515 learned_addr learned my addr 172.21.15.43:0/849297515 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29e7e00640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e815d340 0x7f29e80d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29ecc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29e80d1bc0 0x7f29e80b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 -- 172.21.15.43:0/849297515 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e815d340 msgr2=0x7f29e80d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:15.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e815d340 0x7f29e80d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 -- 172.21.15.43:0/849297515 >> v2:172.21.15.69:3300/0 conn(0x7f29e80d1bc0 msgr2=0x7f29e80b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:15.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 --2- 172.21.15.43:0/849297515 >> v2:172.21.15.69:3300/0 conn(0x7f29e80d1bc0 0x7f29e80b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.449+0000 7f29dfe00640 1 -- 172.21.15.43:0/849297515 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80ce320 con 0x7f29e80cdbb0 2024-01-16T15:13:15.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.451+0000 7f29dfe00640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e80cdbb0 0x7f29e80d1680 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f29e80b9120 tx=0x7f29d002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa8279712df0ef4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.451+0000 7f29e7400640 1 -- 172.21.15.43:0/849297515 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0031860 con 0x7f29e80cdbb0 2024-01-16T15:13:15.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.451+0000 7f29e7400640 1 -- 172.21.15.43:0/849297515 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d00319c0 con 0x7f29e80cdbb0 2024-01-16T15:13:15.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.451+0000 7f29e7400640 1 -- 172.21.15.43:0/849297515 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0031d50 con 0x7f29e80cdbb0 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 -- 172.21.15.43:0/849297515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e80cdbb0 msgr2=0x7f29e80d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e80cdbb0 0x7f29e80d1680 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f29e80b9120 tx=0x7f29d002b900 comp rx=0 tx=0).stop 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 -- 172.21.15.43:0/849297515 shutdown_connections 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 --2- 172.21.15.43:0/849297515 >> v2:172.21.15.69:3300/0 conn(0x7f29e80d1bc0 0x7f29e80b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e80cdbb0 0x7f29e80d1680 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 --2- 172.21.15.43:0/849297515 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e815d340 0x7f29e80d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 -- 172.21.15.43:0/849297515 >> 172.21.15.43:0/849297515 conn(0x7f29e809c370 msgr2=0x7f29e80d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 -- 172.21.15.43:0/849297515 shutdown_connections 2024-01-16T15:13:15.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.452+0000 7f29ed600640 1 -- 172.21.15.43:0/849297515 wait complete. 2024-01-16T15:13:15.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.455+0000 7f29ed600640 1 Processor -- start 2024-01-16T15:13:15.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.455+0000 7f29ed600640 1 -- start start 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80cdbb0 0x7f29e8148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e80d1bc0 0x7f29e815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c6760 con 0x7f29e815d340 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c7870 con 0x7f29e80cdbb0 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.456+0000 7f29ed600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e815e260 con 0x7f29e80d1bc0 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42494/0 (socket says 172.21.15.43:42494) 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 -- 172.21.15.43:0/1895243559 learned_addr learned my addr 172.21.15.43:0/1895243559 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29e7e00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80cdbb0 0x7f29e8148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29dfe00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e80d1bc0 0x7f29e815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 -- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e80d1bc0 msgr2=0x7f29e815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e80d1bc0 0x7f29e815e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 -- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80cdbb0 msgr2=0x7f29e8148470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80cdbb0 0x7f29e8148470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.457+0000 7f29ecc00640 1 -- 172.21.15.43:0/1895243559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d00094d0 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29ecc00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f29e000eed0 tx=0x7f29e000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0010040 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e000bd20 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0014640 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e80c7b10 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.458+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e80c80d0 con 0x7f29e815d340 2024-01-16T15:13:15.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.459+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e80c82f0 con 0x7f29e815d340 2024-01-16T15:13:15.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.460+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f29e00147a0 con 0x7f29e815d340 2024-01-16T15:13:15.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.461+0000 7f29e4c00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 0x7f29c006e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.461+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f29e0088c50 con 0x7f29e815d340 2024-01-16T15:13:15.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.462+0000 7f29e7e00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 0x7f29c006e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.462+0000 7f29e7e00640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 0x7f29c006e430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f29e815ddd0 tx=0x7f29d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.466+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29e0055510 con 0x7f29e815d340 2024-01-16T15:13:15.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.731+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f29e8067d70 con 0x7f29e815d340 2024-01-16T15:13:15.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.735+0000 7f29e4c00640 1 -- 172.21.15.43:0/1895243559 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f29e0018070 con 0x7f29e815d340 2024-01-16T15:13:15.713 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:13:15.714 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:13:15.714 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:13:15.714 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:13:15.714 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:13:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:13:15.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.737+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 msgr2=0x7f29c006e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.737+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 0x7f29c006e430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f29e815ddd0 tx=0x7f29d8008040 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.737+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 msgr2=0x7f29e80c85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.737+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f29e000eed0 tx=0x7f29e000b480 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 shutdown_connections 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f29c006bf70 0x7f29c006e430 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29e815d340 0x7f29e80c85f0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29e80d1bc0 0x7f29e815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 --2- 172.21.15.43:0/1895243559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80cdbb0 0x7f29e8148470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 >> 172.21.15.43:0/1895243559 conn(0x7f29e809c370 msgr2=0x7f29e80ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 shutdown_connections 2024-01-16T15:13:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.738+0000 7f29ed600640 1 -- 172.21.15.43:0/1895243559 wait complete. 2024-01-16T15:13:15.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:15.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.975+0000 7f50ef600640 1 Processor -- start 2024-01-16T15:13:15.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.975+0000 7f50ef600640 1 -- start start 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ef600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50e80719c0 0x7f50e80ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ef600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 0x7f50e806d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ef600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e806deb0 con 0x7f50e8161930 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ef600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e80ad510 con 0x7f50e80719c0 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ef600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e80acc50 con 0x7f50e806d520 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50e7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50e80719c0 0x7f50e80ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ecc00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f50e8161930 0x7f50e80ad9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33058/0 (socket says 172.21.15.43:33058) 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ecc00640 1 -- 172.21.15.43:0/267450672 learned_addr learned my addr 172.21.15.43:0/267450672 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50e7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50e80719c0 0x7f50e80ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42504/0 (socket says 172.21.15.43:42504) 2024-01-16T15:13:15.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.977+0000 7f50ed600640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 0x7f50e806d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7400640 1 -- 172.21.15.43:0/267450672 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381035030 0 0) 0x7f50d8001110 con 0x7f50e8161930 2024-01-16T15:13:15.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7400640 1 -- 172.21.15.43:0/267450672 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c8003520 con 0x7f50e8161930 2024-01-16T15:13:15.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7e00640 1 -- 172.21.15.43:0/267450672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 msgr2=0x7f50e806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:15.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7e00640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 0x7f50e806d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7e00640 1 -- 172.21.15.43:0/267450672 >> v1:172.21.15.43:6789/0 conn(0x7f50e8161930 legacy=0x7f50e80ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50e7e00640 1 -- 172.21.15.43:0/267450672 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8071e60 con 0x7f50e80719c0 2024-01-16T15:13:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.978+0000 7f50ed600640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 0x7f50e806d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.979+0000 7f50e7e00640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e80ae6a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f50d00097a0 tx=0x7f50d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1173ea2d5c574d35 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.979+0000 7f50e7400640 1 -- 172.21.15.43:0/267450672 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031830 con 0x7f50e80719c0 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.979+0000 7f50e7400640 1 -- 172.21.15.43:0/267450672 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d0031990 con 0x7f50e80719c0 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.979+0000 7f50e7400640 1 -- 172.21.15.43:0/267450672 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031d20 con 0x7f50e80719c0 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 -- 172.21.15.43:0/267450672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 msgr2=0x7f50e80ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e80ae6a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f50d00097a0 tx=0x7f50d002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 -- 172.21.15.43:0/267450672 shutdown_connections 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e806d520 0x7f50e806d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 --2- 172.21.15.43:0/267450672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e80ae6a0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 -- 172.21.15.43:0/267450672 >> 172.21.15.43:0/267450672 conn(0x7f50e80b0d80 msgr2=0x7f50e80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 -- 172.21.15.43:0/267450672 shutdown_connections 2024-01-16T15:13:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.980+0000 7f50ef600640 1 -- 172.21.15.43:0/267450672 wait complete. 2024-01-16T15:13:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.982+0000 7f50ef600640 1 Processor -- start 2024-01-16T15:13:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.982+0000 7f50ef600640 1 -- start start 2024-01-16T15:13:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 0x7f50e815dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e8161930 0x7f50e80cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e814d8c0 con 0x7f50e80719c0 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e815e1a0 con 0x7f50e806d520 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50ef600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e815e480 con 0x7f50e8161930 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50e7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42510/0 (socket says 172.21.15.43:42510) 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.983+0000 7f50e7e00640 1 -- 172.21.15.43:0/443942597 learned_addr learned my addr 172.21.15.43:0/443942597 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50ed600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e8161930 0x7f50e80cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50ecc00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 0x7f50e815dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 -- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e8161930 msgr2=0x7f50e80cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e8161930 0x7f50e80cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 -- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 msgr2=0x7f50e815dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 0x7f50e815dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 -- 172.21.15.43:0/443942597 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0009400 con 0x7f50e80719c0 2024-01-16T15:13:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50ecc00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 0x7f50e815dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e7e00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f50d002bee0 tx=0x7f50d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d003a6f0 con 0x7f50e80719c0 2024-01-16T15:13:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.984+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d003a870 con 0x7f50e80719c0 2024-01-16T15:13:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.985+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e815e760 con 0x7f50e80719c0 2024-01-16T15:13:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.985+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d003ac20 con 0x7f50e80719c0 2024-01-16T15:13:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.985+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e815ed20 con 0x7f50e80719c0 2024-01-16T15:13:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.986+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e8074770 con 0x7f50e80719c0 2024-01-16T15:13:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.987+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 30) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f50d004c3b0 con 0x7f50e80719c0 2024-01-16T15:13:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.988+0000 7f50e4c00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 0x7f50c006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.988+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f50d00aa4d0 con 0x7f50e80719c0 2024-01-16T15:13:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.989+0000 7f50ecc00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 0x7f50c006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.989+0000 7f50ecc00640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 0x7f50c006e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f50d8001370 tx=0x7f50d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:15.993+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50d0076b10 con 0x7f50e80719c0 2024-01-16T15:13:16.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.226+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f50d0076ee0 con 0x7f50e80719c0 2024-01-16T15:13:16.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.230+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50e80cb4b0 con 0x7f50e80719c0 2024-01-16T15:13:16.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.231+0000 7f50e4c00640 1 -- 172.21.15.43:0/443942597 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f50d00ae130 con 0x7f50e80719c0 2024-01-16T15:13:16.210 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:13:16.210 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/206348255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":206348255},{"type":"v1","addr":"172.21.15.69:6834","nonce":206348255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3389933703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3389933703},{"type":"v1","addr":"172.21.15.69:6835","nonce":3389933703}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/1040529098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1040529098},{"type":"v1","addr":"172.21.15.43:6835","nonce":1040529098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:13:09.595061+0000","modified":"2024-01-16T15:13:12.678462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.43:6837/3379600529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3379600529},{"type":"v1","addr":"172.21.15.43:6837","nonce":3379600529}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:13:16.210 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 57 2024-01-16T15:13:16.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.234+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 msgr2=0x7f50c006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.234+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 0x7f50c006e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f50d8001370 tx=0x7f50d8009040 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.234+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 msgr2=0x7f50e814d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.234+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f50d002bee0 tx=0x7f50d00046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 shutdown_connections 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f50c006bf20 0x7f50c006e3e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50e8161930 0x7f50e80cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50e80719c0 0x7f50e814d380 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 --2- 172.21.15.43:0/443942597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50e806d520 0x7f50e815dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 >> 172.21.15.43:0/443942597 conn(0x7f50e80b0d80 msgr2=0x7f50e80b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 shutdown_connections 2024-01-16T15:13:16.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.235+0000 7f50ef600640 1 -- 172.21.15.43:0/443942597 wait complete. 2024-01-16T15:13:16.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:13:09.595061+0000', 'modified': '2024-01-16T15:13:12.678462+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8106}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8106': {'gid': 8106, 'name': 'a', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.43:6837/3379600529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 3379600529}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 3379600529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:13:16.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:13:16.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-16T15:13:16.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.887+0000 7ff608c00640 1 Processor -- start 2024-01-16T15:13:16.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.888+0000 7ff608c00640 1 -- start start 2024-01-16T15:13:16.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.889+0000 7ff608c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff604070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff608c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604070a90 con 0x7ff604161700 2024-01-16T15:13:16.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff608c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604070cf0 con 0x7ff6040b2930 2024-01-16T15:13:16.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff608c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff604070fd0 con 0x7ff6040b2550 2024-01-16T15:13:16.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff603e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff604161700 0x7ff6040ac900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33070/0 (socket says 172.21.15.43:33070) 2024-01-16T15:13:16.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff603e00640 1 -- 172.21.15.43:0/2558118693 learned_addr learned my addr 172.21.15.43:0/2558118693 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:16.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.890+0000 7ff603400640 1 --2- 172.21.15.43:0/2558118693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff604070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199966974 0 0) 0x7ff604070a90 con 0x7ff604161700 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e4003520 con 0x7ff604161700 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649273256 0 0) 0x7ff604070cf0 con 0x7ff6040b2930 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604070a90 con 0x7ff6040b2930 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff603400640 1 -- 172.21.15.43:0/2558118693 >> v1:172.21.15.69:6789/0 conn(0x7ff6040b2930 legacy=0x7ff604160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff603400640 1 -- 172.21.15.43:0/2558118693 >> v1:172.21.15.43:6789/0 conn(0x7ff604161700 legacy=0x7ff6040ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652352997 0 0) 0x7ff5e4003520 con 0x7ff604161700 2024-01-16T15:13:16.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.891+0000 7ff603400640 1 -- 172.21.15.43:0/2558118693 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040712b0 con 0x7ff6040b2550 2024-01-16T15:13:16.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.893+0000 7ff603400640 1 --2- 172.21.15.43:0/2558118693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff604070550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40097d0 tx=0x7ff5f402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9affbbbd2b00a7d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.893+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0019f0 con 0x7ff6040b2550 2024-01-16T15:13:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.893+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f402e050 con 0x7ff6040b2550 2024-01-16T15:13:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.893+0000 7ff602a00640 1 -- 172.21.15.43:0/2558118693 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4030710 con 0x7ff6040b2550 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 -- 172.21.15.43:0/2558118693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 msgr2=0x7ff604070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 --2- 172.21.15.43:0/2558118693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff604070550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40097d0 tx=0x7ff5f402b800 comp rx=0 tx=0).stop 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 -- 172.21.15.43:0/2558118693 shutdown_connections 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 --2- 172.21.15.43:0/2558118693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff604070550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 -- 172.21.15.43:0/2558118693 >> 172.21.15.43:0/2558118693 conn(0x7ff6040b0840 msgr2=0x7ff6040b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 -- 172.21.15.43:0/2558118693 shutdown_connections 2024-01-16T15:13:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.894+0000 7ff608c00640 1 -- 172.21.15.43:0/2558118693 wait complete. 2024-01-16T15:13:16.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.896+0000 7ff608c00640 1 Processor -- start 2024-01-16T15:13:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.896+0000 7ff608c00640 1 -- start start 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff60413e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604161700 0x7ff6041580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60413ef60 con 0x7ff6040b2930 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6040ad620 con 0x7ff604161700 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff608c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6040ad900 con 0x7ff6040b2550 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42512/0 (socket says 172.21.15.43:42512) 2024-01-16T15:13:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.897+0000 7ff5fbe00640 1 -- 172.21.15.43:0/4231801913 learned_addr learned my addr 172.21.15.43:0/4231801913 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff603e00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604161700 0x7ff6041580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 -- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 msgr2=0x7ff60413e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff60413e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 -- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604161700 msgr2=0x7ff6041580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604161700 0x7ff6041580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 -- 172.21.15.43:0/4231801913 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4009400 con 0x7ff6040b2930 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00c500 tx=0x7ff5ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fb400640 1 -- 172.21.15.43:0/4231801913 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0052d0 con 0x7ff6040b2930 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fb400640 1 -- 172.21.15.43:0/4231801913 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec005450 con 0x7ff6040b2930 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff5fb400640 1 -- 172.21.15.43:0/4231801913 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec012870 con 0x7ff6040b2930 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6040adbe0 con 0x7ff6040b2930 2024-01-16T15:13:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.898+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6040ae1a0 con 0x7ff6040b2930 2024-01-16T15:13:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.901+0000 7ff5fb400640 1 -- 172.21.15.43:0/4231801913 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff5ec0129d0 con 0x7ff6040b2930 2024-01-16T15:13:16.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.902+0000 7ff5fb400640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 0x7ff5dc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.903+0000 7ff5fb400640 1 -- 172.21.15.43:0/4231801913 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ff5ec0884a0 con 0x7ff6040b2930 2024-01-16T15:13:16.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.903+0000 7ff603400640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 0x7ff5dc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.903+0000 7ff60cdc0740 1 Processor -- start 2024-01-16T15:13:16.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.903+0000 7ff60cdc0740 1 -- start start 2024-01-16T15:13:16.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.903+0000 7ff603400640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 0x7ff5dc06e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4034000 tx=0x7ff5f402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb16d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb0cb880 con 0x5598fb167d70 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb0daae0 con 0x5598fb16d070 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598fb0d81d0 con 0x5598fb16c7b0 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.904+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42526/0 (socket says 172.21.15.43:42526) 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1990783334 learned_addr learned my addr 172.21.15.43:0/1990783334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff603400640 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb16d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 msgr2=0x5598fb1b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 msgr2=0x5598fb16d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb16d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1990783334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598fb09b860 con 0x5598fb167d70 2024-01-16T15:13:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec054c40 tx=0x7ff5ec058e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453030aca8d7a4ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5f3e00640 1 -- 172.21.15.43:0/1990783334 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec060a40 con 0x5598fb167d70 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5f3e00640 1 -- 172.21.15.43:0/1990783334 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec077070 con 0x5598fb167d70 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.905+0000 7ff5f3e00640 1 -- 172.21.15.43:0/1990783334 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec012c80 con 0x5598fb167d70 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1990783334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 msgr2=0x5598fb16c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec054c40 tx=0x7ff5ec058e40 comp rx=0 tx=0).stop 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1990783334 shutdown_connections 2024-01-16T15:13:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb16c270 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1990783334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb16d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1990783334 >> 172.21.15.43:0/1990783334 conn(0x5598fb12ec70 msgr2=0x5598fb1b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1990783334 shutdown_connections 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.906+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1990783334 wait complete. 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.907+0000 7ff60cdc0740 1 -- start start 2024-01-16T15:13:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.907+0000 7ff60cdc0740 1 Processor -- start 2024-01-16T15:13:16.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb196be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb0d3ac0 con 0x5598fb167d70 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb0702f0 con 0x5598fb16d070 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598fb08dc70 con 0x5598fb16c7b0 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff60cdc0740 10 client.0 _ll_register_callbacks cb 0x5598fb0923b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.908+0000 7ff603400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42528/0 (socket says 172.21.15.43:42528) 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 -- 172.21.15.43:0/2043371981 learned_addr learned my addr 172.21.15.43:0/2043371981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603e00640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 -- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 msgr2=0x5598fb1c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 -- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 msgr2=0x5598fb196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 -- 172.21.15.43:0/2043371981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6040ae1a0 con 0x5598fb167d70 2024-01-16T15:13:16.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff603400640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4003c70 tx=0x7ff5f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22346c4ab694625 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.909+0000 7ff5f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4030710 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ff5f4030870 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598fb0ccaa0 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5f4030a30 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff60cdc0740 10 client.8235 fetch_fsmap learned FSMap version 57 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5598fb0d2140 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5598fb0ccaa0 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.910+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff5f4030b90 con 0x5598fb167d70 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 10 client.8235 fetch_fsmap finished waiting for FSMap version 57 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 10 client.8235 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 20 client.8235 populate_metadata read hostname 'smithi043' 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb196280 0x5598fb1c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 4 client.8235 mds_command: new command op to 8106 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:13:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 --> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598fb0cc7c0 con 0x5598fb196280 2024-01-16T15:13:16.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb196280 0x5598fb1c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb196280 0x5598fb1c5fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7936eba024affda server_cookie=314306ab89ca65e7 in_seq=0 out_seq=0 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.911+0000 7ff5f0c00640 10 client.8235 ms_handle_connect on v2:172.21.15.43:6836/3379600529 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.912+0000 7ff5f0c00640 1 -- 172.21.15.43:0/2043371981 <== mds.0 v2:172.21.15.43:6836/3379600529 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5598fb0cc7c0 con 0x5598fb196280 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.912+0000 7ff5f0c00640 10 client.8235 handle_command_reply: tid=0 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.912+0000 7ff60cdc0740 1 client.8235 shutdown 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.912+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 msgr2=0x5598fb1932d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.912+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4003c70 tx=0x7ff5f4004730 comp rx=0 tx=0).stop 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 shutdown_connections 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb196280 0x5598fb1c5fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16d070 0x5598fb196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16c7b0 0x5598fb1c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/2043371981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb167d70 0x5598fb1932d0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff5f0c00640 0 client.8235 ms_handle_reset on v2:172.21.15.43:6836/3379600529 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 >> 172.21.15.43:0/2043371981 conn(0x5598fb12ec70 msgr2=0x5598fb1b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 shutdown_connections 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 1 -- 172.21.15.43:0/2043371981 wait complete. 2024-01-16T15:13:16.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.913+0000 7ff60cdc0740 20 client.8235 trim_cache size 0 max 16384 2024-01-16T15:13:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.930+0000 7ff60cdc0740 1 Processor -- start 2024-01-16T15:13:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.930+0000 7ff60cdc0740 1 -- start start 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb167d70 0x5598fb163740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16c7b0 0x5598fb217670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fae36f20 con 0x5598fb16d070 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb196b50 con 0x5598fb16c7b0 2024-01-16T15:13:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598fb217bb0 con 0x5598fb167d70 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.931+0000 7ff603e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42532/0 (socket says 172.21.15.43:42532) 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 -- 172.21.15.43:0/3819452275 learned_addr learned my addr 172.21.15.43:0/3819452275 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603400640 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb167d70 0x5598fb163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16c7b0 0x5598fb217670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 -- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb167d70 msgr2=0x5598fb163740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb167d70 0x5598fb163740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 -- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16c7b0 msgr2=0x5598fb217670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16c7b0 0x5598fb217670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 -- 172.21.15.43:0/3819452275 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598fb06fb80 con 0x5598fb16d070 2024-01-16T15:13:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.932+0000 7ff603e00640 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005c00 tx=0x7ff5fc00f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cc4ed4bb678692 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff5f8c00640 1 -- 172.21.15.43:0/3819452275 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc00fe90 con 0x5598fb16d070 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff5f8c00640 1 -- 172.21.15.43:0/3819452275 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc00d050 con 0x5598fb16d070 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff5f8c00640 1 -- 172.21.15.43:0/3819452275 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc007310 con 0x5598fb16d070 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 -- 172.21.15.43:0/3819452275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 msgr2=0x5598fb197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005c00 tx=0x7ff5fc00f5a0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 -- 172.21.15.43:0/3819452275 shutdown_connections 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb16c7b0 0x5598fb217670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16d070 0x5598fb197710 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/3819452275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb167d70 0x5598fb163740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.933+0000 7ff60cdc0740 1 -- 172.21.15.43:0/3819452275 >> 172.21.15.43:0/3819452275 conn(0x5598fb12ec70 msgr2=0x5598fb1afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.934+0000 7ff60cdc0740 1 -- 172.21.15.43:0/3819452275 shutdown_connections 2024-01-16T15:13:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.934+0000 7ff60cdc0740 1 -- 172.21.15.43:0/3819452275 wait complete. 2024-01-16T15:13:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.934+0000 7ff60cdc0740 1 -- start start 2024-01-16T15:13:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.934+0000 7ff60cdc0740 1 Processor -- start 2024-01-16T15:13:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb167d70 0x5598fb1a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16d070 0x5598fb2132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb213810 con 0x5598fb16c7b0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598fb1c1700 con 0x5598fb167d70 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598fb1a8930 con 0x5598fb16d070 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.935+0000 7ff60cdc0740 10 client.0 _ll_register_callbacks cb 0x7ff5f4035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42538/0 (socket says 172.21.15.43:42538) 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1258786189 learned_addr learned my addr 172.21.15.43:0/1258786189 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16d070 msgr2=0x5598fb2132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16d070 0x5598fb2132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb167d70 msgr2=0x5598fb1a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb167d70 0x5598fb1a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 -- 172.21.15.43:0/1258786189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc001d50 con 0x5598fb16c7b0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5fbe00640 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec058900 tx=0x7ff5ec000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8301ae1dbad3c0b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.936+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec081ea0 con 0x5598fb16c7b0 2024-01-16T15:13:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ff5ec054e50 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5ec077070 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5ec055010 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 10 client.8241 fetch_fsmap learned FSMap version 57 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5598fb095170 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec077070 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff5ec07e020 con 0x5598fb16c7b0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 10 client.8241 fetch_fsmap finished waiting for FSMap version 57 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 10 client.8241 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 20 client.8241 populate_metadata read hostname 'smithi043' 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb1977f0 0x5598fb197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 4 client.8241 mds_command: new command op to 8106 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 --> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5598fadd46c0 con 0x5598fb1977f0 2024-01-16T15:13:16.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.937+0000 7ff603400640 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb1977f0 0x5598fb197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.938+0000 7ff5f0c00640 10 client.8241 ms_handle_connect on v2:172.21.15.43:6836/3379600529 2024-01-16T15:13:16.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.938+0000 7ff603400640 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb1977f0 0x5598fb197bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee59552d1c6c6ca1 server_cookie=e4d5038c26676754 in_seq=0 out_seq=0 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.938+0000 7ff5f0c00640 1 -- 172.21.15.43:0/1258786189 <== mds.0 v2:172.21.15.43:6836/3379600529 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5598fadd46c0 con 0x5598fb1977f0 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.938+0000 7ff5f0c00640 10 client.8241 handle_command_reply: tid=0 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.938+0000 7ff60cdc0740 1 client.8241 shutdown 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 msgr2=0x5598fb1b6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec058900 tx=0x7ff5ec000ed0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 shutdown_connections 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:6836/3379600529,v1:172.21.15.43:6837/3379600529] conn(0x5598fb1977f0 0x5598fb197bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5598fb16d070 0x5598fb2132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5598fb16c7b0 0x5598fb1b6010 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff5f0c00640 0 client.8241 ms_handle_reset on v2:172.21.15.43:6836/3379600529 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 --2- 172.21.15.43:0/1258786189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598fb167d70 0x5598fb1a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 >> 172.21.15.43:0/1258786189 conn(0x5598fb12ec70 msgr2=0x5598fb1c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 shutdown_connections 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 1 -- 172.21.15.43:0/1258786189 wait complete. 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.939+0000 7ff60cdc0740 20 client.8241 trim_cache size 0 max 16384 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 8106, 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.792902819, 2024-01-16T15:13:16.921 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 57, 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 88, 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 13.774194980000001 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 msgr2=0x7ff5dc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 0x7ff5dc06e350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4034000 tx=0x7ff5f402f000 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 msgr2=0x7ff6041542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00c500 tx=0x7ff5ec00c9c0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 shutdown_connections 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff5dc06be90 0x7ff5dc06e350 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff604161700 0x7ff6041580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6040b2930 0x7ff6041542c0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 --2- 172.21.15.43:0/4231801913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6040b2550 0x7ff60413e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 >> 172.21.15.43:0/4231801913 conn(0x7ff6040b0840 msgr2=0x7ff60415c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 shutdown_connections 2024-01-16T15:13:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:16.940+0000 7ff608c00640 1 -- 172.21.15.43:0/4231801913 wait complete. 2024-01-16T15:13:16.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:13:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.182+0000 7ff023e00640 1 Processor -- start 2024-01-16T15:13:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.183+0000 7ff023e00640 1 -- start start 2024-01-16T15:13:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.184+0000 7ff023e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff0240cf050 0x7ff0240cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.184+0000 7ff023e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.184+0000 7ff023e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02415fa70 con 0x7ff02413dfd0 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff023e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240cfa40 con 0x7ff0240cf050 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff023e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02415c580 con 0x7ff0240cd7b0 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff0240cf050 0x7ff0240cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff0240cf050 0x7ff0240cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42540/0 (socket says 172.21.15.43:42540) 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 -- 172.21.15.43:0/3762129955 learned_addr learned my addr 172.21.15.43:0/3762129955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff023400640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 -- 172.21.15.43:0/3762129955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 msgr2=0x7ff0240cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 -- 172.21.15.43:0/3762129955 >> v1:172.21.15.69:6789/0 conn(0x7ff02413dfd0 legacy=0x7ff02415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.185+0000 7ff01be00640 1 -- 172.21.15.43:0/3762129955 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024140900 con 0x7ff0240cf050 2024-01-16T15:13:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.186+0000 7ff01be00640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0240cf050 0x7ff0240cf500 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff02406e070 tx=0x7ff00c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0acb79f73e77b9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff022000640 1 -- 172.21.15.43:0/3762129955 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c031830 con 0x7ff0240cf050 2024-01-16T15:13:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff022000640 1 -- 172.21.15.43:0/3762129955 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c031990 con 0x7ff0240cf050 2024-01-16T15:13:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff022000640 1 -- 172.21.15.43:0/3762129955 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c031d20 con 0x7ff0240cf050 2024-01-16T15:13:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 -- 172.21.15.43:0/3762129955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0240cf050 msgr2=0x7ff0240cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0240cf050 0x7ff0240cf500 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff02406e070 tx=0x7ff00c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 -- 172.21.15.43:0/3762129955 shutdown_connections 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0240cf050 0x7ff0240cf500 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 --2- 172.21.15.43:0/3762129955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.187+0000 7ff023e00640 1 -- 172.21.15.43:0/3762129955 >> 172.21.15.43:0/3762129955 conn(0x7ff02406a860 msgr2=0x7ff02406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.188+0000 7ff023e00640 1 -- 172.21.15.43:0/3762129955 shutdown_connections 2024-01-16T15:13:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.188+0000 7ff023e00640 1 -- 172.21.15.43:0/3762129955 wait complete. 2024-01-16T15:13:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.189+0000 7ff023e00640 1 Processor -- start 2024-01-16T15:13:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.190+0000 7ff023e00640 1 -- start start 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.190+0000 7ff023e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.190+0000 7ff023e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 0x7ff024155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b4080 con 0x7ff02413dfd0 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b2aa0 con 0x7ff0240cf050 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240b2d80 con 0x7ff0240cd7b0 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42554/0 (socket says 172.21.15.43:42554) 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 -- 172.21.15.43:0/973413586 learned_addr learned my addr 172.21.15.43:0/973413586 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff01be00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 0x7ff024155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff022a00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 -- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 msgr2=0x7ff0240b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 -- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 msgr2=0x7ff024155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 0x7ff024155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff023400640 1 -- 172.21.15.43:0/973413586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c0094d0 con 0x7ff02413dfd0 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.191+0000 7ff01be00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 0x7ff024155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff023400640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff01c00cd70 tx=0x7ff01c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c003b90 con 0x7ff02413dfd0 2024-01-16T15:13:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c003cf0 con 0x7ff02413dfd0 2024-01-16T15:13:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240b3060 con 0x7ff02413dfd0 2024-01-16T15:13:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c004240 con 0x7ff02413dfd0 2024-01-16T15:13:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.192+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240b3620 con 0x7ff02413dfd0 2024-01-16T15:13:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.193+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0240b3810 con 0x7ff02413dfd0 2024-01-16T15:13:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.194+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff01c0043a0 con 0x7ff02413dfd0 2024-01-16T15:13:17.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.196+0000 7ff01aa00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 0x7feffc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:17.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.196+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ff01c012070 con 0x7ff02413dfd0 2024-01-16T15:13:17.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.196+0000 7ff022a00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 0x7feffc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:17.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.197+0000 7ff022a00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 0x7feffc06e350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff01400a750 tx=0x7ff014008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.200+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff01c055860 con 0x7ff02413dfd0 2024-01-16T15:13:17.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.433+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff02404e840 con 0x7ff02413dfd0 2024-01-16T15:13:17.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.434+0000 7ff01aa00640 1 -- 172.21.15.43:0/973413586 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff01c058e80 con 0x7ff02413dfd0 2024-01-16T15:13:17.413 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:13:17.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.436+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 msgr2=0x7feffc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.436+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 0x7feffc06e350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff01400a750 tx=0x7ff014008040 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.436+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 msgr2=0x7ff0240b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.436+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff01c00cd70 tx=0x7ff01c00a660 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 shutdown_connections 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7feffc06be90 0x7feffc06e350 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0240cf050 0x7ff024155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff02413dfd0 0x7ff0240b3b40 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 --2- 172.21.15.43:0/973413586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0240cd7b0 0x7ff0240b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 >> 172.21.15.43:0/973413586 conn(0x7ff02406a860 msgr2=0x7ff02413cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 shutdown_connections 2024-01-16T15:13:17.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:17.437+0000 7ff023e00640 1 -- 172.21.15.43:0/973413586 wait complete. 2024-01-16T15:13:17.436 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:13:17.436 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:13:17.436 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:13:17.436 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:13:17.436 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:17.437 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:13:17.437 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:13:17.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:13:17.437 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:17.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.437 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:17.488 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41641sec preferred_lft 41641sec 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:13:17.489 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:13:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:17.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.491 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:13:17.569 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:13:17.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.573 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:13:17.627 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:13:17.627 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:13:17.628 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:13:17.628 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:13:17.628 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:13:17.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:13:17.628 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:13:17.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.824 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:13:17.878 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:13:17.878 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:13:17.879 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:17.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:17.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:13:17.981 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:13:17.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:17.981 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:18.032 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:18.032 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:18.032 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:18.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:13:18.183 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:13:18.239 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:13:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:18.241 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:18.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:18.242 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:13:18.294 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:13:18.294 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:13:18.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:13:18.385 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:18.432 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:18.433 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:13:18.434 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:18.434 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:18.434 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:18.434 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:18.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:18.435 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:13:18.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:13:18.513+0000 7ffbef04b480 -1 init, newargv = 0x557a96f03c10 newargc=15 2024-01-16T15:13:18.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[68697]: starting ceph client 2024-01-16T15:13:18.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[68697]: starting fuse 2024-01-16T15:13:19.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:13:19.691 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:13:19.751 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:19.752 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:19.753 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:19.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:19.754 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:13:19.809 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:13:19.809 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:13:19.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:13:19.810 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:13:19.811 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:13:19.979 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.68697.asok 2024-01-16T15:13:19.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:13:19.997 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68697.asok 2024-01-16T15:13:19.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:19.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68697.asok status 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "68697", 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:13:20.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 8253, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 8253 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 485040318 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:20.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 485040318 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.8253 192.168.0.1:0/485040318", 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/485040318", 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 57, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 90, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:13:20.247 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:13:20.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:20.262 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:13:20.262 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:20.314 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:13:20.314 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:20.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:20.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:20.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:13:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.663+0000 7fd3b5600640 1 Processor -- start 2024-01-16T15:13:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.663+0000 7fd3b5600640 1 -- start start 2024-01-16T15:13:20.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.664+0000 7fd3b5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3b015a100 0x7fd3b00b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.664+0000 7fd3b5600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b013da30 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.664+0000 7fd3b5600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b006e340 con 0x7fd3b00abff0 2024-01-16T15:13:20.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.664+0000 7fd3b5600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b00ac840 con 0x7fd3b015a100 2024-01-16T15:13:20.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3af400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd3b00ae7e0 0x7fd3b006dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33086/0 (socket says 172.21.15.43:33086) 2024-01-16T15:13:20.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3af400640 1 -- 172.21.15.43:0/3497040165 learned_addr learned my addr 172.21.15.43:0/3497040165 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:20.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3aea00640 1 --2- 172.21.15.43:0/3497040165 >> v2:172.21.15.69:3300/0 conn(0x7fd3b015a100 0x7fd3b00b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169910459 0 0) 0x7fd3b013da30 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd390003520 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1080218407 0 0) 0x7fd390003520 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 >> v2:172.21.15.69:3300/0 conn(0x7fd3b015a100 msgr2=0x7fd3b00b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 --2- 172.21.15.43:0/3497040165 >> v2:172.21.15.69:3300/0 conn(0x7fd3b015a100 0x7fd3b00b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 >> v1:172.21.15.69:6789/0 conn(0x7fd3b00abff0 legacy=0x7fd3b006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b00ac560 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.665+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd3a80019f0 con 0x7fd3b00ae7e0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 >> v1:172.21.15.43:6789/0 conn(0x7fd3b00ae7e0 legacy=0x7fd3b006dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd390003970 0x7fd390003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 0x7fd390004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd390005180 0x7fd39000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b00ac840 con 0x7fd3900047c0 2024-01-16T15:13:20.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39000c020 con 0x7fd390003970 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39000c1e0 con 0x7fd390005180 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 0x7fd390004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3aea00640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd390003970 0x7fd390003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 -- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd390005180 msgr2=0x7fd39000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd390005180 0x7fd39000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 -- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd390003970 msgr2=0x7fd390003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd390003970 0x7fd390003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3ae000640 1 -- 172.21.15.43:0/3497040165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390003520 con 0x7fd3900047c0 2024-01-16T15:13:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.666+0000 7fd3af400640 1 -- 172.21.15.43:0/3497040165 reap_dead start 2024-01-16T15:13:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.667+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 0x7fd390004c40 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fd398001750 tx=0x7fd3900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8480324de6ce865b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3980052d0 con 0x7fd3900047c0 2024-01-16T15:13:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39802e050 con 0x7fd3900047c0 2024-01-16T15:13:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3ad600640 1 -- 172.21.15.43:0/3497040165 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398003830 con 0x7fd3900047c0 2024-01-16T15:13:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 -- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 msgr2=0x7fd390004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 0x7fd390004c40 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fd398001750 tx=0x7fd3900057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 -- 172.21.15.43:0/3497040165 shutdown_connections 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3497040165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3900047c0 0x7fd390004c40 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 -- 172.21.15.43:0/3497040165 >> 172.21.15.43:0/3497040165 conn(0x7fd3b00b0d80 msgr2=0x7fd3b00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 -- 172.21.15.43:0/3497040165 shutdown_connections 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.668+0000 7fd3b5600640 1 -- 172.21.15.43:0/3497040165 wait complete. 2024-01-16T15:13:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.670+0000 7fd3b5600640 1 Processor -- start 2024-01-16T15:13:20.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.670+0000 7fd3b5600640 1 -- start start 2024-01-16T15:13:20.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3900047c0 0x7fd3b0098370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b015bd10 0x7fd3b00cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b006d7a0 con 0x7fd3b014c030 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b014a870 con 0x7fd3b015bd10 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.671+0000 7fd3b5600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b015b880 con 0x7fd3900047c0 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42576/0 (socket says 172.21.15.43:42576) 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 -- 172.21.15.43:0/3395978828 learned_addr learned my addr 172.21.15.43:0/3395978828 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3af400640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b015bd10 0x7fd3b00cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3aea00640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3900047c0 0x7fd3b0098370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 -- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3900047c0 msgr2=0x7fd3b0098370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3900047c0 0x7fd3b0098370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 -- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b015bd10 msgr2=0x7fd3b00cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b015bd10 0x7fd3b00cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.672+0000 7fd3ae000640 1 -- 172.21.15.43:0/3395978828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b015b270 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3ae000640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd398005040 tx=0x7fd398001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3980076c0 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398007a30 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398006be0 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b014a590 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.673+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b00b1910 con 0x7fd3b014c030 2024-01-16T15:13:20.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.674+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b015b0d0 con 0x7fd3b014c030 2024-01-16T15:13:20.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.675+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd398006d40 con 0x7fd3b014c030 2024-01-16T15:13:20.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.676+0000 7fd3a6a00640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 0x7fd38806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.676+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fd3980aa370 con 0x7fd3b014c030 2024-01-16T15:13:20.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.676+0000 7fd3aea00640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 0x7fd38806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.677+0000 7fd3aea00640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 0x7fd38806e350 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0068d90 tx=0x7fd3a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.681+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3b015b0d0 con 0x7fd3b014c030 2024-01-16T15:13:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.913+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd3b015b0d0 con 0x7fd3b014c030 2024-01-16T15:13:20.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.914+0000 7fd3a6a00640 1 -- 172.21.15.43:0/3395978828 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd3b015b0d0 con 0x7fd3b014c030 2024-01-16T15:13:20.893 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 msgr2=0x7fd38806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 0x7fd38806e350 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0068d90 tx=0x7fd3a0008040 comp rx=0 tx=0).stop 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 msgr2=0x7fd3b00b13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd398005040 tx=0x7fd398001120 comp rx=0 tx=0).stop 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 shutdown_connections 2024-01-16T15:13:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd38806be90 0x7fd38806e350 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3b015bd10 0x7fd3b00cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3b014c030 0x7fd3b00b13d0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 --2- 172.21.15.43:0/3395978828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3900047c0 0x7fd3b0098370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 >> 172.21.15.43:0/3395978828 conn(0x7fd3b00b0d80 msgr2=0x7fd3b0157440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 shutdown_connections 2024-01-16T15:13:20.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:20.917+0000 7fd3b5600640 1 -- 172.21.15.43:0/3395978828 wait complete. 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:13:20.913 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:13:20.914 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:20.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:20.914 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:13:20.972 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:13:20.972 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:13:20.972 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:20.973 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41798sec preferred_lft 41798sec 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.974 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:20.975 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:20.975 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:13:20.975 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:20.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:20.976 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:13:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:13:21.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.067 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:13:21.117 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:13:21.117 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:13:21.117 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:13:21.117 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:13:21.117 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:13:21.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.119 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:13:21.119 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:13:21.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:13:21.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:13:21.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:13:21.120 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:13:21.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.303 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:13:21.352 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:13:21.353 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:13:21.353 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:21.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:13:21.442 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:13:21.442 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:21.442 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:21.492 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:21.492 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:21.492 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:21.544 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:13:21.630 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:13:21.684 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:21.685 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:21.686 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:21.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.687 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:13:21.739 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:13:21.739 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:13:21.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:13:21.821 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:13:21.860 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:13:21.861 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:21.862 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:21.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.863 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:13:21.922 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:13:21.943+0000 7fb6d5967480 -1 init, newargv = 0x55f49d3c2c80 newargc=15 2024-01-16T15:13:21.922 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[45873]: starting ceph client 2024-01-16T15:13:21.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[45873]: starting fuse 2024-01-16T15:13:21.945 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:13:21.946 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:13:21.946 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:13:21.947 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:13:21.948 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:13:22.141 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.45873.asok 2024-01-16T15:13:22.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:13:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:13:22.157 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45873.asok 2024-01-16T15:13:22.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:22.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45873.asok status 2024-01-16T15:13:22.390 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "45873", 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6835, 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:13:22.391 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 6835 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3331910536 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3331910536 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:13:22.392 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.6835 192.168.0.1:0/3331910536", 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/3331910536", 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 57, 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 90, 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:13:22.393 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:13:22.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:22.406 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:13:22.406 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:22.456 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:13:22.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:22.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:22.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:22.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:13:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.797+0000 7f2ec5200640 1 Processor -- start 2024-01-16T15:13:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.797+0000 7f2ec5200640 1 -- start start 2024-01-16T15:13:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.798+0000 7f2ec5200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.798+0000 7f2ec5200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ec00b3210 0x7f2ec0167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.798+0000 7f2ec5200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec00b23e0 con 0x7f2ec00b48d0 2024-01-16T15:13:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.798+0000 7f2ec5200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00cdc70 con 0x7f2ec0160bd0 2024-01-16T15:13:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.798+0000 7f2ec5200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00b2870 con 0x7f2ec00b3210 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42590/0 (socket says 172.21.15.43:42590) 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 -- 172.21.15.43:0/2313338097 learned_addr learned my addr 172.21.15.43:0/2313338097 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebe800640 1 --2- 172.21.15.43:0/2313338097 >> v2:172.21.15.69:3300/0 conn(0x7f2ec00b3210 0x7f2ec0167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 -- 172.21.15.43:0/2313338097 >> v2:172.21.15.69:3300/0 conn(0x7f2ec00b3210 msgr2=0x7f2ec0167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 --2- 172.21.15.43:0/2313338097 >> v2:172.21.15.69:3300/0 conn(0x7f2ec00b3210 0x7f2ec0167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 -- 172.21.15.43:0/2313338097 >> v1:172.21.15.43:6789/0 conn(0x7f2ec00b48d0 legacy=0x7f2ec00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.799+0000 7f2ebde00640 1 -- 172.21.15.43:0/2313338097 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec00d41b0 con 0x7f2ec0160bd0 2024-01-16T15:13:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.800+0000 7f2ebde00640 1 --2- 172.21.15.43:0/2313338097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00d2e10 tx=0x7f2eb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f09cd597ceb9074 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.801+0000 7f2ebca00640 1 -- 172.21.15.43:0/2313338097 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0031870 con 0x7f2ec0160bd0 2024-01-16T15:13:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.801+0000 7f2ebca00640 1 -- 172.21.15.43:0/2313338097 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb00319d0 con 0x7f2ec0160bd0 2024-01-16T15:13:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.801+0000 7f2ebca00640 1 -- 172.21.15.43:0/2313338097 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0031d60 con 0x7f2ec0160bd0 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.801+0000 7f2ec5200640 1 -- 172.21.15.43:0/2313338097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec0160bd0 msgr2=0x7f2ec00b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.801+0000 7f2ec5200640 1 --2- 172.21.15.43:0/2313338097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00d2e10 tx=0x7f2eb002b910 comp rx=0 tx=0).stop 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 -- 172.21.15.43:0/2313338097 shutdown_connections 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 --2- 172.21.15.43:0/2313338097 >> v2:172.21.15.69:3300/0 conn(0x7f2ec00b3210 0x7f2ec0167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 --2- 172.21.15.43:0/2313338097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec0160bd0 0x7f2ec00b1ea0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 -- 172.21.15.43:0/2313338097 >> 172.21.15.43:0/2313338097 conn(0x7f2ec009c5a0 msgr2=0x7f2ec00d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 -- 172.21.15.43:0/2313338097 shutdown_connections 2024-01-16T15:13:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.802+0000 7f2ec5200640 1 -- 172.21.15.43:0/2313338097 wait complete. 2024-01-16T15:13:22.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.803+0000 7f2ec5200640 1 Processor -- start 2024-01-16T15:13:22.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.803+0000 7f2ec5200640 1 -- start start 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 0x7f2ec00b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ec0160bd0 0x7f2ec014e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00b83c0 con 0x7f2ec00b48d0 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00b8850 con 0x7f2ec00b3210 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.804+0000 7f2ec5200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00b77b0 con 0x7f2ec0160bd0 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42594/0 (socket says 172.21.15.43:42594) 2024-01-16T15:13:22.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 -- 172.21.15.43:0/3427594573 learned_addr learned my addr 172.21.15.43:0/3427594573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebde00640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 0x7f2ec00b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebe800640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ec0160bd0 0x7f2ec014e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 -- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ec0160bd0 msgr2=0x7f2ec014e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ec0160bd0 0x7f2ec014e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 -- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 msgr2=0x7f2ec00b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 0x7f2ec00b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebd400640 1 -- 172.21.15.43:0/3427594573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00094d0 con 0x7f2ec00b48d0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.805+0000 7f2ebde00640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 0x7f2ec00b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2ebd400640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c7a0 tx=0x7f2ea800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003b70 con 0x7f2ec00b48d0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8003cd0 con 0x7f2ec00b48d0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800aa30 con 0x7f2ec00b48d0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec00b7a90 con 0x7f2ec00b48d0 2024-01-16T15:13:22.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.806+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec00b8050 con 0x7f2ec00b48d0 2024-01-16T15:13:22.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.807+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec00b8570 con 0x7f2ec00b48d0 2024-01-16T15:13:22.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.808+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2ea800ab90 con 0x7f2ec00b48d0 2024-01-16T15:13:22.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.809+0000 7f2eb6000640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 0x7f2e9806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.809+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f2ea80885c0 con 0x7f2ec00b48d0 2024-01-16T15:13:22.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.809+0000 7f2ebde00640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 0x7f2e9806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.810+0000 7f2ebde00640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 0x7f2e9806e380 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0036000 tx=0x7f2eb003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:22.814+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ec00b8570 con 0x7f2ec00b48d0 2024-01-16T15:13:23.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.053+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2ec00a1210 con 0x7f2ec00b48d0 2024-01-16T15:13:23.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.281+0000 7f2eb6000640 1 -- 172.21.15.43:0/3427594573 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 136+0+0 (secure 0 0 0) 0x7f2ea8054c80 con 0x7f2ec00b48d0 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.286+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 msgr2=0x7f2e9806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.286+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 0x7f2e9806e380 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0036000 tx=0x7f2eb003c040 comp rx=0 tx=0).stop 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 msgr2=0x7f2ec014f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c7a0 tx=0x7f2ea800cc60 comp rx=0 tx=0).stop 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 shutdown_connections 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2e9806bec0 0x7f2e9806e380 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ec0160bd0 0x7f2ec014e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ec00b48d0 0x7f2ec014f570 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 --2- 172.21.15.43:0/3427594573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ec00b3210 0x7f2ec00b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 >> 172.21.15.43:0/3427594573 conn(0x7f2ec009c5a0 msgr2=0x7f2ec00ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 shutdown_connections 2024-01-16T15:13:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.287+0000 7f2ec5200640 1 -- 172.21.15.43:0/3427594573 wait complete. 2024-01-16T15:13:23.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:13:23.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.534+0000 7fb62c800640 1 Processor -- start 2024-01-16T15:13:23.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.534+0000 7fb62c800640 1 -- start start 2024-01-16T15:13:23.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.535+0000 7fb62c800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb62415d770 0x7fb6240d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.535+0000 7fb62c800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb6240d2740 0x7fb6240d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.535+0000 7fb62c800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6240d0b40 con 0x7fb6240d37b0 2024-01-16T15:13:23.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.535+0000 7fb62c800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6240d3c30 con 0x7fb6240d2740 2024-01-16T15:13:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.535+0000 7fb62c800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62415db50 con 0x7fb62415d770 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb6240d2740 0x7fb6240d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb6240d2740 0x7fb6240d3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42112/0 (socket says 172.21.15.43:42112) 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 -- 172.21.15.43:0/3985490485 learned_addr learned my addr 172.21.15.43:0/3985490485 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629e00640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb62415d770 0x7fb6240d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 -- 172.21.15.43:0/3985490485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb62415d770 msgr2=0x7fb6240d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb62415d770 0x7fb6240d2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 -- 172.21.15.43:0/3985490485 >> v1:172.21.15.69:6789/0 conn(0x7fb6240d37b0 legacy=0x7fb6240b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.536+0000 7fb629400640 1 -- 172.21.15.43:0/3985490485 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6240d0860 con 0x7fb6240d2740 2024-01-16T15:13:23.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.537+0000 7fb629400640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240d2740 0x7fb6240d3270 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009870 tx=0x7fb60c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209282b1e33b4084 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb628a00640 1 -- 172.21.15.43:0/3985490485 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c031640 con 0x7fb6240d2740 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb628a00640 1 -- 172.21.15.43:0/3985490485 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60c0317a0 con 0x7fb6240d2740 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb628a00640 1 -- 172.21.15.43:0/3985490485 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c031b30 con 0x7fb6240d2740 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 -- 172.21.15.43:0/3985490485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240d2740 msgr2=0x7fb6240d3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240d2740 0x7fb6240d3270 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009870 tx=0x7fb60c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 -- 172.21.15.43:0/3985490485 shutdown_connections 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240d2740 0x7fb6240d3270 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 --2- 172.21.15.43:0/3985490485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb62415d770 0x7fb6240d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 -- 172.21.15.43:0/3985490485 >> 172.21.15.43:0/3985490485 conn(0x7fb6240721a0 msgr2=0x7fb62406b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.538+0000 7fb62c800640 1 -- 172.21.15.43:0/3985490485 shutdown_connections 2024-01-16T15:13:23.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.539+0000 7fb62c800640 1 -- 172.21.15.43:0/3985490485 wait complete. 2024-01-16T15:13:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.540+0000 7fb62c800640 1 Processor -- start 2024-01-16T15:13:23.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.540+0000 7fb62c800640 1 -- start start 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6240d37b0 0x7fb624150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62415d770 0x7fb62415dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62415f920 con 0x7fb6240cc950 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6241604a0 con 0x7fb62415d770 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.541+0000 7fb62c800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb624150b60 con 0x7fb6240d37b0 2024-01-16T15:13:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42126/0 (socket says 172.21.15.43:42126) 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 -- 172.21.15.43:0/4153188473 learned_addr learned my addr 172.21.15.43:0/4153188473 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb629e00640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6240d37b0 0x7fb624150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb629400640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62415d770 0x7fb62415dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 -- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6240d37b0 msgr2=0x7fb624150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6240d37b0 0x7fb624150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 -- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62415d770 msgr2=0x7fb62415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62415d770 0x7fb62415dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.542+0000 7fb62a800640 1 -- 172.21.15.43:0/4153188473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0094d0 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb62a800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c7e0 tx=0x7fb61c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004b10 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c00ce90 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c00a850 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62415fb70 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.543+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb624160130 con 0x7fb6240cc950 2024-01-16T15:13:23.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.544+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb624160630 con 0x7fb6240cc950 2024-01-16T15:13:23.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.545+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb61c00a9b0 con 0x7fb6240cc950 2024-01-16T15:13:23.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.546+0000 7fb622000640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 0x7fb5fc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.547+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fb61c088920 con 0x7fb6240cc950 2024-01-16T15:13:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.547+0000 7fb629e00640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 0x7fb5fc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.547+0000 7fb629e00640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 0x7fb5fc06e350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.551+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb61c054f80 con 0x7fb6240cc950 2024-01-16T15:13:23.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.770+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6240d1ce0 con 0x7fb5fc06be90 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.772+0000 7fb622000640 1 -- 172.21.15.43:0/4153188473 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb61400c050 con 0x7fb5fc06be90 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:13:23.751 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.774+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 msgr2=0x7fb5fc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.774+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 0x7fb5fc06e350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614008040 comp rx=0 tx=0).stop 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.774+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 msgr2=0x7fb62415e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.774+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c7e0 tx=0x7fb61c00cca0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 shutdown_connections 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb5fc06be90 0x7fb5fc06e350 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62415d770 0x7fb62415dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6240cc950 0x7fb62415e230 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 --2- 172.21.15.43:0/4153188473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6240d37b0 0x7fb624150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 >> 172.21.15.43:0/4153188473 conn(0x7fb6240721a0 msgr2=0x7fb62415b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 shutdown_connections 2024-01-16T15:13:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:23.775+0000 7fb62c800640 1 -- 172.21.15.43:0/4153188473 wait complete. 2024-01-16T15:13:23.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:13:23.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.010+0000 7fd55fa00640 1 Processor -- start 2024-01-16T15:13:23.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.010+0000 7fd55fa00640 1 -- start start 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.011+0000 7fd55fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55815aae0 0x7fd55806cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.011+0000 7fd55fa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd55806d2a0 0x7fd5580b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55fa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd5580b2d70 0x7fd55815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55fa00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55815aec0 con 0x7fd5580b2d70 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55fa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd558154440 con 0x7fd55806d2a0 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55806d700 con 0x7fd55815aae0 2024-01-16T15:13:23.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd5580b2d70 0x7fd55815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd5580b2d70 0x7fd55815d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42140/0 (socket says 172.21.15.43:42140) 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 -- 172.21.15.43:0/2127623695 learned_addr learned my addr 172.21.15.43:0/2127623695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 -- 172.21.15.43:0/2127623695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55815aae0 msgr2=0x7fd55806cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 --2- 172.21.15.43:0/2127623695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55815aae0 0x7fd55806cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 -- 172.21.15.43:0/2127623695 >> v2:172.21.15.69:3300/0 conn(0x7fd55806d2a0 msgr2=0x7fd5580b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 --2- 172.21.15.43:0/2127623695 >> v2:172.21.15.69:3300/0 conn(0x7fd55806d2a0 0x7fd5580b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.012+0000 7fd55da00640 1 -- 172.21.15.43:0/2127623695 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558157b30 con 0x7fd5580b2d70 2024-01-16T15:13:23.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd55da00640 1 --2- 172.21.15.43:0/2127623695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5580b2d70 0x7fd55815d4e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fd5580b5130 tx=0x7fd55002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4716883b943b2337 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd557400640 1 -- 172.21.15.43:0/2127623695 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550031860 con 0x7fd5580b2d70 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd557400640 1 -- 172.21.15.43:0/2127623695 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5500319c0 con 0x7fd5580b2d70 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd557400640 1 -- 172.21.15.43:0/2127623695 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550031d50 con 0x7fd5580b2d70 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd55fa00640 1 -- 172.21.15.43:0/2127623695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5580b2d70 msgr2=0x7fd55815d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.014+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2127623695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5580b2d70 0x7fd55815d4e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fd5580b5130 tx=0x7fd55002b900 comp rx=0 tx=0).stop 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 -- 172.21.15.43:0/2127623695 shutdown_connections 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2127623695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5580b2d70 0x7fd55815d4e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2127623695 >> v2:172.21.15.69:3300/0 conn(0x7fd55806d2a0 0x7fd5580b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2127623695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55815aae0 0x7fd55806cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 -- 172.21.15.43:0/2127623695 >> 172.21.15.43:0/2127623695 conn(0x7fd5580b0ba0 msgr2=0x7fd5580b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 -- 172.21.15.43:0/2127623695 shutdown_connections 2024-01-16T15:13:23.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.015+0000 7fd55fa00640 1 -- 172.21.15.43:0/2127623695 wait complete. 2024-01-16T15:13:23.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.016+0000 7fd55fa00640 1 Processor -- start 2024-01-16T15:13:23.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.017+0000 7fd55fa00640 1 -- start start 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.017+0000 7fd55fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55806d2a0 0x7fd558155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.017+0000 7fd55fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5580b2d70 0x7fd55814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558155ae0 con 0x7fd55815aae0 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558154ed0 con 0x7fd5580b2d70 2024-01-16T15:13:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581551b0 con 0x7fd55806d2a0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42156/0 (socket says 172.21.15.43:42156) 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 -- 172.21.15.43:0/2129847656 learned_addr learned my addr 172.21.15.43:0/2129847656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 -- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55806d2a0 msgr2=0x7fd558155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55d000640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55806d2a0 0x7fd558155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55806d2a0 0x7fd558155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 -- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5580b2d70 msgr2=0x7fd55814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd557e00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5580b2d70 0x7fd55814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5580b2d70 0x7fd55814d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.018+0000 7fd55da00640 1 -- 172.21.15.43:0/2129847656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500094d0 con 0x7fd55815aae0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd55da00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fd55002b940 tx=0x7fd5500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55003adf0 con 0x7fd55815aae0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5500443f0 con 0x7fd55815aae0 2024-01-16T15:13:23.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5500447a0 con 0x7fd55815aae0 2024-01-16T15:13:24.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd558155490 con 0x7fd55815aae0 2024-01-16T15:13:24.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.019+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55814ded0 con 0x7fd55815aae0 2024-01-16T15:13:24.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.020+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd558067560 con 0x7fd55815aae0 2024-01-16T15:13:24.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.021+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd550044900 con 0x7fd55815aae0 2024-01-16T15:13:24.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.022+0000 7fd554c00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 0x7fd53006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.022+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fd5500aafe0 con 0x7fd55815aae0 2024-01-16T15:13:24.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.023+0000 7fd55d000640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 0x7fd53006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.023+0000 7fd55d000640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 0x7fd53006e3a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd54800a770 tx=0x7fd548008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.027+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5500776d0 con 0x7fd55815aae0 2024-01-16T15:13:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.223+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd55809d200 con 0x7fd55815aae0 2024-01-16T15:13:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.224+0000 7fd554c00640 1 -- 172.21.15.43:0/2129847656 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd55007acf0 con 0x7fd55815aae0 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.226+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 msgr2=0x7fd53006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.226+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 0x7fd53006e3a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd54800a770 tx=0x7fd548008040 comp rx=0 tx=0).stop 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.226+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 msgr2=0x7fd558154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.226+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fd55002b940 tx=0x7fd5500046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 shutdown_connections 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd53006bee0 0x7fd53006e3a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5580b2d70 0x7fd55814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd55815aae0 0x7fd558154580 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 --2- 172.21.15.43:0/2129847656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd55806d2a0 0x7fd558155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 >> 172.21.15.43:0/2129847656 conn(0x7fd5580b0ba0 msgr2=0x7fd55815b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 shutdown_connections 2024-01-16T15:13:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.227+0000 7fd55fa00640 1 -- 172.21.15.43:0/2129847656 wait complete. 2024-01-16T15:13:24.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:13:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.468+0000 7f1abc800640 1 Processor -- start 2024-01-16T15:13:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.468+0000 7f1abc800640 1 -- start start 2024-01-16T15:13:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.469+0000 7f1abc800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.469+0000 7f1abc800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b3210 0x7f1ab4167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.469+0000 7f1abc800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab40b23e0 con 0x7f1ab4160bd0 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.469+0000 7f1abc800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40cdc70 con 0x7f1ab40b48d0 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.469+0000 7f1abc800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40b2870 con 0x7f1ab40b3210 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42162/0 (socket says 172.21.15.43:42162) 2024-01-16T15:13:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 -- 172.21.15.43:0/1773003540 learned_addr learned my addr 172.21.15.43:0/1773003540 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1aba800640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b3210 0x7f1ab4167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 -- 172.21.15.43:0/1773003540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b3210 msgr2=0x7f1ab4167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b3210 0x7f1ab4167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 -- 172.21.15.43:0/1773003540 >> v1:172.21.15.69:6789/0 conn(0x7f1ab4160bd0 legacy=0x7f1ab40b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.470+0000 7f1ab9400640 1 -- 172.21.15.43:0/1773003540 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40d41b0 con 0x7f1ab40b48d0 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.471+0000 7f1ab9400640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009870 tx=0x7f1a9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee170d3e1e745447 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1ab8a00640 1 -- 172.21.15.43:0/1773003540 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c0317c0 con 0x7f1ab40b48d0 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1ab8a00640 1 -- 172.21.15.43:0/1773003540 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c031920 con 0x7f1ab40b48d0 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1ab8a00640 1 -- 172.21.15.43:0/1773003540 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c031cb0 con 0x7f1ab40b48d0 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 -- 172.21.15.43:0/1773003540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab40b48d0 msgr2=0x7f1ab40b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009870 tx=0x7f1a9c02b860 comp rx=0 tx=0).stop 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 -- 172.21.15.43:0/1773003540 shutdown_connections 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab40b48d0 0x7f1ab40b1ea0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 --2- 172.21.15.43:0/1773003540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b3210 0x7f1ab4167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.472+0000 7f1abc800640 1 -- 172.21.15.43:0/1773003540 >> 172.21.15.43:0/1773003540 conn(0x7f1ab409c5a0 msgr2=0x7f1ab40d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.473+0000 7f1abc800640 1 -- 172.21.15.43:0/1773003540 shutdown_connections 2024-01-16T15:13:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.473+0000 7f1abc800640 1 -- 172.21.15.43:0/1773003540 wait complete. 2024-01-16T15:13:24.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.474+0000 7f1abc800640 1 Processor -- start 2024-01-16T15:13:24.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.474+0000 7f1abc800640 1 -- start start 2024-01-16T15:13:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ab40b3210 0x7f1ab415d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b48d0 0x7f1ab40b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40cb810 con 0x7f1ab4160bd0 2024-01-16T15:13:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40b8320 con 0x7f1ab40b3210 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.475+0000 7f1abc800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40cb530 con 0x7f1ab40b48d0 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42166/0 (socket says 172.21.15.43:42166) 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 -- 172.21.15.43:0/2573599834 learned_addr learned my addr 172.21.15.43:0/2573599834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1ab9400640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b48d0 0x7f1ab40b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1ab9e00640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ab40b3210 0x7f1ab415d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 -- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b48d0 msgr2=0x7f1ab40b87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b48d0 0x7f1ab40b87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 -- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ab40b3210 msgr2=0x7f1ab415d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ab40b3210 0x7f1ab415d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.476+0000 7f1aba800640 1 -- 172.21.15.43:0/2573599834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0094d0 con 0x7f1ab4160bd0 2024-01-16T15:13:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1aba800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f1aac007e40 tx=0x7f1aac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac014620 con 0x7f1ab4160bd0 2024-01-16T15:13:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00ee90 con 0x7f1ab4160bd0 2024-01-16T15:13:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab40b7cd0 con 0x7f1ab4160bd0 2024-01-16T15:13:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac0149e0 con 0x7f1ab4160bd0 2024-01-16T15:13:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.477+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab40b7510 con 0x7f1ab4160bd0 2024-01-16T15:13:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.478+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab40d0980 con 0x7f1ab4160bd0 2024-01-16T15:13:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.479+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1aac014b40 con 0x7f1ab4160bd0 2024-01-16T15:13:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.480+0000 7f1ab2000640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 0x7f1a8c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.480+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f1aac088f70 con 0x7f1ab4160bd0 2024-01-16T15:13:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.480+0000 7f1ab9e00640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 0x7f1a8c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.481+0000 7f1ab9e00640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 0x7f1a8c06e350 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400a750 tx=0x7f1aa4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.485+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1aac055890 con 0x7f1ab4160bd0 2024-01-16T15:13:24.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.677+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f1ab40a1210 con 0x7f1ab4160bd0 2024-01-16T15:13:24.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.682+0000 7f1ab2000640 1 -- 172.21.15.43:0/2573599834 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v31)=0 v31) v1 ==== 131+0+0 (secure 0 0 0) 0x7f1aac058eb0 con 0x7f1ab4160bd0 2024-01-16T15:13:24.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 msgr2=0x7f1a8c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 0x7f1a8c06e350 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400a750 tx=0x7f1aa4008040 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 msgr2=0x7f1ab40caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f1aac007e40 tx=0x7f1aac00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 shutdown_connections 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1a8c06be90 0x7f1a8c06e350 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ab4160bd0 0x7f1ab40caa70 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ab40b48d0 0x7f1ab40b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 --2- 172.21.15.43:0/2573599834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ab40b3210 0x7f1ab415d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 >> 172.21.15.43:0/2573599834 conn(0x7f1ab409c5a0 msgr2=0x7f1ab40d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 shutdown_connections 2024-01-16T15:13:24.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.686+0000 7f1abc800640 1 -- 172.21.15.43:0/2573599834 wait complete. 2024-01-16T15:13:24.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:13:24.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.917+0000 7f40a9400640 1 Processor -- start 2024-01-16T15:13:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.917+0000 7f40a9400640 1 -- start start 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a415b350 0x7f40a40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40a409cb30 0x7f40a40b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f40a40adb60 0x7f40a40adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a415b730 con 0x7f40a409cb30 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40ac740 con 0x7f40a40adb60 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.919+0000 7f40a9400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40ac460 con 0x7f40a415b350 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40a409cb30 0x7f40a40b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40a409cb30 0x7f40a40b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42168/0 (socket says 172.21.15.43:42168) 2024-01-16T15:13:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 -- 172.21.15.43:0/3136854635 learned_addr learned my addr 172.21.15.43:0/3136854635 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f40a2800640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a415b350 0x7f40a40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f40a3200640 1 --2- 172.21.15.43:0/3136854635 >> v2:172.21.15.69:3300/0 conn(0x7f40a40adb60 0x7f40a40adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 -- 172.21.15.43:0/3136854635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a415b350 msgr2=0x7f40a40abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a415b350 0x7f40a40abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 -- 172.21.15.43:0/3136854635 >> v2:172.21.15.69:3300/0 conn(0x7f40a40adb60 msgr2=0x7f40a40adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 --2- 172.21.15.43:0/3136854635 >> v2:172.21.15.69:3300/0 conn(0x7f40a40adb60 0x7f40a40adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.920+0000 7f409be00640 1 -- 172.21.15.43:0/3136854635 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40b2a50 con 0x7f40a409cb30 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.922+0000 7f409be00640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a40b3530 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f40a4072730 tx=0x7f408c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913766915384050 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.922+0000 7f40a1e00640 1 -- 172.21.15.43:0/3136854635 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c031870 con 0x7f40a409cb30 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.922+0000 7f40a1e00640 1 -- 172.21.15.43:0/3136854635 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c0319d0 con 0x7f40a409cb30 2024-01-16T15:13:24.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.922+0000 7f40a1e00640 1 -- 172.21.15.43:0/3136854635 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c031d60 con 0x7f40a409cb30 2024-01-16T15:13:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 -- 172.21.15.43:0/3136854635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 msgr2=0x7f40a40b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a40b3530 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f40a4072730 tx=0x7f408c02b910 comp rx=0 tx=0).stop 2024-01-16T15:13:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 -- 172.21.15.43:0/3136854635 shutdown_connections 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 --2- 172.21.15.43:0/3136854635 >> v2:172.21.15.69:3300/0 conn(0x7f40a40adb60 0x7f40a40adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a40b3530 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 --2- 172.21.15.43:0/3136854635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a415b350 0x7f40a40abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 -- 172.21.15.43:0/3136854635 >> 172.21.15.43:0/3136854635 conn(0x7f40a40b0ba0 msgr2=0x7f40a40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 -- 172.21.15.43:0/3136854635 shutdown_connections 2024-01-16T15:13:24.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.923+0000 7f40a9400640 1 -- 172.21.15.43:0/3136854635 wait complete. 2024-01-16T15:13:24.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.925+0000 7f40a9400640 1 Processor -- start 2024-01-16T15:13:24.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.925+0000 7f40a9400640 1 -- start start 2024-01-16T15:13:24.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a40adb60 0x7f40a4071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 0x7f40a41e1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4154df0 con 0x7f40a409cb30 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4154170 con 0x7f40a415b350 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a9400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4154450 con 0x7f40a40adb60 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42174/0 (socket says 172.21.15.43:42174) 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.926+0000 7f40a2800640 1 -- 172.21.15.43:0/3440494584 learned_addr learned my addr 172.21.15.43:0/3440494584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f409be00640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a40adb60 0x7f40a4071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a3200640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 0x7f40a41e1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 -- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a40adb60 msgr2=0x7f40a4071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a40adb60 0x7f40a4071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 -- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 msgr2=0x7f40a41e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 0x7f40a41e1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 -- 172.21.15.43:0/3440494584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c0094d0 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a3200640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 0x7f40a41e1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f40a2800640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f409400d800 tx=0x7f409400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409400b6c0 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.927+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409400b820 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.928+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409400bbb0 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.928+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a4154730 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.928+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a4153ce0 con 0x7f40a409cb30 2024-01-16T15:13:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.929+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a409d200 con 0x7f40a409cb30 2024-01-16T15:13:24.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.930+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f409400bd10 con 0x7f40a409cb30 2024-01-16T15:13:24.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.931+0000 7f409aa00640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 0x7f407c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.931+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f4094088d50 con 0x7f40a409cb30 2024-01-16T15:13:24.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.931+0000 7f409be00640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 0x7f407c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.932+0000 7f409be00640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 0x7f407c06e350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f408c036000 tx=0x7f408c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:24.935+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4094055410 con 0x7f40a409cb30 2024-01-16T15:13:25.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.135+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f40a404e920 con 0x7f40a409cb30 2024-01-16T15:13:25.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.142+0000 7f409aa00640 1 -- 172.21.15.43:0/3440494584 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v32)=0 v32) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4094058a30 con 0x7f40a409cb30 2024-01-16T15:13:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 msgr2=0x7f407c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 0x7f407c06e350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f408c036000 tx=0x7f408c03c040 comp rx=0 tx=0).stop 2024-01-16T15:13:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 msgr2=0x7f40a4155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f409400d800 tx=0x7f409400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 shutdown_connections 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f407c06be90 0x7f407c06e350 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a415b350 0x7f40a41e1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40a40adb60 0x7f40a4071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 --2- 172.21.15.43:0/3440494584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40a409cb30 0x7f40a4155210 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.146+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 >> 172.21.15.43:0/3440494584 conn(0x7f40a40b0ba0 msgr2=0x7f40a4156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.147+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 shutdown_connections 2024-01-16T15:13:25.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.147+0000 7f40a9400640 1 -- 172.21.15.43:0/3440494584 wait complete. 2024-01-16T15:13:25.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:13:25.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.381+0000 7f6c96600640 1 Processor -- start 2024-01-16T15:13:25.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.381+0000 7f6c96600640 1 -- start start 2024-01-16T15:13:25.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c96600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c90144770 0x7f6c9015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c96600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6c900b35a0 0x7f6c9015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c96600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9015f930 con 0x7f6c9006c9d0 2024-01-16T15:13:25.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c96600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c900b43b0 con 0x7f6c900b35a0 2024-01-16T15:13:25.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c96600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c900b3b10 con 0x7f6c90144770 2024-01-16T15:13:25.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c8fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6c900b35a0 0x7f6c9015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c95c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6c9006c9d0 0x7f6c9006ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60274/0 (socket says 172.21.15.43:60274) 2024-01-16T15:13:25.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.383+0000 7f6c95c00640 1 -- 172.21.15.43:0/8318737 learned_addr learned my addr 172.21.15.43:0/8318737 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:25.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c95200640 1 --2- 172.21.15.43:0/8318737 >> v2:172.21.15.69:3300/0 conn(0x7f6c90144770 0x7f6c9015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c8fe00640 1 -- 172.21.15.43:0/8318737 >> v2:172.21.15.69:3300/0 conn(0x7f6c90144770 msgr2=0x7f6c9015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c8fe00640 1 --2- 172.21.15.43:0/8318737 >> v2:172.21.15.69:3300/0 conn(0x7f6c90144770 0x7f6c9015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c8fe00640 1 -- 172.21.15.43:0/8318737 >> v1:172.21.15.43:6789/0 conn(0x7f6c9006c9d0 legacy=0x7f6c9006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c8fe00640 1 -- 172.21.15.43:0/8318737 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900b3df0 con 0x7f6c900b35a0 2024-01-16T15:13:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.384+0000 7f6c8f400640 1 -- 172.21.15.43:0/8318737 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215329259 0 0) 0x7f6c9015f930 con 0x7f6c9006c9d0 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.385+0000 7f6c8fe00640 1 --2- 172.21.15.43:0/8318737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c900b35a0 0x7f6c9015f0a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6c9006df90 tx=0x7f6c7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdff629a9b50463 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.385+0000 7f6c8f400640 1 -- 172.21.15.43:0/8318737 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78031830 con 0x7f6c900b35a0 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.385+0000 7f6c8f400640 1 -- 172.21.15.43:0/8318737 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c78031990 con 0x7f6c900b35a0 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.385+0000 7f6c8f400640 1 -- 172.21.15.43:0/8318737 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78031d20 con 0x7f6c900b35a0 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 -- 172.21.15.43:0/8318737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c900b35a0 msgr2=0x7f6c9015f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 --2- 172.21.15.43:0/8318737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c900b35a0 0x7f6c9015f0a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6c9006df90 tx=0x7f6c7802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 -- 172.21.15.43:0/8318737 shutdown_connections 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 --2- 172.21.15.43:0/8318737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c900b35a0 0x7f6c9015f0a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 --2- 172.21.15.43:0/8318737 >> v2:172.21.15.69:3300/0 conn(0x7f6c90144770 0x7f6c9015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 -- 172.21.15.43:0/8318737 >> 172.21.15.43:0/8318737 conn(0x7f6c900cec10 msgr2=0x7f6c900cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 -- 172.21.15.43:0/8318737 shutdown_connections 2024-01-16T15:13:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.386+0000 7f6c96600640 1 -- 172.21.15.43:0/8318737 wait complete. 2024-01-16T15:13:25.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.389+0000 7f6c96600640 1 Processor -- start 2024-01-16T15:13:25.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.389+0000 7f6c96600640 1 -- start start 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c900b35a0 0x7f6c9015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c90144770 0x7f6c900719a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f76d0 con 0x7f6c9006c9d0 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f6ac0 con 0x7f6c90144770 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.390+0000 7f6c96600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f6da0 con 0x7f6c900b35a0 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42200/0 (socket says 172.21.15.43:42200) 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 -- 172.21.15.43:0/3044077002 learned_addr learned my addr 172.21.15.43:0/3044077002 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:25.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c8fe00640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c900b35a0 0x7f6c9015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95c00640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c90144770 0x7f6c900719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 -- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c900b35a0 msgr2=0x7f6c9015a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c900b35a0 0x7f6c9015a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 -- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c90144770 msgr2=0x7f6c900719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c90144770 0x7f6c900719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.391+0000 7f6c95200640 1 -- 172.21.15.43:0/3044077002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78009400 con 0x7f6c9006c9d0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c95200640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000d5f0 tx=0x7f6c8000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c800125f0 con 0x7f6c9006c9d0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80012750 con 0x7f6c9006c9d0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80012b00 con 0x7f6c9006c9d0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c901f7080 con 0x7f6c9006c9d0 2024-01-16T15:13:25.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.392+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9015aa30 con 0x7f6c9006c9d0 2024-01-16T15:13:25.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.393+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9004e840 con 0x7f6c9006c9d0 2024-01-16T15:13:25.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.394+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6c80012c60 con 0x7f6c9006c9d0 2024-01-16T15:13:25.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.395+0000 7f6c8cc00640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 0x7f6c6806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.396+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f6c80088d00 con 0x7f6c9006c9d0 2024-01-16T15:13:25.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.396+0000 7f6c8fe00640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 0x7f6c6806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.397+0000 7f6c8fe00640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 0x7f6c6806e350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6c78036000 tx=0x7f6c7803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.400+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c80055360 con 0x7f6c9006c9d0 2024-01-16T15:13:25.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.620+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c90144b50 con 0x7f6c6806be90 2024-01-16T15:13:25.608 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:13:25.628+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:25.608 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:13:25.628+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:25.608 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:13:25.628+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:25.608 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:13:25.628+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:25.608 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:13:25.628+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:25.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.680+0000 7f6c8cc00640 1 -- 172.21.15.43:0/3044077002 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6c7803a420 con 0x7f6c6806be90 2024-01-16T15:13:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 msgr2=0x7f6c6806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 0x7f6c6806e350 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6c78036000 tx=0x7f6c7803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 msgr2=0x7f6c901414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000d5f0 tx=0x7f6c8000dab0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 shutdown_connections 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6c6806be90 0x7f6c6806e350 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c90144770 0x7f6c900719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c900b35a0 0x7f6c9015a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 --2- 172.21.15.43:0/3044077002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c9006c9d0 0x7f6c901414c0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 >> 172.21.15.43:0/3044077002 conn(0x7f6c900cec10 msgr2=0x7f6c9013e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.683+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 shutdown_connections 2024-01-16T15:13:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.684+0000 7f6c96600640 1 -- 172.21.15.43:0/3044077002 wait complete. 2024-01-16T15:13:25.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:13:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.918+0000 7fd304a00640 1 Processor -- start 2024-01-16T15:13:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.918+0000 7fd304a00640 1 -- start start 2024-01-16T15:13:25.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd304a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd30006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd304a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000d0660 con 0x7fd3000cc010 2024-01-16T15:13:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd304a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000d0380 con 0x7fd30015d730 2024-01-16T15:13:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd304a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000cf4e0 con 0x7fd3000cf080 2024-01-16T15:13:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd2ffe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd3000cc010 0x7fd3000cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60282/0 (socket says 172.21.15.43:60282) 2024-01-16T15:13:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.919+0000 7fd2ffe00640 1 -- 172.21.15.43:0/3204012438 learned_addr learned my addr 172.21.15.43:0/3204012438 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fea00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd30006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080367800 0 0) 0x7fd3000d0660 con 0x7fd3000cc010 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0003520 con 0x7fd3000cc010 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692894943 0 0) 0x7fd2e0003520 con 0x7fd3000cc010 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 msgr2=0x7fd30006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd30006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 >> v1:172.21.15.69:6789/0 conn(0x7fd30015d730 legacy=0x7fd30006b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000cb530 con 0x7fd3000cc010 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fd2f8001670 con 0x7fd3000cc010 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 >> v1:172.21.15.43:6789/0 conn(0x7fd3000cc010 legacy=0x7fd3000cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0003a10 0x7fd2e0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.920+0000 7fd2fe000640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd2e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2fe000640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2e0004d00 0x7fd2e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000cf4e0 con 0x7fd2e0004d00 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e000bbb0 con 0x7fd2e0003a10 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e000bd70 con 0x7fd3000cf080 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2e0004d00 0x7fd2e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ff400640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0003a10 0x7fd2e0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2fea00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd2e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 -- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 msgr2=0x7fd2e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000cf080 0x7fd2e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 -- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0003a10 msgr2=0x7fd2e0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0003a10 0x7fd2e0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ffe00640 1 -- 172.21.15.43:0/3204012438 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0003520 con 0x7fd2e0004d00 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.921+0000 7fd2ff400640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0003a10 0x7fd2e0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.922+0000 7fd2ffe00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2e0004d00 0x7fd2e0005180 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800c240 tx=0x7fd2e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad26cacd670086b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8005ad0 con 0x7fd2e0004d00 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f8005c30 con 0x7fd2e0004d00 2024-01-16T15:13:25.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd2fe000640 1 -- 172.21.15.43:0/3204012438 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f802c8b0 con 0x7fd2e0004d00 2024-01-16T15:13:25.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd304a00640 1 -- 172.21.15.43:0/3204012438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2e0004d00 msgr2=0x7fd2e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd304a00640 1 --2- 172.21.15.43:0/3204012438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2e0004d00 0x7fd2e0005180 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800c240 tx=0x7fd2e00057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd2ffe00640 1 -- 172.21.15.43:0/3204012438 reap_dead start 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd304a00640 1 -- 172.21.15.43:0/3204012438 shutdown_connections 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.923+0000 7fd304a00640 1 -- 172.21.15.43:0/3204012438 >> 172.21.15.43:0/3204012438 conn(0x7fd3000d0e40 msgr2=0x7fd3000d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.924+0000 7fd304a00640 1 -- 172.21.15.43:0/3204012438 shutdown_connections 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.924+0000 7fd304a00640 1 -- 172.21.15.43:0/3204012438 wait complete. 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.925+0000 7fd304a00640 1 Processor -- start 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.926+0000 7fd304a00640 1 -- start start 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.926+0000 7fd304a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0004d00 0x7fd3000c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.926+0000 7fd304a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd304a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000c7210 0x7fd30014e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd304a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30015ead0 con 0x7fd3000c6d90 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd304a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000c48a0 con 0x7fd2e0004d00 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd304a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000c4b80 con 0x7fd3000c7210 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42234/0 (socket says 172.21.15.43:42234) 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 -- 172.21.15.43:0/3193396984 learned_addr learned my addr 172.21.15.43:0/3193396984 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 -- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000c7210 msgr2=0x7fd30014e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000c7210 0x7fd30014e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 -- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0004d00 msgr2=0x7fd3000c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0004d00 0x7fd3000c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 -- 172.21.15.43:0/3193396984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000c4e60 con 0x7fd3000c6d90 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.927+0000 7fd2fea00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8007f30 tx=0x7fd2e8011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.928+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800e030 con 0x7fd3000c6d90 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.928+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8016470 con 0x7fd3000c6d90 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.928+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8016820 con 0x7fd3000c6d90 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.928+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3000c5140 con 0x7fd3000c6d90 2024-01-16T15:13:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.928+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30015dd60 con 0x7fd3000c6d90 2024-01-16T15:13:25.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.929+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30004e840 con 0x7fd3000c6d90 2024-01-16T15:13:25.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.930+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd2e8016980 con 0x7fd3000c6d90 2024-01-16T15:13:25.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.931+0000 7fd2f7400640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 0x7fd2d806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.931+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fd2e800f070 con 0x7fd3000c6d90 2024-01-16T15:13:25.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.931+0000 7fd2ff400640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 0x7fd2d806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.932+0000 7fd2ff400640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 0x7fd2d806e380 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000f60 tx=0x7fd2f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:25.935+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2e8055d90 con 0x7fd3000c6d90 2024-01-16T15:13:26.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.159+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3000d0070 con 0x7fd2d806bec0 2024-01-16T15:13:26.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.172+0000 7fd2f7400640 1 -- 172.21.15.43:0/3193396984 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd2f00050d0 con 0x7fd2d806bec0 2024-01-16T15:13:26.151 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:13:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.174+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 msgr2=0x7fd2d806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.174+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 0x7fd2d806e380 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000f60 tx=0x7fd2f0009040 comp rx=0 tx=0).stop 2024-01-16T15:13:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 msgr2=0x7fd30014de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8007f30 tx=0x7fd2e8011660 comp rx=0 tx=0).stop 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 shutdown_connections 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd2d806bec0 0x7fd2d806e380 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3000c7210 0x7fd30014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3000c6d90 0x7fd30014de80 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 --2- 172.21.15.43:0/3193396984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e0004d00 0x7fd3000c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 >> 172.21.15.43:0/3193396984 conn(0x7fd3000d0e40 msgr2=0x7fd3000af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 shutdown_connections 2024-01-16T15:13:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.175+0000 7fd304a00640 1 -- 172.21.15.43:0/3193396984 wait complete. 2024-01-16T15:13:26.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:13:26.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.398+0000 7f554be00640 1 Processor -- start 2024-01-16T15:13:26.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.398+0000 7f554be00640 1 -- start start 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f554be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5544160b10 0x7f554406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f554be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f554406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f554be00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5544071780 con 0x7f554406c900 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f554be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55440b2320 con 0x7f5544160b10 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f554be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554406cd90 con 0x7f5544072280 2024-01-16T15:13:26.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f5549e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f554406c900 0x7f5544160fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60292/0 (socket says 172.21.15.43:60292) 2024-01-16T15:13:26.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f5549e00640 1 -- 172.21.15.43:0/1397471969 learned_addr learned my addr 172.21.15.43:0/1397471969 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:13:26.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f5549400640 1 --2- 172.21.15.43:0/1397471969 >> v2:172.21.15.69:3300/0 conn(0x7f5544160b10 0x7f554406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.400+0000 7f5548a00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f554406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670521200 0 0) 0x7f5544071780 con 0x7f554406c900 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5524003520 con 0x7f554406c900 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039223701 0 0) 0x7f5524003520 con 0x7f554406c900 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 msgr2=0x7f554406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f554406c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 >> v2:172.21.15.69:3300/0 conn(0x7f5544160b10 msgr2=0x7f554406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 --2- 172.21.15.43:0/1397471969 >> v2:172.21.15.69:3300/0 conn(0x7f5544160b10 0x7f554406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440711c0 con 0x7f554406c900 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f553c001670 con 0x7f554406c900 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 >> v1:172.21.15.43:6789/0 conn(0x7f554406c900 legacy=0x7f5544160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524003970 0x7f5524003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f55240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5524004d00 0x7f5524005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440b2320 con 0x7f5524004d00 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554406cd90 con 0x7f5524003970 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.401+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552400b9b0 con 0x7f5544072280 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5524004d00 0x7f5524005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5548a00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f55240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549400640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524003970 0x7f5524003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 -- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 msgr2=0x7f55240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5544072280 0x7f55240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 -- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524003970 msgr2=0x7f5524003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524003970 0x7f5524003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.402+0000 7f5549e00640 1 -- 172.21.15.43:0/1397471969 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524003520 con 0x7f5524004d00 2024-01-16T15:13:26.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.403+0000 7f5549e00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5524004d00 0x7f5524005180 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c160 tx=0x7f55240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c753c7717d4add4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.403+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c005ad0 con 0x7f5524004d00 2024-01-16T15:13:26.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.403+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c005c30 con 0x7f5524004d00 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.403+0000 7f5543e00640 1 -- 172.21.15.43:0/1397471969 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c02c810 con 0x7f5524004d00 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 -- 172.21.15.43:0/1397471969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5524004d00 msgr2=0x7f5524005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 --2- 172.21.15.43:0/1397471969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5524004d00 0x7f5524005180 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c160 tx=0x7f55240057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f5549e00640 1 -- 172.21.15.43:0/1397471969 reap_dead start 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 -- 172.21.15.43:0/1397471969 shutdown_connections 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 -- 172.21.15.43:0/1397471969 >> 172.21.15.43:0/1397471969 conn(0x7f55440b0ba0 msgr2=0x7f55440b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 -- 172.21.15.43:0/1397471969 shutdown_connections 2024-01-16T15:13:26.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.404+0000 7f554be00640 1 -- 172.21.15.43:0/1397471969 wait complete. 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.406+0000 7f554be00640 1 Processor -- start 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.406+0000 7f554be00640 1 -- start start 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 0x7f55440ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 0x7f5544152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554415dad0 con 0x7f55441519d0 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440c9a00 con 0x7f5524004d00 2024-01-16T15:13:26.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f554be00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55440c9ce0 con 0x7f554415d650 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f5548a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f5548a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42262/0 (socket says 172.21.15.43:42262) 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.407+0000 7f5548a00640 1 -- 172.21.15.43:0/2325969489 learned_addr learned my addr 172.21.15.43:0/2325969489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5549e00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 0x7f5544152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5549400640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 0x7f55440ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 -- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 msgr2=0x7f5544152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 0x7f5544152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 -- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 msgr2=0x7f55440ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 0x7f55440ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 -- 172.21.15.43:0/2325969489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440c9fc0 con 0x7f55441519d0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5549e00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 0x7f5544152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5549400640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 0x7f55440ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5548a00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f552c011600 tx=0x7f552c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c00f920 con 0x7f55441519d0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c00fa80 con 0x7f55441519d0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c00fdf0 con 0x7f55441519d0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.408+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55440ca2a0 con 0x7f55441519d0 2024-01-16T15:13:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.409+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5544152030 con 0x7f55441519d0 2024-01-16T15:13:26.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.410+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55440ca430 con 0x7f55441519d0 2024-01-16T15:13:26.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.411+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 31) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f552c015460 con 0x7f55441519d0 2024-01-16T15:13:26.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.411+0000 7f5541600640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 0x7f551c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.412+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f552c089040 con 0x7f55441519d0 2024-01-16T15:13:26.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.412+0000 7f5549400640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 0x7f551c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.412+0000 7f5549400640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 0x7f551c06e3a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5534007440 tx=0x7f553400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.416+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f552c0556e0 con 0x7f55441519d0 2024-01-16T15:13:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.638+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f55440b1f40 con 0x7f551c06bee0 2024-01-16T15:13:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.645+0000 7f5541600640 1 -- 172.21.15.43:0/2325969489 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f5534018470 con 0x7f551c06bee0 2024-01-16T15:13:26.624 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 msgr2=0x7f551c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 0x7f551c06e3a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5534007440 tx=0x7f553400b040 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 msgr2=0x7f554415d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f552c011600 tx=0x7f552c011ac0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 shutdown_connections 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f551c06bee0 0x7f551c06e3a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f554415d650 0x7f5544152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55441519d0 0x7f554415d110 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 --2- 172.21.15.43:0/2325969489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5524004d00 0x7f55440ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.647+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 >> 172.21.15.43:0/2325969489 conn(0x7f55440b0ba0 msgr2=0x7f554415fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.648+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 shutdown_connections 2024-01-16T15:13:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:13:26.648+0000 7f554be00640 1 -- 172.21.15.43:0/2325969489 wait complete. 2024-01-16T15:13:26.645 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:13:26.645 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:13:26.645 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:26.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:26.696 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:13:28.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:28.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:43.492 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:13:43.492 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:13:43.492 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.7119 s, 64.3 MB/s 2024-01-16T15:13:43.494 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:13:43.494 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:43.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:43.552 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:13:57.780 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:13:57.780 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:13:57.780 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.1651 s, 75.8 MB/s 2024-01-16T15:13:57.782 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:13:57.782 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:57.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:57.841 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:13:58.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:58.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:11.946 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:14:11.947 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:14:11.947 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.0424 s, 76.5 MB/s 2024-01-16T15:14:11.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:14:12.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.198+0000 7fb193e00640 1 Processor -- start 2024-01-16T15:14:12.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.199+0000 7fb193e00640 1 -- start start 2024-01-16T15:14:12.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb193e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1940d1fd0 0x7fb1940ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb193e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 0x7fb1940b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb193e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1940b2980 con 0x7fb1940cfa30 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb193e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1940cf220 con 0x7fb1940d1fd0 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb193e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1940cf500 con 0x7fb1940b33d0 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb18be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1940d1fd0 0x7fb1940ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb192a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb1940cfa30 0x7fb1940cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34776/0 (socket says 172.21.15.43:34776) 2024-01-16T15:14:12.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.200+0000 7fb192a00640 1 -- 172.21.15.43:0/2989971889 learned_addr learned my addr 172.21.15.43:0/2989971889 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:12.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb193400640 1 --2- 172.21.15.43:0/2989971889 >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 0x7fb1940b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb18be00640 1 -- 172.21.15.43:0/2989971889 >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 msgr2=0x7fb1940b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:12.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb18be00640 1 --2- 172.21.15.43:0/2989971889 >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 0x7fb1940b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb18be00640 1 -- 172.21.15.43:0/2989971889 >> v1:172.21.15.43:6789/0 conn(0x7fb1940cfa30 legacy=0x7fb1940cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb18be00640 1 -- 172.21.15.43:0/2989971889 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1940b2bd0 con 0x7fb1940d1fd0 2024-01-16T15:14:12.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.201+0000 7fb193400640 1 --2- 172.21.15.43:0/2989971889 >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 0x7fb1940b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.202+0000 7fb18be00640 1 --2- 172.21.15.43:0/2989971889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940d1fd0 0x7fb1940ce920 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ca080 tx=0x7fb17c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd99f24d15bc845 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.202+0000 7fb192000640 1 -- 172.21.15.43:0/2989971889 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c031830 con 0x7fb1940d1fd0 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb192000640 1 -- 172.21.15.43:0/2989971889 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c031990 con 0x7fb1940d1fd0 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb192000640 1 -- 172.21.15.43:0/2989971889 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c031d20 con 0x7fb1940d1fd0 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 -- 172.21.15.43:0/2989971889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940d1fd0 msgr2=0x7fb1940ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 --2- 172.21.15.43:0/2989971889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940d1fd0 0x7fb1940ce920 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ca080 tx=0x7fb17c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 -- 172.21.15.43:0/2989971889 shutdown_connections 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 --2- 172.21.15.43:0/2989971889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940d1fd0 0x7fb1940ce920 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 --2- 172.21.15.43:0/2989971889 >> v2:172.21.15.69:3300/0 conn(0x7fb1940b33d0 0x7fb1940b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.203+0000 7fb193e00640 1 -- 172.21.15.43:0/2989971889 >> 172.21.15.43:0/2989971889 conn(0x7fb1940d3020 msgr2=0x7fb1940d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.204+0000 7fb193e00640 1 -- 172.21.15.43:0/2989971889 shutdown_connections 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.204+0000 7fb193e00640 1 -- 172.21.15.43:0/2989971889 wait complete. 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.205+0000 7fb193e00640 1 Processor -- start 2024-01-16T15:14:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.205+0000 7fb193e00640 1 -- start start 2024-01-16T15:14:12.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1940b33d0 0x7fb1941f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 0x7fb1940ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940cb870 con 0x7fb1940cfa30 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19415dcb0 con 0x7fb1940b33d0 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.206+0000 7fb193e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19415df90 con 0x7fb1940d1fd0 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51282/0 (socket says 172.21.15.43:51282) 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 -- 172.21.15.43:0/590818197 learned_addr learned my addr 172.21.15.43:0/590818197 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb192a00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1940b33d0 0x7fb1941f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb193400640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 0x7fb1940ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 -- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 msgr2=0x7fb1940ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 0x7fb1940ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 -- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1940b33d0 msgr2=0x7fb1941f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1940b33d0 0x7fb1941f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb18be00640 1 -- 172.21.15.43:0/590818197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c009400 con 0x7fb1940cfa30 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.207+0000 7fb193400640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 0x7fb1940ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb18be00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009890 tx=0x7fb17c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c03ba70 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c03bbf0 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0453f0 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19415e270 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.208+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1941f2a20 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.209+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1940d0a40 con 0x7fb1940cfa30 2024-01-16T15:14:12.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.210+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 32) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb17c045550 con 0x7fb1940cfa30 2024-01-16T15:14:12.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.211+0000 7fb18aa00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 0x7fb16c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.211+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fb17c0aab90 con 0x7fb1940cfa30 2024-01-16T15:14:12.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.211+0000 7fb192a00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 0x7fb16c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.212+0000 7fb192a00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 0x7fb16c06e3e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb184005880 tx=0x7fb184009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.216+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb17c0772f0 con 0x7fb1940cfa30 2024-01-16T15:14:12.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.451+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb19415de20 con 0x7fb16c06bf20 2024-01-16T15:14:12.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.482+0000 7fb18aa00640 1 -- 172.21.15.43:0/590818197 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb19415de20 con 0x7fb16c06bf20 2024-01-16T15:14:12.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.484+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 msgr2=0x7fb16c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.484+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 0x7fb16c06e3e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb184005880 tx=0x7fb184009040 comp rx=0 tx=0).stop 2024-01-16T15:14:12.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.484+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 msgr2=0x7fb1940cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.484+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009890 tx=0x7fb17c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 shutdown_connections 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb16c06bf20 0x7fb16c06e3e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1940d1fd0 0x7fb1940ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1940cfa30 0x7fb1940cddd0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 --2- 172.21.15.43:0/590818197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1940b33d0 0x7fb1941f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 >> 172.21.15.43:0/590818197 conn(0x7fb1940d3020 msgr2=0x7fb19406bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 shutdown_connections 2024-01-16T15:14:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.485+0000 7fb193e00640 1 -- 172.21.15.43:0/590818197 wait complete. 2024-01-16T15:14:12.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:14:12.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.730+0000 7f22e5000640 1 Processor -- start 2024-01-16T15:14:12.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.730+0000 7f22e5000640 1 -- start start 2024-01-16T15:14:12.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22e5000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f22e00b43e0 0x7f22e0168090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22e5000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e00b2980 con 0x7f22e00d1fd0 2024-01-16T15:14:12.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22e5000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e00cf220 con 0x7f22e00cfa30 2024-01-16T15:14:12.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22e5000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e00cf030 con 0x7f22e00b43e0 2024-01-16T15:14:12.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f22e00b43e0 0x7f22e0168090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f22e00b43e0 0x7f22e0168090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51288/0 (socket says 172.21.15.43:51288) 2024-01-16T15:14:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.732+0000 7f22dfe00640 1 -- 172.21.15.43:0/110918758 learned_addr learned my addr 172.21.15.43:0/110918758 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.733+0000 7f22dfe00640 1 -- 172.21.15.43:0/110918758 >> v1:172.21.15.69:6789/0 conn(0x7f22e00cfa30 legacy=0x7f22e00cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.733+0000 7f22dfe00640 1 -- 172.21.15.43:0/110918758 >> v1:172.21.15.43:6789/0 conn(0x7f22e00d1fd0 legacy=0x7f22e00ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.733+0000 7f22dfe00640 1 -- 172.21.15.43:0/110918758 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00cf500 con 0x7f22e00b43e0 2024-01-16T15:14:12.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.734+0000 7f22dfe00640 1 --2- 172.21.15.43:0/110918758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e0168090 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009e50 tx=0x7f22d802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c991adcfd0336a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.734+0000 7f22de000640 1 -- 172.21.15.43:0/110918758 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d802cb90 con 0x7f22e00b43e0 2024-01-16T15:14:12.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.734+0000 7f22de000640 1 -- 172.21.15.43:0/110918758 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d802ccf0 con 0x7f22e00b43e0 2024-01-16T15:14:12.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.734+0000 7f22de000640 1 -- 172.21.15.43:0/110918758 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8034610 con 0x7f22e00b43e0 2024-01-16T15:14:12.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 -- 172.21.15.43:0/110918758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 msgr2=0x7f22e0168090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 --2- 172.21.15.43:0/110918758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e0168090 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009e50 tx=0x7f22d802bff0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 -- 172.21.15.43:0/110918758 shutdown_connections 2024-01-16T15:14:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 --2- 172.21.15.43:0/110918758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e0168090 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 -- 172.21.15.43:0/110918758 >> 172.21.15.43:0/110918758 conn(0x7f22e00d3020 msgr2=0x7f22e00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 -- 172.21.15.43:0/110918758 shutdown_connections 2024-01-16T15:14:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.735+0000 7f22e5000640 1 -- 172.21.15.43:0/110918758 wait complete. 2024-01-16T15:14:12.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.737+0000 7f22e5000640 1 Processor -- start 2024-01-16T15:14:12.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.737+0000 7f22e5000640 1 -- start start 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22e00cfa30 0x7f22e00ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 0x7f22e00ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00cd0e0 con 0x7f22e00b43e0 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e015f900 con 0x7f22e00cfa30 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22e5000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00c8df0 con 0x7f22e00d1fd0 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22df400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22df400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51290/0 (socket says 172.21.15.43:51290) 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.738+0000 7f22df400640 1 -- 172.21.15.43:0/210531371 learned_addr learned my addr 172.21.15.43:0/210531371 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22dea00640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22e00cfa30 0x7f22e00ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22dfe00640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 0x7f22e00ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 -- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 msgr2=0x7f22e00ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 0x7f22e00ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 -- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22e00cfa30 msgr2=0x7f22e00ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22e00cfa30 0x7f22e00ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 -- 172.21.15.43:0/210531371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d8009b30 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22dfe00640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 0x7f22e00ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.739+0000 7f22df400640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f22d000c450 tx=0x7f22d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.740+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d00052d0 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.740+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d0005430 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.740+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0012870 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.740+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e015f2b0 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.740+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e00ca630 con 0x7f22e00b43e0 2024-01-16T15:14:12.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.741+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e00d0a40 con 0x7f22e00b43e0 2024-01-16T15:14:12.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.742+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 32) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f22d00129d0 con 0x7f22e00b43e0 2024-01-16T15:14:12.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.743+0000 7f22d7400640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 0x7f22b806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:12.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.743+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f22d0088110 con 0x7f22e00b43e0 2024-01-16T15:14:12.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.743+0000 7f22dea00640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 0x7f22b806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:12.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.744+0000 7f22dea00640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 0x7f22b806e3e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f22e00cc9a0 tx=0x7f22c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:12.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.748+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22d00549d0 con 0x7f22e00b43e0 2024-01-16T15:14:12.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.976+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f22e015f130 con 0x7f22b806bf20 2024-01-16T15:14:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.990+0000 7f22d7400640 1 -- 172.21.15.43:0/210531371 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f22c8001c70 con 0x7f22b806bf20 2024-01-16T15:14:12.971 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.992+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 msgr2=0x7f22b806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 0x7f22b806e3e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f22e00cc9a0 tx=0x7f22c8009040 comp rx=0 tx=0).stop 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 msgr2=0x7f22e00c8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f22d000c450 tx=0x7f22d000c910 comp rx=0 tx=0).stop 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 shutdown_connections 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22b806bf20 0x7f22b806e3e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22e00cfa30 0x7f22e00ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22e00d1fd0 0x7f22e00ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 --2- 172.21.15.43:0/210531371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22e00b43e0 0x7f22e00c8710 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:12.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 >> 172.21.15.43:0/210531371 conn(0x7f22e00d3020 msgr2=0x7f22e00d13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:12.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 shutdown_connections 2024-01-16T15:14:12.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:12.993+0000 7f22e5000640 1 -- 172.21.15.43:0/210531371 wait complete. 2024-01-16T15:14:12.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:12.995 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd) 2024-01-16T15:14:13.059 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:14:13.059 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:14:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:13.060 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:14:15.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:15.061 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd) 2024-01-16T15:14:15.126 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd 2024-01-16T15:14:15.126 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:14:15.126 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:15.127 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:14:15.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:14:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.380+0000 7fe8e7200640 1 Processor -- start 2024-01-16T15:14:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.380+0000 7fe8e7200640 1 -- start start 2024-01-16T15:14:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.381+0000 7fe8e7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8e0160b10 0x7fe8e006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.381+0000 7fe8e7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e0071780 con 0x7fe8e0072280 2024-01-16T15:14:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8e7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00711c0 con 0x7fe8e006cad0 2024-01-16T15:14:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8e7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e00b2320 con 0x7fe8e0160b10 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8d7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe8e0072280 0x7fe8e006c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34852/0 (socket says 172.21.15.43:34852) 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8dfe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8e0160b10 0x7fe8e006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8d7e00640 1 -- 172.21.15.43:0/2076523235 learned_addr learned my addr 172.21.15.43:0/2076523235 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8df400640 1 -- 172.21.15.43:0/2076523235 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473432799 0 0) 0x7fe8e0071780 con 0x7fe8e0072280 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8df400640 1 -- 172.21.15.43:0/2076523235 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0003520 con 0x7fe8e0072280 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8dfe00640 1 -- 172.21.15.43:0/2076523235 >> v1:172.21.15.69:6789/0 conn(0x7fe8e006cad0 legacy=0x7fe8e00b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.382+0000 7fe8dfe00640 1 -- 172.21.15.43:0/2076523235 >> v1:172.21.15.43:6789/0 conn(0x7fe8e0072280 legacy=0x7fe8e006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.383+0000 7fe8dfe00640 1 -- 172.21.15.43:0/2076523235 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00714a0 con 0x7fe8e0160b10 2024-01-16T15:14:15.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8dfe00640 1 --2- 172.21.15.43:0/2076523235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e0160b10 0x7fe8e006e420 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00097a0 tx=0x7fe8d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8009847e618f154c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8df400640 1 -- 172.21.15.43:0/2076523235 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d002e070 con 0x7fe8e0160b10 2024-01-16T15:14:15.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8df400640 1 -- 172.21.15.43:0/2076523235 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d00315a0 con 0x7fe8e0160b10 2024-01-16T15:14:15.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8df400640 1 -- 172.21.15.43:0/2076523235 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0031910 con 0x7fe8e0160b10 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8e7200640 1 -- 172.21.15.43:0/2076523235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e0160b10 msgr2=0x7fe8e006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.384+0000 7fe8e7200640 1 --2- 172.21.15.43:0/2076523235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e0160b10 0x7fe8e006e420 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00097a0 tx=0x7fe8d002b840 comp rx=0 tx=0).stop 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.385+0000 7fe8e7200640 1 -- 172.21.15.43:0/2076523235 shutdown_connections 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.385+0000 7fe8e7200640 1 --2- 172.21.15.43:0/2076523235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e0160b10 0x7fe8e006e420 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.385+0000 7fe8e7200640 1 -- 172.21.15.43:0/2076523235 >> 172.21.15.43:0/2076523235 conn(0x7fe8e00b0ba0 msgr2=0x7fe8e00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.385+0000 7fe8e7200640 1 -- 172.21.15.43:0/2076523235 shutdown_connections 2024-01-16T15:14:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.385+0000 7fe8e7200640 1 -- 172.21.15.43:0/2076523235 wait complete. 2024-01-16T15:14:15.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.386+0000 7fe8e7200640 1 Processor -- start 2024-01-16T15:14:15.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.387+0000 7fe8e7200640 1 -- start start 2024-01-16T15:14:15.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e0072280 0x7fe8e0150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e0160b10 0x7fe8e00aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e009a120 con 0x7fe8e006cad0 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00ca790 con 0x7fe8e0072280 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8e7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01510f0 con 0x7fe8e0160b10 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33532/0 (socket says 172.21.15.43:33532) 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 -- 172.21.15.43:0/1109040129 learned_addr learned my addr 172.21.15.43:0/1109040129 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 -- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e0160b10 msgr2=0x7fe8e00aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e0160b10 0x7fe8e00aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 -- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e0072280 msgr2=0x7fe8e0150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e0072280 0x7fe8e0150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.388+0000 7fe8dfe00640 1 -- 172.21.15.43:0/1109040129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0009400 con 0x7fe8e006cad0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8dfe00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00098c0 tx=0x7fe8d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0032630 con 0x7fe8e006cad0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d0032790 con 0x7fe8e006cad0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0032b40 con 0x7fe8e006cad0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e00cbbe0 con 0x7fe8e006cad0 2024-01-16T15:14:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.389+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e00cc160 con 0x7fe8e006cad0 2024-01-16T15:14:15.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.390+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e009d200 con 0x7fe8e006cad0 2024-01-16T15:14:15.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.391+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 32) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe8d0032ca0 con 0x7fe8e006cad0 2024-01-16T15:14:15.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.392+0000 7fe8dcc00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 0x7fe8b806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.392+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fe8d00aa0b0 con 0x7fe8e006cad0 2024-01-16T15:14:15.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.393+0000 7fe8d7e00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 0x7fe8b806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.393+0000 7fe8d7e00640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 0x7fe8b806e3e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000f60 tx=0x7fe8c80078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.397+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8d0076970 con 0x7fe8e006cad0 2024-01-16T15:14:15.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.628+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8e0071d50 con 0x7fe8b806bf20 2024-01-16T15:14:15.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.770+0000 7fe8dcc00640 1 -- 172.21.15.43:0/1109040129 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe8c8005890 con 0x7fe8b806bf20 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 msgr2=0x7fe8b806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 0x7fe8b806e3e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8000f60 tx=0x7fe8c80078f0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 msgr2=0x7fe8e0099be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00098c0 tx=0x7fe8d00046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 shutdown_connections 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8b806bf20 0x7fe8b806e3e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e0072280 0x7fe8e0150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e0160b10 0x7fe8e00aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.773+0000 7fe8e7200640 1 --2- 172.21.15.43:0/1109040129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e006cad0 0x7fe8e0099be0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.774+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 >> 172.21.15.43:0/1109040129 conn(0x7fe8e00b0ba0 msgr2=0x7fe8e0070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.774+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 shutdown_connections 2024-01-16T15:14:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.774+0000 7fe8e7200640 1 -- 172.21.15.43:0/1109040129 wait complete. 2024-01-16T15:14:15.758 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.777+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.758 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.777+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.758 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.777+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.758 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.777+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.758 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.777+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.777 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:14:15.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:14:15.782 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.799+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.783 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.799+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.783 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.799+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.783 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.799+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.783 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:15.799+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.991+0000 7f3a1a800640 1 Processor -- start 2024-01-16T15:14:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.992+0000 7f3a1a800640 1 -- start start 2024-01-16T15:14:15.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.993+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.993+0000 7f3a1a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a14070490 0x7f3a14070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.993+0000 7f3a1a800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1406d7a0 con 0x7f3a140ae490 2024-01-16T15:14:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.993+0000 7f3a1a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1415a5b0 con 0x7f3a14070490 2024-01-16T15:14:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.993+0000 7f3a1a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1406ce70 con 0x7f3a140acd70 2024-01-16T15:14:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:48588/0 (socket says 172.21.15.43:48588) 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 -- 172.21.15.43:0/2872078334 learned_addr learned my addr 172.21.15.43:0/2872078334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13e00640 1 --2- 172.21.15.43:0/2872078334 >> v2:172.21.15.69:3300/0 conn(0x7f3a14070490 0x7f3a14070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 -- 172.21.15.43:0/2872078334 >> v2:172.21.15.69:3300/0 conn(0x7f3a14070490 msgr2=0x7f3a14070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 --2- 172.21.15.43:0/2872078334 >> v2:172.21.15.69:3300/0 conn(0x7f3a14070490 0x7f3a14070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 -- 172.21.15.43:0/2872078334 >> v1:172.21.15.69:6789/0 conn(0x7f3a140ae490 legacy=0x7f3a140ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.994+0000 7f3a13400640 1 -- 172.21.15.43:0/2872078334 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1406d150 con 0x7f3a140acd70 2024-01-16T15:14:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.996+0000 7f3a13400640 1 --2- 172.21.15.43:0/2872078334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400bb10 tx=0x7f3a0402e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08ddd77015b2bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.996+0000 7f3a12000640 1 -- 172.21.15.43:0/2872078334 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04036440 con 0x7f3a140acd70 2024-01-16T15:14:15.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.996+0000 7f3a12000640 1 -- 172.21.15.43:0/2872078334 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a040365a0 con 0x7f3a140acd70 2024-01-16T15:14:15.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a12000640 1 -- 172.21.15.43:0/2872078334 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04036930 con 0x7f3a140acd70 2024-01-16T15:14:15.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 -- 172.21.15.43:0/2872078334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 msgr2=0x7f3a1406dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 --2- 172.21.15.43:0/2872078334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400bb10 tx=0x7f3a0402e150 comp rx=0 tx=0).stop 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 -- 172.21.15.43:0/2872078334 shutdown_connections 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 --2- 172.21.15.43:0/2872078334 >> v2:172.21.15.69:3300/0 conn(0x7f3a14070490 0x7f3a14070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 --2- 172.21.15.43:0/2872078334 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140acd70 0x7f3a1406dc30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 -- 172.21.15.43:0/2872078334 >> 172.21.15.43:0/2872078334 conn(0x7f3a140b0ba0 msgr2=0x7f3a140b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 -- 172.21.15.43:0/2872078334 shutdown_connections 2024-01-16T15:14:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.997+0000 7f3a1a800640 1 -- 172.21.15.43:0/2872078334 wait complete. 2024-01-16T15:14:15.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.999+0000 7f3a1a800640 1 Processor -- start 2024-01-16T15:14:15.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:15.999+0000 7f3a1a800640 1 -- start start 2024-01-16T15:14:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 0x7f3a140ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140ae490 0x7f3a1409bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a141e4ef0 con 0x7f3a14070490 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14145910 con 0x7f3a140acd70 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a1a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a141e42e0 con 0x7f3a140ae490 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a13400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a13400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33580/0 (socket says 172.21.15.43:33580) 2024-01-16T15:14:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.000+0000 7f3a13400640 1 -- 172.21.15.43:0/902444819 learned_addr learned my addr 172.21.15.43:0/902444819 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a12a00640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 0x7f3a140ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13e00640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140ae490 0x7f3a1409bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 -- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140ae490 msgr2=0x7f3a1409bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140ae490 0x7f3a1409bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 -- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 msgr2=0x7f3a140ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 0x7f3a140ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 -- 172.21.15.43:0/902444819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0400b770 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a12a00640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 0x7f3a140ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.001+0000 7f3a13400640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400b2c0 tx=0x7f3a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.002+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04036440 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.002+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04034ba0 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.002+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04041440 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.002+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a141e45c0 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.002+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a141e4b80 con 0x7f3a14070490 2024-01-16T15:14:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.003+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1409d200 con 0x7f3a14070490 2024-01-16T15:14:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.004+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 32) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f3a0403d040 con 0x7f3a14070490 2024-01-16T15:14:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.005+0000 7f3a0b400640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 0x7f39ec06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.005+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f3a040ad3b0 con 0x7f3a14070490 2024-01-16T15:14:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.005+0000 7f3a12a00640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 0x7f39ec06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:15.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.006+0000 7f3a12a00640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 0x7f39ec06e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:15.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.010+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a04079c70 con 0x7f3a14070490 2024-01-16T15:14:16.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.241+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a140b3d90 con 0x7f39ec06bf20 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.257+0000 7f3a0b400640 1 -- 172.21.15.43:0/902444819 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f39fc001340 con 0x7f39ec06bf20 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:14:16.238 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:14:16.239 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:14:16.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:16.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:16.239 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 msgr2=0x7f39ec06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 0x7f39ec06e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc009040 comp rx=0 tx=0).stop 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 msgr2=0x7f3a141453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400b2c0 tx=0x7f3a040046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 shutdown_connections 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f39ec06bf20 0x7f39ec06e3e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a140acd70 0x7f3a140ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a140ae490 0x7f3a1409bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 --2- 172.21.15.43:0/902444819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a14070490 0x7f3a141453d0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.259+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 >> 172.21.15.43:0/902444819 conn(0x7f3a140b0ba0 msgr2=0x7f3a140aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:16.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.260+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 shutdown_connections 2024-01-16T15:14:16.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.260+0000 7f3a1a800640 1 -- 172.21.15.43:0/902444819 wait complete. 2024-01-16T15:14:16.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-16T15:14:16.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.491+0000 7fde74c00640 1 Processor -- start 2024-01-16T15:14:16.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.492+0000 7fde74c00640 1 -- start start 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde70144770 0x7fde7015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde700b35a0 0x7fde7015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde7006ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde700b43b0 con 0x7fde700b35a0 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7015f930 con 0x7fde70144770 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.493+0000 7fde74c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700b3b10 con 0x7fde7006c9d0 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde700b35a0 0x7fde7015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fde700b35a0 0x7fde7015f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33590/0 (socket says 172.21.15.43:33590) 2024-01-16T15:14:16.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 -- 172.21.15.43:0/1399786438 learned_addr learned my addr 172.21.15.43:0/1399786438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6f400640 1 --2- 172.21.15.43:0/1399786438 >> v2:172.21.15.69:3300/0 conn(0x7fde70144770 0x7fde7015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6fe00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde7006ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 -- 172.21.15.43:0/1399786438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 msgr2=0x7fde7006ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde7006ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 -- 172.21.15.43:0/1399786438 >> v2:172.21.15.69:3300/0 conn(0x7fde70144770 msgr2=0x7fde7015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:16.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 --2- 172.21.15.43:0/1399786438 >> v2:172.21.15.69:3300/0 conn(0x7fde70144770 0x7fde7015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.494+0000 7fde6ea00640 1 -- 172.21.15.43:0/1399786438 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700b3df0 con 0x7fde700b35a0 2024-01-16T15:14:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.495+0000 7fde6ea00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde700b35a0 0x7fde7015f0a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fde7006df90 tx=0x7fde5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e74062b6cbac19 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.496+0000 7fde6e000640 1 -- 172.21.15.43:0/1399786438 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58031870 con 0x7fde700b35a0 2024-01-16T15:14:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.496+0000 7fde6e000640 1 -- 172.21.15.43:0/1399786438 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde580319d0 con 0x7fde700b35a0 2024-01-16T15:14:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.496+0000 7fde6e000640 1 -- 172.21.15.43:0/1399786438 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58031d60 con 0x7fde700b35a0 2024-01-16T15:14:16.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.496+0000 7fde74c00640 1 -- 172.21.15.43:0/1399786438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde700b35a0 msgr2=0x7fde7015f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:16.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.496+0000 7fde74c00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde700b35a0 0x7fde7015f0a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fde7006df90 tx=0x7fde5802b910 comp rx=0 tx=0).stop 2024-01-16T15:14:16.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 -- 172.21.15.43:0/1399786438 shutdown_connections 2024-01-16T15:14:16.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde7006ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 --2- 172.21.15.43:0/1399786438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde700b35a0 0x7fde7015f0a0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 --2- 172.21.15.43:0/1399786438 >> v2:172.21.15.69:3300/0 conn(0x7fde70144770 0x7fde7015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 -- 172.21.15.43:0/1399786438 >> 172.21.15.43:0/1399786438 conn(0x7fde700cec10 msgr2=0x7fde700cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:16.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 -- 172.21.15.43:0/1399786438 shutdown_connections 2024-01-16T15:14:16.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.497+0000 7fde74c00640 1 -- 172.21.15.43:0/1399786438 wait complete. 2024-01-16T15:14:16.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.499+0000 7fde74c00640 1 Processor -- start 2024-01-16T15:14:16.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.499+0000 7fde74c00640 1 -- start start 2024-01-16T15:14:16.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde701649d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 0x7fde70164f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde70165ba0 con 0x7fde70144770 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701658c0 con 0x7fde700b35a0 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde74c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde70161de0 con 0x7fde7006c9d0 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33602/0 (socket says 172.21.15.43:33602) 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde6fe00640 1 -- 172.21.15.43:0/2100276071 learned_addr learned my addr 172.21.15.43:0/2100276071 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.500+0000 7fde6f400640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde701649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6ea00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 0x7fde70164f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 -- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 msgr2=0x7fde701649d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde701649d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 -- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 msgr2=0x7fde70164f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 0x7fde70164f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:16.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 -- 172.21.15.43:0/2100276071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580094d0 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6ea00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 0x7fde70164f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde6fe00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fde68007e40 tx=0x7fde6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68014620 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6800ee90 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde680149e0 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.501+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde701620c0 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.502+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70165450 con 0x7fde70144770 2024-01-16T15:14:16.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.503+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde701625e0 con 0x7fde70144770 2024-01-16T15:14:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.504+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 32) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fde68014b40 con 0x7fde70144770 2024-01-16T15:14:16.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.504+0000 7fde67400640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 0x7fde4806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:16.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.505+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fde68088f50 con 0x7fde70144770 2024-01-16T15:14:16.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.505+0000 7fde6f400640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 0x7fde4806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:16.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.505+0000 7fde6f400640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 0x7fde4806e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fde60006fd0 tx=0x7fde60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:16.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.509+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde701625e0 con 0x7fde70144770 2024-01-16T15:14:16.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:16.740+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fde70144b50 con 0x7fde4806bf50 2024-01-16T15:14:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:18.302+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fde68055be0 con 0x7fde70144770 2024-01-16T15:14:28.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.351+0000 7fde67400640 1 -- 172.21.15.43:0/2100276071 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde6000de40 con 0x7fde4806bf50 2024-01-16T15:14:28.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 msgr2=0x7fde4806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:28.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 0x7fde4806e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fde60006fd0 tx=0x7fde60008040 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 msgr2=0x7fde701628c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fde68007e40 tx=0x7fde6800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 shutdown_connections 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fde4806bf50 0x7fde4806e410 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde700b35a0 0x7fde70164f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde70144770 0x7fde701628c0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 --2- 172.21.15.43:0/2100276071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde7006c9d0 0x7fde701649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 >> 172.21.15.43:0/2100276071 conn(0x7fde700cec10 msgr2=0x7fde70070930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 shutdown_connections 2024-01-16T15:14:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:28.354+0000 7fde74c00640 1 -- 172.21.15.43:0/2100276071 wait complete. 2024-01-16T15:14:28.928 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:28.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:30.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:14:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.608+0000 7feafe800640 1 Processor -- start 2024-01-16T15:14:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.608+0000 7feafe800640 1 -- start start 2024-01-16T15:14:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafe800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf815f270 0x7feaf80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafe800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feaf806d9e0 0x7feaf80cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafe800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf815f930 con 0x7feaf8144770 2024-01-16T15:14:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafe800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf80b43b0 con 0x7feaf806d9e0 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafe800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80b3b10 con 0x7feaf815f270 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feaf806d9e0 0x7feaf80cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafde00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7feaf806d9e0 0x7feaf80cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36730/0 (socket says 172.21.15.43:36730) 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafde00640 1 -- 172.21.15.43:0/826942213 learned_addr learned my addr 172.21.15.43:0/826942213 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafde00640 1 -- 172.21.15.43:0/826942213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf815f270 msgr2=0x7feaf80cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.610+0000 7feafca00640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf815f270 0x7feaf80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.611+0000 7feafde00640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf815f270 0x7feaf80cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.611+0000 7feafde00640 1 -- 172.21.15.43:0/826942213 >> v1:172.21.15.69:6789/0 conn(0x7feaf8144770 legacy=0x7feaf815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.611+0000 7feafde00640 1 -- 172.21.15.43:0/826942213 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80b3df0 con 0x7feaf806d9e0 2024-01-16T15:14:30.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.612+0000 7feafde00640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf806d9e0 0x7feaf80cd9e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009870 tx=0x7feaf002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d4a71454fa411f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:30.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.612+0000 7feaf7e00640 1 -- 172.21.15.43:0/826942213 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0031640 con 0x7feaf806d9e0 2024-01-16T15:14:30.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.612+0000 7feaf7e00640 1 -- 172.21.15.43:0/826942213 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf00317a0 con 0x7feaf806d9e0 2024-01-16T15:14:30.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.612+0000 7feaf7e00640 1 -- 172.21.15.43:0/826942213 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0031b30 con 0x7feaf806d9e0 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 -- 172.21.15.43:0/826942213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf806d9e0 msgr2=0x7feaf80cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf806d9e0 0x7feaf80cd9e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009870 tx=0x7feaf002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 -- 172.21.15.43:0/826942213 shutdown_connections 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf806d9e0 0x7feaf80cd9e0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 --2- 172.21.15.43:0/826942213 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf815f270 0x7feaf80cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 -- 172.21.15.43:0/826942213 >> 172.21.15.43:0/826942213 conn(0x7feaf80cec10 msgr2=0x7feaf80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:30.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 -- 172.21.15.43:0/826942213 shutdown_connections 2024-01-16T15:14:30.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.613+0000 7feafe800640 1 -- 172.21.15.43:0/826942213 wait complete. 2024-01-16T15:14:30.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.615+0000 7feafe800640 1 Processor -- start 2024-01-16T15:14:30.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.615+0000 7feafe800640 1 -- start start 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf806d9e0 0x7feaf813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf8144770 0x7feaf81f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf815ca90 con 0x7feaf815f270 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8071840 con 0x7feaf8144770 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafe800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8071cd0 con 0x7feaf806d9e0 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36742/0 (socket says 172.21.15.43:36742) 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.616+0000 7feafde00640 1 -- 172.21.15.43:0/3072110287 learned_addr learned my addr 172.21.15.43:0/3072110287 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafd400640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf806d9e0 0x7feaf813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafca00640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf8144770 0x7feaf81f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 -- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf806d9e0 msgr2=0x7feaf813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:30.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf806d9e0 0x7feaf813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 -- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf8144770 msgr2=0x7feaf81f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:30.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf8144770 0x7feaf81f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:30.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 -- 172.21.15.43:0/3072110287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00094d0 con 0x7feaf815f270 2024-01-16T15:14:30.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.617+0000 7feafde00640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009870 tx=0x7feaf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.618+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf003ac60 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.618+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf003ade0 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.618+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0044630 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.618+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf813ab30 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.618+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf80711f0 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.619+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf80b3580 con 0x7feaf815f270 2024-01-16T15:14:30.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.620+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7feaf002f050 con 0x7feaf815f270 2024-01-16T15:14:30.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.621+0000 7feaf5600640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 0x7fead006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:30.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.621+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7feaf00aaa10 con 0x7feaf815f270 2024-01-16T15:14:30.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.621+0000 7feafd400640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 0x7fead006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:30.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.622+0000 7feafd400640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 0x7fead006e500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7feae8005520 tx=0x7feae8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:30.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.626+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaf80b3580 con 0x7feaf815f270 2024-01-16T15:14:30.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:30.857+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7feaf8144b50 con 0x7fead006c040 2024-01-16T15:14:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.192+0000 7feaf5600640 1 -- 172.21.15.43:0/3072110287 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7feae8005110 con 0x7fead006c040 2024-01-16T15:14:33.174 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.175 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.176 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 msgr2=0x7fead006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 0x7fead006e500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7feae8005520 tx=0x7feae8009040 comp rx=0 tx=0).stop 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 msgr2=0x7feaf81f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009870 tx=0x7feaf00046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 shutdown_connections 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fead006c040 0x7fead006e500 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf8144770 0x7feaf81f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feaf815f270 0x7feaf81f66d0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.195+0000 7feafe800640 1 --2- 172.21.15.43:0/3072110287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf806d9e0 0x7feaf813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.196+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 >> 172.21.15.43:0/3072110287 conn(0x7feaf80cec10 msgr2=0x7feaf806c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:33.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.196+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 shutdown_connections 2024-01-16T15:14:33.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.196+0000 7feafe800640 1 -- 172.21.15.43:0/3072110287 wait complete. 2024-01-16T15:14:33.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:14:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.440+0000 7f62e4a00640 1 Processor -- start 2024-01-16T15:14:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.441+0000 7f62e4a00640 1 -- start start 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.442+0000 7f62e4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62e0164e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.442+0000 7f62e4a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0071700 con 0x7f62e0143570 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.442+0000 7f62e4a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e015b410 con 0x7f62e00cf760 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.442+0000 7f62e4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e015b130 con 0x7f62e0143950 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.443+0000 7f62df400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f62e0143570 0x7f62e00cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49374/0 (socket says 172.21.15.43:49374) 2024-01-16T15:14:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.443+0000 7f62df400640 1 -- 172.21.15.43:0/1243096807 learned_addr learned my addr 172.21.15.43:0/1243096807 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.443+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933927469 0 0) 0x7f62d0001110 con 0x7f62e0143570 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.443+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c0005730 con 0x7f62e0143570 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.443+0000 7f62dfe00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62e0164e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 593419296 0 0) 0x7f62c0005730 con 0x7f62e0143570 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 msgr2=0x7f62e0164e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62e0164e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 >> v1:172.21.15.69:6789/0 conn(0x7f62e00cf760 legacy=0x7f62e015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e015dbd0 con 0x7f62e0143570 2024-01-16T15:14:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f62d0001800 con 0x7f62e0143570 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 >> v1:172.21.15.43:6789/0 conn(0x7f62e0143570 legacy=0x7f62e00cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62c0005c20 0x7f62c0006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62c00069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0006f60 0x7f62c00073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e015b130 con 0x7f62c0005c20 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c000ddf0 con 0x7f62c0006f60 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c000dfb0 con 0x7f62e0143950 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.444+0000 7f62df400640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62c0005c20 0x7f62c0006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62dfe00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0006f60 0x7f62c00073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62d7e00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62c00069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62df400640 1 -- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 msgr2=0x7f62c00069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62df400640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62c00069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62df400640 1 -- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0006f60 msgr2=0x7f62c00073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62df400640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0006f60 0x7f62c00073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62df400640 1 -- 172.21.15.43:0/1243096807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0005730 con 0x7f62c0005c20 2024-01-16T15:14:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62dfe00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0006f60 0x7f62c00073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.445+0000 7f62d7e00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0143950 0x7f62c00069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.446+0000 7f62df400640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62c0005c20 0x7f62c0006040 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c240 tx=0x7f62c0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cf3c6063ce9782 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:33.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.446+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0005ad0 con 0x7f62c0005c20 2024-01-16T15:14:33.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.446+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0005c50 con 0x7f62c0005c20 2024-01-16T15:14:33.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.446+0000 7f62dea00640 1 -- 172.21.15.43:0/1243096807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d002c8b0 con 0x7f62c0005c20 2024-01-16T15:14:33.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 -- 172.21.15.43:0/1243096807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62c0005c20 msgr2=0x7f62c0006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1243096807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62c0005c20 0x7f62c0006040 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c240 tx=0x7f62c0007a00 comp rx=0 tx=0).stop 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62dfe00640 1 -- 172.21.15.43:0/1243096807 reap_dead start 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 -- 172.21.15.43:0/1243096807 shutdown_connections 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 -- 172.21.15.43:0/1243096807 >> 172.21.15.43:0/1243096807 conn(0x7f62e006a860 msgr2=0x7f62e006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 -- 172.21.15.43:0/1243096807 shutdown_connections 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.447+0000 7f62e4a00640 1 -- 172.21.15.43:0/1243096807 wait complete. 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.449+0000 7f62e4a00640 1 Processor -- start 2024-01-16T15:14:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.449+0000 7f62e4a00640 1 -- start start 2024-01-16T15:14:33.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 0x7f62e013e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 0x7f62e0159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0159c60 con 0x7f62e0158bd0 2024-01-16T15:14:33.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e013e9e0 con 0x7f62c0005c20 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62e4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e013fb10 con 0x7f62e0159290 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62d7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62d7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38814/0 (socket says 172.21.15.43:38814) 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.450+0000 7f62d7e00640 1 -- 172.21.15.43:0/1850360812 learned_addr learned my addr 172.21.15.43:0/1850360812 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62dfe00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 0x7f62e0159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62df400640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 0x7f62e013e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 -- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 msgr2=0x7f62e0159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 0x7f62e0159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 -- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 msgr2=0x7f62e013e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 0x7f62e013e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 -- 172.21.15.43:0/1850360812 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0156620 con 0x7f62e0158bd0 2024-01-16T15:14:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62dfe00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 0x7f62e0159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62df400640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 0x7f62e013e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7e00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f62c800d7a0 tx=0x7f62c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8017070 con 0x7f62e0158bd0 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c8005070 con 0x7f62e0158bd0 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8005420 con 0x7f62e0158bd0 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.451+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0156900 con 0x7f62e0158bd0 2024-01-16T15:14:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.452+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e013fdf0 con 0x7f62e0158bd0 2024-01-16T15:14:33.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.453+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e006b1a0 con 0x7f62e0158bd0 2024-01-16T15:14:33.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.454+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f62c8007030 con 0x7f62e0158bd0 2024-01-16T15:14:33.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.455+0000 7f62d7400640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 0x7f62b806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:33.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.455+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f62c8088200 con 0x7f62e0158bd0 2024-01-16T15:14:33.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.455+0000 7f62df400640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 0x7f62b806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.456+0000 7f62df400640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 0x7f62b806e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f62d0034000 tx=0x7f62d0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:33.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.460+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62c80bde90 con 0x7f62e0158bd0 2024-01-16T15:14:33.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.731+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f62e004e840 con 0x7f62e0158bd0 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.732+0000 7f62d7400640 1 -- 172.21.15.43:0/1850360812 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2306 (secure 0 0 0) 0x7f62c80548d0 con 0x7f62e0158bd0 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:14:33.714 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:14:33.715 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 926, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 90, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:14:33.716 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:14:33.717 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 798, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222169656, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 7033659392, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766060453888, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 511, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 12624, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 3 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:14:33.718 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 58, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 6, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 6, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 8106 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:14:33.719 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:14:33.720 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16, 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:14:15.179510+0000", 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "services": { 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": { 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "daemons": { 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "summary": "", 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "x": { 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "start_epoch": 0, 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "start_stamp": "0.000000", 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 0, 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": {}, 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: "task_status": {} 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": {} 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:14:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 msgr2=0x7f62b806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 0x7f62b806e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f62d0034000 tx=0x7f62d0009030 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 msgr2=0x7f62e013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f62c800d7a0 tx=0x7f62c800dc60 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 shutdown_connections 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b806c040 0x7f62b806e500 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62e0159290 0x7f62e0159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62e0158bd0 0x7f62e013f1a0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 --2- 172.21.15.43:0/1850360812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62c0005c20 0x7f62e013e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 >> 172.21.15.43:0/1850360812 conn(0x7f62e006a860 msgr2=0x7f62e015a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 shutdown_connections 2024-01-16T15:14:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:33.735+0000 7f62e4a00640 1 -- 172.21.15.43:0/1850360812 wait complete. 2024-01-16T15:14:33.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:33.737 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd) 2024-01-16T15:14:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/b8803a62-26c1-4dc4-a0d7-08f7f7738acd': No such file or directory 2024-01-16T15:14:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:14:33.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:14:34.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.042+0000 7f1454e00640 1 Processor -- start 2024-01-16T15:14:34.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.042+0000 7f1454e00640 1 -- start start 2024-01-16T15:14:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f1454e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14500b2550 0x7f1450070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f1454e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f1450160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f1454e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450070a90 con 0x7f14500b2d70 2024-01-16T15:14:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f1454e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1450070cf0 con 0x7f14500b2550 2024-01-16T15:14:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f1454e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1450070fd0 con 0x7f14500b2930 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f144f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14500b2550 0x7f1450070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f144f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14500b2550 0x7f1450070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38830/0 (socket says 172.21.15.43:38830) 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.044+0000 7f144f400640 1 -- 172.21.15.43:0/1380152259 learned_addr learned my addr 172.21.15.43:0/1380152259 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.045+0000 7f144ea00640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f1450160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.045+0000 7f144f400640 1 -- 172.21.15.43:0/1380152259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 msgr2=0x7f1450160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.045+0000 7f144f400640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f1450160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.045+0000 7f144f400640 1 -- 172.21.15.43:0/1380152259 >> v1:172.21.15.69:6789/0 conn(0x7f14500b2d70 legacy=0x7f145015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:34.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.045+0000 7f144f400640 1 -- 172.21.15.43:0/1380152259 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500712b0 con 0x7f14500b2550 2024-01-16T15:14:34.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.046+0000 7f144f400640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2550 0x7f1450070550 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f1440009870 tx=0x7f144002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7910e71ca565308 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.046+0000 7f144e000640 1 -- 172.21.15.43:0/1380152259 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440031640 con 0x7f14500b2550 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.046+0000 7f144e000640 1 -- 172.21.15.43:0/1380152259 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14400317a0 con 0x7f14500b2550 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.046+0000 7f144e000640 1 -- 172.21.15.43:0/1380152259 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440031b30 con 0x7f14500b2550 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 -- 172.21.15.43:0/1380152259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2550 msgr2=0x7f1450070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2550 0x7f1450070550 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f1440009870 tx=0x7f144002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 -- 172.21.15.43:0/1380152259 shutdown_connections 2024-01-16T15:14:34.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f1450160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 --2- 172.21.15.43:0/1380152259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2550 0x7f1450070550 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 -- 172.21.15.43:0/1380152259 >> 172.21.15.43:0/1380152259 conn(0x7f14500b0840 msgr2=0x7f14500b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:34.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 -- 172.21.15.43:0/1380152259 shutdown_connections 2024-01-16T15:14:34.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.047+0000 7f1454e00640 1 -- 172.21.15.43:0/1380152259 wait complete. 2024-01-16T15:14:34.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.049+0000 7f1454e00640 1 Processor -- start 2024-01-16T15:14:34.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.049+0000 7f1454e00640 1 -- start start 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 0x7f145015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f145014faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14501f34d0 con 0x7f14500b2d70 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14501f37b0 con 0x7f14500b2550 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f1454e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14501f3a90 con 0x7f14500b2930 2024-01-16T15:14:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f144fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f144fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38836/0 (socket says 172.21.15.43:38836) 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.050+0000 7f144fe00640 1 -- 172.21.15.43:0/524170925 learned_addr learned my addr 172.21.15.43:0/524170925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144ea00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f145014faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144f400640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 0x7f145015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 -- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 msgr2=0x7f145014faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f145014faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 -- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 msgr2=0x7f145015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 0x7f145015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 -- 172.21.15.43:0/524170925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400094d0 con 0x7f14500b2d70 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144f400640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 0x7f145015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:34.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f144fe00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f144800b970 tx=0x7f144800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144800cb40 con 0x7f14500b2d70 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.051+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144800cca0 con 0x7f14500b2d70 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.052+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1450153e80 con 0x7f14500b2d70 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.052+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448014680 con 0x7f14500b2d70 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.052+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14501f3150 con 0x7f14500b2d70 2024-01-16T15:14:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.053+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14501543a0 con 0x7f14500b2d70 2024-01-16T15:14:34.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.054+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f144801d3b0 con 0x7f14500b2d70 2024-01-16T15:14:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.055+0000 7f1447400640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 0x7f142806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.055+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f1448088830 con 0x7f14500b2d70 2024-01-16T15:14:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.055+0000 7f144f400640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 0x7f142806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.056+0000 7f144f400640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 0x7f142806e500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1440036000 tx=0x7f144003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.059+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14501543a0 con 0x7f14500b2d70 2024-01-16T15:14:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.278+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1450071870 con 0x7f142806c040 2024-01-16T15:14:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.281+0000 7f1447400640 1 -- 172.21.15.43:0/524170925 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1440031ae0 con 0x7f142806c040 2024-01-16T15:14:34.264 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:14:34.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:34.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:14:34.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:34.264 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:14:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 msgr2=0x7f142806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 0x7f142806e500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1440036000 tx=0x7f144003c040 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 msgr2=0x7f14501f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f144800b970 tx=0x7f144800be30 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 shutdown_connections 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f142806c040 0x7f142806e500 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14500b2d70 0x7f14501f2c10 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14500b2930 0x7f145014faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 --2- 172.21.15.43:0/524170925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14500b2550 0x7f145015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 >> 172.21.15.43:0/524170925 conn(0x7f14500b0840 msgr2=0x7f1450157720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 shutdown_connections 2024-01-16T15:14:34.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.284+0000 7f1454e00640 1 -- 172.21.15.43:0/524170925 wait complete. 2024-01-16T15:14:34.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:14:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.535+0000 7fe649600640 1 Processor -- start 2024-01-16T15:14:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.535+0000 7fe649600640 1 -- start start 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe644153ea0 0x7fe6440ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644154280 0x7fe64415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe644070850 0x7fe644071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6440ce650 con 0x7fe644070850 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64415ddd0 con 0x7fe644153ea0 2024-01-16T15:14:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.537+0000 7fe649600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64415e0a0 con 0x7fe644154280 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe644070850 0x7fe644071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe644070850 0x7fe644071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38848/0 (socket says 172.21.15.43:38848) 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 -- 172.21.15.43:0/241677100 learned_addr learned my addr 172.21.15.43:0/241677100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe63b200640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644154280 0x7fe64415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe643e00640 1 --2- 172.21.15.43:0/241677100 >> v2:172.21.15.69:3300/0 conn(0x7fe644153ea0 0x7fe6440ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 -- 172.21.15.43:0/241677100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644154280 msgr2=0x7fe64415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644154280 0x7fe64415d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 -- 172.21.15.43:0/241677100 >> v2:172.21.15.69:3300/0 conn(0x7fe644153ea0 msgr2=0x7fe6440ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 --2- 172.21.15.43:0/241677100 >> v2:172.21.15.69:3300/0 conn(0x7fe644153ea0 0x7fe6440ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.538+0000 7fe648c00640 1 -- 172.21.15.43:0/241677100 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64415e380 con 0x7fe644070850 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.539+0000 7fe648c00640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644070850 0x7fe644071350 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fe6440cfc10 tx=0x7fe63c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb44002071bd3a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.540+0000 7fe643400640 1 -- 172.21.15.43:0/241677100 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c031870 con 0x7fe644070850 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.540+0000 7fe643400640 1 -- 172.21.15.43:0/241677100 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c0319d0 con 0x7fe644070850 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.540+0000 7fe643400640 1 -- 172.21.15.43:0/241677100 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c031d60 con 0x7fe644070850 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.540+0000 7fe649600640 1 -- 172.21.15.43:0/241677100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644070850 msgr2=0x7fe644071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.540+0000 7fe649600640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644070850 0x7fe644071350 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fe6440cfc10 tx=0x7fe63c02b910 comp rx=0 tx=0).stop 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 -- 172.21.15.43:0/241677100 shutdown_connections 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644070850 0x7fe644071350 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 --2- 172.21.15.43:0/241677100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644154280 0x7fe64415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 --2- 172.21.15.43:0/241677100 >> v2:172.21.15.69:3300/0 conn(0x7fe644153ea0 0x7fe6440ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 -- 172.21.15.43:0/241677100 >> 172.21.15.43:0/241677100 conn(0x7fe6440cec10 msgr2=0x7fe6440cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 -- 172.21.15.43:0/241677100 shutdown_connections 2024-01-16T15:14:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.541+0000 7fe649600640 1 -- 172.21.15.43:0/241677100 wait complete. 2024-01-16T15:14:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.542+0000 7fe649600640 1 Processor -- start 2024-01-16T15:14:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.542+0000 7fe649600640 1 -- start start 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644070850 0x7fe64413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe644153ea0 0x7fe64414c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440afbb0 con 0x7fe644154280 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440afe90 con 0x7fe644153ea0 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.543+0000 7fe649600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440b0170 con 0x7fe644070850 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38864/0 (socket says 172.21.15.43:38864) 2024-01-16T15:14:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 -- 172.21.15.43:0/2555059035 learned_addr learned my addr 172.21.15.43:0/2555059035 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe643e00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644070850 0x7fe64413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 -- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644070850 msgr2=0x7fe64413c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644070850 0x7fe64413c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 -- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe644153ea0 msgr2=0x7fe64414c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe644153ea0 0x7fe64414c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.544+0000 7fe648c00640 1 -- 172.21.15.43:0/2555059035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0094d0 con 0x7fe644154280 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe648c00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe63c0090b0 tx=0x7fe63c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c03ac60 con 0x7fe644154280 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c03adc0 con 0x7fe644154280 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c044630 con 0x7fe644154280 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6440b0450 con 0x7fe644154280 2024-01-16T15:14:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.545+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64414c680 con 0x7fe644154280 2024-01-16T15:14:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.546+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6440b0970 con 0x7fe644154280 2024-01-16T15:14:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.547+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fe63c04d450 con 0x7fe644154280 2024-01-16T15:14:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.548+0000 7fe640c00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 0x7fe61c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.548+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fe63c02d040 con 0x7fe644154280 2024-01-16T15:14:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.548+0000 7fe643e00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 0x7fe61c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.549+0000 7fe643e00640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 0x7fe61c06e500 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe6440ae3b0 tx=0x7fe63400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.552+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6440b0970 con 0x7fe644154280 2024-01-16T15:14:34.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.778+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe64406da00 con 0x7fe61c06c040 2024-01-16T15:14:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.787+0000 7fe640c00640 1 -- 172.21.15.43:0/2555059035 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7fe64406da00 con 0x7fe61c06c040 2024-01-16T15:14:34.769 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:34.769 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:14:34.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:34.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:14:34.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:34.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:34.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1clone1" 2024-01-16T15:14:34.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:34.770 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:14:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 msgr2=0x7fe61c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 0x7fe61c06e500 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe6440ae3b0 tx=0x7fe63400b040 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 msgr2=0x7fe6440b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe63c0090b0 tx=0x7fe63c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 shutdown_connections 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe61c06c040 0x7fe61c06e500 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe644153ea0 0x7fe64414c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe644154280 0x7fe6440b0c50 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 --2- 172.21.15.43:0/2555059035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe644070850 0x7fe64413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 >> 172.21.15.43:0/2555059035 conn(0x7fe6440cec10 msgr2=0x7fe64406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 shutdown_connections 2024-01-16T15:14:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:34.790+0000 7fe649600640 1 -- 172.21.15.43:0/2555059035 wait complete. 2024-01-16T15:14:34.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:14:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.029+0000 7f63b1800640 1 Processor -- start 2024-01-16T15:14:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.030+0000 7f63b1800640 1 -- start start 2024-01-16T15:14:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.031+0000 7f63b1800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f63ac15b350 0x7f63ac0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.031+0000 7f63b1800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 0x7f63ac0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.031+0000 7f63b1800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac15b730 con 0x7f63ac09cb30 2024-01-16T15:14:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.031+0000 7f63b1800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac0ac740 con 0x7f63ac15b350 2024-01-16T15:14:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.031+0000 7f63b1800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac0ac460 con 0x7f63ac0adb60 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63a3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f63ac09cb30 0x7f63ac0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49386/0 (socket says 172.21.15.43:49386) 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63aaa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f63ac15b350 0x7f63ac0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63a3e00640 1 -- 172.21.15.43:0/3046996780 learned_addr learned my addr 172.21.15.43:0/3046996780 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63ab400640 1 --2- 172.21.15.43:0/3046996780 >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 0x7f63ac0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63aa000640 1 -- 172.21.15.43:0/3046996780 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259062629 0 0) 0x7f6394001110 con 0x7f63ac09cb30 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.032+0000 7f63aa000640 1 -- 172.21.15.43:0/3046996780 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c003520 con 0x7f63ac09cb30 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.033+0000 7f63aaa00640 1 -- 172.21.15.43:0/3046996780 >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 msgr2=0x7f63ac0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.033+0000 7f63aaa00640 1 --2- 172.21.15.43:0/3046996780 >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 0x7f63ac0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.033+0000 7f63aaa00640 1 -- 172.21.15.43:0/3046996780 >> v1:172.21.15.43:6789/0 conn(0x7f63ac09cb30 legacy=0x7f63ac0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.033+0000 7f63aaa00640 1 -- 172.21.15.43:0/3046996780 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0b2a50 con 0x7f63ac15b350 2024-01-16T15:14:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.033+0000 7f63ab400640 1 --2- 172.21.15.43:0/3046996780 >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 0x7f63ac0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:14:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.034+0000 7f63aaa00640 1 --2- 172.21.15.43:0/3046996780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac15b350 0x7f63ac0abe80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f63ac06dac0 tx=0x7f639c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976d50a5b5e0a8db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.034+0000 7f63aa000640 1 -- 172.21.15.43:0/3046996780 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031830 con 0x7f63ac15b350 2024-01-16T15:14:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.034+0000 7f63aa000640 1 -- 172.21.15.43:0/3046996780 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c031990 con 0x7f63ac15b350 2024-01-16T15:14:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.034+0000 7f63aa000640 1 -- 172.21.15.43:0/3046996780 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031d20 con 0x7f63ac15b350 2024-01-16T15:14:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 -- 172.21.15.43:0/3046996780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac15b350 msgr2=0x7f63ac0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 --2- 172.21.15.43:0/3046996780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac15b350 0x7f63ac0abe80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f63ac06dac0 tx=0x7f639c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 -- 172.21.15.43:0/3046996780 shutdown_connections 2024-01-16T15:14:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 --2- 172.21.15.43:0/3046996780 >> v2:172.21.15.69:3300/0 conn(0x7f63ac0adb60 0x7f63ac0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 --2- 172.21.15.43:0/3046996780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac15b350 0x7f63ac0abe80 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 -- 172.21.15.43:0/3046996780 >> 172.21.15.43:0/3046996780 conn(0x7f63ac0b0ba0 msgr2=0x7f63ac0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 -- 172.21.15.43:0/3046996780 shutdown_connections 2024-01-16T15:14:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.035+0000 7f63b1800640 1 -- 172.21.15.43:0/3046996780 wait complete. 2024-01-16T15:14:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.037+0000 7f63b1800640 1 Processor -- start 2024-01-16T15:14:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.037+0000 7f63b1800640 1 -- start start 2024-01-16T15:14:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63ac09cb30 0x7f63ac09af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 0x7f63ac155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac152e50 con 0x7f63ac0adb60 2024-01-16T15:14:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac1530a0 con 0x7f63ac15b350 2024-01-16T15:14:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63b1800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac153380 con 0x7f63ac09cb30 2024-01-16T15:14:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38890/0 (socket says 172.21.15.43:38890) 2024-01-16T15:14:35.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.038+0000 7f63a3e00640 1 -- 172.21.15.43:0/2363637048 learned_addr learned my addr 172.21.15.43:0/2363637048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63aaa00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63ac09cb30 0x7f63ac09af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 -- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63ac09cb30 msgr2=0x7f63ac09af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63ab400640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 0x7f63ac155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63ac09cb30 0x7f63ac09af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 -- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 msgr2=0x7f63ac155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 0x7f63ac155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 -- 172.21.15.43:0/2363637048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009400 con 0x7f63ac0adb60 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63ab400640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 0x7f63ac155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a3e00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f639400c700 tx=0x7f639400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394005ce0 con 0x7f63ac0adb60 2024-01-16T15:14:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6394005e40 con 0x7f63ac0adb60 2024-01-16T15:14:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394007290 con 0x7f63ac0adb60 2024-01-16T15:14:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.039+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac153660 con 0x7f63ac0adb60 2024-01-16T15:14:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.040+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac153e60 con 0x7f63ac0adb60 2024-01-16T15:14:35.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.041+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac153b80 con 0x7f63ac0adb60 2024-01-16T15:14:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.042+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f63940074b0 con 0x7f63ac0adb60 2024-01-16T15:14:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.043+0000 7f63a2a00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 0x7f638406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.043+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f6394087d60 con 0x7f63ac0adb60 2024-01-16T15:14:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.043+0000 7f63aaa00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 0x7f638406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.044+0000 7f63aaa00640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 0x7f638406e500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f639c036000 tx=0x7f639c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.048+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63ac153b80 con 0x7f63ac0adb60 2024-01-16T15:14:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.275+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63ac0b3010 con 0x7f638406c040 2024-01-16T15:14:35.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.306+0000 7f63a2a00640 1 -- 172.21.15.43:0/2363637048 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f639c031b80 con 0x7f638406c040 2024-01-16T15:14:35.288 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:35.289 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:14:35.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:35.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:14:35.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:35.289 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.309+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 msgr2=0x7f638406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.309+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 0x7f638406e500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f639c036000 tx=0x7f639c03c040 comp rx=0 tx=0).stop 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.309+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 msgr2=0x7f63ac157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.309+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f639400c700 tx=0x7f639400cb00 comp rx=0 tx=0).stop 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 shutdown_connections 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f638406c040 0x7f638406e500 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63ac15b350 0x7f63ac155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63ac0adb60 0x7f63ac157fe0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 --2- 172.21.15.43:0/2363637048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63ac09cb30 0x7f63ac09af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 >> 172.21.15.43:0/2363637048 conn(0x7f63ac0b0ba0 msgr2=0x7f63ac0b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 shutdown_connections 2024-01-16T15:14:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.310+0000 7f63b1800640 1 -- 172.21.15.43:0/2363637048 wait complete. 2024-01-16T15:14:35.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:14:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.551+0000 7ffb4f000640 1 Processor -- start 2024-01-16T15:14:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.551+0000 7ffb4f000640 1 -- start start 2024-01-16T15:14:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.553+0000 7ffb4f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb480d1d90 0x7ffb480d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.553+0000 7ffb4f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb480cdbb0 0x7ffb480b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.553+0000 7ffb4f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb480cef30 con 0x7ffb4815d340 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.553+0000 7ffb4f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb480d11a0 con 0x7ffb480d1d90 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.553+0000 7ffb4f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480ce320 con 0x7ffb480cdbb0 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.554+0000 7ffb4e600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb480cdbb0 0x7ffb480b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.554+0000 7ffb4dc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ffb4815d340 0x7ffb480d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:36580/0 (socket says 172.21.15.43:36580) 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.554+0000 7ffb4dc00640 1 -- 172.21.15.43:0/451019386 learned_addr learned my addr 172.21.15.43:0/451019386 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.554+0000 7ffb4d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb480d1d90 0x7ffb480d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.555+0000 7ffb4d200640 1 -- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb480cdbb0 msgr2=0x7ffb480b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.555+0000 7ffb4d200640 1 --2- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb480cdbb0 0x7ffb480b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.555+0000 7ffb4d200640 1 -- 172.21.15.43:0/451019386 >> v1:172.21.15.69:6789/0 conn(0x7ffb4815d340 legacy=0x7ffb480d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.555+0000 7ffb4d200640 1 -- 172.21.15.43:0/451019386 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480ce600 con 0x7ffb480d1d90 2024-01-16T15:14:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.556+0000 7ffb4d200640 1 --2- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb480d00b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffb300097a0 tx=0x7ffb3002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2f74011c1447ff8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.557+0000 7ffb47e00640 1 -- 172.21.15.43:0/451019386 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3002d040 con 0x7ffb480d1d90 2024-01-16T15:14:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.557+0000 7ffb47e00640 1 -- 172.21.15.43:0/451019386 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30031810 con 0x7ffb480d1d90 2024-01-16T15:14:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.557+0000 7ffb47e00640 1 -- 172.21.15.43:0/451019386 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30031ba0 con 0x7ffb480d1d90 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.557+0000 7ffb4f000640 1 -- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 msgr2=0x7ffb480d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.557+0000 7ffb4f000640 1 --2- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb480d00b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffb300097a0 tx=0x7ffb3002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 -- 172.21.15.43:0/451019386 shutdown_connections 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 --2- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb480d00b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 --2- 172.21.15.43:0/451019386 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb480cdbb0 0x7ffb480b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 -- 172.21.15.43:0/451019386 >> 172.21.15.43:0/451019386 conn(0x7ffb4809c370 msgr2=0x7ffb480d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 -- 172.21.15.43:0/451019386 shutdown_connections 2024-01-16T15:14:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.558+0000 7ffb4f000640 1 -- 172.21.15.43:0/451019386 wait complete. 2024-01-16T15:14:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.559+0000 7ffb4f000640 1 Processor -- start 2024-01-16T15:14:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.560+0000 7ffb4f000640 1 -- start start 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb481f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb4815d340 0x7ffb480b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb48160400 con 0x7ffb480cdbb0 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4815f640 con 0x7ffb480d1d90 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4815f920 con 0x7ffb4815d340 2024-01-16T15:14:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38904/0 (socket says 172.21.15.43:38904) 2024-01-16T15:14:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 -- 172.21.15.43:0/3595506449 learned_addr learned my addr 172.21.15.43:0/3595506449 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 -- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb4815d340 msgr2=0x7ffb480b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb4815d340 0x7ffb480b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 -- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 msgr2=0x7ffb481f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4d200640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb481f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.561+0000 7ffb4dc00640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb481f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb4dc00640 1 -- 172.21.15.43:0/3595506449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30009400 con 0x7ffb480cdbb0 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb4dc00640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c340 tx=0x7ffb3800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb38005ce0 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb38005e40 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380072d0 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4815fc00 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.562+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb480b8ce0 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.563+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb48160120 con 0x7ffb480cdbb0 2024-01-16T15:14:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.564+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7ffb38007430 con 0x7ffb480cdbb0 2024-01-16T15:14:35.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.565+0000 7ffb45600640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 0x7ffb2006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:35.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.566+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ffb38087d80 con 0x7ffb480cdbb0 2024-01-16T15:14:35.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.566+0000 7ffb4d200640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 0x7ffb2006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.567+0000 7ffb4d200640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 0x7ffb2006e530 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ffb30036000 tx=0x7ffb3003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.570+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb48160120 con 0x7ffb480cdbb0 2024-01-16T15:14:35.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.797+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb480cebc0 con 0x7ffb2006c070 2024-01-16T15:14:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.863+0000 7ffb45600640 1 -- 172.21.15.43:0/3595506449 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffb30031b00 con 0x7ffb2006c070 2024-01-16T15:14:35.845 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:35.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 msgr2=0x7ffb2006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 0x7ffb2006e530 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ffb30036000 tx=0x7ffb3003c040 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 msgr2=0x7ffb480c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c340 tx=0x7ffb3800c800 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 shutdown_connections 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffb2006c070 0x7ffb2006e530 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb480d1d90 0x7ffb481f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb4815d340 0x7ffb480b87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 --2- 172.21.15.43:0/3595506449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb480cdbb0 0x7ffb480c7800 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 >> 172.21.15.43:0/3595506449 conn(0x7ffb4809c370 msgr2=0x7ffb480afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 shutdown_connections 2024-01-16T15:14:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:35.866+0000 7ffb4f000640 1 -- 172.21.15.43:0/3595506449 wait complete. 2024-01-16T15:14:35.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:14:36.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.111+0000 7fa788e00640 1 Processor -- start 2024-01-16T15:14:36.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.112+0000 7fa788e00640 1 -- start start 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.113+0000 7fa788e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa784160b10 0x7fa78406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.113+0000 7fa788e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa7840b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.113+0000 7fa788e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784071780 con 0x7fa784072280 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.113+0000 7fa788e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7840b2320 con 0x7fa784160b10 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.113+0000 7fa788e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78406df30 con 0x7fa78406cad0 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa781400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa784072280 0x7fa78406c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49396/0 (socket says 172.21.15.43:49396) 2024-01-16T15:14:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa781400640 1 -- 172.21.15.43:0/4212466326 learned_addr learned my addr 172.21.15.43:0/4212466326 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa781e00640 1 --2- 172.21.15.43:0/4212466326 >> v2:172.21.15.69:3300/0 conn(0x7fa784160b10 0x7fa78406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa782800640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa7840b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457655462 0 0) 0x7fa784071780 con 0x7fa784072280 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.114+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa764003520 con 0x7fa784072280 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674606318 0 0) 0x7fa764003520 con 0x7fa784072280 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 msgr2=0x7fa7840b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa7840b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 >> v2:172.21.15.69:3300/0 conn(0x7fa784160b10 msgr2=0x7fa78406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 --2- 172.21.15.43:0/4212466326 >> v2:172.21.15.69:3300/0 conn(0x7fa784160b10 0x7fa78406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7840711c0 con 0x7fa784072280 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa76c0019f0 con 0x7fa784072280 2024-01-16T15:14:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 >> v1:172.21.15.43:6789/0 conn(0x7fa784072280 legacy=0x7fa78406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa764003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa764003ef0 0x7fa7640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa764004d00 0x7fa764005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840b2320 con 0x7fa764004d00 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78406df30 con 0x7fa764003ef0 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76400ba80 con 0x7fa78406cad0 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.115+0000 7fa782800640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa764004d00 0x7fa764005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa781e00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa764003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa782800640 1 -- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 msgr2=0x7fa764003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa782800640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78406cad0 0x7fa764003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa782800640 1 -- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa764003ef0 msgr2=0x7fa7640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa782800640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa764003ef0 0x7fa7640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.116+0000 7fa782800640 1 -- 172.21.15.43:0/4212466326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa764003520 con 0x7fa764004d00 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.117+0000 7fa782800640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa764004d00 0x7fa764005180 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00ba00 tx=0x7fa7640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49eff548a9e252e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.118+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c033680 con 0x7fa764004d00 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.118+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c0337e0 con 0x7fa764004d00 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.118+0000 7fa780a00640 1 -- 172.21.15.43:0/4212466326 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c033b70 con 0x7fa764004d00 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 -- 172.21.15.43:0/4212466326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa764004d00 msgr2=0x7fa764005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 --2- 172.21.15.43:0/4212466326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa764004d00 0x7fa764005180 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00ba00 tx=0x7fa7640057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa782800640 1 -- 172.21.15.43:0/4212466326 reap_dead start 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 -- 172.21.15.43:0/4212466326 shutdown_connections 2024-01-16T15:14:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 -- 172.21.15.43:0/4212466326 >> 172.21.15.43:0/4212466326 conn(0x7fa7840b0ba0 msgr2=0x7fa7840b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 -- 172.21.15.43:0/4212466326 shutdown_connections 2024-01-16T15:14:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.119+0000 7fa788e00640 1 -- 172.21.15.43:0/4212466326 wait complete. 2024-01-16T15:14:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.121+0000 7fa788e00640 1 Processor -- start 2024-01-16T15:14:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.121+0000 7fa788e00640 1 -- start start 2024-01-16T15:14:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa764004d00 0x7fa7840af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78409bf40 0x7fa78409c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78415a6e0 con 0x7fa7840cba40 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78409c8f0 con 0x7fa78409bf40 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa788e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78409b1a0 con 0x7fa764004d00 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.122+0000 7fa781400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38910/0 (socket says 172.21.15.43:38910) 2024-01-16T15:14:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 -- 172.21.15.43:0/1280992276 learned_addr learned my addr 172.21.15.43:0/1280992276 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa782800640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78409bf40 0x7fa78409c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 -- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa764004d00 msgr2=0x7fa7840af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa764004d00 0x7fa7840af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa764004d00 0x7fa7840af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 -- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78409bf40 msgr2=0x7fa78409c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78409bf40 0x7fa78409c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 -- 172.21.15.43:0/1280992276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78409b480 con 0x7fa7840cba40 2024-01-16T15:14:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.123+0000 7fa781400640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00ca50 tx=0x7fa76c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.125+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00a7a0 con 0x7fa7840cba40 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.125+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c00a900 con 0x7fa7840cba40 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.125+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00acf0 con 0x7fa7840cba40 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.125+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78409b760 con 0x7fa7840cba40 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.125+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7840afaa0 con 0x7fa7840cba40 2024-01-16T15:14:36.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.126+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78409d200 con 0x7fa7840cba40 2024-01-16T15:14:36.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.127+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fa76c0070d0 con 0x7fa7840cba40 2024-01-16T15:14:36.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.128+0000 7fa77a000640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 0x7fa75c06e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.128+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fa76c088520 con 0x7fa7840cba40 2024-01-16T15:14:36.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.128+0000 7fa781e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 0x7fa75c06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.129+0000 7fa781e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 0x7fa75c06e580 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa7740099d0 tx=0x7fa774008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.135+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa76c054b30 con 0x7fa7840cba40 2024-01-16T15:14:36.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.364+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa784071d50 con 0x7fa75c06c0c0 2024-01-16T15:14:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.399+0000 7fa77a000640 1 -- 172.21.15.43:0/1280992276 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa774004020 con 0x7fa75c06c0c0 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 msgr2=0x7fa75c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 0x7fa75c06e580 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa7740099d0 tx=0x7fa774008040 comp rx=0 tx=0).stop 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 msgr2=0x7fa7840cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00ca50 tx=0x7fa76c00cf10 comp rx=0 tx=0).stop 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 shutdown_connections 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa75c06c0c0 0x7fa75c06e580 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78409bf40 0x7fa78409c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7840cba40 0x7fa7840cbec0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 --2- 172.21.15.43:0/1280992276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa764004d00 0x7fa7840af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 >> 172.21.15.43:0/1280992276 conn(0x7fa7840b0ba0 msgr2=0x7fa78406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 shutdown_connections 2024-01-16T15:14:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.403+0000 7fa788e00640 1 -- 172.21.15.43:0/1280992276 wait complete. 2024-01-16T15:14:36.388 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.406+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.388 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.406+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.388 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.406+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.389 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.406+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.389 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.406+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:14:36.409 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.427+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.410 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.427+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.410 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.427+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.410 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.427+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.410 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:14:36.427+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:36.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.619+0000 7ff00f000640 1 Processor -- start 2024-01-16T15:14:36.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.620+0000 7ff00f000640 1 -- start start 2024-01-16T15:14:36.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.621+0000 7ff00f000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff0080acd70 0x7ff00806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.621+0000 7ff00f000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00806d7a0 con 0x7ff0080ae490 2024-01-16T15:14:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.621+0000 7ff00f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00806ce70 con 0x7ff008070490 2024-01-16T15:14:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.621+0000 7ff00f000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00815a5b0 con 0x7ff0080acd70 2024-01-16T15:14:36.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.622+0000 7ff00d000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff008070490 0x7ff008070900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:36592/0 (socket says 172.21.15.43:36592) 2024-01-16T15:14:36.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.622+0000 7ff00d000640 1 -- 172.21.15.43:0/2431664058 learned_addr learned my addr 172.21.15.43:0/2431664058 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:36.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735741806 0 0) 0x7ff00806ce70 con 0x7ff008070490 2024-01-16T15:14:36.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8003520 con 0x7ff008070490 2024-01-16T15:14:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009741342 0 0) 0x7fefe8003520 con 0x7ff008070490 2024-01-16T15:14:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 >> v2:172.21.15.43:3300/0 conn(0x7ff0080acd70 msgr2=0x7ff00806dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 --2- 172.21.15.43:0/2431664058 >> v2:172.21.15.43:3300/0 conn(0x7ff0080acd70 0x7ff00806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 >> v1:172.21.15.43:6789/0 conn(0x7ff0080ae490 legacy=0x7ff0080ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00806d150 con 0x7ff008070490 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff0000019f0 con 0x7ff008070490 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.623+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 >> v1:172.21.15.69:6789/0 conn(0x7ff008070490 legacy=0x7ff008070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefe80039c0 0x7fefe8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefe80052a0 0x7fefe800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00815a5b0 con 0x7fefe80039c0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe800c1f0 con 0x7fefe80047c0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe800c3b0 con 0x7fefe80052a0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7fefffe00640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.624+0000 7ff00d000640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefe80052a0 0x7fefe800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7fefffe00640 1 -- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefe80052a0 msgr2=0x7fefe800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7fefffe00640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fefe80052a0 0x7fefe800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7fefffe00640 1 -- 172.21.15.43:0/2431664058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefe80039c0 msgr2=0x7fefe8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7fefffe00640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefe80039c0 0x7fefe8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7fefffe00640 1 -- 172.21.15.43:0/2431664058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8003520 con 0x7fefe80047c0 2024-01-16T15:14:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.625+0000 7ff00d000640 1 -- 172.21.15.43:0/2431664058 reap_dead start 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.627+0000 7fefffe00640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feff0001620 tx=0x7fefe80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd591d94960d267 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.627+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0005070 con 0x7fefe80047c0 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.627+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff002e070 con 0x7fefe80047c0 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.627+0000 7ff007400640 1 -- 172.21.15.43:0/2431664058 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0005430 con 0x7fefe80047c0 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 -- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 msgr2=0x7fefe8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feff0001620 tx=0x7fefe80057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 -- 172.21.15.43:0/2431664058 shutdown_connections 2024-01-16T15:14:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 --2- 172.21.15.43:0/2431664058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 -- 172.21.15.43:0/2431664058 >> 172.21.15.43:0/2431664058 conn(0x7ff0080b0ba0 msgr2=0x7ff0080b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 -- 172.21.15.43:0/2431664058 shutdown_connections 2024-01-16T15:14:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.628+0000 7ff00f000640 1 -- 172.21.15.43:0/2431664058 wait complete. 2024-01-16T15:14:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.630+0000 7ff00f000640 1 Processor -- start 2024-01-16T15:14:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.630+0000 7ff00f000640 1 -- start start 2024-01-16T15:14:36.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7ff0080ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0080c89d0 0x7ff0080b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff008149450 con 0x7ff00814b4e0 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00814ba50 con 0x7fefe80047c0 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.631+0000 7ff00f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00814bd30 con 0x7ff0080c89d0 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38964/0 (socket says 172.21.15.43:38964) 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 -- 172.21.15.43:0/2788553489 learned_addr learned my addr 172.21.15.43:0/2788553489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7ff007e00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7ff0080ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7ff00d000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0080c89d0 0x7ff0080b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 -- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0080c89d0 msgr2=0x7ff0080b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0080c89d0 0x7ff0080b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 -- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 msgr2=0x7ff0080ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7ff0080ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7fefffe00640 1 -- 172.21.15.43:0/2788553489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00814c010 con 0x7ff00814b4e0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.632+0000 7ff007e00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7ff0080ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7fefffe00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7feff002bac0 tx=0x7feff0000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0007460 con 0x7ff00814b4e0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff00075c0 con 0x7ff00814b4e0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00069b0 con 0x7ff00814b4e0 2024-01-16T15:14:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00814c2f0 con 0x7ff00814b4e0 2024-01-16T15:14:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.633+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0080b3460 con 0x7ff00814b4e0 2024-01-16T15:14:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.634+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0080b3710 con 0x7ff00814b4e0 2024-01-16T15:14:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.635+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7feff0030030 con 0x7ff00814b4e0 2024-01-16T15:14:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.637+0000 7ff004c00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 0x7fefe006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.637+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7feff00a9730 con 0x7ff00814b4e0 2024-01-16T15:14:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.637+0000 7ff007e00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 0x7fefe006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.638+0000 7ff007e00640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 0x7fefe006e500 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:36.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.641+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feff0075c40 con 0x7ff00814b4e0 2024-01-16T15:14:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.868+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0080b3d90 con 0x7fefe006c040 2024-01-16T15:14:36.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.897+0000 7ff004c00640 1 -- 172.21.15.43:0/2788553489 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feff8009600 con 0x7fefe006c040 2024-01-16T15:14:36.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.899+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 msgr2=0x7fefe006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.899+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 0x7fefe006e500 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8008040 comp rx=0 tx=0).stop 2024-01-16T15:14:36.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.899+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 msgr2=0x7ff0080c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:36.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.899+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7feff002bac0 tx=0x7feff0000fb0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 shutdown_connections 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fefe006c040 0x7fefe006e500 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff0080c89d0 0x7ff0080b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff00814b4e0 0x7ff0080c8490 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 --2- 172.21.15.43:0/2788553489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe80047c0 0x7ff0080ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 >> 172.21.15.43:0/2788553489 conn(0x7ff0080b0ba0 msgr2=0x7ff0080b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 shutdown_connections 2024-01-16T15:14:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:36.900+0000 7ff00f000640 1 -- 172.21.15.43:0/2788553489 wait complete. 2024-01-16T15:14:36.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:36.903 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:14:36.956 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:14:36.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:14:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.163+0000 7f0879200640 1 Processor -- start 2024-01-16T15:14:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.163+0000 7f0879200640 1 -- start start 2024-01-16T15:14:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f0879200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0874153ea0 0x7f08740ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f0879200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0874154280 0x7f087415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f0879200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08740ce650 con 0x7f0874070850 2024-01-16T15:14:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f0879200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0874154710 con 0x7f0874154280 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f0879200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087415ddd0 con 0x7f0874153ea0 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f086be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0874154280 0x7f087415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f086be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0874154280 0x7f087415d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38968/0 (socket says 172.21.15.43:38968) 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.165+0000 7f086be00640 1 -- 172.21.15.43:0/1817031634 learned_addr learned my addr 172.21.15.43:0/1817031634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.166+0000 7f086be00640 1 -- 172.21.15.43:0/1817031634 >> v2:172.21.15.69:3300/0 conn(0x7f0874153ea0 msgr2=0x7f08740ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.166+0000 7f0873400640 1 --2- 172.21.15.43:0/1817031634 >> v2:172.21.15.69:3300/0 conn(0x7f0874153ea0 0x7f08740ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.166+0000 7f086be00640 1 --2- 172.21.15.43:0/1817031634 >> v2:172.21.15.69:3300/0 conn(0x7f0874153ea0 0x7f08740ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.166+0000 7f086be00640 1 -- 172.21.15.43:0/1817031634 >> v1:172.21.15.69:6789/0 conn(0x7f0874070850 legacy=0x7f0874071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.166+0000 7f086be00640 1 -- 172.21.15.43:0/1817031634 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087415e0a0 con 0x7f0874154280 2024-01-16T15:14:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.167+0000 7f086be00640 1 --2- 172.21.15.43:0/1817031634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f087415d890 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f08740cfc10 tx=0x7f085c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e452954a44d56c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.167+0000 7f0872a00640 1 -- 172.21.15.43:0/1817031634 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c031870 con 0x7f0874154280 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.167+0000 7f0872a00640 1 -- 172.21.15.43:0/1817031634 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c0319d0 con 0x7f0874154280 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.167+0000 7f0872a00640 1 -- 172.21.15.43:0/1817031634 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c031d60 con 0x7f0874154280 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 -- 172.21.15.43:0/1817031634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 msgr2=0x7f087415d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 --2- 172.21.15.43:0/1817031634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f087415d890 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f08740cfc10 tx=0x7f085c02b910 comp rx=0 tx=0).stop 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 -- 172.21.15.43:0/1817031634 shutdown_connections 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 --2- 172.21.15.43:0/1817031634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f087415d890 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 --2- 172.21.15.43:0/1817031634 >> v2:172.21.15.69:3300/0 conn(0x7f0874153ea0 0x7f08740ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 -- 172.21.15.43:0/1817031634 >> 172.21.15.43:0/1817031634 conn(0x7f08740cec10 msgr2=0x7f08740cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 -- 172.21.15.43:0/1817031634 shutdown_connections 2024-01-16T15:14:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.168+0000 7f0879200640 1 -- 172.21.15.43:0/1817031634 wait complete. 2024-01-16T15:14:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.170+0000 7f0879200640 1 Processor -- start 2024-01-16T15:14:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.170+0000 7f0879200640 1 -- start start 2024-01-16T15:14:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0874070850 0x7f0874143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0874153ea0 0x7f0874154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874143ba0 con 0x7f0874154280 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08741561d0 con 0x7f0874153ea0 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0879200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874155130 con 0x7f0874070850 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38980/0 (socket says 172.21.15.43:38980) 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 -- 172.21.15.43:0/4252708527 learned_addr learned my addr 172.21.15.43:0/4252708527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873400640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0874070850 0x7f0874143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f086be00640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0874153ea0 0x7f0874154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 -- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0874070850 msgr2=0x7f0874143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0874070850 0x7f0874143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 -- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0874153ea0 msgr2=0x7f0874154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0874153ea0 0x7f0874154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.171+0000 7f0873e00640 1 -- 172.21.15.43:0/4252708527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c0094d0 con 0x7f0874154280 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f0873e00640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c780 tx=0x7f086c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c004b60 con 0x7f0874154280 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00ce90 con 0x7f0874154280 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0874155410 con 0x7f0874154280 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00a850 con 0x7f0874154280 2024-01-16T15:14:37.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.172+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08741559d0 con 0x7f0874154280 2024-01-16T15:14:37.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.173+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087404e840 con 0x7f0874154280 2024-01-16T15:14:37.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.174+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 33) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f086c00a9b0 con 0x7f0874154280 2024-01-16T15:14:37.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.175+0000 7f086b400640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 0x7f084c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:37.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.175+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f086c088a10 con 0x7f0874154280 2024-01-16T15:14:37.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.175+0000 7f0873400640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 0x7f084c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.176+0000 7f0873400640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 0x7f084c06e530 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.179+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f086c055020 con 0x7f0874154280 2024-01-16T15:14:37.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:37.395+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f0874070050 con 0x7f084c06c070 2024-01-16T15:14:37.684 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:14:37.700+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:14:37.774 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:14:38.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:38.156+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 34) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f086c0885b0 con 0x7f0874154280 2024-01-16T15:14:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.166+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f086c0553f0 con 0x7f0874154280 2024-01-16T15:14:40.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.756+0000 7f086b400640 1 -- 172.21.15.43:0/4252708527 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f086400c040 con 0x7f084c06c070 2024-01-16T15:14:40.742 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:14:40.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 msgr2=0x7f084c06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 0x7f084c06e530 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864008040 comp rx=0 tx=0).stop 2024-01-16T15:14:40.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 msgr2=0x7f08741570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c780 tx=0x7f086c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 shutdown_connections 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f084c06c070 0x7f084c06e530 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0874153ea0 0x7f0874154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.761+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0874154280 0x7f08741570f0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.762+0000 7f0879200640 1 --2- 172.21.15.43:0/4252708527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0874070850 0x7f0874143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.762+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 >> 172.21.15.43:0/4252708527 conn(0x7f08740cec10 msgr2=0x7f0874160180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.762+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 shutdown_connections 2024-01-16T15:14:40.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:40.763+0000 7f0879200640 1 -- 172.21.15.43:0/4252708527 wait complete. 2024-01-16T15:14:40.775 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:14:40.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:40.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:40.967 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:14:40.967 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:14:40.967 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:41.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:41.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.266+0000 7f5658000640 1 Processor -- start 2024-01-16T15:14:41.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.266+0000 7f5658000640 1 -- start start 2024-01-16T15:14:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.268+0000 7f5658000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f565015b450 0x7f56500ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.268+0000 7f5658000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 0x7f56500adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.268+0000 7f5658000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56500ad4d0 con 0x7f56500af830 2024-01-16T15:14:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.268+0000 7f5658000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56500b2db0 con 0x7f565015b450 2024-01-16T15:14:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.268+0000 7f5658000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b5670 con 0x7f5650099a00 2024-01-16T15:14:41.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5654c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 0x7f56500adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5655600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f565015b450 0x7f56500ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5656000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f56500af830 0x7f565006e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:36610/0 (socket says 172.21.15.43:36610) 2024-01-16T15:14:41.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5656000640 1 -- 172.21.15.43:0/3199602152 learned_addr learned my addr 172.21.15.43:0/3199602152 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5654c00640 1 -- 172.21.15.43:0/3199602152 >> v2:172.21.15.69:3300/0 conn(0x7f565015b450 msgr2=0x7f56500ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5654c00640 1 --2- 172.21.15.43:0/3199602152 >> v2:172.21.15.69:3300/0 conn(0x7f565015b450 0x7f56500ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5654c00640 1 -- 172.21.15.43:0/3199602152 >> v1:172.21.15.69:6789/0 conn(0x7f56500af830 legacy=0x7f565006e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.269+0000 7f5654c00640 1 -- 172.21.15.43:0/3199602152 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565006af80 con 0x7f5650099a00 2024-01-16T15:14:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.271+0000 7f5654c00640 1 --2- 172.21.15.43:0/3199602152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 0x7f56500adbf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56500ad660 tx=0x7f563802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=294be4f8c61c06fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.271+0000 7f564fe00640 1 -- 172.21.15.43:0/3199602152 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638031780 con 0x7f5650099a00 2024-01-16T15:14:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.271+0000 7f564fe00640 1 -- 172.21.15.43:0/3199602152 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56380318e0 con 0x7f5650099a00 2024-01-16T15:14:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.271+0000 7f564fe00640 1 -- 172.21.15.43:0/3199602152 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638031c70 con 0x7f5650099a00 2024-01-16T15:14:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 -- 172.21.15.43:0/3199602152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 msgr2=0x7f56500adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 --2- 172.21.15.43:0/3199602152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 0x7f56500adbf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56500ad660 tx=0x7f563802b820 comp rx=0 tx=0).stop 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 -- 172.21.15.43:0/3199602152 shutdown_connections 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 --2- 172.21.15.43:0/3199602152 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5650099a00 0x7f56500adbf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 --2- 172.21.15.43:0/3199602152 >> v2:172.21.15.69:3300/0 conn(0x7f565015b450 0x7f56500ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 -- 172.21.15.43:0/3199602152 >> 172.21.15.43:0/3199602152 conn(0x7f5650071b00 msgr2=0x7f56500b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 -- 172.21.15.43:0/3199602152 shutdown_connections 2024-01-16T15:14:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.272+0000 7f5658000640 1 -- 172.21.15.43:0/3199602152 wait complete. 2024-01-16T15:14:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.274+0000 7f5658000640 1 Processor -- start 2024-01-16T15:14:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.274+0000 7f5658000640 1 -- start start 2024-01-16T15:14:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 0x7f56501f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 0x7f56500b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56501f45f0 con 0x7f56500af830 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b4190 con 0x7f5650099a00 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5658000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b32a0 con 0x7f565015b450 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5654c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5654c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39016/0 (socket says 172.21.15.43:39016) 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.275+0000 7f5654c00640 1 -- 172.21.15.43:0/3535237274 learned_addr learned my addr 172.21.15.43:0/3535237274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5655600640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 0x7f56501f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5656000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 0x7f56500b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 -- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 msgr2=0x7f56500b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 0x7f56500b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 -- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 msgr2=0x7f56501f40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 0x7f56501f40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 -- 172.21.15.43:0/3535237274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638009400 con 0x7f56500af830 2024-01-16T15:14:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5656000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 0x7f56500b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5655600640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 0x7f56501f40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5654c00640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f56380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638032820 con 0x7f56500af830 2024-01-16T15:14:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638032980 con 0x7f56500af830 2024-01-16T15:14:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638032d10 con 0x7f56500af830 2024-01-16T15:14:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.276+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56500b3580 con 0x7f56500af830 2024-01-16T15:14:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.277+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56500b3b40 con 0x7f56500af830 2024-01-16T15:14:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.278+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56500aec20 con 0x7f56500af830 2024-01-16T15:14:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.279+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f563804e450 con 0x7f56500af830 2024-01-16T15:14:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.279+0000 7f564d600640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 0x7f562806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.280+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f56380aa8e0 con 0x7f56500af830 2024-01-16T15:14:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.280+0000 7f5655600640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 0x7f562806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.280+0000 7f5655600640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 0x7f562806e590 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f56500b1020 tx=0x7f5640008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.284+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5638076f90 con 0x7f56500af830 2024-01-16T15:14:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.534+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56500b2560 con 0x7f56500af830 2024-01-16T15:14:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.535+0000 7f564d600640 1 -- 172.21.15.43:0/3535237274 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12982 (secure 0 0 0) 0x7f563807a5b0 con 0x7f56500af830 2024-01-16T15:14:41.518 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:41.518 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":93,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:40.716799+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3379600529":"2024-01-17T15:14:37.701304+0000","172.21.15.43:6836/3379600529":"2024-01-17T15:14:37.701304+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:41.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 msgr2=0x7f562806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 0x7f562806e590 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f56500b1020 tx=0x7f5640008040 comp rx=0 tx=0).stop 2024-01-16T15:14:41.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 msgr2=0x7f56500b10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f56380046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 shutdown_connections 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f562806c0d0 0x7f562806e590 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565015b450 0x7f56500b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56500af830 0x7f56500b10d0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 --2- 172.21.15.43:0/3535237274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5650099a00 0x7f56501f40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 >> 172.21.15.43:0/3535237274 conn(0x7f5650071b00 msgr2=0x7f565014ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 shutdown_connections 2024-01-16T15:14:41.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.538+0000 7f5658000640 1 -- 172.21.15.43:0/3535237274 wait complete. 2024-01-16T15:14:41.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:41.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.799+0000 7f389ce00640 1 Processor -- start 2024-01-16T15:14:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.799+0000 7f389ce00640 1 -- start start 2024-01-16T15:14:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f389ce00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3898160bd0 0x7f38980b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f389ce00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 0x7f3898167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f389ce00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980b23e0 con 0x7f38980b48d0 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f389ce00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38980cdc70 con 0x7f3898160bd0 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f389ce00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980b2870 con 0x7f38980b3210 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.800+0000 7f3895e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3898160bd0 0x7f38980b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3898160bd0 0x7f38980b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39022/0 (socket says 172.21.15.43:39022) 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f388d200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f38980b48d0 0x7f38980b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49412/0 (socket says 172.21.15.43:49412) 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 -- 172.21.15.43:0/2694336983 learned_addr learned my addr 172.21.15.43:0/2694336983 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3896800640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 0x7f3898167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 -- 172.21.15.43:0/2694336983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 msgr2=0x7f3898167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 0x7f3898167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 -- 172.21.15.43:0/2694336983 >> v1:172.21.15.43:6789/0 conn(0x7f38980b48d0 legacy=0x7f38980b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895400640 1 -- 172.21.15.43:0/2694336983 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103399975 0 0) 0x7f38980b23e0 con 0x7f38980b48d0 2024-01-16T15:14:41.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3895e00640 1 -- 172.21.15.43:0/2694336983 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980d41b0 con 0x7f3898160bd0 2024-01-16T15:14:41.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.801+0000 7f3896800640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 0x7f3898167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.802+0000 7f3895e00640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3898160bd0 0x7f38980b1ea0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3888009e50 tx=0x7f388802bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236d3d8d24139965 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.802+0000 7f3895400640 1 -- 172.21.15.43:0/2694336983 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388802caa0 con 0x7f3898160bd0 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.802+0000 7f3895400640 1 -- 172.21.15.43:0/2694336983 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388802cc00 con 0x7f3898160bd0 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.802+0000 7f3895400640 1 -- 172.21.15.43:0/2694336983 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38880343f0 con 0x7f3898160bd0 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 -- 172.21.15.43:0/2694336983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3898160bd0 msgr2=0x7f38980b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3898160bd0 0x7f38980b1ea0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3888009e50 tx=0x7f388802bf00 comp rx=0 tx=0).stop 2024-01-16T15:14:41.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 -- 172.21.15.43:0/2694336983 shutdown_connections 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38980b3210 0x7f3898167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 --2- 172.21.15.43:0/2694336983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3898160bd0 0x7f38980b1ea0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 -- 172.21.15.43:0/2694336983 >> 172.21.15.43:0/2694336983 conn(0x7f389809c5a0 msgr2=0x7f38980d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 -- 172.21.15.43:0/2694336983 shutdown_connections 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.803+0000 7f389ce00640 1 -- 172.21.15.43:0/2694336983 wait complete. 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.805+0000 7f389ce00640 1 Processor -- start 2024-01-16T15:14:41.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.805+0000 7f389ce00640 1 -- start start 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38980b3210 0x7f38980b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 0x7f389815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980cbbb0 con 0x7f38980b48d0 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980cbe90 con 0x7f38980b3210 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f389ce00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980cc170 con 0x7f3898160bd0 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f388d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f388d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39038/0 (socket says 172.21.15.43:39038) 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.806+0000 7f388d200640 1 -- 172.21.15.43:0/3357020004 learned_addr learned my addr 172.21.15.43:0/3357020004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f3896800640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 0x7f389815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 -- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 msgr2=0x7f389815d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 0x7f389815d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 -- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38980b3210 msgr2=0x7f38980b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38980b3210 0x7f38980b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 -- 172.21.15.43:0/3357020004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888009b30 con 0x7f38980b48d0 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f3896800640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 0x7f389815d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388d200640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f388000c450 tx=0x7f388000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800074e0 con 0x7f38980b48d0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880007660 con 0x7f38980b48d0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880004a20 con 0x7f38980b48d0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980cc450 con 0x7f38980b48d0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.807+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38980ccc50 con 0x7f38980b48d0 2024-01-16T15:14:41.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.809+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3898067f80 con 0x7f38980b48d0 2024-01-16T15:14:41.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.809+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f3880004b80 con 0x7f38980b48d0 2024-01-16T15:14:41.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.810+0000 7f388ea00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 0x7f387006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.811+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f3880087ad0 con 0x7f38980b48d0 2024-01-16T15:14:41.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.811+0000 7f3895e00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 0x7f387006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.811+0000 7f3895e00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 0x7f387006e590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3888035000 tx=0x7f388802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:41.815+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3880054130 con 0x7f38980b48d0 2024-01-16T15:14:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.000+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38980cc970 con 0x7f38980b48d0 2024-01-16T15:14:41.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.001+0000 7f388ea00640 1 -- 172.21.15.43:0/3357020004 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38980cc970 con 0x7f38980b48d0 2024-01-16T15:14:41.983 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:14:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.003+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 msgr2=0x7f387006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.003+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 0x7f387006e590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3888035000 tx=0x7f388802e000 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.003+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 msgr2=0x7f38980cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.003+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f388000c450 tx=0x7f388000c910 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 shutdown_connections 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f387006c0d0 0x7f387006e590 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3898160bd0 0x7f389815d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980b48d0 0x7f38980cb360 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 --2- 172.21.15.43:0/3357020004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38980b3210 0x7f38980b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 >> 172.21.15.43:0/3357020004 conn(0x7f389809c5a0 msgr2=0x7f38980cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 shutdown_connections 2024-01-16T15:14:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.004+0000 7f389ce00640 1 -- 172.21.15.43:0/3357020004 wait complete. 2024-01-16T15:14:42.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:14:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.238+0000 7fafba800640 1 Processor -- start 2024-01-16T15:14:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.238+0000 7fafba800640 1 -- start start 2024-01-16T15:14:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.239+0000 7fafba800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fafb415c5a0 0x7fafb4140930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.239+0000 7fafba800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb4140e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.239+0000 7fafba800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40704e0 con 0x7fafb415bab0 2024-01-16T15:14:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.239+0000 7fafba800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb413e8a0 con 0x7fafb415c5a0 2024-01-16T15:14:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.239+0000 7fafba800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40707c0 con 0x7fafb406ff20 2024-01-16T15:14:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.240+0000 7fafb3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb4140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.240+0000 7fafb3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fafb415bab0 0x7fafb415be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:36626/0 (socket says 172.21.15.43:36626) 2024-01-16T15:14:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb3400640 1 -- 172.21.15.43:0/1663361025 learned_addr learned my addr 172.21.15.43:0/1663361025 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb2a00640 1 --2- 172.21.15.43:0/1663361025 >> v2:172.21.15.69:3300/0 conn(0x7fafb415c5a0 0x7fafb4140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb3e00640 1 -- 172.21.15.43:0/1663361025 >> v2:172.21.15.69:3300/0 conn(0x7fafb415c5a0 msgr2=0x7fafb4140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:42.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb3e00640 1 --2- 172.21.15.43:0/1663361025 >> v2:172.21.15.69:3300/0 conn(0x7fafb415c5a0 0x7fafb4140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb3e00640 1 -- 172.21.15.43:0/1663361025 >> v1:172.21.15.69:6789/0 conn(0x7fafb415bab0 legacy=0x7fafb415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.241+0000 7fafb3e00640 1 -- 172.21.15.43:0/1663361025 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4070aa0 con 0x7fafb406ff20 2024-01-16T15:14:42.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.243+0000 7fafb3e00640 1 --2- 172.21.15.43:0/1663361025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb4140e70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fafac009d90 tx=0x7fafac02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4f40e0fb1e4295 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:42.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.244+0000 7fafb2000640 1 -- 172.21.15.43:0/1663361025 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac02cc60 con 0x7fafb406ff20 2024-01-16T15:14:42.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.244+0000 7fafb2000640 1 -- 172.21.15.43:0/1663361025 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac02cdc0 con 0x7fafb406ff20 2024-01-16T15:14:42.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.244+0000 7fafb2000640 1 -- 172.21.15.43:0/1663361025 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac034660 con 0x7fafb406ff20 2024-01-16T15:14:42.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.244+0000 7fafba800640 1 -- 172.21.15.43:0/1663361025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 msgr2=0x7fafb4140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.244+0000 7fafba800640 1 --2- 172.21.15.43:0/1663361025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb4140e70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fafac009d90 tx=0x7fafac02beb0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 -- 172.21.15.43:0/1663361025 shutdown_connections 2024-01-16T15:14:42.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 --2- 172.21.15.43:0/1663361025 >> v2:172.21.15.69:3300/0 conn(0x7fafb415c5a0 0x7fafb4140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 --2- 172.21.15.43:0/1663361025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb4140e70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 -- 172.21.15.43:0/1663361025 >> 172.21.15.43:0/1663361025 conn(0x7fafb40b2e00 msgr2=0x7fafb40b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:42.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 -- 172.21.15.43:0/1663361025 shutdown_connections 2024-01-16T15:14:42.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.245+0000 7fafba800640 1 -- 172.21.15.43:0/1663361025 wait complete. 2024-01-16T15:14:42.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.246+0000 7fafba800640 1 Processor -- start 2024-01-16T15:14:42.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 -- start start 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb41f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 0x7fafb4152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415df80 con 0x7fafb415bab0 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb41f3580 con 0x7fafb415c5a0 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.247+0000 7fafba800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415d370 con 0x7fafb406ff20 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39054/0 (socket says 172.21.15.43:39054) 2024-01-16T15:14:42.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 -- 172.21.15.43:0/4101966875 learned_addr learned my addr 172.21.15.43:0/4101966875 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb3400640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb41f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb3e00640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 0x7fafb4152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 -- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 msgr2=0x7fafb41f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb41f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 -- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 msgr2=0x7fafb4152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 0x7fafb4152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb2a00640 1 -- 172.21.15.43:0/4101966875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac009a70 con 0x7fafb415bab0 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb3400640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb41f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.248+0000 7fafb3e00640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 0x7fafb4152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:42.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafb2a00640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00e9b0 tx=0x7faf9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:42.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c009670 con 0x7fafb415bab0 2024-01-16T15:14:42.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c00c9a0 con 0x7fafb415bab0 2024-01-16T15:14:42.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c00cd50 con 0x7fafb415bab0 2024-01-16T15:14:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb415d650 con 0x7fafb415bab0 2024-01-16T15:14:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.249+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb415dc10 con 0x7fafb415bab0 2024-01-16T15:14:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.251+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb4074770 con 0x7fafb415bab0 2024-01-16T15:14:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.252+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7faf9c013470 con 0x7fafb415bab0 2024-01-16T15:14:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.252+0000 7fafab400640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 0x7faf8c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.252+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7faf9c088b40 con 0x7fafb415bab0 2024-01-16T15:14:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.253+0000 7fafb3400640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 0x7faf8c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.253+0000 7fafb3400640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 0x7faf8c06e590 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fafa4000f60 tx=0x7fafa4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.257+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf9c055140 con 0x7fafb415bab0 2024-01-16T15:14:42.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.456+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fafb4072af0 con 0x7fafb415bab0 2024-01-16T15:14:42.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.457+0000 7fafab400640 1 -- 172.21.15.43:0/4101966875 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v93) v1 ==== 80+0+122 (secure 0 0 0) 0x7faf9c058760 con 0x7fafb415bab0 2024-01-16T15:14:42.440 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6837/3379600529 2024-01-17T15:14:37.701304+0000 2024-01-16T15:14:42.440 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3379600529 2024-01-17T15:14:37.701304+0000 2024-01-16T15:14:42.440 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 msgr2=0x7faf8c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 0x7faf8c06e590 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fafa4000f60 tx=0x7fafa4009040 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 msgr2=0x7fafb415c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00e9b0 tx=0x7faf9c00ee70 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 shutdown_connections 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf8c06c0d0 0x7faf8c06e590 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafb415c5a0 0x7fafb4152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafb415bab0 0x7fafb415c9b0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 --2- 172.21.15.43:0/4101966875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fafb406ff20 0x7fafb41f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 >> 172.21.15.43:0/4101966875 conn(0x7fafb40b2e00 msgr2=0x7fafb406b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 shutdown_connections 2024-01-16T15:14:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:42.460+0000 7fafba800640 1 -- 172.21.15.43:0/4101966875 wait complete. 2024-01-16T15:14:42.461 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:14:42.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:42.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:14:42.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[68697]: fuse finished with error 0 and tester_r 0 2024-01-16T15:14:42.559 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:14:42.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:42.560 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:14:42.821 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:14:42.821 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:42.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:43.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.127+0000 7f4182000640 1 Processor -- start 2024-01-16T15:14:43.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.127+0000 7f4182000640 1 -- start start 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f4182000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f417c070780 0x7f417c06d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f4182000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f417c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f4182000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417c071cd0 con 0x7f417c160970 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f4182000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c0b2120 con 0x7f417c070780 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f4182000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0b2400 con 0x7f417c070cd0 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f417b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f417c160970 0x7f417c0b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32808/0 (socket says 172.21.15.43:32808) 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.129+0000 7f417b400640 1 -- 172.21.15.43:0/3892993990 learned_addr learned my addr 172.21.15.43:0/3892993990 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:43.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f4172800640 1 --2- 172.21.15.43:0/3892993990 >> v2:172.21.15.69:3300/0 conn(0x7f417c070780 0x7f417c06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417be00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f417c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321068173 0 0) 0x7f417c071cd0 con 0x7f417c160970 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c003520 con 0x7f417c160970 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1441645917 0 0) 0x7f415c003520 con 0x7f417c160970 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 msgr2=0x7f417c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f417c06da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 >> v2:172.21.15.69:3300/0 conn(0x7f417c070780 msgr2=0x7f417c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 --2- 172.21.15.43:0/3892993990 >> v2:172.21.15.69:3300/0 conn(0x7f417c070780 0x7f417c06d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0b26e0 con 0x7f417c160970 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f416c001670 con 0x7f417c160970 2024-01-16T15:14:43.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 >> v1:172.21.15.43:6789/0 conn(0x7f417c160970 legacy=0x7f417c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f415c003970 0x7f415c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f415c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f415c004d00 0x7f415c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0b2120 con 0x7f415c003970 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0b2400 con 0x7f415c004d00 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.130+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c00b980 con 0x7f417c070cd0 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f415c003970 0x7f415c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f4172800640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f415c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417be00640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f415c004d00 0x7f415c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 -- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 msgr2=0x7f415c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f417c070cd0 0x7f415c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 -- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f415c004d00 msgr2=0x7f415c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:43.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f415c004d00 0x7f415c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.131+0000 7f417b400640 1 -- 172.21.15.43:0/3892993990 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0b26e0 con 0x7f415c003970 2024-01-16T15:14:43.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.133+0000 7f417b400640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f415c003970 0x7f415c003d90 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c210 tx=0x7f415c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bcc51ab76e3224 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.133+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c02cae0 con 0x7f415c003970 2024-01-16T15:14:43.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.133+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c02cc60 con 0x7f415c003970 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.133+0000 7f417aa00640 1 -- 172.21.15.43:0/3892993990 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c0058d0 con 0x7f415c003970 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 -- 172.21.15.43:0/3892993990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f415c003970 msgr2=0x7f415c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 --2- 172.21.15.43:0/3892993990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f415c003970 0x7f415c003d90 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c210 tx=0x7f415c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f417be00640 1 -- 172.21.15.43:0/3892993990 reap_dead start 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 -- 172.21.15.43:0/3892993990 shutdown_connections 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 -- 172.21.15.43:0/3892993990 >> 172.21.15.43:0/3892993990 conn(0x7f417c0b0d80 msgr2=0x7f417c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 -- 172.21.15.43:0/3892993990 shutdown_connections 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.134+0000 7f4182000640 1 -- 172.21.15.43:0/3892993990 wait complete. 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.136+0000 7f4182000640 1 Processor -- start 2024-01-16T15:14:43.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.136+0000 7f4182000640 1 -- start start 2024-01-16T15:14:43.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f415c003970 0x7f417c149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 0x7f417c149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c146e80 con 0x7f417c149e90 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c144c00 con 0x7f417c1472f0 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f4182000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c143b20 con 0x7f415c003970 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f417be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f417be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37638/0 (socket says 172.21.15.43:37638) 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.137+0000 7f417be00640 1 -- 172.21.15.43:0/2841055097 learned_addr learned my addr 172.21.15.43:0/2841055097 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417b400640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f415c003970 0x7f417c149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4172800640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 0x7f417c149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 -- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f415c003970 msgr2=0x7f417c149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f415c003970 0x7f417c149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 -- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 msgr2=0x7f417c149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 0x7f417c149950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 -- 172.21.15.43:0/2841055097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c143f80 con 0x7f417c149e90 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4172800640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 0x7f417c149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f417be00640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f417c06b420 tx=0x7f41740115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400e070 con 0x7f417c149e90 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174011db0 con 0x7f417c149e90 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c144260 con 0x7f417c149e90 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.138+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174003990 con 0x7f417c149e90 2024-01-16T15:14:43.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.139+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c144820 con 0x7f417c149e90 2024-01-16T15:14:43.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.140+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c144d40 con 0x7f417c149e90 2024-01-16T15:14:43.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.141+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f4174003af0 con 0x7f417c149e90 2024-01-16T15:14:43.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.142+0000 7f4173e00640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 0x7f415406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.142+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f4174089140 con 0x7f417c149e90 2024-01-16T15:14:43.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.142+0000 7f417b400640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 0x7f415406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.143+0000 7f417b400640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 0x7f415406e590 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f416c034000 tx=0x7f416c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.147+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f417c144d40 con 0x7f417c149e90 2024-01-16T15:14:43.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.341+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f417c144d40 con 0x7f417c149e90 2024-01-16T15:14:43.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.343+0000 7f4173e00640 1 -- 172.21.15.43:0/2841055097 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12982 (secure 0 0 0) 0x7f417c144d40 con 0x7f417c149e90 2024-01-16T15:14:43.326 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:43.326 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":93,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:40.716799+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3379600529":"2024-01-17T15:14:37.701304+0000","172.21.15.43:6836/3379600529":"2024-01-17T15:14:37.701304+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 msgr2=0x7f415406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 0x7f415406e590 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f416c034000 tx=0x7f416c007000 comp rx=0 tx=0).stop 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 msgr2=0x7f417c15a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f417c06b420 tx=0x7f41740115a0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 shutdown_connections 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f415406c0d0 0x7f415406e590 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f417c1472f0 0x7f417c149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f417c149e90 0x7f417c15a9e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 --2- 172.21.15.43:0/2841055097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f415c003970 0x7f417c149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 >> 172.21.15.43:0/2841055097 conn(0x7f417c0b0d80 msgr2=0x7f417c0af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:43.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 shutdown_connections 2024-01-16T15:14:43.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.346+0000 7f4182000640 1 -- 172.21.15.43:0/2841055097 wait complete. 2024-01-16T15:14:43.350 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:43.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:43.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.593+0000 7fdb8b200640 1 Processor -- start 2024-01-16T15:14:43.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.593+0000 7fdb8b200640 1 -- start start 2024-01-16T15:14:43.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb8b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb8406ee10 0x7fdb840d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb8b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb84144650 0x7fdb840d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb8b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb840c79a0 con 0x7fdb84143ff0 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb8b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb840c7ed0 con 0x7fdb8406ee10 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb8b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840ced50 con 0x7fdb84144650 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb83e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdb84143ff0 0x7fdb8406e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32818/0 (socket says 172.21.15.43:32818) 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb83e00640 1 -- 172.21.15.43:0/1667300027 learned_addr learned my addr 172.21.15.43:0/1667300027 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb83400640 1 --2- 172.21.15.43:0/1667300027 >> v2:172.21.15.69:3300/0 conn(0x7fdb8406ee10 0x7fdb840d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.595+0000 7fdb89200640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb84144650 0x7fdb840d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb82a00640 1 -- 172.21.15.43:0/1667300027 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713305197 0 0) 0x7fdb840c79a0 con 0x7fdb84143ff0 2024-01-16T15:14:43.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb82a00640 1 -- 172.21.15.43:0/1667300027 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb64003520 con 0x7fdb84143ff0 2024-01-16T15:14:43.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb83400640 1 -- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb84144650 msgr2=0x7fdb840d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:43.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb83400640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb84144650 0x7fdb840d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb83400640 1 -- 172.21.15.43:0/1667300027 >> v1:172.21.15.43:6789/0 conn(0x7fdb84143ff0 legacy=0x7fdb8406e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.596+0000 7fdb83400640 1 -- 172.21.15.43:0/1667300027 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb841443d0 con 0x7fdb8406ee10 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.597+0000 7fdb83400640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8406ee10 0x7fdb840d08e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0097a0 tx=0x7fdb6c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd624a540a7547bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.598+0000 7fdb82a00640 1 -- 172.21.15.43:0/1667300027 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c02d040 con 0x7fdb8406ee10 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.598+0000 7fdb82a00640 1 -- 172.21.15.43:0/1667300027 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6c031810 con 0x7fdb8406ee10 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.598+0000 7fdb82a00640 1 -- 172.21.15.43:0/1667300027 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c031ba0 con 0x7fdb8406ee10 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.598+0000 7fdb8b200640 1 -- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8406ee10 msgr2=0x7fdb840d08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.598+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8406ee10 0x7fdb840d08e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0097a0 tx=0x7fdb6c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 -- 172.21.15.43:0/1667300027 shutdown_connections 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb84144650 0x7fdb840d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1667300027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8406ee10 0x7fdb840d08e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 -- 172.21.15.43:0/1667300027 >> 172.21.15.43:0/1667300027 conn(0x7fdb840661f0 msgr2=0x7fdb840665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 -- 172.21.15.43:0/1667300027 shutdown_connections 2024-01-16T15:14:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.599+0000 7fdb8b200640 1 -- 172.21.15.43:0/1667300027 wait complete. 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.600+0000 7fdb8b200640 1 Processor -- start 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.600+0000 7fdb8b200640 1 -- start start 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 0x7fdb84140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 0x7fdb840c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840c9330 con 0x7fdb84144650 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840c9050 con 0x7fdb84143ff0 2024-01-16T15:14:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.601+0000 7fdb8b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840b3ee0 con 0x7fdb8406ee10 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37640/0 (socket says 172.21.15.43:37640) 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 -- 172.21.15.43:0/1638621868 learned_addr learned my addr 172.21.15.43:0/1638621868 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb83400640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 0x7fdb840c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb83e00640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 0x7fdb84140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 -- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 msgr2=0x7fdb84140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 0x7fdb84140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 -- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 msgr2=0x7fdb840c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 0x7fdb840c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb89200640 1 -- 172.21.15.43:0/1638621868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c009400 con 0x7fdb84144650 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb83400640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 0x7fdb840c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.602+0000 7fdb83e00640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 0x7fdb84140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb89200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00efc0 tx=0x7fdb7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c009b20 con 0x7fdb84144650 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c00f040 con 0x7fdb84144650 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0146c0 con 0x7fdb84144650 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb840b41c0 con 0x7fdb84144650 2024-01-16T15:14:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.603+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb841eedd0 con 0x7fdb84144650 2024-01-16T15:14:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.604+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb840b3d40 con 0x7fdb84144650 2024-01-16T15:14:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.605+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fdb7c014820 con 0x7fdb84144650 2024-01-16T15:14:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.606+0000 7fdb7be00640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 0x7fdb5c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.606+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fdb7c0896f0 con 0x7fdb84144650 2024-01-16T15:14:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.607+0000 7fdb83e00640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 0x7fdb5c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.607+0000 7fdb83e00640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 0x7fdb5c06e590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdb740011e0 tx=0x7fdb74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.611+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb840b3d40 con 0x7fdb84144650 2024-01-16T15:14:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.805+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb8415f790 con 0x7fdb84144650 2024-01-16T15:14:43.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.806+0000 7fdb7be00640 1 -- 172.21.15.43:0/1638621868 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb8415f790 con 0x7fdb84144650 2024-01-16T15:14:43.789 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:14:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 msgr2=0x7fdb5c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 0x7fdb5c06e590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdb740011e0 tx=0x7fdb74009040 comp rx=0 tx=0).stop 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 msgr2=0x7fdb840ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00efc0 tx=0x7fdb7c00c490 comp rx=0 tx=0).stop 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 shutdown_connections 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb5c06c0d0 0x7fdb5c06e590 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb84143ff0 0x7fdb840c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb84144650 0x7fdb840ca3b0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 --2- 172.21.15.43:0/1638621868 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8406ee10 0x7fdb84140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.809+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 >> 172.21.15.43:0/1638621868 conn(0x7fdb840661f0 msgr2=0x7fdb840c6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.810+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 shutdown_connections 2024-01-16T15:14:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:43.810+0000 7fdb8b200640 1 -- 172.21.15.43:0/1638621868 wait complete. 2024-01-16T15:14:43.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:14:44.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.058+0000 7fab92e00640 1 Processor -- start 2024-01-16T15:14:44.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.058+0000 7fab92e00640 1 -- start start 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fab8c070a40 0x7fab8c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c15ad90 con 0x7fab8c15d3c0 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8c071360 con 0x7fab8c070a40 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c06cf20 con 0x7fab8c06c9a0 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fab8c15d3c0 0x7fab8c15d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32830/0 (socket says 172.21.15.43:32830) 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.060+0000 7fab92400640 1 -- 172.21.15.43:0/2512776486 learned_addr learned my addr 172.21.15.43:0/2512776486 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91a00640 1 --2- 172.21.15.43:0/2512776486 >> v2:172.21.15.43:3300/0 conn(0x7fab8c070a40 0x7fab8c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91000640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab8be00640 1 -- 172.21.15.43:0/2512776486 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230330338 0 0) 0x7fab8c15ad90 con 0x7fab8c15d3c0 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab8be00640 1 -- 172.21.15.43:0/2512776486 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6c003520 con 0x7fab8c15d3c0 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91a00640 1 -- 172.21.15.43:0/2512776486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 msgr2=0x7fab8c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91a00640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91a00640 1 -- 172.21.15.43:0/2512776486 >> v1:172.21.15.43:6789/0 conn(0x7fab8c15d3c0 legacy=0x7fab8c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.061+0000 7fab91a00640 1 -- 172.21.15.43:0/2512776486 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c06d200 con 0x7fab8c070a40 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.062+0000 7fab91a00640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c070a40 0x7fab8c15a850 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0097a0 tx=0x7fab7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9214bfc2d82298c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab8be00640 1 -- 172.21.15.43:0/2512776486 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c02e070 con 0x7fab8c070a40 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab8be00640 1 -- 172.21.15.43:0/2512776486 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c0315a0 con 0x7fab8c070a40 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab8be00640 1 -- 172.21.15.43:0/2512776486 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c031930 con 0x7fab8c070a40 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 -- 172.21.15.43:0/2512776486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c070a40 msgr2=0x7fab8c15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c070a40 0x7fab8c15a850 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0097a0 tx=0x7fab7c02b840 comp rx=0 tx=0).stop 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 -- 172.21.15.43:0/2512776486 shutdown_connections 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 --2- 172.21.15.43:0/2512776486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c070a40 0x7fab8c15a850 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.063+0000 7fab92e00640 1 -- 172.21.15.43:0/2512776486 >> 172.21.15.43:0/2512776486 conn(0x7fab8c0b0840 msgr2=0x7fab8c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:44.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.064+0000 7fab92e00640 1 -- 172.21.15.43:0/2512776486 shutdown_connections 2024-01-16T15:14:44.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.064+0000 7fab92e00640 1 -- 172.21.15.43:0/2512776486 wait complete. 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.065+0000 7fab92e00640 1 Processor -- start 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.065+0000 7fab92e00640 1 -- start start 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab8c070a40 0x7fab8c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1510e0 con 0x7fab8c15d3c0 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c151850 con 0x7fab8c070a40 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.066+0000 7fab92e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c151b30 con 0x7fab8c06c9a0 2024-01-16T15:14:44.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37656/0 (socket says 172.21.15.43:37656) 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 -- 172.21.15.43:0/2188062814 learned_addr learned my addr 172.21.15.43:0/2188062814 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab91000640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab8c070a40 0x7fab8c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab91a00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 -- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 msgr2=0x7fab8c0c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c0c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 -- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab8c070a40 msgr2=0x7fab8c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab8c070a40 0x7fab8c0b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 -- 172.21.15.43:0/2188062814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c009400 con 0x7fab8c15d3c0 2024-01-16T15:14:44.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.067+0000 7fab92400640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fab8400cb40 tx=0x7fab8400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:44.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.068+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400aca0 con 0x7fab8c15d3c0 2024-01-16T15:14:44.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.068+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8400ae00 con 0x7fab8c15d3c0 2024-01-16T15:14:44.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.068+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84012620 con 0x7fab8c15d3c0 2024-01-16T15:14:44.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.068+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c151e10 con 0x7fab8c15d3c0 2024-01-16T15:14:44.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.068+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c0c5050 con 0x7fab8c15d3c0 2024-01-16T15:14:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.069+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c152330 con 0x7fab8c15d3c0 2024-01-16T15:14:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.070+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fab84007070 con 0x7fab8c15d3c0 2024-01-16T15:14:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.071+0000 7fab89600640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 0x7fab6406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.072+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fab84088970 con 0x7fab8c15d3c0 2024-01-16T15:14:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.072+0000 7fab91a00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 0x7fab6406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:44.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.072+0000 7fab91a00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 0x7fab6406e590 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:44.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.076+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab8c152330 con 0x7fab8c15d3c0 2024-01-16T15:14:44.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.286+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fab8c152330 con 0x7fab8c15d3c0 2024-01-16T15:14:44.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.287+0000 7fab89600640 1 -- 172.21.15.43:0/2188062814 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v93) v1 ==== 80+0+122 (secure 0 0 0) 0x7fab8c152330 con 0x7fab8c15d3c0 2024-01-16T15:14:44.270 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6837/3379600529 2024-01-17T15:14:37.701304+0000 2024-01-16T15:14:44.270 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6836/3379600529 2024-01-17T15:14:37.701304+0000 2024-01-16T15:14:44.270 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.289+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 msgr2=0x7fab6406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.289+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 0x7fab6406e590 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c03d040 comp rx=0 tx=0).stop 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.289+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 msgr2=0x7fab8c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.289+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fab8400cb40 tx=0x7fab8400a060 comp rx=0 tx=0).stop 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 shutdown_connections 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab6406c0d0 0x7fab6406e590 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab8c070a40 0x7fab8c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab8c15d3c0 0x7fab8c0b20f0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 --2- 172.21.15.43:0/2188062814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8c06c9a0 0x7fab8c0c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 >> 172.21.15.43:0/2188062814 conn(0x7fab8c0b0840 msgr2=0x7fab8c0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 shutdown_connections 2024-01-16T15:14:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:44.290+0000 7fab92e00640 1 -- 172.21.15.43:0/2188062814 wait complete. 2024-01-16T15:14:44.294 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:14:44.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:44.294 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:14:44.388 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[45873]: fuse finished with error 0 and tester_r 0 2024-01-16T15:14:44.392 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:14:44.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:44.392 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:14:44.566 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:14:44.566 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:44.627 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:14:44.628 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:14:44.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:44.628 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:14:44.684 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:14:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:14:44.685 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:14:44.685 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:44.743 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:14:44.743 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:14:44.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:44.743 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:14:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:14:44.800 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:14:44.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:14:45.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.046+0000 7f9b13000640 1 Processor -- start 2024-01-16T15:14:45.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.046+0000 7f9b13000640 1 -- start start 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b13000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9b0c06ad10 0x7f9b0c1682e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b13000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c15d880 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b13000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c06fba0 con 0x7f9b0c0af890 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b13000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c06eb00 con 0x7f9b0c06ad10 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b0be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9b0c15d4a0 0x7f9b0c0af180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32832/0 (socket says 172.21.15.43:32832) 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.048+0000 7f9b0be00640 1 -- 172.21.15.43:0/2451404839 learned_addr learned my addr 172.21.15.43:0/2451404839 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:45.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b11000640 1 --2- 172.21.15.43:0/2451404839 >> v2:172.21.15.69:3300/0 conn(0x7f9b0c06ad10 0x7f9b0c1682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185154516 0 0) 0x7f9b0c15d880 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec003520 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 708101843 0 0) 0x7f9aec003520 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 >> v2:172.21.15.69:3300/0 conn(0x7f9b0c06ad10 msgr2=0x7f9b0c1682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 --2- 172.21.15.43:0/2451404839 >> v2:172.21.15.69:3300/0 conn(0x7f9b0c06ad10 0x7f9b0c1682e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.049+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 >> v1:172.21.15.69:6789/0 conn(0x7f9b0c0af890 legacy=0x7f9b0c06a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c06ede0 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9afc001670 con 0x7f9b0c15d4a0 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 >> v1:172.21.15.43:6789/0 conn(0x7f9b0c15d4a0 legacy=0x7f9b0c0af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0039c0 0x7f9aec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c06eb00 con 0x7f9aec0047c0 2024-01-16T15:14:45.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00c050 con 0x7f9aec005180 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00c210 con 0x7f9aec0039c0 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b03200640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b0be00640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0039c0 0x7f9aec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.050+0000 7f9b11000640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b03200640 1 -- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0039c0 msgr2=0x7f9aec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b03200640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0039c0 0x7f9aec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b03200640 1 -- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aec005180 msgr2=0x7f9aec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b03200640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b03200640 1 -- 172.21.15.43:0/2451404839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec003520 con 0x7f9aec0047c0 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b11000640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aec005180 0x7f9aec00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.051+0000 7f9b11000640 1 -- 172.21.15.43:0/2451404839 reap_dead start 2024-01-16T15:14:45.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.052+0000 7f9b03200640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f9af40044b0 tx=0x7f9aec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e17d96f8744052 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.052+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4030400 con 0x7f9aec0047c0 2024-01-16T15:14:45.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.052+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4030560 con 0x7f9aec0047c0 2024-01-16T15:14:45.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.052+0000 7f9b0b400640 1 -- 172.21.15.43:0/2451404839 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4030950 con 0x7f9aec0047c0 2024-01-16T15:14:45.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 -- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 msgr2=0x7f9aec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f9af40044b0 tx=0x7f9aec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 -- 172.21.15.43:0/2451404839 shutdown_connections 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 --2- 172.21.15.43:0/2451404839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9aec0047c0 0x7f9aec004c40 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 -- 172.21.15.43:0/2451404839 >> 172.21.15.43:0/2451404839 conn(0x7f9b0c06d490 msgr2=0x7f9b0c06d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 -- 172.21.15.43:0/2451404839 shutdown_connections 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.053+0000 7f9b13000640 1 -- 172.21.15.43:0/2451404839 wait complete. 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.055+0000 7f9b13000640 1 Processor -- start 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.055+0000 7f9b13000640 1 -- start start 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0047c0 0x7f9b0c14d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c1f6680 0x7f9b0c156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0ccab0 con 0x7f9b0c15e760 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c14de50 con 0x7f9b0c1f6680 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b13000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c1f6c40 con 0x7f9aec0047c0 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b11000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b11000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37682/0 (socket says 172.21.15.43:37682) 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b11000640 1 -- 172.21.15.43:0/546645736 learned_addr learned my addr 172.21.15.43:0/546645736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b0be00640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0047c0 0x7f9b0c14d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.056+0000 7f9b03200640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c1f6680 0x7f9b0c156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 -- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0047c0 msgr2=0x7f9b0c14d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0047c0 0x7f9b0c14d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 -- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c1f6680 msgr2=0x7f9b0c156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c1f6680 0x7f9b0c156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 -- 172.21.15.43:0/546645736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c1f6f20 con 0x7f9b0c15e760 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b11000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0acfe0 tx=0x7f9b04010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04004230 con 0x7f9b0c15e760 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04004390 con 0x7f9b0c15e760 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04003810 con 0x7f9b0c15e760 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c1f7200 con 0x7f9b0c15e760 2024-01-16T15:14:45.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.057+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c1f74e0 con 0x7f9b0c15e760 2024-01-16T15:14:45.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.058+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c074770 con 0x7f9b0c15e760 2024-01-16T15:14:45.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.060+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f9b04003970 con 0x7f9b0c15e760 2024-01-16T15:14:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.061+0000 7f9b08c00640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 0x7f9ae406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.061+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9b04088a10 con 0x7f9b0c15e760 2024-01-16T15:14:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.061+0000 7f9b0be00640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 0x7f9ae406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.062+0000 7f9b0be00640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 0x7f9ae406e590 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9afc001eb0 tx=0x7f9afc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.065+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b04055010 con 0x7f9b0c15e760 2024-01-16T15:14:45.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.299+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b0c0ae960 con 0x7f9b0c15e760 2024-01-16T15:14:45.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.300+0000 7f9b08c00640 1 -- 172.21.15.43:0/546645736 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f9b04058630 con 0x7f9b0c15e760 2024-01-16T15:14:45.283 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:45.283 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/206348255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":206348255},{"type":"v1","addr":"172.21.15.69:6834","nonce":206348255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3389933703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3389933703},{"type":"v1","addr":"172.21.15.69:6835","nonce":3389933703}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/1040529098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1040529098},{"type":"v1","addr":"172.21.15.43:6835","nonce":1040529098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2617094504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2617094504},{"type":"v1","addr":"172.21.15.43:6837","nonce":2617094504}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-01-16T15:14:45.284 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 60 2024-01-16T15:14:45.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 msgr2=0x7f9ae406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 0x7f9ae406e590 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9afc001eb0 tx=0x7f9afc009040 comp rx=0 tx=0).stop 2024-01-16T15:14:45.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 msgr2=0x7f9b0c15ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0acfe0 tx=0x7f9b04010c20 comp rx=0 tx=0).stop 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 shutdown_connections 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ae406c0d0 0x7f9ae406e590 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c1f6680 0x7f9b0c156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b0c15e760 0x7f9b0c15ebe0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.303+0000 7f9b13000640 1 --2- 172.21.15.43:0/546645736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aec0047c0 0x7f9b0c14d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.304+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 >> 172.21.15.43:0/546645736 conn(0x7f9b0c06d490 msgr2=0x7f9b0c0ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.304+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 shutdown_connections 2024-01-16T15:14:45.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.304+0000 7f9b13000640 1 -- 172.21.15.43:0/546645736 wait complete. 2024-01-16T15:14:45.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:14:45.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.552+0000 7f535e000640 1 Processor -- start 2024-01-16T15:14:45.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.553+0000 7f535e000640 1 -- start start 2024-01-16T15:14:45.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f535815d3b0 0x7f53580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f53580d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580d2010 con 0x7f53580d1bc0 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535e000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53580c6bf0 con 0x7f535815d3b0 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580d1400 con 0x7f53580d0fa0 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f535815d3b0 0x7f53580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f535815d3b0 0x7f53580ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37694/0 (socket says 172.21.15.43:37694) 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.554+0000 7f535cc00640 1 -- 172.21.15.43:0/368875655 learned_addr learned my addr 172.21.15.43:0/368875655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.555+0000 7f5357e00640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f53580d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.555+0000 7f535cc00640 1 -- 172.21.15.43:0/368875655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 msgr2=0x7f53580d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.555+0000 7f535cc00640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f53580d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.555+0000 7f535cc00640 1 -- 172.21.15.43:0/368875655 >> v1:172.21.15.69:6789/0 conn(0x7f53580d1bc0 legacy=0x7f53580b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.555+0000 7f535cc00640 1 -- 172.21.15.43:0/368875655 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580cedc0 con 0x7f535815d3b0 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.556+0000 7f535cc00640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f535815d3b0 0x7f53580ce6c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5348009870 tx=0x7f534802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668ce94f40208f45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.556+0000 7f5357400640 1 -- 172.21.15.43:0/368875655 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480317c0 con 0x7f535815d3b0 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.556+0000 7f5357400640 1 -- 172.21.15.43:0/368875655 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5348031920 con 0x7f535815d3b0 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f5357400640 1 -- 172.21.15.43:0/368875655 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348031cb0 con 0x7f535815d3b0 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 -- 172.21.15.43:0/368875655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f535815d3b0 msgr2=0x7f53580ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f535815d3b0 0x7f53580ce6c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5348009870 tx=0x7f534802b860 comp rx=0 tx=0).stop 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 -- 172.21.15.43:0/368875655 shutdown_connections 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f53580d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 --2- 172.21.15.43:0/368875655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f535815d3b0 0x7f53580ce6c0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 -- 172.21.15.43:0/368875655 >> 172.21.15.43:0/368875655 conn(0x7f53580d30a0 msgr2=0x7f53580d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 -- 172.21.15.43:0/368875655 shutdown_connections 2024-01-16T15:14:45.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.557+0000 7f535e000640 1 -- 172.21.15.43:0/368875655 wait complete. 2024-01-16T15:14:45.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.559+0000 7f535e000640 1 Processor -- start 2024-01-16T15:14:45.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.559+0000 7f535e000640 1 -- start start 2024-01-16T15:14:45.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f535815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 0x7f53580b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53581f2b30 con 0x7f53580d1bc0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53581f2d40 con 0x7f535815d3b0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.560+0000 7f535e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53581f3020 con 0x7f53580d0fa0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37700/0 (socket says 172.21.15.43:37700) 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 -- 172.21.15.43:0/2841786009 learned_addr learned my addr 172.21.15.43:0/2841786009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f535cc00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f535815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f535d600640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 0x7f53580b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 -- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 msgr2=0x7f535815bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f535815bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 -- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 msgr2=0x7f53580b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 0x7f53580b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 -- 172.21.15.43:0/2841786009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480094d0 con 0x7f53580d1bc0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f535cc00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f535815bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f535d600640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 0x7f53580b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.561+0000 7f5357e00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5340007e40 tx=0x7f534000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.562+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340014620 con 0x7f53580d1bc0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.562+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534000ee90 con 0x7f53580d1bc0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.562+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53400149c0 con 0x7f53580d1bc0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.562+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53581f3300 con 0x7f53580d1bc0 2024-01-16T15:14:45.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.562+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53581f38c0 con 0x7f53580d1bc0 2024-01-16T15:14:45.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.563+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53581f3a80 con 0x7f53580d1bc0 2024-01-16T15:14:45.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.564+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f5340014b20 con 0x7f53580d1bc0 2024-01-16T15:14:45.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.565+0000 7f5354c00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 0x7f533006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.565+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5340089200 con 0x7f53580d1bc0 2024-01-16T15:14:45.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.565+0000 7f535cc00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 0x7f533006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.566+0000 7f535cc00640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 0x7f533006e5c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5348036000 tx=0x7f534803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.569+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5340055800 con 0x7f53580d1bc0 2024-01-16T15:14:45.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.762+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f535804e840 con 0x7f53580d1bc0 2024-01-16T15:14:45.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.770+0000 7f5354c00640 1 -- 172.21.15.43:0/2841786009 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5340058e20 con 0x7f53580d1bc0 2024-01-16T15:14:45.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 msgr2=0x7f533006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 0x7f533006e5c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5348036000 tx=0x7f534803d040 comp rx=0 tx=0).stop 2024-01-16T15:14:45.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 msgr2=0x7f53580b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5340007e40 tx=0x7f534000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 shutdown_connections 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f533006c100 0x7f533006e5c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f535815d3b0 0x7f53580b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53580d1bc0 0x7f53580b7e90 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 --2- 172.21.15.43:0/2841786009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f53580d0fa0 0x7f535815bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 >> 172.21.15.43:0/2841786009 conn(0x7f53580d30a0 msgr2=0x7f53580c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.774+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 shutdown_connections 2024-01-16T15:14:45.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.775+0000 7f535e000640 1 -- 172.21.15.43:0/2841786009 wait complete. 2024-01-16T15:14:45.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:14:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.992+0000 7f6fcdc00640 1 Processor -- start 2024-01-16T15:14:45.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.992+0000 7f6fcdc00640 1 -- start start 2024-01-16T15:14:45.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcdc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6fc806a440 0x7f6fc8070f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcdc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 0x7f6fc80b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcdc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc806d880 con 0x7f6fc8159530 2024-01-16T15:14:45.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcdc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc80b24a0 con 0x7f6fc806a440 2024-01-16T15:14:45.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcdc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc80718b0 con 0x7f6fc8071440 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fc7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6fc806a440 0x7f6fc8070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fc7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6fc806a440 0x7f6fc8070f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37714/0 (socket says 172.21.15.43:37714) 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fc7400640 1 -- 172.21.15.43:0/1562391544 learned_addr learned my addr 172.21.15.43:0/1562391544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fcd200640 1 --2- 172.21.15.43:0/1562391544 >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 0x7f6fc80b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.994+0000 7f6fc7400640 1 -- 172.21.15.43:0/1562391544 >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 msgr2=0x7f6fc80b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.995+0000 7f6fc7400640 1 --2- 172.21.15.43:0/1562391544 >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 0x7f6fc80b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.995+0000 7f6fc7400640 1 -- 172.21.15.43:0/1562391544 >> v1:172.21.15.69:6789/0 conn(0x7f6fc8159530 legacy=0x7f6fc806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.995+0000 7f6fc7400640 1 -- 172.21.15.43:0/1562391544 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc806db60 con 0x7f6fc806a440 2024-01-16T15:14:45.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.995+0000 7f6fcd200640 1 --2- 172.21.15.43:0/1562391544 >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 0x7f6fc80b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fc7400640 1 --2- 172.21.15.43:0/1562391544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc806a440 0x7f6fc8070f00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80b41e0 tx=0x7f6fb802e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd0bbe2c27c3d60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fc6a00640 1 -- 172.21.15.43:0/1562391544 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8038400 con 0x7f6fc806a440 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fc6a00640 1 -- 172.21.15.43:0/1562391544 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8032070 con 0x7f6fc806a440 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fc6a00640 1 -- 172.21.15.43:0/1562391544 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8038780 con 0x7f6fc806a440 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fcdc00640 1 -- 172.21.15.43:0/1562391544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc806a440 msgr2=0x7f6fc8070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.996+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/1562391544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc806a440 0x7f6fc8070f00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80b41e0 tx=0x7f6fb802e140 comp rx=0 tx=0).stop 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 -- 172.21.15.43:0/1562391544 shutdown_connections 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/1562391544 >> v2:172.21.15.69:3300/0 conn(0x7f6fc8071440 0x7f6fc80b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/1562391544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc806a440 0x7f6fc8070f00 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 -- 172.21.15.43:0/1562391544 >> 172.21.15.43:0/1562391544 conn(0x7f6fc80b15c0 msgr2=0x7f6fc80b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 -- 172.21.15.43:0/1562391544 shutdown_connections 2024-01-16T15:14:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.997+0000 7f6fcdc00640 1 -- 172.21.15.43:0/1562391544 wait complete. 2024-01-16T15:14:45.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.998+0000 7f6fcdc00640 1 Processor -- start 2024-01-16T15:14:45.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.999+0000 7f6fcdc00640 1 -- start start 2024-01-16T15:14:45.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.999+0000 7f6fcdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc806a440 0x7f6fc80cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:45.999+0000 7f6fcdc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 0x7f6fc80af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcdc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcdc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f2bb0 con 0x7f6fc8159530 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcdc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f6680 con 0x7f6fc8071440 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcdc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f6960 con 0x7f6fc806a440 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37726/0 (socket says 172.21.15.43:37726) 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 -- 172.21.15.43:0/3098523017 learned_addr learned my addr 172.21.15.43:0/3098523017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc806a440 0x7f6fc80cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fc7400640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 0x7f6fc80af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 -- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc806a440 msgr2=0x7f6fc80cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc806a440 0x7f6fc80cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 -- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 msgr2=0x7f6fc80af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 0x7f6fc80af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fcd200640 1 -- 172.21.15.43:0/3098523017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb800bd40 con 0x7f6fc8159530 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.000+0000 7f6fc7400640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 0x7f6fc80af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fcd200640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000c7f0 tx=0x7f6fc000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0007820 con 0x7f6fc8159530 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0007980 con 0x7f6fc8159530 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0007d70 con 0x7f6fc8159530 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc81f6c40 con 0x7f6fc8159530 2024-01-16T15:14:45.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.001+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc80ccbd0 con 0x7f6fc8159530 2024-01-16T15:14:45.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.002+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc804e840 con 0x7f6fc8159530 2024-01-16T15:14:45.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.003+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f6fc00113f0 con 0x7f6fc8159530 2024-01-16T15:14:45.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.004+0000 7f6fbfe00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 0x7f6fa006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:45.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.004+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6fc0088cb0 con 0x7f6fc8159530 2024-01-16T15:14:45.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.004+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 0x7f6fa006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:45.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.005+0000 7f6fc7e00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 0x7f6fa006e590 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6fc81e3500 tx=0x7f6fb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:45.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.010+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fc0055310 con 0x7f6fc8159530 2024-01-16T15:14:46.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.208+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f6fc80cbd60 con 0x7f6fc8159530 2024-01-16T15:14:46.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.215+0000 7f6fbfe00640 1 -- 172.21.15.43:0/3098523017 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v34) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6fc001f0b0 con 0x7f6fc8159530 2024-01-16T15:14:46.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 msgr2=0x7f6fa006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 0x7f6fa006e590 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6fc81e3500 tx=0x7f6fb0008040 comp rx=0 tx=0).stop 2024-01-16T15:14:46.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 msgr2=0x7f6fc80cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000c7f0 tx=0x7f6fc000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 shutdown_connections 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fa006c0d0 0x7f6fa006e590 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc8071440 0x7f6fc80af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fc8159530 0x7f6fc80cf090 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 --2- 172.21.15.43:0/3098523017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc806a440 0x7f6fc80cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 >> 172.21.15.43:0/3098523017 conn(0x7f6fc80b15c0 msgr2=0x7f6fc8156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 shutdown_connections 2024-01-16T15:14:46.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.218+0000 7f6fcdc00640 1 -- 172.21.15.43:0/3098523017 wait complete. 2024-01-16T15:14:46.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:14:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.464+0000 7f1825a00640 1 Processor -- start 2024-01-16T15:14:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.464+0000 7f1825a00640 1 -- start start 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.465+0000 7f1825a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f18200c8930 0x7f18200ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.465+0000 7f1825a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18200c6bf0 con 0x7f182015d3b0 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.465+0000 7f1825a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18200cedc0 con 0x7f18200d1d90 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.465+0000 7f1825a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18200c6e40 con 0x7f18200c8930 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.466+0000 7f181fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f182015d3b0 0x7f18200d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32842/0 (socket says 172.21.15.43:32842) 2024-01-16T15:14:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.466+0000 7f181fe00640 1 -- 172.21.15.43:0/3820388585 learned_addr learned my addr 172.21.15.43:0/3820388585 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.466+0000 7f1825000640 1 --2- 172.21.15.43:0/3820388585 >> v2:172.21.15.43:3300/0 conn(0x7f18200c8930 0x7f18200ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.466+0000 7f181ea00640 1 -- 172.21.15.43:0/3820388585 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852110853 0 0) 0x7f18200c6bf0 con 0x7f182015d3b0 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.466+0000 7f181ea00640 1 -- 172.21.15.43:0/3820388585 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1800003520 con 0x7f182015d3b0 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.467+0000 7f1825000640 1 -- 172.21.15.43:0/3820388585 >> v1:172.21.15.69:6789/0 conn(0x7f18200d1d90 legacy=0x7f18200b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.467+0000 7f1825000640 1 -- 172.21.15.43:0/3820388585 >> v1:172.21.15.43:6789/0 conn(0x7f182015d3b0 legacy=0x7f18200d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.467+0000 7f1825000640 1 -- 172.21.15.43:0/3820388585 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18200c7120 con 0x7f18200c8930 2024-01-16T15:14:46.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.468+0000 7f1825000640 1 --2- 172.21.15.43:0/3820388585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f18200ce6c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f18180097a0 tx=0x7f181802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619d528a38b82643 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.468+0000 7f181ea00640 1 -- 172.21.15.43:0/3820388585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818031830 con 0x7f18200c8930 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.468+0000 7f181ea00640 1 -- 172.21.15.43:0/3820388585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1818031990 con 0x7f18200c8930 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.468+0000 7f181ea00640 1 -- 172.21.15.43:0/3820388585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818031d20 con 0x7f18200c8930 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 -- 172.21.15.43:0/3820388585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 msgr2=0x7f18200ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 --2- 172.21.15.43:0/3820388585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f18200ce6c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f18180097a0 tx=0x7f181802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 -- 172.21.15.43:0/3820388585 shutdown_connections 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 --2- 172.21.15.43:0/3820388585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f18200ce6c0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 -- 172.21.15.43:0/3820388585 >> 172.21.15.43:0/3820388585 conn(0x7f18200d30a0 msgr2=0x7f18200d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 -- 172.21.15.43:0/3820388585 shutdown_connections 2024-01-16T15:14:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.469+0000 7f1825a00640 1 -- 172.21.15.43:0/3820388585 wait complete. 2024-01-16T15:14:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.470+0000 7f1825a00640 1 Processor -- start 2024-01-16T15:14:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.471+0000 7f1825a00640 1 -- start start 2024-01-16T15:14:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200d1d90 0x7f1820143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f182015d3b0 0x7f1820144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201f5f20 con 0x7f18200c8930 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201578a0 con 0x7f18200d1d90 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f1825a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182015e8b0 con 0x7f182015d3b0 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37740/0 (socket says 172.21.15.43:37740) 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 -- 172.21.15.43:0/2936670023 learned_addr learned my addr 172.21.15.43:0/2936670023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 -- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f182015d3b0 msgr2=0x7f1820144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f182015d3b0 0x7f1820144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 -- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200d1d90 msgr2=0x7f1820143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200d1d90 0x7f1820143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.472+0000 7f181fe00640 1 -- 172.21.15.43:0/2936670023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818009400 con 0x7f18200c8930 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f181fe00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f181000cbf0 tx=0x7f181000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000acc0 con 0x7f18200c8930 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181000ae20 con 0x7f18200c8930 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810012620 con 0x7f18200c8930 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182015eb90 con 0x7f18200c8930 2024-01-16T15:14:46.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.473+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182015f150 con 0x7f18200c8930 2024-01-16T15:14:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.474+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182004e840 con 0x7f18200c8930 2024-01-16T15:14:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.475+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f1810007070 con 0x7f18200c8930 2024-01-16T15:14:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.477+0000 7f1817e00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 0x7f17f806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.477+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1810088870 con 0x7f18200c8930 2024-01-16T15:14:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.477+0000 7f181f400640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 0x7f17f806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.477+0000 7f181f400640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 0x7f17f806e590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1808000f60 tx=0x7f18080079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.481+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1810054ed0 con 0x7f18200c8930 2024-01-16T15:14:46.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.676+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f18200c6b60 con 0x7f18200c8930 2024-01-16T15:14:46.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.681+0000 7f1817e00640 1 -- 172.21.15.43:0/2936670023 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f18100170a0 con 0x7f18200c8930 2024-01-16T15:14:46.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 msgr2=0x7f17f806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 0x7f17f806e590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1808000f60 tx=0x7f18080079d0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 msgr2=0x7f182015f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f181000cbf0 tx=0x7f181000a430 comp rx=0 tx=0).stop 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 shutdown_connections 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17f806c0d0 0x7f17f806e590 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200d1d90 0x7f1820143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f182015d3b0 0x7f1820144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 --2- 172.21.15.43:0/2936670023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18200c8930 0x7f182015f950 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.684+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 >> 172.21.15.43:0/2936670023 conn(0x7f18200d30a0 msgr2=0x7f18200b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.685+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 shutdown_connections 2024-01-16T15:14:46.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.685+0000 7f1825a00640 1 -- 172.21.15.43:0/2936670023 wait complete. 2024-01-16T15:14:46.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:14:46.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.918+0000 7f8337200640 1 Processor -- start 2024-01-16T15:14:46.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.918+0000 7f8337200640 1 -- start start 2024-01-16T15:14:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.919+0000 7f8337200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f83380d00d0 0x7f83380d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.919+0000 7f8337200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380afd30 con 0x7f833815d370 2024-01-16T15:14:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.919+0000 7f8337200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380d0c30 con 0x7f83380b1f00 2024-01-16T15:14:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.919+0000 7f8337200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83380cef60 con 0x7f83380d00d0 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.920+0000 7f8335e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f833815d370 0x7f83380cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32850/0 (socket says 172.21.15.43:32850) 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.920+0000 7f8335e00640 1 -- 172.21.15.43:0/1424528368 learned_addr learned my addr 172.21.15.43:0/1424528368 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.920+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174052739 0 0) 0x7f83380afd30 con 0x7f833815d370 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.920+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8318003520 con 0x7f833815d370 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.920+0000 7f8335400640 1 --2- 172.21.15.43:0/1424528368 >> v2:172.21.15.69:3300/0 conn(0x7f83380d00d0 0x7f83380d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855625981 0 0) 0x7f8318003520 con 0x7f833815d370 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 >> v2:172.21.15.69:3300/0 conn(0x7f83380d00d0 msgr2=0x7f83380d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 --2- 172.21.15.43:0/1424528368 >> v2:172.21.15.69:3300/0 conn(0x7f83380d00d0 0x7f83380d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 >> v1:172.21.15.69:6789/0 conn(0x7f83380b1f00 legacy=0x7f83380b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380d0ef0 con 0x7f833815d370 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8328001670 con 0x7f833815d370 2024-01-16T15:14:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 >> v1:172.21.15.43:6789/0 conn(0x7f833815d370 legacy=0x7f83380cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f8318003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83180047c0 0x7f8318004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83180052a0 0x7f831800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380cef60 con 0x7f83180039e0 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831800c1f0 con 0x7f83180052a0 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831800c3b0 con 0x7f83180047c0 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.921+0000 7f8335e00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f8318003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8335e00640 1 -- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83180047c0 msgr2=0x7f8318004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8335e00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83180047c0 0x7f8318004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8335e00640 1 -- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83180052a0 msgr2=0x7f831800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8335e00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83180052a0 0x7f831800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8335e00640 1 -- 172.21.15.43:0/1424528368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318003520 con 0x7f83180039e0 2024-01-16T15:14:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.922+0000 7f8336800640 1 -- 172.21.15.43:0/1424528368 reap_dead start 2024-01-16T15:14:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.923+0000 7f8335e00640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f8318003e00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f832800c210 tx=0x7f83180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e462145e34c9b992 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.923+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328005ad0 con 0x7f83180039e0 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.923+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328005c50 con 0x7f83180039e0 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.923+0000 7f8334a00640 1 -- 172.21.15.43:0/1424528368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832802c890 con 0x7f83180039e0 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 -- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 msgr2=0x7f8318003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f8318003e00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f832800c210 tx=0x7f83180057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 -- 172.21.15.43:0/1424528368 shutdown_connections 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 --2- 172.21.15.43:0/1424528368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f8318003e00 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 -- 172.21.15.43:0/1424528368 >> 172.21.15.43:0/1424528368 conn(0x7f833809c3d0 msgr2=0x7f83380d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 -- 172.21.15.43:0/1424528368 shutdown_connections 2024-01-16T15:14:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.924+0000 7f8337200640 1 -- 172.21.15.43:0/1424528368 wait complete. 2024-01-16T15:14:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.926+0000 7f8337200640 1 Processor -- start 2024-01-16T15:14:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.926+0000 7f8337200640 1 -- start start 2024-01-16T15:14:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 0x7f83380b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83380b3f30 0x7f83381f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380b43b0 con 0x7f83180039e0 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380ccc20 con 0x7f833815b450 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8337200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83381f6dc0 con 0x7f83380b3f30 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8335e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8335e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37744/0 (socket says 172.21.15.43:37744) 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.927+0000 7f8335e00640 1 -- 172.21.15.43:0/2554147494 learned_addr learned my addr 172.21.15.43:0/2554147494 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335400640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 0x7f83380b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8336800640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83380b3f30 0x7f83381f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 -- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83380b3f30 msgr2=0x7f83381f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83380b3f30 0x7f83381f6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 -- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 msgr2=0x7f83380b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 0x7f83380b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 -- 172.21.15.43:0/2554147494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380cc2f0 con 0x7f83180039e0 2024-01-16T15:14:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335400640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 0x7f83380b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8335e00640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f832800c210 tx=0x7f832802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832803c040 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832802cde0 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328007290 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.928+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83380cc5d0 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.929+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833815b8d0 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.930+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833804e840 con 0x7f83180039e0 2024-01-16T15:14:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.931+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f8328032070 con 0x7f83180039e0 2024-01-16T15:14:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.931+0000 7f832e000640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 0x7f831006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.932+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f83280a9f20 con 0x7f83180039e0 2024-01-16T15:14:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.932+0000 7f8335400640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 0x7f831006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.933+0000 7f8335400640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 0x7f831006e590 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:46.936+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8328076640 con 0x7f83180039e0 2024-01-16T15:14:47.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:47.202+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f83380d2670 con 0x7f83180039e0 2024-01-16T15:14:48.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.140+0000 7f832e000640 1 -- 172.21.15.43:0/2554147494 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v671) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8328079c60 con 0x7f83180039e0 2024-01-16T15:14:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 msgr2=0x7f831006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 0x7f831006e590 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320008040 comp rx=0 tx=0).stop 2024-01-16T15:14:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 msgr2=0x7f83380cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f832800c210 tx=0x7f832802e070 comp rx=0 tx=0).stop 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 shutdown_connections 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f831006c0d0 0x7f831006e590 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833815b450 0x7f83380b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83380b3f30 0x7f83381f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 --2- 172.21.15.43:0/2554147494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83180039e0 0x7f83380cd0b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.143+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 >> 172.21.15.43:0/2554147494 conn(0x7f833809c3d0 msgr2=0x7f83380b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.144+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 shutdown_connections 2024-01-16T15:14:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.144+0000 7f8337200640 1 -- 172.21.15.43:0/2554147494 wait complete. 2024-01-16T15:14:48.149 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:14:48.150 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:14:48.150 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:14:48.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:48.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.383+0000 7fc601400640 1 Processor -- start 2024-01-16T15:14:48.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.383+0000 7fc601400640 1 -- start start 2024-01-16T15:14:48.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc601400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc5fc0ad720 0x7fc5fc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc601400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc15c360 con 0x7fc5fc0abff0 2024-01-16T15:14:48.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc601400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc15ba30 con 0x7fc5fc06d520 2024-01-16T15:14:48.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc601400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5fc0b41a0 con 0x7fc5fc0ad720 2024-01-16T15:14:48.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc5f9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc5fc0ad720 0x7fc5fc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc5fb200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc5fc0abff0 0x7fc5fc0b25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32852/0 (socket says 172.21.15.43:32852) 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.384+0000 7fc5fb200640 1 -- 172.21.15.43:0/1735291030 learned_addr learned my addr 172.21.15.43:0/1735291030 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.385+0000 7fc5f9e00640 1 -- 172.21.15.43:0/1735291030 >> v1:172.21.15.69:6789/0 conn(0x7fc5fc06d520 legacy=0x7fc5fc06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.385+0000 7fc5f9e00640 1 -- 172.21.15.43:0/1735291030 >> v1:172.21.15.43:6789/0 conn(0x7fc5fc0abff0 legacy=0x7fc5fc0b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.385+0000 7fc5f9e00640 1 -- 172.21.15.43:0/1735291030 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc15bd10 con 0x7fc5fc0ad720 2024-01-16T15:14:48.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.386+0000 7fc5f9e00640 1 --2- 172.21.15.43:0/1735291030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0b2080 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40097a0 tx=0x7fc5e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364ce2408a012324 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.386+0000 7fc5f9400640 1 -- 172.21.15.43:0/1735291030 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4031830 con 0x7fc5fc0ad720 2024-01-16T15:14:48.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.386+0000 7fc5f9400640 1 -- 172.21.15.43:0/1735291030 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e4031990 con 0x7fc5fc0ad720 2024-01-16T15:14:48.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.386+0000 7fc5f9400640 1 -- 172.21.15.43:0/1735291030 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4031d20 con 0x7fc5fc0ad720 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 -- 172.21.15.43:0/1735291030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc0ad720 msgr2=0x7fc5fc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 --2- 172.21.15.43:0/1735291030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0b2080 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40097a0 tx=0x7fc5e402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 -- 172.21.15.43:0/1735291030 shutdown_connections 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 --2- 172.21.15.43:0/1735291030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0b2080 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 -- 172.21.15.43:0/1735291030 >> 172.21.15.43:0/1735291030 conn(0x7fc5fc0b0d80 msgr2=0x7fc5fc0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 -- 172.21.15.43:0/1735291030 shutdown_connections 2024-01-16T15:14:48.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.387+0000 7fc601400640 1 -- 172.21.15.43:0/1735291030 wait complete. 2024-01-16T15:14:48.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.389+0000 7fc601400640 1 Processor -- start 2024-01-16T15:14:48.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.389+0000 7fc601400640 1 -- start start 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5fc0abff0 0x7fc5fc1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc148360 con 0x7fc5fc06d520 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1478f0 con 0x7fc5fc0ad720 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc601400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0720b0 con 0x7fc5fc0abff0 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc5fa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.390+0000 7fc5fa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37772/0 (socket says 172.21.15.43:37772) 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 -- 172.21.15.43:0/1199337628 learned_addr learned my addr 172.21.15.43:0/1199337628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 -- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5fc0abff0 msgr2=0x7fc5fc1473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5f9e00640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5fc0abff0 0x7fc5fc1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fb200640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5fc0abff0 0x7fc5fc1473b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 -- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5fc0ad720 msgr2=0x7fc5fc0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 -- 172.21.15.43:0/1199337628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4009400 con 0x7fc5fc06d520 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.391+0000 7fc5fa800640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b3f0 tx=0x7fc5ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.392+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec011830 con 0x7fc5fc06d520 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.392+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec011990 con 0x7fc5fc06d520 2024-01-16T15:14:48.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.392+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec011d40 con 0x7fc5fc06d520 2024-01-16T15:14:48.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.392+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc072390 con 0x7fc5fc06d520 2024-01-16T15:14:48.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.392+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc0c86e0 con 0x7fc5fc06d520 2024-01-16T15:14:48.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.393+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc074770 con 0x7fc5fc06d520 2024-01-16T15:14:48.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.394+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fc5ec01a400 con 0x7fc5fc06d520 2024-01-16T15:14:48.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.395+0000 7fc5f2a00640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 0x7fc5d406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.395+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc5ec088780 con 0x7fc5fc06d520 2024-01-16T15:14:48.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.395+0000 7fc5f9e00640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 0x7fc5d406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.396+0000 7fc5f9e00640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 0x7fc5d406e5e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4036000 tx=0x7fc5e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.400+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5ec054d80 con 0x7fc5fc06d520 2024-01-16T15:14:48.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.597+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5fc0b3c30 con 0x7fc5fc06d520 2024-01-16T15:14:48.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.599+0000 7fc5f2a00640 1 -- 172.21.15.43:0/1199337628 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12982 (secure 0 0 0) 0x7fc5ec015090 con 0x7fc5fc06d520 2024-01-16T15:14:48.582 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:48.582 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":93,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:40.716799+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6837/3379600529":"2024-01-17T15:14:37.701304+0000","172.21.15.43:6836/3379600529":"2024-01-17T15:14:37.701304+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 msgr2=0x7fc5d406e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 0x7fc5d406e5e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4036000 tx=0x7fc5e4008030 comp rx=0 tx=0).stop 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 msgr2=0x7fc5fc147e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b3f0 tx=0x7fc5ec00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 shutdown_connections 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc5d406c120 0x7fc5d406e5e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5fc0ad720 0x7fc5fc0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5fc0abff0 0x7fc5fc1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 --2- 172.21.15.43:0/1199337628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5fc06d520 0x7fc5fc147e20 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 >> 172.21.15.43:0/1199337628 conn(0x7fc5fc0b0d80 msgr2=0x7fc5fc0af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:48.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 shutdown_connections 2024-01-16T15:14:48.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.602+0000 7fc601400640 1 -- 172.21.15.43:0/1199337628 wait complete. 2024-01-16T15:14:48.603 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:48.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.845+0000 7fe256e00640 1 Processor -- start 2024-01-16T15:14:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.845+0000 7fe256e00640 1 -- start start 2024-01-16T15:14:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.846+0000 7fe256e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe250160bd0 0x7fe2500b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.846+0000 7fe256e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2500b3210 0x7fe250167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.846+0000 7fe256e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500b23e0 con 0x7fe2500b48d0 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.846+0000 7fe256e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2500cdc70 con 0x7fe2500b3210 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.846+0000 7fe256e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b2870 con 0x7fe250160bd0 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2500b3210 0x7fe250167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2500b3210 0x7fe250167290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37782/0 (socket says 172.21.15.43:37782) 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 -- 172.21.15.43:0/1738618453 learned_addr learned my addr 172.21.15.43:0/1738618453 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe24fe00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe250160bd0 0x7fe2500b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 -- 172.21.15.43:0/1738618453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe250160bd0 msgr2=0x7fe2500b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe250160bd0 0x7fe2500b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.847+0000 7fe254e00640 1 -- 172.21.15.43:0/1738618453 >> v1:172.21.15.69:6789/0 conn(0x7fe2500b48d0 legacy=0x7fe2500b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.848+0000 7fe254e00640 1 -- 172.21.15.43:0/1738618453 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500d41b0 con 0x7fe2500b3210 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe254e00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2500b3210 0x7fe250167290 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe248009870 tx=0x7fe24802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f734bfb57366d95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe24ea00640 1 -- 172.21.15.43:0/1738618453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248031640 con 0x7fe2500b3210 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe24ea00640 1 -- 172.21.15.43:0/1738618453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2480317a0 con 0x7fe2500b3210 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe24ea00640 1 -- 172.21.15.43:0/1738618453 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248031b30 con 0x7fe2500b3210 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe256e00640 1 -- 172.21.15.43:0/1738618453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2500b3210 msgr2=0x7fe250167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.849+0000 7fe256e00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2500b3210 0x7fe250167290 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe248009870 tx=0x7fe24802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 -- 172.21.15.43:0/1738618453 shutdown_connections 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2500b3210 0x7fe250167290 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 --2- 172.21.15.43:0/1738618453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe250160bd0 0x7fe2500b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 -- 172.21.15.43:0/1738618453 >> 172.21.15.43:0/1738618453 conn(0x7fe25009c5a0 msgr2=0x7fe2500d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 -- 172.21.15.43:0/1738618453 shutdown_connections 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.850+0000 7fe256e00640 1 -- 172.21.15.43:0/1738618453 wait complete. 2024-01-16T15:14:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.851+0000 7fe256e00640 1 Processor -- start 2024-01-16T15:14:48.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.852+0000 7fe256e00640 1 -- start start 2024-01-16T15:14:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.852+0000 7fe256e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2500b3210 0x7fe2500b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe256e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 0x7fe250158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe256e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe256e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b7480 con 0x7fe250160bd0 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe256e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b90a0 con 0x7fe2500b3210 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe256e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500b8e00 con 0x7fe2500b48d0 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37788/0 (socket says 172.21.15.43:37788) 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 -- 172.21.15.43:0/1712064193 learned_addr learned my addr 172.21.15.43:0/1712064193 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe24f400640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 0x7fe250158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe24fe00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2500b3210 0x7fe2500b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 -- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 msgr2=0x7fe250158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 0x7fe250158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 -- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2500b3210 msgr2=0x7fe2500b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2500b3210 0x7fe2500b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe254e00640 1 -- 172.21.15.43:0/1712064193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480094d0 con 0x7fe250160bd0 2024-01-16T15:14:48.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.853+0000 7fe24f400640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 0x7fe250158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe254e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe24802b930 tx=0x7fe2480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24803ac60 con 0x7fe250160bd0 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24803ade0 con 0x7fe250160bd0 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248042630 con 0x7fe250160bd0 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2500b6b50 con 0x7fe250160bd0 2024-01-16T15:14:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.854+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2500b7110 con 0x7fe250160bd0 2024-01-16T15:14:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.856+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2500b7630 con 0x7fe250160bd0 2024-01-16T15:14:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.856+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fe24802f050 con 0x7fe250160bd0 2024-01-16T15:14:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.858+0000 7fe247e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 0x7fe22806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.858+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe248031e40 con 0x7fe250160bd0 2024-01-16T15:14:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.858+0000 7fe24fe00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 0x7fe22806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:48.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.859+0000 7fe24fe00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 0x7fe22806e590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe238006fd0 tx=0x7fe238008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:48.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:48.862+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2500b7630 con 0x7fe250160bd0 2024-01-16T15:14:49.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.055+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2500b7630 con 0x7fe250160bd0 2024-01-16T15:14:49.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.056+0000 7fe247e00640 1 -- 172.21.15.43:0/1712064193 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2500b7630 con 0x7fe250160bd0 2024-01-16T15:14:49.039 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:14:49.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 msgr2=0x7fe22806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 0x7fe22806e590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe238006fd0 tx=0x7fe238008040 comp rx=0 tx=0).stop 2024-01-16T15:14:49.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 msgr2=0x7fe2500b88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe24802b930 tx=0x7fe2480046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 shutdown_connections 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe22806c0d0 0x7fe22806e590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe250160bd0 0x7fe2500b88c0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2500b48d0 0x7fe250158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 --2- 172.21.15.43:0/1712064193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2500b3210 0x7fe2500b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.059+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 >> 172.21.15.43:0/1712064193 conn(0x7fe25009c5a0 msgr2=0x7fe2500cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.060+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 shutdown_connections 2024-01-16T15:14:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.060+0000 7fe256e00640 1 -- 172.21.15.43:0/1712064193 wait complete. 2024-01-16T15:14:49.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:14:49.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.300+0000 7feab9c00640 1 Processor -- start 2024-01-16T15:14:49.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.300+0000 7feab9c00640 1 -- start start 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7feab40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feab40cd7b0 0x7feab40cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab415fa70 con 0x7feab413dfd0 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab415c580 con 0x7feab40cd7b0 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab40cfa10 con 0x7feab40cf050 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7feab413dfd0 0x7feab415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32856/0 (socket says 172.21.15.43:32856) 2024-01-16T15:14:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab3e00640 1 -- 172.21.15.43:0/1321227618 learned_addr learned my addr 172.21.15.43:0/1321227618 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991121134 0 0) 0x7feab415fa70 con 0x7feab413dfd0 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94003520 con 0x7feab413dfd0 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab9200640 1 --2- 172.21.15.43:0/1321227618 >> v2:172.21.15.69:3300/0 conn(0x7feab40cd7b0 0x7feab40cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.302+0000 7feab3400640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7feab40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268331224 0 0) 0x7fea94003520 con 0x7feab413dfd0 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 msgr2=0x7feab40cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7feab40cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 >> v2:172.21.15.69:3300/0 conn(0x7feab40cd7b0 msgr2=0x7feab40cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 --2- 172.21.15.43:0/1321227618 >> v2:172.21.15.69:3300/0 conn(0x7feab40cd7b0 0x7feab40cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4140900 con 0x7feab413dfd0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7feaa4001670 con 0x7feab413dfd0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 >> v1:172.21.15.43:6789/0 conn(0x7feab413dfd0 legacy=0x7feab415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea940039e0 0x7fea94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940047c0 0x7fea94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7fea94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab415c580 con 0x7fea940039e0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab40cfa10 con 0x7fea940047c0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9400b9b0 con 0x7feab40cf050 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.303+0000 7feab3e00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea940039e0 0x7fea94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab9200640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7fea94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3400640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940047c0 0x7fea94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3e00640 1 -- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 msgr2=0x7fea94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3e00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7fea94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3e00640 1 -- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940047c0 msgr2=0x7fea94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3e00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940047c0 0x7fea94004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab3e00640 1 -- 172.21.15.43:0/1321227618 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea94003520 con 0x7fea940039e0 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.304+0000 7feab9200640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab40cf050 0x7fea94005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.305+0000 7feab3e00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea940039e0 0x7fea94003e00 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c0e0 tx=0x7fea940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aec0fd1145879d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.305+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4005ad0 con 0x7fea940039e0 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.305+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa4005c50 con 0x7fea940039e0 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.305+0000 7feab2a00640 1 -- 172.21.15.43:0/1321227618 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa402c810 con 0x7fea940039e0 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 -- 172.21.15.43:0/1321227618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea940039e0 msgr2=0x7fea94003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 --2- 172.21.15.43:0/1321227618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea940039e0 0x7fea94003e00 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c0e0 tx=0x7fea940057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9200640 1 -- 172.21.15.43:0/1321227618 reap_dead start 2024-01-16T15:14:49.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 -- 172.21.15.43:0/1321227618 shutdown_connections 2024-01-16T15:14:49.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 -- 172.21.15.43:0/1321227618 >> 172.21.15.43:0/1321227618 conn(0x7feab406a860 msgr2=0x7feab406ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:49.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 -- 172.21.15.43:0/1321227618 shutdown_connections 2024-01-16T15:14:49.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.306+0000 7feab9c00640 1 -- 172.21.15.43:0/1321227618 wait complete. 2024-01-16T15:14:49.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.308+0000 7feab9c00640 1 Processor -- start 2024-01-16T15:14:49.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.308+0000 7feab9c00640 1 -- start start 2024-01-16T15:14:49.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940039e0 0x7feab41550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab413cb80 0x7feab4070890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab413c5a0 con 0x7feab413d000 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab413b500 con 0x7fea940039e0 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab413b7e0 con 0x7feab413cb80 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37810/0 (socket says 172.21.15.43:37810) 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 -- 172.21.15.43:0/459217198 learned_addr learned my addr 172.21.15.43:0/459217198 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab3e00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940039e0 0x7feab41550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab3400640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab413cb80 0x7feab4070890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 -- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab413cb80 msgr2=0x7feab4070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab413cb80 0x7feab4070890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 -- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940039e0 msgr2=0x7feab41550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940039e0 0x7feab41550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.309+0000 7feab9200640 1 -- 172.21.15.43:0/459217198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab413bac0 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feab9200640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7feab40cfd30 tx=0x7feaac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac017400 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac011040 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab413bda0 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac017780 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.310+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab4155620 con 0x7feab413d000 2024-01-16T15:14:49.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.311+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab413c2c0 con 0x7feab413d000 2024-01-16T15:14:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.312+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7feaac0203b0 con 0x7feab413d000 2024-01-16T15:14:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.313+0000 7feaabe00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 0x7fea8c06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.313+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7feaac088c00 con 0x7feab413d000 2024-01-16T15:14:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.313+0000 7feab3e00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 0x7fea8c06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.315+0000 7feab3e00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 0x7fea8c06e5b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7feaa4034000 tx=0x7feaa4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:49.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.319+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feab413c2c0 con 0x7feab413d000 2024-01-16T15:14:49.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:49.581+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7feab413c2c0 con 0x7feab413d000 2024-01-16T15:14:50.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.152+0000 7feaabe00640 1 -- 172.21.15.43:0/459217198 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v673) v1 ==== 158+0+0 (secure 0 0 0) 0x7feab413c2c0 con 0x7feab413d000 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.155+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 msgr2=0x7fea8c06e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.155+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 0x7fea8c06e5b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7feaa4034000 tx=0x7feaa4007000 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.155+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 msgr2=0x7feab4070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.155+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7feab40cfd30 tx=0x7feaac00f4a0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 shutdown_connections 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fea8c06c0f0 0x7fea8c06e5b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab413d000 0x7feab4070dd0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feab413cb80 0x7feab4070890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 --2- 172.21.15.43:0/459217198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea940039e0 0x7feab41550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 >> 172.21.15.43:0/459217198 conn(0x7feab406a860 msgr2=0x7feab40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 shutdown_connections 2024-01-16T15:14:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.156+0000 7feab9c00640 1 -- 172.21.15.43:0/459217198 wait complete. 2024-01-16T15:14:50.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:14:50.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.408+0000 7f2c5d600640 1 Processor -- start 2024-01-16T15:14:50.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.409+0000 7f2c5d600640 1 -- start start 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.410+0000 7f2c5d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2c580acd70 0x7f2c5806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.410+0000 7f2c5d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c58070490 0x7f2c58070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.410+0000 7f2c5d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5806d7a0 con 0x7f2c580ae490 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.410+0000 7f2c5d600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5815a5b0 con 0x7f2c580acd70 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.410+0000 7f2c5d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5806ce70 con 0x7f2c58070490 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2c580acd70 0x7f2c5806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2c580acd70 0x7f2c5806dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37812/0 (socket says 172.21.15.43:37812) 2024-01-16T15:14:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 -- 172.21.15.43:0/1574713514 learned_addr learned my addr 172.21.15.43:0/1574713514 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:50.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 -- 172.21.15.43:0/1574713514 >> v2:172.21.15.69:3300/0 conn(0x7f2c58070490 msgr2=0x7f2c58070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:50.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 --2- 172.21.15.43:0/1574713514 >> v2:172.21.15.69:3300/0 conn(0x7f2c58070490 0x7f2c58070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 -- 172.21.15.43:0/1574713514 >> v1:172.21.15.69:6789/0 conn(0x7f2c580ae490 legacy=0x7f2c580ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:50.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.411+0000 7f2c56a00640 1 -- 172.21.15.43:0/1574713514 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5806d150 con 0x7f2c580acd70 2024-01-16T15:14:50.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.412+0000 7f2c56a00640 1 --2- 172.21.15.43:0/1574713514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c580acd70 0x7f2c5806dc30 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2c58067e70 tx=0x7f2c4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd44817e4413f966 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:50.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.413+0000 7f2c55600640 1 -- 172.21.15.43:0/1574713514 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4802ca80 con 0x7f2c580acd70 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.413+0000 7f2c55600640 1 -- 172.21.15.43:0/1574713514 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4802cbe0 con 0x7f2c580acd70 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.413+0000 7f2c55600640 1 -- 172.21.15.43:0/1574713514 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c480343f0 con 0x7f2c580acd70 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.413+0000 7f2c5d600640 1 -- 172.21.15.43:0/1574713514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c580acd70 msgr2=0x7f2c5806dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.413+0000 7f2c5d600640 1 --2- 172.21.15.43:0/1574713514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c580acd70 0x7f2c5806dc30 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2c58067e70 tx=0x7f2c4802bee0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 -- 172.21.15.43:0/1574713514 shutdown_connections 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 --2- 172.21.15.43:0/1574713514 >> v2:172.21.15.69:3300/0 conn(0x7f2c58070490 0x7f2c58070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 --2- 172.21.15.43:0/1574713514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c580acd70 0x7f2c5806dc30 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 -- 172.21.15.43:0/1574713514 >> 172.21.15.43:0/1574713514 conn(0x7f2c580b0ba0 msgr2=0x7f2c580b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 -- 172.21.15.43:0/1574713514 shutdown_connections 2024-01-16T15:14:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.414+0000 7f2c5d600640 1 -- 172.21.15.43:0/1574713514 wait complete. 2024-01-16T15:14:50.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.416+0000 7f2c5d600640 1 Processor -- start 2024-01-16T15:14:50.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.416+0000 7f2c5d600640 1 -- start start 2024-01-16T15:14:50.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c580acd70 0x7f2c5815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c580ae490 0x7f2c581e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581f71d0 con 0x7f2c58070490 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581f6600 con 0x7f2c580acd70 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c5d600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581f68a0 con 0x7f2c580ae490 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37828/0 (socket says 172.21.15.43:37828) 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 -- 172.21.15.43:0/3911868483 learned_addr learned my addr 172.21.15.43:0/3911868483 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 -- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c580ae490 msgr2=0x7f2c581e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c580ae490 0x7f2c581e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56a00640 1 -- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c580acd70 msgr2=0x7f2c5815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.417+0000 7f2c56000640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c580acd70 0x7f2c5815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c56a00640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c580acd70 0x7f2c5815b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c56a00640 1 -- 172.21.15.43:0/3911868483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c48009b30 con 0x7f2c58070490 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c56a00640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2c480093e0 tx=0x7f2c48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4803e450 con 0x7f2c58070490 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4803e5b0 con 0x7f2c58070490 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4803e960 con 0x7f2c58070490 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c581f6b80 con 0x7f2c58070490 2024-01-16T15:14:50.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.418+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5814e6e0 con 0x7f2c58070490 2024-01-16T15:14:50.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.420+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5814e870 con 0x7f2c58070490 2024-01-16T15:14:50.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.421+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f2c4802cd50 con 0x7f2c58070490 2024-01-16T15:14:50.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.422+0000 7f2c4ea00640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 0x7f2c3006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:50.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.422+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2c480ab6c0 con 0x7f2c58070490 2024-01-16T15:14:50.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.422+0000 7f2c56000640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 0x7f2c3006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:50.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.423+0000 7f2c56000640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 0x7f2c3006e590 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2c5814f2a0 tx=0x7f2c40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.426+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c480e0e90 con 0x7f2c58070490 2024-01-16T15:14:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.626+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2c5809d200 con 0x7f2c58070490 2024-01-16T15:14:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.634+0000 7f2c4ea00640 1 -- 172.21.15.43:0/3911868483 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2c48043020 con 0x7f2c58070490 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.637+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 msgr2=0x7f2c3006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.637+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 0x7f2c3006e590 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2c5814f2a0 tx=0x7f2c40009040 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.637+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 msgr2=0x7f2c5814e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.637+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f2c480093e0 tx=0x7f2c48004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 shutdown_connections 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c3006c0d0 0x7f2c3006e590 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c580acd70 0x7f2c5815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c580ae490 0x7f2c581e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 --2- 172.21.15.43:0/3911868483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c58070490 0x7f2c5814e1a0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 >> 172.21.15.43:0/3911868483 conn(0x7f2c580b0ba0 msgr2=0x7f2c580ad150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 shutdown_connections 2024-01-16T15:14:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:50.638+0000 7f2c5d600640 1 -- 172.21.15.43:0/3911868483 wait complete. 2024-01-16T15:14:50.641 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:14:50.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:50.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:14:50.891 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:14:50.891 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:14:50.891 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:50.952 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:14:50.952 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:14:50.952 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:51.009 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:14:51.009 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:14:51.009 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:51.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:14:51.068 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:14:51.068 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:51.127 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:14:51.127 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:14:51.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:14:51.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.375+0000 7f9f5e400640 1 Processor -- start 2024-01-16T15:14:51.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.375+0000 7f9f5e400640 1 -- start start 2024-01-16T15:14:51.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5e400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9f58157770 0x7f9f5806cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5815d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5e400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58161670 con 0x7f9f5815ddb0 2024-01-16T15:14:51.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5e400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5806d500 con 0x7f9f58157770 2024-01-16T15:14:51.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58071730 con 0x7f9f5806d080 2024-01-16T15:14:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5da00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9f5815ddb0 0x7f9f580ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:32864/0 (socket says 172.21.15.43:32864) 2024-01-16T15:14:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5da00640 1 -- 172.21.15.43:0/749551831 learned_addr learned my addr 172.21.15.43:0/749551831 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.377+0000 7f9f5d000640 1 --2- 172.21.15.43:0/749551831 >> v2:172.21.15.43:3300/0 conn(0x7f9f58157770 0x7f9f5806cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f4fc00640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f57e00640 1 -- 172.21.15.43:0/749551831 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946798891 0 0) 0x7f9f58161670 con 0x7f9f5815ddb0 2024-01-16T15:14:51.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f5d000640 1 -- 172.21.15.43:0/749551831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 msgr2=0x7f9f5815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:51.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f5d000640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5815d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f5d000640 1 -- 172.21.15.43:0/749551831 >> v1:172.21.15.43:6789/0 conn(0x7f9f5815ddb0 legacy=0x7f9f580ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.378+0000 7f9f5d000640 1 -- 172.21.15.43:0/749551831 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5815e610 con 0x7f9f58157770 2024-01-16T15:14:51.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.380+0000 7f9f5d000640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f5806cb40 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f4802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a73ef7ab93e543 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.380+0000 7f9f57e00640 1 -- 172.21.15.43:0/749551831 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4802cc50 con 0x7f9f58157770 2024-01-16T15:14:51.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.380+0000 7f9f57e00640 1 -- 172.21.15.43:0/749551831 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4802cdb0 con 0x7f9f58157770 2024-01-16T15:14:51.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.380+0000 7f9f57e00640 1 -- 172.21.15.43:0/749551831 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48034660 con 0x7f9f58157770 2024-01-16T15:14:51.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 -- 172.21.15.43:0/749551831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 msgr2=0x7f9f5806cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f5806cb40 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f4802bea0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 -- 172.21.15.43:0/749551831 shutdown_connections 2024-01-16T15:14:51.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 --2- 172.21.15.43:0/749551831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f5806cb40 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 -- 172.21.15.43:0/749551831 >> 172.21.15.43:0/749551831 conn(0x7f9f5806a860 msgr2=0x7f9f5806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 -- 172.21.15.43:0/749551831 shutdown_connections 2024-01-16T15:14:51.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.381+0000 7f9f5e400640 1 -- 172.21.15.43:0/749551831 wait complete. 2024-01-16T15:14:51.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.383+0000 7f9f5e400640 1 Processor -- start 2024-01-16T15:14:51.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.383+0000 7f9f5e400640 1 -- start start 2024-01-16T15:14:51.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 0x7f9f580b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581f6a30 con 0x7f9f58157770 2024-01-16T15:14:51.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580cc550 con 0x7f9f5815ddb0 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.384+0000 7f9f5e400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580cc830 con 0x7f9f5806d080 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37846/0 (socket says 172.21.15.43:37846) 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 -- 172.21.15.43:0/4169904098 learned_addr learned my addr 172.21.15.43:0/4169904098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f5d000640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f5da00640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 0x7f9f580b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 -- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 msgr2=0x7f9f5814eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5814eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 -- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 msgr2=0x7f9f580b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 0x7f9f580b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f4fc00640 1 -- 172.21.15.43:0/4169904098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48009a70 con 0x7f9f58157770 2024-01-16T15:14:51.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.385+0000 7f9f5da00640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 0x7f9f580b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f4fc00640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000d800 tx=0x7f9f4000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4000b6c0 con 0x7f9f58157770 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4000b820 con 0x7f9f58157770 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f581f6370 con 0x7f9f58157770 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4000bbb0 con 0x7f9f58157770 2024-01-16T15:14:51.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.386+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f580ccb10 con 0x7f9f58157770 2024-01-16T15:14:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.387+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f58155a00 con 0x7f9f58157770 2024-01-16T15:14:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.388+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f9f4000bd10 con 0x7f9f58157770 2024-01-16T15:14:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.389+0000 7f9f55600640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 0x7f9f3006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.389+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9f40088e70 con 0x7f9f58157770 2024-01-16T15:14:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.390+0000 7f9f5d000640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 0x7f9f3006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.390+0000 7f9f5d000640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 0x7f9f3006e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9f48036000 tx=0x7f9f48030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.394+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f40055470 con 0x7f9f58157770 2024-01-16T15:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.625+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f5804e840 con 0x7f9f58157770 2024-01-16T15:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.626+0000 7f9f55600640 1 -- 172.21.15.43:0/4169904098 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f9f40058a90 con 0x7f9f58157770 2024-01-16T15:14:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:51.610 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6026,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/206348255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":206348255},{"type":"v1","addr":"172.21.15.69:6834","nonce":206348255}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3389933703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3389933703},{"type":"v1","addr":"172.21.15.69:6835","nonce":3389933703}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/1040529098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1040529098},{"type":"v1","addr":"172.21.15.43:6835","nonce":1040529098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2617094504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":2617094504},{"type":"v1","addr":"172.21.15.43:6837","nonce":2617094504}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-01-16T15:14:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 60 2024-01-16T15:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 msgr2=0x7f9f3006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 0x7f9f3006e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9f48036000 tx=0x7f9f48030000 comp rx=0 tx=0).stop 2024-01-16T15:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 msgr2=0x7f9f581f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000d800 tx=0x7f9f4000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 shutdown_connections 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9f3006c100 0x7f9f3006e5c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f5815ddb0 0x7f9f580b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9f58157770 0x7f9f581f6e50 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 --2- 172.21.15.43:0/4169904098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f5806d080 0x7f9f5814eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 >> 172.21.15.43:0/4169904098 conn(0x7f9f5806a860 msgr2=0x7f9f581552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 shutdown_connections 2024-01-16T15:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.629+0000 7f9f5e400640 1 -- 172.21.15.43:0/4169904098 wait complete. 2024-01-16T15:14:51.633 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:14:51.633 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:14:51.633 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:14:51.634 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:14:51.634 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:14:51.634 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:14:51.634 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:14:51.634 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:14:51.634 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:14:51.635 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:14:51.635 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:14:51.635 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:14:51.635 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:14:51.635 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:14:51.635 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:14:51.636 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:14:51.657 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:14:51.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:14:51.660 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:14:51.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:14:51.661 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:14:51.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:14:51.662 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:14:51.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:14:51.706 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:14:51.707 DEBUG:teuthology.parallel:result is None 2024-01-16T15:14:51.708 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:14:51.709 DEBUG:teuthology.parallel:result is None 2024-01-16T15:14:51.713 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:14:51.713 DEBUG:teuthology.parallel:result is None 2024-01-16T15:14:51.714 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:14:51.714 DEBUG:teuthology.parallel:result is None 2024-01-16T15:14:51.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:14:51.865 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:14:51.865 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:14:51.872 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:14:51.878 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:14:51.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.918+0000 7f9ae6e00640 1 Processor -- start 2024-01-16T15:14:51.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.919+0000 7f9ae6e00640 1 -- start start 2024-01-16T15:14:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae6e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9ae0146c70 0x7f9ae00b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae6e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ae00b3710 0x7f9ae015eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae00b3b60 con 0x7f9ae0146890 2024-01-16T15:14:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae6e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae006cd20 con 0x7f9ae0146c70 2024-01-16T15:14:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae6e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae0161ec0 con 0x7f9ae00b3710 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ad7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9ae0146c70 0x7f9ae00b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9ae4e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ae00b3710 0x7f9ae015eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9adfe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9ae0146890 0x7f9ae006c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:46120/0 (socket says 172.21.15.43:46120) 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.920+0000 7f9adfe00640 1 -- 172.21.15.43:0/3378122944 learned_addr learned my addr 172.21.15.43:0/3378122944 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9adf400640 1 -- 172.21.15.43:0/3378122944 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711072902 0 0) 0x7f9ad0001110 con 0x7f9ae0146890 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9adf400640 1 -- 172.21.15.43:0/3378122944 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac0003520 con 0x7f9ae0146890 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9ae4e00640 1 -- 172.21.15.43:0/3378122944 >> v2:172.21.15.43:3300/0 conn(0x7f9ae0146c70 msgr2=0x7f9ae00b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9ae4e00640 1 --2- 172.21.15.43:0/3378122944 >> v2:172.21.15.43:3300/0 conn(0x7f9ae0146c70 0x7f9ae00b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9ae4e00640 1 -- 172.21.15.43:0/3378122944 >> v1:172.21.15.69:6789/0 conn(0x7f9ae0146890 legacy=0x7f9ae006c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.921+0000 7f9ae4e00640 1 -- 172.21.15.43:0/3378122944 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae0161be0 con 0x7f9ae00b3710 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.922+0000 7f9ae4e00640 1 --2- 172.21.15.43:0/3378122944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae00b3710 0x7f9ae015eb10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9ae015d950 tx=0x7f9ad802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2645668c1389f306 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.922+0000 7f9adf400640 1 -- 172.21.15.43:0/3378122944 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad802cc70 con 0x7f9ae00b3710 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.922+0000 7f9adf400640 1 -- 172.21.15.43:0/3378122944 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad802cdd0 con 0x7f9ae00b3710 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.922+0000 7f9adf400640 1 -- 172.21.15.43:0/3378122944 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8034660 con 0x7f9ae00b3710 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 -- 172.21.15.43:0/3378122944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae00b3710 msgr2=0x7f9ae015eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/3378122944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae00b3710 0x7f9ae015eb10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9ae015d950 tx=0x7f9ad802bec0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 -- 172.21.15.43:0/3378122944 shutdown_connections 2024-01-16T15:14:51.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/3378122944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae00b3710 0x7f9ae015eb10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/3378122944 >> v2:172.21.15.43:3300/0 conn(0x7f9ae0146c70 0x7f9ae00b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 -- 172.21.15.43:0/3378122944 >> 172.21.15.43:0/3378122944 conn(0x7f9ae0066610 msgr2=0x7f9ae0066a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 -- 172.21.15.43:0/3378122944 shutdown_connections 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.923+0000 7f9ae6e00640 1 -- 172.21.15.43:0/3378122944 wait complete. 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.925+0000 7f9ae6e00640 1 Processor -- start 2024-01-16T15:14:51.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.925+0000 7f9ae6e00640 1 -- start start 2024-01-16T15:14:51.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ae0146890 0x7f9ae00d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00d00c0 con 0x7f9ae0146890 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae015d1d0 con 0x7f9ae0146c70 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00cf440 con 0x7f9ae00b3710 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9adfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ad7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ae0146890 0x7f9ae00d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:43956/0 (socket says 172.21.15.43:43956) 2024-01-16T15:14:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9adfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:42962/0 (socket says 172.21.15.43:42962) 2024-01-16T15:14:51.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.926+0000 7f9ae4e00640 1 -- 172.21.15.43:0/1058994839 learned_addr learned my addr 172.21.15.43:0/1058994839 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:51.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 -- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 msgr2=0x7f9ae015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 -- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ae0146890 msgr2=0x7f9ae00d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ae0146890 0x7f9ae00d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 -- 172.21.15.43:0/1058994839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8009a70 con 0x7f9ae0146c70 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9adfe00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae4e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009e80 tx=0x7f9ad8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8040650 con 0x7f9ae0146c70 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad80407b0 con 0x7f9ae0146c70 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8040b60 con 0x7f9ae0146c70 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae00cf720 con 0x7f9ae0146c70 2024-01-16T15:14:51.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.927+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae00cfce0 con 0x7f9ae0146c70 2024-01-16T15:14:51.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.929+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae00d0200 con 0x7f9ae0146c70 2024-01-16T15:14:51.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.930+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f9ad8040cc0 con 0x7f9ae0146c70 2024-01-16T15:14:51.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.931+0000 7f9adcc00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 0x7f9ab806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:51.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.931+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9ad80ab1a0 con 0x7f9ae0146c70 2024-01-16T15:14:51.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.931+0000 7f9adfe00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 0x7f9ab806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:51.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.931+0000 7f9adfe00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 0x7f9ab806e590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0001270 tx=0x7f9ad0007bb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:51.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:51.936+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ae00d0200 con 0x7f9ae0146c70 2024-01-16T15:14:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.145+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9ae00d0200 con 0x7f9ae0146c70 2024-01-16T15:14:52.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.185+0000 7f9adcc00640 1 -- 172.21.15.43:0/1058994839 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9ae00d0200 con 0x7f9ae0146c70 2024-01-16T15:14:52.168 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:14:52.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 msgr2=0x7f9ab806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 0x7f9ab806e590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0001270 tx=0x7f9ad0007bb0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 msgr2=0x7f9ae00d1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009e80 tx=0x7f9ad8004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 shutdown_connections 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9ab806c0d0 0x7f9ab806e590 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ae0146c70 0x7f9ae00d1310 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ae0146890 0x7f9ae00d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 --2- 172.21.15.43:0/1058994839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ae00b3710 0x7f9ae015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 >> 172.21.15.43:0/1058994839 conn(0x7f9ae0066610 msgr2=0x7f9ae0160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 shutdown_connections 2024-01-16T15:14:52.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.188+0000 7f9ae6e00640 1 -- 172.21.15.43:0/1058994839 wait complete. 2024-01-16T15:14:52.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:14:52.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.409+0000 7fe2c2000640 1 Processor -- start 2024-01-16T15:14:52.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.409+0000 7fe2c2000640 1 -- start start 2024-01-16T15:14:52.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2bc145e80 0x7fe2bc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc06a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2bc06a980 0x7fe2bc06d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc06ae10 con 0x7fe2bc06a980 2024-01-16T15:14:52.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc146260 con 0x7fe2bc145e80 2024-01-16T15:14:52.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc0d0c80 con 0x7fe2bc1469a0 2024-01-16T15:14:52.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.411+0000 7fe2c1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2bc06a980 0x7fe2bc06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2bc06a980 0x7fe2bc06d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37884/0 (socket says 172.21.15.43:37884) 2024-01-16T15:14:52.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 -- 172.21.15.43:0/1317401679 learned_addr learned my addr 172.21.15.43:0/1317401679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:52.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c0c00640 1 --2- 172.21.15.43:0/1317401679 >> v2:172.21.15.69:3300/0 conn(0x7fe2bc145e80 0x7fe2bc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2bbe00640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc06a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 -- 172.21.15.43:0/1317401679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 msgr2=0x7fe2bc06a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc06a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 -- 172.21.15.43:0/1317401679 >> v2:172.21.15.69:3300/0 conn(0x7fe2bc145e80 msgr2=0x7fe2bc0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 --2- 172.21.15.43:0/1317401679 >> v2:172.21.15.69:3300/0 conn(0x7fe2bc145e80 0x7fe2bc0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.412+0000 7fe2c1600640 1 -- 172.21.15.43:0/1317401679 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc146730 con 0x7fe2bc06a980 2024-01-16T15:14:52.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.414+0000 7fe2c1600640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc06d490 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc06e940 tx=0x7fe2b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30012f7d7ba5a751 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.414+0000 7fe2bb400640 1 -- 172.21.15.43:0/1317401679 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4031870 con 0x7fe2bc06a980 2024-01-16T15:14:52.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.414+0000 7fe2bb400640 1 -- 172.21.15.43:0/1317401679 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b40319d0 con 0x7fe2bc06a980 2024-01-16T15:14:52.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.414+0000 7fe2bb400640 1 -- 172.21.15.43:0/1317401679 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4031d60 con 0x7fe2bc06a980 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 -- 172.21.15.43:0/1317401679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 msgr2=0x7fe2bc06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc06d490 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc06e940 tx=0x7fe2b402b910 comp rx=0 tx=0).stop 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 -- 172.21.15.43:0/1317401679 shutdown_connections 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc06d490 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 --2- 172.21.15.43:0/1317401679 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc06a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 --2- 172.21.15.43:0/1317401679 >> v2:172.21.15.69:3300/0 conn(0x7fe2bc145e80 0x7fe2bc0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 -- 172.21.15.43:0/1317401679 >> 172.21.15.43:0/1317401679 conn(0x7fe2bc0cfd20 msgr2=0x7fe2bc0d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 -- 172.21.15.43:0/1317401679 shutdown_connections 2024-01-16T15:14:52.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.415+0000 7fe2c2000640 1 -- 172.21.15.43:0/1317401679 wait complete. 2024-01-16T15:14:52.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.417+0000 7fe2c2000640 1 Processor -- start 2024-01-16T15:14:52.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.417+0000 7fe2c2000640 1 -- start start 2024-01-16T15:14:52.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 0x7fe2bc15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc15f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc13f3d0 con 0x7fe2bc06a980 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc15f6c0 con 0x7fe2bc145e80 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc15f9a0 con 0x7fe2bc1469a0 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37896/0 (socket says 172.21.15.43:37896) 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.418+0000 7fe2c0c00640 1 -- 172.21.15.43:0/3933836756 learned_addr learned my addr 172.21.15.43:0/3933836756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c1600640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 -- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 msgr2=0x7fe2bc15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2bbe00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 0x7fe2bc15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 -- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 msgr2=0x7fe2bc15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 0x7fe2bc15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 -- 172.21.15.43:0/3933836756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40094d0 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2bbe00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 0x7fe2bc15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c0c00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac007ef0 tx=0x7fe2ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac014620 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac014780 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac014b30 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.419+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc15fc80 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.420+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc160240 con 0x7fe2bc06a980 2024-01-16T15:14:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.421+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc160400 con 0x7fe2bc06a980 2024-01-16T15:14:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.422+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fe2ac014c90 con 0x7fe2bc06a980 2024-01-16T15:14:52.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.423+0000 7fe2b8c00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 0x7fe29406e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.423+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fe2ac0892b0 con 0x7fe2bc06a980 2024-01-16T15:14:52.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.423+0000 7fe2bbe00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 0x7fe29406e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.424+0000 7fe2bbe00640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 0x7fe29406e610 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0ccbd0 tx=0x7fe2a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.427+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2ac0558f0 con 0x7fe2bc06a980 2024-01-16T15:14:52.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.689+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe2bc04e840 con 0x7fe2bc06a980 2024-01-16T15:14:52.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.689+0000 7fe2b8c00640 1 -- 172.21.15.43:0/3933836756 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe2ac058f10 con 0x7fe2bc06a980 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:14:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:14:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:14:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:14:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:14:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:14:52.685 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 msgr2=0x7fe29406e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 0x7fe29406e610 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0ccbd0 tx=0x7fe2a4008040 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 msgr2=0x7fe2bc15c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac007ef0 tx=0x7fe2ac00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 shutdown_connections 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29406c150 0x7fe29406e610 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2bc145e80 0x7fe2bc15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2bc1469a0 0x7fe2bc15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 --2- 172.21.15.43:0/3933836756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2bc06a980 0x7fe2bc15c750 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.692+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 >> 172.21.15.43:0/3933836756 conn(0x7fe2bc0cfd20 msgr2=0x7fe2bc0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.693+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 shutdown_connections 2024-01-16T15:14:52.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.693+0000 7fe2c2000640 1 -- 172.21.15.43:0/3933836756 wait complete. 2024-01-16T15:14:52.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:52.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.929+0000 7f62e3400640 1 Processor -- start 2024-01-16T15:14:52.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.930+0000 7f62e3400640 1 -- start start 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.932+0000 7f62e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.932+0000 7f62e3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62dc0cd7b0 0x7f62dc0cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.932+0000 7f62e3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc15fa70 con 0x7f62dc13dfd0 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.932+0000 7f62e3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc15c580 con 0x7f62dc0cd7b0 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.932+0000 7f62e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0cfa10 con 0x7f62dc0cf050 2024-01-16T15:14:52.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e2a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62dc0cd7b0 0x7f62dc0cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:42986/0 (socket says 172.21.15.43:42986) 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 -- 172.21.15.43:0/1660288208 learned_addr learned my addr 172.21.15.43:0/1660288208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e2000640 1 --1- 172.21.15.43:0/1660288208 >> v1:172.21.15.69:6789/0 conn(0x7f62dc13dfd0 0x7f62dc15be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:46136/0 (socket says 172.21.15.43:46136) 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 -- 172.21.15.43:0/1660288208 >> v2:172.21.15.69:3300/0 conn(0x7f62dc0cd7b0 msgr2=0x7f62dc0cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 --2- 172.21.15.43:0/1660288208 >> v2:172.21.15.69:3300/0 conn(0x7f62dc0cd7b0 0x7f62dc0cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 -- 172.21.15.43:0/1660288208 >> v1:172.21.15.69:6789/0 conn(0x7f62dc13dfd0 legacy=0x7f62dc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.933+0000 7f62e1600640 1 -- 172.21.15.43:0/1660288208 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc140900 con 0x7f62dc0cf050 2024-01-16T15:14:52.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.935+0000 7f62e1600640 1 --2- 172.21.15.43:0/1660288208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009870 tx=0x7f62c402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=222481a08ce34037 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.936+0000 7f62e0c00640 1 -- 172.21.15.43:0/1660288208 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4030840 con 0x7f62dc0cf050 2024-01-16T15:14:52.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.936+0000 7f62e0c00640 1 -- 172.21.15.43:0/1660288208 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c40309a0 con 0x7f62dc0cf050 2024-01-16T15:14:52.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.936+0000 7f62e0c00640 1 -- 172.21.15.43:0/1660288208 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4030d30 con 0x7f62dc0cf050 2024-01-16T15:14:52.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.936+0000 7f62e3400640 1 -- 172.21.15.43:0/1660288208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 msgr2=0x7f62dc0cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.936+0000 7f62e3400640 1 --2- 172.21.15.43:0/1660288208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009870 tx=0x7f62c402b980 comp rx=0 tx=0).stop 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 -- 172.21.15.43:0/1660288208 shutdown_connections 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 --2- 172.21.15.43:0/1660288208 >> v2:172.21.15.69:3300/0 conn(0x7f62dc0cd7b0 0x7f62dc0cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 --2- 172.21.15.43:0/1660288208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc0cf050 0x7f62dc0cf4d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 -- 172.21.15.43:0/1660288208 >> 172.21.15.43:0/1660288208 conn(0x7f62dc06a860 msgr2=0x7f62dc06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 -- 172.21.15.43:0/1660288208 shutdown_connections 2024-01-16T15:14:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.937+0000 7f62e3400640 1 -- 172.21.15.43:0/1660288208 wait complete. 2024-01-16T15:14:52.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.939+0000 7f62e3400640 1 Processor -- start 2024-01-16T15:14:52.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.939+0000 7f62e3400640 1 -- start start 2024-01-16T15:14:52.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62dc0cd7b0 0x7f62dc155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc13dfd0 0x7f62dc15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc15a760 con 0x7f62dc0cf050 2024-01-16T15:14:52.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc159b50 con 0x7f62dc0cd7b0 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc159e30 con 0x7f62dc13dfd0 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37904/0 (socket says 172.21.15.43:37904) 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e1600640 1 -- 172.21.15.43:0/2165467777 learned_addr learned my addr 172.21.15.43:0/2165467777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e2a00640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc13dfd0 0x7f62dc15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.940+0000 7f62e2000640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62dc0cd7b0 0x7f62dc155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 -- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc13dfd0 msgr2=0x7f62dc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc13dfd0 0x7f62dc15abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 -- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62dc0cd7b0 msgr2=0x7f62dc155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62dc0cd7b0 0x7f62dc155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:52.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 -- 172.21.15.43:0/2165467777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40094d0 con 0x7f62dc0cf050 2024-01-16T15:14:52.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e1600640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f62c4000c00 tx=0x7f62c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4039c90 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c4039df0 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4043640 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.941+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc15a110 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.942+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1597e0 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.943+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc15a910 con 0x7f62dc0cf050 2024-01-16T15:14:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.944+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f62c40437a0 con 0x7f62dc0cf050 2024-01-16T15:14:52.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.945+0000 7f62da000640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 0x7f62b406e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:52.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.945+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f62c403e070 con 0x7f62dc0cf050 2024-01-16T15:14:52.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.945+0000 7f62e2000640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 0x7f62b406e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:52.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.946+0000 7f62e2000640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 0x7f62b406e5c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f62cc000f60 tx=0x7f62cc007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:52.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:52.949+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62dc15a910 con 0x7f62dc0cf050 2024-01-16T15:14:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.144+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62dc04e840 con 0x7f62dc0cf050 2024-01-16T15:14:53.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.145+0000 7f62da000640 1 -- 172.21.15.43:0/2165467777 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+12853 (secure 0 0 0) 0x7f62c4078020 con 0x7f62dc0cf050 2024-01-16T15:14:53.157 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:53.157 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":97,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:52.192388+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:53.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 msgr2=0x7f62b406e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 0x7f62b406e5c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f62cc000f60 tx=0x7f62cc007c80 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 msgr2=0x7f62dc13b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f62c4000c00 tx=0x7f62c4004730 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 shutdown_connections 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62b406c100 0x7f62b406e5c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62dc13dfd0 0x7f62dc15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62dc0cf050 0x7f62dc13b690 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 --2- 172.21.15.43:0/2165467777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62dc0cd7b0 0x7f62dc155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 >> 172.21.15.43:0/2165467777 conn(0x7f62dc06a860 msgr2=0x7f62dc15c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.148+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 shutdown_connections 2024-01-16T15:14:53.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.149+0000 7f62e3400640 1 -- 172.21.15.43:0/2165467777 wait complete. 2024-01-16T15:14:53.160 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:14:53.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:14:53.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.391+0000 7faa27200640 1 Processor -- start 2024-01-16T15:14:53.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.391+0000 7faa27200640 1 -- start start 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa27200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faa2815f270 0x7faa280cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa27200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa2806d9e0 0x7faa280cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa27200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2815f930 con 0x7faa28144770 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa27200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa280b43b0 con 0x7faa2815f270 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa27200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa280b3b10 con 0x7faa2806d9e0 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa25e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7faa28144770 0x7faa2815eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53014/0 (socket says 172.21.15.43:53014) 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa25e00640 1 -- 172.21.15.43:0/2416742719 learned_addr learned my addr 172.21.15.43:0/2416742719 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:53.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2416742719 >> v2:172.21.15.43:3300/0 conn(0x7faa2815f270 0x7faa280cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.393+0000 7faa26800640 1 --2- 172.21.15.43:0/2416742719 >> v2:172.21.15.69:3300/0 conn(0x7faa2806d9e0 0x7faa280cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa25400640 1 -- 172.21.15.43:0/2416742719 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490416791 0 0) 0x7faa2815f930 con 0x7faa28144770 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa25400640 1 -- 172.21.15.43:0/2416742719 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa08003520 con 0x7faa28144770 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa1fe00640 1 -- 172.21.15.43:0/2416742719 >> v2:172.21.15.69:3300/0 conn(0x7faa2806d9e0 msgr2=0x7faa280cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2416742719 >> v2:172.21.15.69:3300/0 conn(0x7faa2806d9e0 0x7faa280cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa1fe00640 1 -- 172.21.15.43:0/2416742719 >> v1:172.21.15.43:6789/0 conn(0x7faa28144770 legacy=0x7faa2815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.394+0000 7faa1fe00640 1 -- 172.21.15.43:0/2416742719 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa280b3df0 con 0x7faa2815f270 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.395+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2416742719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa2815f270 0x7faa280cd4a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7faa28144eb0 tx=0x7faa1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8921fc1807fec89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.395+0000 7faa25400640 1 -- 172.21.15.43:0/2416742719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10031830 con 0x7faa2815f270 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.395+0000 7faa25400640 1 -- 172.21.15.43:0/2416742719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10031990 con 0x7faa2815f270 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.395+0000 7faa25400640 1 -- 172.21.15.43:0/2416742719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10031d20 con 0x7faa2815f270 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 -- 172.21.15.43:0/2416742719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa2815f270 msgr2=0x7faa280cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 --2- 172.21.15.43:0/2416742719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa2815f270 0x7faa280cd4a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7faa28144eb0 tx=0x7faa1002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 -- 172.21.15.43:0/2416742719 shutdown_connections 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 --2- 172.21.15.43:0/2416742719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa2815f270 0x7faa280cd4a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 --2- 172.21.15.43:0/2416742719 >> v2:172.21.15.69:3300/0 conn(0x7faa2806d9e0 0x7faa280cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 -- 172.21.15.43:0/2416742719 >> 172.21.15.43:0/2416742719 conn(0x7faa280cec10 msgr2=0x7faa280cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 -- 172.21.15.43:0/2416742719 shutdown_connections 2024-01-16T15:14:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.396+0000 7faa27200640 1 -- 172.21.15.43:0/2416742719 wait complete. 2024-01-16T15:14:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.398+0000 7faa27200640 1 Processor -- start 2024-01-16T15:14:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.398+0000 7faa27200640 1 -- start start 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa2806d9e0 0x7faa2813a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa2815f270 0x7faa281f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2813acc0 con 0x7faa28144770 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28147c90 con 0x7faa2806d9e0 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa27200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa28147f70 con 0x7faa2815f270 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53968/0 (socket says 172.21.15.43:53968) 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 -- 172.21.15.43:0/2594662137 learned_addr learned my addr 172.21.15.43:0/2594662137 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 -- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa2815f270 msgr2=0x7faa281f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa2815f270 0x7faa281f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 -- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa2806d9e0 msgr2=0x7faa2813a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa2806d9e0 0x7faa2813a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.399+0000 7faa1fe00640 1 -- 172.21.15.43:0/2594662137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009400 con 0x7faa28144770 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa1fe00640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7faa10000c00 tx=0x7faa100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1003a8f0 con 0x7faa28144770 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1003aa70 con 0x7faa28144770 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa28148250 con 0x7faa28144770 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1003ae20 con 0x7faa28144770 2024-01-16T15:14:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.400+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa28148810 con 0x7faa28144770 2024-01-16T15:14:53.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.401+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2804e840 con 0x7faa28144770 2024-01-16T15:14:53.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.402+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7faa1004d450 con 0x7faa28144770 2024-01-16T15:14:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.403+0000 7faa1e000640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 0x7faa0006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.403+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4885+0+0 (secure 0 0 0) 0x7faa100aa9a0 con 0x7faa28144770 2024-01-16T15:14:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.403+0000 7faa25e00640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 0x7faa0006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.404+0000 7faa25e00640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 0x7faa0006e590 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7faa18001610 tx=0x7faa18007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.408+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa10076fa0 con 0x7faa28144770 2024-01-16T15:14:53.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.600+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa280cc520 con 0x7faa28144770 2024-01-16T15:14:53.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.601+0000 7faa1e000640 1 -- 172.21.15.43:0/2594662137 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa10035020 con 0x7faa28144770 2024-01-16T15:14:53.584 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.604+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 msgr2=0x7faa0006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.604+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 0x7faa0006e590 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7faa18001610 tx=0x7faa18007930 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.604+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 msgr2=0x7faa281471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.604+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7faa10000c00 tx=0x7faa100046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 shutdown_connections 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faa0006c0d0 0x7faa0006e590 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa2815f270 0x7faa281f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa28144770 0x7faa281471f0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 --2- 172.21.15.43:0/2594662137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa2806d9e0 0x7faa2813a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 >> 172.21.15.43:0/2594662137 conn(0x7faa280cec10 msgr2=0x7faa2806c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 shutdown_connections 2024-01-16T15:14:53.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.605+0000 7faa27200640 1 -- 172.21.15.43:0/2594662137 wait complete. 2024-01-16T15:14:53.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:14:53.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.849+0000 7f00cbe00640 1 Processor -- start 2024-01-16T15:14:53.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.849+0000 7f00cbe00640 1 -- start start 2024-01-16T15:14:53.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.850+0000 7f00cbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f00cc0d14a0 0x7f00cc0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.850+0000 7f00cbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 0x7f00cc0b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.850+0000 7f00cbe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc0cf0c0 con 0x7f00cc15d5b0 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.850+0000 7f00cbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc0b01b0 con 0x7f00cc0d14a0 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.850+0000 7f00cbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc15d990 con 0x7f00cc0b0640 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00caa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f00cc15d5b0 0x7f00cc0d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53016/0 (socket says 172.21.15.43:53016) 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00caa00640 1 -- 172.21.15.43:0/3271993350 learned_addr learned my addr 172.21.15.43:0/3271993350 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:53.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00ca000640 1 --2- 172.21.15.43:0/3271993350 >> v2:172.21.15.43:3300/0 conn(0x7f00cc0d14a0 0x7f00cc0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00cb400640 1 --2- 172.21.15.43:0/3271993350 >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 0x7f00cc0b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00ca000640 1 -- 172.21.15.43:0/3271993350 >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 msgr2=0x7f00cc0b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00ca000640 1 --2- 172.21.15.43:0/3271993350 >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 0x7f00cc0b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00ca000640 1 -- 172.21.15.43:0/3271993350 >> v1:172.21.15.43:6789/0 conn(0x7f00cc15d5b0 legacy=0x7f00cc0d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00ca000640 1 -- 172.21.15.43:0/3271993350 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc0d1e40 con 0x7f00cc0d14a0 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00c9600640 1 -- 172.21.15.43:0/3271993350 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817581146 0 0) 0x7f00cc0cf0c0 con 0x7f00cc15d5b0 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.851+0000 7f00cb400640 1 --2- 172.21.15.43:0/3271993350 >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 0x7f00cc0b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:53.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.852+0000 7f00ca000640 1 --2- 172.21.15.43:0/3271993350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc0d1900 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0cf3e0 tx=0x7f00b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda2919d2184d1cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.853+0000 7f00c9600640 1 -- 172.21.15.43:0/3271993350 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4031830 con 0x7f00cc0d14a0 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.853+0000 7f00c9600640 1 -- 172.21.15.43:0/3271993350 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b4031990 con 0x7f00cc0d14a0 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.853+0000 7f00c9600640 1 -- 172.21.15.43:0/3271993350 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4031d20 con 0x7f00cc0d14a0 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.853+0000 7f00cbe00640 1 -- 172.21.15.43:0/3271993350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 msgr2=0x7f00cc0d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.853+0000 7f00cbe00640 1 --2- 172.21.15.43:0/3271993350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc0d1900 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0cf3e0 tx=0x7f00b402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 -- 172.21.15.43:0/3271993350 shutdown_connections 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 --2- 172.21.15.43:0/3271993350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc0d1900 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 --2- 172.21.15.43:0/3271993350 >> v2:172.21.15.69:3300/0 conn(0x7f00cc0b0640 0x7f00cc0b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 -- 172.21.15.43:0/3271993350 >> 172.21.15.43:0/3271993350 conn(0x7f00cc09c3b0 msgr2=0x7f00cc0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 -- 172.21.15.43:0/3271993350 shutdown_connections 2024-01-16T15:14:53.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.854+0000 7f00cbe00640 1 -- 172.21.15.43:0/3271993350 wait complete. 2024-01-16T15:14:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.855+0000 7f00cbe00640 1 Processor -- start 2024-01-16T15:14:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.856+0000 7f00cbe00640 1 -- start start 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00cc0b0640 0x7f00cc152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00cc15d5b0 0x7f00cc1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0b86d0 con 0x7f00cc0d14a0 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0b8b80 con 0x7f00cc0b0640 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00cbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0b8e60 con 0x7f00cc15d5b0 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53980/0 (socket says 172.21.15.43:53980) 2024-01-16T15:14:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 -- 172.21.15.43:0/814953429 learned_addr learned my addr 172.21.15.43:0/814953429 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 -- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00cc15d5b0 msgr2=0x7f00cc1f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00cc15d5b0 0x7f00cc1f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00caa00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00cc0b0640 0x7f00cc152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 -- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00cc0b0640 msgr2=0x7f00cc152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00cc0b0640 0x7f00cc152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.857+0000 7f00ca000640 1 -- 172.21.15.43:0/814953429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4009400 con 0x7f00cc0d14a0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00ca000640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b403aa60 con 0x7f00cc0d14a0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b403abe0 con 0x7f00cc0d14a0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40443f0 con 0x7f00cc0d14a0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc15e610 con 0x7f00cc0d14a0 2024-01-16T15:14:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.858+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc15ebd0 con 0x7f00cc0d14a0 2024-01-16T15:14:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.860+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc04e840 con 0x7f00cc0d14a0 2024-01-16T15:14:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.860+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f00b404d450 con 0x7f00cc0d14a0 2024-01-16T15:14:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.861+0000 7f00c2a00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 0x7f00a406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.861+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f00b40aab20 con 0x7f00cc0d14a0 2024-01-16T15:14:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.862+0000 7f00caa00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 0x7f00a406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:53.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.862+0000 7f00caa00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 0x7f00a406e590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000f60 tx=0x7f00bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:53.866+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00b4077180 con 0x7f00cc0d14a0 2024-01-16T15:14:54.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.062+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f00cc0678d0 con 0x7f00cc0d14a0 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.063+0000 7f00c2a00640 1 -- 172.21.15.43:0/814953429 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) v1 ==== 84+0+65 (secure 0 0 0) 0x7f00b4035020 con 0x7f00cc0d14a0 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:14:54.046 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:14:54.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 msgr2=0x7f00a406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 0x7f00a406e590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000f60 tx=0x7f00bc009040 comp rx=0 tx=0).stop 2024-01-16T15:14:54.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 msgr2=0x7f00cc15f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b40046c0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 shutdown_connections 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f00a406c0d0 0x7f00a406e590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00cc15d5b0 0x7f00cc1f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00cc0d14a0 0x7f00cc15f0d0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 --2- 172.21.15.43:0/814953429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00cc0b0640 0x7f00cc152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 >> 172.21.15.43:0/814953429 conn(0x7f00cc09c3b0 msgr2=0x7f00cc0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 shutdown_connections 2024-01-16T15:14:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.066+0000 7f00cbe00640 1 -- 172.21.15.43:0/814953429 wait complete. 2024-01-16T15:14:54.069 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:14:54.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:14:54.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.315+0000 7f64fa800640 1 Processor -- start 2024-01-16T15:14:54.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.316+0000 7f64fa800640 1 -- start start 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.317+0000 7f64fa800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.317+0000 7f64fa800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64f40b33d0 0x7f64f40b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.317+0000 7f64fa800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40b2980 con 0x7f64f40d1fd0 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.317+0000 7f64fa800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f40cf030 con 0x7f64f40b33d0 2024-01-16T15:14:54.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.317+0000 7f64fa800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40d0350 con 0x7f64f40cfa30 2024-01-16T15:14:54.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.318+0000 7f64f9400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.318+0000 7f64f9400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34516/0 (socket says 172.21.15.43:34516) 2024-01-16T15:14:54.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.318+0000 7f64f9400640 1 -- 172.21.15.43:0/1183575258 learned_addr learned my addr 172.21.15.43:0/1183575258 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:54.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.318+0000 7f64f9e00640 1 --2- 172.21.15.43:0/1183575258 >> v2:172.21.15.69:3300/0 conn(0x7f64f40b33d0 0x7f64f40b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.319+0000 7f64f9400640 1 -- 172.21.15.43:0/1183575258 >> v2:172.21.15.69:3300/0 conn(0x7f64f40b33d0 msgr2=0x7f64f40b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:54.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.319+0000 7f64f9400640 1 --2- 172.21.15.43:0/1183575258 >> v2:172.21.15.69:3300/0 conn(0x7f64f40b33d0 0x7f64f40b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.319+0000 7f64f9400640 1 -- 172.21.15.43:0/1183575258 >> v1:172.21.15.69:6789/0 conn(0x7f64f40d1fd0 legacy=0x7f64f40ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.319+0000 7f64f9400640 1 -- 172.21.15.43:0/1183575258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40cf220 con 0x7f64f40cfa30 2024-01-16T15:14:54.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.320+0000 7f64f9400640 1 --2- 172.21.15.43:0/1183575258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009870 tx=0x7f64e402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db7b59d290a21ff8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.321+0000 7f64f3e00640 1 -- 172.21.15.43:0/1183575258 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4030840 con 0x7f64f40cfa30 2024-01-16T15:14:54.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.321+0000 7f64f3e00640 1 -- 172.21.15.43:0/1183575258 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e40309a0 con 0x7f64f40cfa30 2024-01-16T15:14:54.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.321+0000 7f64f3e00640 1 -- 172.21.15.43:0/1183575258 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4030d30 con 0x7f64f40cfa30 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 -- 172.21.15.43:0/1183575258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 msgr2=0x7f64f40cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 --2- 172.21.15.43:0/1183575258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009870 tx=0x7f64e402b980 comp rx=0 tx=0).stop 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 -- 172.21.15.43:0/1183575258 shutdown_connections 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 --2- 172.21.15.43:0/1183575258 >> v2:172.21.15.69:3300/0 conn(0x7f64f40b33d0 0x7f64f40b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 --2- 172.21.15.43:0/1183575258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40cfa30 0x7f64f40cfe10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 -- 172.21.15.43:0/1183575258 >> 172.21.15.43:0/1183575258 conn(0x7f64f40d3020 msgr2=0x7f64f40d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 -- 172.21.15.43:0/1183575258 shutdown_connections 2024-01-16T15:14:54.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.322+0000 7f64fa800640 1 -- 172.21.15.43:0/1183575258 wait complete. 2024-01-16T15:14:54.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.324+0000 7f64fa800640 1 Processor -- start 2024-01-16T15:14:54.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.324+0000 7f64fa800640 1 -- start start 2024-01-16T15:14:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f40b33d0 0x7f64f40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40d1fd0 0x7f64f415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f415eea0 con 0x7f64f40cfa30 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40caff0 con 0x7f64f40b33d0 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.325+0000 7f64fa800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40cbbd0 con 0x7f64f40d1fd0 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53986/0 (socket says 172.21.15.43:53986) 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 -- 172.21.15.43:0/1351900328 learned_addr learned my addr 172.21.15.43:0/1351900328 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f9400640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f40b33d0 0x7f64f40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f9e00640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40d1fd0 0x7f64f415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 -- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40d1fd0 msgr2=0x7f64f415b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40d1fd0 0x7f64f415b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 -- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f40b33d0 msgr2=0x7f64f40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f40b33d0 0x7f64f40caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.326+0000 7f64f8a00640 1 -- 172.21.15.43:0/1351900328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40094d0 con 0x7f64f40cfa30 2024-01-16T15:14:54.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64f8a00640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c400 tx=0x7f64dc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0052d0 con 0x7f64f40cfa30 2024-01-16T15:14:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc005450 con 0x7f64f40cfa30 2024-01-16T15:14:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc007260 con 0x7f64f40cfa30 2024-01-16T15:14:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f40cbeb0 con 0x7f64f40cfa30 2024-01-16T15:14:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.327+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f40cc470 con 0x7f64f40cfa30 2024-01-16T15:14:54.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.328+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f40d0a40 con 0x7f64f40cfa30 2024-01-16T15:14:54.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.329+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f64dc0073c0 con 0x7f64f40cfa30 2024-01-16T15:14:54.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.330+0000 7f64f1600640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 0x7f64cc06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:54.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.330+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f64dc087d40 con 0x7f64f40cfa30 2024-01-16T15:14:54.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.331+0000 7f64f9400640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 0x7f64cc06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:54.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.331+0000 7f64f9400640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 0x7f64cc06e5c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f64e4036000 tx=0x7f64e403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:54.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.334+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64dc054340 con 0x7f64f40cfa30 2024-01-16T15:14:54.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:54.528+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f64f404e840 con 0x7f64f40cfa30 2024-01-16T15:14:55.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.209+0000 7f64f1600640 1 -- 172.21.15.43:0/1351900328 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v98) v1 ==== 138+0+0 (secure 0 0 0) 0x7f64dc057960 con 0x7f64f40cfa30 2024-01-16T15:14:55.192 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:14:55.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.214+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 msgr2=0x7f64cc06e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.214+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 0x7f64cc06e5c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f64e4036000 tx=0x7f64e403b040 comp rx=0 tx=0).stop 2024-01-16T15:14:55.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.214+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 msgr2=0x7f64f4154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.214+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c400 tx=0x7f64dc00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 shutdown_connections 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64cc06c100 0x7f64cc06e5c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64f40d1fd0 0x7f64f415b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64f40cfa30 0x7f64f4154520 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 --2- 172.21.15.43:0/1351900328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f40b33d0 0x7f64f40caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 >> 172.21.15.43:0/1351900328 conn(0x7f64f40d3020 msgr2=0x7f64f40ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 shutdown_connections 2024-01-16T15:14:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.215+0000 7f64fa800640 1 -- 172.21.15.43:0/1351900328 wait complete. 2024-01-16T15:14:55.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:14:55.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.455+0000 7fe12a600640 1 Processor -- start 2024-01-16T15:14:55.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.455+0000 7fe12a600640 1 -- start start 2024-01-16T15:14:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe12a600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe124144770 0x7fe12415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe12a600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12415f930 con 0x7fe12415f0a0 2024-01-16T15:14:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe12a600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1240b3b10 con 0x7fe1240b35a0 2024-01-16T15:14:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe12a600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1240b43b0 con 0x7fe124144770 2024-01-16T15:14:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe129200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe124144770 0x7fe12415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe129200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe124144770 0x7fe12415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53992/0 (socket says 172.21.15.43:53992) 2024-01-16T15:14:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.457+0000 7fe129200640 1 -- 172.21.15.43:0/3992448079 learned_addr learned my addr 172.21.15.43:0/3992448079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.458+0000 7fe129200640 1 -- 172.21.15.43:0/3992448079 >> v1:172.21.15.69:6789/0 conn(0x7fe1240b35a0 legacy=0x7fe12406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.458+0000 7fe129200640 1 -- 172.21.15.43:0/3992448079 >> v1:172.21.15.43:6789/0 conn(0x7fe12415f0a0 legacy=0x7fe1240ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.458+0000 7fe129200640 1 -- 172.21.15.43:0/3992448079 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1240b3df0 con 0x7fe124144770 2024-01-16T15:14:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.459+0000 7fe129200640 1 --2- 172.21.15.43:0/3992448079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe12415eb60 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29b576bfea743cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.460+0000 7fe123400640 1 -- 172.21.15.43:0/3992448079 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11402f850 con 0x7fe124144770 2024-01-16T15:14:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.460+0000 7fe123400640 1 -- 172.21.15.43:0/3992448079 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11402f9b0 con 0x7fe124144770 2024-01-16T15:14:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.460+0000 7fe123400640 1 -- 172.21.15.43:0/3992448079 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11402fd40 con 0x7fe124144770 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.460+0000 7fe12a600640 1 -- 172.21.15.43:0/3992448079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 msgr2=0x7fe12415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.460+0000 7fe12a600640 1 --2- 172.21.15.43:0/3992448079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe12415eb60 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402ba30 comp rx=0 tx=0).stop 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.461+0000 7fe12a600640 1 -- 172.21.15.43:0/3992448079 shutdown_connections 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.461+0000 7fe12a600640 1 --2- 172.21.15.43:0/3992448079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe12415eb60 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.461+0000 7fe12a600640 1 -- 172.21.15.43:0/3992448079 >> 172.21.15.43:0/3992448079 conn(0x7fe1240cec10 msgr2=0x7fe1240cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.461+0000 7fe12a600640 1 -- 172.21.15.43:0/3992448079 shutdown_connections 2024-01-16T15:14:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.461+0000 7fe12a600640 1 -- 172.21.15.43:0/3992448079 wait complete. 2024-01-16T15:14:55.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.463+0000 7fe12a600640 1 Processor -- start 2024-01-16T15:14:55.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.463+0000 7fe12a600640 1 -- start start 2024-01-16T15:14:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 0x7fe12413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 0x7fe124141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12413b0d0 con 0x7fe124144770 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe124142e60 con 0x7fe12415f0a0 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe12a600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1241429d0 con 0x7fe1240b35a0 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe123e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe123e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53994/0 (socket says 172.21.15.43:53994) 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe123e00640 1 -- 172.21.15.43:0/2787893429 learned_addr learned my addr 172.21.15.43:0/2787893429 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.464+0000 7fe129200640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 0x7fe12413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe129c00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 0x7fe124141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 -- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 msgr2=0x7fe12413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 0x7fe12413f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 -- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 msgr2=0x7fe124141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 0x7fe124141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 -- 172.21.15.43:0/2787893429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1140094d0 con 0x7fe124144770 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe129200640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 0x7fe12413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe129c00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 0x7fe124141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe123e00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c480 tx=0x7fe10c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0052d0 con 0x7fe124144770 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c005430 con 0x7fe124144770 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c012870 con 0x7fe124144770 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.465+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe124141dc0 con 0x7fe124144770 2024-01-16T15:14:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.466+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe124142380 con 0x7fe124144770 2024-01-16T15:14:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.467+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe124142b80 con 0x7fe124144770 2024-01-16T15:14:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.468+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fe10c0129d0 con 0x7fe124144770 2024-01-16T15:14:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.469+0000 7fe120c00640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 0x7fe0fc06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.469+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fe10c0884d0 con 0x7fe124144770 2024-01-16T15:14:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.469+0000 7fe129200640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 0x7fe0fc06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.470+0000 7fe129200640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 0x7fe0fc06e5c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe114036000 tx=0x7fe114039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.474+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe10c0bde90 con 0x7fe124144770 2024-01-16T15:14:55.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:55.670+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe12404e840 con 0x7fe124144770 2024-01-16T15:14:56.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.216+0000 7fe120c00640 1 -- 172.21.15.43:0/2787893429 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v99) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe124142b80 con 0x7fe124144770 2024-01-16T15:14:56.199 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.219+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 msgr2=0x7fe0fc06e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.219+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 0x7fe0fc06e5c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe114036000 tx=0x7fe114039040 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.219+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 msgr2=0x7fe124141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.219+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c480 tx=0x7fe10c00c940 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 shutdown_connections 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0fc06c100 0x7fe0fc06e5c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe12415f0a0 0x7fe124141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe124144770 0x7fe124141200 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 --2- 172.21.15.43:0/2787893429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240b35a0 0x7fe12413f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 >> 172.21.15.43:0/2787893429 conn(0x7fe1240cec10 msgr2=0x7fe12415bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 shutdown_connections 2024-01-16T15:14:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.220+0000 7fe12a600640 1 -- 172.21.15.43:0/2787893429 wait complete. 2024-01-16T15:14:56.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:14:56.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.440+0000 7facf3400640 1 Processor -- start 2024-01-16T15:14:56.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.441+0000 7facf3400640 1 -- start start 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 0x7facec0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facec0adb60 0x7facec0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf3400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec15b730 con 0x7facec15b350 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf3400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0ac740 con 0x7facec0adb60 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0ac460 con 0x7facec09cb30 2024-01-16T15:14:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf1400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facec0adb60 0x7facec0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf1400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7facec0adb60 0x7facec0adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54006/0 (socket says 172.21.15.43:54006) 2024-01-16T15:14:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf1400640 1 -- 172.21.15.43:0/907271136 learned_addr learned my addr 172.21.15.43:0/907271136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.443+0000 7facf0a00640 1 --1- 172.21.15.43:0/907271136 >> v1:172.21.15.43:6789/0 conn(0x7facec15b350 0x7facec0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53032/0 (socket says 172.21.15.43:53032) 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7facf1400640 1 -- 172.21.15.43:0/907271136 >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 msgr2=0x7facec0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7face3e00640 1 --2- 172.21.15.43:0/907271136 >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 0x7facec0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7facf1400640 1 --2- 172.21.15.43:0/907271136 >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 0x7facec0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7facf1400640 1 -- 172.21.15.43:0/907271136 >> v1:172.21.15.43:6789/0 conn(0x7facec15b350 legacy=0x7facec0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7facf1400640 1 -- 172.21.15.43:0/907271136 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0b2a50 con 0x7facec0adb60 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7facebe00640 1 -- 172.21.15.43:0/907271136 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307942919 0 0) 0x7facdc001110 con 0x7facec15b350 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.444+0000 7face3e00640 1 --2- 172.21.15.43:0/907271136 >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 0x7facec0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.445+0000 7facf1400640 1 --2- 172.21.15.43:0/907271136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec0adb60 0x7facec0adf70 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7facec0d0060 tx=0x7face402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2352f4f5d801e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.445+0000 7facebe00640 1 -- 172.21.15.43:0/907271136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face402cb20 con 0x7facec0adb60 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.445+0000 7facebe00640 1 -- 172.21.15.43:0/907271136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face402cc80 con 0x7facec0adb60 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.445+0000 7facebe00640 1 -- 172.21.15.43:0/907271136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face40343f0 con 0x7facec0adb60 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 -- 172.21.15.43:0/907271136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec0adb60 msgr2=0x7facec0adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 --2- 172.21.15.43:0/907271136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec0adb60 0x7facec0adf70 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7facec0d0060 tx=0x7face402bf80 comp rx=0 tx=0).stop 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 -- 172.21.15.43:0/907271136 shutdown_connections 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 --2- 172.21.15.43:0/907271136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec0adb60 0x7facec0adf70 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 --2- 172.21.15.43:0/907271136 >> v2:172.21.15.69:3300/0 conn(0x7facec09cb30 0x7facec0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 -- 172.21.15.43:0/907271136 >> 172.21.15.43:0/907271136 conn(0x7facec0b0ba0 msgr2=0x7facec0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 -- 172.21.15.43:0/907271136 shutdown_connections 2024-01-16T15:14:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.446+0000 7facf3400640 1 -- 172.21.15.43:0/907271136 wait complete. 2024-01-16T15:14:56.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.448+0000 7facf3400640 1 Processor -- start 2024-01-16T15:14:56.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.448+0000 7facf3400640 1 -- start start 2024-01-16T15:14:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 0x7facec13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facec0adb60 0x7facec154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec13f1f0 con 0x7facec15b350 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec153f90 con 0x7facec09cb30 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf3400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec152ef0 con 0x7facec0adb60 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54016/0 (socket says 172.21.15.43:54016) 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.449+0000 7facf1400640 1 -- 172.21.15.43:0/2604177445 learned_addr learned my addr 172.21.15.43:0/2604177445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7face3e00640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facec0adb60 0x7facec154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf0a00640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 0x7facec13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 -- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facec0adb60 msgr2=0x7facec154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facec0adb60 0x7facec154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 -- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 msgr2=0x7facec13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 0x7facec13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 -- 172.21.15.43:0/2604177445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4009b30 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf0a00640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 0x7facec13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf1400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7face4036040 tx=0x7face4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face402c920 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face402ca80 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face403e690 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.450+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec1531d0 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.451+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec153790 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.452+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec09d200 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.453+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7face403e7f0 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.454+0000 7face9600640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 0x7facc406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.454+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5637+0+0 (secure 0 0 0) 0x7face40ab9d0 con 0x7facec15b350 2024-01-16T15:14:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.454+0000 7facf0a00640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 0x7facc406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.455+0000 7facf0a00640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 0x7facc406e590 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7facdc000f60 tx=0x7facdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.458+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7face4077fd0 con 0x7facec15b350 2024-01-16T15:14:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:56.688+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7facec04e920 con 0x7facec15b350 2024-01-16T15:14:57.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.240+0000 7face9600640 1 -- 172.21.15.43:0/2604177445 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7face40430c0 con 0x7facec15b350 2024-01-16T15:14:57.223 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 14 and data pool 15 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 msgr2=0x7facc406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 0x7facc406e590 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7facdc000f60 tx=0x7facdc009040 comp rx=0 tx=0).stop 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 msgr2=0x7facec154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7face4036040 tx=0x7face4004a40 comp rx=0 tx=0).stop 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 shutdown_connections 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facc406c0d0 0x7facc406e590 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facec15b350 0x7facec154f90 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facec0adb60 0x7facec154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 --2- 172.21.15.43:0/2604177445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facec09cb30 0x7facec13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.244+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 >> 172.21.15.43:0/2604177445 conn(0x7facec0b0ba0 msgr2=0x7facec0af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:57.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.245+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 shutdown_connections 2024-01-16T15:14:57.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.245+0000 7facf3400640 1 -- 172.21.15.43:0/2604177445 wait complete. 2024-01-16T15:14:57.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:57.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.469+0000 7f0e42200640 1 Processor -- start 2024-01-16T15:14:57.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.469+0000 7f0e42200640 1 -- start start 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e42200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e3c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e42200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c071cd0 con 0x7f0e3c0b3140 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e42200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c0b2120 con 0x7f0e3c06dc30 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e42200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0b2400 con 0x7f0e3c160970 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e3aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0e3c0b3140 0x7f0e3c06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53038/0 (socket says 172.21.15.43:53038) 2024-01-16T15:14:57.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.471+0000 7f0e3aa00640 1 -- 172.21.15.43:0/2686151530 learned_addr learned my addr 172.21.15.43:0/2686151530 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3b400640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e3c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145552609 0 0) 0x7f0e3c071cd0 con 0x7f0e3c0b3140 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e1c003520 con 0x7f0e3c0b3140 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833091823 0 0) 0x7f0e1c003520 con 0x7f0e3c0b3140 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 msgr2=0x7f0e3c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e3c0b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 >> v1:172.21.15.69:6789/0 conn(0x7f0e3c06dc30 legacy=0x7f0e3c0acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0b26e0 con 0x7f0e3c0b3140 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0e24001670 con 0x7f0e3c0b3140 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 >> v1:172.21.15.43:6789/0 conn(0x7f0e3c0b3140 legacy=0x7f0e3c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c0039a0 0x7f0e1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0b2400 con 0x7f0e1c0047c0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c00bab0 con 0x7f0e1c0039a0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.472+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c00bc70 con 0x7f0e3c160970 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3b400640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c0039a0 0x7f0e1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3be00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 -- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 msgr2=0x7f0e1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 -- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c0039a0 msgr2=0x7f0e1c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c0039a0 0x7f0e1c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3aa00640 1 -- 172.21.15.43:0/2686151530 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c003520 con 0x7f0e1c0047c0 2024-01-16T15:14:57.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.473+0000 7f0e3be00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c160970 0x7f0e1c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.474+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e1c004c40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0e24007fd0 tx=0x7f0e1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc4c90ce1c73ac7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24005ad0 con 0x7f0e1c0047c0 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e24005c30 con 0x7f0e1c0047c0 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e3a000640 1 -- 172.21.15.43:0/2686151530 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2402c870 con 0x7f0e1c0047c0 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 -- 172.21.15.43:0/2686151530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 msgr2=0x7f0e1c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 --2- 172.21.15.43:0/2686151530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e1c004c40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0e24007fd0 tx=0x7f0e1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e3be00640 1 -- 172.21.15.43:0/2686151530 reap_dead start 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 -- 172.21.15.43:0/2686151530 shutdown_connections 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 -- 172.21.15.43:0/2686151530 >> 172.21.15.43:0/2686151530 conn(0x7f0e3c0b0d80 msgr2=0x7f0e3c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 -- 172.21.15.43:0/2686151530 shutdown_connections 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.475+0000 7f0e42200640 1 -- 172.21.15.43:0/2686151530 wait complete. 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.477+0000 7f0e42200640 1 Processor -- start 2024-01-16T15:14:57.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.477+0000 7f0e42200640 1 -- start start 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c1573e0 0x7f0e3c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3c099e10 0x7f0e3c09a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c15d940 con 0x7f0e1c0047c0 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c157860 con 0x7f0e3c099e10 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e42200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c154940 con 0x7f0e3c1573e0 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.478+0000 7f0e3b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54032/0 (socket says 172.21.15.43:54032) 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 -- 172.21.15.43:0/2149833444 learned_addr learned my addr 172.21.15.43:0/2149833444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:57.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c1573e0 0x7f0e3c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3be00640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3c099e10 0x7f0e3c09a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 -- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c1573e0 msgr2=0x7f0e3c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c1573e0 0x7f0e3c15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 -- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3c099e10 msgr2=0x7f0e3c09a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3c099e10 0x7f0e3c09a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 -- 172.21.15.43:0/2149833444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c156340 con 0x7f0e1c0047c0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.479+0000 7f0e3b400640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c011600 tx=0x7f0e2c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.480+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00f920 con 0x7f0e1c0047c0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.480+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c00fa80 con 0x7f0e1c0047c0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.480+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00fdf0 con 0x7f0e1c0047c0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.480+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c156620 con 0x7f0e1c0047c0 2024-01-16T15:14:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.480+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c156be0 con 0x7f0e1c0047c0 2024-01-16T15:14:57.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.481+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c074770 con 0x7f0e1c0047c0 2024-01-16T15:14:57.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.482+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f0e2c015460 con 0x7f0e1c0047c0 2024-01-16T15:14:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.483+0000 7f0e33400640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 0x7f0e1406e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.483+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0e2c0891c0 con 0x7f0e1c0047c0 2024-01-16T15:14:57.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.483+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 0x7f0e1406e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.484+0000 7f0e3aa00640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 0x7f0e1406e610 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0e24034000 tx=0x7f0e24009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.487+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e2c08de60 con 0x7f0e1c0047c0 2024-01-16T15:14:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.680+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e3c06e290 con 0x7f0e1c0047c0 2024-01-16T15:14:57.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.682+0000 7f0e33400640 1 -- 172.21.15.43:0/2149833444 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f0e2c0557c0 con 0x7f0e1c0047c0 2024-01-16T15:14:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":100,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:56.689941+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:14:54.585377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:14:55.729389+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 msgr2=0x7f0e1406e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 0x7f0e1406e610 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0e24034000 tx=0x7f0e24009030 comp rx=0 tx=0).stop 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 msgr2=0x7f0e3c155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c011600 tx=0x7f0e2c011ac0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 shutdown_connections 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e1406c150 0x7f0e1406e610 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e3c099e10 0x7f0e3c09a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e3c1573e0 0x7f0e3c15d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 --2- 172.21.15.43:0/2149833444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e1c0047c0 0x7f0e3c155c80 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.685+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 >> 172.21.15.43:0/2149833444 conn(0x7f0e3c0b0d80 msgr2=0x7f0e3c0aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:57.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.686+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 shutdown_connections 2024-01-16T15:14:57.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.686+0000 7f0e42200640 1 -- 172.21.15.43:0/2149833444 wait complete. 2024-01-16T15:14:57.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:14:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.928+0000 7fc15f200640 1 Processor -- start 2024-01-16T15:14:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.928+0000 7fc15f200640 1 -- start start 2024-01-16T15:14:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc15f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc16014b1c0 0x7fc160162670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc15f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc160162bb0 con 0x7fc1601651a0 2024-01-16T15:14:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc15f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc160165ca0 con 0x7fc1600cf760 2024-01-16T15:14:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc15f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc160165f80 con 0x7fc16014b1c0 2024-01-16T15:14:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc155200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc1601651a0 0x7fc1600cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53050/0 (socket says 172.21.15.43:53050) 2024-01-16T15:14:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc15de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc16014b1c0 0x7fc160162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.930+0000 7fc155200640 1 -- 172.21.15.43:0/364964055 learned_addr learned my addr 172.21.15.43:0/364964055 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.931+0000 7fc15d400640 1 -- 172.21.15.43:0/364964055 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792650396 0 0) 0x7fc160162bb0 con 0x7fc1601651a0 2024-01-16T15:14:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.931+0000 7fc15d400640 1 -- 172.21.15.43:0/364964055 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc140003520 con 0x7fc1601651a0 2024-01-16T15:14:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.931+0000 7fc15de00640 1 -- 172.21.15.43:0/364964055 >> v1:172.21.15.69:6789/0 conn(0x7fc1600cf760 legacy=0x7fc16006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.931+0000 7fc15de00640 1 -- 172.21.15.43:0/364964055 >> v1:172.21.15.43:6789/0 conn(0x7fc1601651a0 legacy=0x7fc1600cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.931+0000 7fc15de00640 1 -- 172.21.15.43:0/364964055 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160161e70 con 0x7fc16014b1c0 2024-01-16T15:14:57.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.932+0000 7fc15de00640 1 --2- 172.21.15.43:0/364964055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16014b1c0 0x7fc160162670 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc1500097a0 tx=0x7fc15002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8818000fa2c31e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15d400640 1 -- 172.21.15.43:0/364964055 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150031830 con 0x7fc16014b1c0 2024-01-16T15:14:57.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15d400640 1 -- 172.21.15.43:0/364964055 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc150031990 con 0x7fc16014b1c0 2024-01-16T15:14:57.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15d400640 1 -- 172.21.15.43:0/364964055 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150031d20 con 0x7fc16014b1c0 2024-01-16T15:14:57.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 -- 172.21.15.43:0/364964055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16014b1c0 msgr2=0x7fc160162670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 --2- 172.21.15.43:0/364964055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16014b1c0 0x7fc160162670 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc1500097a0 tx=0x7fc15002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 -- 172.21.15.43:0/364964055 shutdown_connections 2024-01-16T15:14:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 --2- 172.21.15.43:0/364964055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc16014b1c0 0x7fc160162670 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 -- 172.21.15.43:0/364964055 >> 172.21.15.43:0/364964055 conn(0x7fc16006a860 msgr2=0x7fc16006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 -- 172.21.15.43:0/364964055 shutdown_connections 2024-01-16T15:14:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.933+0000 7fc15f200640 1 -- 172.21.15.43:0/364964055 wait complete. 2024-01-16T15:14:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.935+0000 7fc15f200640 1 Processor -- start 2024-01-16T15:14:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.935+0000 7fc15f200640 1 -- start start 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1600cf760 0x7fc1600cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 0x7fc160160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc160149070 con 0x7fc1601651a0 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16013fe30 con 0x7fc1600cf760 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1600cdf90 con 0x7fc16014b1c0 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.936+0000 7fc15e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54060/0 (socket says 172.21.15.43:54060) 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 -- 172.21.15.43:0/3987168859 learned_addr learned my addr 172.21.15.43:0/3987168859 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc155200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 0x7fc160160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15de00640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1600cf760 0x7fc1600cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 -- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 msgr2=0x7fc160160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 0x7fc160160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 -- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1600cf760 msgr2=0x7fc1600cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1600cf760 0x7fc1600cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 -- 172.21.15.43:0/3987168859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150009400 con 0x7fc1601651a0 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc155200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 0x7fc160160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:57.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc15e800640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c7d0 tx=0x7fc15800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.937+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158003b70 con 0x7fc1601651a0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.938+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158003cf0 con 0x7fc1601651a0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.938+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800aa90 con 0x7fc1601651a0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.938+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc160149490 con 0x7fc1601651a0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.938+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1601489b0 con 0x7fc1601651a0 2024-01-16T15:14:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.939+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1601491b0 con 0x7fc1601651a0 2024-01-16T15:14:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.940+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fc15800abf0 con 0x7fc1601651a0 2024-01-16T15:14:57.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.941+0000 7fc156a00640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 0x7fc13806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:57.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.941+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc158088810 con 0x7fc1601651a0 2024-01-16T15:14:57.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.941+0000 7fc15de00640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 0x7fc13806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:57.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.942+0000 7fc15de00640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 0x7fc13806e590 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc150036000 tx=0x7fc150008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:57.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:57.945+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1601491b0 con 0x7fc1601651a0 2024-01-16T15:14:58.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.141+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1601491b0 con 0x7fc1601651a0 2024-01-16T15:14:58.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.143+0000 7fc156a00640 1 -- 172.21.15.43:0/3987168859 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16519 (secure 0 0 0) 0x7fc1601491b0 con 0x7fc1601651a0 2024-01-16T15:14:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:14:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":100,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:56.689941+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:14:54.585377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:14:55.729389+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:14:58.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 msgr2=0x7fc13806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 0x7fc13806e590 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc150036000 tx=0x7fc150008030 comp rx=0 tx=0).stop 2024-01-16T15:14:58.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 msgr2=0x7fc1600cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c7d0 tx=0x7fc15800cc90 comp rx=0 tx=0).stop 2024-01-16T15:14:58.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 shutdown_connections 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc13806c0d0 0x7fc13806e590 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601651a0 0x7fc1600cd760 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc16014b1c0 0x7fc160160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 --2- 172.21.15.43:0/3987168859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1600cf760 0x7fc1600cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 >> 172.21.15.43:0/3987168859 conn(0x7fc16006a860 msgr2=0x7fc1600afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 shutdown_connections 2024-01-16T15:14:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.146+0000 7fc15f200640 1 -- 172.21.15.43:0/3987168859 wait complete. 2024-01-16T15:14:58.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:14:58.165 INFO:tasks.ceph.mds.b.smithi069.stderr:2024-01-16T15:14:58.182+0000 7f45e1800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:14:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.391+0000 7f517a000640 1 Processor -- start 2024-01-16T15:14:58.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.392+0000 7f517a000640 1 -- start start 2024-01-16T15:14:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.393+0000 7f517a000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f51740b48d0 0x7f51740b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.393+0000 7f517a000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740b23e0 con 0x7f5174160bd0 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.393+0000 7f517a000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740d41b0 con 0x7f51740b3210 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.393+0000 7f517a000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51740cdc70 con 0x7f51740b48d0 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5173400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5174160bd0 0x7f51740b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53056/0 (socket says 172.21.15.43:53056) 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5173400640 1 -- 172.21.15.43:0/3586163853 learned_addr learned my addr 172.21.15.43:0/3586163853 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172a00640 1 --2- 172.21.15.43:0/3586163853 >> v2:172.21.15.43:3300/0 conn(0x7f51740b48d0 0x7f51740b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172000640 1 -- 172.21.15.43:0/3586163853 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958066704 0 0) 0x7f51740b23e0 con 0x7f5174160bd0 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172000640 1 -- 172.21.15.43:0/3586163853 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5154003520 con 0x7f5174160bd0 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172a00640 1 -- 172.21.15.43:0/3586163853 >> v1:172.21.15.69:6789/0 conn(0x7f51740b3210 legacy=0x7f5174167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172a00640 1 -- 172.21.15.43:0/3586163853 >> v1:172.21.15.43:6789/0 conn(0x7f5174160bd0 legacy=0x7f51740b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.394+0000 7f5172a00640 1 -- 172.21.15.43:0/3586163853 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174160fb0 con 0x7f51740b48d0 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.395+0000 7f5172a00640 1 --2- 172.21.15.43:0/3586163853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51740b48d0 0x7f51740b1ea0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f51740cddb0 tx=0x7f515c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7f72897476eae9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f5172000640 1 -- 172.21.15.43:0/3586163853 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c02e070 con 0x7f51740b48d0 2024-01-16T15:14:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f5172000640 1 -- 172.21.15.43:0/3586163853 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0315a0 con 0x7f51740b48d0 2024-01-16T15:14:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f5172000640 1 -- 172.21.15.43:0/3586163853 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c031930 con 0x7f51740b48d0 2024-01-16T15:14:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 -- 172.21.15.43:0/3586163853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51740b48d0 msgr2=0x7f51740b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 --2- 172.21.15.43:0/3586163853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51740b48d0 0x7f51740b1ea0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f51740cddb0 tx=0x7f515c02b840 comp rx=0 tx=0).stop 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 -- 172.21.15.43:0/3586163853 shutdown_connections 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 --2- 172.21.15.43:0/3586163853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51740b48d0 0x7f51740b1ea0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 -- 172.21.15.43:0/3586163853 >> 172.21.15.43:0/3586163853 conn(0x7f517409c5a0 msgr2=0x7f51740d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 -- 172.21.15.43:0/3586163853 shutdown_connections 2024-01-16T15:14:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.396+0000 7f517a000640 1 -- 172.21.15.43:0/3586163853 wait complete. 2024-01-16T15:14:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.398+0000 7f517a000640 1 Processor -- start 2024-01-16T15:14:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.398+0000 7f517a000640 1 -- start start 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51740b3210 0x7f517414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 0x7f517414cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517414c7f0 con 0x7f5174160bd0 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517414d670 con 0x7f51740b3210 2024-01-16T15:14:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.399+0000 7f517a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517414d950 con 0x7f51740b48d0 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54074/0 (socket says 172.21.15.43:54074) 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 -- 172.21.15.43:0/3594000938 learned_addr learned my addr 172.21.15.43:0/3594000938 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5172a00640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 0x7f517414cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173400640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51740b3210 0x7f517414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 -- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 msgr2=0x7f517414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 0x7f517414cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 -- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51740b3210 msgr2=0x7f517414c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51740b3210 0x7f517414c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 -- 172.21.15.43:0/3594000938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c009400 con 0x7f5174160bd0 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5172a00640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 0x7f517414cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.400+0000 7f5173e00640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c410 tx=0x7f516c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.401+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c011790 con 0x7f5174160bd0 2024-01-16T15:14:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.401+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c0118f0 con 0x7f5174160bd0 2024-01-16T15:14:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.401+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c011ce0 con 0x7f5174160bd0 2024-01-16T15:14:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.401+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517414dc30 con 0x7f5174160bd0 2024-01-16T15:14:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.401+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517414e1f0 con 0x7f5174160bd0 2024-01-16T15:14:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.402+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51740a1210 con 0x7f5174160bd0 2024-01-16T15:14:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.403+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f516c01c460 con 0x7f5174160bd0 2024-01-16T15:14:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.404+0000 7f516b400640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 0x7f514c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.404+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f516c0886c0 con 0x7f5174160bd0 2024-01-16T15:14:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.404+0000 7f5173400640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 0x7f514c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.405+0000 7f5173400640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 0x7f514c06e590 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5164001560 tx=0x7f5164009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:58.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.409+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f516c054c00 con 0x7f5174160bd0 2024-01-16T15:14:58.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:58.645+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f51740b1eb0 con 0x7f5174160bd0 2024-01-16T15:14:59.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:59.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.284+0000 7f516b400640 1 -- 172.21.15.43:0/3594000938 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f516c058220 con 0x7f5174160bd0 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 msgr2=0x7f514c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 0x7f514c06e590 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5164001560 tx=0x7f5164009040 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 msgr2=0x7f517414d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c410 tx=0x7f516c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 shutdown_connections 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f514c06c0d0 0x7f514c06e590 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5174160bd0 0x7f517414d040 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f51740b48d0 0x7f517414cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 --2- 172.21.15.43:0/3594000938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51740b3210 0x7f517414c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.288+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 >> 172.21.15.43:0/3594000938 conn(0x7f517409c5a0 msgr2=0x7f51740d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.289+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 shutdown_connections 2024-01-16T15:14:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.289+0000 7f517a000640 1 -- 172.21.15.43:0/3594000938 wait complete. 2024-01-16T15:14:59.294 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:14:59.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:14:59.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.535+0000 7fcd9a400640 1 Processor -- start 2024-01-16T15:14:59.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.536+0000 7fcd9a400640 1 -- start start 2024-01-16T15:14:59.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd9a400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcd9415d3b0 0x7fcd940ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd9a400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd940b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd9a400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940c6bf0 con 0x7fcd940d0fa0 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd9a400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940d1430 con 0x7fcd9415d3b0 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd9a400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cedc0 con 0x7fcd940d1d90 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd93e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcd940d0fa0 0x7fcd940d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53066/0 (socket says 172.21.15.43:53066) 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.537+0000 7fcd93e00640 1 -- 172.21.15.43:0/1274762483 learned_addr learned my addr 172.21.15.43:0/1274762483 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd99000640 1 --2- 172.21.15.43:0/1274762483 >> v2:172.21.15.69:3300/0 conn(0x7fcd9415d3b0 0x7fcd940ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd99a00640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd940b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964431876 0 0) 0x7fcd940c6bf0 con 0x7fcd940d0fa0 2024-01-16T15:14:59.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd74003520 con 0x7fcd940d0fa0 2024-01-16T15:14:59.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 223503916 0 0) 0x7fcd74003520 con 0x7fcd940d0fa0 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 msgr2=0x7fcd940b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd940b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 >> v2:172.21.15.69:3300/0 conn(0x7fcd9415d3b0 msgr2=0x7fcd940ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 --2- 172.21.15.43:0/1274762483 >> v2:172.21.15.69:3300/0 conn(0x7fcd9415d3b0 0x7fcd940ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940c6e40 con 0x7fcd940d0fa0 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcd7c0019f0 con 0x7fcd940d0fa0 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 >> v1:172.21.15.43:6789/0 conn(0x7fcd940d0fa0 legacy=0x7fcd940d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd74003fb0 0x7fcd740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd74004d00 0x7fcd74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d1430 con 0x7fcd74003fb0 2024-01-16T15:14:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cedc0 con 0x7fcd74004d00 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.538+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7400ba80 con 0x7fcd940d1d90 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd74003fb0 0x7fcd740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd99000640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 -- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 msgr2=0x7fcd740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 -- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd74004d00 msgr2=0x7fcd74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd74004d00 0x7fcd74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd93e00640 1 -- 172.21.15.43:0/1274762483 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd74003520 con 0x7fcd74003fb0 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.539+0000 7fcd99000640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940d1d90 0x7fcd740039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.540+0000 7fcd93e00640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd74003fb0 0x7fcd740047c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00a790 tx=0x7fcd740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd89fc73062ac01 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.540+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c034630 con 0x7fcd74003fb0 2024-01-16T15:14:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.540+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c02f070 con 0x7fcd74003fb0 2024-01-16T15:14:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.540+0000 7fcd93400640 1 -- 172.21.15.43:0/1274762483 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c034a30 con 0x7fcd74003fb0 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 -- 172.21.15.43:0/1274762483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd74003fb0 msgr2=0x7fcd740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 --2- 172.21.15.43:0/1274762483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd74003fb0 0x7fcd740047c0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00a790 tx=0x7fcd740057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd99a00640 1 -- 172.21.15.43:0/1274762483 reap_dead start 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 -- 172.21.15.43:0/1274762483 shutdown_connections 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 -- 172.21.15.43:0/1274762483 >> 172.21.15.43:0/1274762483 conn(0x7fcd940d30a0 msgr2=0x7fcd940d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 -- 172.21.15.43:0/1274762483 shutdown_connections 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.541+0000 7fcd9a400640 1 -- 172.21.15.43:0/1274762483 wait complete. 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.543+0000 7fcd9a400640 1 Processor -- start 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.543+0000 7fcd9a400640 1 -- start start 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 0x7fcd940b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 0x7fcd9415d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cad40 con 0x7fcd94158ce0 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cbad0 con 0x7fcd941590c0 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd9a400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cb230 con 0x7fcd940ca170 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd99000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd99000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54084/0 (socket says 172.21.15.43:54084) 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.544+0000 7fcd99000640 1 -- 172.21.15.43:0/2433748263 learned_addr learned my addr 172.21.15.43:0/2433748263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd93e00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 0x7fcd940b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99a00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 0x7fcd9415d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 -- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 msgr2=0x7fcd940b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 0x7fcd940b79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 -- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 msgr2=0x7fcd9415d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 0x7fcd9415d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 -- 172.21.15.43:0/2433748263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940cb510 con 0x7fcd94158ce0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd93e00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 0x7fcd940b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99a00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 0x7fcd9415d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.545+0000 7fcd99000640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fcd840109d0 tx=0x7fcd84010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.546+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400ec20 con 0x7fcd94158ce0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.546+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8400ed80 con 0x7fcd94158ce0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.546+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940cb7f0 con 0x7fcd94158ce0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.546+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8401c620 con 0x7fcd94158ce0 2024-01-16T15:14:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.546+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940caee0 con 0x7fcd94158ce0 2024-01-16T15:14:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.547+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404e840 con 0x7fcd94158ce0 2024-01-16T15:14:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.548+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fcd8401c780 con 0x7fcd94158ce0 2024-01-16T15:14:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.549+0000 7fcd90c00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 0x7fcd6c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.549+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcd84088fd0 con 0x7fcd94158ce0 2024-01-16T15:14:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.549+0000 7fcd93e00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 0x7fcd6c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.550+0000 7fcd93e00640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 0x7fcd6c06e590 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c034000 tx=0x7fcd7c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.553+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd84091020 con 0x7fcd94158ce0 2024-01-16T15:14:59.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:14:59.787+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcd9406c9c0 con 0x7fcd94158ce0 2024-01-16T15:15:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.298+0000 7fcd90c00640 1 -- 172.21.15.43:0/2433748263 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd840555d0 con 0x7fcd94158ce0 2024-01-16T15:15:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 msgr2=0x7fcd6c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 0x7fcd6c06e590 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c034000 tx=0x7fcd7c009030 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 msgr2=0x7fcd940b7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fcd840109d0 tx=0x7fcd84010e90 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 shutdown_connections 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd6c06c0d0 0x7fcd6c06e590 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd941590c0 0x7fcd9415d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd940ca170 0x7fcd940b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 --2- 172.21.15.43:0/2433748263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd94158ce0 0x7fcd940b7480 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 >> 172.21.15.43:0/2433748263 conn(0x7fcd940d30a0 msgr2=0x7fcd940c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.302+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 shutdown_connections 2024-01-16T15:15:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.303+0000 7fcd9a400640 1 -- 172.21.15.43:0/2433748263 wait complete. 2024-01-16T15:15:00.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:15:00.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:00.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.541+0000 7fe733400640 1 Processor -- start 2024-01-16T15:15:00.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.541+0000 7fe733400640 1 -- start start 2024-01-16T15:15:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 0x7fe72c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0cff50 0x7fe72c0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe72c0d0630 0x7fe72c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0cbe30 con 0x7fe72c0d0630 2024-01-16T15:15:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c15b530 con 0x7fe72c15b150 2024-01-16T15:15:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.542+0000 7fe733400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0cb620 con 0x7fe72c0cff50 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe72c0d0630 0x7fe72c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe72c0d0630 0x7fe72c15ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54094/0 (socket says 172.21.15.43:54094) 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 -- 172.21.15.43:0/1456264255 learned_addr learned my addr 172.21.15.43:0/1456264255 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe72be00640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0cff50 0x7fe72c0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe730a00640 1 --2- 172.21.15.43:0/1456264255 >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 0x7fe72c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 -- 172.21.15.43:0/1456264255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0cff50 msgr2=0x7fe72c0b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:00.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0cff50 0x7fe72c0b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 -- 172.21.15.43:0/1456264255 >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 msgr2=0x7fe72c0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:00.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 --2- 172.21.15.43:0/1456264255 >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 0x7fe72c0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe731400640 1 -- 172.21.15.43:0/1456264255 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0b3ea0 con 0x7fe72c0d0630 2024-01-16T15:15:00.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.543+0000 7fe730a00640 1 --2- 172.21.15.43:0/1456264255 >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 0x7fe72c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.545+0000 7fe731400640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c0d0630 0x7fe72c15ec60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe72c06e8d0 tx=0x7fe72402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cba3d47bb3304f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.545+0000 7fe72b400640 1 -- 172.21.15.43:0/1456264255 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72402c9f0 con 0x7fe72c0d0630 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.545+0000 7fe72b400640 1 -- 172.21.15.43:0/1456264255 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72402cb50 con 0x7fe72c0d0630 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.545+0000 7fe72b400640 1 -- 172.21.15.43:0/1456264255 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240343f0 con 0x7fe72c0d0630 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 -- 172.21.15.43:0/1456264255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c0d0630 msgr2=0x7fe72c15ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c0d0630 0x7fe72c15ec60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe72c06e8d0 tx=0x7fe72402be50 comp rx=0 tx=0).stop 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 -- 172.21.15.43:0/1456264255 shutdown_connections 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c0d0630 0x7fe72c15ec60 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 --2- 172.21.15.43:0/1456264255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0cff50 0x7fe72c0b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 --2- 172.21.15.43:0/1456264255 >> v2:172.21.15.69:3300/0 conn(0x7fe72c15b150 0x7fe72c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 -- 172.21.15.43:0/1456264255 >> 172.21.15.43:0/1456264255 conn(0x7fe72c070b00 msgr2=0x7fe72c0661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 -- 172.21.15.43:0/1456264255 shutdown_connections 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.546+0000 7fe733400640 1 -- 172.21.15.43:0/1456264255 wait complete. 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.548+0000 7fe733400640 1 Processor -- start 2024-01-16T15:15:00.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.548+0000 7fe733400640 1 -- start start 2024-01-16T15:15:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe72c0cff50 0x7fe72c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0d0630 0x7fe72c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c06d290 con 0x7fe72c15b150 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c158570 con 0x7fe72c0cff50 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe733400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c157fb0 con 0x7fe72c0d0630 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe731400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe731400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54102/0 (socket says 172.21.15.43:54102) 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe731400640 1 -- 172.21.15.43:0/2242685485 learned_addr learned my addr 172.21.15.43:0/2242685485 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe72be00640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0d0630 0x7fe72c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe731400640 1 -- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0d0630 msgr2=0x7fe72c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.549+0000 7fe730a00640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe72c0cff50 0x7fe72c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe731400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0d0630 0x7fe72c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe731400640 1 -- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe72c0cff50 msgr2=0x7fe72c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe731400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe72c0cff50 0x7fe72c156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe731400640 1 -- 172.21.15.43:0/2242685485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009b30 con 0x7fe72c15b150 2024-01-16T15:15:00.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe731400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe724009f40 tx=0x7fe724004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724009510 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72403e450 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c158290 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72403e7e0 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.550+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c15e900 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.552+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c0c6200 con 0x7fe72c15b150 2024-01-16T15:15:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.552+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fe72403e940 con 0x7fe72c15b150 2024-01-16T15:15:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.553+0000 7fe728c00640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 0x7fe70406e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.553+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe7240ab7a0 con 0x7fe72c15b150 2024-01-16T15:15:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.554+0000 7fe730a00640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 0x7fe70406e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.554+0000 7fe730a00640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 0x7fe70406e5c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00b840 tx=0x7fe71c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.558+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe72c0c6200 con 0x7fe72c15b150 2024-01-16T15:15:00.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.790+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe72c0c65e0 con 0x7fe72c15b150 2024-01-16T15:15:00.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.791+0000 7fe728c00640 1 -- 172.21.15.43:0/2242685485 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fe72c0c65e0 con 0x7fe72c15b150 2024-01-16T15:15:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:00.774 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/1137375185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1137375185},{"type":"v1","addr":"172.21.15.43:6836","nonce":1137375185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2958556276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2958556276},{"type":"v1","addr":"172.21.15.43:6837","nonce":2958556276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/208706681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":208706681},{"type":"v1","addr":"172.21.15.69:6835","nonce":208706681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:14:56.690837+0000","modified":"2024-01-16T15:14:59.788652+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8568":{"gid":8568,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6833/3755062795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3755062795},{"type":"v1","addr":"172.21.15.69:6833","nonce":3755062795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:15:00.774 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 66 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.793+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 msgr2=0x7fe70406e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.793+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 0x7fe70406e5c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00b840 tx=0x7fe71c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.793+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 msgr2=0x7fe72c1f3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.793+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe724009f40 tx=0x7fe724004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 shutdown_connections 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe70406c100 0x7fe70406e5c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72c15b150 0x7fe72c1f3360 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe72c0d0630 0x7fe72c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 --2- 172.21.15.43:0/2242685485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe72c0cff50 0x7fe72c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 >> 172.21.15.43:0/2242685485 conn(0x7fe72c070b00 msgr2=0x7fe72c0661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 shutdown_connections 2024-01-16T15:15:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:00.794+0000 7fe733400640 1 -- 172.21.15.43:0/2242685485 wait complete. 2024-01-16T15:15:00.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:01.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.038+0000 7fa51fe00640 1 Processor -- start 2024-01-16T15:15:01.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.039+0000 7fa51fe00640 1 -- start start 2024-01-16T15:15:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.040+0000 7fa51fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa52006d250 0x7fa52015d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.040+0000 7fa51fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa520070720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa51fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa520161670 con 0x7fa520157770 2024-01-16T15:15:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa51fe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52015e610 con 0x7fa52006d250 2024-01-16T15:15:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa51fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa520160d40 con 0x7fa52015ddb0 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa515e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa52006d250 0x7fa52015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa515e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa52006d250 0x7fa52015d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54116/0 (socket says 172.21.15.43:54116) 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa515e00640 1 -- 172.21.15.43:0/1298586059 learned_addr learned my addr 172.21.15.43:0/1298586059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa51ea00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa520157770 0x7fa52006cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53074/0 (socket says 172.21.15.43:53074) 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa51f400640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa520070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.041+0000 7fa515e00640 1 -- 172.21.15.43:0/1298586059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 msgr2=0x7fa520070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.042+0000 7fa515e00640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa520070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.042+0000 7fa515e00640 1 -- 172.21.15.43:0/1298586059 >> v1:172.21.15.43:6789/0 conn(0x7fa520157770 legacy=0x7fa52006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.042+0000 7fa515e00640 1 -- 172.21.15.43:0/1298586059 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa520161020 con 0x7fa52006d250 2024-01-16T15:15:01.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.043+0000 7fa515e00640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52006d250 0x7fa52015d870 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa508009870 tx=0x7fa50802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f70dd3199926 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.043+0000 7fa51e000640 1 -- 172.21.15.43:0/1298586059 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508030840 con 0x7fa52006d250 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.043+0000 7fa51e000640 1 -- 172.21.15.43:0/1298586059 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5080309a0 con 0x7fa52006d250 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.043+0000 7fa51e000640 1 -- 172.21.15.43:0/1298586059 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508030d30 con 0x7fa52006d250 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 -- 172.21.15.43:0/1298586059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52006d250 msgr2=0x7fa52015d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52006d250 0x7fa52015d870 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa508009870 tx=0x7fa50802b980 comp rx=0 tx=0).stop 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 -- 172.21.15.43:0/1298586059 shutdown_connections 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa520070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1298586059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa52006d250 0x7fa52015d870 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 -- 172.21.15.43:0/1298586059 >> 172.21.15.43:0/1298586059 conn(0x7fa52006a860 msgr2=0x7fa52006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 -- 172.21.15.43:0/1298586059 shutdown_connections 2024-01-16T15:15:01.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.044+0000 7fa51fe00640 1 -- 172.21.15.43:0/1298586059 wait complete. 2024-01-16T15:15:01.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.046+0000 7fa51fe00640 1 Processor -- start 2024-01-16T15:15:01.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.046+0000 7fa51fe00640 1 -- start start 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa52006d250 0x7fa52015e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa5200cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa520151e20 con 0x7fa520157770 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52015ee70 con 0x7fa52006d250 2024-01-16T15:15:01.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52015f150 con 0x7fa52015ddb0 2024-01-16T15:15:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa515e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa515e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54122/0 (socket says 172.21.15.43:54122) 2024-01-16T15:15:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa515e00640 1 -- 172.21.15.43:0/1472753391 learned_addr learned my addr 172.21.15.43:0/1472753391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.047+0000 7fa51f400640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa5200cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 -- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 msgr2=0x7fa5200cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa51ea00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa52006d250 0x7fa52015e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa5200cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 -- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa52006d250 msgr2=0x7fa52015e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa52006d250 0x7fa52015e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 -- 172.21.15.43:0/1472753391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080094d0 con 0x7fa520157770 2024-01-16T15:15:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa515e00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa508000c00 tx=0x7fa508004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508039c90 con 0x7fa520157770 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa508039df0 con 0x7fa520157770 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.048+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508043640 con 0x7fa520157770 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.049+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa520161c80 con 0x7fa520157770 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.049+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52015f710 con 0x7fa520157770 2024-01-16T15:15:01.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.050+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52004e840 con 0x7fa520157770 2024-01-16T15:15:01.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.051+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fa5080437a0 con 0x7fa520157770 2024-01-16T15:15:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.052+0000 7fa517400640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 0x7fa4f806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.052+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa50803e070 con 0x7fa520157770 2024-01-16T15:15:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.052+0000 7fa51ea00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 0x7fa4f806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.053+0000 7fa51ea00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 0x7fa4f806e590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa510007fd0 tx=0x7fa510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.056+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa508078020 con 0x7fa520157770 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.253+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5200704e0 con 0x7fa520157770 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.255+0000 7fa517400640 1 -- 172.21.15.43:0/1472753391 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16506 (secure 0 0 0) 0x7fa50807b640 con 0x7fa520157770 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:01.239 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":101,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:14:58.256140+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:14:54.585377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:14:55.729389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 msgr2=0x7fa4f806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 0x7fa4f806e590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa510007fd0 tx=0x7fa510009040 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 msgr2=0x7fa5201518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa508000c00 tx=0x7fa508004730 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 shutdown_connections 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa4f806c0d0 0x7fa4f806e590 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa52015ddb0 0x7fa5200cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.258+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa520157770 0x7fa5201518e0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.259+0000 7fa51fe00640 1 --2- 172.21.15.43:0/1472753391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa52006d250 0x7fa52015e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.259+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 >> 172.21.15.43:0/1472753391 conn(0x7fa52006a860 msgr2=0x7fa52006ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.259+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 shutdown_connections 2024-01-16T15:15:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.259+0000 7fa51fe00640 1 -- 172.21.15.43:0/1472753391 wait complete. 2024-01-16T15:15:01.260 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:15:01.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.502+0000 7f891e000640 1 Processor -- start 2024-01-16T15:15:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.502+0000 7f891e000640 1 -- start start 2024-01-16T15:15:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f891e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f891806d520 0x7f891806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f891e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f891815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f891e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891815c360 con 0x7f89180ad720 2024-01-16T15:15:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f891e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89180b41a0 con 0x7f891806d520 2024-01-16T15:15:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f891e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891806de40 con 0x7f89180abff0 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f890fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f89180ad720 0x7f891815ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53086/0 (socket says 172.21.15.43:53086) 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.504+0000 7f890fe00640 1 -- 172.21.15.43:0/1882215917 learned_addr learned my addr 172.21.15.43:0/1882215917 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8917400640 1 --2- 172.21.15.43:0/1882215917 >> v2:172.21.15.69:3300/0 conn(0x7f891806d520 0x7f891806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8917e00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f891815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38512407 0 0) 0x7f891815c360 con 0x7f89180ad720 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f8003520 con 0x7f89180ad720 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054981193 0 0) 0x7f88f8003520 con 0x7f89180ad720 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 msgr2=0x7f891815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f891815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 >> v2:172.21.15.69:3300/0 conn(0x7f891806d520 msgr2=0x7f891806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 --2- 172.21.15.43:0/1882215917 >> v2:172.21.15.69:3300/0 conn(0x7f891806d520 0x7f891806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891815ba30 con 0x7f89180ad720 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f8900001670 con 0x7f89180ad720 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 >> v1:172.21.15.43:6789/0 conn(0x7f89180ad720 legacy=0x7f891815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f88f8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f88f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f8004d00 0x7f88f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891806de40 con 0x7f88f8003f40 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f800b980 con 0x7f88f8004d00 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.505+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f800bac0 con 0x7f89180abff0 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f88f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f8917400640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f88f8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 -- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 msgr2=0x7f88f8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180abff0 0x7f88f8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 -- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f8004d00 msgr2=0x7f88f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88f8004d00 0x7f88f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.506+0000 7f890fe00640 1 -- 172.21.15.43:0/1882215917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8003520 con 0x7f88f8003f40 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.507+0000 7f890fe00640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f88f80047c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f890000c0e0 tx=0x7f88f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e923dbeabdea5290 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900005ad0 con 0x7f88f8003f40 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8900005c50 con 0x7f88f8003f40 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f8916a00640 1 -- 172.21.15.43:0/1882215917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890002c7c0 con 0x7f88f8003f40 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f891e000640 1 -- 172.21.15.43:0/1882215917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 msgr2=0x7f88f80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f891e000640 1 --2- 172.21.15.43:0/1882215917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f88f80047c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f890000c0e0 tx=0x7f88f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f8917e00640 1 -- 172.21.15.43:0/1882215917 reap_dead start 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f891e000640 1 -- 172.21.15.43:0/1882215917 shutdown_connections 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.508+0000 7f891e000640 1 -- 172.21.15.43:0/1882215917 >> 172.21.15.43:0/1882215917 conn(0x7f89180b0d80 msgr2=0x7f89180b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.509+0000 7f891e000640 1 -- 172.21.15.43:0/1882215917 shutdown_connections 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.509+0000 7f891e000640 1 -- 172.21.15.43:0/1882215917 wait complete. 2024-01-16T15:15:01.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.511+0000 7f891e000640 1 Processor -- start 2024-01-16T15:15:01.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.511+0000 7f891e000640 1 -- start start 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 0x7f8918070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180c8d90 0x7f89180c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89180c9780 con 0x7f88f8003f40 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89180cc770 con 0x7f89180c8400 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.512+0000 7f891e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8918157c80 con 0x7f89180c8d90 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54130/0 (socket says 172.21.15.43:54130) 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 -- 172.21.15.43:0/297617521 learned_addr learned my addr 172.21.15.43:0/297617521 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917e00640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180c8d90 0x7f89180c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f890fe00640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 0x7f8918070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 -- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180c8d90 msgr2=0x7f89180c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180c8d90 0x7f89180c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 -- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 msgr2=0x7f8918070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 0x7f8918070ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f8917400640 1 -- 172.21.15.43:0/297617521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180cc440 con 0x7f88f8003f40 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.513+0000 7f890fe00640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 0x7f8918070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f8917400640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8908011600 tx=0x7f8908011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800f8c0 con 0x7f88f8003f40 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890800fa20 con 0x7f88f8003f40 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800fd90 con 0x7f88f8003f40 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8918071710 con 0x7f88f8003f40 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.514+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8918071cd0 con 0x7f88f8003f40 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.516+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8918074770 con 0x7f88f8003f40 2024-01-16T15:15:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.517+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f8908015460 con 0x7f88f8003f40 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.518+0000 7f890f400640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 0x7f88f006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.518+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8908089100 con 0x7f88f8003f40 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.518+0000 7f890fe00640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 0x7f88f006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.519+0000 7f890fe00640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 0x7f88f006e5c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8900034000 tx=0x7f8900007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.522+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8908055760 con 0x7f88f8003f40 2024-01-16T15:15:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.715+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f891815b6e0 con 0x7f88f006c100 2024-01-16T15:15:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.738+0000 7f890f400640 1 -- 172.21.15.43:0/297617521 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286862 (secure 0 0 0) 0x7f8900001b80 con 0x7f88f006c100 2024-01-16T15:15:01.726 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:15:01.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 msgr2=0x7f88f006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 0x7f88f006e5c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8900034000 tx=0x7f8900007000 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 msgr2=0x7f8918157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8908011600 tx=0x7f8908011ac0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 shutdown_connections 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f88f006c100 0x7f88f006e5c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89180c8400 0x7f8918070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89180c8d90 0x7f89180c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 --2- 172.21.15.43:0/297617521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88f8003f40 0x7f8918157740 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.745+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 >> 172.21.15.43:0/297617521 conn(0x7f89180b0d80 msgr2=0x7f89181583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.746+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 shutdown_connections 2024-01-16T15:15:01.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:01.746+0000 7f891e000640 1 -- 172.21.15.43:0/297617521 wait complete. 2024-01-16T15:15:01.751 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-01-16T15:15:01.194875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":586124,"kb_used_data":10952,"kb_used_omap":12,"kb_used_meta":575027,"kb_avail":754388596,"statfs":{"total":773094113280,"available":772493922304,"internally_reserved":0,"allocated":11214848,"data_stored":5487388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":588828204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.982515"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375451+0000","last_change":"2024-01-16T15:14:57.268738+0000","last_active":"2024-01-16T15:14:58.375451+0000","last_peered":"2024-01-16T15:14:58.375451+0000","last_clean":"2024-01-16T15:14:58.375451+0000","last_became_active":"2024-01-16T15:14:57.267639+0000","last_became_peered":"2024-01-16T15:14:57.267639+0000","last_unstale":"2024-01-16T15:14:58.375451+0000","last_undegraded":"2024-01-16T15:14:58.375451+0000","last_fullsized":"2024-01-16T15:14:58.375451+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:55.550976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376602+0000","last_change":"2024-01-16T15:14:57.277847+0000","last_active":"2024-01-16T15:14:58.376602+0000","last_peered":"2024-01-16T15:14:58.376602+0000","last_clean":"2024-01-16T15:14:58.376602+0000","last_became_active":"2024-01-16T15:14:57.277511+0000","last_became_peered":"2024-01-16T15:14:57.277511+0000","last_unstale":"2024-01-16T15:14:58.376602+0000","last_undegraded":"2024-01-16T15:14:58.376602+0000","last_fullsized":"2024-01-16T15:14:58.376602+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:04.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375490+0000","last_change":"2024-01-16T15:14:57.268421+0000","last_active":"2024-01-16T15:14:58.375490+0000","last_peered":"2024-01-16T15:14:58.375490+0000","last_clean":"2024-01-16T15:14:58.375490+0000","last_became_active":"2024-01-16T15:14:57.268158+0000","last_became_peered":"2024-01-16T15:14:57.268158+0000","last_unstale":"2024-01-16T15:14:58.375490+0000","last_undegraded":"2024-01-16T15:14:58.375490+0000","last_fullsized":"2024-01-16T15:14:58.375490+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:22.247546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377946+0000","last_change":"2024-01-16T15:14:57.276753+0000","last_active":"2024-01-16T15:14:58.377946+0000","last_peered":"2024-01-16T15:14:58.377946+0000","last_clean":"2024-01-16T15:14:58.377946+0000","last_became_active":"2024-01-16T15:14:57.272254+0000","last_became_peered":"2024-01-16T15:14:57.272254+0000","last_unstale":"2024-01-16T15:14:58.377946+0000","last_undegraded":"2024-01-16T15:14:58.377946+0000","last_fullsized":"2024-01-16T15:14:58.377946+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:58.689825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267888+0000","last_change":"2024-01-16T15:14:57.260787+0000","last_active":"2024-01-16T15:14:58.267888+0000","last_peered":"2024-01-16T15:14:58.267888+0000","last_clean":"2024-01-16T15:14:58.267888+0000","last_became_active":"2024-01-16T15:14:57.260490+0000","last_became_peered":"2024-01-16T15:14:57.260490+0000","last_unstale":"2024-01-16T15:14:58.267888+0000","last_undegraded":"2024-01-16T15:14:58.267888+0000","last_fullsized":"2024-01-16T15:14:58.267888+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:57.049352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376590+0000","last_change":"2024-01-16T15:14:57.276912+0000","last_active":"2024-01-16T15:14:58.376590+0000","last_peered":"2024-01-16T15:14:58.376590+0000","last_clean":"2024-01-16T15:14:58.376590+0000","last_became_active":"2024-01-16T15:14:57.276487+0000","last_became_peered":"2024-01-16T15:14:57.276487+0000","last_unstale":"2024-01-16T15:14:58.376590+0000","last_undegraded":"2024-01-16T15:14:58.376590+0000","last_fullsized":"2024-01-16T15:14:58.376590+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:09.470583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375735+0000","last_change":"2024-01-16T15:14:57.276540+0000","last_active":"2024-01-16T15:14:58.375735+0000","last_peered":"2024-01-16T15:14:58.375735+0000","last_clean":"2024-01-16T15:14:58.375735+0000","last_became_active":"2024-01-16T15:14:57.276244+0000","last_became_peered":"2024-01-16T15:14:57.276244+0000","last_unstale":"2024-01-16T15:14:58.375735+0000","last_undegraded":"2024-01-16T15:14:58.375735+0000","last_fullsized":"2024-01-16T15:14:58.375735+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:31.613245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375171+0000","last_change":"2024-01-16T15:14:57.269832+0000","last_active":"2024-01-16T15:14:58.375171+0000","last_peered":"2024-01-16T15:14:58.375171+0000","last_clean":"2024-01-16T15:14:58.375171+0000","last_became_active":"2024-01-16T15:14:57.269536+0000","last_became_peered":"2024-01-16T15:14:57.269536+0000","last_unstale":"2024-01-16T15:14:58.375171+0000","last_undegraded":"2024-01-16T15:14:58.375171+0000","last_fullsized":"2024-01-16T15:14:58.375171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:51:10.476996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266714+0000","last_change":"2024-01-16T15:14:57.275114+0000","last_active":"2024-01-16T15:14:58.266714+0000","last_peered":"2024-01-16T15:14:58.266714+0000","last_clean":"2024-01-16T15:14:58.266714+0000","last_became_active":"2024-01-16T15:14:57.274090+0000","last_became_peered":"2024-01-16T15:14:57.274090+0000","last_unstale":"2024-01-16T15:14:58.266714+0000","last_undegraded":"2024-01-16T15:14:58.266714+0000","last_fullsized":"2024-01-16T15:14:58.266714+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.150989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269239+0000","last_change":"2024-01-16T15:14:57.267010+0000","last_active":"2024-01-16T15:14:58.269239+0000","last_peered":"2024-01-16T15:14:58.269239+0000","last_clean":"2024-01-16T15:14:58.269239+0000","last_became_active":"2024-01-16T15:14:57.266733+0000","last_became_peered":"2024-01-16T15:14:57.266733+0000","last_unstale":"2024-01-16T15:14:58.269239+0000","last_undegraded":"2024-01-16T15:14:58.269239+0000","last_fullsized":"2024-01-16T15:14:58.269239+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:41.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375755+0000","last_change":"2024-01-16T15:14:57.280588+0000","last_active":"2024-01-16T15:14:58.375755+0000","last_peered":"2024-01-16T15:14:58.375755+0000","last_clean":"2024-01-16T15:14:58.375755+0000","last_became_active":"2024-01-16T15:14:57.280107+0000","last_became_peered":"2024-01-16T15:14:57.280107+0000","last_unstale":"2024-01-16T15:14:58.375755+0000","last_undegraded":"2024-01-16T15:14:58.375755+0000","last_fullsized":"2024-01-16T15:14:58.375755+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:31.434341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265809+0000","last_change":"2024-01-16T15:14:57.282533+0000","last_active":"2024-01-16T15:14:58.265809+0000","last_peered":"2024-01-16T15:14:58.265809+0000","last_clean":"2024-01-16T15:14:58.265809+0000","last_became_active":"2024-01-16T15:14:57.282165+0000","last_became_peered":"2024-01-16T15:14:57.282165+0000","last_unstale":"2024-01-16T15:14:58.265809+0000","last_undegraded":"2024-01-16T15:14:58.265809+0000","last_fullsized":"2024-01-16T15:14:58.265809+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:08.595609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378098+0000","last_change":"2024-01-16T15:14:57.276729+0000","last_active":"2024-01-16T15:14:58.378098+0000","last_peered":"2024-01-16T15:14:58.378098+0000","last_clean":"2024-01-16T15:14:58.378098+0000","last_became_active":"2024-01-16T15:14:57.275755+0000","last_became_peered":"2024-01-16T15:14:57.275755+0000","last_unstale":"2024-01-16T15:14:58.378098+0000","last_undegraded":"2024-01-16T15:14:58.378098+0000","last_fullsized":"2024-01-16T15:14:58.378098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:31.892776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263427+0000","last_change":"2024-01-16T15:14:57.277001+0000","last_active":"2024-01-16T15:14:58.263427+0000","last_peered":"2024-01-16T15:14:58.263427+0000","last_clean":"2024-01-16T15:14:58.263427+0000","last_became_active":"2024-01-16T15:14:57.276555+0000","last_became_peered":"2024-01-16T15:14:57.276555+0000","last_unstale":"2024-01-16T15:14:58.263427+0000","last_undegraded":"2024-01-16T15:14:58.263427+0000","last_fullsized":"2024-01-16T15:14:58.263427+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:18.037106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378059+0000","last_change":"2024-01-16T15:14:57.268931+0000","last_active":"2024-01-16T15:14:58.378059+0000","last_peered":"2024-01-16T15:14:58.378059+0000","last_clean":"2024-01-16T15:14:58.378059+0000","last_became_active":"2024-01-16T15:14:57.268661+0000","last_became_peered":"2024-01-16T15:14:57.268661+0000","last_unstale":"2024-01-16T15:14:58.378059+0000","last_undegraded":"2024-01-16T15:14:58.378059+0000","last_fullsized":"2024-01-16T15:14:58.378059+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:21.445627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375715+0000","last_change":"2024-01-16T15:14:57.279996+0000","last_active":"2024-01-16T15:14:58.375715+0000","last_peered":"2024-01-16T15:14:58.375715+0000","last_clean":"2024-01-16T15:14:58.375715+0000","last_became_active":"2024-01-16T15:14:57.279695+0000","last_became_peered":"2024-01-16T15:14:57.279695+0000","last_unstale":"2024-01-16T15:14:58.375715+0000","last_undegraded":"2024-01-16T15:14:58.375715+0000","last_fullsized":"2024-01-16T15:14:58.375715+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:51.570860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376819+0000","last_change":"2024-01-16T15:14:57.263987+0000","last_active":"2024-01-16T15:14:58.376819+0000","last_peered":"2024-01-16T15:14:58.376819+0000","last_clean":"2024-01-16T15:14:58.376819+0000","last_became_active":"2024-01-16T15:14:57.258913+0000","last_became_peered":"2024-01-16T15:14:57.258913+0000","last_unstale":"2024-01-16T15:14:58.376819+0000","last_undegraded":"2024-01-16T15:14:58.376819+0000","last_fullsized":"2024-01-16T15:14:58.376819+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:22.248118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378592+0000","last_change":"2024-01-16T15:14:57.282890+0000","last_active":"2024-01-16T15:14:58.378592+0000","last_peered":"2024-01-16T15:14:58.378592+0000","last_clean":"2024-01-16T15:14:58.378592+0000","last_became_active":"2024-01-16T15:14:57.282602+0000","last_became_peered":"2024-01-16T15:14:57.282602+0000","last_unstale":"2024-01-16T15:14:58.378592+0000","last_undegraded":"2024-01-16T15:14:58.378592+0000","last_fullsized":"2024-01-16T15:14:58.378592+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:51.570860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264370+0000","last_change":"2024-01-16T15:14:57.268679+0000","last_active":"2024-01-16T15:14:58.264370+0000","last_peered":"2024-01-16T15:14:58.264370+0000","last_clean":"2024-01-16T15:14:58.264370+0000","last_became_active":"2024-01-16T15:14:57.268395+0000","last_became_peered":"2024-01-16T15:14:57.268395+0000","last_unstale":"2024-01-16T15:14:58.264370+0000","last_undegraded":"2024-01-16T15:14:58.264370+0000","last_fullsized":"2024-01-16T15:14:58.264370+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:23.789757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266237+0000","last_change":"2024-01-16T15:14:57.277443+0000","last_active":"2024-01-16T15:14:58.266237+0000","last_peered":"2024-01-16T15:14:58.266237+0000","last_clean":"2024-01-16T15:14:58.266237+0000","last_became_active":"2024-01-16T15:14:57.277120+0000","last_became_peered":"2024-01-16T15:14:57.277120+0000","last_unstale":"2024-01-16T15:14:58.266237+0000","last_undegraded":"2024-01-16T15:14:58.266237+0000","last_fullsized":"2024-01-16T15:14:58.266237+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:23.813457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263388+0000","last_change":"2024-01-16T15:14:57.265176+0000","last_active":"2024-01-16T15:14:58.263388+0000","last_peered":"2024-01-16T15:14:58.263388+0000","last_clean":"2024-01-16T15:14:58.263388+0000","last_became_active":"2024-01-16T15:14:57.264777+0000","last_became_peered":"2024-01-16T15:14:57.264777+0000","last_unstale":"2024-01-16T15:14:58.263388+0000","last_undegraded":"2024-01-16T15:14:58.263388+0000","last_fullsized":"2024-01-16T15:14:58.263388+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:58.989677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375965+0000","last_change":"2024-01-16T15:14:57.279642+0000","last_active":"2024-01-16T15:14:58.375965+0000","last_peered":"2024-01-16T15:14:58.375965+0000","last_clean":"2024-01-16T15:14:58.375965+0000","last_became_active":"2024-01-16T15:14:57.277939+0000","last_became_peered":"2024-01-16T15:14:57.277939+0000","last_unstale":"2024-01-16T15:14:58.375965+0000","last_undegraded":"2024-01-16T15:14:58.375965+0000","last_fullsized":"2024-01-16T15:14:58.375965+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:58.954219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375910+0000","last_change":"2024-01-16T15:14:57.279745+0000","last_active":"2024-01-16T15:14:58.375910+0000","last_peered":"2024-01-16T15:14:58.375910+0000","last_clean":"2024-01-16T15:14:58.375910+0000","last_became_active":"2024-01-16T15:14:57.279048+0000","last_became_peered":"2024-01-16T15:14:57.279048+0000","last_unstale":"2024-01-16T15:14:58.375910+0000","last_undegraded":"2024-01-16T15:14:58.375910+0000","last_fullsized":"2024-01-16T15:14:58.375910+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:25.612959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376512+0000","last_change":"2024-01-16T15:14:57.279969+0000","last_active":"2024-01-16T15:14:58.376512+0000","last_peered":"2024-01-16T15:14:58.376512+0000","last_clean":"2024-01-16T15:14:58.376512+0000","last_became_active":"2024-01-16T15:14:57.279515+0000","last_became_peered":"2024-01-16T15:14:57.279515+0000","last_unstale":"2024-01-16T15:14:58.376512+0000","last_undegraded":"2024-01-16T15:14:58.376512+0000","last_fullsized":"2024-01-16T15:14:58.376512+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:24.282448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269602+0000","last_change":"2024-01-16T15:14:57.272062+0000","last_active":"2024-01-16T15:14:58.269602+0000","last_peered":"2024-01-16T15:14:58.269602+0000","last_clean":"2024-01-16T15:14:58.269602+0000","last_became_active":"2024-01-16T15:14:57.271101+0000","last_became_peered":"2024-01-16T15:14:57.271101+0000","last_unstale":"2024-01-16T15:14:58.269602+0000","last_undegraded":"2024-01-16T15:14:58.269602+0000","last_fullsized":"2024-01-16T15:14:58.269602+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:13.227145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376372+0000","last_change":"2024-01-16T15:14:57.270151+0000","last_active":"2024-01-16T15:14:58.376372+0000","last_peered":"2024-01-16T15:14:58.376372+0000","last_clean":"2024-01-16T15:14:58.376372+0000","last_became_active":"2024-01-16T15:14:57.269840+0000","last_became_peered":"2024-01-16T15:14:57.269840+0000","last_unstale":"2024-01-16T15:14:58.376372+0000","last_undegraded":"2024-01-16T15:14:58.376372+0000","last_fullsized":"2024-01-16T15:14:58.376372+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:43.938439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376112+0000","last_change":"2024-01-16T15:14:57.276338+0000","last_active":"2024-01-16T15:14:58.376112+0000","last_peered":"2024-01-16T15:14:58.376112+0000","last_clean":"2024-01-16T15:14:58.376112+0000","last_became_active":"2024-01-16T15:14:57.276027+0000","last_became_peered":"2024-01-16T15:14:57.276027+0000","last_unstale":"2024-01-16T15:14:58.376112+0000","last_undegraded":"2024-01-16T15:14:58.376112+0000","last_fullsized":"2024-01-16T15:14:58.376112+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269043+0000","last_change":"2024-01-16T15:14:57.264934+0000","last_active":"2024-01-16T15:14:58.269043+0000","last_peered":"2024-01-16T15:14:58.269043+0000","last_clean":"2024-01-16T15:14:58.269043+0000","last_became_active":"2024-01-16T15:14:57.264636+0000","last_became_peered":"2024-01-16T15:14:57.264636+0000","last_unstale":"2024-01-16T15:14:58.269043+0000","last_undegraded":"2024-01-16T15:14:58.269043+0000","last_fullsized":"2024-01-16T15:14:58.269043+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:08.246934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376050+0000","last_change":"2024-01-16T15:14:57.270864+0000","last_active":"2024-01-16T15:14:58.376050+0000","last_peered":"2024-01-16T15:14:58.376050+0000","last_clean":"2024-01-16T15:14:58.376050+0000","last_became_active":"2024-01-16T15:14:57.270588+0000","last_became_peered":"2024-01-16T15:14:57.270588+0000","last_unstale":"2024-01-16T15:14:58.376050+0000","last_undegraded":"2024-01-16T15:14:58.376050+0000","last_fullsized":"2024-01-16T15:14:58.376050+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:59.418210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263603+0000","last_change":"2024-01-16T15:14:57.276822+0000","last_active":"2024-01-16T15:14:58.263603+0000","last_peered":"2024-01-16T15:14:58.263603+0000","last_clean":"2024-01-16T15:14:58.263603+0000","last_became_active":"2024-01-16T15:14:57.276263+0000","last_became_peered":"2024-01-16T15:14:57.276263+0000","last_unstale":"2024-01-16T15:14:58.263603+0000","last_undegraded":"2024-01-16T15:14:58.263603+0000","last_fullsized":"2024-01-16T15:14:58.263603+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:50.333191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378125+0000","last_change":"2024-01-16T15:14:57.268384+0000","last_active":"2024-01-16T15:14:58.378125+0000","last_peered":"2024-01-16T15:14:58.378125+0000","last_clean":"2024-01-16T15:14:58.378125+0000","last_became_active":"2024-01-16T15:14:57.267319+0000","last_became_peered":"2024-01-16T15:14:57.267319+0000","last_unstale":"2024-01-16T15:14:58.378125+0000","last_undegraded":"2024-01-16T15:14:58.378125+0000","last_fullsized":"2024-01-16T15:14:58.378125+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:31.093985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.374812+0000","last_change":"2024-01-16T15:14:57.286809+0000","last_active":"2024-01-16T15:14:58.374812+0000","last_peered":"2024-01-16T15:14:58.374812+0000","last_clean":"2024-01-16T15:14:58.374812+0000","last_became_active":"2024-01-16T15:14:57.284949+0000","last_became_peered":"2024-01-16T15:14:57.284949+0000","last_unstale":"2024-01-16T15:14:58.374812+0000","last_undegraded":"2024-01-16T15:14:58.374812+0000","last_fullsized":"2024-01-16T15:14:58.374812+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:19.648713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264387+0000","last_change":"2024-01-16T15:14:57.268766+0000","last_active":"2024-01-16T15:14:58.264387+0000","last_peered":"2024-01-16T15:14:58.264387+0000","last_clean":"2024-01-16T15:14:58.264387+0000","last_became_active":"2024-01-16T15:14:57.268488+0000","last_became_peered":"2024-01-16T15:14:57.268488+0000","last_unstale":"2024-01-16T15:14:58.264387+0000","last_undegraded":"2024-01-16T15:14:58.264387+0000","last_fullsized":"2024-01-16T15:14:58.264387+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:43.440293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"100'4","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.284752+0000","last_change":"2024-01-16T15:14:56.247628+0000","last_active":"2024-01-16T15:14:57.284752+0000","last_peered":"2024-01-16T15:14:57.284752+0000","last_clean":"2024-01-16T15:14:57.284752+0000","last_became_active":"2024-01-16T15:14:56.247338+0000","last_became_peered":"2024-01-16T15:14:56.247338+0000","last_unstale":"2024-01-16T15:14:57.284752+0000","last_undegraded":"2024-01-16T15:14:57.284752+0000","last_fullsized":"2024-01-16T15:14:57.284752+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:30.335125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.379230+0000","last_change":"2024-01-16T15:14:57.270923+0000","last_active":"2024-01-16T15:14:58.379230+0000","last_peered":"2024-01-16T15:14:58.379230+0000","last_clean":"2024-01-16T15:14:58.379230+0000","last_became_active":"2024-01-16T15:14:57.270658+0000","last_became_peered":"2024-01-16T15:14:57.270658+0000","last_unstale":"2024-01-16T15:14:58.379230+0000","last_undegraded":"2024-01-16T15:14:58.379230+0000","last_fullsized":"2024-01-16T15:14:58.379230+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:31.613245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.274812+0000","last_change":"2024-01-16T15:14:56.257668+0000","last_active":"2024-01-16T15:14:57.274812+0000","last_peered":"2024-01-16T15:14:57.274812+0000","last_clean":"2024-01-16T15:14:57.274812+0000","last_became_active":"2024-01-16T15:14:56.257372+0000","last_became_peered":"2024-01-16T15:14:56.257372+0000","last_unstale":"2024-01-16T15:14:57.274812+0000","last_undegraded":"2024-01-16T15:14:57.274812+0000","last_fullsized":"2024-01-16T15:14:57.274812+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:30.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376455+0000","last_change":"2024-01-16T15:14:57.283445+0000","last_active":"2024-01-16T15:14:58.376455+0000","last_peered":"2024-01-16T15:14:58.376455+0000","last_clean":"2024-01-16T15:14:58.376455+0000","last_became_active":"2024-01-16T15:14:57.283138+0000","last_became_peered":"2024-01-16T15:14:57.283138+0000","last_unstale":"2024-01-16T15:14:58.376455+0000","last_undegraded":"2024-01-16T15:14:58.376455+0000","last_fullsized":"2024-01-16T15:14:58.376455+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:26.406696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288805+0000","last_change":"2024-01-16T15:14:56.251801+0000","last_active":"2024-01-16T15:14:57.288805+0000","last_peered":"2024-01-16T15:14:57.288805+0000","last_clean":"2024-01-16T15:14:57.288805+0000","last_became_active":"2024-01-16T15:14:56.248230+0000","last_became_peered":"2024-01-16T15:14:56.248230+0000","last_unstale":"2024-01-16T15:14:57.288805+0000","last_undegraded":"2024-01-16T15:14:57.288805+0000","last_fullsized":"2024-01-16T15:14:57.288805+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:14.641251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266838+0000","last_change":"2024-01-16T15:14:57.275944+0000","last_active":"2024-01-16T15:14:58.266838+0000","last_peered":"2024-01-16T15:14:58.266838+0000","last_clean":"2024-01-16T15:14:58.266838+0000","last_became_active":"2024-01-16T15:14:57.275675+0000","last_became_peered":"2024-01-16T15:14:57.275675+0000","last_unstale":"2024-01-16T15:14:58.266838+0000","last_undegraded":"2024-01-16T15:14:58.266838+0000","last_fullsized":"2024-01-16T15:14:58.266838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.284502+0000","last_change":"2024-01-16T15:14:56.251154+0000","last_active":"2024-01-16T15:14:57.284502+0000","last_peered":"2024-01-16T15:14:57.284502+0000","last_clean":"2024-01-16T15:14:57.284502+0000","last_became_active":"2024-01-16T15:14:56.247511+0000","last_became_peered":"2024-01-16T15:14:56.247511+0000","last_unstale":"2024-01-16T15:14:57.284502+0000","last_undegraded":"2024-01-16T15:14:57.284502+0000","last_fullsized":"2024-01-16T15:14:57.284502+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:19.175215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375888+0000","last_change":"2024-01-16T15:14:57.279137+0000","last_active":"2024-01-16T15:14:58.375888+0000","last_peered":"2024-01-16T15:14:58.375888+0000","last_clean":"2024-01-16T15:14:58.375888+0000","last_became_active":"2024-01-16T15:14:57.278824+0000","last_became_peered":"2024-01-16T15:14:57.278824+0000","last_unstale":"2024-01-16T15:14:58.375888+0000","last_undegraded":"2024-01-16T15:14:58.375888+0000","last_fullsized":"2024-01-16T15:14:58.375888+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:26.406696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.274380+0000","last_change":"2024-01-16T15:14:56.255146+0000","last_active":"2024-01-16T15:14:57.274380+0000","last_peered":"2024-01-16T15:14:57.274380+0000","last_clean":"2024-01-16T15:14:57.274380+0000","last_became_active":"2024-01-16T15:14:56.253956+0000","last_became_peered":"2024-01-16T15:14:56.253956+0000","last_unstale":"2024-01-16T15:14:57.274380+0000","last_undegraded":"2024-01-16T15:14:57.274380+0000","last_fullsized":"2024-01-16T15:14:57.274380+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:30.054925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376516+0000","last_change":"2024-01-16T15:14:57.276953+0000","last_active":"2024-01-16T15:14:58.376516+0000","last_peered":"2024-01-16T15:14:58.376516+0000","last_clean":"2024-01-16T15:14:58.376516+0000","last_became_active":"2024-01-16T15:14:57.276709+0000","last_became_peered":"2024-01-16T15:14:57.276709+0000","last_unstale":"2024-01-16T15:14:58.376516+0000","last_undegraded":"2024-01-16T15:14:58.376516+0000","last_fullsized":"2024-01-16T15:14:58.376516+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:03:52.157955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.286528+0000","last_change":"2024-01-16T15:14:56.257803+0000","last_active":"2024-01-16T15:14:57.286528+0000","last_peered":"2024-01-16T15:14:57.286528+0000","last_clean":"2024-01-16T15:14:57.286528+0000","last_became_active":"2024-01-16T15:14:56.257475+0000","last_became_peered":"2024-01-16T15:14:56.257475+0000","last_unstale":"2024-01-16T15:14:57.286528+0000","last_undegraded":"2024-01-16T15:14:57.286528+0000","last_fullsized":"2024-01-16T15:14:57.286528+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:07.231523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377565+0000","last_change":"2024-01-16T15:14:57.274379+0000","last_active":"2024-01-16T15:14:58.377565+0000","last_peered":"2024-01-16T15:14:58.377565+0000","last_clean":"2024-01-16T15:14:58.377565+0000","last_became_active":"2024-01-16T15:14:57.272853+0000","last_became_peered":"2024-01-16T15:14:57.272853+0000","last_unstale":"2024-01-16T15:14:58.377565+0000","last_undegraded":"2024-01-16T15:14:58.377565+0000","last_fullsized":"2024-01-16T15:14:58.377565+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:15.374800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.251642+0000","last_change":"2024-01-16T15:14:56.247282+0000","last_active":"2024-01-16T15:14:57.251642+0000","last_peered":"2024-01-16T15:14:57.251642+0000","last_clean":"2024-01-16T15:14:57.251642+0000","last_became_active":"2024-01-16T15:14:56.246887+0000","last_became_peered":"2024-01-16T15:14:56.246887+0000","last_unstale":"2024-01-16T15:14:57.251642+0000","last_undegraded":"2024-01-16T15:14:57.251642+0000","last_fullsized":"2024-01-16T15:14:57.251642+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:44:56.232403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265296+0000","last_change":"2024-01-16T15:14:57.262925+0000","last_active":"2024-01-16T15:14:58.265296+0000","last_peered":"2024-01-16T15:14:58.265296+0000","last_clean":"2024-01-16T15:14:58.265296+0000","last_became_active":"2024-01-16T15:14:57.261447+0000","last_became_peered":"2024-01-16T15:14:57.261447+0000","last_unstale":"2024-01-16T15:14:58.265296+0000","last_undegraded":"2024-01-16T15:14:58.265296+0000","last_fullsized":"2024-01-16T15:14:58.265296+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:19.313378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.259804+0000","last_change":"2024-01-16T15:14:56.260607+0000","last_active":"2024-01-16T15:14:57.259804+0000","last_peered":"2024-01-16T15:14:57.259804+0000","last_clean":"2024-01-16T15:14:57.259804+0000","last_became_active":"2024-01-16T15:14:56.258899+0000","last_became_peered":"2024-01-16T15:14:56.258899+0000","last_unstale":"2024-01-16T15:14:57.259804+0000","last_undegraded":"2024-01-16T15:14:57.259804+0000","last_fullsized":"2024-01-16T15:14:57.259804+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:22.774345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376497+0000","last_change":"2024-01-16T15:14:57.261589+0000","last_active":"2024-01-16T15:14:58.376497+0000","last_peered":"2024-01-16T15:14:58.376497+0000","last_clean":"2024-01-16T15:14:58.376497+0000","last_became_active":"2024-01-16T15:14:57.261259+0000","last_became_peered":"2024-01-16T15:14:57.261259+0000","last_unstale":"2024-01-16T15:14:58.376497+0000","last_undegraded":"2024-01-16T15:14:58.376497+0000","last_fullsized":"2024-01-16T15:14:58.376497+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:39.211207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.263636+0000","last_change":"2024-01-16T15:14:56.258700+0000","last_active":"2024-01-16T15:14:57.263636+0000","last_peered":"2024-01-16T15:14:57.263636+0000","last_clean":"2024-01-16T15:14:57.263636+0000","last_became_active":"2024-01-16T15:14:56.256930+0000","last_became_peered":"2024-01-16T15:14:56.256930+0000","last_unstale":"2024-01-16T15:14:57.263636+0000","last_undegraded":"2024-01-16T15:14:57.263636+0000","last_fullsized":"2024-01-16T15:14:57.263636+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:33.636833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267108+0000","last_change":"2024-01-16T15:14:57.274399+0000","last_active":"2024-01-16T15:14:58.267108+0000","last_peered":"2024-01-16T15:14:58.267108+0000","last_clean":"2024-01-16T15:14:58.267108+0000","last_became_active":"2024-01-16T15:14:57.273766+0000","last_became_peered":"2024-01-16T15:14:57.273766+0000","last_unstale":"2024-01-16T15:14:58.267108+0000","last_undegraded":"2024-01-16T15:14:58.267108+0000","last_fullsized":"2024-01-16T15:14:58.267108+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:36.466764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.250085+0000","last_change":"2024-01-16T15:14:56.234171+0000","last_active":"2024-01-16T15:14:57.250085+0000","last_peered":"2024-01-16T15:14:57.250085+0000","last_clean":"2024-01-16T15:14:57.250085+0000","last_became_active":"2024-01-16T15:14:56.233313+0000","last_became_peered":"2024-01-16T15:14:56.233313+0000","last_unstale":"2024-01-16T15:14:57.250085+0000","last_undegraded":"2024-01-16T15:14:57.250085+0000","last_fullsized":"2024-01-16T15:14:57.250085+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:40.475359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.268766+0000","last_change":"2024-01-16T15:14:57.262136+0000","last_active":"2024-01-16T15:14:58.268766+0000","last_peered":"2024-01-16T15:14:58.268766+0000","last_clean":"2024-01-16T15:14:58.268766+0000","last_became_active":"2024-01-16T15:14:57.261858+0000","last_became_peered":"2024-01-16T15:14:57.261858+0000","last_unstale":"2024-01-16T15:14:58.268766+0000","last_undegraded":"2024-01-16T15:14:58.268766+0000","last_fullsized":"2024-01-16T15:14:58.268766+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:20.190626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"100'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.281892+0000","last_change":"2024-01-16T15:14:56.254644+0000","last_active":"2024-01-16T15:14:57.281892+0000","last_peered":"2024-01-16T15:14:57.281892+0000","last_clean":"2024-01-16T15:14:57.281892+0000","last_became_active":"2024-01-16T15:14:56.254201+0000","last_became_peered":"2024-01-16T15:14:56.254201+0000","last_unstale":"2024-01-16T15:14:57.281892+0000","last_undegraded":"2024-01-16T15:14:57.281892+0000","last_fullsized":"2024-01-16T15:14:57.281892+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:20.430216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377526+0000","last_change":"2024-01-16T15:14:57.274051+0000","last_active":"2024-01-16T15:14:58.377526+0000","last_peered":"2024-01-16T15:14:58.377526+0000","last_clean":"2024-01-16T15:14:58.377526+0000","last_became_active":"2024-01-16T15:14:57.273775+0000","last_became_peered":"2024-01-16T15:14:57.273775+0000","last_unstale":"2024-01-16T15:14:58.377526+0000","last_undegraded":"2024-01-16T15:14:58.377526+0000","last_fullsized":"2024-01-16T15:14:58.377526+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:28.098955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"100'6","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288248+0000","last_change":"2024-01-16T15:14:56.249720+0000","last_active":"2024-01-16T15:14:57.288248+0000","last_peered":"2024-01-16T15:14:57.288248+0000","last_clean":"2024-01-16T15:14:57.288248+0000","last_became_active":"2024-01-16T15:14:56.248747+0000","last_became_peered":"2024-01-16T15:14:56.248747+0000","last_unstale":"2024-01-16T15:14:57.288248+0000","last_undegraded":"2024-01-16T15:14:57.288248+0000","last_fullsized":"2024-01-16T15:14:57.288248+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:18.335638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377698+0000","last_change":"2024-01-16T15:14:57.267525+0000","last_active":"2024-01-16T15:14:58.377698+0000","last_peered":"2024-01-16T15:14:58.377698+0000","last_clean":"2024-01-16T15:14:58.377698+0000","last_became_active":"2024-01-16T15:14:57.267246+0000","last_became_peered":"2024-01-16T15:14:57.267246+0000","last_unstale":"2024-01-16T15:14:58.377698+0000","last_undegraded":"2024-01-16T15:14:58.377698+0000","last_fullsized":"2024-01-16T15:14:58.377698+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:17.356858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.285170+0000","last_change":"2024-01-16T15:14:56.252697+0000","last_active":"2024-01-16T15:14:57.285170+0000","last_peered":"2024-01-16T15:14:57.285170+0000","last_clean":"2024-01-16T15:14:57.285170+0000","last_became_active":"2024-01-16T15:14:56.252235+0000","last_became_peered":"2024-01-16T15:14:56.252235+0000","last_unstale":"2024-01-16T15:14:57.285170+0000","last_undegraded":"2024-01-16T15:14:57.285170+0000","last_fullsized":"2024-01-16T15:14:57.285170+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.646300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375077+0000","last_change":"2024-01-16T15:14:57.279952+0000","last_active":"2024-01-16T15:14:58.375077+0000","last_peered":"2024-01-16T15:14:58.375077+0000","last_clean":"2024-01-16T15:14:58.375077+0000","last_became_active":"2024-01-16T15:14:57.279605+0000","last_became_peered":"2024-01-16T15:14:57.279605+0000","last_unstale":"2024-01-16T15:14:58.375077+0000","last_undegraded":"2024-01-16T15:14:58.375077+0000","last_fullsized":"2024-01-16T15:14:58.375077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:16.022404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.262802+0000","last_change":"2024-01-16T15:14:56.258480+0000","last_active":"2024-01-16T15:14:57.262802+0000","last_peered":"2024-01-16T15:14:57.262802+0000","last_clean":"2024-01-16T15:14:57.262802+0000","last_became_active":"2024-01-16T15:14:56.258207+0000","last_became_peered":"2024-01-16T15:14:56.258207+0000","last_unstale":"2024-01-16T15:14:57.262802+0000","last_undegraded":"2024-01-16T15:14:57.262802+0000","last_fullsized":"2024-01-16T15:14:57.262802+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:25.707694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264907+0000","last_change":"2024-01-16T15:14:57.278967+0000","last_active":"2024-01-16T15:14:58.264907+0000","last_peered":"2024-01-16T15:14:58.264907+0000","last_clean":"2024-01-16T15:14:58.264907+0000","last_became_active":"2024-01-16T15:14:57.277612+0000","last_became_peered":"2024-01-16T15:14:57.277612+0000","last_unstale":"2024-01-16T15:14:58.264907+0000","last_undegraded":"2024-01-16T15:14:58.264907+0000","last_fullsized":"2024-01-16T15:14:58.264907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:49.246274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.263440+0000","last_change":"2024-01-16T15:14:56.247113+0000","last_active":"2024-01-16T15:14:57.263440+0000","last_peered":"2024-01-16T15:14:57.263440+0000","last_clean":"2024-01-16T15:14:57.263440+0000","last_became_active":"2024-01-16T15:14:56.246632+0000","last_became_peered":"2024-01-16T15:14:56.246632+0000","last_unstale":"2024-01-16T15:14:57.263440+0000","last_undegraded":"2024-01-16T15:14:57.263440+0000","last_fullsized":"2024-01-16T15:14:57.263440+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:41.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263164+0000","last_change":"2024-01-16T15:14:57.287255+0000","last_active":"2024-01-16T15:14:58.263164+0000","last_peered":"2024-01-16T15:14:58.263164+0000","last_clean":"2024-01-16T15:14:58.263164+0000","last_became_active":"2024-01-16T15:14:57.286264+0000","last_became_peered":"2024-01-16T15:14:57.286264+0000","last_unstale":"2024-01-16T15:14:58.263164+0000","last_undegraded":"2024-01-16T15:14:58.263164+0000","last_fullsized":"2024-01-16T15:14:58.263164+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:11.979730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"100'8","reported_seq":20,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.286645+0000","last_change":"2024-01-16T15:14:56.254042+0000","last_active":"2024-01-16T15:14:57.286645+0000","last_peered":"2024-01-16T15:14:57.286645+0000","last_clean":"2024-01-16T15:14:57.286645+0000","last_became_active":"2024-01-16T15:14:56.253721+0000","last_became_peered":"2024-01-16T15:14:56.253721+0000","last_unstale":"2024-01-16T15:14:57.286645+0000","last_undegraded":"2024-01-16T15:14:57.286645+0000","last_fullsized":"2024-01-16T15:14:57.286645+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:42.424881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269616+0000","last_change":"2024-01-16T15:14:57.275824+0000","last_active":"2024-01-16T15:14:58.269616+0000","last_peered":"2024-01-16T15:14:58.269616+0000","last_clean":"2024-01-16T15:14:58.269616+0000","last_became_active":"2024-01-16T15:14:57.275379+0000","last_became_peered":"2024-01-16T15:14:57.275379+0000","last_unstale":"2024-01-16T15:14:58.269616+0000","last_undegraded":"2024-01-16T15:14:58.269616+0000","last_fullsized":"2024-01-16T15:14:58.269616+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:31.350537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288484+0000","last_change":"2024-01-16T15:14:56.247405+0000","last_active":"2024-01-16T15:14:57.288484+0000","last_peered":"2024-01-16T15:14:57.288484+0000","last_clean":"2024-01-16T15:14:57.288484+0000","last_became_active":"2024-01-16T15:14:56.247123+0000","last_became_peered":"2024-01-16T15:14:56.247123+0000","last_unstale":"2024-01-16T15:14:57.288484+0000","last_undegraded":"2024-01-16T15:14:57.288484+0000","last_fullsized":"2024-01-16T15:14:57.288484+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:21:12.250548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376994+0000","last_change":"2024-01-16T15:14:57.267134+0000","last_active":"2024-01-16T15:14:58.376994+0000","last_peered":"2024-01-16T15:14:58.376994+0000","last_clean":"2024-01-16T15:14:58.376994+0000","last_became_active":"2024-01-16T15:14:57.266832+0000","last_became_peered":"2024-01-16T15:14:57.266832+0000","last_unstale":"2024-01-16T15:14:58.376994+0000","last_undegraded":"2024-01-16T15:14:58.376994+0000","last_fullsized":"2024-01-16T15:14:58.376994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:29.973542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.266039+0000","last_change":"2024-01-16T15:14:56.241803+0000","last_active":"2024-01-16T15:14:57.266039+0000","last_peered":"2024-01-16T15:14:57.266039+0000","last_clean":"2024-01-16T15:14:57.266039+0000","last_became_active":"2024-01-16T15:14:56.240380+0000","last_became_peered":"2024-01-16T15:14:56.240380+0000","last_unstale":"2024-01-16T15:14:57.266039+0000","last_undegraded":"2024-01-16T15:14:57.266039+0000","last_fullsized":"2024-01-16T15:14:57.266039+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:22.270194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377591+0000","last_change":"2024-01-16T15:14:57.281920+0000","last_active":"2024-01-16T15:14:58.377591+0000","last_peered":"2024-01-16T15:14:58.377591+0000","last_clean":"2024-01-16T15:14:58.377591+0000","last_became_active":"2024-01-16T15:14:57.281480+0000","last_became_peered":"2024-01-16T15:14:57.281480+0000","last_unstale":"2024-01-16T15:14:58.377591+0000","last_undegraded":"2024-01-16T15:14:58.377591+0000","last_fullsized":"2024-01-16T15:14:58.377591+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:54.127537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257517+0000","last_change":"2024-01-16T15:14:56.254269+0000","last_active":"2024-01-16T15:14:57.257517+0000","last_peered":"2024-01-16T15:14:57.257517+0000","last_clean":"2024-01-16T15:14:57.257517+0000","last_became_active":"2024-01-16T15:14:56.246024+0000","last_became_peered":"2024-01-16T15:14:56.246024+0000","last_unstale":"2024-01-16T15:14:57.257517+0000","last_undegraded":"2024-01-16T15:14:57.257517+0000","last_fullsized":"2024-01-16T15:14:57.257517+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:40.497506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263758+0000","last_change":"2024-01-16T15:14:57.276053+0000","last_active":"2024-01-16T15:14:58.263758+0000","last_peered":"2024-01-16T15:14:58.263758+0000","last_clean":"2024-01-16T15:14:58.263758+0000","last_became_active":"2024-01-16T15:14:57.275755+0000","last_became_peered":"2024-01-16T15:14:57.275755+0000","last_unstale":"2024-01-16T15:14:58.263758+0000","last_undegraded":"2024-01-16T15:14:58.263758+0000","last_fullsized":"2024-01-16T15:14:58.263758+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:23.192004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288743+0000","last_change":"2024-01-16T15:14:56.255721+0000","last_active":"2024-01-16T15:14:57.288743+0000","last_peered":"2024-01-16T15:14:57.288743+0000","last_clean":"2024-01-16T15:14:57.288743+0000","last_became_active":"2024-01-16T15:14:56.250495+0000","last_became_peered":"2024-01-16T15:14:56.250495+0000","last_unstale":"2024-01-16T15:14:57.288743+0000","last_undegraded":"2024-01-16T15:14:57.288743+0000","last_fullsized":"2024-01-16T15:14:57.288743+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:58.630719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264108+0000","last_change":"2024-01-16T15:14:57.275382+0000","last_active":"2024-01-16T15:14:58.264108+0000","last_peered":"2024-01-16T15:14:58.264108+0000","last_clean":"2024-01-16T15:14:58.264108+0000","last_became_active":"2024-01-16T15:14:57.275125+0000","last_became_peered":"2024-01-16T15:14:57.275125+0000","last_unstale":"2024-01-16T15:14:58.264108+0000","last_undegraded":"2024-01-16T15:14:58.264108+0000","last_fullsized":"2024-01-16T15:14:58.264108+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:39:34.553063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.247724+0000","last_change":"2024-01-16T15:14:56.260376+0000","last_active":"2024-01-16T15:14:57.247724+0000","last_peered":"2024-01-16T15:14:57.247724+0000","last_clean":"2024-01-16T15:14:57.247724+0000","last_became_active":"2024-01-16T15:14:56.260078+0000","last_became_peered":"2024-01-16T15:14:56.260078+0000","last_unstale":"2024-01-16T15:14:57.247724+0000","last_undegraded":"2024-01-16T15:14:57.247724+0000","last_fullsized":"2024-01-16T15:14:57.247724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:21.232134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263099+0000","last_change":"2024-01-16T15:14:57.281739+0000","last_active":"2024-01-16T15:14:58.263099+0000","last_peered":"2024-01-16T15:14:58.263099+0000","last_clean":"2024-01-16T15:14:58.263099+0000","last_became_active":"2024-01-16T15:14:57.280568+0000","last_became_peered":"2024-01-16T15:14:57.280568+0000","last_unstale":"2024-01-16T15:14:58.263099+0000","last_undegraded":"2024-01-16T15:14:58.263099+0000","last_fullsized":"2024-01-16T15:14:58.263099+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:08.863102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.247300+0000","last_change":"2024-01-16T15:14:56.250720+0000","last_active":"2024-01-16T15:14:57.247300+0000","last_peered":"2024-01-16T15:14:57.247300+0000","last_clean":"2024-01-16T15:14:57.247300+0000","last_became_active":"2024-01-16T15:14:56.250425+0000","last_became_peered":"2024-01-16T15:14:56.250425+0000","last_unstale":"2024-01-16T15:14:57.247300+0000","last_undegraded":"2024-01-16T15:14:57.247300+0000","last_fullsized":"2024-01-16T15:14:57.247300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:54.535564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267577+0000","last_change":"2024-01-16T15:14:57.271475+0000","last_active":"2024-01-16T15:14:58.267577+0000","last_peered":"2024-01-16T15:14:58.267577+0000","last_clean":"2024-01-16T15:14:58.267577+0000","last_became_active":"2024-01-16T15:14:57.269885+0000","last_became_peered":"2024-01-16T15:14:57.269885+0000","last_unstale":"2024-01-16T15:14:58.267577+0000","last_undegraded":"2024-01-16T15:14:58.267577+0000","last_fullsized":"2024-01-16T15:14:58.267577+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:29.843051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.256714+0000","last_change":"2024-01-16T15:14:56.259100+0000","last_active":"2024-01-16T15:14:57.256714+0000","last_peered":"2024-01-16T15:14:57.256714+0000","last_clean":"2024-01-16T15:14:57.256714+0000","last_became_active":"2024-01-16T15:14:56.258797+0000","last_became_peered":"2024-01-16T15:14:56.258797+0000","last_unstale":"2024-01-16T15:14:57.256714+0000","last_undegraded":"2024-01-16T15:14:57.256714+0000","last_fullsized":"2024-01-16T15:14:57.256714+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:28.827639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376643+0000","last_change":"2024-01-16T15:14:57.262723+0000","last_active":"2024-01-16T15:14:58.376643+0000","last_peered":"2024-01-16T15:14:58.376643+0000","last_clean":"2024-01-16T15:14:58.376643+0000","last_became_active":"2024-01-16T15:14:57.262413+0000","last_became_peered":"2024-01-16T15:14:57.262413+0000","last_unstale":"2024-01-16T15:14:58.376643+0000","last_undegraded":"2024-01-16T15:14:58.376643+0000","last_fullsized":"2024-01-16T15:14:58.376643+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:37.124421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.283433+0000","last_change":"2024-01-16T15:14:56.250815+0000","last_active":"2024-01-16T15:14:57.283433+0000","last_peered":"2024-01-16T15:14:57.283433+0000","last_clean":"2024-01-16T15:14:57.283433+0000","last_became_active":"2024-01-16T15:14:56.250559+0000","last_became_peered":"2024-01-16T15:14:56.250559+0000","last_unstale":"2024-01-16T15:14:57.283433+0000","last_undegraded":"2024-01-16T15:14:57.283433+0000","last_fullsized":"2024-01-16T15:14:57.283433+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:06.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375590+0000","last_change":"2024-01-16T15:14:57.278308+0000","last_active":"2024-01-16T15:14:58.375590+0000","last_peered":"2024-01-16T15:14:58.375590+0000","last_clean":"2024-01-16T15:14:58.375590+0000","last_became_active":"2024-01-16T15:14:57.277548+0000","last_became_peered":"2024-01-16T15:14:57.277548+0000","last_unstale":"2024-01-16T15:14:58.375590+0000","last_undegraded":"2024-01-16T15:14:58.375590+0000","last_fullsized":"2024-01-16T15:14:58.375590+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:07.179160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257987+0000","last_change":"2024-01-16T15:14:56.253570+0000","last_active":"2024-01-16T15:14:57.257987+0000","last_peered":"2024-01-16T15:14:57.257987+0000","last_clean":"2024-01-16T15:14:57.257987+0000","last_became_active":"2024-01-16T15:14:56.249988+0000","last_became_peered":"2024-01-16T15:14:56.249988+0000","last_unstale":"2024-01-16T15:14:57.257987+0000","last_undegraded":"2024-01-16T15:14:57.257987+0000","last_fullsized":"2024-01-16T15:14:57.257987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:12.211734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376795+0000","last_change":"2024-01-16T15:14:57.271437+0000","last_active":"2024-01-16T15:14:58.376795+0000","last_peered":"2024-01-16T15:14:58.376795+0000","last_clean":"2024-01-16T15:14:58.376795+0000","last_became_active":"2024-01-16T15:14:57.270628+0000","last_became_peered":"2024-01-16T15:14:57.270628+0000","last_unstale":"2024-01-16T15:14:58.376795+0000","last_undegraded":"2024-01-16T15:14:58.376795+0000","last_fullsized":"2024-01-16T15:14:58.376795+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:29.973542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257309+0000","last_change":"2024-01-16T15:14:56.253017+0000","last_active":"2024-01-16T15:14:57.257309+0000","last_peered":"2024-01-16T15:14:57.257309+0000","last_clean":"2024-01-16T15:14:57.257309+0000","last_became_active":"2024-01-16T15:14:56.252713+0000","last_became_peered":"2024-01-16T15:14:56.252713+0000","last_unstale":"2024-01-16T15:14:57.257309+0000","last_undegraded":"2024-01-16T15:14:57.257309+0000","last_fullsized":"2024-01-16T15:14:57.257309+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:23.441321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.268721+0000","last_change":"2024-01-16T15:14:57.255863+0000","last_active":"2024-01-16T15:14:58.268721+0000","last_peered":"2024-01-16T15:14:58.268721+0000","last_clean":"2024-01-16T15:14:58.268721+0000","last_became_active":"2024-01-16T15:14:57.255566+0000","last_became_peered":"2024-01-16T15:14:57.255566+0000","last_unstale":"2024-01-16T15:14:58.268721+0000","last_undegraded":"2024-01-16T15:14:58.268721+0000","last_fullsized":"2024-01-16T15:14:58.268721+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:24.456733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.252495+0000","last_change":"2024-01-16T15:14:56.245838+0000","last_active":"2024-01-16T15:14:57.252495+0000","last_peered":"2024-01-16T15:14:57.252495+0000","last_clean":"2024-01-16T15:14:57.252495+0000","last_became_active":"2024-01-16T15:14:56.243709+0000","last_became_peered":"2024-01-16T15:14:56.243709+0000","last_unstale":"2024-01-16T15:14:57.252495+0000","last_undegraded":"2024-01-16T15:14:57.252495+0000","last_fullsized":"2024-01-16T15:14:57.252495+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:58.630719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263953+0000","last_change":"2024-01-16T15:14:57.285072+0000","last_active":"2024-01-16T15:14:58.263953+0000","last_peered":"2024-01-16T15:14:58.263953+0000","last_clean":"2024-01-16T15:14:58.263953+0000","last_became_active":"2024-01-16T15:14:57.284598+0000","last_became_peered":"2024-01-16T15:14:57.284598+0000","last_unstale":"2024-01-16T15:14:58.263953+0000","last_undegraded":"2024-01-16T15:14:58.263953+0000","last_fullsized":"2024-01-16T15:14:58.263953+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:44.684310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.256403+0000","last_change":"2024-01-16T15:14:56.241536+0000","last_active":"2024-01-16T15:14:57.256403+0000","last_peered":"2024-01-16T15:14:57.256403+0000","last_clean":"2024-01-16T15:14:57.256403+0000","last_became_active":"2024-01-16T15:14:56.241265+0000","last_became_peered":"2024-01-16T15:14:56.241265+0000","last_unstale":"2024-01-16T15:14:57.256403+0000","last_undegraded":"2024-01-16T15:14:57.256403+0000","last_fullsized":"2024-01-16T15:14:57.256403+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:22.270194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265621+0000","last_change":"2024-01-16T15:14:57.268919+0000","last_active":"2024-01-16T15:14:58.265621+0000","last_peered":"2024-01-16T15:14:58.265621+0000","last_clean":"2024-01-16T15:14:58.265621+0000","last_became_active":"2024-01-16T15:14:57.268601+0000","last_became_peered":"2024-01-16T15:14:57.268601+0000","last_unstale":"2024-01-16T15:14:58.265621+0000","last_undegraded":"2024-01-16T15:14:58.265621+0000","last_fullsized":"2024-01-16T15:14:58.265621+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:25.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.251818+0000","last_change":"2024-01-16T15:14:56.257392+0000","last_active":"2024-01-16T15:14:57.251818+0000","last_peered":"2024-01-16T15:14:57.251818+0000","last_clean":"2024-01-16T15:14:57.251818+0000","last_became_active":"2024-01-16T15:14:56.256861+0000","last_became_peered":"2024-01-16T15:14:56.256861+0000","last_unstale":"2024-01-16T15:14:57.251818+0000","last_undegraded":"2024-01-16T15:14:57.251818+0000","last_fullsized":"2024-01-16T15:14:57.251818+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:17.250545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263300+0000","last_change":"2024-01-16T15:14:57.285776+0000","last_active":"2024-01-16T15:14:58.263300+0000","last_peered":"2024-01-16T15:14:58.263300+0000","last_clean":"2024-01-16T15:14:58.263300+0000","last_became_active":"2024-01-16T15:14:57.285440+0000","last_became_peered":"2024-01-16T15:14:57.285440+0000","last_unstale":"2024-01-16T15:14:58.263300+0000","last_undegraded":"2024-01-16T15:14:58.263300+0000","last_fullsized":"2024-01-16T15:14:58.263300+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:31.070337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.248936+0000","last_change":"2024-01-16T15:14:56.257589+0000","last_active":"2024-01-16T15:14:57.248936+0000","last_peered":"2024-01-16T15:14:57.248936+0000","last_clean":"2024-01-16T15:14:57.248936+0000","last_became_active":"2024-01-16T15:14:56.257240+0000","last_became_peered":"2024-01-16T15:14:56.257240+0000","last_unstale":"2024-01-16T15:14:57.248936+0000","last_undegraded":"2024-01-16T15:14:57.248936+0000","last_fullsized":"2024-01-16T15:14:57.248936+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:44:09.006145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267922+0000","last_change":"2024-01-16T15:14:57.255621+0000","last_active":"2024-01-16T15:14:58.267922+0000","last_peered":"2024-01-16T15:14:58.267922+0000","last_clean":"2024-01-16T15:14:58.267922+0000","last_became_active":"2024-01-16T15:14:57.255297+0000","last_became_peered":"2024-01-16T15:14:57.255297+0000","last_unstale":"2024-01-16T15:14:58.267922+0000","last_undegraded":"2024-01-16T15:14:58.267922+0000","last_fullsized":"2024-01-16T15:14:58.267922+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:15.656662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.264990+0000","last_change":"2024-01-16T15:14:56.251203+0000","last_active":"2024-01-16T15:14:57.264990+0000","last_peered":"2024-01-16T15:14:57.264990+0000","last_clean":"2024-01-16T15:14:57.264990+0000","last_became_active":"2024-01-16T15:14:56.250531+0000","last_became_peered":"2024-01-16T15:14:56.250531+0000","last_unstale":"2024-01-16T15:14:57.264990+0000","last_undegraded":"2024-01-16T15:14:57.264990+0000","last_fullsized":"2024-01-16T15:14:57.264990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:33:59.694443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264464+0000","last_change":"2024-01-16T15:14:57.269312+0000","last_active":"2024-01-16T15:14:58.264464+0000","last_peered":"2024-01-16T15:14:58.264464+0000","last_clean":"2024-01-16T15:14:58.264464+0000","last_became_active":"2024-01-16T15:14:57.268972+0000","last_became_peered":"2024-01-16T15:14:57.268972+0000","last_unstale":"2024-01-16T15:14:58.264464+0000","last_undegraded":"2024-01-16T15:14:58.264464+0000","last_fullsized":"2024-01-16T15:14:58.264464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:34.652244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.272405+0000","last_change":"2024-01-16T15:14:56.252064+0000","last_active":"2024-01-16T15:14:57.272405+0000","last_peered":"2024-01-16T15:14:57.272405+0000","last_clean":"2024-01-16T15:14:57.272405+0000","last_became_active":"2024-01-16T15:14:56.251632+0000","last_became_peered":"2024-01-16T15:14:56.251632+0000","last_unstale":"2024-01-16T15:14:57.272405+0000","last_undegraded":"2024-01-16T15:14:57.272405+0000","last_fullsized":"2024-01-16T15:14:57.272405+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:57.674413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":235,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.262940+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:14:57.262940+0000","last_peered":"2024-01-16T15:14:57.262940+0000","last_clean":"2024-01-16T15:14:57.262940+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:14:57.262940+0000","last_undegraded":"2024-01-16T15:14:57.262940+0000","last_fullsized":"2024-01-16T15:14:57.262940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542335,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73180,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298660,"statfs":{"total":96636764160,"available":96561827840,"internally_reserved":0,"allocated":1847296,"data_stored":1128209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.4359999999999999,"15min":1.486},"min":{"1min":0.63500000000000001,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":2.0779999999999998,"5min":15.449999999999999,"15min":27.469000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.7909999999999999,"15min":1.6319999999999999},"min":{"1min":0.79500000000000004,"5min":0.60299999999999998,"15min":0.47099999999999997},"max":{"1min":2.3069999999999999,"5min":25.503,"15min":25.503},"last":1.605}]},{"osd":1,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1899999999999999,"15min":1.2909999999999999},"min":{"1min":0.55100000000000005,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.679,"5min":5.5010000000000003,"15min":6.718},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.21,"15min":1.347},"min":{"1min":0.75700000000000001,"5min":0.64700000000000002,"15min":0.373},"max":{"1min":2.0110000000000001,"5min":4.3369999999999997,"15min":15.452},"last":1.571}]},{"osd":2,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.6040000000000001,"15min":1.4970000000000001},"min":{"1min":0.81999999999999995,"5min":0.629,"15min":0.48299999999999998},"max":{"1min":11.536,"5min":11.536,"15min":11.536},"last":1.347},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.899,"15min":1.5820000000000001},"min":{"1min":1.0740000000000001,"5min":0.64400000000000002,"15min":0.504},"max":{"1min":11.576000000000001,"5min":25.478000000000002,"15min":25.478000000000002},"last":1.3899999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.4830000000000001,"15min":1.571},"min":{"1min":0.67000000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.7789999999999999,"5min":13.907999999999999,"15min":13.907999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.526,"15min":1.5680000000000001},"min":{"1min":0.93200000000000005,"5min":0.66900000000000004,"15min":0.62},"max":{"1min":2.032,"5min":8.6999999999999993,"15min":13.193},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.764,"15min":1.496},"min":{"1min":0.90900000000000003,"5min":0.47799999999999998,"15min":0.376},"max":{"1min":1.742,"5min":25.401,"15min":25.401},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.59599999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.645,"5min":6.7000000000000002,"15min":10.134},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.5880000000000001,"15min":1.4990000000000001},"min":{"1min":0.69399999999999995,"5min":0.432,"15min":0.432},"max":{"1min":2.1259999999999999,"5min":12.528,"15min":12.528},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.802,"15min":1.6160000000000001},"min":{"1min":0.92900000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.8420000000000001,"5min":26.327000000000002,"15min":26.327000000000002},"last":1.5109999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.994,"15min":1.73},"min":{"1min":1.087,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.8520000000000001,"5min":25.451000000000001,"15min":25.451000000000001},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.663,"15min":1.6499999999999999},"min":{"1min":0.89800000000000002,"5min":0.60199999999999998,"15min":0.44700000000000001},"max":{"1min":1.8080000000000001,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.55}]}]},{"osd":6,"up_from":14,"seq":60129542334,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":1253376,"data_stored":538381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2030000000000001,"15min":1.6539999999999999},"min":{"1min":0.73699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.8120000000000001,"5min":4.6600000000000001,"15min":30.164999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3029999999999999,"15min":1.732},"min":{"1min":0.83799999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.8560000000000001,"5min":4.8150000000000004,"15min":30.280999999999999},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.292,"15min":1.482},"min":{"1min":0.65800000000000003,"5min":0.54100000000000004,"15min":0.47899999999999998},"max":{"1min":1.891,"5min":5.9859999999999998,"15min":28.122},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.71699999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":3.7069999999999999,"5min":7.7489999999999997,"15min":8.9719999999999995},"last":0.79000000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.5840000000000001,"15min":1.7},"min":{"1min":0.90900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":1.673,"5min":11.869,"15min":26.109000000000002},"last":1.169},{"interface":"front","average":{"1min":1.446,"5min":1.726,"15min":1.8500000000000001},"min":{"1min":1.107,"5min":0.71099999999999997,"15min":0.65800000000000003},"max":{"1min":1.952,"5min":12.179,"15min":26.161000000000001},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4119999999999999,"15min":1.609},"min":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.44700000000000001},"max":{"1min":1.7490000000000001,"5min":9.0199999999999996,"15min":17.75},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.538,"15min":1.6200000000000001},"min":{"1min":0.77100000000000002,"5min":0.623,"15min":0.55700000000000005},"max":{"1min":2.028,"5min":9.0660000000000007,"15min":17.794},"last":1.224}]},{"osd":4,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4530000000000001,"15min":1.546},"min":{"1min":0.94899999999999995,"5min":0.45500000000000002,"15min":0.45000000000000001},"max":{"1min":2.2639999999999998,"5min":6.2220000000000004,"15min":15.571},"last":1.331},{"interface":"front","average":{"1min":1.423,"5min":1.327,"15min":1.5740000000000001},"min":{"1min":0.80300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.9510000000000001,"5min":5.0330000000000004,"15min":23.667999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.5249999999999999,"15min":1.46},"min":{"1min":0.86599999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1829999999999998,"5min":23.137,"15min":23.137},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.7290000000000001,"15min":1.6779999999999999},"min":{"1min":1.2609999999999999,"5min":0.48099999999999998,"15min":0.47099999999999997},"max":{"1min":2.1200000000000001,"5min":25.684000000000001,"15min":25.684000000000001},"last":1.282}]},{"osd":7,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.48,"15min":1.518},"min":{"1min":1.036,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.4500000000000002,"5min":5.1239999999999997,"15min":11.597},"last":1.359},{"interface":"front","average":{"1min":1.653,"5min":1.6890000000000001,"15min":1.6910000000000001},"min":{"1min":1.1930000000000001,"5min":0.61599999999999999,"15min":0.499},"max":{"1min":2.375,"5min":6.2770000000000001,"15min":11.788},"last":1.4019999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80920,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290920,"statfs":{"total":96636764160,"available":96553902080,"internally_reserved":0,"allocated":1249280,"data_stored":537799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.349,"15min":1.579},"min":{"1min":0.61399999999999999,"5min":0.41299999999999998,"15min":0.39000000000000001},"max":{"1min":1.6479999999999999,"5min":22.289000000000001,"15min":26.228999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.1639999999999999,"15min":1.845},"min":{"1min":0.73799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.7889999999999999,"5min":5.4290000000000003,"15min":56.011000000000003},"last":0.93000000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.179,"15min":1.375},"min":{"1min":0.69899999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":4.2400000000000002,"15min":16.687999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.7250000000000001,"15min":1.913},"min":{"1min":0.83099999999999996,"5min":0.46999999999999997,"15min":0.40999999999999998},"max":{"1min":1.577,"5min":22.451000000000001,"15min":67.930000000000007},"last":1.589}]},{"osd":2,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.325,"15min":1.55},"min":{"1min":0.88800000000000001,"5min":0.54800000000000004,"15min":0.47099999999999997},"max":{"1min":1.5700000000000001,"5min":5.4950000000000001,"15min":13.193},"last":1.05},{"interface":"front","average":{"1min":1.375,"5min":1.454,"15min":1.5},"min":{"1min":0.80000000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":3.9590000000000001,"5min":16.721,"15min":16.721},"last":1.3600000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.5960000000000001,"15min":1.7889999999999999},"min":{"1min":1.0189999999999999,"5min":0.59599999999999997,"15min":0.57999999999999996},"max":{"1min":1.7410000000000001,"5min":22.382000000000001,"15min":22.382000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.5089999999999999,"15min":1.8080000000000001},"min":{"1min":0.89900000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.915,"5min":6.0540000000000003,"15min":14.685},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4259999999999999,"15min":1.456},"min":{"1min":0.621,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.6759999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.619,"15min":1.581},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.379},"max":{"1min":1.948,"5min":27.006,"15min":27.006},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.454,"15min":1.5820000000000001},"min":{"1min":0.72799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.9339999999999999,"5min":5.1230000000000002,"15min":13.263999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.641},"min":{"1min":0.82099999999999995,"5min":0.57599999999999996,"15min":0.505},"max":{"1min":1.9099999999999999,"5min":5.9710000000000001,"15min":9.5269999999999992},"last":1.619}]},{"osd":7,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.8520000000000001,"15min":1.802},"min":{"1min":0.78000000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.988,"5min":35.127000000000002,"15min":35.127000000000002},"last":1.649},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6120000000000001,"15min":1.79},"min":{"1min":0.94999999999999996,"5min":0.55800000000000005,"15min":0.42999999999999999},"max":{"1min":2.0459999999999998,"5min":5.1959999999999997,"15min":13.491},"last":1.4139999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542334,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64296,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307544,"statfs":{"total":96636764160,"available":96570925056,"internally_reserved":0,"allocated":1265664,"data_stored":539465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.103,"15min":1.2509999999999999},"min":{"1min":0.628,"5min":0.51900000000000002,"15min":0.502},"max":{"1min":2.23,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.53},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.129,"15min":1.4830000000000001},"min":{"1min":0.621,"5min":0.54400000000000004,"15min":0.501},"max":{"1min":1.917,"5min":1.917,"15min":60.106000000000002},"last":1.702}]},{"osd":1,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.379,"15min":1.6919999999999999},"min":{"1min":0.89400000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8420000000000001,"5min":3.3029999999999999,"15min":22.143000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.369,"15min":1.5369999999999999},"min":{"1min":0.76000000000000001,"5min":0.628,"15min":0.57099999999999995},"max":{"1min":1.7849999999999999,"5min":6.3719999999999999,"15min":22.181999999999999},"last":0.91600000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.294,"15min":1.319},"min":{"1min":0.82599999999999996,"5min":0.58099999999999996,"15min":0.52800000000000002},"max":{"1min":2.008,"5min":6.9480000000000004,"15min":9.7140000000000004},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3109999999999999,"15min":1.73},"min":{"1min":0.89900000000000002,"5min":0.63100000000000001,"15min":0.52400000000000002},"max":{"1min":1.752,"5min":3.1720000000000002,"15min":35.024999999999999},"last":1.486}]},{"osd":3,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3360000000000001,"15min":1.5820000000000001},"min":{"1min":0.85899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.8879999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.631},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.387,"15min":1.391},"min":{"1min":1.0069999999999999,"5min":0.623,"15min":0.57999999999999996},"max":{"1min":2.165,"5min":5.4690000000000003,"15min":7.8140000000000001},"last":1.413}]},{"osd":5,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.385,"15min":1.4810000000000001},"min":{"1min":0.94699999999999995,"5min":0.47199999999999998,"15min":0.38900000000000001},"max":{"1min":2.0600000000000001,"5min":2.1299999999999999,"15min":14.568},"last":1.95},{"interface":"front","average":{"1min":1.552,"5min":1.4510000000000001,"15min":1.4630000000000001},"min":{"1min":1.016,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.3570000000000002,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":1.304}]},{"osd":6,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.395,"15min":1.4019999999999999},"min":{"1min":0.89300000000000002,"5min":0.49099999999999999,"15min":0.436},"max":{"1min":2.4350000000000001,"5min":3.8929999999999998,"15min":9.6630000000000003},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.585,"15min":1.6679999999999999},"min":{"1min":1.004,"5min":0.79200000000000004,"15min":0.45100000000000001},"max":{"1min":2.3980000000000001,"5min":8.8650000000000002,"15min":14.317},"last":1.869}]},{"osd":7,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.647,"15min":1.583},"min":{"1min":1.073,"5min":0.61099999999999999,"15min":0.52100000000000002},"max":{"1min":2.9319999999999999,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6679999999999999,"15min":1.5920000000000001},"min":{"1min":1.1499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":2.3570000000000002,"5min":9.2270000000000003,"15min":9.2270000000000003},"last":1.764}]}]},{"osd":3,"up_from":14,"seq":60129542336,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76768,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295072,"statfs":{"total":96636764160,"available":96558153728,"internally_reserved":0,"allocated":1261568,"data_stored":539363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.1240000000000001,"15min":1.5169999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.9299999999999999,"5min":7.2229999999999999,"15min":50.988999999999997},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.4039999999999999,"15min":1.474},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5940000000000001,"5min":20.274000000000001,"15min":20.274000000000001},"last":1.4770000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.359,"15min":1.403},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.6070000000000002,"5min":13.576000000000001,"15min":18.192},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.5449999999999999,"15min":1.744},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":1.8839999999999999,"5min":12.696,"15min":20.385999999999999},"last":0.92400000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.865,"15min":1.7150000000000001},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.952,"5min":42.840000000000003,"15min":42.840000000000003},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.853,"15min":1.833},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.9550000000000001,"5min":38.707000000000001,"15min":38.707000000000001},"last":1.0529999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.6060000000000001,"15min":1.7669999999999999},"min":{"1min":0.78200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.327,"5min":14.148,"15min":18.952000000000002},"last":1.385},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.7090000000000001,"15min":1.7829999999999999},"min":{"1min":0.745,"5min":0.61599999999999999,"15min":0.54800000000000004},"max":{"1min":1.893,"5min":14.454000000000001,"15min":18.088999999999999},"last":1.512}]},{"osd":5,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.611,"15min":1.881},"min":{"1min":0.73199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.1509999999999998,"5min":5.7910000000000004,"15min":24.349},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.788,"15min":1.9390000000000001},"min":{"1min":0.753,"5min":0.66100000000000003,"15min":0.61899999999999999},"max":{"1min":2.5379999999999998,"5min":14.192,"15min":14.192},"last":1.5840000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.931,"15min":1.8620000000000001},"min":{"1min":0.77200000000000002,"5min":0.69699999999999995,"15min":0.52200000000000002},"max":{"1min":2.0449999999999999,"5min":22.687000000000001,"15min":22.687000000000001},"last":1.29},{"interface":"front","average":{"1min":1.512,"5min":1.716,"15min":1.9650000000000001},"min":{"1min":0.73799999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2949999999999999,"5min":6.1299999999999999,"15min":24.058},"last":1.54}]},{"osd":7,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":2.2200000000000002,"15min":2.0819999999999999},"min":{"1min":0.85099999999999998,"5min":0.76900000000000002,"15min":0.55800000000000005},"max":{"1min":2.464,"5min":35.570999999999998,"15min":35.570999999999998},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":2.359,"15min":2.0529999999999999},"min":{"1min":0.81100000000000005,"5min":0.64700000000000002,"15min":0.58899999999999997},"max":{"1min":2.0019999999999998,"5min":37.607999999999997,"15min":37.607999999999997},"last":1.4450000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76756,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295084,"statfs":{"total":96636764160,"available":96558166016,"internally_reserved":0,"allocated":1249280,"data_stored":537763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0980000000000001,"15min":1.488},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.24399999999999999},"max":{"1min":1.3939999999999999,"5min":11.112,"15min":64.387},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.3799999999999999,"15min":1.389},"min":{"1min":0.67000000000000004,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.514,"5min":24.308,"15min":24.308},"last":0.89500000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1639999999999999,"15min":1.464},"min":{"1min":0.66300000000000003,"5min":0.42299999999999999,"15min":0.28299999999999997},"max":{"1min":2.0579999999999998,"5min":11.406000000000001,"15min":27.263000000000002},"last":0.755},{"interface":"front","average":{"1min":1.409,"5min":1.7270000000000001,"15min":1.819},"min":{"1min":0.70999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.901,"5min":23.602,"15min":23.602},"last":0.71299999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.306,"15min":1.5920000000000001},"min":{"1min":0.755,"5min":0.53400000000000003,"15min":0.33100000000000002},"max":{"1min":1.9299999999999999,"5min":3.1080000000000001,"15min":27.311},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.421,"15min":1.649},"min":{"1min":0.86899999999999999,"5min":0.49099999999999999,"15min":0.47099999999999997},"max":{"1min":1.782,"5min":9.7390000000000008,"15min":26.555},"last":0.82999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5369999999999999,"15min":1.6739999999999999},"min":{"1min":1.0649999999999999,"5min":0.54000000000000004,"15min":0.38900000000000001},"max":{"1min":2.1040000000000001,"5min":12.779,"15min":26.503},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.833,"15min":1.7390000000000001},"min":{"1min":0.98999999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.4319999999999999,"5min":25.661000000000001,"15min":25.661000000000001},"last":1.294}]},{"osd":5,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.478,"15min":1.7290000000000001},"min":{"1min":0.81599999999999995,"5min":0.56699999999999995,"15min":0.45800000000000002},"max":{"1min":2.3490000000000002,"5min":3.0209999999999999,"15min":27.149000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.581,"5min":1.6200000000000001,"15min":1.7649999999999999},"min":{"1min":1.1739999999999999,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":2.6589999999999998,"5min":9.7850000000000001,"15min":26.349},"last":1.2789999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":2.0590000000000002,"15min":1.9399999999999999},"min":{"1min":1.0189999999999999,"5min":0.504,"15min":0.504},"max":{"1min":2.7149999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":1.073},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.5640000000000001,"15min":1.899},"min":{"1min":1.0209999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":15.968999999999999,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.137}]},{"osd":7,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":2,"15min":1.9510000000000001},"min":{"1min":1.036,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.4889999999999999,"5min":24.509,"15min":24.509},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.617,"5min":1.73,"15min":1.9159999999999999},"min":{"1min":1.089,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.3479999999999999,"5min":10.455,"15min":27.207999999999998},"last":1.3420000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542334,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76752,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295088,"statfs":{"total":96636764160,"available":96558170112,"internally_reserved":0,"allocated":1245184,"data_stored":537729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0309999999999999,"15min":1.2689999999999999},"min":{"1min":0.61199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.8560000000000001,"5min":6.2249999999999996,"15min":14.407},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.075,"5min":1.1739999999999999,"15min":1.306},"min":{"1min":0.66100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.3999999999999999,"5min":3.5699999999999998,"15min":13.930999999999999},"last":1.4239999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.238,"15min":1.3899999999999999},"min":{"1min":0.57799999999999996,"5min":0.31,"15min":0.31},"max":{"1min":14.269,"5min":14.269,"15min":16.488},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.421,"15min":1.702},"min":{"1min":0.53000000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4530000000000001,"5min":21.446000000000002,"15min":22.550999999999998},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.647,"15min":1.7649999999999999},"min":{"1min":0.63400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.9510000000000001,"5min":15.436,"15min":22.471},"last":1.323},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.504,"15min":1.653},"min":{"1min":0.77500000000000002,"5min":0.46600000000000003,"15min":0.35699999999999998},"max":{"1min":1.9610000000000001,"5min":14.763999999999999,"15min":22.550999999999998},"last":1.5029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.52,"15min":1.6699999999999999},"min":{"1min":0.96799999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.863,"5min":5.3789999999999996,"15min":12.675000000000001},"last":1.296},{"interface":"front","average":{"1min":1.613,"5min":1.5900000000000001,"15min":1.728},"min":{"1min":1.036,"5min":0.433,"15min":0.433},"max":{"1min":5.1619999999999999,"5min":5.6849999999999996,"15min":19.236999999999998},"last":1.3580000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.79,"15min":1.7629999999999999},"min":{"1min":1.155,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.2040000000000002,"5min":21.117000000000001,"15min":21.117000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.651,"15min":1.744},"min":{"1min":1.099,"5min":0.65800000000000003,"15min":0.64800000000000002},"max":{"1min":5.2359999999999998,"5min":5.9009999999999998,"15min":17.138000000000002},"last":1.2649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.595,"15min":1.8440000000000001},"min":{"1min":1.0169999999999999,"5min":0.79800000000000004,"15min":0.72199999999999998},"max":{"1min":2.298,"5min":6.3710000000000004,"15min":14.781000000000001},"last":1.194},{"interface":"front","average":{"1min":1.671,"5min":1.663,"15min":1.849},"min":{"1min":1.2310000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2719999999999998,"5min":5.4630000000000001,"15min":23.420999999999999},"last":1.6080000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7190000000000001,"15min":1.8280000000000001},"min":{"1min":1.1619999999999999,"5min":0.71299999999999997,"15min":0.60699999999999998},"max":{"1min":5.2910000000000004,"5min":5.8300000000000001,"15min":17.010000000000002},"last":1.147},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.7609999999999999,"15min":1.9730000000000001},"min":{"1min":0.91600000000000004,"5min":0.78700000000000003,"15min":0.75700000000000001},"max":{"1min":2.1539999999999999,"5min":10.316000000000001,"15min":14.884},"last":1.472}]}]},{"osd":0,"up_from":14,"seq":60129542335,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69012,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302828,"statfs":{"total":96636764160,"available":96566095872,"internally_reserved":0,"allocated":1843200,"data_stored":1128679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":68811201},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.48,"15min":1.494},"min":{"1min":0.47399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.8380000000000001,"5min":43.459000000000003,"15min":43.459000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.7609999999999999,"15min":1.6739999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":4.9779999999999998,"5min":43.627000000000002,"15min":43.627000000000002},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2649999999999999,"15min":1.4359999999999999},"min":{"1min":0.55100000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.0570000000000004,"5min":7.5940000000000003,"15min":18.861000000000001},"last":1.341},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.353,"15min":1.5069999999999999},"min":{"1min":0.49099999999999999,"5min":0.42999999999999999,"15min":0.378},"max":{"1min":5.3550000000000004,"5min":8.9979999999999993,"15min":38.893000000000001},"last":0.95299999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.135,"15min":1.5880000000000001},"min":{"1min":0.77100000000000002,"5min":0.45300000000000001,"15min":0.31900000000000001},"max":{"1min":4.9050000000000002,"5min":4.9050000000000002,"15min":23.687999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.417,"5min":1.516,"15min":1.6659999999999999},"min":{"1min":0.624,"5min":0.434,"15min":0.434},"max":{"1min":5.1980000000000004,"5min":16.978000000000002,"15min":18.966999999999999},"last":1.569}]},{"osd":4,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4039999999999999,"15min":1.524},"min":{"1min":0.96999999999999997,"5min":0.63400000000000001,"15min":0.58899999999999997},"max":{"1min":5.1040000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.536},{"interface":"front","average":{"1min":1.607,"5min":1.492,"15min":1.718},"min":{"1min":0.83099999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":5.3109999999999999,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.7989999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.6279999999999999,"15min":1.861},"min":{"1min":0.80700000000000005,"5min":0.55600000000000005,"15min":0.54600000000000004},"max":{"1min":5.1589999999999998,"5min":13.016,"15min":24.611000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.633,"15min":1.7170000000000001},"min":{"1min":0.871,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":7.2699999999999996,"5min":8.3949999999999996,"15min":20.199000000000002},"last":1.2669999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.7210000000000001,"15min":2.0539999999999998},"min":{"1min":1.079,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":6.3339999999999996,"5min":10.342000000000001,"15min":26.998000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.718,"15min":1.982},"min":{"1min":1.0069999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":5.9690000000000003,"5min":10.279999999999999,"15min":24.478999999999999},"last":1.0609999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.734,"15min":1.901},"min":{"1min":0.91700000000000004,"5min":0.60899999999999999,"15min":0.58399999999999996},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.776,"5min":1.659,"15min":1.7569999999999999},"min":{"1min":1.165,"5min":0.65800000000000003,"15min":0.52800000000000002},"max":{"1min":6.149,"5min":6.149,"15min":20.279},"last":1.452}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:01.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:02.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.032+0000 7f5807e00640 1 Processor -- start 2024-01-16T15:15:02.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.033+0000 7f5807e00640 1 -- start start 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.034+0000 7f5807e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f58080b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.034+0000 7f5807e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58080cf0c0 con 0x7f580815d5b0 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.034+0000 7f5807e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58080b01b0 con 0x7f58080d14a0 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.034+0000 7f5807e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580815d990 con 0x7f58080d1900 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.035+0000 7f5806a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f580815d5b0 0x7f58080d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53090/0 (socket says 172.21.15.43:53090) 2024-01-16T15:15:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.035+0000 7f5806a00640 1 -- 172.21.15.43:0/4113574494 learned_addr learned my addr 172.21.15.43:0/4113574494 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.035+0000 7f5807400640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f58080b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.035+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722717564 0 0) 0x7f58080cf0c0 con 0x7f580815d5b0 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.035+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e8003520 con 0x7f580815d5b0 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18079359 0 0) 0x7f57e8003520 con 0x7f580815d5b0 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 msgr2=0x7f58080b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f58080b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 >> v1:172.21.15.69:6789/0 conn(0x7f58080d14a0 legacy=0x7f58080b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080ceb00 con 0x7f580815d5b0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f57f8001670 con 0x7f580815d5b0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 >> v1:172.21.15.43:6789/0 conn(0x7f580815d5b0 legacy=0x7f58080d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f57e8003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8003fe0 0x7f57e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57e8004d00 0x7f57e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580815d990 con 0x7f57e8004d00 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e800bb40 con 0x7f57e8003fe0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e800bd00 con 0x7f58080d1900 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.036+0000 7f5807400640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57e8004d00 0x7f57e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5806a00640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f57e8003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f57ffe00640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8003fe0 0x7f57e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5807400640 1 -- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 msgr2=0x7f57e8003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5807400640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58080d1900 0x7f57e8003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5807400640 1 -- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8003fe0 msgr2=0x7f57e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5807400640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8003fe0 0x7f57e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f5807400640 1 -- 172.21.15.43:0/4113574494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8003520 con 0x7f57e8004d00 2024-01-16T15:15:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.037+0000 7f57ffe00640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8003fe0 0x7f57e80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.038+0000 7f5807400640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57e8004d00 0x7f57e8005180 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f5800009fc0 tx=0x7f57e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2edb9bbfeecb41 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.038+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580002cc20 con 0x7f57e8004d00 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.038+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580000f040 con 0x7f57e8004d00 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.038+0000 7f5806000640 1 -- 172.21.15.43:0/4113574494 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800034440 con 0x7f57e8004d00 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 -- 172.21.15.43:0/4113574494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57e8004d00 msgr2=0x7f57e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 --2- 172.21.15.43:0/4113574494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57e8004d00 0x7f57e8005180 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f5800009fc0 tx=0x7f57e80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807400640 1 -- 172.21.15.43:0/4113574494 reap_dead start 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 -- 172.21.15.43:0/4113574494 shutdown_connections 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 -- 172.21.15.43:0/4113574494 >> 172.21.15.43:0/4113574494 conn(0x7f580809c3b0 msgr2=0x7f58080d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 -- 172.21.15.43:0/4113574494 shutdown_connections 2024-01-16T15:15:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.039+0000 7f5807e00640 1 -- 172.21.15.43:0/4113574494 wait complete. 2024-01-16T15:15:02.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.041+0000 7f5807e00640 1 Processor -- start 2024-01-16T15:15:02.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.041+0000 7f5807e00640 1 -- start start 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8004d00 0x7f5808159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f580815f730 0x7f580815fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58080b63f0 con 0x7f58080c88c0 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58081600f0 con 0x7f57e8004d00 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58080c9640 con 0x7f580815f730 2024-01-16T15:15:02.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54144/0 (socket says 172.21.15.43:54144) 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.042+0000 7f5807400640 1 -- 172.21.15.43:0/3388860087 learned_addr learned my addr 172.21.15.43:0/3388860087 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f57ffe00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f580815f730 0x7f580815fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5806a00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8004d00 0x7f5808159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 -- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f580815f730 msgr2=0x7f580815fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f580815f730 0x7f580815fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 -- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8004d00 msgr2=0x7f5808159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8004d00 0x7f5808159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 -- 172.21.15.43:0/3388860087 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080c9920 con 0x7f58080c88c0 2024-01-16T15:15:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f5807400640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f5800002230 tx=0x7f5800009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.043+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800034cd0 con 0x7f58080c88c0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.044+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800034e30 con 0x7f58080c88c0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.044+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800003bd0 con 0x7f58080c88c0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.044+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58080b5d30 con 0x7f58080c88c0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.044+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58080c92b0 con 0x7f58080c88c0 2024-01-16T15:15:02.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.045+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58080b6530 con 0x7f58080c88c0 2024-01-16T15:15:02.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.046+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f5800003d30 con 0x7f58080c88c0 2024-01-16T15:15:02.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.047+0000 7f57fea00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 0x7f57e006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.047+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58000a9d50 con 0x7f58080c88c0 2024-01-16T15:15:02.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.047+0000 7f5806a00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 0x7f57e006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.048+0000 7f5806a00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 0x7f57e006e590 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f57f80059f0 tx=0x7f57f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.051+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58080b6530 con 0x7f58080c88c0 2024-01-16T15:15:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.242+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58080d1d90 con 0x7f57e006c0d0 2024-01-16T15:15:02.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.264+0000 7f57fea00640 1 -- 172.21.15.43:0/3388860087 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286862 (secure 0 0 0) 0x7f57f8010860 con 0x7f57e006c0d0 2024-01-16T15:15:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:02.253 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:15:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 msgr2=0x7f57e006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 0x7f57e006e590 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f57f80059f0 tx=0x7f57f8009040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 msgr2=0x7f58080c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f5800002230 tx=0x7f5800009750 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 shutdown_connections 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f57e006c0d0 0x7f57e006e590 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58080c88c0 0x7f58080c8d70 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f580815f730 0x7f580815fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 --2- 172.21.15.43:0/3388860087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e8004d00 0x7f5808159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 >> 172.21.15.43:0/3388860087 conn(0x7f580809c3b0 msgr2=0x7f580806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 shutdown_connections 2024-01-16T15:15:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.271+0000 7f5807e00640 1 -- 172.21.15.43:0/3388860087 wait complete. 2024-01-16T15:15:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-01-16T15:15:01.194875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":586124,"kb_used_data":10952,"kb_used_omap":12,"kb_used_meta":575027,"kb_avail":754388596,"statfs":{"total":773094113280,"available":772493922304,"internally_reserved":0,"allocated":11214848,"data_stored":5487388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":588828204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.982515"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375451+0000","last_change":"2024-01-16T15:14:57.268738+0000","last_active":"2024-01-16T15:14:58.375451+0000","last_peered":"2024-01-16T15:14:58.375451+0000","last_clean":"2024-01-16T15:14:58.375451+0000","last_became_active":"2024-01-16T15:14:57.267639+0000","last_became_peered":"2024-01-16T15:14:57.267639+0000","last_unstale":"2024-01-16T15:14:58.375451+0000","last_undegraded":"2024-01-16T15:14:58.375451+0000","last_fullsized":"2024-01-16T15:14:58.375451+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:55.550976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376602+0000","last_change":"2024-01-16T15:14:57.277847+0000","last_active":"2024-01-16T15:14:58.376602+0000","last_peered":"2024-01-16T15:14:58.376602+0000","last_clean":"2024-01-16T15:14:58.376602+0000","last_became_active":"2024-01-16T15:14:57.277511+0000","last_became_peered":"2024-01-16T15:14:57.277511+0000","last_unstale":"2024-01-16T15:14:58.376602+0000","last_undegraded":"2024-01-16T15:14:58.376602+0000","last_fullsized":"2024-01-16T15:14:58.376602+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:04.129379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375490+0000","last_change":"2024-01-16T15:14:57.268421+0000","last_active":"2024-01-16T15:14:58.375490+0000","last_peered":"2024-01-16T15:14:58.375490+0000","last_clean":"2024-01-16T15:14:58.375490+0000","last_became_active":"2024-01-16T15:14:57.268158+0000","last_became_peered":"2024-01-16T15:14:57.268158+0000","last_unstale":"2024-01-16T15:14:58.375490+0000","last_undegraded":"2024-01-16T15:14:58.375490+0000","last_fullsized":"2024-01-16T15:14:58.375490+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:22.247546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377946+0000","last_change":"2024-01-16T15:14:57.276753+0000","last_active":"2024-01-16T15:14:58.377946+0000","last_peered":"2024-01-16T15:14:58.377946+0000","last_clean":"2024-01-16T15:14:58.377946+0000","last_became_active":"2024-01-16T15:14:57.272254+0000","last_became_peered":"2024-01-16T15:14:57.272254+0000","last_unstale":"2024-01-16T15:14:58.377946+0000","last_undegraded":"2024-01-16T15:14:58.377946+0000","last_fullsized":"2024-01-16T15:14:58.377946+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:58.689825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267888+0000","last_change":"2024-01-16T15:14:57.260787+0000","last_active":"2024-01-16T15:14:58.267888+0000","last_peered":"2024-01-16T15:14:58.267888+0000","last_clean":"2024-01-16T15:14:58.267888+0000","last_became_active":"2024-01-16T15:14:57.260490+0000","last_became_peered":"2024-01-16T15:14:57.260490+0000","last_unstale":"2024-01-16T15:14:58.267888+0000","last_undegraded":"2024-01-16T15:14:58.267888+0000","last_fullsized":"2024-01-16T15:14:58.267888+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:57.049352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376590+0000","last_change":"2024-01-16T15:14:57.276912+0000","last_active":"2024-01-16T15:14:58.376590+0000","last_peered":"2024-01-16T15:14:58.376590+0000","last_clean":"2024-01-16T15:14:58.376590+0000","last_became_active":"2024-01-16T15:14:57.276487+0000","last_became_peered":"2024-01-16T15:14:57.276487+0000","last_unstale":"2024-01-16T15:14:58.376590+0000","last_undegraded":"2024-01-16T15:14:58.376590+0000","last_fullsized":"2024-01-16T15:14:58.376590+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:09.470583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375735+0000","last_change":"2024-01-16T15:14:57.276540+0000","last_active":"2024-01-16T15:14:58.375735+0000","last_peered":"2024-01-16T15:14:58.375735+0000","last_clean":"2024-01-16T15:14:58.375735+0000","last_became_active":"2024-01-16T15:14:57.276244+0000","last_became_peered":"2024-01-16T15:14:57.276244+0000","last_unstale":"2024-01-16T15:14:58.375735+0000","last_undegraded":"2024-01-16T15:14:58.375735+0000","last_fullsized":"2024-01-16T15:14:58.375735+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:31.613245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375171+0000","last_change":"2024-01-16T15:14:57.269832+0000","last_active":"2024-01-16T15:14:58.375171+0000","last_peered":"2024-01-16T15:14:58.375171+0000","last_clean":"2024-01-16T15:14:58.375171+0000","last_became_active":"2024-01-16T15:14:57.269536+0000","last_became_peered":"2024-01-16T15:14:57.269536+0000","last_unstale":"2024-01-16T15:14:58.375171+0000","last_undegraded":"2024-01-16T15:14:58.375171+0000","last_fullsized":"2024-01-16T15:14:58.375171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:51:10.476996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266714+0000","last_change":"2024-01-16T15:14:57.275114+0000","last_active":"2024-01-16T15:14:58.266714+0000","last_peered":"2024-01-16T15:14:58.266714+0000","last_clean":"2024-01-16T15:14:58.266714+0000","last_became_active":"2024-01-16T15:14:57.274090+0000","last_became_peered":"2024-01-16T15:14:57.274090+0000","last_unstale":"2024-01-16T15:14:58.266714+0000","last_undegraded":"2024-01-16T15:14:58.266714+0000","last_fullsized":"2024-01-16T15:14:58.266714+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.150989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269239+0000","last_change":"2024-01-16T15:14:57.267010+0000","last_active":"2024-01-16T15:14:58.269239+0000","last_peered":"2024-01-16T15:14:58.269239+0000","last_clean":"2024-01-16T15:14:58.269239+0000","last_became_active":"2024-01-16T15:14:57.266733+0000","last_became_peered":"2024-01-16T15:14:57.266733+0000","last_unstale":"2024-01-16T15:14:58.269239+0000","last_undegraded":"2024-01-16T15:14:58.269239+0000","last_fullsized":"2024-01-16T15:14:58.269239+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:41.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375755+0000","last_change":"2024-01-16T15:14:57.280588+0000","last_active":"2024-01-16T15:14:58.375755+0000","last_peered":"2024-01-16T15:14:58.375755+0000","last_clean":"2024-01-16T15:14:58.375755+0000","last_became_active":"2024-01-16T15:14:57.280107+0000","last_became_peered":"2024-01-16T15:14:57.280107+0000","last_unstale":"2024-01-16T15:14:58.375755+0000","last_undegraded":"2024-01-16T15:14:58.375755+0000","last_fullsized":"2024-01-16T15:14:58.375755+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:31.434341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265809+0000","last_change":"2024-01-16T15:14:57.282533+0000","last_active":"2024-01-16T15:14:58.265809+0000","last_peered":"2024-01-16T15:14:58.265809+0000","last_clean":"2024-01-16T15:14:58.265809+0000","last_became_active":"2024-01-16T15:14:57.282165+0000","last_became_peered":"2024-01-16T15:14:57.282165+0000","last_unstale":"2024-01-16T15:14:58.265809+0000","last_undegraded":"2024-01-16T15:14:58.265809+0000","last_fullsized":"2024-01-16T15:14:58.265809+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:08.595609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378098+0000","last_change":"2024-01-16T15:14:57.276729+0000","last_active":"2024-01-16T15:14:58.378098+0000","last_peered":"2024-01-16T15:14:58.378098+0000","last_clean":"2024-01-16T15:14:58.378098+0000","last_became_active":"2024-01-16T15:14:57.275755+0000","last_became_peered":"2024-01-16T15:14:57.275755+0000","last_unstale":"2024-01-16T15:14:58.378098+0000","last_undegraded":"2024-01-16T15:14:58.378098+0000","last_fullsized":"2024-01-16T15:14:58.378098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:31.892776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263427+0000","last_change":"2024-01-16T15:14:57.277001+0000","last_active":"2024-01-16T15:14:58.263427+0000","last_peered":"2024-01-16T15:14:58.263427+0000","last_clean":"2024-01-16T15:14:58.263427+0000","last_became_active":"2024-01-16T15:14:57.276555+0000","last_became_peered":"2024-01-16T15:14:57.276555+0000","last_unstale":"2024-01-16T15:14:58.263427+0000","last_undegraded":"2024-01-16T15:14:58.263427+0000","last_fullsized":"2024-01-16T15:14:58.263427+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:18.037106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378059+0000","last_change":"2024-01-16T15:14:57.268931+0000","last_active":"2024-01-16T15:14:58.378059+0000","last_peered":"2024-01-16T15:14:58.378059+0000","last_clean":"2024-01-16T15:14:58.378059+0000","last_became_active":"2024-01-16T15:14:57.268661+0000","last_became_peered":"2024-01-16T15:14:57.268661+0000","last_unstale":"2024-01-16T15:14:58.378059+0000","last_undegraded":"2024-01-16T15:14:58.378059+0000","last_fullsized":"2024-01-16T15:14:58.378059+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:21.445627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375715+0000","last_change":"2024-01-16T15:14:57.279996+0000","last_active":"2024-01-16T15:14:58.375715+0000","last_peered":"2024-01-16T15:14:58.375715+0000","last_clean":"2024-01-16T15:14:58.375715+0000","last_became_active":"2024-01-16T15:14:57.279695+0000","last_became_peered":"2024-01-16T15:14:57.279695+0000","last_unstale":"2024-01-16T15:14:58.375715+0000","last_undegraded":"2024-01-16T15:14:58.375715+0000","last_fullsized":"2024-01-16T15:14:58.375715+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:51.570860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376819+0000","last_change":"2024-01-16T15:14:57.263987+0000","last_active":"2024-01-16T15:14:58.376819+0000","last_peered":"2024-01-16T15:14:58.376819+0000","last_clean":"2024-01-16T15:14:58.376819+0000","last_became_active":"2024-01-16T15:14:57.258913+0000","last_became_peered":"2024-01-16T15:14:57.258913+0000","last_unstale":"2024-01-16T15:14:58.376819+0000","last_undegraded":"2024-01-16T15:14:58.376819+0000","last_fullsized":"2024-01-16T15:14:58.376819+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:22.248118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378592+0000","last_change":"2024-01-16T15:14:57.282890+0000","last_active":"2024-01-16T15:14:58.378592+0000","last_peered":"2024-01-16T15:14:58.378592+0000","last_clean":"2024-01-16T15:14:58.378592+0000","last_became_active":"2024-01-16T15:14:57.282602+0000","last_became_peered":"2024-01-16T15:14:57.282602+0000","last_unstale":"2024-01-16T15:14:58.378592+0000","last_undegraded":"2024-01-16T15:14:58.378592+0000","last_fullsized":"2024-01-16T15:14:58.378592+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:51.570860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264370+0000","last_change":"2024-01-16T15:14:57.268679+0000","last_active":"2024-01-16T15:14:58.264370+0000","last_peered":"2024-01-16T15:14:58.264370+0000","last_clean":"2024-01-16T15:14:58.264370+0000","last_became_active":"2024-01-16T15:14:57.268395+0000","last_became_peered":"2024-01-16T15:14:57.268395+0000","last_unstale":"2024-01-16T15:14:58.264370+0000","last_undegraded":"2024-01-16T15:14:58.264370+0000","last_fullsized":"2024-01-16T15:14:58.264370+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:23.789757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266237+0000","last_change":"2024-01-16T15:14:57.277443+0000","last_active":"2024-01-16T15:14:58.266237+0000","last_peered":"2024-01-16T15:14:58.266237+0000","last_clean":"2024-01-16T15:14:58.266237+0000","last_became_active":"2024-01-16T15:14:57.277120+0000","last_became_peered":"2024-01-16T15:14:57.277120+0000","last_unstale":"2024-01-16T15:14:58.266237+0000","last_undegraded":"2024-01-16T15:14:58.266237+0000","last_fullsized":"2024-01-16T15:14:58.266237+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:23.813457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263388+0000","last_change":"2024-01-16T15:14:57.265176+0000","last_active":"2024-01-16T15:14:58.263388+0000","last_peered":"2024-01-16T15:14:58.263388+0000","last_clean":"2024-01-16T15:14:58.263388+0000","last_became_active":"2024-01-16T15:14:57.264777+0000","last_became_peered":"2024-01-16T15:14:57.264777+0000","last_unstale":"2024-01-16T15:14:58.263388+0000","last_undegraded":"2024-01-16T15:14:58.263388+0000","last_fullsized":"2024-01-16T15:14:58.263388+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:58.989677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375965+0000","last_change":"2024-01-16T15:14:57.279642+0000","last_active":"2024-01-16T15:14:58.375965+0000","last_peered":"2024-01-16T15:14:58.375965+0000","last_clean":"2024-01-16T15:14:58.375965+0000","last_became_active":"2024-01-16T15:14:57.277939+0000","last_became_peered":"2024-01-16T15:14:57.277939+0000","last_unstale":"2024-01-16T15:14:58.375965+0000","last_undegraded":"2024-01-16T15:14:58.375965+0000","last_fullsized":"2024-01-16T15:14:58.375965+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:58.954219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375910+0000","last_change":"2024-01-16T15:14:57.279745+0000","last_active":"2024-01-16T15:14:58.375910+0000","last_peered":"2024-01-16T15:14:58.375910+0000","last_clean":"2024-01-16T15:14:58.375910+0000","last_became_active":"2024-01-16T15:14:57.279048+0000","last_became_peered":"2024-01-16T15:14:57.279048+0000","last_unstale":"2024-01-16T15:14:58.375910+0000","last_undegraded":"2024-01-16T15:14:58.375910+0000","last_fullsized":"2024-01-16T15:14:58.375910+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:56:25.612959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376512+0000","last_change":"2024-01-16T15:14:57.279969+0000","last_active":"2024-01-16T15:14:58.376512+0000","last_peered":"2024-01-16T15:14:58.376512+0000","last_clean":"2024-01-16T15:14:58.376512+0000","last_became_active":"2024-01-16T15:14:57.279515+0000","last_became_peered":"2024-01-16T15:14:57.279515+0000","last_unstale":"2024-01-16T15:14:58.376512+0000","last_undegraded":"2024-01-16T15:14:58.376512+0000","last_fullsized":"2024-01-16T15:14:58.376512+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:54:24.282448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269602+0000","last_change":"2024-01-16T15:14:57.272062+0000","last_active":"2024-01-16T15:14:58.269602+0000","last_peered":"2024-01-16T15:14:58.269602+0000","last_clean":"2024-01-16T15:14:58.269602+0000","last_became_active":"2024-01-16T15:14:57.271101+0000","last_became_peered":"2024-01-16T15:14:57.271101+0000","last_unstale":"2024-01-16T15:14:58.269602+0000","last_undegraded":"2024-01-16T15:14:58.269602+0000","last_fullsized":"2024-01-16T15:14:58.269602+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:13.227145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376372+0000","last_change":"2024-01-16T15:14:57.270151+0000","last_active":"2024-01-16T15:14:58.376372+0000","last_peered":"2024-01-16T15:14:58.376372+0000","last_clean":"2024-01-16T15:14:58.376372+0000","last_became_active":"2024-01-16T15:14:57.269840+0000","last_became_peered":"2024-01-16T15:14:57.269840+0000","last_unstale":"2024-01-16T15:14:58.376372+0000","last_undegraded":"2024-01-16T15:14:58.376372+0000","last_fullsized":"2024-01-16T15:14:58.376372+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:43.938439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376112+0000","last_change":"2024-01-16T15:14:57.276338+0000","last_active":"2024-01-16T15:14:58.376112+0000","last_peered":"2024-01-16T15:14:58.376112+0000","last_clean":"2024-01-16T15:14:58.376112+0000","last_became_active":"2024-01-16T15:14:57.276027+0000","last_became_peered":"2024-01-16T15:14:57.276027+0000","last_unstale":"2024-01-16T15:14:58.376112+0000","last_undegraded":"2024-01-16T15:14:58.376112+0000","last_fullsized":"2024-01-16T15:14:58.376112+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269043+0000","last_change":"2024-01-16T15:14:57.264934+0000","last_active":"2024-01-16T15:14:58.269043+0000","last_peered":"2024-01-16T15:14:58.269043+0000","last_clean":"2024-01-16T15:14:58.269043+0000","last_became_active":"2024-01-16T15:14:57.264636+0000","last_became_peered":"2024-01-16T15:14:57.264636+0000","last_unstale":"2024-01-16T15:14:58.269043+0000","last_undegraded":"2024-01-16T15:14:58.269043+0000","last_fullsized":"2024-01-16T15:14:58.269043+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:08.246934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376050+0000","last_change":"2024-01-16T15:14:57.270864+0000","last_active":"2024-01-16T15:14:58.376050+0000","last_peered":"2024-01-16T15:14:58.376050+0000","last_clean":"2024-01-16T15:14:58.376050+0000","last_became_active":"2024-01-16T15:14:57.270588+0000","last_became_peered":"2024-01-16T15:14:57.270588+0000","last_unstale":"2024-01-16T15:14:58.376050+0000","last_undegraded":"2024-01-16T15:14:58.376050+0000","last_fullsized":"2024-01-16T15:14:58.376050+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:59.418210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263603+0000","last_change":"2024-01-16T15:14:57.276822+0000","last_active":"2024-01-16T15:14:58.263603+0000","last_peered":"2024-01-16T15:14:58.263603+0000","last_clean":"2024-01-16T15:14:58.263603+0000","last_became_active":"2024-01-16T15:14:57.276263+0000","last_became_peered":"2024-01-16T15:14:57.276263+0000","last_unstale":"2024-01-16T15:14:58.263603+0000","last_undegraded":"2024-01-16T15:14:58.263603+0000","last_fullsized":"2024-01-16T15:14:58.263603+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:50.333191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.378125+0000","last_change":"2024-01-16T15:14:57.268384+0000","last_active":"2024-01-16T15:14:58.378125+0000","last_peered":"2024-01-16T15:14:58.378125+0000","last_clean":"2024-01-16T15:14:58.378125+0000","last_became_active":"2024-01-16T15:14:57.267319+0000","last_became_peered":"2024-01-16T15:14:57.267319+0000","last_unstale":"2024-01-16T15:14:58.378125+0000","last_undegraded":"2024-01-16T15:14:58.378125+0000","last_fullsized":"2024-01-16T15:14:58.378125+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:31.093985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.374812+0000","last_change":"2024-01-16T15:14:57.286809+0000","last_active":"2024-01-16T15:14:58.374812+0000","last_peered":"2024-01-16T15:14:58.374812+0000","last_clean":"2024-01-16T15:14:58.374812+0000","last_became_active":"2024-01-16T15:14:57.284949+0000","last_became_peered":"2024-01-16T15:14:57.284949+0000","last_unstale":"2024-01-16T15:14:58.374812+0000","last_undegraded":"2024-01-16T15:14:58.374812+0000","last_fullsized":"2024-01-16T15:14:58.374812+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:19.648713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264387+0000","last_change":"2024-01-16T15:14:57.268766+0000","last_active":"2024-01-16T15:14:58.264387+0000","last_peered":"2024-01-16T15:14:58.264387+0000","last_clean":"2024-01-16T15:14:58.264387+0000","last_became_active":"2024-01-16T15:14:57.268488+0000","last_became_peered":"2024-01-16T15:14:57.268488+0000","last_unstale":"2024-01-16T15:14:58.264387+0000","last_undegraded":"2024-01-16T15:14:58.264387+0000","last_fullsized":"2024-01-16T15:14:58.264387+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:43.440293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"100'4","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.284752+0000","last_change":"2024-01-16T15:14:56.247628+0000","last_active":"2024-01-16T15:14:57.284752+0000","last_peered":"2024-01-16T15:14:57.284752+0000","last_clean":"2024-01-16T15:14:57.284752+0000","last_became_active":"2024-01-16T15:14:56.247338+0000","last_became_peered":"2024-01-16T15:14:56.247338+0000","last_unstale":"2024-01-16T15:14:57.284752+0000","last_undegraded":"2024-01-16T15:14:57.284752+0000","last_fullsized":"2024-01-16T15:14:57.284752+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:30.335125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.379230+0000","last_change":"2024-01-16T15:14:57.270923+0000","last_active":"2024-01-16T15:14:58.379230+0000","last_peered":"2024-01-16T15:14:58.379230+0000","last_clean":"2024-01-16T15:14:58.379230+0000","last_became_active":"2024-01-16T15:14:57.270658+0000","last_became_peered":"2024-01-16T15:14:57.270658+0000","last_unstale":"2024-01-16T15:14:58.379230+0000","last_undegraded":"2024-01-16T15:14:58.379230+0000","last_fullsized":"2024-01-16T15:14:58.379230+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:31.613245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.274812+0000","last_change":"2024-01-16T15:14:56.257668+0000","last_active":"2024-01-16T15:14:57.274812+0000","last_peered":"2024-01-16T15:14:57.274812+0000","last_clean":"2024-01-16T15:14:57.274812+0000","last_became_active":"2024-01-16T15:14:56.257372+0000","last_became_peered":"2024-01-16T15:14:56.257372+0000","last_unstale":"2024-01-16T15:14:57.274812+0000","last_undegraded":"2024-01-16T15:14:57.274812+0000","last_fullsized":"2024-01-16T15:14:57.274812+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:30.877365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376455+0000","last_change":"2024-01-16T15:14:57.283445+0000","last_active":"2024-01-16T15:14:58.376455+0000","last_peered":"2024-01-16T15:14:58.376455+0000","last_clean":"2024-01-16T15:14:58.376455+0000","last_became_active":"2024-01-16T15:14:57.283138+0000","last_became_peered":"2024-01-16T15:14:57.283138+0000","last_unstale":"2024-01-16T15:14:58.376455+0000","last_undegraded":"2024-01-16T15:14:58.376455+0000","last_fullsized":"2024-01-16T15:14:58.376455+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:26.406696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288805+0000","last_change":"2024-01-16T15:14:56.251801+0000","last_active":"2024-01-16T15:14:57.288805+0000","last_peered":"2024-01-16T15:14:57.288805+0000","last_clean":"2024-01-16T15:14:57.288805+0000","last_became_active":"2024-01-16T15:14:56.248230+0000","last_became_peered":"2024-01-16T15:14:56.248230+0000","last_unstale":"2024-01-16T15:14:57.288805+0000","last_undegraded":"2024-01-16T15:14:57.288805+0000","last_fullsized":"2024-01-16T15:14:57.288805+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:14.641251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.266838+0000","last_change":"2024-01-16T15:14:57.275944+0000","last_active":"2024-01-16T15:14:58.266838+0000","last_peered":"2024-01-16T15:14:58.266838+0000","last_clean":"2024-01-16T15:14:58.266838+0000","last_became_active":"2024-01-16T15:14:57.275675+0000","last_became_peered":"2024-01-16T15:14:57.275675+0000","last_unstale":"2024-01-16T15:14:58.266838+0000","last_undegraded":"2024-01-16T15:14:58.266838+0000","last_fullsized":"2024-01-16T15:14:58.266838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:14.535235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.284502+0000","last_change":"2024-01-16T15:14:56.251154+0000","last_active":"2024-01-16T15:14:57.284502+0000","last_peered":"2024-01-16T15:14:57.284502+0000","last_clean":"2024-01-16T15:14:57.284502+0000","last_became_active":"2024-01-16T15:14:56.247511+0000","last_became_peered":"2024-01-16T15:14:56.247511+0000","last_unstale":"2024-01-16T15:14:57.284502+0000","last_undegraded":"2024-01-16T15:14:57.284502+0000","last_fullsized":"2024-01-16T15:14:57.284502+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:19.175215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375888+0000","last_change":"2024-01-16T15:14:57.279137+0000","last_active":"2024-01-16T15:14:58.375888+0000","last_peered":"2024-01-16T15:14:58.375888+0000","last_clean":"2024-01-16T15:14:58.375888+0000","last_became_active":"2024-01-16T15:14:57.278824+0000","last_became_peered":"2024-01-16T15:14:57.278824+0000","last_unstale":"2024-01-16T15:14:58.375888+0000","last_undegraded":"2024-01-16T15:14:58.375888+0000","last_fullsized":"2024-01-16T15:14:58.375888+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:26.406696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.274380+0000","last_change":"2024-01-16T15:14:56.255146+0000","last_active":"2024-01-16T15:14:57.274380+0000","last_peered":"2024-01-16T15:14:57.274380+0000","last_clean":"2024-01-16T15:14:57.274380+0000","last_became_active":"2024-01-16T15:14:56.253956+0000","last_became_peered":"2024-01-16T15:14:56.253956+0000","last_unstale":"2024-01-16T15:14:57.274380+0000","last_undegraded":"2024-01-16T15:14:57.274380+0000","last_fullsized":"2024-01-16T15:14:57.274380+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:30.054925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376516+0000","last_change":"2024-01-16T15:14:57.276953+0000","last_active":"2024-01-16T15:14:58.376516+0000","last_peered":"2024-01-16T15:14:58.376516+0000","last_clean":"2024-01-16T15:14:58.376516+0000","last_became_active":"2024-01-16T15:14:57.276709+0000","last_became_peered":"2024-01-16T15:14:57.276709+0000","last_unstale":"2024-01-16T15:14:58.376516+0000","last_undegraded":"2024-01-16T15:14:58.376516+0000","last_fullsized":"2024-01-16T15:14:58.376516+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:03:52.157955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.286528+0000","last_change":"2024-01-16T15:14:56.257803+0000","last_active":"2024-01-16T15:14:57.286528+0000","last_peered":"2024-01-16T15:14:57.286528+0000","last_clean":"2024-01-16T15:14:57.286528+0000","last_became_active":"2024-01-16T15:14:56.257475+0000","last_became_peered":"2024-01-16T15:14:56.257475+0000","last_unstale":"2024-01-16T15:14:57.286528+0000","last_undegraded":"2024-01-16T15:14:57.286528+0000","last_fullsized":"2024-01-16T15:14:57.286528+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:07.231523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377565+0000","last_change":"2024-01-16T15:14:57.274379+0000","last_active":"2024-01-16T15:14:58.377565+0000","last_peered":"2024-01-16T15:14:58.377565+0000","last_clean":"2024-01-16T15:14:58.377565+0000","last_became_active":"2024-01-16T15:14:57.272853+0000","last_became_peered":"2024-01-16T15:14:57.272853+0000","last_unstale":"2024-01-16T15:14:58.377565+0000","last_undegraded":"2024-01-16T15:14:58.377565+0000","last_fullsized":"2024-01-16T15:14:58.377565+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:15.374800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.251642+0000","last_change":"2024-01-16T15:14:56.247282+0000","last_active":"2024-01-16T15:14:57.251642+0000","last_peered":"2024-01-16T15:14:57.251642+0000","last_clean":"2024-01-16T15:14:57.251642+0000","last_became_active":"2024-01-16T15:14:56.246887+0000","last_became_peered":"2024-01-16T15:14:56.246887+0000","last_unstale":"2024-01-16T15:14:57.251642+0000","last_undegraded":"2024-01-16T15:14:57.251642+0000","last_fullsized":"2024-01-16T15:14:57.251642+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:44:56.232403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265296+0000","last_change":"2024-01-16T15:14:57.262925+0000","last_active":"2024-01-16T15:14:58.265296+0000","last_peered":"2024-01-16T15:14:58.265296+0000","last_clean":"2024-01-16T15:14:58.265296+0000","last_became_active":"2024-01-16T15:14:57.261447+0000","last_became_peered":"2024-01-16T15:14:57.261447+0000","last_unstale":"2024-01-16T15:14:58.265296+0000","last_undegraded":"2024-01-16T15:14:58.265296+0000","last_fullsized":"2024-01-16T15:14:58.265296+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:19.313378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.259804+0000","last_change":"2024-01-16T15:14:56.260607+0000","last_active":"2024-01-16T15:14:57.259804+0000","last_peered":"2024-01-16T15:14:57.259804+0000","last_clean":"2024-01-16T15:14:57.259804+0000","last_became_active":"2024-01-16T15:14:56.258899+0000","last_became_peered":"2024-01-16T15:14:56.258899+0000","last_unstale":"2024-01-16T15:14:57.259804+0000","last_undegraded":"2024-01-16T15:14:57.259804+0000","last_fullsized":"2024-01-16T15:14:57.259804+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:22.774345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376497+0000","last_change":"2024-01-16T15:14:57.261589+0000","last_active":"2024-01-16T15:14:58.376497+0000","last_peered":"2024-01-16T15:14:58.376497+0000","last_clean":"2024-01-16T15:14:58.376497+0000","last_became_active":"2024-01-16T15:14:57.261259+0000","last_became_peered":"2024-01-16T15:14:57.261259+0000","last_unstale":"2024-01-16T15:14:58.376497+0000","last_undegraded":"2024-01-16T15:14:58.376497+0000","last_fullsized":"2024-01-16T15:14:58.376497+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:39.211207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.263636+0000","last_change":"2024-01-16T15:14:56.258700+0000","last_active":"2024-01-16T15:14:57.263636+0000","last_peered":"2024-01-16T15:14:57.263636+0000","last_clean":"2024-01-16T15:14:57.263636+0000","last_became_active":"2024-01-16T15:14:56.256930+0000","last_became_peered":"2024-01-16T15:14:56.256930+0000","last_unstale":"2024-01-16T15:14:57.263636+0000","last_undegraded":"2024-01-16T15:14:57.263636+0000","last_fullsized":"2024-01-16T15:14:57.263636+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:33.636833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267108+0000","last_change":"2024-01-16T15:14:57.274399+0000","last_active":"2024-01-16T15:14:58.267108+0000","last_peered":"2024-01-16T15:14:58.267108+0000","last_clean":"2024-01-16T15:14:58.267108+0000","last_became_active":"2024-01-16T15:14:57.273766+0000","last_became_peered":"2024-01-16T15:14:57.273766+0000","last_unstale":"2024-01-16T15:14:58.267108+0000","last_undegraded":"2024-01-16T15:14:58.267108+0000","last_fullsized":"2024-01-16T15:14:58.267108+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:36.466764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.250085+0000","last_change":"2024-01-16T15:14:56.234171+0000","last_active":"2024-01-16T15:14:57.250085+0000","last_peered":"2024-01-16T15:14:57.250085+0000","last_clean":"2024-01-16T15:14:57.250085+0000","last_became_active":"2024-01-16T15:14:56.233313+0000","last_became_peered":"2024-01-16T15:14:56.233313+0000","last_unstale":"2024-01-16T15:14:57.250085+0000","last_undegraded":"2024-01-16T15:14:57.250085+0000","last_fullsized":"2024-01-16T15:14:57.250085+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:40.475359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.268766+0000","last_change":"2024-01-16T15:14:57.262136+0000","last_active":"2024-01-16T15:14:58.268766+0000","last_peered":"2024-01-16T15:14:58.268766+0000","last_clean":"2024-01-16T15:14:58.268766+0000","last_became_active":"2024-01-16T15:14:57.261858+0000","last_became_peered":"2024-01-16T15:14:57.261858+0000","last_unstale":"2024-01-16T15:14:58.268766+0000","last_undegraded":"2024-01-16T15:14:58.268766+0000","last_fullsized":"2024-01-16T15:14:58.268766+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:15:20.190626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"100'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.281892+0000","last_change":"2024-01-16T15:14:56.254644+0000","last_active":"2024-01-16T15:14:57.281892+0000","last_peered":"2024-01-16T15:14:57.281892+0000","last_clean":"2024-01-16T15:14:57.281892+0000","last_became_active":"2024-01-16T15:14:56.254201+0000","last_became_peered":"2024-01-16T15:14:56.254201+0000","last_unstale":"2024-01-16T15:14:57.281892+0000","last_undegraded":"2024-01-16T15:14:57.281892+0000","last_fullsized":"2024-01-16T15:14:57.281892+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:20.430216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377526+0000","last_change":"2024-01-16T15:14:57.274051+0000","last_active":"2024-01-16T15:14:58.377526+0000","last_peered":"2024-01-16T15:14:58.377526+0000","last_clean":"2024-01-16T15:14:58.377526+0000","last_became_active":"2024-01-16T15:14:57.273775+0000","last_became_peered":"2024-01-16T15:14:57.273775+0000","last_unstale":"2024-01-16T15:14:58.377526+0000","last_undegraded":"2024-01-16T15:14:58.377526+0000","last_fullsized":"2024-01-16T15:14:58.377526+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:28.098955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"100'6","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288248+0000","last_change":"2024-01-16T15:14:56.249720+0000","last_active":"2024-01-16T15:14:57.288248+0000","last_peered":"2024-01-16T15:14:57.288248+0000","last_clean":"2024-01-16T15:14:57.288248+0000","last_became_active":"2024-01-16T15:14:56.248747+0000","last_became_peered":"2024-01-16T15:14:56.248747+0000","last_unstale":"2024-01-16T15:14:57.288248+0000","last_undegraded":"2024-01-16T15:14:57.288248+0000","last_fullsized":"2024-01-16T15:14:57.288248+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:18.335638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377698+0000","last_change":"2024-01-16T15:14:57.267525+0000","last_active":"2024-01-16T15:14:58.377698+0000","last_peered":"2024-01-16T15:14:58.377698+0000","last_clean":"2024-01-16T15:14:58.377698+0000","last_became_active":"2024-01-16T15:14:57.267246+0000","last_became_peered":"2024-01-16T15:14:57.267246+0000","last_unstale":"2024-01-16T15:14:58.377698+0000","last_undegraded":"2024-01-16T15:14:58.377698+0000","last_fullsized":"2024-01-16T15:14:58.377698+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:17.356858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.285170+0000","last_change":"2024-01-16T15:14:56.252697+0000","last_active":"2024-01-16T15:14:57.285170+0000","last_peered":"2024-01-16T15:14:57.285170+0000","last_clean":"2024-01-16T15:14:57.285170+0000","last_became_active":"2024-01-16T15:14:56.252235+0000","last_became_peered":"2024-01-16T15:14:56.252235+0000","last_unstale":"2024-01-16T15:14:57.285170+0000","last_undegraded":"2024-01-16T15:14:57.285170+0000","last_fullsized":"2024-01-16T15:14:57.285170+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:30.646300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375077+0000","last_change":"2024-01-16T15:14:57.279952+0000","last_active":"2024-01-16T15:14:58.375077+0000","last_peered":"2024-01-16T15:14:58.375077+0000","last_clean":"2024-01-16T15:14:58.375077+0000","last_became_active":"2024-01-16T15:14:57.279605+0000","last_became_peered":"2024-01-16T15:14:57.279605+0000","last_unstale":"2024-01-16T15:14:58.375077+0000","last_undegraded":"2024-01-16T15:14:58.375077+0000","last_fullsized":"2024-01-16T15:14:58.375077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:16.022404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.262802+0000","last_change":"2024-01-16T15:14:56.258480+0000","last_active":"2024-01-16T15:14:57.262802+0000","last_peered":"2024-01-16T15:14:57.262802+0000","last_clean":"2024-01-16T15:14:57.262802+0000","last_became_active":"2024-01-16T15:14:56.258207+0000","last_became_peered":"2024-01-16T15:14:56.258207+0000","last_unstale":"2024-01-16T15:14:57.262802+0000","last_undegraded":"2024-01-16T15:14:57.262802+0000","last_fullsized":"2024-01-16T15:14:57.262802+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:25.707694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264907+0000","last_change":"2024-01-16T15:14:57.278967+0000","last_active":"2024-01-16T15:14:58.264907+0000","last_peered":"2024-01-16T15:14:58.264907+0000","last_clean":"2024-01-16T15:14:58.264907+0000","last_became_active":"2024-01-16T15:14:57.277612+0000","last_became_peered":"2024-01-16T15:14:57.277612+0000","last_unstale":"2024-01-16T15:14:58.264907+0000","last_undegraded":"2024-01-16T15:14:58.264907+0000","last_fullsized":"2024-01-16T15:14:58.264907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:17:49.246274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.263440+0000","last_change":"2024-01-16T15:14:56.247113+0000","last_active":"2024-01-16T15:14:57.263440+0000","last_peered":"2024-01-16T15:14:57.263440+0000","last_clean":"2024-01-16T15:14:57.263440+0000","last_became_active":"2024-01-16T15:14:56.246632+0000","last_became_peered":"2024-01-16T15:14:56.246632+0000","last_unstale":"2024-01-16T15:14:57.263440+0000","last_undegraded":"2024-01-16T15:14:57.263440+0000","last_fullsized":"2024-01-16T15:14:57.263440+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:41.615685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263164+0000","last_change":"2024-01-16T15:14:57.287255+0000","last_active":"2024-01-16T15:14:58.263164+0000","last_peered":"2024-01-16T15:14:58.263164+0000","last_clean":"2024-01-16T15:14:58.263164+0000","last_became_active":"2024-01-16T15:14:57.286264+0000","last_became_peered":"2024-01-16T15:14:57.286264+0000","last_unstale":"2024-01-16T15:14:58.263164+0000","last_undegraded":"2024-01-16T15:14:58.263164+0000","last_fullsized":"2024-01-16T15:14:58.263164+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:11.979730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"100'8","reported_seq":20,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.286645+0000","last_change":"2024-01-16T15:14:56.254042+0000","last_active":"2024-01-16T15:14:57.286645+0000","last_peered":"2024-01-16T15:14:57.286645+0000","last_clean":"2024-01-16T15:14:57.286645+0000","last_became_active":"2024-01-16T15:14:56.253721+0000","last_became_peered":"2024-01-16T15:14:56.253721+0000","last_unstale":"2024-01-16T15:14:57.286645+0000","last_undegraded":"2024-01-16T15:14:57.286645+0000","last_fullsized":"2024-01-16T15:14:57.286645+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:06:42.424881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.269616+0000","last_change":"2024-01-16T15:14:57.275824+0000","last_active":"2024-01-16T15:14:58.269616+0000","last_peered":"2024-01-16T15:14:58.269616+0000","last_clean":"2024-01-16T15:14:58.269616+0000","last_became_active":"2024-01-16T15:14:57.275379+0000","last_became_peered":"2024-01-16T15:14:57.275379+0000","last_unstale":"2024-01-16T15:14:58.269616+0000","last_undegraded":"2024-01-16T15:14:58.269616+0000","last_fullsized":"2024-01-16T15:14:58.269616+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:24:31.350537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288484+0000","last_change":"2024-01-16T15:14:56.247405+0000","last_active":"2024-01-16T15:14:57.288484+0000","last_peered":"2024-01-16T15:14:57.288484+0000","last_clean":"2024-01-16T15:14:57.288484+0000","last_became_active":"2024-01-16T15:14:56.247123+0000","last_became_peered":"2024-01-16T15:14:56.247123+0000","last_unstale":"2024-01-16T15:14:57.288484+0000","last_undegraded":"2024-01-16T15:14:57.288484+0000","last_fullsized":"2024-01-16T15:14:57.288484+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:21:12.250548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376994+0000","last_change":"2024-01-16T15:14:57.267134+0000","last_active":"2024-01-16T15:14:58.376994+0000","last_peered":"2024-01-16T15:14:58.376994+0000","last_clean":"2024-01-16T15:14:58.376994+0000","last_became_active":"2024-01-16T15:14:57.266832+0000","last_became_peered":"2024-01-16T15:14:57.266832+0000","last_unstale":"2024-01-16T15:14:58.376994+0000","last_undegraded":"2024-01-16T15:14:58.376994+0000","last_fullsized":"2024-01-16T15:14:58.376994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:29.973542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.266039+0000","last_change":"2024-01-16T15:14:56.241803+0000","last_active":"2024-01-16T15:14:57.266039+0000","last_peered":"2024-01-16T15:14:57.266039+0000","last_clean":"2024-01-16T15:14:57.266039+0000","last_became_active":"2024-01-16T15:14:56.240380+0000","last_became_peered":"2024-01-16T15:14:56.240380+0000","last_unstale":"2024-01-16T15:14:57.266039+0000","last_undegraded":"2024-01-16T15:14:57.266039+0000","last_fullsized":"2024-01-16T15:14:57.266039+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:22.270194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.377591+0000","last_change":"2024-01-16T15:14:57.281920+0000","last_active":"2024-01-16T15:14:58.377591+0000","last_peered":"2024-01-16T15:14:58.377591+0000","last_clean":"2024-01-16T15:14:58.377591+0000","last_became_active":"2024-01-16T15:14:57.281480+0000","last_became_peered":"2024-01-16T15:14:57.281480+0000","last_unstale":"2024-01-16T15:14:58.377591+0000","last_undegraded":"2024-01-16T15:14:58.377591+0000","last_fullsized":"2024-01-16T15:14:58.377591+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:52:54.127537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257517+0000","last_change":"2024-01-16T15:14:56.254269+0000","last_active":"2024-01-16T15:14:57.257517+0000","last_peered":"2024-01-16T15:14:57.257517+0000","last_clean":"2024-01-16T15:14:57.257517+0000","last_became_active":"2024-01-16T15:14:56.246024+0000","last_became_peered":"2024-01-16T15:14:56.246024+0000","last_unstale":"2024-01-16T15:14:57.257517+0000","last_undegraded":"2024-01-16T15:14:57.257517+0000","last_fullsized":"2024-01-16T15:14:57.257517+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:40.497506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263758+0000","last_change":"2024-01-16T15:14:57.276053+0000","last_active":"2024-01-16T15:14:58.263758+0000","last_peered":"2024-01-16T15:14:58.263758+0000","last_clean":"2024-01-16T15:14:58.263758+0000","last_became_active":"2024-01-16T15:14:57.275755+0000","last_became_peered":"2024-01-16T15:14:57.275755+0000","last_unstale":"2024-01-16T15:14:58.263758+0000","last_undegraded":"2024-01-16T15:14:58.263758+0000","last_fullsized":"2024-01-16T15:14:58.263758+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:23.192004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.288743+0000","last_change":"2024-01-16T15:14:56.255721+0000","last_active":"2024-01-16T15:14:57.288743+0000","last_peered":"2024-01-16T15:14:57.288743+0000","last_clean":"2024-01-16T15:14:57.288743+0000","last_became_active":"2024-01-16T15:14:56.250495+0000","last_became_peered":"2024-01-16T15:14:56.250495+0000","last_unstale":"2024-01-16T15:14:57.288743+0000","last_undegraded":"2024-01-16T15:14:57.288743+0000","last_fullsized":"2024-01-16T15:14:57.288743+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:58.630719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264108+0000","last_change":"2024-01-16T15:14:57.275382+0000","last_active":"2024-01-16T15:14:58.264108+0000","last_peered":"2024-01-16T15:14:58.264108+0000","last_clean":"2024-01-16T15:14:58.264108+0000","last_became_active":"2024-01-16T15:14:57.275125+0000","last_became_peered":"2024-01-16T15:14:57.275125+0000","last_unstale":"2024-01-16T15:14:58.264108+0000","last_undegraded":"2024-01-16T15:14:58.264108+0000","last_fullsized":"2024-01-16T15:14:58.264108+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:39:34.553063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.247724+0000","last_change":"2024-01-16T15:14:56.260376+0000","last_active":"2024-01-16T15:14:57.247724+0000","last_peered":"2024-01-16T15:14:57.247724+0000","last_clean":"2024-01-16T15:14:57.247724+0000","last_became_active":"2024-01-16T15:14:56.260078+0000","last_became_peered":"2024-01-16T15:14:56.260078+0000","last_unstale":"2024-01-16T15:14:57.247724+0000","last_undegraded":"2024-01-16T15:14:57.247724+0000","last_fullsized":"2024-01-16T15:14:57.247724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:21.232134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263099+0000","last_change":"2024-01-16T15:14:57.281739+0000","last_active":"2024-01-16T15:14:58.263099+0000","last_peered":"2024-01-16T15:14:58.263099+0000","last_clean":"2024-01-16T15:14:58.263099+0000","last_became_active":"2024-01-16T15:14:57.280568+0000","last_became_peered":"2024-01-16T15:14:57.280568+0000","last_unstale":"2024-01-16T15:14:58.263099+0000","last_undegraded":"2024-01-16T15:14:58.263099+0000","last_fullsized":"2024-01-16T15:14:58.263099+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:08.863102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.247300+0000","last_change":"2024-01-16T15:14:56.250720+0000","last_active":"2024-01-16T15:14:57.247300+0000","last_peered":"2024-01-16T15:14:57.247300+0000","last_clean":"2024-01-16T15:14:57.247300+0000","last_became_active":"2024-01-16T15:14:56.250425+0000","last_became_peered":"2024-01-16T15:14:56.250425+0000","last_unstale":"2024-01-16T15:14:57.247300+0000","last_undegraded":"2024-01-16T15:14:57.247300+0000","last_fullsized":"2024-01-16T15:14:57.247300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:07:54.535564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267577+0000","last_change":"2024-01-16T15:14:57.271475+0000","last_active":"2024-01-16T15:14:58.267577+0000","last_peered":"2024-01-16T15:14:58.267577+0000","last_clean":"2024-01-16T15:14:58.267577+0000","last_became_active":"2024-01-16T15:14:57.269885+0000","last_became_peered":"2024-01-16T15:14:57.269885+0000","last_unstale":"2024-01-16T15:14:58.267577+0000","last_undegraded":"2024-01-16T15:14:58.267577+0000","last_fullsized":"2024-01-16T15:14:58.267577+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:29.843051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.256714+0000","last_change":"2024-01-16T15:14:56.259100+0000","last_active":"2024-01-16T15:14:57.256714+0000","last_peered":"2024-01-16T15:14:57.256714+0000","last_clean":"2024-01-16T15:14:57.256714+0000","last_became_active":"2024-01-16T15:14:56.258797+0000","last_became_peered":"2024-01-16T15:14:56.258797+0000","last_unstale":"2024-01-16T15:14:57.256714+0000","last_undegraded":"2024-01-16T15:14:57.256714+0000","last_fullsized":"2024-01-16T15:14:57.256714+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:20:28.827639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376643+0000","last_change":"2024-01-16T15:14:57.262723+0000","last_active":"2024-01-16T15:14:58.376643+0000","last_peered":"2024-01-16T15:14:58.376643+0000","last_clean":"2024-01-16T15:14:58.376643+0000","last_became_active":"2024-01-16T15:14:57.262413+0000","last_became_peered":"2024-01-16T15:14:57.262413+0000","last_unstale":"2024-01-16T15:14:58.376643+0000","last_undegraded":"2024-01-16T15:14:58.376643+0000","last_fullsized":"2024-01-16T15:14:58.376643+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:00:37.124421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"100'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.283433+0000","last_change":"2024-01-16T15:14:56.250815+0000","last_active":"2024-01-16T15:14:57.283433+0000","last_peered":"2024-01-16T15:14:57.283433+0000","last_clean":"2024-01-16T15:14:57.283433+0000","last_became_active":"2024-01-16T15:14:56.250559+0000","last_became_peered":"2024-01-16T15:14:56.250559+0000","last_unstale":"2024-01-16T15:14:57.283433+0000","last_undegraded":"2024-01-16T15:14:57.283433+0000","last_fullsized":"2024-01-16T15:14:57.283433+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:06.163748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.375590+0000","last_change":"2024-01-16T15:14:57.278308+0000","last_active":"2024-01-16T15:14:58.375590+0000","last_peered":"2024-01-16T15:14:58.375590+0000","last_clean":"2024-01-16T15:14:58.375590+0000","last_became_active":"2024-01-16T15:14:57.277548+0000","last_became_peered":"2024-01-16T15:14:57.277548+0000","last_unstale":"2024-01-16T15:14:58.375590+0000","last_undegraded":"2024-01-16T15:14:58.375590+0000","last_fullsized":"2024-01-16T15:14:58.375590+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:07.179160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257987+0000","last_change":"2024-01-16T15:14:56.253570+0000","last_active":"2024-01-16T15:14:57.257987+0000","last_peered":"2024-01-16T15:14:57.257987+0000","last_clean":"2024-01-16T15:14:57.257987+0000","last_became_active":"2024-01-16T15:14:56.249988+0000","last_became_peered":"2024-01-16T15:14:56.249988+0000","last_unstale":"2024-01-16T15:14:57.257987+0000","last_undegraded":"2024-01-16T15:14:57.257987+0000","last_fullsized":"2024-01-16T15:14:57.257987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:12.211734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.376795+0000","last_change":"2024-01-16T15:14:57.271437+0000","last_active":"2024-01-16T15:14:58.376795+0000","last_peered":"2024-01-16T15:14:58.376795+0000","last_clean":"2024-01-16T15:14:58.376795+0000","last_became_active":"2024-01-16T15:14:57.270628+0000","last_became_peered":"2024-01-16T15:14:57.270628+0000","last_unstale":"2024-01-16T15:14:58.376795+0000","last_undegraded":"2024-01-16T15:14:58.376795+0000","last_fullsized":"2024-01-16T15:14:58.376795+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:29.973542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.257309+0000","last_change":"2024-01-16T15:14:56.253017+0000","last_active":"2024-01-16T15:14:57.257309+0000","last_peered":"2024-01-16T15:14:57.257309+0000","last_clean":"2024-01-16T15:14:57.257309+0000","last_became_active":"2024-01-16T15:14:56.252713+0000","last_became_peered":"2024-01-16T15:14:56.252713+0000","last_unstale":"2024-01-16T15:14:57.257309+0000","last_undegraded":"2024-01-16T15:14:57.257309+0000","last_fullsized":"2024-01-16T15:14:57.257309+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:23.441321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.268721+0000","last_change":"2024-01-16T15:14:57.255863+0000","last_active":"2024-01-16T15:14:58.268721+0000","last_peered":"2024-01-16T15:14:58.268721+0000","last_clean":"2024-01-16T15:14:58.268721+0000","last_became_active":"2024-01-16T15:14:57.255566+0000","last_became_peered":"2024-01-16T15:14:57.255566+0000","last_unstale":"2024-01-16T15:14:58.268721+0000","last_undegraded":"2024-01-16T15:14:58.268721+0000","last_fullsized":"2024-01-16T15:14:58.268721+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:27:24.456733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.252495+0000","last_change":"2024-01-16T15:14:56.245838+0000","last_active":"2024-01-16T15:14:57.252495+0000","last_peered":"2024-01-16T15:14:57.252495+0000","last_clean":"2024-01-16T15:14:57.252495+0000","last_became_active":"2024-01-16T15:14:56.243709+0000","last_became_peered":"2024-01-16T15:14:56.243709+0000","last_unstale":"2024-01-16T15:14:57.252495+0000","last_undegraded":"2024-01-16T15:14:57.252495+0000","last_fullsized":"2024-01-16T15:14:57.252495+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:58.630719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263953+0000","last_change":"2024-01-16T15:14:57.285072+0000","last_active":"2024-01-16T15:14:58.263953+0000","last_peered":"2024-01-16T15:14:58.263953+0000","last_clean":"2024-01-16T15:14:58.263953+0000","last_became_active":"2024-01-16T15:14:57.284598+0000","last_became_peered":"2024-01-16T15:14:57.284598+0000","last_unstale":"2024-01-16T15:14:58.263953+0000","last_undegraded":"2024-01-16T15:14:58.263953+0000","last_fullsized":"2024-01-16T15:14:58.263953+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:36:44.684310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.256403+0000","last_change":"2024-01-16T15:14:56.241536+0000","last_active":"2024-01-16T15:14:57.256403+0000","last_peered":"2024-01-16T15:14:57.256403+0000","last_clean":"2024-01-16T15:14:57.256403+0000","last_became_active":"2024-01-16T15:14:56.241265+0000","last_became_peered":"2024-01-16T15:14:56.241265+0000","last_unstale":"2024-01-16T15:14:57.256403+0000","last_undegraded":"2024-01-16T15:14:57.256403+0000","last_fullsized":"2024-01-16T15:14:57.256403+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:22.270194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.265621+0000","last_change":"2024-01-16T15:14:57.268919+0000","last_active":"2024-01-16T15:14:58.265621+0000","last_peered":"2024-01-16T15:14:58.265621+0000","last_clean":"2024-01-16T15:14:58.265621+0000","last_became_active":"2024-01-16T15:14:57.268601+0000","last_became_peered":"2024-01-16T15:14:57.268601+0000","last_unstale":"2024-01-16T15:14:58.265621+0000","last_undegraded":"2024-01-16T15:14:58.265621+0000","last_fullsized":"2024-01-16T15:14:58.265621+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:25.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.251818+0000","last_change":"2024-01-16T15:14:56.257392+0000","last_active":"2024-01-16T15:14:57.251818+0000","last_peered":"2024-01-16T15:14:57.251818+0000","last_clean":"2024-01-16T15:14:57.251818+0000","last_became_active":"2024-01-16T15:14:56.256861+0000","last_became_peered":"2024-01-16T15:14:56.256861+0000","last_unstale":"2024-01-16T15:14:57.251818+0000","last_undegraded":"2024-01-16T15:14:57.251818+0000","last_fullsized":"2024-01-16T15:14:57.251818+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:53:17.250545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.263300+0000","last_change":"2024-01-16T15:14:57.285776+0000","last_active":"2024-01-16T15:14:58.263300+0000","last_peered":"2024-01-16T15:14:58.263300+0000","last_clean":"2024-01-16T15:14:58.263300+0000","last_became_active":"2024-01-16T15:14:57.285440+0000","last_became_peered":"2024-01-16T15:14:57.285440+0000","last_unstale":"2024-01-16T15:14:58.263300+0000","last_undegraded":"2024-01-16T15:14:58.263300+0000","last_fullsized":"2024-01-16T15:14:58.263300+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:31.070337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.248936+0000","last_change":"2024-01-16T15:14:56.257589+0000","last_active":"2024-01-16T15:14:57.248936+0000","last_peered":"2024-01-16T15:14:57.248936+0000","last_clean":"2024-01-16T15:14:57.248936+0000","last_became_active":"2024-01-16T15:14:56.257240+0000","last_became_peered":"2024-01-16T15:14:56.257240+0000","last_unstale":"2024-01-16T15:14:57.248936+0000","last_undegraded":"2024-01-16T15:14:57.248936+0000","last_fullsized":"2024-01-16T15:14:57.248936+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:44:09.006145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.267922+0000","last_change":"2024-01-16T15:14:57.255621+0000","last_active":"2024-01-16T15:14:58.267922+0000","last_peered":"2024-01-16T15:14:58.267922+0000","last_clean":"2024-01-16T15:14:58.267922+0000","last_became_active":"2024-01-16T15:14:57.255297+0000","last_became_peered":"2024-01-16T15:14:57.255297+0000","last_unstale":"2024-01-16T15:14:58.267922+0000","last_undegraded":"2024-01-16T15:14:58.267922+0000","last_fullsized":"2024-01-16T15:14:58.267922+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:48:15.656662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.264990+0000","last_change":"2024-01-16T15:14:56.251203+0000","last_active":"2024-01-16T15:14:57.264990+0000","last_peered":"2024-01-16T15:14:57.264990+0000","last_clean":"2024-01-16T15:14:57.264990+0000","last_became_active":"2024-01-16T15:14:56.250531+0000","last_became_peered":"2024-01-16T15:14:56.250531+0000","last_unstale":"2024-01-16T15:14:57.264990+0000","last_undegraded":"2024-01-16T15:14:57.264990+0000","last_fullsized":"2024-01-16T15:14:57.264990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:33:59.694443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-01-16T15:14:58.264464+0000","last_change":"2024-01-16T15:14:57.269312+0000","last_active":"2024-01-16T15:14:58.264464+0000","last_peered":"2024-01-16T15:14:58.264464+0000","last_clean":"2024-01-16T15:14:58.264464+0000","last_became_active":"2024-01-16T15:14:57.268972+0000","last_became_peered":"2024-01-16T15:14:57.268972+0000","last_unstale":"2024-01-16T15:14:58.264464+0000","last_undegraded":"2024-01-16T15:14:58.264464+0000","last_fullsized":"2024-01-16T15:14:58.264464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:56.209775+0000","last_clean_scrub_stamp":"2024-01-16T15:14:56.209775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:34.652244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"100'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.272405+0000","last_change":"2024-01-16T15:14:56.252064+0000","last_active":"2024-01-16T15:14:57.272405+0000","last_peered":"2024-01-16T15:14:57.272405+0000","last_clean":"2024-01-16T15:14:57.272405+0000","last_became_active":"2024-01-16T15:14:56.251632+0000","last_became_peered":"2024-01-16T15:14:56.251632+0000","last_unstale":"2024-01-16T15:14:57.272405+0000","last_undegraded":"2024-01-16T15:14:57.272405+0000","last_fullsized":"2024-01-16T15:14:57.272405+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:14:55.194364+0000","last_clean_scrub_stamp":"2024-01-16T15:14:55.194364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:45:57.674413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":235,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-16T15:14:57.262940+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:14:57.262940+0000","last_peered":"2024-01-16T15:14:57.262940+0000","last_clean":"2024-01-16T15:14:57.262940+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:14:57.262940+0000","last_undegraded":"2024-01-16T15:14:57.262940+0000","last_fullsized":"2024-01-16T15:14:57.262940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542335,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73180,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298660,"statfs":{"total":96636764160,"available":96561827840,"internally_reserved":0,"allocated":1847296,"data_stored":1128209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.4359999999999999,"15min":1.486},"min":{"1min":0.63500000000000001,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":2.0779999999999998,"5min":15.449999999999999,"15min":27.469000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.7909999999999999,"15min":1.6319999999999999},"min":{"1min":0.79500000000000004,"5min":0.60299999999999998,"15min":0.47099999999999997},"max":{"1min":2.3069999999999999,"5min":25.503,"15min":25.503},"last":1.605}]},{"osd":1,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1899999999999999,"15min":1.2909999999999999},"min":{"1min":0.55100000000000005,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.679,"5min":5.5010000000000003,"15min":6.718},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.21,"15min":1.347},"min":{"1min":0.75700000000000001,"5min":0.64700000000000002,"15min":0.373},"max":{"1min":2.0110000000000001,"5min":4.3369999999999997,"15min":15.452},"last":1.571}]},{"osd":2,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.6040000000000001,"15min":1.4970000000000001},"min":{"1min":0.81999999999999995,"5min":0.629,"15min":0.48299999999999998},"max":{"1min":11.536,"5min":11.536,"15min":11.536},"last":1.347},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.899,"15min":1.5820000000000001},"min":{"1min":1.0740000000000001,"5min":0.64400000000000002,"15min":0.504},"max":{"1min":11.576000000000001,"5min":25.478000000000002,"15min":25.478000000000002},"last":1.3899999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.4830000000000001,"15min":1.571},"min":{"1min":0.67000000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.7789999999999999,"5min":13.907999999999999,"15min":13.907999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.526,"15min":1.5680000000000001},"min":{"1min":0.93200000000000005,"5min":0.66900000000000004,"15min":0.62},"max":{"1min":2.032,"5min":8.6999999999999993,"15min":13.193},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.764,"15min":1.496},"min":{"1min":0.90900000000000003,"5min":0.47799999999999998,"15min":0.376},"max":{"1min":1.742,"5min":25.401,"15min":25.401},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.59599999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.645,"5min":6.7000000000000002,"15min":10.134},"last":1.4850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.5880000000000001,"15min":1.4990000000000001},"min":{"1min":0.69399999999999995,"5min":0.432,"15min":0.432},"max":{"1min":2.1259999999999999,"5min":12.528,"15min":12.528},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.802,"15min":1.6160000000000001},"min":{"1min":0.92900000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.8420000000000001,"5min":26.327000000000002,"15min":26.327000000000002},"last":1.5109999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:41 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.994,"15min":1.73},"min":{"1min":1.087,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.8520000000000001,"5min":25.451000000000001,"15min":25.451000000000001},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.663,"15min":1.6499999999999999},"min":{"1min":0.89800000000000002,"5min":0.60199999999999998,"15min":0.44700000000000001},"max":{"1min":1.8080000000000001,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.55}]}]},{"osd":6,"up_from":14,"seq":60129542334,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":1253376,"data_stored":538381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2030000000000001,"15min":1.6539999999999999},"min":{"1min":0.73699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.8120000000000001,"5min":4.6600000000000001,"15min":30.164999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3029999999999999,"15min":1.732},"min":{"1min":0.83799999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.8560000000000001,"5min":4.8150000000000004,"15min":30.280999999999999},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.292,"15min":1.482},"min":{"1min":0.65800000000000003,"5min":0.54100000000000004,"15min":0.47899999999999998},"max":{"1min":1.891,"5min":5.9859999999999998,"15min":28.122},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.71699999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":3.7069999999999999,"5min":7.7489999999999997,"15min":8.9719999999999995},"last":0.79000000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.5840000000000001,"15min":1.7},"min":{"1min":0.90900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":1.673,"5min":11.869,"15min":26.109000000000002},"last":1.169},{"interface":"front","average":{"1min":1.446,"5min":1.726,"15min":1.8500000000000001},"min":{"1min":1.107,"5min":0.71099999999999997,"15min":0.65800000000000003},"max":{"1min":1.952,"5min":12.179,"15min":26.161000000000001},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4119999999999999,"15min":1.609},"min":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.44700000000000001},"max":{"1min":1.7490000000000001,"5min":9.0199999999999996,"15min":17.75},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.538,"15min":1.6200000000000001},"min":{"1min":0.77100000000000002,"5min":0.623,"15min":0.55700000000000005},"max":{"1min":2.028,"5min":9.0660000000000007,"15min":17.794},"last":1.224}]},{"osd":4,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4530000000000001,"15min":1.546},"min":{"1min":0.94899999999999995,"5min":0.45500000000000002,"15min":0.45000000000000001},"max":{"1min":2.2639999999999998,"5min":6.2220000000000004,"15min":15.571},"last":1.331},{"interface":"front","average":{"1min":1.423,"5min":1.327,"15min":1.5740000000000001},"min":{"1min":0.80300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.9510000000000001,"5min":5.0330000000000004,"15min":23.667999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.5249999999999999,"15min":1.46},"min":{"1min":0.86599999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1829999999999998,"5min":23.137,"15min":23.137},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.7290000000000001,"15min":1.6779999999999999},"min":{"1min":1.2609999999999999,"5min":0.48099999999999998,"15min":0.47099999999999997},"max":{"1min":2.1200000000000001,"5min":25.684000000000001,"15min":25.684000000000001},"last":1.282}]},{"osd":7,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.48,"15min":1.518},"min":{"1min":1.036,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.4500000000000002,"5min":5.1239999999999997,"15min":11.597},"last":1.359},{"interface":"front","average":{"1min":1.653,"5min":1.6890000000000001,"15min":1.6910000000000001},"min":{"1min":1.1930000000000001,"5min":0.61599999999999999,"15min":0.499},"max":{"1min":2.375,"5min":6.2770000000000001,"15min":11.788},"last":1.4019999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80920,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290920,"statfs":{"total":96636764160,"available":96553902080,"internally_reserved":0,"allocated":1249280,"data_stored":537799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.349,"15min":1.579},"min":{"1min":0.61399999999999999,"5min":0.41299999999999998,"15min":0.39000000000000001},"max":{"1min":1.6479999999999999,"5min":22.289000000000001,"15min":26.228999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.1639999999999999,"15min":1.845},"min":{"1min":0.73799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.7889999999999999,"5min":5.4290000000000003,"15min":56.011000000000003},"last":0.93000000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.179,"15min":1.375},"min":{"1min":0.69899999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":4.2400000000000002,"15min":16.687999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.7250000000000001,"15min":1.913},"min":{"1min":0.83099999999999996,"5min":0.46999999999999997,"15min":0.40999999999999998},"max":{"1min":1.577,"5min":22.451000000000001,"15min":67.930000000000007},"last":1.589}]},{"osd":2,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.325,"15min":1.55},"min":{"1min":0.88800000000000001,"5min":0.54800000000000004,"15min":0.47099999999999997},"max":{"1min":1.5700000000000001,"5min":5.4950000000000001,"15min":13.193},"last":1.05},{"interface":"front","average":{"1min":1.375,"5min":1.454,"15min":1.5},"min":{"1min":0.80000000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":3.9590000000000001,"5min":16.721,"15min":16.721},"last":1.3600000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.5960000000000001,"15min":1.7889999999999999},"min":{"1min":1.0189999999999999,"5min":0.59599999999999997,"15min":0.57999999999999996},"max":{"1min":1.7410000000000001,"5min":22.382000000000001,"15min":22.382000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.5089999999999999,"15min":1.8080000000000001},"min":{"1min":0.89900000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.915,"5min":6.0540000000000003,"15min":14.685},"last":1.3140000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4259999999999999,"15min":1.456},"min":{"1min":0.621,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.6759999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.619,"15min":1.581},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.379},"max":{"1min":1.948,"5min":27.006,"15min":27.006},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.454,"15min":1.5820000000000001},"min":{"1min":0.72799999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.9339999999999999,"5min":5.1230000000000002,"15min":13.263999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.641},"min":{"1min":0.82099999999999995,"5min":0.57599999999999996,"15min":0.505},"max":{"1min":1.9099999999999999,"5min":5.9710000000000001,"15min":9.5269999999999992},"last":1.619}]},{"osd":7,"last update":"Tue Jan 16 15:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.8520000000000001,"15min":1.802},"min":{"1min":0.78000000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.988,"5min":35.127000000000002,"15min":35.127000000000002},"last":1.649},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6120000000000001,"15min":1.79},"min":{"1min":0.94999999999999996,"5min":0.55800000000000005,"15min":0.42999999999999999},"max":{"1min":2.0459999999999998,"5min":5.1959999999999997,"15min":13.491},"last":1.4139999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542334,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64296,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307544,"statfs":{"total":96636764160,"available":96570925056,"internally_reserved":0,"allocated":1265664,"data_stored":539465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.103,"15min":1.2509999999999999},"min":{"1min":0.628,"5min":0.51900000000000002,"15min":0.502},"max":{"1min":2.23,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.53},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.129,"15min":1.4830000000000001},"min":{"1min":0.621,"5min":0.54400000000000004,"15min":0.501},"max":{"1min":1.917,"5min":1.917,"15min":60.106000000000002},"last":1.702}]},{"osd":1,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.379,"15min":1.6919999999999999},"min":{"1min":0.89400000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8420000000000001,"5min":3.3029999999999999,"15min":22.143000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.369,"15min":1.5369999999999999},"min":{"1min":0.76000000000000001,"5min":0.628,"15min":0.57099999999999995},"max":{"1min":1.7849999999999999,"5min":6.3719999999999999,"15min":22.181999999999999},"last":0.91600000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.294,"15min":1.319},"min":{"1min":0.82599999999999996,"5min":0.58099999999999996,"15min":0.52800000000000002},"max":{"1min":2.008,"5min":6.9480000000000004,"15min":9.7140000000000004},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3109999999999999,"15min":1.73},"min":{"1min":0.89900000000000002,"5min":0.63100000000000001,"15min":0.52400000000000002},"max":{"1min":1.752,"5min":3.1720000000000002,"15min":35.024999999999999},"last":1.486}]},{"osd":3,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3360000000000001,"15min":1.5820000000000001},"min":{"1min":0.85899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.8879999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.631},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.387,"15min":1.391},"min":{"1min":1.0069999999999999,"5min":0.623,"15min":0.57999999999999996},"max":{"1min":2.165,"5min":5.4690000000000003,"15min":7.8140000000000001},"last":1.413}]},{"osd":5,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.385,"15min":1.4810000000000001},"min":{"1min":0.94699999999999995,"5min":0.47199999999999998,"15min":0.38900000000000001},"max":{"1min":2.0600000000000001,"5min":2.1299999999999999,"15min":14.568},"last":1.95},{"interface":"front","average":{"1min":1.552,"5min":1.4510000000000001,"15min":1.4630000000000001},"min":{"1min":1.016,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.3570000000000002,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":1.304}]},{"osd":6,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.395,"15min":1.4019999999999999},"min":{"1min":0.89300000000000002,"5min":0.49099999999999999,"15min":0.436},"max":{"1min":2.4350000000000001,"5min":3.8929999999999998,"15min":9.6630000000000003},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.585,"15min":1.6679999999999999},"min":{"1min":1.004,"5min":0.79200000000000004,"15min":0.45100000000000001},"max":{"1min":2.3980000000000001,"5min":8.8650000000000002,"15min":14.317},"last":1.869}]},{"osd":7,"last update":"Tue Jan 16 15:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.647,"15min":1.583},"min":{"1min":1.073,"5min":0.61099999999999999,"15min":0.52100000000000002},"max":{"1min":2.9319999999999999,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6679999999999999,"15min":1.5920000000000001},"min":{"1min":1.1499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":2.3570000000000002,"5min":9.2270000000000003,"15min":9.2270000000000003},"last":1.764}]}]},{"osd":3,"up_from":14,"seq":60129542336,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76768,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295072,"statfs":{"total":96636764160,"available":96558153728,"internally_reserved":0,"allocated":1261568,"data_stored":539363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.1240000000000001,"15min":1.5169999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.9299999999999999,"5min":7.2229999999999999,"15min":50.988999999999997},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.4039999999999999,"15min":1.474},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5940000000000001,"5min":20.274000000000001,"15min":20.274000000000001},"last":1.4770000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.359,"15min":1.403},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.6070000000000002,"5min":13.576000000000001,"15min":18.192},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.5449999999999999,"15min":1.744},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":1.8839999999999999,"5min":12.696,"15min":20.385999999999999},"last":0.92400000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.865,"15min":1.7150000000000001},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.952,"5min":42.840000000000003,"15min":42.840000000000003},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.853,"15min":1.833},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.9550000000000001,"5min":38.707000000000001,"15min":38.707000000000001},"last":1.0529999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.6060000000000001,"15min":1.7669999999999999},"min":{"1min":0.78200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.327,"5min":14.148,"15min":18.952000000000002},"last":1.385},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.7090000000000001,"15min":1.7829999999999999},"min":{"1min":0.745,"5min":0.61599999999999999,"15min":0.54800000000000004},"max":{"1min":1.893,"5min":14.454000000000001,"15min":18.088999999999999},"last":1.512}]},{"osd":5,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.611,"15min":1.881},"min":{"1min":0.73199999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.1509999999999998,"5min":5.7910000000000004,"15min":24.349},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.788,"15min":1.9390000000000001},"min":{"1min":0.753,"5min":0.66100000000000003,"15min":0.61899999999999999},"max":{"1min":2.5379999999999998,"5min":14.192,"15min":14.192},"last":1.5840000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.931,"15min":1.8620000000000001},"min":{"1min":0.77200000000000002,"5min":0.69699999999999995,"15min":0.52200000000000002},"max":{"1min":2.0449999999999999,"5min":22.687000000000001,"15min":22.687000000000001},"last":1.29},{"interface":"front","average":{"1min":1.512,"5min":1.716,"15min":1.9650000000000001},"min":{"1min":0.73799999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2949999999999999,"5min":6.1299999999999999,"15min":24.058},"last":1.54}]},{"osd":7,"last update":"Tue Jan 16 15:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":2.2200000000000002,"15min":2.0819999999999999},"min":{"1min":0.85099999999999998,"5min":0.76900000000000002,"15min":0.55800000000000005},"max":{"1min":2.464,"5min":35.570999999999998,"15min":35.570999999999998},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":2.359,"15min":2.0529999999999999},"min":{"1min":0.81100000000000005,"5min":0.64700000000000002,"15min":0.58899999999999997},"max":{"1min":2.0019999999999998,"5min":37.607999999999997,"15min":37.607999999999997},"last":1.4450000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76756,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295084,"statfs":{"total":96636764160,"available":96558166016,"internally_reserved":0,"allocated":1249280,"data_stored":537763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0980000000000001,"15min":1.488},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.24399999999999999},"max":{"1min":1.3939999999999999,"5min":11.112,"15min":64.387},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.3799999999999999,"15min":1.389},"min":{"1min":0.67000000000000004,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.514,"5min":24.308,"15min":24.308},"last":0.89500000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1639999999999999,"15min":1.464},"min":{"1min":0.66300000000000003,"5min":0.42299999999999999,"15min":0.28299999999999997},"max":{"1min":2.0579999999999998,"5min":11.406000000000001,"15min":27.263000000000002},"last":0.755},{"interface":"front","average":{"1min":1.409,"5min":1.7270000000000001,"15min":1.819},"min":{"1min":0.70999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.901,"5min":23.602,"15min":23.602},"last":0.71299999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.306,"15min":1.5920000000000001},"min":{"1min":0.755,"5min":0.53400000000000003,"15min":0.33100000000000002},"max":{"1min":1.9299999999999999,"5min":3.1080000000000001,"15min":27.311},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.421,"15min":1.649},"min":{"1min":0.86899999999999999,"5min":0.49099999999999999,"15min":0.47099999999999997},"max":{"1min":1.782,"5min":9.7390000000000008,"15min":26.555},"last":0.82999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5369999999999999,"15min":1.6739999999999999},"min":{"1min":1.0649999999999999,"5min":0.54000000000000004,"15min":0.38900000000000001},"max":{"1min":2.1040000000000001,"5min":12.779,"15min":26.503},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.833,"15min":1.7390000000000001},"min":{"1min":0.98999999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.4319999999999999,"5min":25.661000000000001,"15min":25.661000000000001},"last":1.294}]},{"osd":5,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.478,"15min":1.7290000000000001},"min":{"1min":0.81599999999999995,"5min":0.56699999999999995,"15min":0.45800000000000002},"max":{"1min":2.3490000000000002,"5min":3.0209999999999999,"15min":27.149000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.581,"5min":1.6200000000000001,"15min":1.7649999999999999},"min":{"1min":1.1739999999999999,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":2.6589999999999998,"5min":9.7850000000000001,"15min":26.349},"last":1.2789999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":2.0590000000000002,"15min":1.9399999999999999},"min":{"1min":1.0189999999999999,"5min":0.504,"15min":0.504},"max":{"1min":2.7149999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":1.073},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.5640000000000001,"15min":1.899},"min":{"1min":1.0209999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":15.968999999999999,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.137}]},{"osd":7,"last update":"Tue Jan 16 15:14:54 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":2,"15min":1.9510000000000001},"min":{"1min":1.036,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.4889999999999999,"5min":24.509,"15min":24.509},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.617,"5min":1.73,"15min":1.9159999999999999},"min":{"1min":1.089,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.3479999999999999,"5min":10.455,"15min":27.207999999999998},"last":1.3420000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542334,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76752,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295088,"statfs":{"total":96636764160,"available":96558170112,"internally_reserved":0,"allocated":1245184,"data_stored":537729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0309999999999999,"15min":1.2689999999999999},"min":{"1min":0.61199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.8560000000000001,"5min":6.2249999999999996,"15min":14.407},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.075,"5min":1.1739999999999999,"15min":1.306},"min":{"1min":0.66100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.3999999999999999,"5min":3.5699999999999998,"15min":13.930999999999999},"last":1.4239999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.238,"15min":1.3899999999999999},"min":{"1min":0.57799999999999996,"5min":0.31,"15min":0.31},"max":{"1min":14.269,"5min":14.269,"15min":16.488},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.421,"15min":1.702},"min":{"1min":0.53000000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4530000000000001,"5min":21.446000000000002,"15min":22.550999999999998},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.647,"15min":1.7649999999999999},"min":{"1min":0.63400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.9510000000000001,"5min":15.436,"15min":22.471},"last":1.323},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.504,"15min":1.653},"min":{"1min":0.77500000000000002,"5min":0.46600000000000003,"15min":0.35699999999999998},"max":{"1min":1.9610000000000001,"5min":14.763999999999999,"15min":22.550999999999998},"last":1.5029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.52,"15min":1.6699999999999999},"min":{"1min":0.96799999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.863,"5min":5.3789999999999996,"15min":12.675000000000001},"last":1.296},{"interface":"front","average":{"1min":1.613,"5min":1.5900000000000001,"15min":1.728},"min":{"1min":1.036,"5min":0.433,"15min":0.433},"max":{"1min":5.1619999999999999,"5min":5.6849999999999996,"15min":19.236999999999998},"last":1.3580000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.79,"15min":1.7629999999999999},"min":{"1min":1.155,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.2040000000000002,"5min":21.117000000000001,"15min":21.117000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.651,"15min":1.744},"min":{"1min":1.099,"5min":0.65800000000000003,"15min":0.64800000000000002},"max":{"1min":5.2359999999999998,"5min":5.9009999999999998,"15min":17.138000000000002},"last":1.2649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.595,"15min":1.8440000000000001},"min":{"1min":1.0169999999999999,"5min":0.79800000000000004,"15min":0.72199999999999998},"max":{"1min":2.298,"5min":6.3710000000000004,"15min":14.781000000000001},"last":1.194},{"interface":"front","average":{"1min":1.671,"5min":1.663,"15min":1.849},"min":{"1min":1.2310000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2719999999999998,"5min":5.4630000000000001,"15min":23.420999999999999},"last":1.6080000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7190000000000001,"15min":1.8280000000000001},"min":{"1min":1.1619999999999999,"5min":0.71299999999999997,"15min":0.60699999999999998},"max":{"1min":5.2910000000000004,"5min":5.8300000000000001,"15min":17.010000000000002},"last":1.147},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.7609999999999999,"15min":1.9730000000000001},"min":{"1min":0.91600000000000004,"5min":0.78700000000000003,"15min":0.75700000000000001},"max":{"1min":2.1539999999999999,"5min":10.316000000000001,"15min":14.884},"last":1.472}]}]},{"osd":0,"up_from":14,"seq":60129542335,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69012,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302828,"statfs":{"total":96636764160,"available":96566095872,"internally_reserved":0,"allocated":1843200,"data_stored":1128679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":68811201},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.48,"15min":1.494},"min":{"1min":0.47399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.8380000000000001,"5min":43.459000000000003,"15min":43.459000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.7609999999999999,"15min":1.6739999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28999999999999998},"max":{"1min":4.9779999999999998,"5min":43.627000000000002,"15min":43.627000000000002},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2649999999999999,"15min":1.4359999999999999},"min":{"1min":0.55100000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.0570000000000004,"5min":7.5940000000000003,"15min":18.861000000000001},"last":1.341},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.353,"15min":1.5069999999999999},"min":{"1min":0.49099999999999999,"5min":0.42999999999999999,"15min":0.378},"max":{"1min":5.3550000000000004,"5min":8.9979999999999993,"15min":38.893000000000001},"last":0.95299999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.135,"15min":1.5880000000000001},"min":{"1min":0.77100000000000002,"5min":0.45300000000000001,"15min":0.31900000000000001},"max":{"1min":4.9050000000000002,"5min":4.9050000000000002,"15min":23.687999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.417,"5min":1.516,"15min":1.6659999999999999},"min":{"1min":0.624,"5min":0.434,"15min":0.434},"max":{"1min":5.1980000000000004,"5min":16.978000000000002,"15min":18.966999999999999},"last":1.569}]},{"osd":4,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4039999999999999,"15min":1.524},"min":{"1min":0.96999999999999997,"5min":0.63400000000000001,"15min":0.58899999999999997},"max":{"1min":5.1040000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.536},{"interface":"front","average":{"1min":1.607,"5min":1.492,"15min":1.718},"min":{"1min":0.83099999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":5.3109999999999999,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.7989999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.6279999999999999,"15min":1.861},"min":{"1min":0.80700000000000005,"5min":0.55600000000000005,"15min":0.54600000000000004},"max":{"1min":5.1589999999999998,"5min":13.016,"15min":24.611000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.633,"15min":1.7170000000000001},"min":{"1min":0.871,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":7.2699999999999996,"5min":8.3949999999999996,"15min":20.199000000000002},"last":1.2669999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.7210000000000001,"15min":2.0539999999999998},"min":{"1min":1.079,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":6.3339999999999996,"5min":10.342000000000001,"15min":26.998000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.718,"15min":1.982},"min":{"1min":1.0069999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":5.9690000000000003,"5min":10.279999999999999,"15min":24.478999999999999},"last":1.0609999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.734,"15min":1.901},"min":{"1min":0.91700000000000004,"5min":0.60899999999999999,"15min":0.58399999999999996},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.776,"5min":1.659,"15min":1.7569999999999999},"min":{"1min":1.165,"5min":0.65800000000000003,"15min":0.52800000000000002},"max":{"1min":6.149,"5min":6.149,"15min":20.279},"last":1.452}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:02.280 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:15:02.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:15:02.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.568+0000 7f5dab200640 1 Processor -- start 2024-01-16T15:15:02.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.569+0000 7f5dab200640 1 -- start start 2024-01-16T15:15:02.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.570+0000 7f5dab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 0x7f5dac0ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.570+0000 7f5dab200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dac070280 0x7f5dac0ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.570+0000 7f5dab200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac0b3e20 con 0x7f5dac06d3d0 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.570+0000 7f5dab200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac0706e0 con 0x7f5dac070280 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.570+0000 7f5dab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0ae820 con 0x7f5dac143590 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dac070280 0x7f5dac0ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dac070280 0x7f5dac0ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54152/0 (socket says 172.21.15.43:54152) 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 -- 172.21.15.43:0/252303154 learned_addr learned my addr 172.21.15.43:0/252303154 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5daa800640 1 --1- 172.21.15.43:0/252303154 >> v1:172.21.15.43:6789/0 conn(0x7f5dac06d3d0 0x7f5dac0b42b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:53106/0 (socket says 172.21.15.43:53106) 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da9e00640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 0x7f5dac0ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 -- 172.21.15.43:0/252303154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 msgr2=0x7f5dac0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 0x7f5dac0ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 -- 172.21.15.43:0/252303154 >> v1:172.21.15.43:6789/0 conn(0x7f5dac06d3d0 legacy=0x7f5dac0b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da3e00640 1 -- 172.21.15.43:0/252303154 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac0acee0 con 0x7f5dac070280 2024-01-16T15:15:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.571+0000 7f5da9e00640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 0x7f5dac0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:02.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5da3e00640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0ae2e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009850 tx=0x7f5d9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5047b7b7e8f878 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5da9400640 1 -- 172.21.15.43:0/252303154 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94004230 con 0x7f5dac070280 2024-01-16T15:15:02.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5da9400640 1 -- 172.21.15.43:0/252303154 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d94004390 con 0x7f5dac070280 2024-01-16T15:15:02.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5da9400640 1 -- 172.21.15.43:0/252303154 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94031830 con 0x7f5dac070280 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5dab200640 1 -- 172.21.15.43:0/252303154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 msgr2=0x7f5dac0ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.573+0000 7f5dab200640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0ae2e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009850 tx=0x7f5d9402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 -- 172.21.15.43:0/252303154 shutdown_connections 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0ae2e0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 --2- 172.21.15.43:0/252303154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac143590 0x7f5dac0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 -- 172.21.15.43:0/252303154 >> 172.21.15.43:0/252303154 conn(0x7f5dac0b0aa0 msgr2=0x7f5dac0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 -- 172.21.15.43:0/252303154 shutdown_connections 2024-01-16T15:15:02.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.574+0000 7f5dab200640 1 -- 172.21.15.43:0/252303154 wait complete. 2024-01-16T15:15:02.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.576+0000 7f5dab200640 1 Processor -- start 2024-01-16T15:15:02.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.576+0000 7f5dab200640 1 -- start start 2024-01-16T15:15:02.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac06d3d0 0x7f5dac15f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dac143590 0x7f5dac0c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac160670 con 0x7f5dac070280 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0c4010 con 0x7f5dac143590 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5dab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac15fdd0 con 0x7f5dac06d3d0 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54160/0 (socket says 172.21.15.43:54160) 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 -- 172.21.15.43:0/3034265256 learned_addr learned my addr 172.21.15.43:0/3034265256 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 -- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac06d3d0 msgr2=0x7f5dac15f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac06d3d0 0x7f5dac15f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 -- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dac143590 msgr2=0x7f5dac0c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dac143590 0x7f5dac0c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.577+0000 7f5da3e00640 1 -- 172.21.15.43:0/3034265256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94009460 con 0x7f5dac070280 2024-01-16T15:15:02.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5da3e00640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5d9402b910 tx=0x7f5d940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94031a40 con 0x7f5dac070280 2024-01-16T15:15:02.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d94031ba0 con 0x7f5dac070280 2024-01-16T15:15:02.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94003610 con 0x7f5dac070280 2024-01-16T15:15:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac1600b0 con 0x7f5dac070280 2024-01-16T15:15:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.578+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac0c5ed0 con 0x7f5dac070280 2024-01-16T15:15:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.579+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac0c5d30 con 0x7f5dac070280 2024-01-16T15:15:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.580+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f5d94003770 con 0x7f5dac070280 2024-01-16T15:15:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.581+0000 7f5da2000640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 0x7f5d8406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.581+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5d940aadd0 con 0x7f5dac070280 2024-01-16T15:15:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.581+0000 7f5da9e00640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 0x7f5d8406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.582+0000 7f5da9e00640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 0x7f5d8406e590 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006fd0 tx=0x7f5d9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.586+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5dac0c5d30 con 0x7f5dac070280 2024-01-16T15:15:02.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.850+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5dac0c5d30 con 0x7f5dac070280 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.855+0000 7f5da2000640 1 -- 172.21.15.43:0/3034265256 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5d9403f0b0 con 0x7f5dac070280 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:15:02.838 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:02.839 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 msgr2=0x7f5d8406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 0x7f5d8406e590 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006fd0 tx=0x7f5d9c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 msgr2=0x7f5dac0af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5d9402b910 tx=0x7f5d940046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 shutdown_connections 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d8406c0d0 0x7f5d8406e590 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dac143590 0x7f5dac0c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dac070280 0x7f5dac0af130 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 --2- 172.21.15.43:0/3034265256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5dac06d3d0 0x7f5dac15f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.858+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 >> 172.21.15.43:0/3034265256 conn(0x7f5dac0b0aa0 msgr2=0x7f5dac1427c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.859+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 shutdown_connections 2024-01-16T15:15:02.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:02.859+0000 7f5dab200640 1 -- 172.21.15.43:0/3034265256 wait complete. 2024-01-16T15:15:02.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:03.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.099+0000 7fec95000640 1 Processor -- start 2024-01-16T15:15:03.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.099+0000 7fec95000640 1 -- start start 2024-01-16T15:15:03.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec95000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec900cf050 0x7fec900cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec95000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec90140900 con 0x7fec900cd7b0 2024-01-16T15:15:03.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec95000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9015fa70 con 0x7fec9013dfd0 2024-01-16T15:15:03.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec95000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec900cfa40 con 0x7fec900cf050 2024-01-16T15:15:03.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec8fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fec900cd7b0 0x7fec900cdbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36922/0 (socket says 172.21.15.43:36922) 2024-01-16T15:15:03.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.101+0000 7fec8fe00640 1 -- 172.21.15.43:0/1610123807 learned_addr learned my addr 172.21.15.43:0/1610123807 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350820485 0 0) 0x7fec90140900 con 0x7fec900cd7b0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec86800640 1 --2- 172.21.15.43:0/1610123807 >> v2:172.21.15.69:3300/0 conn(0x7fec900cf050 0x7fec900cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec70003520 con 0x7fec900cd7b0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343604582 0 0) 0x7fec70003520 con 0x7fec900cd7b0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 >> v2:172.21.15.69:3300/0 conn(0x7fec900cf050 msgr2=0x7fec900cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 --2- 172.21.15.43:0/1610123807 >> v2:172.21.15.69:3300/0 conn(0x7fec900cf050 0x7fec900cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 >> v1:172.21.15.69:6789/0 conn(0x7fec9013dfd0 legacy=0x7fec9015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9015ee60 con 0x7fec900cd7b0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fec88001670 con 0x7fec900cd7b0 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 >> v1:172.21.15.43:6789/0 conn(0x7fec900cd7b0 legacy=0x7fec900cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 0x7fec70003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec700047c0 0x7fec70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700051d0 0x7fec7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec900cfa40 con 0x7fec70003a30 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7000c100 con 0x7fec700051d0 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.102+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7000c2c0 con 0x7fec700047c0 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 0x7fec70003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec86800640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec700047c0 0x7fec70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8fe00640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700051d0 0x7fec7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 -- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec700047c0 msgr2=0x7fec70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec700047c0 0x7fec70004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 -- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700051d0 msgr2=0x7fec7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700051d0 0x7fec7000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8f400640 1 -- 172.21.15.43:0/1610123807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70003520 con 0x7fec70003a30 2024-01-16T15:15:03.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.103+0000 7fec8fe00640 1 -- 172.21.15.43:0/1610123807 reap_dead start 2024-01-16T15:15:03.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.104+0000 7fec8f400640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 0x7fec70003e50 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fec80001750 tx=0x7fec700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723c67bdee8db2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec800052d0 con 0x7fec70003a30 2024-01-16T15:15:03.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8002e050 con 0x7fec70003a30 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec8ea00640 1 -- 172.21.15.43:0/1610123807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80003830 con 0x7fec70003a30 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec95000640 1 -- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 msgr2=0x7fec70003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec95000640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 0x7fec70003e50 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fec80001750 tx=0x7fec700057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec95000640 1 -- 172.21.15.43:0/1610123807 shutdown_connections 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec95000640 1 --2- 172.21.15.43:0/1610123807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec70003a30 0x7fec70003e50 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.105+0000 7fec95000640 1 -- 172.21.15.43:0/1610123807 >> 172.21.15.43:0/1610123807 conn(0x7fec9006a860 msgr2=0x7fec9006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.106+0000 7fec95000640 1 -- 172.21.15.43:0/1610123807 shutdown_connections 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.106+0000 7fec95000640 1 -- 172.21.15.43:0/1610123807 wait complete. 2024-01-16T15:15:03.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.107+0000 7fec95000640 1 Processor -- start 2024-01-16T15:15:03.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.108+0000 7fec95000640 1 -- start start 2024-01-16T15:15:03.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.108+0000 7fec95000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec70003a30 0x7fec900cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec95000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec901f3230 0x7fec9015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec95000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec95000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90157a90 con 0x7fec901570d0 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec95000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9015b380 con 0x7fec901f3230 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec95000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec901f36b0 con 0x7fec70003a30 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55362/0 (socket says 172.21.15.43:55362) 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 -- 172.21.15.43:0/1033842146 learned_addr learned my addr 172.21.15.43:0/1033842146 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8f400640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec70003a30 0x7fec900cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 -- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec70003a30 msgr2=0x7fec900cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec70003a30 0x7fec900cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 -- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec901f3230 msgr2=0x7fec9015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec901f3230 0x7fec9015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.109+0000 7fec8fe00640 1 -- 172.21.15.43:0/1033842146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9015dda0 con 0x7fec901570d0 2024-01-16T15:15:03.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec8fe00640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec88001200 tx=0x7fec8800d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88007440 con 0x7fec901570d0 2024-01-16T15:15:03.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec880075c0 con 0x7fec901570d0 2024-01-16T15:15:03.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88004a20 con 0x7fec901570d0 2024-01-16T15:15:03.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90159460 con 0x7fec901570d0 2024-01-16T15:15:03.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.110+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9015d7e0 con 0x7fec901570d0 2024-01-16T15:15:03.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.111+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9004e840 con 0x7fec901570d0 2024-01-16T15:15:03.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.115+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fec88004b80 con 0x7fec901570d0 2024-01-16T15:15:03.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.116+0000 7fec87e00640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 0x7fec6806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.116+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fec88087f50 con 0x7fec901570d0 2024-01-16T15:15:03.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.116+0000 7fec8f400640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 0x7fec6806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.117+0000 7fec8f400640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 0x7fec6806e5c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fec80033000 tx=0x7fec80009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.119+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec88054550 con 0x7fec901570d0 2024-01-16T15:15:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.354+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec901595f0 con 0x7fec901570d0 2024-01-16T15:15:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.355+0000 7fec87e00640 1 -- 172.21.15.43:0/1033842146 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fec88057b70 con 0x7fec901570d0 2024-01-16T15:15:03.338 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:03.338 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/1137375185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1137375185},{"type":"v1","addr":"172.21.15.43:6836","nonce":1137375185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2958556276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2958556276},{"type":"v1","addr":"172.21.15.43:6837","nonce":2958556276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/208706681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":208706681},{"type":"v1","addr":"172.21.15.69:6835","nonce":208706681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:14:56.690837+0000","modified":"2024-01-16T15:14:59.788652+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8568":{"gid":8568,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6833/3755062795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3755062795},{"type":"v1","addr":"172.21.15.69:6833","nonce":3755062795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:15:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 66 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 msgr2=0x7fec6806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 0x7fec6806e5c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fec80033000 tx=0x7fec80009030 comp rx=0 tx=0).stop 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 msgr2=0x7fec90157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec88001200 tx=0x7fec8800d810 comp rx=0 tx=0).stop 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 shutdown_connections 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fec6806c100 0x7fec6806e5c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec901f3230 0x7fec9015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec901570d0 0x7fec90157550 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 --2- 172.21.15.43:0/1033842146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec70003a30 0x7fec900cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.357+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 >> 172.21.15.43:0/1033842146 conn(0x7fec9006a860 msgr2=0x7fec9015e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.358+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 shutdown_connections 2024-01-16T15:15:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:03.358+0000 7fec95000640 1 -- 172.21.15.43:0/1033842146 wait complete. 2024-01-16T15:15:03.764 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:14:56.690837+0000', 'modified': '2024-01-16T15:14:59.788652+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8568}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8568': {'gid': 8568, 'name': 'b', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6833/3755062795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 3755062795}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 3755062795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:15:03.764 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:15:03.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-16T15:15:03.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.008+0000 7faf48a00640 1 Processor -- start 2024-01-16T15:15:03.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.008+0000 7faf48a00640 1 -- start start 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf4015b340 0x7faf400abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf400cbcb0 0x7faf4006d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf400ce880 0x7faf4006da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf400ac530 con 0x7faf400cbcb0 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf400b2080 con 0x7faf400ce880 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.009+0000 7faf48a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400b2360 con 0x7faf4015b340 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf400cbcb0 0x7faf4006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faf400cbcb0 0x7faf4006d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55366/0 (socket says 172.21.15.43:55366) 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 -- 172.21.15.43:0/3302245376 learned_addr learned my addr 172.21.15.43:0/3302245376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf46000640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf4015b340 0x7faf400abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf46a00640 1 --2- 172.21.15.43:0/3302245376 >> v2:172.21.15.69:3300/0 conn(0x7faf400ce880 0x7faf4006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 -- 172.21.15.43:0/3302245376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf4015b340 msgr2=0x7faf400abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf4015b340 0x7faf400abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 -- 172.21.15.43:0/3302245376 >> v2:172.21.15.69:3300/0 conn(0x7faf400ce880 msgr2=0x7faf4006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 --2- 172.21.15.43:0/3302245376 >> v2:172.21.15.69:3300/0 conn(0x7faf400ce880 0x7faf4006da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.010+0000 7faf45600640 1 -- 172.21.15.43:0/3302245376 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400b2640 con 0x7faf400cbcb0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.011+0000 7faf45600640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400cbcb0 0x7faf4006d520 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7faf400ce240 tx=0x7faf2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4358a31053119e4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf44c00640 1 -- 172.21.15.43:0/3302245376 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28031870 con 0x7faf400cbcb0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf44c00640 1 -- 172.21.15.43:0/3302245376 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf280319d0 con 0x7faf400cbcb0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf44c00640 1 -- 172.21.15.43:0/3302245376 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28031d60 con 0x7faf400cbcb0 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 -- 172.21.15.43:0/3302245376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400cbcb0 msgr2=0x7faf4006d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400cbcb0 0x7faf4006d520 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7faf400ce240 tx=0x7faf2802b910 comp rx=0 tx=0).stop 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 -- 172.21.15.43:0/3302245376 shutdown_connections 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 --2- 172.21.15.43:0/3302245376 >> v2:172.21.15.69:3300/0 conn(0x7faf400ce880 0x7faf4006da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400cbcb0 0x7faf4006d520 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 --2- 172.21.15.43:0/3302245376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf4015b340 0x7faf400abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 -- 172.21.15.43:0/3302245376 >> 172.21.15.43:0/3302245376 conn(0x7faf400b0d80 msgr2=0x7faf400b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.012+0000 7faf48a00640 1 -- 172.21.15.43:0/3302245376 shutdown_connections 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.013+0000 7faf48a00640 1 -- 172.21.15.43:0/3302245376 wait complete. 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.015+0000 7faf48a00640 1 Processor -- start 2024-01-16T15:15:04.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.015+0000 7faf48a00640 1 -- start start 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf400cbcb0 0x7faf4015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf4015b340 0x7faf40158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400c8400 con 0x7faf400ce880 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400c8650 con 0x7faf4015b340 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf48a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400c8930 con 0x7faf400cbcb0 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf45600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf45600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55380/0 (socket says 172.21.15.43:55380) 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf45600640 1 -- 172.21.15.43:0/2742588645 learned_addr learned my addr 172.21.15.43:0/2742588645 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf46000640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf400cbcb0 0x7faf4015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.016+0000 7faf46a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf4015b340 0x7faf40158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 -- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf400cbcb0 msgr2=0x7faf4015c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf400cbcb0 0x7faf4015c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 -- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf4015b340 msgr2=0x7faf40158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf4015b340 0x7faf40158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 -- 172.21.15.43:0/2742588645 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280094d0 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf45600640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf3e000640 1 -- 172.21.15.43:0/2742588645 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2803ad30 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf3e000640 1 -- 172.21.15.43:0/2742588645 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2803ae90 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf3e000640 1 -- 172.21.15.43:0/2742588645 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28044630 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.017+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf400c8c10 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.018+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4015c670 con 0x7faf400ce880 2024-01-16T15:15:04.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.020+0000 7faf3e000640 1 -- 172.21.15.43:0/2742588645 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7faf2804d450 con 0x7faf400ce880 2024-01-16T15:15:04.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.021+0000 7faf3e000640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 0x7faf1806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.021+0000 7faf3e000640 1 -- 172.21.15.43:0/2742588645 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faf2802d040 con 0x7faf400ce880 2024-01-16T15:15:04.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.021+0000 7faf46000640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 0x7faf1806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.022+0000 7faf4b7e0740 1 Processor -- start 2024-01-16T15:15:04.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.022+0000 7faf4b7e0740 1 -- start start 2024-01-16T15:15:04.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.022+0000 7faf46000640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 0x7faf1806e590 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf30006fd0 tx=0x7faf30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x56051123c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511280560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051119b880 con 0x56051123cea0 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605111aaae0 con 0x560511237ba0 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605111a81d0 con 0x56051123c5e0 2024-01-16T15:15:04.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55382/0 (socket says 172.21.15.43:55382) 2024-01-16T15:15:04.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 -- 172.21.15.43:0/425176658 learned_addr learned my addr 172.21.15.43:0/425176658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 -- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 msgr2=0x560511280560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511280560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 -- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 msgr2=0x56051123c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x56051123c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.023+0000 7faf46000640 1 -- 172.21.15.43:0/425176658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56051116b860 con 0x56051123cea0 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.024+0000 7faf46000640 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7faf3000f990 tx=0x7faf30012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e3f95f9c84ab75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.024+0000 7faf36a00640 1 -- 172.21.15.43:0/425176658 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30014be0 con 0x56051123cea0 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.024+0000 7faf36a00640 1 -- 172.21.15.43:0/425176658 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf30014d40 con 0x56051123cea0 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.024+0000 7faf36a00640 1 -- 172.21.15.43:0/425176658 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3001f610 con 0x56051123cea0 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 -- 172.21.15.43:0/425176658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 msgr2=0x56051123d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7faf3000f990 tx=0x7faf30012000 comp rx=0 tx=0).stop 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 -- 172.21.15.43:0/425176658 shutdown_connections 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x56051123c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511280560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/425176658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123cea0 0x56051123d280 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 -- 172.21.15.43:0/425176658 >> 172.21.15.43:0/425176658 conn(0x5605111fec70 msgr2=0x560511282990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 -- 172.21.15.43:0/425176658 shutdown_connections 2024-01-16T15:15:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.025+0000 7faf4b7e0740 1 -- 172.21.15.43:0/425176658 wait complete. 2024-01-16T15:15:04.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.026+0000 7faf4b7e0740 1 -- start start 2024-01-16T15:15:04.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.026+0000 7faf4b7e0740 1 Processor -- start 2024-01-16T15:15:04.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x560511263090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511288310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605111a3ac0 con 0x56051123c5e0 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605111402f0 con 0x560511237ba0 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x56051115dc70 con 0x56051123cea0 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf4b7e0740 10 client.0 _ll_register_callbacks cb 0x5605111623b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf45600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf45600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55392/0 (socket says 172.21.15.43:55392) 2024-01-16T15:15:04.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.027+0000 7faf45600640 1 -- 172.21.15.43:0/170559481 learned_addr learned my addr 172.21.15.43:0/170559481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf46000640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x560511263090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf46a00640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511288310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 -- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 msgr2=0x560511288310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511288310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 -- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 msgr2=0x560511263090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x560511263090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 -- 172.21.15.43:0/170559481 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3000e040 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf46a00640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511288310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf46000640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x560511263090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf45600640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7faf2807af60 tx=0x7faf2807b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b970d10655958ac9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf2f400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280a8070 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56051119caa0 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.028+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faf2807f160 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf2807b4c0 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 10 client.8709 fetch_fsmap learned FSMap version 66 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5605111a2140 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56051119caa0 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faf2807a750 con 0x56051123c5e0 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 10 client.8709 fetch_fsmap finished waiting for FSMap version 66 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 10 client.8709 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T15:15:04.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 20 client.8709 populate_metadata read hostname 'smithi043' 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x56051127b1b0 0x56051128a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 4 client.8709 mds_command: new command op to 8568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.029+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 --> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56051119c7c0 con 0x56051127b1b0 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.030+0000 7faf46000640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x56051127b1b0 0x56051128a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.031+0000 7faf46000640 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x56051127b1b0 0x56051128a130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d46c6522df2f1a0 server_cookie=89ee1d7db7ee17cd in_seq=0 out_seq=0 2024-01-16T15:15:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.031+0000 7faf2f400640 10 client.8709 ms_handle_connect on v2:172.21.15.69:6832/3755062795 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.033+0000 7faf2f400640 1 -- 172.21.15.43:0/170559481 <== mds.0 v2:172.21.15.69:6832/3755062795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56051119c7c0 con 0x56051127b1b0 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.033+0000 7faf2f400640 10 client.8709 handle_command_reply: tid=0 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.033+0000 7faf4b7e0740 1 client.8709 shutdown 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.033+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 msgr2=0x56051127a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7faf2807af60 tx=0x7faf2807b260 comp rx=0 tx=0).stop 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 shutdown_connections 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x56051127b1b0 0x56051128a130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511288310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56051123c5e0 0x56051127a450 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/170559481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560511237ba0 0x560511263090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 >> 172.21.15.43:0/170559481 conn(0x5605111fec70 msgr2=0x560511236b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf2f400640 0 client.8709 ms_handle_reset on v2:172.21.15.69:6832/3755062795 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 shutdown_connections 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 1 -- 172.21.15.43:0/170559481 wait complete. 2024-01-16T15:15:04.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.034+0000 7faf4b7e0740 20 client.8709 trim_cache size 0 max 16384 2024-01-16T15:15:04.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.051+0000 7faf4b7e0740 1 Processor -- start 2024-01-16T15:15:04.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.051+0000 7faf4b7e0740 1 -- start start 2024-01-16T15:15:04.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123c5e0 0x560511288d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511281d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x560510f06f20 con 0x560511237ba0 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605111a5a40 con 0x56051123c5e0 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x56051127a850 con 0x56051123cea0 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55398/0 (socket says 172.21.15.43:55398) 2024-01-16T15:15:04.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.052+0000 7faf46000640 1 -- 172.21.15.43:0/34248373 learned_addr learned my addr 172.21.15.43:0/34248373 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46a00640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511281d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf45600640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123c5e0 0x560511288d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 -- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 msgr2=0x560511281d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511281d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 -- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123c5e0 msgr2=0x560511288d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123c5e0 0x560511288d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 -- 172.21.15.43:0/34248373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56051113fb80 con 0x560511237ba0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46a00640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511281d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.053+0000 7faf46000640 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7faf3000f960 tx=0x7faf30014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551ab57c44988962 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf37400640 1 -- 172.21.15.43:0/34248373 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30035070 con 0x560511237ba0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf37400640 1 -- 172.21.15.43:0/34248373 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3002e040 con 0x560511237ba0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf37400640 1 -- 172.21.15.43:0/34248373 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3002d070 con 0x560511237ba0 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 -- 172.21.15.43:0/34248373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 msgr2=0x560511295480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7faf3000f960 tx=0x7faf30014000 comp rx=0 tx=0).stop 2024-01-16T15:15:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 -- 172.21.15.43:0/34248373 shutdown_connections 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123c5e0 0x560511288d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123cea0 0x560511281d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/34248373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x560511295480 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 -- 172.21.15.43:0/34248373 >> 172.21.15.43:0/34248373 conn(0x5605111fec70 msgr2=0x5605112332a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 -- 172.21.15.43:0/34248373 shutdown_connections 2024-01-16T15:15:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.054+0000 7faf4b7e0740 1 -- 172.21.15.43:0/34248373 wait complete. 2024-01-16T15:15:04.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.055+0000 7faf4b7e0740 1 -- start start 2024-01-16T15:15:04.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.055+0000 7faf4b7e0740 1 Processor -- start 2024-01-16T15:15:04.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511281d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123cea0 0x560511282270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605112ed180 con 0x560511237ba0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x560511294b90 con 0x56051123cea0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605112ecc50 con 0x56051123c5e0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf4b7e0740 10 client.0 _ll_register_callbacks cb 0x7faf280775a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.056+0000 7faf46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55406/0 (socket says 172.21.15.43:55406) 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 -- 172.21.15.43:0/588767840 learned_addr learned my addr 172.21.15.43:0/588767840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 -- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 msgr2=0x560511281d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511281d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 -- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123cea0 msgr2=0x560511282270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123cea0 0x560511282270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 -- 172.21.15.43:0/588767840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56051116b860 con 0x560511237ba0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf46000640 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7faf30033040 tx=0x7faf30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6504ea3bb749d790 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf2f400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:15:04.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3002d070 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faf3001a040 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.057+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf30014d40 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf30035040 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 10 client.8715 fetch_fsmap learned FSMap version 66 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x560511165170 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf30014d40 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faf30028450 con 0x560511237ba0 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 10 client.8715 fetch_fsmap finished waiting for FSMap version 66 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 10 client.8715 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 20 client.8715 populate_metadata read hostname 'smithi043' 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x560511293ce0 0x5605112940c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 4 client.8715 mds_command: new command op to 8568 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:15:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.058+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 --> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560510ea46c0 con 0x560511293ce0 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.059+0000 7faf45600640 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x560511293ce0 0x5605112940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.059+0000 7faf45600640 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x560511293ce0 0x5605112940c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b4765f52bf8dac server_cookie=461931282770fc10 in_seq=0 out_seq=0 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.059+0000 7faf2f400640 10 client.8715 ms_handle_connect on v2:172.21.15.69:6832/3755062795 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf2f400640 1 -- 172.21.15.43:0/588767840 <== mds.0 v2:172.21.15.69:6832/3755062795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x560510ea46c0 con 0x560511293ce0 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf2f400640 10 client.8715 handle_command_reply: tid=0 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 client.8715 shutdown 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 msgr2=0x5605112a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7faf30033040 tx=0x7faf30004730 comp rx=0 tx=0).stop 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 shutdown_connections 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56051123cea0 0x560511282270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:6832/3755062795,v1:172.21.15.69:6833/3755062795] conn(0x560511293ce0 0x5605112940c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56051123c5e0 0x560511281d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 --2- 172.21.15.43:0/588767840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560511237ba0 0x5605112a53b0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf2f400640 0 client.8715 ms_handle_reset on v2:172.21.15.69:6832/3755062795 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.060+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 >> 172.21.15.43:0/588767840 conn(0x5605111fec70 msgr2=0x56051123b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 shutdown_connections 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf4b7e0740 1 -- 172.21.15.43:0/588767840 wait complete. 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf4b7e0740 20 client.8715 trim_cache size 0 max 16384 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 8568, 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:15:04.046 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 6.8176191060000004, 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 66, 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 101, 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 101, 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 12.166443044999999 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:15:04.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 msgr2=0x7faf1806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 0x7faf1806e590 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf30006fd0 tx=0x7faf30008040 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 msgr2=0x7faf4015d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf280046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 shutdown_connections 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7faf1806c0d0 0x7faf1806e590 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf4015b340 0x7faf40158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf400ce880 0x7faf4015d630 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 --2- 172.21.15.43:0/2742588645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf400cbcb0 0x7faf4015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.061+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 >> 172.21.15.43:0/2742588645 conn(0x7faf400b0d80 msgr2=0x7faf400ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.062+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 shutdown_connections 2024-01-16T15:15:04.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.062+0000 7faf48a00640 1 -- 172.21.15.43:0/2742588645 wait complete. 2024-01-16T15:15:04.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:04.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.301+0000 7f9906400640 1 Processor -- start 2024-01-16T15:15:04.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.301+0000 7f9906400640 1 -- start start 2024-01-16T15:15:04.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9906400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f990006cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9906400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9900161670 con 0x7f990015df80 2024-01-16T15:15:04.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9906400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990015e610 con 0x7f990006d080 2024-01-16T15:15:04.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9906400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990015e410 con 0x7f9900157770 2024-01-16T15:15:04.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9905a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f990015df80 0x7f9900070720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36932/0 (socket says 172.21.15.43:36932) 2024-01-16T15:15:04.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.303+0000 7f9905a00640 1 -- 172.21.15.43:0/2050641749 learned_addr learned my addr 172.21.15.43:0/2050641749 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:04.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f9905000640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f990006cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037424200 0 0) 0x7f9900161670 con 0x7f990015df80 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e0003520 con 0x7f990015df80 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1412412493 0 0) 0x7f98e0003520 con 0x7f990015df80 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 msgr2=0x7f990006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f990006cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 >> v1:172.21.15.69:6789/0 conn(0x7f990006d080 legacy=0x7f990015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900160d40 con 0x7f990015df80 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f98f80019f0 con 0x7f990015df80 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 >> v1:172.21.15.43:6789/0 conn(0x7f990015df80 legacy=0x7f9900070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0003a10 0x7f98e0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f98e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98e0004d00 0x7f98e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990015e410 con 0x7f98e0004d00 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e000ba90 con 0x7f98e0003a10 2024-01-16T15:15:04.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.304+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e000bc50 con 0x7f9900157770 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98e0004d00 0x7f98e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905000640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0003a10 0x7f98e0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f98ffe00640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f98e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 -- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 msgr2=0x7f98e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9900157770 0x7f98e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 -- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0003a10 msgr2=0x7f98e0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0003a10 0x7f98e0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.305+0000 7f9905a00640 1 -- 172.21.15.43:0/2050641749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e0003520 con 0x7f98e0004d00 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.306+0000 7f9905a00640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98e0004d00 0x7f98e0005180 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f98f800a7f0 tx=0x7f98e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba5d7e4fb6c8f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.306+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f80072d0 con 0x7f98e0004d00 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.306+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f802f070 con 0x7f98e0004d00 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.306+0000 7f98ff400640 1 -- 172.21.15.43:0/2050641749 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f80076b0 con 0x7f98e0004d00 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 -- 172.21.15.43:0/2050641749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98e0004d00 msgr2=0x7f98e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 --2- 172.21.15.43:0/2050641749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98e0004d00 0x7f98e0005180 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f98f800a7f0 tx=0x7f98e00057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9905a00640 1 -- 172.21.15.43:0/2050641749 reap_dead start 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 -- 172.21.15.43:0/2050641749 shutdown_connections 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 -- 172.21.15.43:0/2050641749 >> 172.21.15.43:0/2050641749 conn(0x7f990006a860 msgr2=0x7f990006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 -- 172.21.15.43:0/2050641749 shutdown_connections 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.307+0000 7f9906400640 1 -- 172.21.15.43:0/2050641749 wait complete. 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.309+0000 7f9906400640 1 Processor -- start 2024-01-16T15:15:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.309+0000 7f9906400640 1 -- start start 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0004d00 0x7f990014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000b3020 0x7f99000b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99000cc060 con 0x7f99001606d0 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99000b39e0 con 0x7f98e0004d00 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f9906400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990014bcf0 con 0x7f99000b3020 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f98ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f98ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55430/0 (socket says 172.21.15.43:55430) 2024-01-16T15:15:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.310+0000 7f98ffe00640 1 -- 172.21.15.43:0/4162869604 learned_addr learned my addr 172.21.15.43:0/4162869604 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f9905a00640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000b3020 0x7f99000b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 -- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000b3020 msgr2=0x7f99000b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000b3020 0x7f99000b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 -- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0004d00 msgr2=0x7f990014b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0004d00 0x7f990014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 -- 172.21.15.43:0/4162869604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990014d080 con 0x7f99001606d0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98ffe00640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f98e800e550 tx=0x7f98e800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80052e0 con 0x7f99001606d0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8005440 con 0x7f99001606d0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99000cb7c0 con 0x7f99001606d0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.311+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8003810 con 0x7f99001606d0 2024-01-16T15:15:04.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.312+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99000cbd80 con 0x7f99001606d0 2024-01-16T15:15:04.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.313+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990004e840 con 0x7f99001606d0 2024-01-16T15:15:04.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.314+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f98e8003970 con 0x7f99001606d0 2024-01-16T15:15:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.314+0000 7f98fcc00640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 0x7f98d806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.315+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f98e80887b0 con 0x7f99001606d0 2024-01-16T15:15:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.315+0000 7f9905000640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 0x7f98d806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.315+0000 7f9905000640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 0x7f98d806e5c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009910 tx=0x7f98f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.319+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98e8054cb0 con 0x7f99001606d0 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.555+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9900155ad0 con 0x7f99001606d0 2024-01-16T15:15:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.555+0000 7f98fcc00640 1 -- 172.21.15.43:0/4162869604 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f98e80582d0 con 0x7f99001606d0 2024-01-16T15:15:04.539 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:04.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 msgr2=0x7f98d806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 0x7f98d806e5c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009910 tx=0x7f98f0008040 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 msgr2=0x7f99000caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f98e800e550 tx=0x7f98e800ea10 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 shutdown_connections 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f98d806c100 0x7f98d806e5c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99000b3020 0x7f99000b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99001606d0 0x7f99000caee0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 --2- 172.21.15.43:0/4162869604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e0004d00 0x7f990014b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 >> 172.21.15.43:0/4162869604 conn(0x7f990006a860 msgr2=0x7f99000cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 shutdown_connections 2024-01-16T15:15:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:04.558+0000 7f9906400640 1 -- 172.21.15.43:0/4162869604 wait complete. 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:04.561 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:04.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.562 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:15:04.612 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:04.612 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:04.612 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:04.612 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.612 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41534sec preferred_lft 41534sec 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:15:04.613 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:04.614 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:15:04.615 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:04.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.616 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:04.700 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:15:04.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.705 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:15:04.757 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:15:04.758 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:04.758 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:15:04.758 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:15:04.758 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:04.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:15:04.759 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:15:04.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:04.951 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:15:05.003 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:15:05.003 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:15:05.004 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:05.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:05.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:15:05.106 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:15:05.106 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:05.106 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:05.162 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:05.162 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:05.162 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:05.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:15:05.313 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:15:05.363 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:05.364 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:05.365 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:05.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:05.366 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:15:05.417 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:05.417 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:15:05.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:15:05.498 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:15:05.541 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:05.542 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:05.543 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:05.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:05.544 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:15:05.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:15:05.631+0000 7f7b33371480 -1 init, newargv = 0x557c7febac00 newargc=15 2024-01-16T15:15:05.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[72177]: starting ceph client 2024-01-16T15:15:05.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[72177]: starting fuse 2024-01-16T15:15:05.635 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:15:05.635 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:15:05.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:05.635 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:15:05.635 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:15:05.635 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:15:05.635 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:15:05.636 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:15:05.637 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:15:05.637 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:05.637 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:15:05.637 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:15:05.637 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:15:05.836 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.72177.asok 2024-01-16T15:15:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:05.853 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72177.asok 2024-01-16T15:15:05.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:05.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72177.asok status 2024-01-16T15:15:06.098 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:15:06.098 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:15:06.098 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "72177", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 8727, 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 8727 2024-01-16T15:15:06.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3948536089 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3948536089 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.8727 192.168.0.1:0/3948536089", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/3948536089", 2024-01-16T15:15:06.100 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 66, 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 101, 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:15:06.101 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:15:06.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.115 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:15:06.115 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:06.172 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:15:06.172 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:06.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:06.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.518+0000 7f10f1400640 1 Processor -- start 2024-01-16T15:15:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.518+0000 7f10f1400640 1 -- start start 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.519+0000 7f10f1400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10ec0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.519+0000 7f10f1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10ec0ae7e0 0x7f10ec0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.519+0000 7f10f1400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec13da30 con 0x7f10ec0abff0 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.519+0000 7f10f1400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec0ac840 con 0x7f10ec0ae7e0 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10f1400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0ac560 con 0x7f10ec15a100 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10e9e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f10ec0abff0 0x7f10ec06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36934/0 (socket says 172.21.15.43:36934) 2024-01-16T15:15:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10e9e00640 1 -- 172.21.15.43:0/2904607509 learned_addr learned my addr 172.21.15.43:0/2904607509 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10ea800640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10ec0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10eb200640 1 --2- 172.21.15.43:0/2904607509 >> v2:172.21.15.69:3300/0 conn(0x7f10ec0ae7e0 0x7f10ec0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355015371 0 0) 0x7f10ec13da30 con 0x7f10ec0abff0 2024-01-16T15:15:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.520+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc005730 con 0x7f10ec0abff0 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945162947 0 0) 0x7f10cc005730 con 0x7f10ec0abff0 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 msgr2=0x7f10ec0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10ec0b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 >> v2:172.21.15.69:3300/0 conn(0x7f10ec0ae7e0 msgr2=0x7f10ec0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 --2- 172.21.15.43:0/2904607509 >> v2:172.21.15.69:3300/0 conn(0x7f10ec0ae7e0 0x7f10ec0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec070650 con 0x7f10ec0abff0 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f10dc001670 con 0x7f10ec0abff0 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 >> v1:172.21.15.43:6789/0 conn(0x7f10ec0abff0 legacy=0x7f10ec06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10cc005b80 0x7f10cc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10cc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10cc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0ac840 con 0x7f10cc006f10 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0ac560 con 0x7f10cc005b80 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc00db90 con 0x7f10ec15a100 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.521+0000 7f10eb200640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10cc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10e9e00640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10cc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10eb200640 1 -- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 msgr2=0x7f10cc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10ea800640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10cc005b80 0x7f10cc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10eb200640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15a100 0x7f10cc0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10eb200640 1 -- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10cc005b80 msgr2=0x7f10cc005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10eb200640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10cc005b80 0x7f10cc005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.522+0000 7f10eb200640 1 -- 172.21.15.43:0/2904607509 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc005730 con 0x7f10cc006f10 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.523+0000 7f10eb200640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10cc007390 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009fc0 tx=0x7f10cc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691d54e4e4ed6d43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.523+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e402cc60 con 0x7f10cc006f10 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.523+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e400f040 con 0x7f10cc006f10 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.523+0000 7f10e9400640 1 -- 172.21.15.43:0/2904607509 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e4034440 con 0x7f10cc006f10 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 -- 172.21.15.43:0/2904607509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 msgr2=0x7f10cc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 --2- 172.21.15.43:0/2904607509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10cc007390 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009fc0 tx=0x7f10cc007a00 comp rx=0 tx=0).stop 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10eb200640 1 -- 172.21.15.43:0/2904607509 reap_dead start 2024-01-16T15:15:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 -- 172.21.15.43:0/2904607509 shutdown_connections 2024-01-16T15:15:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 -- 172.21.15.43:0/2904607509 >> 172.21.15.43:0/2904607509 conn(0x7f10ec0b0d80 msgr2=0x7f10ec0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 -- 172.21.15.43:0/2904607509 shutdown_connections 2024-01-16T15:15:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.524+0000 7f10f1400640 1 -- 172.21.15.43:0/2904607509 wait complete. 2024-01-16T15:15:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.526+0000 7f10f1400640 1 Processor -- start 2024-01-16T15:15:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.526+0000 7f10f1400640 1 -- start start 2024-01-16T15:15:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15bc10 0x7f10ec14b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec157ed0 0x7f10ec158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec1588c0 con 0x7f10cc006f10 2024-01-16T15:15:06.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec15c070 con 0x7f10ec157ed0 2024-01-16T15:15:06.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10f1400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0cba60 con 0x7f10ec15bc10 2024-01-16T15:15:06.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10ea800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10ea800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55456/0 (socket says 172.21.15.43:55456) 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.527+0000 7f10ea800640 1 -- 172.21.15.43:0/1429443685 learned_addr learned my addr 172.21.15.43:0/1429443685 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10e9e00640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15bc10 0x7f10ec14b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 -- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15bc10 msgr2=0x7f10ec14b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10eb200640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec157ed0 0x7f10ec158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15bc10 0x7f10ec14b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 -- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec157ed0 msgr2=0x7f10ec158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec157ed0 0x7f10ec158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 -- 172.21.15.43:0/1429443685 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0cbd40 con 0x7f10cc006f10 2024-01-16T15:15:06.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.528+0000 7f10ea800640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f10d4010860 tx=0x7f10d4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.529+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4009610 con 0x7f10cc006f10 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.529+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d400eb00 con 0x7f10cc006f10 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.529+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400ee90 con 0x7f10cc006f10 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.529+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec0cc020 con 0x7f10cc006f10 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.529+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec0cc5e0 con 0x7f10cc006f10 2024-01-16T15:15:06.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.530+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec074770 con 0x7f10cc006f10 2024-01-16T15:15:06.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.531+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f10d401c400 con 0x7f10cc006f10 2024-01-16T15:15:06.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.532+0000 7f10e2a00640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 0x7f10c406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:06.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.532+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f10d4088ee0 con 0x7f10cc006f10 2024-01-16T15:15:06.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.532+0000 7f10e9e00640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 0x7f10c406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:06.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.533+0000 7f10e9e00640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 0x7f10c406e590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f10dc001eb0 tx=0x7f10dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:06.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.537+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10d40554e0 con 0x7f10cc006f10 2024-01-16T15:15:06.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.772+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f10ec14b9b0 con 0x7f10cc006f10 2024-01-16T15:15:06.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.773+0000 7f10e2a00640 1 -- 172.21.15.43:0/1429443685 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f10d4058b00 con 0x7f10cc006f10 2024-01-16T15:15:06.757 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:06.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 msgr2=0x7f10c406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 0x7f10c406e590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f10dc001eb0 tx=0x7f10dc009040 comp rx=0 tx=0).stop 2024-01-16T15:15:06.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 msgr2=0x7f10ec15a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f10d4010860 tx=0x7f10d4010d20 comp rx=0 tx=0).stop 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 shutdown_connections 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10c406c0d0 0x7f10c406e590 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec157ed0 0x7f10ec158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10ec15bc10 0x7f10ec14b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 --2- 172.21.15.43:0/1429443685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10cc006f10 0x7f10ec15a660 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 >> 172.21.15.43:0/1429443685 conn(0x7f10ec0b0d80 msgr2=0x7f10ec06d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 shutdown_connections 2024-01-16T15:15:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:06.776+0000 7f10f1400640 1 -- 172.21.15.43:0/1429443685 wait complete. 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:06.778 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:06.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.778 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:06.837 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41693sec preferred_lft 41693sec 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:06.838 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:15:06.839 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:06.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.840 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:06.929 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:15:06.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.933 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:15:06.985 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:15:06.985 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:06.985 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:06.985 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:06.985 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:06.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:06.986 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:15:06.986 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:15:06.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:06.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:06.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:06.987 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:15:07.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:07.168 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:15:07.220 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:15:07.220 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:15:07.221 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:07.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:07.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:15:07.313 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:15:07.313 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:07.313 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:07.365 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:07.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:07.365 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:07.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:15:07.507 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:07.560 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:07.561 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:07.562 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:07.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:07.563 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:15:07.611 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:07.611 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:15:07.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:15:07.689 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:07.728 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:07.730 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:07.731 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:15:07.731 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:07.731 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:07.731 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:07.731 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:07.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:07.732 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:15:07.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:15:07.810+0000 7fc87d447480 -1 init, newargv = 0x56249c2ffb60 newargc=15 2024-01-16T15:15:07.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[46566]: starting ceph client 2024-01-16T15:15:07.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[46566]: starting fuse 2024-01-16T15:15:07.816 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:15:07.817 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:15:07.817 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:15:07.818 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:15:08.017 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.46566.asok 2024-01-16T15:15:08.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:15:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:08.035 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46566.asok 2024-01-16T15:15:08.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:08.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46566.asok status 2024-01-16T15:15:08.263 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:15:08.263 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:15:08.263 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "46566", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7069, 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7069 2024-01-16T15:15:08.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4191745735 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4191745735 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7069 192.168.0.1:0/4191745735", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/4191745735", 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 66, 2024-01-16T15:15:08.265 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 101, 2024-01-16T15:15:08.266 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:08.266 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:15:08.266 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:15:08.266 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:15:08.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:08.278 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:15:08.278 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:08.331 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:15:08.331 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:08.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:08.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:08.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:15:08.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.675+0000 7f18a7200640 1 Processor -- start 2024-01-16T15:15:08.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.675+0000 7f18a7200640 1 -- start start 2024-01-16T15:15:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.677+0000 7f18a7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.677+0000 7f18a7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 0x7f18a8070050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.677+0000 7f18a7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a8070d40 con 0x7f18a81439b0 2024-01-16T15:15:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.677+0000 7f18a7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a8071020 con 0x7f18a80b4140 2024-01-16T15:15:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.677+0000 7f18a7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a8071300 con 0x7f18a81435d0 2024-01-16T15:15:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.678+0000 7f18a5e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.678+0000 7f18a6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 0x7f18a8070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.678+0000 7f18a5e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:47512/0 (socket says 172.21.15.43:47512) 2024-01-16T15:15:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.678+0000 7f18a5e00640 1 -- 172.21.15.43:0/3510126154 learned_addr learned my addr 172.21.15.43:0/3510126154 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.679+0000 7f18a5e00640 1 -- 172.21.15.43:0/3510126154 >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 msgr2=0x7f18a8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.679+0000 7f18a5e00640 1 --2- 172.21.15.43:0/3510126154 >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 0x7f18a8070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.679+0000 7f18a5e00640 1 -- 172.21.15.43:0/3510126154 >> v1:172.21.15.69:6789/0 conn(0x7f18a81439b0 legacy=0x7f18a80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.679+0000 7f18a5e00640 1 -- 172.21.15.43:0/3510126154 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a80715e0 con 0x7f18a81435d0 2024-01-16T15:15:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.679+0000 7f18a6800640 1 --2- 172.21.15.43:0/3510126154 >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 0x7f18a8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.680+0000 7f18a5e00640 1 --2- 172.21.15.43:0/3510126154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f18980098a0 tx=0x7f189802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ae4db8382a533f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a5400640 1 -- 172.21.15.43:0/3510126154 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898030840 con 0x7f18a81435d0 2024-01-16T15:15:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a5400640 1 -- 172.21.15.43:0/3510126154 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18980309a0 con 0x7f18a81435d0 2024-01-16T15:15:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a5400640 1 -- 172.21.15.43:0/3510126154 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898030d30 con 0x7f18a81435d0 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 -- 172.21.15.43:0/3510126154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 msgr2=0x7f18a8071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 --2- 172.21.15.43:0/3510126154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f18980098a0 tx=0x7f189802b980 comp rx=0 tx=0).stop 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 -- 172.21.15.43:0/3510126154 shutdown_connections 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 --2- 172.21.15.43:0/3510126154 >> v2:172.21.15.69:3300/0 conn(0x7f18a80b4140 0x7f18a8070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 --2- 172.21.15.43:0/3510126154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a81435d0 0x7f18a8071de0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.681+0000 7f18a7200640 1 -- 172.21.15.43:0/3510126154 >> 172.21.15.43:0/3510126154 conn(0x7f18a80cec10 msgr2=0x7f18a80cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.682+0000 7f18a7200640 1 -- 172.21.15.43:0/3510126154 shutdown_connections 2024-01-16T15:15:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.682+0000 7f18a7200640 1 -- 172.21.15.43:0/3510126154 wait complete. 2024-01-16T15:15:08.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.684+0000 7f18a7200640 1 Processor -- start 2024-01-16T15:15:08.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.684+0000 7f18a7200640 1 -- start start 2024-01-16T15:15:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a80b4140 0x7f18a81f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a81435d0 0x7f18a80b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80c4d20 con 0x7f18a81439b0 2024-01-16T15:15:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81f3470 con 0x7f18a81435d0 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.685+0000 7f18a7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81f3750 con 0x7f18a80b4140 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55458/0 (socket says 172.21.15.43:55458) 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 -- 172.21.15.43:0/1011720259 learned_addr learned my addr 172.21.15.43:0/1011720259 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a5e00640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a80b4140 0x7f18a81f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f189fe00640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a81435d0 0x7f18a80b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 -- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a80b4140 msgr2=0x7f18a81f2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a80b4140 0x7f18a81f2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 -- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a81435d0 msgr2=0x7f18a80b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a81435d0 0x7f18a80b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.686+0000 7f18a6800640 1 -- 172.21.15.43:0/1011720259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0009a70 con 0x7f18a81439b0 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f18a6800640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f18a000efb0 tx=0x7f18a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0009d50 con 0x7f18a81439b0 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000f040 con 0x7f18a81439b0 2024-01-16T15:15:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a00146a0 con 0x7f18a81439b0 2024-01-16T15:15:08.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18980094d0 con 0x7f18a81439b0 2024-01-16T15:15:08.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.687+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a81f3a30 con 0x7f18a81439b0 2024-01-16T15:15:08.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.688+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a804e840 con 0x7f18a81439b0 2024-01-16T15:15:08.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.690+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f18a0014800 con 0x7f18a81439b0 2024-01-16T15:15:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.691+0000 7f189e000640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 0x7f188006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.691+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f18a0089870 con 0x7f18a81439b0 2024-01-16T15:15:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.691+0000 7f18a5e00640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 0x7f188006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.692+0000 7f18a5e00640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 0x7f188006e590 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1898036000 tx=0x7f1898008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.695+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18a0055e70 con 0x7f18a81439b0 2024-01-16T15:15:08.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:08.932+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f18a80cb820 con 0x7f18a81439b0 2024-01-16T15:15:09.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.363+0000 7f189e000640 1 -- 172.21.15.43:0/1011720259 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v67) v1 ==== 136+0+0 (secure 0 0 0) 0x7f18a0059490 con 0x7f18a81439b0 2024-01-16T15:15:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 msgr2=0x7f188006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 0x7f188006e590 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1898036000 tx=0x7f1898008030 comp rx=0 tx=0).stop 2024-01-16T15:15:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 msgr2=0x7f18a80c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f18a000efb0 tx=0x7f18a000c490 comp rx=0 tx=0).stop 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 shutdown_connections 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f188006c0d0 0x7f188006e590 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a81435d0 0x7f18a80b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a81439b0 0x7f18a80c54b0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 --2- 172.21.15.43:0/1011720259 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18a80b4140 0x7f18a81f2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 >> 172.21.15.43:0/1011720259 conn(0x7f18a80cec10 msgr2=0x7f18a80ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 shutdown_connections 2024-01-16T15:15:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.367+0000 7f18a7200640 1 -- 172.21.15.43:0/1011720259 wait complete. 2024-01-16T15:15:09.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:15:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.618+0000 7fc2a8600640 1 Processor -- start 2024-01-16T15:15:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.618+0000 7fc2a8600640 1 -- start start 2024-01-16T15:15:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a8600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2a015d6d0 0x7fc2a00d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a8600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 0x7fc2a00b6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a8600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a00d1300 con 0x7fc2a00d2d80 2024-01-16T15:15:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a8600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00d21d0 con 0x7fc2a015d6d0 2024-01-16T15:15:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a8600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00b2830 con 0x7fc2a00b5390 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a5200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc2a00d2d80 0x7fc2a00b4c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36936/0 (socket says 172.21.15.43:36936) 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a5200640 1 -- 172.21.15.43:0/3266023882 learned_addr learned my addr 172.21.15.43:0/3266023882 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.619+0000 7fc2a5c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc2a015d6d0 0x7fc2a00d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a6600640 1 --2- 172.21.15.43:0/3266023882 >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 0x7fc2a00b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc29fe00640 1 -- 172.21.15.43:0/3266023882 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822199326 0 0) 0x7fc2a00d1300 con 0x7fc2a00d2d80 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a5c00640 1 -- 172.21.15.43:0/3266023882 >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 msgr2=0x7fc2a00b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a5c00640 1 --2- 172.21.15.43:0/3266023882 >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 0x7fc2a00b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a5c00640 1 -- 172.21.15.43:0/3266023882 >> v1:172.21.15.43:6789/0 conn(0x7fc2a00d2d80 legacy=0x7fc2a00b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a5c00640 1 -- 172.21.15.43:0/3266023882 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00d1550 con 0x7fc2a015d6d0 2024-01-16T15:15:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.620+0000 7fc2a6600640 1 --2- 172.21.15.43:0/3266023882 >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 0x7fc2a00b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:09.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.621+0000 7fc2a5c00640 1 --2- 172.21.15.43:0/3266023882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a015d6d0 0x7fc2a00d2840 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00ca260 tx=0x7fc28802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2fc9dc581c576f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc29fe00640 1 -- 172.21.15.43:0/3266023882 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288030610 con 0x7fc2a015d6d0 2024-01-16T15:15:09.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc29fe00640 1 -- 172.21.15.43:0/3266023882 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288030770 con 0x7fc2a015d6d0 2024-01-16T15:15:09.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc29fe00640 1 -- 172.21.15.43:0/3266023882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288030b00 con 0x7fc2a015d6d0 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 -- 172.21.15.43:0/3266023882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a015d6d0 msgr2=0x7fc2a00d2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 --2- 172.21.15.43:0/3266023882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a015d6d0 0x7fc2a00d2840 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00ca260 tx=0x7fc28802b960 comp rx=0 tx=0).stop 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 -- 172.21.15.43:0/3266023882 shutdown_connections 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 --2- 172.21.15.43:0/3266023882 >> v2:172.21.15.69:3300/0 conn(0x7fc2a00b5390 0x7fc2a00b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 --2- 172.21.15.43:0/3266023882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a015d6d0 0x7fc2a00d2840 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 -- 172.21.15.43:0/3266023882 >> 172.21.15.43:0/3266023882 conn(0x7fc2a00721a0 msgr2=0x7fc2a00cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 -- 172.21.15.43:0/3266023882 shutdown_connections 2024-01-16T15:15:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.622+0000 7fc2a8600640 1 -- 172.21.15.43:0/3266023882 wait complete. 2024-01-16T15:15:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.624+0000 7fc2a8600640 1 Processor -- start 2024-01-16T15:15:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.624+0000 7fc2a8600640 1 -- start start 2024-01-16T15:15:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00d2d80 0x7fc2a01530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a015d6d0 0x7fc2a01f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0159880 con 0x7fc2a00b5390 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0153e30 con 0x7fc2a015d6d0 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.625+0000 7fc2a8600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0154110 con 0x7fc2a00d2d80 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55478/0 (socket says 172.21.15.43:55478) 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 -- 172.21.15.43:0/1359060573 learned_addr learned my addr 172.21.15.43:0/1359060573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5200640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00d2d80 0x7fc2a01530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a6600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a015d6d0 0x7fc2a01f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 -- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00d2d80 msgr2=0x7fc2a01530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00d2d80 0x7fc2a01530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 -- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a015d6d0 msgr2=0x7fc2a01f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a015d6d0 0x7fc2a01f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.626+0000 7fc2a5c00640 1 -- 172.21.15.43:0/1359060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc288009400 con 0x7fc2a00b5390 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc2a5c00640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc288009890 tx=0x7fc2880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28803e070 con 0x7fc2a00b5390 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288039800 con 0x7fc2a00b5390 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288039b90 con 0x7fc2a00b5390 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a015e880 con 0x7fc2a00b5390 2024-01-16T15:15:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.627+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a015ee40 con 0x7fc2a00b5390 2024-01-16T15:15:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.628+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a015f8a0 con 0x7fc2a00b5390 2024-01-16T15:15:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.630+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fc288042070 con 0x7fc2a00b5390 2024-01-16T15:15:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.630+0000 7fc29d600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 0x7fc27806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.631+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc2880aa8f0 con 0x7fc2a00b5390 2024-01-16T15:15:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.631+0000 7fc2a5200640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 0x7fc27806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.631+0000 7fc2a5200640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 0x7fc27806e590 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00cd3d0 tx=0x7fc290007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.635+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc288076ef0 con 0x7fc2a00b5390 2024-01-16T15:15:09.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.852+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2a00d1df0 con 0x7fc27806c0d0 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.854+0000 7fc29d600640 1 -- 172.21.15.43:0/1359060573 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc2900079d0 con 0x7fc27806c0d0 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:15:09.838 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 msgr2=0x7fc27806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 0x7fc27806e590 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00cd3d0 tx=0x7fc290007ed0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 msgr2=0x7fc2a015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc288009890 tx=0x7fc2880046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 shutdown_connections 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc27806c0d0 0x7fc27806e590 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a015d6d0 0x7fc2a01f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00d2d80 0x7fc2a01530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 --2- 172.21.15.43:0/1359060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2a00b5390 0x7fc2a015f360 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 >> 172.21.15.43:0/1359060573 conn(0x7fc2a00721a0 msgr2=0x7fc2a015c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 shutdown_connections 2024-01-16T15:15:09.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:09.857+0000 7fc2a8600640 1 -- 172.21.15.43:0/1359060573 wait complete. 2024-01-16T15:15:09.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:10.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.106+0000 7f589de00640 1 Processor -- start 2024-01-16T15:15:10.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.107+0000 7f589de00640 1 -- start start 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.108+0000 7f589de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f589815f100 0x7f5898161c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.108+0000 7f589de00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5898070070 con 0x7f5898165e20 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.108+0000 7f589de00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5898070f40 con 0x7f589806d3d0 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.108+0000 7f589de00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58981624e0 con 0x7f589815f100 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f589ca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f589815f100 0x7f5898161c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f588fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5898165e20 0x7f5898070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36942/0 (socket says 172.21.15.43:36942) 2024-01-16T15:15:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f588fe00640 1 -- 172.21.15.43:0/390909831 learned_addr learned my addr 172.21.15.43:0/390909831 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:10.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f589ca00640 1 -- 172.21.15.43:0/390909831 >> v1:172.21.15.69:6789/0 conn(0x7f589806d3d0 legacy=0x7f589806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f5897e00640 1 -- 172.21.15.43:0/390909831 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867637585 0 0) 0x7f5898070070 con 0x7f5898165e20 2024-01-16T15:15:10.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f589ca00640 1 -- 172.21.15.43:0/390909831 >> v1:172.21.15.43:6789/0 conn(0x7f5898165e20 legacy=0x7f5898070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.109+0000 7f589ca00640 1 -- 172.21.15.43:0/390909831 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58981627c0 con 0x7f589815f100 2024-01-16T15:15:10.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f589ca00640 1 --2- 172.21.15.43:0/390909831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f589815f100 0x7f5898161c20 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f58880097a0 tx=0x7f588802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60666c1046ebe9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f5897e00640 1 -- 172.21.15.43:0/390909831 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888030610 con 0x7f589815f100 2024-01-16T15:15:10.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f5897e00640 1 -- 172.21.15.43:0/390909831 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888030770 con 0x7f589815f100 2024-01-16T15:15:10.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f5897e00640 1 -- 172.21.15.43:0/390909831 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888030b00 con 0x7f589815f100 2024-01-16T15:15:10.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f589de00640 1 -- 172.21.15.43:0/390909831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f589815f100 msgr2=0x7f5898161c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.111+0000 7f589de00640 1 --2- 172.21.15.43:0/390909831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f589815f100 0x7f5898161c20 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f58880097a0 tx=0x7f588802b960 comp rx=0 tx=0).stop 2024-01-16T15:15:10.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.112+0000 7f589de00640 1 -- 172.21.15.43:0/390909831 shutdown_connections 2024-01-16T15:15:10.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.112+0000 7f589de00640 1 --2- 172.21.15.43:0/390909831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f589815f100 0x7f5898161c20 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.112+0000 7f589de00640 1 -- 172.21.15.43:0/390909831 >> 172.21.15.43:0/390909831 conn(0x7f58980b0aa0 msgr2=0x7f58980b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.112+0000 7f589de00640 1 -- 172.21.15.43:0/390909831 shutdown_connections 2024-01-16T15:15:10.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.112+0000 7f589de00640 1 -- 172.21.15.43:0/390909831 wait complete. 2024-01-16T15:15:10.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.113+0000 7f589de00640 1 Processor -- start 2024-01-16T15:15:10.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.114+0000 7f589de00640 1 -- start start 2024-01-16T15:15:10.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f589806d3d0 0x7f5898162e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f589815f100 0x7f5898156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898157e90 con 0x7f5898165e20 2024-01-16T15:15:10.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898157040 con 0x7f589806d3d0 2024-01-16T15:15:10.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5898163410 con 0x7f589815f100 2024-01-16T15:15:10.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55488/0 (socket says 172.21.15.43:55488) 2024-01-16T15:15:10.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 -- 172.21.15.43:0/51398693 learned_addr learned my addr 172.21.15.43:0/51398693 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f588fe00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f589815f100 0x7f5898156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589ca00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f589806d3d0 0x7f5898162e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 -- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f589815f100 msgr2=0x7f5898156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f589815f100 0x7f5898156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 -- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f589806d3d0 msgr2=0x7f5898162e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f589806d3d0 0x7f5898162e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.115+0000 7f589d400640 1 -- 172.21.15.43:0/51398693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888009400 con 0x7f5898165e20 2024-01-16T15:15:10.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f589d400640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f589000c6e0 tx=0x7f589000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890003b70 con 0x7f5898165e20 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5890003cf0 con 0x7f5898165e20 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890005310 con 0x7f5898165e20 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58981636f0 con 0x7f5898165e20 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.116+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5898158350 con 0x7f5898165e20 2024-01-16T15:15:10.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.117+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5898163930 con 0x7f5898165e20 2024-01-16T15:15:10.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.119+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f5890005470 con 0x7f5898165e20 2024-01-16T15:15:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.119+0000 7f5895600640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 0x7f587006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.120+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58900886f0 con 0x7f5898165e20 2024-01-16T15:15:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.120+0000 7f589ca00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 0x7f587006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.120+0000 7f589ca00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 0x7f587006e590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5888036000 tx=0x7f588803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.124+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5898163930 con 0x7f5898165e20 2024-01-16T15:15:10.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.323+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5898163930 con 0x7f5898165e20 2024-01-16T15:15:10.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.324+0000 7f5895600640 1 -- 172.21.15.43:0/51398693 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5898163930 con 0x7f5898165e20 2024-01-16T15:15:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 msgr2=0x7f587006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 0x7f587006e590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5888036000 tx=0x7f588803b040 comp rx=0 tx=0).stop 2024-01-16T15:15:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 msgr2=0x7f5898158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f589000c6e0 tx=0x7f589000cba0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 shutdown_connections 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f587006c0d0 0x7f587006e590 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5898165e20 0x7f5898158870 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f589815f100 0x7f5898156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 --2- 172.21.15.43:0/51398693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f589806d3d0 0x7f5898162e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.327+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 >> 172.21.15.43:0/51398693 conn(0x7f58980b0aa0 msgr2=0x7f589815e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.328+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 shutdown_connections 2024-01-16T15:15:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.328+0000 7f589de00640 1 -- 172.21.15.43:0/51398693 wait complete. 2024-01-16T15:15:10.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:15:10.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.560+0000 7f2aba600640 1 Processor -- start 2024-01-16T15:15:10.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.560+0000 7f2aba600640 1 -- start start 2024-01-16T15:15:10.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab415d550 0x7f2ab40d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ab40cfab0 0x7f2ab40cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ab40ceaa0 0x7f2ab40cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab40d2380 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab40d0450 con 0x7f2ab40cfab0 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.562+0000 7f2aba600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40d19b0 con 0x7f2ab415d550 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ab40ceaa0 0x7f2ab40cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2ab40ceaa0 0x7f2ab40cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55500/0 (socket says 172.21.15.43:55500) 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 -- 172.21.15.43:0/2530658664 learned_addr learned my addr 172.21.15.43:0/2530658664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2aabe00640 1 --2- 172.21.15.43:0/2530658664 >> v2:172.21.15.69:3300/0 conn(0x7f2ab40cfab0 0x7f2ab40cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9200640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab415d550 0x7f2ab40d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 -- 172.21.15.43:0/2530658664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab415d550 msgr2=0x7f2ab40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab415d550 0x7f2ab40d1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 -- 172.21.15.43:0/2530658664 >> v2:172.21.15.69:3300/0 conn(0x7f2ab40cfab0 msgr2=0x7f2ab40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/2530658664 >> v2:172.21.15.69:3300/0 conn(0x7f2ab40cfab0 0x7f2ab40cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.563+0000 7f2ab9c00640 1 -- 172.21.15.43:0/2530658664 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab40d0da0 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.564+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab40cef30 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4068140 tx=0x7f2aac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820f484fea6de66e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2ab3e00640 1 -- 172.21.15.43:0/2530658664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac02c9e0 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2ab3e00640 1 -- 172.21.15.43:0/2530658664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac02cb40 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2ab3e00640 1 -- 172.21.15.43:0/2530658664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0343f0 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 -- 172.21.15.43:0/2530658664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab40ceaa0 msgr2=0x7f2ab40cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab40cef30 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4068140 tx=0x7f2aac02be40 comp rx=0 tx=0).stop 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 -- 172.21.15.43:0/2530658664 shutdown_connections 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab40cef30 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 --2- 172.21.15.43:0/2530658664 >> v2:172.21.15.69:3300/0 conn(0x7f2ab40cfab0 0x7f2ab40cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 --2- 172.21.15.43:0/2530658664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab415d550 0x7f2ab40d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.565+0000 7f2aba600640 1 -- 172.21.15.43:0/2530658664 >> 172.21.15.43:0/2530658664 conn(0x7f2ab40d30a0 msgr2=0x7f2ab40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.566+0000 7f2aba600640 1 -- 172.21.15.43:0/2530658664 shutdown_connections 2024-01-16T15:15:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.566+0000 7f2aba600640 1 -- 172.21.15.43:0/2530658664 wait complete. 2024-01-16T15:15:10.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.568+0000 7f2aba600640 1 Processor -- start 2024-01-16T15:15:10.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.568+0000 7f2aba600640 1 -- start start 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.569+0000 7f2aba600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab4151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aba600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab40cfab0 0x7f2ab40b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aba600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aba600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40b8ce0 con 0x7f2ab415d550 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aba600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41f3e10 con 0x7f2ab40ceaa0 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aba600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41f3c20 con 0x7f2ab40cfab0 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55516/0 (socket says 172.21.15.43:55516) 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 -- 172.21.15.43:0/1859106221 learned_addr learned my addr 172.21.15.43:0/1859106221 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9200640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab4151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2aabe00640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab40cfab0 0x7f2ab40b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 -- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab40cfab0 msgr2=0x7f2ab40b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab40cfab0 0x7f2ab40b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 -- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ab40ceaa0 msgr2=0x7f2ab4151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.570+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab4151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2ab9c00640 1 -- 172.21.15.43:0/1859106221 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac009b30 con 0x7f2ab415d550 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2ab9c00640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009f40 tx=0x7f2aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac009510 con 0x7f2ab415d550 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac03e450 con 0x7f2ab415d550 2024-01-16T15:15:10.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab41f3940 con 0x7f2ab415d550 2024-01-16T15:15:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac03e7e0 con 0x7f2ab415d550 2024-01-16T15:15:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.571+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab40c8160 con 0x7f2ab415d550 2024-01-16T15:15:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.573+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4067960 con 0x7f2ab415d550 2024-01-16T15:15:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.574+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f2aac03e940 con 0x7f2ab415d550 2024-01-16T15:15:10.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.575+0000 7f2ab1600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 0x7f2a8c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.575+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2aac0abb30 con 0x7f2ab415d550 2024-01-16T15:15:10.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.575+0000 7f2ab9200640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 0x7f2a8c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.576+0000 7f2ab9200640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 0x7f2a8c06e590 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006fd0 tx=0x7f2aa4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.580+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aac078250 con 0x7f2ab415d550 2024-01-16T15:15:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.777+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f2ab404e840 con 0x7f2ab415d550 2024-01-16T15:15:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.784+0000 7f2ab1600640 1 -- 172.21.15.43:0/1859106221 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v37)=0 v37) v1 ==== 131+0+0 (secure 0 0 0) 0x7f2aac07b870 con 0x7f2ab415d550 2024-01-16T15:15:10.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 msgr2=0x7f2a8c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 0x7f2a8c06e590 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006fd0 tx=0x7f2aa4008040 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 msgr2=0x7f2ab40b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009f40 tx=0x7f2aac004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 shutdown_connections 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2a8c06c0d0 0x7f2a8c06e590 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab415d550 0x7f2ab40b0d90 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ab40cfab0 0x7f2ab40b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 --2- 172.21.15.43:0/1859106221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ab40ceaa0 0x7f2ab4151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.788+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 >> 172.21.15.43:0/1859106221 conn(0x7f2ab40d30a0 msgr2=0x7f2ab40afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.789+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 shutdown_connections 2024-01-16T15:15:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:10.789+0000 7f2aba600640 1 -- 172.21.15.43:0/1859106221 wait complete. 2024-01-16T15:15:10.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:15:11.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.030+0000 7f06e6e00640 1 Processor -- start 2024-01-16T15:15:11.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.030+0000 7f06e6e00640 1 -- start start 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e6e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06e00cfa30 0x7f06e00cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 0x7f06e00ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e00b2980 con 0x7f06e00d0e00 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e6e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e00d0350 con 0x7f06e00cfa30 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00cf220 con 0x7f06e00d1fd0 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e5a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06e00cfa30 0x7f06e00cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e5a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06e00cfa30 0x7f06e00cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55522/0 (socket says 172.21.15.43:55522) 2024-01-16T15:15:11.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.032+0000 7f06e5a00640 1 -- 172.21.15.43:0/3821896867 learned_addr learned my addr 172.21.15.43:0/3821896867 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5000640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 0x7f06e00ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5a00640 1 -- 172.21.15.43:0/3821896867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 msgr2=0x7f06e00ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5a00640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 0x7f06e00ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5a00640 1 -- 172.21.15.43:0/3821896867 >> v1:172.21.15.69:6789/0 conn(0x7f06e00d0e00 legacy=0x7f06e00d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5a00640 1 -- 172.21.15.43:0/3821896867 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e00cf500 con 0x7f06e00cfa30 2024-01-16T15:15:11.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.033+0000 7f06e5000640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 0x7f06e00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.034+0000 7f06e5a00640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00cfa30 0x7f06e00cfe10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009870 tx=0x7f06d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581f7b9a955b2b26 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06dfe00640 1 -- 172.21.15.43:0/3821896867 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d0030840 con 0x7f06e00cfa30 2024-01-16T15:15:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06dfe00640 1 -- 172.21.15.43:0/3821896867 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d00309a0 con 0x7f06e00cfa30 2024-01-16T15:15:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06dfe00640 1 -- 172.21.15.43:0/3821896867 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d0030d30 con 0x7f06e00cfa30 2024-01-16T15:15:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 -- 172.21.15.43:0/3821896867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00cfa30 msgr2=0x7f06e00cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00cfa30 0x7f06e00cfe10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009870 tx=0x7f06d002b980 comp rx=0 tx=0).stop 2024-01-16T15:15:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 -- 172.21.15.43:0/3821896867 shutdown_connections 2024-01-16T15:15:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00d1fd0 0x7f06e00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 --2- 172.21.15.43:0/3821896867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00cfa30 0x7f06e00cfe10 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 -- 172.21.15.43:0/3821896867 >> 172.21.15.43:0/3821896867 conn(0x7f06e00d3020 msgr2=0x7f06e00d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 -- 172.21.15.43:0/3821896867 shutdown_connections 2024-01-16T15:15:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.035+0000 7f06e6e00640 1 -- 172.21.15.43:0/3821896867 wait complete. 2024-01-16T15:15:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.037+0000 7f06e6e00640 1 Processor -- start 2024-01-16T15:15:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.037+0000 7f06e6e00640 1 -- start start 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00cfa30 0x7f06e01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06e00d1fd0 0x7f06e00cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e015d7f0 con 0x7f06e00d0e00 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00b8180 con 0x7f06e00d1fd0 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.038+0000 7f06e6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00b8460 con 0x7f06e00cfa30 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55536/0 (socket says 172.21.15.43:55536) 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 -- 172.21.15.43:0/2736032714 learned_addr learned my addr 172.21.15.43:0/2736032714 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e6400640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06e00d1fd0 0x7f06e00cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5a00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00cfa30 0x7f06e01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 -- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00cfa30 msgr2=0x7f06e01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00cfa30 0x7f06e01555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 -- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06e00d1fd0 msgr2=0x7f06e00cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06e00d1fd0 0x7f06e00cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.039+0000 7f06e5000640 1 -- 172.21.15.43:0/2736032714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00094d0 con 0x7f06e00d0e00 2024-01-16T15:15:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06e5000640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c7f0 tx=0x7f06c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8007820 con 0x7f06e00d0e00 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c8007980 con 0x7f06e00d0e00 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8007cf0 con 0x7f06e00d0e00 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e00b8740 con 0x7f06e00d0e00 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.040+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e0155b20 con 0x7f06e00d0e00 2024-01-16T15:15:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.041+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e00d0a40 con 0x7f06e00d0e00 2024-01-16T15:15:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.042+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f06c80113f0 con 0x7f06e00d0e00 2024-01-16T15:15:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.043+0000 7f06dd600640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 0x7f06b806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.043+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f06c8088b90 con 0x7f06e00d0e00 2024-01-16T15:15:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.044+0000 7f06e5a00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 0x7f06b806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.044+0000 7f06e5a00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 0x7f06b806e590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f06d0036000 tx=0x7f06d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.048+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06c8055240 con 0x7f06e00d0e00 2024-01-16T15:15:11.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.242+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f06e004e840 con 0x7f06e00d0e00 2024-01-16T15:15:11.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.248+0000 7f06dd600640 1 -- 172.21.15.43:0/2736032714 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v38)=0 v38) v1 ==== 137+0+0 (secure 0 0 0) 0x7f06c8058860 con 0x7f06e00d0e00 2024-01-16T15:15:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.251+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 msgr2=0x7f06b806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.251+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 0x7f06b806e590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f06d0036000 tx=0x7f06d0008030 comp rx=0 tx=0).stop 2024-01-16T15:15:11.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.251+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 msgr2=0x7f06e00cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.251+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c7f0 tx=0x7f06c800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 shutdown_connections 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f06b806c0d0 0x7f06b806e590 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06e00d1fd0 0x7f06e00cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06e00d0e00 0x7f06e00cbc80 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 --2- 172.21.15.43:0/2736032714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06e00cfa30 0x7f06e01555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 >> 172.21.15.43:0/2736032714 conn(0x7f06e00d3020 msgr2=0x7f06e00b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 shutdown_connections 2024-01-16T15:15:11.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.252+0000 7f06e6e00640 1 -- 172.21.15.43:0/2736032714 wait complete. 2024-01-16T15:15:11.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:15:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.498+0000 7f424b800640 1 Processor -- start 2024-01-16T15:15:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.498+0000 7f424b800640 1 -- start start 2024-01-16T15:15:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.500+0000 7f424b800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 0x7f4244162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.500+0000 7f424b800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4244162fa0 con 0x7f424406c3c0 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.500+0000 7f424b800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42440706a0 con 0x7f4244162220 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.500+0000 7f424b800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4244070c60 con 0x7f4244162600 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4249800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f424406c3c0 0x7f42440710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36948/0 (socket says 172.21.15.43:36948) 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4249800640 1 -- 172.21.15.43:0/1225048301 learned_addr learned my addr 172.21.15.43:0/1225048301 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f423be00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 0x7f4244162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185916673 0 0) 0x7f4244162fa0 con 0x7f424406c3c0 2024-01-16T15:15:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4224003520 con 0x7f424406c3c0 2024-01-16T15:15:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3276035926 0 0) 0x7f4224003520 con 0x7f424406c3c0 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 msgr2=0x7f4244162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 0x7f4244162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.501+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 >> v1:172.21.15.69:6789/0 conn(0x7f4244162220 legacy=0x7f42440b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440722b0 con 0x7f424406c3c0 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f423c0019f0 con 0x7f424406c3c0 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 >> v1:172.21.15.43:6789/0 conn(0x7f424406c3c0 legacy=0x7f42440710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42240039a0 0x7f4224003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 0x7f4224005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4244070c60 con 0x7f42240039a0 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422400ba30 con 0x7f42240047c0 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422400bbf0 con 0x7f4244162600 2024-01-16T15:15:11.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42240039a0 0x7f4224003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f423be00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 -- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 msgr2=0x7f4224005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4244162600 0x7f4224005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 -- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42240047c0 msgr2=0x7f4224004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.502+0000 7f4248e00640 1 -- 172.21.15.43:0/1225048301 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440722b0 con 0x7f42240039a0 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f4248e00640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42240039a0 0x7f4224003dc0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4234001720 tx=0x7f42240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc0382faa534db2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42340052d0 con 0x7f42240039a0 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423402e050 con 0x7f42240039a0 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f4243e00640 1 -- 172.21.15.43:0/1225048301 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42340035f0 con 0x7f42240039a0 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f424b800640 1 -- 172.21.15.43:0/1225048301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42240039a0 msgr2=0x7f4224003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f424b800640 1 --2- 172.21.15.43:0/1225048301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42240039a0 0x7f4224003dc0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4234001720 tx=0x7f42240057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.504+0000 7f4249800640 1 -- 172.21.15.43:0/1225048301 reap_dead start 2024-01-16T15:15:11.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.505+0000 7f424b800640 1 -- 172.21.15.43:0/1225048301 shutdown_connections 2024-01-16T15:15:11.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.505+0000 7f424b800640 1 -- 172.21.15.43:0/1225048301 >> 172.21.15.43:0/1225048301 conn(0x7f42440b0ba0 msgr2=0x7f42440b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.505+0000 7f424b800640 1 -- 172.21.15.43:0/1225048301 shutdown_connections 2024-01-16T15:15:11.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.505+0000 7f424b800640 1 -- 172.21.15.43:0/1225048301 wait complete. 2024-01-16T15:15:11.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.507+0000 7f424b800640 1 Processor -- start 2024-01-16T15:15:11.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.507+0000 7f424b800640 1 -- start start 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42240039a0 0x7f424409c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 0x7f4244142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440ad260 con 0x7f42440cdf70 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42441f7ca0 con 0x7f42440ad6f0 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f424b800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440ac930 con 0x7f42240039a0 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f423be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f423be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55550/0 (socket says 172.21.15.43:55550) 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f423be00640 1 -- 172.21.15.43:0/3538948067 learned_addr learned my addr 172.21.15.43:0/3538948067 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f4248e00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42240039a0 0x7f424409c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.508+0000 7f4249800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 0x7f4244142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 -- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42240039a0 msgr2=0x7f424409c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42240039a0 0x7f424409c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 -- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 msgr2=0x7f4244142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 0x7f4244142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 -- 172.21.15.43:0/3538948067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440acc10 con 0x7f42440cdf70 2024-01-16T15:15:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f4249800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 0x7f4244142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f423be00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f422c010860 tx=0x7f422c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c009610 con 0x7f42440cdf70 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c00eb60 con 0x7f42440cdf70 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c01c400 con 0x7f42440cdf70 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42440acef0 con 0x7f42440cdf70 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.509+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424409c760 con 0x7f42440cdf70 2024-01-16T15:15:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.510+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424406e730 con 0x7f42440cdf70 2024-01-16T15:15:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.512+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f422c01c560 con 0x7f42440cdf70 2024-01-16T15:15:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.512+0000 7f4241600640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 0x7f421c06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.513+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f422c088dc0 con 0x7f42440cdf70 2024-01-16T15:15:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.513+0000 7f4248e00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 0x7f421c06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.514+0000 7f4248e00640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 0x7f421c06e5c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4234033000 tx=0x7f4234009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.517+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f422c055470 con 0x7f42440cdf70 2024-01-16T15:15:11.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.736+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f42440acaa0 con 0x7f421c06c100 2024-01-16T15:15:11.728 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:11.745+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:11.728 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:11.745+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:11.729 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:11.745+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:11.729 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:11.745+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:11.729 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:11.745+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:11.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.817+0000 7f4241600640 1 -- 172.21.15.43:0/3538948067 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f42440acaa0 con 0x7f421c06c100 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.819+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 msgr2=0x7f421c06e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.819+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 0x7f421c06e5c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4234033000 tx=0x7f4234009030 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.819+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 msgr2=0x7f42441f7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.819+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f422c010860 tx=0x7f422c010d20 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 shutdown_connections 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f421c06c100 0x7f421c06e5c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42440ad6f0 0x7f4244142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42440cdf70 0x7f42441f7760 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 --2- 172.21.15.43:0/3538948067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42240039a0 0x7f424409c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 >> 172.21.15.43:0/3538948067 conn(0x7f42440b0ba0 msgr2=0x7f4244073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 shutdown_connections 2024-01-16T15:15:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:11.820+0000 7f424b800640 1 -- 172.21.15.43:0/3538948067 wait complete. 2024-01-16T15:15:11.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:12.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.044+0000 7f80f4a00640 1 Processor -- start 2024-01-16T15:15:12.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.045+0000 7f80f4a00640 1 -- start start 2024-01-16T15:15:12.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.046+0000 7f80f4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80ec09cb30 0x7f80ec0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.046+0000 7f80f4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f80ec0adb60 0x7f80ec0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.046+0000 7f80f4a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec15b730 con 0x7f80ec15b350 2024-01-16T15:15:12.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.046+0000 7f80f4a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec0ac740 con 0x7f80ec0adb60 2024-01-16T15:15:12.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.046+0000 7f80f4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec0ac460 con 0x7f80ec09cb30 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f80ec0adb60 0x7f80ec0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f80ec0adb60 0x7f80ec0adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55566/0 (socket says 172.21.15.43:55566) 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 -- 172.21.15.43:0/619284501 learned_addr learned my addr 172.21.15.43:0/619284501 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f1600640 1 --2- 172.21.15.43:0/619284501 >> v2:172.21.15.69:3300/0 conn(0x7f80ec09cb30 0x7f80ec0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 -- 172.21.15.43:0/619284501 >> v2:172.21.15.69:3300/0 conn(0x7f80ec09cb30 msgr2=0x7f80ec0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 --2- 172.21.15.43:0/619284501 >> v2:172.21.15.69:3300/0 conn(0x7f80ec09cb30 0x7f80ec0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.047+0000 7f80f2a00640 1 -- 172.21.15.43:0/619284501 >> v1:172.21.15.69:6789/0 conn(0x7f80ec15b350 legacy=0x7f80ec0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.048+0000 7f80f2a00640 1 -- 172.21.15.43:0/619284501 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0b2a50 con 0x7f80ec0adb60 2024-01-16T15:15:12.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.049+0000 7f80f2a00640 1 --2- 172.21.15.43:0/619284501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec0adf70 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0d0060 tx=0x7f80dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39778fe5cb3308bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.050+0000 7f80f0c00640 1 -- 172.21.15.43:0/619284501 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc030860 con 0x7f80ec0adb60 2024-01-16T15:15:12.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.050+0000 7f80f0c00640 1 -- 172.21.15.43:0/619284501 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc0309c0 con 0x7f80ec0adb60 2024-01-16T15:15:12.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.050+0000 7f80f0c00640 1 -- 172.21.15.43:0/619284501 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc030d50 con 0x7f80ec0adb60 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.050+0000 7f80f4a00640 1 -- 172.21.15.43:0/619284501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 msgr2=0x7f80ec0adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.050+0000 7f80f4a00640 1 --2- 172.21.15.43:0/619284501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec0adf70 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0d0060 tx=0x7f80dc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 -- 172.21.15.43:0/619284501 shutdown_connections 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 --2- 172.21.15.43:0/619284501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec0adf70 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 --2- 172.21.15.43:0/619284501 >> v2:172.21.15.69:3300/0 conn(0x7f80ec09cb30 0x7f80ec0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 -- 172.21.15.43:0/619284501 >> 172.21.15.43:0/619284501 conn(0x7f80ec0b0ba0 msgr2=0x7f80ec0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 -- 172.21.15.43:0/619284501 shutdown_connections 2024-01-16T15:15:12.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.051+0000 7f80f4a00640 1 -- 172.21.15.43:0/619284501 wait complete. 2024-01-16T15:15:12.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.053+0000 7f80f4a00640 1 Processor -- start 2024-01-16T15:15:12.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.053+0000 7f80f4a00640 1 -- start start 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.054+0000 7f80f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80ec09cb30 0x7f80ec1512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.054+0000 7f80f4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.054+0000 7f80f4a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80ec15b350 0x7f80ec1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec0c8390 con 0x7f80ec0adb60 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec157270 con 0x7f80ec09cb30 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f4a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec157550 con 0x7f80ec15b350 2024-01-16T15:15:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55568/0 (socket says 172.21.15.43:55568) 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 -- 172.21.15.43:0/1953145268 learned_addr learned my addr 172.21.15.43:0/1953145268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f2a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80ec15b350 0x7f80ec1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f2000640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80ec09cb30 0x7f80ec1512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 -- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80ec15b350 msgr2=0x7f80ec1f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80ec15b350 0x7f80ec1f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.055+0000 7f80f1600640 1 -- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80ec09cb30 msgr2=0x7f80ec1512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80f1600640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80ec09cb30 0x7f80ec1512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80f1600640 1 -- 172.21.15.43:0/1953145268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc0094d0 con 0x7f80ec0adb60 2024-01-16T15:15:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80f1600640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f80d400efc0 tx=0x7f80d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4014480 con 0x7f80ec0adb60 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d400f040 con 0x7f80ec0adb60 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4014800 con 0x7f80ec0adb60 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec157830 con 0x7f80ec0adb60 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.056+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec157df0 con 0x7f80ec0adb60 2024-01-16T15:15:12.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.058+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ec157f80 con 0x7f80ec0adb60 2024-01-16T15:15:12.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.059+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f80d4014960 con 0x7f80ec0adb60 2024-01-16T15:15:12.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.060+0000 7f80ea000640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 0x7f80c406e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.060+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f80d4089570 con 0x7f80ec0adb60 2024-01-16T15:15:12.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.060+0000 7f80f2000640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 0x7f80c406e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.061+0000 7f80f2000640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 0x7f80c406e5c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f80e4000f60 tx=0x7f80e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.064+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80d40bee90 con 0x7f80ec0adb60 2024-01-16T15:15:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.291+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f80ec06e720 con 0x7f80c406c100 2024-01-16T15:15:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.297+0000 7f80ea000640 1 -- 172.21.15.43:0/1953145268 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f80e40050d0 con 0x7f80c406c100 2024-01-16T15:15:12.281 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 msgr2=0x7f80c406e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 0x7f80c406e5c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f80e4000f60 tx=0x7f80e4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 msgr2=0x7f80ec151b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f80d400efc0 tx=0x7f80d400c490 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 shutdown_connections 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f80c406c100 0x7f80c406e5c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80ec15b350 0x7f80ec1f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80ec0adb60 0x7f80ec151b60 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 --2- 172.21.15.43:0/1953145268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80ec09cb30 0x7f80ec1512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 >> 172.21.15.43:0/1953145268 conn(0x7f80ec0b0ba0 msgr2=0x7f80ec09ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.300+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 shutdown_connections 2024-01-16T15:15:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.301+0000 7f80f4a00640 1 -- 172.21.15.43:0/1953145268 wait complete. 2024-01-16T15:15:12.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:12.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.530+0000 7ff9f0000640 1 Processor -- start 2024-01-16T15:15:12.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.530+0000 7ff9f0000640 1 -- start start 2024-01-16T15:15:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.531+0000 7ff9f0000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9e815b350 0x7ff9e80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9f0000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 0x7ff9e80b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9f0000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e815b730 con 0x7ff9e80adb60 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9f0000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80ac740 con 0x7ff9e815b350 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9f0000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80ac460 con 0x7ff9e809cb30 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9ee000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff9e80adb60 0x7ff9e80adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36952/0 (socket says 172.21.15.43:36952) 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9ee000640 1 -- 172.21.15.43:0/4294917410 learned_addr learned my addr 172.21.15.43:0/4294917410 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9ed600640 1 --2- 172.21.15.43:0/4294917410 >> v2:172.21.15.69:3300/0 conn(0x7ff9e815b350 0x7ff9e80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021115600 0 0) 0x7ff9e0001490 con 0x7ff9e80adb60 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 0x7ff9e80b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.532+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c8003520 con 0x7ff9e80adb60 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178727709 0 0) 0x7ff9c8003520 con 0x7ff9e80adb60 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 msgr2=0x7ff9e80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 0x7ff9e80b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 >> v2:172.21.15.69:3300/0 conn(0x7ff9e815b350 msgr2=0x7ff9e80abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 --2- 172.21.15.43:0/4294917410 >> v2:172.21.15.69:3300/0 conn(0x7ff9e815b350 0x7ff9e80abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80b2a50 con 0x7ff9e80adb60 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff9e0001b80 con 0x7ff9e80adb60 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 >> v1:172.21.15.43:6789/0 conn(0x7ff9e80adb60 legacy=0x7ff9e80adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9c8003970 0x7ff9c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 0x7ff9c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80ac740 con 0x7ff9c80047c0 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80ac460 con 0x7ff9c8003970 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c800b980 con 0x7ff9e809cb30 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.533+0000 7ff9ed600640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9c8003970 0x7ff9c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.534+0000 7ff9ecc00640 1 -- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 msgr2=0x7ff9c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.534+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e809cb30 0x7ff9c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.534+0000 7ff9ecc00640 1 -- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9c8003970 msgr2=0x7ff9c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.534+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9c8003970 0x7ff9c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.534+0000 7ff9ecc00640 1 -- 172.21.15.43:0/4294917410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8003520 con 0x7ff9c80047c0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9c8004c40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009ac0 tx=0x7ff9c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7045b1e21e57997 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d002c6d0 con 0x7ff9c80047c0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d002c830 con 0x7ff9c80047c0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9e7e00640 1 -- 172.21.15.43:0/4294917410 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d002cbc0 con 0x7ff9c80047c0 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9f0000640 1 -- 172.21.15.43:0/4294917410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 msgr2=0x7ff9c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9f0000640 1 --2- 172.21.15.43:0/4294917410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9c8004c40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009ac0 tx=0x7ff9c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.535+0000 7ff9ee000640 1 -- 172.21.15.43:0/4294917410 reap_dead start 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.536+0000 7ff9f0000640 1 -- 172.21.15.43:0/4294917410 shutdown_connections 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.536+0000 7ff9f0000640 1 -- 172.21.15.43:0/4294917410 >> 172.21.15.43:0/4294917410 conn(0x7ff9e80b0ba0 msgr2=0x7ff9e80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.536+0000 7ff9f0000640 1 -- 172.21.15.43:0/4294917410 shutdown_connections 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.536+0000 7ff9f0000640 1 -- 172.21.15.43:0/4294917410 wait complete. 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.537+0000 7ff9f0000640 1 Processor -- start 2024-01-16T15:15:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.538+0000 7ff9f0000640 1 -- start start 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.538+0000 7ff9f0000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9f0000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8152fa0 0x7ff9e80c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9f0000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8155a90 0x7ff9e81595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9f0000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81f4270 con 0x7ff9c80047c0 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9f0000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80c8cf0 con 0x7ff9e8152fa0 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9f0000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81f39d0 con 0x7ff9e8155a90 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55586/0 (socket says 172.21.15.43:55586) 2024-01-16T15:15:12.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 -- 172.21.15.43:0/2399365568 learned_addr learned my addr 172.21.15.43:0/2399365568 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8152fa0 0x7ff9e80c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ee000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8155a90 0x7ff9e81595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 -- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8155a90 msgr2=0x7ff9e81595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8155a90 0x7ff9e81595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 -- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8152fa0 msgr2=0x7ff9e80c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8152fa0 0x7ff9e80c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.539+0000 7ff9ed600640 1 -- 172.21.15.43:0/2399365568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e81f3cb0 con 0x7ff9c80047c0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9ed600640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80116d0 tx=0x7ff9d8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800f9c0 con 0x7ff9c80047c0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800fb20 con 0x7ff9c80047c0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e81f3f90 con 0x7ff9c80047c0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800fe90 con 0x7ff9c80047c0 2024-01-16T15:15:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.540+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e8153420 con 0x7ff9c80047c0 2024-01-16T15:15:12.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.541+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80b1f40 con 0x7ff9c80047c0 2024-01-16T15:15:12.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.543+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 35) v1 ==== 90814+0+0 (secure 0 0 0) 0x7ff9d8015460 con 0x7ff9c80047c0 2024-01-16T15:15:12.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.544+0000 7ff9e5600640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 0x7ff9c006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.544+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff9d8088ea0 con 0x7ff9c80047c0 2024-01-16T15:15:12.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.544+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 0x7ff9c006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.545+0000 7ff9ecc00640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 0x7ff9c006e5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.548+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9d8055550 con 0x7ff9c80047c0 2024-01-16T15:15:12.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.766+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e806e720 con 0x7ff9c006c100 2024-01-16T15:15:12.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.772+0000 7ff9e5600640 1 -- 172.21.15.43:0/2399365568 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ff9d0004040 con 0x7ff9c006c100 2024-01-16T15:15:12.756 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3 2024-01-16T15:15:12.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 msgr2=0x7ff9c006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 0x7ff9c006e5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d0030000 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 msgr2=0x7ff9e81f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80116d0 tx=0x7ff9d8011b90 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 shutdown_connections 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff9c006c100 0x7ff9c006e5c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9e8152fa0 0x7ff9e80c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9e8155a90 0x7ff9e81595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.774+0000 7ff9f0000640 1 --2- 172.21.15.43:0/2399365568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9c80047c0 0x7ff9e81f32d0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.775+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 >> 172.21.15.43:0/2399365568 conn(0x7ff9e80b0ba0 msgr2=0x7ff9e8153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.775+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 shutdown_connections 2024-01-16T15:15:12.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:12.775+0000 7ff9f0000640 1 -- 172.21.15.43:0/2399365568 wait complete. 2024-01-16T15:15:12.780 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3 2024-01-16T15:15:12.780 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:15:12.780 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:12.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:12.833 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:28.652 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:15:28.652 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:15:28.652 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.7363 s, 68.2 MB/s 2024-01-16T15:15:28.654 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:15:28.654 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:28.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:28.712 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:29.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:29.222 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:42.432 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:15:42.433 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:15:42.433 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6511 s, 78.7 MB/s 2024-01-16T15:15:42.435 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:15:42.435 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:42.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:42.489 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:15:56.299 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:15:56.299 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.7478 s, 78.1 MB/s 2024-01-16T15:15:56.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:15:56.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.545+0000 7f6966c00640 1 Processor -- start 2024-01-16T15:15:56.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.545+0000 7f6966c00640 1 -- start start 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6960144770 0x7f696015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f696015f0a0 0x7f69600ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696015f930 con 0x7f69600b35a0 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69600b43b0 con 0x7f6960144770 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696015f510 con 0x7f696015f0a0 2024-01-16T15:15:56.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6964e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f69600b35a0 0x7f696006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33174/0 (socket says 172.21.15.43:33174) 2024-01-16T15:15:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6965800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6960144770 0x7f696015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6964e00640 1 -- 172.21.15.43:0/1584855819 learned_addr learned my addr 172.21.15.43:0/1584855819 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6966200640 1 --2- 172.21.15.43:0/1584855819 >> v2:172.21.15.69:3300/0 conn(0x7f696015f0a0 0x7f69600ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f695fe00640 1 -- 172.21.15.43:0/1584855819 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921894327 0 0) 0x7f696015f930 con 0x7f69600b35a0 2024-01-16T15:15:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6965800640 1 -- 172.21.15.43:0/1584855819 >> v2:172.21.15.69:3300/0 conn(0x7f696015f0a0 msgr2=0x7f69600ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6965800640 1 --2- 172.21.15.43:0/1584855819 >> v2:172.21.15.69:3300/0 conn(0x7f696015f0a0 0x7f69600ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6965800640 1 -- 172.21.15.43:0/1584855819 >> v1:172.21.15.43:6789/0 conn(0x7f69600b35a0 legacy=0x7f696006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.547+0000 7f6965800640 1 -- 172.21.15.43:0/1584855819 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69600ce9f0 con 0x7f6960144770 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.548+0000 7f6965800640 1 --2- 172.21.15.43:0/1584855819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6960144770 0x7f696015eb60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f69600700e0 tx=0x7f695002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fba344f34d5173 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.548+0000 7f695fe00640 1 -- 172.21.15.43:0/1584855819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950031830 con 0x7f6960144770 2024-01-16T15:15:56.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.548+0000 7f695fe00640 1 -- 172.21.15.43:0/1584855819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6950031990 con 0x7f6960144770 2024-01-16T15:15:56.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.548+0000 7f695fe00640 1 -- 172.21.15.43:0/1584855819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950031d20 con 0x7f6960144770 2024-01-16T15:15:56.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 -- 172.21.15.43:0/1584855819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6960144770 msgr2=0x7f696015eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 --2- 172.21.15.43:0/1584855819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6960144770 0x7f696015eb60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f69600700e0 tx=0x7f695002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 -- 172.21.15.43:0/1584855819 shutdown_connections 2024-01-16T15:15:56.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 --2- 172.21.15.43:0/1584855819 >> v2:172.21.15.69:3300/0 conn(0x7f696015f0a0 0x7f69600ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 --2- 172.21.15.43:0/1584855819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6960144770 0x7f696015eb60 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 -- 172.21.15.43:0/1584855819 >> 172.21.15.43:0/1584855819 conn(0x7f69600cec10 msgr2=0x7f69600cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:56.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 -- 172.21.15.43:0/1584855819 shutdown_connections 2024-01-16T15:15:56.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.549+0000 7f6966c00640 1 -- 172.21.15.43:0/1584855819 wait complete. 2024-01-16T15:15:56.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.551+0000 7f6966c00640 1 Processor -- start 2024-01-16T15:15:56.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.551+0000 7f6966c00640 1 -- start start 2024-01-16T15:15:56.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6960144770 0x7f696013a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f696015f0a0 0x7f69600713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69600ae320 con 0x7f69600b35a0 2024-01-16T15:15:56.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69600c8620 con 0x7f696015f0a0 2024-01-16T15:15:56.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69600ab570 con 0x7f6960144770 2024-01-16T15:15:56.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:56.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58830/0 (socket says 172.21.15.43:58830) 2024-01-16T15:15:56.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 -- 172.21.15.43:0/985546843 learned_addr learned my addr 172.21.15.43:0/985546843 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 -- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6960144770 msgr2=0x7f696013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6960144770 0x7f696013a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 -- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f696015f0a0 msgr2=0x7f69600713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f696015f0a0 0x7f69600713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 -- 172.21.15.43:0/985546843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950009400 con 0x7f69600b35a0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6965800640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f695002bee0 tx=0x7f69500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695003a6f0 con 0x7f69600b35a0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695003a870 con 0x7f69600b35a0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695003ac20 con 0x7f69600b35a0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.552+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6960070be0 con 0x7f69600b35a0 2024-01-16T15:15:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.553+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6960070e80 con 0x7f69600b35a0 2024-01-16T15:15:56.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.554+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6960071070 con 0x7f69600b35a0 2024-01-16T15:15:56.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.555+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 36) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f695004c3b0 con 0x7f69600b35a0 2024-01-16T15:15:56.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.557+0000 7f695d600640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 0x7f693806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:56.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.557+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f69500aa7a0 con 0x7f69600b35a0 2024-01-16T15:15:56.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.557+0000 7f6964e00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 0x7f693806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:56.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.557+0000 7f6964e00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 0x7f693806e670 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6948007fd0 tx=0x7f6948007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:56.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.562+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6950076ea0 con 0x7f69600b35a0 2024-01-16T15:15:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.784+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f69600b40d0 con 0x7f693806c1b0 2024-01-16T15:15:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.816+0000 7f695d600640 1 -- 172.21.15.43:0/985546843 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f69480070d0 con 0x7f693806c1b0 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 msgr2=0x7f693806e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 0x7f693806e670 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6948007fd0 tx=0x7f6948007920 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 msgr2=0x7f69600c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f695002bee0 tx=0x7f69500046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 shutdown_connections 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f693806c1b0 0x7f693806e670 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f696015f0a0 0x7f69600713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6960144770 0x7f696013a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 --2- 172.21.15.43:0/985546843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69600b35a0 0x7f69600c8040 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.819+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 >> 172.21.15.43:0/985546843 conn(0x7f69600cec10 msgr2=0x7f69600b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.820+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 shutdown_connections 2024-01-16T15:15:56.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:56.820+0000 7f6966c00640 1 -- 172.21.15.43:0/985546843 wait complete. 2024-01-16T15:15:56.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:57.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.067+0000 7fe201c00640 1 Processor -- start 2024-01-16T15:15:57.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.067+0000 7fe201c00640 1 -- start start 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1fc153ea0 0x7fe1fc0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 0x7fe1fc071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0ce650 con 0x7fe1fc154280 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc154710 con 0x7fe1fc153ea0 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc15e0a0 con 0x7fe1fc070850 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1fc153ea0 0x7fe1fc0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1fc153ea0 0x7fe1fc0ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58840/0 (socket says 172.21.15.43:58840) 2024-01-16T15:15:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 -- 172.21.15.43:0/2296097954 learned_addr learned my addr 172.21.15.43:0/2296097954 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fb400640 1 --1- 172.21.15.43:0/2296097954 >> v1:172.21.15.43:6789/0 conn(0x7fe1fc154280 0x7fe1fc15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33176/0 (socket says 172.21.15.43:33176) 2024-01-16T15:15:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 -- 172.21.15.43:0/2296097954 >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 msgr2=0x7fe1fc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201200640 1 --2- 172.21.15.43:0/2296097954 >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 0x7fe1fc071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 --2- 172.21.15.43:0/2296097954 >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 0x7fe1fc071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 -- 172.21.15.43:0/2296097954 >> v1:172.21.15.43:6789/0 conn(0x7fe1fc154280 legacy=0x7fe1fc15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe1fbe00640 1 -- 172.21.15.43:0/2296097954 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc15e380 con 0x7fe1fc153ea0 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.069+0000 7fe201200640 1 --2- 172.21.15.43:0/2296097954 >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 0x7fe1fc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.070+0000 7fe1fbe00640 1 --2- 172.21.15.43:0/2296097954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc0ad450 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc157e50 tx=0x7fe1ec02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474063490b6c5dc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.070+0000 7fe1faa00640 1 -- 172.21.15.43:0/2296097954 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec030870 con 0x7fe1fc153ea0 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.070+0000 7fe1faa00640 1 -- 172.21.15.43:0/2296097954 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec0309d0 con 0x7fe1fc153ea0 2024-01-16T15:15:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.070+0000 7fe1faa00640 1 -- 172.21.15.43:0/2296097954 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec030d60 con 0x7fe1fc153ea0 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 -- 172.21.15.43:0/2296097954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc153ea0 msgr2=0x7fe1fc0ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 --2- 172.21.15.43:0/2296097954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc0ad450 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc157e50 tx=0x7fe1ec02b9b0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 -- 172.21.15.43:0/2296097954 shutdown_connections 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 --2- 172.21.15.43:0/2296097954 >> v2:172.21.15.69:3300/0 conn(0x7fe1fc070850 0x7fe1fc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 --2- 172.21.15.43:0/2296097954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc0ad450 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 -- 172.21.15.43:0/2296097954 >> 172.21.15.43:0/2296097954 conn(0x7fe1fc0cec10 msgr2=0x7fe1fc0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:57.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 -- 172.21.15.43:0/2296097954 shutdown_connections 2024-01-16T15:15:57.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.071+0000 7fe201c00640 1 -- 172.21.15.43:0/2296097954 wait complete. 2024-01-16T15:15:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.073+0000 7fe201c00640 1 Processor -- start 2024-01-16T15:15:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.073+0000 7fe201c00640 1 -- start start 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1fc154280 0x7fe1fc13a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0b0290 con 0x7fe1fc070850 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0b0b30 con 0x7fe1fc153ea0 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc14f2b0 con 0x7fe1fc154280 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58852/0 (socket says 172.21.15.43:58852) 2024-01-16T15:15:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 -- 172.21.15.43:0/3747470797 learned_addr learned my addr 172.21.15.43:0/3747470797 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 -- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1fc154280 msgr2=0x7fe1fc13a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1fc154280 0x7fe1fc13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 -- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc153ea0 msgr2=0x7fe1fc1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 -- 172.21.15.43:0/3747470797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0094d0 con 0x7fe1fc070850 2024-01-16T15:15:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1fbe00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec030860 con 0x7fe1fc070850 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec044480 con 0x7fe1fc070850 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec044830 con 0x7fe1fc070850 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.075+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc0b04e0 con 0x7fe1fc070850 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.076+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc14c680 con 0x7fe1fc070850 2024-01-16T15:15:57.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.078+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc04e840 con 0x7fe1fc070850 2024-01-16T15:15:57.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.079+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 36) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe1ec02f3c0 con 0x7fe1fc070850 2024-01-16T15:15:57.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.079+0000 7fe1f3e00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 0x7fe1d406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.079+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe1ec03e070 con 0x7fe1fc070850 2024-01-16T15:15:57.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.079+0000 7fe1fb400640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 0x7fe1d406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.080+0000 7fe1fb400640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 0x7fe1d406e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc1f6290 tx=0x7fe1e40079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.082+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1ec0783c0 con 0x7fe1fc070850 2024-01-16T15:15:57.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.308+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1fc070330 con 0x7fe1d406c160 2024-01-16T15:15:57.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.320+0000 7fe1f3e00640 1 -- 172.21.15.43:0/3747470797 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fe1e40070d0 con 0x7fe1d406c160 2024-01-16T15:15:57.305 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3 2024-01-16T15:15:57.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 msgr2=0x7fe1d406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 0x7fe1d406e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc1f6290 tx=0x7fe1e40079c0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 msgr2=0x7fe1fc14c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec004730 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 shutdown_connections 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe1d406c160 0x7fe1d406e620 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1fc153ea0 0x7fe1fc1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1fc154280 0x7fe1fc13a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 --2- 172.21.15.43:0/3747470797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1fc070850 0x7fe1fc14c140 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.322+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 >> 172.21.15.43:0/3747470797 conn(0x7fe1fc0cec10 msgr2=0x7fe1fc15d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.323+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 shutdown_connections 2024-01-16T15:15:57.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.323+0000 7fe201c00640 1 -- 172.21.15.43:0/3747470797 wait complete. 2024-01-16T15:15:57.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:57.328 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3) 2024-01-16T15:15:57.393 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3 2024-01-16T15:15:57.394 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:15:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:15:57.395 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:15:57.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone2461 2024-01-16T15:15:57.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.643+0000 7f48a0a00640 1 Processor -- start 2024-01-16T15:15:57.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.643+0000 7f48a0a00640 1 -- start start 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f48a0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c0ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f48a0a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f489c070850 0x7f489c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f48a0a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c0ce650 con 0x7f489c154280 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f48a0a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c15e0a0 con 0x7f489c070850 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f48a0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c15e380 con 0x7f489c153ea0 2024-01-16T15:15:57.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f4893e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f489c154280 0x7f489c15d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33192/0 (socket says 172.21.15.43:33192) 2024-01-16T15:15:57.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.645+0000 7f4893e00640 1 -- 172.21.15.43:0/1033169075 learned_addr learned my addr 172.21.15.43:0/1033169075 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489a800640 1 --2- 172.21.15.43:0/1033169075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c0ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489b200640 1 --2- 172.21.15.43:0/1033169075 >> v2:172.21.15.69:3300/0 conn(0x7f489c070850 0x7f489c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444933084 0 0) 0x7f489c0ce650 con 0x7f489c154280 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f487c003520 con 0x7f489c154280 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489a800640 1 -- 172.21.15.43:0/1033169075 >> v2:172.21.15.69:3300/0 conn(0x7f489c070850 msgr2=0x7f489c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489a800640 1 --2- 172.21.15.43:0/1033169075 >> v2:172.21.15.69:3300/0 conn(0x7f489c070850 0x7f489c071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489a800640 1 -- 172.21.15.43:0/1033169075 >> v1:172.21.15.43:6789/0 conn(0x7f489c154280 legacy=0x7f489c15d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004982517 0 0) 0x7f487c003520 con 0x7f489c154280 2024-01-16T15:15:57.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.646+0000 7f489a800640 1 -- 172.21.15.43:0/1033169075 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c15e660 con 0x7f489c153ea0 2024-01-16T15:15:57.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.648+0000 7f489a800640 1 --2- 172.21.15.43:0/1033169075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c0ad450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f488c0097a0 tx=0x7f488c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e30a13a3cb4659 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.648+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884001670 con 0x7f489c153ea0 2024-01-16T15:15:57.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.648+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c02e050 con 0x7f489c153ea0 2024-01-16T15:15:57.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.648+0000 7f4899e00640 1 -- 172.21.15.43:0/1033169075 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c030730 con 0x7f489c153ea0 2024-01-16T15:15:57.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 -- 172.21.15.43:0/1033169075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 msgr2=0x7f489c0ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 --2- 172.21.15.43:0/1033169075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c0ad450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f488c0097a0 tx=0x7f488c02b790 comp rx=0 tx=0).stop 2024-01-16T15:15:57.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 -- 172.21.15.43:0/1033169075 shutdown_connections 2024-01-16T15:15:57.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 --2- 172.21.15.43:0/1033169075 >> v2:172.21.15.69:3300/0 conn(0x7f489c070850 0x7f489c071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 --2- 172.21.15.43:0/1033169075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c0ad450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 -- 172.21.15.43:0/1033169075 >> 172.21.15.43:0/1033169075 conn(0x7f489c0cec10 msgr2=0x7f489c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:57.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 -- 172.21.15.43:0/1033169075 shutdown_connections 2024-01-16T15:15:57.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.649+0000 7f48a0a00640 1 -- 172.21.15.43:0/1033169075 wait complete. 2024-01-16T15:15:57.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.651+0000 7f48a0a00640 1 Processor -- start 2024-01-16T15:15:57.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.651+0000 7f48a0a00640 1 -- start start 2024-01-16T15:15:57.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 0x7f489c156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b1630 con 0x7f489c070850 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b09b0 con 0x7f489c154280 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f48a0a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b0c90 con 0x7f489c153ea0 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f489a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f489a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58860/0 (socket says 172.21.15.43:58860) 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.652+0000 7f489a800640 1 -- 172.21.15.43:0/614413989 learned_addr learned my addr 172.21.15.43:0/614413989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f4893e00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489b200640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 0x7f489c156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 -- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 msgr2=0x7f489c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 -- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 msgr2=0x7f489c156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:57.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 0x7f489c156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 -- 172.21.15.43:0/614413989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c009400 con 0x7f489c070850 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489b200640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 0x7f489c156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f489a800640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098c0 tx=0x7f488c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.653+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c03c820 con 0x7f489c070850 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.654+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c03c9a0 con 0x7f489c070850 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.654+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c03cd50 con 0x7f489c070850 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.654+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c0b0f70 con 0x7f489c070850 2024-01-16T15:15:57.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.654+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0b06a0 con 0x7f489c070850 2024-01-16T15:15:57.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.655+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c155820 con 0x7f489c070850 2024-01-16T15:15:57.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.656+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 36) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f488c039070 con 0x7f489c070850 2024-01-16T15:15:57.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.657+0000 7f4892a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 0x7f487406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:57.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.657+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f488c0aa380 con 0x7f489c070850 2024-01-16T15:15:57.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.657+0000 7f4893e00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 0x7f487406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:57.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.658+0000 7f4893e00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 0x7f487406e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f489c157920 tx=0x7f4884009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:57.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.662+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f488c076960 con 0x7f489c070850 2024-01-16T15:15:57.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:57.889+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone2461", "target": ["mon-mgr", ""]}) v1 -- 0x7f489c070330 con 0x7f487406c160 2024-01-16T15:15:58.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.042+0000 7f4892a00640 1 -- 172.21.15.43:0/614413989 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4884001430 con 0x7f487406c160 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 msgr2=0x7f487406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 0x7f487406e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f489c157920 tx=0x7f4884009040 comp rx=0 tx=0).stop 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 msgr2=0x7f489c0b0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098c0 tx=0x7f488c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 shutdown_connections 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f487406c160 0x7f487406e620 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f489c154280 0x7f489c156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f489c153ea0 0x7f489c156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 --2- 172.21.15.43:0/614413989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f489c070850 0x7f489c0b0160 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.045+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 >> 172.21.15.43:0/614413989 conn(0x7f489c0cec10 msgr2=0x7f489c06d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.046+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 shutdown_connections 2024-01-16T15:15:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.046+0000 7f48a0a00640 1 -- 172.21.15.43:0/614413989 wait complete. 2024-01-16T15:15:58.037 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.052+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.037 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.052+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.038 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.052+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.038 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.052+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.038 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.052+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone2462 2024-01-16T15:15:58.067 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.082+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.068 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.082+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.068 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.082+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.068 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.082+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.068 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:15:58.082+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.262+0000 7f56e2a00640 1 Processor -- start 2024-01-16T15:15:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.262+0000 7f56e2a00640 1 -- start start 2024-01-16T15:15:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56e2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f56dc06dc30 0x7f56dc0ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56e2a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc070490 0x7f56dc070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56e2a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc06d7a0 con 0x7f56dc0acd70 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56e2a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc15a5b0 con 0x7f56dc06dc30 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56e2a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0ae9d0 con 0x7f56dc070490 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.264+0000 7f56db400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f56dc06dc30 0x7f56dc0ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f56dc06dc30 0x7f56dc0ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58900/0 (socket says 172.21.15.43:58900) 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 -- 172.21.15.43:0/3909257813 learned_addr learned my addr 172.21.15.43:0/3909257813 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56e0a00640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc070490 0x7f56dc070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 -- 172.21.15.43:0/3909257813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc070490 msgr2=0x7f56dc070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc070490 0x7f56dc070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 -- 172.21.15.43:0/3909257813 >> v1:172.21.15.69:6789/0 conn(0x7f56dc0acd70 legacy=0x7f56dc0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.265+0000 7f56db400640 1 -- 172.21.15.43:0/3909257813 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc06ce70 con 0x7f56dc06dc30 2024-01-16T15:15:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.266+0000 7f56db400640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc0ae490 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009870 tx=0x7f56c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574481c6c837b8a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56daa00640 1 -- 172.21.15.43:0/3909257813 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4030840 con 0x7f56dc06dc30 2024-01-16T15:15:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56daa00640 1 -- 172.21.15.43:0/3909257813 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40309a0 con 0x7f56dc06dc30 2024-01-16T15:15:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56daa00640 1 -- 172.21.15.43:0/3909257813 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4030d30 con 0x7f56dc06dc30 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 -- 172.21.15.43:0/3909257813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 msgr2=0x7f56dc0ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc0ae490 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009870 tx=0x7f56c402b980 comp rx=0 tx=0).stop 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 -- 172.21.15.43:0/3909257813 shutdown_connections 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc0ae490 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 --2- 172.21.15.43:0/3909257813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc070490 0x7f56dc070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.267+0000 7f56e2a00640 1 -- 172.21.15.43:0/3909257813 >> 172.21.15.43:0/3909257813 conn(0x7f56dc0b0ba0 msgr2=0x7f56dc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.268+0000 7f56e2a00640 1 -- 172.21.15.43:0/3909257813 shutdown_connections 2024-01-16T15:15:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.268+0000 7f56e2a00640 1 -- 172.21.15.43:0/3909257813 wait complete. 2024-01-16T15:15:58.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.269+0000 7f56e2a00640 1 Processor -- start 2024-01-16T15:15:58.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.270+0000 7f56e2a00640 1 -- start start 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.270+0000 7f56e2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 0x7f56dc166710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e2a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc0acd70 0x7f56dc14fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e2a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1662f0 con 0x7f56dc06dc30 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e2a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc09d030 con 0x7f56dc070490 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e2a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc09bf50 con 0x7f56dc0acd70 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58912/0 (socket says 172.21.15.43:58912) 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 -- 172.21.15.43:0/1362745388 learned_addr learned my addr 172.21.15.43:0/1362745388 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56db400640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 0x7f56dc166710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56e0a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc0acd70 0x7f56dc14fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 -- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc0acd70 msgr2=0x7f56dc14fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc0acd70 0x7f56dc14fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 -- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 msgr2=0x7f56dc166710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 0x7f56dc166710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56dbe00640 1 -- 172.21.15.43:0/1362745388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c40094d0 con 0x7f56dc06dc30 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.271+0000 7f56db400640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 0x7f56dc166710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56dbe00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c400 tx=0x7f56cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0052d0 con 0x7f56dc06dc30 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc005430 con 0x7f56dc06dc30 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc003830 con 0x7f56dc06dc30 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc165c30 con 0x7f56dc06dc30 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.272+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc166c50 con 0x7f56dc06dc30 2024-01-16T15:15:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.273+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc166560 con 0x7f56dc06dc30 2024-01-16T15:15:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.274+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 36) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f56cc003990 con 0x7f56dc06dc30 2024-01-16T15:15:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.275+0000 7f56d3e00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 0x7f56b406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.276+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f56cc087ea0 con 0x7f56dc06dc30 2024-01-16T15:15:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.276+0000 7f56db400640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 0x7f56b406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.276+0000 7f56db400640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 0x7f56b406e620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f56c4036000 tx=0x7f56c403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.280+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56cc054480 con 0x7f56dc06dc30 2024-01-16T15:15:58.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.514+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone2462", "target": ["mon-mgr", ""]}) v1 -- 0x7f56dc0b3d90 con 0x7f56b406c160 2024-01-16T15:15:58.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.662+0000 7f56d3e00640 1 -- 172.21.15.43:0/1362745388 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f56c4030cf0 con 0x7f56b406c160 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.666+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 msgr2=0x7f56b406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.666+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 0x7f56b406e620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f56c4036000 tx=0x7f56c403b040 comp rx=0 tx=0).stop 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.666+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 msgr2=0x7f56dc09caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.666+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c400 tx=0x7f56cc00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 shutdown_connections 2024-01-16T15:15:58.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f56b406c160 0x7f56b406e620 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56dc070490 0x7f56dc166710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56dc0acd70 0x7f56dc14fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 --2- 172.21.15.43:0/1362745388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56dc06dc30 0x7f56dc09caf0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 >> 172.21.15.43:0/1362745388 conn(0x7f56dc0b0ba0 msgr2=0x7f56dc06c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 shutdown_connections 2024-01-16T15:15:58.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.667+0000 7f56e2a00640 1 -- 172.21.15.43:0/1362745388 wait complete. 2024-01-16T15:15:58.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone2463 2024-01-16T15:15:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.965+0000 7fa8aec00640 1 Processor -- start 2024-01-16T15:15:58.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.965+0000 7fa8aec00640 1 -- start start 2024-01-16T15:15:58.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.966+0000 7fa8aec00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa8a8162220 0x7fa8a80b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.966+0000 7fa8aec00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa8a8162600 0x7fa8a8162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.966+0000 7fa8aec00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a8162fa0 con 0x7fa8a806c3c0 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.966+0000 7fa8aec00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a80706a0 con 0x7fa8a8162600 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.966+0000 7fa8aec00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a8070c60 con 0x7fa8a8162220 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa8a8162600 0x7fa8a8162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa8a8162600 0x7fa8a8162a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58928/0 (socket says 172.21.15.43:58928) 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 -- 172.21.15.43:0/190413253 learned_addr learned my addr 172.21.15.43:0/190413253 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7e00640 1 --2- 172.21.15.43:0/190413253 >> v2:172.21.15.69:3300/0 conn(0x7fa8a8162220 0x7fa8a80b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 -- 172.21.15.43:0/190413253 >> v2:172.21.15.69:3300/0 conn(0x7fa8a8162220 msgr2=0x7fa8a80b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 --2- 172.21.15.43:0/190413253 >> v2:172.21.15.69:3300/0 conn(0x7fa8a8162220 0x7fa8a80b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 -- 172.21.15.43:0/190413253 >> v1:172.21.15.43:6789/0 conn(0x7fa8a806c3c0 legacy=0x7fa8a80710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.967+0000 7fa8a7400640 1 -- 172.21.15.43:0/190413253 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80722b0 con 0x7fa8a8162600 2024-01-16T15:15:58.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.968+0000 7fa8a7400640 1 --2- 172.21.15.43:0/190413253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162600 0x7fa8a8162a60 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80b4070 tx=0x7fa89002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b9a824d4b779f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.969+0000 7fa8a6a00640 1 -- 172.21.15.43:0/190413253 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890030860 con 0x7fa8a8162600 2024-01-16T15:15:58.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.969+0000 7fa8a6a00640 1 -- 172.21.15.43:0/190413253 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8900309c0 con 0x7fa8a8162600 2024-01-16T15:15:58.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.969+0000 7fa8a6a00640 1 -- 172.21.15.43:0/190413253 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890030d50 con 0x7fa8a8162600 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.969+0000 7fa8aec00640 1 -- 172.21.15.43:0/190413253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162600 msgr2=0x7fa8a8162a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.969+0000 7fa8aec00640 1 --2- 172.21.15.43:0/190413253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162600 0x7fa8a8162a60 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80b4070 tx=0x7fa89002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 -- 172.21.15.43:0/190413253 shutdown_connections 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 --2- 172.21.15.43:0/190413253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162600 0x7fa8a8162a60 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 --2- 172.21.15.43:0/190413253 >> v2:172.21.15.69:3300/0 conn(0x7fa8a8162220 0x7fa8a80b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 -- 172.21.15.43:0/190413253 >> 172.21.15.43:0/190413253 conn(0x7fa8a80b0ba0 msgr2=0x7fa8a80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 -- 172.21.15.43:0/190413253 shutdown_connections 2024-01-16T15:15:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.970+0000 7fa8aec00640 1 -- 172.21.15.43:0/190413253 wait complete. 2024-01-16T15:15:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.971+0000 7fa8aec00640 1 Processor -- start 2024-01-16T15:15:58.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.972+0000 7fa8aec00640 1 -- start start 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8a806c3c0 0x7fa8a81f8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 0x7fa8a809a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a809c7c0 con 0x7fa8a8162220 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a81f8ec0 con 0x7fa8a8162600 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8aec00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a809bbb0 con 0x7fa8a806c3c0 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58944/0 (socket says 172.21.15.43:58944) 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 -- 172.21.15.43:0/3972066172 learned_addr learned my addr 172.21.15.43:0/3972066172 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7e00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8a806c3c0 0x7fa8a81f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8acc00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 0x7fa8a809a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 -- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8a806c3c0 msgr2=0x7fa8a81f8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8a806c3c0 0x7fa8a81f8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 -- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 msgr2=0x7fa8a809a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 0x7fa8a809a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8a7400640 1 -- 172.21.15.43:0/3972066172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900094d0 con 0x7fa8a8162220 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.973+0000 7fa8acc00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 0x7fa8a809a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa8a7400640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa890000c00 tx=0x7fa890004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890039e20 con 0x7fa8a8162220 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa890043400 con 0x7fa8a8162220 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900437b0 con 0x7fa8a8162220 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a809be90 con 0x7fa8a8162220 2024-01-16T15:15:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.974+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a809c450 con 0x7fa8a8162220 2024-01-16T15:15:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.976+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a809c610 con 0x7fa8a8162220 2024-01-16T15:15:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.977+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 36) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa890043910 con 0x7fa8a8162220 2024-01-16T15:15:58.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.977+0000 7fa89fe00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 0x7fa88006e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:58.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.978+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa89003e070 con 0x7fa8a8162220 2024-01-16T15:15:58.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.978+0000 7fa8a7e00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 0x7fa88006e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:58.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.978+0000 7fa8a7e00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 0x7fa88006e670 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa8980070b0 tx=0x7fa898007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:58.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:58.982+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa890078180 con 0x7fa8a8162220 2024-01-16T15:15:59.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:15:59.215+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone2463", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8a80b23e0 con 0x7fa88006c1b0 2024-01-16T15:15:59.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:59.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:00.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:00.332+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 37) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fa890078550 con 0x7fa8a8162220 2024-01-16T15:16:02.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.078+0000 7fa89fe00640 1 -- 172.21.15.43:0/3972066172 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa898007bc0 con 0x7fa88006c1b0 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 msgr2=0x7fa88006e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 0x7fa88006e670 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa8980070b0 tx=0x7fa898007040 comp rx=0 tx=0).stop 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 msgr2=0x7fa8a8099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa890000c00 tx=0x7fa890004730 comp rx=0 tx=0).stop 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 shutdown_connections 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa88006c1b0 0x7fa88006e670 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8a8162600 0x7fa8a809a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8a8162220 0x7fa8a8099ed0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 --2- 172.21.15.43:0/3972066172 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8a806c3c0 0x7fa8a81f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 >> 172.21.15.43:0/3972066172 conn(0x7fa8a80b0ba0 msgr2=0x7fa8a8160f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:02.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 shutdown_connections 2024-01-16T15:16:02.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.082+0000 7fa8aec00640 1 -- 172.21.15.43:0/3972066172 wait complete. 2024-01-16T15:16:02.094 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:16:02.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2463 2024-01-16T15:16:02.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.335+0000 7f4c67200640 1 Processor -- start 2024-01-16T15:16:02.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.336+0000 7f4c67200640 1 -- start start 2024-01-16T15:16:02.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.337+0000 7f4c67200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4c6015b350 0x7f4c600abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.337+0000 7f4c67200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4c600adb60 0x7f4c600adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.337+0000 7f4c67200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6015b730 con 0x7f4c6009cb30 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c67200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600ac740 con 0x7f4c6015b350 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c67200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600ac460 con 0x7f4c600adb60 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c57e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4c6009cb30 0x7f4c600b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33220/0 (socket says 172.21.15.43:33220) 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c57e00640 1 -- 172.21.15.43:0/2911887368 learned_addr learned my addr 172.21.15.43:0/2911887368 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/2911887368 >> v2:172.21.15.43:3300/0 conn(0x7f4c6015b350 0x7f4c600abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c65200640 1 --2- 172.21.15.43:0/2911887368 >> v2:172.21.15.69:3300/0 conn(0x7f4c600adb60 0x7f4c600adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.338+0000 7f4c5f400640 1 -- 172.21.15.43:0/2911887368 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690251060 0 0) 0x7f4c48001110 con 0x7f4c6009cb30 2024-01-16T15:16:02.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.339+0000 7f4c5f400640 1 -- 172.21.15.43:0/2911887368 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c40003520 con 0x7f4c6009cb30 2024-01-16T15:16:02.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.339+0000 7f4c5fe00640 1 -- 172.21.15.43:0/2911887368 >> v2:172.21.15.69:3300/0 conn(0x7f4c600adb60 msgr2=0x7f4c600adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:02.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.339+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/2911887368 >> v2:172.21.15.69:3300/0 conn(0x7f4c600adb60 0x7f4c600adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.339+0000 7f4c5fe00640 1 -- 172.21.15.43:0/2911887368 >> v1:172.21.15.43:6789/0 conn(0x7f4c6009cb30 legacy=0x7f4c600b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.339+0000 7f4c5fe00640 1 -- 172.21.15.43:0/2911887368 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600b2a50 con 0x7f4c6015b350 2024-01-16T15:16:02.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.340+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/2911887368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6015b350 0x7f4c600abe80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f4c6006dac0 tx=0x7f4c5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1a7d2317b2cd46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:02.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.340+0000 7f4c5f400640 1 -- 172.21.15.43:0/2911887368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50031830 con 0x7f4c6015b350 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.340+0000 7f4c5f400640 1 -- 172.21.15.43:0/2911887368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50031990 con 0x7f4c6015b350 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.340+0000 7f4c5f400640 1 -- 172.21.15.43:0/2911887368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50031d00 con 0x7f4c6015b350 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 -- 172.21.15.43:0/2911887368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6015b350 msgr2=0x7f4c600abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 --2- 172.21.15.43:0/2911887368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6015b350 0x7f4c600abe80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f4c6006dac0 tx=0x7f4c5002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 -- 172.21.15.43:0/2911887368 shutdown_connections 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 --2- 172.21.15.43:0/2911887368 >> v2:172.21.15.69:3300/0 conn(0x7f4c600adb60 0x7f4c600adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 --2- 172.21.15.43:0/2911887368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6015b350 0x7f4c600abe80 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 -- 172.21.15.43:0/2911887368 >> 172.21.15.43:0/2911887368 conn(0x7f4c600b0ba0 msgr2=0x7f4c600b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:02.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 -- 172.21.15.43:0/2911887368 shutdown_connections 2024-01-16T15:16:02.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.341+0000 7f4c67200640 1 -- 172.21.15.43:0/2911887368 wait complete. 2024-01-16T15:16:02.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.343+0000 7f4c67200640 1 Processor -- start 2024-01-16T15:16:02.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.343+0000 7f4c67200640 1 -- start start 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600adb60 0x7f4c6014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6015b350 0x7f4c601f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6009bf00 con 0x7f4c6009cb30 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6014cca0 con 0x7f4c600adb60 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.344+0000 7f4c67200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6014ceb0 con 0x7f4c6015b350 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58956/0 (socket says 172.21.15.43:58956) 2024-01-16T15:16:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3808095793 learned_addr learned my addr 172.21.15.43:0/3808095793 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c57e00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600adb60 0x7f4c6014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c65200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6015b350 0x7f4c601f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6015b350 msgr2=0x7f4c601f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6015b350 0x7f4c601f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600adb60 msgr2=0x7f4c6014c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600adb60 0x7f4c6014c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 -- 172.21.15.43:0/3808095793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50009400 con 0x7f4c6009cb30 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.345+0000 7f4c5fe00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4c5002b910 tx=0x7f4c500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.346+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50004450 con 0x7f4c6009cb30 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.346+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5003f070 con 0x7f4c6009cb30 2024-01-16T15:16:02.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.346+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50003830 con 0x7f4c6009cb30 2024-01-16T15:16:02.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.346+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6014d190 con 0x7f4c6009cb30 2024-01-16T15:16:02.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.346+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6014d750 con 0x7f4c6009cb30 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.347+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6014d320 con 0x7f4c6009cb30 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.348+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 37) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f4c50003990 con 0x7f4c6009cb30 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.349+0000 7f4c5cc00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 0x7f4c3806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.349+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4c50041070 con 0x7f4c6009cb30 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.349+0000 7f4c57e00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 0x7f4c3806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.350+0000 7f4c57e00640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 0x7f4c3806e740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c6013f130 tx=0x7f4c48009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:02.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.353+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c50076e50 con 0x7f4c6009cb30 2024-01-16T15:16:02.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:02.587+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2463", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c600b3010 con 0x7f4c3806c280 2024-01-16T15:16:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.936+0000 7f4c5cc00640 1 -- 172.21.15.43:0/3808095793 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f4c48001c90 con 0x7f4c3806c280 2024-01-16T15:16:06.922 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:16:06.923 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 msgr2=0x7f4c3806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 0x7f4c3806e740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c6013f130 tx=0x7f4c48009040 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 msgr2=0x7f4c6009b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f4c5002b910 tx=0x7f4c500046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 shutdown_connections 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4c3806c280 0x7f4c3806e740 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c600adb60 0x7f4c6014c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4c6015b350 0x7f4c601f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 --2- 172.21.15.43:0/3808095793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c6009cb30 0x7f4c6009b9c0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 >> 172.21.15.43:0/3808095793 conn(0x7f4c600b0ba0 msgr2=0x7f4c6006da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 shutdown_connections 2024-01-16T15:16:06.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:06.940+0000 7f4c67200640 1 -- 172.21.15.43:0/3808095793 wait complete. 2024-01-16T15:16:06.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone2461 2024-01-16T15:16:07.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.187+0000 7fdd91600640 1 Processor -- start 2024-01-16T15:16:07.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.187+0000 7fdd91600640 1 -- start start 2024-01-16T15:16:07.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd91600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd8c0b48d0 0x7fdd8c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd91600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd8c0b3210 0x7fdd8c167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd91600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8c0b23e0 con 0x7fdd8c160bd0 2024-01-16T15:16:07.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd91600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c0cdc70 con 0x7fdd8c0b3210 2024-01-16T15:16:07.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd91600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c0b2870 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd8b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd8c0b3210 0x7fdd8c167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd8b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd8c0b3210 0x7fdd8c167290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52586/0 (socket says 172.21.15.43:52586) 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd8b400640 1 -- 172.21.15.43:0/533507300 learned_addr learned my addr 172.21.15.43:0/533507300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.189+0000 7fdd8a000640 1 --2- 172.21.15.43:0/533507300 >> v2:172.21.15.69:3300/0 conn(0x7fdd8c0b48d0 0x7fdd8c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.190+0000 7fdd8b400640 1 -- 172.21.15.43:0/533507300 >> v2:172.21.15.69:3300/0 conn(0x7fdd8c0b48d0 msgr2=0x7fdd8c0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.190+0000 7fdd8b400640 1 --2- 172.21.15.43:0/533507300 >> v2:172.21.15.69:3300/0 conn(0x7fdd8c0b48d0 0x7fdd8c0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.190+0000 7fdd8b400640 1 -- 172.21.15.43:0/533507300 >> v1:172.21.15.69:6789/0 conn(0x7fdd8c160bd0 legacy=0x7fdd8c0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:07.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.190+0000 7fdd8b400640 1 -- 172.21.15.43:0/533507300 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c0d41b0 con 0x7fdd8c0b3210 2024-01-16T15:16:07.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.191+0000 7fdd8b400640 1 --2- 172.21.15.43:0/533507300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b3210 0x7fdd8c167290 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0d30f0 tx=0x7fdd8402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4bfc9d263db6cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.191+0000 7fdd89600640 1 -- 172.21.15.43:0/533507300 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8402c9f0 con 0x7fdd8c0b3210 2024-01-16T15:16:07.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.191+0000 7fdd89600640 1 -- 172.21.15.43:0/533507300 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8402cb50 con 0x7fdd8c0b3210 2024-01-16T15:16:07.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.191+0000 7fdd89600640 1 -- 172.21.15.43:0/533507300 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd840343f0 con 0x7fdd8c0b3210 2024-01-16T15:16:07.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 -- 172.21.15.43:0/533507300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b3210 msgr2=0x7fdd8c167290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 --2- 172.21.15.43:0/533507300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b3210 0x7fdd8c167290 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0d30f0 tx=0x7fdd8402be50 comp rx=0 tx=0).stop 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 -- 172.21.15.43:0/533507300 shutdown_connections 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 --2- 172.21.15.43:0/533507300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b3210 0x7fdd8c167290 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 --2- 172.21.15.43:0/533507300 >> v2:172.21.15.69:3300/0 conn(0x7fdd8c0b48d0 0x7fdd8c0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 -- 172.21.15.43:0/533507300 >> 172.21.15.43:0/533507300 conn(0x7fdd8c09c5a0 msgr2=0x7fdd8c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 -- 172.21.15.43:0/533507300 shutdown_connections 2024-01-16T15:16:07.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.192+0000 7fdd91600640 1 -- 172.21.15.43:0/533507300 wait complete. 2024-01-16T15:16:07.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.194+0000 7fdd91600640 1 Processor -- start 2024-01-16T15:16:07.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.194+0000 7fdd91600640 1 -- start start 2024-01-16T15:16:07.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd8c0b3210 0x7fdd8c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd8c160bd0 0x7fdd8c1f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c15b830 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c158eb0 con 0x7fdd8c160bd0 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd91600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c15c000 con 0x7fdd8c0b3210 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd8a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd8a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52592/0 (socket says 172.21.15.43:52592) 2024-01-16T15:16:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.195+0000 7fdd8a000640 1 -- 172.21.15.43:0/1618878462 learned_addr learned my addr 172.21.15.43:0/1618878462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:16:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 -- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd8c0b3210 msgr2=0x7fdd8c15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd8c0b3210 0x7fdd8c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 -- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd8c160bd0 msgr2=0x7fdd8c1f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd8c160bd0 0x7fdd8c1f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 -- 172.21.15.43:0/1618878462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84009b30 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd8a000640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d7d0 tx=0x7fdd7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74004030 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74004190 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400b6c0 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.196+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c15ba40 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.197+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c0b6130 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.198+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c0a1210 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.199+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 37) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fdd7400b820 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.200+0000 7fdd82a00640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 0x7fdd6406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:07.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.200+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd740888f0 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.200+0000 7fdd8aa00640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 0x7fdd6406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:07.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.201+0000 7fdd8aa00640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 0x7fdd6406e740 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006fd0 tx=0x7fdd7c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:07.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.205+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd740551b0 con 0x7fdd8c0b48d0 2024-01-16T15:16:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:07.437+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone2461", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd8c0b3760 con 0x7fdd6406c280 2024-01-16T15:16:11.928 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:16:11.942+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:11.928 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:16:11.942+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:11.928 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:16:11.942+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:11.928 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:16:11.942+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:11.928 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:16:11.942+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.160+0000 7fdd82a00640 1 -- 172.21.15.43:0/1618878462 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd7c009280 con 0x7fdd6406c280 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.163+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 msgr2=0x7fdd6406e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 0x7fdd6406e740 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006fd0 tx=0x7fdd7c006bc0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 msgr2=0x7fdd8c1582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d7d0 tx=0x7fdd7400dc90 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 shutdown_connections 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd6406c280 0x7fdd6406e740 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd8c160bd0 0x7fdd8c1f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd8c0b48d0 0x7fdd8c1582e0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 --2- 172.21.15.43:0/1618878462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd8c0b3210 0x7fdd8c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 >> 172.21.15.43:0/1618878462 conn(0x7fdd8c09c5a0 msgr2=0x7fdd8c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 shutdown_connections 2024-01-16T15:16:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:12.164+0000 7fdd91600640 1 -- 172.21.15.43:0/1618878462 wait complete. 2024-01-16T15:16:14.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone2462 2024-01-16T15:16:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.430+0000 7fe2bee00640 1 Processor -- start 2024-01-16T15:16:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.430+0000 7fe2bee00640 1 -- start start 2024-01-16T15:16:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bee00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe2b80adb60 0x7fe2b80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bee00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b815b730 con 0x7fe2b809cb30 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bee00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b80ac740 con 0x7fe2b80adb60 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80ac460 con 0x7fe2b815b350 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2b7400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe2b809cb30 0x7fe2b80b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33290/0 (socket says 172.21.15.43:33290) 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2b7400640 1 -- 172.21.15.43:0/3266744257 learned_addr learned my addr 172.21.15.43:0/3266744257 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:16:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2bce00640 1 --2- 172.21.15.43:0/3266744257 >> v2:172.21.15.43:3300/0 conn(0x7fe2b80adb60 0x7fe2b80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.432+0000 7fe2b7e00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2b6a00640 1 -- 172.21.15.43:0/3266744257 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648058557 0 0) 0x7fe2a0001110 con 0x7fe2b809cb30 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2b6a00640 1 -- 172.21.15.43:0/3266744257 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe298003520 con 0x7fe2b809cb30 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2bce00640 1 -- 172.21.15.43:0/3266744257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 msgr2=0x7fe2b80abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2bce00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b80abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2bce00640 1 -- 172.21.15.43:0/3266744257 >> v1:172.21.15.43:6789/0 conn(0x7fe2b809cb30 legacy=0x7fe2b80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.433+0000 7fe2bce00640 1 -- 172.21.15.43:0/3266744257 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80b2a50 con 0x7fe2b80adb60 2024-01-16T15:16:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.434+0000 7fe2bce00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b80adb60 0x7fe2b80adfb0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009d90 tx=0x7fe2b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afa8a50a133dc27 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2b6a00640 1 -- 172.21.15.43:0/3266744257 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b002cc50 con 0x7fe2b80adb60 2024-01-16T15:16:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2b6a00640 1 -- 172.21.15.43:0/3266744257 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b002cdb0 con 0x7fe2b80adb60 2024-01-16T15:16:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2b6a00640 1 -- 172.21.15.43:0/3266744257 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0034640 con 0x7fe2b80adb60 2024-01-16T15:16:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 -- 172.21.15.43:0/3266744257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b80adb60 msgr2=0x7fe2b80adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b80adb60 0x7fe2b80adfb0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009d90 tx=0x7fe2b002bea0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 -- 172.21.15.43:0/3266744257 shutdown_connections 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b80adb60 0x7fe2b80adfb0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 --2- 172.21.15.43:0/3266744257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b80abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.435+0000 7fe2bee00640 1 -- 172.21.15.43:0/3266744257 >> 172.21.15.43:0/3266744257 conn(0x7fe2b80b0ba0 msgr2=0x7fe2b80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.436+0000 7fe2bee00640 1 -- 172.21.15.43:0/3266744257 shutdown_connections 2024-01-16T15:16:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.436+0000 7fe2bee00640 1 -- 172.21.15.43:0/3266744257 wait complete. 2024-01-16T15:16:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.437+0000 7fe2bee00640 1 Processor -- start 2024-01-16T15:16:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.438+0000 7fe2bee00640 1 -- start start 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b80adb60 0x7fe2b8158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b81e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b8156cc0 con 0x7fe2b809cb30 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b814ea00 con 0x7fe2b80adb60 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2bee00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b81563d0 con 0x7fe2b815b350 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57906/0 (socket says 172.21.15.43:57906) 2024-01-16T15:16:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 -- 172.21.15.43:0/1133014274 learned_addr learned my addr 172.21.15.43:0/1133014274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:16:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 -- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 msgr2=0x7fe2b81e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b81e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 -- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b80adb60 msgr2=0x7fe2b8158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b80adb60 0x7fe2b8158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.439+0000 7fe2b7e00640 1 -- 172.21.15.43:0/1133014274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b0009a70 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2b7e00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800d800 tx=0x7fe2a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800b6c0 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a800b820 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800bbb0 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b8158ab0 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.440+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b8156950 con 0x7fe2b809cb30 2024-01-16T15:16:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.442+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b80b5200 con 0x7fe2b809cb30 2024-01-16T15:16:14.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.442+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 38) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fe2a800bd10 con 0x7fe2b809cb30 2024-01-16T15:16:14.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.443+0000 7fe2afe00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 0x7fe29006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:14.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.443+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe2a8088c10 con 0x7fe2b809cb30 2024-01-16T15:16:14.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.444+0000 7fe2b7400640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 0x7fe29006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:14.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.444+0000 7fe2b7400640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 0x7fe29006e800 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0000f60 tx=0x7fe2a0001110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:14.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.448+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2a8055530 con 0x7fe2b809cb30 2024-01-16T15:16:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:16:14.684+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone2462", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2b80ae4f0 con 0x7fe29006c340 2024-01-16T15:16:29.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:29.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:59.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:59.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:16.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:17:16.333+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 39) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fe2a8055900 con 0x7fe2b809cb30 2024-01-16T15:17:29.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:29.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:59.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:59.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:12.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.411+0000 7fe2afe00640 1 -- 172.21.15.43:0/1133014274 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe2a0005890 con 0x7fe29006c340 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 msgr2=0x7fe29006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 0x7fe29006e800 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0000f60 tx=0x7fe2a0001110 comp rx=0 tx=0).stop 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 msgr2=0x7fe2b814e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800d800 tx=0x7fe2a800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 shutdown_connections 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe29006c340 0x7fe29006e800 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b80adb60 0x7fe2b8158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2b815b350 0x7fe2b81e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 --2- 172.21.15.43:0/1133014274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2b809cb30 0x7fe2b814e4c0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 >> 172.21.15.43:0/1133014274 conn(0x7fe2b80b0ba0 msgr2=0x7fe2b80b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 shutdown_connections 2024-01-16T15:18:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:12.415+0000 7fe2bee00640 1 -- 172.21.15.43:0/1133014274 wait complete. 2024-01-16T15:18:14.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone2463 2024-01-16T15:18:14.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.674+0000 7f74a5a00640 1 Processor -- start 2024-01-16T15:18:14.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.675+0000 7f74a5a00640 1 -- start start 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f74a5a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f74a00cf600 0x7f74a00ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f74a5a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a00cf440 con 0x7f74a006a440 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f74a5a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a006ee00 con 0x7f74a00cefb0 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f74a5a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a006fa40 con 0x7f74a00cf600 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f749fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f74a006a440 0x7f74a006a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40598/0 (socket says 172.21.15.43:40598) 2024-01-16T15:18:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f749fe00640 1 -- 172.21.15.43:0/1667766555 learned_addr learned my addr 172.21.15.43:0/1667766555 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404358812 0 0) 0x7f7490001110 con 0x7f74a006a440 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7480003520 con 0x7f74a006a440 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.677+0000 7f74a5000640 1 --2- 172.21.15.43:0/1667766555 >> v2:172.21.15.69:3300/0 conn(0x7f74a00cf600 0x7f74a00ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122592501 0 0) 0x7f7480003520 con 0x7f74a006a440 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 >> v2:172.21.15.69:3300/0 conn(0x7f74a00cf600 msgr2=0x7f74a00ad4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 --2- 172.21.15.43:0/1667766555 >> v2:172.21.15.69:3300/0 conn(0x7f74a00cf600 0x7f74a00ad4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 >> v1:172.21.15.69:6789/0 conn(0x7f74a00cefb0 legacy=0x7f74a00acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a006f0a0 con 0x7f74a006a440 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7490001800 con 0x7f74a006a440 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 >> v1:172.21.15.43:6789/0 conn(0x7f74a006a440 legacy=0x7f74a006a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7480003a30 0x7f7480003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74800047c0 0x7f7480004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 0x7f748000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a006fa40 con 0x7f7480005380 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748000c220 con 0x7f74800047c0 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748000c3e0 con 0x7f7480003a30 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.678+0000 7f74a5000640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 0x7f748000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f749fe00640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7480003a30 0x7f7480003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f7497e00640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74800047c0 0x7f7480004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f74a5000640 1 -- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7480003a30 msgr2=0x7f7480003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f74a5000640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7480003a30 0x7f7480003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f74a5000640 1 -- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74800047c0 msgr2=0x7f7480004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:14.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f74a5000640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74800047c0 0x7f7480004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.679+0000 7f74a5000640 1 -- 172.21.15.43:0/1667766555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480003520 con 0x7f7480005380 2024-01-16T15:18:14.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.680+0000 7f74a5000640 1 -- 172.21.15.43:0/1667766555 reap_dead start 2024-01-16T15:18:14.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.680+0000 7f74a5000640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 0x7f748000b980 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f749800ba30 tx=0x7f74800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f269a9ac28099a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:14.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.680+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498004020 con 0x7f7480005380 2024-01-16T15:18:14.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.680+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498004180 con 0x7f7480005380 2024-01-16T15:18:14.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.680+0000 7f749f400640 1 -- 172.21.15.43:0/1667766555 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498003610 con 0x7f7480005380 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 -- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 msgr2=0x7f748000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 0x7f748000b980 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f749800ba30 tx=0x7f74800057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 -- 172.21.15.43:0/1667766555 shutdown_connections 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1667766555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7480005380 0x7f748000b980 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 -- 172.21.15.43:0/1667766555 >> 172.21.15.43:0/1667766555 conn(0x7f74a00cfd80 msgr2=0x7f74a00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 -- 172.21.15.43:0/1667766555 shutdown_connections 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.681+0000 7f74a5a00640 1 -- 172.21.15.43:0/1667766555 wait complete. 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.683+0000 7f74a5a00640 1 Processor -- start 2024-01-16T15:18:14.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.683+0000 7f74a5a00640 1 -- start start 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7480005380 0x7f74a015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74a0154590 0x7f74a009b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a015c940 con 0x7f74a015df20 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a015e690 con 0x7f7480005380 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f74a5a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a015e3b0 con 0x7f74a0154590 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f7497e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f7497e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46758/0 (socket says 172.21.15.43:46758) 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.684+0000 7f7497e00640 1 -- 172.21.15.43:0/1820289502 learned_addr learned my addr 172.21.15.43:0/1820289502 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f749fe00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7480005380 0x7f74a015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 -- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74a0154590 msgr2=0x7f74a009b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74a0154590 0x7f74a009b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 -- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7480005380 msgr2=0x7f74a015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7480005380 0x7f74a015cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 -- 172.21.15.43:0/1820289502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a015c2f0 con 0x7f74a015df20 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f7497e00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f748800e820 tx=0x7f748800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488003b80 con 0x7f74a015df20 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488003ce0 con 0x7f74a015df20 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488005310 con 0x7f74a015df20 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.685+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a015c5d0 con 0x7f74a015df20 2024-01-16T15:18:14.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.686+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a0154e10 con 0x7f74a015df20 2024-01-16T15:18:14.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.687+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a0066f80 con 0x7f74a015df20 2024-01-16T15:18:14.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.688+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 39) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f7488005470 con 0x7f74a015df20 2024-01-16T15:18:14.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.689+0000 7f749cc00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 0x7f747806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:14.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.690+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7488088850 con 0x7f74a015df20 2024-01-16T15:18:14.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.690+0000 7f749fe00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 0x7f747806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:14.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.691+0000 7f749fe00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 0x7f747806e740 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7490001270 tx=0x7f7490009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:14.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.694+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7488055090 con 0x7f74a015df20 2024-01-16T15:18:14.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:14.926+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone2463", "target": ["mon-mgr", ""]}) v1 -- 0x7f74a015c170 con 0x7f747806c280 2024-01-16T15:18:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.381+0000 7f749cc00640 1 -- 172.21.15.43:0/1820289502 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7490001ce0 con 0x7f747806c280 2024-01-16T15:18:17.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 msgr2=0x7f747806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 0x7f747806e740 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7490001270 tx=0x7f7490009040 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 msgr2=0x7f74a0153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f748800e820 tx=0x7f748800ece0 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 shutdown_connections 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f747806c280 0x7f747806e740 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74a0154590 0x7f74a009b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a015df20 0x7f74a0153f20 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 --2- 172.21.15.43:0/1820289502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7480005380 0x7f74a015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.384+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 >> 172.21.15.43:0/1820289502 conn(0x7f74a00cfd80 msgr2=0x7f74a0155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.385+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 shutdown_connections 2024-01-16T15:18:17.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:17.385+0000 7f74a5a00640 1 -- 172.21.15.43:0/1820289502 wait complete. 2024-01-16T15:18:19.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2461 2024-01-16T15:18:19.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.645+0000 7f3285200640 1 Processor -- start 2024-01-16T15:18:19.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.645+0000 7f3285200640 1 -- start start 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3280144770 0x7f328015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 0x7f328015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f328006c9d0 0x7f328006ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32800b43b0 con 0x7f328006c9d0 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328015f930 con 0x7f32800b35a0 2024-01-16T15:18:19.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f3285200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32800b3b10 con 0x7f3280144770 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f327fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f328006c9d0 0x7f328006ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f327fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f328006c9d0 0x7f328006ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46760/0 (socket says 172.21.15.43:46760) 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.647+0000 7f327fe00640 1 -- 172.21.15.43:0/3376816700 learned_addr learned my addr 172.21.15.43:0/3376816700 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327fe00640 1 -- 172.21.15.43:0/3376816700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3280144770 msgr2=0x7f328015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327f400640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3280144770 0x7f328015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327ea00640 1 --2- 172.21.15.43:0/3376816700 >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 0x7f328015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327fe00640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3280144770 0x7f328015eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327fe00640 1 -- 172.21.15.43:0/3376816700 >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 msgr2=0x7f328015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327fe00640 1 --2- 172.21.15.43:0/3376816700 >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 0x7f328015f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327fe00640 1 -- 172.21.15.43:0/3376816700 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800b3df0 con 0x7f328006c9d0 2024-01-16T15:18:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.648+0000 7f327ea00640 1 --2- 172.21.15.43:0/3376816700 >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 0x7f328015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.649+0000 7f327fe00640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f328006c9d0 0x7f328006ce60 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3278009870 tx=0x7f327802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b489496df0454a2c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.649+0000 7f327e000640 1 -- 172.21.15.43:0/3376816700 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278031640 con 0x7f328006c9d0 2024-01-16T15:18:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.649+0000 7f327e000640 1 -- 172.21.15.43:0/3376816700 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32780317a0 con 0x7f328006c9d0 2024-01-16T15:18:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.649+0000 7f327e000640 1 -- 172.21.15.43:0/3376816700 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278031b30 con 0x7f328006c9d0 2024-01-16T15:18:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 -- 172.21.15.43:0/3376816700 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f328006c9d0 msgr2=0x7f328006ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:19.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f328006c9d0 0x7f328006ce60 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3278009870 tx=0x7f327802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 -- 172.21.15.43:0/3376816700 shutdown_connections 2024-01-16T15:18:19.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f328006c9d0 0x7f328006ce60 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 --2- 172.21.15.43:0/3376816700 >> v2:172.21.15.69:3300/0 conn(0x7f32800b35a0 0x7f328015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 --2- 172.21.15.43:0/3376816700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3280144770 0x7f328015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 -- 172.21.15.43:0/3376816700 >> 172.21.15.43:0/3376816700 conn(0x7f32800cec10 msgr2=0x7f32800cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 -- 172.21.15.43:0/3376816700 shutdown_connections 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.650+0000 7f3285200640 1 -- 172.21.15.43:0/3376816700 wait complete. 2024-01-16T15:18:19.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.652+0000 7f3285200640 1 Processor -- start 2024-01-16T15:18:19.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.652+0000 7f3285200640 1 -- start start 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 0x7f328013b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280144770 0x7f3280070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280159530 con 0x7f32800b35a0 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328013b6f0 con 0x7f3280144770 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.653+0000 7f3285200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32801570d0 con 0x7f328006c9d0 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46774/0 (socket says 172.21.15.43:46774) 2024-01-16T15:18:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 -- 172.21.15.43:0/3333380566 learned_addr learned my addr 172.21.15.43:0/3333380566 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327fe00640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280144770 0x7f3280070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327f400640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 0x7f328013b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 -- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 msgr2=0x7f328013b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 0x7f328013b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 -- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280144770 msgr2=0x7f3280070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280144770 0x7f3280070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327ea00640 1 -- 172.21.15.43:0/3333380566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780094d0 con 0x7f32800b35a0 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.654+0000 7f327f400640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 0x7f328013b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:19.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f327ea00640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f326800c7f0 tx=0x7f326800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268007a30 con 0x7f32800b35a0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268007b90 con 0x7f32800b35a0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3280141670 con 0x7f32800b35a0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326800ce40 con 0x7f32800b35a0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.655+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3280141c30 con 0x7f32800b35a0 2024-01-16T15:18:19.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.656+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328004e840 con 0x7f32800b35a0 2024-01-16T15:18:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.657+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 39) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f32680113f0 con 0x7f32800b35a0 2024-01-16T15:18:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.658+0000 7f3277400640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 0x7f325806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.658+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3268088bf0 con 0x7f32800b35a0 2024-01-16T15:18:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.659+0000 7f327f400640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 0x7f325806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.659+0000 7f327f400640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 0x7f325806e740 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3270007e30 tx=0x7f3270007380 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:19.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.663+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32680551a0 con 0x7f32800b35a0 2024-01-16T15:18:19.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.897+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2461", "target": ["mon-mgr", ""]}) v1 -- 0x7f3280144b50 con 0x7f325806c280 2024-01-16T15:18:19.895 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:19.906+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:19.895 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:19.906+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:19.895 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:19.906+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:19.895 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:19.906+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:19.895 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:19.906+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.934+0000 7f3277400640 1 -- 172.21.15.43:0/3333380566 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3270007a40 con 0x7f325806c280 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:18:19.924 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:19.925 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:18:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 msgr2=0x7f325806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 0x7f325806e740 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3270007e30 tx=0x7f3270007380 comp rx=0 tx=0).stop 2024-01-16T15:18:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 msgr2=0x7f3280156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f326800c7f0 tx=0x7f326800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 shutdown_connections 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f325806c280 0x7f325806e740 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280144770 0x7f3280070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32800b35a0 0x7f3280156b90 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 --2- 172.21.15.43:0/3333380566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f328006c9d0 0x7f328013b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.937+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 >> 172.21.15.43:0/3333380566 conn(0x7f32800cec10 msgr2=0x7f328013dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.938+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 shutdown_connections 2024-01-16T15:18:19.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:19.938+0000 7f3285200640 1 -- 172.21.15.43:0/3333380566 wait complete. 2024-01-16T15:18:19.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2462 2024-01-16T15:18:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.171+0000 7f9e81800640 1 Processor -- start 2024-01-16T15:18:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.171+0000 7f9e81800640 1 -- start start 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.172+0000 7f9e81800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e7c09cb30 0x7f9e7c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.172+0000 7f9e81800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c15b730 con 0x7f9e7c15b350 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.172+0000 7f9e81800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0ac740 con 0x7f9e7c0adb60 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.172+0000 7f9e81800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c0ac460 con 0x7f9e7c09cb30 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.173+0000 7f9e7aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9e7c15b350 0x7f9e7c0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40600/0 (socket says 172.21.15.43:40600) 2024-01-16T15:18:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.173+0000 7f9e7aa00640 1 -- 172.21.15.43:0/2250448396 learned_addr learned my addr 172.21.15.43:0/2250448396 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.173+0000 7f9e7a000640 1 --2- 172.21.15.43:0/2250448396 >> v2:172.21.15.69:3300/0 conn(0x7f9e7c09cb30 0x7f9e7c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.173+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027925487 0 0) 0x7f9e6c001110 con 0x7f9e7c15b350 2024-01-16T15:18:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.173+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c003520 con 0x7f9e7c15b350 2024-01-16T15:18:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540414711 0 0) 0x7f9e5c003520 con 0x7f9e7c15b350 2024-01-16T15:18:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 >> v2:172.21.15.69:3300/0 conn(0x7f9e7c09cb30 msgr2=0x7f9e7c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 --2- 172.21.15.43:0/2250448396 >> v2:172.21.15.69:3300/0 conn(0x7f9e7c09cb30 0x7f9e7c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 >> v1:172.21.15.69:6789/0 conn(0x7f9e7c0adb60 legacy=0x7f9e7c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0b2a50 con 0x7f9e7c15b350 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9e6c001800 con 0x7f9e7c15b350 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 >> v1:172.21.15.43:6789/0 conn(0x7f9e7c15b350 legacy=0x7f9e7c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 0x7f9e5c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e5c0047c0 0x7f9e5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c005180 0x7f9e5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0ac460 con 0x7f9e5c0039c0 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c00c080 con 0x7f9e5c0047c0 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c00c240 con 0x7f9e5c005180 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.174+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 0x7f9e5c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7a000640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e5c0047c0 0x7f9e5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7b400640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c005180 0x7f9e5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7aa00640 1 -- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c005180 msgr2=0x7f9e5c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c005180 0x7f9e5c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7aa00640 1 -- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e5c0047c0 msgr2=0x7f9e5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e5c0047c0 0x7f9e5c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7b400640 1 -- 172.21.15.43:0/2250448396 reap_dead start 2024-01-16T15:18:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.175+0000 7f9e7aa00640 1 -- 172.21.15.43:0/2250448396 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c003520 con 0x7f9e5c0039c0 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.177+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 0x7f9e5c003de0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c007fc0 tx=0x7f9e5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0030cf5b089978 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.177+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c005ad0 con 0x7f9e5c0039c0 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.177+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c005c50 con 0x7f9e5c0039c0 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.177+0000 7f9e79600640 1 -- 172.21.15.43:0/2250448396 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c02c7c0 con 0x7f9e5c0039c0 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 -- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 msgr2=0x7f9e5c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 0x7f9e5c003de0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c007fc0 tx=0x7f9e5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 -- 172.21.15.43:0/2250448396 shutdown_connections 2024-01-16T15:18:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 --2- 172.21.15.43:0/2250448396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e5c0039c0 0x7f9e5c003de0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 -- 172.21.15.43:0/2250448396 >> 172.21.15.43:0/2250448396 conn(0x7f9e7c0b0ba0 msgr2=0x7f9e7c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 -- 172.21.15.43:0/2250448396 shutdown_connections 2024-01-16T15:18:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.178+0000 7f9e81800640 1 -- 172.21.15.43:0/2250448396 wait complete. 2024-01-16T15:18:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.180+0000 7f9e81800640 1 Processor -- start 2024-01-16T15:18:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.180+0000 7f9e81800640 1 -- start start 2024-01-16T15:18:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c0039c0 0x7f9e7c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 0x7f9e7c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c157a90 con 0x7f9e7c14cac0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c157d70 con 0x7f9e7c15e1e0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e81800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c158050 con 0x7f9e5c0039c0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e7b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e7b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46802/0 (socket says 172.21.15.43:46802) 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.181+0000 7f9e7b400640 1 -- 172.21.15.43:0/1785987176 learned_addr learned my addr 172.21.15.43:0/1785987176 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7a000640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 0x7f9e7c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c0039c0 0x7f9e7c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 -- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c0039c0 msgr2=0x7f9e7c157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c0039c0 0x7f9e7c157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 -- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 msgr2=0x7f9e7c0b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 0x7f9e7c0b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 -- 172.21.15.43:0/1785987176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c158330 con 0x7f9e7c14cac0 2024-01-16T15:18:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7a000640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 0x7f9e7c0b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e7b400640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0b1bf0 tx=0x7f9e7400f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740053a0 con 0x7f9e7c14cac0 2024-01-16T15:18:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e740035f0 con 0x7f9e7c14cac0 2024-01-16T15:18:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.182+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740039a0 con 0x7f9e7c14cac0 2024-01-16T15:18:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.183+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c158610 con 0x7f9e7c14cac0 2024-01-16T15:18:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.183+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c0b2440 con 0x7f9e7c14cac0 2024-01-16T15:18:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.184+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c158800 con 0x7f9e7c14cac0 2024-01-16T15:18:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.185+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 39) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f9e74003b00 con 0x7f9e7c14cac0 2024-01-16T15:18:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.186+0000 7f9e72a00640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 0x7f9e5406e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.186+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9e740884f0 con 0x7f9e7c14cac0 2024-01-16T15:18:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.186+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 0x7f9e5406e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.187+0000 7f9e7aa00640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 0x7f9e5406e740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c034000 tx=0x7f9e6c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.190+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e74054a10 con 0x7f9e7c14cac0 2024-01-16T15:18:20.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.429+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2462", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e7c06e720 con 0x7f9e5406c280 2024-01-16T15:18:20.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.452+0000 7f9e72a00640 1 -- 172.21.15.43:0/1785987176 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f9e6c001600 con 0x7f9e5406c280 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:18:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:18:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:20.443 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:18:20.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 msgr2=0x7f9e5406e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 0x7f9e5406e740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c034000 tx=0x7f9e6c009030 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 msgr2=0x7f9e7c14acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0b1bf0 tx=0x7f9e7400f960 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 shutdown_connections 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5406c280 0x7f9e5406e740 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e7c15e1e0 0x7f9e7c0b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e7c14cac0 0x7f9e7c14acb0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 --2- 172.21.15.43:0/1785987176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e5c0039c0 0x7f9e7c157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 >> 172.21.15.43:0/1785987176 conn(0x7f9e7c0b0ba0 msgr2=0x7f9e7c15a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 shutdown_connections 2024-01-16T15:18:20.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.455+0000 7f9e81800640 1 -- 172.21.15.43:0/1785987176 wait complete. 2024-01-16T15:18:20.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone2463 2024-01-16T15:18:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.695+0000 7f0ce9000640 1 Processor -- start 2024-01-16T15:18:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.695+0000 7f0ce9000640 1 -- start start 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce9000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0ce40b43e0 0x7f0ce4168090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce9000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40b2980 con 0x7f0ce40cfa30 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce9000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40cf220 con 0x7f0ce40d1fd0 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce9000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce40cf030 con 0x7f0ce40b43e0 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce3400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0ce40cfa30 0x7f0ce40cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40604/0 (socket says 172.21.15.43:40604) 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.697+0000 7f0ce3400640 1 -- 172.21.15.43:0/2639499605 learned_addr learned my addr 172.21.15.43:0/2639499605 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce3e00640 1 --2- 172.21.15.43:0/2639499605 >> v2:172.21.15.69:3300/0 conn(0x7f0ce40b43e0 0x7f0ce4168090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910952238 0 0) 0x7f0ce40b2980 con 0x7f0ce40cfa30 2024-01-16T15:18:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4003520 con 0x7f0ce40cfa30 2024-01-16T15:18:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce3e00640 1 -- 172.21.15.43:0/2639499605 >> v1:172.21.15.69:6789/0 conn(0x7f0ce40d1fd0 legacy=0x7f0ce40ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1053984526 0 0) 0x7f0cc4003520 con 0x7f0ce40cfa30 2024-01-16T15:18:20.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce3e00640 1 -- 172.21.15.43:0/2639499605 >> v1:172.21.15.43:6789/0 conn(0x7f0ce40cfa30 legacy=0x7f0ce40cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.698+0000 7f0ce3e00640 1 -- 172.21.15.43:0/2639499605 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40cf500 con 0x7f0ce40b43e0 2024-01-16T15:18:20.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.700+0000 7f0ce3e00640 1 --2- 172.21.15.43:0/2639499605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40b43e0 0x7f0ce4168090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009d90 tx=0x7f0cdc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d10dde9c86fa594 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.700+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4001670 con 0x7f0ce40b43e0 2024-01-16T15:18:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.700+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc02c940 con 0x7f0ce40b43e0 2024-01-16T15:18:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.700+0000 7f0ce2000640 1 -- 172.21.15.43:0/2639499605 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc02ccd0 con 0x7f0ce40b43e0 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 -- 172.21.15.43:0/2639499605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40b43e0 msgr2=0x7f0ce4168090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2639499605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40b43e0 0x7f0ce4168090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009d90 tx=0x7f0cdc02be10 comp rx=0 tx=0).stop 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 -- 172.21.15.43:0/2639499605 shutdown_connections 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2639499605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40b43e0 0x7f0ce4168090 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 -- 172.21.15.43:0/2639499605 >> 172.21.15.43:0/2639499605 conn(0x7f0ce40d3020 msgr2=0x7f0ce40d3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 -- 172.21.15.43:0/2639499605 shutdown_connections 2024-01-16T15:18:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.701+0000 7f0ce9000640 1 -- 172.21.15.43:0/2639499605 wait complete. 2024-01-16T15:18:20.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.703+0000 7f0ce9000640 1 Processor -- start 2024-01-16T15:18:20.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.703+0000 7f0ce9000640 1 -- start start 2024-01-16T15:18:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 0x7f0ce40cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40d1fd0 0x7f0ce40cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce40cc330 con 0x7f0ce40cfa30 2024-01-16T15:18:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce415a820 con 0x7f0ce40d1fd0 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce9000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce40cb720 con 0x7f0ce40b43e0 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46812/0 (socket says 172.21.15.43:46812) 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.704+0000 7f0ce2a00640 1 -- 172.21.15.43:0/2058314324 learned_addr learned my addr 172.21.15.43:0/2058314324 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce3e00640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40d1fd0 0x7f0ce40cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 -- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 msgr2=0x7f0ce40cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce3400640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 0x7f0ce40cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 0x7f0ce40cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 -- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40d1fd0 msgr2=0x7f0ce40cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40d1fd0 0x7f0ce40cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 -- 172.21.15.43:0/2058314324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc009a70 con 0x7f0ce40cfa30 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce3400640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 0x7f0ce40cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce2a00640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00c450 tx=0x7f0ccc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0052d0 con 0x7f0ce40cfa30 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc005430 con 0x7f0ce40cfa30 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc012870 con 0x7f0ce40cfa30 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.705+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce40cba00 con 0x7f0ce40cfa30 2024-01-16T15:18:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.706+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce40cbfc0 con 0x7f0ce40cfa30 2024-01-16T15:18:20.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.707+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce40d0a40 con 0x7f0ce40cfa30 2024-01-16T15:18:20.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.708+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 39) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f0ccc0129d0 con 0x7f0ce40cfa30 2024-01-16T15:18:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.709+0000 7f0cdb400640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 0x7f0cbc06e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.709+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0ccc088350 con 0x7f0ce40cfa30 2024-01-16T15:18:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.710+0000 7f0ce3400640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 0x7f0cbc06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.710+0000 7f0ce3400640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 0x7f0cbc06e740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4000f60 tx=0x7f0cd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.714+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ccc054b60 con 0x7f0ce40cfa30 2024-01-16T15:18:20.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:20.946+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone2463", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ce40b2eb0 con 0x7f0cbc06c280 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.927+0000 7f0cdb400640 1 -- 172.21.15.43:0/2058314324 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f0cd40058d0 con 0x7f0cbc06c280 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "canceled", 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: "failure": { 2024-01-16T15:18:21.918 INFO:teuthology.orchestra.run.smithi043.stdout: "errno": "4", 2024-01-16T15:18:21.918 INFO:teuthology.orchestra.run.smithi043.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:21.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:21.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:21.918 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 msgr2=0x7f0cbc06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 0x7f0cbc06e740 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4000f60 tx=0x7f0cd4009040 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 msgr2=0x7f0ce415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00c450 tx=0x7f0ccc00c910 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 shutdown_connections 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cbc06c280 0x7f0cbc06e740 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce40d1fd0 0x7f0ce40cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce40cfa30 0x7f0ce415a2e0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 --2- 172.21.15.43:0/2058314324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ce40b43e0 0x7f0ce40cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 >> 172.21.15.43:0/2058314324 conn(0x7f0ce40d3020 msgr2=0x7f0ce40d3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 shutdown_connections 2024-01-16T15:18:21.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:21.931+0000 7f0ce9000640 1 -- 172.21.15.43:0/2058314324 wait complete. 2024-01-16T15:18:21.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:18:22.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.178+0000 7f151fe00640 1 Processor -- start 2024-01-16T15:18:22.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.178+0000 7f151fe00640 1 -- start start 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f152013dfd0 0x7f152015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 0x7f15200cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15200ce7c0 con 0x7f152006c130 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151fe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200cfa40 con 0x7f152013dfd0 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1520140900 con 0x7f15200cf050 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f152013dfd0 0x7f152015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f152013dfd0 0x7f152015be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46820/0 (socket says 172.21.15.43:46820) 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151ea00640 1 -- 172.21.15.43:0/1456801467 learned_addr learned my addr 172.21.15.43:0/1456801467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:22.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151ea00640 1 -- 172.21.15.43:0/1456801467 >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 msgr2=0x7f15200cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.180+0000 7f151e000640 1 --2- 172.21.15.43:0/1456801467 >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 0x7f15200cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.181+0000 7f151ea00640 1 --2- 172.21.15.43:0/1456801467 >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 0x7f15200cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.181+0000 7f151ea00640 1 -- 172.21.15.43:0/1456801467 >> v1:172.21.15.43:6789/0 conn(0x7f152006c130 legacy=0x7f152006c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.181+0000 7f151ea00640 1 -- 172.21.15.43:0/1456801467 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152015fa70 con 0x7f152013dfd0 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.181+0000 7f151e000640 1 --2- 172.21.15.43:0/1456801467 >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 0x7f15200cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.181+0000 7f151d600640 1 -- 172.21.15.43:0/1456801467 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178683562 0 0) 0x7f15200ce7c0 con 0x7f152006c130 2024-01-16T15:18:22.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.182+0000 7f151ea00640 1 --2- 172.21.15.43:0/1456801467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152015be70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f152006e070 tx=0x7f151002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b5a0b421bf4839 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.182+0000 7f151d600640 1 -- 172.21.15.43:0/1456801467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510030870 con 0x7f152013dfd0 2024-01-16T15:18:22.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.182+0000 7f151d600640 1 -- 172.21.15.43:0/1456801467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15100309d0 con 0x7f152013dfd0 2024-01-16T15:18:22.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.182+0000 7f151d600640 1 -- 172.21.15.43:0/1456801467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510030d60 con 0x7f152013dfd0 2024-01-16T15:18:22.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 -- 172.21.15.43:0/1456801467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 msgr2=0x7f152015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 --2- 172.21.15.43:0/1456801467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152015be70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f152006e070 tx=0x7f151002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 -- 172.21.15.43:0/1456801467 shutdown_connections 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 --2- 172.21.15.43:0/1456801467 >> v2:172.21.15.69:3300/0 conn(0x7f15200cf050 0x7f15200cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 --2- 172.21.15.43:0/1456801467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152015be70 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 -- 172.21.15.43:0/1456801467 >> 172.21.15.43:0/1456801467 conn(0x7f152006a860 msgr2=0x7f152006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 -- 172.21.15.43:0/1456801467 shutdown_connections 2024-01-16T15:18:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.183+0000 7f151fe00640 1 -- 172.21.15.43:0/1456801467 wait complete. 2024-01-16T15:18:22.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.185+0000 7f151fe00640 1 Processor -- start 2024-01-16T15:18:22.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.185+0000 7f151fe00640 1 -- start start 2024-01-16T15:18:22.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 0x7f152015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15200cf050 0x7f15201f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15201f3290 con 0x7f152013dfd0 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15201f3570 con 0x7f15200cf050 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15201f3850 con 0x7f152006c130 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46822/0 (socket says 172.21.15.43:46822) 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.186+0000 7f151f400640 1 -- 172.21.15.43:0/1100912300 learned_addr learned my addr 172.21.15.43:0/1100912300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151e000640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15200cf050 0x7f15201f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151ea00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 0x7f152015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 -- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 msgr2=0x7f152015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 0x7f152015d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 -- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15200cf050 msgr2=0x7f15201f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15200cf050 0x7f15201f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 -- 172.21.15.43:0/1100912300 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100094d0 con 0x7f152013dfd0 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151ea00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 0x7f152015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f151f400640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1518001560 tx=0x7f151800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.187+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518003da0 con 0x7f152013dfd0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.188+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518005070 con 0x7f152013dfd0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.188+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152015bc40 con 0x7f152013dfd0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.188+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518005420 con 0x7f152013dfd0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.188+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15201f2f30 con 0x7f152013dfd0 2024-01-16T15:18:22.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.189+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152015c160 con 0x7f152013dfd0 2024-01-16T15:18:22.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.190+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f1518005580 con 0x7f152013dfd0 2024-01-16T15:18:22.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.191+0000 7f1517400640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 0x7f14f806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.191+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1518088310 con 0x7f152013dfd0 2024-01-16T15:18:22.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.192+0000 7f151ea00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 0x7f14f806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.192+0000 7f151ea00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 0x7f14f806e7d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1510036000 tx=0x7f151003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.196+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f152015c160 con 0x7f152013dfd0 2024-01-16T15:18:22.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.465+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f152015c160 con 0x7f152013dfd0 2024-01-16T15:18:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.466+0000 7f1517400640 1 -- 172.21.15.43:0/1100912300 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1855 (secure 0 0 0) 0x7f152015c160 con 0x7f152013dfd0 2024-01-16T15:18:22.455 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:22.455 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:18:22.455 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:18:22.455 INFO:teuthology.orchestra.run.smithi043.stdout: "health": { 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "HEALTH_OK", 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "checks": {}, 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "mutes": [] 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 4, 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_names": [ 2024-01-16T15:18:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: "a", 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "b", 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "c" 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1154, 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "reef", 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "num_mons": 3 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap": { 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 101, 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "num_osds": 8, 2024-01-16T15:18:22.457 INFO:teuthology.orchestra.run.smithi043.stdout: "num_up_osds": 8, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_up_since": 1705417159, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "num_in_osds": 8, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_in_since": 1705417151, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "num_remapped_pgs": 0 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap": { 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "pgs_by_state": [ 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "state_name": "active+clean", 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "count": 97 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pgs": 97, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "num_pools": 3, 2024-01-16T15:18:22.458 INFO:teuthology.orchestra.run.smithi043.stdout: "num_objects": 800, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "data_bytes": 3222407863, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_used": 7062278144, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_avail": 766031835136, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bytes_total": 773094113280, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "read_bytes_sec": 767, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "write_bytes_sec": 11941, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "read_op_per_sec": 0, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "write_op_per_sec": 3 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "fsmap": { 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 67, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 7, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1, 2024-01-16T15:18:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: "by_rank": [ 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "filesystem_id": 7, 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "up:active", 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 8568 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "up:standby": 3 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "mgrmap": { 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-01-16T15:18:22.460 INFO:teuthology.orchestra.run.smithi043.stdout: "num_standbys": 1, 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "restful" 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "servicemap": { 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19, 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-01-16T15:16:03.223164+0000", 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "services": {} 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout: "progress_events": {} 2024-01-16T15:18:22.461 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:18:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 msgr2=0x7f14f806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 0x7f14f806e7d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1510036000 tx=0x7f151003b040 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 msgr2=0x7f152013e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1518001560 tx=0x7f151800cc70 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 shutdown_connections 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14f806c310 0x7f14f806e7d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15200cf050 0x7f15201f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f152013dfd0 0x7f152013e620 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 --2- 172.21.15.43:0/1100912300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f152006c130 0x7f152015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.468+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 >> 172.21.15.43:0/1100912300 conn(0x7f152006a860 msgr2=0x7f152015e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.469+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 shutdown_connections 2024-01-16T15:18:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.469+0000 7f151fe00640 1 -- 172.21.15.43:0/1100912300 wait complete. 2024-01-16T15:18:22.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:22.478 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone2461/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3) 2024-01-16T15:18:22.538 INFO:teuthology.orchestra.run.smithi043.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone2461/b596fcb0-ebd6-4c5c-b149-efa0cb243fe3': No such file or directory 2024-01-16T15:18:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:18:22.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:18:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.772+0000 7fc430c00640 1 Processor -- start 2024-01-16T15:18:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.772+0000 7fc430c00640 1 -- start start 2024-01-16T15:18:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc430c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc42c1449d0 0x7fc42c0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc430c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c06c9e0 0x7fc42c145360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc430c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42c070a70 con 0x7fc42c070610 2024-01-16T15:18:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc430c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c144ed0 con 0x7fc42c1449d0 2024-01-16T15:18:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc430c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c06d040 con 0x7fc42c06c9e0 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc42b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc42c1449d0 0x7fc42c0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc42be00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c06c9e0 0x7fc42c145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc42b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc42c1449d0 0x7fc42c0700d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:38278/0 (socket says 172.21.15.43:38278) 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.774+0000 7fc42b400640 1 -- 172.21.15.43:0/480902472 learned_addr learned my addr 172.21.15.43:0/480902472 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.775+0000 7fc42b400640 1 -- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c06c9e0 msgr2=0x7fc42c145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.775+0000 7fc42b400640 1 --2- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c06c9e0 0x7fc42c145360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.775+0000 7fc42b400640 1 -- 172.21.15.43:0/480902472 >> v1:172.21.15.69:6789/0 conn(0x7fc42c070610 legacy=0x7fc42c06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.775+0000 7fc42b400640 1 -- 172.21.15.43:0/480902472 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c06d320 con 0x7fc42c1449d0 2024-01-16T15:18:22.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.777+0000 7fc42b400640 1 --2- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c1449d0 0x7fc42c0700d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009870 tx=0x7fc41c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9225bc328e389f8d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc42a000640 1 -- 172.21.15.43:0/480902472 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c030840 con 0x7fc42c1449d0 2024-01-16T15:18:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc42a000640 1 -- 172.21.15.43:0/480902472 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c0309a0 con 0x7fc42c1449d0 2024-01-16T15:18:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc42a000640 1 -- 172.21.15.43:0/480902472 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c030d30 con 0x7fc42c1449d0 2024-01-16T15:18:22.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 -- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c1449d0 msgr2=0x7fc42c0700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 --2- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c1449d0 0x7fc42c0700d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009870 tx=0x7fc41c02b980 comp rx=0 tx=0).stop 2024-01-16T15:18:22.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 -- 172.21.15.43:0/480902472 shutdown_connections 2024-01-16T15:18:22.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 --2- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c06c9e0 0x7fc42c145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 --2- 172.21.15.43:0/480902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c1449d0 0x7fc42c0700d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.778+0000 7fc430c00640 1 -- 172.21.15.43:0/480902472 >> 172.21.15.43:0/480902472 conn(0x7fc42c0b0840 msgr2=0x7fc42c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:22.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.779+0000 7fc430c00640 1 -- 172.21.15.43:0/480902472 shutdown_connections 2024-01-16T15:18:22.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.779+0000 7fc430c00640 1 -- 172.21.15.43:0/480902472 wait complete. 2024-01-16T15:18:22.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.780+0000 7fc430c00640 1 Processor -- start 2024-01-16T15:18:22.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.780+0000 7fc430c00640 1 -- start start 2024-01-16T15:18:22.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.781+0000 7fc430c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c06c9e0 0x7fc42c143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.781+0000 7fc430c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c070610 0x7fc42c153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc430c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc430c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c15e070 con 0x7fc42c1449d0 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc430c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c15d420 con 0x7fc42c06c9e0 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc430c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c15da20 con 0x7fc42c070610 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc42be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc42be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46826/0 (socket says 172.21.15.43:46826) 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc42be00640 1 -- 172.21.15.43:0/2017251283 learned_addr learned my addr 172.21.15.43:0/2017251283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc42b400640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c06c9e0 0x7fc42c143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.782+0000 7fc42aa00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c070610 0x7fc42c153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 -- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c070610 msgr2=0x7fc42c153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c070610 0x7fc42c153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 -- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c06c9e0 msgr2=0x7fc42c143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c06c9e0 0x7fc42c143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 -- 172.21.15.43:0/2017251283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0094d0 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc42be00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc42400efc0 tx=0x7fc42400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424014480 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.783+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42400f040 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.784+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c15dd00 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.784+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424014820 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.784+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c15d700 con 0x7fc42c1449d0 2024-01-16T15:18:22.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.785+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42c04e840 con 0x7fc42c1449d0 2024-01-16T15:18:22.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.786+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fc424014980 con 0x7fc42c1449d0 2024-01-16T15:18:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.787+0000 7fc423400640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 0x7fc40406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.787+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc4240897d0 con 0x7fc42c1449d0 2024-01-16T15:18:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.787+0000 7fc42b400640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 0x7fc40406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.788+0000 7fc42b400640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 0x7fc40406e7d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc41c036000 tx=0x7fc41c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:22.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:22.791+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc424055e20 con 0x7fc42c1449d0 2024-01-16T15:18:23.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.011+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc42c066ec0 con 0x7fc40406c310 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.013+0000 7fc423400640 1 -- 172.21.15.43:0/2017251283 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc42c066ec0 con 0x7fc40406c310 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:23.003 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 msgr2=0x7fc40406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 0x7fc40406e7d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc41c036000 tx=0x7fc41c008030 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 msgr2=0x7fc42c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc42400efc0 tx=0x7fc42400c490 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 shutdown_connections 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc40406c310 0x7fc40406e7d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c1449d0 0x7fc42c0b2210 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc42c070610 0x7fc42c153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 --2- 172.21.15.43:0/2017251283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc42c06c9e0 0x7fc42c143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 >> 172.21.15.43:0/2017251283 conn(0x7fc42c0b0840 msgr2=0x7fc42c161150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 shutdown_connections 2024-01-16T15:18:23.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.016+0000 7fc430c00640 1 -- 172.21.15.43:0/2017251283 wait complete. 2024-01-16T15:18:23.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:18:23.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.261+0000 7f6a1ea00640 1 Processor -- start 2024-01-16T15:18:23.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.261+0000 7f6a1ea00640 1 -- start start 2024-01-16T15:18:23.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.263+0000 7f6a1ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a18160b10 0x7f6a1806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.263+0000 7f6a1ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f6a1806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.263+0000 7f6a1ea00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a18071780 con 0x7f6a1806c900 2024-01-16T15:18:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.263+0000 7f6a1ea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a180b2320 con 0x7f6a18160b10 2024-01-16T15:18:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.263+0000 7f6a1ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1806cd90 con 0x7f6a18072280 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a1ca00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6a1806c900 0x7f6a18160fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46714/0 (socket says 172.21.15.43:46714) 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a1ca00640 1 -- 172.21.15.43:0/772889927 learned_addr learned my addr 172.21.15.43:0/772889927 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a17e00640 1 --2- 172.21.15.43:0/772889927 >> v2:172.21.15.69:3300/0 conn(0x7f6a18160b10 0x7f6a1806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f6a1806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295158726 0 0) 0x7f6a18071780 con 0x7f6a1806c900 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.264+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8003520 con 0x7f6a1806c900 2024-01-16T15:18:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107217589 0 0) 0x7f69f8003520 con 0x7f6a1806c900 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 msgr2=0x7f6a1806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f6a1806c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 >> v2:172.21.15.69:3300/0 conn(0x7f6a18160b10 msgr2=0x7f6a1806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 --2- 172.21.15.43:0/772889927 >> v2:172.21.15.69:3300/0 conn(0x7f6a18160b10 0x7f6a1806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180711c0 con 0x7f6a1806c900 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6a100019f0 con 0x7f6a1806c900 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 >> v1:172.21.15.43:6789/0 conn(0x7f6a1806c900 legacy=0x7f6a18160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69f8003970 0x7f69f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f69f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f69f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180b2320 con 0x7f69f80047c0 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1806cd90 con 0x7f69f8003970 2024-01-16T15:18:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f800b9d0 con 0x7f6a18072280 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f69f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a0fe00640 1 -- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 msgr2=0x7f69f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a1ca00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f69f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.265+0000 7f6a17e00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69f8003970 0x7f69f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f69f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a0fe00640 1 -- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69f8003970 msgr2=0x7f69f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69f8003970 0x7f69f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a0fe00640 1 -- 172.21.15.43:0/772889927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8003520 con 0x7f69f80047c0 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a1ca00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18072280 0x7f69f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.266+0000 7f6a17e00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f69f8003970 0x7f69f8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:23.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.267+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f69f8004c40 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009af0 tx=0x7f69f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf8ecca4b22e217 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.267+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0002c6b0 con 0x7f69f80047c0 2024-01-16T15:18:23.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.267+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0002c810 con 0x7f69f80047c0 2024-01-16T15:18:23.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.267+0000 7f6a17400640 1 -- 172.21.15.43:0/772889927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0002cba0 con 0x7f69f80047c0 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 -- 172.21.15.43:0/772889927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 msgr2=0x7f69f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/772889927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f69f8004c40 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009af0 tx=0x7f69f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ca00640 1 -- 172.21.15.43:0/772889927 reap_dead start 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 -- 172.21.15.43:0/772889927 shutdown_connections 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 -- 172.21.15.43:0/772889927 >> 172.21.15.43:0/772889927 conn(0x7f6a180b0ba0 msgr2=0x7f6a180b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 -- 172.21.15.43:0/772889927 shutdown_connections 2024-01-16T15:18:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.268+0000 7f6a1ea00640 1 -- 172.21.15.43:0/772889927 wait complete. 2024-01-16T15:18:23.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.269+0000 7f6a1ea00640 1 Processor -- start 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.270+0000 7f6a1ea00640 1 -- start start 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.270+0000 7f6a1ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a1ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18150660 0x7f6a180ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a1ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a18150ae0 0x7f6a180ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a1ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180ccf20 con 0x7f69f80047c0 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a1ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180cd130 con 0x7f6a18150ae0 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a1ea00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180cd410 con 0x7f6a18150660 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41272/0 (socket says 172.21.15.43:41272) 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 -- 172.21.15.43:0/3085027772 learned_addr learned my addr 172.21.15.43:0/3085027772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 -- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18150660 msgr2=0x7f6a180ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18150660 0x7f6a180ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 -- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a18150ae0 msgr2=0x7f6a180ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a18150ae0 0x7f6a180ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.271+0000 7f6a17e00640 1 -- 172.21.15.43:0/3085027772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180cd6f0 con 0x7f69f80047c0 2024-01-16T15:18:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a17e00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f6a080109d0 tx=0x7f6a08010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0800ec20 con 0x7f69f80047c0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0800ed80 con 0x7f69f80047c0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0801c620 con 0x7f69f80047c0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a180cd9d0 con 0x7f69f80047c0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.272+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a180cdef0 con 0x7f69f80047c0 2024-01-16T15:18:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.273+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1809d200 con 0x7f69f80047c0 2024-01-16T15:18:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.275+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f6a0801c780 con 0x7f69f80047c0 2024-01-16T15:18:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.275+0000 7f6a14c00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 0x7f69f006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.276+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6a08089140 con 0x7f69f80047c0 2024-01-16T15:18:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.276+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 0x7f69f006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.277+0000 7f6a0fe00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 0x7f69f006e7d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6a00036000 tx=0x7f6a0002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.280+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a08055790 con 0x7f69f80047c0 2024-01-16T15:18:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.507+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a180b1f40 con 0x7f69f006c310 2024-01-16T15:18:23.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.523+0000 7f6a14c00640 1 -- 172.21.15.43:0/3085027772 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+182 (secure 0 0 0) 0x7f6a0002c980 con 0x7f69f006c310 2024-01-16T15:18:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone2461" 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone2462" 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone2463" 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:23.513 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:18:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 msgr2=0x7f69f006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 0x7f69f006e7d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6a00036000 tx=0x7f6a0002e000 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 msgr2=0x7f6a1814c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f6a080109d0 tx=0x7f6a08010e90 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 shutdown_connections 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f69f006c310 0x7f69f006e7d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a18150ae0 0x7f6a180ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a18150660 0x7f6a180ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 --2- 172.21.15.43:0/3085027772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f80047c0 0x7f6a1814c8f0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 >> 172.21.15.43:0/3085027772 conn(0x7f6a180b0ba0 msgr2=0x7f6a1815fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 shutdown_connections 2024-01-16T15:18:23.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.526+0000 7f6a1ea00640 1 -- 172.21.15.43:0/3085027772 wait complete. 2024-01-16T15:18:23.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:18:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.773+0000 7f1fdbe00640 1 Processor -- start 2024-01-16T15:18:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.773+0000 7f1fdbe00640 1 -- start start 2024-01-16T15:18:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.774+0000 7f1fdbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1fdc160ab0 0x7f1fdc0d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.774+0000 7f1fdbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1fdc0ce6f0 0x7f1fdc0cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.774+0000 7f1fdbe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc0cf190 con 0x7f1fdc0cfae0 2024-01-16T15:18:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.774+0000 7f1fdbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc0cff50 con 0x7f1fdc160ab0 2024-01-16T15:18:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.774+0000 7f1fdbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc0b2e60 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdaa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1fdc160ab0 0x7f1fdc0d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdb400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1fdc0cfae0 0x7f1fdc0d0f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46730/0 (socket says 172.21.15.43:46730) 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdb400640 1 -- 172.21.15.43:0/3283188877 learned_addr learned my addr 172.21.15.43:0/3283188877 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fda000640 1 --2- 172.21.15.43:0/3283188877 >> v2:172.21.15.69:3300/0 conn(0x7f1fdc0ce6f0 0x7f1fdc0cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3283188877 >> v2:172.21.15.69:3300/0 conn(0x7f1fdc0ce6f0 msgr2=0x7f1fdc0cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdaa00640 1 --2- 172.21.15.43:0/3283188877 >> v2:172.21.15.69:3300/0 conn(0x7f1fdc0ce6f0 0x7f1fdc0cf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3283188877 >> v1:172.21.15.43:6789/0 conn(0x7f1fdc0cfae0 legacy=0x7f1fdc0d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.775+0000 7f1fd9600640 1 -- 172.21.15.43:0/3283188877 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667083973 0 0) 0x7f1fdc0cf190 con 0x7f1fdc0cfae0 2024-01-16T15:18:23.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.776+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3283188877 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0cebd0 con 0x7f1fdc160ab0 2024-01-16T15:18:23.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.777+0000 7f1fdaa00640 1 --2- 172.21.15.43:0/3283188877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc0d3b80 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc08c9a0 tx=0x7f1fcc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6ab19b05bd195c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.777+0000 7f1fd9600640 1 -- 172.21.15.43:0/3283188877 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc02e070 con 0x7f1fdc160ab0 2024-01-16T15:18:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.777+0000 7f1fd9600640 1 -- 172.21.15.43:0/3283188877 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc0315a0 con 0x7f1fdc160ab0 2024-01-16T15:18:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.777+0000 7f1fd9600640 1 -- 172.21.15.43:0/3283188877 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc031930 con 0x7f1fdc160ab0 2024-01-16T15:18:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3283188877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc160ab0 msgr2=0x7f1fdc0d3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3283188877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc0d3b80 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc08c9a0 tx=0x7f1fcc02b840 comp rx=0 tx=0).stop 2024-01-16T15:18:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3283188877 shutdown_connections 2024-01-16T15:18:23.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3283188877 >> v2:172.21.15.69:3300/0 conn(0x7f1fdc0ce6f0 0x7f1fdc0cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3283188877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc0d3b80 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3283188877 >> 172.21.15.43:0/3283188877 conn(0x7f1fdc0d3060 msgr2=0x7f1fdc0d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:23.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3283188877 shutdown_connections 2024-01-16T15:18:23.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.778+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3283188877 wait complete. 2024-01-16T15:18:23.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.780+0000 7f1fdbe00640 1 Processor -- start 2024-01-16T15:18:23.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.780+0000 7f1fdbe00640 1 -- start start 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 0x7f1fdc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc14c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0cade0 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc1f6bf0 con 0x7f1fdc160ab0 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0ca1d0 con 0x7f1fdc0cfae0 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41282/0 (socket says 172.21.15.43:41282) 2024-01-16T15:18:23.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.781+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3696943935 learned_addr learned my addr 172.21.15.43:0/3696943935 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fda000640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 0x7f1fdc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdb400640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc14c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 msgr2=0x7f1fdc0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 0x7f1fdc0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 msgr2=0x7f1fdc14c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc14c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 -- 172.21.15.43:0/3696943935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc009400 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdb400640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc14c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fda000640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 0x7f1fdc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdaa00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009890 tx=0x7f1fcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc032840 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc0329a0 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc032d50 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.782+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc0ca4b0 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.783+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc0caa70 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.784+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fdc04e840 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.785+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f1fcc03b3f0 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.786+0000 7f1fd2a00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 0x7f1fb406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:23.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.786+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1fcc0aa650 con 0x7f1fdc0ce6f0 2024-01-16T15:18:23.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.787+0000 7f1fda000640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 0x7f1fb406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:23.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.787+0000 7f1fda000640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 0x7f1fb406e7d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4006fd0 tx=0x7f1fc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:23.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:23.791+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fcc076f70 con 0x7f1fdc0ce6f0 2024-01-16T15:18:24.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:24.019+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fdc160e90 con 0x7f1fb406c310 2024-01-16T15:18:26.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.930+0000 7f1fd2a00640 1 -- 172.21.15.43:0/3696943935 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1fc4009530 con 0x7f1fb406c310 2024-01-16T15:18:26.920 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:26.920 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:18:26.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:26.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:18:26.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:26.921 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:18:26.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 msgr2=0x7f1fb406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:26.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 0x7f1fb406e7d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4006fd0 tx=0x7f1fc4008040 comp rx=0 tx=0).stop 2024-01-16T15:18:26.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 msgr2=0x7f1fdc1f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:26.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009890 tx=0x7f1fcc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:26.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 shutdown_connections 2024-01-16T15:18:26.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fb406c310 0x7f1fb406e7d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fdc160ab0 0x7f1fdc14c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fdc0cfae0 0x7f1fdc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 --2- 172.21.15.43:0/3696943935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fdc0ce6f0 0x7f1fdc1f66b0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 >> 172.21.15.43:0/3696943935 conn(0x7f1fdc0d3060 msgr2=0x7f1fdc15e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.934+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 shutdown_connections 2024-01-16T15:18:26.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:26.935+0000 7f1fdbe00640 1 -- 172.21.15.43:0/3696943935 wait complete. 2024-01-16T15:18:26.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:18:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.184+0000 7f34b7000640 1 Processor -- start 2024-01-16T15:18:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.184+0000 7f34b7000640 1 -- start start 2024-01-16T15:18:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b7000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00d1d90 0x7f34b00d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b7000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34b00cdbb0 0x7f34b00b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b7000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00cef30 con 0x7f34b015d340 2024-01-16T15:18:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b7000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b00d11a0 con 0x7f34b00cdbb0 2024-01-16T15:18:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b7000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00ce320 con 0x7f34b00d1d90 2024-01-16T15:18:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b5200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00d1d90 0x7f34b00d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b6600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34b00cdbb0 0x7f34b00b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.186+0000 7f34b5c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f34b015d340 0x7f34b00d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:42566/0 (socket says 172.21.15.43:42566) 2024-01-16T15:18:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.187+0000 7f34b5c00640 1 -- 172.21.15.43:0/1564269040 learned_addr learned my addr 172.21.15.43:0/1564269040 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:27.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.187+0000 7f34b6600640 1 -- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00d1d90 msgr2=0x7f34b00d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:27.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.187+0000 7f34b6600640 1 --2- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00d1d90 0x7f34b00d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.187+0000 7f34b6600640 1 -- 172.21.15.43:0/1564269040 >> v1:172.21.15.69:6789/0 conn(0x7f34b015d340 legacy=0x7f34b00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.187+0000 7f34b6600640 1 -- 172.21.15.43:0/1564269040 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00ce600 con 0x7f34b00cdbb0 2024-01-16T15:18:27.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.189+0000 7f34b6600640 1 --2- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b00cdbb0 0x7f34b00b3dc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097a0 tx=0x7f34a802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd1666ef3f2970f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.189+0000 7f34afe00640 1 -- 172.21.15.43:0/1564269040 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8031830 con 0x7f34b00cdbb0 2024-01-16T15:18:27.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.189+0000 7f34afe00640 1 -- 172.21.15.43:0/1564269040 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a8031990 con 0x7f34b00cdbb0 2024-01-16T15:18:27.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.189+0000 7f34afe00640 1 -- 172.21.15.43:0/1564269040 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8031d20 con 0x7f34b00cdbb0 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 -- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b00cdbb0 msgr2=0x7f34b00b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 --2- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b00cdbb0 0x7f34b00b3dc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097a0 tx=0x7f34a802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 -- 172.21.15.43:0/1564269040 shutdown_connections 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 --2- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b00cdbb0 0x7f34b00b3dc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 --2- 172.21.15.43:0/1564269040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00d1d90 0x7f34b00d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 -- 172.21.15.43:0/1564269040 >> 172.21.15.43:0/1564269040 conn(0x7f34b009c370 msgr2=0x7f34b00d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 -- 172.21.15.43:0/1564269040 shutdown_connections 2024-01-16T15:18:27.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.190+0000 7f34b7000640 1 -- 172.21.15.43:0/1564269040 wait complete. 2024-01-16T15:18:27.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.192+0000 7f34b7000640 1 Processor -- start 2024-01-16T15:18:27.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.193+0000 7f34b7000640 1 -- start start 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00cdbb0 0x7f34b0159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b015d340 0x7f34b00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b015ffe0 con 0x7f34b00d1d90 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b01602c0 con 0x7f34b015d340 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b7000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00b7080 con 0x7f34b00cdbb0 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41284/0 (socket says 172.21.15.43:41284) 2024-01-16T15:18:27.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.194+0000 7f34b5200640 1 -- 172.21.15.43:0/603046422 learned_addr learned my addr 172.21.15.43:0/603046422 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:27.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b6600640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b015d340 0x7f34b00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5c00640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00cdbb0 0x7f34b0159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 -- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00cdbb0 msgr2=0x7f34b0159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00cdbb0 0x7f34b0159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 -- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b015d340 msgr2=0x7f34b00b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b015d340 0x7f34b00b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 -- 172.21.15.43:0/603046422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8009400 con 0x7f34b00d1d90 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34b5200640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f349800e870 tx=0x7f349800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980094f0 con 0x7f34b00d1d90 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349800c940 con 0x7f34b00d1d90 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.195+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800ccf0 con 0x7f34b00d1d90 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.196+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b00b7360 con 0x7f34b00d1d90 2024-01-16T15:18:27.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.196+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b00b3430 con 0x7f34b00d1d90 2024-01-16T15:18:27.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.197+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b00b36b0 con 0x7f34b00d1d90 2024-01-16T15:18:27.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.198+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f3498013470 con 0x7f34b00d1d90 2024-01-16T15:18:27.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.200+0000 7f34ad600640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 0x7f348806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.200+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3498088860 con 0x7f34b00d1d90 2024-01-16T15:18:27.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.200+0000 7f34b5c00640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 0x7f348806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.201+0000 7f34b5c00640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 0x7f348806e7d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f34a0005c00 tx=0x7f34a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.204+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3498055120 con 0x7f34b00d1d90 2024-01-16T15:18:27.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.434+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34b00ce020 con 0x7f348806c310 2024-01-16T15:18:27.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.497+0000 7f34ad600640 1 -- 172.21.15.43:0/603046422 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34a0001df0 con 0x7f348806c310 2024-01-16T15:18:27.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 msgr2=0x7f348806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 0x7f348806e7d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f34a0005c00 tx=0x7f34a0009040 comp rx=0 tx=0).stop 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 msgr2=0x7f34b00b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f349800e870 tx=0x7f349800ed30 comp rx=0 tx=0).stop 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 shutdown_connections 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f348806c310 0x7f348806e7d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b015d340 0x7f34b00b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b00d1d90 0x7f34b00b2ef0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 --2- 172.21.15.43:0/603046422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b00cdbb0 0x7f34b0159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 >> 172.21.15.43:0/603046422 conn(0x7f34b009c370 msgr2=0x7f34b00b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:27.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 shutdown_connections 2024-01-16T15:18:27.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.502+0000 7f34b7000640 1 -- 172.21.15.43:0/603046422 wait complete. 2024-01-16T15:18:27.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:18:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.749+0000 7f095fa00640 1 Processor -- start 2024-01-16T15:18:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.750+0000 7f095fa00640 1 -- start start 2024-01-16T15:18:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.751+0000 7f095fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f095806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.751+0000 7f095fa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f095806ec60 0x7f09580cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.751+0000 7f095fa00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958142d50 con 0x7f09580cd150 2024-01-16T15:18:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.751+0000 7f095fa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580cd530 con 0x7f095806ec60 2024-01-16T15:18:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.751+0000 7f095fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095806f430 con 0x7f0958070470 2024-01-16T15:18:27.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f095d000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f09580cd150 0x7f095806fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46746/0 (socket says 172.21.15.43:46746) 2024-01-16T15:18:27.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f095d000640 1 -- 172.21.15.43:0/1436895247 learned_addr learned my addr 172.21.15.43:0/1436895247 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f095da00640 1 --2- 172.21.15.43:0/1436895247 >> v2:172.21.15.69:3300/0 conn(0x7f095806ec60 0x7f09580cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f0957e00640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f095806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281609106 0 0) 0x7f0958142d50 con 0x7f09580cd150 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.752+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0938003520 con 0x7f09580cd150 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420650223 0 0) 0x7f0938003520 con 0x7f09580cd150 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 msgr2=0x7f095806e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f095806e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 >> v2:172.21.15.69:3300/0 conn(0x7f095806ec60 msgr2=0x7f09580cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:27.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 --2- 172.21.15.43:0/1436895247 >> v2:172.21.15.69:3300/0 conn(0x7f095806ec60 0x7f09580cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958142fa0 con 0x7f09580cd150 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0948001670 con 0x7f09580cd150 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 >> v1:172.21.15.43:6789/0 conn(0x7f09580cd150 legacy=0x7f095806fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0938003970 0x7f0938003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09380047c0 0x7f0938004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f0938005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580cd530 con 0x7f0938003970 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095806f430 con 0x7f09380047c0 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093800b980 con 0x7f0958070470 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.753+0000 7f095d000640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0938003970 0x7f0938003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095da00640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f0938005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f0957e00640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09380047c0 0x7f0938004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095d000640 1 -- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 msgr2=0x7f0938005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095d000640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0958070470 0x7f0938005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095d000640 1 -- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09380047c0 msgr2=0x7f0938004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095d000640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09380047c0 0x7f0938004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.754+0000 7f095d000640 1 -- 172.21.15.43:0/1436895247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938003520 con 0x7f0938003970 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.755+0000 7f095d000640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0938003970 0x7f0938003d90 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f094800c270 tx=0x7f09380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792003dce53cf95f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.755+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094802cae0 con 0x7f0938003970 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.755+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094802cc40 con 0x7f0938003970 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.755+0000 7f0957400640 1 -- 172.21.15.43:0/1436895247 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480058d0 con 0x7f0938003970 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 -- 172.21.15.43:0/1436895247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0938003970 msgr2=0x7f0938003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 --2- 172.21.15.43:0/1436895247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0938003970 0x7f0938003d90 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f094800c270 tx=0x7f09380057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095da00640 1 -- 172.21.15.43:0/1436895247 reap_dead start 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 -- 172.21.15.43:0/1436895247 shutdown_connections 2024-01-16T15:18:27.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 -- 172.21.15.43:0/1436895247 >> 172.21.15.43:0/1436895247 conn(0x7f09580cdb20 msgr2=0x7f09580cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:27.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 -- 172.21.15.43:0/1436895247 shutdown_connections 2024-01-16T15:18:27.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.756+0000 7f095fa00640 1 -- 172.21.15.43:0/1436895247 wait complete. 2024-01-16T15:18:27.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.758+0000 7f095fa00640 1 Processor -- start 2024-01-16T15:18:27.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.758+0000 7f095fa00640 1 -- start start 2024-01-16T15:18:27.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0938003970 0x7f09581454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 0x7f09581529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0958145090 con 0x7f0958152ef0 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095815cea0 con 0x7f0958146b40 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095fa00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580c5df0 con 0x7f0938003970 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41296/0 (socket says 172.21.15.43:41296) 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.759+0000 7f095da00640 1 -- 172.21.15.43:0/675043298 learned_addr learned my addr 172.21.15.43:0/675043298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095d000640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0938003970 0x7f09581454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f0957e00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 0x7f09581529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 -- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0938003970 msgr2=0x7f09581454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0938003970 0x7f09581454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 -- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 msgr2=0x7f09581529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 0x7f09581529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 -- 172.21.15.43:0/675043298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09581449d0 con 0x7f0958152ef0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f0957e00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 0x7f09581529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095da00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0958160f20 tx=0x7f09500115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000e070 con 0x7f0958152ef0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950003990 con 0x7f0958152ef0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950003b40 con 0x7f0958152ef0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.760+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0958144cb0 con 0x7f0958152ef0 2024-01-16T15:18:27.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.761+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09580c7130 con 0x7f0958152ef0 2024-01-16T15:18:27.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.762+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09581451d0 con 0x7f0958152ef0 2024-01-16T15:18:27.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.763+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f0950003ca0 con 0x7f0958152ef0 2024-01-16T15:18:27.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.763+0000 7f0954c00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 0x7f093006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:27.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.764+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0950089030 con 0x7f0958152ef0 2024-01-16T15:18:27.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.764+0000 7f095d000640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 0x7f093006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:27.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.765+0000 7f095d000640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 0x7f093006e7d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0948034000 tx=0x7f0948007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:27.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.768+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09581451d0 con 0x7f0958152ef0 2024-01-16T15:18:27.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:27.989+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f09581471a0 con 0x7f093006c310 2024-01-16T15:18:28.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.031+0000 7f0954c00640 1 -- 172.21.15.43:0/675043298 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f09581471a0 con 0x7f093006c310 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 msgr2=0x7f093006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 0x7f093006e7d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0948034000 tx=0x7f0948007000 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 msgr2=0x7f09580c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0958160f20 tx=0x7f09500115a0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 shutdown_connections 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f093006c310 0x7f093006e7d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0958146b40 0x7f09581529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0958152ef0 0x7f09580c6bf0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 --2- 172.21.15.43:0/675043298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0938003970 0x7f09581454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 >> 172.21.15.43:0/675043298 conn(0x7f09580cdb20 msgr2=0x7f09580d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:28.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.034+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 shutdown_connections 2024-01-16T15:18:28.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.035+0000 7f095fa00640 1 -- 172.21.15.43:0/675043298 wait complete. 2024-01-16T15:18:28.028 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.038+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.028 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.038+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.028 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.038+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.028 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.038+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.028 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.038+0000 7fd4e8200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone2461 2024-01-16T15:18:28.056 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.067+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.056 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.067+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.057 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.067+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.057 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.067+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.057 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:18:28.067+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.253+0000 7fb497200640 1 Processor -- start 2024-01-16T15:18:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.253+0000 7fb497200640 1 -- start start 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb497200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4901403f0 0x7fb49015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb497200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49013f800 con 0x7fb49006dbe0 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb497200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49015e230 con 0x7fb49015a6f0 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb497200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4900cc8f0 con 0x7fb4901403f0 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb495200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4901403f0 0x7fb49015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb495200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4901403f0 0x7fb49015aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41342/0 (socket says 172.21.15.43:41342) 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb48f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb49006dbe0 0x7fb49013fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46762/0 (socket says 172.21.15.43:46762) 2024-01-16T15:18:28.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.255+0000 7fb495200640 1 -- 172.21.15.43:0/653230790 learned_addr learned my addr 172.21.15.43:0/653230790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:28.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.256+0000 7fb495200640 1 -- 172.21.15.43:0/653230790 >> v1:172.21.15.69:6789/0 conn(0x7fb49015a6f0 legacy=0x7fb49006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:28.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.256+0000 7fb495200640 1 -- 172.21.15.43:0/653230790 >> v1:172.21.15.43:6789/0 conn(0x7fb49006dbe0 legacy=0x7fb49013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.256+0000 7fb495200640 1 -- 172.21.15.43:0/653230790 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49015e510 con 0x7fb4901403f0 2024-01-16T15:18:28.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.256+0000 7fb48ea00640 1 -- 172.21.15.43:0/653230790 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089103407 0 0) 0x7fb49013f800 con 0x7fb49006dbe0 2024-01-16T15:18:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.257+0000 7fb495200640 1 --2- 172.21.15.43:0/653230790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb49015aad0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fb488009e50 tx=0x7fb48802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d226691666f7047 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.257+0000 7fb48ea00640 1 -- 172.21.15.43:0/653230790 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48802cb20 con 0x7fb4901403f0 2024-01-16T15:18:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.257+0000 7fb48ea00640 1 -- 172.21.15.43:0/653230790 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48802cc80 con 0x7fb4901403f0 2024-01-16T15:18:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.257+0000 7fb48ea00640 1 -- 172.21.15.43:0/653230790 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880343f0 con 0x7fb4901403f0 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 -- 172.21.15.43:0/653230790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 msgr2=0x7fb49015aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 --2- 172.21.15.43:0/653230790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb49015aad0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fb488009e50 tx=0x7fb48802bf80 comp rx=0 tx=0).stop 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 -- 172.21.15.43:0/653230790 shutdown_connections 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 --2- 172.21.15.43:0/653230790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb49015aad0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 -- 172.21.15.43:0/653230790 >> 172.21.15.43:0/653230790 conn(0x7fb4900661f0 msgr2=0x7fb4900665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 -- 172.21.15.43:0/653230790 shutdown_connections 2024-01-16T15:18:28.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.258+0000 7fb497200640 1 -- 172.21.15.43:0/653230790 wait complete. 2024-01-16T15:18:28.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.260+0000 7fb497200640 1 Processor -- start 2024-01-16T15:18:28.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.260+0000 7fb497200640 1 -- start start 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 0x7fb490150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49015a6f0 0x7fb490159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb490154570 con 0x7fb4901403f0 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900b4010 con 0x7fb49006dbe0 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb497200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900b36e0 con 0x7fb49015a6f0 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb48f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb48f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41346/0 (socket says 172.21.15.43:41346) 2024-01-16T15:18:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.261+0000 7fb48f400640 1 -- 172.21.15.43:0/3795976879 learned_addr learned my addr 172.21.15.43:0/3795976879 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48fe00640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 0x7fb490150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb495200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49015a6f0 0x7fb490159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 -- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49015a6f0 msgr2=0x7fb490159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49015a6f0 0x7fb490159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 -- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 msgr2=0x7fb490150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 0x7fb490150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 -- 172.21.15.43:0/3795976879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009b30 con 0x7fb4901403f0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48fe00640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 0x7fb490150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb48f400640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c3f0 tx=0x7fb47800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780074e0 con 0x7fb4901403f0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478007640 con 0x7fb4901403f0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478004a20 con 0x7fb4901403f0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.262+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4900b39c0 con 0x7fb4901403f0 2024-01-16T15:18:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.263+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb490150f70 con 0x7fb4901403f0 2024-01-16T15:18:28.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.264+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49006e800 con 0x7fb4901403f0 2024-01-16T15:18:28.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.265+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fb478004b80 con 0x7fb4901403f0 2024-01-16T15:18:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.266+0000 7fb487e00640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 0x7fb46806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.266+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb478088840 con 0x7fb4901403f0 2024-01-16T15:18:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.266+0000 7fb48fe00640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 0x7fb46806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.267+0000 7fb48fe00640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 0x7fb46806e7d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb480000f60 tx=0x7fb480009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.270+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb478054de0 con 0x7fb4901403f0 2024-01-16T15:18:28.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.489+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone2461", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4900b3850 con 0x7fb46806c310 2024-01-16T15:18:28.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.528+0000 7fb487e00640 1 -- 172.21.15.43:0/3795976879 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb4900b3850 con 0x7fb46806c310 2024-01-16T15:18:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 msgr2=0x7fb46806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 0x7fb46806e7d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb480000f60 tx=0x7fb480009040 comp rx=0 tx=0).stop 2024-01-16T15:18:28.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 msgr2=0x7fb4901590c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c3f0 tx=0x7fb47800c8b0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 shutdown_connections 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb46806c310 0x7fb46806e7d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49015a6f0 0x7fb490159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4901403f0 0x7fb4901590c0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 --2- 172.21.15.43:0/3795976879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb49006dbe0 0x7fb490150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 >> 172.21.15.43:0/3795976879 conn(0x7fb4900661f0 msgr2=0x7fb49013d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.531+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 shutdown_connections 2024-01-16T15:18:28.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.532+0000 7fb497200640 1 -- 172.21.15.43:0/3795976879 wait complete. 2024-01-16T15:18:28.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone2462 2024-01-16T15:18:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.790+0000 7fa5f8a00640 1 Processor -- start 2024-01-16T15:18:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.791+0000 7fa5f8a00640 1 -- start start 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.792+0000 7fa5f8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5f4144770 0x7fa5f415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.792+0000 7fa5f8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5f40b35a0 0x7fa5f415f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.792+0000 7fa5f8a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f415f930 con 0x7fa5f406c9d0 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.792+0000 7fa5f8a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40b43b0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.792+0000 7fa5f8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40b3b10 con 0x7fa5f4144770 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5f40b35a0 0x7fa5f415f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa5f40b35a0 0x7fa5f415f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41358/0 (socket says 172.21.15.43:41358) 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 -- 172.21.15.43:0/2438385427 learned_addr learned my addr 172.21.15.43:0/2438385427 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5f3400640 1 --2- 172.21.15.43:0/2438385427 >> v2:172.21.15.69:3300/0 conn(0x7fa5f4144770 0x7fa5f415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 -- 172.21.15.43:0/2438385427 >> v2:172.21.15.69:3300/0 conn(0x7fa5f4144770 msgr2=0x7fa5f415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 --2- 172.21.15.43:0/2438385427 >> v2:172.21.15.69:3300/0 conn(0x7fa5f4144770 0x7fa5f415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 -- 172.21.15.43:0/2438385427 >> v1:172.21.15.43:6789/0 conn(0x7fa5f406c9d0 legacy=0x7fa5f406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5ebe00640 1 -- 172.21.15.43:0/2438385427 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40b3df0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.793+0000 7fa5f2a00640 1 -- 172.21.15.43:0/2438385427 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102600369 0 0) 0x7fa5f415f930 con 0x7fa5f406c9d0 2024-01-16T15:18:28.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.794+0000 7fa5ebe00640 1 --2- 172.21.15.43:0/2438385427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f415f0a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa5f406df90 tx=0x7fa5dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7787f8d2a1a36733 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f2a00640 1 -- 172.21.15.43:0/2438385427 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc030860 con 0x7fa5f40b35a0 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f2a00640 1 -- 172.21.15.43:0/2438385427 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc0309c0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f2a00640 1 -- 172.21.15.43:0/2438385427 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc030d50 con 0x7fa5f40b35a0 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 -- 172.21.15.43:0/2438385427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 msgr2=0x7fa5f415f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/2438385427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f415f0a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa5f406df90 tx=0x7fa5dc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 -- 172.21.15.43:0/2438385427 shutdown_connections 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/2438385427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f415f0a0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/2438385427 >> v2:172.21.15.69:3300/0 conn(0x7fa5f4144770 0x7fa5f415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.795+0000 7fa5f8a00640 1 -- 172.21.15.43:0/2438385427 >> 172.21.15.43:0/2438385427 conn(0x7fa5f40cec10 msgr2=0x7fa5f40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.796+0000 7fa5f8a00640 1 -- 172.21.15.43:0/2438385427 shutdown_connections 2024-01-16T15:18:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.796+0000 7fa5f8a00640 1 -- 172.21.15.43:0/2438385427 wait complete. 2024-01-16T15:18:28.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.797+0000 7fa5f8a00640 1 Processor -- start 2024-01-16T15:18:28.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.798+0000 7fa5f8a00640 1 -- start start 2024-01-16T15:18:28.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.798+0000 7fa5f8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5f406c9d0 0x7fa5f4152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 0x7fa5f4140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f415afa0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4070fe0 con 0x7fa5f4144770 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4070130 con 0x7fa5f406c9d0 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41370/0 (socket says 172.21.15.43:41370) 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 -- 172.21.15.43:0/4111940314 learned_addr learned my addr 172.21.15.43:0/4111940314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f3400640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5f406c9d0 0x7fa5f4152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f3e00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 0x7fa5f4140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 -- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5f406c9d0 msgr2=0x7fa5f4152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5f406c9d0 0x7fa5f4152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 -- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 msgr2=0x7fa5f4140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 0x7fa5f4140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5ebe00640 1 -- 172.21.15.43:0/4111940314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc0094d0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.799+0000 7fa5f3e00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 0x7fa5f4140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5ebe00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000c00 tx=0x7fa5dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc039d60 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc044480 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc044830 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f40703d0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.800+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4070990 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.801+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f4071170 con 0x7fa5f40b35a0 2024-01-16T15:18:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.802+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fa5dc04d3b0 con 0x7fa5f40b35a0 2024-01-16T15:18:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.803+0000 7fa5eb400640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 0x7fa5cc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.803+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa5dc03e070 con 0x7fa5f40b35a0 2024-01-16T15:18:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.804+0000 7fa5f3400640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 0x7fa5cc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.804+0000 7fa5f3400640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 0x7fa5cc06e7d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006fd0 tx=0x7fa5e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:28.808+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5dc077cb0 con 0x7fa5f40b35a0 2024-01-16T15:18:29.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.030+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone2462", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5f4144b50 con 0x7fa5cc06c310 2024-01-16T15:18:29.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.070+0000 7fa5eb400640 1 -- 172.21.15.43:0/4111940314 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa5e400f860 con 0x7fa5cc06c310 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.073+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 msgr2=0x7fa5cc06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.073+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 0x7fa5cc06e7d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006fd0 tx=0x7fa5e4008040 comp rx=0 tx=0).stop 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.073+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 msgr2=0x7fa5f4140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.073+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000c00 tx=0x7fa5dc004730 comp rx=0 tx=0).stop 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 shutdown_connections 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa5cc06c310 0x7fa5cc06e7d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5f4144770 0x7fa5f4140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f40b35a0 0x7fa5f4140990 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 --2- 172.21.15.43:0/4111940314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5f406c9d0 0x7fa5f4152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 >> 172.21.15.43:0/4111940314 conn(0x7fa5f40cec10 msgr2=0x7fa5f413e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:29.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 shutdown_connections 2024-01-16T15:18:29.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.074+0000 7fa5f8a00640 1 -- 172.21.15.43:0/4111940314 wait complete. 2024-01-16T15:18:29.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone2463 2024-01-16T15:18:29.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.307+0000 7fd875c00640 1 Processor -- start 2024-01-16T15:18:29.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.307+0000 7fd875c00640 1 -- start start 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.308+0000 7fd875c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd8700acd70 0x7fd87006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.308+0000 7fd875c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8700ae490 0x7fd8700ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd875c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87006d7a0 con 0x7fd8700ad690 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd875c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87015a5b0 con 0x7fd8700acd70 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd875c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87006ce70 con 0x7fd8700ae490 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd8700acd70 0x7fd87006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd8700acd70 0x7fd87006dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41376/0 (socket says 172.21.15.43:41376) 2024-01-16T15:18:29.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 -- 172.21.15.43:0/3582059015 learned_addr learned my addr 172.21.15.43:0/3582059015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 -- 172.21.15.43:0/3582059015 >> v2:172.21.15.69:3300/0 conn(0x7fd8700ae490 msgr2=0x7fd8700ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 --2- 172.21.15.43:0/3582059015 >> v2:172.21.15.69:3300/0 conn(0x7fd8700ae490 0x7fd8700ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 -- 172.21.15.43:0/3582059015 >> v1:172.21.15.43:6789/0 conn(0x7fd8700ad690 legacy=0x7fd870070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.309+0000 7fd86ea00640 1 -- 172.21.15.43:0/3582059015 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87006d150 con 0x7fd8700acd70 2024-01-16T15:18:29.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.310+0000 7fd86ea00640 1 --2- 172.21.15.43:0/3582059015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700acd70 0x7fd87006dc30 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd87006e720 tx=0x7fd86002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7203f2d97a6b3c91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.311+0000 7fd86e000640 1 -- 172.21.15.43:0/3582059015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860030860 con 0x7fd8700acd70 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.311+0000 7fd86e000640 1 -- 172.21.15.43:0/3582059015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600309c0 con 0x7fd8700acd70 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.311+0000 7fd86e000640 1 -- 172.21.15.43:0/3582059015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860030d50 con 0x7fd8700acd70 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.311+0000 7fd875c00640 1 -- 172.21.15.43:0/3582059015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700acd70 msgr2=0x7fd87006dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.311+0000 7fd875c00640 1 --2- 172.21.15.43:0/3582059015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700acd70 0x7fd87006dc30 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd87006e720 tx=0x7fd86002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 -- 172.21.15.43:0/3582059015 shutdown_connections 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 --2- 172.21.15.43:0/3582059015 >> v2:172.21.15.69:3300/0 conn(0x7fd8700ae490 0x7fd8700ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 --2- 172.21.15.43:0/3582059015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700acd70 0x7fd87006dc30 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 -- 172.21.15.43:0/3582059015 >> 172.21.15.43:0/3582059015 conn(0x7fd8700b0ba0 msgr2=0x7fd8700b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 -- 172.21.15.43:0/3582059015 shutdown_connections 2024-01-16T15:18:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.312+0000 7fd875c00640 1 -- 172.21.15.43:0/3582059015 wait complete. 2024-01-16T15:18:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.313+0000 7fd875c00640 1 Processor -- start 2024-01-16T15:18:29.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.314+0000 7fd875c00640 1 -- start start 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.314+0000 7fd875c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8700acd70 0x7fd87014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.314+0000 7fd875c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.314+0000 7fd875c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8700ae490 0x7fd8700b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd875c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700cd690 con 0x7fd8700ad690 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd875c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700cca10 con 0x7fd8700ae490 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd875c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700cccf0 con 0x7fd8700acd70 2024-01-16T15:18:29.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41390/0 (socket says 172.21.15.43:41390) 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 -- 172.21.15.43:0/3302479991 learned_addr learned my addr 172.21.15.43:0/3302479991 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd86ea00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8700acd70 0x7fd87014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd86f400640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8700ae490 0x7fd8700b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 -- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8700acd70 msgr2=0x7fd87014ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8700acd70 0x7fd87014ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 -- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8700ae490 msgr2=0x7fd8700b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8700ae490 0x7fd8700b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.315+0000 7fd867e00640 1 -- 172.21.15.43:0/3302479991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600094d0 con 0x7fd8700ad690 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd867e00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b4a0 tx=0x7fd85800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858010610 con 0x7fd8700ad690 2024-01-16T15:18:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858010770 con 0x7fd8700ad690 2024-01-16T15:18:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858010b20 con 0x7fd8700ad690 2024-01-16T15:18:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8700ccfd0 con 0x7fd8700ad690 2024-01-16T15:18:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.316+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87014f1f0 con 0x7fd8700ad690 2024-01-16T15:18:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.317+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8700cd7d0 con 0x7fd8700ad690 2024-01-16T15:18:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.318+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7fd858010c80 con 0x7fd8700ad690 2024-01-16T15:18:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.319+0000 7fd866a00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 0x7fd84806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.320+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd858088630 con 0x7fd8700ad690 2024-01-16T15:18:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.320+0000 7fd86ea00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 0x7fd84806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.320+0000 7fd86ea00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 0x7fd84806e800 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.324+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8700cd7d0 con 0x7fd8700ad690 2024-01-16T15:18:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.547+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone2463", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8700ad430 con 0x7fd84806c340 2024-01-16T15:18:29.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.601+0000 7fd866a00640 1 -- 172.21.15.43:0/3302479991 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd8700ad430 con 0x7fd84806c340 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 msgr2=0x7fd84806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 0x7fd84806e800 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003b040 comp rx=0 tx=0).stop 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 msgr2=0x7fd87015cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b4a0 tx=0x7fd85800b960 comp rx=0 tx=0).stop 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 shutdown_connections 2024-01-16T15:18:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd84806c340 0x7fd84806e800 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8700ae490 0x7fd8700b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8700ad690 0x7fd87015cf20 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 --2- 172.21.15.43:0/3302479991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8700acd70 0x7fd87014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 >> 172.21.15.43:0/3302479991 conn(0x7fd8700b0ba0 msgr2=0x7fd870166410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 shutdown_connections 2024-01-16T15:18:29.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.604+0000 7fd875c00640 1 -- 172.21.15.43:0/3302479991 wait complete. 2024-01-16T15:18:29.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:29.617 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:18:29.680 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-01-16T15:18:29.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:18:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.918+0000 7f4f17c00640 1 Processor -- start 2024-01-16T15:18:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.918+0000 7f4f17c00640 1 -- start start 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f17c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4f100abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f17c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1015b730 con 0x7f4f1015b350 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f17c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f100ac740 con 0x7f4f1009cb30 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f17c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100ac460 con 0x7f4f100ae270 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f15200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4f1015b350 0x7f4f100b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46776/0 (socket says 172.21.15.43:46776) 2024-01-16T15:18:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.920+0000 7f4f15200640 1 -- 172.21.15.43:0/834434036 learned_addr learned my addr 172.21.15.43:0/834434036 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f15c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4f100abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982669301 0 0) 0x7f4f00001110 con 0x7f4f1015b350 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef0003520 con 0x7f4f1015b350 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065681218 0 0) 0x7f4ef0003520 con 0x7f4f1015b350 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 msgr2=0x7f4f100abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4f100abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.921+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 >> v1:172.21.15.69:6789/0 conn(0x7f4f1009cb30 legacy=0x7f4f100adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f100b2a50 con 0x7f4f1015b350 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4f00001800 con 0x7f4f1015b350 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 >> v1:172.21.15.43:6789/0 conn(0x7f4f1015b350 legacy=0x7f4f100b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef00039a0 0x7f4ef0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4ef00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ef0004d00 0x7f4ef0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100ac460 con 0x7f4ef00039a0 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000ba60 con 0x7f4ef0004d00 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000bc20 con 0x7f4f100ae270 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef00039a0 0x7f4ef0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f07c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4ef00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ef0004d00 0x7f4ef0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 -- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 msgr2=0x7f4ef00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4ef00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 -- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ef0004d00 msgr2=0x7f4ef0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ef0004d00 0x7f4ef0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.922+0000 7f4f15200640 1 -- 172.21.15.43:0/834434036 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef0003520 con 0x7f4ef00039a0 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.923+0000 7f4f15c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ef0004d00 0x7f4ef0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.923+0000 7f4f07c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f100ae270 0x7f4ef00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.924+0000 7f4f15200640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef00039a0 0x7f4ef0003dc0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4f00001eb0 tx=0x7f4ef00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f6690f3efe0281 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.924+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00005ad0 con 0x7f4ef00039a0 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.924+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f00005c50 con 0x7f4ef00039a0 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.924+0000 7f4f0fe00640 1 -- 172.21.15.43:0/834434036 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0002c7a0 con 0x7f4ef00039a0 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 -- 172.21.15.43:0/834434036 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef00039a0 msgr2=0x7f4ef0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 --2- 172.21.15.43:0/834434036 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef00039a0 0x7f4ef0003dc0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4f00001eb0 tx=0x7f4ef00057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f15c00640 1 -- 172.21.15.43:0/834434036 reap_dead start 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 -- 172.21.15.43:0/834434036 shutdown_connections 2024-01-16T15:18:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 -- 172.21.15.43:0/834434036 >> 172.21.15.43:0/834434036 conn(0x7f4f100b0ba0 msgr2=0x7f4f100b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 -- 172.21.15.43:0/834434036 shutdown_connections 2024-01-16T15:18:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.925+0000 7f4f17c00640 1 -- 172.21.15.43:0/834434036 wait complete. 2024-01-16T15:18:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.927+0000 7f4f17c00640 1 Processor -- start 2024-01-16T15:18:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.927+0000 7f4f17c00640 1 -- start start 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ef00039a0 0x7f4f10071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f10070490 0x7f4f10148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1015db60 con 0x7f4f1015d1a0 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f10070a00 con 0x7f4f10070490 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f17c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f10070ce0 con 0x7f4ef00039a0 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41406/0 (socket says 172.21.15.43:41406) 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 -- 172.21.15.43:0/4008657554 learned_addr learned my addr 172.21.15.43:0/4008657554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15200640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ef00039a0 0x7f4f10071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f07c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f10070490 0x7f4f10148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 -- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ef00039a0 msgr2=0x7f4f10071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ef00039a0 0x7f4f10071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 -- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f10070490 msgr2=0x7f4f10148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f10070490 0x7f4f10148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.928+0000 7f4f15c00640 1 -- 172.21.15.43:0/4008657554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10070fc0 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f15c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4f10067e90 tx=0x7f4f0800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08017400 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08011040 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f100712a0 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08017780 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.929+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f10072210 con 0x7f4f1015d1a0 2024-01-16T15:18:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.930+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1009d200 con 0x7f4f1015d1a0 2024-01-16T15:18:29.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.931+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 40) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f4f080203b0 con 0x7f4f1015d1a0 2024-01-16T15:18:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.933+0000 7f4f0d600640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 0x7f4ee806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:29.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.933+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4f08088cd0 con 0x7f4f1015d1a0 2024-01-16T15:18:29.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.933+0000 7f4f15200640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 0x7f4ee806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:29.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.934+0000 7f4f15200640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 0x7f4ee806e7d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4f00034000 tx=0x7f4f00009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:29.937+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f08055270 con 0x7f4f1015d1a0 2024-01-16T15:18:30.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:30.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:30.162+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f1006e720 con 0x7f4ee806c310 2024-01-16T15:18:30.165 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-01-16T15:18:30.175+0000 7f83f5800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:18:30.299 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:18:30.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:30.376+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 41) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4f08055640 con 0x7f4f1015d1a0 2024-01-16T15:18:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:32.391+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4f08070a50 con 0x7f4f1015d1a0 2024-01-16T15:18:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.210+0000 7f4f0d600640 1 -- 172.21.15.43:0/4008657554 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f4f00001110 con 0x7f4ee806c310 2024-01-16T15:18:33.201 INFO:teuthology.orchestra.run.smithi043.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:18:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 msgr2=0x7f4ee806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 0x7f4ee806e7d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4f00034000 tx=0x7f4f00009030 comp rx=0 tx=0).stop 2024-01-16T15:18:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 msgr2=0x7f4f1015d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4f10067e90 tx=0x7f4f0800f4a0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 shutdown_connections 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4ee806c310 0x7f4ee806e7d0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f10070490 0x7f4f10148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f1015d1a0 0x7f4f1015d620 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 --2- 172.21.15.43:0/4008657554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ef00039a0 0x7f4f10071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.215+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 >> 172.21.15.43:0/4008657554 conn(0x7f4f100b0ba0 msgr2=0x7f4f100adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.216+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 shutdown_connections 2024-01-16T15:18:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.216+0000 7f4f17c00640 1 -- 172.21.15.43:0/4008657554 wait complete. 2024-01-16T15:18:33.239 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:18:33.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:18:33.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:18:33.426 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:18:33.426 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:18:33.426 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:33.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.771+0000 7f7d19400640 1 Processor -- start 2024-01-16T15:18:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.772+0000 7f7d19400640 1 -- start start 2024-01-16T15:18:33.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.775+0000 7f7d19400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 0x7f7d1406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.775+0000 7f7d19400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d141616a0 0x7f7d1406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.775+0000 7f7d19400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d14147140 con 0x7f7d1406efb0 2024-01-16T15:18:33.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.775+0000 7f7d19400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d140b4060 con 0x7f7d141616a0 2024-01-16T15:18:33.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.775+0000 7f7d19400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1406f3f0 con 0x7f7d14146d60 2024-01-16T15:18:33.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.776+0000 7f7d11e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7d1406efb0 0x7f7d14160f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:43514/0 (socket says 172.21.15.43:43514) 2024-01-16T15:18:33.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.776+0000 7f7d11e00640 1 -- 172.21.15.43:0/4263818434 learned_addr learned my addr 172.21.15.43:0/4263818434 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.777+0000 7f7d12800640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 0x7f7d1406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.777+0000 7f7d13200640 1 --2- 172.21.15.43:0/4263818434 >> v2:172.21.15.69:3300/0 conn(0x7f7d141616a0 0x7f7d1406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.777+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502214408 0 0) 0x7f7d14147140 con 0x7f7d1406efb0 2024-01-16T15:18:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.777+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf4003520 con 0x7f7d1406efb0 2024-01-16T15:18:33.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d13200640 1 -- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 msgr2=0x7f7d1406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:33.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d13200640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 0x7f7d1406ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454088039 0 0) 0x7f7cf4003520 con 0x7f7d1406efb0 2024-01-16T15:18:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d13200640 1 -- 172.21.15.43:0/4263818434 >> v1:172.21.15.69:6789/0 conn(0x7f7d1406efb0 legacy=0x7f7d14160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d13200640 1 -- 172.21.15.43:0/4263818434 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14161ca0 con 0x7f7d141616a0 2024-01-16T15:18:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.778+0000 7f7d12800640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 0x7f7d1406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:18:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.780+0000 7f7d13200640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d141616a0 0x7f7d1406b9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009d90 tx=0x7f7d0c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d8ffb29792598f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.781+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc001670 con 0x7f7d141616a0 2024-01-16T15:18:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.781+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c02c8a0 con 0x7f7d141616a0 2024-01-16T15:18:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.781+0000 7f7d11400640 1 -- 172.21.15.43:0/4263818434 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c02cc30 con 0x7f7d141616a0 2024-01-16T15:18:33.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.783+0000 7f7d19400640 1 -- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d141616a0 msgr2=0x7f7d1406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.783+0000 7f7d19400640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d141616a0 0x7f7d1406b9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009d90 tx=0x7f7d0c02bd70 comp rx=0 tx=0).stop 2024-01-16T15:18:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.783+0000 7f7d19400640 1 -- 172.21.15.43:0/4263818434 shutdown_connections 2024-01-16T15:18:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.783+0000 7f7d19400640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d141616a0 0x7f7d1406b9e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.783+0000 7f7d19400640 1 --2- 172.21.15.43:0/4263818434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d14146d60 0x7f7d1406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.784+0000 7f7d19400640 1 -- 172.21.15.43:0/4263818434 >> 172.21.15.43:0/4263818434 conn(0x7f7d140663d0 msgr2=0x7f7d140667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:33.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.787+0000 7f7d19400640 1 -- 172.21.15.43:0/4263818434 shutdown_connections 2024-01-16T15:18:33.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.787+0000 7f7d19400640 1 -- 172.21.15.43:0/4263818434 wait complete. 2024-01-16T15:18:33.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.790+0000 7f7d19400640 1 Processor -- start 2024-01-16T15:18:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.791+0000 7f7d19400640 1 -- start start 2024-01-16T15:18:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.792+0000 7f7d19400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1406efb0 0x7f7d140c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.792+0000 7f7d19400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 0x7f7d140c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d19400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 0x7f7d140d0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d19400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140d0a80 con 0x7f7d14146d60 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d19400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14144e50 con 0x7f7d1406efb0 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d19400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140c6aa0 con 0x7f7d141616a0 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d11e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 0x7f7d140c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d11e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 0x7f7d140c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57386/0 (socket says 172.21.15.43:57386) 2024-01-16T15:18:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d11e00640 1 -- 172.21.15.43:0/3818639595 learned_addr learned my addr 172.21.15.43:0/3818639595 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.793+0000 7f7d12800640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1406efb0 0x7f7d140c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 0x7f7d140d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 -- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1406efb0 msgr2=0x7f7d140c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1406efb0 0x7f7d140c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 -- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 msgr2=0x7f7d140c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 0x7f7d140c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.794+0000 7f7d13200640 1 -- 172.21.15.43:0/3818639595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d04009400 con 0x7f7d141616a0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.795+0000 7f7d13200640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 0x7f7d140d0540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c033000 tx=0x7f7d0c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.796+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c034bd0 con 0x7f7d141616a0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.796+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c034d30 con 0x7f7d141616a0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.796+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c009a70 con 0x7f7d141616a0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.796+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c034ea0 con 0x7f7d141616a0 2024-01-16T15:18:33.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.796+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d140c6450 con 0x7f7d141616a0 2024-01-16T15:18:33.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.799+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4005250 con 0x7f7d141616a0 2024-01-16T15:18:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.801+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f7d0c0677b0 con 0x7f7d141616a0 2024-01-16T15:18:33.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.803+0000 7f7d0aa00640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 0x7f7cec06e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:33.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.803+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7d0c0ab090 con 0x7f7d141616a0 2024-01-16T15:18:33.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.804+0000 7f7d12800640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 0x7f7cec06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:33.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.804+0000 7f7d12800640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 0x7f7cec06e890 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c30 tx=0x7f7d04009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:33.808+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d0c077b60 con 0x7f7d141616a0 2024-01-16T15:18:33.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.005+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce40057d0 con 0x7f7d141616a0 2024-01-16T15:18:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.007+0000 7f7d0aa00640 1 -- 172.21.15.43:0/3818639595 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12983 (secure 0 0 0) 0x7f7d0c04d090 con 0x7f7d141616a0 2024-01-16T15:18:33.998 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:33.998 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":104,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:33.192509+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6833/3755062795":"2024-01-17T15:18:30.175901+0000","172.21.15.69:6832/3755062795":"2024-01-17T15:18:30.175901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:34.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 msgr2=0x7f7cec06e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 0x7f7cec06e890 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c30 tx=0x7f7d04009190 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 msgr2=0x7f7d140d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 0x7f7d140d0540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c033000 tx=0x7f7d0c004a40 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 shutdown_connections 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7cec06c3d0 0x7f7cec06e890 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d141616a0 0x7f7d140d0540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d14146d60 0x7f7d140c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 --2- 172.21.15.43:0/3818639595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1406efb0 0x7f7d140c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 >> 172.21.15.43:0/3818639595 conn(0x7f7d140663d0 msgr2=0x7f7d1415feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 shutdown_connections 2024-01-16T15:18:34.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.010+0000 7f7d19400640 1 -- 172.21.15.43:0/3818639595 wait complete. 2024-01-16T15:18:34.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:18:34.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:18:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.249+0000 7fcd5d800640 1 Processor -- start 2024-01-16T15:18:34.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.250+0000 7fcd5d800640 1 -- start start 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.251+0000 7fcd5d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcd5815d3b0 0x7fcd580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.251+0000 7fcd5d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd580b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.251+0000 7fcd5d800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580c6bf0 con 0x7fcd580d0fa0 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.251+0000 7fcd5d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd580d1430 con 0x7fcd5815d3b0 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.251+0000 7fcd5d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580cedc0 con 0x7fcd580d1d90 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd57400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcd580d0fa0 0x7fcd580d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56114/0 (socket says 172.21.15.43:56114) 2024-01-16T15:18:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd57400640 1 -- 172.21.15.43:0/2612021599 learned_addr learned my addr 172.21.15.43:0/2612021599 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd57e00640 1 --2- 172.21.15.43:0/2612021599 >> v2:172.21.15.69:3300/0 conn(0x7fcd5815d3b0 0x7fcd580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd5ce00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd580b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551673436 0 0) 0x7fcd580c6bf0 con 0x7fcd580d0fa0 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd38003520 con 0x7fcd580d0fa0 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607188961 0 0) 0x7fcd38003520 con 0x7fcd580d0fa0 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 msgr2=0x7fcd580b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd580b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 >> v2:172.21.15.69:3300/0 conn(0x7fcd5815d3b0 msgr2=0x7fcd580ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 --2- 172.21.15.43:0/2612021599 >> v2:172.21.15.69:3300/0 conn(0x7fcd5815d3b0 0x7fcd580ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580c6e40 con 0x7fcd580d0fa0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.252+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcd400019f0 con 0x7fcd580d0fa0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 >> v1:172.21.15.43:6789/0 conn(0x7fcd580d0fa0 legacy=0x7fcd580d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd380039e0 0x7fcd38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd380047c0 0x7fcd38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580d1430 con 0x7fcd380039e0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580cedc0 con 0x7fcd380047c0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3800bae0 con 0x7fcd580d1d90 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd57e00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd380039e0 0x7fcd38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd57400640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd380047c0 0x7fcd38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.253+0000 7fcd5ce00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57e00640 1 -- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 msgr2=0x7fcd38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57e00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd580d1d90 0x7fcd38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57e00640 1 -- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd380047c0 msgr2=0x7fcd38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57e00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd380047c0 0x7fcd38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57e00640 1 -- 172.21.15.43:0/2612021599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38003520 con 0x7fcd380039e0 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.254+0000 7fcd57400640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd380047c0 0x7fcd38004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.255+0000 7fcd57e00640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd380039e0 0x7fcd38003e00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800ba00 tx=0x7fcd380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bf82981f313786 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.255+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48032630 con 0x7fcd380039e0 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.255+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48032790 con 0x7fcd380039e0 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.255+0000 7fcd56a00640 1 -- 172.21.15.43:0/2612021599 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48032b20 con 0x7fcd380039e0 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 -- 172.21.15.43:0/2612021599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd380039e0 msgr2=0x7fcd38003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 --2- 172.21.15.43:0/2612021599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd380039e0 0x7fcd38003e00 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800ba00 tx=0x7fcd380057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5ce00640 1 -- 172.21.15.43:0/2612021599 reap_dead start 2024-01-16T15:18:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 -- 172.21.15.43:0/2612021599 shutdown_connections 2024-01-16T15:18:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 -- 172.21.15.43:0/2612021599 >> 172.21.15.43:0/2612021599 conn(0x7fcd580d30a0 msgr2=0x7fcd580d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 -- 172.21.15.43:0/2612021599 shutdown_connections 2024-01-16T15:18:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.256+0000 7fcd5d800640 1 -- 172.21.15.43:0/2612021599 wait complete. 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.258+0000 7fcd5d800640 1 Processor -- start 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.258+0000 7fcd5d800640 1 -- start start 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd380039e0 0x7fcd580b8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd58160580 0x7fcd580cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581542b0 con 0x7fcd581544d0 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580b8680 con 0x7fcd58160580 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd5d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580b70a0 con 0x7fcd380039e0 2024-01-16T15:18:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd57400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd57400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57396/0 (socket says 172.21.15.43:57396) 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.259+0000 7fcd57400640 1 -- 172.21.15.43:0/1869083004 learned_addr learned my addr 172.21.15.43:0/1869083004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57e00640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd380039e0 0x7fcd580b8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd5ce00640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd58160580 0x7fcd580cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 -- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd380039e0 msgr2=0x7fcd580b8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd380039e0 0x7fcd580b8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 -- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd58160580 msgr2=0x7fcd580cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd58160580 0x7fcd580cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 -- 172.21.15.43:0/1869083004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580b7380 con 0x7fcd581544d0 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.260+0000 7fcd57400640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fcd40001c60 tx=0x7fcd4000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.261+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000edf0 con 0x7fcd581544d0 2024-01-16T15:18:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.261+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd40013420 con 0x7fcd581544d0 2024-01-16T15:18:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.261+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd580b7660 con 0x7fcd581544d0 2024-01-16T15:18:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.261+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40013790 con 0x7fcd581544d0 2024-01-16T15:18:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.261+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd580b7c20 con 0x7fcd581544d0 2024-01-16T15:18:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.262+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5804e840 con 0x7fcd581544d0 2024-01-16T15:18:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.263+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fcd400138f0 con 0x7fcd581544d0 2024-01-16T15:18:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.264+0000 7fcd4fe00640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 0x7fcd3006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.264+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcd40088e90 con 0x7fcd581544d0 2024-01-16T15:18:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.264+0000 7fcd57e00640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 0x7fcd3006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.265+0000 7fcd57e00640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 0x7fcd3006e890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcd380057a0 tx=0x7fcd4803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.268+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd40055750 con 0x7fcd581544d0 2024-01-16T15:18:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.464+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd580cc440 con 0x7fcd581544d0 2024-01-16T15:18:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.465+0000 7fcd4fe00640 1 -- 172.21.15.43:0/1869083004 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd40058d70 con 0x7fcd581544d0 2024-01-16T15:18:34.455 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 msgr2=0x7fcd3006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 0x7fcd3006e890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcd380057a0 tx=0x7fcd4803d040 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 msgr2=0x7fcd580cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fcd40001c60 tx=0x7fcd4000e530 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 shutdown_connections 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd3006c3d0 0x7fcd3006e890 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd58160580 0x7fcd580cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd581544d0 0x7fcd580cb8a0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 --2- 172.21.15.43:0/1869083004 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd380039e0 0x7fcd580b8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.467+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 >> 172.21.15.43:0/1869083004 conn(0x7fcd580d30a0 msgr2=0x7fcd5806af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.468+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 shutdown_connections 2024-01-16T15:18:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.468+0000 7fcd5d800640 1 -- 172.21.15.43:0/1869083004 wait complete. 2024-01-16T15:18:34.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:18:34.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.713+0000 7f73abe00640 1 Processor -- start 2024-01-16T15:18:34.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.714+0000 7f73abe00640 1 -- start start 2024-01-16T15:18:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73abe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f73ac0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73abe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73ac0d1670 0x7f73ac0d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73abe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0d04b0 con 0x7f73ac15d5d0 2024-01-16T15:18:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73abe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0ce4e0 con 0x7f73ac0d1670 2024-01-16T15:18:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73abe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0cf010 con 0x7f73ac0cf710 2024-01-16T15:18:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73aaa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f73ac15d5d0 0x7f73ac0ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56130/0 (socket says 172.21.15.43:56130) 2024-01-16T15:18:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.715+0000 7f73aaa00640 1 -- 172.21.15.43:0/1593865420 learned_addr learned my addr 172.21.15.43:0/1593865420 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f73ac0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222095410 0 0) 0x7f73ac0d04b0 con 0x7f73ac15d5d0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73ab400640 1 --2- 172.21.15.43:0/1593865420 >> v2:172.21.15.69:3300/0 conn(0x7f73ac0d1670 0x7f73ac0d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c003520 con 0x7f73ac15d5d0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389530710 0 0) 0x7f738c003520 con 0x7f73ac15d5d0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 msgr2=0x7f73ac0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f73ac0cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 >> v2:172.21.15.69:3300/0 conn(0x7f73ac0d1670 msgr2=0x7f73ac0d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 --2- 172.21.15.43:0/1593865420 >> v2:172.21.15.69:3300/0 conn(0x7f73ac0d1670 0x7f73ac0d1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0cdb80 con 0x7f73ac15d5d0 2024-01-16T15:18:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7394001670 con 0x7f73ac15d5d0 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f73ac0cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 >> v1:172.21.15.43:6789/0 conn(0x7f73ac15d5d0 legacy=0x7f73ac0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f738c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f738c003fd0 0x7f738c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c004d00 0x7f738c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0ce4e0 con 0x7f738c003fd0 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0cf010 con 0x7f738c004d00 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.716+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c00bb60 con 0x7f73ac0cf710 2024-01-16T15:18:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73ab400640 1 --2- 172.21.15.43:0/1593865420 >> v2:172.21.15.69:3300/0 conn(0x7f73ac0d1670 0x7f73ac0d1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f738c003fd0 0x7f738c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aaa00640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f738c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 -- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 msgr2=0x7f738c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f738c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 -- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c004d00 msgr2=0x7f738c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73ab400640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c004d00 0x7f738c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c004d00 0x7f738c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aa000640 1 -- 172.21.15.43:0/1593865420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c003520 con 0x7f738c003fd0 2024-01-16T15:18:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73aaa00640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac0cf710 0x7f738c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.717+0000 7f73ab400640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c004d00 0x7f738c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.718+0000 7f73aa000640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f738c003fd0 0x7f738c0047c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f739c007410 tx=0x7f738c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39865e12e2fba8cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.718+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c02cd40 con 0x7f738c003fd0 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.718+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c034440 con 0x7f738c003fd0 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.718+0000 7f73a9600640 1 -- 172.21.15.43:0/1593865420 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c02cea0 con 0x7f738c003fd0 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 -- 172.21.15.43:0/1593865420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f738c003fd0 msgr2=0x7f738c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 --2- 172.21.15.43:0/1593865420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f738c003fd0 0x7f738c0047c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f739c007410 tx=0x7f738c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73ab400640 1 -- 172.21.15.43:0/1593865420 reap_dead start 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 -- 172.21.15.43:0/1593865420 shutdown_connections 2024-01-16T15:18:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 -- 172.21.15.43:0/1593865420 >> 172.21.15.43:0/1593865420 conn(0x7f73ac09c3d0 msgr2=0x7f73ac0d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 -- 172.21.15.43:0/1593865420 shutdown_connections 2024-01-16T15:18:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.719+0000 7f73abe00640 1 -- 172.21.15.43:0/1593865420 wait complete. 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.721+0000 7f73abe00640 1 Processor -- start 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.721+0000 7f73abe00640 1 -- start start 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c003fd0 0x7f73ac1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac157120 0x7f73ac0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac1583f0 con 0x7f73ac1575a0 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0ca390 con 0x7f738c003fd0 2024-01-16T15:18:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73abe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac157b50 con 0x7f73ac157120 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.722+0000 7f73ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57412/0 (socket says 172.21.15.43:57412) 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 -- 172.21.15.43:0/3133214377 learned_addr learned my addr 172.21.15.43:0/3133214377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73aaa00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c003fd0 0x7f73ac1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73aa000640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac157120 0x7f73ac0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 -- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac157120 msgr2=0x7f73ac0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac157120 0x7f73ac0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 -- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c003fd0 msgr2=0x7f73ac1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c003fd0 0x7f73ac1f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 -- 172.21.15.43:0/3133214377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac157e30 con 0x7f73ac1575a0 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.723+0000 7f73ab400640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0b90c0 tx=0x7f73a400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.724+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4004030 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.724+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a400c070 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.724+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40043d0 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.724+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac158110 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.724+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac158a50 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.725+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac0ca160 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.726+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f73a4003610 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.727+0000 7f73a2a00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 0x7f738406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.727+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f73a4088400 con 0x7f73ac1575a0 2024-01-16T15:18:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.727+0000 7f73aaa00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 0x7f738406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.728+0000 7f73aaa00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 0x7f738406e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7394001eb0 tx=0x7f7394009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.731+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73a4054cc0 con 0x7f73ac1575a0 2024-01-16T15:18:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.937+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f73ac04e840 con 0x7f73ac1575a0 2024-01-16T15:18:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.938+0000 7f73a2a00640 1 -- 172.21.15.43:0/3133214377 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v104) v1 ==== 80+0+122 (secure 0 0 0) 0x7f73a40582e0 con 0x7f73ac1575a0 2024-01-16T15:18:34.928 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6833/3755062795 2024-01-17T15:18:30.175901+0000 2024-01-16T15:18:34.928 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/3755062795 2024-01-17T15:18:30.175901+0000 2024-01-16T15:18:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:18:34.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.940+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 msgr2=0x7f738406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.940+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 0x7f738406e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7394001eb0 tx=0x7f7394009040 comp rx=0 tx=0).stop 2024-01-16T15:18:34.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.940+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 msgr2=0x7f73ac0b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.940+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0b90c0 tx=0x7f73a400f670 comp rx=0 tx=0).stop 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 shutdown_connections 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f738406c3a0 0x7f738406e860 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73ac1575a0 0x7f73ac0b4240 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ac157120 0x7f73ac0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 --2- 172.21.15.43:0/3133214377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f738c003fd0 0x7f73ac1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 >> 172.21.15.43:0/3133214377 conn(0x7f73ac09c3d0 msgr2=0x7f73ac0b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 shutdown_connections 2024-01-16T15:18:34.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:34.941+0000 7f73abe00640 1 -- 172.21.15.43:0/3133214377 wait complete. 2024-01-16T15:18:34.949 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:18:34.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:34.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:18:35.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[72177]: fuse finished with error 0 and tester_r 0 2024-01-16T15:18:35.048 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:18:35.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:35.048 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:18:35.258 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:18:35.258 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:35.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.559+0000 7f83bdc00640 1 Processor -- start 2024-01-16T15:18:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.559+0000 7f83bdc00640 1 -- start start 2024-01-16T15:18:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83bdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 0x7f83b815aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83bdc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80ad6b0 con 0x7f83b815b050 2024-01-16T15:18:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83bdc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80b2860 con 0x7f83b806ddb0 2024-01-16T15:18:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83bdc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80b2b40 con 0x7f83b8153f10 2024-01-16T15:18:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83ade00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f83b815b050 0x7f83b80b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56146/0 (socket says 172.21.15.43:56146) 2024-01-16T15:18:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83ade00640 1 -- 172.21.15.43:0/3772540155 learned_addr learned my addr 172.21.15.43:0/3772540155 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.561+0000 7f83b6a00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 0x7f83b815aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911724967 0 0) 0x7f83b80ad6b0 con 0x7f83b815b050 2024-01-16T15:18:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398003520 con 0x7f83b815b050 2024-01-16T15:18:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591642088 0 0) 0x7f8398003520 con 0x7f83b815b050 2024-01-16T15:18:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 msgr2=0x7f83b815aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 0x7f83b815aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 >> v1:172.21.15.69:6789/0 conn(0x7f83b806ddb0 legacy=0x7f83b80adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80b2e20 con 0x7f83b815b050 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f83a0001670 con 0x7f83b815b050 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 >> v1:172.21.15.43:6789/0 conn(0x7f83b815b050 legacy=0x7f83b80b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83980039a0 0x7f8398003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 0x7f83980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8398004d00 0x7f8398005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80b2b40 con 0x7f83980039a0 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839800ba60 con 0x7f8398004d00 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839800de30 con 0x7f83b8153f10 2024-01-16T15:18:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.562+0000 7f83b6a00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83980039a0 0x7f8398003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b7400640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8398004d00 0x7f8398005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b6a00640 1 -- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 msgr2=0x7f83980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b6a00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b8153f10 0x7f83980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b6a00640 1 -- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8398004d00 msgr2=0x7f8398005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b6a00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8398004d00 0x7f8398005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b6a00640 1 -- 172.21.15.43:0/3772540155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398003520 con 0x7f83980039a0 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.563+0000 7f83b7400640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8398004d00 0x7f8398005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.564+0000 7f83b6a00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83980039a0 0x7f8398003dc0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009f90 tx=0x7f83980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3340249ea3894b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.564+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a802cc20 con 0x7f83980039a0 2024-01-16T15:18:35.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.564+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a800f040 con 0x7f83980039a0 2024-01-16T15:18:35.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83b6000640 1 -- 172.21.15.43:0/3772540155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8034440 con 0x7f83980039a0 2024-01-16T15:18:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 -- 172.21.15.43:0/3772540155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83980039a0 msgr2=0x7f8398003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 --2- 172.21.15.43:0/3772540155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83980039a0 0x7f8398003dc0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009f90 tx=0x7f83980057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83b7400640 1 -- 172.21.15.43:0/3772540155 reap_dead start 2024-01-16T15:18:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 -- 172.21.15.43:0/3772540155 shutdown_connections 2024-01-16T15:18:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 -- 172.21.15.43:0/3772540155 >> 172.21.15.43:0/3772540155 conn(0x7f83b80b0d80 msgr2=0x7f83b80b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 -- 172.21.15.43:0/3772540155 shutdown_connections 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.565+0000 7f83bdc00640 1 -- 172.21.15.43:0/3772540155 wait complete. 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.567+0000 7f83bdc00640 1 Processor -- start 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.567+0000 7f83bdc00640 1 -- start start 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83980039a0 0x7f83b80c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 0x7f83b8159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8141650 con 0x7f83b8153680 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80c8fa0 con 0x7f83b81411d0 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.568+0000 7f83bdc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b814fc80 con 0x7f83980039a0 2024-01-16T15:18:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57422/0 (socket says 172.21.15.43:57422) 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 -- 172.21.15.43:0/2245536947 learned_addr learned my addr 172.21.15.43:0/2245536947 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83b7400640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 0x7f83b8159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83b6a00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83980039a0 0x7f83b80c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 -- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83980039a0 msgr2=0x7f83b80c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83980039a0 0x7f83b80c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 -- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 msgr2=0x7f83b8159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 0x7f83b8159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83ade00640 1 -- 172.21.15.43:0/2245536947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80c89e0 con 0x7f83b8153680 2024-01-16T15:18:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.569+0000 7f83b7400640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 0x7f83b8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83ade00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f83a000c770 tx=0x7f83a000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00074e0 con 0x7f83b8153680 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0007640 con 0x7f83b8153680 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0004a20 con 0x7f83b8153680 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80c8cc0 con 0x7f83b8153680 2024-01-16T15:18:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.570+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b8141a60 con 0x7f83b8153680 2024-01-16T15:18:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.571+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8067960 con 0x7f83b8153680 2024-01-16T15:18:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.573+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f83a0004b80 con 0x7f83b8153680 2024-01-16T15:18:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.573+0000 7f83af400640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 0x7f839006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.574+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f83a0087ac0 con 0x7f83b8153680 2024-01-16T15:18:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.574+0000 7f83b6a00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 0x7f839006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.575+0000 7f83b6a00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 0x7f839006e890 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f83a8036000 tx=0x7f83a802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.578+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83a0054380 con 0x7f83b8153680 2024-01-16T15:18:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.773+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b814fdf0 con 0x7f83b8153680 2024-01-16T15:18:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.776+0000 7f83af400640 1 -- 172.21.15.43:0/2245536947 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12983 (secure 0 0 0) 0x7f83a0018020 con 0x7f83b8153680 2024-01-16T15:18:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:35.766 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":104,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:33.192509+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6833/3755062795":"2024-01-17T15:18:30.175901+0000","172.21.15.69:6832/3755062795":"2024-01-17T15:18:30.175901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 msgr2=0x7f839006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 0x7f839006e890 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f83a8036000 tx=0x7f83a802e000 comp rx=0 tx=0).stop 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 msgr2=0x7f83b8140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f83a000c770 tx=0x7f83a000cc30 comp rx=0 tx=0).stop 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 shutdown_connections 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f839006c3d0 0x7f839006e890 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b81411d0 0x7f83b8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b8153680 0x7f83b8140c90 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 --2- 172.21.15.43:0/2245536947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83980039a0 0x7f83b80c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 >> 172.21.15.43:0/2245536947 conn(0x7f83b80b0d80 msgr2=0x7f83b806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.778+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 shutdown_connections 2024-01-16T15:18:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:35.779+0000 7f83bdc00640 1 -- 172.21.15.43:0/2245536947 wait complete. 2024-01-16T15:18:35.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:18:35.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:18:36.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.030+0000 7f5663800640 1 Processor -- start 2024-01-16T15:18:36.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.030+0000 7f5663800640 1 -- start start 2024-01-16T15:18:36.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.031+0000 7f5663800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f565c160bd0 0x7f565c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.031+0000 7f5663800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f565c0b48d0 0x7f565c0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.031+0000 7f5663800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0b23e0 con 0x7f565c0b3040 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.031+0000 7f5663800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0cdc70 con 0x7f565c0b48d0 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.031+0000 7f5663800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0b2870 con 0x7f565c160bd0 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f565c0b48d0 0x7f565c0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5661800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f565c0b3040 0x7f565c166280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56158/0 (socket says 172.21.15.43:56158) 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f565c0b48d0 0x7f565c0b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57424/0 (socket says 172.21.15.43:57424) 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5661800640 1 -- 172.21.15.43:0/2622965710 learned_addr learned my addr 172.21.15.43:0/2622965710 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 -- 172.21.15.43:0/2622965710 >> v2:172.21.15.69:3300/0 conn(0x7f565c160bd0 msgr2=0x7f565c0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 --2- 172.21.15.43:0/2622965710 >> v2:172.21.15.69:3300/0 conn(0x7f565c160bd0 0x7f565c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 -- 172.21.15.43:0/2622965710 >> v1:172.21.15.43:6789/0 conn(0x7f565c0b3040 legacy=0x7f565c166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.032+0000 7f5653c00640 1 -- 172.21.15.43:0/2622965710 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0d41b0 con 0x7f565c0b48d0 2024-01-16T15:18:36.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.034+0000 7f5653c00640 1 --2- 172.21.15.43:0/2622965710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b48d0 0x7f565c0b2b00 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f565c0cda60 tx=0x7f564402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14747ea7067c67e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.034+0000 7f565be00640 1 -- 172.21.15.43:0/2622965710 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564402e070 con 0x7f565c0b48d0 2024-01-16T15:18:36.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.034+0000 7f565be00640 1 -- 172.21.15.43:0/2622965710 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56440315a0 con 0x7f565c0b48d0 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.034+0000 7f565be00640 1 -- 172.21.15.43:0/2622965710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644031930 con 0x7f565c0b48d0 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 -- 172.21.15.43:0/2622965710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b48d0 msgr2=0x7f565c0b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 --2- 172.21.15.43:0/2622965710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b48d0 0x7f565c0b2b00 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f565c0cda60 tx=0x7f564402b840 comp rx=0 tx=0).stop 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 -- 172.21.15.43:0/2622965710 shutdown_connections 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 --2- 172.21.15.43:0/2622965710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b48d0 0x7f565c0b2b00 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 --2- 172.21.15.43:0/2622965710 >> v2:172.21.15.69:3300/0 conn(0x7f565c160bd0 0x7f565c0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 -- 172.21.15.43:0/2622965710 >> 172.21.15.43:0/2622965710 conn(0x7f565c09c5a0 msgr2=0x7f565c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 -- 172.21.15.43:0/2622965710 shutdown_connections 2024-01-16T15:18:36.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.035+0000 7f5663800640 1 -- 172.21.15.43:0/2622965710 wait complete. 2024-01-16T15:18:36.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.037+0000 7f5663800640 1 Processor -- start 2024-01-16T15:18:36.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.037+0000 7f5663800640 1 -- start start 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f565c0b48d0 0x7f565c0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565c160bd0 0x7f565c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0b7170 con 0x7f565c0b3040 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0b7ad0 con 0x7f565c0b48d0 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5663800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0b8cb0 con 0x7f565c160bd0 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5660e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5660e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57440/0 (socket says 172.21.15.43:57440) 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5660e00640 1 -- 172.21.15.43:0/2933959836 learned_addr learned my addr 172.21.15.43:0/2933959836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:36.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5653c00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f565c0b48d0 0x7f565c0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5661800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565c160bd0 0x7f565c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.038+0000 7f5660e00640 1 -- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565c160bd0 msgr2=0x7f565c0b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5660e00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565c160bd0 0x7f565c0b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5660e00640 1 -- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f565c0b48d0 msgr2=0x7f565c0b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5660e00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f565c0b48d0 0x7f565c0b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5660e00640 1 -- 172.21.15.43:0/2933959836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644009400 con 0x7f565c0b3040 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5660e00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f564c00b440 tx=0x7f564c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c011890 con 0x7f565c0b3040 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c0119f0 con 0x7f565c0b3040 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c011da0 con 0x7f565c0b3040 2024-01-16T15:18:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c0b64f0 con 0x7f565c0b3040 2024-01-16T15:18:36.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.039+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c0b6ab0 con 0x7f565c0b3040 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.041+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c0a1210 con 0x7f565c0b3040 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.042+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f564c01a400 con 0x7f565c0b3040 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.042+0000 7f5659600640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 0x7f563406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.042+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f564c088a40 con 0x7f565c0b3040 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.043+0000 7f5653c00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 0x7f563406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.044+0000 7f5653c00640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 0x7f563406e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f564403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.047+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f564c08de60 con 0x7f565c0b3040 2024-01-16T15:18:36.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.242+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f565c0665c0 con 0x7f565c0b3040 2024-01-16T15:18:36.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.243+0000 7f5659600640 1 -- 172.21.15.43:0/2933959836 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f564c015020 con 0x7f565c0b3040 2024-01-16T15:18:36.233 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.245+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 msgr2=0x7f563406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.245+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 0x7f563406e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f564403d040 comp rx=0 tx=0).stop 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.245+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 msgr2=0x7f565c0b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.245+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f564c00b440 tx=0x7f564c00b900 comp rx=0 tx=0).stop 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 shutdown_connections 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f563406c3a0 0x7f563406e860 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f565c0b48d0 0x7f565c0b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f565c160bd0 0x7f565c0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 --2- 172.21.15.43:0/2933959836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f565c0b3040 0x7f565c0b5e60 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 >> 172.21.15.43:0/2933959836 conn(0x7f565c09c5a0 msgr2=0x7f565c06be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:36.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 shutdown_connections 2024-01-16T15:18:36.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.246+0000 7f5663800640 1 -- 172.21.15.43:0/2933959836 wait complete. 2024-01-16T15:18:36.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:18:36.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.486+0000 7fe816c00640 1 Processor -- start 2024-01-16T15:18:36.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.486+0000 7fe816c00640 1 -- start start 2024-01-16T15:18:36.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.487+0000 7fe816c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8100c3e20 0x7fe810162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe816c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81006b8f0 con 0x7fe8100b01f0 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe816c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe810162bd0 con 0x7fe81006bd80 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe816c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81006b5f0 con 0x7fe8100c3e20 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8100c3e20 0x7fe810162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8100c3e20 0x7fe810162690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57454/0 (socket says 172.21.15.43:57454) 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe814e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe8100b01f0 0x7fe8100c4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56164/0 (socket says 172.21.15.43:56164) 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 -- 172.21.15.43:0/6774322 learned_addr learned my addr 172.21.15.43:0/6774322 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 -- 172.21.15.43:0/6774322 >> v1:172.21.15.69:6789/0 conn(0x7fe81006bd80 legacy=0x7fe81006c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 -- 172.21.15.43:0/6774322 >> v1:172.21.15.43:6789/0 conn(0x7fe8100b01f0 legacy=0x7fe8100c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe815800640 1 -- 172.21.15.43:0/6774322 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81006e570 con 0x7fe8100c3e20 2024-01-16T15:18:36.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.488+0000 7fe80fe00640 1 -- 172.21.15.43:0/6774322 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171267932 0 0) 0x7fe81006b8f0 con 0x7fe8100b01f0 2024-01-16T15:18:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe815800640 1 --2- 172.21.15.43:0/6774322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8100c3e20 0x7fe810162690 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe800009870 tx=0x7fe80002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3501f1c55219ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe80fe00640 1 -- 172.21.15.43:0/6774322 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe800031860 con 0x7fe8100c3e20 2024-01-16T15:18:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe80fe00640 1 -- 172.21.15.43:0/6774322 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8000319c0 con 0x7fe8100c3e20 2024-01-16T15:18:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe80fe00640 1 -- 172.21.15.43:0/6774322 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe800031d50 con 0x7fe8100c3e20 2024-01-16T15:18:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe816c00640 1 -- 172.21.15.43:0/6774322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8100c3e20 msgr2=0x7fe810162690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.490+0000 7fe816c00640 1 --2- 172.21.15.43:0/6774322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8100c3e20 0x7fe810162690 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe800009870 tx=0x7fe80002b900 comp rx=0 tx=0).stop 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.491+0000 7fe816c00640 1 -- 172.21.15.43:0/6774322 shutdown_connections 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.491+0000 7fe816c00640 1 --2- 172.21.15.43:0/6774322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8100c3e20 0x7fe810162690 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.491+0000 7fe816c00640 1 -- 172.21.15.43:0/6774322 >> 172.21.15.43:0/6774322 conn(0x7fe8100d0e50 msgr2=0x7fe8100d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.491+0000 7fe816c00640 1 -- 172.21.15.43:0/6774322 shutdown_connections 2024-01-16T15:18:36.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.491+0000 7fe816c00640 1 -- 172.21.15.43:0/6774322 wait complete. 2024-01-16T15:18:36.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.492+0000 7fe816c00640 1 Processor -- start 2024-01-16T15:18:36.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.493+0000 7fe816c00640 1 -- start start 2024-01-16T15:18:36.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.493+0000 7fe816c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100b01f0 0x7fe81015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100c3e20 0x7fe81015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100b0f00 con 0x7fe81006bd80 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe810155b50 con 0x7fe8100b01f0 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81015f000 con 0x7fe8100c3e20 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57464/0 (socket says 172.21.15.43:57464) 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 -- 172.21.15.43:0/4061549569 learned_addr learned my addr 172.21.15.43:0/4061549569 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe814e00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100b01f0 0x7fe81015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe816200640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100c3e20 0x7fe81015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 -- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100c3e20 msgr2=0x7fe81015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100c3e20 0x7fe81015ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 -- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100b01f0 msgr2=0x7fe81015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100b01f0 0x7fe81015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.494+0000 7fe815800640 1 -- 172.21.15.43:0/4061549569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000094d0 con 0x7fe81006bd80 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe815800640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe800009870 tx=0x7fe8000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe800031810 con 0x7fe81006bd80 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8000443f0 con 0x7fe81006bd80 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8000447a0 con 0x7fe81006bd80 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81015ed20 con 0x7fe81006bd80 2024-01-16T15:18:36.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.495+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8100b0b20 con 0x7fe81006bd80 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.496+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8100b1040 con 0x7fe81006bd80 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.497+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fe80004d450 con 0x7fe81006bd80 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.498+0000 7fe80d600640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 0x7fe7e806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.498+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe8000aae80 con 0x7fe81006bd80 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.498+0000 7fe814e00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 0x7fe7e806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:36.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.499+0000 7fe814e00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 0x7fe7e806e860 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8007fd0 tx=0x7fe7f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.503+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe800077740 con 0x7fe81006bd80 2024-01-16T15:18:36.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.711+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe81004e840 con 0x7fe81006bd80 2024-01-16T15:18:36.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.712+0000 7fe80d600640 1 -- 172.21.15.43:0/4061549569 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v104) v1 ==== 80+0+122 (secure 0 0 0) 0x7fe80007ad60 con 0x7fe81006bd80 2024-01-16T15:18:36.702 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6833/3755062795 2024-01-17T15:18:30.175901+0000 2024-01-16T15:18:36.702 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.69:6832/3755062795 2024-01-17T15:18:30.175901+0000 2024-01-16T15:18:36.702 INFO:teuthology.orchestra.run.smithi043.stderr:listed 2 entries 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 msgr2=0x7fe7e806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 0x7fe7e806e860 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8007fd0 tx=0x7fe7f8009040 comp rx=0 tx=0).stop 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 msgr2=0x7fe81015c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe800009870 tx=0x7fe8000046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 shutdown_connections 2024-01-16T15:18:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7e806c3a0 0x7fe7e806e860 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8100b01f0 0x7fe81015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8100c3e20 0x7fe81015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 --2- 172.21.15.43:0/4061549569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81006bd80 0x7fe81015c1e0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 >> 172.21.15.43:0/4061549569 conn(0x7fe8100d0e50 msgr2=0x7fe810160950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 shutdown_connections 2024-01-16T15:18:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:36.715+0000 7fe816c00640 1 -- 172.21.15.43:0/4061549569 wait complete. 2024-01-16T15:18:36.723 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:18:36.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:36.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:18:36.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[46566]: fuse finished with error 0 and tester_r 0 2024-01-16T15:18:36.823 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:18:36.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:36.823 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:18:37.006 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:18:37.006 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:37.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:18:37.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:18:37.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:37.066 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:18:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:18:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:18:37.125 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:18:37.125 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:37.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:18:37.183 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:18:37.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:37.183 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:18:37.234 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:18:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:18:37.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:18:37.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.475+0000 7f5d28e00640 1 Processor -- start 2024-01-16T15:18:37.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.476+0000 7f5d28e00640 1 -- start start 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d28e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d2406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d28e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d2406cae0 0x7f5d240b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d28e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d24161290 con 0x7f5d240b3640 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d28e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2415b410 con 0x7f5d2406cae0 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d28e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240b3ad0 con 0x7f5d2415b030 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d22800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5d240b3640 0x7f5d2415d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56176/0 (socket says 172.21.15.43:56176) 2024-01-16T15:18:37.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d22800640 1 -- 172.21.15.43:0/459027030 learned_addr learned my addr 172.21.15.43:0/459027030 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d21e00640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d2406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d1be00640 1 --2- 172.21.15.43:0/459027030 >> v2:172.21.15.69:3300/0 conn(0x7f5d2406cae0 0x7f5d240b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.478+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892846565 0 0) 0x7f5d24161290 con 0x7f5d240b3640 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d04003520 con 0x7f5d240b3640 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608294820 0 0) 0x7f5d04003520 con 0x7f5d240b3640 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 msgr2=0x7f5d2406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d2406c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 >> v2:172.21.15.69:3300/0 conn(0x7f5d2406cae0 msgr2=0x7f5d240b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 --2- 172.21.15.43:0/459027030 >> v2:172.21.15.69:3300/0 conn(0x7f5d2406cae0 0x7f5d240b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2415e580 con 0x7f5d240b3640 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f5d1c0019f0 con 0x7f5d240b3640 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 >> v1:172.21.15.43:6789/0 conn(0x7f5d240b3640 legacy=0x7f5d2415d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d04003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04003f40 0x7f5d040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d04004d00 0x7f5d04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2415b410 con 0x7f5d04004d00 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240b3ad0 con 0x7f5d04003f40 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.479+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0400b980 con 0x7f5d2415b030 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d04004d00 0x7f5d04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d1be00640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04003f40 0x7f5d040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d21e00640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d04003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 -- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 msgr2=0x7f5d04003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d2415b030 0x7f5d04003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 -- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04003f40 msgr2=0x7f5d040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:37.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04003f40 0x7f5d040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.480+0000 7f5d22800640 1 -- 172.21.15.43:0/459027030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04003520 con 0x7f5d04004d00 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.481+0000 7f5d22800640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d04004d00 0x7f5d04005180 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00a7b0 tx=0x7f5d040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0af18318c020b6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c034630 con 0x7f5d04004d00 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c02f070 con 0x7f5d04004d00 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d21400640 1 -- 172.21.15.43:0/459027030 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c034a10 con 0x7f5d04004d00 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 -- 172.21.15.43:0/459027030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d04004d00 msgr2=0x7f5d04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 --2- 172.21.15.43:0/459027030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d04004d00 0x7f5d04005180 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00a7b0 tx=0x7f5d040057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d22800640 1 -- 172.21.15.43:0/459027030 reap_dead start 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 -- 172.21.15.43:0/459027030 shutdown_connections 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 -- 172.21.15.43:0/459027030 >> 172.21.15.43:0/459027030 conn(0x7f5d240663d0 msgr2=0x7f5d240667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 -- 172.21.15.43:0/459027030 shutdown_connections 2024-01-16T15:18:37.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.482+0000 7f5d28e00640 1 -- 172.21.15.43:0/459027030 wait complete. 2024-01-16T15:18:37.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.484+0000 7f5d28e00640 1 Processor -- start 2024-01-16T15:18:37.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.484+0000 7f5d28e00640 1 -- start start 2024-01-16T15:18:37.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.485+0000 7f5d28e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04004d00 0x7f5d241446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.485+0000 7f5d28e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d240ccc70 0x7f5d241f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.485+0000 7f5d28e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.485+0000 7f5d28e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2414b110 con 0x7f5d240cf660 2024-01-16T15:18:37.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.485+0000 7f5d28e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240d0030 con 0x7f5d04004d00 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d28e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240ca540 con 0x7f5d240ccc70 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57492/0 (socket says 172.21.15.43:57492) 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 -- 172.21.15.43:0/689137551 learned_addr learned my addr 172.21.15.43:0/689137551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d21e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04004d00 0x7f5d241446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d1be00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d240ccc70 0x7f5d241f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 -- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d240ccc70 msgr2=0x7f5d241f6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d240ccc70 0x7f5d241f6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 -- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04004d00 msgr2=0x7f5d241446e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04004d00 0x7f5d241446e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.486+0000 7f5d22800640 1 -- 172.21.15.43:0/689137551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240d1820 con 0x7f5d240cf660 2024-01-16T15:18:37.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d22800640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c02cab0 tx=0x7f5d1c001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c034630 con 0x7f5d240cf660 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c034790 con 0x7f5d240cf660 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c034b00 con 0x7f5d240cf660 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d241f72d0 con 0x7f5d240cf660 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.487+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240d1a90 con 0x7f5d240cf660 2024-01-16T15:18:37.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.488+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240ca6b0 con 0x7f5d240cf660 2024-01-16T15:18:37.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.489+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f5d1c031030 con 0x7f5d240cf660 2024-01-16T15:18:37.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.490+0000 7f5d1a000640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 0x7f5cfc06e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.490+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5d1c0aaab0 con 0x7f5d240cf660 2024-01-16T15:18:37.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.490+0000 7f5d21e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 0x7f5cfc06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.491+0000 7f5d21e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 0x7f5cfc06e890 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009a10 tx=0x7f5d14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.494+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d1c077490 con 0x7f5d240cf660 2024-01-16T15:18:37.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.729+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d240ca6b0 con 0x7f5d240cf660 2024-01-16T15:18:37.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.730+0000 7f5d1a000640 1 -- 172.21.15.43:0/689137551 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f5d1c03ee60 con 0x7f5d240cf660 2024-01-16T15:18:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:37.720 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/1137375185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1137375185},{"type":"v1","addr":"172.21.15.43:6836","nonce":1137375185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2958556276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2958556276},{"type":"v1","addr":"172.21.15.43:6837","nonce":2958556276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6437,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2525540945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2525540945},{"type":"v1","addr":"172.21.15.69:6833","nonce":2525540945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/208706681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":208706681},{"type":"v1","addr":"172.21.15.69:6835","nonce":208706681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[]} 2024-01-16T15:18:37.721 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 69 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 msgr2=0x7f5cfc06e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 0x7f5cfc06e890 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009a10 tx=0x7f5d14008040 comp rx=0 tx=0).stop 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 msgr2=0x7f5d240cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c02cab0 tx=0x7f5d1c001360 comp rx=0 tx=0).stop 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 shutdown_connections 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5cfc06c3d0 0x7f5cfc06e890 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240cf660 0x7f5d240cfaf0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d240ccc70 0x7f5d241f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 --2- 172.21.15.43:0/689137551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04004d00 0x7f5d241446e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 >> 172.21.15.43:0/689137551 conn(0x7f5d240663d0 msgr2=0x7f5d2415b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 shutdown_connections 2024-01-16T15:18:37.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.733+0000 7f5d28e00640 1 -- 172.21.15.43:0/689137551 wait complete. 2024-01-16T15:18:37.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:18:37.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.978+0000 7f26c8400640 1 Processor -- start 2024-01-16T15:18:37.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.979+0000 7f26c8400640 1 -- start start 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.980+0000 7f26c8400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26c0160b10 0x7f26c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.980+0000 7f26c8400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c0072280 0x7f26c006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.980+0000 7f26c8400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0071780 con 0x7f26c006c900 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.980+0000 7f26c8400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c00b2320 con 0x7f26c0160b10 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.980+0000 7f26c8400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c006cd90 con 0x7f26c0072280 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26c0160b10 0x7f26c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f26c0160b10 0x7f26c006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57494/0 (socket says 172.21.15.43:57494) 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 -- 172.21.15.43:0/1620022315 learned_addr learned my addr 172.21.15.43:0/1620022315 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:37.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c6400640 1 --1- 172.21.15.43:0/1620022315 >> v1:172.21.15.43:6789/0 conn(0x7f26c006c900 0x7f26c0160fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56178/0 (socket says 172.21.15.43:56178) 2024-01-16T15:18:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5000640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c0072280 0x7f26c006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 -- 172.21.15.43:0/1620022315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c0072280 msgr2=0x7f26c006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c0072280 0x7f26c006c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 -- 172.21.15.43:0/1620022315 >> v1:172.21.15.43:6789/0 conn(0x7f26c006c900 legacy=0x7f26c0160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26bfe00640 1 -- 172.21.15.43:0/1620022315 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565443538 0 0) 0x7f26c0071780 con 0x7f26c006c900 2024-01-16T15:18:37.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.981+0000 7f26c5a00640 1 -- 172.21.15.43:0/1620022315 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00711c0 con 0x7f26c0160b10 2024-01-16T15:18:37.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.983+0000 7f26c5a00640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0160b10 0x7f26c006e420 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009870 tx=0x7f26b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16caf46971bcfc07 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.983+0000 7f26bfe00640 1 -- 172.21.15.43:0/1620022315 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0031640 con 0x7f26c0160b10 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.983+0000 7f26bfe00640 1 -- 172.21.15.43:0/1620022315 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b00317a0 con 0x7f26c0160b10 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.983+0000 7f26bfe00640 1 -- 172.21.15.43:0/1620022315 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0031b30 con 0x7f26c0160b10 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 -- 172.21.15.43:0/1620022315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0160b10 msgr2=0x7f26c006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0160b10 0x7f26c006e420 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009870 tx=0x7f26b002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 -- 172.21.15.43:0/1620022315 shutdown_connections 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c0072280 0x7f26c006c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 --2- 172.21.15.43:0/1620022315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0160b10 0x7f26c006e420 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 -- 172.21.15.43:0/1620022315 >> 172.21.15.43:0/1620022315 conn(0x7f26c00b0ba0 msgr2=0x7f26c00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 -- 172.21.15.43:0/1620022315 shutdown_connections 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.984+0000 7f26c8400640 1 -- 172.21.15.43:0/1620022315 wait complete. 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.986+0000 7f26c8400640 1 Processor -- start 2024-01-16T15:18:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.986+0000 7f26c8400640 1 -- start start 2024-01-16T15:18:37.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c006c900 0x7f26c009a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26c0160b10 0x7f26c00ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c00cbcb0 con 0x7f26c0072280 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c009a920 con 0x7f26c0160b10 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c8400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c0156bb0 con 0x7f26c006c900 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57504/0 (socket says 172.21.15.43:57504) 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.987+0000 7f26c5000640 1 -- 172.21.15.43:0/3026778829 learned_addr learned my addr 172.21.15.43:0/3026778829 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 -- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c006c900 msgr2=0x7f26c009a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c006c900 0x7f26c009a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 -- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26c0160b10 msgr2=0x7f26c00ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c6400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26c0160b10 0x7f26c00ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26c0160b10 0x7f26c00ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:37.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 -- 172.21.15.43:0/3026778829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b00094d0 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c5000640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f26a800d5f0 tx=0x7f26a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80125f0 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a8012750 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c0156e30 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.988+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8012b00 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.989+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c009ae60 con 0x7f26c0072280 2024-01-16T15:18:37.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.990+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c009d200 con 0x7f26c0072280 2024-01-16T15:18:37.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.991+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f26a8012c60 con 0x7f26c0072280 2024-01-16T15:18:37.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.992+0000 7f26bd600640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 0x7f269806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:37.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.992+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f26a8088cc0 con 0x7f26c0072280 2024-01-16T15:18:37.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.992+0000 7f26c5a00640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 0x7f269806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:37.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.993+0000 7f26c5a00640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 0x7f269806e860 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f26b0036000 tx=0x7f26b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:37.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:37.996+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26a8055580 con 0x7f26c0072280 2024-01-16T15:18:38.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.191+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f26c0156ff0 con 0x7f26c0072280 2024-01-16T15:18:38.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.197+0000 7f26bd600640 1 -- 172.21.15.43:0/3026778829 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f26a8058ba0 con 0x7f26c0072280 2024-01-16T15:18:38.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 msgr2=0x7f269806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 0x7f269806e860 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f26b0036000 tx=0x7f26b0030000 comp rx=0 tx=0).stop 2024-01-16T15:18:38.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 msgr2=0x7f26c00cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f26a800d5f0 tx=0x7f26a800dab0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 shutdown_connections 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f269806c3a0 0x7f269806e860 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26c0160b10 0x7f26c00ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26c0072280 0x7f26c00cdc10 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 --2- 172.21.15.43:0/3026778829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26c006c900 0x7f26c009a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 >> 172.21.15.43:0/3026778829 conn(0x7f26c00b0ba0 msgr2=0x7f26c015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 shutdown_connections 2024-01-16T15:18:38.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.201+0000 7f26c8400640 1 -- 172.21.15.43:0/3026778829 wait complete. 2024-01-16T15:18:38.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:18:38.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.450+0000 7fd135600640 1 Processor -- start 2024-01-16T15:18:38.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.450+0000 7fd135600640 1 -- start start 2024-01-16T15:18:38.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1300d0fa0 0x7fd1300d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd1300d1bc0 0x7fd1300b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300d2010 con 0x7fd1300d0fa0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300c6bf0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd135600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300d1400 con 0x7fd13015d3b0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.452+0000 7fd127e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1300d0fa0 0x7fd1300d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd1300d0fa0 0x7fd1300d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57506/0 (socket says 172.21.15.43:57506) 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 -- 172.21.15.43:0/2026167718 learned_addr learned my addr 172.21.15.43:0/2026167718 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd134c00640 1 --2- 172.21.15.43:0/2026167718 >> v2:172.21.15.69:3300/0 conn(0x7fd1300d1bc0 0x7fd1300b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd12fe00640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 -- 172.21.15.43:0/2026167718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 msgr2=0x7fd1300ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 -- 172.21.15.43:0/2026167718 >> v2:172.21.15.69:3300/0 conn(0x7fd1300d1bc0 msgr2=0x7fd1300b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 --2- 172.21.15.43:0/2026167718 >> v2:172.21.15.69:3300/0 conn(0x7fd1300d1bc0 0x7fd1300b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.453+0000 7fd127e00640 1 -- 172.21.15.43:0/2026167718 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1300cedc0 con 0x7fd1300d0fa0 2024-01-16T15:18:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.454+0000 7fd127e00640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d0fa0 0x7fd1300d1680 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fd1300cf310 tx=0x7fd11802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b8f5e5bb4e474d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.454+0000 7fd12f400640 1 -- 172.21.15.43:0/2026167718 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1180317e0 con 0x7fd1300d0fa0 2024-01-16T15:18:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd12f400640 1 -- 172.21.15.43:0/2026167718 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd118031940 con 0x7fd1300d0fa0 2024-01-16T15:18:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd12f400640 1 -- 172.21.15.43:0/2026167718 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118031cd0 con 0x7fd1300d0fa0 2024-01-16T15:18:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 -- 172.21.15.43:0/2026167718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d0fa0 msgr2=0x7fd1300d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d0fa0 0x7fd1300d1680 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fd1300cf310 tx=0x7fd11802b880 comp rx=0 tx=0).stop 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 -- 172.21.15.43:0/2026167718 shutdown_connections 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 --2- 172.21.15.43:0/2026167718 >> v2:172.21.15.69:3300/0 conn(0x7fd1300d1bc0 0x7fd1300b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d0fa0 0x7fd1300d1680 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 --2- 172.21.15.43:0/2026167718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 -- 172.21.15.43:0/2026167718 >> 172.21.15.43:0/2026167718 conn(0x7fd1300d30a0 msgr2=0x7fd1300d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 -- 172.21.15.43:0/2026167718 shutdown_connections 2024-01-16T15:18:38.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.455+0000 7fd135600640 1 -- 172.21.15.43:0/2026167718 wait complete. 2024-01-16T15:18:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.457+0000 7fd135600640 1 Processor -- start 2024-01-16T15:18:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.457+0000 7fd135600640 1 -- start start 2024-01-16T15:18:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 0x7fd13009cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13015c3a0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13009d320 con 0x7fd1300d0fa0 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd135600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300ca170 con 0x7fd13015d3b0 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd127e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd127e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57508/0 (socket says 172.21.15.43:57508) 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.458+0000 7fd127e00640 1 -- 172.21.15.43:0/1866924751 learned_addr learned my addr 172.21.15.43:0/1866924751 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd134c00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd12fe00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 0x7fd13009cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 -- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 msgr2=0x7fd1300c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 -- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 msgr2=0x7fd13009cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 0x7fd13009cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 -- 172.21.15.43:0/1866924751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1180094d0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd12fe00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 0x7fd13009cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd134c00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.459+0000 7fd127e00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fd11802b8c0 tx=0x7fd1180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.460+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118032880 con 0x7fd1300d1bc0 2024-01-16T15:18:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.460+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd118032a20 con 0x7fd1300d1bc0 2024-01-16T15:18:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.460+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1300a0c00 con 0x7fd1300d1bc0 2024-01-16T15:18:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.460+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118032dd0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.460+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1300ca710 con 0x7fd1300d1bc0 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.461+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13004e840 con 0x7fd1300d1bc0 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.462+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fd1180453f0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.463+0000 7fd12cc00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 0x7fd10806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.463+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd1180aa900 con 0x7fd1300d1bc0 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.463+0000 7fd12fe00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 0x7fd10806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.464+0000 7fd12fe00640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 0x7fd10806e860 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd120006fd0 tx=0x7fd120008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.468+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd118077280 con 0x7fd1300d1bc0 2024-01-16T15:18:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.661+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fd1300d3640 con 0x7fd1300d1bc0 2024-01-16T15:18:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.669+0000 7fd12cc00640 1 -- 172.21.15.43:0/1866924751 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v40) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd11807a8a0 con 0x7fd1300d1bc0 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 msgr2=0x7fd10806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 0x7fd10806e860 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd120006fd0 tx=0x7fd120008040 comp rx=0 tx=0).stop 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 msgr2=0x7fd13014f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fd11802b8c0 tx=0x7fd1180046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 shutdown_connections 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd10806c3a0 0x7fd10806e860 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd13015d3b0 0x7fd1300c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1300d1bc0 0x7fd13014f400 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 --2- 172.21.15.43:0/1866924751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1300d0fa0 0x7fd13009cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 >> 172.21.15.43:0/1866924751 conn(0x7fd1300d30a0 msgr2=0x7fd13006c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 shutdown_connections 2024-01-16T15:18:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.672+0000 7fd135600640 1 -- 172.21.15.43:0/1866924751 wait complete. 2024-01-16T15:18:38.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:18:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.905+0000 7fe7e8c00640 1 Processor -- start 2024-01-16T15:18:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.905+0000 7fe7e8c00640 1 -- start start 2024-01-16T15:18:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.906+0000 7fe7e8c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7e40d1d90 0x7fe7e40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.906+0000 7fe7e8c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 0x7fe7e40b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.906+0000 7fe7e8c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40c6bf0 con 0x7fe7e415d3b0 2024-01-16T15:18:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.906+0000 7fe7e8c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40d13f0 con 0x7fe7e40d1d90 2024-01-16T15:18:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.906+0000 7fe7e8c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40cedc0 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe7e40d1d90 0x7fe7e40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e3400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe7e415d3b0 0x7fe7e40d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56188/0 (socket says 172.21.15.43:56188) 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e3400640 1 -- 172.21.15.43:0/1238443632 learned_addr learned my addr 172.21.15.43:0/1238443632 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2a00640 1 -- 172.21.15.43:0/1238443632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 msgr2=0x7fe7e40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e3e00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 0x7fe7e40b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2a00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 0x7fe7e40b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2a00640 1 -- 172.21.15.43:0/1238443632 >> v1:172.21.15.43:6789/0 conn(0x7fe7e415d3b0 legacy=0x7fe7e40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2a00640 1 -- 172.21.15.43:0/1238443632 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40c6e40 con 0x7fe7e40d1d90 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e2000640 1 -- 172.21.15.43:0/1238443632 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942183824 0 0) 0x7fe7e40c6bf0 con 0x7fe7e415d3b0 2024-01-16T15:18:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.907+0000 7fe7e3e00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 0x7fe7e40b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.908+0000 7fe7e2a00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d1d90 0x7fe7e40ce6c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dce3daa476198b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e2000640 1 -- 172.21.15.43:0/1238443632 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc02e070 con 0x7fe7e40d1d90 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e2000640 1 -- 172.21.15.43:0/1238443632 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc0315a0 con 0x7fe7e40d1d90 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e2000640 1 -- 172.21.15.43:0/1238443632 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc031930 con 0x7fe7e40d1d90 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 -- 172.21.15.43:0/1238443632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d1d90 msgr2=0x7fe7e40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d1d90 0x7fe7e40ce6c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b840 comp rx=0 tx=0).stop 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 -- 172.21.15.43:0/1238443632 shutdown_connections 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d0fa0 0x7fe7e40b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/1238443632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d1d90 0x7fe7e40ce6c0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.909+0000 7fe7e8c00640 1 -- 172.21.15.43:0/1238443632 >> 172.21.15.43:0/1238443632 conn(0x7fe7e40d30a0 msgr2=0x7fe7e40d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.910+0000 7fe7e8c00640 1 -- 172.21.15.43:0/1238443632 shutdown_connections 2024-01-16T15:18:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.910+0000 7fe7e8c00640 1 -- 172.21.15.43:0/1238443632 wait complete. 2024-01-16T15:18:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.911+0000 7fe7e8c00640 1 Processor -- start 2024-01-16T15:18:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.911+0000 7fe7e8c00640 1 -- start start 2024-01-16T15:18:38.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d1d90 0x7fe7e40b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 0x7fe7e40b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40cbf80 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40b7870 con 0x7fe7e415d3b0 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e8c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40b67d0 con 0x7fe7e40d1d90 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.912+0000 7fe7e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57532/0 (socket says 172.21.15.43:57532) 2024-01-16T15:18:38.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 -- 172.21.15.43:0/2194071043 learned_addr learned my addr 172.21.15.43:0/2194071043 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e2a00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d1d90 0x7fe7e40b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3e00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 0x7fe7e40b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 -- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d1d90 msgr2=0x7fe7e40b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d1d90 0x7fe7e40b5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 -- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 msgr2=0x7fe7e40b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 0x7fe7e40b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 -- 172.21.15.43:0/2194071043 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc009400 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3e00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 0x7fe7e40b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.913+0000 7fe7e3400640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400c430 tx=0x7fe7d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.914+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40074e0 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.914+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d4007660 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.914+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4004a20 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.914+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e40b6ab0 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.914+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e40b7070 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.915+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e404e840 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.916+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fe7d4004b80 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.916+0000 7fe7db400640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 0x7fe7bc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.917+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe7d40888e0 con 0x7fe7e40d0fa0 2024-01-16T15:18:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.917+0000 7fe7e2a00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 0x7fe7bc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.917+0000 7fe7e2a00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 0x7fe7bc06e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036000 tx=0x7fe7cc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:38.921+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7d4054ec0 con 0x7fe7e40d0fa0 2024-01-16T15:18:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.115+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe7e40d1960 con 0x7fe7e40d0fa0 2024-01-16T15:18:39.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.121+0000 7fe7db400640 1 -- 172.21.15.43:0/2194071043 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe7d40584e0 con 0x7fe7e40d0fa0 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.123+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 msgr2=0x7fe7bc06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.123+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 0x7fe7bc06e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036000 tx=0x7fe7cc03d040 comp rx=0 tx=0).stop 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.123+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 msgr2=0x7fe7e414d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.123+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400c430 tx=0x7fe7d400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 shutdown_connections 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe7bc06c3a0 0x7fe7bc06e860 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e415d3b0 0x7fe7e40b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e40d1d90 0x7fe7e40b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 --2- 172.21.15.43:0/2194071043 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe7e40d0fa0 0x7fe7e414d800 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 >> 172.21.15.43:0/2194071043 conn(0x7fe7e40d30a0 msgr2=0x7fe7e40c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:39.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 shutdown_connections 2024-01-16T15:18:39.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.124+0000 7fe7e8c00640 1 -- 172.21.15.43:0/2194071043 wait complete. 2024-01-16T15:18:39.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:18:39.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.362+0000 7fb777e00640 1 Processor -- start 2024-01-16T15:18:39.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.363+0000 7fb777e00640 1 -- start start 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb777e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7780d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb777e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780cf0c0 con 0x7fb77815d5b0 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb777e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780d1e40 con 0x7fb7780b0640 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb777e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b01b0 con 0x7fb7780d14a0 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb776a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb77815d5b0 0x7fb7780d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56204/0 (socket says 172.21.15.43:56204) 2024-01-16T15:18:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.364+0000 7fb776a00640 1 -- 172.21.15.43:0/2156092157 learned_addr learned my addr 172.21.15.43:0/2156092157 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb776000640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7780d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352317185 0 0) 0x7fb7780cf0c0 con 0x7fb77815d5b0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758003520 con 0x7fb77815d5b0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046723798 0 0) 0x7fb758003520 con 0x7fb77815d5b0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 msgr2=0x7fb7780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7780d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 >> v1:172.21.15.69:6789/0 conn(0x7fb7780b0640 legacy=0x7fb7780b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77815d990 con 0x7fb77815d5b0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb768001670 con 0x7fb77815d5b0 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 >> v1:172.21.15.43:6789/0 conn(0x7fb77815d5b0 legacy=0x7fb7780d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb758003a10 0x7fb758003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb758004d00 0x7fb758005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b01b0 con 0x7fb758004d00 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75800bb10 con 0x7fb758003a10 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.365+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75800bca0 con 0x7fb7780d14a0 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb758004d00 0x7fb758005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb776000640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb776a00640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb758003a10 0x7fb758003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 -- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 msgr2=0x7fb7580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 -- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb758003a10 msgr2=0x7fb758003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb758003a10 0x7fb758003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb777400640 1 -- 172.21.15.43:0/2156092157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758003520 con 0x7fb758004d00 2024-01-16T15:18:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.366+0000 7fb776000640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7780d14a0 0x7fb7580047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.367+0000 7fb777400640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb758004d00 0x7fb758005180 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fb770001750 tx=0x7fb7580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7c40429acf4974 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700052d0 con 0x7fb758004d00 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77002e050 con 0x7fb758004d00 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb775600640 1 -- 172.21.15.43:0/2156092157 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700035f0 con 0x7fb758004d00 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 -- 172.21.15.43:0/2156092157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb758004d00 msgr2=0x7fb758005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 --2- 172.21.15.43:0/2156092157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb758004d00 0x7fb758005180 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fb770001750 tx=0x7fb7580057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777400640 1 -- 172.21.15.43:0/2156092157 reap_dead start 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 -- 172.21.15.43:0/2156092157 shutdown_connections 2024-01-16T15:18:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 -- 172.21.15.43:0/2156092157 >> 172.21.15.43:0/2156092157 conn(0x7fb77809c3b0 msgr2=0x7fb7780d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 -- 172.21.15.43:0/2156092157 shutdown_connections 2024-01-16T15:18:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.368+0000 7fb777e00640 1 -- 172.21.15.43:0/2156092157 wait complete. 2024-01-16T15:18:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.370+0000 7fb777e00640 1 Processor -- start 2024-01-16T15:18:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.370+0000 7fb777e00640 1 -- start start 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb758004d00 0x7fb778151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7780b2510 0x7fb7780b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780c6e40 con 0x7fb7780c90d0 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b3600 con 0x7fb7780b2510 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780c9a90 con 0x7fb758004d00 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.371+0000 7fb777400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57558/0 (socket says 172.21.15.43:57558) 2024-01-16T15:18:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 -- 172.21.15.43:0/1370286417 learned_addr learned my addr 172.21.15.43:0/1370286417 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb776000640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7780b2510 0x7fb7780b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb776a00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb758004d00 0x7fb778151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 -- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb758004d00 msgr2=0x7fb778151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb758004d00 0x7fb778151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 -- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7780b2510 msgr2=0x7fb7780b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7780b2510 0x7fb7780b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 -- 172.21.15.43:0/1370286417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b2d60 con 0x7fb7780c90d0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.372+0000 7fb777400640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb770005040 tx=0x7fb770001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.373+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770007a30 con 0x7fb7780c90d0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.373+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700067d0 con 0x7fb7780c90d0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.373+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770006b60 con 0x7fb7780c90d0 2024-01-16T15:18:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.373+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7780b3040 con 0x7fb7780c90d0 2024-01-16T15:18:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.373+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7780b2990 con 0x7fb7780c90d0 2024-01-16T15:18:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.374+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780c6ba0 con 0x7fb7780c90d0 2024-01-16T15:18:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.375+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fb770006cc0 con 0x7fb7780c90d0 2024-01-16T15:18:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.376+0000 7fb76ea00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 0x7fb75006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.376+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb7700aa040 con 0x7fb7780c90d0 2024-01-16T15:18:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.376+0000 7fb776a00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 0x7fb75006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.377+0000 7fb776a00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 0x7fb75006e860 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb768001050 tx=0x7fb768009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.380+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7780c6ba0 con 0x7fb7780c90d0 2024-01-16T15:18:39.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:39.648+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7fb77804e840 con 0x7fb7780c90d0 2024-01-16T15:18:40.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.416+0000 7fb76ea00640 1 -- 172.21.15.43:0/1370286417 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v815) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb770076900 con 0x7fb7780c90d0 2024-01-16T15:18:40.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 msgr2=0x7fb75006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 0x7fb75006e860 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb768001050 tx=0x7fb768009040 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 msgr2=0x7fb7780c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb770005040 tx=0x7fb770001120 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 shutdown_connections 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb75006c3a0 0x7fb75006e860 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7780b2510 0x7fb7780b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7780c90d0 0x7fb7780c9550 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 --2- 172.21.15.43:0/1370286417 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb758004d00 0x7fb778151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 >> 172.21.15.43:0/1370286417 conn(0x7fb77809c3b0 msgr2=0x7fb77815de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.419+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 shutdown_connections 2024-01-16T15:18:40.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.420+0000 7fb777e00640 1 -- 172.21.15.43:0/1370286417 wait complete. 2024-01-16T15:18:40.432 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:18:40.432 INFO:tasks.cephfs_test_runner:Starting test: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:18:40.432 INFO:tasks.cephfs_test_runner:test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:18:40.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:40.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.667+0000 7f8815c00640 1 Processor -- start 2024-01-16T15:18:40.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.667+0000 7f8815c00640 1 -- start start 2024-01-16T15:18:40.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f8815c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f881006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f8815c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8810165ca0 con 0x7f8810162670 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f8815c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8810165f80 con 0x7f881014b1c0 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f8815c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8810161e70 con 0x7f88100cf760 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f880f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8810162670 0x7f88100cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56218/0 (socket says 172.21.15.43:56218) 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.669+0000 7f880f400640 1 -- 172.21.15.43:0/587708799 learned_addr learned my addr 172.21.15.43:0/587708799 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f8815200640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f881006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541098758 0 0) 0x7f8810165ca0 con 0x7f8810162670 2024-01-16T15:18:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f0003520 con 0x7f8810162670 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291287370 0 0) 0x7f87f0003520 con 0x7f8810162670 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 msgr2=0x7f881006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f881006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 >> v1:172.21.15.69:6789/0 conn(0x7f881014b1c0 legacy=0x7f88101651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810162150 con 0x7f8810162670 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f87f8001670 con 0x7f8810162670 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 >> v1:172.21.15.43:6789/0 conn(0x7f8810162670 legacy=0x7f88100cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f0003a10 0x7f87f0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87f00047c0 0x7f87f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f87f0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8810161e70 con 0x7f87f00047c0 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000bb40 con 0x7f87f0003a10 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.670+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000bd00 con 0x7f88100cf760 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87f00047c0 0x7f87f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f8815200640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f87f0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880fe00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f0003a10 0x7f87f0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 -- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 msgr2=0x7f87f0005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88100cf760 0x7f87f0005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 -- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f0003a10 msgr2=0x7f87f0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87f0003a10 0x7f87f0003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.671+0000 7f880f400640 1 -- 172.21.15.43:0/587708799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0003520 con 0x7f87f00047c0 2024-01-16T15:18:40.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.672+0000 7f880f400640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87f00047c0 0x7f87f0004c40 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c210 tx=0x7f87f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285255f649ca67c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:40.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8005ad0 con 0x7f87f00047c0 2024-01-16T15:18:40.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8005c30 con 0x7f87f00047c0 2024-01-16T15:18:40.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f880ea00640 1 -- 172.21.15.43:0/587708799 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f802cad0 con 0x7f87f00047c0 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 -- 172.21.15.43:0/587708799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87f00047c0 msgr2=0x7f87f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 --2- 172.21.15.43:0/587708799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87f00047c0 0x7f87f0004c40 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c210 tx=0x7f87f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815200640 1 -- 172.21.15.43:0/587708799 reap_dead start 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 -- 172.21.15.43:0/587708799 shutdown_connections 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 -- 172.21.15.43:0/587708799 >> 172.21.15.43:0/587708799 conn(0x7f881006a860 msgr2=0x7f881006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 -- 172.21.15.43:0/587708799 shutdown_connections 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.673+0000 7f8815c00640 1 -- 172.21.15.43:0/587708799 wait complete. 2024-01-16T15:18:40.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.675+0000 7f8815c00640 1 Processor -- start 2024-01-16T15:18:40.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.675+0000 7f8815c00640 1 -- start start 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 0x7f88100cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88100cbca0 0x7f881015f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cce30 con 0x7f881015f6d0 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cc1b0 con 0x7f88100cbca0 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cc490 con 0x7f87f00047c0 2024-01-16T15:18:40.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.676+0000 7f8815200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57576/0 (socket says 172.21.15.43:57576) 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 -- 172.21.15.43:0/4119692840 learned_addr learned my addr 172.21.15.43:0/4119692840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f880fe00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 0x7f88100cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f880f400640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88100cbca0 0x7f881015f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 -- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 msgr2=0x7f88100cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 0x7f88100cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 -- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88100cbca0 msgr2=0x7f881015f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88100cbca0 0x7f881015f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 -- 172.21.15.43:0/4119692840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100cc770 con 0x7f881015f6d0 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f880fe00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 0x7f88100cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:40.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.677+0000 7f8815200640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f88100ae760 tx=0x7f88080115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:40.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.678+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800e070 con 0x7f881015f6d0 2024-01-16T15:18:40.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.678+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808011ed0 con 0x7f881015f6d0 2024-01-16T15:18:40.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.678+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808003990 con 0x7f881015f6d0 2024-01-16T15:18:40.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.678+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88100cca50 con 0x7f881015f6d0 2024-01-16T15:18:40.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.678+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88100cd250 con 0x7f881015f6d0 2024-01-16T15:18:40.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.679+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100ccf70 con 0x7f881015f6d0 2024-01-16T15:18:40.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.680+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f8808003b10 con 0x7f881015f6d0 2024-01-16T15:18:40.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.681+0000 7f8807e00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 0x7f87e806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:40.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.681+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f88080890a0 con 0x7f881015f6d0 2024-01-16T15:18:40.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.681+0000 7f880fe00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 0x7f87e806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:40.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.682+0000 7f880fe00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 0x7f87e806e860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f88000015f0 tx=0x7f8800009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:40.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.685+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88100ccf70 con 0x7f881015f6d0 2024-01-16T15:18:40.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.883+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88100ccf70 con 0x7f881015f6d0 2024-01-16T15:18:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.885+0000 7f8807e00640 1 -- 172.21.15.43:0/4119692840 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12983 (secure 0 0 0) 0x7f88100ccf70 con 0x7f881015f6d0 2024-01-16T15:18:40.875 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:40.875 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":104,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:33.192509+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6833/3755062795":"2024-01-17T15:18:30.175901+0000","172.21.15.69:6832/3755062795":"2024-01-17T15:18:30.175901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 msgr2=0x7f87e806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 0x7f87e806e860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f88000015f0 tx=0x7f8800009040 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 msgr2=0x7f881015fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f88100ae760 tx=0x7f88080115a0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 shutdown_connections 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f87e806c3a0 0x7f87e806e860 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88100cbca0 0x7f881015f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f881015f6d0 0x7f881015fb50 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 --2- 172.21.15.43:0/4119692840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87f00047c0 0x7f88100cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.887+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 >> 172.21.15.43:0/4119692840 conn(0x7f881006a860 msgr2=0x7f88100b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.888+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 shutdown_connections 2024-01-16T15:18:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:40.888+0000 7f8815c00640 1 -- 172.21.15.43:0/4119692840 wait complete. 2024-01-16T15:18:40.898 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:18:40.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:18:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.133+0000 7f9197a00640 1 Processor -- start 2024-01-16T15:18:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.133+0000 7f9197a00640 1 -- start start 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9197a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91900ae270 0x7f91900abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9197a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919015b730 con 0x7f919015b350 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9197a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91900ac740 con 0x7f919009cb30 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9197a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91900ac460 con 0x7f91900ae270 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9195000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f919015b350 0x7f91900b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56232/0 (socket says 172.21.15.43:56232) 2024-01-16T15:18:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.135+0000 7f9195000640 1 -- 172.21.15.43:0/2027464946 learned_addr learned my addr 172.21.15.43:0/2027464946 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f9195a00640 1 --2- 172.21.15.43:0/2027464946 >> v2:172.21.15.69:3300/0 conn(0x7f91900ae270 0x7f91900abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758486034 0 0) 0x7f9180001110 con 0x7f919015b350 2024-01-16T15:18:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9170003520 con 0x7f919015b350 2024-01-16T15:18:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1661147748 0 0) 0x7f9170003520 con 0x7f919015b350 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 >> v2:172.21.15.69:3300/0 conn(0x7f91900ae270 msgr2=0x7f91900abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 --2- 172.21.15.43:0/2027464946 >> v2:172.21.15.69:3300/0 conn(0x7f91900ae270 0x7f91900abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 >> v1:172.21.15.69:6789/0 conn(0x7f919009cb30 legacy=0x7f91900adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900b2a50 con 0x7f919015b350 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f9180001800 con 0x7f919015b350 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 >> v1:172.21.15.43:6789/0 conn(0x7f919015b350 legacy=0x7f91900b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 0x7f9170003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91700047c0 0x7f9170004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170005180 0x7f917000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91900ac460 con 0x7f9170003970 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917000c020 con 0x7f9170005180 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.136+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917000c1e0 con 0x7f91700047c0 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 0x7f9170003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f918fe00640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91700047c0 0x7f9170004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195a00640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170005180 0x7f917000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 -- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91700047c0 msgr2=0x7f9170004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91700047c0 0x7f9170004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 -- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170005180 msgr2=0x7f917000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170005180 0x7f917000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195000640 1 -- 172.21.15.43:0/2027464946 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170003520 con 0x7f9170003970 2024-01-16T15:18:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.137+0000 7f9195a00640 1 -- 172.21.15.43:0/2027464946 reap_dead start 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f9195000640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 0x7f9170003d90 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f918000c240 tx=0x7f91700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4063dfbbb85ce758 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180005ad0 con 0x7f9170003970 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9180005c50 con 0x7f9170003970 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f918f400640 1 -- 172.21.15.43:0/2027464946 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918002ca90 con 0x7f9170003970 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f9197a00640 1 -- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 msgr2=0x7f9170003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.139+0000 7f9197a00640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 0x7f9170003d90 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f918000c240 tx=0x7f91700057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.140+0000 7f9197a00640 1 -- 172.21.15.43:0/2027464946 shutdown_connections 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.140+0000 7f9197a00640 1 --2- 172.21.15.43:0/2027464946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9170003970 0x7f9170003d90 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.140+0000 7f9197a00640 1 -- 172.21.15.43:0/2027464946 >> 172.21.15.43:0/2027464946 conn(0x7f91900b0ba0 msgr2=0x7f91900b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.140+0000 7f9197a00640 1 -- 172.21.15.43:0/2027464946 shutdown_connections 2024-01-16T15:18:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.140+0000 7f9197a00640 1 -- 172.21.15.43:0/2027464946 wait complete. 2024-01-16T15:18:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.141+0000 7f9197a00640 1 Processor -- start 2024-01-16T15:18:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.142+0000 7f9197a00640 1 -- start start 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170003970 0x7f9190070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91901515b0 0x7f9190151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190151f70 con 0x7f919015db30 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190071480 con 0x7f9170003970 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f9197a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9190071760 con 0x7f91901515b0 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57588/0 (socket says 172.21.15.43:57588) 2024-01-16T15:18:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 -- 172.21.15.43:0/489269173 learned_addr learned my addr 172.21.15.43:0/489269173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 -- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91901515b0 msgr2=0x7f9190151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91901515b0 0x7f9190151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 -- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170003970 msgr2=0x7f9190070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170003970 0x7f9190070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.143+0000 7f918fe00640 1 -- 172.21.15.43:0/489269173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190071a40 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f918fe00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f917800e4d0 tx=0x7f917800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780052e0 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9178005440 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178003810 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9190071d20 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.144+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919014c5f0 con 0x7f919015db30 2024-01-16T15:18:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.145+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919009d200 con 0x7f919015db30 2024-01-16T15:18:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.146+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9178003970 con 0x7f919015db30 2024-01-16T15:18:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.147+0000 7f918cc00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 0x7f916806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.147+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f91780882f0 con 0x7f919015db30 2024-01-16T15:18:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.147+0000 7f9195000640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 0x7f916806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.148+0000 7f9195000640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 0x7f916806e860 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9180034000 tx=0x7f9180007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.152+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9178054bb0 con 0x7f919015db30 2024-01-16T15:18:41.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.346+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91900ac740 con 0x7f919015db30 2024-01-16T15:18:41.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.347+0000 7f918cc00640 1 -- 172.21.15.43:0/489269173 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91900ac740 con 0x7f919015db30 2024-01-16T15:18:41.337 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 msgr2=0x7f916806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 0x7f916806e860 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9180034000 tx=0x7f9180007000 comp rx=0 tx=0).stop 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 msgr2=0x7f919014c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f917800e4d0 tx=0x7f917800e990 comp rx=0 tx=0).stop 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 shutdown_connections 2024-01-16T15:18:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f916806c3a0 0x7f916806e860 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91901515b0 0x7f9190151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f919015db30 0x7f919014c0b0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 --2- 172.21.15.43:0/489269173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9170003970 0x7f9190070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 >> 172.21.15.43:0/489269173 conn(0x7f91900b0ba0 msgr2=0x7f91900adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 shutdown_connections 2024-01-16T15:18:41.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.350+0000 7f9197a00640 1 -- 172.21.15.43:0/489269173 wait complete. 2024-01-16T15:18:41.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled' 2024-01-16T15:18:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.580+0000 7fd4cf800640 1 Processor -- start 2024-01-16T15:18:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.581+0000 7fd4cf800640 1 -- start start 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cf800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd4c8159790 0x7fd4c815de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cf800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd4c80b42f0 0x7fd4c806a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cf800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c815e340 con 0x7fd4c80ce920 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cf800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c806dd10 con 0x7fd4c8159790 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cf800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c806dff0 con 0x7fd4c80b42f0 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.582+0000 7fd4cce00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd4c8159790 0x7fd4c815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4c7e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd4c80ce920 0x7fd4c80b3be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:56238/0 (socket says 172.21.15.43:56238) 2024-01-16T15:18:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4c7e00640 1 -- 172.21.15.43:0/2246422719 learned_addr learned my addr 172.21.15.43:0/2246422719 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4cd800640 1 --2- 172.21.15.43:0/2246422719 >> v2:172.21.15.69:3300/0 conn(0x7fd4c80b42f0 0x7fd4c806a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4cce00640 1 -- 172.21.15.43:0/2246422719 >> v2:172.21.15.69:3300/0 conn(0x7fd4c80b42f0 msgr2=0x7fd4c806a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4cce00640 1 --2- 172.21.15.43:0/2246422719 >> v2:172.21.15.69:3300/0 conn(0x7fd4c80b42f0 0x7fd4c806a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4cce00640 1 -- 172.21.15.43:0/2246422719 >> v1:172.21.15.43:6789/0 conn(0x7fd4c80ce920 legacy=0x7fd4c80b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4c7400640 1 -- 172.21.15.43:0/2246422719 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977390447 0 0) 0x7fd4c815e340 con 0x7fd4c80ce920 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.583+0000 7fd4cce00640 1 -- 172.21.15.43:0/2246422719 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c806e2d0 con 0x7fd4c8159790 2024-01-16T15:18:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.584+0000 7fd4cce00640 1 --2- 172.21.15.43:0/2246422719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c8159790 0x7fd4c815de00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80ce1f0 tx=0x7fd4b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcda5d8a5510e6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4c7400640 1 -- 172.21.15.43:0/2246422719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8031830 con 0x7fd4c8159790 2024-01-16T15:18:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4c7400640 1 -- 172.21.15.43:0/2246422719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b8031990 con 0x7fd4c8159790 2024-01-16T15:18:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4c7400640 1 -- 172.21.15.43:0/2246422719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8031d20 con 0x7fd4c8159790 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 -- 172.21.15.43:0/2246422719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c8159790 msgr2=0x7fd4c815de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2246422719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c8159790 0x7fd4c815de00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80ce1f0 tx=0x7fd4b802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 -- 172.21.15.43:0/2246422719 shutdown_connections 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2246422719 >> v2:172.21.15.69:3300/0 conn(0x7fd4c80b42f0 0x7fd4c806a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2246422719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c8159790 0x7fd4c815de00 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 -- 172.21.15.43:0/2246422719 >> 172.21.15.43:0/2246422719 conn(0x7fd4c80b23a0 msgr2=0x7fd4c80cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 -- 172.21.15.43:0/2246422719 shutdown_connections 2024-01-16T15:18:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.585+0000 7fd4cf800640 1 -- 172.21.15.43:0/2246422719 wait complete. 2024-01-16T15:18:41.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.587+0000 7fd4cf800640 1 Processor -- start 2024-01-16T15:18:41.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.587+0000 7fd4cf800640 1 -- start start 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4c80ce920 0x7fd4c81f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4c8159790 0x7fd4c81f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8153ed0 con 0x7fd4c80b42f0 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81f4620 con 0x7fd4c80ce920 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.588+0000 7fd4cf800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81f4060 con 0x7fd4c8159790 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57600/0 (socket says 172.21.15.43:57600) 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 -- 172.21.15.43:0/2382464805 learned_addr learned my addr 172.21.15.43:0/2382464805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 -- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4c8159790 msgr2=0x7fd4c81f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4c7e00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4c80ce920 0x7fd4c81f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4c8159790 0x7fd4c81f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 -- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4c80ce920 msgr2=0x7fd4c81f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:41.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4c80ce920 0x7fd4c81f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:41.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 -- 172.21.15.43:0/2382464805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8009400 con 0x7fd4c80b42f0 2024-01-16T15:18:41.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.589+0000 7fd4cce00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009890 tx=0x7fd4b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.590+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80317d0 con 0x7fd4c80b42f0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.590+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b803aa60 con 0x7fd4c80b42f0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.590+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b803ae10 con 0x7fd4c80b42f0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.590+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c81f4340 con 0x7fd4c80b42f0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.590+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81f7060 con 0x7fd4c80b42f0 2024-01-16T15:18:41.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.591+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c8074770 con 0x7fd4c80b42f0 2024-01-16T15:18:41.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.592+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fd4b804b3b0 con 0x7fd4c80b42f0 2024-01-16T15:18:41.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.593+0000 7fd4c4c00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 0x7fd4a006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:41.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.593+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd4b80aa6e0 con 0x7fd4c80b42f0 2024-01-16T15:18:41.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.593+0000 7fd4c7e00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 0x7fd4a006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.594+0000 7fd4c7e00640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 0x7fd4a006e860 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0007fd0 tx=0x7fd4b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.597+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4b8077120 con 0x7fd4c80b42f0 2024-01-16T15:18:41.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:41.865+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]} v 0) v1 -- 0x7fd4c80b1580 con 0x7fd4c80b42f0 2024-01-16T15:18:42.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.427+0000 7fd4c4c00640 1 -- 172.21.15.43:0/2382464805 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]}]=0 v817) v1 ==== 158+0+0 (secure 0 0 0) 0x7fd4b80350e0 con 0x7fd4c80b42f0 2024-01-16T15:18:42.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.430+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 msgr2=0x7fd4a006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.430+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 0x7fd4a006e860 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0007fd0 tx=0x7fd4b0009040 comp rx=0 tx=0).stop 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 msgr2=0x7fd4c8153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009890 tx=0x7fd4b80046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 shutdown_connections 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd4a006c3a0 0x7fd4a006e860 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4c80ce920 0x7fd4c81f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4c8159790 0x7fd4c81f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 --2- 172.21.15.43:0/2382464805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c80b42f0 0x7fd4c8153990 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 >> 172.21.15.43:0/2382464805 conn(0x7fd4c80b23a0 msgr2=0x7fd4c814e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:42.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 shutdown_connections 2024-01-16T15:18:42.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.431+0000 7fd4cf800640 1 -- 172.21.15.43:0/2382464805 wait complete. 2024-01-16T15:18:42.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:18:42.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.688+0000 7f9e63e00640 1 Processor -- start 2024-01-16T15:18:42.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.688+0000 7f9e63e00640 1 -- start start 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.690+0000 7f9e63e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e6413cf40 0x7f9e6406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.690+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 0x7f9e640cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.690+0000 7f9e63e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e64140430 con 0x7f9e6406d910 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.690+0000 7f9e63e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6406dd90 con 0x7f9e6413cf40 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.690+0000 7f9e63e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64071220 con 0x7f9e6413fe10 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e6413cf40 0x7f9e6406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e6413cf40 0x7f9e6406d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57614/0 (socket says 172.21.15.43:57614) 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 -- 172.21.15.43:0/1825464697 learned_addr learned my addr 172.21.15.43:0/1825464697 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 -- 172.21.15.43:0/1825464697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 msgr2=0x7f9e640cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e63400640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 0x7f9e640cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 0x7f9e640cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 -- 172.21.15.43:0/1825464697 >> v1:172.21.15.43:6789/0 conn(0x7f9e6406d910 legacy=0x7f9e6413f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e62a00640 1 -- 172.21.15.43:0/1825464697 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64071500 con 0x7f9e6413cf40 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e61600640 1 -- 172.21.15.43:0/1825464697 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489945259 0 0) 0x7f9e64140430 con 0x7f9e6406d910 2024-01-16T15:18:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.691+0000 7f9e63400640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 0x7f9e640cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:42.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e62a00640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6406d3d0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009870 tx=0x7f9e5402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8ffe42fcbd7deb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:42.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e61600640 1 -- 172.21.15.43:0/1825464697 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54030840 con 0x7f9e6413cf40 2024-01-16T15:18:42.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e61600640 1 -- 172.21.15.43:0/1825464697 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e540309a0 con 0x7f9e6413cf40 2024-01-16T15:18:42.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e61600640 1 -- 172.21.15.43:0/1825464697 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54030d30 con 0x7f9e6413cf40 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e63e00640 1 -- 172.21.15.43:0/1825464697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 msgr2=0x7f9e6406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.693+0000 7f9e63e00640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6406d3d0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009870 tx=0x7f9e5402b980 comp rx=0 tx=0).stop 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 -- 172.21.15.43:0/1825464697 shutdown_connections 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6413fe10 0x7f9e640cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 --2- 172.21.15.43:0/1825464697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6406d3d0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 -- 172.21.15.43:0/1825464697 >> 172.21.15.43:0/1825464697 conn(0x7f9e640b0aa0 msgr2=0x7f9e640b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 -- 172.21.15.43:0/1825464697 shutdown_connections 2024-01-16T15:18:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.694+0000 7f9e63e00640 1 -- 172.21.15.43:0/1825464697 wait complete. 2024-01-16T15:18:42.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.695+0000 7f9e63e00640 1 Processor -- start 2024-01-16T15:18:42.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 -- start start 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6406d910 0x7f9e640702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6413fe10 0x7f9e64156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64141100 con 0x7f9e6413cf40 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64140ad0 con 0x7f9e6413fe10 2024-01-16T15:18:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.696+0000 7f9e63e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6415e280 con 0x7f9e6406d910 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57620/0 (socket says 172.21.15.43:57620) 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 -- 172.21.15.43:0/425768428 learned_addr learned my addr 172.21.15.43:0/425768428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62a00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6406d910 0x7f9e640702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e63400640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6413fe10 0x7f9e64156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 -- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6406d910 msgr2=0x7f9e640702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6406d910 0x7f9e640702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 -- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6413fe10 msgr2=0x7f9e64156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6413fe10 0x7f9e64156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.697+0000 7f9e62000640 1 -- 172.21.15.43:0/425768428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540094d0 con 0x7f9e6413cf40 2024-01-16T15:18:42.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e62000640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c460 tx=0x7f9e4c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:42.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c005480 con 0x7f9e6413cf40 2024-01-16T15:18:42.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c0035f0 con 0x7f9e6413cf40 2024-01-16T15:18:42.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0039a0 con 0x7f9e6413cf40 2024-01-16T15:18:42.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64072160 con 0x7f9e6413cf40 2024-01-16T15:18:42.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.698+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64140d20 con 0x7f9e6413cf40 2024-01-16T15:18:42.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.699+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64141240 con 0x7f9e6413cf40 2024-01-16T15:18:42.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.700+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9e4c003b00 con 0x7f9e6413cf40 2024-01-16T15:18:42.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.701+0000 7f9e5aa00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 0x7f9e3c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:42.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.701+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9e4c087f00 con 0x7f9e6413cf40 2024-01-16T15:18:42.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.701+0000 7f9e62a00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 0x7f9e3c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:42.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.702+0000 7f9e62a00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 0x7f9e3c06e860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9e54036000 tx=0x7f9e54008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:42.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.705+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e64141240 con 0x7f9e6413cf40 2024-01-16T15:18:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.898+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9e64141240 con 0x7f9e6413cf40 2024-01-16T15:18:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.906+0000 7f9e5aa00640 1 -- 172.21.15.43:0/425768428 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9e64141240 con 0x7f9e6413cf40 2024-01-16T15:18:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 msgr2=0x7f9e3c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 0x7f9e3c06e860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9e54036000 tx=0x7f9e54008030 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 msgr2=0x7f9e6415d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c460 tx=0x7f9e4c00c920 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 shutdown_connections 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e3c06c3a0 0x7f9e3c06e860 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e6413fe10 0x7f9e64156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e6413cf40 0x7f9e6415d850 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 --2- 172.21.15.43:0/425768428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e6406d910 0x7f9e640702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.909+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 >> 172.21.15.43:0/425768428 conn(0x7f9e640b0aa0 msgr2=0x7f9e640b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.910+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 shutdown_connections 2024-01-16T15:18:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:42.910+0000 7f9e63e00640 1 -- 172.21.15.43:0/425768428 wait complete. 2024-01-16T15:18:42.920 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:18:42.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:18:43.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:18:43.153 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:18:43.153 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:18:43.153 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:43.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:18:43.212 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:18:43.212 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:43.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:18:43.267 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:18:43.267 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:43.329 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:18:43.329 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:18:43.329 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:18:43.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:18:43.385 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:18:43.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:18:43.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.626+0000 7fab84c00640 1 Processor -- start 2024-01-16T15:18:43.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.627+0000 7fab84c00640 1 -- start start 2024-01-16T15:18:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab84c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab8015d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab84c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab80161670 con 0x7fab80157770 2024-01-16T15:18:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab84c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8015e610 con 0x7fab8015ddb0 2024-01-16T15:18:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab84c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab80160d40 con 0x7fab8006d250 2024-01-16T15:18:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab7f400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fab80157770 0x7fab8006cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36484/0 (socket says 172.21.15.43:36484) 2024-01-16T15:18:43.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab7f400640 1 -- 172.21.15.43:0/817852389 learned_addr learned my addr 172.21.15.43:0/817852389 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:43.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab77e00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab8015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809409207 0 0) 0x7fab80161670 con 0x7fab80157770 2024-01-16T15:18:43.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.628+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60003520 con 0x7fab80157770 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168267498 0 0) 0x7fab60003520 con 0x7fab80157770 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 msgr2=0x7fab8015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab8015d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 >> v1:172.21.15.69:6789/0 conn(0x7fab8015ddb0 legacy=0x7fab80070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80161020 con 0x7fab80157770 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fab70001670 con 0x7fab80157770 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 >> v1:172.21.15.43:6789/0 conn(0x7fab80157770 legacy=0x7fab8006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab60003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab60003f90 0x7fab600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab60004d00 0x7fab60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80160d40 con 0x7fab60004d00 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000bae0 con 0x7fab60003f90 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000bca0 con 0x7fab8006d250 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.629+0000 7fab7fe00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab60004d00 0x7fab60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7f400640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab60003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab77e00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab60003f90 0x7fab600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7fe00640 1 -- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 msgr2=0x7fab60003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7fe00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab8006d250 0x7fab60003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7fe00640 1 -- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab60003f90 msgr2=0x7fab600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7fe00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab60003f90 0x7fab600047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.630+0000 7fab7fe00640 1 -- 172.21.15.43:0/817852389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60003520 con 0x7fab60004d00 2024-01-16T15:18:43.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.631+0000 7fab7fe00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab60004d00 0x7fab60005180 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fab78001e50 tx=0x7fab600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e939652bcc00a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.631+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78003b70 con 0x7fab60004d00 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.631+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78003cd0 con 0x7fab60004d00 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.631+0000 7fab7ea00640 1 -- 172.21.15.43:0/817852389 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005310 con 0x7fab60004d00 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 -- 172.21.15.43:0/817852389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab60004d00 msgr2=0x7fab60005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 --2- 172.21.15.43:0/817852389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab60004d00 0x7fab60005180 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fab78001e50 tx=0x7fab600057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab7fe00640 1 -- 172.21.15.43:0/817852389 reap_dead start 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 -- 172.21.15.43:0/817852389 shutdown_connections 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 -- 172.21.15.43:0/817852389 >> 172.21.15.43:0/817852389 conn(0x7fab8006a860 msgr2=0x7fab8006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 -- 172.21.15.43:0/817852389 shutdown_connections 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.632+0000 7fab84c00640 1 -- 172.21.15.43:0/817852389 wait complete. 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.634+0000 7fab84c00640 1 Processor -- start 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.634+0000 7fab84c00640 1 -- start start 2024-01-16T15:18:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 0x7fab8014bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 0x7fab8015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800b2ea0 con 0x7fab800b24e0 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8015f1f0 con 0x7fab801606d0 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab84c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8015f4d0 con 0x7fab60004d00 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39140/0 (socket says 172.21.15.43:39140) 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 -- 172.21.15.43:0/2309518734 learned_addr learned my addr 172.21.15.43:0/2309518734 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7f400640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 0x7fab8014bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab77e00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 0x7fab8015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 -- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 msgr2=0x7fab8014bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 0x7fab8014bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.636+0000 7fab7fe00640 1 -- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 msgr2=0x7fab8015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab7fe00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 0x7fab8015e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab7fe00640 1 -- 172.21.15.43:0/2309518734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80161c80 con 0x7fab800b24e0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab7f400640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 0x7fab8014bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab77e00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 0x7fab8015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab7fe00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fab78001f70 tx=0x7fab780014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7803e070 con 0x7fab800b24e0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78006b50 con 0x7fab800b24e0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8015f7b0 con 0x7fab800b24e0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78007030 con 0x7fab800b24e0 2024-01-16T15:18:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.637+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab800cb220 con 0x7fab800b24e0 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.639+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8004e840 con 0x7fab800b24e0 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.639+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fab78007190 con 0x7fab800b24e0 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.640+0000 7fab77400640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 0x7fab5806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.641+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fab780a9d50 con 0x7fab800b24e0 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.641+0000 7fab7f400640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 0x7fab5806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.641+0000 7fab7f400640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 0x7fab5806e860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.645+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab780762d0 con 0x7fab800b24e0 2024-01-16T15:18:43.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.877+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab80070500 con 0x7fab800b24e0 2024-01-16T15:18:43.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.878+0000 7fab77400640 1 -- 172.21.15.43:0/2309518734 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fab78048020 con 0x7fab800b24e0 2024-01-16T15:18:43.869 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:43.869 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/1137375185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":1137375185},{"type":"v1","addr":"172.21.15.43:6836","nonce":1137375185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2958556276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2958556276},{"type":"v1","addr":"172.21.15.43:6837","nonce":2958556276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6437,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2525540945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2525540945},{"type":"v1","addr":"172.21.15.69:6833","nonce":2525540945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/208706681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":208706681},{"type":"v1","addr":"172.21.15.69:6835","nonce":208706681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[]} 2024-01-16T15:18:43.869 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 69 2024-01-16T15:18:43.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 msgr2=0x7fab5806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 0x7fab5806e860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70000ed0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 msgr2=0x7fab800b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fab78001f70 tx=0x7fab780014a0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 shutdown_connections 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fab5806c3a0 0x7fab5806e860 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab801606d0 0x7fab8015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab800b24e0 0x7fab800b2960 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 --2- 172.21.15.43:0/2309518734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab60004d00 0x7fab8014bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 >> 172.21.15.43:0/2309518734 conn(0x7fab8006a860 msgr2=0x7fab80071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 shutdown_connections 2024-01-16T15:18:43.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:43.881+0000 7fab84c00640 1 -- 172.21.15.43:0/2309518734 wait complete. 2024-01-16T15:18:43.891 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:18:43.891 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:18:43.891 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:18:43.891 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:18:43.891 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:18:43.891 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:18:43.891 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:18:43.892 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:18:43.892 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:18:43.892 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:18:43.892 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:18:43.892 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:18:43.892 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:18:43.892 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:18:43.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:18:43.893 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:18:43.908 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:18:43.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:18:43.912 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:18:43.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:18:43.924 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:18:43.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:18:43.931 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:18:43.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:18:43.959 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:18:43.959 DEBUG:teuthology.parallel:result is None 2024-01-16T15:18:43.966 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:18:43.966 DEBUG:teuthology.parallel:result is None 2024-01-16T15:18:43.978 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:18:43.979 DEBUG:teuthology.parallel:result is None 2024-01-16T15:18:43.985 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:18:43.985 DEBUG:teuthology.parallel:result is None 2024-01-16T15:18:43.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:18:44.121 INFO:tasks.ceph.mds.c.smithi043.stdout:starting mds.c at 2024-01-16T15:18:44.122 INFO:tasks.ceph.mds.a.smithi043.stdout:starting mds.a at 2024-01-16T15:18:44.137 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-01-16T15:18:44.140 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-01-16T15:18:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 Processor -- start 2024-01-16T15:18:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 -- start start 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f944015b340 0x7f94400abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944006dc30 0x7f94400b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94400ac530 con 0x7f94400cbcb0 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400b2080 con 0x7f944015b340 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.184+0000 7f9448600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400b2360 con 0x7f944006dc30 2024-01-16T15:18:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f944015b340 0x7f94400abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f944015b340 0x7f94400abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39182/0 (socket says 172.21.15.43:39182) 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 -- 172.21.15.43:0/1673195599 learned_addr learned my addr 172.21.15.43:0/1673195599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9446600640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944006dc30 0x7f94400b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 -- 172.21.15.43:0/1673195599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944006dc30 msgr2=0x7f94400b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944006dc30 0x7f94400b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 -- 172.21.15.43:0/1673195599 >> v1:172.21.15.69:6789/0 conn(0x7f94400cbcb0 legacy=0x7f944006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 -- 172.21.15.43:0/1673195599 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94400b2640 con 0x7f944015b340 2024-01-16T15:18:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.186+0000 7f9445c00640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f944015b340 0x7f94400abff0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9430009e50 tx=0x7f943002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81e92d8fdfedb7d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f943fe00640 1 -- 172.21.15.43:0/1673195599 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943002cb00 con 0x7f944015b340 2024-01-16T15:18:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f943fe00640 1 -- 172.21.15.43:0/1673195599 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943002cc60 con 0x7f944015b340 2024-01-16T15:18:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f943fe00640 1 -- 172.21.15.43:0/1673195599 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94300343f0 con 0x7f944015b340 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 -- 172.21.15.43:0/1673195599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f944015b340 msgr2=0x7f94400abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f944015b340 0x7f94400abff0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9430009e50 tx=0x7f943002bf60 comp rx=0 tx=0).stop 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 -- 172.21.15.43:0/1673195599 shutdown_connections 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944006dc30 0x7f94400b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 --2- 172.21.15.43:0/1673195599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f944015b340 0x7f94400abff0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.187+0000 7f9448600640 1 -- 172.21.15.43:0/1673195599 >> 172.21.15.43:0/1673195599 conn(0x7f94400b0d80 msgr2=0x7f94400b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.188+0000 7f9448600640 1 -- 172.21.15.43:0/1673195599 shutdown_connections 2024-01-16T15:18:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.188+0000 7f9448600640 1 -- 172.21.15.43:0/1673195599 wait complete. 2024-01-16T15:18:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.189+0000 7f9448600640 1 Processor -- start 2024-01-16T15:18:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.189+0000 7f9448600640 1 -- start start 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944006dc30 0x7f94401f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944015b340 0x7f9440159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94401f4430 con 0x7f94400cbcb0 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94401587d0 con 0x7f944006dc30 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.190+0000 7f9448600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9440158ab0 con 0x7f944015b340 2024-01-16T15:18:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39198/0 (socket says 172.21.15.43:39198) 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 -- 172.21.15.43:0/148155077 learned_addr learned my addr 172.21.15.43:0/148155077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9446600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944015b340 0x7f9440159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445c00640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944006dc30 0x7f94401f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 -- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944015b340 msgr2=0x7f9440159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944015b340 0x7f9440159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 -- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944006dc30 msgr2=0x7f94401f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944006dc30 0x7f94401f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 -- 172.21.15.43:0/148155077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430009b30 con 0x7f94400cbcb0 2024-01-16T15:18:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.191+0000 7f9445200640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f942800c400 tx=0x7f942800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.192+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94280052d0 con 0x7f94400cbcb0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.192+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428005430 con 0x7f94400cbcb0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.192+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944015b7c0 con 0x7f94400cbcb0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.192+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428003830 con 0x7f94400cbcb0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.192+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944015bd80 con 0x7f94400cbcb0 2024-01-16T15:18:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.193+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9440074770 con 0x7f94400cbcb0 2024-01-16T15:18:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.195+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9428003990 con 0x7f94400cbcb0 2024-01-16T15:18:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.195+0000 7f943d600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 0x7f941806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.195+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9428087f80 con 0x7f94400cbcb0 2024-01-16T15:18:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.196+0000 7f9445c00640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 0x7f941806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.196+0000 7f9445c00640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 0x7f941806e860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9430035000 tx=0x7f9430030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.200+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94280545b0 con 0x7f94400cbcb0 2024-01-16T15:18:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:44.412+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f944006e050 con 0x7f94400cbcb0 2024-01-16T15:18:45.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.252+0000 7f943d600640 1 -- 172.21.15.43:0/148155077 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v109) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9428017020 con 0x7f94400cbcb0 2024-01-16T15:18:45.242 INFO:teuthology.orchestra.run.smithi043.stderr: removed all blocklist entries 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 msgr2=0x7f941806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 0x7f941806e860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9430035000 tx=0x7f9430030000 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 msgr2=0x7f94401595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f942800c400 tx=0x7f942800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 shutdown_connections 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f941806c3a0 0x7f941806e860 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f944015b340 0x7f9440159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94400cbcb0 0x7f94401595c0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 --2- 172.21.15.43:0/148155077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944006dc30 0x7f94401f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 >> 172.21.15.43:0/148155077 conn(0x7f94400b0d80 msgr2=0x7f94400aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:45.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 shutdown_connections 2024-01-16T15:18:45.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.256+0000 7f9448600640 1 -- 172.21.15.43:0/148155077 wait complete. 2024-01-16T15:18:45.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:18:45.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.491+0000 7f4f7f200640 1 Processor -- start 2024-01-16T15:18:45.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.492+0000 7f4f7f200640 1 -- start start 2024-01-16T15:18:45.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.493+0000 7f4f7f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f7806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.493+0000 7f4f7f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7806cae0 con 0x7f4f7815ed20 2024-01-16T15:18:45.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.493+0000 7f4f7f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f78071e40 con 0x7f4f78141e80 2024-01-16T15:18:45.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.493+0000 7f4f7f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f781422e0 con 0x7f4f78071820 2024-01-16T15:18:45.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7de00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4f7815ed20 0x7f4f7815f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36500/0 (socket says 172.21.15.43:36500) 2024-01-16T15:18:45.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7de00640 1 -- 172.21.15.43:0/3376377845 learned_addr learned my addr 172.21.15.43:0/3376377845 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:45.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049666777 0 0) 0x7f4f7806cae0 con 0x7f4f7815ed20 2024-01-16T15:18:45.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f58003520 con 0x7f4f7815ed20 2024-01-16T15:18:45.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7e800640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f7806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836130672 0 0) 0x7f4f58003520 con 0x7f4f7815ed20 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 msgr2=0x7f4f7806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f7806cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 >> v1:172.21.15.69:6789/0 conn(0x7f4f78141e80 legacy=0x7f4f78071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7806c1b0 con 0x7f4f7815ed20 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.494+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4f68001670 con 0x7f4f7815ed20 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 >> v1:172.21.15.43:6789/0 conn(0x7f4f7815ed20 legacy=0x7f4f7815f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f58003a10 0x7f4f58003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f58004d70 0x7f4f580051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5800bb90 con 0x7f4f58004d70 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5800bd00 con 0x7f4f58003a10 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5800bec0 con 0x7f4f78071820 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f58004d70 0x7f4f580051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7de00640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f58003a10 0x7f4f58003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7d400640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 -- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 msgr2=0x7f4f580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78071820 0x7f4f580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 -- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f58003a10 msgr2=0x7f4f58003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f58003a10 0x7f4f58003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.495+0000 7f4f7e800640 1 -- 172.21.15.43:0/3376377845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58003520 con 0x7f4f58004d70 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.497+0000 7f4f7e800640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f58004d70 0x7f4f580051f0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007330 tx=0x7f4f580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57b0f964e9000e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.497+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7002cbf0 con 0x7f4f58004d70 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.497+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7002cd50 con 0x7f4f58004d70 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.497+0000 7f4f7ca00640 1 -- 172.21.15.43:0/3376377845 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70034660 con 0x7f4f58004d70 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 -- 172.21.15.43:0/3376377845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f58004d70 msgr2=0x7f4f580051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 --2- 172.21.15.43:0/3376377845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f58004d70 0x7f4f580051f0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007330 tx=0x7f4f580057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7e800640 1 -- 172.21.15.43:0/3376377845 reap_dead start 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 -- 172.21.15.43:0/3376377845 shutdown_connections 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 -- 172.21.15.43:0/3376377845 >> 172.21.15.43:0/3376377845 conn(0x7f4f780b0840 msgr2=0x7f4f780b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 -- 172.21.15.43:0/3376377845 shutdown_connections 2024-01-16T15:18:45.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.498+0000 7f4f7f200640 1 -- 172.21.15.43:0/3376377845 wait complete. 2024-01-16T15:18:45.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.500+0000 7f4f7f200640 1 Processor -- start 2024-01-16T15:18:45.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.500+0000 7f4f7f200640 1 -- start start 2024-01-16T15:18:45.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 0x7f4f7813d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78158220 0x7f4f78155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7813d350 con 0x7f4f7813dcb0 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f781586a0 con 0x7f4f78138180 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78138560 con 0x7f4f78158220 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39224/0 (socket says 172.21.15.43:39224) 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.501+0000 7f4f7d400640 1 -- 172.21.15.43:0/452974374 learned_addr learned my addr 172.21.15.43:0/452974374 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7de00640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 0x7f4f7813d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7e800640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78158220 0x7f4f78155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 -- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78158220 msgr2=0x7f4f78155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78158220 0x7f4f78155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 -- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 msgr2=0x7f4f7813d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 0x7f4f7813d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 -- 172.21.15.43:0/452974374 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7813c6d0 con 0x7f4f7813dcb0 2024-01-16T15:18:45.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7de00640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 0x7f4f7813d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:45.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7d400640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000e580 tx=0x7f4f6000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:45.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f600052e0 con 0x7f4f7813dcb0 2024-01-16T15:18:45.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60005440 con 0x7f4f7813dcb0 2024-01-16T15:18:45.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7813c9b0 con 0x7f4f7813dcb0 2024-01-16T15:18:45.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.502+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60003810 con 0x7f4f7813dcb0 2024-01-16T15:18:45.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.503+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7813cf70 con 0x7f4f7813dcb0 2024-01-16T15:18:45.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.504+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7813d490 con 0x7f4f7813dcb0 2024-01-16T15:18:45.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.505+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4f60003970 con 0x7f4f7813dcb0 2024-01-16T15:18:45.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.505+0000 7f4f76000640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 0x7f4f5006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:45.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.506+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f4f60088570 con 0x7f4f7813dcb0 2024-01-16T15:18:45.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.506+0000 7f4f7de00640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 0x7f4f5006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:45.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.507+0000 7f4f7de00640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 0x7f4f5006e860 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4f68001770 tx=0x7f4f68000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:45.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.510+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f7813d490 con 0x7f4f7813dcb0 2024-01-16T15:18:45.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.780+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4f7813d490 con 0x7f4f7813dcb0 2024-01-16T15:18:45.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.781+0000 7f4f76000640 1 -- 172.21.15.43:0/452974374 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4f7813d490 con 0x7f4f7813dcb0 2024-01-16T15:18:45.771 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:45.771 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_dump": [ 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.a", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlU0WPJxAAfRcJrIkew8QylLYN07Uw9Q==", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.b", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlaqu3DhAAoNUI7ZYBYwkwzSepV3RSBA==", 2024-01-16T15:18:45.772 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.c", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlUtTCOBAApvb3+ZG0X09gqY9efBhEDg==", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mds.d", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlXcNpHxAA0v269zzgLnUhMny1ZPzSsQ==", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.0", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCXmaZltihLExAAiAaQu2xEra+z/fEZXSqOvA==", 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.1", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCamaZlxxhjLRAA2MuPSZTjGNVMpWeKBt9dPg==", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.2", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCemaZlPky+CRAAr6HktZ6ErCsai0bFVgcipw==", 2024-01-16T15:18:45.775 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.3", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQChmaZlpvCpIhAAx1RtH9zEiMKYqUZhlRH7Ww==", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.4", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQComaZluwuJKxAAKjEmanAZxetKuxLZ888BbQ==", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.5", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCsmaZlALvzCBAA36C471D6yFaauBTaDsDDBg==", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.6", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCvmaZlsTEoIRAAQBxmxr/Y5hSdpCaSjrvKNA==", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "osd.7", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCymaZlmeGmORAAdIolaYA6p2pZAkkaofhrnA==", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow profile osd", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile osd", 2024-01-16T15:18:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.0", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ==", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow r", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.1", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZlP8QOCRAA+5q/S/lLH9Qv8zz/VEwBXA==", 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.2", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCUmaZlu0yxORAAeXzxT3sRgWCeQbl18SDT5w==", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.3", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCVmaZl2F5oExAAon/f/Up7DOC1S0folm4MDA==", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow r", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow rw", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow rwx" 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "client.admin", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCSmaZlNFQGGxAAA8SB+dJnMzJQ/OzYnp3PEw==", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": "allow *", 2024-01-16T15:18:45.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow *", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.x", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlIjFfGxAAIiK+6U+mhoikeYYVfgxUeg==", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "entity": "mgr.y", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "AQCTmaZlH70YDxAA7t43Nc5PnDtGwnJtprsImA==", 2024-01-16T15:18:45.782 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": { 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": "allow *", 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": "allow profile mgr", 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": "allow *" 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-01-16T15:18:45.783 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 msgr2=0x7f4f5006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 0x7f4f5006e860 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4f68001770 tx=0x7f4f68000ed0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 msgr2=0x7f4f78157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000e580 tx=0x7f4f6000ea40 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 shutdown_connections 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f5006c3a0 0x7f4f5006e860 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f78158220 0x7f4f78155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f7813dcb0 0x7f4f78157ce0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 --2- 172.21.15.43:0/452974374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f78138180 0x7f4f7813d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 >> 172.21.15.43:0/452974374 conn(0x7f4f780b0840 msgr2=0x7f4f780ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 shutdown_connections 2024-01-16T15:18:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:45.784+0000 7f4f7f200640 1 -- 172.21.15.43:0/452974374 wait complete. 2024-01-16T15:18:45.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:46.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.011+0000 7f1ff7e00640 1 Processor -- start 2024-01-16T15:18:46.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.011+0000 7f1ff7e00640 1 -- start start 2024-01-16T15:18:46.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 0x7f1ff80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff815fa70 con 0x7f1ff813dfd0 2024-01-16T15:18:46.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff80cfa40 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff815c580 con 0x7f1ff80cf050 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39240/0 (socket says 172.21.15.43:39240) 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.013+0000 7f1ff7400640 1 -- 172.21.15.43:0/3894882271 learned_addr learned my addr 172.21.15.43:0/3894882271 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1fede00640 1 --2- 172.21.15.43:0/3894882271 >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 0x7f1ff80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1ff7400640 1 -- 172.21.15.43:0/3894882271 >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 msgr2=0x7f1ff80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1ff7400640 1 --2- 172.21.15.43:0/3894882271 >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 0x7f1ff80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1ff7400640 1 -- 172.21.15.43:0/3894882271 >> v1:172.21.15.43:6789/0 conn(0x7f1ff813dfd0 legacy=0x7f1ff815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1ff7400640 1 -- 172.21.15.43:0/3894882271 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8140900 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.014+0000 7f1fede00640 1 --2- 172.21.15.43:0/3894882271 >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 0x7f1ff80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:46.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.016+0000 7f1ff7400640 1 --2- 172.21.15.43:0/3894882271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f1ff806e070 tx=0x7f1ff002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8445e7a2046b9e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.016+0000 7f1ff6000640 1 -- 172.21.15.43:0/3894882271 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0030870 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.016+0000 7f1ff6000640 1 -- 172.21.15.43:0/3894882271 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff00309d0 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.016+0000 7f1ff6000640 1 -- 172.21.15.43:0/3894882271 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0030d60 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 -- 172.21.15.43:0/3894882271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff80cd7b0 msgr2=0x7f1ff80cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/3894882271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f1ff806e070 tx=0x7f1ff002b9b0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 -- 172.21.15.43:0/3894882271 shutdown_connections 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/3894882271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff80cd7b0 0x7f1ff80cdbb0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/3894882271 >> v2:172.21.15.69:3300/0 conn(0x7f1ff80cf050 0x7f1ff80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 -- 172.21.15.43:0/3894882271 >> 172.21.15.43:0/3894882271 conn(0x7f1ff806a860 msgr2=0x7f1ff806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:46.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 -- 172.21.15.43:0/3894882271 shutdown_connections 2024-01-16T15:18:46.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.017+0000 7f1ff7e00640 1 -- 172.21.15.43:0/3894882271 wait complete. 2024-01-16T15:18:46.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.019+0000 7f1ff7e00640 1 Processor -- start 2024-01-16T15:18:46.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.019+0000 7f1ff7e00640 1 -- start start 2024-01-16T15:18:46.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ff80cd7b0 0x7f1ff80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff80cf050 0x7f1ff8131150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80c92a0 con 0x7f1ff813dfd0 2024-01-16T15:18:46.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80b2440 con 0x7f1ff80cf050 2024-01-16T15:18:46.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80b1ba0 con 0x7f1ff80cd7b0 2024-01-16T15:18:46.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39242/0 (socket says 172.21.15.43:39242) 2024-01-16T15:18:46.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.020+0000 7f1ff7400640 1 -- 172.21.15.43:0/431628251 learned_addr learned my addr 172.21.15.43:0/431628251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff6a00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ff80cd7b0 0x7f1ff80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1fede00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff80cf050 0x7f1ff8131150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 -- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ff80cd7b0 msgr2=0x7f1ff80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ff80cd7b0 0x7f1ff80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 -- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff80cf050 msgr2=0x7f1ff8131150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff80cf050 0x7f1ff8131150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 -- 172.21.15.43:0/431628251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00094d0 con 0x7f1ff813dfd0 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.021+0000 7f1ff7400640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00090b0 tx=0x7f1ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.022+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0039c90 con 0x7f1ff813dfd0 2024-01-16T15:18:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.022+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0039df0 con 0x7f1ff813dfd0 2024-01-16T15:18:46.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.022+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0043640 con 0x7f1ff813dfd0 2024-01-16T15:18:46.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.022+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff80b1e80 con 0x7f1ff813dfd0 2024-01-16T15:18:46.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.022+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff80b1770 con 0x7f1ff813dfd0 2024-01-16T15:18:46.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.023+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff80b1960 con 0x7f1ff813dfd0 2024-01-16T15:18:46.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.024+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f1ff00437a0 con 0x7f1ff813dfd0 2024-01-16T15:18:46.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.025+0000 7f1fef400640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 0x7f1fd006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.025+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f1ff003e070 con 0x7f1ff813dfd0 2024-01-16T15:18:46.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.025+0000 7f1ff6a00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 0x7f1fd006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.026+0000 7f1ff6a00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 0x7f1fd006e860 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80012e0 tx=0x7f1fe8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.030+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ff80b1960 con 0x7f1ff813dfd0 2024-01-16T15:18:46.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.228+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff80c9000 con 0x7f1ff813dfd0 2024-01-16T15:18:46.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.230+0000 7f1fef400640 1 -- 172.21.15.43:0/431628251 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+12854 (secure 0 0 0) 0x7f1ff80c9000 con 0x7f1ff813dfd0 2024-01-16T15:18:46.220 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:46.220 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":109,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:45.249352+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:46.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 msgr2=0x7f1fd006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 0x7f1fd006e860 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80012e0 tx=0x7f1fe8009040 comp rx=0 tx=0).stop 2024-01-16T15:18:46.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 msgr2=0x7f1ff8131690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00090b0 tx=0x7f1ff0004730 comp rx=0 tx=0).stop 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 shutdown_connections 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1fd006c3a0 0x7f1fd006e860 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff80cf050 0x7f1ff8131150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ff813dfd0 0x7f1ff8131690 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 --2- 172.21.15.43:0/431628251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ff80cd7b0 0x7f1ff80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 >> 172.21.15.43:0/431628251 conn(0x7f1ff806a860 msgr2=0x7f1ff815e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.233+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 shutdown_connections 2024-01-16T15:18:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.234+0000 7f1ff7e00640 1 -- 172.21.15.43:0/431628251 wait complete. 2024-01-16T15:18:46.242 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:18:46.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:18:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.477+0000 7f643da00640 1 Processor -- start 2024-01-16T15:18:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.477+0000 7f643da00640 1 -- start start 2024-01-16T15:18:46.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f643da00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f64380ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f643da00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64380c6bf0 con 0x7f643815d3b0 2024-01-16T15:18:46.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f643da00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64380cedc0 con 0x7f64380d1d90 2024-01-16T15:18:46.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f643da00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64380c6e40 con 0x7f64380c8930 2024-01-16T15:18:46.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f6437e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f643815d3b0 0x7f64380d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36530/0 (socket says 172.21.15.43:36530) 2024-01-16T15:18:46.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.479+0000 7f6437e00640 1 -- 172.21.15.43:0/1156295607 learned_addr learned my addr 172.21.15.43:0/1156295607 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:46.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f643d000640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f64380ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854854105 0 0) 0x7f64380c6bf0 con 0x7f643815d3b0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6418003520 con 0x7f643815d3b0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692255546 0 0) 0x7f6418003520 con 0x7f643815d3b0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 msgr2=0x7f64380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f64380ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 >> v1:172.21.15.69:6789/0 conn(0x7f64380d1d90 legacy=0x7f64380b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64380c7120 con 0x7f643815d3b0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6428001670 con 0x7f643815d3b0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 >> v1:172.21.15.43:6789/0 conn(0x7f643815d3b0 legacy=0x7f64380d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f6418003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64180047c0 0x7f6418004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f6418005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64380c6e40 con 0x7f6418003a10 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.480+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641800bb70 con 0x7f64180047c0 2024-01-16T15:18:46.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641800bd30 con 0x7f64380c8930 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f6418003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f642fe00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64180047c0 0x7f6418004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f643d000640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f6418005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 -- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 msgr2=0x7f6418005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380c8930 0x7f6418005230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 -- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64180047c0 msgr2=0x7f6418004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64180047c0 0x7f6418004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f6437e00640 1 -- 172.21.15.43:0/1156295607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418003520 con 0x7f6418003a10 2024-01-16T15:18:46.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.481+0000 7f642fe00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64180047c0 0x7f6418004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.482+0000 7f6437e00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f6418003e30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f642800c210 tx=0x7f64180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb02ae65ee086f18 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428005ad0 con 0x7f6418003a10 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6428005c50 con 0x7f6418003a10 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f6437400640 1 -- 172.21.15.43:0/1156295607 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642802c890 con 0x7f6418003a10 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f643da00640 1 -- 172.21.15.43:0/1156295607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 msgr2=0x7f6418003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f643da00640 1 --2- 172.21.15.43:0/1156295607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f6418003e30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f642800c210 tx=0x7f64180057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f643d000640 1 -- 172.21.15.43:0/1156295607 reap_dead start 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f643da00640 1 -- 172.21.15.43:0/1156295607 shutdown_connections 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.483+0000 7f643da00640 1 -- 172.21.15.43:0/1156295607 >> 172.21.15.43:0/1156295607 conn(0x7f64380d30a0 msgr2=0x7f64380d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.484+0000 7f643da00640 1 -- 172.21.15.43:0/1156295607 shutdown_connections 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.484+0000 7f643da00640 1 -- 172.21.15.43:0/1156295607 wait complete. 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.485+0000 7f643da00640 1 Processor -- start 2024-01-16T15:18:46.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.486+0000 7f643da00640 1 -- start start 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 0x7f643815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64380b8580 0x7f643815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64380caf20 con 0x7f6418003a10 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643815c760 con 0x7f64380b8580 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643da00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64380ca2a0 con 0x7f64380b8100 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39262/0 (socket says 172.21.15.43:39262) 2024-01-16T15:18:46.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 -- 172.21.15.43:0/2509819708 learned_addr learned my addr 172.21.15.43:0/2509819708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f642fe00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 0x7f643815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f643d000640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64380b8580 0x7f643815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 -- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 msgr2=0x7f643815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 0x7f643815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 -- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64380b8580 msgr2=0x7f643815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.487+0000 7f6437e00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64380b8580 0x7f643815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f6437e00640 1 -- 172.21.15.43:0/2509819708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64380ca580 con 0x7f6418003a10 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f642fe00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 0x7f643815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f6437e00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f642800c210 tx=0x7f6428001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428039070 con 0x7f6418003a10 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642802cca0 con 0x7f6418003a10 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642803c040 con 0x7f6418003a10 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64380ca860 con 0x7f6418003a10 2024-01-16T15:18:46.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.488+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64381f3730 con 0x7f6418003a10 2024-01-16T15:18:46.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.490+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643804e840 con 0x7f6418003a10 2024-01-16T15:18:46.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.491+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f6428032070 con 0x7f6418003a10 2024-01-16T15:18:46.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.491+0000 7f6434c00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 0x7f641006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.492+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f64280a9df0 con 0x7f6418003a10 2024-01-16T15:18:46.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.492+0000 7f642fe00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 0x7f641006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.492+0000 7f642fe00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 0x7f641006e860 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f64380b8ad0 tx=0x7f6420009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.496+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64280763d0 con 0x7f6418003a10 2024-01-16T15:18:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.695+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64380c8f90 con 0x7f6418003a10 2024-01-16T15:18:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.696+0000 7f6434c00640 1 -- 172.21.15.43:0/2509819708 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64280799f0 con 0x7f6418003a10 2024-01-16T15:18:46.687 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:18:46.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 msgr2=0x7f641006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 0x7f641006e860 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f64380b8ad0 tx=0x7f6420009040 comp rx=0 tx=0).stop 2024-01-16T15:18:46.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 msgr2=0x7f64381f31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f642800c210 tx=0x7f6428001a00 comp rx=0 tx=0).stop 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 shutdown_connections 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f641006c3a0 0x7f641006e860 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64380b8580 0x7f643815c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64380b8100 0x7f643815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 --2- 172.21.15.43:0/2509819708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6418003a10 0x7f64381f31f0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.699+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 >> 172.21.15.43:0/2509819708 conn(0x7f64380d30a0 msgr2=0x7f64380d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.700+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 shutdown_connections 2024-01-16T15:18:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.700+0000 7f643da00640 1 -- 172.21.15.43:0/2509819708 wait complete. 2024-01-16T15:18:46.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:18:46.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.935+0000 7f4012c00640 1 Processor -- start 2024-01-16T15:18:46.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.935+0000 7f4012c00640 1 -- start start 2024-01-16T15:18:46.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.936+0000 7f4012c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 0x7f400c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.936+0000 7f4012c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f400c15d840 0x7f400c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.936+0000 7f4012c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0cbf60 con 0x7f400c0d19f0 2024-01-16T15:18:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.936+0000 7f4012c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400c0cc520 con 0x7f400c15d840 2024-01-16T15:18:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.936+0000 7f4012c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0d03b0 con 0x7f400c15d460 2024-01-16T15:18:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.937+0000 7f4010e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f400c15d840 0x7f400c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.937+0000 7f4010e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f400c15d840 0x7f400c06b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39274/0 (socket says 172.21.15.43:39274) 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.937+0000 7f4010e00640 1 -- 172.21.15.43:0/1301006391 learned_addr learned my addr 172.21.15.43:0/1301006391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.937+0000 7f4011800640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 0x7f400c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.938+0000 7f4010e00640 1 -- 172.21.15.43:0/1301006391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 msgr2=0x7f400c0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.938+0000 7f4010e00640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 0x7f400c0b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.938+0000 7f4010e00640 1 -- 172.21.15.43:0/1301006391 >> v1:172.21.15.69:6789/0 conn(0x7f400c0d19f0 legacy=0x7f400c0cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.938+0000 7f4010e00640 1 -- 172.21.15.43:0/1301006391 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c06c3c0 con 0x7f400c15d840 2024-01-16T15:18:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.938+0000 7f4011800640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 0x7f400c0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:46.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.939+0000 7f4010e00640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c15d840 0x7f400c06b760 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009870 tx=0x7f3ff402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4653cd39363ecc36 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.939+0000 7f400be00640 1 -- 172.21.15.43:0/1301006391 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4030840 con 0x7f400c15d840 2024-01-16T15:18:46.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.939+0000 7f400be00640 1 -- 172.21.15.43:0/1301006391 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff40309a0 con 0x7f400c15d840 2024-01-16T15:18:46.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f400be00640 1 -- 172.21.15.43:0/1301006391 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4030d30 con 0x7f400c15d840 2024-01-16T15:18:46.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 -- 172.21.15.43:0/1301006391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c15d840 msgr2=0x7f400c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c15d840 0x7f400c06b760 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009870 tx=0x7f3ff402b980 comp rx=0 tx=0).stop 2024-01-16T15:18:46.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 -- 172.21.15.43:0/1301006391 shutdown_connections 2024-01-16T15:18:46.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c15d840 0x7f400c06b760 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 --2- 172.21.15.43:0/1301006391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d460 0x7f400c0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 -- 172.21.15.43:0/1301006391 >> 172.21.15.43:0/1301006391 conn(0x7f400c0d0f00 msgr2=0x7f400c0d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:46.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 -- 172.21.15.43:0/1301006391 shutdown_connections 2024-01-16T15:18:46.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.940+0000 7f4012c00640 1 -- 172.21.15.43:0/1301006391 wait complete. 2024-01-16T15:18:46.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.942+0000 7f4012c00640 1 Processor -- start 2024-01-16T15:18:46.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.942+0000 7f4012c00640 1 -- start start 2024-01-16T15:18:46.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f400c15d460 0x7f400c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d840 0x7f400c0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c09e3f0 con 0x7f400c0d19f0 2024-01-16T15:18:46.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c09b900 con 0x7f400c15d460 2024-01-16T15:18:46.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.943+0000 7f4012c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c09d7e0 con 0x7f400c15d840 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39288/0 (socket says 172.21.15.43:39288) 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 -- 172.21.15.43:0/3732630949 learned_addr learned my addr 172.21.15.43:0/3732630949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4012200640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d840 0x7f400c0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4010e00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f400c15d460 0x7f400c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 -- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d840 msgr2=0x7f400c0ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d840 0x7f400c0ae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 -- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f400c15d460 msgr2=0x7f400c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f400c15d460 0x7f400c15b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.944+0000 7f4011800640 1 -- 172.21.15.43:0/3732630949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff40094d0 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4011800640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00d7d0 tx=0x7f3ffc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc004030 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc004190 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc004520 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c09dac0 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.945+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c09e080 con 0x7f400c0d19f0 2024-01-16T15:18:46.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.946+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c04e840 con 0x7f400c0d19f0 2024-01-16T15:18:46.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.948+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f3ffc0096a0 con 0x7f400c0d19f0 2024-01-16T15:18:46.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.949+0000 7f4009600640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 0x7f3fe406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:46.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.949+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f3ffc089780 con 0x7f400c0d19f0 2024-01-16T15:18:46.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.949+0000 7f4010e00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 0x7f3fe406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:46.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.950+0000 7f4010e00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 0x7f3fe406e860 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4036000 tx=0x7f3ff4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:46.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:46.952+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ffc055d40 con 0x7f400c0d19f0 2024-01-16T15:18:47.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.149+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f400c0d0860 con 0x7f400c0d19f0 2024-01-16T15:18:47.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.150+0000 7f4009600640 1 -- 172.21.15.43:0/3732630949 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3ffc059360 con 0x7f400c0d19f0 2024-01-16T15:18:47.140 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:47.140 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:18:47.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:18:47.140 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-01-16T15:18:47.141 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-01-16T15:18:47.141 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:18:47.141 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 msgr2=0x7f3fe406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 0x7f3fe406e860 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4036000 tx=0x7f3ff4008030 comp rx=0 tx=0).stop 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 msgr2=0x7f400c09e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00d7d0 tx=0x7f3ffc00dc90 comp rx=0 tx=0).stop 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 shutdown_connections 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3fe406c3a0 0x7f3fe406e860 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f400c15d460 0x7f400c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f400c15d840 0x7f400c0ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 --2- 172.21.15.43:0/3732630949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f400c0d19f0 0x7f400c09e880 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 >> 172.21.15.43:0/3732630949 conn(0x7f400c0d0f00 msgr2=0x7f400c15dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:47.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 shutdown_connections 2024-01-16T15:18:47.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.153+0000 7f4012c00640 1 -- 172.21.15.43:0/3732630949 wait complete. 2024-01-16T15:18:47.164 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:18:47.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:18:47.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.402+0000 7fdf79e00640 1 Processor -- start 2024-01-16T15:18:47.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.403+0000 7fdf79e00640 1 -- start start 2024-01-16T15:18:47.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf79e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdf740ce6f0 0x7fdf740d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:47.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf79e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740cf190 con 0x7fdf74160ab0 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf79e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b2e60 con 0x7fdf740cfcb0 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf79e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740cebd0 con 0x7fdf740ce6f0 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf78a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdf74160ab0 0x7fdf740cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36534/0 (socket says 172.21.15.43:36534) 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.404+0000 7fdf78a00640 1 -- 172.21.15.43:0/2914618562 learned_addr learned my addr 172.21.15.43:0/2914618562 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf79400640 1 --2- 172.21.15.43:0/2914618562 >> v2:172.21.15.43:3300/0 conn(0x7fdf740ce6f0 0x7fdf740d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf73400640 1 -- 172.21.15.43:0/2914618562 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408307570 0 0) 0x7fdf740cf190 con 0x7fdf74160ab0 2024-01-16T15:18:47.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf73400640 1 -- 172.21.15.43:0/2914618562 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54003520 con 0x7fdf74160ab0 2024-01-16T15:18:47.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf79400640 1 -- 172.21.15.43:0/2914618562 >> v1:172.21.15.69:6789/0 conn(0x7fdf740cfcb0 legacy=0x7fdf740b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf79400640 1 -- 172.21.15.43:0/2914618562 >> v1:172.21.15.43:6789/0 conn(0x7fdf74160ab0 legacy=0x7fdf740cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.405+0000 7fdf79400640 1 -- 172.21.15.43:0/2914618562 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740ceeb0 con 0x7fdf740ce6f0 2024-01-16T15:18:47.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.406+0000 7fdf79400640 1 --2- 172.21.15.43:0/2914618562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf740ce6f0 0x7fdf740d3b80 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0097a0 tx=0x7fdf6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d09413fe1126500 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf73400640 1 -- 172.21.15.43:0/2914618562 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c031830 con 0x7fdf740ce6f0 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf73400640 1 -- 172.21.15.43:0/2914618562 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c031990 con 0x7fdf740ce6f0 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf73400640 1 -- 172.21.15.43:0/2914618562 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c031d20 con 0x7fdf740ce6f0 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf79e00640 1 -- 172.21.15.43:0/2914618562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf740ce6f0 msgr2=0x7fdf740d3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf79e00640 1 --2- 172.21.15.43:0/2914618562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf740ce6f0 0x7fdf740d3b80 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0097a0 tx=0x7fdf6c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf79e00640 1 -- 172.21.15.43:0/2914618562 shutdown_connections 2024-01-16T15:18:47.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf79e00640 1 --2- 172.21.15.43:0/2914618562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf740ce6f0 0x7fdf740d3b80 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.407+0000 7fdf79e00640 1 -- 172.21.15.43:0/2914618562 >> 172.21.15.43:0/2914618562 conn(0x7fdf740d3060 msgr2=0x7fdf740d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:47.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.408+0000 7fdf79e00640 1 -- 172.21.15.43:0/2914618562 shutdown_connections 2024-01-16T15:18:47.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.408+0000 7fdf79e00640 1 -- 172.21.15.43:0/2914618562 wait complete. 2024-01-16T15:18:47.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.409+0000 7fdf79e00640 1 Processor -- start 2024-01-16T15:18:47.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.409+0000 7fdf79e00640 1 -- start start 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 0x7fdf740cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cfcb0 0x7fdf740ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740cc0a0 con 0x7fdf74160ab0 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7414bf60 con 0x7fdf740ce6f0 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.410+0000 7fdf79e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c9cd0 con 0x7fdf740cfcb0 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39300/0 (socket says 172.21.15.43:39300) 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 -- 172.21.15.43:0/700932592 learned_addr learned my addr 172.21.15.43:0/700932592 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf73e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cfcb0 0x7fdf740ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:47.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf78a00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 0x7fdf740cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:47.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 -- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cfcb0 msgr2=0x7fdf740ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cfcb0 0x7fdf740ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 -- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 msgr2=0x7fdf740cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 0x7fdf740cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf79400640 1 -- 172.21.15.43:0/700932592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009400 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.411+0000 7fdf78a00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 0x7fdf740cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf79400640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098c0 tx=0x7fdf6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c03a630 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c03a790 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf740c99f0 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c03ab40 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.412+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7414c620 con 0x7fdf74160ab0 2024-01-16T15:18:47.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.413+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7404e840 con 0x7fdf74160ab0 2024-01-16T15:18:47.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.414+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fdf6c03aca0 con 0x7fdf74160ab0 2024-01-16T15:18:47.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.415+0000 7fdf70c00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 0x7fdf4c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:47.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.415+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fdf6c0aa520 con 0x7fdf74160ab0 2024-01-16T15:18:47.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.416+0000 7fdf78a00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 0x7fdf4c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:47.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.416+0000 7fdf78a00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 0x7fdf4c06e860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf640011e0 tx=0x7fdf64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:47.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.420+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf6c076f00 con 0x7fdf74160ab0 2024-01-16T15:18:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:47.616+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdf740b3fa0 con 0x7fdf74160ab0 2024-01-16T15:18:48.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.468+0000 7fdf70c00640 1 -- 172.21.15.43:0/700932592 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdf6c048020 con 0x7fdf74160ab0 2024-01-16T15:18:48.459 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-01-16T15:18:48.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.473+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 msgr2=0x7fdf4c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.473+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 0x7fdf4c06e860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf640011e0 tx=0x7fdf64009040 comp rx=0 tx=0).stop 2024-01-16T15:18:48.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.473+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 msgr2=0x7fdf740caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.473+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098c0 tx=0x7fdf6c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 shutdown_connections 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdf4c06c3a0 0x7fdf4c06e860 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf74160ab0 0x7fdf740caaf0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cfcb0 0x7fdf740ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 --2- 172.21.15.43:0/700932592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf740ce6f0 0x7fdf740cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 >> 172.21.15.43:0/700932592 conn(0x7fdf740d3060 msgr2=0x7fdf7415c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:48.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.474+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 shutdown_connections 2024-01-16T15:18:48.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.475+0000 7fdf79e00640 1 -- 172.21.15.43:0/700932592 wait complete. 2024-01-16T15:18:48.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:18:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.711+0000 7f476f200640 1 Processor -- start 2024-01-16T15:18:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.711+0000 7f476f200640 1 -- start start 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f477015efa0 0x7f4770161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f477006da40 0x7f477006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477006c9a0 con 0x7f47700b2970 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477006cc80 con 0x7f477015efa0 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477006cf60 con 0x7f477006da40 2024-01-16T15:18:48.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f477015efa0 0x7f4770161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f477015efa0 0x7f4770161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39310/0 (socket says 172.21.15.43:39310) 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.713+0000 7f476de00640 1 -- 172.21.15.43:0/930443093 learned_addr learned my addr 172.21.15.43:0/930443093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.714+0000 7f4767e00640 1 --2- 172.21.15.43:0/930443093 >> v2:172.21.15.69:3300/0 conn(0x7f477006da40 0x7f477006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.714+0000 7f476de00640 1 -- 172.21.15.43:0/930443093 >> v2:172.21.15.69:3300/0 conn(0x7f477006da40 msgr2=0x7f477006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.714+0000 7f476de00640 1 --2- 172.21.15.43:0/930443093 >> v2:172.21.15.69:3300/0 conn(0x7f477006da40 0x7f477006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.714+0000 7f476de00640 1 -- 172.21.15.43:0/930443093 >> v1:172.21.15.69:6789/0 conn(0x7f47700b2970 legacy=0x7f47701650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.714+0000 7f476de00640 1 -- 172.21.15.43:0/930443093 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477006d240 con 0x7f477015efa0 2024-01-16T15:18:48.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.715+0000 7f476de00640 1 --2- 172.21.15.43:0/930443093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770161ac0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f47700b4670 tx=0x7f476002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615aa857885b1be6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:48.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.716+0000 7f476d400640 1 -- 172.21.15.43:0/930443093 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760031870 con 0x7f477015efa0 2024-01-16T15:18:48.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.716+0000 7f476d400640 1 -- 172.21.15.43:0/930443093 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47600319d0 con 0x7f477015efa0 2024-01-16T15:18:48.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.716+0000 7f476d400640 1 -- 172.21.15.43:0/930443093 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760031d60 con 0x7f477015efa0 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.716+0000 7f476f200640 1 -- 172.21.15.43:0/930443093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 msgr2=0x7f4770161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.716+0000 7f476f200640 1 --2- 172.21.15.43:0/930443093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770161ac0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f47700b4670 tx=0x7f476002b910 comp rx=0 tx=0).stop 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 -- 172.21.15.43:0/930443093 shutdown_connections 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 --2- 172.21.15.43:0/930443093 >> v2:172.21.15.69:3300/0 conn(0x7f477006da40 0x7f477006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 --2- 172.21.15.43:0/930443093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770161ac0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 -- 172.21.15.43:0/930443093 >> 172.21.15.43:0/930443093 conn(0x7f47700cec10 msgr2=0x7f47700cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 -- 172.21.15.43:0/930443093 shutdown_connections 2024-01-16T15:18:48.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.717+0000 7f476f200640 1 -- 172.21.15.43:0/930443093 wait complete. 2024-01-16T15:18:48.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.719+0000 7f476f200640 1 Processor -- start 2024-01-16T15:18:48.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.719+0000 7f476f200640 1 -- start start 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 0x7f47700b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 0x7f4770144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4770154ef0 con 0x7f477015efa0 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4770145360 con 0x7f47700b2970 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4770155380 con 0x7f477006da40 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39312/0 (socket says 172.21.15.43:39312) 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f476e800640 1 -- 172.21.15.43:0/64941244 learned_addr learned my addr 172.21.15.43:0/64941244 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:48.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.720+0000 7f4767e00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 0x7f4770144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476de00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 0x7f47700b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 -- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 msgr2=0x7f47700b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 0x7f47700b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 -- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 msgr2=0x7f4770144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 0x7f4770144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 -- 172.21.15.43:0/64941244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47600094d0 con 0x7f477015efa0 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f4767e00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 0x7f4770144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476de00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 0x7f47700b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:48.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476e800640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f476800c7a0 tx=0x7f476800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768003b70 con 0x7f477015efa0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4768003cf0 con 0x7f477015efa0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768003ea0 con 0x7f477015efa0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47701536f0 con 0x7f477015efa0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.721+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47701545c0 con 0x7f477015efa0 2024-01-16T15:18:48.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.722+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4770155060 con 0x7f477015efa0 2024-01-16T15:18:48.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.724+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f476800aa90 con 0x7f477015efa0 2024-01-16T15:18:48.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.725+0000 7f4766000640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 0x7f474806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:48.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.725+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f47680893a0 con 0x7f477015efa0 2024-01-16T15:18:48.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.725+0000 7f476de00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 0x7f474806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:48.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.726+0000 7f476de00640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 0x7f474806e890 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4760036000 tx=0x7f476003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:48.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.729+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47680559d0 con 0x7f477015efa0 2024-01-16T15:18:48.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:48.924+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f477004e840 con 0x7f477015efa0 2024-01-16T15:18:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.486+0000 7f4766000640 1 -- 172.21.15.43:0/64941244 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4768058ff0 con 0x7f477015efa0 2024-01-16T15:18:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-01-16T15:18:49.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.492+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 msgr2=0x7f474806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.492+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 0x7f474806e890 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4760036000 tx=0x7f476003c040 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.493+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 msgr2=0x7f4770163e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.493+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f476800c7a0 tx=0x7f476800cc60 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 shutdown_connections 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f474806c3d0 0x7f474806e890 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47700b2970 0x7f4770144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f477015efa0 0x7f4770163e00 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 --2- 172.21.15.43:0/64941244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f477006da40 0x7f47700b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.494+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 >> 172.21.15.43:0/64941244 conn(0x7f47700cec10 msgr2=0x7f4770161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:49.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.495+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 shutdown_connections 2024-01-16T15:18:49.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.495+0000 7f476f200640 1 -- 172.21.15.43:0/64941244 wait complete. 2024-01-16T15:18:49.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:18:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.718+0000 7f38bc800640 1 Processor -- start 2024-01-16T15:18:49.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.718+0000 7f38bc800640 1 -- start start 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38bc800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f38b415b350 0x7f38b40abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38bc800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f38b409cb30 0x7f38b40b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38bc800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b415b730 con 0x7f38b40adb60 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38bc800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b40ac740 con 0x7f38b415b350 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38bc800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b40ac460 con 0x7f38b409cb30 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38b9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f38b415b350 0x7f38b40abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38ba800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f38b40adb60 0x7f38b40adf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36538/0 (socket says 172.21.15.43:36538) 2024-01-16T15:18:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38b9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f38b415b350 0x7f38b40abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39320/0 (socket says 172.21.15.43:39320) 2024-01-16T15:18:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38ba800640 1 -- 172.21.15.43:0/607190338 learned_addr learned my addr 172.21.15.43:0/607190338 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.720+0000 7f38b9400640 1 --2- 172.21.15.43:0/607190338 >> v2:172.21.15.69:3300/0 conn(0x7f38b409cb30 0x7f38b40b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.721+0000 7f38b9e00640 1 -- 172.21.15.43:0/607190338 >> v2:172.21.15.69:3300/0 conn(0x7f38b409cb30 msgr2=0x7f38b40b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.721+0000 7f38b9e00640 1 --2- 172.21.15.43:0/607190338 >> v2:172.21.15.69:3300/0 conn(0x7f38b409cb30 0x7f38b40b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.721+0000 7f38b9e00640 1 -- 172.21.15.43:0/607190338 >> v1:172.21.15.43:6789/0 conn(0x7f38b40adb60 legacy=0x7f38b40adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.721+0000 7f38b9e00640 1 -- 172.21.15.43:0/607190338 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b40b2a50 con 0x7f38b415b350 2024-01-16T15:18:49.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.721+0000 7f38b8a00640 1 -- 172.21.15.43:0/607190338 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960555161 0 0) 0x7f38ac001490 con 0x7f38b40adb60 2024-01-16T15:18:49.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.722+0000 7f38b9e00640 1 --2- 172.21.15.43:0/607190338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b415b350 0x7f38b40abe80 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f38b40d0060 tx=0x7f38a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3100f6a6287ef38c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.722+0000 7f38b8a00640 1 -- 172.21.15.43:0/607190338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4031830 con 0x7f38b415b350 2024-01-16T15:18:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.722+0000 7f38b8a00640 1 -- 172.21.15.43:0/607190338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4031990 con 0x7f38b415b350 2024-01-16T15:18:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.722+0000 7f38b8a00640 1 -- 172.21.15.43:0/607190338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4031d20 con 0x7f38b415b350 2024-01-16T15:18:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 -- 172.21.15.43:0/607190338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b415b350 msgr2=0x7f38b40abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 --2- 172.21.15.43:0/607190338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b415b350 0x7f38b40abe80 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f38b40d0060 tx=0x7f38a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 -- 172.21.15.43:0/607190338 shutdown_connections 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 --2- 172.21.15.43:0/607190338 >> v2:172.21.15.69:3300/0 conn(0x7f38b409cb30 0x7f38b40b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 --2- 172.21.15.43:0/607190338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b415b350 0x7f38b40abe80 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 -- 172.21.15.43:0/607190338 >> 172.21.15.43:0/607190338 conn(0x7f38b40b0ba0 msgr2=0x7f38b40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 -- 172.21.15.43:0/607190338 shutdown_connections 2024-01-16T15:18:49.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.723+0000 7f38bc800640 1 -- 172.21.15.43:0/607190338 wait complete. 2024-01-16T15:18:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.725+0000 7f38bc800640 1 Processor -- start 2024-01-16T15:18:49.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.725+0000 7f38bc800640 1 -- start start 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38b40adb60 0x7f38b4154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 0x7f38b4155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b414c200 con 0x7f38b409cb30 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41546e0 con 0x7f38b415b350 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38bc800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4153ad0 con 0x7f38b40adb60 2024-01-16T15:18:49.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38b9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38b9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39324/0 (socket says 172.21.15.43:39324) 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.726+0000 7f38b9e00640 1 -- 172.21.15.43:0/2577355162 learned_addr learned my addr 172.21.15.43:0/2577355162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38ba800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 0x7f38b4155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9400640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38b40adb60 0x7f38b4154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 -- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38b40adb60 msgr2=0x7f38b4154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38b40adb60 0x7f38b4154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 -- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 msgr2=0x7f38b4155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 0x7f38b4155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 -- 172.21.15.43:0/2577355162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a4009400 con 0x7f38b409cb30 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38ba800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 0x7f38b4155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b9e00640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4004230 con 0x7f38b409cb30 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4004390 con 0x7f38b409cb30 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.727+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a403a870 con 0x7f38b409cb30 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.728+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b4153db0 con 0x7f38b409cb30 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.728+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b4154370 con 0x7f38b409cb30 2024-01-16T15:18:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.729+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b409d200 con 0x7f38b409cb30 2024-01-16T15:18:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.730+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f38a403a9d0 con 0x7f38b409cb30 2024-01-16T15:18:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.731+0000 7f38b2000640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 0x7f388c06e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.731+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f38a40aae20 con 0x7f38b409cb30 2024-01-16T15:18:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.731+0000 7f38b9400640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 0x7f388c06e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.732+0000 7f38b9400640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 0x7f388c06e8b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f38b41f4710 tx=0x7f389c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:49.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.735+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38a40776e0 con 0x7f38b409cb30 2024-01-16T15:18:49.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:49.962+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f38b404e920 con 0x7f38b409cb30 2024-01-16T15:18:50.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.501+0000 7f38b2000640 1 -- 172.21.15.43:0/2577355162 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f38a407ad00 con 0x7f38b409cb30 2024-01-16T15:18:50.493 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 16 and data pool 17 2024-01-16T15:18:50.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.514+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 msgr2=0x7f388c06e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.514+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 0x7f388c06e8b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f38b41f4710 tx=0x7f389c008040 comp rx=0 tx=0).stop 2024-01-16T15:18:50.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.514+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 msgr2=0x7f38b413b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.514+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a40046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 shutdown_connections 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f388c06c3f0 0x7f388c06e8b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b415b350 0x7f38b4155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38b40adb60 0x7f38b4154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 --2- 172.21.15.43:0/2577355162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38b409cb30 0x7f38b413b8e0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.516+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 >> 172.21.15.43:0/2577355162 conn(0x7f38b40b0ba0 msgr2=0x7f38b40afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:50.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.518+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 shutdown_connections 2024-01-16T15:18:50.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.519+0000 7f38bc800640 1 -- 172.21.15.43:0/2577355162 wait complete. 2024-01-16T15:18:50.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.728+0000 7f8d1e000640 1 Processor -- start 2024-01-16T15:18:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.729+0000 7f8d1e000640 1 -- start start 2024-01-16T15:18:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d1e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d1815d4d0 0x7f8d180cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d1e000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8d180ce260 0x7f8d180d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d1e000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1815d8b0 con 0x7f8d180ce6c0 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d1e000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d180b91b0 con 0x7f8d180ce260 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d1e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d180cdfb0 con 0x7f8d1815d4d0 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d16a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8d180ce260 0x7f8d180d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d16a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8d180ce260 0x7f8d180d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39340/0 (socket says 172.21.15.43:39340) 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d16a00640 1 -- 172.21.15.43:0/1742804935 learned_addr learned my addr 172.21.15.43:0/1742804935 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.730+0000 7f8d17e00640 1 --1- 172.21.15.43:0/1742804935 >> v1:172.21.15.43:6789/0 conn(0x7f8d180ce6c0 0x7f8d180b4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36542/0 (socket says 172.21.15.43:36542) 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d17400640 1 --2- 172.21.15.43:0/1742804935 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815d4d0 0x7f8d180cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d16a00640 1 -- 172.21.15.43:0/1742804935 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815d4d0 msgr2=0x7f8d180cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d16a00640 1 --2- 172.21.15.43:0/1742804935 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815d4d0 0x7f8d180cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d16a00640 1 -- 172.21.15.43:0/1742804935 >> v1:172.21.15.43:6789/0 conn(0x7f8d180ce6c0 legacy=0x7f8d180b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d16000640 1 -- 172.21.15.43:0/1742804935 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623999289 0 0) 0x7f8d1815d8b0 con 0x7f8d180ce6c0 2024-01-16T15:18:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.731+0000 7f8d16a00640 1 -- 172.21.15.43:0/1742804935 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180b2400 con 0x7f8d180ce260 2024-01-16T15:18:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.733+0000 7f8d16a00640 1 --2- 172.21.15.43:0/1742804935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180d1850 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f8d1806c470 tx=0x7f8d0002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de019b22fedd356 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.733+0000 7f8d16000640 1 -- 172.21.15.43:0/1742804935 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00030860 con 0x7f8d180ce260 2024-01-16T15:18:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.733+0000 7f8d16000640 1 -- 172.21.15.43:0/1742804935 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d000309c0 con 0x7f8d180ce260 2024-01-16T15:18:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.733+0000 7f8d16000640 1 -- 172.21.15.43:0/1742804935 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00030d50 con 0x7f8d180ce260 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 -- 172.21.15.43:0/1742804935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 msgr2=0x7f8d180d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 --2- 172.21.15.43:0/1742804935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180d1850 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f8d1806c470 tx=0x7f8d0002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 -- 172.21.15.43:0/1742804935 shutdown_connections 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 --2- 172.21.15.43:0/1742804935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180d1850 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 --2- 172.21.15.43:0/1742804935 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815d4d0 0x7f8d180cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 -- 172.21.15.43:0/1742804935 >> 172.21.15.43:0/1742804935 conn(0x7f8d18071f90 msgr2=0x7f8d180723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 -- 172.21.15.43:0/1742804935 shutdown_connections 2024-01-16T15:18:50.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.734+0000 7f8d1e000640 1 -- 172.21.15.43:0/1742804935 wait complete. 2024-01-16T15:18:50.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.735+0000 7f8d1e000640 1 Processor -- start 2024-01-16T15:18:50.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.736+0000 7f8d1e000640 1 -- start start 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.736+0000 7f8d1e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d1e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180ce6c0 0x7f8d180ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d1e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 0x7f8d180ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d1e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180cadc0 con 0x7f8d180ce260 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d1e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18154be0 con 0x7f8d1815d4d0 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d1e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180cb030 con 0x7f8d180ce6c0 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39350/0 (socket says 172.21.15.43:39350) 2024-01-16T15:18:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 -- 172.21.15.43:0/798094536 learned_addr learned my addr 172.21.15.43:0/798094536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17e00640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 0x7f8d180ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 -- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180ce6c0 msgr2=0x7f8d180ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180ce6c0 0x7f8d180ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 -- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 msgr2=0x7f8d180ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 0x7f8d180ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17400640 1 -- 172.21.15.43:0/798094536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d000094d0 con 0x7f8d180ce260 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.737+0000 7f8d17e00640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 0x7f8d180ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d17400640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8d08007e60 tx=0x7f8d0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08014620 con 0x7f8d180ce260 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0800ee90 con 0x7f8d180ce260 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d080149e0 con 0x7f8d180ce260 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d180cb310 con 0x7f8d180ce260 2024-01-16T15:18:50.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.738+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d180cb8d0 con 0x7f8d180ce260 2024-01-16T15:18:50.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.739+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d18074770 con 0x7f8d180ce260 2024-01-16T15:18:50.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.740+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f8d08014b40 con 0x7f8d180ce260 2024-01-16T15:18:50.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.741+0000 7f8d0f400640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 0x7f8cf006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.741+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8d08089140 con 0x7f8d180ce260 2024-01-16T15:18:50.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.741+0000 7f8d16a00640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 0x7f8cf006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.742+0000 7f8d16a00640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 0x7f8cf006e860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8d00036000 tx=0x7f8d0003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.745+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d08055a00 con 0x7f8d180ce260 2024-01-16T15:18:50.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.938+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d1806c1a0 con 0x7f8d180ce260 2024-01-16T15:18:50.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.941+0000 7f8d0f400640 1 -- 172.21.15.43:0/798094536 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16535 (secure 0 0 0) 0x7f8d08059020 con 0x7f8d180ce260 2024-01-16T15:18:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:50.932 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":112,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:49.964607+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.943+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 msgr2=0x7f8cf006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.943+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 0x7f8cf006e860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8d00036000 tx=0x7f8d0003b040 comp rx=0 tx=0).stop 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.943+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 msgr2=0x7f8d180cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.943+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8d08007e60 tx=0x7f8d0800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 shutdown_connections 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8cf006c3a0 0x7f8cf006e860 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1815d4d0 0x7f8d180ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180ce6c0 0x7f8d180ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 --2- 172.21.15.43:0/798094536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d180ce260 0x7f8d180cc0d0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 >> 172.21.15.43:0/798094536 conn(0x7f8d18071f90 msgr2=0x7f8d18158650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 shutdown_connections 2024-01-16T15:18:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:50.944+0000 7f8d1e000640 1 -- 172.21.15.43:0/798094536 wait complete. 2024-01-16T15:18:50.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:51.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.177+0000 7f9c0d200640 1 Processor -- start 2024-01-16T15:18:51.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.178+0000 7f9c0d200640 1 -- start start 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.179+0000 7f9c0d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9c080cf050 0x7f9c080cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9c0d200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c08140900 con 0x7f9c080cd7b0 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9c0d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0815fa70 con 0x7f9c0813dfd0 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9c0d200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c080cfa40 con 0x7f9c080cf050 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9c080cf050 0x7f9c080cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9c080cf050 0x7f9c080cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39364/0 (socket says 172.21.15.43:39364) 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9c07e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9c080cd7b0 0x7f9c080cdbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36548/0 (socket says 172.21.15.43:36548) 2024-01-16T15:18:51.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 -- 172.21.15.43:0/2056622393 learned_addr learned my addr 172.21.15.43:0/2056622393 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:51.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 -- 172.21.15.43:0/2056622393 >> v1:172.21.15.69:6789/0 conn(0x7f9c0813dfd0 legacy=0x7f9c0815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:51.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 -- 172.21.15.43:0/2056622393 >> v1:172.21.15.43:6789/0 conn(0x7f9c080cd7b0 legacy=0x7f9c080cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.180+0000 7f9bffe00640 1 -- 172.21.15.43:0/2056622393 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0815ee60 con 0x7f9c080cf050 2024-01-16T15:18:51.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.181+0000 7f9bffe00640 1 --2- 172.21.15.43:0/2056622393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080cf500 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009870 tx=0x7f9bf002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2becd7cae4e3f209 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.182+0000 7f9c06a00640 1 -- 172.21.15.43:0/2056622393 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0030860 con 0x7f9c080cf050 2024-01-16T15:18:51.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.182+0000 7f9c06a00640 1 -- 172.21.15.43:0/2056622393 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf00309c0 con 0x7f9c080cf050 2024-01-16T15:18:51.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.182+0000 7f9c06a00640 1 -- 172.21.15.43:0/2056622393 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0030d50 con 0x7f9c080cf050 2024-01-16T15:18:51.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.182+0000 7f9c0d200640 1 -- 172.21.15.43:0/2056622393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 msgr2=0x7f9c080cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.182+0000 7f9c0d200640 1 --2- 172.21.15.43:0/2056622393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080cf500 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009870 tx=0x7f9bf002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.183+0000 7f9c0d200640 1 -- 172.21.15.43:0/2056622393 shutdown_connections 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.183+0000 7f9c0d200640 1 --2- 172.21.15.43:0/2056622393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080cf500 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.183+0000 7f9c0d200640 1 -- 172.21.15.43:0/2056622393 >> 172.21.15.43:0/2056622393 conn(0x7f9c0806a860 msgr2=0x7f9c0806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.183+0000 7f9c0d200640 1 -- 172.21.15.43:0/2056622393 shutdown_connections 2024-01-16T15:18:51.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.183+0000 7f9c0d200640 1 -- 172.21.15.43:0/2056622393 wait complete. 2024-01-16T15:18:51.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.184+0000 7f9c0d200640 1 Processor -- start 2024-01-16T15:18:51.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.184+0000 7f9c0d200640 1 -- start start 2024-01-16T15:18:51.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 0x7f9c0813cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 0x7f9c081f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080c95a0 con 0x7f9c080cf050 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080c9880 con 0x7f9c0813dfd0 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9c0d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080c9b60 con 0x7f9c080cd7b0 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.185+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39380/0 (socket says 172.21.15.43:39380) 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 -- 172.21.15.43:0/3451109297 learned_addr learned my addr 172.21.15.43:0/3451109297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9c07400640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 0x7f9c0813cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9c07e00640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 0x7f9c081f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 -- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 msgr2=0x7f9c0813cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 0x7f9c0813cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 -- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 msgr2=0x7f9c081f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 0x7f9c081f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 -- 172.21.15.43:0/3451109297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00094d0 con 0x7f9c080cf050 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9c07e00640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 0x7f9c081f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9c07400640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 0x7f9c0813cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.186+0000 7f9bffe00640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009870 tx=0x7f9bf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.187+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0030810 con 0x7f9c080cf050 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.187+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0043400 con 0x7f9c080cf050 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.187+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf00437b0 con 0x7f9c080cf050 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.187+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c080c9e40 con 0x7f9c080cf050 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.187+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0813d290 con 0x7f9c080cf050 2024-01-16T15:18:51.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.188+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0815eb10 con 0x7f9c080cf050 2024-01-16T15:18:51.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.189+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9bf0043910 con 0x7f9c080cf050 2024-01-16T15:18:51.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.190+0000 7f9bff400640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 0x7f9be006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.190+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9bf003e070 con 0x7f9c080cf050 2024-01-16T15:18:51.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.190+0000 7f9c07400640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 0x7f9be006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.191+0000 7f9c07400640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 0x7f9be006e860 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800a830 tx=0x7f9bf8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.194+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c0815eb10 con 0x7f9c080cf050 2024-01-16T15:18:51.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.392+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c080ca360 con 0x7f9c080cf050 2024-01-16T15:18:51.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.395+0000 7f9bff400640 1 -- 172.21.15.43:0/3451109297 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16535 (secure 0 0 0) 0x7f9c080ca360 con 0x7f9c080cf050 2024-01-16T15:18:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:51.385 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":112,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:49.964607+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:51.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 msgr2=0x7f9be006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 0x7f9be006e860 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800a830 tx=0x7f9bf8008040 comp rx=0 tx=0).stop 2024-01-16T15:18:51.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 msgr2=0x7f9c080c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009870 tx=0x7f9bf0004730 comp rx=0 tx=0).stop 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 shutdown_connections 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9be006c3a0 0x7f9be006e860 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c0813dfd0 0x7f9c081f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c080cf050 0x7f9c080c8c30 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 --2- 172.21.15.43:0/3451109297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c080cd7b0 0x7f9c0813cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 >> 172.21.15.43:0/3451109297 conn(0x7f9c0806a860 msgr2=0x7f9c0815d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 shutdown_connections 2024-01-16T15:18:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.398+0000 7f9c0d200640 1 -- 172.21.15.43:0/3451109297 wait complete. 2024-01-16T15:18:51.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:18:51.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.635+0000 7f7fd1600640 1 Processor -- start 2024-01-16T15:18:51.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.636+0000 7f7fd1600640 1 -- start start 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.637+0000 7f7fd1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 0x7f7fcc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.637+0000 7f7fd1600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7fcc0d0fa0 0x7f7fcc0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.637+0000 7f7fd1600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0c6bf0 con 0x7f7fcc15d3b0 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.637+0000 7f7fd1600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0d13f0 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.637+0000 7f7fd1600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cedc0 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fcbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7fcc15d3b0 0x7f7fcc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36556/0 (socket says 172.21.15.43:36556) 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fcbe00640 1 -- 172.21.15.43:0/552256912 learned_addr learned my addr 172.21.15.43:0/552256912 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:51.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fd0c00640 1 --2- 172.21.15.43:0/552256912 >> v2:172.21.15.43:3300/0 conn(0x7f7fcc0d0fa0 0x7f7fcc0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fcb400640 1 --2- 172.21.15.43:0/552256912 >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 0x7f7fcc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fd0c00640 1 -- 172.21.15.43:0/552256912 >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 msgr2=0x7f7fcc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fd0c00640 1 --2- 172.21.15.43:0/552256912 >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 0x7f7fcc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fd0c00640 1 -- 172.21.15.43:0/552256912 >> v1:172.21.15.43:6789/0 conn(0x7f7fcc15d3b0 legacy=0x7f7fcc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fd0c00640 1 -- 172.21.15.43:0/552256912 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0c6e40 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.638+0000 7f7fcb400640 1 --2- 172.21.15.43:0/552256912 >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 0x7f7fcc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.639+0000 7f7fd0c00640 1 --2- 172.21.15.43:0/552256912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc0b3ca0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0ca100 tx=0x7f7fc402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1bca736ca51434 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fcaa00640 1 -- 172.21.15.43:0/552256912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc402cbe0 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fcaa00640 1 -- 172.21.15.43:0/552256912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc402cd40 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fcaa00640 1 -- 172.21.15.43:0/552256912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4034660 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 -- 172.21.15.43:0/552256912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d0fa0 msgr2=0x7f7fcc0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 --2- 172.21.15.43:0/552256912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc0b3ca0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0ca100 tx=0x7f7fc402be30 comp rx=0 tx=0).stop 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 -- 172.21.15.43:0/552256912 shutdown_connections 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 --2- 172.21.15.43:0/552256912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc0b3ca0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 --2- 172.21.15.43:0/552256912 >> v2:172.21.15.69:3300/0 conn(0x7f7fcc0d1d90 0x7f7fcc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 -- 172.21.15.43:0/552256912 >> 172.21.15.43:0/552256912 conn(0x7f7fcc0d30a0 msgr2=0x7f7fcc0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.640+0000 7f7fd1600640 1 -- 172.21.15.43:0/552256912 shutdown_connections 2024-01-16T15:18:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.641+0000 7f7fd1600640 1 -- 172.21.15.43:0/552256912 wait complete. 2024-01-16T15:18:51.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.642+0000 7f7fd1600640 1 Processor -- start 2024-01-16T15:18:51.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.642+0000 7f7fd1600640 1 -- start start 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc14d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc15d3b0 0x7f7fcc15a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cbc00 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc14db20 con 0x7f7fcc0d0fa0 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fd1600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0caff0 con 0x7f7fcc15d3b0 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fcb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fcb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39412/0 (socket says 172.21.15.43:39412) 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.643+0000 7f7fcb400640 1 -- 172.21.15.43:0/891170212 learned_addr learned my addr 172.21.15.43:0/891170212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcbe00640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc14d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fd0c00640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc15d3b0 0x7f7fcc15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 -- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc15d3b0 msgr2=0x7f7fcc15a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc15d3b0 0x7f7fcc15a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 -- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc0d0fa0 msgr2=0x7f7fcc14d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc14d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 -- 172.21.15.43:0/891170212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4009a70 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.644+0000 7f7fcb400640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c7f0 tx=0x7f7fb400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.645+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4007a30 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.645+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4007b90 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.645+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400ce40 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.645+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc0cb2d0 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.645+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc0cb890 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.646+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc0cbd70 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.647+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f7fb40113f0 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.648+0000 7f7fc3e00640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 0x7f7fa406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:51.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.648+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7fb4088c50 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.648+0000 7f7fcbe00640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 0x7f7fa406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:51.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.649+0000 7f7fcbe00640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 0x7f7fa406e860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc005880 tx=0x7f7fbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:51.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.653+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fb40551d0 con 0x7f7fcc0d1d90 2024-01-16T15:18:51.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:51.883+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7fcc04e840 con 0x7f7fcc0d1d90 2024-01-16T15:18:52.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.511+0000 7f7fc3e00640 1 -- 172.21.15.43:0/891170212 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7fb401f0b0 con 0x7f7fcc0d1d90 2024-01-16T15:18:52.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 msgr2=0x7f7fa406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 0x7f7fa406e860 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc005880 tx=0x7f7fbc009040 comp rx=0 tx=0).stop 2024-01-16T15:18:52.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 msgr2=0x7f7fcc0b80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c7f0 tx=0x7f7fb400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 shutdown_connections 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7fa406c3a0 0x7f7fa406e860 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fcc15d3b0 0x7f7fcc15a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fcc0d1d90 0x7f7fcc0b80d0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 --2- 172.21.15.43:0/891170212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fcc0d0fa0 0x7f7fcc14d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.515+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 >> 172.21.15.43:0/891170212 conn(0x7f7fcc0d30a0 msgr2=0x7f7fcc06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:52.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.516+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 shutdown_connections 2024-01-16T15:18:52.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.516+0000 7f7fd1600640 1 -- 172.21.15.43:0/891170212 wait complete. 2024-01-16T15:18:52.526 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:18:52.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:18:52.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.764+0000 7f334ca00640 1 Processor -- start 2024-01-16T15:18:52.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.764+0000 7f334ca00640 1 -- start start 2024-01-16T15:18:52.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.766+0000 7f334ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f334815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.766+0000 7f334ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f33480b35a0 0x7f334815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.766+0000 7f334ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334815f930 con 0x7f334806c9d0 2024-01-16T15:18:52.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.766+0000 7f334ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33480b43b0 con 0x7f33480b35a0 2024-01-16T15:18:52.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.766+0000 7f334ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33480b3b10 con 0x7f3348144770 2024-01-16T15:18:52.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f33480b35a0 0x7f334815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f3347400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f334815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f33480b35a0 0x7f334815f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:52550/0 (socket says 172.21.15.43:52550) 2024-01-16T15:18:52.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 -- 172.21.15.43:0/55824613 learned_addr learned my addr 172.21.15.43:0/55824613 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:52.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 -- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 msgr2=0x7f334815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:52.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 --2- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f334815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 -- 172.21.15.43:0/55824613 >> v1:172.21.15.69:6789/0 conn(0x7f334806c9d0 legacy=0x7f334806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.767+0000 7f333e800640 1 -- 172.21.15.43:0/55824613 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480b3df0 con 0x7f33480b35a0 2024-01-16T15:18:52.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.769+0000 7f333e800640 1 --2- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33480b35a0 0x7f334815f0a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f33300098a0 tx=0x7f333002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc0c7fe3612d347 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:52.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.769+0000 7f3346a00640 1 -- 172.21.15.43:0/55824613 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330030840 con 0x7f33480b35a0 2024-01-16T15:18:52.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.769+0000 7f3346a00640 1 -- 172.21.15.43:0/55824613 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33300309a0 con 0x7f33480b35a0 2024-01-16T15:18:52.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.769+0000 7f3346a00640 1 -- 172.21.15.43:0/55824613 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330030d30 con 0x7f33480b35a0 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 -- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33480b35a0 msgr2=0x7f334815f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 --2- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33480b35a0 0x7f334815f0a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f33300098a0 tx=0x7f333002b980 comp rx=0 tx=0).stop 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 -- 172.21.15.43:0/55824613 shutdown_connections 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 --2- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33480b35a0 0x7f334815f0a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 --2- 172.21.15.43:0/55824613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f334815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 -- 172.21.15.43:0/55824613 >> 172.21.15.43:0/55824613 conn(0x7f33480cec10 msgr2=0x7f33480cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 -- 172.21.15.43:0/55824613 shutdown_connections 2024-01-16T15:18:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.770+0000 7f334ca00640 1 -- 172.21.15.43:0/55824613 wait complete. 2024-01-16T15:18:52.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.772+0000 7f334ca00640 1 Processor -- start 2024-01-16T15:18:52.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.772+0000 7f334ca00640 1 -- start start 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f334806c9d0 0x7f334813e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f3348159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481f7fb0 con 0x7f33480b35a0 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334813eba0 con 0x7f334806c9d0 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.773+0000 7f334ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334813ee80 con 0x7f3348144770 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39418/0 (socket says 172.21.15.43:39418) 2024-01-16T15:18:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 -- 172.21.15.43:0/1915190650 learned_addr learned my addr 172.21.15.43:0/1915190650 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f3347400640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f334806c9d0 0x7f334813e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f3347e00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f3348159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 -- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 msgr2=0x7f3348159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f3348159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 -- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f334806c9d0 msgr2=0x7f334813e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f334806c9d0 0x7f334813e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f333e800640 1 -- 172.21.15.43:0/1915190650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33300094d0 con 0x7f33480b35a0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.774+0000 7f3347e00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f3348159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f333e800640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f3330000c00 tx=0x7f3330004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330039c90 con 0x7f33480b35a0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3330039e10 con 0x7f33480b35a0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330043640 con 0x7f33480b35a0 2024-01-16T15:18:52.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334813f160 con 0x7f33480b35a0 2024-01-16T15:18:52.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.775+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334813f720 con 0x7f33480b35a0 2024-01-16T15:18:52.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.776+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334813f910 con 0x7f33480b35a0 2024-01-16T15:18:52.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.777+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f33300437a0 con 0x7f33480b35a0 2024-01-16T15:18:52.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.779+0000 7f333fe00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 0x7f332006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:52.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.779+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f333003e070 con 0x7f33480b35a0 2024-01-16T15:18:52.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.779+0000 7f3347400640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 0x7f332006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:52.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.780+0000 7f3347400640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 0x7f332006e860 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f334815b240 tx=0x7f3338008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:52.783+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3330078060 con 0x7f33480b35a0 2024-01-16T15:18:53.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.021+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f334804e840 con 0x7f33480b35a0 2024-01-16T15:18:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.516+0000 7f333fe00640 1 -- 172.21.15.43:0/1915190650 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7f33300340c0 con 0x7f33480b35a0 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 msgr2=0x7f332006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 0x7f332006e860 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f334815b240 tx=0x7f3338008040 comp rx=0 tx=0).stop 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 msgr2=0x7f33481f7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f3330000c00 tx=0x7f3330004730 comp rx=0 tx=0).stop 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 shutdown_connections 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f332006c3a0 0x7f332006e860 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3348144770 0x7f3348159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33480b35a0 0x7f33481f7a70 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 --2- 172.21.15.43:0/1915190650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f334806c9d0 0x7f334813e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 >> 172.21.15.43:0/1915190650 conn(0x7f33480cec10 msgr2=0x7f3348143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 shutdown_connections 2024-01-16T15:18:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.520+0000 7f334ca00640 1 -- 172.21.15.43:0/1915190650 wait complete. 2024-01-16T15:18:53.535 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:18:53.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:18:53.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.754+0000 7f99d6800640 1 Processor -- start 2024-01-16T15:18:53.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.755+0000 7f99d6800640 1 -- start start 2024-01-16T15:18:53.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.756+0000 7f99d6800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99d00d14a0 0x7f99d00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.756+0000 7f99d6800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d00b0640 0x7f99d00b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.756+0000 7f99d6800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d00cf0c0 con 0x7f99d015d5b0 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.756+0000 7f99d6800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d00b01b0 con 0x7f99d00d14a0 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.756+0000 7f99d6800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d015d990 con 0x7f99d00b0640 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.757+0000 7f99d4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99d00d14a0 0x7f99d00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.757+0000 7f99d4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99d00d14a0 0x7f99d00d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38248/0 (socket says 172.21.15.43:38248) 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.757+0000 7f99d4a00640 1 -- 172.21.15.43:0/12385754 learned_addr learned my addr 172.21.15.43:0/12385754 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:53.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.757+0000 7f99d5e00640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d00b0640 0x7f99d00b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.758+0000 7f99d4a00640 1 -- 172.21.15.43:0/12385754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d00b0640 msgr2=0x7f99d00b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.758+0000 7f99d4a00640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d00b0640 0x7f99d00b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.758+0000 7f99d4a00640 1 -- 172.21.15.43:0/12385754 >> v1:172.21.15.69:6789/0 conn(0x7f99d015d5b0 legacy=0x7f99d00d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.758+0000 7f99d4a00640 1 -- 172.21.15.43:0/12385754 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d00d1e40 con 0x7f99d00d14a0 2024-01-16T15:18:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.759+0000 7f99d4a00640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d00d1900 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009870 tx=0x7f99b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2270cdca2dea8104 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:53.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.759+0000 7f99cfe00640 1 -- 172.21.15.43:0/12385754 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8030840 con 0x7f99d00d14a0 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.759+0000 7f99cfe00640 1 -- 172.21.15.43:0/12385754 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b80309a0 con 0x7f99d00d14a0 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.759+0000 7f99cfe00640 1 -- 172.21.15.43:0/12385754 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8030d30 con 0x7f99d00d14a0 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 -- 172.21.15.43:0/12385754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 msgr2=0x7f99d00d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d00d1900 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009870 tx=0x7f99b802b980 comp rx=0 tx=0).stop 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 -- 172.21.15.43:0/12385754 shutdown_connections 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d00d1900 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 --2- 172.21.15.43:0/12385754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d00b0640 0x7f99d00b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 -- 172.21.15.43:0/12385754 >> 172.21.15.43:0/12385754 conn(0x7f99d009c3b0 msgr2=0x7f99d00d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 -- 172.21.15.43:0/12385754 shutdown_connections 2024-01-16T15:18:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.760+0000 7f99d6800640 1 -- 172.21.15.43:0/12385754 wait complete. 2024-01-16T15:18:53.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.762+0000 7f99d6800640 1 Processor -- start 2024-01-16T15:18:53.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.762+0000 7f99d6800640 1 -- start start 2024-01-16T15:18:53.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 0x7f99d015b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d015d5b0 0x7f99d00cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d015b650 con 0x7f99d00d14a0 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d0151d50 con 0x7f99d00b0640 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d6800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00cc100 con 0x7f99d015d5b0 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38256/0 (socket says 172.21.15.43:38256) 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.763+0000 7f99d4a00640 1 -- 172.21.15.43:0/3747881454 learned_addr learned my addr 172.21.15.43:0/3747881454 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d5400640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 0x7f99d015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d5e00640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d015d5b0 0x7f99d00cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 -- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d015d5b0 msgr2=0x7f99d00cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d015d5b0 0x7f99d00cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 -- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 msgr2=0x7f99d015b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 0x7f99d015b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 -- 172.21.15.43:0/3747881454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80094d0 con 0x7f99d00d14a0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d5400640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 0x7f99d015b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.764+0000 7f99d4a00640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f99b8000c00 tx=0x7f99b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.765+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8039a90 con 0x7f99d00d14a0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.765+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b8039bf0 con 0x7f99d00d14a0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.765+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8043410 con 0x7f99d00d14a0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.765+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d00cc3e0 con 0x7f99d00d14a0 2024-01-16T15:18:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.765+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d00c6c40 con 0x7f99d00d14a0 2024-01-16T15:18:53.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.766+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d004e840 con 0x7f99d00d14a0 2024-01-16T15:18:53.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.767+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f99b802f390 con 0x7f99d00d14a0 2024-01-16T15:18:53.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.768+0000 7f99cd600640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 0x7f99a806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:53.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.768+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99b80aa950 con 0x7f99d00d14a0 2024-01-16T15:18:53.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.769+0000 7f99d5400640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 0x7f99a806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:53.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.769+0000 7f99d5400640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 0x7f99a806e860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001660 tx=0x7f99c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:53.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:53.773+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99b8077210 con 0x7f99d00d14a0 2024-01-16T15:18:53.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.007+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99d00678d0 con 0x7f99d00d14a0 2024-01-16T15:18:53.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.008+0000 7f99cd600640 1 -- 172.21.15.43:0/3747881454 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f99b8048090 con 0x7f99d00d14a0 2024-01-16T15:18:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:53.999 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3583583840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":3583583840},{"type":"v1","addr":"172.21.15.43:6837","nonce":3583583840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2071477429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2071477429},{"type":"v1","addr":"172.21.15.69:6833","nonce":2071477429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/4168252148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4168252148},{"type":"v1","addr":"172.21.15.43:6836","nonce":4168252148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:18:49.965316+0000","modified":"2024-01-16T15:18:53.023347+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9132":{"gid":9132,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/3715375194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3715375194},{"type":"v1","addr":"172.21.15.69:6835","nonce":3715375194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:18:53.999 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 75 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 msgr2=0x7f99a806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 0x7f99a806e860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001660 tx=0x7f99c0009040 comp rx=0 tx=0).stop 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 msgr2=0x7f99d0151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f99b8000c00 tx=0x7f99b8004730 comp rx=0 tx=0).stop 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 shutdown_connections 2024-01-16T15:18:54.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f99a806c3a0 0x7f99a806e860 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d015d5b0 0x7f99d00cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99d00d14a0 0x7f99d0151810 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 --2- 172.21.15.43:0/3747881454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d00b0640 0x7f99d015b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 >> 172.21.15.43:0/3747881454 conn(0x7f99d009c3b0 msgr2=0x7f99d00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 shutdown_connections 2024-01-16T15:18:54.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.011+0000 7f99d6800640 1 -- 172.21.15.43:0/3747881454 wait complete. 2024-01-16T15:18:54.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:18:54.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.240+0000 7f64dba00640 1 Processor -- start 2024-01-16T15:18:54.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.241+0000 7f64dba00640 1 -- start start 2024-01-16T15:18:54.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64dba00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64d406d520 0x7f64d406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64dba00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f64d40abff0 0x7f64d415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64dba00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d415c360 con 0x7f64d40ad720 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64dba00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d40b41a0 con 0x7f64d40abff0 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64dba00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d406de40 con 0x7f64d406d520 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.242+0000 7f64cbe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f64d40ad720 0x7f64d415ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35508/0 (socket says 172.21.15.43:35508) 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64cbe00640 1 -- 172.21.15.43:0/992710498 learned_addr learned my addr 172.21.15.43:0/992710498 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9a00640 1 --2- 172.21.15.43:0/992710498 >> v2:172.21.15.43:3300/0 conn(0x7f64d40abff0 0x7f64d415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9000640 1 --2- 172.21.15.43:0/992710498 >> v2:172.21.15.69:3300/0 conn(0x7f64d406d520 0x7f64d406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d3e00640 1 -- 172.21.15.43:0/992710498 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867166140 0 0) 0x7f64d415c360 con 0x7f64d40ad720 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d3e00640 1 -- 172.21.15.43:0/992710498 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4003520 con 0x7f64d40ad720 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9a00640 1 -- 172.21.15.43:0/992710498 >> v2:172.21.15.69:3300/0 conn(0x7f64d406d520 msgr2=0x7f64d406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:54.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9a00640 1 --2- 172.21.15.43:0/992710498 >> v2:172.21.15.69:3300/0 conn(0x7f64d406d520 0x7f64d406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9a00640 1 -- 172.21.15.43:0/992710498 >> v1:172.21.15.43:6789/0 conn(0x7f64d40ad720 legacy=0x7f64d415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.243+0000 7f64d9a00640 1 -- 172.21.15.43:0/992710498 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d415ba30 con 0x7f64d40abff0 2024-01-16T15:18:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.244+0000 7f64d9a00640 1 --2- 172.21.15.43:0/992710498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d40abff0 0x7f64d415f3e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f64d406c070 tx=0x7f64cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdf6bff2ad9691d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64d3e00640 1 -- 172.21.15.43:0/992710498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc02cc80 con 0x7f64d40abff0 2024-01-16T15:18:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64d3e00640 1 -- 172.21.15.43:0/992710498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc02cde0 con 0x7f64d40abff0 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64d3e00640 1 -- 172.21.15.43:0/992710498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc034660 con 0x7f64d40abff0 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 -- 172.21.15.43:0/992710498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d40abff0 msgr2=0x7f64d415f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 --2- 172.21.15.43:0/992710498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d40abff0 0x7f64d415f3e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f64d406c070 tx=0x7f64cc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 -- 172.21.15.43:0/992710498 shutdown_connections 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 --2- 172.21.15.43:0/992710498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d40abff0 0x7f64d415f3e0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 --2- 172.21.15.43:0/992710498 >> v2:172.21.15.69:3300/0 conn(0x7f64d406d520 0x7f64d406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.245+0000 7f64dba00640 1 -- 172.21.15.43:0/992710498 >> 172.21.15.43:0/992710498 conn(0x7f64d40b0d80 msgr2=0x7f64d40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.246+0000 7f64dba00640 1 -- 172.21.15.43:0/992710498 shutdown_connections 2024-01-16T15:18:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.246+0000 7f64dba00640 1 -- 172.21.15.43:0/992710498 wait complete. 2024-01-16T15:18:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.247+0000 7f64dba00640 1 Processor -- start 2024-01-16T15:18:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.247+0000 7f64dba00640 1 -- start start 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 0x7f64d4070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64d40ad720 0x7f64d415d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d415de60 con 0x7f64d406d520 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40704b0 con 0x7f64d40abff0 2024-01-16T15:18:54.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.248+0000 7f64dba00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4070a70 con 0x7f64d40ad720 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38284/0 (socket says 172.21.15.43:38284) 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 -- 172.21.15.43:0/2382004711 learned_addr learned my addr 172.21.15.43:0/2382004711 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64cbe00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 0x7f64d4070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9a00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64d40ad720 0x7f64d415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 -- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64d40ad720 msgr2=0x7f64d415d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64d40ad720 0x7f64d415d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 -- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 msgr2=0x7f64d4070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 0x7f64d4070f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64d9000640 1 -- 172.21.15.43:0/2382004711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc009a70 con 0x7f64d406d520 2024-01-16T15:18:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.249+0000 7f64cbe00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 0x7f64d4070f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64d9000640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e9b0 tx=0x7f64c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4009670 con 0x7f64d406d520 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c400c9a0 con 0x7f64d406d520 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c400cd50 con 0x7f64d406d520 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d415e070 con 0x7f64d406d520 2024-01-16T15:18:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.250+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d4070700 con 0x7f64d406d520 2024-01-16T15:18:54.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.251+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d4074770 con 0x7f64d406d520 2024-01-16T15:18:54.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.252+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f64c4013470 con 0x7f64d406d520 2024-01-16T15:18:54.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.253+0000 7f64d1600640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 0x7f64ac06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.253+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f64c4089070 con 0x7f64d406d520 2024-01-16T15:18:54.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.253+0000 7f64cbe00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 0x7f64ac06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.254+0000 7f64cbe00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 0x7f64ac06e860 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f64d41540d0 tx=0x7f64bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.257+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64c40555f0 con 0x7f64d406d520 2024-01-16T15:18:54.439 INFO:tasks.ceph.mds.d.smithi069.stderr:2024-01-16T15:18:54.447+0000 7f5b5ac00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:18:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.452+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64d40b0850 con 0x7f64d406d520 2024-01-16T15:18:54.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.455+0000 7f64d1600640 1 -- 172.21.15.43:0/2382004711 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16522 (secure 0 0 0) 0x7f64c4021090 con 0x7f64d406d520 2024-01-16T15:18:54.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:54.446 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":113,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:18:51.499510+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:18:54.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.457+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 msgr2=0x7f64ac06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.457+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 0x7f64ac06e860 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f64d41540d0 tx=0x7f64bc009040 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.457+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 msgr2=0x7f64d40720b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.457+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e9b0 tx=0x7f64c400ee70 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 shutdown_connections 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f64ac06c3a0 0x7f64ac06e860 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64d40abff0 0x7f64d4070f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64d40ad720 0x7f64d415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 --2- 172.21.15.43:0/2382004711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64d406d520 0x7f64d40720b0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 >> 172.21.15.43:0/2382004711 conn(0x7f64d40b0d80 msgr2=0x7f64d40aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 shutdown_connections 2024-01-16T15:18:54.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.458+0000 7f64dba00640 1 -- 172.21.15.43:0/2382004711 wait complete. 2024-01-16T15:18:54.468 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:18:54.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:18:54.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.701+0000 7f0d2a400640 1 Processor -- start 2024-01-16T15:18:54.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.701+0000 7f0d2a400640 1 -- start start 2024-01-16T15:18:54.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d2406d520 0x7f0d2406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d240ad720 0x7f0d2415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d240abff0 0x7f0d2415f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240b41a0 con 0x7f0d240ad720 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2415c360 con 0x7f0d2406d520 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d2a400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2406de40 con 0x7f0d240abff0 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d1a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d240ad720 0x7f0d2415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d1a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d240ad720 0x7f0d2415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38296/0 (socket says 172.21.15.43:38296) 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.703+0000 7f0d1a800640 1 -- 172.21.15.43:0/2732383231 learned_addr learned my addr 172.21.15.43:0/2732383231 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d23e00640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d240abff0 0x7f0d2415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d23400640 1 --2- 172.21.15.43:0/2732383231 >> v2:172.21.15.69:3300/0 conn(0x7f0d2406d520 0x7f0d2406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d1a800640 1 -- 172.21.15.43:0/2732383231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d240abff0 msgr2=0x7f0d2415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d240abff0 0x7f0d2415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d1a800640 1 -- 172.21.15.43:0/2732383231 >> v2:172.21.15.69:3300/0 conn(0x7f0d2406d520 msgr2=0x7f0d2406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2732383231 >> v2:172.21.15.69:3300/0 conn(0x7f0d2406d520 0x7f0d2406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.704+0000 7f0d1a800640 1 -- 172.21.15.43:0/2732383231 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2415ba30 con 0x7f0d240ad720 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.705+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240ad720 0x7f0d2415ecd0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f0d2406c070 tx=0x7f0d0c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bb5910a6be9bfe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.705+0000 7f0d22a00640 1 -- 172.21.15.43:0/2732383231 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c031860 con 0x7f0d240ad720 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.705+0000 7f0d22a00640 1 -- 172.21.15.43:0/2732383231 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c0319c0 con 0x7f0d240ad720 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.705+0000 7f0d22a00640 1 -- 172.21.15.43:0/2732383231 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c031d50 con 0x7f0d240ad720 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 -- 172.21.15.43:0/2732383231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240ad720 msgr2=0x7f0d2415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240ad720 0x7f0d2415ecd0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f0d2406c070 tx=0x7f0d0c02b900 comp rx=0 tx=0).stop 2024-01-16T15:18:54.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 -- 172.21.15.43:0/2732383231 shutdown_connections 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d240abff0 0x7f0d2415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2732383231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240ad720 0x7f0d2415ecd0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2732383231 >> v2:172.21.15.69:3300/0 conn(0x7f0d2406d520 0x7f0d2406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 -- 172.21.15.43:0/2732383231 >> 172.21.15.43:0/2732383231 conn(0x7f0d240b0d80 msgr2=0x7f0d240b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 -- 172.21.15.43:0/2732383231 shutdown_connections 2024-01-16T15:18:54.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.706+0000 7f0d2a400640 1 -- 172.21.15.43:0/2732383231 wait complete. 2024-01-16T15:18:54.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.708+0000 7f0d2a400640 1 Processor -- start 2024-01-16T15:18:54.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.708+0000 7f0d2a400640 1 -- start start 2024-01-16T15:18:54.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 0x7f0d24155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d240ad720 0x7f0d2415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24155bb0 con 0x7f0d240abff0 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240cadc0 con 0x7f0d240ad720 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d2a400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24071290 con 0x7f0d2406d520 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.709+0000 7f0d1a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38310/0 (socket says 172.21.15.43:38310) 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 -- 172.21.15.43:0/2857922033 learned_addr learned my addr 172.21.15.43:0/2857922033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d23400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 0x7f0d24155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d23e00640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d240ad720 0x7f0d2415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 -- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 msgr2=0x7f0d24155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 0x7f0d24155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 -- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d240ad720 msgr2=0x7f0d2415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d240ad720 0x7f0d2415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 -- 172.21.15.43:0/2857922033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0094d0 con 0x7f0d240abff0 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d23400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 0x7f0d24155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.710+0000 7f0d1a800640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c02b940 tx=0x7f0d0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.711+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c03ad30 con 0x7f0d240abff0 2024-01-16T15:18:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.711+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c0443f0 con 0x7f0d240abff0 2024-01-16T15:18:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.711+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c0447a0 con 0x7f0d240abff0 2024-01-16T15:18:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.711+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d240c9da0 con 0x7f0d240abff0 2024-01-16T15:18:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.711+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d240ca360 con 0x7f0d240abff0 2024-01-16T15:18:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.712+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d240ac840 con 0x7f0d240abff0 2024-01-16T15:18:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.713+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0d0c04d450 con 0x7f0d240abff0 2024-01-16T15:18:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.714+0000 7f0d1be00640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 0x7f0cfc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.714+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0d0c0aad20 con 0x7f0d240abff0 2024-01-16T15:18:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.714+0000 7f0d23400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 0x7f0cfc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.715+0000 7f0d23400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 0x7f0cfc06e860 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0d14006fd0 tx=0x7f0d14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.718+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d0c0775e0 con 0x7f0d240abff0 2024-01-16T15:18:54.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.909+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d2415b6e0 con 0x7f0cfc06c3a0 2024-01-16T15:18:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.933+0000 7f0d1be00640 1 -- 172.21.15.43:0/2857922033 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282614 (secure 0 0 0) 0x7f0d1400f830 con 0x7f0cfc06c3a0 2024-01-16T15:18:54.928 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 msgr2=0x7f0cfc06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 0x7f0cfc06e860 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0d14006fd0 tx=0x7f0d14008040 comp rx=0 tx=0).stop 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 msgr2=0x7f0d240ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c02b940 tx=0x7f0d0c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 shutdown_connections 2024-01-16T15:18:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0cfc06c3a0 0x7f0cfc06e860 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d240ad720 0x7f0d2415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d240abff0 0x7f0d240ca880 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 --2- 172.21.15.43:0/2857922033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d2406d520 0x7f0d24155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.938+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 >> 172.21.15.43:0/2857922033 conn(0x7f0d240b0d80 msgr2=0x7f0d240af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.939+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 shutdown_connections 2024-01-16T15:18:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:54.939+0000 7f0d2a400640 1 -- 172.21.15.43:0/2857922033 wait complete. 2024-01-16T15:18:54.951 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2024-01-16T15:18:53.289793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592092,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":170,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606496,"kb_used_data":11900,"kb_used_omap":12,"kb_used_meta":594483,"kb_avail":754368224,"statfs":{"total":773094113280,"available":772473061376,"internally_reserved":0,"allocated":12185600,"data_stored":5922525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":608751148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.818006"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520878+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:18:50.520878+0000","last_peered":"2024-01-16T15:18:50.520878+0000","last_clean":"2024-01-16T15:18:50.520878+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:18:50.520878+0000","last_undegraded":"2024-01-16T15:18:50.520878+0000","last_fullsized":"2024-01-16T15:18:50.520878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:02.958841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496455+0000","last_change":"2024-01-16T15:18:49.496455+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496455+0000","last_undegraded":"2024-01-16T15:18:49.496455+0000","last_fullsized":"2024-01-16T15:18:49.496455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511018+0000","last_change":"2024-01-16T15:18:49.511018+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511018+0000","last_undegraded":"2024-01-16T15:18:49.511018+0000","last_fullsized":"2024-01-16T15:18:49.511018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527282+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:18:50.527282+0000","last_peered":"2024-01-16T15:18:50.527282+0000","last_clean":"2024-01-16T15:18:50.527282+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:18:50.527282+0000","last_undegraded":"2024-01-16T15:18:50.527282+0000","last_fullsized":"2024-01-16T15:18:50.527282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:40.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494552+0000","last_change":"2024-01-16T15:18:49.494552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494552+0000","last_undegraded":"2024-01-16T15:18:49.494552+0000","last_fullsized":"2024-01-16T15:18:49.494552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511599+0000","last_change":"2024-01-16T15:18:49.511599+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511599+0000","last_undegraded":"2024-01-16T15:18:49.511599+0000","last_fullsized":"2024-01-16T15:18:49.511599+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502722+0000","last_change":"2024-01-16T15:18:49.502722+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502722+0000","last_undegraded":"2024-01-16T15:18:49.502722+0000","last_fullsized":"2024-01-16T15:18:49.502722+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507251+0000","last_change":"2024-01-16T15:18:49.507251+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507251+0000","last_undegraded":"2024-01-16T15:18:49.507251+0000","last_fullsized":"2024-01-16T15:18:49.507251+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527409+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:18:50.527409+0000","last_peered":"2024-01-16T15:18:50.527409+0000","last_clean":"2024-01-16T15:18:50.527409+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:18:50.527409+0000","last_undegraded":"2024-01-16T15:18:50.527409+0000","last_fullsized":"2024-01-16T15:18:50.527409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:44.209808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513531+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:18:50.513531+0000","last_peered":"2024-01-16T15:18:50.513531+0000","last_clean":"2024-01-16T15:18:50.513531+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:18:50.513531+0000","last_undegraded":"2024-01-16T15:18:50.513531+0000","last_fullsized":"2024-01-16T15:18:50.513531+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:37.753051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496236+0000","last_change":"2024-01-16T15:18:49.496236+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496236+0000","last_undegraded":"2024-01-16T15:18:49.496236+0000","last_fullsized":"2024-01-16T15:18:49.496236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513845+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:18:50.513845+0000","last_peered":"2024-01-16T15:18:50.513845+0000","last_clean":"2024-01-16T15:18:50.513845+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:18:50.513845+0000","last_undegraded":"2024-01-16T15:18:50.513845+0000","last_fullsized":"2024-01-16T15:18:50.513845+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509738+0000","last_change":"2024-01-16T15:18:49.509738+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509738+0000","last_undegraded":"2024-01-16T15:18:49.509738+0000","last_fullsized":"2024-01-16T15:18:49.509738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505304+0000","last_change":"2024-01-16T15:18:49.505304+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505304+0000","last_undegraded":"2024-01-16T15:18:49.505304+0000","last_fullsized":"2024-01-16T15:18:49.505304+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509868+0000","last_change":"2024-01-16T15:18:49.509868+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509868+0000","last_undegraded":"2024-01-16T15:18:49.509868+0000","last_fullsized":"2024-01-16T15:18:49.509868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518744+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:18:50.518744+0000","last_peered":"2024-01-16T15:18:50.518744+0000","last_clean":"2024-01-16T15:18:50.518744+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:18:50.518744+0000","last_undegraded":"2024-01-16T15:18:50.518744+0000","last_fullsized":"2024-01-16T15:18:50.518744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:42.136923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495707+0000","last_change":"2024-01-16T15:18:49.495707+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495707+0000","last_undegraded":"2024-01-16T15:18:49.495707+0000","last_fullsized":"2024-01-16T15:18:49.495707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501685+0000","last_change":"2024-01-16T15:18:49.501685+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501685+0000","last_undegraded":"2024-01-16T15:18:49.501685+0000","last_fullsized":"2024-01-16T15:18:49.501685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509552+0000","last_change":"2024-01-16T15:18:49.509552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509552+0000","last_undegraded":"2024-01-16T15:18:49.509552+0000","last_fullsized":"2024-01-16T15:18:49.509552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494776+0000","last_change":"2024-01-16T15:18:49.494776+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494776+0000","last_undegraded":"2024-01-16T15:18:49.494776+0000","last_fullsized":"2024-01-16T15:18:49.494776+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494903+0000","last_change":"2024-01-16T15:18:49.494903+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494903+0000","last_undegraded":"2024-01-16T15:18:49.494903+0000","last_fullsized":"2024-01-16T15:18:49.494903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498780+0000","last_change":"2024-01-16T15:18:49.498780+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498780+0000","last_undegraded":"2024-01-16T15:18:49.498780+0000","last_fullsized":"2024-01-16T15:18:49.498780+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494905+0000","last_change":"2024-01-16T15:18:49.494905+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494905+0000","last_undegraded":"2024-01-16T15:18:49.494905+0000","last_fullsized":"2024-01-16T15:18:49.494905+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498751+0000","last_change":"2024-01-16T15:18:49.498751+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498751+0000","last_undegraded":"2024-01-16T15:18:49.498751+0000","last_fullsized":"2024-01-16T15:18:49.498751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495191+0000","last_change":"2024-01-16T15:18:49.495191+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495191+0000","last_undegraded":"2024-01-16T15:18:49.495191+0000","last_fullsized":"2024-01-16T15:18:49.495191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518712+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:18:50.518712+0000","last_peered":"2024-01-16T15:18:50.518712+0000","last_clean":"2024-01-16T15:18:50.518712+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:18:50.518712+0000","last_undegraded":"2024-01-16T15:18:50.518712+0000","last_fullsized":"2024-01-16T15:18:50.518712+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:03.552086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511948+0000","last_change":"2024-01-16T15:18:49.511948+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511948+0000","last_undegraded":"2024-01-16T15:18:49.511948+0000","last_fullsized":"2024-01-16T15:18:49.511948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501471+0000","last_change":"2024-01-16T15:18:49.501471+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501471+0000","last_undegraded":"2024-01-16T15:18:49.501471+0000","last_fullsized":"2024-01-16T15:18:49.501471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498605+0000","last_change":"2024-01-16T15:18:49.498605+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498605+0000","last_undegraded":"2024-01-16T15:18:49.498605+0000","last_fullsized":"2024-01-16T15:18:49.498605+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502074+0000","last_change":"2024-01-16T15:18:49.502074+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502074+0000","last_undegraded":"2024-01-16T15:18:49.502074+0000","last_fullsized":"2024-01-16T15:18:49.502074+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.517526+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:18:50.517526+0000","last_peered":"2024-01-16T15:18:50.517526+0000","last_clean":"2024-01-16T15:18:50.517526+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:18:50.517526+0000","last_undegraded":"2024-01-16T15:18:50.517526+0000","last_fullsized":"2024-01-16T15:18:50.517526+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.126247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505422+0000","last_change":"2024-01-16T15:18:49.505422+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505422+0000","last_undegraded":"2024-01-16T15:18:49.505422+0000","last_fullsized":"2024-01-16T15:18:49.505422+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.513783+0000","last_change":"2024-01-16T15:18:49.513783+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.513783+0000","last_undegraded":"2024-01-16T15:18:49.513783+0000","last_fullsized":"2024-01-16T15:18:49.513783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520299+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:18:49.520299+0000","last_peered":"2024-01-16T15:18:49.520299+0000","last_clean":"2024-01-16T15:18:49.520299+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:18:49.520299+0000","last_undegraded":"2024-01-16T15:18:49.520299+0000","last_fullsized":"2024-01-16T15:18:49.520299+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501201+0000","last_change":"2024-01-16T15:18:49.501201+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501201+0000","last_undegraded":"2024-01-16T15:18:49.501201+0000","last_fullsized":"2024-01-16T15:18:49.501201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524289+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:18:49.524289+0000","last_peered":"2024-01-16T15:18:49.524289+0000","last_clean":"2024-01-16T15:18:49.524289+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:18:49.524289+0000","last_undegraded":"2024-01-16T15:18:49.524289+0000","last_fullsized":"2024-01-16T15:18:49.524289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:21.931597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.512665+0000","last_change":"2024-01-16T15:18:49.512665+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.512665+0000","last_undegraded":"2024-01-16T15:18:49.512665+0000","last_fullsized":"2024-01-16T15:18:49.512665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522310+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:18:49.522310+0000","last_peered":"2024-01-16T15:18:49.522310+0000","last_clean":"2024-01-16T15:18:49.522310+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:18:49.522310+0000","last_undegraded":"2024-01-16T15:18:49.522310+0000","last_fullsized":"2024-01-16T15:18:49.522310+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:02.444449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507992+0000","last_change":"2024-01-16T15:18:49.507992+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507992+0000","last_undegraded":"2024-01-16T15:18:49.507992+0000","last_fullsized":"2024-01-16T15:18:49.507992+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522279+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:18:49.522279+0000","last_peered":"2024-01-16T15:18:49.522279+0000","last_clean":"2024-01-16T15:18:49.522279+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:18:49.522279+0000","last_undegraded":"2024-01-16T15:18:49.522279+0000","last_fullsized":"2024-01-16T15:18:49.522279+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:44.623306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513485+0000","last_change":"2024-01-16T15:18:50.510180+0000","last_active":"2024-01-16T15:18:50.513485+0000","last_peered":"2024-01-16T15:18:50.513485+0000","last_clean":"2024-01-16T15:18:50.513485+0000","last_became_active":"2024-01-16T15:18:50.509888+0000","last_became_peered":"2024-01-16T15:18:50.509888+0000","last_unstale":"2024-01-16T15:18:50.513485+0000","last_undegraded":"2024-01-16T15:18:50.513485+0000","last_fullsized":"2024-01-16T15:18:50.513485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:49.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524377+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:18:49.524377+0000","last_peered":"2024-01-16T15:18:49.524377+0000","last_clean":"2024-01-16T15:18:49.524377+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:18:49.524377+0000","last_undegraded":"2024-01-16T15:18:49.524377+0000","last_fullsized":"2024-01-16T15:18:49.524377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497629+0000","last_change":"2024-01-16T15:18:49.497629+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497629+0000","last_undegraded":"2024-01-16T15:18:49.497629+0000","last_fullsized":"2024-01-16T15:18:49.497629+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.518795+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:18:49.518795+0000","last_peered":"2024-01-16T15:18:49.518795+0000","last_clean":"2024-01-16T15:18:49.518795+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:18:49.518795+0000","last_undegraded":"2024-01-16T15:18:49.518795+0000","last_fullsized":"2024-01-16T15:18:49.518795+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:43.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518433+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:18:50.518433+0000","last_peered":"2024-01-16T15:18:50.518433+0000","last_clean":"2024-01-16T15:18:50.518433+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:18:50.518433+0000","last_undegraded":"2024-01-16T15:18:50.518433+0000","last_fullsized":"2024-01-16T15:18:50.518433+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:48.856213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511369+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:18:49.511369+0000","last_peered":"2024-01-16T15:18:49.511369+0000","last_clean":"2024-01-16T15:18:49.511369+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:18:49.511369+0000","last_undegraded":"2024-01-16T15:18:49.511369+0000","last_fullsized":"2024-01-16T15:18:49.511369+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505254+0000","last_change":"2024-01-16T15:18:49.505254+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505254+0000","last_undegraded":"2024-01-16T15:18:49.505254+0000","last_fullsized":"2024-01-16T15:18:49.505254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503470+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:18:50.503470+0000","last_peered":"2024-01-16T15:18:50.503470+0000","last_clean":"2024-01-16T15:18:50.503470+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:18:50.503470+0000","last_undegraded":"2024-01-16T15:18:50.503470+0000","last_fullsized":"2024-01-16T15:18:50.503470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.508797+0000","last_change":"2024-01-16T15:18:49.508797+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.508797+0000","last_undegraded":"2024-01-16T15:18:49.508797+0000","last_fullsized":"2024-01-16T15:18:49.508797+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.525562+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:18:49.525562+0000","last_peered":"2024-01-16T15:18:49.525562+0000","last_clean":"2024-01-16T15:18:49.525562+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:18:49.525562+0000","last_undegraded":"2024-01-16T15:18:49.525562+0000","last_fullsized":"2024-01-16T15:18:49.525562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:10.112260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500474+0000","last_change":"2024-01-16T15:18:49.500474+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500474+0000","last_undegraded":"2024-01-16T15:18:49.500474+0000","last_fullsized":"2024-01-16T15:18:49.500474+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.515357+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:18:49.515357+0000","last_peered":"2024-01-16T15:18:49.515357+0000","last_clean":"2024-01-16T15:18:49.515357+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:18:49.515357+0000","last_undegraded":"2024-01-16T15:18:49.515357+0000","last_fullsized":"2024-01-16T15:18:49.515357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:56.122268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500974+0000","last_change":"2024-01-16T15:18:49.500974+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500974+0000","last_undegraded":"2024-01-16T15:18:49.500974+0000","last_fullsized":"2024-01-16T15:18:49.500974+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.536241+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:18:50.536241+0000","last_peered":"2024-01-16T15:18:50.536241+0000","last_clean":"2024-01-16T15:18:50.536241+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:18:50.536241+0000","last_undegraded":"2024-01-16T15:18:50.536241+0000","last_fullsized":"2024-01-16T15:18:50.536241+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.503096+0000","last_change":"2024-01-16T15:18:49.503096+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.503096+0000","last_undegraded":"2024-01-16T15:18:49.503096+0000","last_fullsized":"2024-01-16T15:18:49.503096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.538150+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:18:50.538150+0000","last_peered":"2024-01-16T15:18:50.538150+0000","last_clean":"2024-01-16T15:18:50.538150+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:18:50.538150+0000","last_undegraded":"2024-01-16T15:18:50.538150+0000","last_fullsized":"2024-01-16T15:18:50.538150+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498243+0000","last_change":"2024-01-16T15:18:49.498243+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498243+0000","last_undegraded":"2024-01-16T15:18:49.498243+0000","last_fullsized":"2024-01-16T15:18:49.498243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520668+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:18:49.520668+0000","last_peered":"2024-01-16T15:18:49.520668+0000","last_clean":"2024-01-16T15:18:49.520668+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:18:49.520668+0000","last_undegraded":"2024-01-16T15:18:49.520668+0000","last_fullsized":"2024-01-16T15:18:49.520668+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499199+0000","last_change":"2024-01-16T15:18:49.499199+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499199+0000","last_undegraded":"2024-01-16T15:18:49.499199+0000","last_fullsized":"2024-01-16T15:18:49.499199+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516723+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:18:49.516723+0000","last_peered":"2024-01-16T15:18:49.516723+0000","last_clean":"2024-01-16T15:18:49.516723+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:18:49.516723+0000","last_undegraded":"2024-01-16T15:18:49.516723+0000","last_fullsized":"2024-01-16T15:18:49.516723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:21.691380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499391+0000","last_change":"2024-01-16T15:18:49.499391+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499391+0000","last_undegraded":"2024-01-16T15:18:49.499391+0000","last_fullsized":"2024-01-16T15:18:49.499391+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511289+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:18:49.511289+0000","last_peered":"2024-01-16T15:18:49.511289+0000","last_clean":"2024-01-16T15:18:49.511289+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:18:49.511289+0000","last_undegraded":"2024-01-16T15:18:49.511289+0000","last_fullsized":"2024-01-16T15:18:49.511289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:29.197614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495765+0000","last_change":"2024-01-16T15:18:49.495765+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495765+0000","last_undegraded":"2024-01-16T15:18:49.495765+0000","last_fullsized":"2024-01-16T15:18:49.495765+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533470+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:18:50.533470+0000","last_peered":"2024-01-16T15:18:50.533470+0000","last_clean":"2024-01-16T15:18:50.533470+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:18:50.533470+0000","last_undegraded":"2024-01-16T15:18:50.533470+0000","last_fullsized":"2024-01-16T15:18:50.533470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.516561+0000","last_change":"2024-01-16T15:18:49.516561+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.516561+0000","last_undegraded":"2024-01-16T15:18:49.516561+0000","last_fullsized":"2024-01-16T15:18:49.516561+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511184+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:18:49.511184+0000","last_peered":"2024-01-16T15:18:49.511184+0000","last_clean":"2024-01-16T15:18:49.511184+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:18:49.511184+0000","last_undegraded":"2024-01-16T15:18:49.511184+0000","last_fullsized":"2024-01-16T15:18:49.511184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:53.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.510804+0000","last_change":"2024-01-16T15:18:49.510804+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.510804+0000","last_undegraded":"2024-01-16T15:18:49.510804+0000","last_fullsized":"2024-01-16T15:18:49.510804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520479+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:18:49.520479+0000","last_peered":"2024-01-16T15:18:49.520479+0000","last_clean":"2024-01-16T15:18:49.520479+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:18:49.520479+0000","last_undegraded":"2024-01-16T15:18:49.520479+0000","last_fullsized":"2024-01-16T15:18:49.520479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:53.321942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507278+0000","last_change":"2024-01-16T15:18:49.507278+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507278+0000","last_undegraded":"2024-01-16T15:18:49.507278+0000","last_fullsized":"2024-01-16T15:18:49.507278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520339+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:18:49.520339+0000","last_peered":"2024-01-16T15:18:49.520339+0000","last_clean":"2024-01-16T15:18:49.520339+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:18:49.520339+0000","last_undegraded":"2024-01-16T15:18:49.520339+0000","last_fullsized":"2024-01-16T15:18:49.520339+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.504828+0000","last_change":"2024-01-16T15:18:49.504828+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.504828+0000","last_undegraded":"2024-01-16T15:18:49.504828+0000","last_fullsized":"2024-01-16T15:18:49.504828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523509+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:18:49.523509+0000","last_peered":"2024-01-16T15:18:49.523509+0000","last_clean":"2024-01-16T15:18:49.523509+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:18:49.523509+0000","last_undegraded":"2024-01-16T15:18:49.523509+0000","last_fullsized":"2024-01-16T15:18:49.523509+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518601+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:18:50.518601+0000","last_peered":"2024-01-16T15:18:50.518601+0000","last_clean":"2024-01-16T15:18:50.518601+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:18:50.518601+0000","last_undegraded":"2024-01-16T15:18:50.518601+0000","last_fullsized":"2024-01-16T15:18:50.518601+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:27.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522193+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:18:49.522193+0000","last_peered":"2024-01-16T15:18:49.522193+0000","last_clean":"2024-01-16T15:18:49.522193+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:18:49.522193+0000","last_undegraded":"2024-01-16T15:18:49.522193+0000","last_fullsized":"2024-01-16T15:18:49.522193+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:54.524825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.506081+0000","last_change":"2024-01-16T15:18:49.506081+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.506081+0000","last_undegraded":"2024-01-16T15:18:49.506081+0000","last_fullsized":"2024-01-16T15:18:49.506081+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523683+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:18:49.523683+0000","last_peered":"2024-01-16T15:18:49.523683+0000","last_clean":"2024-01-16T15:18:49.523683+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:18:49.523683+0000","last_undegraded":"2024-01-16T15:18:49.523683+0000","last_fullsized":"2024-01-16T15:18:49.523683+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498896+0000","last_change":"2024-01-16T15:18:49.498896+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498896+0000","last_undegraded":"2024-01-16T15:18:49.498896+0000","last_fullsized":"2024-01-16T15:18:49.498896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503708+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:18:50.503708+0000","last_peered":"2024-01-16T15:18:50.503708+0000","last_clean":"2024-01-16T15:18:50.503708+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:18:50.503708+0000","last_undegraded":"2024-01-16T15:18:50.503708+0000","last_fullsized":"2024-01-16T15:18:50.503708+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500913+0000","last_change":"2024-01-16T15:18:49.500913+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500913+0000","last_undegraded":"2024-01-16T15:18:49.500913+0000","last_fullsized":"2024-01-16T15:18:49.500913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522640+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:18:49.522640+0000","last_peered":"2024-01-16T15:18:49.522640+0000","last_clean":"2024-01-16T15:18:49.522640+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:18:49.522640+0000","last_undegraded":"2024-01-16T15:18:49.522640+0000","last_fullsized":"2024-01-16T15:18:49.522640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:48.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.526632+0000","last_change":"2024-01-16T15:18:50.522199+0000","last_active":"2024-01-16T15:18:50.526632+0000","last_peered":"2024-01-16T15:18:50.526632+0000","last_clean":"2024-01-16T15:18:50.526632+0000","last_became_active":"2024-01-16T15:18:50.521902+0000","last_became_peered":"2024-01-16T15:18:50.521902+0000","last_unstale":"2024-01-16T15:18:50.526632+0000","last_undegraded":"2024-01-16T15:18:50.526632+0000","last_fullsized":"2024-01-16T15:18:50.526632+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:11.568326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516769+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:18:49.516769+0000","last_peered":"2024-01-16T15:18:49.516769+0000","last_clean":"2024-01-16T15:18:49.516769+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:18:49.516769+0000","last_undegraded":"2024-01-16T15:18:49.516769+0000","last_fullsized":"2024-01-16T15:18:49.516769+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:17.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518602+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:18:50.518602+0000","last_peered":"2024-01-16T15:18:50.518602+0000","last_clean":"2024-01-16T15:18:50.518602+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:18:50.518602+0000","last_undegraded":"2024-01-16T15:18:50.518602+0000","last_fullsized":"2024-01-16T15:18:50.518602+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:38.334522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.505052+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:18:50.505052+0000","last_peered":"2024-01-16T15:18:50.505052+0000","last_clean":"2024-01-16T15:18:50.505052+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:18:50.505052+0000","last_undegraded":"2024-01-16T15:18:50.505052+0000","last_fullsized":"2024-01-16T15:18:50.505052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500292+0000","last_change":"2024-01-16T15:18:49.500292+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500292+0000","last_undegraded":"2024-01-16T15:18:49.500292+0000","last_fullsized":"2024-01-16T15:18:49.500292+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520661+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:18:49.520661+0000","last_peered":"2024-01-16T15:18:49.520661+0000","last_clean":"2024-01-16T15:18:49.520661+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:18:49.520661+0000","last_undegraded":"2024-01-16T15:18:49.520661+0000","last_fullsized":"2024-01-16T15:18:49.520661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:24.291444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497200+0000","last_change":"2024-01-16T15:18:49.497200+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497200+0000","last_undegraded":"2024-01-16T15:18:49.497200+0000","last_fullsized":"2024-01-16T15:18:49.497200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.510956+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:18:49.510956+0000","last_peered":"2024-01-16T15:18:49.510956+0000","last_clean":"2024-01-16T15:18:49.510956+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:18:49.510956+0000","last_undegraded":"2024-01-16T15:18:49.510956+0000","last_fullsized":"2024-01-16T15:18:49.510956+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496009+0000","last_change":"2024-01-16T15:18:49.496009+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496009+0000","last_undegraded":"2024-01-16T15:18:49.496009+0000","last_fullsized":"2024-01-16T15:18:49.496009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524006+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:18:49.524006+0000","last_peered":"2024-01-16T15:18:49.524006+0000","last_clean":"2024-01-16T15:18:49.524006+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:18:49.524006+0000","last_undegraded":"2024-01-16T15:18:49.524006+0000","last_fullsized":"2024-01-16T15:18:49.524006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505239+0000","last_change":"2024-01-16T15:18:49.505239+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505239+0000","last_undegraded":"2024-01-16T15:18:49.505239+0000","last_fullsized":"2024-01-16T15:18:49.505239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.519663+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:18:49.519663+0000","last_peered":"2024-01-16T15:18:49.519663+0000","last_clean":"2024-01-16T15:18:49.519663+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:18:49.519663+0000","last_undegraded":"2024-01-16T15:18:49.519663+0000","last_fullsized":"2024-01-16T15:18:49.519663+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:27.624738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520862+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:18:50.520862+0000","last_peered":"2024-01-16T15:18:50.520862+0000","last_clean":"2024-01-16T15:18:50.520862+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:18:50.520862+0000","last_undegraded":"2024-01-16T15:18:50.520862+0000","last_fullsized":"2024-01-16T15:18:50.520862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:09.166757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503918+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:18:50.503918+0000","last_peered":"2024-01-16T15:18:50.503918+0000","last_clean":"2024-01-16T15:18:50.503918+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:18:50.503918+0000","last_undegraded":"2024-01-16T15:18:50.503918+0000","last_fullsized":"2024-01-16T15:18:50.503918+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498189+0000","last_change":"2024-01-16T15:18:49.498189+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498189+0000","last_undegraded":"2024-01-16T15:18:49.498189+0000","last_fullsized":"2024-01-16T15:18:49.498189+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524346+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:18:49.524346+0000","last_peered":"2024-01-16T15:18:49.524346+0000","last_clean":"2024-01-16T15:18:49.524346+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:18:49.524346+0000","last_undegraded":"2024-01-16T15:18:49.524346+0000","last_fullsized":"2024-01-16T15:18:49.524346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:18:48.485739+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:18:48.485739+0000","last_peered":"2024-01-16T15:18:48.485739+0000","last_clean":"2024-01-16T15:18:48.485739+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:18:48.485739+0000","last_undegraded":"2024-01-16T15:18:48.485739+0000","last_fullsized":"2024-01-16T15:18:48.485739+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542381,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1970176,"data_stored":1183817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.349,"15min":1.5429999999999999},"min":{"1min":0.72099999999999997,"5min":0.63500000000000001,"15min":0.28199999999999997},"max":{"1min":1.768,"5min":5.2690000000000001,"15min":27.469000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4410000000000001,"15min":1.6859999999999999},"min":{"1min":0.86099999999999999,"5min":0.79500000000000004,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":6.5599999999999996,"15min":25.503},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.139,"15min":1.2689999999999999},"min":{"1min":0.77600000000000002,"5min":0.55100000000000005,"15min":0.34499999999999997},"max":{"1min":1.712,"5min":1.742,"15min":6.718},"last":1.343},{"interface":"front","average":{"1min":1.262,"5min":1.2070000000000001,"15min":1.3520000000000001},"min":{"1min":0.85499999999999998,"5min":0.75700000000000001,"15min":0.373},"max":{"1min":1.7669999999999999,"5min":2.0110000000000001,"15min":15.452},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.4530000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.48299999999999998},"max":{"1min":1.7709999999999999,"5min":11.536,"15min":11.536},"last":1.25},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5409999999999999,"15min":1.5629999999999999},"min":{"1min":1.0720000000000001,"5min":0.79800000000000004,"15min":0.504},"max":{"1min":1.978,"5min":11.576000000000001,"15min":25.478000000000002},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.222,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.67000000000000004,"15min":0.40899999999999997},"max":{"1min":1.593,"5min":2.133,"15min":13.907999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.357,"5min":1.365,"15min":1.5029999999999999},"min":{"1min":1.022,"5min":0.89300000000000002,"15min":0.62},"max":{"1min":1.8580000000000001,"5min":2.1960000000000002,"15min":8.6999999999999993},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.452},"min":{"1min":0.89200000000000002,"5min":0.85299999999999998,"15min":0.376},"max":{"1min":1.7829999999999999,"5min":6.7599999999999998,"15min":25.401},"last":1.171},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3280000000000001,"15min":1.472},"min":{"1min":0.84999999999999998,"5min":0.59599999999999997,"15min":0.45000000000000001},"max":{"1min":2.069,"5min":2.069,"15min":10.134},"last":1.5489999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":0.96399999999999997,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.7310000000000001,"5min":2.1259999999999999,"15min":12.528},"last":1.486},{"interface":"front","average":{"1min":1.498,"5min":1.544,"15min":1.639},"min":{"1min":1.0640000000000001,"5min":0.86599999999999999,"15min":0.41399999999999998},"max":{"1min":1.9159999999999999,"5min":6.8719999999999999,"15min":26.327000000000002},"last":1.302}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.6759999999999999,"15min":1.734},"min":{"1min":1.3640000000000001,"5min":1.087,"15min":0.45200000000000001},"max":{"1min":2.0870000000000002,"5min":6.9690000000000003,"15min":25.451000000000001},"last":1.462},{"interface":"front","average":{"1min":1.524,"5min":1.4930000000000001,"15min":1.605},"min":{"1min":0.92400000000000004,"5min":0.89800000000000002,"15min":0.44700000000000001},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":12.577999999999999},"last":1.6120000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72716,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299124,"statfs":{"total":96636764160,"available":96562302976,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.452,"15min":1.744},"min":{"1min":0.752,"5min":0.71299999999999997,"15min":0.35399999999999998},"max":{"1min":1.53,"5min":29.77,"15min":30.164999999999999},"last":1.53},{"interface":"front","average":{"1min":1.292,"5min":1.6870000000000001,"15min":1.8779999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.48899999999999999},"max":{"1min":1.5780000000000001,"5min":36.405999999999999,"15min":36.405999999999999},"last":1.49}]},{"osd":1,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1850000000000001,"15min":1.508},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.47899999999999998},"max":{"1min":1.52,"5min":5.2859999999999996,"15min":28.122},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.2549999999999999,"15min":1.298},"min":{"1min":0.78200000000000003,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":1.548,"5min":10.494,"15min":10.494},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.347,"15min":1.734},"min":{"1min":1,"5min":0.76900000000000002,"15min":0.501},"max":{"1min":1.845,"5min":5.0549999999999997,"15min":26.109000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.51,"15min":1.893},"min":{"1min":1.0329999999999999,"5min":1.016,"15min":0.67100000000000004},"max":{"1min":2.0449999999999999,"5min":5.1459999999999999,"15min":26.161000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3029999999999999,"15min":1.615},"min":{"1min":1.1060000000000001,"5min":0.68500000000000005,"15min":0.44700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8079999999999998,"15min":17.75},"last":1.363},{"interface":"front","average":{"1min":1.395,"5min":1.681,"15min":1.728},"min":{"1min":1.0349999999999999,"5min":0.77100000000000002,"15min":0.55700000000000005},"max":{"1min":2.1789999999999998,"5min":21.456,"15min":21.456},"last":1.5600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3540000000000001,"15min":1.5069999999999999},"min":{"1min":0.91400000000000003,"5min":0.82099999999999995,"15min":0.45000000000000001},"max":{"1min":1.6699999999999999,"5min":5.3310000000000004,"15min":15.571},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.325,"15min":1.5469999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.42299999999999999},"max":{"1min":1.639,"5min":3.6160000000000001,"15min":23.667999999999999},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4350000000000001,"15min":1.4890000000000001},"min":{"1min":0.83499999999999996,"5min":0.73699999999999999,"15min":0.42299999999999999},"max":{"1min":1.982,"5min":13.949999999999999,"15min":23.137},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.6100000000000001,"15min":1.6970000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":0.47099999999999997},"max":{"1min":1.714,"5min":11.298,"15min":25.684000000000001},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.6020000000000001,"15min":1.552},"min":{"1min":0.754,"5min":0.71499999999999997,"15min":0.47099999999999997},"max":{"1min":2.1000000000000001,"5min":14.037000000000001,"15min":14.037000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6240000000000001,"15min":1.724},"min":{"1min":0.95899999999999996,"5min":0.92300000000000004,"15min":0.499},"max":{"1min":2.1659999999999999,"5min":4.9829999999999997,"15min":11.788},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.075,"15min":1.571},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.39000000000000001},"max":{"1min":1.5649999999999999,"5min":1.651,"15min":26.228999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.135,"15min":1.831},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.34799999999999998},"max":{"1min":1.7070000000000001,"5min":1.849,"15min":56.011000000000003},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1419999999999999,"15min":1.363},"min":{"1min":0.79900000000000004,"5min":0.65200000000000002,"15min":0.40999999999999998},"max":{"1min":1.6419999999999999,"5min":2.2450000000000001,"15min":16.687999999999999},"last":1.034},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2430000000000001,"15min":1.8600000000000001},"min":{"1min":0.88900000000000001,"5min":0.745,"15min":0.40999999999999998},"max":{"1min":1.667,"5min":4.8499999999999996,"15min":67.930000000000007},"last":1.125}]},{"osd":2,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1779999999999999,"15min":1.52},"min":{"1min":0.85199999999999998,"5min":0.61099999999999999,"15min":0.47099999999999997},"max":{"1min":1.55,"5min":1.7529999999999999,"15min":13.193},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.246,"5min":1.258,"15min":1.4850000000000001},"min":{"1min":0.93700000000000006,"5min":0.76300000000000001,"15min":0.45800000000000002},"max":{"1min":1.7769999999999999,"5min":3.9590000000000001,"15min":16.721},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.379,"15min":1.712},"min":{"1min":1.0509999999999999,"5min":0.81100000000000005,"15min":0.57999999999999996},"max":{"1min":1.8680000000000001,"5min":4.8010000000000002,"15min":22.382000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.3819999999999999,"15min":1.7529999999999999},"min":{"1min":0.94199999999999995,"5min":0.72399999999999998,"15min":0.51200000000000001},"max":{"1min":1.921,"5min":1.921,"15min":14.685},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2649999999999999,"15min":1.419},"min":{"1min":0.95199999999999996,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.8680000000000001,"5min":2.2549999999999999,"15min":16.382999999999999},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.377,"15min":1.5269999999999999},"min":{"1min":0.98899999999999999,"5min":0.502,"15min":0.379},"max":{"1min":1.8129999999999999,"5min":4.6580000000000004,"15min":27.006},"last":1.6539999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3600000000000001,"15min":1.5600000000000001},"min":{"1min":0.83999999999999997,"5min":0.52400000000000002,"15min":0.39700000000000002},"max":{"1min":1.974,"5min":1.9770000000000001,"15min":13.263999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.4890000000000001,"15min":1.609},"min":{"1min":1.304,"5min":0.629,"15min":0.505},"max":{"1min":1.9670000000000001,"5min":2.1640000000000001,"15min":9.5269999999999992},"last":1.704}]},{"osd":7,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.52,"15min":1.7470000000000001},"min":{"1min":1.2410000000000001,"5min":0.59099999999999997,"15min":0.45000000000000001},"max":{"1min":1.9119999999999999,"5min":4.742,"15min":35.127000000000002},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5489999999999999,"15min":1.7290000000000001},"min":{"1min":0.91200000000000003,"5min":0.61099999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":13.491},"last":1.601}]}]},{"osd":4,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1619999999999999,"15min":1.244},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6180000000000001,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.21},{"interface":"front","average":{"1min":1.095,"5min":1.123,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.679,"5min":1.958,"15min":60.106000000000002},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.629,"15min":1.79},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51100000000000001},"max":{"1min":1.738,"5min":24.399999999999999,"15min":24.399999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.611,"15min":1.627},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57099999999999995},"max":{"1min":1.7290000000000001,"5min":24.456,"15min":24.456},"last":1.7090000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.3029999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.52800000000000002},"max":{"1min":1.6299999999999999,"5min":2.1699999999999999,"15min":9.7140000000000004},"last":1.145},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2889999999999999,"15min":1.6970000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.52400000000000002},"max":{"1min":1.6479999999999999,"5min":1.855,"15min":35.024999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3300000000000001,"15min":1.556},"min":{"1min":0.503,"5min":0.503,"15min":0.47799999999999998},"max":{"1min":1.6499999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":6.4589999999999996,"15min":7.8140000000000001},"last":1.375}]},{"osd":5,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.454},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":2.1299999999999999,"15min":14.568},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4690000000000001,"15min":1.4319999999999999},"min":{"1min":0.74299999999999999,"5min":0.496,"15min":0.34399999999999997},"max":{"1min":1.8160000000000001,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":2.0720000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.436},"max":{"1min":1.696,"5min":3.0190000000000001,"15min":9.6630000000000003},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4239999999999999,"15min":1.6299999999999999},"min":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.45100000000000001},"max":{"1min":1.675,"5min":2.3980000000000001,"15min":14.317},"last":2.0129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5640000000000001,"15min":1.5449999999999999},"min":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.52100000000000002},"max":{"1min":1.927,"5min":6.5250000000000004,"15min":9.2799999999999994},"last":1.903},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.552,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.443},"max":{"1min":1.8540000000000001,"5min":3.3439999999999999,"15min":9.2270000000000003},"last":2.2229999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82432,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289408,"statfs":{"total":96636764160,"available":96552353792,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.93899999999999995,"15min":1.429},"min":{"1min":0.748,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.45,"5min":1.9299999999999999,"15min":50.988999999999997},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.087,"15min":1.4750000000000001},"min":{"1min":0.80100000000000005,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5840000000000001,"5min":1.786,"15min":20.274000000000001},"last":1.141}]},{"osd":1,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.101,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.371,"15min":0.32100000000000001},"max":{"1min":1.613,"5min":3.6640000000000001,"15min":18.192},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.21,"15min":1.712},"min":{"1min":0.89000000000000001,"5min":0.39400000000000002,"15min":0.313},"max":{"1min":1.8440000000000001,"5min":1.9239999999999999,"15min":20.385999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3089999999999999,"15min":1.7250000000000001},"min":{"1min":0.871,"5min":0.45500000000000002,"15min":0.36699999999999999},"max":{"1min":1.978,"5min":3.5249999999999999,"15min":42.840000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.397,"15min":1.806},"min":{"1min":1.016,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.631,"5min":3.702,"15min":38.707000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.339,"15min":1.762},"min":{"1min":0.99199999999999999,"5min":0.78200000000000003,"15min":0.44500000000000001},"max":{"1min":1.734,"5min":2.327,"15min":18.952000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4650000000000001,"15min":1.79},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.54800000000000004},"max":{"1min":1.718,"5min":3.9529999999999998,"15min":18.088999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4890000000000001,"15min":1.8160000000000001},"min":{"1min":1.212,"5min":0.73199999999999998,"15min":0.55500000000000005},"max":{"1min":1.7729999999999999,"5min":2.661,"15min":24.349},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.464,"15min":1.9199999999999999},"min":{"1min":1.105,"5min":0.753,"15min":0.61899999999999999},"max":{"1min":1.9119999999999999,"5min":3.4239999999999999,"15min":14.192},"last":1.306}]},{"osd":6,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.621,"15min":1.877},"min":{"1min":1.159,"5min":0.77200000000000002,"15min":0.52200000000000002},"max":{"1min":1.7929999999999999,"5min":3.9009999999999998,"15min":22.687000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.623,"15min":1.931},"min":{"1min":1.262,"5min":0.73799999999999999,"15min":0.59699999999999998},"max":{"1min":1.7969999999999999,"5min":3.9300000000000002,"15min":24.058},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6040000000000001,"15min":2.1000000000000001},"min":{"1min":1.111,"5min":0.85099999999999998,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":6.3209999999999997,"15min":35.570999999999998},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.6659999999999999,"15min":2.0619999999999998},"min":{"1min":1.2969999999999999,"5min":0.81100000000000005,"15min":0.58899999999999997},"max":{"1min":2.0750000000000002,"5min":3.8279999999999998,"15min":37.607999999999997},"last":1.633}]}]},{"osd":2,"up_from":14,"seq":60129542380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94924,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94276916,"statfs":{"total":96636764160,"available":96539561984,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90400000000000003,"15min":1.4430000000000001},"min":{"1min":0.67700000000000005,"5min":0.36099999999999999,"15min":0.24399999999999999},"max":{"1min":1.135,"5min":1.96,"15min":64.387},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.022,"15min":1.387},"min":{"1min":0.76500000000000001,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7030000000000001,"5min":1.9890000000000001,"15min":24.308},"last":1.7490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0580000000000001,"15min":1.466},"min":{"1min":0.752,"5min":0.46300000000000002,"15min":0.34699999999999998},"max":{"1min":1.2490000000000001,"5min":2.093,"15min":27.263000000000002},"last":1.27},{"interface":"front","average":{"1min":1.381,"5min":1.3879999999999999,"15min":1.8520000000000001},"min":{"1min":1.0940000000000001,"5min":0.51300000000000001,"15min":0.38500000000000001},"max":{"1min":1.8620000000000001,"5min":2.1539999999999999,"15min":23.602},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.385,"15min":1.627},"min":{"1min":1.1499999999999999,"5min":0.58499999999999996,"15min":0.40100000000000002},"max":{"1min":1.9299999999999999,"5min":2.085,"15min":27.311},"last":1.081},{"interface":"front","average":{"1min":1.296,"5min":1.327,"15min":1.6299999999999999},"min":{"1min":0.96599999999999997,"5min":0.498,"15min":0.47399999999999998},"max":{"1min":1.698,"5min":1.998,"15min":26.555},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.389,"15min":1.637},"min":{"1min":1.0960000000000001,"5min":0.66100000000000003,"15min":0.38900000000000001},"max":{"1min":1.764,"5min":2.1040000000000001,"15min":26.503},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.3819999999999999,"15min":1.7490000000000001},"min":{"1min":1.0009999999999999,"5min":0.56399999999999995,"15min":0.51200000000000001},"max":{"1min":1.9079999999999999,"5min":2.4319999999999999,"15min":25.661000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.4610000000000001,"15min":1.74},"min":{"1min":1.3120000000000001,"5min":0.74199999999999999,"15min":0.45800000000000002},"max":{"1min":2.077,"5min":2.3490000000000002,"15min":27.149000000000001},"last":1.49},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.538,"15min":1.7330000000000001},"min":{"1min":1.1930000000000001,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":1.8540000000000001,"5min":2.6589999999999998,"15min":26.349},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4910000000000001,"15min":1.9610000000000001},"min":{"1min":1.0700000000000001,"5min":0.64500000000000002,"15min":0.504},"max":{"1min":2.1909999999999998,"5min":2.7149999999999999,"15min":25.603999999999999},"last":1.548},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.597,"15min":1.843},"min":{"1min":1.1579999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":2.157,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.591,"15min":1.9470000000000001},"min":{"1min":1.343,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1200000000000001,"5min":2.4889999999999999,"15min":24.509},"last":1.649},{"interface":"front","average":{"1min":1.657,"5min":1.5880000000000001,"15min":1.927},"min":{"1min":1.282,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.2229999999999999,"5min":2.3479999999999999,"15min":27.207999999999998},"last":1.8080000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.92400000000000004,"15min":1.153},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.28399999999999997},"max":{"1min":1.696,"5min":1.865,"15min":11.699},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0489999999999999,"15min":1.276},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.33700000000000002},"max":{"1min":1.5089999999999999,"5min":2.3860000000000001,"15min":13.930999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.238,"15min":1.385},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.31},"max":{"1min":1.8,"5min":15.557,"15min":16.488},"last":1.274},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1559999999999999,"15min":1.5680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39000000000000001},"max":{"1min":1.968,"5min":2.306,"15min":22.550999999999998},"last":1.4650000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.6879999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.439},"max":{"1min":1.7450000000000001,"5min":1.9790000000000001,"15min":22.471},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3660000000000001,"15min":1.6359999999999999},"min":{"1min":0.54100000000000004,"5min":0.49099999999999999,"15min":0.35699999999999998},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":22.550999999999998},"last":0.82899999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4870000000000001,"15min":1.6240000000000001},"min":{"1min":0.96699999999999997,"5min":0.95699999999999996,"15min":0.60899999999999999},"max":{"1min":1.95,"5min":2.7050000000000001,"15min":8.3620000000000001},"last":1.333},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5960000000000001,"15min":1.7070000000000001},"min":{"1min":0.97399999999999998,"5min":0.69999999999999996,"15min":0.433},"max":{"1min":1.97,"5min":15.723000000000001,"15min":19.236999999999998},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.706},"min":{"1min":0.92800000000000005,"5min":0.73799999999999999,"15min":0.57199999999999995},"max":{"1min":1.843,"5min":2.2040000000000002,"15min":21.117000000000001},"last":1.175},{"interface":"front","average":{"1min":1.425,"5min":1.544,"15min":1.758},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.65800000000000003},"max":{"1min":1.8009999999999999,"5min":15.785,"15min":17.138000000000002},"last":1.7569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.4850000000000001,"15min":1.698},"min":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.72199999999999998},"max":{"1min":2.2759999999999998,"5min":2.298,"15min":11.901999999999999},"last":1.798},{"interface":"front","average":{"1min":1.629,"5min":1.5840000000000001,"15min":1.8069999999999999},"min":{"1min":1.044,"5min":0.86399999999999999,"15min":0.63300000000000001},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":23.420999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.631,"15min":1.8380000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.085,"5min":15.875,"15min":17.010000000000002},"last":1.423},{"interface":"front","average":{"1min":1.591,"5min":1.573,"15min":1.825},"min":{"1min":0.998,"5min":0.91600000000000004,"15min":0.75700000000000001},"max":{"1min":2.508,"5min":2.508,"15min":11.962},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542381,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73308,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298532,"statfs":{"total":96636764160,"available":96561696768,"internally_reserved":0,"allocated":1982464,"data_stored":1190620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":73071041},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.016,"15min":1.468},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":1.355,"5min":4.8380000000000001,"15min":43.459000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.2290000000000001,"15min":1.706},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.956,"5min":4.9779999999999998,"15min":43.627000000000002},"last":1.042}]},{"osd":2,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1919999999999999,"15min":1.4099999999999999},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":5.0570000000000004,"15min":18.861000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.218,"15min":1.5229999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":2.2330000000000001,"5min":5.3550000000000004,"15min":38.893000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0800000000000001,"15min":1.47},"min":{"1min":0.37,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":1.4410000000000001,"5min":4.9050000000000002,"15min":23.687999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.357,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.434},"max":{"1min":1.6080000000000001,"5min":5.1980000000000004,"15min":18.966999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3460000000000001,"15min":1.5129999999999999},"min":{"1min":0.76500000000000001,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.8420000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.496,"15min":1.7030000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.873,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.423}]},{"osd":5,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.508,"15min":1.891},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.792,"5min":5.1589999999999998,"15min":24.611000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.6970000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.56000000000000005},"max":{"1min":1.907,"5min":7.2699999999999996,"15min":20.199000000000002},"last":1.75}]},{"osd":6,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6499999999999999,"15min":2.0510000000000002},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.55900000000000005},"max":{"1min":2.3119999999999998,"5min":6.3339999999999996,"15min":26.998000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.6140000000000001,"15min":1.988},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":2.1669999999999998,"5min":5.9690000000000003,"15min":24.478999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6699999999999999,"15min":1.9139999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.60899999999999999},"max":{"1min":2.2599999999999998,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.657},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.5629999999999999,"15min":1.7310000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.52800000000000002},"max":{"1min":1.8999999999999999,"5min":6.149,"15min":20.279},"last":1.8280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:18:54.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:18:55.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.228+0000 7f3b18a00640 1 Processor -- start 2024-01-16T15:18:55.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.228+0000 7f3b18a00640 1 -- start start 2024-01-16T15:18:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.229+0000 7f3b18a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b1015d460 0x7f3b100b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.229+0000 7f3b18a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3b100d1300 0x7f3b100b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.229+0000 7f3b18a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100d1760 con 0x7f3b100d0230 2024-01-16T15:18:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.229+0000 7f3b18a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1015d840 con 0x7f3b100d1300 2024-01-16T15:18:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.230+0000 7f3b18a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100b69a0 con 0x7f3b1015d460 2024-01-16T15:18:55.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.230+0000 7f3b15600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3b100d1300 0x7f3b100b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.230+0000 7f3b16000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b1015d460 0x7f3b100b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.230+0000 7f3b15600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3b100d1300 0x7f3b100b8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:45458/0 (socket says 172.21.15.43:45458) 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.230+0000 7f3b15600640 1 -- 172.21.15.43:0/2997883317 learned_addr learned my addr 172.21.15.43:0/2997883317 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.231+0000 7f3b15600640 1 -- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b1015d460 msgr2=0x7f3b100b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.231+0000 7f3b15600640 1 --2- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b1015d460 0x7f3b100b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.231+0000 7f3b15600640 1 -- 172.21.15.43:0/2997883317 >> v1:172.21.15.69:6789/0 conn(0x7f3b100d0230 legacy=0x7f3b100b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.231+0000 7f3b15600640 1 -- 172.21.15.43:0/2997883317 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100b6c80 con 0x7f3b100d1300 2024-01-16T15:18:55.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.232+0000 7f3b15600640 1 --2- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d1300 0x7f3b100b8c70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3b1015d980 tx=0x7f3af802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbdad8e68a441822 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.233+0000 7f3b14c00640 1 -- 172.21.15.43:0/2997883317 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8030840 con 0x7f3b100d1300 2024-01-16T15:18:55.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.233+0000 7f3b14c00640 1 -- 172.21.15.43:0/2997883317 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af80309a0 con 0x7f3b100d1300 2024-01-16T15:18:55.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.233+0000 7f3b14c00640 1 -- 172.21.15.43:0/2997883317 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8030d30 con 0x7f3b100d1300 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.233+0000 7f3b18a00640 1 -- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d1300 msgr2=0x7f3b100b8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.233+0000 7f3b18a00640 1 --2- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d1300 0x7f3b100b8c70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3b1015d980 tx=0x7f3af802b980 comp rx=0 tx=0).stop 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 -- 172.21.15.43:0/2997883317 shutdown_connections 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 --2- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d1300 0x7f3b100b8c70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 --2- 172.21.15.43:0/2997883317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b1015d460 0x7f3b100b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 -- 172.21.15.43:0/2997883317 >> 172.21.15.43:0/2997883317 conn(0x7f3b10071f90 msgr2=0x7f3b100723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 -- 172.21.15.43:0/2997883317 shutdown_connections 2024-01-16T15:18:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.234+0000 7f3b18a00640 1 -- 172.21.15.43:0/2997883317 wait complete. 2024-01-16T15:18:55.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.235+0000 7f3b18a00640 1 Processor -- start 2024-01-16T15:18:55.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.236+0000 7f3b18a00640 1 -- start start 2024-01-16T15:18:55.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.236+0000 7f3b18a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d0230 0x7f3b101518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.236+0000 7f3b18a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b100d1300 0x7f3b100ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b18a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b18a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101f7280 con 0x7f3b1015d460 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b18a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100cc2d0 con 0x7f3b100d0230 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b18a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100cc5b0 con 0x7f3b100d1300 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38326/0 (socket says 172.21.15.43:38326) 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 -- 172.21.15.43:0/763779792 learned_addr learned my addr 172.21.15.43:0/763779792 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:55.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16000640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d0230 0x7f3b101518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b15600640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b100d1300 0x7f3b100ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 -- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b100d1300 msgr2=0x7f3b100ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b100d1300 0x7f3b100ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 -- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d0230 msgr2=0x7f3b101518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d0230 0x7f3b101518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.237+0000 7f3b16a00640 1 -- 172.21.15.43:0/763779792 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af80094d0 con 0x7f3b1015d460 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b16a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f3b00001eb0 tx=0x7f3b0000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000052d0 con 0x7f3b1015d460 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00005430 con 0x7f3b1015d460 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003810 con 0x7f3b1015d460 2024-01-16T15:18:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b100cc890 con 0x7f3b1015d460 2024-01-16T15:18:55.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.238+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b101f7560 con 0x7f3b1015d460 2024-01-16T15:18:55.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.239+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b100cca80 con 0x7f3b1015d460 2024-01-16T15:18:55.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.240+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f3b00003970 con 0x7f3b1015d460 2024-01-16T15:18:55.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.241+0000 7f3b0e000640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 0x7f3ae806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.241+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3b000880f0 con 0x7f3b1015d460 2024-01-16T15:18:55.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.242+0000 7f3b16000640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 0x7f3ae806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.242+0000 7f3b16000640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 0x7f3ae806e860 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3b080073f0 tx=0x7f3b0800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.246+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b000545f0 con 0x7f3b1015d460 2024-01-16T15:18:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.438+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b100cc440 con 0x7f3ae806c3a0 2024-01-16T15:18:55.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.447+0000 7f3b0e000640 1 -- 172.21.15.43:0/763779792 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282614 (secure 0 0 0) 0x7f3b100cc440 con 0x7f3ae806c3a0 2024-01-16T15:18:55.442 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:18:55.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 msgr2=0x7f3ae806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 0x7f3ae806e860 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3b080073f0 tx=0x7f3b0800b040 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 msgr2=0x7f3b101f6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f3b00001eb0 tx=0x7f3b0000c890 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 shutdown_connections 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3ae806c3a0 0x7f3ae806e860 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b1015d460 0x7f3b101f6710 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b100d1300 0x7f3b100ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 --2- 172.21.15.43:0/763779792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b100d0230 0x7f3b101518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 >> 172.21.15.43:0/763779792 conn(0x7f3b10071f90 msgr2=0x7f3b100b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 shutdown_connections 2024-01-16T15:18:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.453+0000 7f3b18a00640 1 -- 172.21.15.43:0/763779792 wait complete. 2024-01-16T15:18:55.463 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-01-16T15:18:55.290428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592092,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":170,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606496,"kb_used_data":11900,"kb_used_omap":12,"kb_used_meta":594483,"kb_avail":754368224,"statfs":{"total":773094113280,"available":772473061376,"internally_reserved":0,"allocated":12185600,"data_stored":5922525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":608751148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001986"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520878+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:18:50.520878+0000","last_peered":"2024-01-16T15:18:50.520878+0000","last_clean":"2024-01-16T15:18:50.520878+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:18:50.520878+0000","last_undegraded":"2024-01-16T15:18:50.520878+0000","last_fullsized":"2024-01-16T15:18:50.520878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:02.958841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496455+0000","last_change":"2024-01-16T15:18:49.496455+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496455+0000","last_undegraded":"2024-01-16T15:18:49.496455+0000","last_fullsized":"2024-01-16T15:18:49.496455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511018+0000","last_change":"2024-01-16T15:18:49.511018+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511018+0000","last_undegraded":"2024-01-16T15:18:49.511018+0000","last_fullsized":"2024-01-16T15:18:49.511018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527282+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:18:50.527282+0000","last_peered":"2024-01-16T15:18:50.527282+0000","last_clean":"2024-01-16T15:18:50.527282+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:18:50.527282+0000","last_undegraded":"2024-01-16T15:18:50.527282+0000","last_fullsized":"2024-01-16T15:18:50.527282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:40.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494552+0000","last_change":"2024-01-16T15:18:49.494552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494552+0000","last_undegraded":"2024-01-16T15:18:49.494552+0000","last_fullsized":"2024-01-16T15:18:49.494552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511599+0000","last_change":"2024-01-16T15:18:49.511599+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511599+0000","last_undegraded":"2024-01-16T15:18:49.511599+0000","last_fullsized":"2024-01-16T15:18:49.511599+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502722+0000","last_change":"2024-01-16T15:18:49.502722+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502722+0000","last_undegraded":"2024-01-16T15:18:49.502722+0000","last_fullsized":"2024-01-16T15:18:49.502722+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507251+0000","last_change":"2024-01-16T15:18:49.507251+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507251+0000","last_undegraded":"2024-01-16T15:18:49.507251+0000","last_fullsized":"2024-01-16T15:18:49.507251+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527409+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:18:50.527409+0000","last_peered":"2024-01-16T15:18:50.527409+0000","last_clean":"2024-01-16T15:18:50.527409+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:18:50.527409+0000","last_undegraded":"2024-01-16T15:18:50.527409+0000","last_fullsized":"2024-01-16T15:18:50.527409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:44.209808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513531+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:18:50.513531+0000","last_peered":"2024-01-16T15:18:50.513531+0000","last_clean":"2024-01-16T15:18:50.513531+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:18:50.513531+0000","last_undegraded":"2024-01-16T15:18:50.513531+0000","last_fullsized":"2024-01-16T15:18:50.513531+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:37.753051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496236+0000","last_change":"2024-01-16T15:18:49.496236+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496236+0000","last_undegraded":"2024-01-16T15:18:49.496236+0000","last_fullsized":"2024-01-16T15:18:49.496236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513845+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:18:50.513845+0000","last_peered":"2024-01-16T15:18:50.513845+0000","last_clean":"2024-01-16T15:18:50.513845+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:18:50.513845+0000","last_undegraded":"2024-01-16T15:18:50.513845+0000","last_fullsized":"2024-01-16T15:18:50.513845+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509738+0000","last_change":"2024-01-16T15:18:49.509738+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509738+0000","last_undegraded":"2024-01-16T15:18:49.509738+0000","last_fullsized":"2024-01-16T15:18:49.509738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505304+0000","last_change":"2024-01-16T15:18:49.505304+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505304+0000","last_undegraded":"2024-01-16T15:18:49.505304+0000","last_fullsized":"2024-01-16T15:18:49.505304+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509868+0000","last_change":"2024-01-16T15:18:49.509868+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509868+0000","last_undegraded":"2024-01-16T15:18:49.509868+0000","last_fullsized":"2024-01-16T15:18:49.509868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518744+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:18:50.518744+0000","last_peered":"2024-01-16T15:18:50.518744+0000","last_clean":"2024-01-16T15:18:50.518744+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:18:50.518744+0000","last_undegraded":"2024-01-16T15:18:50.518744+0000","last_fullsized":"2024-01-16T15:18:50.518744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:42.136923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495707+0000","last_change":"2024-01-16T15:18:49.495707+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495707+0000","last_undegraded":"2024-01-16T15:18:49.495707+0000","last_fullsized":"2024-01-16T15:18:49.495707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501685+0000","last_change":"2024-01-16T15:18:49.501685+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501685+0000","last_undegraded":"2024-01-16T15:18:49.501685+0000","last_fullsized":"2024-01-16T15:18:49.501685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509552+0000","last_change":"2024-01-16T15:18:49.509552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509552+0000","last_undegraded":"2024-01-16T15:18:49.509552+0000","last_fullsized":"2024-01-16T15:18:49.509552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494776+0000","last_change":"2024-01-16T15:18:49.494776+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494776+0000","last_undegraded":"2024-01-16T15:18:49.494776+0000","last_fullsized":"2024-01-16T15:18:49.494776+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494903+0000","last_change":"2024-01-16T15:18:49.494903+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494903+0000","last_undegraded":"2024-01-16T15:18:49.494903+0000","last_fullsized":"2024-01-16T15:18:49.494903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498780+0000","last_change":"2024-01-16T15:18:49.498780+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498780+0000","last_undegraded":"2024-01-16T15:18:49.498780+0000","last_fullsized":"2024-01-16T15:18:49.498780+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494905+0000","last_change":"2024-01-16T15:18:49.494905+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494905+0000","last_undegraded":"2024-01-16T15:18:49.494905+0000","last_fullsized":"2024-01-16T15:18:49.494905+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498751+0000","last_change":"2024-01-16T15:18:49.498751+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498751+0000","last_undegraded":"2024-01-16T15:18:49.498751+0000","last_fullsized":"2024-01-16T15:18:49.498751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495191+0000","last_change":"2024-01-16T15:18:49.495191+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495191+0000","last_undegraded":"2024-01-16T15:18:49.495191+0000","last_fullsized":"2024-01-16T15:18:49.495191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518712+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:18:50.518712+0000","last_peered":"2024-01-16T15:18:50.518712+0000","last_clean":"2024-01-16T15:18:50.518712+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:18:50.518712+0000","last_undegraded":"2024-01-16T15:18:50.518712+0000","last_fullsized":"2024-01-16T15:18:50.518712+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:03.552086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511948+0000","last_change":"2024-01-16T15:18:49.511948+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511948+0000","last_undegraded":"2024-01-16T15:18:49.511948+0000","last_fullsized":"2024-01-16T15:18:49.511948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501471+0000","last_change":"2024-01-16T15:18:49.501471+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501471+0000","last_undegraded":"2024-01-16T15:18:49.501471+0000","last_fullsized":"2024-01-16T15:18:49.501471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498605+0000","last_change":"2024-01-16T15:18:49.498605+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498605+0000","last_undegraded":"2024-01-16T15:18:49.498605+0000","last_fullsized":"2024-01-16T15:18:49.498605+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502074+0000","last_change":"2024-01-16T15:18:49.502074+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502074+0000","last_undegraded":"2024-01-16T15:18:49.502074+0000","last_fullsized":"2024-01-16T15:18:49.502074+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.517526+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:18:50.517526+0000","last_peered":"2024-01-16T15:18:50.517526+0000","last_clean":"2024-01-16T15:18:50.517526+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:18:50.517526+0000","last_undegraded":"2024-01-16T15:18:50.517526+0000","last_fullsized":"2024-01-16T15:18:50.517526+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.126247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505422+0000","last_change":"2024-01-16T15:18:49.505422+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505422+0000","last_undegraded":"2024-01-16T15:18:49.505422+0000","last_fullsized":"2024-01-16T15:18:49.505422+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.513783+0000","last_change":"2024-01-16T15:18:49.513783+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.513783+0000","last_undegraded":"2024-01-16T15:18:49.513783+0000","last_fullsized":"2024-01-16T15:18:49.513783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520299+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:18:49.520299+0000","last_peered":"2024-01-16T15:18:49.520299+0000","last_clean":"2024-01-16T15:18:49.520299+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:18:49.520299+0000","last_undegraded":"2024-01-16T15:18:49.520299+0000","last_fullsized":"2024-01-16T15:18:49.520299+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501201+0000","last_change":"2024-01-16T15:18:49.501201+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501201+0000","last_undegraded":"2024-01-16T15:18:49.501201+0000","last_fullsized":"2024-01-16T15:18:49.501201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524289+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:18:49.524289+0000","last_peered":"2024-01-16T15:18:49.524289+0000","last_clean":"2024-01-16T15:18:49.524289+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:18:49.524289+0000","last_undegraded":"2024-01-16T15:18:49.524289+0000","last_fullsized":"2024-01-16T15:18:49.524289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:21.931597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.512665+0000","last_change":"2024-01-16T15:18:49.512665+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.512665+0000","last_undegraded":"2024-01-16T15:18:49.512665+0000","last_fullsized":"2024-01-16T15:18:49.512665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522310+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:18:49.522310+0000","last_peered":"2024-01-16T15:18:49.522310+0000","last_clean":"2024-01-16T15:18:49.522310+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:18:49.522310+0000","last_undegraded":"2024-01-16T15:18:49.522310+0000","last_fullsized":"2024-01-16T15:18:49.522310+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:02.444449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507992+0000","last_change":"2024-01-16T15:18:49.507992+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507992+0000","last_undegraded":"2024-01-16T15:18:49.507992+0000","last_fullsized":"2024-01-16T15:18:49.507992+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522279+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:18:49.522279+0000","last_peered":"2024-01-16T15:18:49.522279+0000","last_clean":"2024-01-16T15:18:49.522279+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:18:49.522279+0000","last_undegraded":"2024-01-16T15:18:49.522279+0000","last_fullsized":"2024-01-16T15:18:49.522279+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:44.623306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513485+0000","last_change":"2024-01-16T15:18:50.510180+0000","last_active":"2024-01-16T15:18:50.513485+0000","last_peered":"2024-01-16T15:18:50.513485+0000","last_clean":"2024-01-16T15:18:50.513485+0000","last_became_active":"2024-01-16T15:18:50.509888+0000","last_became_peered":"2024-01-16T15:18:50.509888+0000","last_unstale":"2024-01-16T15:18:50.513485+0000","last_undegraded":"2024-01-16T15:18:50.513485+0000","last_fullsized":"2024-01-16T15:18:50.513485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:49.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524377+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:18:49.524377+0000","last_peered":"2024-01-16T15:18:49.524377+0000","last_clean":"2024-01-16T15:18:49.524377+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:18:49.524377+0000","last_undegraded":"2024-01-16T15:18:49.524377+0000","last_fullsized":"2024-01-16T15:18:49.524377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497629+0000","last_change":"2024-01-16T15:18:49.497629+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497629+0000","last_undegraded":"2024-01-16T15:18:49.497629+0000","last_fullsized":"2024-01-16T15:18:49.497629+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.518795+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:18:49.518795+0000","last_peered":"2024-01-16T15:18:49.518795+0000","last_clean":"2024-01-16T15:18:49.518795+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:18:49.518795+0000","last_undegraded":"2024-01-16T15:18:49.518795+0000","last_fullsized":"2024-01-16T15:18:49.518795+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:43.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518433+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:18:50.518433+0000","last_peered":"2024-01-16T15:18:50.518433+0000","last_clean":"2024-01-16T15:18:50.518433+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:18:50.518433+0000","last_undegraded":"2024-01-16T15:18:50.518433+0000","last_fullsized":"2024-01-16T15:18:50.518433+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:48.856213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511369+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:18:49.511369+0000","last_peered":"2024-01-16T15:18:49.511369+0000","last_clean":"2024-01-16T15:18:49.511369+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:18:49.511369+0000","last_undegraded":"2024-01-16T15:18:49.511369+0000","last_fullsized":"2024-01-16T15:18:49.511369+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505254+0000","last_change":"2024-01-16T15:18:49.505254+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505254+0000","last_undegraded":"2024-01-16T15:18:49.505254+0000","last_fullsized":"2024-01-16T15:18:49.505254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503470+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:18:50.503470+0000","last_peered":"2024-01-16T15:18:50.503470+0000","last_clean":"2024-01-16T15:18:50.503470+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:18:50.503470+0000","last_undegraded":"2024-01-16T15:18:50.503470+0000","last_fullsized":"2024-01-16T15:18:50.503470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.508797+0000","last_change":"2024-01-16T15:18:49.508797+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.508797+0000","last_undegraded":"2024-01-16T15:18:49.508797+0000","last_fullsized":"2024-01-16T15:18:49.508797+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.525562+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:18:49.525562+0000","last_peered":"2024-01-16T15:18:49.525562+0000","last_clean":"2024-01-16T15:18:49.525562+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:18:49.525562+0000","last_undegraded":"2024-01-16T15:18:49.525562+0000","last_fullsized":"2024-01-16T15:18:49.525562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:10.112260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500474+0000","last_change":"2024-01-16T15:18:49.500474+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500474+0000","last_undegraded":"2024-01-16T15:18:49.500474+0000","last_fullsized":"2024-01-16T15:18:49.500474+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.515357+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:18:49.515357+0000","last_peered":"2024-01-16T15:18:49.515357+0000","last_clean":"2024-01-16T15:18:49.515357+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:18:49.515357+0000","last_undegraded":"2024-01-16T15:18:49.515357+0000","last_fullsized":"2024-01-16T15:18:49.515357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:56.122268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500974+0000","last_change":"2024-01-16T15:18:49.500974+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500974+0000","last_undegraded":"2024-01-16T15:18:49.500974+0000","last_fullsized":"2024-01-16T15:18:49.500974+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.536241+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:18:50.536241+0000","last_peered":"2024-01-16T15:18:50.536241+0000","last_clean":"2024-01-16T15:18:50.536241+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:18:50.536241+0000","last_undegraded":"2024-01-16T15:18:50.536241+0000","last_fullsized":"2024-01-16T15:18:50.536241+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.503096+0000","last_change":"2024-01-16T15:18:49.503096+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.503096+0000","last_undegraded":"2024-01-16T15:18:49.503096+0000","last_fullsized":"2024-01-16T15:18:49.503096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.538150+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:18:50.538150+0000","last_peered":"2024-01-16T15:18:50.538150+0000","last_clean":"2024-01-16T15:18:50.538150+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:18:50.538150+0000","last_undegraded":"2024-01-16T15:18:50.538150+0000","last_fullsized":"2024-01-16T15:18:50.538150+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498243+0000","last_change":"2024-01-16T15:18:49.498243+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498243+0000","last_undegraded":"2024-01-16T15:18:49.498243+0000","last_fullsized":"2024-01-16T15:18:49.498243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520668+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:18:49.520668+0000","last_peered":"2024-01-16T15:18:49.520668+0000","last_clean":"2024-01-16T15:18:49.520668+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:18:49.520668+0000","last_undegraded":"2024-01-16T15:18:49.520668+0000","last_fullsized":"2024-01-16T15:18:49.520668+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499199+0000","last_change":"2024-01-16T15:18:49.499199+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499199+0000","last_undegraded":"2024-01-16T15:18:49.499199+0000","last_fullsized":"2024-01-16T15:18:49.499199+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516723+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:18:49.516723+0000","last_peered":"2024-01-16T15:18:49.516723+0000","last_clean":"2024-01-16T15:18:49.516723+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:18:49.516723+0000","last_undegraded":"2024-01-16T15:18:49.516723+0000","last_fullsized":"2024-01-16T15:18:49.516723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:21.691380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499391+0000","last_change":"2024-01-16T15:18:49.499391+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499391+0000","last_undegraded":"2024-01-16T15:18:49.499391+0000","last_fullsized":"2024-01-16T15:18:49.499391+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511289+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:18:49.511289+0000","last_peered":"2024-01-16T15:18:49.511289+0000","last_clean":"2024-01-16T15:18:49.511289+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:18:49.511289+0000","last_undegraded":"2024-01-16T15:18:49.511289+0000","last_fullsized":"2024-01-16T15:18:49.511289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:29.197614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495765+0000","last_change":"2024-01-16T15:18:49.495765+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495765+0000","last_undegraded":"2024-01-16T15:18:49.495765+0000","last_fullsized":"2024-01-16T15:18:49.495765+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533470+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:18:50.533470+0000","last_peered":"2024-01-16T15:18:50.533470+0000","last_clean":"2024-01-16T15:18:50.533470+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:18:50.533470+0000","last_undegraded":"2024-01-16T15:18:50.533470+0000","last_fullsized":"2024-01-16T15:18:50.533470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.516561+0000","last_change":"2024-01-16T15:18:49.516561+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.516561+0000","last_undegraded":"2024-01-16T15:18:49.516561+0000","last_fullsized":"2024-01-16T15:18:49.516561+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511184+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:18:49.511184+0000","last_peered":"2024-01-16T15:18:49.511184+0000","last_clean":"2024-01-16T15:18:49.511184+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:18:49.511184+0000","last_undegraded":"2024-01-16T15:18:49.511184+0000","last_fullsized":"2024-01-16T15:18:49.511184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:53.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.510804+0000","last_change":"2024-01-16T15:18:49.510804+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.510804+0000","last_undegraded":"2024-01-16T15:18:49.510804+0000","last_fullsized":"2024-01-16T15:18:49.510804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520479+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:18:49.520479+0000","last_peered":"2024-01-16T15:18:49.520479+0000","last_clean":"2024-01-16T15:18:49.520479+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:18:49.520479+0000","last_undegraded":"2024-01-16T15:18:49.520479+0000","last_fullsized":"2024-01-16T15:18:49.520479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:53.321942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507278+0000","last_change":"2024-01-16T15:18:49.507278+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507278+0000","last_undegraded":"2024-01-16T15:18:49.507278+0000","last_fullsized":"2024-01-16T15:18:49.507278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520339+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:18:49.520339+0000","last_peered":"2024-01-16T15:18:49.520339+0000","last_clean":"2024-01-16T15:18:49.520339+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:18:49.520339+0000","last_undegraded":"2024-01-16T15:18:49.520339+0000","last_fullsized":"2024-01-16T15:18:49.520339+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.504828+0000","last_change":"2024-01-16T15:18:49.504828+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.504828+0000","last_undegraded":"2024-01-16T15:18:49.504828+0000","last_fullsized":"2024-01-16T15:18:49.504828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523509+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:18:49.523509+0000","last_peered":"2024-01-16T15:18:49.523509+0000","last_clean":"2024-01-16T15:18:49.523509+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:18:49.523509+0000","last_undegraded":"2024-01-16T15:18:49.523509+0000","last_fullsized":"2024-01-16T15:18:49.523509+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518601+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:18:50.518601+0000","last_peered":"2024-01-16T15:18:50.518601+0000","last_clean":"2024-01-16T15:18:50.518601+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:18:50.518601+0000","last_undegraded":"2024-01-16T15:18:50.518601+0000","last_fullsized":"2024-01-16T15:18:50.518601+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:27.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522193+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:18:49.522193+0000","last_peered":"2024-01-16T15:18:49.522193+0000","last_clean":"2024-01-16T15:18:49.522193+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:18:49.522193+0000","last_undegraded":"2024-01-16T15:18:49.522193+0000","last_fullsized":"2024-01-16T15:18:49.522193+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:54.524825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.506081+0000","last_change":"2024-01-16T15:18:49.506081+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.506081+0000","last_undegraded":"2024-01-16T15:18:49.506081+0000","last_fullsized":"2024-01-16T15:18:49.506081+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523683+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:18:49.523683+0000","last_peered":"2024-01-16T15:18:49.523683+0000","last_clean":"2024-01-16T15:18:49.523683+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:18:49.523683+0000","last_undegraded":"2024-01-16T15:18:49.523683+0000","last_fullsized":"2024-01-16T15:18:49.523683+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498896+0000","last_change":"2024-01-16T15:18:49.498896+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498896+0000","last_undegraded":"2024-01-16T15:18:49.498896+0000","last_fullsized":"2024-01-16T15:18:49.498896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503708+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:18:50.503708+0000","last_peered":"2024-01-16T15:18:50.503708+0000","last_clean":"2024-01-16T15:18:50.503708+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:18:50.503708+0000","last_undegraded":"2024-01-16T15:18:50.503708+0000","last_fullsized":"2024-01-16T15:18:50.503708+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500913+0000","last_change":"2024-01-16T15:18:49.500913+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500913+0000","last_undegraded":"2024-01-16T15:18:49.500913+0000","last_fullsized":"2024-01-16T15:18:49.500913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522640+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:18:49.522640+0000","last_peered":"2024-01-16T15:18:49.522640+0000","last_clean":"2024-01-16T15:18:49.522640+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:18:49.522640+0000","last_undegraded":"2024-01-16T15:18:49.522640+0000","last_fullsized":"2024-01-16T15:18:49.522640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:48.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.526632+0000","last_change":"2024-01-16T15:18:50.522199+0000","last_active":"2024-01-16T15:18:50.526632+0000","last_peered":"2024-01-16T15:18:50.526632+0000","last_clean":"2024-01-16T15:18:50.526632+0000","last_became_active":"2024-01-16T15:18:50.521902+0000","last_became_peered":"2024-01-16T15:18:50.521902+0000","last_unstale":"2024-01-16T15:18:50.526632+0000","last_undegraded":"2024-01-16T15:18:50.526632+0000","last_fullsized":"2024-01-16T15:18:50.526632+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:11.568326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516769+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:18:49.516769+0000","last_peered":"2024-01-16T15:18:49.516769+0000","last_clean":"2024-01-16T15:18:49.516769+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:18:49.516769+0000","last_undegraded":"2024-01-16T15:18:49.516769+0000","last_fullsized":"2024-01-16T15:18:49.516769+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:17.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518602+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:18:50.518602+0000","last_peered":"2024-01-16T15:18:50.518602+0000","last_clean":"2024-01-16T15:18:50.518602+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:18:50.518602+0000","last_undegraded":"2024-01-16T15:18:50.518602+0000","last_fullsized":"2024-01-16T15:18:50.518602+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:38.334522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.505052+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:18:50.505052+0000","last_peered":"2024-01-16T15:18:50.505052+0000","last_clean":"2024-01-16T15:18:50.505052+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:18:50.505052+0000","last_undegraded":"2024-01-16T15:18:50.505052+0000","last_fullsized":"2024-01-16T15:18:50.505052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500292+0000","last_change":"2024-01-16T15:18:49.500292+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500292+0000","last_undegraded":"2024-01-16T15:18:49.500292+0000","last_fullsized":"2024-01-16T15:18:49.500292+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520661+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:18:49.520661+0000","last_peered":"2024-01-16T15:18:49.520661+0000","last_clean":"2024-01-16T15:18:49.520661+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:18:49.520661+0000","last_undegraded":"2024-01-16T15:18:49.520661+0000","last_fullsized":"2024-01-16T15:18:49.520661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:24.291444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497200+0000","last_change":"2024-01-16T15:18:49.497200+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497200+0000","last_undegraded":"2024-01-16T15:18:49.497200+0000","last_fullsized":"2024-01-16T15:18:49.497200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.510956+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:18:49.510956+0000","last_peered":"2024-01-16T15:18:49.510956+0000","last_clean":"2024-01-16T15:18:49.510956+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:18:49.510956+0000","last_undegraded":"2024-01-16T15:18:49.510956+0000","last_fullsized":"2024-01-16T15:18:49.510956+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496009+0000","last_change":"2024-01-16T15:18:49.496009+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496009+0000","last_undegraded":"2024-01-16T15:18:49.496009+0000","last_fullsized":"2024-01-16T15:18:49.496009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524006+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:18:49.524006+0000","last_peered":"2024-01-16T15:18:49.524006+0000","last_clean":"2024-01-16T15:18:49.524006+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:18:49.524006+0000","last_undegraded":"2024-01-16T15:18:49.524006+0000","last_fullsized":"2024-01-16T15:18:49.524006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505239+0000","last_change":"2024-01-16T15:18:49.505239+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505239+0000","last_undegraded":"2024-01-16T15:18:49.505239+0000","last_fullsized":"2024-01-16T15:18:49.505239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.519663+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:18:49.519663+0000","last_peered":"2024-01-16T15:18:49.519663+0000","last_clean":"2024-01-16T15:18:49.519663+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:18:49.519663+0000","last_undegraded":"2024-01-16T15:18:49.519663+0000","last_fullsized":"2024-01-16T15:18:49.519663+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:27.624738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520862+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:18:50.520862+0000","last_peered":"2024-01-16T15:18:50.520862+0000","last_clean":"2024-01-16T15:18:50.520862+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:18:50.520862+0000","last_undegraded":"2024-01-16T15:18:50.520862+0000","last_fullsized":"2024-01-16T15:18:50.520862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:09.166757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503918+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:18:50.503918+0000","last_peered":"2024-01-16T15:18:50.503918+0000","last_clean":"2024-01-16T15:18:50.503918+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:18:50.503918+0000","last_undegraded":"2024-01-16T15:18:50.503918+0000","last_fullsized":"2024-01-16T15:18:50.503918+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498189+0000","last_change":"2024-01-16T15:18:49.498189+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498189+0000","last_undegraded":"2024-01-16T15:18:49.498189+0000","last_fullsized":"2024-01-16T15:18:49.498189+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524346+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:18:49.524346+0000","last_peered":"2024-01-16T15:18:49.524346+0000","last_clean":"2024-01-16T15:18:49.524346+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:18:49.524346+0000","last_undegraded":"2024-01-16T15:18:49.524346+0000","last_fullsized":"2024-01-16T15:18:49.524346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:18:48.485739+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:18:48.485739+0000","last_peered":"2024-01-16T15:18:48.485739+0000","last_clean":"2024-01-16T15:18:48.485739+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:18:48.485739+0000","last_undegraded":"2024-01-16T15:18:48.485739+0000","last_fullsized":"2024-01-16T15:18:48.485739+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542381,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1970176,"data_stored":1183817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.349,"15min":1.5429999999999999},"min":{"1min":0.72099999999999997,"5min":0.63500000000000001,"15min":0.28199999999999997},"max":{"1min":1.768,"5min":5.2690000000000001,"15min":27.469000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4410000000000001,"15min":1.6859999999999999},"min":{"1min":0.86099999999999999,"5min":0.79500000000000004,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":6.5599999999999996,"15min":25.503},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.139,"15min":1.2689999999999999},"min":{"1min":0.77600000000000002,"5min":0.55100000000000005,"15min":0.34499999999999997},"max":{"1min":1.712,"5min":1.742,"15min":6.718},"last":1.343},{"interface":"front","average":{"1min":1.262,"5min":1.2070000000000001,"15min":1.3520000000000001},"min":{"1min":0.85499999999999998,"5min":0.75700000000000001,"15min":0.373},"max":{"1min":1.7669999999999999,"5min":2.0110000000000001,"15min":15.452},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.4530000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.48299999999999998},"max":{"1min":1.7709999999999999,"5min":11.536,"15min":11.536},"last":1.25},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5409999999999999,"15min":1.5629999999999999},"min":{"1min":1.0720000000000001,"5min":0.79800000000000004,"15min":0.504},"max":{"1min":1.978,"5min":11.576000000000001,"15min":25.478000000000002},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.222,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.67000000000000004,"15min":0.40899999999999997},"max":{"1min":1.593,"5min":2.133,"15min":13.907999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.357,"5min":1.365,"15min":1.5029999999999999},"min":{"1min":1.022,"5min":0.89300000000000002,"15min":0.62},"max":{"1min":1.8580000000000001,"5min":2.1960000000000002,"15min":8.6999999999999993},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.452},"min":{"1min":0.89200000000000002,"5min":0.85299999999999998,"15min":0.376},"max":{"1min":1.7829999999999999,"5min":6.7599999999999998,"15min":25.401},"last":1.171},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3280000000000001,"15min":1.472},"min":{"1min":0.84999999999999998,"5min":0.59599999999999997,"15min":0.45000000000000001},"max":{"1min":2.069,"5min":2.069,"15min":10.134},"last":1.5489999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":0.96399999999999997,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.7310000000000001,"5min":2.1259999999999999,"15min":12.528},"last":1.486},{"interface":"front","average":{"1min":1.498,"5min":1.544,"15min":1.639},"min":{"1min":1.0640000000000001,"5min":0.86599999999999999,"15min":0.41399999999999998},"max":{"1min":1.9159999999999999,"5min":6.8719999999999999,"15min":26.327000000000002},"last":1.302}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.6759999999999999,"15min":1.734},"min":{"1min":1.3640000000000001,"5min":1.087,"15min":0.45200000000000001},"max":{"1min":2.0870000000000002,"5min":6.9690000000000003,"15min":25.451000000000001},"last":1.462},{"interface":"front","average":{"1min":1.524,"5min":1.4930000000000001,"15min":1.605},"min":{"1min":0.92400000000000004,"5min":0.89800000000000002,"15min":0.44700000000000001},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":12.577999999999999},"last":1.6120000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72716,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299124,"statfs":{"total":96636764160,"available":96562302976,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.452,"15min":1.744},"min":{"1min":0.752,"5min":0.71299999999999997,"15min":0.35399999999999998},"max":{"1min":1.53,"5min":29.77,"15min":30.164999999999999},"last":1.53},{"interface":"front","average":{"1min":1.292,"5min":1.6870000000000001,"15min":1.8779999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.48899999999999999},"max":{"1min":1.5780000000000001,"5min":36.405999999999999,"15min":36.405999999999999},"last":1.49}]},{"osd":1,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1850000000000001,"15min":1.508},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.47899999999999998},"max":{"1min":1.52,"5min":5.2859999999999996,"15min":28.122},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.2549999999999999,"15min":1.298},"min":{"1min":0.78200000000000003,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":1.548,"5min":10.494,"15min":10.494},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.347,"15min":1.734},"min":{"1min":1,"5min":0.76900000000000002,"15min":0.501},"max":{"1min":1.845,"5min":5.0549999999999997,"15min":26.109000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.51,"15min":1.893},"min":{"1min":1.0329999999999999,"5min":1.016,"15min":0.67100000000000004},"max":{"1min":2.0449999999999999,"5min":5.1459999999999999,"15min":26.161000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3029999999999999,"15min":1.615},"min":{"1min":1.1060000000000001,"5min":0.68500000000000005,"15min":0.44700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8079999999999998,"15min":17.75},"last":1.363},{"interface":"front","average":{"1min":1.395,"5min":1.681,"15min":1.728},"min":{"1min":1.0349999999999999,"5min":0.77100000000000002,"15min":0.55700000000000005},"max":{"1min":2.1789999999999998,"5min":21.456,"15min":21.456},"last":1.5600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3540000000000001,"15min":1.5069999999999999},"min":{"1min":0.91400000000000003,"5min":0.82099999999999995,"15min":0.45000000000000001},"max":{"1min":1.6699999999999999,"5min":5.3310000000000004,"15min":15.571},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.325,"15min":1.5469999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.42299999999999999},"max":{"1min":1.639,"5min":3.6160000000000001,"15min":23.667999999999999},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4350000000000001,"15min":1.4890000000000001},"min":{"1min":0.83499999999999996,"5min":0.73699999999999999,"15min":0.42299999999999999},"max":{"1min":1.982,"5min":13.949999999999999,"15min":23.137},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.6100000000000001,"15min":1.6970000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":0.47099999999999997},"max":{"1min":1.714,"5min":11.298,"15min":25.684000000000001},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.6020000000000001,"15min":1.552},"min":{"1min":0.754,"5min":0.71499999999999997,"15min":0.47099999999999997},"max":{"1min":2.1000000000000001,"5min":14.037000000000001,"15min":14.037000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6240000000000001,"15min":1.724},"min":{"1min":0.95899999999999996,"5min":0.92300000000000004,"15min":0.499},"max":{"1min":2.1659999999999999,"5min":4.9829999999999997,"15min":11.788},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.075,"15min":1.571},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.39000000000000001},"max":{"1min":1.5649999999999999,"5min":1.651,"15min":26.228999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.135,"15min":1.831},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.34799999999999998},"max":{"1min":1.7070000000000001,"5min":1.849,"15min":56.011000000000003},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1419999999999999,"15min":1.363},"min":{"1min":0.79900000000000004,"5min":0.65200000000000002,"15min":0.40999999999999998},"max":{"1min":1.6419999999999999,"5min":2.2450000000000001,"15min":16.687999999999999},"last":1.034},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2430000000000001,"15min":1.8600000000000001},"min":{"1min":0.88900000000000001,"5min":0.745,"15min":0.40999999999999998},"max":{"1min":1.667,"5min":4.8499999999999996,"15min":67.930000000000007},"last":1.125}]},{"osd":2,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1779999999999999,"15min":1.52},"min":{"1min":0.85199999999999998,"5min":0.61099999999999999,"15min":0.47099999999999997},"max":{"1min":1.55,"5min":1.7529999999999999,"15min":13.193},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.246,"5min":1.258,"15min":1.4850000000000001},"min":{"1min":0.93700000000000006,"5min":0.76300000000000001,"15min":0.45800000000000002},"max":{"1min":1.7769999999999999,"5min":3.9590000000000001,"15min":16.721},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.379,"15min":1.712},"min":{"1min":1.0509999999999999,"5min":0.81100000000000005,"15min":0.57999999999999996},"max":{"1min":1.8680000000000001,"5min":4.8010000000000002,"15min":22.382000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.3819999999999999,"15min":1.7529999999999999},"min":{"1min":0.94199999999999995,"5min":0.72399999999999998,"15min":0.51200000000000001},"max":{"1min":1.921,"5min":1.921,"15min":14.685},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2649999999999999,"15min":1.419},"min":{"1min":0.95199999999999996,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.8680000000000001,"5min":2.2549999999999999,"15min":16.382999999999999},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.377,"15min":1.5269999999999999},"min":{"1min":0.98899999999999999,"5min":0.502,"15min":0.379},"max":{"1min":1.8129999999999999,"5min":4.6580000000000004,"15min":27.006},"last":1.6539999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3600000000000001,"15min":1.5600000000000001},"min":{"1min":0.83999999999999997,"5min":0.52400000000000002,"15min":0.39700000000000002},"max":{"1min":1.974,"5min":1.9770000000000001,"15min":13.263999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.4890000000000001,"15min":1.609},"min":{"1min":1.304,"5min":0.629,"15min":0.505},"max":{"1min":1.9670000000000001,"5min":2.1640000000000001,"15min":9.5269999999999992},"last":1.704}]},{"osd":7,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.52,"15min":1.7470000000000001},"min":{"1min":1.2410000000000001,"5min":0.59099999999999997,"15min":0.45000000000000001},"max":{"1min":1.9119999999999999,"5min":4.742,"15min":35.127000000000002},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5489999999999999,"15min":1.7290000000000001},"min":{"1min":0.91200000000000003,"5min":0.61099999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":13.491},"last":1.601}]}]},{"osd":4,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1619999999999999,"15min":1.244},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6180000000000001,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.21},{"interface":"front","average":{"1min":1.095,"5min":1.123,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.679,"5min":1.958,"15min":60.106000000000002},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.629,"15min":1.79},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51100000000000001},"max":{"1min":1.738,"5min":24.399999999999999,"15min":24.399999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.611,"15min":1.627},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57099999999999995},"max":{"1min":1.7290000000000001,"5min":24.456,"15min":24.456},"last":1.7090000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.3029999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.52800000000000002},"max":{"1min":1.6299999999999999,"5min":2.1699999999999999,"15min":9.7140000000000004},"last":1.145},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2889999999999999,"15min":1.6970000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.52400000000000002},"max":{"1min":1.6479999999999999,"5min":1.855,"15min":35.024999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3300000000000001,"15min":1.556},"min":{"1min":0.503,"5min":0.503,"15min":0.47799999999999998},"max":{"1min":1.6499999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":6.4589999999999996,"15min":7.8140000000000001},"last":1.375}]},{"osd":5,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.454},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":2.1299999999999999,"15min":14.568},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4690000000000001,"15min":1.4319999999999999},"min":{"1min":0.74299999999999999,"5min":0.496,"15min":0.34399999999999997},"max":{"1min":1.8160000000000001,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":2.0720000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.436},"max":{"1min":1.696,"5min":3.0190000000000001,"15min":9.6630000000000003},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4239999999999999,"15min":1.6299999999999999},"min":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.45100000000000001},"max":{"1min":1.675,"5min":2.3980000000000001,"15min":14.317},"last":2.0129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5640000000000001,"15min":1.5449999999999999},"min":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.52100000000000002},"max":{"1min":1.927,"5min":6.5250000000000004,"15min":9.2799999999999994},"last":1.903},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.552,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.443},"max":{"1min":1.8540000000000001,"5min":3.3439999999999999,"15min":9.2270000000000003},"last":2.2229999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82432,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289408,"statfs":{"total":96636764160,"available":96552353792,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.93899999999999995,"15min":1.429},"min":{"1min":0.748,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.45,"5min":1.9299999999999999,"15min":50.988999999999997},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.087,"15min":1.4750000000000001},"min":{"1min":0.80100000000000005,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5840000000000001,"5min":1.786,"15min":20.274000000000001},"last":1.141}]},{"osd":1,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.101,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.371,"15min":0.32100000000000001},"max":{"1min":1.613,"5min":3.6640000000000001,"15min":18.192},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.21,"15min":1.712},"min":{"1min":0.89000000000000001,"5min":0.39400000000000002,"15min":0.313},"max":{"1min":1.8440000000000001,"5min":1.9239999999999999,"15min":20.385999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3089999999999999,"15min":1.7250000000000001},"min":{"1min":0.871,"5min":0.45500000000000002,"15min":0.36699999999999999},"max":{"1min":1.978,"5min":3.5249999999999999,"15min":42.840000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.397,"15min":1.806},"min":{"1min":1.016,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.631,"5min":3.702,"15min":38.707000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.339,"15min":1.762},"min":{"1min":0.99199999999999999,"5min":0.78200000000000003,"15min":0.44500000000000001},"max":{"1min":1.734,"5min":2.327,"15min":18.952000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4650000000000001,"15min":1.79},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.54800000000000004},"max":{"1min":1.718,"5min":3.9529999999999998,"15min":18.088999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4890000000000001,"15min":1.8160000000000001},"min":{"1min":1.212,"5min":0.73199999999999998,"15min":0.55500000000000005},"max":{"1min":1.7729999999999999,"5min":2.661,"15min":24.349},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.464,"15min":1.9199999999999999},"min":{"1min":1.105,"5min":0.753,"15min":0.61899999999999999},"max":{"1min":1.9119999999999999,"5min":3.4239999999999999,"15min":14.192},"last":1.306}]},{"osd":6,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.621,"15min":1.877},"min":{"1min":1.159,"5min":0.77200000000000002,"15min":0.52200000000000002},"max":{"1min":1.7929999999999999,"5min":3.9009999999999998,"15min":22.687000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.623,"15min":1.931},"min":{"1min":1.262,"5min":0.73799999999999999,"15min":0.59699999999999998},"max":{"1min":1.7969999999999999,"5min":3.9300000000000002,"15min":24.058},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6040000000000001,"15min":2.1000000000000001},"min":{"1min":1.111,"5min":0.85099999999999998,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":6.3209999999999997,"15min":35.570999999999998},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.6659999999999999,"15min":2.0619999999999998},"min":{"1min":1.2969999999999999,"5min":0.81100000000000005,"15min":0.58899999999999997},"max":{"1min":2.0750000000000002,"5min":3.8279999999999998,"15min":37.607999999999997},"last":1.633}]}]},{"osd":2,"up_from":14,"seq":60129542380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94924,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94276916,"statfs":{"total":96636764160,"available":96539561984,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90400000000000003,"15min":1.4430000000000001},"min":{"1min":0.67700000000000005,"5min":0.36099999999999999,"15min":0.24399999999999999},"max":{"1min":1.135,"5min":1.96,"15min":64.387},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.022,"15min":1.387},"min":{"1min":0.76500000000000001,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7030000000000001,"5min":1.9890000000000001,"15min":24.308},"last":1.7490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0580000000000001,"15min":1.466},"min":{"1min":0.752,"5min":0.46300000000000002,"15min":0.34699999999999998},"max":{"1min":1.2490000000000001,"5min":2.093,"15min":27.263000000000002},"last":1.27},{"interface":"front","average":{"1min":1.381,"5min":1.3879999999999999,"15min":1.8520000000000001},"min":{"1min":1.0940000000000001,"5min":0.51300000000000001,"15min":0.38500000000000001},"max":{"1min":1.8620000000000001,"5min":2.1539999999999999,"15min":23.602},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.385,"15min":1.627},"min":{"1min":1.1499999999999999,"5min":0.58499999999999996,"15min":0.40100000000000002},"max":{"1min":1.9299999999999999,"5min":2.085,"15min":27.311},"last":1.081},{"interface":"front","average":{"1min":1.296,"5min":1.327,"15min":1.6299999999999999},"min":{"1min":0.96599999999999997,"5min":0.498,"15min":0.47399999999999998},"max":{"1min":1.698,"5min":1.998,"15min":26.555},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.389,"15min":1.637},"min":{"1min":1.0960000000000001,"5min":0.66100000000000003,"15min":0.38900000000000001},"max":{"1min":1.764,"5min":2.1040000000000001,"15min":26.503},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.3819999999999999,"15min":1.7490000000000001},"min":{"1min":1.0009999999999999,"5min":0.56399999999999995,"15min":0.51200000000000001},"max":{"1min":1.9079999999999999,"5min":2.4319999999999999,"15min":25.661000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.4610000000000001,"15min":1.74},"min":{"1min":1.3120000000000001,"5min":0.74199999999999999,"15min":0.45800000000000002},"max":{"1min":2.077,"5min":2.3490000000000002,"15min":27.149000000000001},"last":1.49},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.538,"15min":1.7330000000000001},"min":{"1min":1.1930000000000001,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":1.8540000000000001,"5min":2.6589999999999998,"15min":26.349},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4910000000000001,"15min":1.9610000000000001},"min":{"1min":1.0700000000000001,"5min":0.64500000000000002,"15min":0.504},"max":{"1min":2.1909999999999998,"5min":2.7149999999999999,"15min":25.603999999999999},"last":1.548},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.597,"15min":1.843},"min":{"1min":1.1579999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":2.157,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.591,"15min":1.9470000000000001},"min":{"1min":1.343,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1200000000000001,"5min":2.4889999999999999,"15min":24.509},"last":1.649},{"interface":"front","average":{"1min":1.657,"5min":1.5880000000000001,"15min":1.927},"min":{"1min":1.282,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.2229999999999999,"5min":2.3479999999999999,"15min":27.207999999999998},"last":1.8080000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.92400000000000004,"15min":1.153},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.28399999999999997},"max":{"1min":1.696,"5min":1.865,"15min":11.699},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0489999999999999,"15min":1.276},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.33700000000000002},"max":{"1min":1.5089999999999999,"5min":2.3860000000000001,"15min":13.930999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.238,"15min":1.385},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.31},"max":{"1min":1.8,"5min":15.557,"15min":16.488},"last":1.274},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1559999999999999,"15min":1.5680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39000000000000001},"max":{"1min":1.968,"5min":2.306,"15min":22.550999999999998},"last":1.4650000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.6879999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.439},"max":{"1min":1.7450000000000001,"5min":1.9790000000000001,"15min":22.471},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3660000000000001,"15min":1.6359999999999999},"min":{"1min":0.54100000000000004,"5min":0.49099999999999999,"15min":0.35699999999999998},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":22.550999999999998},"last":0.82899999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4870000000000001,"15min":1.6240000000000001},"min":{"1min":0.96699999999999997,"5min":0.95699999999999996,"15min":0.60899999999999999},"max":{"1min":1.95,"5min":2.7050000000000001,"15min":8.3620000000000001},"last":1.333},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5960000000000001,"15min":1.7070000000000001},"min":{"1min":0.97399999999999998,"5min":0.69999999999999996,"15min":0.433},"max":{"1min":1.97,"5min":15.723000000000001,"15min":19.236999999999998},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.706},"min":{"1min":0.92800000000000005,"5min":0.73799999999999999,"15min":0.57199999999999995},"max":{"1min":1.843,"5min":2.2040000000000002,"15min":21.117000000000001},"last":1.175},{"interface":"front","average":{"1min":1.425,"5min":1.544,"15min":1.758},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.65800000000000003},"max":{"1min":1.8009999999999999,"5min":15.785,"15min":17.138000000000002},"last":1.7569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.4850000000000001,"15min":1.698},"min":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.72199999999999998},"max":{"1min":2.2759999999999998,"5min":2.298,"15min":11.901999999999999},"last":1.798},{"interface":"front","average":{"1min":1.629,"5min":1.5840000000000001,"15min":1.8069999999999999},"min":{"1min":1.044,"5min":0.86399999999999999,"15min":0.63300000000000001},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":23.420999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.631,"15min":1.8380000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.085,"5min":15.875,"15min":17.010000000000002},"last":1.423},{"interface":"front","average":{"1min":1.591,"5min":1.573,"15min":1.825},"min":{"1min":0.998,"5min":0.91600000000000004,"15min":0.75700000000000001},"max":{"1min":2.508,"5min":2.508,"15min":11.962},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542381,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73308,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298532,"statfs":{"total":96636764160,"available":96561696768,"internally_reserved":0,"allocated":1982464,"data_stored":1190620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":73071041},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.016,"15min":1.468},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":1.355,"5min":4.8380000000000001,"15min":43.459000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.2290000000000001,"15min":1.706},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.956,"5min":4.9779999999999998,"15min":43.627000000000002},"last":1.042}]},{"osd":2,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1919999999999999,"15min":1.4099999999999999},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":5.0570000000000004,"15min":18.861000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.218,"15min":1.5229999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":2.2330000000000001,"5min":5.3550000000000004,"15min":38.893000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0800000000000001,"15min":1.47},"min":{"1min":0.37,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":1.4410000000000001,"5min":4.9050000000000002,"15min":23.687999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.357,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.434},"max":{"1min":1.6080000000000001,"5min":5.1980000000000004,"15min":18.966999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3460000000000001,"15min":1.5129999999999999},"min":{"1min":0.76500000000000001,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.8420000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.496,"15min":1.7030000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.873,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.423}]},{"osd":5,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.508,"15min":1.891},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.792,"5min":5.1589999999999998,"15min":24.611000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.6970000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.56000000000000005},"max":{"1min":1.907,"5min":7.2699999999999996,"15min":20.199000000000002},"last":1.75}]},{"osd":6,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6499999999999999,"15min":2.0510000000000002},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.55900000000000005},"max":{"1min":2.3119999999999998,"5min":6.3339999999999996,"15min":26.998000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.6140000000000001,"15min":1.988},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":2.1669999999999998,"5min":5.9690000000000003,"15min":24.478999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6699999999999999,"15min":1.9139999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.60899999999999999},"max":{"1min":2.2599999999999998,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.657},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.5629999999999999,"15min":1.7310000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.52800000000000002},"max":{"1min":1.8999999999999999,"5min":6.149,"15min":20.279},"last":1.8280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:18:55.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:18:55.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.742+0000 7f0eb6800640 1 Processor -- start 2024-01-16T15:18:55.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.742+0000 7f0eb6800640 1 -- start start 2024-01-16T15:18:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb013dfd0 0x7f0eb015be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0eb00cf050 0x7f0eb00cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0eb006c130 0x7f0eb006c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb00cfa10 con 0x7f0eb00cf050 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb015fa70 con 0x7f0eb006c130 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb6800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb015ee60 con 0x7f0eb013dfd0 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.744+0000 7f0eb4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0eb00cf050 0x7f0eb00cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0eb00cf050 0x7f0eb00cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38342/0 (socket says 172.21.15.43:38342) 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 -- 172.21.15.43:0/763796740 learned_addr learned my addr 172.21.15.43:0/763796740 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb5400640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb013dfd0 0x7f0eb015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb5e00640 1 --2- 172.21.15.43:0/763796740 >> v2:172.21.15.69:3300/0 conn(0x7f0eb006c130 0x7f0eb006c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 -- 172.21.15.43:0/763796740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb013dfd0 msgr2=0x7f0eb015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb013dfd0 0x7f0eb015be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 -- 172.21.15.43:0/763796740 >> v2:172.21.15.69:3300/0 conn(0x7f0eb006c130 msgr2=0x7f0eb006c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/763796740 >> v2:172.21.15.69:3300/0 conn(0x7f0eb006c130 0x7f0eb006c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.745+0000 7f0eb4a00640 1 -- 172.21.15.43:0/763796740 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb015f140 con 0x7f0eb00cf050 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.746+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00cf4d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0eb006d520 tx=0x7f0e9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd6a39cc9c7745a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eafe00640 1 -- 172.21.15.43:0/763796740 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98031870 con 0x7f0eb00cf050 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eafe00640 1 -- 172.21.15.43:0/763796740 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e980319d0 con 0x7f0eb00cf050 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eafe00640 1 -- 172.21.15.43:0/763796740 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98031d60 con 0x7f0eb00cf050 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eb6800640 1 -- 172.21.15.43:0/763796740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 msgr2=0x7f0eb00cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eb6800640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00cf4d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0eb006d520 tx=0x7f0e9802b910 comp rx=0 tx=0).stop 2024-01-16T15:18:55.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eb6800640 1 -- 172.21.15.43:0/763796740 shutdown_connections 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.747+0000 7f0eb6800640 1 --2- 172.21.15.43:0/763796740 >> v2:172.21.15.69:3300/0 conn(0x7f0eb006c130 0x7f0eb006c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.748+0000 7f0eb6800640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00cf4d0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.748+0000 7f0eb6800640 1 --2- 172.21.15.43:0/763796740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb013dfd0 0x7f0eb015be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.748+0000 7f0eb6800640 1 -- 172.21.15.43:0/763796740 >> 172.21.15.43:0/763796740 conn(0x7f0eb006a860 msgr2=0x7f0eb006ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.748+0000 7f0eb6800640 1 -- 172.21.15.43:0/763796740 shutdown_connections 2024-01-16T15:18:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.748+0000 7f0eb6800640 1 -- 172.21.15.43:0/763796740 wait complete. 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.749+0000 7f0eb6800640 1 Processor -- start 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.749+0000 7f0eb6800640 1 -- start start 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb006c130 0x7f0eb0154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb013dfd0 0x7f0eb0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb01552f0 con 0x7f0eb00cf050 2024-01-16T15:18:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0155fc0 con 0x7f0eb013dfd0 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.750+0000 7f0eb6800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb01562a0 con 0x7f0eb006c130 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38356/0 (socket says 172.21.15.43:38356) 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 -- 172.21.15.43:0/661436517 learned_addr learned my addr 172.21.15.43:0/661436517 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb5400640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb006c130 0x7f0eb0154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb5e00640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb013dfd0 0x7f0eb0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 -- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb006c130 msgr2=0x7f0eb0154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb006c130 0x7f0eb0154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 -- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb013dfd0 msgr2=0x7f0eb0157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb013dfd0 0x7f0eb0157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.751+0000 7f0eb4a00640 1 -- 172.21.15.43:0/661436517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980094d0 con 0x7f0eb00cf050 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0eb4a00640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0e9802b950 tx=0x7f0e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9803abe0 con 0x7f0eb00cf050 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9803ad60 con 0x7f0eb00cf050 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb0156580 con 0x7f0eb00cf050 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98044630 con 0x7f0eb00cf050 2024-01-16T15:18:55.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.752+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb0156b40 con 0x7f0eb00cf050 2024-01-16T15:18:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.753+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb004e840 con 0x7f0eb00cf050 2024-01-16T15:18:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.754+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0e9804d450 con 0x7f0eb00cf050 2024-01-16T15:18:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.755+0000 7f0ead600640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 0x7f0e8806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.755+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0e9802d040 con 0x7f0eb00cf050 2024-01-16T15:18:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.755+0000 7f0eb5400640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 0x7f0e8806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.756+0000 7f0eb5400640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 0x7f0e8806e860 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006fd0 tx=0x7f0ea0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.759+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e980775b0 con 0x7f0eb00cf050 2024-01-16T15:18:55.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.951+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0eb015eb30 con 0x7f0e8806c3a0 2024-01-16T15:18:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.974+0000 7f0ead600640 1 -- 172.21.15.43:0/661436517 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282614 (secure 0 0 0) 0x7f0ea000de40 con 0x7f0e8806c3a0 2024-01-16T15:18:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:55.971 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:18:55.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 msgr2=0x7f0e8806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 0x7f0e8806e860 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006fd0 tx=0x7f0ea0008040 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 msgr2=0x7f0eb00b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0e9802b950 tx=0x7f0e980046c0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 shutdown_connections 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0e8806c3a0 0x7f0e8806e860 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0eb013dfd0 0x7f0eb0157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0eb00cf050 0x7f0eb00b3520 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 --2- 172.21.15.43:0/661436517 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0eb006c130 0x7f0eb0154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.981+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 >> 172.21.15.43:0/661436517 conn(0x7f0eb006a860 msgr2=0x7f0eb0159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.982+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 shutdown_connections 2024-01-16T15:18:55.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:55.982+0000 7f0eb6800640 1 -- 172.21.15.43:0/661436517 wait complete. 2024-01-16T15:18:55.994 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-01-16T15:18:55.290428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592092,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":170,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606496,"kb_used_data":11900,"kb_used_omap":12,"kb_used_meta":594483,"kb_avail":754368224,"statfs":{"total":773094113280,"available":772473061376,"internally_reserved":0,"allocated":12185600,"data_stored":5922525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":608751148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001986"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520878+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:18:50.520878+0000","last_peered":"2024-01-16T15:18:50.520878+0000","last_clean":"2024-01-16T15:18:50.520878+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:18:50.520878+0000","last_undegraded":"2024-01-16T15:18:50.520878+0000","last_fullsized":"2024-01-16T15:18:50.520878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:02.958841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496455+0000","last_change":"2024-01-16T15:18:49.496455+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496455+0000","last_undegraded":"2024-01-16T15:18:49.496455+0000","last_fullsized":"2024-01-16T15:18:49.496455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511018+0000","last_change":"2024-01-16T15:18:49.511018+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511018+0000","last_undegraded":"2024-01-16T15:18:49.511018+0000","last_fullsized":"2024-01-16T15:18:49.511018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527282+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:18:50.527282+0000","last_peered":"2024-01-16T15:18:50.527282+0000","last_clean":"2024-01-16T15:18:50.527282+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:18:50.527282+0000","last_undegraded":"2024-01-16T15:18:50.527282+0000","last_fullsized":"2024-01-16T15:18:50.527282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:40.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494552+0000","last_change":"2024-01-16T15:18:49.494552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494552+0000","last_undegraded":"2024-01-16T15:18:49.494552+0000","last_fullsized":"2024-01-16T15:18:49.494552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511599+0000","last_change":"2024-01-16T15:18:49.511599+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511599+0000","last_undegraded":"2024-01-16T15:18:49.511599+0000","last_fullsized":"2024-01-16T15:18:49.511599+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502722+0000","last_change":"2024-01-16T15:18:49.502722+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502722+0000","last_undegraded":"2024-01-16T15:18:49.502722+0000","last_fullsized":"2024-01-16T15:18:49.502722+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507251+0000","last_change":"2024-01-16T15:18:49.507251+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507251+0000","last_undegraded":"2024-01-16T15:18:49.507251+0000","last_fullsized":"2024-01-16T15:18:49.507251+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527409+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:18:50.527409+0000","last_peered":"2024-01-16T15:18:50.527409+0000","last_clean":"2024-01-16T15:18:50.527409+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:18:50.527409+0000","last_undegraded":"2024-01-16T15:18:50.527409+0000","last_fullsized":"2024-01-16T15:18:50.527409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:44.209808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513531+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:18:50.513531+0000","last_peered":"2024-01-16T15:18:50.513531+0000","last_clean":"2024-01-16T15:18:50.513531+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:18:50.513531+0000","last_undegraded":"2024-01-16T15:18:50.513531+0000","last_fullsized":"2024-01-16T15:18:50.513531+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:37.753051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496236+0000","last_change":"2024-01-16T15:18:49.496236+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496236+0000","last_undegraded":"2024-01-16T15:18:49.496236+0000","last_fullsized":"2024-01-16T15:18:49.496236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513845+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:18:50.513845+0000","last_peered":"2024-01-16T15:18:50.513845+0000","last_clean":"2024-01-16T15:18:50.513845+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:18:50.513845+0000","last_undegraded":"2024-01-16T15:18:50.513845+0000","last_fullsized":"2024-01-16T15:18:50.513845+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509738+0000","last_change":"2024-01-16T15:18:49.509738+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509738+0000","last_undegraded":"2024-01-16T15:18:49.509738+0000","last_fullsized":"2024-01-16T15:18:49.509738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505304+0000","last_change":"2024-01-16T15:18:49.505304+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505304+0000","last_undegraded":"2024-01-16T15:18:49.505304+0000","last_fullsized":"2024-01-16T15:18:49.505304+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509868+0000","last_change":"2024-01-16T15:18:49.509868+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509868+0000","last_undegraded":"2024-01-16T15:18:49.509868+0000","last_fullsized":"2024-01-16T15:18:49.509868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518744+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:18:50.518744+0000","last_peered":"2024-01-16T15:18:50.518744+0000","last_clean":"2024-01-16T15:18:50.518744+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:18:50.518744+0000","last_undegraded":"2024-01-16T15:18:50.518744+0000","last_fullsized":"2024-01-16T15:18:50.518744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:42.136923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495707+0000","last_change":"2024-01-16T15:18:49.495707+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495707+0000","last_undegraded":"2024-01-16T15:18:49.495707+0000","last_fullsized":"2024-01-16T15:18:49.495707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501685+0000","last_change":"2024-01-16T15:18:49.501685+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501685+0000","last_undegraded":"2024-01-16T15:18:49.501685+0000","last_fullsized":"2024-01-16T15:18:49.501685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509552+0000","last_change":"2024-01-16T15:18:49.509552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509552+0000","last_undegraded":"2024-01-16T15:18:49.509552+0000","last_fullsized":"2024-01-16T15:18:49.509552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494776+0000","last_change":"2024-01-16T15:18:49.494776+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494776+0000","last_undegraded":"2024-01-16T15:18:49.494776+0000","last_fullsized":"2024-01-16T15:18:49.494776+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494903+0000","last_change":"2024-01-16T15:18:49.494903+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494903+0000","last_undegraded":"2024-01-16T15:18:49.494903+0000","last_fullsized":"2024-01-16T15:18:49.494903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498780+0000","last_change":"2024-01-16T15:18:49.498780+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498780+0000","last_undegraded":"2024-01-16T15:18:49.498780+0000","last_fullsized":"2024-01-16T15:18:49.498780+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494905+0000","last_change":"2024-01-16T15:18:49.494905+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494905+0000","last_undegraded":"2024-01-16T15:18:49.494905+0000","last_fullsized":"2024-01-16T15:18:49.494905+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498751+0000","last_change":"2024-01-16T15:18:49.498751+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498751+0000","last_undegraded":"2024-01-16T15:18:49.498751+0000","last_fullsized":"2024-01-16T15:18:49.498751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495191+0000","last_change":"2024-01-16T15:18:49.495191+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495191+0000","last_undegraded":"2024-01-16T15:18:49.495191+0000","last_fullsized":"2024-01-16T15:18:49.495191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518712+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:18:50.518712+0000","last_peered":"2024-01-16T15:18:50.518712+0000","last_clean":"2024-01-16T15:18:50.518712+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:18:50.518712+0000","last_undegraded":"2024-01-16T15:18:50.518712+0000","last_fullsized":"2024-01-16T15:18:50.518712+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:03.552086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511948+0000","last_change":"2024-01-16T15:18:49.511948+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511948+0000","last_undegraded":"2024-01-16T15:18:49.511948+0000","last_fullsized":"2024-01-16T15:18:49.511948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501471+0000","last_change":"2024-01-16T15:18:49.501471+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501471+0000","last_undegraded":"2024-01-16T15:18:49.501471+0000","last_fullsized":"2024-01-16T15:18:49.501471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498605+0000","last_change":"2024-01-16T15:18:49.498605+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498605+0000","last_undegraded":"2024-01-16T15:18:49.498605+0000","last_fullsized":"2024-01-16T15:18:49.498605+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502074+0000","last_change":"2024-01-16T15:18:49.502074+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502074+0000","last_undegraded":"2024-01-16T15:18:49.502074+0000","last_fullsized":"2024-01-16T15:18:49.502074+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.517526+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:18:50.517526+0000","last_peered":"2024-01-16T15:18:50.517526+0000","last_clean":"2024-01-16T15:18:50.517526+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:18:50.517526+0000","last_undegraded":"2024-01-16T15:18:50.517526+0000","last_fullsized":"2024-01-16T15:18:50.517526+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.126247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505422+0000","last_change":"2024-01-16T15:18:49.505422+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505422+0000","last_undegraded":"2024-01-16T15:18:49.505422+0000","last_fullsized":"2024-01-16T15:18:49.505422+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.513783+0000","last_change":"2024-01-16T15:18:49.513783+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.513783+0000","last_undegraded":"2024-01-16T15:18:49.513783+0000","last_fullsized":"2024-01-16T15:18:49.513783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520299+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:18:49.520299+0000","last_peered":"2024-01-16T15:18:49.520299+0000","last_clean":"2024-01-16T15:18:49.520299+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:18:49.520299+0000","last_undegraded":"2024-01-16T15:18:49.520299+0000","last_fullsized":"2024-01-16T15:18:49.520299+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501201+0000","last_change":"2024-01-16T15:18:49.501201+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501201+0000","last_undegraded":"2024-01-16T15:18:49.501201+0000","last_fullsized":"2024-01-16T15:18:49.501201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524289+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:18:49.524289+0000","last_peered":"2024-01-16T15:18:49.524289+0000","last_clean":"2024-01-16T15:18:49.524289+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:18:49.524289+0000","last_undegraded":"2024-01-16T15:18:49.524289+0000","last_fullsized":"2024-01-16T15:18:49.524289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:21.931597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.512665+0000","last_change":"2024-01-16T15:18:49.512665+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.512665+0000","last_undegraded":"2024-01-16T15:18:49.512665+0000","last_fullsized":"2024-01-16T15:18:49.512665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522310+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:18:49.522310+0000","last_peered":"2024-01-16T15:18:49.522310+0000","last_clean":"2024-01-16T15:18:49.522310+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:18:49.522310+0000","last_undegraded":"2024-01-16T15:18:49.522310+0000","last_fullsized":"2024-01-16T15:18:49.522310+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:02.444449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507992+0000","last_change":"2024-01-16T15:18:49.507992+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507992+0000","last_undegraded":"2024-01-16T15:18:49.507992+0000","last_fullsized":"2024-01-16T15:18:49.507992+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522279+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:18:49.522279+0000","last_peered":"2024-01-16T15:18:49.522279+0000","last_clean":"2024-01-16T15:18:49.522279+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:18:49.522279+0000","last_undegraded":"2024-01-16T15:18:49.522279+0000","last_fullsized":"2024-01-16T15:18:49.522279+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:44.623306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513485+0000","last_change":"2024-01-16T15:18:50.510180+0000","last_active":"2024-01-16T15:18:50.513485+0000","last_peered":"2024-01-16T15:18:50.513485+0000","last_clean":"2024-01-16T15:18:50.513485+0000","last_became_active":"2024-01-16T15:18:50.509888+0000","last_became_peered":"2024-01-16T15:18:50.509888+0000","last_unstale":"2024-01-16T15:18:50.513485+0000","last_undegraded":"2024-01-16T15:18:50.513485+0000","last_fullsized":"2024-01-16T15:18:50.513485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:49.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524377+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:18:49.524377+0000","last_peered":"2024-01-16T15:18:49.524377+0000","last_clean":"2024-01-16T15:18:49.524377+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:18:49.524377+0000","last_undegraded":"2024-01-16T15:18:49.524377+0000","last_fullsized":"2024-01-16T15:18:49.524377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497629+0000","last_change":"2024-01-16T15:18:49.497629+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497629+0000","last_undegraded":"2024-01-16T15:18:49.497629+0000","last_fullsized":"2024-01-16T15:18:49.497629+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.518795+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:18:49.518795+0000","last_peered":"2024-01-16T15:18:49.518795+0000","last_clean":"2024-01-16T15:18:49.518795+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:18:49.518795+0000","last_undegraded":"2024-01-16T15:18:49.518795+0000","last_fullsized":"2024-01-16T15:18:49.518795+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:43.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518433+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:18:50.518433+0000","last_peered":"2024-01-16T15:18:50.518433+0000","last_clean":"2024-01-16T15:18:50.518433+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:18:50.518433+0000","last_undegraded":"2024-01-16T15:18:50.518433+0000","last_fullsized":"2024-01-16T15:18:50.518433+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:48.856213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511369+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:18:49.511369+0000","last_peered":"2024-01-16T15:18:49.511369+0000","last_clean":"2024-01-16T15:18:49.511369+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:18:49.511369+0000","last_undegraded":"2024-01-16T15:18:49.511369+0000","last_fullsized":"2024-01-16T15:18:49.511369+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505254+0000","last_change":"2024-01-16T15:18:49.505254+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505254+0000","last_undegraded":"2024-01-16T15:18:49.505254+0000","last_fullsized":"2024-01-16T15:18:49.505254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503470+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:18:50.503470+0000","last_peered":"2024-01-16T15:18:50.503470+0000","last_clean":"2024-01-16T15:18:50.503470+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:18:50.503470+0000","last_undegraded":"2024-01-16T15:18:50.503470+0000","last_fullsized":"2024-01-16T15:18:50.503470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.508797+0000","last_change":"2024-01-16T15:18:49.508797+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.508797+0000","last_undegraded":"2024-01-16T15:18:49.508797+0000","last_fullsized":"2024-01-16T15:18:49.508797+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.525562+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:18:49.525562+0000","last_peered":"2024-01-16T15:18:49.525562+0000","last_clean":"2024-01-16T15:18:49.525562+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:18:49.525562+0000","last_undegraded":"2024-01-16T15:18:49.525562+0000","last_fullsized":"2024-01-16T15:18:49.525562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:10.112260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500474+0000","last_change":"2024-01-16T15:18:49.500474+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500474+0000","last_undegraded":"2024-01-16T15:18:49.500474+0000","last_fullsized":"2024-01-16T15:18:49.500474+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.515357+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:18:49.515357+0000","last_peered":"2024-01-16T15:18:49.515357+0000","last_clean":"2024-01-16T15:18:49.515357+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:18:49.515357+0000","last_undegraded":"2024-01-16T15:18:49.515357+0000","last_fullsized":"2024-01-16T15:18:49.515357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:56.122268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500974+0000","last_change":"2024-01-16T15:18:49.500974+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500974+0000","last_undegraded":"2024-01-16T15:18:49.500974+0000","last_fullsized":"2024-01-16T15:18:49.500974+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.536241+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:18:50.536241+0000","last_peered":"2024-01-16T15:18:50.536241+0000","last_clean":"2024-01-16T15:18:50.536241+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:18:50.536241+0000","last_undegraded":"2024-01-16T15:18:50.536241+0000","last_fullsized":"2024-01-16T15:18:50.536241+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.503096+0000","last_change":"2024-01-16T15:18:49.503096+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.503096+0000","last_undegraded":"2024-01-16T15:18:49.503096+0000","last_fullsized":"2024-01-16T15:18:49.503096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.538150+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:18:50.538150+0000","last_peered":"2024-01-16T15:18:50.538150+0000","last_clean":"2024-01-16T15:18:50.538150+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:18:50.538150+0000","last_undegraded":"2024-01-16T15:18:50.538150+0000","last_fullsized":"2024-01-16T15:18:50.538150+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498243+0000","last_change":"2024-01-16T15:18:49.498243+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498243+0000","last_undegraded":"2024-01-16T15:18:49.498243+0000","last_fullsized":"2024-01-16T15:18:49.498243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520668+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:18:49.520668+0000","last_peered":"2024-01-16T15:18:49.520668+0000","last_clean":"2024-01-16T15:18:49.520668+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:18:49.520668+0000","last_undegraded":"2024-01-16T15:18:49.520668+0000","last_fullsized":"2024-01-16T15:18:49.520668+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499199+0000","last_change":"2024-01-16T15:18:49.499199+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499199+0000","last_undegraded":"2024-01-16T15:18:49.499199+0000","last_fullsized":"2024-01-16T15:18:49.499199+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516723+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:18:49.516723+0000","last_peered":"2024-01-16T15:18:49.516723+0000","last_clean":"2024-01-16T15:18:49.516723+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:18:49.516723+0000","last_undegraded":"2024-01-16T15:18:49.516723+0000","last_fullsized":"2024-01-16T15:18:49.516723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:21.691380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499391+0000","last_change":"2024-01-16T15:18:49.499391+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499391+0000","last_undegraded":"2024-01-16T15:18:49.499391+0000","last_fullsized":"2024-01-16T15:18:49.499391+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511289+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:18:49.511289+0000","last_peered":"2024-01-16T15:18:49.511289+0000","last_clean":"2024-01-16T15:18:49.511289+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:18:49.511289+0000","last_undegraded":"2024-01-16T15:18:49.511289+0000","last_fullsized":"2024-01-16T15:18:49.511289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:29.197614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495765+0000","last_change":"2024-01-16T15:18:49.495765+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495765+0000","last_undegraded":"2024-01-16T15:18:49.495765+0000","last_fullsized":"2024-01-16T15:18:49.495765+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533470+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:18:50.533470+0000","last_peered":"2024-01-16T15:18:50.533470+0000","last_clean":"2024-01-16T15:18:50.533470+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:18:50.533470+0000","last_undegraded":"2024-01-16T15:18:50.533470+0000","last_fullsized":"2024-01-16T15:18:50.533470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.516561+0000","last_change":"2024-01-16T15:18:49.516561+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.516561+0000","last_undegraded":"2024-01-16T15:18:49.516561+0000","last_fullsized":"2024-01-16T15:18:49.516561+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511184+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:18:49.511184+0000","last_peered":"2024-01-16T15:18:49.511184+0000","last_clean":"2024-01-16T15:18:49.511184+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:18:49.511184+0000","last_undegraded":"2024-01-16T15:18:49.511184+0000","last_fullsized":"2024-01-16T15:18:49.511184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:53.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.510804+0000","last_change":"2024-01-16T15:18:49.510804+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.510804+0000","last_undegraded":"2024-01-16T15:18:49.510804+0000","last_fullsized":"2024-01-16T15:18:49.510804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520479+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:18:49.520479+0000","last_peered":"2024-01-16T15:18:49.520479+0000","last_clean":"2024-01-16T15:18:49.520479+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:18:49.520479+0000","last_undegraded":"2024-01-16T15:18:49.520479+0000","last_fullsized":"2024-01-16T15:18:49.520479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:53.321942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507278+0000","last_change":"2024-01-16T15:18:49.507278+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507278+0000","last_undegraded":"2024-01-16T15:18:49.507278+0000","last_fullsized":"2024-01-16T15:18:49.507278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520339+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:18:49.520339+0000","last_peered":"2024-01-16T15:18:49.520339+0000","last_clean":"2024-01-16T15:18:49.520339+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:18:49.520339+0000","last_undegraded":"2024-01-16T15:18:49.520339+0000","last_fullsized":"2024-01-16T15:18:49.520339+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.504828+0000","last_change":"2024-01-16T15:18:49.504828+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.504828+0000","last_undegraded":"2024-01-16T15:18:49.504828+0000","last_fullsized":"2024-01-16T15:18:49.504828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523509+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:18:49.523509+0000","last_peered":"2024-01-16T15:18:49.523509+0000","last_clean":"2024-01-16T15:18:49.523509+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:18:49.523509+0000","last_undegraded":"2024-01-16T15:18:49.523509+0000","last_fullsized":"2024-01-16T15:18:49.523509+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518601+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:18:50.518601+0000","last_peered":"2024-01-16T15:18:50.518601+0000","last_clean":"2024-01-16T15:18:50.518601+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:18:50.518601+0000","last_undegraded":"2024-01-16T15:18:50.518601+0000","last_fullsized":"2024-01-16T15:18:50.518601+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:27.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522193+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:18:49.522193+0000","last_peered":"2024-01-16T15:18:49.522193+0000","last_clean":"2024-01-16T15:18:49.522193+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:18:49.522193+0000","last_undegraded":"2024-01-16T15:18:49.522193+0000","last_fullsized":"2024-01-16T15:18:49.522193+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:54.524825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.506081+0000","last_change":"2024-01-16T15:18:49.506081+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.506081+0000","last_undegraded":"2024-01-16T15:18:49.506081+0000","last_fullsized":"2024-01-16T15:18:49.506081+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523683+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:18:49.523683+0000","last_peered":"2024-01-16T15:18:49.523683+0000","last_clean":"2024-01-16T15:18:49.523683+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:18:49.523683+0000","last_undegraded":"2024-01-16T15:18:49.523683+0000","last_fullsized":"2024-01-16T15:18:49.523683+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498896+0000","last_change":"2024-01-16T15:18:49.498896+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498896+0000","last_undegraded":"2024-01-16T15:18:49.498896+0000","last_fullsized":"2024-01-16T15:18:49.498896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503708+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:18:50.503708+0000","last_peered":"2024-01-16T15:18:50.503708+0000","last_clean":"2024-01-16T15:18:50.503708+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:18:50.503708+0000","last_undegraded":"2024-01-16T15:18:50.503708+0000","last_fullsized":"2024-01-16T15:18:50.503708+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500913+0000","last_change":"2024-01-16T15:18:49.500913+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500913+0000","last_undegraded":"2024-01-16T15:18:49.500913+0000","last_fullsized":"2024-01-16T15:18:49.500913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522640+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:18:49.522640+0000","last_peered":"2024-01-16T15:18:49.522640+0000","last_clean":"2024-01-16T15:18:49.522640+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:18:49.522640+0000","last_undegraded":"2024-01-16T15:18:49.522640+0000","last_fullsized":"2024-01-16T15:18:49.522640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:48.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.526632+0000","last_change":"2024-01-16T15:18:50.522199+0000","last_active":"2024-01-16T15:18:50.526632+0000","last_peered":"2024-01-16T15:18:50.526632+0000","last_clean":"2024-01-16T15:18:50.526632+0000","last_became_active":"2024-01-16T15:18:50.521902+0000","last_became_peered":"2024-01-16T15:18:50.521902+0000","last_unstale":"2024-01-16T15:18:50.526632+0000","last_undegraded":"2024-01-16T15:18:50.526632+0000","last_fullsized":"2024-01-16T15:18:50.526632+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:11.568326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516769+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:18:49.516769+0000","last_peered":"2024-01-16T15:18:49.516769+0000","last_clean":"2024-01-16T15:18:49.516769+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:18:49.516769+0000","last_undegraded":"2024-01-16T15:18:49.516769+0000","last_fullsized":"2024-01-16T15:18:49.516769+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:17.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518602+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:18:50.518602+0000","last_peered":"2024-01-16T15:18:50.518602+0000","last_clean":"2024-01-16T15:18:50.518602+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:18:50.518602+0000","last_undegraded":"2024-01-16T15:18:50.518602+0000","last_fullsized":"2024-01-16T15:18:50.518602+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:38.334522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.505052+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:18:50.505052+0000","last_peered":"2024-01-16T15:18:50.505052+0000","last_clean":"2024-01-16T15:18:50.505052+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:18:50.505052+0000","last_undegraded":"2024-01-16T15:18:50.505052+0000","last_fullsized":"2024-01-16T15:18:50.505052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500292+0000","last_change":"2024-01-16T15:18:49.500292+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500292+0000","last_undegraded":"2024-01-16T15:18:49.500292+0000","last_fullsized":"2024-01-16T15:18:49.500292+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520661+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:18:49.520661+0000","last_peered":"2024-01-16T15:18:49.520661+0000","last_clean":"2024-01-16T15:18:49.520661+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:18:49.520661+0000","last_undegraded":"2024-01-16T15:18:49.520661+0000","last_fullsized":"2024-01-16T15:18:49.520661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:24.291444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497200+0000","last_change":"2024-01-16T15:18:49.497200+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497200+0000","last_undegraded":"2024-01-16T15:18:49.497200+0000","last_fullsized":"2024-01-16T15:18:49.497200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.510956+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:18:49.510956+0000","last_peered":"2024-01-16T15:18:49.510956+0000","last_clean":"2024-01-16T15:18:49.510956+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:18:49.510956+0000","last_undegraded":"2024-01-16T15:18:49.510956+0000","last_fullsized":"2024-01-16T15:18:49.510956+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496009+0000","last_change":"2024-01-16T15:18:49.496009+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496009+0000","last_undegraded":"2024-01-16T15:18:49.496009+0000","last_fullsized":"2024-01-16T15:18:49.496009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524006+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:18:49.524006+0000","last_peered":"2024-01-16T15:18:49.524006+0000","last_clean":"2024-01-16T15:18:49.524006+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:18:49.524006+0000","last_undegraded":"2024-01-16T15:18:49.524006+0000","last_fullsized":"2024-01-16T15:18:49.524006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505239+0000","last_change":"2024-01-16T15:18:49.505239+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505239+0000","last_undegraded":"2024-01-16T15:18:49.505239+0000","last_fullsized":"2024-01-16T15:18:49.505239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.519663+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:18:49.519663+0000","last_peered":"2024-01-16T15:18:49.519663+0000","last_clean":"2024-01-16T15:18:49.519663+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:18:49.519663+0000","last_undegraded":"2024-01-16T15:18:49.519663+0000","last_fullsized":"2024-01-16T15:18:49.519663+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:27.624738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520862+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:18:50.520862+0000","last_peered":"2024-01-16T15:18:50.520862+0000","last_clean":"2024-01-16T15:18:50.520862+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:18:50.520862+0000","last_undegraded":"2024-01-16T15:18:50.520862+0000","last_fullsized":"2024-01-16T15:18:50.520862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:09.166757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503918+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:18:50.503918+0000","last_peered":"2024-01-16T15:18:50.503918+0000","last_clean":"2024-01-16T15:18:50.503918+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:18:50.503918+0000","last_undegraded":"2024-01-16T15:18:50.503918+0000","last_fullsized":"2024-01-16T15:18:50.503918+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498189+0000","last_change":"2024-01-16T15:18:49.498189+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498189+0000","last_undegraded":"2024-01-16T15:18:49.498189+0000","last_fullsized":"2024-01-16T15:18:49.498189+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524346+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:18:49.524346+0000","last_peered":"2024-01-16T15:18:49.524346+0000","last_clean":"2024-01-16T15:18:49.524346+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:18:49.524346+0000","last_undegraded":"2024-01-16T15:18:49.524346+0000","last_fullsized":"2024-01-16T15:18:49.524346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:18:48.485739+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:18:48.485739+0000","last_peered":"2024-01-16T15:18:48.485739+0000","last_clean":"2024-01-16T15:18:48.485739+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:18:48.485739+0000","last_undegraded":"2024-01-16T15:18:48.485739+0000","last_fullsized":"2024-01-16T15:18:48.485739+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542381,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1970176,"data_stored":1183817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.349,"15min":1.5429999999999999},"min":{"1min":0.72099999999999997,"5min":0.63500000000000001,"15min":0.28199999999999997},"max":{"1min":1.768,"5min":5.2690000000000001,"15min":27.469000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4410000000000001,"15min":1.6859999999999999},"min":{"1min":0.86099999999999999,"5min":0.79500000000000004,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":6.5599999999999996,"15min":25.503},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.139,"15min":1.2689999999999999},"min":{"1min":0.77600000000000002,"5min":0.55100000000000005,"15min":0.34499999999999997},"max":{"1min":1.712,"5min":1.742,"15min":6.718},"last":1.343},{"interface":"front","average":{"1min":1.262,"5min":1.2070000000000001,"15min":1.3520000000000001},"min":{"1min":0.85499999999999998,"5min":0.75700000000000001,"15min":0.373},"max":{"1min":1.7669999999999999,"5min":2.0110000000000001,"15min":15.452},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.4530000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.48299999999999998},"max":{"1min":1.7709999999999999,"5min":11.536,"15min":11.536},"last":1.25},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5409999999999999,"15min":1.5629999999999999},"min":{"1min":1.0720000000000001,"5min":0.79800000000000004,"15min":0.504},"max":{"1min":1.978,"5min":11.576000000000001,"15min":25.478000000000002},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.222,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.67000000000000004,"15min":0.40899999999999997},"max":{"1min":1.593,"5min":2.133,"15min":13.907999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.357,"5min":1.365,"15min":1.5029999999999999},"min":{"1min":1.022,"5min":0.89300000000000002,"15min":0.62},"max":{"1min":1.8580000000000001,"5min":2.1960000000000002,"15min":8.6999999999999993},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.452},"min":{"1min":0.89200000000000002,"5min":0.85299999999999998,"15min":0.376},"max":{"1min":1.7829999999999999,"5min":6.7599999999999998,"15min":25.401},"last":1.171},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3280000000000001,"15min":1.472},"min":{"1min":0.84999999999999998,"5min":0.59599999999999997,"15min":0.45000000000000001},"max":{"1min":2.069,"5min":2.069,"15min":10.134},"last":1.5489999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":0.96399999999999997,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.7310000000000001,"5min":2.1259999999999999,"15min":12.528},"last":1.486},{"interface":"front","average":{"1min":1.498,"5min":1.544,"15min":1.639},"min":{"1min":1.0640000000000001,"5min":0.86599999999999999,"15min":0.41399999999999998},"max":{"1min":1.9159999999999999,"5min":6.8719999999999999,"15min":26.327000000000002},"last":1.302}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.6759999999999999,"15min":1.734},"min":{"1min":1.3640000000000001,"5min":1.087,"15min":0.45200000000000001},"max":{"1min":2.0870000000000002,"5min":6.9690000000000003,"15min":25.451000000000001},"last":1.462},{"interface":"front","average":{"1min":1.524,"5min":1.4930000000000001,"15min":1.605},"min":{"1min":0.92400000000000004,"5min":0.89800000000000002,"15min":0.44700000000000001},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":12.577999999999999},"last":1.6120000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72716,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299124,"statfs":{"total":96636764160,"available":96562302976,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.452,"15min":1.744},"min":{"1min":0.752,"5min":0.71299999999999997,"15min":0.35399999999999998},"max":{"1min":1.53,"5min":29.77,"15min":30.164999999999999},"last":1.53},{"interface":"front","average":{"1min":1.292,"5min":1.6870000000000001,"15min":1.8779999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.48899999999999999},"max":{"1min":1.5780000000000001,"5min":36.405999999999999,"15min":36.405999999999999},"last":1.49}]},{"osd":1,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1850000000000001,"15min":1.508},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.47899999999999998},"max":{"1min":1.52,"5min":5.2859999999999996,"15min":28.122},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.2549999999999999,"15min":1.298},"min":{"1min":0.78200000000000003,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":1.548,"5min":10.494,"15min":10.494},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.347,"15min":1.734},"min":{"1min":1,"5min":0.76900000000000002,"15min":0.501},"max":{"1min":1.845,"5min":5.0549999999999997,"15min":26.109000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.51,"15min":1.893},"min":{"1min":1.0329999999999999,"5min":1.016,"15min":0.67100000000000004},"max":{"1min":2.0449999999999999,"5min":5.1459999999999999,"15min":26.161000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3029999999999999,"15min":1.615},"min":{"1min":1.1060000000000001,"5min":0.68500000000000005,"15min":0.44700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8079999999999998,"15min":17.75},"last":1.363},{"interface":"front","average":{"1min":1.395,"5min":1.681,"15min":1.728},"min":{"1min":1.0349999999999999,"5min":0.77100000000000002,"15min":0.55700000000000005},"max":{"1min":2.1789999999999998,"5min":21.456,"15min":21.456},"last":1.5600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3540000000000001,"15min":1.5069999999999999},"min":{"1min":0.91400000000000003,"5min":0.82099999999999995,"15min":0.45000000000000001},"max":{"1min":1.6699999999999999,"5min":5.3310000000000004,"15min":15.571},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.325,"15min":1.5469999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.42299999999999999},"max":{"1min":1.639,"5min":3.6160000000000001,"15min":23.667999999999999},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4350000000000001,"15min":1.4890000000000001},"min":{"1min":0.83499999999999996,"5min":0.73699999999999999,"15min":0.42299999999999999},"max":{"1min":1.982,"5min":13.949999999999999,"15min":23.137},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.6100000000000001,"15min":1.6970000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":0.47099999999999997},"max":{"1min":1.714,"5min":11.298,"15min":25.684000000000001},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.6020000000000001,"15min":1.552},"min":{"1min":0.754,"5min":0.71499999999999997,"15min":0.47099999999999997},"max":{"1min":2.1000000000000001,"5min":14.037000000000001,"15min":14.037000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6240000000000001,"15min":1.724},"min":{"1min":0.95899999999999996,"5min":0.92300000000000004,"15min":0.499},"max":{"1min":2.1659999999999999,"5min":4.9829999999999997,"15min":11.788},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.075,"15min":1.571},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.39000000000000001},"max":{"1min":1.5649999999999999,"5min":1.651,"15min":26.228999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.135,"15min":1.831},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.34799999999999998},"max":{"1min":1.7070000000000001,"5min":1.849,"15min":56.011000000000003},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1419999999999999,"15min":1.363},"min":{"1min":0.79900000000000004,"5min":0.65200000000000002,"15min":0.40999999999999998},"max":{"1min":1.6419999999999999,"5min":2.2450000000000001,"15min":16.687999999999999},"last":1.034},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2430000000000001,"15min":1.8600000000000001},"min":{"1min":0.88900000000000001,"5min":0.745,"15min":0.40999999999999998},"max":{"1min":1.667,"5min":4.8499999999999996,"15min":67.930000000000007},"last":1.125}]},{"osd":2,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1779999999999999,"15min":1.52},"min":{"1min":0.85199999999999998,"5min":0.61099999999999999,"15min":0.47099999999999997},"max":{"1min":1.55,"5min":1.7529999999999999,"15min":13.193},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.246,"5min":1.258,"15min":1.4850000000000001},"min":{"1min":0.93700000000000006,"5min":0.76300000000000001,"15min":0.45800000000000002},"max":{"1min":1.7769999999999999,"5min":3.9590000000000001,"15min":16.721},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.379,"15min":1.712},"min":{"1min":1.0509999999999999,"5min":0.81100000000000005,"15min":0.57999999999999996},"max":{"1min":1.8680000000000001,"5min":4.8010000000000002,"15min":22.382000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.3819999999999999,"15min":1.7529999999999999},"min":{"1min":0.94199999999999995,"5min":0.72399999999999998,"15min":0.51200000000000001},"max":{"1min":1.921,"5min":1.921,"15min":14.685},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2649999999999999,"15min":1.419},"min":{"1min":0.95199999999999996,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.8680000000000001,"5min":2.2549999999999999,"15min":16.382999999999999},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.377,"15min":1.5269999999999999},"min":{"1min":0.98899999999999999,"5min":0.502,"15min":0.379},"max":{"1min":1.8129999999999999,"5min":4.6580000000000004,"15min":27.006},"last":1.6539999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3600000000000001,"15min":1.5600000000000001},"min":{"1min":0.83999999999999997,"5min":0.52400000000000002,"15min":0.39700000000000002},"max":{"1min":1.974,"5min":1.9770000000000001,"15min":13.263999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.4890000000000001,"15min":1.609},"min":{"1min":1.304,"5min":0.629,"15min":0.505},"max":{"1min":1.9670000000000001,"5min":2.1640000000000001,"15min":9.5269999999999992},"last":1.704}]},{"osd":7,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.52,"15min":1.7470000000000001},"min":{"1min":1.2410000000000001,"5min":0.59099999999999997,"15min":0.45000000000000001},"max":{"1min":1.9119999999999999,"5min":4.742,"15min":35.127000000000002},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5489999999999999,"15min":1.7290000000000001},"min":{"1min":0.91200000000000003,"5min":0.61099999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":13.491},"last":1.601}]}]},{"osd":4,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1619999999999999,"15min":1.244},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6180000000000001,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.21},{"interface":"front","average":{"1min":1.095,"5min":1.123,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.679,"5min":1.958,"15min":60.106000000000002},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.629,"15min":1.79},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51100000000000001},"max":{"1min":1.738,"5min":24.399999999999999,"15min":24.399999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.611,"15min":1.627},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57099999999999995},"max":{"1min":1.7290000000000001,"5min":24.456,"15min":24.456},"last":1.7090000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.3029999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.52800000000000002},"max":{"1min":1.6299999999999999,"5min":2.1699999999999999,"15min":9.7140000000000004},"last":1.145},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2889999999999999,"15min":1.6970000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.52400000000000002},"max":{"1min":1.6479999999999999,"5min":1.855,"15min":35.024999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3300000000000001,"15min":1.556},"min":{"1min":0.503,"5min":0.503,"15min":0.47799999999999998},"max":{"1min":1.6499999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":6.4589999999999996,"15min":7.8140000000000001},"last":1.375}]},{"osd":5,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.454},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":2.1299999999999999,"15min":14.568},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4690000000000001,"15min":1.4319999999999999},"min":{"1min":0.74299999999999999,"5min":0.496,"15min":0.34399999999999997},"max":{"1min":1.8160000000000001,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":2.0720000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.436},"max":{"1min":1.696,"5min":3.0190000000000001,"15min":9.6630000000000003},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4239999999999999,"15min":1.6299999999999999},"min":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.45100000000000001},"max":{"1min":1.675,"5min":2.3980000000000001,"15min":14.317},"last":2.0129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5640000000000001,"15min":1.5449999999999999},"min":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.52100000000000002},"max":{"1min":1.927,"5min":6.5250000000000004,"15min":9.2799999999999994},"last":1.903},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.552,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.443},"max":{"1min":1.8540000000000001,"5min":3.3439999999999999,"15min":9.2270000000000003},"last":2.2229999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82432,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289408,"statfs":{"total":96636764160,"available":96552353792,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.93899999999999995,"15min":1.429},"min":{"1min":0.748,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.45,"5min":1.9299999999999999,"15min":50.988999999999997},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.087,"15min":1.4750000000000001},"min":{"1min":0.80100000000000005,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5840000000000001,"5min":1.786,"15min":20.274000000000001},"last":1.141}]},{"osd":1,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.101,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.371,"15min":0.32100000000000001},"max":{"1min":1.613,"5min":3.6640000000000001,"15min":18.192},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.21,"15min":1.712},"min":{"1min":0.89000000000000001,"5min":0.39400000000000002,"15min":0.313},"max":{"1min":1.8440000000000001,"5min":1.9239999999999999,"15min":20.385999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3089999999999999,"15min":1.7250000000000001},"min":{"1min":0.871,"5min":0.45500000000000002,"15min":0.36699999999999999},"max":{"1min":1.978,"5min":3.5249999999999999,"15min":42.840000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.397,"15min":1.806},"min":{"1min":1.016,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.631,"5min":3.702,"15min":38.707000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.339,"15min":1.762},"min":{"1min":0.99199999999999999,"5min":0.78200000000000003,"15min":0.44500000000000001},"max":{"1min":1.734,"5min":2.327,"15min":18.952000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4650000000000001,"15min":1.79},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.54800000000000004},"max":{"1min":1.718,"5min":3.9529999999999998,"15min":18.088999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4890000000000001,"15min":1.8160000000000001},"min":{"1min":1.212,"5min":0.73199999999999998,"15min":0.55500000000000005},"max":{"1min":1.7729999999999999,"5min":2.661,"15min":24.349},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.464,"15min":1.9199999999999999},"min":{"1min":1.105,"5min":0.753,"15min":0.61899999999999999},"max":{"1min":1.9119999999999999,"5min":3.4239999999999999,"15min":14.192},"last":1.306}]},{"osd":6,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.621,"15min":1.877},"min":{"1min":1.159,"5min":0.77200000000000002,"15min":0.52200000000000002},"max":{"1min":1.7929999999999999,"5min":3.9009999999999998,"15min":22.687000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.623,"15min":1.931},"min":{"1min":1.262,"5min":0.73799999999999999,"15min":0.59699999999999998},"max":{"1min":1.7969999999999999,"5min":3.9300000000000002,"15min":24.058},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6040000000000001,"15min":2.1000000000000001},"min":{"1min":1.111,"5min":0.85099999999999998,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":6.3209999999999997,"15min":35.570999999999998},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.6659999999999999,"15min":2.0619999999999998},"min":{"1min":1.2969999999999999,"5min":0.81100000000000005,"15min":0.58899999999999997},"max":{"1min":2.0750000000000002,"5min":3.8279999999999998,"15min":37.607999999999997},"last":1.633}]}]},{"osd":2,"up_from":14,"seq":60129542380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94924,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94276916,"statfs":{"total":96636764160,"available":96539561984,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90400000000000003,"15min":1.4430000000000001},"min":{"1min":0.67700000000000005,"5min":0.36099999999999999,"15min":0.24399999999999999},"max":{"1min":1.135,"5min":1.96,"15min":64.387},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.022,"15min":1.387},"min":{"1min":0.76500000000000001,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7030000000000001,"5min":1.9890000000000001,"15min":24.308},"last":1.7490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0580000000000001,"15min":1.466},"min":{"1min":0.752,"5min":0.46300000000000002,"15min":0.34699999999999998},"max":{"1min":1.2490000000000001,"5min":2.093,"15min":27.263000000000002},"last":1.27},{"interface":"front","average":{"1min":1.381,"5min":1.3879999999999999,"15min":1.8520000000000001},"min":{"1min":1.0940000000000001,"5min":0.51300000000000001,"15min":0.38500000000000001},"max":{"1min":1.8620000000000001,"5min":2.1539999999999999,"15min":23.602},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.385,"15min":1.627},"min":{"1min":1.1499999999999999,"5min":0.58499999999999996,"15min":0.40100000000000002},"max":{"1min":1.9299999999999999,"5min":2.085,"15min":27.311},"last":1.081},{"interface":"front","average":{"1min":1.296,"5min":1.327,"15min":1.6299999999999999},"min":{"1min":0.96599999999999997,"5min":0.498,"15min":0.47399999999999998},"max":{"1min":1.698,"5min":1.998,"15min":26.555},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.389,"15min":1.637},"min":{"1min":1.0960000000000001,"5min":0.66100000000000003,"15min":0.38900000000000001},"max":{"1min":1.764,"5min":2.1040000000000001,"15min":26.503},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.3819999999999999,"15min":1.7490000000000001},"min":{"1min":1.0009999999999999,"5min":0.56399999999999995,"15min":0.51200000000000001},"max":{"1min":1.9079999999999999,"5min":2.4319999999999999,"15min":25.661000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.4610000000000001,"15min":1.74},"min":{"1min":1.3120000000000001,"5min":0.74199999999999999,"15min":0.45800000000000002},"max":{"1min":2.077,"5min":2.3490000000000002,"15min":27.149000000000001},"last":1.49},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.538,"15min":1.7330000000000001},"min":{"1min":1.1930000000000001,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":1.8540000000000001,"5min":2.6589999999999998,"15min":26.349},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4910000000000001,"15min":1.9610000000000001},"min":{"1min":1.0700000000000001,"5min":0.64500000000000002,"15min":0.504},"max":{"1min":2.1909999999999998,"5min":2.7149999999999999,"15min":25.603999999999999},"last":1.548},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.597,"15min":1.843},"min":{"1min":1.1579999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":2.157,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.591,"15min":1.9470000000000001},"min":{"1min":1.343,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1200000000000001,"5min":2.4889999999999999,"15min":24.509},"last":1.649},{"interface":"front","average":{"1min":1.657,"5min":1.5880000000000001,"15min":1.927},"min":{"1min":1.282,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.2229999999999999,"5min":2.3479999999999999,"15min":27.207999999999998},"last":1.8080000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.92400000000000004,"15min":1.153},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.28399999999999997},"max":{"1min":1.696,"5min":1.865,"15min":11.699},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0489999999999999,"15min":1.276},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.33700000000000002},"max":{"1min":1.5089999999999999,"5min":2.3860000000000001,"15min":13.930999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.238,"15min":1.385},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.31},"max":{"1min":1.8,"5min":15.557,"15min":16.488},"last":1.274},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1559999999999999,"15min":1.5680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39000000000000001},"max":{"1min":1.968,"5min":2.306,"15min":22.550999999999998},"last":1.4650000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.6879999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.439},"max":{"1min":1.7450000000000001,"5min":1.9790000000000001,"15min":22.471},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3660000000000001,"15min":1.6359999999999999},"min":{"1min":0.54100000000000004,"5min":0.49099999999999999,"15min":0.35699999999999998},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":22.550999999999998},"last":0.82899999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4870000000000001,"15min":1.6240000000000001},"min":{"1min":0.96699999999999997,"5min":0.95699999999999996,"15min":0.60899999999999999},"max":{"1min":1.95,"5min":2.7050000000000001,"15min":8.3620000000000001},"last":1.333},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5960000000000001,"15min":1.7070000000000001},"min":{"1min":0.97399999999999998,"5min":0.69999999999999996,"15min":0.433},"max":{"1min":1.97,"5min":15.723000000000001,"15min":19.236999999999998},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.706},"min":{"1min":0.92800000000000005,"5min":0.73799999999999999,"15min":0.57199999999999995},"max":{"1min":1.843,"5min":2.2040000000000002,"15min":21.117000000000001},"last":1.175},{"interface":"front","average":{"1min":1.425,"5min":1.544,"15min":1.758},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.65800000000000003},"max":{"1min":1.8009999999999999,"5min":15.785,"15min":17.138000000000002},"last":1.7569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.4850000000000001,"15min":1.698},"min":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.72199999999999998},"max":{"1min":2.2759999999999998,"5min":2.298,"15min":11.901999999999999},"last":1.798},{"interface":"front","average":{"1min":1.629,"5min":1.5840000000000001,"15min":1.8069999999999999},"min":{"1min":1.044,"5min":0.86399999999999999,"15min":0.63300000000000001},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":23.420999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.631,"15min":1.8380000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.085,"5min":15.875,"15min":17.010000000000002},"last":1.423},{"interface":"front","average":{"1min":1.591,"5min":1.573,"15min":1.825},"min":{"1min":0.998,"5min":0.91600000000000004,"15min":0.75700000000000001},"max":{"1min":2.508,"5min":2.508,"15min":11.962},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542381,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73308,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298532,"statfs":{"total":96636764160,"available":96561696768,"internally_reserved":0,"allocated":1982464,"data_stored":1190620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":73071041},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.016,"15min":1.468},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":1.355,"5min":4.8380000000000001,"15min":43.459000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.2290000000000001,"15min":1.706},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.956,"5min":4.9779999999999998,"15min":43.627000000000002},"last":1.042}]},{"osd":2,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1919999999999999,"15min":1.4099999999999999},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":5.0570000000000004,"15min":18.861000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.218,"15min":1.5229999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":2.2330000000000001,"5min":5.3550000000000004,"15min":38.893000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0800000000000001,"15min":1.47},"min":{"1min":0.37,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":1.4410000000000001,"5min":4.9050000000000002,"15min":23.687999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.357,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.434},"max":{"1min":1.6080000000000001,"5min":5.1980000000000004,"15min":18.966999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3460000000000001,"15min":1.5129999999999999},"min":{"1min":0.76500000000000001,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.8420000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.496,"15min":1.7030000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.873,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.423}]},{"osd":5,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.508,"15min":1.891},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.792,"5min":5.1589999999999998,"15min":24.611000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.6970000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.56000000000000005},"max":{"1min":1.907,"5min":7.2699999999999996,"15min":20.199000000000002},"last":1.75}]},{"osd":6,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6499999999999999,"15min":2.0510000000000002},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.55900000000000005},"max":{"1min":2.3119999999999998,"5min":6.3339999999999996,"15min":26.998000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.6140000000000001,"15min":1.988},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":2.1669999999999998,"5min":5.9690000000000003,"15min":24.478999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6699999999999999,"15min":1.9139999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.60899999999999999},"max":{"1min":2.2599999999999998,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.657},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.5629999999999999,"15min":1.7310000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.52800000000000002},"max":{"1min":1.8999999999999999,"5min":6.149,"15min":20.279},"last":1.8280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.496455+0000', 'last_change': '2024-01-16T15:18:49.496455+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.496455+0000', 'last_undegraded': '2024-01-16T15:18:49.496455+0000', 'last_fullsized': '2024-01-16T15:18:49.496455+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.511018+0000', 'last_change': '2024-01-16T15:18:49.511018+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.511018+0000', 'last_undegraded': '2024-01-16T15:18:49.511018+0000', 'last_fullsized': '2024-01-16T15:18:49.511018+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.494552+0000', 'last_change': '2024-01-16T15:18:49.494552+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.494552+0000', 'last_undegraded': '2024-01-16T15:18:49.494552+0000', 'last_fullsized': '2024-01-16T15:18:49.494552+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3a is not active+clean 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.511599+0000', 'last_change': '2024-01-16T15:18:49.511599+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.511599+0000', 'last_undegraded': '2024-01-16T15:18:49.511599+0000', 'last_fullsized': '2024-01-16T15:18:49.511599+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-01-16T15:18:55.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.502722+0000', 'last_change': '2024-01-16T15:18:49.502722+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.502722+0000', 'last_undegraded': '2024-01-16T15:18:49.502722+0000', 'last_fullsized': '2024-01-16T15:18:49.502722+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.507251+0000', 'last_change': '2024-01-16T15:18:49.507251+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.507251+0000', 'last_undegraded': '2024-01-16T15:18:49.507251+0000', 'last_fullsized': '2024-01-16T15:18:49.507251+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.496236+0000', 'last_change': '2024-01-16T15:18:49.496236+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.496236+0000', 'last_undegraded': '2024-01-16T15:18:49.496236+0000', 'last_fullsized': '2024-01-16T15:18:49.496236+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.509738+0000', 'last_change': '2024-01-16T15:18:49.509738+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.509738+0000', 'last_undegraded': '2024-01-16T15:18:49.509738+0000', 'last_fullsized': '2024-01-16T15:18:49.509738+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.505304+0000', 'last_change': '2024-01-16T15:18:49.505304+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.505304+0000', 'last_undegraded': '2024-01-16T15:18:49.505304+0000', 'last_fullsized': '2024-01-16T15:18:49.505304+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.31 is not active+clean 2024-01-16T15:18:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.509868+0000', 'last_change': '2024-01-16T15:18:49.509868+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.509868+0000', 'last_undegraded': '2024-01-16T15:18:49.509868+0000', 'last_fullsized': '2024-01-16T15:18:49.509868+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.495707+0000', 'last_change': '2024-01-16T15:18:49.495707+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.495707+0000', 'last_undegraded': '2024-01-16T15:18:49.495707+0000', 'last_fullsized': '2024-01-16T15:18:49.495707+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2e is not active+clean 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.501685+0000', 'last_change': '2024-01-16T15:18:49.501685+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.501685+0000', 'last_undegraded': '2024-01-16T15:18:49.501685+0000', 'last_fullsized': '2024-01-16T15:18:49.501685+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.509552+0000', 'last_change': '2024-01-16T15:18:49.509552+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.509552+0000', 'last_undegraded': '2024-01-16T15:18:49.509552+0000', 'last_fullsized': '2024-01-16T15:18:49.509552+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.494776+0000', 'last_change': '2024-01-16T15:18:49.494776+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.494776+0000', 'last_undegraded': '2024-01-16T15:18:49.494776+0000', 'last_fullsized': '2024-01-16T15:18:49.494776+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.494903+0000', 'last_change': '2024-01-16T15:18:49.494903+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.494903+0000', 'last_undegraded': '2024-01-16T15:18:49.494903+0000', 'last_fullsized': '2024-01-16T15:18:49.494903+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2a is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498780+0000', 'last_change': '2024-01-16T15:18:49.498780+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498780+0000', 'last_undegraded': '2024-01-16T15:18:49.498780+0000', 'last_fullsized': '2024-01-16T15:18:49.498780+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.494905+0000', 'last_change': '2024-01-16T15:18:49.494905+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.494905+0000', 'last_undegraded': '2024-01-16T15:18:49.494905+0000', 'last_fullsized': '2024-01-16T15:18:49.494905+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.28 is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498751+0000', 'last_change': '2024-01-16T15:18:49.498751+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498751+0000', 'last_undegraded': '2024-01-16T15:18:49.498751+0000', 'last_fullsized': '2024-01-16T15:18:49.498751+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.495191+0000', 'last_change': '2024-01-16T15:18:49.495191+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.495191+0000', 'last_undegraded': '2024-01-16T15:18:49.495191+0000', 'last_fullsized': '2024-01-16T15:18:49.495191+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.25 is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.511948+0000', 'last_change': '2024-01-16T15:18:49.511948+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.511948+0000', 'last_undegraded': '2024-01-16T15:18:49.511948+0000', 'last_fullsized': '2024-01-16T15:18:49.511948+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.24 is not active+clean 2024-01-16T15:18:56.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.501471+0000', 'last_change': '2024-01-16T15:18:49.501471+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.501471+0000', 'last_undegraded': '2024-01-16T15:18:49.501471+0000', 'last_fullsized': '2024-01-16T15:18:49.501471+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498605+0000', 'last_change': '2024-01-16T15:18:49.498605+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498605+0000', 'last_undegraded': '2024-01-16T15:18:49.498605+0000', 'last_fullsized': '2024-01-16T15:18:49.498605+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.502074+0000', 'last_change': '2024-01-16T15:18:49.502074+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.502074+0000', 'last_undegraded': '2024-01-16T15:18:49.502074+0000', 'last_fullsized': '2024-01-16T15:18:49.502074+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.20 is not active+clean 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.505422+0000', 'last_change': '2024-01-16T15:18:49.505422+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.505422+0000', 'last_undegraded': '2024-01-16T15:18:49.505422+0000', 'last_fullsized': '2024-01-16T15:18:49.505422+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.513783+0000', 'last_change': '2024-01-16T15:18:49.513783+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.513783+0000', 'last_undegraded': '2024-01-16T15:18:49.513783+0000', 'last_fullsized': '2024-01-16T15:18:49.513783+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-01-16T15:18:56.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.501201+0000', 'last_change': '2024-01-16T15:18:49.501201+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.501201+0000', 'last_undegraded': '2024-01-16T15:18:49.501201+0000', 'last_fullsized': '2024-01-16T15:18:49.501201+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.512665+0000', 'last_change': '2024-01-16T15:18:49.512665+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.512665+0000', 'last_undegraded': '2024-01-16T15:18:49.512665+0000', 'last_fullsized': '2024-01-16T15:18:49.512665+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.507992+0000', 'last_change': '2024-01-16T15:18:49.507992+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.507992+0000', 'last_undegraded': '2024-01-16T15:18:49.507992+0000', 'last_fullsized': '2024-01-16T15:18:49.507992+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.497629+0000', 'last_change': '2024-01-16T15:18:49.497629+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.497629+0000', 'last_undegraded': '2024-01-16T15:18:49.497629+0000', 'last_fullsized': '2024-01-16T15:18:49.497629+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.505254+0000', 'last_change': '2024-01-16T15:18:49.505254+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.505254+0000', 'last_undegraded': '2024-01-16T15:18:49.505254+0000', 'last_fullsized': '2024-01-16T15:18:49.505254+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-01-16T15:18:56.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.508797+0000', 'last_change': '2024-01-16T15:18:49.508797+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.508797+0000', 'last_undegraded': '2024-01-16T15:18:49.508797+0000', 'last_fullsized': '2024-01-16T15:18:49.508797+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.500474+0000', 'last_change': '2024-01-16T15:18:49.500474+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.500474+0000', 'last_undegraded': '2024-01-16T15:18:49.500474+0000', 'last_fullsized': '2024-01-16T15:18:49.500474+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.500974+0000', 'last_change': '2024-01-16T15:18:49.500974+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.500974+0000', 'last_undegraded': '2024-01-16T15:18:49.500974+0000', 'last_fullsized': '2024-01-16T15:18:49.500974+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.503096+0000', 'last_change': '2024-01-16T15:18:49.503096+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.503096+0000', 'last_undegraded': '2024-01-16T15:18:49.503096+0000', 'last_fullsized': '2024-01-16T15:18:49.503096+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498243+0000', 'last_change': '2024-01-16T15:18:49.498243+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498243+0000', 'last_undegraded': '2024-01-16T15:18:49.498243+0000', 'last_fullsized': '2024-01-16T15:18:49.498243+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.499199+0000', 'last_change': '2024-01-16T15:18:49.499199+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.499199+0000', 'last_undegraded': '2024-01-16T15:18:49.499199+0000', 'last_fullsized': '2024-01-16T15:18:49.499199+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-01-16T15:18:56.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.499391+0000', 'last_change': '2024-01-16T15:18:49.499391+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.499391+0000', 'last_undegraded': '2024-01-16T15:18:49.499391+0000', 'last_fullsized': '2024-01-16T15:18:49.499391+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.495765+0000', 'last_change': '2024-01-16T15:18:49.495765+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.495765+0000', 'last_undegraded': '2024-01-16T15:18:49.495765+0000', 'last_fullsized': '2024-01-16T15:18:49.495765+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.516561+0000', 'last_change': '2024-01-16T15:18:49.516561+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.516561+0000', 'last_undegraded': '2024-01-16T15:18:49.516561+0000', 'last_fullsized': '2024-01-16T15:18:49.516561+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.510804+0000', 'last_change': '2024-01-16T15:18:49.510804+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.510804+0000', 'last_undegraded': '2024-01-16T15:18:49.510804+0000', 'last_fullsized': '2024-01-16T15:18:49.510804+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.507278+0000', 'last_change': '2024-01-16T15:18:49.507278+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.507278+0000', 'last_undegraded': '2024-01-16T15:18:49.507278+0000', 'last_fullsized': '2024-01-16T15:18:49.507278+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-01-16T15:18:56.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.504828+0000', 'last_change': '2024-01-16T15:18:49.504828+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.504828+0000', 'last_undegraded': '2024-01-16T15:18:49.504828+0000', 'last_fullsized': '2024-01-16T15:18:49.504828+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.506081+0000', 'last_change': '2024-01-16T15:18:49.506081+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.506081+0000', 'last_undegraded': '2024-01-16T15:18:49.506081+0000', 'last_fullsized': '2024-01-16T15:18:49.506081+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498896+0000', 'last_change': '2024-01-16T15:18:49.498896+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498896+0000', 'last_undegraded': '2024-01-16T15:18:49.498896+0000', 'last_fullsized': '2024-01-16T15:18:49.498896+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.500913+0000', 'last_change': '2024-01-16T15:18:49.500913+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.500913+0000', 'last_undegraded': '2024-01-16T15:18:49.500913+0000', 'last_fullsized': '2024-01-16T15:18:49.500913+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.500292+0000', 'last_change': '2024-01-16T15:18:49.500292+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.500292+0000', 'last_undegraded': '2024-01-16T15:18:49.500292+0000', 'last_fullsized': '2024-01-16T15:18:49.500292+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.497200+0000', 'last_change': '2024-01-16T15:18:49.497200+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.497200+0000', 'last_undegraded': '2024-01-16T15:18:49.497200+0000', 'last_fullsized': '2024-01-16T15:18:49.497200+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-01-16T15:18:56.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.496009+0000', 'last_change': '2024-01-16T15:18:49.496009+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.496009+0000', 'last_undegraded': '2024-01-16T15:18:49.496009+0000', 'last_fullsized': '2024-01-16T15:18:49.496009+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:18:56.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-01-16T15:18:56.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.505239+0000', 'last_change': '2024-01-16T15:18:49.505239+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.505239+0000', 'last_undegraded': '2024-01-16T15:18:49.505239+0000', 'last_fullsized': '2024-01-16T15:18:49.505239+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-16T15:18:56.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-01-16T15:18:56.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 111, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:18:49.498189+0000', 'last_change': '2024-01-16T15:18:49.498189+0000', 'last_active': '2024-01-16T15:18:49.470527+0000', 'last_peered': '2024-01-16T15:18:49.470527+0000', 'last_clean': '2024-01-16T15:18:49.470527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:18:49.498189+0000', 'last_undegraded': '2024-01-16T15:18:49.498189+0000', 'last_fullsized': '2024-01-16T15:18:49.498189+0000', 'mapping_epoch': 111, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 111, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'last_clean_scrub_stamp': '2024-01-16T15:18:49.470527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T15:18:56.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T15:18:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.278+0000 7f617b400640 1 Processor -- start 2024-01-16T15:18:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.278+0000 7f617b400640 1 -- start start 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f617b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f617415b050 0x7f617415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f617b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f617406e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f617b400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740ad6b0 con 0x7f6174153f10 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f617b400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617406e7a0 con 0x7f617415b050 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f617b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61740b2860 con 0x7f617406ddb0 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f6178a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6174153f10 0x7f61740b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35512/0 (socket says 172.21.15.43:35512) 2024-01-16T15:18:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.280+0000 7f6178a00640 1 -- 172.21.15.43:0/3244426590 learned_addr learned my addr 172.21.15.43:0/3244426590 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:18:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173e00640 1 --2- 172.21.15.43:0/3244426590 >> v2:172.21.15.69:3300/0 conn(0x7f617415b050 0x7f617415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6179400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f617406e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484871738 0 0) 0x7f61740ad6b0 con 0x7f6174153f10 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154005730 con 0x7f6174153f10 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997070636 0 0) 0x7f6154005730 con 0x7f6174153f10 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 msgr2=0x7f617406e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f617406e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 >> v2:172.21.15.69:3300/0 conn(0x7f617415b050 msgr2=0x7f617415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 --2- 172.21.15.43:0/3244426590 >> v2:172.21.15.69:3300/0 conn(0x7f617415b050 0x7f617415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740b2b40 con 0x7f6174153f10 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.281+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f615c001670 con 0x7f6174153f10 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 >> v1:172.21.15.43:6789/0 conn(0x7f6174153f10 legacy=0x7f61740b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f6154005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006100 0x7f61540069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6154006f10 0x7f6154007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617406e7a0 con 0x7f6154006f10 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740b2860 con 0x7f6154006100 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615400db90 con 0x7f617406ddb0 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6179400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6154006f10 0x7f6154007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6173e00640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006100 0x7f61540069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.282+0000 7f6178a00640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f6154005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6179400640 1 -- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 msgr2=0x7f6154005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6179400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f617406ddb0 0x7f6154005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6179400640 1 -- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006100 msgr2=0x7f61540069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6179400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006100 0x7f61540069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6179400640 1 -- 172.21.15.43:0/3244426590 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6154005730 con 0x7f6154006f10 2024-01-16T15:18:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.283+0000 7f6173e00640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006100 0x7f61540069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:56.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.284+0000 7f6179400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6154006f10 0x7f6154007390 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f616c009ac0 tx=0x7f6154007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e87288d1cef8d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.284+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c02c6b0 con 0x7f6154006f10 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.284+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c02c810 con 0x7f6154006f10 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.284+0000 7f6173400640 1 -- 172.21.15.43:0/3244426590 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c02cba0 con 0x7f6154006f10 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 -- 172.21.15.43:0/3244426590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6154006f10 msgr2=0x7f6154007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 --2- 172.21.15.43:0/3244426590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6154006f10 0x7f6154007390 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f616c009ac0 tx=0x7f6154007a00 comp rx=0 tx=0).stop 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f6179400640 1 -- 172.21.15.43:0/3244426590 reap_dead start 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 -- 172.21.15.43:0/3244426590 shutdown_connections 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 -- 172.21.15.43:0/3244426590 >> 172.21.15.43:0/3244426590 conn(0x7f61740b0d80 msgr2=0x7f61740b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 -- 172.21.15.43:0/3244426590 shutdown_connections 2024-01-16T15:18:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.285+0000 7f617b400640 1 -- 172.21.15.43:0/3244426590 wait complete. 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.287+0000 7f617b400640 1 Processor -- start 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.287+0000 7f617b400640 1 -- start start 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006f10 0x7f6174150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61741f2250 0x7f6174157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6174150550 con 0x7f61741f26d0 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617414d080 con 0x7f6154006f10 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f617b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617414d360 con 0x7f61741f2250 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f6179400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f6179400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38364/0 (socket says 172.21.15.43:38364) 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f6179400640 1 -- 172.21.15.43:0/2832594825 learned_addr learned my addr 172.21.15.43:0/2832594825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.288+0000 7f6173e00640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61741f2250 0x7f6174157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6178a00640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006f10 0x7f6174150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 -- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61741f2250 msgr2=0x7f6174157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61741f2250 0x7f6174157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 -- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006f10 msgr2=0x7f6174150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006f10 0x7f6174150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 -- 172.21.15.43:0/2832594825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617414d640 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.289+0000 7f6179400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f616c037040 tx=0x7f616c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.290+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c02cd90 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.290+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c003610 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.290+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c0039c0 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.290+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617414d920 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.290+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174158990 con 0x7f61741f26d0 2024-01-16T15:18:56.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.291+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6174074770 con 0x7f61741f26d0 2024-01-16T15:18:56.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.292+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f616c003b20 con 0x7f61741f26d0 2024-01-16T15:18:56.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.293+0000 7f6170c00640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 0x7f614c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.293+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f616c0a9660 con 0x7f61741f26d0 2024-01-16T15:18:56.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.293+0000 7f6178a00640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 0x7f614c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.294+0000 7f6178a00640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 0x7f614c06e860 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f615c005b10 tx=0x7f615c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.297+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f616c075fd0 con 0x7f61741f26d0 2024-01-16T15:18:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.568+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f61740b3340 con 0x7f61741f26d0 2024-01-16T15:18:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.569+0000 7f6170c00640 1 -- 172.21.15.43:0/2832594825 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1099 (secure 0 0 0) 0x7f616c039460 con 0x7f61741f26d0 2024-01-16T15:18:56.560 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:56.560 INFO:teuthology.orchestra.run.smithi043.stdout:{"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1188,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":113,"num_osds":8,"num_up_osds":8,"osd_up_since":1705417159,"num_in_osds":8,"osd_in_since":1705417151,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":50},{"state_name":"active+clean","count":47}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":592092,"bytes_used":621051904,"bytes_avail":772473061376,"bytes_total":773094113280,"inactive_pgs_ratio":0.51546388864517212,"write_bytes_sec":853,"read_op_per_sec":0,"write_op_per_sec":1},"fsmap":{"epoch":75,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"d","status":"up:active","gid":9132}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-01-16T15:16:03.223164+0000","services":{}},"progress_events":{}} 2024-01-16T15:18:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.572+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 msgr2=0x7f614c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.572+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 0x7f614c06e860 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f615c005b10 tx=0x7f615c009040 comp rx=0 tx=0).stop 2024-01-16T15:18:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.572+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 msgr2=0x7f6174158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.572+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f616c037040 tx=0x7f616c004730 comp rx=0 tx=0).stop 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 shutdown_connections 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f614c06c3a0 0x7f614c06e860 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61741f26d0 0x7f6174158450 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61741f2250 0x7f6174157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 --2- 172.21.15.43:0/2832594825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6154006f10 0x7f6174150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 >> 172.21.15.43:0/2832594825 conn(0x7f61740b0d80 msgr2=0x7f61741531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 shutdown_connections 2024-01-16T15:18:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.573+0000 7f617b400640 1 -- 172.21.15.43:0/2832594825 wait complete. 2024-01-16T15:18:56.584 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T15:18:56.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:18:56.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.817+0000 7f1120000640 1 Processor -- start 2024-01-16T15:18:56.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.818+0000 7f1120000640 1 -- start start 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.819+0000 7f1120000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111815d4a0 0x7f11180af180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.819+0000 7f1120000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f11180af6c0 0x7f111806a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.819+0000 7f1120000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11180afb00 con 0x7f111806ab40 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.819+0000 7f1120000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111806afd0 con 0x7f11180af6c0 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.819+0000 7f1120000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111815d880 con 0x7f111815d4a0 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f11180af6c0 0x7f111806a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f11180af6c0 0x7f111806a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38374/0 (socket says 172.21.15.43:38374) 2024-01-16T15:18:56.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 -- 172.21.15.43:0/753715592 learned_addr learned my addr 172.21.15.43:0/753715592 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:56.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111d600640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111815d4a0 0x7f11180af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 -- 172.21.15.43:0/753715592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111815d4a0 msgr2=0x7f11180af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111815d4a0 0x7f11180af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 -- 172.21.15.43:0/753715592 >> v1:172.21.15.43:6789/0 conn(0x7f111806ab40 legacy=0x7f1118160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f111cc00640 1 -- 172.21.15.43:0/753715592 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111806fba0 con 0x7f11180af6c0 2024-01-16T15:18:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.820+0000 7f1117e00640 1 -- 172.21.15.43:0/753715592 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611691488 0 0) 0x7f11180afb00 con 0x7f111806ab40 2024-01-16T15:18:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f111cc00640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11180af6c0 0x7f111806a600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f1100009870 tx=0x7f110002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870b0a89af99484 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f1117e00640 1 -- 172.21.15.43:0/753715592 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100030840 con 0x7f11180af6c0 2024-01-16T15:18:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f1117e00640 1 -- 172.21.15.43:0/753715592 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11000309a0 con 0x7f11180af6c0 2024-01-16T15:18:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f1117e00640 1 -- 172.21.15.43:0/753715592 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100030d30 con 0x7f11180af6c0 2024-01-16T15:18:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f1120000640 1 -- 172.21.15.43:0/753715592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11180af6c0 msgr2=0x7f111806a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.822+0000 7f1120000640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11180af6c0 0x7f111806a600 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f1100009870 tx=0x7f110002b980 comp rx=0 tx=0).stop 2024-01-16T15:18:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 -- 172.21.15.43:0/753715592 shutdown_connections 2024-01-16T15:18:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11180af6c0 0x7f111806a600 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 --2- 172.21.15.43:0/753715592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f111815d4a0 0x7f11180af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 -- 172.21.15.43:0/753715592 >> 172.21.15.43:0/753715592 conn(0x7f111806d490 msgr2=0x7f111806d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 -- 172.21.15.43:0/753715592 shutdown_connections 2024-01-16T15:18:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.823+0000 7f1120000640 1 -- 172.21.15.43:0/753715592 wait complete. 2024-01-16T15:18:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.825+0000 7f1120000640 1 Processor -- start 2024-01-16T15:18:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.825+0000 7f1120000640 1 -- start start 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11180af6c0 0x7f1118152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111815d4a0 0x7f11180ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11181f4040 con 0x7f111806ab40 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180ccdb0 con 0x7f111815d4a0 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f1120000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11180cd090 con 0x7f11180af6c0 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38390/0 (socket says 172.21.15.43:38390) 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 -- 172.21.15.43:0/613270539 learned_addr learned my addr 172.21.15.43:0/613270539 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 -- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11180af6c0 msgr2=0x7f1118152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11180af6c0 0x7f1118152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 -- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111815d4a0 msgr2=0x7f11180ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111815d4a0 0x7f11180ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.826+0000 7f111d600640 1 -- 172.21.15.43:0/613270539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11000094d0 con 0x7f111806ab40 2024-01-16T15:18:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f111d600640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f110800d800 tx=0x7f110800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110800b6c0 con 0x7f111806ab40 2024-01-16T15:18:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110800b820 con 0x7f111806ab40 2024-01-16T15:18:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110800bbb0 con 0x7f111806ab40 2024-01-16T15:18:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11180cd370 con 0x7f111806ab40 2024-01-16T15:18:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.827+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111815ad00 con 0x7f111806ab40 2024-01-16T15:18:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.829+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111815cfe0 con 0x7f111806ab40 2024-01-16T15:18:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.830+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f110800bd10 con 0x7f111806ab40 2024-01-16T15:18:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.830+0000 7f1115600640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 0x7f10f006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.831+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1108088cb0 con 0x7f111806ab40 2024-01-16T15:18:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.831+0000 7f111cc00640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 0x7f10f006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.832+0000 7f111cc00640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 0x7f10f006e860 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1100036000 tx=0x7f110003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:56.835+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11080555d0 con 0x7f111806ab40 2024-01-16T15:18:57.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.028+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f111806f0c0 con 0x7f10f006c3a0 2024-01-16T15:18:57.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.037+0000 7f1115600640 1 -- 172.21.15.43:0/613270539 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282614 (secure 0 0 0) 0x7f1100030b10 con 0x7f10f006c3a0 2024-01-16T15:18:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:18:57.033 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:18:57.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 msgr2=0x7f10f006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 0x7f10f006e860 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1100036000 tx=0x7f110003b040 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 msgr2=0x7f11181f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f110800d800 tx=0x7f110800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 shutdown_connections 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f10f006c3a0 0x7f10f006e860 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f111815d4a0 0x7f11180ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11180af6c0 0x7f1118152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 --2- 172.21.15.43:0/613270539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f111806ab40 0x7f11181f3b00 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 >> 172.21.15.43:0/613270539 conn(0x7f111806d490 msgr2=0x7f11180ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 shutdown_connections 2024-01-16T15:18:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:18:57.043+0000 7f1120000640 1 -- 172.21.15.43:0/613270539 wait complete. 2024-01-16T15:18:57.052 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-01-16T15:18:55.290428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592092,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":170,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606496,"kb_used_data":11900,"kb_used_omap":12,"kb_used_meta":594483,"kb_avail":754368224,"statfs":{"total":773094113280,"available":772473061376,"internally_reserved":0,"allocated":12185600,"data_stored":5922525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":608751148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001986"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520878+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:18:50.520878+0000","last_peered":"2024-01-16T15:18:50.520878+0000","last_clean":"2024-01-16T15:18:50.520878+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:18:50.520878+0000","last_undegraded":"2024-01-16T15:18:50.520878+0000","last_fullsized":"2024-01-16T15:18:50.520878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:02.958841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496455+0000","last_change":"2024-01-16T15:18:49.496455+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496455+0000","last_undegraded":"2024-01-16T15:18:49.496455+0000","last_fullsized":"2024-01-16T15:18:49.496455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511018+0000","last_change":"2024-01-16T15:18:49.511018+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511018+0000","last_undegraded":"2024-01-16T15:18:49.511018+0000","last_fullsized":"2024-01-16T15:18:49.511018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527282+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:18:50.527282+0000","last_peered":"2024-01-16T15:18:50.527282+0000","last_clean":"2024-01-16T15:18:50.527282+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:18:50.527282+0000","last_undegraded":"2024-01-16T15:18:50.527282+0000","last_fullsized":"2024-01-16T15:18:50.527282+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:40.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494552+0000","last_change":"2024-01-16T15:18:49.494552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494552+0000","last_undegraded":"2024-01-16T15:18:49.494552+0000","last_fullsized":"2024-01-16T15:18:49.494552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511599+0000","last_change":"2024-01-16T15:18:49.511599+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511599+0000","last_undegraded":"2024-01-16T15:18:49.511599+0000","last_fullsized":"2024-01-16T15:18:49.511599+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502722+0000","last_change":"2024-01-16T15:18:49.502722+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502722+0000","last_undegraded":"2024-01-16T15:18:49.502722+0000","last_fullsized":"2024-01-16T15:18:49.502722+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507251+0000","last_change":"2024-01-16T15:18:49.507251+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507251+0000","last_undegraded":"2024-01-16T15:18:49.507251+0000","last_fullsized":"2024-01-16T15:18:49.507251+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527409+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:18:50.527409+0000","last_peered":"2024-01-16T15:18:50.527409+0000","last_clean":"2024-01-16T15:18:50.527409+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:18:50.527409+0000","last_undegraded":"2024-01-16T15:18:50.527409+0000","last_fullsized":"2024-01-16T15:18:50.527409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:44.209808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513531+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:18:50.513531+0000","last_peered":"2024-01-16T15:18:50.513531+0000","last_clean":"2024-01-16T15:18:50.513531+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:18:50.513531+0000","last_undegraded":"2024-01-16T15:18:50.513531+0000","last_fullsized":"2024-01-16T15:18:50.513531+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:37.753051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496236+0000","last_change":"2024-01-16T15:18:49.496236+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496236+0000","last_undegraded":"2024-01-16T15:18:49.496236+0000","last_fullsized":"2024-01-16T15:18:49.496236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513845+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:18:50.513845+0000","last_peered":"2024-01-16T15:18:50.513845+0000","last_clean":"2024-01-16T15:18:50.513845+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:18:50.513845+0000","last_undegraded":"2024-01-16T15:18:50.513845+0000","last_fullsized":"2024-01-16T15:18:50.513845+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509738+0000","last_change":"2024-01-16T15:18:49.509738+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509738+0000","last_undegraded":"2024-01-16T15:18:49.509738+0000","last_fullsized":"2024-01-16T15:18:49.509738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505304+0000","last_change":"2024-01-16T15:18:49.505304+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505304+0000","last_undegraded":"2024-01-16T15:18:49.505304+0000","last_fullsized":"2024-01-16T15:18:49.505304+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509868+0000","last_change":"2024-01-16T15:18:49.509868+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509868+0000","last_undegraded":"2024-01-16T15:18:49.509868+0000","last_fullsized":"2024-01-16T15:18:49.509868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518744+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:18:50.518744+0000","last_peered":"2024-01-16T15:18:50.518744+0000","last_clean":"2024-01-16T15:18:50.518744+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:18:50.518744+0000","last_undegraded":"2024-01-16T15:18:50.518744+0000","last_fullsized":"2024-01-16T15:18:50.518744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:11:42.136923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495707+0000","last_change":"2024-01-16T15:18:49.495707+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495707+0000","last_undegraded":"2024-01-16T15:18:49.495707+0000","last_fullsized":"2024-01-16T15:18:49.495707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501685+0000","last_change":"2024-01-16T15:18:49.501685+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501685+0000","last_undegraded":"2024-01-16T15:18:49.501685+0000","last_fullsized":"2024-01-16T15:18:49.501685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.509552+0000","last_change":"2024-01-16T15:18:49.509552+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.509552+0000","last_undegraded":"2024-01-16T15:18:49.509552+0000","last_fullsized":"2024-01-16T15:18:49.509552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494776+0000","last_change":"2024-01-16T15:18:49.494776+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494776+0000","last_undegraded":"2024-01-16T15:18:49.494776+0000","last_fullsized":"2024-01-16T15:18:49.494776+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494903+0000","last_change":"2024-01-16T15:18:49.494903+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494903+0000","last_undegraded":"2024-01-16T15:18:49.494903+0000","last_fullsized":"2024-01-16T15:18:49.494903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498780+0000","last_change":"2024-01-16T15:18:49.498780+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498780+0000","last_undegraded":"2024-01-16T15:18:49.498780+0000","last_fullsized":"2024-01-16T15:18:49.498780+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.494905+0000","last_change":"2024-01-16T15:18:49.494905+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.494905+0000","last_undegraded":"2024-01-16T15:18:49.494905+0000","last_fullsized":"2024-01-16T15:18:49.494905+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498751+0000","last_change":"2024-01-16T15:18:49.498751+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498751+0000","last_undegraded":"2024-01-16T15:18:49.498751+0000","last_fullsized":"2024-01-16T15:18:49.498751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495191+0000","last_change":"2024-01-16T15:18:49.495191+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495191+0000","last_undegraded":"2024-01-16T15:18:49.495191+0000","last_fullsized":"2024-01-16T15:18:49.495191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518712+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:18:50.518712+0000","last_peered":"2024-01-16T15:18:50.518712+0000","last_clean":"2024-01-16T15:18:50.518712+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:18:50.518712+0000","last_undegraded":"2024-01-16T15:18:50.518712+0000","last_fullsized":"2024-01-16T15:18:50.518712+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:03.552086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.511948+0000","last_change":"2024-01-16T15:18:49.511948+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.511948+0000","last_undegraded":"2024-01-16T15:18:49.511948+0000","last_fullsized":"2024-01-16T15:18:49.511948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501471+0000","last_change":"2024-01-16T15:18:49.501471+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501471+0000","last_undegraded":"2024-01-16T15:18:49.501471+0000","last_fullsized":"2024-01-16T15:18:49.501471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498605+0000","last_change":"2024-01-16T15:18:49.498605+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498605+0000","last_undegraded":"2024-01-16T15:18:49.498605+0000","last_fullsized":"2024-01-16T15:18:49.498605+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.502074+0000","last_change":"2024-01-16T15:18:49.502074+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.502074+0000","last_undegraded":"2024-01-16T15:18:49.502074+0000","last_fullsized":"2024-01-16T15:18:49.502074+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.517526+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:18:50.517526+0000","last_peered":"2024-01-16T15:18:50.517526+0000","last_clean":"2024-01-16T15:18:50.517526+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:18:50.517526+0000","last_undegraded":"2024-01-16T15:18:50.517526+0000","last_fullsized":"2024-01-16T15:18:50.517526+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:59.126247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505422+0000","last_change":"2024-01-16T15:18:49.505422+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505422+0000","last_undegraded":"2024-01-16T15:18:49.505422+0000","last_fullsized":"2024-01-16T15:18:49.505422+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.513783+0000","last_change":"2024-01-16T15:18:49.513783+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.513783+0000","last_undegraded":"2024-01-16T15:18:49.513783+0000","last_fullsized":"2024-01-16T15:18:49.513783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520299+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:18:49.520299+0000","last_peered":"2024-01-16T15:18:49.520299+0000","last_clean":"2024-01-16T15:18:49.520299+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:18:49.520299+0000","last_undegraded":"2024-01-16T15:18:49.520299+0000","last_fullsized":"2024-01-16T15:18:49.520299+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.501201+0000","last_change":"2024-01-16T15:18:49.501201+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.501201+0000","last_undegraded":"2024-01-16T15:18:49.501201+0000","last_fullsized":"2024-01-16T15:18:49.501201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524289+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:18:49.524289+0000","last_peered":"2024-01-16T15:18:49.524289+0000","last_clean":"2024-01-16T15:18:49.524289+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:18:49.524289+0000","last_undegraded":"2024-01-16T15:18:49.524289+0000","last_fullsized":"2024-01-16T15:18:49.524289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:21.931597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.512665+0000","last_change":"2024-01-16T15:18:49.512665+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.512665+0000","last_undegraded":"2024-01-16T15:18:49.512665+0000","last_fullsized":"2024-01-16T15:18:49.512665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522310+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:18:49.522310+0000","last_peered":"2024-01-16T15:18:49.522310+0000","last_clean":"2024-01-16T15:18:49.522310+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:18:49.522310+0000","last_undegraded":"2024-01-16T15:18:49.522310+0000","last_fullsized":"2024-01-16T15:18:49.522310+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:02.444449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507992+0000","last_change":"2024-01-16T15:18:49.507992+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507992+0000","last_undegraded":"2024-01-16T15:18:49.507992+0000","last_fullsized":"2024-01-16T15:18:49.507992+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522279+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:18:49.522279+0000","last_peered":"2024-01-16T15:18:49.522279+0000","last_clean":"2024-01-16T15:18:49.522279+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:18:49.522279+0000","last_undegraded":"2024-01-16T15:18:49.522279+0000","last_fullsized":"2024-01-16T15:18:49.522279+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:03:44.623306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.513485+0000","last_change":"2024-01-16T15:18:50.510180+0000","last_active":"2024-01-16T15:18:50.513485+0000","last_peered":"2024-01-16T15:18:50.513485+0000","last_clean":"2024-01-16T15:18:50.513485+0000","last_became_active":"2024-01-16T15:18:50.509888+0000","last_became_peered":"2024-01-16T15:18:50.509888+0000","last_unstale":"2024-01-16T15:18:50.513485+0000","last_undegraded":"2024-01-16T15:18:50.513485+0000","last_fullsized":"2024-01-16T15:18:50.513485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:00:49.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524377+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:18:49.524377+0000","last_peered":"2024-01-16T15:18:49.524377+0000","last_clean":"2024-01-16T15:18:49.524377+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:18:49.524377+0000","last_undegraded":"2024-01-16T15:18:49.524377+0000","last_fullsized":"2024-01-16T15:18:49.524377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497629+0000","last_change":"2024-01-16T15:18:49.497629+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497629+0000","last_undegraded":"2024-01-16T15:18:49.497629+0000","last_fullsized":"2024-01-16T15:18:49.497629+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.518795+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:18:49.518795+0000","last_peered":"2024-01-16T15:18:49.518795+0000","last_clean":"2024-01-16T15:18:49.518795+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:18:49.518795+0000","last_undegraded":"2024-01-16T15:18:49.518795+0000","last_fullsized":"2024-01-16T15:18:49.518795+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:43.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518433+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:18:50.518433+0000","last_peered":"2024-01-16T15:18:50.518433+0000","last_clean":"2024-01-16T15:18:50.518433+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:18:50.518433+0000","last_undegraded":"2024-01-16T15:18:50.518433+0000","last_fullsized":"2024-01-16T15:18:50.518433+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:48.856213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511369+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:18:49.511369+0000","last_peered":"2024-01-16T15:18:49.511369+0000","last_clean":"2024-01-16T15:18:49.511369+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:18:49.511369+0000","last_undegraded":"2024-01-16T15:18:49.511369+0000","last_fullsized":"2024-01-16T15:18:49.511369+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505254+0000","last_change":"2024-01-16T15:18:49.505254+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505254+0000","last_undegraded":"2024-01-16T15:18:49.505254+0000","last_fullsized":"2024-01-16T15:18:49.505254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503470+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:18:50.503470+0000","last_peered":"2024-01-16T15:18:50.503470+0000","last_clean":"2024-01-16T15:18:50.503470+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:18:50.503470+0000","last_undegraded":"2024-01-16T15:18:50.503470+0000","last_fullsized":"2024-01-16T15:18:50.503470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.508797+0000","last_change":"2024-01-16T15:18:49.508797+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.508797+0000","last_undegraded":"2024-01-16T15:18:49.508797+0000","last_fullsized":"2024-01-16T15:18:49.508797+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.525562+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:18:49.525562+0000","last_peered":"2024-01-16T15:18:49.525562+0000","last_clean":"2024-01-16T15:18:49.525562+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:18:49.525562+0000","last_undegraded":"2024-01-16T15:18:49.525562+0000","last_fullsized":"2024-01-16T15:18:49.525562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:10.112260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500474+0000","last_change":"2024-01-16T15:18:49.500474+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500474+0000","last_undegraded":"2024-01-16T15:18:49.500474+0000","last_fullsized":"2024-01-16T15:18:49.500474+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.515357+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:18:49.515357+0000","last_peered":"2024-01-16T15:18:49.515357+0000","last_clean":"2024-01-16T15:18:49.515357+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:18:49.515357+0000","last_undegraded":"2024-01-16T15:18:49.515357+0000","last_fullsized":"2024-01-16T15:18:49.515357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:56.122268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500974+0000","last_change":"2024-01-16T15:18:49.500974+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500974+0000","last_undegraded":"2024-01-16T15:18:49.500974+0000","last_fullsized":"2024-01-16T15:18:49.500974+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.536241+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:18:50.536241+0000","last_peered":"2024-01-16T15:18:50.536241+0000","last_clean":"2024-01-16T15:18:50.536241+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:18:50.536241+0000","last_undegraded":"2024-01-16T15:18:50.536241+0000","last_fullsized":"2024-01-16T15:18:50.536241+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.503096+0000","last_change":"2024-01-16T15:18:49.503096+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.503096+0000","last_undegraded":"2024-01-16T15:18:49.503096+0000","last_fullsized":"2024-01-16T15:18:49.503096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.538150+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:18:50.538150+0000","last_peered":"2024-01-16T15:18:50.538150+0000","last_clean":"2024-01-16T15:18:50.538150+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:18:50.538150+0000","last_undegraded":"2024-01-16T15:18:50.538150+0000","last_fullsized":"2024-01-16T15:18:50.538150+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498243+0000","last_change":"2024-01-16T15:18:49.498243+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498243+0000","last_undegraded":"2024-01-16T15:18:49.498243+0000","last_fullsized":"2024-01-16T15:18:49.498243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520668+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:18:49.520668+0000","last_peered":"2024-01-16T15:18:49.520668+0000","last_clean":"2024-01-16T15:18:49.520668+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:18:49.520668+0000","last_undegraded":"2024-01-16T15:18:49.520668+0000","last_fullsized":"2024-01-16T15:18:49.520668+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499199+0000","last_change":"2024-01-16T15:18:49.499199+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499199+0000","last_undegraded":"2024-01-16T15:18:49.499199+0000","last_fullsized":"2024-01-16T15:18:49.499199+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516723+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:18:49.516723+0000","last_peered":"2024-01-16T15:18:49.516723+0000","last_clean":"2024-01-16T15:18:49.516723+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:18:49.516723+0000","last_undegraded":"2024-01-16T15:18:49.516723+0000","last_fullsized":"2024-01-16T15:18:49.516723+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:21.691380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.499391+0000","last_change":"2024-01-16T15:18:49.499391+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.499391+0000","last_undegraded":"2024-01-16T15:18:49.499391+0000","last_fullsized":"2024-01-16T15:18:49.499391+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511289+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:18:49.511289+0000","last_peered":"2024-01-16T15:18:49.511289+0000","last_clean":"2024-01-16T15:18:49.511289+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:18:49.511289+0000","last_undegraded":"2024-01-16T15:18:49.511289+0000","last_fullsized":"2024-01-16T15:18:49.511289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:29.197614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.495765+0000","last_change":"2024-01-16T15:18:49.495765+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.495765+0000","last_undegraded":"2024-01-16T15:18:49.495765+0000","last_fullsized":"2024-01-16T15:18:49.495765+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533470+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:18:50.533470+0000","last_peered":"2024-01-16T15:18:50.533470+0000","last_clean":"2024-01-16T15:18:50.533470+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:18:50.533470+0000","last_undegraded":"2024-01-16T15:18:50.533470+0000","last_fullsized":"2024-01-16T15:18:50.533470+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.516561+0000","last_change":"2024-01-16T15:18:49.516561+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.516561+0000","last_undegraded":"2024-01-16T15:18:49.516561+0000","last_fullsized":"2024-01-16T15:18:49.516561+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.511184+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:18:49.511184+0000","last_peered":"2024-01-16T15:18:49.511184+0000","last_clean":"2024-01-16T15:18:49.511184+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:18:49.511184+0000","last_undegraded":"2024-01-16T15:18:49.511184+0000","last_fullsized":"2024-01-16T15:18:49.511184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:53.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.510804+0000","last_change":"2024-01-16T15:18:49.510804+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.510804+0000","last_undegraded":"2024-01-16T15:18:49.510804+0000","last_fullsized":"2024-01-16T15:18:49.510804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520479+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:18:49.520479+0000","last_peered":"2024-01-16T15:18:49.520479+0000","last_clean":"2024-01-16T15:18:49.520479+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:18:49.520479+0000","last_undegraded":"2024-01-16T15:18:49.520479+0000","last_fullsized":"2024-01-16T15:18:49.520479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:53.321942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.507278+0000","last_change":"2024-01-16T15:18:49.507278+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.507278+0000","last_undegraded":"2024-01-16T15:18:49.507278+0000","last_fullsized":"2024-01-16T15:18:49.507278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520339+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:18:49.520339+0000","last_peered":"2024-01-16T15:18:49.520339+0000","last_clean":"2024-01-16T15:18:49.520339+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:18:49.520339+0000","last_undegraded":"2024-01-16T15:18:49.520339+0000","last_fullsized":"2024-01-16T15:18:49.520339+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.504828+0000","last_change":"2024-01-16T15:18:49.504828+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.504828+0000","last_undegraded":"2024-01-16T15:18:49.504828+0000","last_fullsized":"2024-01-16T15:18:49.504828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523509+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:18:49.523509+0000","last_peered":"2024-01-16T15:18:49.523509+0000","last_clean":"2024-01-16T15:18:49.523509+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:18:49.523509+0000","last_undegraded":"2024-01-16T15:18:49.523509+0000","last_fullsized":"2024-01-16T15:18:49.523509+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518601+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:18:50.518601+0000","last_peered":"2024-01-16T15:18:50.518601+0000","last_clean":"2024-01-16T15:18:50.518601+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:18:50.518601+0000","last_undegraded":"2024-01-16T15:18:50.518601+0000","last_fullsized":"2024-01-16T15:18:50.518601+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:27.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522193+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:18:49.522193+0000","last_peered":"2024-01-16T15:18:49.522193+0000","last_clean":"2024-01-16T15:18:49.522193+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:18:49.522193+0000","last_undegraded":"2024-01-16T15:18:49.522193+0000","last_fullsized":"2024-01-16T15:18:49.522193+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:54.524825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.506081+0000","last_change":"2024-01-16T15:18:49.506081+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.506081+0000","last_undegraded":"2024-01-16T15:18:49.506081+0000","last_fullsized":"2024-01-16T15:18:49.506081+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.523683+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:18:49.523683+0000","last_peered":"2024-01-16T15:18:49.523683+0000","last_clean":"2024-01-16T15:18:49.523683+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:18:49.523683+0000","last_undegraded":"2024-01-16T15:18:49.523683+0000","last_fullsized":"2024-01-16T15:18:49.523683+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498896+0000","last_change":"2024-01-16T15:18:49.498896+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498896+0000","last_undegraded":"2024-01-16T15:18:49.498896+0000","last_fullsized":"2024-01-16T15:18:49.498896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503708+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:18:50.503708+0000","last_peered":"2024-01-16T15:18:50.503708+0000","last_clean":"2024-01-16T15:18:50.503708+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:18:50.503708+0000","last_undegraded":"2024-01-16T15:18:50.503708+0000","last_fullsized":"2024-01-16T15:18:50.503708+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500913+0000","last_change":"2024-01-16T15:18:49.500913+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500913+0000","last_undegraded":"2024-01-16T15:18:49.500913+0000","last_fullsized":"2024-01-16T15:18:49.500913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.522640+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:18:49.522640+0000","last_peered":"2024-01-16T15:18:49.522640+0000","last_clean":"2024-01-16T15:18:49.522640+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:18:49.522640+0000","last_undegraded":"2024-01-16T15:18:49.522640+0000","last_fullsized":"2024-01-16T15:18:49.522640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:48.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.526632+0000","last_change":"2024-01-16T15:18:50.522199+0000","last_active":"2024-01-16T15:18:50.526632+0000","last_peered":"2024-01-16T15:18:50.526632+0000","last_clean":"2024-01-16T15:18:50.526632+0000","last_became_active":"2024-01-16T15:18:50.521902+0000","last_became_peered":"2024-01-16T15:18:50.521902+0000","last_unstale":"2024-01-16T15:18:50.526632+0000","last_undegraded":"2024-01-16T15:18:50.526632+0000","last_fullsized":"2024-01-16T15:18:50.526632+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:11.568326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.516769+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:18:49.516769+0000","last_peered":"2024-01-16T15:18:49.516769+0000","last_clean":"2024-01-16T15:18:49.516769+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:18:49.516769+0000","last_undegraded":"2024-01-16T15:18:49.516769+0000","last_fullsized":"2024-01-16T15:18:49.516769+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:17.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.518602+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:18:50.518602+0000","last_peered":"2024-01-16T15:18:50.518602+0000","last_clean":"2024-01-16T15:18:50.518602+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:18:50.518602+0000","last_undegraded":"2024-01-16T15:18:50.518602+0000","last_fullsized":"2024-01-16T15:18:50.518602+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:38.334522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.505052+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:18:50.505052+0000","last_peered":"2024-01-16T15:18:50.505052+0000","last_clean":"2024-01-16T15:18:50.505052+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:18:50.505052+0000","last_undegraded":"2024-01-16T15:18:50.505052+0000","last_fullsized":"2024-01-16T15:18:50.505052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.500292+0000","last_change":"2024-01-16T15:18:49.500292+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.500292+0000","last_undegraded":"2024-01-16T15:18:49.500292+0000","last_fullsized":"2024-01-16T15:18:49.500292+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.520661+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:18:49.520661+0000","last_peered":"2024-01-16T15:18:49.520661+0000","last_clean":"2024-01-16T15:18:49.520661+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:18:49.520661+0000","last_undegraded":"2024-01-16T15:18:49.520661+0000","last_fullsized":"2024-01-16T15:18:49.520661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:24.291444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.497200+0000","last_change":"2024-01-16T15:18:49.497200+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.497200+0000","last_undegraded":"2024-01-16T15:18:49.497200+0000","last_fullsized":"2024-01-16T15:18:49.497200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.510956+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:18:49.510956+0000","last_peered":"2024-01-16T15:18:49.510956+0000","last_clean":"2024-01-16T15:18:49.510956+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:18:49.510956+0000","last_undegraded":"2024-01-16T15:18:49.510956+0000","last_fullsized":"2024-01-16T15:18:49.510956+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.496009+0000","last_change":"2024-01-16T15:18:49.496009+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.496009+0000","last_undegraded":"2024-01-16T15:18:49.496009+0000","last_fullsized":"2024-01-16T15:18:49.496009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524006+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:18:49.524006+0000","last_peered":"2024-01-16T15:18:49.524006+0000","last_clean":"2024-01-16T15:18:49.524006+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:18:49.524006+0000","last_undegraded":"2024-01-16T15:18:49.524006+0000","last_fullsized":"2024-01-16T15:18:49.524006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.505239+0000","last_change":"2024-01-16T15:18:49.505239+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.505239+0000","last_undegraded":"2024-01-16T15:18:49.505239+0000","last_fullsized":"2024-01-16T15:18:49.505239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.519663+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:18:49.519663+0000","last_peered":"2024-01-16T15:18:49.519663+0000","last_clean":"2024-01-16T15:18:49.519663+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:18:49.519663+0000","last_undegraded":"2024-01-16T15:18:49.519663+0000","last_fullsized":"2024-01-16T15:18:49.519663+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:08:27.624738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.520862+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:18:50.520862+0000","last_peered":"2024-01-16T15:18:50.520862+0000","last_clean":"2024-01-16T15:18:50.520862+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:18:50.520862+0000","last_undegraded":"2024-01-16T15:18:50.520862+0000","last_fullsized":"2024-01-16T15:18:50.520862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:09.166757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503918+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:18:50.503918+0000","last_peered":"2024-01-16T15:18:50.503918+0000","last_clean":"2024-01-16T15:18:50.503918+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:18:50.503918+0000","last_undegraded":"2024-01-16T15:18:50.503918+0000","last_fullsized":"2024-01-16T15:18:50.503918+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-16T15:18:49.498189+0000","last_change":"2024-01-16T15:18:49.498189+0000","last_active":"2024-01-16T15:18:49.470527+0000","last_peered":"2024-01-16T15:18:49.470527+0000","last_clean":"2024-01-16T15:18:49.470527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:18:49.498189+0000","last_undegraded":"2024-01-16T15:18:49.498189+0000","last_fullsized":"2024-01-16T15:18:49.498189+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-16T15:18:49.524346+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:18:49.524346+0000","last_peered":"2024-01-16T15:18:49.524346+0000","last_clean":"2024-01-16T15:18:49.524346+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:18:49.524346+0000","last_undegraded":"2024-01-16T15:18:49.524346+0000","last_fullsized":"2024-01-16T15:18:49.524346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:18:48.485739+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:18:48.485739+0000","last_peered":"2024-01-16T15:18:48.485739+0000","last_clean":"2024-01-16T15:18:48.485739+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:18:48.485739+0000","last_undegraded":"2024-01-16T15:18:48.485739+0000","last_fullsized":"2024-01-16T15:18:48.485739+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1724,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542381,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1970176,"data_stored":1183817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.349,"15min":1.5429999999999999},"min":{"1min":0.72099999999999997,"5min":0.63500000000000001,"15min":0.28199999999999997},"max":{"1min":1.768,"5min":5.2690000000000001,"15min":27.469000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4410000000000001,"15min":1.6859999999999999},"min":{"1min":0.86099999999999999,"5min":0.79500000000000004,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":6.5599999999999996,"15min":25.503},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.139,"15min":1.2689999999999999},"min":{"1min":0.77600000000000002,"5min":0.55100000000000005,"15min":0.34499999999999997},"max":{"1min":1.712,"5min":1.742,"15min":6.718},"last":1.343},{"interface":"front","average":{"1min":1.262,"5min":1.2070000000000001,"15min":1.3520000000000001},"min":{"1min":0.85499999999999998,"5min":0.75700000000000001,"15min":0.373},"max":{"1min":1.7669999999999999,"5min":2.0110000000000001,"15min":15.452},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.4530000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.48299999999999998},"max":{"1min":1.7709999999999999,"5min":11.536,"15min":11.536},"last":1.25},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5409999999999999,"15min":1.5629999999999999},"min":{"1min":1.0720000000000001,"5min":0.79800000000000004,"15min":0.504},"max":{"1min":1.978,"5min":11.576000000000001,"15min":25.478000000000002},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.222,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.67000000000000004,"15min":0.40899999999999997},"max":{"1min":1.593,"5min":2.133,"15min":13.907999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.357,"5min":1.365,"15min":1.5029999999999999},"min":{"1min":1.022,"5min":0.89300000000000002,"15min":0.62},"max":{"1min":1.8580000000000001,"5min":2.1960000000000002,"15min":8.6999999999999993},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.452},"min":{"1min":0.89200000000000002,"5min":0.85299999999999998,"15min":0.376},"max":{"1min":1.7829999999999999,"5min":6.7599999999999998,"15min":25.401},"last":1.171},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3280000000000001,"15min":1.472},"min":{"1min":0.84999999999999998,"5min":0.59599999999999997,"15min":0.45000000000000001},"max":{"1min":2.069,"5min":2.069,"15min":10.134},"last":1.5489999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":0.96399999999999997,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.7310000000000001,"5min":2.1259999999999999,"15min":12.528},"last":1.486},{"interface":"front","average":{"1min":1.498,"5min":1.544,"15min":1.639},"min":{"1min":1.0640000000000001,"5min":0.86599999999999999,"15min":0.41399999999999998},"max":{"1min":1.9159999999999999,"5min":6.8719999999999999,"15min":26.327000000000002},"last":1.302}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.6759999999999999,"15min":1.734},"min":{"1min":1.3640000000000001,"5min":1.087,"15min":0.45200000000000001},"max":{"1min":2.0870000000000002,"5min":6.9690000000000003,"15min":25.451000000000001},"last":1.462},{"interface":"front","average":{"1min":1.524,"5min":1.4930000000000001,"15min":1.605},"min":{"1min":0.92400000000000004,"5min":0.89800000000000002,"15min":0.44700000000000001},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":12.577999999999999},"last":1.6120000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72716,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299124,"statfs":{"total":96636764160,"available":96562302976,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.452,"15min":1.744},"min":{"1min":0.752,"5min":0.71299999999999997,"15min":0.35399999999999998},"max":{"1min":1.53,"5min":29.77,"15min":30.164999999999999},"last":1.53},{"interface":"front","average":{"1min":1.292,"5min":1.6870000000000001,"15min":1.8779999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.48899999999999999},"max":{"1min":1.5780000000000001,"5min":36.405999999999999,"15min":36.405999999999999},"last":1.49}]},{"osd":1,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1850000000000001,"15min":1.508},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.47899999999999998},"max":{"1min":1.52,"5min":5.2859999999999996,"15min":28.122},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.2549999999999999,"15min":1.298},"min":{"1min":0.78200000000000003,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":1.548,"5min":10.494,"15min":10.494},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.347,"15min":1.734},"min":{"1min":1,"5min":0.76900000000000002,"15min":0.501},"max":{"1min":1.845,"5min":5.0549999999999997,"15min":26.109000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.51,"15min":1.893},"min":{"1min":1.0329999999999999,"5min":1.016,"15min":0.67100000000000004},"max":{"1min":2.0449999999999999,"5min":5.1459999999999999,"15min":26.161000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3029999999999999,"15min":1.615},"min":{"1min":1.1060000000000001,"5min":0.68500000000000005,"15min":0.44700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8079999999999998,"15min":17.75},"last":1.363},{"interface":"front","average":{"1min":1.395,"5min":1.681,"15min":1.728},"min":{"1min":1.0349999999999999,"5min":0.77100000000000002,"15min":0.55700000000000005},"max":{"1min":2.1789999999999998,"5min":21.456,"15min":21.456},"last":1.5600000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3540000000000001,"15min":1.5069999999999999},"min":{"1min":0.91400000000000003,"5min":0.82099999999999995,"15min":0.45000000000000001},"max":{"1min":1.6699999999999999,"5min":5.3310000000000004,"15min":15.571},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.325,"15min":1.5469999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.42299999999999999},"max":{"1min":1.639,"5min":3.6160000000000001,"15min":23.667999999999999},"last":1.212}]},{"osd":5,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4350000000000001,"15min":1.4890000000000001},"min":{"1min":0.83499999999999996,"5min":0.73699999999999999,"15min":0.42299999999999999},"max":{"1min":1.982,"5min":13.949999999999999,"15min":23.137},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.6100000000000001,"15min":1.6970000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":0.47099999999999997},"max":{"1min":1.714,"5min":11.298,"15min":25.684000000000001},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.6020000000000001,"15min":1.552},"min":{"1min":0.754,"5min":0.71499999999999997,"15min":0.47099999999999997},"max":{"1min":2.1000000000000001,"5min":14.037000000000001,"15min":14.037000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6240000000000001,"15min":1.724},"min":{"1min":0.95899999999999996,"5min":0.92300000000000004,"15min":0.499},"max":{"1min":2.1659999999999999,"5min":4.9829999999999997,"15min":11.788},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64400,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307440,"statfs":{"total":96636764160,"available":96570818560,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.075,"15min":1.571},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.39000000000000001},"max":{"1min":1.5649999999999999,"5min":1.651,"15min":26.228999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.135,"15min":1.831},"min":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.34799999999999998},"max":{"1min":1.7070000000000001,"5min":1.849,"15min":56.011000000000003},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1419999999999999,"15min":1.363},"min":{"1min":0.79900000000000004,"5min":0.65200000000000002,"15min":0.40999999999999998},"max":{"1min":1.6419999999999999,"5min":2.2450000000000001,"15min":16.687999999999999},"last":1.034},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2430000000000001,"15min":1.8600000000000001},"min":{"1min":0.88900000000000001,"5min":0.745,"15min":0.40999999999999998},"max":{"1min":1.667,"5min":4.8499999999999996,"15min":67.930000000000007},"last":1.125}]},{"osd":2,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1779999999999999,"15min":1.52},"min":{"1min":0.85199999999999998,"5min":0.61099999999999999,"15min":0.47099999999999997},"max":{"1min":1.55,"5min":1.7529999999999999,"15min":13.193},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.246,"5min":1.258,"15min":1.4850000000000001},"min":{"1min":0.93700000000000006,"5min":0.76300000000000001,"15min":0.45800000000000002},"max":{"1min":1.7769999999999999,"5min":3.9590000000000001,"15min":16.721},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.379,"15min":1.712},"min":{"1min":1.0509999999999999,"5min":0.81100000000000005,"15min":0.57999999999999996},"max":{"1min":1.8680000000000001,"5min":4.8010000000000002,"15min":22.382000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.3819999999999999,"15min":1.7529999999999999},"min":{"1min":0.94199999999999995,"5min":0.72399999999999998,"15min":0.51200000000000001},"max":{"1min":1.921,"5min":1.921,"15min":14.685},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2649999999999999,"15min":1.419},"min":{"1min":0.95199999999999996,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.8680000000000001,"5min":2.2549999999999999,"15min":16.382999999999999},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.377,"15min":1.5269999999999999},"min":{"1min":0.98899999999999999,"5min":0.502,"15min":0.379},"max":{"1min":1.8129999999999999,"5min":4.6580000000000004,"15min":27.006},"last":1.6539999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3600000000000001,"15min":1.5600000000000001},"min":{"1min":0.83999999999999997,"5min":0.52400000000000002,"15min":0.39700000000000002},"max":{"1min":1.974,"5min":1.9770000000000001,"15min":13.263999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.4890000000000001,"15min":1.609},"min":{"1min":1.304,"5min":0.629,"15min":0.505},"max":{"1min":1.9670000000000001,"5min":2.1640000000000001,"15min":9.5269999999999992},"last":1.704}]},{"osd":7,"last update":"Tue Jan 16 15:17:51 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.52,"15min":1.7470000000000001},"min":{"1min":1.2410000000000001,"5min":0.59099999999999997,"15min":0.45000000000000001},"max":{"1min":1.9119999999999999,"5min":4.742,"15min":35.127000000000002},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5489999999999999,"15min":1.7290000000000001},"min":{"1min":0.91200000000000003,"5min":0.61099999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":13.491},"last":1.601}]}]},{"osd":4,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1619999999999999,"15min":1.244},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6180000000000001,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.21},{"interface":"front","average":{"1min":1.095,"5min":1.123,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.679,"5min":1.958,"15min":60.106000000000002},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.629,"15min":1.79},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51100000000000001},"max":{"1min":1.738,"5min":24.399999999999999,"15min":24.399999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.611,"15min":1.627},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57099999999999995},"max":{"1min":1.7290000000000001,"5min":24.456,"15min":24.456},"last":1.7090000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.3029999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.52800000000000002},"max":{"1min":1.6299999999999999,"5min":2.1699999999999999,"15min":9.7140000000000004},"last":1.145},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2889999999999999,"15min":1.6970000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.52400000000000002},"max":{"1min":1.6479999999999999,"5min":1.855,"15min":35.024999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3300000000000001,"15min":1.556},"min":{"1min":0.503,"5min":0.503,"15min":0.47799999999999998},"max":{"1min":1.6499999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":6.4589999999999996,"15min":7.8140000000000001},"last":1.375}]},{"osd":5,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.454},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":2.1299999999999999,"15min":14.568},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.4690000000000001,"15min":1.4319999999999999},"min":{"1min":0.74299999999999999,"5min":0.496,"15min":0.34399999999999997},"max":{"1min":1.8160000000000001,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":2.0720000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.436},"max":{"1min":1.696,"5min":3.0190000000000001,"15min":9.6630000000000003},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4239999999999999,"15min":1.6299999999999999},"min":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.45100000000000001},"max":{"1min":1.675,"5min":2.3980000000000001,"15min":14.317},"last":2.0129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5640000000000001,"15min":1.5449999999999999},"min":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.52100000000000002},"max":{"1min":1.927,"5min":6.5250000000000004,"15min":9.2799999999999994},"last":1.903},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.552,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.443},"max":{"1min":1.8540000000000001,"5min":3.3439999999999999,"15min":9.2270000000000003},"last":2.2229999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82432,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289408,"statfs":{"total":96636764160,"available":96552353792,"internally_reserved":0,"allocated":1363968,"data_stored":587146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.93899999999999995,"15min":1.429},"min":{"1min":0.748,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.45,"5min":1.9299999999999999,"15min":50.988999999999997},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.087,"15min":1.4750000000000001},"min":{"1min":0.80100000000000005,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.5840000000000001,"5min":1.786,"15min":20.274000000000001},"last":1.141}]},{"osd":1,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.101,"15min":1.385},"min":{"1min":0.71799999999999997,"5min":0.371,"15min":0.32100000000000001},"max":{"1min":1.613,"5min":3.6640000000000001,"15min":18.192},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.21,"15min":1.712},"min":{"1min":0.89000000000000001,"5min":0.39400000000000002,"15min":0.313},"max":{"1min":1.8440000000000001,"5min":1.9239999999999999,"15min":20.385999999999999},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3089999999999999,"15min":1.7250000000000001},"min":{"1min":0.871,"5min":0.45500000000000002,"15min":0.36699999999999999},"max":{"1min":1.978,"5min":3.5249999999999999,"15min":42.840000000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.397,"15min":1.806},"min":{"1min":1.016,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":1.631,"5min":3.702,"15min":38.707000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.339,"15min":1.762},"min":{"1min":0.99199999999999999,"5min":0.78200000000000003,"15min":0.44500000000000001},"max":{"1min":1.734,"5min":2.327,"15min":18.952000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4650000000000001,"15min":1.79},"min":{"1min":1.1859999999999999,"5min":0.745,"15min":0.54800000000000004},"max":{"1min":1.718,"5min":3.9529999999999998,"15min":18.088999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.4890000000000001,"15min":1.8160000000000001},"min":{"1min":1.212,"5min":0.73199999999999998,"15min":0.55500000000000005},"max":{"1min":1.7729999999999999,"5min":2.661,"15min":24.349},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.464,"15min":1.9199999999999999},"min":{"1min":1.105,"5min":0.753,"15min":0.61899999999999999},"max":{"1min":1.9119999999999999,"5min":3.4239999999999999,"15min":14.192},"last":1.306}]},{"osd":6,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.621,"15min":1.877},"min":{"1min":1.159,"5min":0.77200000000000002,"15min":0.52200000000000002},"max":{"1min":1.7929999999999999,"5min":3.9009999999999998,"15min":22.687000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.623,"15min":1.931},"min":{"1min":1.262,"5min":0.73799999999999999,"15min":0.59699999999999998},"max":{"1min":1.7969999999999999,"5min":3.9300000000000002,"15min":24.058},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:17:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6040000000000001,"15min":2.1000000000000001},"min":{"1min":1.111,"5min":0.85099999999999998,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":6.3209999999999997,"15min":35.570999999999998},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.6659999999999999,"15min":2.0619999999999998},"min":{"1min":1.2969999999999999,"5min":0.81100000000000005,"15min":0.58899999999999997},"max":{"1min":2.0750000000000002,"5min":3.8279999999999998,"15min":37.607999999999997},"last":1.633}]}]},{"osd":2,"up_from":14,"seq":60129542380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94924,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94276916,"statfs":{"total":96636764160,"available":96539561984,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90400000000000003,"15min":1.4430000000000001},"min":{"1min":0.67700000000000005,"5min":0.36099999999999999,"15min":0.24399999999999999},"max":{"1min":1.135,"5min":1.96,"15min":64.387},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.022,"15min":1.387},"min":{"1min":0.76500000000000001,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7030000000000001,"5min":1.9890000000000001,"15min":24.308},"last":1.7490000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0580000000000001,"15min":1.466},"min":{"1min":0.752,"5min":0.46300000000000002,"15min":0.34699999999999998},"max":{"1min":1.2490000000000001,"5min":2.093,"15min":27.263000000000002},"last":1.27},{"interface":"front","average":{"1min":1.381,"5min":1.3879999999999999,"15min":1.8520000000000001},"min":{"1min":1.0940000000000001,"5min":0.51300000000000001,"15min":0.38500000000000001},"max":{"1min":1.8620000000000001,"5min":2.1539999999999999,"15min":23.602},"last":1.234}]},{"osd":3,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.385,"15min":1.627},"min":{"1min":1.1499999999999999,"5min":0.58499999999999996,"15min":0.40100000000000002},"max":{"1min":1.9299999999999999,"5min":2.085,"15min":27.311},"last":1.081},{"interface":"front","average":{"1min":1.296,"5min":1.327,"15min":1.6299999999999999},"min":{"1min":0.96599999999999997,"5min":0.498,"15min":0.47399999999999998},"max":{"1min":1.698,"5min":1.998,"15min":26.555},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.389,"15min":1.637},"min":{"1min":1.0960000000000001,"5min":0.66100000000000003,"15min":0.38900000000000001},"max":{"1min":1.764,"5min":2.1040000000000001,"15min":26.503},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.3819999999999999,"15min":1.7490000000000001},"min":{"1min":1.0009999999999999,"5min":0.56399999999999995,"15min":0.51200000000000001},"max":{"1min":1.9079999999999999,"5min":2.4319999999999999,"15min":25.661000000000001},"last":1.4350000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.4610000000000001,"15min":1.74},"min":{"1min":1.3120000000000001,"5min":0.74199999999999999,"15min":0.45800000000000002},"max":{"1min":2.077,"5min":2.3490000000000002,"15min":27.149000000000001},"last":1.49},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.538,"15min":1.7330000000000001},"min":{"1min":1.1930000000000001,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":1.8540000000000001,"5min":2.6589999999999998,"15min":26.349},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4910000000000001,"15min":1.9610000000000001},"min":{"1min":1.0700000000000001,"5min":0.64500000000000002,"15min":0.504},"max":{"1min":2.1909999999999998,"5min":2.7149999999999999,"15min":25.603999999999999},"last":1.548},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.597,"15min":1.843},"min":{"1min":1.1579999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":2.157,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.591,"15min":1.9470000000000001},"min":{"1min":1.343,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1200000000000001,"5min":2.4889999999999999,"15min":24.509},"last":1.649},{"interface":"front","average":{"1min":1.657,"5min":1.5880000000000001,"15min":1.927},"min":{"1min":1.282,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.2229999999999999,"5min":2.3479999999999999,"15min":27.207999999999998},"last":1.8080000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":1376256,"data_stored":593449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.92400000000000004,"15min":1.153},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.28399999999999997},"max":{"1min":1.696,"5min":1.865,"15min":11.699},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0489999999999999,"15min":1.276},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.33700000000000002},"max":{"1min":1.5089999999999999,"5min":2.3860000000000001,"15min":13.930999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.238,"15min":1.385},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.31},"max":{"1min":1.8,"5min":15.557,"15min":16.488},"last":1.274},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1559999999999999,"15min":1.5680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39000000000000001},"max":{"1min":1.968,"5min":2.306,"15min":22.550999999999998},"last":1.4650000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.6879999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.439},"max":{"1min":1.7450000000000001,"5min":1.9790000000000001,"15min":22.471},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3660000000000001,"15min":1.6359999999999999},"min":{"1min":0.54100000000000004,"5min":0.49099999999999999,"15min":0.35699999999999998},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":22.550999999999998},"last":0.82899999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4870000000000001,"15min":1.6240000000000001},"min":{"1min":0.96699999999999997,"5min":0.95699999999999996,"15min":0.60899999999999999},"max":{"1min":1.95,"5min":2.7050000000000001,"15min":8.3620000000000001},"last":1.333},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5960000000000001,"15min":1.7070000000000001},"min":{"1min":0.97399999999999998,"5min":0.69999999999999996,"15min":0.433},"max":{"1min":1.97,"5min":15.723000000000001,"15min":19.236999999999998},"last":1.5960000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.706},"min":{"1min":0.92800000000000005,"5min":0.73799999999999999,"15min":0.57199999999999995},"max":{"1min":1.843,"5min":2.2040000000000002,"15min":21.117000000000001},"last":1.175},{"interface":"front","average":{"1min":1.425,"5min":1.544,"15min":1.758},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.65800000000000003},"max":{"1min":1.8009999999999999,"5min":15.785,"15min":17.138000000000002},"last":1.7569999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.4850000000000001,"15min":1.698},"min":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.72199999999999998},"max":{"1min":2.2759999999999998,"5min":2.298,"15min":11.901999999999999},"last":1.798},{"interface":"front","average":{"1min":1.629,"5min":1.5840000000000001,"15min":1.8069999999999999},"min":{"1min":1.044,"5min":0.86399999999999999,"15min":0.63300000000000001},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":23.420999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.631,"15min":1.8380000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.085,"5min":15.875,"15min":17.010000000000002},"last":1.423},{"interface":"front","average":{"1min":1.591,"5min":1.573,"15min":1.825},"min":{"1min":0.998,"5min":0.91600000000000004,"15min":0.75700000000000001},"max":{"1min":2.508,"5min":2.508,"15min":11.962},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542381,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73308,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298532,"statfs":{"total":96636764160,"available":96561696768,"internally_reserved":0,"allocated":1982464,"data_stored":1190620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":73071041},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.016,"15min":1.468},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":1.355,"5min":4.8380000000000001,"15min":43.459000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.2290000000000001,"15min":1.706},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.956,"5min":4.9779999999999998,"15min":43.627000000000002},"last":1.042}]},{"osd":2,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1919999999999999,"15min":1.4099999999999999},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":5.0570000000000004,"15min":18.861000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.218,"15min":1.5229999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":2.2330000000000001,"5min":5.3550000000000004,"15min":38.893000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0800000000000001,"15min":1.47},"min":{"1min":0.37,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":1.4410000000000001,"5min":4.9050000000000002,"15min":23.687999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.357,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.434},"max":{"1min":1.6080000000000001,"5min":5.1980000000000004,"15min":18.966999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3460000000000001,"15min":1.5129999999999999},"min":{"1min":0.76500000000000001,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.8420000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.496,"15min":1.7030000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.873,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.423}]},{"osd":5,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.508,"15min":1.891},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.792,"5min":5.1589999999999998,"15min":24.611000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.6970000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.56000000000000005},"max":{"1min":1.907,"5min":7.2699999999999996,"15min":20.199000000000002},"last":1.75}]},{"osd":6,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6499999999999999,"15min":2.0510000000000002},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.55900000000000005},"max":{"1min":2.3119999999999998,"5min":6.3339999999999996,"15min":26.998000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.6140000000000001,"15min":1.988},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":2.1669999999999998,"5min":5.9690000000000003,"15min":24.478999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6699999999999999,"15min":1.9139999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.60899999999999999},"max":{"1min":2.2599999999999998,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.657},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.5629999999999999,"15min":1.7310000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.52800000000000002},"max":{"1min":1.8999999999999999,"5min":6.149,"15min":20.279},"last":1.8280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:19:00.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:19:00.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:00.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.295+0000 7f3e40a00640 1 Processor -- start 2024-01-16T15:19:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.295+0000 7f3e40a00640 1 -- start start 2024-01-16T15:19:00.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e40a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e380ce260 0x7f3e380d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e40a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3e380ce6a0 0x7f3e380d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e40a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3815d8b0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e40a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380b91b0 con 0x7f3e380ce260 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e40a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380b2400 con 0x7f3e380ce6a0 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e3d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e380ce260 0x7f3e380d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e3e000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f3e3815d4d0 0x7f3e380cda70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35524/0 (socket says 172.21.15.43:35524) 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e3d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3e380ce260 0x7f3e380d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38402/0 (socket says 172.21.15.43:38402) 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e3e000640 1 -- 172.21.15.43:0/1782988406 learned_addr learned my addr 172.21.15.43:0/1782988406 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:00.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.297+0000 7f3e3ea00640 1 --2- 172.21.15.43:0/1782988406 >> v2:172.21.15.69:3300/0 conn(0x7f3e380ce6a0 0x7f3e380d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.298+0000 7f3e3d600640 1 -- 172.21.15.43:0/1782988406 >> v2:172.21.15.69:3300/0 conn(0x7f3e380ce6a0 msgr2=0x7f3e380d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.298+0000 7f3e3d600640 1 --2- 172.21.15.43:0/1782988406 >> v2:172.21.15.69:3300/0 conn(0x7f3e380ce6a0 0x7f3e380d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.298+0000 7f3e3d600640 1 -- 172.21.15.43:0/1782988406 >> v1:172.21.15.43:6789/0 conn(0x7f3e3815d4d0 legacy=0x7f3e380cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.298+0000 7f3e3cc00640 1 -- 172.21.15.43:0/1782988406 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255812173 0 0) 0x7f3e3815d8b0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.298+0000 7f3e3d600640 1 -- 172.21.15.43:0/1782988406 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380b26e0 con 0x7f3e380ce260 2024-01-16T15:19:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.299+0000 7f3e3d600640 1 --2- 172.21.15.43:0/1782988406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e380ce260 0x7f3e380d1850 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3e380d3cd0 tx=0x7f3e2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e993f13eb93a794b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.299+0000 7f3e3cc00640 1 -- 172.21.15.43:0/1782988406 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031830 con 0x7f3e380ce260 2024-01-16T15:19:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.299+0000 7f3e3cc00640 1 -- 172.21.15.43:0/1782988406 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e20031990 con 0x7f3e380ce260 2024-01-16T15:19:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.299+0000 7f3e3cc00640 1 -- 172.21.15.43:0/1782988406 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031d20 con 0x7f3e380ce260 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 -- 172.21.15.43:0/1782988406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e380ce260 msgr2=0x7f3e380d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 --2- 172.21.15.43:0/1782988406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e380ce260 0x7f3e380d1850 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3e380d3cd0 tx=0x7f3e2002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 -- 172.21.15.43:0/1782988406 shutdown_connections 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 --2- 172.21.15.43:0/1782988406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e380ce260 0x7f3e380d1850 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 --2- 172.21.15.43:0/1782988406 >> v2:172.21.15.69:3300/0 conn(0x7f3e380ce6a0 0x7f3e380d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 -- 172.21.15.43:0/1782988406 >> 172.21.15.43:0/1782988406 conn(0x7f3e38071f90 msgr2=0x7f3e380723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 -- 172.21.15.43:0/1782988406 shutdown_connections 2024-01-16T15:19:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.300+0000 7f3e40a00640 1 -- 172.21.15.43:0/1782988406 wait complete. 2024-01-16T15:19:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.302+0000 7f3e40a00640 1 Processor -- start 2024-01-16T15:19:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.302+0000 7f3e40a00640 1 -- start start 2024-01-16T15:19:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 0x7f3e380cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e380ce6a0 0x7f3e3814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 0x7f3e3814b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380c7d50 con 0x7f3e3815d4d0 2024-01-16T15:19:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380c8030 con 0x7f3e380ce6a0 2024-01-16T15:19:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.303+0000 7f3e40a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380c8310 con 0x7f3e380ce260 2024-01-16T15:19:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 0x7f3e380cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e380ce6a0 0x7f3e3814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 0x7f3e3814b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 0x7f3e380cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:35426/0 (socket says 172.21.15.43:35426) 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3e000640 1 -- 172.21.15.43:0/2026807617 learned_addr learned my addr 172.21.15.43:0/2026807617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 -- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 msgr2=0x7f3e380cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 0x7f3e380cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 -- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e380ce6a0 msgr2=0x7f3e3814b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e380ce6a0 0x7f3e3814b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.304+0000 7f3e3ea00640 1 -- 172.21.15.43:0/2026807617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20009400 con 0x7f3e3815d4d0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e3ea00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 0x7f3e3814b830 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f3e30007d90 tx=0x7f3e3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30018070 con 0x7f3e3815d4d0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3000edd0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30003bb0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e380c85f0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.305+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e380c8df0 con 0x7f3e3815d4d0 2024-01-16T15:19:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.307+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380c8b10 con 0x7f3e3815d4d0 2024-01-16T15:19:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.308+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f3e30014420 con 0x7f3e3815d4d0 2024-01-16T15:19:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.309+0000 7f3e36000640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 0x7f3e1006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.309+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3e30088900 con 0x7f3e3815d4d0 2024-01-16T15:19:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.310+0000 7f3e3e000640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 0x7f3e1006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.310+0000 7f3e3e000640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 0x7f3e1006e860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3e28000f60 tx=0x7f3e28007ea0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.314+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e380c8b10 con 0x7f3e3815d4d0 2024-01-16T15:19:00.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.506+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e380d0260 con 0x7f3e1006c3a0 2024-01-16T15:19:00.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.519+0000 7f3e36000640 1 -- 172.21.15.43:0/2026807617 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287040 (secure 0 0 0) 0x7f3e380d0260 con 0x7f3e1006c3a0 2024-01-16T15:19:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:19:00.515 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 msgr2=0x7f3e1006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 0x7f3e1006e860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3e28000f60 tx=0x7f3e28007ea0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 msgr2=0x7f3e3814b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 0x7f3e3814b830 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f3e30007d90 tx=0x7f3e3000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 shutdown_connections 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3e1006c3a0 0x7f3e1006e860 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e380ce6a0 0x7f3e3814b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e3815d4d0 0x7f3e3814b830 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 --2- 172.21.15.43:0/2026807617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e380ce260 0x7f3e380cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 >> 172.21.15.43:0/2026807617 conn(0x7f3e38071f90 msgr2=0x7f3e380d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 shutdown_connections 2024-01-16T15:19:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.525+0000 7f3e40a00640 1 -- 172.21.15.43:0/2026807617 wait complete. 2024-01-16T15:19:00.535 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2024-01-16T15:18:59.292250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606780,"kb_used_data":12168,"kb_used_omap":12,"kb_used_meta":594483,"kb_avail":754367940,"statfs":{"total":773094113280,"available":772472770560,"internally_reserved":0,"allocated":12460032,"data_stored":6038196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":608751148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.811215"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.533294+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:18:53.533294+0000","last_peered":"2024-01-16T15:18:53.533294+0000","last_clean":"2024-01-16T15:18:53.533294+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:18:53.533294+0000","last_undegraded":"2024-01-16T15:18:53.533294+0000","last_fullsized":"2024-01-16T15:18:53.533294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:47.552196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.520142+0000","last_change":"2024-01-16T15:18:50.514868+0000","last_active":"2024-01-16T15:18:51.520142+0000","last_peered":"2024-01-16T15:18:51.520142+0000","last_clean":"2024-01-16T15:18:51.520142+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:18:51.520142+0000","last_undegraded":"2024-01-16T15:18:51.520142+0000","last_fullsized":"2024-01-16T15:18:51.520142+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:22.704331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.044164+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:18:52.044164+0000","last_peered":"2024-01-16T15:18:52.044164+0000","last_clean":"2024-01-16T15:18:52.044164+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:18:52.044164+0000","last_undegraded":"2024-01-16T15:18:52.044164+0000","last_fullsized":"2024-01-16T15:18:52.044164+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532717+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:18:53.532717+0000","last_peered":"2024-01-16T15:18:53.532717+0000","last_clean":"2024-01-16T15:18:53.532717+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:18:53.532717+0000","last_undegraded":"2024-01-16T15:18:53.532717+0000","last_fullsized":"2024-01-16T15:18:53.532717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:27.895459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.165620+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:18:52.165620+0000","last_peered":"2024-01-16T15:18:52.165620+0000","last_clean":"2024-01-16T15:18:52.165620+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:18:52.165620+0000","last_undegraded":"2024-01-16T15:18:52.165620+0000","last_fullsized":"2024-01-16T15:18:52.165620+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:32.606162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.910621+0000","last_change":"2024-01-16T15:18:50.517889+0000","last_active":"2024-01-16T15:18:52.910621+0000","last_peered":"2024-01-16T15:18:52.910621+0000","last_clean":"2024-01-16T15:18:52.910621+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:18:52.910621+0000","last_undegraded":"2024-01-16T15:18:52.910621+0000","last_fullsized":"2024-01-16T15:18:52.910621+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.518091+0000","last_change":"2024-01-16T15:18:50.506124+0000","last_active":"2024-01-16T15:18:51.518091+0000","last_peered":"2024-01-16T15:18:51.518091+0000","last_clean":"2024-01-16T15:18:51.518091+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:18:51.518091+0000","last_undegraded":"2024-01-16T15:18:51.518091+0000","last_fullsized":"2024-01-16T15:18:51.518091+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043522+0000","last_change":"2024-01-16T15:18:50.513082+0000","last_active":"2024-01-16T15:18:52.043522+0000","last_peered":"2024-01-16T15:18:52.043522+0000","last_clean":"2024-01-16T15:18:52.043522+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:18:52.043522+0000","last_undegraded":"2024-01-16T15:18:52.043522+0000","last_fullsized":"2024-01-16T15:18:52.043522+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.533485+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:18:53.533485+0000","last_peered":"2024-01-16T15:18:53.533485+0000","last_clean":"2024-01-16T15:18:53.533485+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:18:53.533485+0000","last_undegraded":"2024-01-16T15:18:53.533485+0000","last_fullsized":"2024-01-16T15:18:53.533485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:57.207916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.533331+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:18:53.533331+0000","last_peered":"2024-01-16T15:18:53.533331+0000","last_clean":"2024-01-16T15:18:53.533331+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:18:53.533331+0000","last_undegraded":"2024-01-16T15:18:53.533331+0000","last_fullsized":"2024-01-16T15:18:53.533331+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:26.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519583+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:18:51.519583+0000","last_peered":"2024-01-16T15:18:51.519583+0000","last_clean":"2024-01-16T15:18:51.519583+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:18:51.519583+0000","last_undegraded":"2024-01-16T15:18:51.519583+0000","last_fullsized":"2024-01-16T15:18:51.519583+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532775+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:18:53.532775+0000","last_peered":"2024-01-16T15:18:53.532775+0000","last_clean":"2024-01-16T15:18:53.532775+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:18:53.532775+0000","last_undegraded":"2024-01-16T15:18:53.532775+0000","last_fullsized":"2024-01-16T15:18:53.532775+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:28:07.149636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.044511+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:18:52.044511+0000","last_peered":"2024-01-16T15:18:52.044511+0000","last_clean":"2024-01-16T15:18:52.044511+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:18:52.044511+0000","last_undegraded":"2024-01-16T15:18:52.044511+0000","last_fullsized":"2024-01-16T15:18:52.044511+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:35.209083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043542+0000","last_change":"2024-01-16T15:18:50.517177+0000","last_active":"2024-01-16T15:18:52.043542+0000","last_peered":"2024-01-16T15:18:52.043542+0000","last_clean":"2024-01-16T15:18:52.043542+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:18:52.043542+0000","last_undegraded":"2024-01-16T15:18:52.043542+0000","last_fullsized":"2024-01-16T15:18:52.043542+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.910399+0000","last_change":"2024-01-16T15:18:50.518134+0000","last_active":"2024-01-16T15:18:52.910399+0000","last_peered":"2024-01-16T15:18:52.910399+0000","last_clean":"2024-01-16T15:18:52.910399+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:18:52.910399+0000","last_undegraded":"2024-01-16T15:18:52.910399+0000","last_fullsized":"2024-01-16T15:18:52.910399+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:10.936165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.531926+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:18:53.531926+0000","last_peered":"2024-01-16T15:18:53.531926+0000","last_clean":"2024-01-16T15:18:53.531926+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:18:53.531926+0000","last_undegraded":"2024-01-16T15:18:53.531926+0000","last_fullsized":"2024-01-16T15:18:53.531926+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:18.639887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.520241+0000","last_change":"2024-01-16T15:18:50.514080+0000","last_active":"2024-01-16T15:18:51.520241+0000","last_peered":"2024-01-16T15:18:51.520241+0000","last_clean":"2024-01-16T15:18:51.520241+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:18:51.520241+0000","last_undegraded":"2024-01-16T15:18:51.520241+0000","last_fullsized":"2024-01-16T15:18:51.520241+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:18.432791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.516890+0000","last_change":"2024-01-16T15:18:50.512290+0000","last_active":"2024-01-16T15:18:51.516890+0000","last_peered":"2024-01-16T15:18:51.516890+0000","last_clean":"2024-01-16T15:18:51.516890+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:18:51.516890+0000","last_undegraded":"2024-01-16T15:18:51.516890+0000","last_fullsized":"2024-01-16T15:18:51.516890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:03.196990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.044214+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:18:52.044214+0000","last_peered":"2024-01-16T15:18:52.044214+0000","last_clean":"2024-01-16T15:18:52.044214+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:18:52.044214+0000","last_undegraded":"2024-01-16T15:18:52.044214+0000","last_fullsized":"2024-01-16T15:18:52.044214+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.753500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519305+0000","last_change":"2024-01-16T15:18:50.523238+0000","last_active":"2024-01-16T15:18:51.519305+0000","last_peered":"2024-01-16T15:18:51.519305+0000","last_clean":"2024-01-16T15:18:51.519305+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:18:51.519305+0000","last_undegraded":"2024-01-16T15:18:51.519305+0000","last_fullsized":"2024-01-16T15:18:51.519305+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:10.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.520248+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:18:51.520248+0000","last_peered":"2024-01-16T15:18:51.520248+0000","last_clean":"2024-01-16T15:18:51.520248+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:18:51.520248+0000","last_undegraded":"2024-01-16T15:18:51.520248+0000","last_fullsized":"2024-01-16T15:18:51.520248+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:04.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.513532+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:18:51.513532+0000","last_peered":"2024-01-16T15:18:51.513532+0000","last_clean":"2024-01-16T15:18:51.513532+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:18:51.513532+0000","last_undegraded":"2024-01-16T15:18:51.513532+0000","last_fullsized":"2024-01-16T15:18:51.513532+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.518540+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:18:51.518540+0000","last_peered":"2024-01-16T15:18:51.518540+0000","last_clean":"2024-01-16T15:18:51.518540+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:18:51.518540+0000","last_undegraded":"2024-01-16T15:18:51.518540+0000","last_fullsized":"2024-01-16T15:18:51.518540+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:49.070110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.512793+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:18:51.512793+0000","last_peered":"2024-01-16T15:18:51.512793+0000","last_clean":"2024-01-16T15:18:51.512793+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:18:51.512793+0000","last_undegraded":"2024-01-16T15:18:51.512793+0000","last_fullsized":"2024-01-16T15:18:51.512793+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:09.748440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.520141+0000","last_change":"2024-01-16T15:18:50.513030+0000","last_active":"2024-01-16T15:18:51.520141+0000","last_peered":"2024-01-16T15:18:51.520141+0000","last_clean":"2024-01-16T15:18:51.520141+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:18:51.520141+0000","last_undegraded":"2024-01-16T15:18:51.520141+0000","last_fullsized":"2024-01-16T15:18:51.520141+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:54.334893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.533372+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:18:53.533372+0000","last_peered":"2024-01-16T15:18:53.533372+0000","last_clean":"2024-01-16T15:18:53.533372+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:18:53.533372+0000","last_undegraded":"2024-01-16T15:18:53.533372+0000","last_fullsized":"2024-01-16T15:18:53.533372+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:49.522744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.911283+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:18:52.911283+0000","last_peered":"2024-01-16T15:18:52.911283+0000","last_clean":"2024-01-16T15:18:52.911283+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:18:52.911283+0000","last_undegraded":"2024-01-16T15:18:52.911283+0000","last_fullsized":"2024-01-16T15:18:52.911283+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.513875+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:18:51.513875+0000","last_peered":"2024-01-16T15:18:51.513875+0000","last_clean":"2024-01-16T15:18:51.513875+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:18:51.513875+0000","last_undegraded":"2024-01-16T15:18:51.513875+0000","last_fullsized":"2024-01-16T15:18:51.513875+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519648+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:18:51.519648+0000","last_peered":"2024-01-16T15:18:51.519648+0000","last_clean":"2024-01-16T15:18:51.519648+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:18:51.519648+0000","last_undegraded":"2024-01-16T15:18:51.519648+0000","last_fullsized":"2024-01-16T15:18:51.519648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:38.948857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.518653+0000","last_change":"2024-01-16T15:18:50.491931+0000","last_active":"2024-01-16T15:18:51.518653+0000","last_peered":"2024-01-16T15:18:51.518653+0000","last_clean":"2024-01-16T15:18:51.518653+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:18:51.518653+0000","last_undegraded":"2024-01-16T15:18:51.518653+0000","last_fullsized":"2024-01-16T15:18:51.518653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532374+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:18:53.532374+0000","last_peered":"2024-01-16T15:18:53.532374+0000","last_clean":"2024-01-16T15:18:53.532374+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:18:53.532374+0000","last_undegraded":"2024-01-16T15:18:53.532374+0000","last_fullsized":"2024-01-16T15:18:53.532374+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:03.259116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.910491+0000","last_change":"2024-01-16T15:18:50.510830+0000","last_active":"2024-01-16T15:18:52.910491+0000","last_peered":"2024-01-16T15:18:52.910491+0000","last_clean":"2024-01-16T15:18:52.910491+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:18:52.910491+0000","last_undegraded":"2024-01-16T15:18:52.910491+0000","last_fullsized":"2024-01-16T15:18:52.910491+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:30.055863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.911374+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:18:52.911374+0000","last_peered":"2024-01-16T15:18:52.911374+0000","last_clean":"2024-01-16T15:18:52.911374+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:18:52.911374+0000","last_undegraded":"2024-01-16T15:18:52.911374+0000","last_fullsized":"2024-01-16T15:18:52.911374+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.530191+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:18:50.530191+0000","last_peered":"2024-01-16T15:18:50.530191+0000","last_clean":"2024-01-16T15:18:50.530191+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:18:50.530191+0000","last_undegraded":"2024-01-16T15:18:50.530191+0000","last_fullsized":"2024-01-16T15:18:50.530191+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.517439+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:18:51.517439+0000","last_peered":"2024-01-16T15:18:51.517439+0000","last_clean":"2024-01-16T15:18:51.517439+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:18:51.517439+0000","last_undegraded":"2024-01-16T15:18:51.517439+0000","last_fullsized":"2024-01-16T15:18:51.517439+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:37.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.529722+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:18:50.529722+0000","last_peered":"2024-01-16T15:18:50.529722+0000","last_clean":"2024-01-16T15:18:50.529722+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:18:50.529722+0000","last_undegraded":"2024-01-16T15:18:50.529722+0000","last_fullsized":"2024-01-16T15:18:50.529722+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:51.215798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.911059+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:18:52.911059+0000","last_peered":"2024-01-16T15:18:52.911059+0000","last_clean":"2024-01-16T15:18:52.911059+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:18:52.911059+0000","last_undegraded":"2024-01-16T15:18:52.911059+0000","last_fullsized":"2024-01-16T15:18:52.911059+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.535659+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:18:50.535659+0000","last_peered":"2024-01-16T15:18:50.535659+0000","last_clean":"2024-01-16T15:18:50.535659+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:18:50.535659+0000","last_undegraded":"2024-01-16T15:18:50.535659+0000","last_fullsized":"2024-01-16T15:18:50.535659+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.044549+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:18:52.044549+0000","last_peered":"2024-01-16T15:18:52.044549+0000","last_clean":"2024-01-16T15:18:52.044549+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:18:52.044549+0000","last_undegraded":"2024-01-16T15:18:52.044549+0000","last_fullsized":"2024-01-16T15:18:52.044549+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:37.406427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.535624+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:18:50.535624+0000","last_peered":"2024-01-16T15:18:50.535624+0000","last_clean":"2024-01-16T15:18:50.535624+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:18:50.535624+0000","last_undegraded":"2024-01-16T15:18:50.535624+0000","last_fullsized":"2024-01-16T15:18:50.535624+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532415+0000","last_change":"2024-01-16T15:18:50.510180+0000","last_active":"2024-01-16T15:18:53.532415+0000","last_peered":"2024-01-16T15:18:53.532415+0000","last_clean":"2024-01-16T15:18:53.532415+0000","last_became_active":"2024-01-16T15:18:50.509888+0000","last_became_peered":"2024-01-16T15:18:50.509888+0000","last_unstale":"2024-01-16T15:18:53.532415+0000","last_undegraded":"2024-01-16T15:18:53.532415+0000","last_fullsized":"2024-01-16T15:18:53.532415+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:06.436719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.526080+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:18:50.526080+0000","last_peered":"2024-01-16T15:18:50.526080+0000","last_clean":"2024-01-16T15:18:50.526080+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:18:50.526080+0000","last_undegraded":"2024-01-16T15:18:50.526080+0000","last_fullsized":"2024-01-16T15:18:50.526080+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:31.593211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519694+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:18:51.519694+0000","last_peered":"2024-01-16T15:18:51.519694+0000","last_clean":"2024-01-16T15:18:51.519694+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:18:51.519694+0000","last_undegraded":"2024-01-16T15:18:51.519694+0000","last_fullsized":"2024-01-16T15:18:51.519694+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:59.415403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.531947+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:18:50.531947+0000","last_peered":"2024-01-16T15:18:50.531947+0000","last_clean":"2024-01-16T15:18:50.531947+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:18:50.531947+0000","last_undegraded":"2024-01-16T15:18:50.531947+0000","last_fullsized":"2024-01-16T15:18:50.531947+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:57.932540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.531902+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:18:53.531902+0000","last_peered":"2024-01-16T15:18:53.531902+0000","last_clean":"2024-01-16T15:18:53.531902+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:18:53.531902+0000","last_undegraded":"2024-01-16T15:18:53.531902+0000","last_fullsized":"2024-01-16T15:18:53.531902+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:45.776442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.503651+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:18:50.503651+0000","last_peered":"2024-01-16T15:18:50.503651+0000","last_clean":"2024-01-16T15:18:50.503651+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:18:50.503651+0000","last_undegraded":"2024-01-16T15:18:50.503651+0000","last_fullsized":"2024-01-16T15:18:50.503651+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:07.440889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.518139+0000","last_change":"2024-01-16T15:18:50.506340+0000","last_active":"2024-01-16T15:18:51.518139+0000","last_peered":"2024-01-16T15:18:51.518139+0000","last_clean":"2024-01-16T15:18:51.518139+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:18:51.518139+0000","last_undegraded":"2024-01-16T15:18:51.518139+0000","last_fullsized":"2024-01-16T15:18:51.518139+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557186+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:18:50.557186+0000","last_peered":"2024-01-16T15:18:50.557186+0000","last_clean":"2024-01-16T15:18:50.557186+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:18:50.557186+0000","last_undegraded":"2024-01-16T15:18:50.557186+0000","last_fullsized":"2024-01-16T15:18:50.557186+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519598+0000","last_change":"2024-01-16T15:18:50.507182+0000","last_active":"2024-01-16T15:18:51.519598+0000","last_peered":"2024-01-16T15:18:51.519598+0000","last_clean":"2024-01-16T15:18:51.519598+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:18:51.519598+0000","last_undegraded":"2024-01-16T15:18:51.519598+0000","last_fullsized":"2024-01-16T15:18:51.519598+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.500764+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:18:50.500764+0000","last_peered":"2024-01-16T15:18:50.500764+0000","last_clean":"2024-01-16T15:18:50.500764+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:18:50.500764+0000","last_undegraded":"2024-01-16T15:18:50.500764+0000","last_fullsized":"2024-01-16T15:18:50.500764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.517593+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:18:51.517593+0000","last_peered":"2024-01-16T15:18:51.517593+0000","last_clean":"2024-01-16T15:18:51.517593+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:18:51.517593+0000","last_undegraded":"2024-01-16T15:18:51.517593+0000","last_fullsized":"2024-01-16T15:18:51.517593+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:14.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.484869+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:18:50.484869+0000","last_peered":"2024-01-16T15:18:50.484869+0000","last_clean":"2024-01-16T15:18:50.484869+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:18:50.484869+0000","last_undegraded":"2024-01-16T15:18:50.484869+0000","last_fullsized":"2024-01-16T15:18:50.484869+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:31.105453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.514061+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:18:51.514061+0000","last_peered":"2024-01-16T15:18:51.514061+0000","last_clean":"2024-01-16T15:18:51.514061+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:18:51.514061+0000","last_undegraded":"2024-01-16T15:18:51.514061+0000","last_fullsized":"2024-01-16T15:18:51.514061+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557354+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:18:50.557354+0000","last_peered":"2024-01-16T15:18:50.557354+0000","last_clean":"2024-01-16T15:18:50.557354+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:18:50.557354+0000","last_undegraded":"2024-01-16T15:18:50.557354+0000","last_fullsized":"2024-01-16T15:18:50.557354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519083+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:18:51.519083+0000","last_peered":"2024-01-16T15:18:51.519083+0000","last_clean":"2024-01-16T15:18:51.519083+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:18:51.519083+0000","last_undegraded":"2024-01-16T15:18:51.519083+0000","last_fullsized":"2024-01-16T15:18:51.519083+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557341+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:18:50.557341+0000","last_peered":"2024-01-16T15:18:50.557341+0000","last_clean":"2024-01-16T15:18:50.557341+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:18:50.557341+0000","last_undegraded":"2024-01-16T15:18:50.557341+0000","last_fullsized":"2024-01-16T15:18:50.557341+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.514166+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:18:51.514166+0000","last_peered":"2024-01-16T15:18:51.514166+0000","last_clean":"2024-01-16T15:18:51.514166+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:18:51.514166+0000","last_undegraded":"2024-01-16T15:18:51.514166+0000","last_fullsized":"2024-01-16T15:18:51.514166+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.529717+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:18:50.529717+0000","last_peered":"2024-01-16T15:18:50.529717+0000","last_clean":"2024-01-16T15:18:50.529717+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:18:50.529717+0000","last_undegraded":"2024-01-16T15:18:50.529717+0000","last_fullsized":"2024-01-16T15:18:50.529717+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.513239+0000","last_change":"2024-01-16T15:18:50.517675+0000","last_active":"2024-01-16T15:18:51.513239+0000","last_peered":"2024-01-16T15:18:51.513239+0000","last_clean":"2024-01-16T15:18:51.513239+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:18:51.513239+0000","last_undegraded":"2024-01-16T15:18:51.513239+0000","last_fullsized":"2024-01-16T15:18:51.513239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.492828+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:18:50.492828+0000","last_peered":"2024-01-16T15:18:50.492828+0000","last_clean":"2024-01-16T15:18:50.492828+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:18:50.492828+0000","last_undegraded":"2024-01-16T15:18:50.492828+0000","last_fullsized":"2024-01-16T15:18:50.492828+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:28.315056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.512971+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:18:51.512971+0000","last_peered":"2024-01-16T15:18:51.512971+0000","last_clean":"2024-01-16T15:18:51.512971+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:18:51.512971+0000","last_undegraded":"2024-01-16T15:18:51.512971+0000","last_fullsized":"2024-01-16T15:18:51.512971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.537537+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:18:50.537537+0000","last_peered":"2024-01-16T15:18:50.537537+0000","last_clean":"2024-01-16T15:18:50.537537+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:18:50.537537+0000","last_undegraded":"2024-01-16T15:18:50.537537+0000","last_fullsized":"2024-01-16T15:18:50.537537+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:58.071689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.165877+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:18:52.165877+0000","last_peered":"2024-01-16T15:18:52.165877+0000","last_clean":"2024-01-16T15:18:52.165877+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:18:52.165877+0000","last_undegraded":"2024-01-16T15:18:52.165877+0000","last_fullsized":"2024-01-16T15:18:52.165877+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:40.855053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557423+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:18:50.557423+0000","last_peered":"2024-01-16T15:18:50.557423+0000","last_clean":"2024-01-16T15:18:50.557423+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:18:50.557423+0000","last_undegraded":"2024-01-16T15:18:50.557423+0000","last_fullsized":"2024-01-16T15:18:50.557423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.911471+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:18:52.911471+0000","last_peered":"2024-01-16T15:18:52.911471+0000","last_clean":"2024-01-16T15:18:52.911471+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:18:52.911471+0000","last_undegraded":"2024-01-16T15:18:52.911471+0000","last_fullsized":"2024-01-16T15:18:52.911471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.535507+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:18:50.535507+0000","last_peered":"2024-01-16T15:18:50.535507+0000","last_clean":"2024-01-16T15:18:50.535507+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:18:50.535507+0000","last_undegraded":"2024-01-16T15:18:50.535507+0000","last_fullsized":"2024-01-16T15:18:50.535507+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043800+0000","last_change":"2024-01-16T15:18:50.517233+0000","last_active":"2024-01-16T15:18:52.043800+0000","last_peered":"2024-01-16T15:18:52.043800+0000","last_clean":"2024-01-16T15:18:52.043800+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:18:52.043800+0000","last_undegraded":"2024-01-16T15:18:52.043800+0000","last_fullsized":"2024-01-16T15:18:52.043800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533667+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:18:50.533667+0000","last_peered":"2024-01-16T15:18:50.533667+0000","last_clean":"2024-01-16T15:18:50.533667+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:18:50.533667+0000","last_undegraded":"2024-01-16T15:18:50.533667+0000","last_fullsized":"2024-01-16T15:18:50.533667+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:04.731539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.044702+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:18:52.044702+0000","last_peered":"2024-01-16T15:18:52.044702+0000","last_clean":"2024-01-16T15:18:52.044702+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:18:52.044702+0000","last_undegraded":"2024-01-16T15:18:52.044702+0000","last_fullsized":"2024-01-16T15:18:52.044702+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:16.245190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.486645+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:18:50.486645+0000","last_peered":"2024-01-16T15:18:50.486645+0000","last_clean":"2024-01-16T15:18:50.486645+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:18:50.486645+0000","last_undegraded":"2024-01-16T15:18:50.486645+0000","last_fullsized":"2024-01-16T15:18:50.486645+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043546+0000","last_change":"2024-01-16T15:18:50.521667+0000","last_active":"2024-01-16T15:18:52.043546+0000","last_peered":"2024-01-16T15:18:52.043546+0000","last_clean":"2024-01-16T15:18:52.043546+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:18:52.043546+0000","last_undegraded":"2024-01-16T15:18:52.043546+0000","last_fullsized":"2024-01-16T15:18:52.043546+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.527428+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:18:50.527428+0000","last_peered":"2024-01-16T15:18:50.527428+0000","last_clean":"2024-01-16T15:18:50.527428+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:18:50.527428+0000","last_undegraded":"2024-01-16T15:18:50.527428+0000","last_fullsized":"2024-01-16T15:18:50.527428+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532415+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:18:53.532415+0000","last_peered":"2024-01-16T15:18:53.532415+0000","last_clean":"2024-01-16T15:18:53.532415+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:18:53.532415+0000","last_undegraded":"2024-01-16T15:18:53.532415+0000","last_fullsized":"2024-01-16T15:18:53.532415+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:59.502905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.502556+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:18:50.502556+0000","last_peered":"2024-01-16T15:18:50.502556+0000","last_clean":"2024-01-16T15:18:50.502556+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:18:50.502556+0000","last_undegraded":"2024-01-16T15:18:50.502556+0000","last_fullsized":"2024-01-16T15:18:50.502556+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:30.184973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043929+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:18:52.043929+0000","last_peered":"2024-01-16T15:18:52.043929+0000","last_clean":"2024-01-16T15:18:52.043929+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:18:52.043929+0000","last_undegraded":"2024-01-16T15:18:52.043929+0000","last_fullsized":"2024-01-16T15:18:52.043929+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.507950+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:18:50.507950+0000","last_peered":"2024-01-16T15:18:50.507950+0000","last_clean":"2024-01-16T15:18:50.507950+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:18:50.507950+0000","last_undegraded":"2024-01-16T15:18:50.507950+0000","last_fullsized":"2024-01-16T15:18:50.507950+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.512917+0000","last_change":"2024-01-16T15:18:50.516592+0000","last_active":"2024-01-16T15:18:51.512917+0000","last_peered":"2024-01-16T15:18:51.512917+0000","last_clean":"2024-01-16T15:18:51.512917+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:18:51.512917+0000","last_undegraded":"2024-01-16T15:18:51.512917+0000","last_fullsized":"2024-01-16T15:18:51.512917+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:17.446551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557197+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:18:50.557197+0000","last_peered":"2024-01-16T15:18:50.557197+0000","last_clean":"2024-01-16T15:18:50.557197+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:18:50.557197+0000","last_undegraded":"2024-01-16T15:18:50.557197+0000","last_fullsized":"2024-01-16T15:18:50.557197+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.516824+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:18:51.516824+0000","last_peered":"2024-01-16T15:18:51.516824+0000","last_clean":"2024-01-16T15:18:51.516824+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:18:51.516824+0000","last_undegraded":"2024-01-16T15:18:51.516824+0000","last_fullsized":"2024-01-16T15:18:51.516824+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:09.913343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.533671+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:18:50.533671+0000","last_peered":"2024-01-16T15:18:50.533671+0000","last_clean":"2024-01-16T15:18:50.533671+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:18:50.533671+0000","last_undegraded":"2024-01-16T15:18:50.533671+0000","last_fullsized":"2024-01-16T15:18:50.533671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:17.281805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.533641+0000","last_change":"2024-01-16T15:18:50.522199+0000","last_active":"2024-01-16T15:18:53.533641+0000","last_peered":"2024-01-16T15:18:53.533641+0000","last_clean":"2024-01-16T15:18:53.533641+0000","last_became_active":"2024-01-16T15:18:50.521902+0000","last_became_peered":"2024-01-16T15:18:50.521902+0000","last_unstale":"2024-01-16T15:18:53.533641+0000","last_undegraded":"2024-01-16T15:18:53.533641+0000","last_fullsized":"2024-01-16T15:18:53.533641+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:19.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.516939+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:18:51.516939+0000","last_peered":"2024-01-16T15:18:51.516939+0000","last_clean":"2024-01-16T15:18:51.516939+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:18:51.516939+0000","last_undegraded":"2024-01-16T15:18:51.516939+0000","last_fullsized":"2024-01-16T15:18:51.516939+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:07.461774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532874+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:18:53.532874+0000","last_peered":"2024-01-16T15:18:53.532874+0000","last_clean":"2024-01-16T15:18:53.532874+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:18:53.532874+0000","last_undegraded":"2024-01-16T15:18:53.532874+0000","last_fullsized":"2024-01-16T15:18:53.532874+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:49.736642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557365+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:18:50.557365+0000","last_peered":"2024-01-16T15:18:50.557365+0000","last_clean":"2024-01-16T15:18:50.557365+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:18:50.557365+0000","last_undegraded":"2024-01-16T15:18:50.557365+0000","last_fullsized":"2024-01-16T15:18:50.557365+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.516690+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:18:51.516690+0000","last_peered":"2024-01-16T15:18:51.516690+0000","last_clean":"2024-01-16T15:18:51.516690+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:18:51.516690+0000","last_undegraded":"2024-01-16T15:18:51.516690+0000","last_fullsized":"2024-01-16T15:18:51.516690+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:27.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.485849+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:18:50.485849+0000","last_peered":"2024-01-16T15:18:50.485849+0000","last_clean":"2024-01-16T15:18:50.485849+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:18:50.485849+0000","last_undegraded":"2024-01-16T15:18:50.485849+0000","last_fullsized":"2024-01-16T15:18:50.485849+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.618165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.519125+0000","last_change":"2024-01-16T15:18:50.516022+0000","last_active":"2024-01-16T15:18:51.519125+0000","last_peered":"2024-01-16T15:18:51.519125+0000","last_clean":"2024-01-16T15:18:51.519125+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:18:51.519125+0000","last_undegraded":"2024-01-16T15:18:51.519125+0000","last_fullsized":"2024-01-16T15:18:51.519125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.486103+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:18:50.486103+0000","last_peered":"2024-01-16T15:18:50.486103+0000","last_clean":"2024-01-16T15:18:50.486103+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:18:50.486103+0000","last_undegraded":"2024-01-16T15:18:50.486103+0000","last_fullsized":"2024-01-16T15:18:50.486103+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:39.842102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.520476+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:18:51.520476+0000","last_peered":"2024-01-16T15:18:51.520476+0000","last_clean":"2024-01-16T15:18:51.520476+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:18:51.520476+0000","last_undegraded":"2024-01-16T15:18:51.520476+0000","last_fullsized":"2024-01-16T15:18:51.520476+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:03.949006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.508271+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:18:50.508271+0000","last_peered":"2024-01-16T15:18:50.508271+0000","last_clean":"2024-01-16T15:18:50.508271+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:18:50.508271+0000","last_undegraded":"2024-01-16T15:18:50.508271+0000","last_fullsized":"2024-01-16T15:18:50.508271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:26.402495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:52.043924+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:18:52.043924+0000","last_peered":"2024-01-16T15:18:52.043924+0000","last_clean":"2024-01-16T15:18:52.043924+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:18:52.043924+0000","last_undegraded":"2024-01-16T15:18:52.043924+0000","last_fullsized":"2024-01-16T15:18:52.043924+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.500121+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:18:50.500121+0000","last_peered":"2024-01-16T15:18:50.500121+0000","last_clean":"2024-01-16T15:18:50.500121+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:18:50.500121+0000","last_undegraded":"2024-01-16T15:18:50.500121+0000","last_fullsized":"2024-01-16T15:18:50.500121+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:53.532672+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:18:53.532672+0000","last_peered":"2024-01-16T15:18:53.532672+0000","last_clean":"2024-01-16T15:18:53.532672+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:18:53.532672+0000","last_undegraded":"2024-01-16T15:18:53.532672+0000","last_fullsized":"2024-01-16T15:18:53.532672+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:01.372900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.557212+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:18:50.557212+0000","last_peered":"2024-01-16T15:18:50.557212+0000","last_clean":"2024-01-16T15:18:50.557212+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:18:50.557212+0000","last_undegraded":"2024-01-16T15:18:50.557212+0000","last_fullsized":"2024-01-16T15:18:50.557212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-16T15:18:51.513065+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:18:51.513065+0000","last_peered":"2024-01-16T15:18:51.513065+0000","last_clean":"2024-01-16T15:18:51.513065+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:18:51.513065+0000","last_undegraded":"2024-01-16T15:18:51.513065+0000","last_fullsized":"2024-01-16T15:18:51.513065+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.487416+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:18:50.487416+0000","last_peered":"2024-01-16T15:18:50.487416+0000","last_clean":"2024-01-16T15:18:50.487416+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:18:50.487416+0000","last_undegraded":"2024-01-16T15:18:50.487416+0000","last_fullsized":"2024-01-16T15:18:50.487416+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:24.899317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":246,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:18:50.507369+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:18:50.507369+0000","last_peered":"2024-01-16T15:18:50.507369+0000","last_clean":"2024-01-16T15:18:50.507369+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:18:50.507369+0000","last_undegraded":"2024-01-16T15:18:50.507369+0000","last_fullsized":"2024-01-16T15:18:50.507369+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73328,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298512,"statfs":{"total":96636764160,"available":96561676288,"internally_reserved":0,"allocated":2002944,"data_stored":1197564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.349,"15min":1.5429999999999999},"min":{"1min":0.72099999999999997,"5min":0.63500000000000001,"15min":0.28199999999999997},"max":{"1min":1.768,"5min":5.2690000000000001,"15min":27.469000000000001},"last":1.355},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4410000000000001,"15min":1.6859999999999999},"min":{"1min":0.86099999999999999,"5min":0.79500000000000004,"15min":0.47099999999999997},"max":{"1min":1.651,"5min":6.5599999999999996,"15min":25.503},"last":1.462}]},{"osd":1,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.139,"15min":1.2689999999999999},"min":{"1min":0.77600000000000002,"5min":0.55100000000000005,"15min":0.34499999999999997},"max":{"1min":1.712,"5min":1.742,"15min":6.718},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2070000000000001,"15min":1.3520000000000001},"min":{"1min":0.85499999999999998,"5min":0.75700000000000001,"15min":0.373},"max":{"1min":1.7669999999999999,"5min":2.0110000000000001,"15min":15.452},"last":1.1910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.4530000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.48299999999999998},"max":{"1min":1.7709999999999999,"5min":11.536,"15min":11.536},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.5409999999999999,"15min":1.5629999999999999},"min":{"1min":1.0720000000000001,"5min":0.79800000000000004,"15min":0.504},"max":{"1min":1.978,"5min":11.576000000000001,"15min":25.478000000000002},"last":1.6020000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.222,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.67000000000000004,"15min":0.40899999999999997},"max":{"1min":1.593,"5min":2.133,"15min":13.907999999999999},"last":1.143},{"interface":"front","average":{"1min":1.357,"5min":1.365,"15min":1.5029999999999999},"min":{"1min":1.022,"5min":0.89300000000000002,"15min":0.62},"max":{"1min":1.8580000000000001,"5min":2.1960000000000002,"15min":8.6999999999999993},"last":1.4259999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.341,"15min":1.452},"min":{"1min":0.89200000000000002,"5min":0.85299999999999998,"15min":0.376},"max":{"1min":1.7829999999999999,"5min":6.7599999999999998,"15min":25.401},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3280000000000001,"15min":1.472},"min":{"1min":0.84999999999999998,"5min":0.59599999999999997,"15min":0.45000000000000001},"max":{"1min":2.069,"5min":2.069,"15min":10.134},"last":1.3009999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3640000000000001,"15min":1.472},"min":{"1min":0.96399999999999997,"5min":0.69399999999999995,"15min":0.432},"max":{"1min":1.7310000000000001,"5min":2.1259999999999999,"15min":12.528},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.544,"15min":1.639},"min":{"1min":1.0640000000000001,"5min":0.86599999999999999,"15min":0.41399999999999998},"max":{"1min":1.9159999999999999,"5min":6.8719999999999999,"15min":26.327000000000002},"last":1.2370000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.6759999999999999,"15min":1.734},"min":{"1min":1.3640000000000001,"5min":1.087,"15min":0.45200000000000001},"max":{"1min":2.0870000000000002,"5min":6.9690000000000003,"15min":25.451000000000001},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.4930000000000001,"15min":1.605},"min":{"1min":0.92400000000000004,"5min":0.89800000000000002,"15min":0.44700000000000001},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":12.577999999999999},"last":1.5569999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542381,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1404928,"data_stored":606650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.452,"15min":1.744},"min":{"1min":0.752,"5min":0.71299999999999997,"15min":0.35399999999999998},"max":{"1min":1.53,"5min":29.77,"15min":30.164999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.292,"5min":1.6870000000000001,"15min":1.8779999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.48899999999999999},"max":{"1min":1.5780000000000001,"5min":36.405999999999999,"15min":36.405999999999999},"last":1.179}]},{"osd":1,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1850000000000001,"15min":1.508},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.47899999999999998},"max":{"1min":1.52,"5min":5.2859999999999996,"15min":28.122},"last":1.083},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.2549999999999999,"15min":1.298},"min":{"1min":0.78200000000000003,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":1.548,"5min":10.494,"15min":10.494},"last":1.0229999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.347,"15min":1.734},"min":{"1min":1,"5min":0.76900000000000002,"15min":0.501},"max":{"1min":1.845,"5min":5.0549999999999997,"15min":26.109000000000002},"last":1.385},{"interface":"front","average":{"1min":1.397,"5min":1.51,"15min":1.893},"min":{"1min":1.0329999999999999,"5min":1.016,"15min":0.67100000000000004},"max":{"1min":2.0449999999999999,"5min":5.1459999999999999,"15min":26.161000000000001},"last":1.464}]},{"osd":3,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3029999999999999,"15min":1.615},"min":{"1min":1.1060000000000001,"5min":0.68500000000000005,"15min":0.44700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8079999999999998,"15min":17.75},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.681,"15min":1.728},"min":{"1min":1.0349999999999999,"5min":0.77100000000000002,"15min":0.55700000000000005},"max":{"1min":2.1789999999999998,"5min":21.456,"15min":21.456},"last":1.3280000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3540000000000001,"15min":1.5069999999999999},"min":{"1min":0.91400000000000003,"5min":0.82099999999999995,"15min":0.45000000000000001},"max":{"1min":1.6699999999999999,"5min":5.3310000000000004,"15min":15.571},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.325,"15min":1.5469999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.42299999999999999},"max":{"1min":1.639,"5min":3.6160000000000001,"15min":23.667999999999999},"last":1.502}]},{"osd":5,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.4350000000000001,"15min":1.4890000000000001},"min":{"1min":0.83499999999999996,"5min":0.73699999999999999,"15min":0.42299999999999999},"max":{"1min":1.982,"5min":13.949999999999999,"15min":23.137},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.6100000000000001,"15min":1.6970000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":0.47099999999999997},"max":{"1min":1.714,"5min":11.298,"15min":25.684000000000001},"last":1.4350000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.6020000000000001,"15min":1.552},"min":{"1min":0.754,"5min":0.71499999999999997,"15min":0.47099999999999997},"max":{"1min":2.1000000000000001,"5min":14.037000000000001,"15min":14.037000000000001},"last":1.677},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.6240000000000001,"15min":1.724},"min":{"1min":0.95899999999999996,"5min":0.92300000000000004,"15min":0.499},"max":{"1min":2.1659999999999999,"5min":4.9829999999999997,"15min":11.788},"last":1.6160000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542382,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64428,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307412,"statfs":{"total":96636764160,"available":96570789888,"internally_reserved":0,"allocated":1409024,"data_stored":607184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.099,"15min":1.577},"min":{"1min":0.74199999999999999,"5min":0.61399999999999999,"15min":0.41299999999999998},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":26.228999999999999},"last":1.3},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1379999999999999,"15min":1.8360000000000001},"min":{"1min":0.74299999999999999,"5min":0.73799999999999999,"15min":0.34799999999999998},"max":{"1min":2.5609999999999999,"5min":2.5609999999999999,"15min":56.011000000000003},"last":0.93000000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.145,"15min":1.365},"min":{"1min":0.65500000000000003,"5min":0.65200000000000002,"15min":0.40999999999999998},"max":{"1min":1.6040000000000001,"5min":2.2450000000000001,"15min":16.687999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.236,"15min":1.8540000000000001},"min":{"1min":0.79100000000000004,"5min":0.745,"15min":0.40999999999999998},"max":{"1min":1.569,"5min":4.8499999999999996,"15min":67.930000000000007},"last":1.155}]},{"osd":2,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.167,"15min":1.524},"min":{"1min":0.752,"5min":0.61099999999999999,"15min":0.47099999999999997},"max":{"1min":2.4340000000000002,"5min":2.4340000000000002,"15min":13.193},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.286,"15min":1.498},"min":{"1min":0.82699999999999996,"5min":0.78400000000000003,"15min":0.45800000000000002},"max":{"1min":3.282,"5min":3.9590000000000001,"15min":16.721},"last":1.363}]},{"osd":3,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.389,"15min":1.7110000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.57999999999999996},"max":{"1min":1.716,"5min":4.8010000000000002,"15min":22.382000000000001},"last":1.673},{"interface":"front","average":{"1min":1.365,"5min":1.3680000000000001,"15min":1.758},"min":{"1min":0.871,"5min":0.72399999999999998,"15min":0.51200000000000001},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":14.685},"last":1.6180000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.274,"15min":1.3999999999999999},"min":{"1min":0.86599999999999999,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":16.382999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.369,"15min":1.53},"min":{"1min":0.84299999999999997,"5min":0.502,"15min":0.379},"max":{"1min":1.7749999999999999,"5min":4.6580000000000004,"15min":27.006},"last":1.5489999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.3640000000000001,"15min":1.571},"min":{"1min":1.0329999999999999,"5min":0.52400000000000002,"15min":0.39700000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":13.263999999999999},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.5149999999999999,"15min":1.6220000000000001},"min":{"1min":1.1379999999999999,"5min":0.629,"15min":0.505},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":9.5269999999999992},"last":1.6539999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5109999999999999,"15min":1.7509999999999999},"min":{"1min":0.88600000000000001,"5min":0.59099999999999997,"15min":0.45000000000000001},"max":{"1min":1.9159999999999999,"5min":4.742,"15min":35.127000000000002},"last":1.776},{"interface":"front","average":{"1min":1.575,"5min":1.54,"15min":1.734},"min":{"1min":1.155,"5min":0.61099999999999999,"15min":0.42999999999999999},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":13.491},"last":1.819}]}]},{"osd":4,"up_from":14,"seq":60129542381,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68584,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303256,"statfs":{"total":96636764160,"available":96566534144,"internally_reserved":0,"allocated":1400832,"data_stored":606580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1619999999999999,"15min":1.244},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6180000000000001,"5min":6.4240000000000004,"15min":12.587999999999999},"last":1.008},{"interface":"front","average":{"1min":1.095,"5min":1.123,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.679,"5min":1.958,"15min":60.106000000000002},"last":1.5680000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.629,"15min":1.79},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51100000000000001},"max":{"1min":1.738,"5min":24.399999999999999,"15min":24.399999999999999},"last":1.244},{"interface":"front","average":{"1min":1.24,"5min":1.611,"15min":1.627},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57099999999999995},"max":{"1min":1.7290000000000001,"5min":24.456,"15min":24.456},"last":1.4299999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.3029999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.52800000000000002},"max":{"1min":1.6299999999999999,"5min":2.1699999999999999,"15min":9.7140000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2889999999999999,"15min":1.6970000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.52400000000000002},"max":{"1min":1.6479999999999999,"5min":1.855,"15min":35.024999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3300000000000001,"15min":1.556},"min":{"1min":0.503,"5min":0.503,"15min":0.47799999999999998},"max":{"1min":1.6499999999999999,"5min":6.4770000000000003,"15min":32.133000000000003},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":6.4589999999999996,"15min":7.8140000000000001},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.401,"15min":1.454},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":1.641,"5min":2.1299999999999999,"15min":14.568},"last":1.373},{"interface":"front","average":{"1min":1.361,"5min":1.4690000000000001,"15min":1.4319999999999999},"min":{"1min":0.74299999999999999,"5min":0.496,"15min":0.34399999999999997},"max":{"1min":1.8160000000000001,"5min":6.2990000000000004,"15min":7.1479999999999997},"last":1.5369999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.436},"max":{"1min":1.696,"5min":3.0190000000000001,"15min":9.6630000000000003},"last":1.28},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4239999999999999,"15min":1.6299999999999999},"min":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.45100000000000001},"max":{"1min":1.675,"5min":2.3980000000000001,"15min":14.317},"last":1.4790000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5640000000000001,"15min":1.5449999999999999},"min":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.52100000000000002},"max":{"1min":1.927,"5min":6.5250000000000004,"15min":9.2799999999999994},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.552,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77200000000000002,"15min":0.443},"max":{"1min":1.8540000000000001,"5min":3.3439999999999999,"15min":9.2270000000000003},"last":1.4119999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542383,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82492,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":1417216,"data_stored":608262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83032519},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.94199999999999995,"15min":1.4159999999999999},"min":{"1min":0.628,"5min":0.34699999999999998,"15min":0.26700000000000002},"max":{"1min":1.665,"5min":1.9299999999999999,"15min":50.988999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0780000000000001,"15min":1.47},"min":{"1min":0.58399999999999996,"5min":0.34300000000000003,"15min":0.29699999999999999},"max":{"1min":1.45,"5min":1.786,"15min":20.274000000000001},"last":0.77400000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0820000000000001,"15min":1.381},"min":{"1min":0.59099999999999997,"5min":0.371,"15min":0.32100000000000001},"max":{"1min":2.996,"5min":3.6640000000000001,"15min":18.192},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.284,"5min":1.21,"15min":1.718},"min":{"1min":0.623,"5min":0.39400000000000002,"15min":0.313},"max":{"1min":2.8420000000000001,"5min":2.8420000000000001,"15min":20.385999999999999},"last":0.623}]},{"osd":2,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.294,"15min":1.72},"min":{"1min":0.65800000000000003,"5min":0.45500000000000002,"15min":0.36699999999999999},"max":{"1min":3.069,"5min":3.5249999999999999,"15min":42.840000000000003},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3759999999999999,"15min":1.8109999999999999},"min":{"1min":0.93200000000000005,"5min":0.45200000000000001,"15min":0.35199999999999998},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":38.707000000000001},"last":1.1459999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3169999999999999,"15min":1.752},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.44500000000000001},"max":{"1min":1.8979999999999999,"5min":2.327,"15min":18.952000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.4330000000000001,"15min":1.7729999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.54800000000000004},"max":{"1min":2.2290000000000001,"5min":3.9529999999999998,"15min":18.088999999999999},"last":1.2430000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.4850000000000001,"15min":1.8149999999999999},"min":{"1min":1.042,"5min":0.73199999999999998,"15min":0.55500000000000005},"max":{"1min":2.262,"5min":2.661,"15min":24.349},"last":1.042},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4119999999999999,"15min":1.9079999999999999},"min":{"1min":0.76400000000000001,"5min":0.753,"15min":0.61899999999999999},"max":{"1min":1.9690000000000001,"5min":2.5379999999999998,"15min":14.192},"last":1.54}]},{"osd":6,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.5620000000000001,"15min":1.867},"min":{"1min":0.84399999999999997,"5min":0.77200000000000002,"15min":0.52200000000000002},"max":{"1min":2.8940000000000001,"5min":3.9009999999999998,"15min":22.687000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.5900000000000001,"15min":1.927},"min":{"1min":1.181,"5min":0.73799999999999999,"15min":0.59699999999999998},"max":{"1min":2.375,"5min":2.7890000000000001,"15min":24.058},"last":1.3979999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.577,"15min":2.0939999999999999},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.64600000000000002},"max":{"1min":2.1899999999999999,"5min":6.3209999999999997,"15min":35.570999999999998},"last":1.655},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6279999999999999,"15min":2.048},"min":{"1min":0.879,"5min":0.81100000000000005,"15min":0.58899999999999997},"max":{"1min":2.1789999999999998,"5min":3.8279999999999998,"15min":37.607999999999997},"last":1.7110000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542381,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94948,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94276892,"statfs":{"total":96636764160,"available":96539537408,"internally_reserved":0,"allocated":1400832,"data_stored":606580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90400000000000003,"15min":1.4430000000000001},"min":{"1min":0.67700000000000005,"5min":0.36099999999999999,"15min":0.24399999999999999},"max":{"1min":1.135,"5min":1.96,"15min":64.387},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.022,"15min":1.387},"min":{"1min":0.76500000000000001,"5min":0.36299999999999999,"15min":0.33600000000000002},"max":{"1min":1.7030000000000001,"5min":1.9890000000000001,"15min":24.308},"last":0.78900000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0580000000000001,"15min":1.466},"min":{"1min":0.752,"5min":0.46300000000000002,"15min":0.34699999999999998},"max":{"1min":1.2490000000000001,"5min":2.093,"15min":27.263000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.3879999999999999,"15min":1.8520000000000001},"min":{"1min":1.0940000000000001,"5min":0.51300000000000001,"15min":0.38500000000000001},"max":{"1min":1.8620000000000001,"5min":2.1539999999999999,"15min":23.602},"last":0.69999999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.385,"15min":1.627},"min":{"1min":1.1499999999999999,"5min":0.58499999999999996,"15min":0.40100000000000002},"max":{"1min":1.9299999999999999,"5min":2.085,"15min":27.311},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.296,"5min":1.327,"15min":1.6299999999999999},"min":{"1min":0.96599999999999997,"5min":0.498,"15min":0.47399999999999998},"max":{"1min":1.698,"5min":1.998,"15min":26.555},"last":0.747}]},{"osd":4,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.389,"15min":1.637},"min":{"1min":1.0960000000000001,"5min":0.66100000000000003,"15min":0.38900000000000001},"max":{"1min":1.764,"5min":2.1040000000000001,"15min":26.503},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.498,"5min":1.3819999999999999,"15min":1.7490000000000001},"min":{"1min":1.0009999999999999,"5min":0.56399999999999995,"15min":0.51200000000000001},"max":{"1min":1.9079999999999999,"5min":2.4319999999999999,"15min":25.661000000000001},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.4610000000000001,"15min":1.74},"min":{"1min":1.3120000000000001,"5min":0.74199999999999999,"15min":0.45800000000000002},"max":{"1min":2.077,"5min":2.3490000000000002,"15min":27.149000000000001},"last":1.131},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.538,"15min":1.7330000000000001},"min":{"1min":1.1930000000000001,"5min":0.59799999999999998,"15min":0.43099999999999999},"max":{"1min":1.8540000000000001,"5min":2.6589999999999998,"15min":26.349},"last":1.175}]},{"osd":6,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4910000000000001,"15min":1.9610000000000001},"min":{"1min":1.0700000000000001,"5min":0.64500000000000002,"15min":0.504},"max":{"1min":2.1909999999999998,"5min":2.7149999999999999,"15min":25.603999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.597,"15min":1.843},"min":{"1min":1.1579999999999999,"5min":0.629,"15min":0.42399999999999999},"max":{"1min":2.157,"5min":15.968999999999999,"15min":27.076000000000001},"last":1.01}]},{"osd":7,"last update":"Tue Jan 16 15:18:03 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.591,"15min":1.9470000000000001},"min":{"1min":1.343,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1200000000000001,"5min":2.4889999999999999,"15min":24.509},"last":1.272},{"interface":"front","average":{"1min":1.657,"5min":1.5880000000000001,"15min":1.927},"min":{"1min":1.282,"5min":0.68600000000000005,"15min":0.51600000000000001},"max":{"1min":2.2229999999999999,"5min":2.3479999999999999,"15min":27.207999999999998},"last":1.0589999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542381,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76908,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294932,"statfs":{"total":96636764160,"available":96558010368,"internally_reserved":0,"allocated":1409024,"data_stored":606704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.92400000000000004,"15min":1.153},"min":{"1min":0.437,"5min":0.40600000000000003,"15min":0.28399999999999997},"max":{"1min":1.696,"5min":1.865,"15min":11.699},"last":1.012},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0489999999999999,"15min":1.276},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.33700000000000002},"max":{"1min":1.5089999999999999,"5min":2.3860000000000001,"15min":13.930999999999999},"last":0.74299999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.238,"15min":1.385},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.31},"max":{"1min":1.8,"5min":15.557,"15min":16.488},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1559999999999999,"15min":1.5680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.39000000000000001},"max":{"1min":1.968,"5min":2.306,"15min":22.550999999999998},"last":1.931}]},{"osd":3,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.6879999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.439},"max":{"1min":1.7450000000000001,"5min":1.9790000000000001,"15min":22.471},"last":2.0329999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.3660000000000001,"15min":1.6359999999999999},"min":{"1min":0.54100000000000004,"5min":0.49099999999999999,"15min":0.35699999999999998},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":22.550999999999998},"last":1.3029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4870000000000001,"15min":1.6240000000000001},"min":{"1min":0.96699999999999997,"5min":0.95699999999999996,"15min":0.60899999999999999},"max":{"1min":1.95,"5min":2.7050000000000001,"15min":8.3620000000000001},"last":1.722},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.5960000000000001,"15min":1.7070000000000001},"min":{"1min":0.97399999999999998,"5min":0.69999999999999996,"15min":0.433},"max":{"1min":1.97,"5min":15.723000000000001,"15min":19.236999999999998},"last":1.5049999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.706},"min":{"1min":0.92800000000000005,"5min":0.73799999999999999,"15min":0.57199999999999995},"max":{"1min":1.843,"5min":2.2040000000000002,"15min":21.117000000000001},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.544,"15min":1.758},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.65800000000000003},"max":{"1min":1.8009999999999999,"5min":15.785,"15min":17.138000000000002},"last":1.661}]},{"osd":6,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.4850000000000001,"15min":1.698},"min":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.72199999999999998},"max":{"1min":2.2759999999999998,"5min":2.298,"15min":11.901999999999999},"last":2.2890000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.5840000000000001,"15min":1.8069999999999999},"min":{"1min":1.044,"5min":0.86399999999999999,"15min":0.63300000000000001},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":23.420999999999999},"last":1.5740000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.631,"15min":1.8380000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.085,"5min":15.875,"15min":17.010000000000002},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.573,"15min":1.825},"min":{"1min":0.998,"5min":0.91600000000000004,"15min":0.75700000000000001},"max":{"1min":2.508,"5min":2.508,"15min":11.962},"last":2.1669999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542382,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73340,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298500,"statfs":{"total":96636764160,"available":96561664000,"internally_reserved":0,"allocated":2015232,"data_stored":1198672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":73071041},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.016,"15min":1.468},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":1.355,"5min":4.8380000000000001,"15min":43.459000000000003},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.2290000000000001,"15min":1.706},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.956,"5min":4.9779999999999998,"15min":43.627000000000002},"last":1.226}]},{"osd":2,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1919999999999999,"15min":1.4099999999999999},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":5.0570000000000004,"15min":18.861000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.181,"5min":1.218,"15min":1.5229999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":2.2330000000000001,"5min":5.3550000000000004,"15min":38.893000000000001},"last":0.75}]},{"osd":3,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0800000000000001,"15min":1.47},"min":{"1min":0.37,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":1.4410000000000001,"5min":4.9050000000000002,"15min":23.687999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.357,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.434},"max":{"1min":1.6080000000000001,"5min":5.1980000000000004,"15min":18.966999999999999},"last":0.80000000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3460000000000001,"15min":1.5129999999999999},"min":{"1min":0.76500000000000001,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.8420000000000001,"5min":5.1040000000000001,"15min":20.327000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.496,"15min":1.7030000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.873,"5min":5.3109999999999999,"15min":19.004999999999999},"last":1.1910000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.508,"15min":1.891},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.792,"5min":5.1589999999999998,"15min":24.611000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4390000000000001,"15min":1.6970000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.56000000000000005},"max":{"1min":1.907,"5min":7.2699999999999996,"15min":20.199000000000002},"last":1.0660000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6499999999999999,"15min":2.0510000000000002},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.55900000000000005},"max":{"1min":2.3119999999999998,"5min":6.3339999999999996,"15min":26.998000000000001},"last":1.258},{"interface":"front","average":{"1min":1.524,"5min":1.6140000000000001,"15min":1.988},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":2.1669999999999998,"5min":5.9690000000000003,"15min":24.478999999999999},"last":1.2729999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6699999999999999,"15min":1.9139999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.60899999999999999},"max":{"1min":2.2599999999999998,"5min":5.7729999999999997,"15min":17.957000000000001},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.5629999999999999,"15min":1.7310000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.52800000000000002},"max":{"1min":1.8999999999999999,"5min":6.149,"15min":20.279},"last":1.1539999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:19:00.540 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:19:00.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:19:00.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.821+0000 7f92d0800640 1 Processor -- start 2024-01-16T15:19:00.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.822+0000 7f92d0800640 1 -- start start 2024-01-16T15:19:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.823+0000 7f92d0800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c815b350 0x7f92c80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92d0800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92c80adb60 0x7f92c80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92d0800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c815b730 con 0x7f92c809cb30 2024-01-16T15:19:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92d0800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c80ac740 con 0x7f92c80adb60 2024-01-16T15:19:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92d0800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80ac460 con 0x7f92c815b350 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92ce800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92c80adb60 0x7f92c80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92ce800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f92c80adb60 0x7f92c80adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38420/0 (socket says 172.21.15.43:38420) 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92ce800640 1 -- 172.21.15.43:0/2754532581 learned_addr learned my addr 172.21.15.43:0/2754532581 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.824+0000 7f92cde00640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c815b350 0x7f92c80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.825+0000 7f92ce800640 1 -- 172.21.15.43:0/2754532581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c815b350 msgr2=0x7f92c80abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.825+0000 7f92ce800640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c815b350 0x7f92c80abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.825+0000 7f92ce800640 1 -- 172.21.15.43:0/2754532581 >> v1:172.21.15.69:6789/0 conn(0x7f92c809cb30 legacy=0x7f92c80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:00.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.825+0000 7f92ce800640 1 -- 172.21.15.43:0/2754532581 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c80b2a50 con 0x7f92c80adb60 2024-01-16T15:19:00.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.826+0000 7f92ce800640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c80adb60 0x7f92c80adfb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009870 tx=0x7f92c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918e3b2ddb680625 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.826+0000 7f92cca00640 1 -- 172.21.15.43:0/2754532581 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0031640 con 0x7f92c80adb60 2024-01-16T15:19:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.826+0000 7f92cca00640 1 -- 172.21.15.43:0/2754532581 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c00317a0 con 0x7f92c80adb60 2024-01-16T15:19:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.826+0000 7f92cca00640 1 -- 172.21.15.43:0/2754532581 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0031b30 con 0x7f92c80adb60 2024-01-16T15:19:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 -- 172.21.15.43:0/2754532581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c80adb60 msgr2=0x7f92c80adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c80adb60 0x7f92c80adfb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009870 tx=0x7f92c002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 -- 172.21.15.43:0/2754532581 shutdown_connections 2024-01-16T15:19:00.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c80adb60 0x7f92c80adfb0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 --2- 172.21.15.43:0/2754532581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c815b350 0x7f92c80abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 -- 172.21.15.43:0/2754532581 >> 172.21.15.43:0/2754532581 conn(0x7f92c80b0ba0 msgr2=0x7f92c80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:00.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 -- 172.21.15.43:0/2754532581 shutdown_connections 2024-01-16T15:19:00.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.827+0000 7f92d0800640 1 -- 172.21.15.43:0/2754532581 wait complete. 2024-01-16T15:19:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.829+0000 7f92d0800640 1 Processor -- start 2024-01-16T15:19:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.829+0000 7f92d0800640 1 -- start start 2024-01-16T15:19:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c80adb60 0x7f92c809aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815b350 0x7f92c809af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c814e570 con 0x7f92c809cb30 2024-01-16T15:19:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8158060 con 0x7f92c815b350 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92d0800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8158340 con 0x7f92c80adb60 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.830+0000 7f92cde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38424/0 (socket says 172.21.15.43:38424) 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 -- 172.21.15.43:0/3929013594 learned_addr learned my addr 172.21.15.43:0/3929013594 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cd400640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c80adb60 0x7f92c809aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92ce800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815b350 0x7f92c809af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 -- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c80adb60 msgr2=0x7f92c809aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c80adb60 0x7f92c809aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 -- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815b350 msgr2=0x7f92c809af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815b350 0x7f92c809af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:00.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 -- 172.21.15.43:0/3929013594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00094d0 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.831+0000 7f92cde00640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007e40 tx=0x7f92b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.832+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8014620 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.832+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b800ee90 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.832+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80149c0 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.832+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c8158620 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.832+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c8158be0 con 0x7f92c809cb30 2024-01-16T15:19:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.833+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8158d70 con 0x7f92c809cb30 2024-01-16T15:19:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.834+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f92b8014b20 con 0x7f92c809cb30 2024-01-16T15:19:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.836+0000 7f92c6000640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 0x7f92a006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.836+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f92b80890b0 con 0x7f92c809cb30 2024-01-16T15:19:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.836+0000 7f92cd400640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 0x7f92a006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.837+0000 7f92cd400640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 0x7f92a006e860 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f92c81f6490 tx=0x7f92b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:00.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:00.842+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92b8055970 con 0x7f92c809cb30 2024-01-16T15:19:01.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.106+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f92c809d200 con 0x7f92c809cb30 2024-01-16T15:19:01.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.110+0000 7f92c6000640 1 -- 172.21.15.43:0/3929013594 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f92b8058f90 con 0x7f92c809cb30 2024-01-16T15:19:01.101 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-01-16T15:19:01.101 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQCUmaZlX9i3LRAASR8r5806yVM9wSZL5h0RZQ== 2024-01-16T15:19:01.101 INFO:teuthology.orchestra.run.smithi043.stdout: caps mds = "allow" 2024-01-16T15:19:01.102 INFO:teuthology.orchestra.run.smithi043.stdout: caps mon = "allow r" 2024-01-16T15:19:01.102 INFO:teuthology.orchestra.run.smithi043.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:19:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.0 2024-01-16T15:19:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 msgr2=0x7f92a006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 0x7f92a006e860 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f92c81f6490 tx=0x7f92b0009040 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 msgr2=0x7f92c8140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007e40 tx=0x7f92b800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 shutdown_connections 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f92a006c3a0 0x7f92a006e860 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92c815b350 0x7f92c809af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f92c80adb60 0x7f92c809aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 --2- 172.21.15.43:0/3929013594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c809cb30 0x7f92c8140ad0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 >> 172.21.15.43:0/3929013594 conn(0x7f92c80b0ba0 msgr2=0x7f92c80afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 shutdown_connections 2024-01-16T15:19:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.114+0000 7f92d0800640 1 -- 172.21.15.43:0/3929013594 wait complete. 2024-01-16T15:19:01.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:19:01.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.351+0000 7ff8efe00640 1 Processor -- start 2024-01-16T15:19:01.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.352+0000 7ff8efe00640 1 -- start start 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.353+0000 7ff8efe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.353+0000 7ff8efe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e813e500 con 0x7ff8e813e120 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.353+0000 7ff8efe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e815b7c0 con 0x7ff8e80b07d0 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.353+0000 7ff8efe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e815c440 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38428/0 (socket says 172.21.15.43:38428) 2024-01-16T15:19:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 -- 172.21.15.43:0/3475052299 learned_addr learned my addr 172.21.15.43:0/3475052299 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 -- 172.21.15.43:0/3475052299 >> v1:172.21.15.69:6789/0 conn(0x7ff8e80b07d0 legacy=0x7ff8e80cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 -- 172.21.15.43:0/3475052299 >> v1:172.21.15.43:6789/0 conn(0x7ff8e813e120 legacy=0x7ff8e806ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.354+0000 7ff8ede00640 1 -- 172.21.15.43:0/3475052299 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e815baa0 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.355+0000 7ff8ede00640 1 --2- 172.21.15.43:0/3475052299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd71f37812bce1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:01.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8e7e00640 1 -- 172.21.15.43:0/3475052299 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80317e0 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8e7e00640 1 -- 172.21.15.43:0/3475052299 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8031940 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8e7e00640 1 -- 172.21.15.43:0/3475052299 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8031cd0 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 -- 172.21.15.43:0/3475052299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e80ce1e0 msgr2=0x7ff8e8165fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 --2- 172.21.15.43:0/3475052299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d802b880 comp rx=0 tx=0).stop 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 -- 172.21.15.43:0/3475052299 shutdown_connections 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 --2- 172.21.15.43:0/3475052299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e8165fc0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 -- 172.21.15.43:0/3475052299 >> 172.21.15.43:0/3475052299 conn(0x7ff8e80663d0 msgr2=0x7ff8e80667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.356+0000 7ff8efe00640 1 -- 172.21.15.43:0/3475052299 shutdown_connections 2024-01-16T15:19:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.357+0000 7ff8efe00640 1 -- 172.21.15.43:0/3475052299 wait complete. 2024-01-16T15:19:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.358+0000 7ff8efe00640 1 Processor -- start 2024-01-16T15:19:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.358+0000 7ff8efe00640 1 -- start start 2024-01-16T15:19:01.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e80b07d0 0x7ff8e81586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:01.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e813d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:01.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:01.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8141b10 con 0x7ff8e813e120 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80b6580 con 0x7ff8e80ce1e0 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8efe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e81584e0 con 0x7ff8e80b07d0 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8ede00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8ede00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38444/0 (socket says 172.21.15.43:38444) 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.359+0000 7ff8ede00640 1 -- 172.21.15.43:0/2020354312 learned_addr learned my addr 172.21.15.43:0/2020354312 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 -- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e80b07d0 msgr2=0x7ff8e81586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e80b07d0 0x7ff8e81586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 -- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e80ce1e0 msgr2=0x7ff8e813d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e813d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 -- 172.21.15.43:0/2020354312 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80094d0 con 0x7ff8e813e120 2024-01-16T15:19:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8ede00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.360+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8032980 con 0x7ff8e813e120 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.361+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8032ae0 con 0x7ff8e813e120 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.361+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8032e90 con 0x7ff8e813e120 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.361+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e80b5fc0 con 0x7ff8e813e120 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.361+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e80b4820 con 0x7ff8e813e120 2024-01-16T15:19:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.362+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8067d60 con 0x7ff8e813e120 2024-01-16T15:19:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.363+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7ff8d804e450 con 0x7ff8e813e120 2024-01-16T15:19:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.363+0000 7ff8e5600640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 0x7ff8c006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.364+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff8d80aaaf0 con 0x7ff8e813e120 2024-01-16T15:19:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.364+0000 7ff8ed400640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 0x7ff8c006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.364+0000 7ff8ed400640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 0x7ff8c006e860 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00012e0 tx=0x7ff8e0007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:01.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.368+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8d8077460 con 0x7ff8e813e120 2024-01-16T15:19:01.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.603+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e8074770 con 0x7ff8e813e120 2024-01-16T15:19:01.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.605+0000 7ff8e5600640 1 -- 172.21.15.43:0/2020354312 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4915 (secure 0 0 0) 0x7ff8d807aa80 con 0x7ff8e813e120 2024-01-16T15:19:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:19:01.595 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3583583840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":3583583840},{"type":"v1","addr":"172.21.15.43:6837","nonce":3583583840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2071477429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2071477429},{"type":"v1","addr":"172.21.15.69:6833","nonce":2071477429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/4168252148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4168252148},{"type":"v1","addr":"172.21.15.43:6836","nonce":4168252148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:18:49.965316+0000","modified":"2024-01-16T15:18:53.023347+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9132":{"gid":9132,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/3715375194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3715375194},{"type":"v1","addr":"172.21.15.69:6835","nonce":3715375194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:19:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 75 2024-01-16T15:19:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 msgr2=0x7ff8c006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 0x7ff8c006e860 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00012e0 tx=0x7ff8e0007d30 comp rx=0 tx=0).stop 2024-01-16T15:19:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 msgr2=0x7ff8e8157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d80046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 shutdown_connections 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff8c006c3a0 0x7ff8c006e860 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8e80ce1e0 0x7ff8e813d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8e813e120 0x7ff8e8157fa0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 --2- 172.21.15.43:0/2020354312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8e80b07d0 0x7ff8e81586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 >> 172.21.15.43:0/2020354312 conn(0x7ff8e80663d0 msgr2=0x7ff8e80667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 shutdown_connections 2024-01-16T15:19:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:01.608+0000 7ff8efe00640 1 -- 172.21.15.43:0/2020354312 wait complete. 2024-01-16T15:19:02.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:18:49.965316+0000', 'modified': '2024-01-16T15:18:53.023347+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9132}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9132': {'gid': 9132, 'name': 'd', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6835/3715375194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 3715375194}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 3715375194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:19:02.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:19:02.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:19:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.262+0000 7f628f200640 1 Processor -- start 2024-01-16T15:19:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.262+0000 7f628f200640 1 -- start start 2024-01-16T15:19:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 0x7f62880700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6288070610 0x7f628806c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6288070a70 con 0x7f628806c810 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6288144ed0 con 0x7f6288070610 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628806d040 con 0x7f62881449d0 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6288070610 0x7f628806c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6288070610 0x7f628806c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38454/0 (socket says 172.21.15.43:38454) 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628d400640 1 -- 172.21.15.43:0/2141047309 learned_addr learned my addr 172.21.15.43:0/2141047309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.264+0000 7f628de00640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 0x7f62880700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628d400640 1 -- 172.21.15.43:0/2141047309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 msgr2=0x7f62880700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628d400640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 0x7f62880700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628d400640 1 -- 172.21.15.43:0/2141047309 >> v1:172.21.15.43:6789/0 conn(0x7f628806c810 legacy=0x7f6288146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628ca00640 1 -- 172.21.15.43:0/2141047309 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214839179 0 0) 0x7f6288070a70 con 0x7f628806c810 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628d400640 1 -- 172.21.15.43:0/2141047309 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628806d320 con 0x7f6288070610 2024-01-16T15:19:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.265+0000 7f628de00640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 0x7f62880700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628d400640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6288070610 0x7f628806c2d0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6270009870 tx=0x7f627002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96d9d80714bb829 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628ca00640 1 -- 172.21.15.43:0/2141047309 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270030840 con 0x7f6288070610 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628ca00640 1 -- 172.21.15.43:0/2141047309 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62700309a0 con 0x7f6288070610 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628ca00640 1 -- 172.21.15.43:0/2141047309 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270030d30 con 0x7f6288070610 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628f200640 1 -- 172.21.15.43:0/2141047309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6288070610 msgr2=0x7f628806c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.266+0000 7f628f200640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6288070610 0x7f628806c2d0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f6270009870 tx=0x7f627002b980 comp rx=0 tx=0).stop 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 -- 172.21.15.43:0/2141047309 shutdown_connections 2024-01-16T15:19:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6288070610 0x7f628806c2d0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 --2- 172.21.15.43:0/2141047309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62881449d0 0x7f62880700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 -- 172.21.15.43:0/2141047309 >> 172.21.15.43:0/2141047309 conn(0x7f62880b0840 msgr2=0x7f62880b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 -- 172.21.15.43:0/2141047309 shutdown_connections 2024-01-16T15:19:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.267+0000 7f628f200640 1 -- 172.21.15.43:0/2141047309 wait complete. 2024-01-16T15:19:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.268+0000 7f628f200640 1 Processor -- start 2024-01-16T15:19:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.269+0000 7f628f200640 1 -- start start 2024-01-16T15:19:02.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.269+0000 7f628f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f628806c810 0x7f62881583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.269+0000 7f628f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6288070610 0x7f6288158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62881f81d0 con 0x7f62881449d0 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6288158f30 con 0x7f6288070610 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628815f570 con 0x7f628806c810 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38458/0 (socket says 172.21.15.43:38458) 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 -- 172.21.15.43:0/2567747586 learned_addr learned my addr 172.21.15.43:0/2567747586 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628de00640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f628806c810 0x7f62881583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628d400640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6288070610 0x7f6288158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 -- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f628806c810 msgr2=0x7f62881583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f628806c810 0x7f62881583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 -- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6288070610 msgr2=0x7f6288158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6288070610 0x7f6288158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.270+0000 7f628e800640 1 -- 172.21.15.43:0/2567747586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62700094d0 con 0x7f62881449d0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f628e800640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f6280001560 tx=0x7f628000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f6286000640 1 -- 172.21.15.43:0/2567747586 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280003da0 con 0x7f62881449d0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f6286000640 1 -- 172.21.15.43:0/2567747586 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280005070 con 0x7f62881449d0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f6286000640 1 -- 172.21.15.43:0/2567747586 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280005420 con 0x7f62881449d0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62881f7410 con 0x7f62881449d0 2024-01-16T15:19:02.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.271+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62881f79d0 con 0x7f62881449d0 2024-01-16T15:19:02.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.274+0000 7f6286000640 1 -- 172.21.15.43:0/2567747586 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f6280005580 con 0x7f62881449d0 2024-01-16T15:19:02.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.275+0000 7f6286000640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 0x7f626006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.275+0000 7f62935f3740 1 Processor -- start 2024-01-16T15:19:02.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.275+0000 7f6286000640 1 -- 172.21.15.43:0/2567747586 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6280088360 con 0x7f62881449d0 2024-01-16T15:19:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.275+0000 7f62935f3740 1 -- start start 2024-01-16T15:19:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.275+0000 7f628de00640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 0x7f626006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f628de00640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 0x7f626006e860 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6288144650 tx=0x7f6278008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe20f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe253730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe16e880 con 0x556cfe210070 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe17dae0 con 0x556cfe20ad70 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cfe17b1d0 con 0x556cfe20f7b0 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f628de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f628de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38474/0 (socket says 172.21.15.43:38474) 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.276+0000 7f628de00640 1 -- 172.21.15.43:0/3906475909 learned_addr learned my addr 172.21.15.43:0/3906475909 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 -- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 msgr2=0x556cfe253730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628d400640 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe20f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe253730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 -- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 msgr2=0x556cfe20f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe20f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 -- 172.21.15.43:0/3906475909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cfe13e860 con 0x556cfe210070 2024-01-16T15:19:02.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f628de00640 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f627800dbc0 tx=0x7f6278010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00507fef48e8e74 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f627ea00640 1 -- 172.21.15.43:0/3906475909 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627800b050 con 0x556cfe210070 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f627ea00640 1 -- 172.21.15.43:0/3906475909 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278017070 con 0x556cfe210070 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.277+0000 7f627ea00640 1 -- 172.21.15.43:0/3906475909 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278012dc0 con 0x556cfe210070 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 -- 172.21.15.43:0/3906475909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 msgr2=0x556cfe210450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f627800dbc0 tx=0x7f6278010000 comp rx=0 tx=0).stop 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 -- 172.21.15.43:0/3906475909 shutdown_connections 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe20f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe253730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 --2- 172.21.15.43:0/3906475909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe210450 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 -- 172.21.15.43:0/3906475909 >> 172.21.15.43:0/3906475909 conn(0x556cfe1d1c70 msgr2=0x556cfe255b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 -- 172.21.15.43:0/3906475909 shutdown_connections 2024-01-16T15:19:02.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.278+0000 7f62935f3740 1 -- 172.21.15.43:0/3906475909 wait complete. 2024-01-16T15:19:02.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.279+0000 7f62935f3740 1 -- start start 2024-01-16T15:19:02.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.279+0000 7f62935f3740 1 Processor -- start 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20f7b0 0x556cfe2394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe2399f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe176ac0 con 0x556cfe20ad70 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe1132f0 con 0x556cfe20f7b0 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cfe130c70 con 0x556cfe210070 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.280+0000 7f62935f3740 10 client.0 _ll_register_callbacks cb 0x556cfe1353b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38484/0 (socket says 172.21.15.43:38484) 2024-01-16T15:19:02.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 -- 172.21.15.43:0/764224276 learned_addr learned my addr 172.21.15.43:0/764224276 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628d400640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20f7b0 0x556cfe2394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628e800640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe2399f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 -- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 msgr2=0x556cfe2399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe2399f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 -- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20f7b0 msgr2=0x556cfe2394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20f7b0 0x556cfe2394b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 -- 172.21.15.43:0/764224276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627800d840 con 0x556cfe20ad70 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f628de00640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6278008010 tx=0x7f6278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75d572944993e5e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.281+0000 7f6277400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627800b040 con 0x556cfe20ad70 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f627802e070 con 0x556cfe20ad70 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556cfe16faa0 con 0x556cfe20ad70 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627800dec0 con 0x556cfe20ad70 2024-01-16T15:19:02.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 10 client.9303 fetch_fsmap learned FSMap version 75 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x556cfe175140 con 0x556cfe20ad70 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556cfe16faa0 con 0x556cfe20ad70 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f627801ec00 con 0x556cfe20ad70 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 10 client.9303 fetch_fsmap finished waiting for FSMap version 75 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 10 client.9303 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 20 client.9303 populate_metadata read hostname 'smithi043' 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe2370c0 0x556cfe23d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 4 client.9303 mds_command: new command op to 9132 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.282+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 --> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556cfe16f7c0 con 0x556cfe2370c0 2024-01-16T15:19:02.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.283+0000 7f628d400640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe2370c0 0x556cfe23d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.284+0000 7f628d400640 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe2370c0 0x556cfe23d6d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c278ee9b9911e36b server_cookie=7e4af2ee5bfc0e3b in_seq=0 out_seq=0 2024-01-16T15:19:02.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.284+0000 7f6277400640 10 client.9303 ms_handle_connect on v2:172.21.15.69:6834/3715375194 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f6277400640 1 -- 172.21.15.43:0/764224276 <== mds.0 v2:172.21.15.69:6834/3715375194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556cfe16f7c0 con 0x556cfe2370c0 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f6277400640 10 client.9303 handle_command_reply: tid=0 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 client.9303 shutdown 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 msgr2=0x556cfe235f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6278008010 tx=0x7f6278004730 comp rx=0 tx=0).stop 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 shutdown_connections 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe2370c0 0x556cfe23d6d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20f7b0 0x556cfe2394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe2399f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 --2- 172.21.15.43:0/764224276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20ad70 0x556cfe235f30 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 >> 172.21.15.43:0/764224276 conn(0x556cfe1d1c70 msgr2=0x556cfe255b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.286+0000 7f6277400640 0 client.9303 ms_handle_reset on v2:172.21.15.69:6834/3715375194 2024-01-16T15:19:02.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.287+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 shutdown_connections 2024-01-16T15:19:02.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.287+0000 7f62935f3740 1 -- 172.21.15.43:0/764224276 wait complete. 2024-01-16T15:19:02.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.287+0000 7f62935f3740 20 client.9303 trim_cache size 0 max 16384 2024-01-16T15:19:02.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.304+0000 7f62935f3740 1 Processor -- start 2024-01-16T15:19:02.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.304+0000 7f62935f3740 1 -- start start 2024-01-16T15:19:02.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe209dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe254830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfded9f20 con 0x556cfe20f7b0 2024-01-16T15:19:02.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe20e5e0 con 0x556cfe20ad70 2024-01-16T15:19:02.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cfe237870 con 0x556cfe210070 2024-01-16T15:19:02.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38496/0 (socket says 172.21.15.43:38496) 2024-01-16T15:19:02.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 -- 172.21.15.43:0/2162534819 learned_addr learned my addr 172.21.15.43:0/2162534819 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 -- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 msgr2=0x556cfe254830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe254830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 -- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 msgr2=0x556cfe209dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628de00640 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe209dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe209dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628d400640 1 -- 172.21.15.43:0/2162534819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cfe112b80 con 0x556cfe20f7b0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.305+0000 7f628de00640 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe209dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.306+0000 7f628d400640 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6270000c00 tx=0x7f6270030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa06f8a0d7421424 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.306+0000 7f627f400640 1 -- 172.21.15.43:0/2162534819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270030d10 con 0x556cfe20f7b0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.306+0000 7f627f400640 1 -- 172.21.15.43:0/2162534819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6270030e70 con 0x556cfe20f7b0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.306+0000 7f627f400640 1 -- 172.21.15.43:0/2162534819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270004260 con 0x556cfe20f7b0 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- 172.21.15.43:0/2162534819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 msgr2=0x556cfe20a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6270000c00 tx=0x7f6270030000 comp rx=0 tx=0).stop 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- 172.21.15.43:0/2162534819 shutdown_connections 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe210070 0x556cfe254830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe20f7b0 0x556cfe20a310 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 --2- 172.21.15.43:0/2162534819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe209dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- 172.21.15.43:0/2162534819 >> 172.21.15.43:0/2162534819 conn(0x556cfe1d1c70 msgr2=0x556cfe210450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- 172.21.15.43:0/2162534819 shutdown_connections 2024-01-16T15:19:02.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- 172.21.15.43:0/2162534819 wait complete. 2024-01-16T15:19:02.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.307+0000 7f62935f3740 1 -- start start 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.308+0000 7f62935f3740 1 Processor -- start 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe256240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe2befc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe26ef00 con 0x556cfe210070 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cfe252d90 con 0x556cfe20ad70 2024-01-16T15:19:02.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cfe20a720 con 0x556cfe20f7b0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f62935f3740 10 client.0 _ll_register_callbacks cb 0x7f6278012dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38498/0 (socket says 172.21.15.43:38498) 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 -- 172.21.15.43:0/300372958 learned_addr learned my addr 172.21.15.43:0/300372958 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 -- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 msgr2=0x556cfe2befc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe2befc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 -- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 msgr2=0x556cfe256240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628de00640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe256240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe256240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.309+0000 7f628e800640 1 -- 172.21.15.43:0/300372958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628806d320 con 0x556cfe210070 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f628e800640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6280058530 tx=0x7f6280057fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6399f062108350bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628005c6f0 con 0x556cfe210070 2024-01-16T15:19:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f62800608a0 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6278017070 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6280052550 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f62935f3740 10 client.9309 fetch_fsmap learned FSMap version 75 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x556cfe138170 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278017070 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6280057a90 con 0x556cfe210070 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f62935f3740 10 client.9309 fetch_fsmap finished waiting for FSMap version 75 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.310+0000 7f62935f3740 10 client.9309 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.311+0000 7f62935f3740 20 client.9309 populate_metadata read hostname 'smithi043' 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.311+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe267fe0 0x556cfe2683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.311+0000 7f62935f3740 4 client.9309 mds_command: new command op to 9132 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:19:02.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.311+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 --> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556cfde776c0 con 0x556cfe267fe0 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.311+0000 7f628de00640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe267fe0 0x556cfe2683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.312+0000 7f628de00640 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe267fe0 0x556cfe2683c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6818987aad573c6 server_cookie=52764a2dde28567b in_seq=0 out_seq=0 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.312+0000 7f6277400640 10 client.9309 ms_handle_connect on v2:172.21.15.69:6834/3715375194 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f6277400640 1 -- 172.21.15.43:0/300372958 <== mds.0 v2:172.21.15.69:6834/3715375194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x556cfde776c0 con 0x556cfe267fe0 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f6277400640 10 client.9309 handle_command_reply: tid=0 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 client.9309 shutdown 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 msgr2=0x556cfe258ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6280058530 tx=0x7f6280057fc0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 shutdown_connections 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:6834/3715375194,v1:172.21.15.69:6835/3715375194] conn(0x556cfe267fe0 0x556cfe2683c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556cfe210070 0x556cfe258ca0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x556cfe20f7b0 0x556cfe2befc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 --2- 172.21.15.43:0/300372958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556cfe20ad70 0x556cfe256240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f6277400640 0 client.9309 ms_handle_reset on v2:172.21.15.69:6834/3715375194 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 >> 172.21.15.43:0/300372958 conn(0x556cfe1d1c70 msgr2=0x556cfe2064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 shutdown_connections 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 1 -- 172.21.15.43:0/300372958 wait complete. 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.313+0000 7f62935f3740 20 client.9309 trim_cache size 0 max 16384 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "eb89d060-5b4a-4a34-8ffb-6d8b15e6800a", 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 9132, 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-01-16T15:19:02.306 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 11.807591841000001, 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 75, 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 113, 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 113, 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 18.158292357000001 2024-01-16T15:19:02.307 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 msgr2=0x7f626006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 0x7f626006e860 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6288144650 tx=0x7f6278008040 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 msgr2=0x7f62880b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f6280001560 tx=0x7f628000cc70 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 shutdown_connections 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f626006c3a0 0x7f626006e860 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6288070610 0x7f6288158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62881449d0 0x7f62880b2df0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 --2- 172.21.15.43:0/2567747586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f628806c810 0x7f62881583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 >> 172.21.15.43:0/2567747586 conn(0x7f62880b0840 msgr2=0x7f62880afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 shutdown_connections 2024-01-16T15:19:02.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.314+0000 7f628f200640 1 -- 172.21.15.43:0/2567747586 wait complete. 2024-01-16T15:19:02.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:19:02.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.536+0000 7f2fbf000640 1 Processor -- start 2024-01-16T15:19:02.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.536+0000 7f2fbf000640 1 -- start start 2024-01-16T15:19:02.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 0x7f2fb80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fb815e6b0 0x7f2fb815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb8071700 con 0x7f2fb815e6b0 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb815b410 con 0x7f2fb8143570 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbf000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815b130 con 0x7f2fb8143950 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbe600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fb815e6b0 0x7f2fb815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbe600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fb815e6b0 0x7f2fb815a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38512/0 (socket says 172.21.15.43:38512) 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.538+0000 7f2fbe600640 1 -- 172.21.15.43:0/546155894 learned_addr learned my addr 172.21.15.43:0/546155894 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbd200640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbdc00640 1 --2- 172.21.15.43:0/546155894 >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 0x7f2fb80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbe600640 1 -- 172.21.15.43:0/546155894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 msgr2=0x7f2fb80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbe600640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbe600640 1 -- 172.21.15.43:0/546155894 >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 msgr2=0x7f2fb80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbe600640 1 --2- 172.21.15.43:0/546155894 >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 0x7f2fb80cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbe600640 1 -- 172.21.15.43:0/546155894 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb815dbd0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.539+0000 7f2fbdc00640 1 --2- 172.21.15.43:0/546155894 >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 0x7f2fb80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:02.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.540+0000 7f2fbe600640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb815a4b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8070c70 tx=0x7f2fb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d98d95cc56ad27d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fb7e00640 1 -- 172.21.15.43:0/546155894 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb002c9e0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fb7e00640 1 -- 172.21.15.43:0/546155894 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb002cb40 con 0x7f2fb815e6b0 2024-01-16T15:19:02.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fb7e00640 1 -- 172.21.15.43:0/546155894 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb00343f0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 -- 172.21.15.43:0/546155894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 msgr2=0x7f2fb815a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb815a4b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8070c70 tx=0x7f2fb002be40 comp rx=0 tx=0).stop 2024-01-16T15:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 -- 172.21.15.43:0/546155894 shutdown_connections 2024-01-16T15:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb815a4b0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 --2- 172.21.15.43:0/546155894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 --2- 172.21.15.43:0/546155894 >> v2:172.21.15.69:3300/0 conn(0x7f2fb8143570 0x7f2fb80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.541+0000 7f2fbf000640 1 -- 172.21.15.43:0/546155894 >> 172.21.15.43:0/546155894 conn(0x7f2fb806a860 msgr2=0x7f2fb806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.542+0000 7f2fbf000640 1 -- 172.21.15.43:0/546155894 shutdown_connections 2024-01-16T15:19:02.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.542+0000 7f2fbf000640 1 -- 172.21.15.43:0/546155894 wait complete. 2024-01-16T15:19:02.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.543+0000 7f2fbf000640 1 Processor -- start 2024-01-16T15:19:02.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.544+0000 7f2fbf000640 1 -- start start 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.544+0000 7f2fbf000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8143570 0x7f2fb813bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.544+0000 7f2fbf000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbf000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbf000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb813b850 con 0x7f2fb815e6b0 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbf000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb8138e30 con 0x7f2fb8143570 2024-01-16T15:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbf000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb813abd0 con 0x7f2fb8143950 2024-01-16T15:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38518/0 (socket says 172.21.15.43:38518) 2024-01-16T15:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 -- 172.21.15.43:0/3599422915 learned_addr learned my addr 172.21.15.43:0/3599422915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbd200640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbdc00640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8143570 0x7f2fb813bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 -- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 msgr2=0x7f2fb80b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 -- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8143570 msgr2=0x7f2fb813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8143570 0x7f2fb813bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.545+0000 7f2fbe600640 1 -- 172.21.15.43:0/3599422915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0009b30 con 0x7f2fb815e6b0 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fbe600640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0036040 tx=0x7f2fb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb003e450 con 0x7f2fb815e6b0 2024-01-16T15:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb003e5d0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb003e980 con 0x7f2fb815e6b0 2024-01-16T15:19:02.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb813aeb0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.546+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb813b470 con 0x7f2fb815e6b0 2024-01-16T15:19:02.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.548+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb80cb440 con 0x7f2fb815e6b0 2024-01-16T15:19:02.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.548+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f2fb003eae0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.549+0000 7f2fb5600640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 0x7f2f9006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.549+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2fb00ab780 con 0x7f2fb815e6b0 2024-01-16T15:19:02.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.550+0000 7f2fbdc00640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 0x7f2f9006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.550+0000 7f2fbdc00640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 0x7f2f9006e860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006fd0 tx=0x7f2fa8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.554+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fb00781c0 con 0x7f2fb815e6b0 2024-01-16T15:19:02.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.794+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2fb804e840 con 0x7f2fb815e6b0 2024-01-16T15:19:02.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.795+0000 7f2fb5600640 1 -- 172.21.15.43:0/3599422915 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2fb0043060 con 0x7f2fb815e6b0 2024-01-16T15:19:02.785 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 msgr2=0x7f2f9006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 0x7f2f9006e860 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006fd0 tx=0x7f2fa8008040 comp rx=0 tx=0).stop 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 msgr2=0x7f2fb80cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0036040 tx=0x7f2fb0004a40 comp rx=0 tx=0).stop 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 shutdown_connections 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9006c3a0 0x7f2f9006e860 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fb815e6b0 0x7f2fb80cdad0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb8143950 0x7f2fb80b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 --2- 172.21.15.43:0/3599422915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb8143570 0x7f2fb813bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.797+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 >> 172.21.15.43:0/3599422915 conn(0x7f2fb806a860 msgr2=0x7f2fb80cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.798+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 shutdown_connections 2024-01-16T15:19:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:02.798+0000 7f2fbf000640 1 -- 172.21.15.43:0/3599422915 wait complete. 2024-01-16T15:19:02.807 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:19:02.807 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:19:02.808 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:02.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:02.808 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.865 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41296sec preferred_lft 41296sec 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::1ce7:e8ff:fe70:8eb0/64 scope link 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 52:c5:03:ee:72:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::50c5:3ff:feee:7273/64 scope link 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 66:79:33:89:4c:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::6479:33ff:fe89:4c36/64 scope link 2024-01-16T15:19:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:02.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:02.868 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:19:02.965 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-01-16T15:19:02.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:02.970 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-01-16T15:19:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-01-16T15:19:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:19:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:19:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:19:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:19:03.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:03.026 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec bash -c ' 2024-01-16T15:19:03.026 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-01-16T15:19:03.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:19:03.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:19:03.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:19:03.027 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-01-16T15:19:03.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:03.228 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:19:03.282 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:19:03.282 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:19:03.283 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:03.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:03.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-01-16T15:19:03.382 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:19:03.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:03.382 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:03.441 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:03.441 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:03.441 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:03.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:19:03.598 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:03.657 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:19:03.657 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:19:03.657 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:19:03.657 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:03.658 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:03.659 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:03.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:03.660 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:19:03.714 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:19:03.714 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:19:03.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-01-16T15:19:03.794 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:03.835 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-01-16T15:19:03.836 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:19:03.837 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-01-16T15:19:03.838 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:03.838 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:03.838 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:03.838 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:03.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:03.839 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-01-16T15:19:03.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-01-16T15:19:03.915+0000 7f18ae23c480 -1 init, newargv = 0x55ae9ced8e60 newargc=15 2024-01-16T15:19:03.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[76178]: starting ceph client 2024-01-16T15:19:03.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[76178]: starting fuse 2024-01-16T15:19:03.928 INFO:teuthology.orchestra.run.smithi043.stdout:41 2024-01-16T15:19:03.928 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-01-16T15:19:03.929 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-01-16T15:19:03.930 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-01-16T15:19:04.141 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.76178.asok 2024-01-16T15:19:04.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:19:04.159 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76178.asok 2024-01-16T15:19:04.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:04.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76178.asok status 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:19:04.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "76178", 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 9321, 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 9321 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3055526421 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:19:04.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3055526421 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.9321 192.168.0.1:0/3055526421", 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/3055526421", 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 75, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 113, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-01-16T15:19:04.410 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:19:04.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:04.423 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-01-16T15:19:04.424 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:04.479 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-01-16T15:19:04.479 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:04.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:04.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:04.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:19:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.818+0000 7f4a6f600640 1 Processor -- start 2024-01-16T15:19:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.819+0000 7f4a6f600640 1 -- start start 2024-01-16T15:19:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.820+0000 7f4a6f600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a680aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.820+0000 7f4a6f600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a680b3470 con 0x7f4a680ac870 2024-01-16T15:19:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.820+0000 7f4a6f600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a680acca0 con 0x7f4a68146cc0 2024-01-16T15:19:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.820+0000 7f4a6f600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6806dbf0 con 0x7f4a68071230 2024-01-16T15:19:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.821+0000 7f4a6d600640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4a680ac870 0x7f4a6806e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38496/0 (socket says 172.21.15.43:38496) 2024-01-16T15:19:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.821+0000 7f4a6d600640 1 -- 172.21.15.43:0/1269459401 learned_addr learned my addr 172.21.15.43:0/1269459401 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.821+0000 7f4a67e00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a680aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.821+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352544574 0 0) 0x7f4a680b3470 con 0x7f4a680ac870 2024-01-16T15:19:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.821+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48003520 con 0x7f4a680ac870 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147810918 0 0) 0x7f4a48003520 con 0x7f4a680ac870 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 msgr2=0x7f4a680aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a680aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 >> v1:172.21.15.69:6789/0 conn(0x7f4a68146cc0 legacy=0x7f4a68070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a680b2bd0 con 0x7f4a680ac870 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4a600019f0 con 0x7f4a680ac870 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 >> v1:172.21.15.43:6789/0 conn(0x7f4a680ac870 legacy=0x7f4a6806e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a480039a0 0x7f4a48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6806dbf0 con 0x7f4a480047c0 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800ba60 con 0x7f4a480039a0 2024-01-16T15:19:04.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800bc20 con 0x7f4a68071230 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a67e00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.822+0000 7f4a6cc00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a480039a0 0x7f4a48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a67e00640 1 -- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 msgr2=0x7f4a48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a6d600640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a67e00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a48005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a67e00640 1 -- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a480039a0 msgr2=0x7f4a48003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a67e00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a480039a0 0x7f4a48003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a67e00640 1 -- 172.21.15.43:0/1269459401 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48003520 con 0x7f4a480047c0 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a6cc00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a480039a0 0x7f4a48003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.823+0000 7f4a6d600640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a68071230 0x7f4a48005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a67e00640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a48004c40 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009a90 tx=0x7f4a480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bacfa0e23f728bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5002c660 con 0x7f4a480047c0 2024-01-16T15:19:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5002c7c0 con 0x7f4a480047c0 2024-01-16T15:19:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a67400640 1 -- 172.21.15.43:0/1269459401 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5002cb50 con 0x7f4a480047c0 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a6f600640 1 -- 172.21.15.43:0/1269459401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 msgr2=0x7f4a48004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.824+0000 7f4a6f600640 1 --2- 172.21.15.43:0/1269459401 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a48004c40 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009a90 tx=0x7f4a480057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.825+0000 7f4a6d600640 1 -- 172.21.15.43:0/1269459401 reap_dead start 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.825+0000 7f4a6f600640 1 -- 172.21.15.43:0/1269459401 shutdown_connections 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.825+0000 7f4a6f600640 1 -- 172.21.15.43:0/1269459401 >> 172.21.15.43:0/1269459401 conn(0x7f4a6806a620 msgr2=0x7f4a6806aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.825+0000 7f4a6f600640 1 -- 172.21.15.43:0/1269459401 shutdown_connections 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.825+0000 7f4a6f600640 1 -- 172.21.15.43:0/1269459401 wait complete. 2024-01-16T15:19:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.827+0000 7f4a6f600640 1 Processor -- start 2024-01-16T15:19:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.827+0000 7f4a6f600640 1 -- start start 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6806e220 0x7f4a6806e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a681f6540 0x7f4a681f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a681f3fb0 con 0x7f4a480047c0 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6806ebe0 con 0x7f4a6806e220 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6f600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68138b30 con 0x7f4a681f6540 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47316/0 (socket says 172.21.15.43:47316) 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 -- 172.21.15.43:0/3042730138 learned_addr learned my addr 172.21.15.43:0/3042730138 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 -- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a681f6540 msgr2=0x7f4a681f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a681f6540 0x7f4a681f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 -- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6806e220 msgr2=0x7f4a6806e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6806e220 0x7f4a6806e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.828+0000 7f4a6cc00640 1 -- 172.21.15.43:0/3042730138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a681f39f0 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a6cc00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800d7a0 tx=0x7f4a5800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58017070 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a58005070 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58005420 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a681f3cd0 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.829+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a681f4590 con 0x7f4a480047c0 2024-01-16T15:19:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.830+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a681f4110 con 0x7f4a480047c0 2024-01-16T15:19:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.832+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4a58007030 con 0x7f4a480047c0 2024-01-16T15:19:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.832+0000 7f4a64c00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 0x7f4a4006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.833+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4a58088070 con 0x7f4a480047c0 2024-01-16T15:19:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.833+0000 7f4a67e00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 0x7f4a4006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.834+0000 7f4a67e00640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 0x7f4a4006e890 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4a50036000 tx=0x7f4a5002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:04.837+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a681f4110 con 0x7f4a480047c0 2024-01-16T15:19:05.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.076+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4a681f4110 con 0x7f4a480047c0 2024-01-16T15:19:05.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.077+0000 7f4a64c00640 1 -- 172.21.15.43:0/3042730138 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4a681f4110 con 0x7f4a480047c0 2024-01-16T15:19:05.067 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:19:05.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.079+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 msgr2=0x7f4a4006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.079+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 0x7f4a4006e890 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4a50036000 tx=0x7f4a5002e000 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 msgr2=0x7f4a681385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800d7a0 tx=0x7f4a5800dc60 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 shutdown_connections 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4a4006c3d0 0x7f4a4006e890 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6806e220 0x7f4a6806e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a681f6540 0x7f4a681f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 --2- 172.21.15.43:0/3042730138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a480047c0 0x7f4a681385f0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 >> 172.21.15.43:0/3042730138 conn(0x7f4a6806a620 msgr2=0x7f4a680af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 shutdown_connections 2024-01-16T15:19:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:05.080+0000 7f4a6f600640 1 -- 172.21.15.43:0/3042730138 wait complete. 2024-01-16T15:19:05.090 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:19:05.091 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:05.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.091 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-01-16T15:19:05.148 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:19:05.148 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:19:05.148 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:19:05.148 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:19:05.149 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41454sec preferred_lft 41454sec 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::5c44:54ff:fe90:1044/64 scope link 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 26:98:d5:7a:5a:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::2498:d5ff:fe7a:5ae8/64 scope link 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0a:fd:15:c8:0c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:05.150 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::8fd:15ff:fec8:cf9/64 scope link 2024-01-16T15:19:05.151 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:05.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.151 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:19:05.243 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-01-16T15:19:05.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.248 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-01-16T15:19:05.297 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-01-16T15:19:05.297 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:19:05.297 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:19:05.298 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:19:05.298 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:19:05.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.298 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-01-16T15:19:05.299 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-01-16T15:19:05.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:19:05.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:19:05.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:19:05.299 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-01-16T15:19:05.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.475 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:19:05.524 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:19:05.524 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:19:05.525 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:05.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-01-16T15:19:05.612 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:19:05.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:05.613 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:05.663 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:05.663 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:05.663 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:05.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:19:05.845 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:05.899 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:19:05.899 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:19:05.899 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:19:05.900 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:05.901 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:05.902 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:05.902 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:05.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:05.903 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:19:05.953 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:19:05.953 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:19:05.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-01-16T15:19:06.039 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-01-16T15:19:06.085 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-01-16T15:19:06.086 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:06.087 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:06.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:06.088 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-01-16T15:19:06.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-01-16T15:19:06.156+0000 7f52923f9480 -1 init, newargv = 0x559fb6c1dc40 newargc=15 2024-01-16T15:19:06.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[47352]: starting ceph client 2024-01-16T15:19:06.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[47352]: starting fuse 2024-01-16T15:19:06.173 INFO:teuthology.orchestra.run.smithi069.stdout:41 2024-01-16T15:19:06.173 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:19:06.174 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> 2024-01-16T15:19:06.175 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-01-16T15:19:06.176 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-01-16T15:19:06.372 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.47352.asok 2024-01-16T15:19:06.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-16T15:19:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:19:06.389 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47352.asok 2024-01-16T15:19:06.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:06.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47352.asok status 2024-01-16T15:19:06.625 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "47352", 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-01-16T15:19:06.626 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7417, 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7417 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:19:06.627 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 657669809 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 657669809 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7417 192.168.0.1:0/657669809", 2024-01-16T15:19:06.628 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/657669809", 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 75, 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 113, 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-01-16T15:19:06.629 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-16T15:19:06.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:06.641 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-01-16T15:19:06.641 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:06.695 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-01-16T15:19:06.695 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:06.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:06.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:06.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:19:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.033+0000 7fc0c7200640 1 Processor -- start 2024-01-16T15:19:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.034+0000 7fc0c7200640 1 -- start start 2024-01-16T15:19:07.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.035+0000 7fc0c7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0c80d1d90 0x7fc0c80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.035+0000 7fc0c7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c80c6bf0 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.035+0000 7fc0c7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c80d13f0 con 0x7fc0c815d3b0 2024-01-16T15:19:07.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.035+0000 7fc0c7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c80cedc0 con 0x7fc0c80d1d90 2024-01-16T15:19:07.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.036+0000 7fc0c6800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc0c80d0fa0 0x7fc0c80b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38498/0 (socket says 172.21.15.43:38498) 2024-01-16T15:19:07.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.036+0000 7fc0c6800640 1 -- 172.21.15.43:0/2088781886 learned_addr learned my addr 172.21.15.43:0/2088781886 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:07.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.036+0000 7fc0bd200640 1 --2- 172.21.15.43:0/2088781886 >> v2:172.21.15.69:3300/0 conn(0x7fc0c80d1d90 0x7fc0c80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.036+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853942733 0 0) 0x7fc0c80c6bf0 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.036+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a8003520 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156174305 0 0) 0x7fc0a8003520 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 >> v2:172.21.15.69:3300/0 conn(0x7fc0c80d1d90 msgr2=0x7fc0c80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 --2- 172.21.15.43:0/2088781886 >> v2:172.21.15.69:3300/0 conn(0x7fc0c80d1d90 0x7fc0c80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 >> v1:172.21.15.69:6789/0 conn(0x7fc0c815d3b0 legacy=0x7fc0c80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80c6e40 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc0c0001670 con 0x7fc0c80d0fa0 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 >> v1:172.21.15.43:6789/0 conn(0x7fc0c80d0fa0 legacy=0x7fc0c80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 0x7fc0a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80047c0 0x7fc0a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80052a0 0x7fc0a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c80cedc0 con 0x7fc0a80039e0 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a800c1f0 con 0x7fc0a80047c0 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a800c3b0 con 0x7fc0a80052a0 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0bd200640 1 --2- 172.21.15.43:0/2088781886 >> v2:172.21.15.69:3300/0 conn(0x7fc0c80d1d90 0x7fc0c80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.037+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 0x7fc0a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c5e00640 1 -- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80052a0 msgr2=0x7fc0a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80052a0 0x7fc0a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0bd200640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80047c0 0x7fc0a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c5e00640 1 -- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80047c0 msgr2=0x7fc0a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80047c0 0x7fc0a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c5e00640 1 -- 172.21.15.43:0/2088781886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a8003520 con 0x7fc0a80039e0 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0c6800640 1 -- 172.21.15.43:0/2088781886 reap_dead start 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.038+0000 7fc0bd200640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0a80047c0 0x7fc0a8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.039+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 0x7fc0a8003e00 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8001e20 tx=0x7fc0a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f66971dd312dad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.039+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8003b70 con 0x7fc0a80039e0 2024-01-16T15:19:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.039+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b8003cd0 con 0x7fc0a80039e0 2024-01-16T15:19:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.039+0000 7fc0c5400640 1 -- 172.21.15.43:0/2088781886 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8005310 con 0x7fc0a80039e0 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 -- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 msgr2=0x7fc0a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 0x7fc0a8003e00 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8001e20 tx=0x7fc0a80057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 -- 172.21.15.43:0/2088781886 shutdown_connections 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 --2- 172.21.15.43:0/2088781886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0a80039e0 0x7fc0a8003e00 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 -- 172.21.15.43:0/2088781886 >> 172.21.15.43:0/2088781886 conn(0x7fc0c80d30a0 msgr2=0x7fc0c80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 -- 172.21.15.43:0/2088781886 shutdown_connections 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.040+0000 7fc0c7200640 1 -- 172.21.15.43:0/2088781886 wait complete. 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.042+0000 7fc0c7200640 1 Processor -- start 2024-01-16T15:19:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.042+0000 7fc0c7200640 1 -- start start 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039e0 0x7fc0c81f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c80b7f10 0x7fc0c81f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c80b7a80 con 0x7fc0c81f3900 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c81f7130 con 0x7fc0c80b7f10 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0c7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c80b6e90 con 0x7fc0a80039e0 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0bd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0bd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47344/0 (socket says 172.21.15.43:47344) 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.043+0000 7fc0bd200640 1 -- 172.21.15.43:0/977037470 learned_addr learned my addr 172.21.15.43:0/977037470 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039e0 0x7fc0c81f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0c6800640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c80b7f10 0x7fc0c81f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 -- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039e0 msgr2=0x7fc0c81f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039e0 0x7fc0c81f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 -- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c80b7f10 msgr2=0x7fc0c81f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c80b7f10 0x7fc0c81f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 -- 172.21.15.43:0/977037470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80b7150 con 0x7fc0c81f3900 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.044+0000 7fc0bd200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000f4f0 tx=0x7fc0b000f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.045+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0014890 con 0x7fc0c81f3900 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.045+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000c030 con 0x7fc0c81f3900 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.045+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0014c30 con 0x7fc0c81f3900 2024-01-16T15:19:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.045+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c80b7430 con 0x7fc0c81f3900 2024-01-16T15:19:07.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.045+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c8147de0 con 0x7fc0c81f3900 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.046+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c804e840 con 0x7fc0c81f3900 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.047+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fc0b00133e0 con 0x7fc0c81f3900 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.048+0000 7fc0bea00640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 0x7fc0a006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.048+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc0b0088cf0 con 0x7fc0c81f3900 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.048+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 0x7fc0a006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.049+0000 7fc0c5e00640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 0x7fc0a006e860 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.053+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0b0055270 con 0x7fc0c81f3900 2024-01-16T15:19:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.288+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc0c80d23b0 con 0x7fc0c81f3900 2024-01-16T15:19:07.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.593+0000 7fc0bea00640 1 -- 172.21.15.43:0/977037470 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v76) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc0b00220c0 con 0x7fc0c81f3900 2024-01-16T15:19:07.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 msgr2=0x7fc0a006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 0x7fc0a006e860 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b8009030 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 msgr2=0x7fc0c81478a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000f4f0 tx=0x7fc0b000f9b0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 shutdown_connections 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc0a006c3a0 0x7fc0a006e860 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c80b7f10 0x7fc0c81f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0c81f3900 0x7fc0c81478a0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 --2- 172.21.15.43:0/977037470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0a80039e0 0x7fc0c81f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.597+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 >> 172.21.15.43:0/977037470 conn(0x7fc0c80d30a0 msgr2=0x7fc0c80c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:07.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.598+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 shutdown_connections 2024-01-16T15:19:07.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.598+0000 7fc0c7200640 1 -- 172.21.15.43:0/977037470 wait complete. 2024-01-16T15:19:07.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:19:07.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.841+0000 7f46c6e00640 1 Processor -- start 2024-01-16T15:19:07.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.842+0000 7f46c6e00640 1 -- start start 2024-01-16T15:19:07.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.843+0000 7f46c6e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 0x7f46c0070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.843+0000 7f46c6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.843+0000 7f46c6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c0070a90 con 0x7f46c00b2930 2024-01-16T15:19:07.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.843+0000 7f46c6e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c0070cf0 con 0x7f46c00b2550 2024-01-16T15:19:07.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.843+0000 7f46c6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0070fd0 con 0x7f46c0161700 2024-01-16T15:19:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.844+0000 7f46c6400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.844+0000 7f46c6400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:34884/0 (socket says 172.21.15.43:34884) 2024-01-16T15:19:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.844+0000 7f46c6400640 1 -- 172.21.15.43:0/2844213175 learned_addr learned my addr 172.21.15.43:0/2844213175 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.844+0000 7f46c5a00640 1 --2- 172.21.15.43:0/2844213175 >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 0x7f46c0070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.845+0000 7f46c6400640 1 -- 172.21.15.43:0/2844213175 >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 msgr2=0x7f46c0070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:07.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.845+0000 7f46c6400640 1 --2- 172.21.15.43:0/2844213175 >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 0x7f46c0070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.845+0000 7f46c6400640 1 -- 172.21.15.43:0/2844213175 >> v1:172.21.15.69:6789/0 conn(0x7f46c00b2930 legacy=0x7f46c0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.845+0000 7f46c6400640 1 -- 172.21.15.43:0/2844213175 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00712b0 con 0x7f46c0161700 2024-01-16T15:19:07.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.845+0000 7f46c5a00640 1 --2- 172.21.15.43:0/2844213175 >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 0x7f46c0070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:19:07.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.847+0000 7f46c6400640 1 --2- 172.21.15.43:0/2844213175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009e50 tx=0x7f46b802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47eaa2b9db1b956 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.847+0000 7f46bfe00640 1 -- 172.21.15.43:0/2844213175 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b802cb00 con 0x7f46c0161700 2024-01-16T15:19:07.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46bfe00640 1 -- 172.21.15.43:0/2844213175 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b802cc60 con 0x7f46c0161700 2024-01-16T15:19:07.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46bfe00640 1 -- 172.21.15.43:0/2844213175 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80343f0 con 0x7f46c0161700 2024-01-16T15:19:07.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 -- 172.21.15.43:0/2844213175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 msgr2=0x7f46c00ac900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 --2- 172.21.15.43:0/2844213175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009e50 tx=0x7f46b802bf60 comp rx=0 tx=0).stop 2024-01-16T15:19:07.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 -- 172.21.15.43:0/2844213175 shutdown_connections 2024-01-16T15:19:07.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 --2- 172.21.15.43:0/2844213175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c00ac900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 --2- 172.21.15.43:0/2844213175 >> v2:172.21.15.69:3300/0 conn(0x7f46c00b2550 0x7f46c0070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 -- 172.21.15.43:0/2844213175 >> 172.21.15.43:0/2844213175 conn(0x7f46c00b0840 msgr2=0x7f46c00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:07.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 -- 172.21.15.43:0/2844213175 shutdown_connections 2024-01-16T15:19:07.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.848+0000 7f46c6e00640 1 -- 172.21.15.43:0/2844213175 wait complete. 2024-01-16T15:19:07.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.850+0000 7f46c6e00640 1 Processor -- start 2024-01-16T15:19:07.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.850+0000 7f46c6e00640 1 -- start start 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b2930 0x7f46c015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c01f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00ac870 con 0x7f46c00b2550 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c015a830 con 0x7f46c00b2930 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.851+0000 7f46c6e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c01f6400 con 0x7f46c0161700 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47354/0 (socket says 172.21.15.43:47354) 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 -- 172.21.15.43:0/328894839 learned_addr learned my addr 172.21.15.43:0/328894839 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c6400640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c01f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5000640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b2930 0x7f46c015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 -- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 msgr2=0x7f46c01f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c01f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 -- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b2930 msgr2=0x7f46c015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b2930 0x7f46c015a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.852+0000 7f46c5a00640 1 -- 172.21.15.43:0/328894839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0009400 con 0x7f46c00b2550 2024-01-16T15:19:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46c5a00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e960 tx=0x7f46b000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000ca00 con 0x7f46c00b2550 2024-01-16T15:19:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b000cb60 con 0x7f46c00b2550 2024-01-16T15:19:07.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0013470 con 0x7f46c00b2550 2024-01-16T15:19:07.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b8009b30 con 0x7f46c00b2550 2024-01-16T15:19:07.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.853+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c01f6980 con 0x7f46c00b2550 2024-01-16T15:19:07.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.854+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c004e840 con 0x7f46c00b2550 2024-01-16T15:19:07.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.855+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f46b00135d0 con 0x7f46c00b2550 2024-01-16T15:19:07.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.856+0000 7f46bd600640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 0x7f469806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:07.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.857+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f46b0088ca0 con 0x7f46c00b2550 2024-01-16T15:19:07.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.857+0000 7f46c5000640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 0x7f469806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:07.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.857+0000 7f46c5000640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 0x7f469806e890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f46c00af970 tx=0x7f46a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:07.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:07.861+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46b0055220 con 0x7f46c00b2550 2024-01-16T15:19:08.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.082+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c0071870 con 0x7f469806c3d0 2024-01-16T15:19:08.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.085+0000 7f46bd600640 1 -- 172.21.15.43:0/328894839 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f46a8001c70 con 0x7f469806c3d0 2024-01-16T15:19:08.076 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:19:08.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:19:08.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:19:08.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:19:08.076 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 msgr2=0x7f469806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 0x7f469806e890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f46c00af970 tx=0x7f46a8009040 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 msgr2=0x7f46c014a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e960 tx=0x7f46b000ee20 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 shutdown_connections 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f469806c3d0 0x7f469806e890 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b2930 0x7f46c015a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c0161700 0x7f46c01f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 --2- 172.21.15.43:0/328894839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46c00b2550 0x7f46c014a070 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 >> 172.21.15.43:0/328894839 conn(0x7f46c00b0840 msgr2=0x7f46c00b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 shutdown_connections 2024-01-16T15:19:08.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.088+0000 7f46c6e00640 1 -- 172.21.15.43:0/328894839 wait complete. 2024-01-16T15:19:08.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:19:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.333+0000 7f3263000640 1 Processor -- start 2024-01-16T15:19:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.334+0000 7f3263000640 1 -- start start 2024-01-16T15:19:08.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.335+0000 7f3263000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.335+0000 7f3263000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f325c15e600 0x7f325c0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.335+0000 7f3263000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c15db60 con 0x7f325c0b43e0 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.335+0000 7f3263000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c15d950 con 0x7f325c15e600 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.335+0000 7f3263000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b3450 con 0x7f325c0b2b30 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f3261000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f325c0b43e0 0x7f325c06b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38506/0 (socket says 172.21.15.43:38506) 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f3261000640 1 -- 172.21.15.43:0/3019897436 learned_addr learned my addr 172.21.15.43:0/3019897436 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f325be00640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f325b400640 1 --2- 172.21.15.43:0/3019897436 >> v2:172.21.15.69:3300/0 conn(0x7f325c15e600 0x7f325c0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744822829 0 0) 0x7f325c15db60 con 0x7f325c0b43e0 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.336+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f323c003520 con 0x7f325c0b43e0 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.337+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952362582 0 0) 0x7f323c003520 con 0x7f325c0b43e0 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.337+0000 7f325b400640 1 -- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 msgr2=0x7f325c0b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.337+0000 7f325b400640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.337+0000 7f325b400640 1 -- 172.21.15.43:0/3019897436 >> v1:172.21.15.43:6789/0 conn(0x7f325c0b43e0 legacy=0x7f325c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.337+0000 7f325b400640 1 -- 172.21.15.43:0/3019897436 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0b4bd0 con 0x7f325c15e600 2024-01-16T15:19:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.338+0000 7f325b400640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c15e600 0x7f325c0b3ea0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f32440097a0 tx=0x7f324402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99ea74d87792f981 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.339+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32540019f0 con 0x7f325c15e600 2024-01-16T15:19:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.339+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324402e050 con 0x7f325c15e600 2024-01-16T15:19:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.339+0000 7f325aa00640 1 -- 172.21.15.43:0/3019897436 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244030730 con 0x7f325c15e600 2024-01-16T15:19:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.339+0000 7f3263000640 1 -- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c15e600 msgr2=0x7f325c0b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.339+0000 7f3263000640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c15e600 0x7f325c0b3ea0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f32440097a0 tx=0x7f324402b710 comp rx=0 tx=0).stop 2024-01-16T15:19:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 -- 172.21.15.43:0/3019897436 shutdown_connections 2024-01-16T15:19:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c15e600 0x7f325c0b3ea0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 --2- 172.21.15.43:0/3019897436 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 -- 172.21.15.43:0/3019897436 >> 172.21.15.43:0/3019897436 conn(0x7f325c0b63c0 msgr2=0x7f325c0b67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 -- 172.21.15.43:0/3019897436 shutdown_connections 2024-01-16T15:19:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.340+0000 7f3263000640 1 -- 172.21.15.43:0/3019897436 wait complete. 2024-01-16T15:19:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.341+0000 7f3263000640 1 Processor -- start 2024-01-16T15:19:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 -- start start 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c0b43e0 0x7f325c1e8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1e7c10 con 0x7f325c15e600 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c135580 con 0x7f325c0b43e0 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.342+0000 7f3263000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1e7e20 con 0x7f325c0b2b30 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47358/0 (socket says 172.21.15.43:47358) 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 -- 172.21.15.43:0/1746067837 learned_addr learned my addr 172.21.15.43:0/1746067837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f325be00640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f325b400640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c0b43e0 0x7f325c1e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 -- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 msgr2=0x7f325c0cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 -- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c0b43e0 msgr2=0x7f325c1e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c0b43e0 0x7f325c1e8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.343+0000 7f3261000640 1 -- 172.21.15.43:0/1746067837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3244009400 con 0x7f325c15e600 2024-01-16T15:19:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3261000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f3254001560 tx=0x7f325400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254012400 con 0x7f325c15e600 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254012560 con 0x7f325c15e600 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254012910 con 0x7f325c15e600 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c1e8100 con 0x7f325c15e600 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.344+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c0d1480 con 0x7f325c15e600 2024-01-16T15:19:08.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.345+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c1e8620 con 0x7f325c15e600 2024-01-16T15:19:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.346+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f3254012a70 con 0x7f325c15e600 2024-01-16T15:19:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.347+0000 7f3253e00640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 0x7f323406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.347+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3254089300 con 0x7f325c15e600 2024-01-16T15:19:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.348+0000 7f325be00640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 0x7f323406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.348+0000 7f325be00640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 0x7f323406e860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f324c009ad0 tx=0x7f324c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.352+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f325c1e8620 con 0x7f325c15e600 2024-01-16T15:19:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.545+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f325c1e8620 con 0x7f325c15e600 2024-01-16T15:19:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.546+0000 7f3253e00640 1 -- 172.21.15.43:0/1746067837 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f325c1e8620 con 0x7f325c15e600 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 msgr2=0x7f323406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 0x7f323406e860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f324c009ad0 tx=0x7f324c008040 comp rx=0 tx=0).stop 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 msgr2=0x7f325c0d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f3254001560 tx=0x7f325400a430 comp rx=0 tx=0).stop 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 shutdown_connections 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f323406c3a0 0x7f323406e860 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f325c0b43e0 0x7f325c1e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f325c15e600 0x7f325c0d0f40 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 --2- 172.21.15.43:0/1746067837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f325c0b2b30 0x7f325c0cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 >> 172.21.15.43:0/1746067837 conn(0x7f325c0b63c0 msgr2=0x7f325c0b67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 shutdown_connections 2024-01-16T15:19:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.549+0000 7f3263000640 1 -- 172.21.15.43:0/1746067837 wait complete. 2024-01-16T15:19:08.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 30 2024-01-16T15:19:08.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.794+0000 7f972f000640 1 Processor -- start 2024-01-16T15:19:08.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.794+0000 7f972f000640 1 -- start start 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.795+0000 7f972f000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f972815f270 0x7f97280cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972f000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972815f930 con 0x7f9728144770 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97280b3b10 con 0x7f972806d9e0 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972f000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97280b43b0 con 0x7f972815f270 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f972815f270 0x7f97280cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972dc00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9728144770 0x7f972815eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38508/0 (socket says 172.21.15.43:38508) 2024-01-16T15:19:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972dc00640 1 -- 172.21.15.43:0/740016763 learned_addr learned my addr 172.21.15.43:0/740016763 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:08.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972d200640 1 -- 172.21.15.43:0/740016763 >> v1:172.21.15.69:6789/0 conn(0x7f972806d9e0 legacy=0x7f97280cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:08.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972d200640 1 -- 172.21.15.43:0/740016763 >> v1:172.21.15.43:6789/0 conn(0x7f9728144770 legacy=0x7f972815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f9727e00640 1 -- 172.21.15.43:0/740016763 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481894038 0 0) 0x7f972815f930 con 0x7f9728144770 2024-01-16T15:19:08.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.796+0000 7f972d200640 1 -- 172.21.15.43:0/740016763 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97280b3df0 con 0x7f972815f270 2024-01-16T15:19:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f972d200640 1 --2- 172.21.15.43:0/740016763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f972815f270 0x7f97280cd4a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f97100097a0 tx=0x7f971002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63c527deba90a21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f9727e00640 1 -- 172.21.15.43:0/740016763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9710031830 con 0x7f972815f270 2024-01-16T15:19:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f9727e00640 1 -- 172.21.15.43:0/740016763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9710031990 con 0x7f972815f270 2024-01-16T15:19:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f9727e00640 1 -- 172.21.15.43:0/740016763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9710031d20 con 0x7f972815f270 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f972f000640 1 -- 172.21.15.43:0/740016763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f972815f270 msgr2=0x7f97280cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.798+0000 7f972f000640 1 --2- 172.21.15.43:0/740016763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f972815f270 0x7f97280cd4a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f97100097a0 tx=0x7f971002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.799+0000 7f972f000640 1 -- 172.21.15.43:0/740016763 shutdown_connections 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.799+0000 7f972f000640 1 --2- 172.21.15.43:0/740016763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f972815f270 0x7f97280cd4a0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.799+0000 7f972f000640 1 -- 172.21.15.43:0/740016763 >> 172.21.15.43:0/740016763 conn(0x7f97280cec10 msgr2=0x7f97280cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.799+0000 7f972f000640 1 -- 172.21.15.43:0/740016763 shutdown_connections 2024-01-16T15:19:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.799+0000 7f972f000640 1 -- 172.21.15.43:0/740016763 wait complete. 2024-01-16T15:19:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.800+0000 7f972f000640 1 Processor -- start 2024-01-16T15:19:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.801+0000 7f972f000640 1 -- start start 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f972806d9e0 0x7f972813ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f972815f270 0x7f972813ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97281409c0 con 0x7f9728144770 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728140ca0 con 0x7f972815f270 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972813c350 con 0x7f972806d9e0 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47362/0 (socket says 172.21.15.43:47362) 2024-01-16T15:19:08.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 -- 172.21.15.43:0/2271750009 learned_addr learned my addr 172.21.15.43:0/2271750009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972dc00640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f972806d9e0 0x7f972813ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972e600640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f972815f270 0x7f972813ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 -- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f972806d9e0 msgr2=0x7f972813ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f972806d9e0 0x7f972813ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 -- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f972815f270 msgr2=0x7f972813ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f972815f270 0x7f972813ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.802+0000 7f972d200640 1 -- 172.21.15.43:0/2271750009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710009400 con 0x7f9728144770 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f972d200640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f97100097a0 tx=0x7f97100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971003abe0 con 0x7f9728144770 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971003ad40 con 0x7f9728144770 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9710044630 con 0x7f9728144770 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972813c630 con 0x7f9728144770 2024-01-16T15:19:08.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.803+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972813ab10 con 0x7f9728144770 2024-01-16T15:19:08.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.804+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972813cb50 con 0x7f9728144770 2024-01-16T15:19:08.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.805+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f971004d450 con 0x7f9728144770 2024-01-16T15:19:08.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.806+0000 7f9725600640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 0x7f970006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:08.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.806+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f97100aa9b0 con 0x7f9728144770 2024-01-16T15:19:08.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.807+0000 7f972dc00640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 0x7f970006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:08.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.807+0000 7f972dc00640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 0x7f970006e860 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9718007fd0 tx=0x7f9718009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:08.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:08.811+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f972813cb50 con 0x7f9728144770 2024-01-16T15:19:08.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.005+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f972813cb50 con 0x7f9728144770 2024-01-16T15:19:09.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.011+0000 7f9725600640 1 -- 172.21.15.43:0/2271750009 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v43)=0 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7f972813cb50 con 0x7f9728144770 2024-01-16T15:19:09.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 msgr2=0x7f970006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 0x7f970006e860 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9718007fd0 tx=0x7f9718009040 comp rx=0 tx=0).stop 2024-01-16T15:19:09.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 msgr2=0x7f972813a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f97100097a0 tx=0x7f97100046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 shutdown_connections 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f970006c3a0 0x7f970006e860 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f972815f270 0x7f972813ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9728144770 0x7f972813a5d0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 --2- 172.21.15.43:0/2271750009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f972806d9e0 0x7f972813ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 >> 172.21.15.43:0/2271750009 conn(0x7f97280cec10 msgr2=0x7f9728070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 shutdown_connections 2024-01-16T15:19:09.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.014+0000 7f972f000640 1 -- 172.21.15.43:0/2271750009 wait complete. 2024-01-16T15:19:09.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:19:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.258+0000 7face4400640 1 Processor -- start 2024-01-16T15:19:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.259+0000 7face4400640 1 -- start start 2024-01-16T15:19:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.260+0000 7face4400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7facdc15b350 0x7facdc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.260+0000 7face4400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 0x7facdc0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.260+0000 7face4400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc15b730 con 0x7facdc0adb60 2024-01-16T15:19:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.260+0000 7face4400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0ac740 con 0x7facdc15b350 2024-01-16T15:19:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.260+0000 7face4400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0ac460 con 0x7facdc09cb30 2024-01-16T15:19:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.261+0000 7face1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 0x7facdc0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.261+0000 7face2400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7facdc0adb60 0x7facdc0adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:52014/0 (socket says 172.21.15.43:52014) 2024-01-16T15:19:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.261+0000 7face2400640 1 -- 172.21.15.43:0/2051142231 learned_addr learned my addr 172.21.15.43:0/2051142231 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.261+0000 7face1a00640 1 --2- 172.21.15.43:0/2051142231 >> v2:172.21.15.69:3300/0 conn(0x7facdc15b350 0x7facdc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7facdbe00640 1 -- 172.21.15.43:0/2051142231 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275080542 0 0) 0x7facd4001490 con 0x7facdc0adb60 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7facdbe00640 1 -- 172.21.15.43:0/2051142231 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc003520 con 0x7facdc0adb60 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7face1000640 1 -- 172.21.15.43:0/2051142231 >> v2:172.21.15.69:3300/0 conn(0x7facdc15b350 msgr2=0x7facdc0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7face1000640 1 --2- 172.21.15.43:0/2051142231 >> v2:172.21.15.69:3300/0 conn(0x7facdc15b350 0x7facdc0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7face1000640 1 -- 172.21.15.43:0/2051142231 >> v1:172.21.15.69:6789/0 conn(0x7facdc0adb60 legacy=0x7facdc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.262+0000 7face1000640 1 -- 172.21.15.43:0/2051142231 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc0b2a50 con 0x7facdc09cb30 2024-01-16T15:19:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.263+0000 7face1000640 1 --2- 172.21.15.43:0/2051142231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 0x7facdc0b3530 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7facc40097a0 tx=0x7facc402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=530fd8791e5bd9c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.264+0000 7facdbe00640 1 -- 172.21.15.43:0/2051142231 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4031830 con 0x7facdc09cb30 2024-01-16T15:19:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.264+0000 7facdbe00640 1 -- 172.21.15.43:0/2051142231 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc4031990 con 0x7facdc09cb30 2024-01-16T15:19:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.264+0000 7facdbe00640 1 -- 172.21.15.43:0/2051142231 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4031d20 con 0x7facdc09cb30 2024-01-16T15:19:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.264+0000 7face4400640 1 -- 172.21.15.43:0/2051142231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 msgr2=0x7facdc0b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.264+0000 7face4400640 1 --2- 172.21.15.43:0/2051142231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 0x7facdc0b3530 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7facc40097a0 tx=0x7facc402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 -- 172.21.15.43:0/2051142231 shutdown_connections 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 --2- 172.21.15.43:0/2051142231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc09cb30 0x7facdc0b3530 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 --2- 172.21.15.43:0/2051142231 >> v2:172.21.15.69:3300/0 conn(0x7facdc15b350 0x7facdc0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 -- 172.21.15.43:0/2051142231 >> 172.21.15.43:0/2051142231 conn(0x7facdc0b0ba0 msgr2=0x7facdc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 -- 172.21.15.43:0/2051142231 shutdown_connections 2024-01-16T15:19:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.265+0000 7face4400640 1 -- 172.21.15.43:0/2051142231 wait complete. 2024-01-16T15:19:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.267+0000 7face4400640 1 Processor -- start 2024-01-16T15:19:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.267+0000 7face4400640 1 -- start start 2024-01-16T15:19:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facdc09cb30 0x7facdc0aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc15b350 0x7facdc1e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc1e6240 con 0x7facdc0adb60 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc14c450 con 0x7facdc09cb30 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face4400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc1e55c0 con 0x7facdc15b350 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47372/0 (socket says 172.21.15.43:47372) 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.268+0000 7face1000640 1 -- 172.21.15.43:0/1646134376 learned_addr learned my addr 172.21.15.43:0/1646134376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1a00640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facdc09cb30 0x7facdc0aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face2400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc15b350 0x7facdc1e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 -- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc15b350 msgr2=0x7facdc1e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc15b350 0x7facdc1e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 -- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facdc09cb30 msgr2=0x7facdc0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facdc09cb30 0x7facdc0aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 -- 172.21.15.43:0/1646134376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4009400 con 0x7facdc0adb60 2024-01-16T15:19:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face1000640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7facc4009890 tx=0x7facc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc403a8f0 con 0x7facdc0adb60 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc403aa70 con 0x7facdc0adb60 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc1e58a0 con 0x7facdc0adb60 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.269+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc403ae20 con 0x7facdc0adb60 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.270+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc1e5e60 con 0x7facdc0adb60 2024-01-16T15:19:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.271+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc09d200 con 0x7facdc0adb60 2024-01-16T15:19:09.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.272+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7facc404d440 con 0x7facdc0adb60 2024-01-16T15:19:09.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.273+0000 7facd9600640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 0x7facb406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.273+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7facc40aa990 con 0x7facdc0adb60 2024-01-16T15:19:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.273+0000 7face1a00640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 0x7facb406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.274+0000 7face1a00640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 0x7facb406e860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faccc009970 tx=0x7faccc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.277+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facc4077250 con 0x7facdc0adb60 2024-01-16T15:19:09.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.473+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7facdc04e920 con 0x7facdc0adb60 2024-01-16T15:19:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.479+0000 7facd9600640 1 -- 172.21.15.43:0/1646134376 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v44)=0 v44) v1 ==== 137+0+0 (secure 0 0 0) 0x7facc407a870 con 0x7facdc0adb60 2024-01-16T15:19:09.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 msgr2=0x7facb406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 0x7facb406e860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faccc009970 tx=0x7faccc008040 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 msgr2=0x7facdc14bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7facc4009890 tx=0x7facc40046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 shutdown_connections 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7facb406c3a0 0x7facb406e860 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7facdc15b350 0x7facdc1e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facdc0adb60 0x7facdc14bf10 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 --2- 172.21.15.43:0/1646134376 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facdc09cb30 0x7facdc0aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 >> 172.21.15.43:0/1646134376 conn(0x7facdc0b0ba0 msgr2=0x7facdc09ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 shutdown_connections 2024-01-16T15:19:09.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.482+0000 7face4400640 1 -- 172.21.15.43:0/1646134376 wait complete. 2024-01-16T15:19:09.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:19:09.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.728+0000 7f2306600640 1 Processor -- start 2024-01-16T15:19:09.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.729+0000 7f2306600640 1 -- start start 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f2300071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23000cc920 0x7f23000af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f230006a440 0x7f230006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230006ae10 con 0x7f230006a440 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000723c0 con 0x7f23000cc920 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f2306600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23000cfe50 con 0x7f23000cd440 2024-01-16T15:19:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f22ffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f230006a440 0x7f230006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f22ffe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f230006a440 0x7f230006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47376/0 (socket says 172.21.15.43:47376) 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.730+0000 7f22ffe00640 1 -- 172.21.15.43:0/2981325385 learned_addr learned my addr 172.21.15.43:0/2981325385 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ff400640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f2300071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22fea00640 1 --2- 172.21.15.43:0/2981325385 >> v2:172.21.15.69:3300/0 conn(0x7f23000cc920 0x7f23000af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ffe00640 1 -- 172.21.15.43:0/2981325385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 msgr2=0x7f2300071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ffe00640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f2300071ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ffe00640 1 -- 172.21.15.43:0/2981325385 >> v2:172.21.15.69:3300/0 conn(0x7f23000cc920 msgr2=0x7f23000af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ffe00640 1 --2- 172.21.15.43:0/2981325385 >> v2:172.21.15.69:3300/0 conn(0x7f23000cc920 0x7f23000af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.731+0000 7f22ffe00640 1 -- 172.21.15.43:0/2981325385 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000ccd60 con 0x7f230006a440 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.732+0000 7f22ffe00640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230006a8d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f23000b2d50 tx=0x7f22f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccf654354c62566 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f22fe000640 1 -- 172.21.15.43:0/2981325385 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8031870 con 0x7f230006a440 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f22fe000640 1 -- 172.21.15.43:0/2981325385 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f80319d0 con 0x7f230006a440 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f22fe000640 1 -- 172.21.15.43:0/2981325385 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8031d60 con 0x7f230006a440 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 -- 172.21.15.43:0/2981325385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 msgr2=0x7f230006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230006a8d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f23000b2d50 tx=0x7f22f802b910 comp rx=0 tx=0).stop 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 -- 172.21.15.43:0/2981325385 shutdown_connections 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230006a8d0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 --2- 172.21.15.43:0/2981325385 >> v2:172.21.15.69:3300/0 conn(0x7f23000cc920 0x7f23000af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 --2- 172.21.15.43:0/2981325385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f2300071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 -- 172.21.15.43:0/2981325385 >> 172.21.15.43:0/2981325385 conn(0x7f23000b1c90 msgr2=0x7f23000b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 -- 172.21.15.43:0/2981325385 shutdown_connections 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.733+0000 7f2306600640 1 -- 172.21.15.43:0/2981325385 wait complete. 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.735+0000 7f2306600640 1 Processor -- start 2024-01-16T15:19:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.735+0000 7f2306600640 1 -- start start 2024-01-16T15:19:09.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 0x7f23000b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f23000b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230015fec0 con 0x7f230006a440 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230014cbb0 con 0x7f23000cc920 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f2306600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230014c1f0 con 0x7f23000cd440 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f22ff400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f22ff400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47384/0 (socket says 172.21.15.43:47384) 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.736+0000 7f22ff400640 1 -- 172.21.15.43:0/2220267498 learned_addr learned my addr 172.21.15.43:0/2220267498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ffe00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f23000b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22fea00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 0x7f23000b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 -- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 msgr2=0x7f23000b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f23000b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 -- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 msgr2=0x7f23000b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 0x7f23000b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 -- 172.21.15.43:0/2220267498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80094d0 con 0x7f230006a440 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22fea00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 0x7f23000b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ffe00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f23000b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22ff400640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d860 tx=0x7f22e800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.737+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800de50 con 0x7f230006a440 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.738+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800b8d0 con 0x7f230006a440 2024-01-16T15:19:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.738+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23000b5390 con 0x7f230006a440 2024-01-16T15:19:09.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.738+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800bc60 con 0x7f230006a440 2024-01-16T15:19:09.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.738+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230014c560 con 0x7f230006a440 2024-01-16T15:19:09.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.739+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230009e2f0 con 0x7f230006a440 2024-01-16T15:19:09.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.740+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f22e801e460 con 0x7f230006a440 2024-01-16T15:19:09.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.741+0000 7f22f7400640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 0x7f22d806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.741+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f22e8088c80 con 0x7f230006a440 2024-01-16T15:19:09.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.741+0000 7f22fea00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 0x7f22d806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.742+0000 7f22fea00640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 0x7f22d806e890 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f23000c3bc0 tx=0x7f22f000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.745+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22e8055540 con 0x7f230006a440 2024-01-16T15:19:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:09.965+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f23000cf800 con 0x7f22d806c3d0 2024-01-16T15:19:09.966 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:19:09.975+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:09.966 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:19:09.975+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:09.966 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:19:09.975+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:09.966 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:19:09.975+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:09.966 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:19:09.975+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.040+0000 7f22f7400640 1 -- 172.21.15.43:0/2220267498 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f22f00096f0 con 0x7f22d806c3d0 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.043+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 msgr2=0x7f22d806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.043+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 0x7f22d806e890 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f23000c3bc0 tx=0x7f22f000b040 comp rx=0 tx=0).stop 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 msgr2=0x7f230014bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d860 tx=0x7f22e800dd20 comp rx=0 tx=0).stop 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 shutdown_connections 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f22d806c3d0 0x7f22d806e890 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23000cc920 0x7f23000b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23000cd440 0x7f23000b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 --2- 172.21.15.43:0/2220267498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230006a440 0x7f230014bcb0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 >> 172.21.15.43:0/2220267498 conn(0x7f23000b1c90 msgr2=0x7f23000ad070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 shutdown_connections 2024-01-16T15:19:10.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.044+0000 7f2306600640 1 -- 172.21.15.43:0/2220267498 wait complete. 2024-01-16T15:19:10.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:19:10.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.285+0000 7f7e53e00640 1 Processor -- start 2024-01-16T15:19:10.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.285+0000 7f7e53e00640 1 -- start start 2024-01-16T15:19:10.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e5415efa0 0x7f7e54161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e540b3080 0x7f7e5406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5406c9a0 con 0x7f7e5406da40 2024-01-16T15:19:10.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5406cc80 con 0x7f7e540b3080 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5406cf60 con 0x7f7e5415efa0 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e540b3080 0x7f7e5406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7e540b3080 0x7f7e5406c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47402/0 (socket says 172.21.15.43:47402) 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.287+0000 7f7e53400640 1 -- 172.21.15.43:0/2477065967 learned_addr learned my addr 172.21.15.43:0/2477065967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.288+0000 7f7e52a00640 1 --2- 172.21.15.43:0/2477065967 >> v2:172.21.15.69:3300/0 conn(0x7f7e5415efa0 0x7f7e54161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.288+0000 7f7e53400640 1 -- 172.21.15.43:0/2477065967 >> v2:172.21.15.69:3300/0 conn(0x7f7e5415efa0 msgr2=0x7f7e54161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.288+0000 7f7e53400640 1 --2- 172.21.15.43:0/2477065967 >> v2:172.21.15.69:3300/0 conn(0x7f7e5415efa0 0x7f7e54161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.288+0000 7f7e53400640 1 -- 172.21.15.43:0/2477065967 >> v1:172.21.15.69:6789/0 conn(0x7f7e5406da40 legacy=0x7f7e540b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.288+0000 7f7e53400640 1 -- 172.21.15.43:0/2477065967 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5406d240 con 0x7f7e540b3080 2024-01-16T15:19:10.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.289+0000 7f7e53400640 1 --2- 172.21.15.43:0/2477065967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e5406c310 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7e54070970 tx=0x7f7e4c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd1e5c09b409c45 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.289+0000 7f7e52000640 1 -- 172.21.15.43:0/2477065967 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c02c9f0 con 0x7f7e540b3080 2024-01-16T15:19:10.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.289+0000 7f7e52000640 1 -- 172.21.15.43:0/2477065967 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c02cb50 con 0x7f7e540b3080 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.289+0000 7f7e52000640 1 -- 172.21.15.43:0/2477065967 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0343f0 con 0x7f7e540b3080 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 -- 172.21.15.43:0/2477065967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 msgr2=0x7f7e5406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 --2- 172.21.15.43:0/2477065967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e5406c310 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7e54070970 tx=0x7f7e4c02be50 comp rx=0 tx=0).stop 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 -- 172.21.15.43:0/2477065967 shutdown_connections 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 --2- 172.21.15.43:0/2477065967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e5406c310 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 --2- 172.21.15.43:0/2477065967 >> v2:172.21.15.69:3300/0 conn(0x7f7e5415efa0 0x7f7e54161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 -- 172.21.15.43:0/2477065967 >> 172.21.15.43:0/2477065967 conn(0x7f7e540cec10 msgr2=0x7f7e540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 -- 172.21.15.43:0/2477065967 shutdown_connections 2024-01-16T15:19:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.290+0000 7f7e53e00640 1 -- 172.21.15.43:0/2477065967 wait complete. 2024-01-16T15:19:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.292+0000 7f7e53e00640 1 Processor -- start 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.292+0000 7f7e53e00640 1 -- start start 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5406da40 0x7f7e540af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415efa0 0x7f7e541ece60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e541edc40 con 0x7f7e540b3080 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540ae890 con 0x7f7e5415efa0 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e53e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e54147250 con 0x7f7e5406da40 2024-01-16T15:19:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.293+0000 7f7e4be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47404/0 (socket says 172.21.15.43:47404) 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 -- 172.21.15.43:0/3581041100 learned_addr learned my addr 172.21.15.43:0/3581041100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e52a00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5406da40 0x7f7e540af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 -- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5406da40 msgr2=0x7f7e540af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5406da40 0x7f7e540af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e53400640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415efa0 0x7f7e541ece60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 -- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415efa0 msgr2=0x7f7e541ece60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415efa0 0x7f7e541ece60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 -- 172.21.15.43:0/3581041100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c009b30 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.294+0000 7f7e4be00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00c3c0 tx=0x7f7e3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.295+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c016070 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.295+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c007020 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.295+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c0073d0 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.295+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e541ed960 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.295+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e540aee10 con 0x7f7e540b3080 2024-01-16T15:19:10.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.296+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e540af030 con 0x7f7e540b3080 2024-01-16T15:19:10.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.297+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f7e3c007530 con 0x7f7e540b3080 2024-01-16T15:19:10.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.298+0000 7f7e4b400640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 0x7f7e2c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.298+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7e3c087f90 con 0x7f7e540b3080 2024-01-16T15:19:10.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.298+0000 7f7e52a00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 0x7f7e2c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.299+0000 7f7e52a00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 0x7f7e2c06e860 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f7e44006fd0 tx=0x7f7e44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.302+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e3c054850 con 0x7f7e540b3080 2024-01-16T15:19:10.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.529+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e540b4d20 con 0x7f7e2c06c3a0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.536+0000 7f7e4b400640 1 -- 172.21.15.43:0/3581041100 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7e4400f860 con 0x7f7e2c06c3a0 2024-01-16T15:19:10.527 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.538+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 msgr2=0x7f7e2c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.538+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 0x7f7e2c06e860 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f7e44006fd0 tx=0x7f7e44008040 comp rx=0 tx=0).stop 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.538+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 msgr2=0x7f7e54146d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00c3c0 tx=0x7f7e3c00c880 comp rx=0 tx=0).stop 2024-01-16T15:19:10.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 shutdown_connections 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7e2c06c3a0 0x7f7e2c06e860 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415efa0 0x7f7e541ece60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e540b3080 0x7f7e54146d10 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 --2- 172.21.15.43:0/3581041100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e5406da40 0x7f7e540af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 >> 172.21.15.43:0/3581041100 conn(0x7f7e540cec10 msgr2=0x7f7e54161e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 shutdown_connections 2024-01-16T15:19:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.539+0000 7f7e53e00640 1 -- 172.21.15.43:0/3581041100 wait complete. 2024-01-16T15:19:10.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:19:10.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.786+0000 7f0596c00640 1 Processor -- start 2024-01-16T15:19:10.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.786+0000 7f0596c00640 1 -- start start 2024-01-16T15:19:10.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.787+0000 7f0596c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05900ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.787+0000 7f0596c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05900d0fa0 0x7f05900b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.787+0000 7f0596c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900c6bf0 con 0x7f059015d3b0 2024-01-16T15:19:10.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.787+0000 7f0596c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05900d13d0 con 0x7f05900d0fa0 2024-01-16T15:19:10.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.787+0000 7f0596c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cedc0 con 0x7f05900d1d90 2024-01-16T15:19:10.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f0595800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f059015d3b0 0x7f05900d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38522/0 (socket says 172.21.15.43:38522) 2024-01-16T15:19:10.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f0595800640 1 -- 172.21.15.43:0/2886687012 learned_addr learned my addr 172.21.15.43:0/2886687012 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f0594e00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05900ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f0596200640 1 --2- 172.21.15.43:0/2886687012 >> v2:172.21.15.69:3300/0 conn(0x7f05900d0fa0 0x7f05900b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024118302 0 0) 0x7f05900c6bf0 con 0x7f059015d3b0 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.788+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0570003520 con 0x7f059015d3b0 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2967462928 0 0) 0x7f0570003520 con 0x7f059015d3b0 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 msgr2=0x7f05900ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05900ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 >> v2:172.21.15.69:3300/0 conn(0x7f05900d0fa0 msgr2=0x7f05900b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 --2- 172.21.15.43:0/2886687012 >> v2:172.21.15.69:3300/0 conn(0x7f05900d0fa0 0x7f05900b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900c6e40 con 0x7f059015d3b0 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0580001670 con 0x7f059015d3b0 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f0594e00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05900ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 >> v1:172.21.15.43:6789/0 conn(0x7f059015d3b0 legacy=0x7f05900d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05700039e0 0x7f0570003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0570004d00 0x7f0570005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900d13d0 con 0x7f0570004d00 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cedc0 con 0x7f05700039e0 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057000ba80 con 0x7f05900d1d90 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f0596200640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0570004d00 0x7f0570005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.789+0000 7f0594e00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0596200640 1 -- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 msgr2=0x7f05700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0596200640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0596200640 1 -- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05700039e0 msgr2=0x7f0570003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0596200640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05700039e0 0x7f0570003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0596200640 1 -- 172.21.15.43:0/2886687012 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570003520 con 0x7f0570004d00 2024-01-16T15:19:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.790+0000 7f0594e00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900d1d90 0x7f05700047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:10.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.791+0000 7f0596200640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0570004d00 0x7f0570005180 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0588009fc0 tx=0x7f05700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c266238b75b9bf12 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.791+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058802cc60 con 0x7f0570004d00 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.791+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058800f040 con 0x7f0570004d00 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.791+0000 7f058fe00640 1 -- 172.21.15.43:0/2886687012 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588034440 con 0x7f0570004d00 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 -- 172.21.15.43:0/2886687012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0570004d00 msgr2=0x7f0570005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 --2- 172.21.15.43:0/2886687012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0570004d00 0x7f0570005180 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0588009fc0 tx=0x7f05700057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596200640 1 -- 172.21.15.43:0/2886687012 reap_dead start 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 -- 172.21.15.43:0/2886687012 shutdown_connections 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 -- 172.21.15.43:0/2886687012 >> 172.21.15.43:0/2886687012 conn(0x7f05900d30a0 msgr2=0x7f05900d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 -- 172.21.15.43:0/2886687012 shutdown_connections 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.792+0000 7f0596c00640 1 -- 172.21.15.43:0/2886687012 wait complete. 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.794+0000 7f0596c00640 1 Processor -- start 2024-01-16T15:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.794+0000 7f0596c00640 1 -- start start 2024-01-16T15:19:10.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0570004d00 0x7f05901f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900ccd40 0x7f05901f6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cba00 con 0x7f05900a0b70 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cd1f0 con 0x7f0570004d00 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059015ae60 con 0x7f05900ccd40 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47410/0 (socket says 172.21.15.43:47410) 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 -- 172.21.15.43:0/3198119615 learned_addr learned my addr 172.21.15.43:0/3198119615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0596200640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900ccd40 0x7f05901f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 -- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900ccd40 msgr2=0x7f05901f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0595800640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0570004d00 0x7f05901f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900ccd40 0x7f05901f6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 -- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0570004d00 msgr2=0x7f05901f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.795+0000 7f0594e00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0570004d00 0x7f05901f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f0594e00640 1 -- 172.21.15.43:0/3198119615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900cad80 con 0x7f05900a0b70 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f0594e00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f057800f550 tx=0x7f057800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578014890 con 0x7f05900a0b70 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057800c030 con 0x7f05900a0b70 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578014bf0 con 0x7f05900a0b70 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900cb060 con 0x7f05900a0b70 2024-01-16T15:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.796+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05900cb620 con 0x7f05900a0b70 2024-01-16T15:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.797+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059004e840 con 0x7f05900a0b70 2024-01-16T15:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.798+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 42) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0578014d50 con 0x7f05900a0b70 2024-01-16T15:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.799+0000 7f058d600640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 0x7f056806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.800+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0578088d90 con 0x7f05900a0b70 2024-01-16T15:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.800+0000 7f0595800640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 0x7f056806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.800+0000 7f0595800640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 0x7f056806e890 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0580000f60 tx=0x7f0580009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:10.804+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0578055310 con 0x7f05900a0b70 2024-01-16T15:19:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.029+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f05901f3a20 con 0x7f056806c3d0 2024-01-16T15:19:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.036+0000 7f058d600640 1 -- 172.21.15.43:0/3198119615 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f05901f3a20 con 0x7f056806c3d0 2024-01-16T15:19:11.027 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 msgr2=0x7f056806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 0x7f056806e890 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0580000f60 tx=0x7f0580009040 comp rx=0 tx=0).stop 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 msgr2=0x7f05900cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f057800f550 tx=0x7f057800fa10 comp rx=0 tx=0).stop 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 shutdown_connections 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f056806c3d0 0x7f056806e890 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05900ccd40 0x7f05901f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05900a0b70 0x7f05900cc800 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 --2- 172.21.15.43:0/3198119615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0570004d00 0x7f05901f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 >> 172.21.15.43:0/3198119615 conn(0x7f05900d30a0 msgr2=0x7f05900b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 shutdown_connections 2024-01-16T15:19:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:11.039+0000 7f0596c00640 1 -- 172.21.15.43:0/3198119615 wait complete. 2024-01-16T15:19:11.048 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:11.048 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:19:11.048 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:11.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:11.102 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:19:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:19:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:19:27.087 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.9005 s, 67.5 MB/s 2024-01-16T15:19:27.089 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:19:27.089 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:27.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:27.147 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:19:30.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:30.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:19:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:19:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.415 s, 80.0 MB/s 2024-01-16T15:19:40.628 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:19:40.628 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:40.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:40.686 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:19:54.694 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records in 2024-01-16T15:19:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:1024+0 records out 2024-01-16T15:19:54.695 INFO:teuthology.orchestra.run.smithi043.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.9396 s, 77.0 MB/s 2024-01-16T15:19:54.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:19:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.930+0000 7ff85f400640 1 Processor -- start 2024-01-16T15:19:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.930+0000 7ff85f400640 1 -- start start 2024-01-16T15:19:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8580ad450 0x7ff858154280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff858071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580ce650 con 0x7ff858153ea0 2024-01-16T15:19:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85f400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8580ad8e0 con 0x7ff8580ad450 2024-01-16T15:19:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8581547c0 con 0x7ff858070850 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.932+0000 7ff85d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8580ad450 0x7ff858154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff8580ad450 0x7ff858154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55224/0 (socket says 172.21.15.43:55224) 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 -- 172.21.15.43:0/1614707660 learned_addr learned my addr 172.21.15.43:0/1614707660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85ea00640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff858071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 -- 172.21.15.43:0/1614707660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 msgr2=0x7ff858071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff858071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 -- 172.21.15.43:0/1614707660 >> v1:172.21.15.69:6789/0 conn(0x7ff858153ea0 legacy=0x7ff85815d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:54.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.933+0000 7ff85d600640 1 -- 172.21.15.43:0/1614707660 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85815e0a0 con 0x7ff8580ad450 2024-01-16T15:19:54.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.934+0000 7ff85d600640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff858154280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff8400098a0 tx=0x7ff84002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c00d54d05a27c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:54.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.935+0000 7ff85cc00640 1 -- 172.21.15.43:0/1614707660 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840030840 con 0x7ff8580ad450 2024-01-16T15:19:54.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.935+0000 7ff85cc00640 1 -- 172.21.15.43:0/1614707660 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8400309a0 con 0x7ff8580ad450 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.935+0000 7ff85cc00640 1 -- 172.21.15.43:0/1614707660 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840030d30 con 0x7ff8580ad450 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.935+0000 7ff85f400640 1 -- 172.21.15.43:0/1614707660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 msgr2=0x7ff858154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.935+0000 7ff85f400640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff858154280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff8400098a0 tx=0x7ff84002b980 comp rx=0 tx=0).stop 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 -- 172.21.15.43:0/1614707660 shutdown_connections 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff858154280 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 --2- 172.21.15.43:0/1614707660 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff858071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 -- 172.21.15.43:0/1614707660 >> 172.21.15.43:0/1614707660 conn(0x7ff8580cec10 msgr2=0x7ff8580cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 -- 172.21.15.43:0/1614707660 shutdown_connections 2024-01-16T15:19:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.936+0000 7ff85f400640 1 -- 172.21.15.43:0/1614707660 wait complete. 2024-01-16T15:19:54.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.937+0000 7ff85f400640 1 Processor -- start 2024-01-16T15:19:54.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.938+0000 7ff85f400640 1 -- start start 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff8581f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff858153ea0 0x7ff858156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8581f3370 con 0x7ff8580ad450 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85813cbc0 con 0x7ff858153ea0 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85814a570 con 0x7ff858070850 2024-01-16T15:19:54.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55240/0 (socket says 172.21.15.43:55240) 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 -- 172.21.15.43:0/325249797 learned_addr learned my addr 172.21.15.43:0/325249797 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85e000640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff8581f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 -- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 msgr2=0x7ff8581f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff8581f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85ea00640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff858153ea0 0x7ff858156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 -- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff858153ea0 msgr2=0x7ff858156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff858153ea0 0x7ff858156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:54.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.939+0000 7ff85d600640 1 -- 172.21.15.43:0/325249797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400094d0 con 0x7ff8580ad450 2024-01-16T15:19:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.940+0000 7ff85d600640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff8400090b0 tx=0x7ff840004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.940+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840047070 con 0x7ff8580ad450 2024-01-16T15:19:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.940+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840039b50 con 0x7ff8580ad450 2024-01-16T15:19:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.940+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85814b0f0 con 0x7ff8580ad450 2024-01-16T15:19:54.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.941+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840043400 con 0x7ff8580ad450 2024-01-16T15:19:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.941+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85814aaa0 con 0x7ff8580ad450 2024-01-16T15:19:54.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.942+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85814b280 con 0x7ff8580ad450 2024-01-16T15:19:54.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.943+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 43) v1 ==== 91120+0+0 (secure 0 0 0) 0x7ff84004c410 con 0x7ff8580ad450 2024-01-16T15:19:54.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.944+0000 7ff856000640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 0x7ff83006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:54.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.944+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff84003e070 con 0x7ff8580ad450 2024-01-16T15:19:54.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.944+0000 7ff85e000640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 0x7ff83006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:54.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.945+0000 7ff85e000640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 0x7ff83006e8f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff848001500 tx=0x7ff848009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:54.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:54.948+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff840078150 con 0x7ff8580ad450 2024-01-16T15:19:55.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.174+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff858070050 con 0x7ff83006c430 2024-01-16T15:19:55.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.208+0000 7ff856000640 1 -- 172.21.15.43:0/325249797 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff8480011a0 con 0x7ff83006c430 2024-01-16T15:19:55.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.210+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 msgr2=0x7ff83006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.210+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 0x7ff83006e8f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff848001500 tx=0x7ff848009040 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.210+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 msgr2=0x7ff85813c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.210+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff8400090b0 tx=0x7ff840004730 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 shutdown_connections 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff83006c430 0x7ff83006e8f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff858153ea0 0x7ff858156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8580ad450 0x7ff85813c680 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 --2- 172.21.15.43:0/325249797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff858070850 0x7ff8581f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 >> 172.21.15.43:0/325249797 conn(0x7ff8580cec10 msgr2=0x7ff85815f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 shutdown_connections 2024-01-16T15:19:55.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.211+0000 7ff85f400640 1 -- 172.21.15.43:0/325249797 wait complete. 2024-01-16T15:19:55.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:19:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.457+0000 7f4f73e00640 1 Processor -- start 2024-01-16T15:19:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.458+0000 7f4f73e00640 1 -- start start 2024-01-16T15:19:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.459+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f740ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.459+0000 7f4f73e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740c6bf0 con 0x7f4f7415d3b0 2024-01-16T15:19:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.459+0000 7f4f73e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740cedc0 con 0x7f4f740d1d90 2024-01-16T15:19:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.459+0000 7f4f73e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740c6e40 con 0x7f4f740c8930 2024-01-16T15:19:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f72a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4f7415d3b0 0x7f4f740d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38192/0 (socket says 172.21.15.43:38192) 2024-01-16T15:19:55.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f72a00640 1 -- 172.21.15.43:0/3477694554 learned_addr learned my addr 172.21.15.43:0/3477694554 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544337230 0 0) 0x7f4f740c6bf0 con 0x7f4f7415d3b0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54003520 con 0x7f4f7415d3b0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f73400640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f740ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.460+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913721344 0 0) 0x7f4f54003520 con 0x7f4f7415d3b0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 msgr2=0x7f4f740ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f740ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 >> v1:172.21.15.69:6789/0 conn(0x7f4f740d1d90 legacy=0x7f4f740b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740c7120 con 0x7f4f7415d3b0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4f6c0019f0 con 0x7f4f7415d3b0 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 >> v1:172.21.15.43:6789/0 conn(0x7f4f7415d3b0 legacy=0x7f4f740d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f54003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f54003fe0 0x7f4f540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f54004d00 0x7f4f54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740c6e40 con 0x7f4f54003fe0 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5400bbc0 con 0x7f4f54004d00 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5400bd80 con 0x7f4f740c8930 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f72000640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f54003fe0 0x7f4f540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.461+0000 7f4f72a00640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f54003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f73400640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f54004d00 0x7f4f54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72000640 1 -- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 msgr2=0x7f4f54003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72000640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f54003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72000640 1 -- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f54004d00 msgr2=0x7f4f54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72000640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f54004d00 0x7f4f54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72000640 1 -- 172.21.15.43:0/3477694554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54003520 con 0x7f4f54003fe0 2024-01-16T15:19:55.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.462+0000 7f4f72a00640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f740c8930 0x7f4f54003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.463+0000 7f4f72000640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f54003fe0 0x7f4f540047c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001750 tx=0x7f4f540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59db490b30032fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.463+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0052d0 con 0x7f4f54003fe0 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.463+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c02e050 con 0x7f4f54003fe0 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.463+0000 7f4f71600640 1 -- 172.21.15.43:0/3477694554 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c003830 con 0x7f4f54003fe0 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 -- 172.21.15.43:0/3477694554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f54003fe0 msgr2=0x7f4f540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 --2- 172.21.15.43:0/3477694554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f54003fe0 0x7f4f540047c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001750 tx=0x7f4f540057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73400640 1 -- 172.21.15.43:0/3477694554 reap_dead start 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 -- 172.21.15.43:0/3477694554 shutdown_connections 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 -- 172.21.15.43:0/3477694554 >> 172.21.15.43:0/3477694554 conn(0x7f4f740d30a0 msgr2=0x7f4f740d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 -- 172.21.15.43:0/3477694554 shutdown_connections 2024-01-16T15:19:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.464+0000 7f4f73e00640 1 -- 172.21.15.43:0/3477694554 wait complete. 2024-01-16T15:19:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.466+0000 7f4f73e00640 1 Processor -- start 2024-01-16T15:19:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.466+0000 7f4f73e00640 1 -- start start 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f54003fe0 0x7f4f740cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f741f6780 0x7f4f741f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741f6be0 con 0x7f4f741f00c0 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740ccb30 con 0x7f4f741f6780 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f73e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740cb5e0 con 0x7f4f54003fe0 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f72000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f72000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55254/0 (socket says 172.21.15.43:55254) 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.467+0000 7f4f72000640 1 -- 172.21.15.43:0/2376013260 learned_addr learned my addr 172.21.15.43:0/2376013260 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:19:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72a00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f54003fe0 0x7f4f740cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f73400640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f741f6780 0x7f4f741f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 -- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f54003fe0 msgr2=0x7f4f740cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f54003fe0 0x7f4f740cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 -- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f741f6780 msgr2=0x7f4f741f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f741f6780 0x7f4f741f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 -- 172.21.15.43:0/2376013260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7409db20 con 0x7f4f741f00c0 2024-01-16T15:19:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.468+0000 7f4f72000640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c008fd0 tx=0x7f4f5c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.469+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0076c0 con 0x7f4f741f00c0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.469+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c007a30 con 0x7f4f741f00c0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.469+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c006be0 con 0x7f4f741f00c0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.469+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f740cc4e0 con 0x7f4f741f00c0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.469+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7409d080 con 0x7f4f741f00c0 2024-01-16T15:19:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.470+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f741f6b60 con 0x7f4f741f00c0 2024-01-16T15:19:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.471+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 43) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4f5c006d40 con 0x7f4f741f00c0 2024-01-16T15:19:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.472+0000 7f4f6aa00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 0x7f4f4c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.472+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4f5c0aa7c0 con 0x7f4f741f00c0 2024-01-16T15:19:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.472+0000 7f4f72a00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 0x7f4f4c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.473+0000 7f4f72a00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 0x7f4f4c06e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c002230 tx=0x7f4f6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.477+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f5c076eb0 con 0x7f4f741f00c0 2024-01-16T15:19:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.701+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f7415d790 con 0x7f4f4c06c460 2024-01-16T15:19:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.711+0000 7f4f6aa00640 1 -- 172.21.15.43:0/2376013260 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f4f6c005d10 con 0x7f4f4c06c460 2024-01-16T15:19:55.703 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 msgr2=0x7f4f4c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 0x7f4f4c06e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c002230 tx=0x7f4f6c009040 comp rx=0 tx=0).stop 2024-01-16T15:19:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 msgr2=0x7f4f741f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c008fd0 tx=0x7f4f5c001120 comp rx=0 tx=0).stop 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 shutdown_connections 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4f4c06c460 0x7f4f4c06e920 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f741f6780 0x7f4f741f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f741f00c0 0x7f4f741f6240 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 --2- 172.21.15.43:0/2376013260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4f54003fe0 0x7f4f740cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 >> 172.21.15.43:0/2376013260 conn(0x7f4f740d30a0 msgr2=0x7f4f741602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 shutdown_connections 2024-01-16T15:19:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:19:55.714+0000 7f4f73e00640 1 -- 172.21.15.43:0/2376013260 wait complete. 2024-01-16T15:19:55.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:55.726 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b) 2024-01-16T15:19:55.791 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:55.792 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:19:55.792 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:19:55.793 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:19:57.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:57.794 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b) 2024-01-16T15:19:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:19:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:19:57.864 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:19:59.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:59.865 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b) 2024-01-16T15:19:59.933 INFO:teuthology.orchestra.run.smithi043.stdout:# file: volumes/_nogroup/sv1/0b7f9b29-fefa-4c28-8ade-caefae8eaa7b 2024-01-16T15:19:59.934 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:19:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:19:59.935 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:19:59.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42751 2024-01-16T15:20:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.174+0000 7f2d7d800640 1 Processor -- start 2024-01-16T15:20:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.174+0000 7f2d7d800640 1 -- start start 2024-01-16T15:20:00.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 0x7f2d7806d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2d780c45b0 0x7f2d7806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d7806b1b0 0x7f2d78070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7806a7e0 con 0x7f2d780c45b0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7806a4e0 con 0x7f2d781595a0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d7d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7806ddc0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d77400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2d780c45b0 0x7f2d7806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d77400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2d780c45b0 0x7f2d7806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55264/0 (socket says 172.21.15.43:55264) 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.176+0000 7f2d77400640 1 -- 172.21.15.43:0/1597407922 learned_addr learned my addr 172.21.15.43:0/1597407922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d7ce00640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d7806b1b0 0x7f2d78070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77e00640 1 --2- 172.21.15.43:0/1597407922 >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 0x7f2d7806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77400640 1 -- 172.21.15.43:0/1597407922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d7806b1b0 msgr2=0x7f2d78070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77400640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d7806b1b0 0x7f2d78070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77400640 1 -- 172.21.15.43:0/1597407922 >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 msgr2=0x7f2d7806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77400640 1 --2- 172.21.15.43:0/1597407922 >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 0x7f2d7806d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77400640 1 -- 172.21.15.43:0/1597407922 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780c4b70 con 0x7f2d780c45b0 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.177+0000 7f2d77e00640 1 --2- 172.21.15.43:0/1597407922 >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 0x7f2d7806d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.178+0000 7f2d77400640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d780c45b0 0x7f2d7806ac70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f2d78071df0 tx=0x7f2d6002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d397d3bbdb6021a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.178+0000 7f2d76a00640 1 -- 172.21.15.43:0/1597407922 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d600317d0 con 0x7f2d780c45b0 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.178+0000 7f2d76a00640 1 -- 172.21.15.43:0/1597407922 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60031930 con 0x7f2d780c45b0 2024-01-16T15:20:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.178+0000 7f2d76a00640 1 -- 172.21.15.43:0/1597407922 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60031cc0 con 0x7f2d780c45b0 2024-01-16T15:20:00.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 -- 172.21.15.43:0/1597407922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d780c45b0 msgr2=0x7f2d7806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d780c45b0 0x7f2d7806ac70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f2d78071df0 tx=0x7f2d6002b870 comp rx=0 tx=0).stop 2024-01-16T15:20:00.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 -- 172.21.15.43:0/1597407922 shutdown_connections 2024-01-16T15:20:00.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d7806b1b0 0x7f2d78070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 --2- 172.21.15.43:0/1597407922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d780c45b0 0x7f2d7806ac70 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 --2- 172.21.15.43:0/1597407922 >> v2:172.21.15.69:3300/0 conn(0x7f2d781595a0 0x7f2d7806d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 -- 172.21.15.43:0/1597407922 >> 172.21.15.43:0/1597407922 conn(0x7f2d780cf500 msgr2=0x7f2d780cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:00.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 -- 172.21.15.43:0/1597407922 shutdown_connections 2024-01-16T15:20:00.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.179+0000 7f2d7d800640 1 -- 172.21.15.43:0/1597407922 wait complete. 2024-01-16T15:20:00.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.181+0000 7f2d7d800640 1 Processor -- start 2024-01-16T15:20:00.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.181+0000 7f2d7d800640 1 -- start start 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d780c45b0 0x7f2d78143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 0x7f2d780ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815a8b0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815ae70 con 0x7f2d781595a0 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d7d800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780b2a80 con 0x7f2d780c45b0 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.182+0000 7f2d77e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55266/0 (socket says 172.21.15.43:55266) 2024-01-16T15:20:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 -- 172.21.15.43:0/671931619 learned_addr learned my addr 172.21.15.43:0/671931619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77400640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d780c45b0 0x7f2d78143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d7ce00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 0x7f2d780ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 -- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d780c45b0 msgr2=0x7f2d78143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d780c45b0 0x7f2d78143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 -- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 msgr2=0x7f2d780ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 0x7f2d780ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 -- 172.21.15.43:0/671931619 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600094d0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d7ce00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 0x7f2d780ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:00.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.183+0000 7f2d77e00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c820 tx=0x7f2d6800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.184+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68007a30 con 0x7f2d7806b1b0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.184+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68007b90 con 0x7f2d7806b1b0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.184+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800ce70 con 0x7f2d7806b1b0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.184+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d780b2d60 con 0x7f2d7806b1b0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.184+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d781444a0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.185+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d780b2fa0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.186+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 43) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2d680113f0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.187+0000 7f2d6fe00640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 0x7f2d5006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.187+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d68088cf0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.187+0000 7f2d77400640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 0x7f2d5006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.188+0000 7f2d77400640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 0x7f2d5006e920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036000 tx=0x7f2d6003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.192+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d680551d0 con 0x7f2d7806b1b0 2024-01-16T15:20:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.424+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42751", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d78071c40 con 0x7f2d5006c460 2024-01-16T15:20:00.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:00.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.577+0000 7f2d6fe00640 1 -- 172.21.15.43:0/671931619 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2d78071c40 con 0x7f2d5006c460 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 msgr2=0x7f2d5006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 0x7f2d5006e920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036000 tx=0x7f2d6003d040 comp rx=0 tx=0).stop 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 msgr2=0x7f2d780b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c820 tx=0x7f2d6800cc20 comp rx=0 tx=0).stop 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 shutdown_connections 2024-01-16T15:20:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2d5006c460 0x7f2d5006e920 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d781595a0 0x7f2d780ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d780c45b0 0x7f2d78143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 --2- 172.21.15.43:0/671931619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d7806b1b0 0x7f2d780b23e0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.581+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 >> 172.21.15.43:0/671931619 conn(0x7f2d780cf500 msgr2=0x7f2d7815d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.582+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 shutdown_connections 2024-01-16T15:20:00.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.582+0000 7f2d7d800640 1 -- 172.21.15.43:0/671931619 wait complete. 2024-01-16T15:20:00.576 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.584+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.576 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.584+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.576 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.584+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.576 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.584+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.576 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.584+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42752 2024-01-16T15:20:00.600 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.608+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.600 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.608+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.600 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.608+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.600 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.608+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.600 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:00.608+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.801+0000 7fad4b200640 1 Processor -- start 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.801+0000 7fad4b200640 1 -- start start 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.802+0000 7fad4b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.802+0000 7fad4b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad4c0d0fa0 0x7fad4c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.802+0000 7fad4b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0c6bf0 con 0x7fad4c15d3b0 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.802+0000 7fad4b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c0d13d0 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.802+0000 7fad4b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0cedc0 con 0x7fad4c0d1d90 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.803+0000 7fad43e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.803+0000 7fad43e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36656/0 (socket says 172.21.15.43:36656) 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.803+0000 7fad4a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad4c0d0fa0 0x7fad4c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.803+0000 7fad43e00640 1 -- 172.21.15.43:0/2584336280 learned_addr learned my addr 172.21.15.43:0/2584336280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.803+0000 7fad49e00640 1 --1- 172.21.15.43:0/2584336280 >> v1:172.21.15.69:6789/0 conn(0x7fad4c15d3b0 0x7fad4c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:48878/0 (socket says 172.21.15.43:48878) 2024-01-16T15:20:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.804+0000 7fad4a800640 1 -- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 msgr2=0x7fad4c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.804+0000 7fad4a800640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.804+0000 7fad4a800640 1 -- 172.21.15.43:0/2584336280 >> v1:172.21.15.69:6789/0 conn(0x7fad4c15d3b0 legacy=0x7fad4c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.804+0000 7fad4a800640 1 -- 172.21.15.43:0/2584336280 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0c6e40 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.804+0000 7fad43e00640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.806+0000 7fad4a800640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c0b3ca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fad44009d90 tx=0x7fad4402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e3efa5a78531b86 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.806+0000 7fad49400640 1 -- 172.21.15.43:0/2584336280 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4402cbc0 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.806+0000 7fad49400640 1 -- 172.21.15.43:0/2584336280 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4402cd20 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.806+0000 7fad49400640 1 -- 172.21.15.43:0/2584336280 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44034660 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 -- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 msgr2=0x7fad4c0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c0b3ca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fad44009d90 tx=0x7fad4402be10 comp rx=0 tx=0).stop 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 -- 172.21.15.43:0/2584336280 shutdown_connections 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c0b3ca0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 --2- 172.21.15.43:0/2584336280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c0d1d90 0x7fad4c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 -- 172.21.15.43:0/2584336280 >> 172.21.15.43:0/2584336280 conn(0x7fad4c0d30a0 msgr2=0x7fad4c0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 -- 172.21.15.43:0/2584336280 shutdown_connections 2024-01-16T15:20:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.807+0000 7fad4b200640 1 -- 172.21.15.43:0/2584336280 wait complete. 2024-01-16T15:20:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.809+0000 7fad4b200640 1 Processor -- start 2024-01-16T15:20:00.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.809+0000 7fad4b200640 1 -- start start 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 0x7fad4c15fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c156d40 con 0x7fad4c0d1d90 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c15d9b0 con 0x7fad4c0d0fa0 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad4b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c15dc90 con 0x7fad4c15d3b0 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad43e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad43e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55300/0 (socket says 172.21.15.43:55300) 2024-01-16T15:20:00.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.810+0000 7fad43e00640 1 -- 172.21.15.43:0/2572803657 learned_addr learned my addr 172.21.15.43:0/2572803657 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad4a800640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 0x7fad4c15fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad49e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 -- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 msgr2=0x7fad4c15fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 0x7fad4c15fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 -- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 msgr2=0x7fad4c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 -- 172.21.15.43:0/2572803657 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad340094d0 con 0x7fad4c0d1d90 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad4a800640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 0x7fad4c15fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad49e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad43e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fad3400e9d0 tx=0x7fad3400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400ca60 con 0x7fad4c0d1d90 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3400cbc0 con 0x7fad4c0d1d90 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34013470 con 0x7fad4c0d1d90 2024-01-16T15:20:00.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.811+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad44009a70 con 0x7fad4c0d1d90 2024-01-16T15:20:00.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.812+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c15e250 con 0x7fad4c0d1d90 2024-01-16T15:20:00.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.813+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c04e840 con 0x7fad4c0d1d90 2024-01-16T15:20:00.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.814+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 43) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fad340135d0 con 0x7fad4c0d1d90 2024-01-16T15:20:00.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.815+0000 7fad42000640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 0x7fad2406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:00.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.815+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fad34088d50 con 0x7fad4c0d1d90 2024-01-16T15:20:00.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.815+0000 7fad49e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 0x7fad2406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.816+0000 7fad49e00640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 0x7fad2406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000f60 tx=0x7fad3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:00.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:00.819+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad34055320 con 0x7fad4c0d1d90 2024-01-16T15:20:01.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.052+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42752", "target": ["mon-mgr", ""]}) v1 -- 0x7fad4c15d790 con 0x7fad2406c430 2024-01-16T15:20:01.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.221+0000 7fad42000640 1 -- 172.21.15.43:0/2572803657 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad3c0012a0 con 0x7fad2406c430 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 msgr2=0x7fad2406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 0x7fad2406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000f60 tx=0x7fad3c009040 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 msgr2=0x7fad4c15ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fad3400e9d0 tx=0x7fad3400ee90 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 shutdown_connections 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fad2406c430 0x7fad2406e8f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad4c15d3b0 0x7fad4c15fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad4c0d1d90 0x7fad4c15ea50 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 --2- 172.21.15.43:0/2572803657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad4c0d0fa0 0x7fad4c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 >> 172.21.15.43:0/2572803657 conn(0x7fad4c0d30a0 msgr2=0x7fad4c0c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 shutdown_connections 2024-01-16T15:20:01.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.225+0000 7fad4b200640 1 -- 172.21.15.43:0/2572803657 wait complete. 2024-01-16T15:20:01.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42753 2024-01-16T15:20:01.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.473+0000 7f1365a00640 1 Processor -- start 2024-01-16T15:20:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.474+0000 7f1365a00640 1 -- start start 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.475+0000 7f1365a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f136015d3b0 0x7f13600ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.475+0000 7f1365a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 0x7f13600b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.475+0000 7f1365a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600c6bf0 con 0x7f13600d0fa0 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.475+0000 7f1365a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13600d1430 con 0x7f136015d3b0 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.475+0000 7f1365a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cedc0 con 0x7f13600d1d90 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f1357200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f13600d0fa0 0x7f13600d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38194/0 (socket says 172.21.15.43:38194) 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f1357200640 1 -- 172.21.15.43:0/1224502630 learned_addr learned my addr 172.21.15.43:0/1224502630 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135fe00640 1 --2- 172.21.15.43:0/1224502630 >> v2:172.21.15.43:3300/0 conn(0x7f136015d3b0 0x7f13600ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f1365000640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 0x7f13600b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135f400640 1 -- 172.21.15.43:0/1224502630 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411017113 0 0) 0x7f13600c6bf0 con 0x7f13600d0fa0 2024-01-16T15:20:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135f400640 1 -- 172.21.15.43:0/1224502630 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1340003520 con 0x7f13600d0fa0 2024-01-16T15:20:01.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135fe00640 1 -- 172.21.15.43:0/1224502630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 msgr2=0x7f13600b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135fe00640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 0x7f13600b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135fe00640 1 -- 172.21.15.43:0/1224502630 >> v1:172.21.15.43:6789/0 conn(0x7f13600d0fa0 legacy=0x7f13600d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.476+0000 7f135fe00640 1 -- 172.21.15.43:0/1224502630 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600c6e40 con 0x7f136015d3b0 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.477+0000 7f1365000640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 0x7f13600b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f135fe00640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136015d3b0 0x7f13600ce6c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f13500097a0 tx=0x7f135002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d635b448e6cae451 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f135f400640 1 -- 172.21.15.43:0/1224502630 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350031780 con 0x7f136015d3b0 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f135f400640 1 -- 172.21.15.43:0/1224502630 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13500318e0 con 0x7f136015d3b0 2024-01-16T15:20:01.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f135f400640 1 -- 172.21.15.43:0/1224502630 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350031c70 con 0x7f136015d3b0 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f1365a00640 1 -- 172.21.15.43:0/1224502630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136015d3b0 msgr2=0x7f13600ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.478+0000 7f1365a00640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136015d3b0 0x7f13600ce6c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f13500097a0 tx=0x7f135002b820 comp rx=0 tx=0).stop 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 -- 172.21.15.43:0/1224502630 shutdown_connections 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f13600d1d90 0x7f13600b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 --2- 172.21.15.43:0/1224502630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136015d3b0 0x7f13600ce6c0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 -- 172.21.15.43:0/1224502630 >> 172.21.15.43:0/1224502630 conn(0x7f13600d30a0 msgr2=0x7f13600d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 -- 172.21.15.43:0/1224502630 shutdown_connections 2024-01-16T15:20:01.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.479+0000 7f1365a00640 1 -- 172.21.15.43:0/1224502630 wait complete. 2024-01-16T15:20:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.480+0000 7f1365a00640 1 Processor -- start 2024-01-16T15:20:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.481+0000 7f1365a00640 1 -- start start 2024-01-16T15:20:01.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.481+0000 7f1365a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 0x7f13600cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f136015d3b0 0x7f136009cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cc390 con 0x7f13600d0fa0 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cadb0 con 0x7f13600d1d90 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cb090 con 0x7f136015d3b0 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55310/0 (socket says 172.21.15.43:55310) 2024-01-16T15:20:01.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 -- 172.21.15.43:0/1557353901 learned_addr learned my addr 172.21.15.43:0/1557353901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:01.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1357200640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 0x7f13600cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1365000640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f136015d3b0 0x7f136009cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 -- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f136015d3b0 msgr2=0x7f136009cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f136015d3b0 0x7f136009cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 -- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 msgr2=0x7f13600cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 0x7f13600cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f135fe00640 1 -- 172.21.15.43:0/1557353901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350009400 con 0x7f13600d0fa0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.482+0000 7f1357200640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 0x7f13600cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f135fe00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1350034040 tx=0x7f13500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350032820 con 0x7f13600d0fa0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1350032980 con 0x7f13600d0fa0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350032d30 con 0x7f13600d0fa0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13600cb370 con 0x7f13600d0fa0 2024-01-16T15:20:01.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.483+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13600cb930 con 0x7f13600d0fa0 2024-01-16T15:20:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.484+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13600cbb20 con 0x7f13600d0fa0 2024-01-16T15:20:01.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.485+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 43) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f135004e440 con 0x7f13600d0fa0 2024-01-16T15:20:01.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.487+0000 7f135cc00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 0x7f133806e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.487+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f13500aab90 con 0x7f13600d0fa0 2024-01-16T15:20:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.487+0000 7f1357200640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 0x7f133806e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:01.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.488+0000 7f1357200640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 0x7f133806e940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f136015c510 tx=0x7f1348009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.492+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1350077260 con 0x7f13600d0fa0 2024-01-16T15:20:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:01.728+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42753", "target": ["mon-mgr", ""]}) v1 -- 0x7f136015d790 con 0x7f133806c480 2024-01-16T15:20:02.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:02.418+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f135004ed30 con 0x7f13600d0fa0 2024-01-16T15:20:04.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.359+0000 7f135cc00640 1 -- 172.21.15.43:0/1557353901 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1348001d20 con 0x7f133806c480 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 msgr2=0x7f133806e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 0x7f133806e940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f136015c510 tx=0x7f1348009040 comp rx=0 tx=0).stop 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 msgr2=0x7f13600ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f1350034040 tx=0x7f13500046c0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 shutdown_connections 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f133806c480 0x7f133806e940 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13600d1d90 0x7f13600cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f136015d3b0 0x7f136009cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 --2- 172.21.15.43:0/1557353901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13600d0fa0 0x7f13600ca7d0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 >> 172.21.15.43:0/1557353901 conn(0x7f13600d30a0 msgr2=0x7f13600c8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:04.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 shutdown_connections 2024-01-16T15:20:04.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.363+0000 7f1365a00640 1 -- 172.21.15.43:0/1557353901 wait complete. 2024-01-16T15:20:04.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42754 2024-01-16T15:20:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.608+0000 7fb1b1000640 1 Processor -- start 2024-01-16T15:20:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.608+0000 7fb1b1000640 1 -- start start 2024-01-16T15:20:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.610+0000 7fb1b1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.610+0000 7fb1b1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1ac0b35a0 0x7fb1ac15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.610+0000 7fb1b1000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac15f930 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.610+0000 7fb1b1000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ac0b43b0 con 0x7fb1ac0b35a0 2024-01-16T15:20:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.610+0000 7fb1b1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0b3b10 con 0x7fb1ac144770 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.611+0000 7fb1ab400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.611+0000 7fb1ab400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:59066/0 (socket says 172.21.15.43:59066) 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.611+0000 7fb1ab400640 1 -- 172.21.15.43:0/815664815 learned_addr learned my addr 172.21.15.43:0/815664815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.611+0000 7fb1a2800640 1 --2- 172.21.15.43:0/815664815 >> v2:172.21.15.69:3300/0 conn(0x7fb1ac0b35a0 0x7fb1ac15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1aaa00640 1 -- 172.21.15.43:0/815664815 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792397120 0 0) 0x7fb1ac15f930 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1aaa00640 1 -- 172.21.15.43:0/815664815 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c003520 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1ab400640 1 -- 172.21.15.43:0/815664815 >> v2:172.21.15.69:3300/0 conn(0x7fb1ac0b35a0 msgr2=0x7fb1ac15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1ab400640 1 --2- 172.21.15.43:0/815664815 >> v2:172.21.15.69:3300/0 conn(0x7fb1ac0b35a0 0x7fb1ac15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1ab400640 1 -- 172.21.15.43:0/815664815 >> v1:172.21.15.69:6789/0 conn(0x7fb1ac06c9d0 legacy=0x7fb1ac06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.612+0000 7fb1ab400640 1 -- 172.21.15.43:0/815664815 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac0b3df0 con 0x7fb1ac144770 2024-01-16T15:20:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1ab400640 1 --2- 172.21.15.43:0/815664815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0098a0 tx=0x7fb19c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd2f3625785e3a4c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1aaa00640 1 -- 172.21.15.43:0/815664815 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c030840 con 0x7fb1ac144770 2024-01-16T15:20:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1aaa00640 1 -- 172.21.15.43:0/815664815 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c0309a0 con 0x7fb1ac144770 2024-01-16T15:20:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1aaa00640 1 -- 172.21.15.43:0/815664815 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c030d30 con 0x7fb1ac144770 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 -- 172.21.15.43:0/815664815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 msgr2=0x7fb1ac15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 --2- 172.21.15.43:0/815664815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0098a0 tx=0x7fb19c02b980 comp rx=0 tx=0).stop 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 -- 172.21.15.43:0/815664815 shutdown_connections 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 --2- 172.21.15.43:0/815664815 >> v2:172.21.15.69:3300/0 conn(0x7fb1ac0b35a0 0x7fb1ac15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 --2- 172.21.15.43:0/815664815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac144770 0x7fb1ac15eb60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 -- 172.21.15.43:0/815664815 >> 172.21.15.43:0/815664815 conn(0x7fb1ac0cec10 msgr2=0x7fb1ac0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 -- 172.21.15.43:0/815664815 shutdown_connections 2024-01-16T15:20:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.615+0000 7fb1b1000640 1 -- 172.21.15.43:0/815664815 wait complete. 2024-01-16T15:20:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 Processor -- start 2024-01-16T15:20:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 -- start start 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac0b35a0 0x7fb1ac1f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 0x7fb1ac139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac140370 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac13d410 con 0x7fb1ac144770 2024-01-16T15:20:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1b1000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac13d6f0 con 0x7fb1ac0b35a0 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.618+0000 7fb1ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56532/0 (socket says 172.21.15.43:56532) 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 -- 172.21.15.43:0/1686148223 learned_addr learned my addr 172.21.15.43:0/1686148223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1a2800640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac0b35a0 0x7fb1ac1f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1abe00640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 0x7fb1ac139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 -- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac0b35a0 msgr2=0x7fb1ac1f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac0b35a0 0x7fb1ac1f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 -- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 msgr2=0x7fb1ac139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 0x7fb1ac139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 -- 172.21.15.43:0/1686148223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0094d0 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1abe00640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 0x7fb1ac139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.619+0000 7fb1ab400640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb19c02b9c0 tx=0x7fb19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.620+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c03e070 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.620+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c039a10 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.620+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c039da0 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.620+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac13d9d0 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.620+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac1f4c30 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.621+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac04e840 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.622+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb19c04b400 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.623+0000 7fb1a3e00640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 0x7fb18406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.623+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb19c0ab040 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.623+0000 7fb1a2800640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 0x7fb18406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.624+0000 7fb1a2800640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 0x7fb18406ea10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0b1270 tx=0x7fb19400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.628+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb19c077580 con 0x7fb1ac06c9d0 2024-01-16T15:20:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:04.858+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42754", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1ac144b50 con 0x7fb18406c550 2024-01-16T15:20:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.361+0000 7fb1a3e00640 1 -- 172.21.15.43:0/1686148223 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb19400f050 con 0x7fb18406c550 2024-01-16T15:20:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 msgr2=0x7fb18406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 0x7fb18406ea10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0b1270 tx=0x7fb19400b040 comp rx=0 tx=0).stop 2024-01-16T15:20:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 msgr2=0x7fb1ac13fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb19c02b9c0 tx=0x7fb19c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 shutdown_connections 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18406c550 0x7fb18406ea10 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.364+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1ac144770 0x7fb1ac139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.365+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1ac0b35a0 0x7fb1ac1f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.365+0000 7fb1b1000640 1 --2- 172.21.15.43:0/1686148223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1ac06c9d0 0x7fb1ac13fd20 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.365+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 >> 172.21.15.43:0/1686148223 conn(0x7fb1ac0cec10 msgr2=0x7fb1ac15e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.365+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 shutdown_connections 2024-01-16T15:20:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.365+0000 7fb1b1000640 1 -- 172.21.15.43:0/1686148223 wait complete. 2024-01-16T15:20:09.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42755 2024-01-16T15:20:09.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.615+0000 7fb282400640 1 Processor -- start 2024-01-16T15:20:09.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.615+0000 7fb282400640 1 -- start start 2024-01-16T15:20:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.616+0000 7fb282400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb27c162220 0x7fb27c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.616+0000 7fb282400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.616+0000 7fb282400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c162fa0 con 0x7fb27c06c3c0 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb282400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c0706a0 con 0x7fb27c162220 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb282400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c070c60 con 0x7fb27c162600 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb27c162220 0x7fb27c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb27c162220 0x7fb27c0b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56542/0 (socket says 172.21.15.43:56542) 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 -- 172.21.15.43:0/3792873682 learned_addr learned my addr 172.21.15.43:0/3792873682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 -- 172.21.15.43:0/3792873682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 msgr2=0x7fb27c162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27aa00640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c162a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 -- 172.21.15.43:0/3792873682 >> v1:172.21.15.69:6789/0 conn(0x7fb27c06c3c0 legacy=0x7fb27c0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.617+0000 7fb27b400640 1 -- 172.21.15.43:0/3792873682 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0722b0 con 0x7fb27c162220 2024-01-16T15:20:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.619+0000 7fb27b400640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0b1ea0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009e50 tx=0x7fb26c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ae590e43fba9e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.619+0000 7fb27a000640 1 -- 172.21.15.43:0/3792873682 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c02ca70 con 0x7fb27c162220 2024-01-16T15:20:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.619+0000 7fb27a000640 1 -- 172.21.15.43:0/3792873682 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c02cbd0 con 0x7fb27c162220 2024-01-16T15:20:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.619+0000 7fb27a000640 1 -- 172.21.15.43:0/3792873682 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0343f0 con 0x7fb27c162220 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 -- 172.21.15.43:0/3792873682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 msgr2=0x7fb27c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0b1ea0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009e50 tx=0x7fb26c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 -- 172.21.15.43:0/3792873682 shutdown_connections 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 --2- 172.21.15.43:0/3792873682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0b1ea0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 -- 172.21.15.43:0/3792873682 >> 172.21.15.43:0/3792873682 conn(0x7fb27c0b0ba0 msgr2=0x7fb27c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 -- 172.21.15.43:0/3792873682 shutdown_connections 2024-01-16T15:20:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.620+0000 7fb282400640 1 -- 172.21.15.43:0/3792873682 wait complete. 2024-01-16T15:20:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.622+0000 7fb282400640 1 Processor -- start 2024-01-16T15:20:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.622+0000 7fb282400640 1 -- start start 2024-01-16T15:20:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 0x7fb27c0ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c09b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c09c080 con 0x7fb27c162220 2024-01-16T15:20:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c15c660 con 0x7fb27c06c3c0 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb282400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c099ba0 con 0x7fb27c162600 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56548/0 (socket says 172.21.15.43:56548) 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27aa00640 1 -- 172.21.15.43:0/903991541 learned_addr learned my addr 172.21.15.43:0/903991541 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27b400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 0x7fb27c0ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27aa00640 1 -- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 msgr2=0x7fb27c09b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.623+0000 7fb27be00640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c09b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27aa00640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c09b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27aa00640 1 -- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 msgr2=0x7fb27c0ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27aa00640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 0x7fb27c0ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27aa00640 1 -- 172.21.15.43:0/903991541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009b30 con 0x7fb27c162220 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27b400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 0x7fb27c0ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27be00640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c09b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb27aa00640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fb26400c7f0 tx=0x7fb26400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264004220 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2640043a0 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264007860 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c099e80 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.624+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c09a440 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.625+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c067960 con 0x7fb27c162220 2024-01-16T15:20:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.626+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb264007bd0 con 0x7fb27c162220 2024-01-16T15:20:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.627+0000 7fb273400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 0x7fb25406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.627+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb2640887d0 con 0x7fb27c162220 2024-01-16T15:20:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.628+0000 7fb27b400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 0x7fb25406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.628+0000 7fb27b400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 0x7fb25406ea10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb26c035000 tx=0x7fb26c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:09.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.632+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb264055090 con 0x7fb27c162220 2024-01-16T15:20:09.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:09.864+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42755", "target": ["mon-mgr", ""]}) v1 -- 0x7fb27c0b23e0 con 0x7fb25406c550 2024-01-16T15:20:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.344+0000 7fb273400640 1 -- 172.21.15.43:0/903991541 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb26c040450 con 0x7fb25406c550 2024-01-16T15:20:14.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 msgr2=0x7fb25406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 0x7fb25406ea10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb26c035000 tx=0x7fb26c030000 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 msgr2=0x7fb27c0adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fb26400c7f0 tx=0x7fb26400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 shutdown_connections 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25406c550 0x7fb25406ea10 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb27c162600 0x7fb27c09b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c162220 0x7fb27c0adb80 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 --2- 172.21.15.43:0/903991541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb27c06c3c0 0x7fb27c0ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.348+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 >> 172.21.15.43:0/903991541 conn(0x7fb27c0b0ba0 msgr2=0x7fb27c160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.349+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 shutdown_connections 2024-01-16T15:20:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.349+0000 7fb282400640 1 -- 172.21.15.43:0/903991541 wait complete. 2024-01-16T15:20:14.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42756 2024-01-16T15:20:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.601+0000 7f30f5000640 1 Processor -- start 2024-01-16T15:20:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.602+0000 7f30f5000640 1 -- start start 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.603+0000 7f30f5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30f00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30f5000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f015f930 con 0x7f30f006d9e0 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30f5000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00b3b10 con 0x7f30f0144770 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30f5000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00b43b0 con 0x7f30f015f270 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30efe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f30f006d9e0 0x7f30f00cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52134/0 (socket says 172.21.15.43:52134) 2024-01-16T15:20:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30efe00640 1 -- 172.21.15.43:0/1838208555 learned_addr learned my addr 172.21.15.43:0/1838208555 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:20:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30e7e00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30f00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.604+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750247739 0 0) 0x7f30f015f930 con 0x7f30f006d9e0 2024-01-16T15:20:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d0003520 con 0x7f30f006d9e0 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377937344 0 0) 0x7f30f00b3b10 con 0x7f30f0144770 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f015f930 con 0x7f30f0144770 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271103710 0 0) 0x7f30d0003520 con 0x7f30f006d9e0 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 msgr2=0x7f30f00cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30f00cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 >> v1:172.21.15.69:6789/0 conn(0x7f30f0144770 legacy=0x7f30f015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00b3df0 con 0x7f30f006d9e0 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f30e80019f0 con 0x7f30f006d9e0 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 >> v1:172.21.15.43:6789/0 conn(0x7f30f006d9e0 legacy=0x7f30f00cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30d0003ba0 0x7f30d0003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30d0004d00 0x7f30d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00b43b0 con 0x7f30d0004d00 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d000ba60 con 0x7f30d0003ba0 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.605+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d000bc20 con 0x7f30f015f270 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30d0004d00 0x7f30d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30ef400640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30d0003ba0 0x7f30d0003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30e7e00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 -- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 msgr2=0x7f30d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 -- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30d0003ba0 msgr2=0x7f30d0003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30d0003ba0 0x7f30d0003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30efe00640 1 -- 172.21.15.43:0/1838208555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d0003520 con 0x7f30d0004d00 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30e7e00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30f015f270 0x7f30d00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.606+0000 7f30ef400640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30d0003ba0 0x7f30d0003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:20:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.607+0000 7f30efe00640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30d0004d00 0x7f30d0005180 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f30e8002230 tx=0x7f30d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975b852e959db1bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.607+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8034630 con 0x7f30d0004d00 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.607+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e802f070 con 0x7f30d0004d00 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.607+0000 7f30eea00640 1 -- 172.21.15.43:0/1838208555 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80349f0 con 0x7f30d0004d00 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 -- 172.21.15.43:0/1838208555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30d0004d00 msgr2=0x7f30d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 --2- 172.21.15.43:0/1838208555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30d0004d00 0x7f30d0005180 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f30e8002230 tx=0x7f30d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30efe00640 1 -- 172.21.15.43:0/1838208555 reap_dead start 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 -- 172.21.15.43:0/1838208555 shutdown_connections 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 -- 172.21.15.43:0/1838208555 >> 172.21.15.43:0/1838208555 conn(0x7f30f00cec10 msgr2=0x7f30f00cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 -- 172.21.15.43:0/1838208555 shutdown_connections 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.608+0000 7f30f5000640 1 -- 172.21.15.43:0/1838208555 wait complete. 2024-01-16T15:20:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.611+0000 7f30f5000640 1 Processor -- start 2024-01-16T15:20:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.611+0000 7f30f5000640 1 -- start start 2024-01-16T15:20:14.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30d0004d00 0x7f30f013efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f014f6f0 0x7f30f0142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01431c0 con 0x7f30f0142d40 2024-01-16T15:20:14.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f013f4e0 con 0x7f30f014f6f0 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.612+0000 7f30f5000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0152dc0 con 0x7f30d0004d00 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54774/0 (socket says 172.21.15.43:54774) 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 -- 172.21.15.43:0/371953192 learned_addr learned my addr 172.21.15.43:0/371953192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30ef400640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30d0004d00 0x7f30f013efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30e7e00640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f014f6f0 0x7f30f0142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 -- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30d0004d00 msgr2=0x7f30f013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30d0004d00 0x7f30f013efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 -- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f014f6f0 msgr2=0x7f30f0142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f014f6f0 0x7f30f0142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:14.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.613+0000 7f30efe00640 1 -- 172.21.15.43:0/371953192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f01fa460 con 0x7f30f0142d40 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30efe00640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f30e8001c30 tx=0x7f30e8001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8043070 con 0x7f30f0142d40 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e8001a00 con 0x7f30f0142d40 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8034630 con 0x7f30f0142d40 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f01fa740 con 0x7f30f0142d40 2024-01-16T15:20:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.614+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f01fad00 con 0x7f30f0142d40 2024-01-16T15:20:14.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.615+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f01faef0 con 0x7f30f0142d40 2024-01-16T15:20:14.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.616+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f30e8031030 con 0x7f30f0142d40 2024-01-16T15:20:14.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.617+0000 7f30e7400640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 0x7f30c806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:14.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.617+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f30e80a9f10 con 0x7f30f0142d40 2024-01-16T15:20:14.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.617+0000 7f30ef400640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 0x7f30c806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:14.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.618+0000 7f30ef400640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 0x7f30c806ea40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f30e0000f60 tx=0x7f30e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.624+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30e8076450 con 0x7f30f0142d40 2024-01-16T15:20:14.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:14.859+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42756", "target": ["mon-mgr", ""]}) v1 -- 0x7f30f0144b50 con 0x7f30c806c580 2024-01-16T15:20:15.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.020+0000 7f30e7400640 1 -- 172.21.15.43:0/371953192 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f30e0007c10 con 0x7f30c806c580 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.023+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 msgr2=0x7f30c806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.023+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 0x7f30c806ea40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f30e0000f60 tx=0x7f30e0009040 comp rx=0 tx=0).stop 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.023+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 msgr2=0x7f30f013b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.023+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f30e8001c30 tx=0x7f30e8001d00 comp rx=0 tx=0).stop 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 shutdown_connections 2024-01-16T15:20:15.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f30c806c580 0x7f30c806ea40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30f014f6f0 0x7f30f0142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30f0142d40 0x7f30f013b260 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 --2- 172.21.15.43:0/371953192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30d0004d00 0x7f30f013efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 >> 172.21.15.43:0/371953192 conn(0x7f30f00cec10 msgr2=0x7f30f015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 shutdown_connections 2024-01-16T15:20:15.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.024+0000 7f30f5000640 1 -- 172.21.15.43:0/371953192 wait complete. 2024-01-16T15:20:15.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone42757 2024-01-16T15:20:15.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.274+0000 7f72bae00640 1 Processor -- start 2024-01-16T15:20:15.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.275+0000 7f72bae00640 1 -- start start 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f72b40cd150 0x7f72b406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 0x7f72b406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b4160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40cd530 con 0x7f72b40cd150 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b4142d50 con 0x7f72b40702a0 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72bae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40706c0 con 0x7f72b406ec60 2024-01-16T15:20:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.276+0000 7f72b3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f72b40cd150 0x7f72b406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f72b40cd150 0x7f72b406fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54786/0 (socket says 172.21.15.43:54786) 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 -- 172.21.15.43:0/1591226785 learned_addr learned my addr 172.21.15.43:0/1591226785 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b8e00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b4160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3400640 1 --2- 172.21.15.43:0/1591226785 >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 0x7f72b406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 -- 172.21.15.43:0/1591226785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 msgr2=0x7f72b4160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b4160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 -- 172.21.15.43:0/1591226785 >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 msgr2=0x7f72b406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 --2- 172.21.15.43:0/1591226785 >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 0x7f72b406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3e00640 1 -- 172.21.15.43:0/1591226785 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b406f430 con 0x7f72b40cd150 2024-01-16T15:20:15.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.277+0000 7f72b3400640 1 --2- 172.21.15.43:0/1591226785 >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 0x7f72b406e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:15.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.278+0000 7f72b3e00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40cd150 0x7f72b406fd60 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f72b4147310 tx=0x7f72a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54573c33053ec44 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.278+0000 7f72b2a00640 1 -- 172.21.15.43:0/1591226785 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a40317e0 con 0x7f72b40cd150 2024-01-16T15:20:15.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.278+0000 7f72b2a00640 1 -- 172.21.15.43:0/1591226785 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4031940 con 0x7f72b40cd150 2024-01-16T15:20:15.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.278+0000 7f72b2a00640 1 -- 172.21.15.43:0/1591226785 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4031cd0 con 0x7f72b40cd150 2024-01-16T15:20:15.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 -- 172.21.15.43:0/1591226785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40cd150 msgr2=0x7f72b406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40cd150 0x7f72b406fd60 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f72b4147310 tx=0x7f72a402b880 comp rx=0 tx=0).stop 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 -- 172.21.15.43:0/1591226785 shutdown_connections 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b4160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 --2- 172.21.15.43:0/1591226785 >> v2:172.21.15.69:3300/0 conn(0x7f72b40702a0 0x7f72b406e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 --2- 172.21.15.43:0/1591226785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40cd150 0x7f72b406fd60 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 -- 172.21.15.43:0/1591226785 >> 172.21.15.43:0/1591226785 conn(0x7f72b40cdb20 msgr2=0x7f72b40cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 -- 172.21.15.43:0/1591226785 shutdown_connections 2024-01-16T15:20:15.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.279+0000 7f72bae00640 1 -- 172.21.15.43:0/1591226785 wait complete. 2024-01-16T15:20:15.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.281+0000 7f72bae00640 1 Processor -- start 2024-01-16T15:20:15.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.281+0000 7f72bae00640 1 -- start start 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 0x7f72b415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40ca620 con 0x7f72b40702a0 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41e7c60 con 0x7f72b40cd150 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72bae00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40caae0 con 0x7f72b406ec60 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.282+0000 7f72b3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54794/0 (socket says 172.21.15.43:54794) 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 -- 172.21.15.43:0/496524071 learned_addr learned my addr 172.21.15.43:0/496524071 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:15.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3e00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b8e00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 0x7f72b415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 -- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 msgr2=0x7f72b415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 -- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 msgr2=0x7f72b415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 0x7f72b415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 -- 172.21.15.43:0/496524071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40094d0 con 0x7f72b40702a0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b8e00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 0x7f72b415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.283+0000 7f72b3400640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c7f0 tx=0x7f729c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.284+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c007820 con 0x7f72b40702a0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.284+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c007980 con 0x7f72b40702a0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.284+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c007d30 con 0x7f72b40702a0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.284+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b415c100 con 0x7f72b40702a0 2024-01-16T15:20:15.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.284+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b40c8cb0 con 0x7f72b40702a0 2024-01-16T15:20:15.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.285+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b406fb50 con 0x7f72b40702a0 2024-01-16T15:20:15.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.286+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f729c0113f0 con 0x7f72b40702a0 2024-01-16T15:20:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.287+0000 7f72abe00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 0x7f728c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.287+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f729c088740 con 0x7f72b40702a0 2024-01-16T15:20:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.287+0000 7f72b3e00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 0x7f728c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.288+0000 7f72b3e00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 0x7f728c06ea40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f72a4036000 tx=0x7f72a403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.291+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f729c055000 con 0x7f72b40702a0 2024-01-16T15:20:15.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.520+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone42757", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b40c8b30 con 0x7f728c06c580 2024-01-16T15:20:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.685+0000 7f72abe00640 1 -- 172.21.15.43:0/496524071 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f72a4031b30 con 0x7f728c06c580 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 msgr2=0x7f728c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 0x7f728c06ea40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f72a4036000 tx=0x7f72a403d040 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 msgr2=0x7f72b40c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c7f0 tx=0x7f729c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 shutdown_connections 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f728c06c580 0x7f728c06ea40 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40cd150 0x7f72b415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72b40702a0 0x7f72b40c9790 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 --2- 172.21.15.43:0/496524071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72b406ec60 0x7f72b415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.688+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 >> 172.21.15.43:0/496524071 conn(0x7f72b40cdb20 msgr2=0x7f72b415f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.689+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 shutdown_connections 2024-01-16T15:20:15.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.689+0000 7f72bae00640 1 -- 172.21.15.43:0/496524071 wait complete. 2024-01-16T15:20:15.702 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:20:15.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone42757 2024-01-16T15:20:15.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.937+0000 7faa12c00640 1 Processor -- start 2024-01-16T15:20:15.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.937+0000 7faa12c00640 1 -- start start 2024-01-16T15:20:15.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa12c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faa0c144770 0x7faa0c15eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa12c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa0c15f0a0 0x7faa0c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa12c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c15f930 con 0x7faa0c0b35a0 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa12c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c0b43b0 con 0x7faa0c144770 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa12c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c15f510 con 0x7faa0c15f0a0 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faa0c144770 0x7faa0c15eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7faa0c144770 0x7faa0c15eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54800/0 (socket says 172.21.15.43:54800) 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 -- 172.21.15.43:0/1496957812 learned_addr learned my addr 172.21.15.43:0/1496957812 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 -- 172.21.15.43:0/1496957812 >> v2:172.21.15.69:3300/0 conn(0x7faa0c15f0a0 msgr2=0x7faa0c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 --2- 172.21.15.43:0/1496957812 >> v2:172.21.15.69:3300/0 conn(0x7faa0c15f0a0 0x7faa0c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 -- 172.21.15.43:0/1496957812 >> v1:172.21.15.69:6789/0 conn(0x7faa0c0b35a0 legacy=0x7faa0c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:15.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.939+0000 7faa11800640 1 -- 172.21.15.43:0/1496957812 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c0ce9f0 con 0x7faa0c144770 2024-01-16T15:20:15.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.941+0000 7faa11800640 1 --2- 172.21.15.43:0/1496957812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c144770 0x7faa0c15eb60 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0700e0 tx=0x7faa0402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b4b2efe7d16fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.941+0000 7faa0be00640 1 -- 172.21.15.43:0/1496957812 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0402ca80 con 0x7faa0c144770 2024-01-16T15:20:15.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.941+0000 7faa0be00640 1 -- 172.21.15.43:0/1496957812 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0402cbe0 con 0x7faa0c144770 2024-01-16T15:20:15.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.941+0000 7faa0be00640 1 -- 172.21.15.43:0/1496957812 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa040343f0 con 0x7faa0c144770 2024-01-16T15:20:15.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 -- 172.21.15.43:0/1496957812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c144770 msgr2=0x7faa0c15eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 --2- 172.21.15.43:0/1496957812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c144770 0x7faa0c15eb60 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0700e0 tx=0x7faa0402bee0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 -- 172.21.15.43:0/1496957812 shutdown_connections 2024-01-16T15:20:15.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 --2- 172.21.15.43:0/1496957812 >> v2:172.21.15.69:3300/0 conn(0x7faa0c15f0a0 0x7faa0c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 --2- 172.21.15.43:0/1496957812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c144770 0x7faa0c15eb60 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 -- 172.21.15.43:0/1496957812 >> 172.21.15.43:0/1496957812 conn(0x7faa0c0cec10 msgr2=0x7faa0c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:15.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 -- 172.21.15.43:0/1496957812 shutdown_connections 2024-01-16T15:20:15.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.942+0000 7faa12c00640 1 -- 172.21.15.43:0/1496957812 wait complete. 2024-01-16T15:20:15.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.944+0000 7faa12c00640 1 Processor -- start 2024-01-16T15:20:15.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.944+0000 7faa12c00640 1 -- start start 2024-01-16T15:20:15.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 0x7faa0c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 0x7faa0c0ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c1e7130 con 0x7faa0c15f0a0 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15af20 con 0x7faa0c144770 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c1e65b0 con 0x7faa0c0b35a0 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54812/0 (socket says 172.21.15.43:54812) 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa12200640 1 -- 172.21.15.43:0/2354054927 learned_addr learned my addr 172.21.15.43:0/2354054927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa11800640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 0x7faa0c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.945+0000 7faa10e00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 0x7faa0c0ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 -- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 msgr2=0x7faa0c159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 0x7faa0c159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 -- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 msgr2=0x7faa0c0ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 0x7faa0c0ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:15.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 -- 172.21.15.43:0/2354054927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04009b30 con 0x7faa0c15f0a0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa11800640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 0x7faa0c159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa10e00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 0x7faa0c0ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12200640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009fd0 tx=0x7fa9fc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc011830 con 0x7faa0c15f0a0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc011990 con 0x7faa0c15f0a0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc011d20 con 0x7faa0c15f0a0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c1e6890 con 0x7faa0c15f0a0 2024-01-16T15:20:15.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.946+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c1e6e50 con 0x7faa0c15f0a0 2024-01-16T15:20:15.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.948+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0c15f750 con 0x7faa0c15f0a0 2024-01-16T15:20:15.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.949+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fa9fc01b460 con 0x7faa0c15f0a0 2024-01-16T15:20:15.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.950+0000 7faa09600640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 0x7fa9e406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:15.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.950+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa9fc0888b0 con 0x7faa0c15f0a0 2024-01-16T15:20:15.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.950+0000 7faa11800640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 0x7fa9e406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:15.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.951+0000 7faa11800640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 0x7fa9e406ea10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa04035000 tx=0x7faa04030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:15.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:15.955+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9fc054e50 con 0x7faa0c15f0a0 2024-01-16T15:20:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.188+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone42757", "target": ["mon-mgr", ""]}) v1 -- 0x7faa0c0b40d0 con 0x7fa9e406c550 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "status": { 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "pending", 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "volume": "cephfs", 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "subvolume": "sv1", 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot": "ss1" 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:20:16.193 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:20:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.200+0000 7faa09600640 1 -- 172.21.15.43:0/2354054927 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7faa04040450 con 0x7fa9e406c550 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 msgr2=0x7fa9e406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 0x7fa9e406ea10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa04035000 tx=0x7faa04030000 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 msgr2=0x7faa0c0adf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009fd0 tx=0x7fa9fc00b8e0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 shutdown_connections 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa9e406c550 0x7fa9e406ea10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa0c144770 0x7faa0c0ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa0c15f0a0 0x7faa0c0adf30 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 --2- 172.21.15.43:0/2354054927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa0c0b35a0 0x7faa0c159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.203+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 >> 172.21.15.43:0/2354054927 conn(0x7faa0c0cec10 msgr2=0x7faa0c0af4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:16.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.204+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 shutdown_connections 2024-01-16T15:20:16.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.204+0000 7faa12c00640 1 -- 172.21.15.43:0/2354054927 wait complete. 2024-01-16T15:20:16.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42751 2024-01-16T15:20:16.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.439+0000 7f04a4000640 1 Processor -- start 2024-01-16T15:20:16.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.440+0000 7f04a4000640 1 -- start start 2024-01-16T15:20:16.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.441+0000 7f04a4000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f049c0cd150 0x7f049c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.441+0000 7f04a4000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f049c0702a0 0x7f049c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.441+0000 7f04a4000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c142d50 con 0x7f049c06ec60 2024-01-16T15:20:16.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.441+0000 7f04a4000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0cd530 con 0x7f049c0702a0 2024-01-16T15:20:16.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.441+0000 7f04a4000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0706c0 con 0x7f049c0cd150 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.442+0000 7f04a1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f049c0cd150 0x7f049c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.442+0000 7f04a0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f049c0702a0 0x7f049c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.442+0000 7f04a2000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f049c06ec60 0x7f049c160fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:52142/0 (socket says 172.21.15.43:52142) 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.442+0000 7f04a2000640 1 -- 172.21.15.43:0/586903144 learned_addr learned my addr 172.21.15.43:0/586903144 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.443+0000 7f04a0c00640 1 -- 172.21.15.43:0/586903144 >> v2:172.21.15.69:3300/0 conn(0x7f049c0cd150 msgr2=0x7f049c06fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.443+0000 7f04a0c00640 1 --2- 172.21.15.43:0/586903144 >> v2:172.21.15.69:3300/0 conn(0x7f049c0cd150 0x7f049c06fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.443+0000 7f04a0c00640 1 -- 172.21.15.43:0/586903144 >> v1:172.21.15.43:6789/0 conn(0x7f049c06ec60 legacy=0x7f049c160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.443+0000 7f049be00640 1 -- 172.21.15.43:0/586903144 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821457539 0 0) 0x7f049c142d50 con 0x7f049c06ec60 2024-01-16T15:20:16.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.443+0000 7f04a0c00640 1 -- 172.21.15.43:0/586903144 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c06f430 con 0x7f049c0702a0 2024-01-16T15:20:16.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.444+0000 7f04a0c00640 1 --2- 172.21.15.43:0/586903144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c0702a0 0x7f049c06e720 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f049c072180 tx=0x7f048c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34786ecfcb85313f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:16.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.444+0000 7f049be00640 1 -- 172.21.15.43:0/586903144 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c02cbf0 con 0x7f049c0702a0 2024-01-16T15:20:16.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.444+0000 7f049be00640 1 -- 172.21.15.43:0/586903144 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c02cd50 con 0x7f049c0702a0 2024-01-16T15:20:16.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.444+0000 7f049be00640 1 -- 172.21.15.43:0/586903144 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c034660 con 0x7f049c0702a0 2024-01-16T15:20:16.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 -- 172.21.15.43:0/586903144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c0702a0 msgr2=0x7f049c06e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 --2- 172.21.15.43:0/586903144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c0702a0 0x7f049c06e720 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f049c072180 tx=0x7f048c02be40 comp rx=0 tx=0).stop 2024-01-16T15:20:16.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 -- 172.21.15.43:0/586903144 shutdown_connections 2024-01-16T15:20:16.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 --2- 172.21.15.43:0/586903144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c0702a0 0x7f049c06e720 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 --2- 172.21.15.43:0/586903144 >> v2:172.21.15.69:3300/0 conn(0x7f049c0cd150 0x7f049c06fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 -- 172.21.15.43:0/586903144 >> 172.21.15.43:0/586903144 conn(0x7f049c0cdb20 msgr2=0x7f049c0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:16.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 -- 172.21.15.43:0/586903144 shutdown_connections 2024-01-16T15:20:16.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.445+0000 7f04a4000640 1 -- 172.21.15.43:0/586903144 wait complete. 2024-01-16T15:20:16.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.447+0000 7f04a4000640 1 Processor -- start 2024-01-16T15:20:16.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.448+0000 7f04a4000640 1 -- start start 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049c0702a0 0x7f049c0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049c0cd150 0x7f049c0c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0c7ab0 con 0x7f049c06ec60 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c155e10 con 0x7f049c0cd150 2024-01-16T15:20:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a4000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0c6f30 con 0x7f049c0702a0 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54824/0 (socket says 172.21.15.43:54824) 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.449+0000 7f04a1600640 1 -- 172.21.15.43:0/1570717772 learned_addr learned my addr 172.21.15.43:0/1570717772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 -- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049c0702a0 msgr2=0x7f049c0c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049c0702a0 0x7f049c0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 -- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049c0cd150 msgr2=0x7f049c0c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a2000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049c0cd150 0x7f049c0c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049c0cd150 0x7f049c0c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 -- 172.21.15.43:0/1570717772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c009a70 con 0x7f049c06ec60 2024-01-16T15:20:16.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.450+0000 7f04a1600640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f048400e870 tx=0x7f048400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.451+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840094f0 con 0x7f049c06ec60 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.451+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048400c940 con 0x7f049c06ec60 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.451+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048400ccf0 con 0x7f049c06ec60 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.451+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c0c7210 con 0x7f049c06ec60 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.451+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c0c77d0 con 0x7f049c06ec60 2024-01-16T15:20:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.452+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c06fb50 con 0x7f049c06ec60 2024-01-16T15:20:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.453+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 44) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0484013470 con 0x7f049c06ec60 2024-01-16T15:20:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.454+0000 7f0499600640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 0x7f047406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.454+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0484088930 con 0x7f049c06ec60 2024-01-16T15:20:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.455+0000 7f04a0c00640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 0x7f047406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.455+0000 7f04a0c00640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 0x7f047406ea10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f048c036000 tx=0x7f048c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:16.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.458+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04840551f0 con 0x7f049c06ec60 2024-01-16T15:20:16.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:16.696+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42751", "target": ["mon-mgr", ""]}) v1 -- 0x7f049c143560 con 0x7f047406c550 2024-01-16T15:20:30.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:30.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:34.214 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:34.221+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:34.214 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:34.221+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:34.214 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:34.221+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:34.214 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:34.221+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:34.214 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:34.221+0000 7fd4eaa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.453+0000 7f0499600640 1 -- 172.21.15.43:0/1570717772 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f048c034440 con 0x7f047406c550 2024-01-16T15:20:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 msgr2=0x7f047406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 0x7f047406ea10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f048c036000 tx=0x7f048c02e000 comp rx=0 tx=0).stop 2024-01-16T15:20:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 msgr2=0x7f049c1558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f048400e870 tx=0x7f048400ed30 comp rx=0 tx=0).stop 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 shutdown_connections 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f047406c550 0x7f047406ea10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f049c0cd150 0x7f049c0c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f049c0702a0 0x7f049c0c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 --2- 172.21.15.43:0/1570717772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f049c06ec60 0x7f049c1558d0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 >> 172.21.15.43:0/1570717772 conn(0x7f049c0cdb20 msgr2=0x7f049c1600a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 shutdown_connections 2024-01-16T15:20:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:34.457+0000 7f04a4000640 1 -- 172.21.15.43:0/1570717772 wait complete. 2024-01-16T15:20:36.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42752 2024-01-16T15:20:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.712+0000 7fbcd5400640 1 Processor -- start 2024-01-16T15:20:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.712+0000 7fbcd5400640 1 -- start start 2024-01-16T15:20:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 0x7fbcd00b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbcd006d570 0x7fbcd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd006ae00 con 0x7fbcd006d570 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd00b4dc0 con 0x7fbcd01439e0 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.713+0000 7fbcd5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0143dc0 con 0x7fbcd006a440 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbcd006d570 0x7fbcd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbcd006d570 0x7fbcd006d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60322/0 (socket says 172.21.15.43:60322) 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 -- 172.21.15.43:0/103066864 learned_addr learned my addr 172.21.15.43:0/103066864 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:36.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbccfe00640 1 --2- 172.21.15.43:0/103066864 >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 0x7fbcd00b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbccf400640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 -- 172.21.15.43:0/103066864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 msgr2=0x7fbcd006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd006a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 -- 172.21.15.43:0/103066864 >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 msgr2=0x7fbcd00b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 --2- 172.21.15.43:0/103066864 >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 0x7fbcd00b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbcd4a00640 1 -- 172.21.15.43:0/103066864 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0161350 con 0x7fbcd006d570 2024-01-16T15:20:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.714+0000 7fbccfe00640 1 --2- 172.21.15.43:0/103066864 >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 0x7fbcd00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.715+0000 7fbcd4a00640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd006d9c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fbcd006c4c0 tx=0x7fbcc802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b6842f5a67ee08 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbccea00640 1 -- 172.21.15.43:0/103066864 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc802ca80 con 0x7fbcd006d570 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbccea00640 1 -- 172.21.15.43:0/103066864 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc802cbe0 con 0x7fbcd006d570 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbccea00640 1 -- 172.21.15.43:0/103066864 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80343f0 con 0x7fbcd006d570 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 -- 172.21.15.43:0/103066864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 msgr2=0x7fbcd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd006d9c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fbcd006c4c0 tx=0x7fbcc802bee0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 -- 172.21.15.43:0/103066864 shutdown_connections 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd006d9c0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 --2- 172.21.15.43:0/103066864 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 --2- 172.21.15.43:0/103066864 >> v2:172.21.15.69:3300/0 conn(0x7fbcd01439e0 0x7fbcd00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.716+0000 7fbcd5400640 1 -- 172.21.15.43:0/103066864 >> 172.21.15.43:0/103066864 conn(0x7fbcd006bbb0 msgr2=0x7fbcd00b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.717+0000 7fbcd5400640 1 -- 172.21.15.43:0/103066864 shutdown_connections 2024-01-16T15:20:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.717+0000 7fbcd5400640 1 -- 172.21.15.43:0/103066864 wait complete. 2024-01-16T15:20:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.718+0000 7fbcd5400640 1 Processor -- start 2024-01-16T15:20:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.719+0000 7fbcd5400640 1 -- start start 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd013dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd01439e0 0x7fbcd00b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd015df20 con 0x7fbcd006d570 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd01f5be0 con 0x7fbcd01439e0 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbcd5400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd01f5ec0 con 0x7fbcd006a440 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbccf400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbccf400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60324/0 (socket says 172.21.15.43:60324) 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.720+0000 7fbccf400640 1 -- 172.21.15.43:0/2533824061 learned_addr learned my addr 172.21.15.43:0/2533824061 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccfe00640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd013dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbcd4a00640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd01439e0 0x7fbcd00b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 -- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 msgr2=0x7fbcd013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:36.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd013dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 -- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd01439e0 msgr2=0x7fbcd00b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd01439e0 0x7fbcd00b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 -- 172.21.15.43:0/2533824061 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc8009b30 con 0x7fbcd006d570 2024-01-16T15:20:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbccf400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800c570 tx=0x7fbcb800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:36.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.721+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8011800 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.722+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb8011960 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.722+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8011d10 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.722+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd01f61a0 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.722+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd015efd0 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.723+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd01f66c0 con 0x7fbcd006d570 2024-01-16T15:20:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.724+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 45) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fbcb8018360 con 0x7fbcd006d570 2024-01-16T15:20:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.725+0000 7fbcc7e00640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 0x7fbca806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.725+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbcb8088b80 con 0x7fbcd006d570 2024-01-16T15:20:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.725+0000 7fbccfe00640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 0x7fbca806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.726+0000 7fbccfe00640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 0x7fbca806ead0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000a990 tx=0x7fbcc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.729+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbcd01f66c0 con 0x7fbcd006d570 2024-01-16T15:20:36.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:36.961+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42752", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcd01f5d50 con 0x7fbca806c610 2024-01-16T15:20:44.194 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:44.200+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:44.194 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:44.200+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:44.194 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:44.201+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:44.194 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:44.201+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:44.194 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:20:44.201+0000 7fd4eb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:20:46.418+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 46) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fbcb8055470 con 0x7fbcd006d570 2024-01-16T15:21:00.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:00.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:05.527 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:21:05.533+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:05.527 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:21:05.533+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:05.527 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:21:05.533+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:05.527 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:21:05.533+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:05.527 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:21:05.533+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:06.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:21:06.427+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 47) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fbcb8003610 con 0x7fbcd006d570 2024-01-16T15:21:31.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:31.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:21:46.445+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mon.0 v2:172.21.15.43:3300/0 9 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fbcb8071220 con 0x7fbcd006d570 2024-01-16T15:22:01.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:01.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:31.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:31.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:34.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.675+0000 7fbcc7e00640 1 -- 172.21.15.43:0/2533824061 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbcd01f5d50 con 0x7fbca806c610 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 msgr2=0x7fbca806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 0x7fbca806ead0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000a990 tx=0x7fbcc0008040 comp rx=0 tx=0).stop 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 msgr2=0x7fbcd015ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800c570 tx=0x7fbcb800ca30 comp rx=0 tx=0).stop 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 shutdown_connections 2024-01-16T15:22:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fbca806c610 0x7fbca806ead0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd01439e0 0x7fbcd00b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbcd006d570 0x7fbcd015ea90 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 --2- 172.21.15.43:0/2533824061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbcd006a440 0x7fbcd013dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 >> 172.21.15.43:0/2533824061 conn(0x7fbcd006bbb0 msgr2=0x7fbcd013e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 shutdown_connections 2024-01-16T15:22:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:34.679+0000 7fbcd5400640 1 -- 172.21.15.43:0/2533824061 wait complete. 2024-01-16T15:22:36.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42753 2024-01-16T15:22:36.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.938+0000 7fb49de00640 1 Processor -- start 2024-01-16T15:22:36.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.939+0000 7fb49de00640 1 -- start start 2024-01-16T15:22:36.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb49de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb4980d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb49de00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4980cf240 con 0x7fb4980d1840 2024-01-16T15:22:36.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb49de00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4980d1fb0 con 0x7fb49815d350 2024-01-16T15:22:36.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb49de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980ce050 con 0x7fb49815d730 2024-01-16T15:22:36.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb497400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb4980d1840 0x7fb4980cfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50726/0 (socket says 172.21.15.43:50726) 2024-01-16T15:22:36.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.940+0000 7fb497400640 1 -- 172.21.15.43:0/4131873275 learned_addr learned my addr 172.21.15.43:0/4131873275 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:22:36.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb496000640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb4980d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842800629 0 0) 0x7fb4980cf240 con 0x7fb4980d1840 2024-01-16T15:22:36.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb478003520 con 0x7fb4980d1840 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218828927 0 0) 0x7fb4980d1fb0 con 0x7fb49815d350 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4980cf240 con 0x7fb49815d350 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334145791 0 0) 0x7fb478003520 con 0x7fb4980d1840 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 msgr2=0x7fb4980d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.941+0000 7fb495600640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb4980d3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 >> v1:172.21.15.69:6789/0 conn(0x7fb49815d350 legacy=0x7fb4980ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980ce330 con 0x7fb4980d1840 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb4900019f0 con 0x7fb4980d1840 2024-01-16T15:22:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 >> v1:172.21.15.43:6789/0 conn(0x7fb4980d1840 legacy=0x7fb4980cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb478003b30 0x7fb478003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4780047c0 0x7fb478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb478005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980ce050 con 0x7fb4780047c0 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800ba30 con 0x7fb478003b30 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800bbc0 con 0x7fb49815d730 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb496000640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4780047c0 0x7fb478004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb496a00640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb478003b30 0x7fb478003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb496000640 1 -- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 msgr2=0x7fb478005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.942+0000 7fb497400640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb478005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb496000640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb478005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb496000640 1 -- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb478003b30 msgr2=0x7fb478003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:36.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb496000640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb478003b30 0x7fb478003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb496000640 1 -- 172.21.15.43:0/4131873275 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478003520 con 0x7fb4780047c0 2024-01-16T15:22:36.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb497400640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb49815d730 0x7fb478005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:36.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.943+0000 7fb496a00640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb478003b30 0x7fb478003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb496000640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4780047c0 0x7fb478004c40 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fb480009a90 tx=0x7fb4780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceee0a5fa02142da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48002c630 con 0x7fb4780047c0 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48002c790 con 0x7fb4780047c0 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb495600640 1 -- 172.21.15.43:0/4131873275 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48002cb20 con 0x7fb4780047c0 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb49de00640 1 -- 172.21.15.43:0/4131873275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4780047c0 msgr2=0x7fb478004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.944+0000 7fb49de00640 1 --2- 172.21.15.43:0/4131873275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4780047c0 0x7fb478004c40 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fb480009a90 tx=0x7fb4780057f0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.945+0000 7fb497400640 1 -- 172.21.15.43:0/4131873275 reap_dead start 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.945+0000 7fb49de00640 1 -- 172.21.15.43:0/4131873275 shutdown_connections 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.945+0000 7fb49de00640 1 -- 172.21.15.43:0/4131873275 >> 172.21.15.43:0/4131873275 conn(0x7fb49809c5c0 msgr2=0x7fb4980d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.945+0000 7fb49de00640 1 -- 172.21.15.43:0/4131873275 shutdown_connections 2024-01-16T15:22:36.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.945+0000 7fb49de00640 1 -- 172.21.15.43:0/4131873275 wait complete. 2024-01-16T15:22:36.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.947+0000 7fb49de00640 1 Processor -- start 2024-01-16T15:22:36.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.947+0000 7fb49de00640 1 -- start start 2024-01-16T15:22:36.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4780047c0 0x7fb4981f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4980b3690 0x7fb4980b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49814bc70 con 0x7fb49814b2b0 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4981498c0 con 0x7fb4980b3690 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb49de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980b4050 con 0x7fb4780047c0 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47672/0 (socket says 172.21.15.43:47672) 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 -- 172.21.15.43:0/2583424634 learned_addr learned my addr 172.21.15.43:0/2583424634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496a00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4780047c0 0x7fb4981f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 -- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4780047c0 msgr2=0x7fb4981f6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4780047c0 0x7fb4981f6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 -- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4980b3690 msgr2=0x7fb4980b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4980b3690 0x7fb4980b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:36.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.948+0000 7fb496000640 1 -- 172.21.15.43:0/2583424634 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49815c670 con 0x7fb49814b2b0 2024-01-16T15:22:36.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb496000640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb480037040 tx=0x7fb480004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480003810 con 0x7fb49814b2b0 2024-01-16T15:22:36.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480003970 con 0x7fb49814b2b0 2024-01-16T15:22:36.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480003ce0 con 0x7fb49814b2b0 2024-01-16T15:22:36.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49814be70 con 0x7fb49814b2b0 2024-01-16T15:22:36.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.949+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb498149270 con 0x7fb49814b2b0 2024-01-16T15:22:36.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.950+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb498149a70 con 0x7fb49814b2b0 2024-01-16T15:22:36.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.951+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fb480034440 con 0x7fb49814b2b0 2024-01-16T15:22:36.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.952+0000 7fb48f400640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 0x7fb47006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:36.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.952+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb4800aa990 con 0x7fb49814b2b0 2024-01-16T15:22:36.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.953+0000 7fb496a00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 0x7fb47006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:36.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.953+0000 7fb496a00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 0x7fb47006eb60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb4880016f0 tx=0x7fb488009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:36.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:36.957+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb498149a70 con 0x7fb49814b2b0 2024-01-16T15:22:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:37.186+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42753", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4980cdd00 con 0x7fb47006c6a0 2024-01-16T15:22:39.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.644+0000 7fb48f400640 1 -- 172.21.15.43:0/2583424634 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb48800d040 con 0x7fb47006c6a0 2024-01-16T15:22:39.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.647+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 msgr2=0x7fb47006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.647+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 0x7fb47006eb60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb4880016f0 tx=0x7fb488009040 comp rx=0 tx=0).stop 2024-01-16T15:22:39.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.647+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 msgr2=0x7fb49814b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:39.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.647+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb480037040 tx=0x7fb480004730 comp rx=0 tx=0).stop 2024-01-16T15:22:39.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 shutdown_connections 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb47006c6a0 0x7fb47006eb60 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4980b3690 0x7fb4980b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb49814b2b0 0x7fb49814b730 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 --2- 172.21.15.43:0/2583424634 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4780047c0 0x7fb4981f6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 >> 172.21.15.43:0/2583424634 conn(0x7fb49809c5c0 msgr2=0x7fb4980cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 shutdown_connections 2024-01-16T15:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:39.648+0000 7fb49de00640 1 -- 172.21.15.43:0/2583424634 wait complete. 2024-01-16T15:22:41.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42754 2024-01-16T15:22:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.902+0000 7fe90ca00640 1 Processor -- start 2024-01-16T15:22:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.902+0000 7fe90ca00640 1 -- start start 2024-01-16T15:22:41.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.903+0000 7fe90ca00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe90406cad0 0x7fe90406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.903+0000 7fe90ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904072280 0x7fe9040b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.903+0000 7fe90ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe904071780 con 0x7fe904160b10 2024-01-16T15:22:41.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.903+0000 7fe90ca00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9040b2320 con 0x7fe90406cad0 2024-01-16T15:22:41.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.903+0000 7fe90ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90406df30 con 0x7fe904072280 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe90406cad0 0x7fe90406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe90406cad0 0x7fe90406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47680/0 (socket says 172.21.15.43:47680) 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 -- 172.21.15.43:0/3475256286 learned_addr learned my addr 172.21.15.43:0/3475256286 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe90aa00640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904072280 0x7fe9040b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 -- 172.21.15.43:0/3475256286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904072280 msgr2=0x7fe9040b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904072280 0x7fe9040b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 -- 172.21.15.43:0/3475256286 >> v1:172.21.15.69:6789/0 conn(0x7fe904160b10 legacy=0x7fe90406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.904+0000 7fe909600640 1 -- 172.21.15.43:0/3475256286 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9040711c0 con 0x7fe90406cad0 2024-01-16T15:22:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.906+0000 7fe909600640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe90406cad0 0x7fe90406e420 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e50 tx=0x7fe8ec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235c22505e3182b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.906+0000 7fe908c00640 1 -- 172.21.15.43:0/3475256286 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec02c9e0 con 0x7fe90406cad0 2024-01-16T15:22:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.906+0000 7fe908c00640 1 -- 172.21.15.43:0/3475256286 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec02cb40 con 0x7fe90406cad0 2024-01-16T15:22:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.906+0000 7fe908c00640 1 -- 172.21.15.43:0/3475256286 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0343f0 con 0x7fe90406cad0 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 -- 172.21.15.43:0/3475256286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe90406cad0 msgr2=0x7fe90406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe90406cad0 0x7fe90406e420 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e50 tx=0x7fe8ec02be40 comp rx=0 tx=0).stop 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 -- 172.21.15.43:0/3475256286 shutdown_connections 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe90406cad0 0x7fe90406e420 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 --2- 172.21.15.43:0/3475256286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904072280 0x7fe9040b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 -- 172.21.15.43:0/3475256286 >> 172.21.15.43:0/3475256286 conn(0x7fe9040b0ba0 msgr2=0x7fe9040b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 -- 172.21.15.43:0/3475256286 shutdown_connections 2024-01-16T15:22:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.907+0000 7fe90ca00640 1 -- 172.21.15.43:0/3475256286 wait complete. 2024-01-16T15:22:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.909+0000 7fe90ca00640 1 Processor -- start 2024-01-16T15:22:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.909+0000 7fe90ca00640 1 -- start start 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 0x7fe9040ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904160b10 0x7fe90415a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90409cba0 con 0x7fe904072280 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9040cf290 con 0x7fe90406cad0 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe90ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90409c8c0 con 0x7fe904160b10 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe909600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe909600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47688/0 (socket says 172.21.15.43:47688) 2024-01-16T15:22:41.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.910+0000 7fe909600640 1 -- 172.21.15.43:0/1440796717 learned_addr learned my addr 172.21.15.43:0/1440796717 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 -- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904160b10 msgr2=0x7fe90415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904160b10 0x7fe90415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe90a000640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 0x7fe9040ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 -- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 msgr2=0x7fe9040ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 0x7fe9040ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 -- 172.21.15.43:0/1440796717 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009b30 con 0x7fe904072280 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe90a000640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 0x7fe9040ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe909600640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e50 tx=0x7fe8ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec009510 con 0x7fe904072280 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec03e450 con 0x7fe904072280 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec03e800 con 0x7fe904072280 2024-01-16T15:22:41.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.911+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe904099ba0 con 0x7fe904072280 2024-01-16T15:22:41.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.912+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90414f080 con 0x7fe904072280 2024-01-16T15:22:41.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.913+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe904099a00 con 0x7fe904072280 2024-01-16T15:22:41.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.914+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 48) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fe8ec03e960 con 0x7fe904072280 2024-01-16T15:22:41.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.914+0000 7fe902000640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 0x7fe8dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:41.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.915+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe8ec0aba70 con 0x7fe904072280 2024-01-16T15:22:41.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.915+0000 7fe90a000640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 0x7fe8dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:41.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.916+0000 7fe90a000640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 0x7fe8dc06eb30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400a830 tx=0x7fe8f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:41.919+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe904099a00 con 0x7fe904072280 2024-01-16T15:22:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:42.154+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42754", "target": ["mon-mgr", ""]}) v1 -- 0x7fe90409cf20 con 0x7fe8dc06c670 2024-01-16T15:22:44.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.644+0000 7fe902000640 1 -- 172.21.15.43:0/1440796717 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe90409cf20 con 0x7fe8dc06c670 2024-01-16T15:22:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 msgr2=0x7fe8dc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 0x7fe8dc06eb30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400a830 tx=0x7fe8f4008040 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 msgr2=0x7fe90414eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e50 tx=0x7fe8ec004a40 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 shutdown_connections 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe8dc06c670 0x7fe8dc06eb30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe904160b10 0x7fe90415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe904072280 0x7fe90414eb40 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 --2- 172.21.15.43:0/1440796717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90406cad0 0x7fe9040ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 >> 172.21.15.43:0/1440796717 conn(0x7fe9040b0ba0 msgr2=0x7fe9040ad2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 shutdown_connections 2024-01-16T15:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:44.648+0000 7fe90ca00640 1 -- 172.21.15.43:0/1440796717 wait complete. 2024-01-16T15:22:46.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42755 2024-01-16T15:22:46.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.905+0000 7f4e57e00640 1 Processor -- start 2024-01-16T15:22:46.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.905+0000 7f4e57e00640 1 -- start start 2024-01-16T15:22:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e57e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4e58144770 0x7f4e5815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e57e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e5815f0a0 0x7f4e580ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e57e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5815f930 con 0x7f4e580b35a0 2024-01-16T15:22:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e57e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e580b43b0 con 0x7f4e58144770 2024-01-16T15:22:46.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e57e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5815f510 con 0x7f4e5815f0a0 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e56000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4e580b35a0 0x7f4e5806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33368/0 (socket says 172.21.15.43:33368) 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e56000640 1 -- 172.21.15.43:0/3703102501 learned_addr learned my addr 172.21.15.43:0/3703102501 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.907+0000 7f4e56a00640 1 --2- 172.21.15.43:0/3703102501 >> v2:172.21.15.43:3300/0 conn(0x7f4e58144770 0x7f4e5815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e57400640 1 --2- 172.21.15.43:0/3703102501 >> v2:172.21.15.69:3300/0 conn(0x7f4e5815f0a0 0x7f4e580ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e55600640 1 -- 172.21.15.43:0/3703102501 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002600683 0 0) 0x7f4e5815f930 con 0x7f4e580b35a0 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e55600640 1 -- 172.21.15.43:0/3703102501 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e38003520 con 0x7f4e580b35a0 2024-01-16T15:22:46.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e56a00640 1 -- 172.21.15.43:0/3703102501 >> v2:172.21.15.69:3300/0 conn(0x7f4e5815f0a0 msgr2=0x7f4e580ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:22:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e56a00640 1 --2- 172.21.15.43:0/3703102501 >> v2:172.21.15.69:3300/0 conn(0x7f4e5815f0a0 0x7f4e580ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e56a00640 1 -- 172.21.15.43:0/3703102501 >> v1:172.21.15.43:6789/0 conn(0x7f4e580b35a0 legacy=0x7f4e5806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.908+0000 7f4e56a00640 1 -- 172.21.15.43:0/3703102501 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580ce9f0 con 0x7f4e58144770 2024-01-16T15:22:46.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.909+0000 7f4e56a00640 1 --2- 172.21.15.43:0/3703102501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e5815eb60 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4e580700e0 tx=0x7f4e4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86f3461d7f666c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.909+0000 7f4e55600640 1 -- 172.21.15.43:0/3703102501 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48031830 con 0x7f4e58144770 2024-01-16T15:22:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e55600640 1 -- 172.21.15.43:0/3703102501 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48031990 con 0x7f4e58144770 2024-01-16T15:22:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e55600640 1 -- 172.21.15.43:0/3703102501 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48031d00 con 0x7f4e58144770 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 -- 172.21.15.43:0/3703102501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 msgr2=0x7f4e5815eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 --2- 172.21.15.43:0/3703102501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e5815eb60 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4e580700e0 tx=0x7f4e4802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 -- 172.21.15.43:0/3703102501 shutdown_connections 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 --2- 172.21.15.43:0/3703102501 >> v2:172.21.15.69:3300/0 conn(0x7f4e5815f0a0 0x7f4e580ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 --2- 172.21.15.43:0/3703102501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e5815eb60 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 -- 172.21.15.43:0/3703102501 >> 172.21.15.43:0/3703102501 conn(0x7f4e580cec10 msgr2=0x7f4e580cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 -- 172.21.15.43:0/3703102501 shutdown_connections 2024-01-16T15:22:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.910+0000 7f4e57e00640 1 -- 172.21.15.43:0/3703102501 wait complete. 2024-01-16T15:22:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.912+0000 7f4e57e00640 1 Processor -- start 2024-01-16T15:22:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.913+0000 7f4e57e00640 1 -- start start 2024-01-16T15:22:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.913+0000 7f4e57e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 0x7f4e58140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815f0a0 0x7f4e581f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58158ef0 con 0x7f4e58144770 2024-01-16T15:22:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58140640 con 0x7f4e580b35a0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58140920 con 0x7f4e5815f0a0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51866/0 (socket says 172.21.15.43:51866) 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 -- 172.21.15.43:0/263114832 learned_addr learned my addr 172.21.15.43:0/263114832 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56a00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 0x7f4e58140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e57400640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815f0a0 0x7f4e581f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 -- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815f0a0 msgr2=0x7f4e581f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815f0a0 0x7f4e581f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 -- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 msgr2=0x7f4e58140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 0x7f4e58140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56000640 1 -- 172.21.15.43:0/263114832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48009400 con 0x7f4e58144770 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.914+0000 7f4e56a00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 0x7f4e58140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e56000640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000c430 tx=0x7f4e4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40005ce0 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40005e40 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e400072f0 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e58140c00 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.915+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e581411c0 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.916+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5804e840 con 0x7f4e58144770 2024-01-16T15:22:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.917+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f4e40007450 con 0x7f4e58144770 2024-01-16T15:22:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.919+0000 7f4e4ea00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 0x7f4e3006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:22:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.919+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e40088270 con 0x7f4e58144770 2024-01-16T15:22:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.919+0000 7f4e56a00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 0x7f4e3006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:22:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.920+0000 7f4e56a00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 0x7f4e3006eaa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:22:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:46.924+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e4008f030 con 0x7f4e58144770 2024-01-16T15:22:47.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:22:47.159+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42755", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e580b1de0 con 0x7f4e3006c5e0 2024-01-16T15:23:01.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:01.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:06.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.018+0000 7f4e4ea00640 1 -- 172.21.15.43:0/263114832 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4e580b1de0 con 0x7f4e3006c5e0 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 msgr2=0x7f4e3006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 0x7f4e3006eaa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48008030 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 msgr2=0x7f4e581589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000c430 tx=0x7f4e4000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 shutdown_connections 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4e3006c5e0 0x7f4e3006eaa0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e5815f0a0 0x7f4e581f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e58144770 0x7f4e581589b0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 --2- 172.21.15.43:0/263114832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e580b35a0 0x7f4e58140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:06.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 >> 172.21.15.43:0/263114832 conn(0x7f4e580cec10 msgr2=0x7f4e58159420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:06.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 shutdown_connections 2024-01-16T15:23:06.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:06.023+0000 7f4e57e00640 1 -- 172.21.15.43:0/263114832 wait complete. 2024-01-16T15:23:08.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42756 2024-01-16T15:23:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.243+0000 7f1165600640 1 Processor -- start 2024-01-16T15:23:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.243+0000 7f1165600640 1 -- start start 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f1165600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f116006cad0 0x7f116006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f1165600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1160072280 0x7f11600b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f1165600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1160071780 con 0x7f1160160b10 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f1165600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11600b2320 con 0x7f1160072280 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f1165600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116006df30 con 0x7f116006cad0 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.245+0000 7f115f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1160072280 0x7f11600b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1160072280 0x7f11600b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45918/0 (socket says 172.21.15.43:45918) 2024-01-16T15:23:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 -- 172.21.15.43:0/2386705512 learned_addr learned my addr 172.21.15.43:0/2386705512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:23:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 -- 172.21.15.43:0/2386705512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f116006cad0 msgr2=0x7f116006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 --2- 172.21.15.43:0/2386705512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f116006cad0 0x7f116006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 -- 172.21.15.43:0/2386705512 >> v1:172.21.15.69:6789/0 conn(0x7f1160160b10 legacy=0x7f116006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.246+0000 7f115f400640 1 -- 172.21.15.43:0/2386705512 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11600711c0 con 0x7f1160072280 2024-01-16T15:23:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.247+0000 7f115f400640 1 --2- 172.21.15.43:0/2386705512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600b2740 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f1158009e80 tx=0x7f115802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50faa1712910d753 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.247+0000 7f115d600640 1 -- 172.21.15.43:0/2386705512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115802c9c0 con 0x7f1160072280 2024-01-16T15:23:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.247+0000 7f115d600640 1 -- 172.21.15.43:0/2386705512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115802cb20 con 0x7f1160072280 2024-01-16T15:23:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f115d600640 1 -- 172.21.15.43:0/2386705512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11580343f0 con 0x7f1160072280 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 -- 172.21.15.43:0/2386705512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 msgr2=0x7f11600b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 --2- 172.21.15.43:0/2386705512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600b2740 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f1158009e80 tx=0x7f115802be20 comp rx=0 tx=0).stop 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 -- 172.21.15.43:0/2386705512 shutdown_connections 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 --2- 172.21.15.43:0/2386705512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600b2740 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 --2- 172.21.15.43:0/2386705512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f116006cad0 0x7f116006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 -- 172.21.15.43:0/2386705512 >> 172.21.15.43:0/2386705512 conn(0x7f11600b0ba0 msgr2=0x7f11600b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:08.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 -- 172.21.15.43:0/2386705512 shutdown_connections 2024-01-16T15:23:08.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.248+0000 7f1165600640 1 -- 172.21.15.43:0/2386705512 wait complete. 2024-01-16T15:23:08.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.250+0000 7f1165600640 1 Processor -- start 2024-01-16T15:23:08.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.250+0000 7f1165600640 1 -- start start 2024-01-16T15:23:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f116006cad0 0x7f116015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1160160b10 0x7f11600ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11601e77e0 con 0x7f1160072280 2024-01-16T15:23:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11601e7a30 con 0x7f116006cad0 2024-01-16T15:23:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.251+0000 7f1165600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11601e7d10 con 0x7f1160160b10 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45930/0 (socket says 172.21.15.43:45930) 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 -- 172.21.15.43:0/145520373 learned_addr learned my addr 172.21.15.43:0/145520373 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115ea00640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f116006cad0 0x7f116015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115f400640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1160160b10 0x7f11600ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 -- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1160160b10 msgr2=0x7f11600ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1160160b10 0x7f11600ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 -- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f116006cad0 msgr2=0x7f116015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f116006cad0 0x7f116015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 -- 172.21.15.43:0/145520373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158009b30 con 0x7f1160072280 2024-01-16T15:23:08.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115f400640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1160160b10 0x7f11600ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:08.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115ea00640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f116006cad0 0x7f116015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:23:08.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.252+0000 7f115e000640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160072280 0x7f11600cde50 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f114800b4a0 tx=0x7f114800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.253+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148010610 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.253+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1148010770 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.253+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148010b20 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.253+0000 7f1165600640 1 -- 172.21.15.43:0/145520373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11601e7ff0 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.253+0000 7f1165600640 1 -- 172.21.15.43:0/145520373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11601e85b0 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.254+0000 7f1165600640 1 -- 172.21.15.43:0/145520373 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116009adb0 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.255+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f1148010c80 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.256+0000 7f1156a00640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f113806c5e0 0x7f113806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.256+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1148088800 con 0x7f1160072280 2024-01-16T15:23:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.256+0000 7f115ea00640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f113806c5e0 0x7f113806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:08.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.257+0000 7f115ea00640 1 --2- 172.21.15.43:0/145520373 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f113806c5e0 0x7f113806eaa0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f1150006fd0 tx=0x7f1150008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:08.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.261+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1148054d20 con 0x7f1160072280 2024-01-16T15:23:08.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:08.493+0000 7f1165600640 1 -- 172.21.15.43:0/145520373 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone42756", "target": ["mon-mgr", ""]}) v1 -- 0x7f1160071d50 con 0x7f113806c5e0 2024-01-16T15:23:16.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:23:16.481+0000 7f1156a00640 1 -- 172.21.15.43:0/145520373 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f11480550f0 con 0x7f1160072280 2024-01-16T15:23:31.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:31.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:01.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:01.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:32.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:32.022 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:02.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:02.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-16T15:25:08.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:25:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.378+0000 7f62f3200640 1 Processor -- start 2024-01-16T15:25:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.378+0000 7f62f3200640 1 -- start start 2024-01-16T15:25:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.380+0000 7f62f3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62ec0b43a0 0x7f62ec0b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.380+0000 7f62f3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec162f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.380+0000 7f62f3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec0705a0 con 0x7f62ec161ed0 2024-01-16T15:25:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.380+0000 7f62f3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec0b4cc0 con 0x7f62ec0b43a0 2024-01-16T15:25:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.380+0000 7f62f3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec070800 con 0x7f62ec06c2c0 2024-01-16T15:25:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62f1200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec162f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62eb400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f62ec161ed0 0x7f62ec06fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50350/0 (socket says 172.21.15.43:50350) 2024-01-16T15:25:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62eb400640 1 -- 172.21.15.43:0/3753719718 learned_addr learned my addr 172.21.15.43:0/3753719718 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62ebe00640 1 --2- 172.21.15.43:0/3753719718 >> v2:172.21.15.69:3300/0 conn(0x7f62ec0b43a0 0x7f62ec0b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62f1200640 1 -- 172.21.15.43:0/3753719718 >> v2:172.21.15.69:3300/0 conn(0x7f62ec0b43a0 msgr2=0x7f62ec0b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62f1200640 1 --2- 172.21.15.43:0/3753719718 >> v2:172.21.15.69:3300/0 conn(0x7f62ec0b43a0 0x7f62ec0b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62f1200640 1 -- 172.21.15.43:0/3753719718 >> v1:172.21.15.69:6789/0 conn(0x7f62ec161ed0 legacy=0x7f62ec06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62eaa00640 1 -- 172.21.15.43:0/3753719718 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401795797 0 0) 0x7f62ec0705a0 con 0x7f62ec161ed0 2024-01-16T15:25:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.381+0000 7f62f1200640 1 -- 172.21.15.43:0/3753719718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec070ae0 con 0x7f62ec06c2c0 2024-01-16T15:25:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.383+0000 7f62f1200640 1 --2- 172.21.15.43:0/3753719718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec162f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009d90 tx=0x7f62e402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23e1af39dbe7bb39 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.383+0000 7f62eaa00640 1 -- 172.21.15.43:0/3753719718 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e402cc50 con 0x7f62ec06c2c0 2024-01-16T15:25:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.383+0000 7f62eaa00640 1 -- 172.21.15.43:0/3753719718 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e402cdb0 con 0x7f62ec06c2c0 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.383+0000 7f62eaa00640 1 -- 172.21.15.43:0/3753719718 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4034660 con 0x7f62ec06c2c0 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 -- 172.21.15.43:0/3753719718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 msgr2=0x7f62ec162f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 --2- 172.21.15.43:0/3753719718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec162f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009d90 tx=0x7f62e402bea0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 -- 172.21.15.43:0/3753719718 shutdown_connections 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 --2- 172.21.15.43:0/3753719718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec162f40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 --2- 172.21.15.43:0/3753719718 >> v2:172.21.15.69:3300/0 conn(0x7f62ec0b43a0 0x7f62ec0b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 -- 172.21.15.43:0/3753719718 >> 172.21.15.43:0/3753719718 conn(0x7f62ec0b2d90 msgr2=0x7f62ec0b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 -- 172.21.15.43:0/3753719718 shutdown_connections 2024-01-16T15:25:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.384+0000 7f62f3200640 1 -- 172.21.15.43:0/3753719718 wait complete. 2024-01-16T15:25:08.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.386+0000 7f62f3200640 1 Processor -- start 2024-01-16T15:25:08.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.386+0000 7f62f3200640 1 -- start start 2024-01-16T15:25:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec15e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ec0b43a0 0x7f62ec15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec15a780 con 0x7f62ec161ed0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec159860 con 0x7f62ec0b43a0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.387+0000 7f62f3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec15ec70 con 0x7f62ec06c2c0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56648/0 (socket says 172.21.15.43:56648) 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 -- 172.21.15.43:0/1942177106 learned_addr learned my addr 172.21.15.43:0/1942177106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62ebe00640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62eb400640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ec0b43a0 0x7f62ec15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 -- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 msgr2=0x7f62ec15e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec15e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 -- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ec0b43a0 msgr2=0x7f62ec15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ec0b43a0 0x7f62ec15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.388+0000 7f62f1200640 1 -- 172.21.15.43:0/1942177106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4009a70 con 0x7f62ec161ed0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62f1200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009d90 tx=0x7f62e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4003b90 con 0x7f62ec161ed0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e4003d10 con 0x7f62ec161ed0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4004270 con 0x7f62ec161ed0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec159b00 con 0x7f62ec161ed0 2024-01-16T15:25:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.389+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec15a0c0 con 0x7f62ec161ed0 2024-01-16T15:25:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.391+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec15a8c0 con 0x7f62ec161ed0 2024-01-16T15:25:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.392+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f62e403b050 con 0x7f62ec161ed0 2024-01-16T15:25:08.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.392+0000 7f62e3e00640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 0x7f62c406ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.393+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f62e40ab020 con 0x7f62ec161ed0 2024-01-16T15:25:08.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.393+0000 7f62ebe00640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 0x7f62c406ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.393+0000 7f62ebe00640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 0x7f62c406ea60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009ad0 tx=0x7f62dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.397+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62e40775a0 con 0x7f62ec161ed0 2024-01-16T15:25:08.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.615+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f62ec06bda0 con 0x7f62c406c5a0 2024-01-16T15:25:08.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.618+0000 7f62e3e00640 1 -- 172.21.15.43:0/1942177106 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f62dc009800 con 0x7f62c406c5a0 2024-01-16T15:25:08.615 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:25:08.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:08.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephfs" 2024-01-16T15:25:08.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:25:08.616 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 msgr2=0x7f62c406ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 0x7f62c406ea60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009ad0 tx=0x7f62dc008040 comp rx=0 tx=0).stop 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 msgr2=0x7f62ec15c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009d90 tx=0x7f62e4004a40 comp rx=0 tx=0).stop 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 shutdown_connections 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f62c406c5a0 0x7f62c406ea60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ec0b43a0 0x7f62ec15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62ec161ed0 0x7f62ec15c520 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.620+0000 7f62f3200640 1 --2- 172.21.15.43:0/1942177106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f62ec06c2c0 0x7f62ec15e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.621+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 >> 172.21.15.43:0/1942177106 conn(0x7f62ec0b2d90 msgr2=0x7f62ec0b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.621+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 shutdown_connections 2024-01-16T15:25:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.621+0000 7f62f3200640 1 -- 172.21.15.43:0/1942177106 wait complete. 2024-01-16T15:25:08.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:25:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.858+0000 7f7b41600640 1 Processor -- start 2024-01-16T15:25:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.858+0000 7f7b41600640 1 -- start start 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b41600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b3c06dc30 0x7f7b3c0ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b41600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b3c070490 0x7f7b3c070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b41600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3c06d7a0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b41600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3c15a5b0 con 0x7f7b3c06dc30 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b41600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3c0ae9d0 con 0x7f7b3c070490 2024-01-16T15:25:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b3c06dc30 0x7f7b3c0ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7b3c06dc30 0x7f7b3c0ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56654/0 (socket says 172.21.15.43:56654) 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 -- 172.21.15.43:0/1055066712 learned_addr learned my addr 172.21.15.43:0/1055066712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 -- 172.21.15.43:0/1055066712 >> v2:172.21.15.69:3300/0 conn(0x7f7b3c070490 msgr2=0x7f7b3c070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 --2- 172.21.15.43:0/1055066712 >> v2:172.21.15.69:3300/0 conn(0x7f7b3c070490 0x7f7b3c070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 -- 172.21.15.43:0/1055066712 >> v1:172.21.15.69:6789/0 conn(0x7f7b3c0acd70 legacy=0x7f7b3c0ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.860+0000 7f7b3a000640 1 -- 172.21.15.43:0/1055066712 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c06ce70 con 0x7f7b3c06dc30 2024-01-16T15:25:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.862+0000 7f7b3a000640 1 --2- 172.21.15.43:0/1055066712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c06dc30 0x7f7b3c0ae490 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c067e70 tx=0x7f7b2402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa8095b635f58bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.862+0000 7f7b39600640 1 -- 172.21.15.43:0/1055066712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2402f8e0 con 0x7f7b3c06dc30 2024-01-16T15:25:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.862+0000 7f7b39600640 1 -- 172.21.15.43:0/1055066712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2402fa40 con 0x7f7b3c06dc30 2024-01-16T15:25:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.862+0000 7f7b39600640 1 -- 172.21.15.43:0/1055066712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2402fdd0 con 0x7f7b3c06dc30 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 -- 172.21.15.43:0/1055066712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c06dc30 msgr2=0x7f7b3c0ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 --2- 172.21.15.43:0/1055066712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c06dc30 0x7f7b3c0ae490 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c067e70 tx=0x7f7b2402db20 comp rx=0 tx=0).stop 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 -- 172.21.15.43:0/1055066712 shutdown_connections 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 --2- 172.21.15.43:0/1055066712 >> v2:172.21.15.69:3300/0 conn(0x7f7b3c070490 0x7f7b3c070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 --2- 172.21.15.43:0/1055066712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c06dc30 0x7f7b3c0ae490 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 -- 172.21.15.43:0/1055066712 >> 172.21.15.43:0/1055066712 conn(0x7f7b3c0b0ba0 msgr2=0x7f7b3c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 -- 172.21.15.43:0/1055066712 shutdown_connections 2024-01-16T15:25:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.863+0000 7f7b41600640 1 -- 172.21.15.43:0/1055066712 wait complete. 2024-01-16T15:25:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.865+0000 7f7b41600640 1 Processor -- start 2024-01-16T15:25:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.865+0000 7f7b41600640 1 -- start start 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3c06dc30 0x7f7b3c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b3c070490 0x7f7b3c0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c0c87a0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1586e0 con 0x7f7b3c070490 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b41600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1589c0 con 0x7f7b3c06dc30 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b3b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b3b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56668/0 (socket says 172.21.15.43:56668) 2024-01-16T15:25:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.866+0000 7f7b3b400640 1 -- 172.21.15.43:0/219921671 learned_addr learned my addr 172.21.15.43:0/219921671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3a000640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b3c070490 0x7f7b3c0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3aa00640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3c06dc30 0x7f7b3c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 -- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3c06dc30 msgr2=0x7f7b3c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3c06dc30 0x7f7b3c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 -- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b3c070490 msgr2=0x7f7b3c0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b3c070490 0x7f7b3c0c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 -- 172.21.15.43:0/219921671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240094d0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b3b400640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400b970 tx=0x7f7b3400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400cb40 con 0x7f7b3c0acd70 2024-01-16T15:25:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3400ccc0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34014680 con 0x7f7b3c0acd70 2024-01-16T15:25:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.867+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c158ca0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.868+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c159260 con 0x7f7b3c0acd70 2024-01-16T15:25:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.869+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c09d200 con 0x7f7b3c0acd70 2024-01-16T15:25:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.870+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7b3401d3b0 con 0x7f7b3c0acd70 2024-01-16T15:25:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.871+0000 7f7b32a00640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 0x7f7b1406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.871+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7b34088b90 con 0x7f7b3c0acd70 2024-01-16T15:25:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.871+0000 7f7b3aa00640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 0x7f7b1406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.872+0000 7f7b3aa00640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 0x7f7b1406ea40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c004ea0 tx=0x7f7b2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:08.875+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b34055040 con 0x7f7b3c0acd70 2024-01-16T15:25:09.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.094+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b3c072670 con 0x7f7b1406c580 2024-01-16T15:25:09.100 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.103+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.100 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.103+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.100 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.103+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.100 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.103+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.100 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.103+0000 7fd4e5000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.131+0000 7f7b32a00640 1 -- 172.21.15.43:0/219921671 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7f7b3c072670 con 0x7f7b1406c580 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42753" 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sv1" 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42751" 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42755" 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42754" 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42757" 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42756" 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subvol_clone42752" 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:25:09.131 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 msgr2=0x7f7b1406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 0x7f7b1406ea40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c004ea0 tx=0x7f7b2c009040 comp rx=0 tx=0).stop 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 msgr2=0x7f7b3c145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400b970 tx=0x7f7b3400be30 comp rx=0 tx=0).stop 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 shutdown_connections 2024-01-16T15:25:09.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7b1406c580 0x7f7b1406ea40 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b3c070490 0x7f7b3c0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b3c0acd70 0x7f7b3c145980 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 --2- 172.21.15.43:0/219921671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b3c06dc30 0x7f7b3c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 >> 172.21.15.43:0/219921671 conn(0x7f7b3c0b0ba0 msgr2=0x7f7b3c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 shutdown_connections 2024-01-16T15:25:09.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.134+0000 7f7b41600640 1 -- 172.21.15.43:0/219921671 wait complete. 2024-01-16T15:25:09.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone42753 2024-01-16T15:25:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.354+0000 7f0fb0400640 1 Processor -- start 2024-01-16T15:25:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.354+0000 7f0fb0400640 1 -- start start 2024-01-16T15:25:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fb0400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0fa80b48d0 0x7f0fa80b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fb0400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0fa80b3210 0x7f0fa8167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fb0400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa80b23e0 con 0x7f0fa8160bd0 2024-01-16T15:25:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fb0400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa80cdc70 con 0x7f0fa80b48d0 2024-01-16T15:25:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fb0400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa80b2870 con 0x7f0fa80b3210 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.356+0000 7f0fad000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0fa80b48d0 0x7f0fa80b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fada00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0fa8160bd0 0x7f0fa80b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:50370/0 (socket says 172.21.15.43:50370) 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fada00640 1 -- 172.21.15.43:0/4229065196 learned_addr learned my addr 172.21.15.43:0/4229065196 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fae400640 1 --2- 172.21.15.43:0/4229065196 >> v2:172.21.15.69:3300/0 conn(0x7f0fa80b3210 0x7f0fa8167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fad000640 1 -- 172.21.15.43:0/4229065196 >> v2:172.21.15.69:3300/0 conn(0x7f0fa80b3210 msgr2=0x7f0fa8167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fad000640 1 --2- 172.21.15.43:0/4229065196 >> v2:172.21.15.69:3300/0 conn(0x7f0fa80b3210 0x7f0fa8167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fad000640 1 -- 172.21.15.43:0/4229065196 >> v1:172.21.15.69:6789/0 conn(0x7f0fa8160bd0 legacy=0x7f0fa80b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.357+0000 7f0fad000640 1 -- 172.21.15.43:0/4229065196 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa80d41b0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.359+0000 7f0fad000640 1 --2- 172.21.15.43:0/4229065196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa80b1ea0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80d2e10 tx=0x7f0f9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678774153d8fff82 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.359+0000 7f0fa7e00640 1 -- 172.21.15.43:0/4229065196 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9002cbf0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.359+0000 7f0fa7e00640 1 -- 172.21.15.43:0/4229065196 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9002cd50 con 0x7f0fa80b48d0 2024-01-16T15:25:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.359+0000 7f0fa7e00640 1 -- 172.21.15.43:0/4229065196 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90034660 con 0x7f0fa80b48d0 2024-01-16T15:25:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 -- 172.21.15.43:0/4229065196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 msgr2=0x7f0fa80b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 --2- 172.21.15.43:0/4229065196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa80b1ea0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80d2e10 tx=0x7f0f9002be40 comp rx=0 tx=0).stop 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 -- 172.21.15.43:0/4229065196 shutdown_connections 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 --2- 172.21.15.43:0/4229065196 >> v2:172.21.15.69:3300/0 conn(0x7f0fa80b3210 0x7f0fa8167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 --2- 172.21.15.43:0/4229065196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa80b1ea0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 -- 172.21.15.43:0/4229065196 >> 172.21.15.43:0/4229065196 conn(0x7f0fa809c5a0 msgr2=0x7f0fa80d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 -- 172.21.15.43:0/4229065196 shutdown_connections 2024-01-16T15:25:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.360+0000 7f0fb0400640 1 -- 172.21.15.43:0/4229065196 wait complete. 2024-01-16T15:25:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.362+0000 7f0fb0400640 1 Processor -- start 2024-01-16T15:25:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.362+0000 7f0fb0400640 1 -- start start 2024-01-16T15:25:09.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa80b3210 0x7f0fa80c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa8160bd0 0x7f0fa80b8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c8fa0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c8110 con 0x7f0fa80b3210 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fb0400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa8155d40 con 0x7f0fa8160bd0 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fad000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fad000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56692/0 (socket says 172.21.15.43:56692) 2024-01-16T15:25:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.363+0000 7f0fad000640 1 -- 172.21.15.43:0/2829020656 learned_addr learned my addr 172.21.15.43:0/2829020656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:09.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fae400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa8160bd0 0x7f0fa80b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fada00640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa80b3210 0x7f0fa80c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 -- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa8160bd0 msgr2=0x7f0fa80b8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa8160bd0 0x7f0fa80b8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 -- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa80b3210 msgr2=0x7f0fa80c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa80b3210 0x7f0fa80c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 -- 172.21.15.43:0/2829020656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f90009a70 con 0x7f0fa80b48d0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.364+0000 7f0fad000640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0f90037040 tx=0x7f0f90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.365+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90040650 con 0x7f0fa80b48d0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.365+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f900407b0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.365+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90040b60 con 0x7f0fa80b48d0 2024-01-16T15:25:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.365+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa80c8320 con 0x7f0fa80b48d0 2024-01-16T15:25:09.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.365+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa80c88e0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.366+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa80a1210 con 0x7f0fa80b48d0 2024-01-16T15:25:09.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.367+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0f90040cc0 con 0x7f0fa80b48d0 2024-01-16T15:25:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.368+0000 7f0fa5600640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 0x7f0f8006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.368+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0f900aba30 con 0x7f0fa80b48d0 2024-01-16T15:25:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.368+0000 7f0fada00640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 0x7f0f8006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.369+0000 7f0fada00640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 0x7f0f8006ea10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005880 tx=0x7f0fa0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.372+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f90077f70 con 0x7f0fa80b48d0 2024-01-16T15:25:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.600+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone42753", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fa80b3760 con 0x7f0f8006c550 2024-01-16T15:25:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.655+0000 7f0fa5600640 1 -- 172.21.15.43:0/2829020656 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0fa00058b0 con 0x7f0f8006c550 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 msgr2=0x7f0f8006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 0x7f0f8006ea10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005880 tx=0x7f0fa0009040 comp rx=0 tx=0).stop 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 msgr2=0x7f0fa8155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0f90037040 tx=0x7f0f90004a40 comp rx=0 tx=0).stop 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 shutdown_connections 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0f8006c550 0x7f0f8006ea10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fa8160bd0 0x7f0fa80b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fa80b48d0 0x7f0fa8155800 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 --2- 172.21.15.43:0/2829020656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fa80b3210 0x7f0fa80c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 >> 172.21.15.43:0/2829020656 conn(0x7f0fa809c5a0 msgr2=0x7f0fa80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:09.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 shutdown_connections 2024-01-16T15:25:09.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.658+0000 7f0fb0400640 1 -- 172.21.15.43:0/2829020656 wait complete. 2024-01-16T15:25:09.660 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.662+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.660 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.662+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.660 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.662+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.660 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.662+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.660 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.662+0000 7fd4e8c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:25:09.690 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.693+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.690 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.693+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.690 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.693+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.690 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.693+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.690 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:09.693+0000 7fd4e6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.878+0000 7f9e8a000640 1 Processor -- start 2024-01-16T15:25:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.879+0000 7f9e8a000640 1 -- start start 2024-01-16T15:25:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.880+0000 7f9e8a000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e8415b350 0x7f9e840abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.880+0000 7f9e8a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e8409cb30 0x7f9e840b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.880+0000 7f9e8a000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8415b730 con 0x7f9e840adb60 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.880+0000 7f9e8a000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e840ac740 con 0x7f9e8415b350 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.880+0000 7f9e8a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e840ac460 con 0x7f9e8409cb30 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e8415b350 0x7f9e840abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9e8415b350 0x7f9e840abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56750/0 (socket says 172.21.15.43:56750) 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 -- 172.21.15.43:0/3232360268 learned_addr learned my addr 172.21.15.43:0/3232360268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e82a00640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e8409cb30 0x7f9e840b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 -- 172.21.15.43:0/3232360268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e8409cb30 msgr2=0x7f9e840b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e8409cb30 0x7f9e840b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 -- 172.21.15.43:0/3232360268 >> v1:172.21.15.69:6789/0 conn(0x7f9e840adb60 legacy=0x7f9e840adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.881+0000 7f9e83400640 1 -- 172.21.15.43:0/3232360268 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840b2a50 con 0x7f9e8415b350 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.882+0000 7f9e83400640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e840abe80 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009870 tx=0x7f9e7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a4ad4f504c2d8c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.883+0000 7f9e82000640 1 -- 172.21.15.43:0/3232360268 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74031640 con 0x7f9e8415b350 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.883+0000 7f9e82000640 1 -- 172.21.15.43:0/3232360268 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e740317a0 con 0x7f9e8415b350 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.883+0000 7f9e82000640 1 -- 172.21.15.43:0/3232360268 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74031b30 con 0x7f9e8415b350 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.883+0000 7f9e8a000640 1 -- 172.21.15.43:0/3232360268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 msgr2=0x7f9e840abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.883+0000 7f9e8a000640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e840abe80 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009870 tx=0x7f9e7402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 -- 172.21.15.43:0/3232360268 shutdown_connections 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e8409cb30 0x7f9e840b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 --2- 172.21.15.43:0/3232360268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e840abe80 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 -- 172.21.15.43:0/3232360268 >> 172.21.15.43:0/3232360268 conn(0x7f9e840b0ba0 msgr2=0x7f9e840b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 -- 172.21.15.43:0/3232360268 shutdown_connections 2024-01-16T15:25:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.884+0000 7f9e8a000640 1 -- 172.21.15.43:0/3232360268 wait complete. 2024-01-16T15:25:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.885+0000 7f9e8a000640 1 Processor -- start 2024-01-16T15:25:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.886+0000 7f9e8a000640 1 -- start start 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 0x7f9e84146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e840adb60 0x7f9e8414eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841472b0 con 0x7f9e8415b350 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841f6810 con 0x7f9e8409cb30 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e8a000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8414e240 con 0x7f9e840adb60 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56760/0 (socket says 172.21.15.43:56760) 2024-01-16T15:25:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e83e00640 1 -- 172.21.15.43:0/946837153 learned_addr learned my addr 172.21.15.43:0/946837153 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e83400640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 0x7f9e84146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.887+0000 7f9e82a00640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e840adb60 0x7f9e8414eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 -- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e840adb60 msgr2=0x7f9e8414eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e840adb60 0x7f9e8414eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 -- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 msgr2=0x7f9e84146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 0x7f9e84146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 -- 172.21.15.43:0/946837153 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740094d0 con 0x7f9e8415b350 2024-01-16T15:25:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83400640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 0x7f9e84146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e83e00640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00c780 tx=0x7f9e7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c004b60 con 0x7f9e8415b350 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c00ce90 con 0x7f9e8415b350 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c00a850 con 0x7f9e8415b350 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8414e520 con 0x7f9e8415b350 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.888+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8414e800 con 0x7f9e8415b350 2024-01-16T15:25:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.890+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e841f69d0 con 0x7f9e8415b350 2024-01-16T15:25:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.891+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9e7c00a9b0 con 0x7f9e8415b350 2024-01-16T15:25:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.892+0000 7f9e7b400640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 0x7f9e5c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.892+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9e7c0889f0 con 0x7f9e8415b350 2024-01-16T15:25:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.892+0000 7f9e83400640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 0x7f9e5c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.893+0000 7f9e83400640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 0x7f9e5c06ea10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9e74036000 tx=0x7f9e7403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:09.896+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e7c0552b0 con 0x7f9e8415b350 2024-01-16T15:25:10.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.128+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e840b3a70 con 0x7f9e5c06c550 2024-01-16T15:25:10.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.160+0000 7f9e7b400640 1 -- 172.21.15.43:0/946837153 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9e74031b00 con 0x7f9e5c06c550 2024-01-16T15:25:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:10.158 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-01-16T15:25:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-01-16T15:25:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ss1" 2024-01-16T15:25:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-01-16T15:25:10.159 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-01-16T15:25:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 msgr2=0x7f9e5c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 0x7f9e5c06ea10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9e74036000 tx=0x7f9e7403c040 comp rx=0 tx=0).stop 2024-01-16T15:25:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 msgr2=0x7f9e841f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00c780 tx=0x7f9e7c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 shutdown_connections 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9e5c06c550 0x7f9e5c06ea10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e8415b350 0x7f9e841f3180 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e840adb60 0x7f9e8414eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 --2- 172.21.15.43:0/946837153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e8409cb30 0x7f9e84146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 >> 172.21.15.43:0/946837153 conn(0x7f9e840b0ba0 msgr2=0x7f9e84070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 shutdown_connections 2024-01-16T15:25:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.164+0000 7f9e8a000640 1 -- 172.21.15.43:0/946837153 wait complete. 2024-01-16T15:25:10.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:10.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.458+0000 7fe83d000640 1 Processor -- start 2024-01-16T15:25:10.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.458+0000 7fe83d000640 1 -- start start 2024-01-16T15:25:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8380acd70 0x7fe83806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8380ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8380ad690 0x7fe838070490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83815a5b0 con 0x7fe8380acd70 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83806d7a0 con 0x7fe8380ad690 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe83d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83806ce70 con 0x7fe8380ae490 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe835e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8380acd70 0x7fe83806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe835e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8380acd70 0x7fe83806dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56774/0 (socket says 172.21.15.43:56774) 2024-01-16T15:25:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.460+0000 7fe835e00640 1 -- 172.21.15.43:0/3537339019 learned_addr learned my addr 172.21.15.43:0/3537339019 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe836800640 1 --2- 172.21.15.43:0/3537339019 >> v2:172.21.15.69:3300/0 conn(0x7fe8380ad690 0x7fe838070490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe82fe00640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8380ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe835e00640 1 -- 172.21.15.43:0/3537339019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 msgr2=0x7fe8380ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe835e00640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8380ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe835e00640 1 -- 172.21.15.43:0/3537339019 >> v2:172.21.15.69:3300/0 conn(0x7fe8380ad690 msgr2=0x7fe838070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe835e00640 1 --2- 172.21.15.43:0/3537339019 >> v2:172.21.15.69:3300/0 conn(0x7fe8380ad690 0x7fe838070490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe835e00640 1 -- 172.21.15.43:0/3537339019 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83806d150 con 0x7fe8380acd70 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.461+0000 7fe82fe00640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8380ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:10.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.462+0000 7fe835e00640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe83806dc30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe838072660 tx=0x7fe82802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4ef19b1e7738d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:10.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.462+0000 7fe835400640 1 -- 172.21.15.43:0/3537339019 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828031860 con 0x7fe8380acd70 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.462+0000 7fe835400640 1 -- 172.21.15.43:0/3537339019 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8280319c0 con 0x7fe8380acd70 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.462+0000 7fe835400640 1 -- 172.21.15.43:0/3537339019 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828031d50 con 0x7fe8380acd70 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 -- 172.21.15.43:0/3537339019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 msgr2=0x7fe83806dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe83806dc30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe838072660 tx=0x7fe82802b900 comp rx=0 tx=0).stop 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 -- 172.21.15.43:0/3537339019 shutdown_connections 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 --2- 172.21.15.43:0/3537339019 >> v2:172.21.15.69:3300/0 conn(0x7fe8380ad690 0x7fe838070490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8380ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 --2- 172.21.15.43:0/3537339019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe83806dc30 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 -- 172.21.15.43:0/3537339019 >> 172.21.15.43:0/3537339019 conn(0x7fe8380b0ba0 msgr2=0x7fe8380b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 -- 172.21.15.43:0/3537339019 shutdown_connections 2024-01-16T15:25:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.463+0000 7fe83d000640 1 -- 172.21.15.43:0/3537339019 wait complete. 2024-01-16T15:25:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.465+0000 7fe83d000640 1 Processor -- start 2024-01-16T15:25:10.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.465+0000 7fe83d000640 1 -- start start 2024-01-16T15:25:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe8380cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8380ad690 0x7fe8381ec9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8381ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380c8260 con 0x7fe8380acd70 2024-01-16T15:25:10.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380cbe30 con 0x7fe8380ad690 2024-01-16T15:25:10.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe83d000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380cc110 con 0x7fe8380ae490 2024-01-16T15:25:10.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe835e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe8380cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe835e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe8380cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56786/0 (socket says 172.21.15.43:56786) 2024-01-16T15:25:10.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.466+0000 7fe835e00640 1 -- 172.21.15.43:0/1879736478 learned_addr learned my addr 172.21.15.43:0/1879736478 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe82fe00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8380ad690 0x7fe8381ec9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 -- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 msgr2=0x7fe8381ecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8380ae490 0x7fe8381ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 -- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8380ad690 msgr2=0x7fe8381ec9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8380ad690 0x7fe8381ec9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 -- 172.21.15.43:0/1879736478 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8280094d0 con 0x7fe8380acd70 2024-01-16T15:25:10.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe82fe00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8380ad690 0x7fe8381ec9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe835e00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8380acd70 0x7fe8380cb550 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe82802b940 tx=0x7fe8280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82803ad30 con 0x7fe8380acd70 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8280443f0 con 0x7fe8380acd70 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8280447a0 con 0x7fe8380acd70 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.467+0000 7fe83d000640 1 -- 172.21.15.43:0/1879736478 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8380cc3f0 con 0x7fe8380acd70 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.468+0000 7fe83d000640 1 -- 172.21.15.43:0/1879736478 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8380cba90 con 0x7fe8380acd70 2024-01-16T15:25:10.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.469+0000 7fe83d000640 1 -- 172.21.15.43:0/1879736478 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83809d200 con 0x7fe8380acd70 2024-01-16T15:25:10.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.470+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 50) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe82804d450 con 0x7fe8380acd70 2024-01-16T15:25:10.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.470+0000 7fe82e000640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe81006c580 0x7fe81006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:10.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.471+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe8280aad20 con 0x7fe8380acd70 2024-01-16T15:25:10.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.471+0000 7fe82fe00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe81006c580 0x7fe81006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:10.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.472+0000 7fe82fe00640 1 --2- 172.21.15.43:0/1879736478 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe81006c580 0x7fe81006ea40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe820006fd0 tx=0x7fe820008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:10.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.475+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8280775e0 con 0x7fe8380acd70 2024-01-16T15:25:10.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.546+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fe8280779b0 con 0x7fe8380acd70 2024-01-16T15:25:10.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.707+0000 7fe83d000640 1 -- 172.21.15.43:0/1879736478 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8380b3d90 con 0x7fe81006c580 2024-01-16T15:25:10.717 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:10.719+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:10.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:10.720+0000 7fe82e000640 1 -- 172.21.15.43:0/1879736478 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe82000c040 con 0x7fe81006c580 2024-01-16T15:25:10.717 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:10.717 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:12.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.063+0000 7f6434000640 1 Processor -- start 2024-01-16T15:25:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.063+0000 7f6434000640 1 -- start start 2024-01-16T15:25:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6434000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f642c09cb30 0x7f642c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6434000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f642c0adb60 0x7f642c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6434000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c15b730 con 0x7f642c15b350 2024-01-16T15:25:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6434000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0ac740 con 0x7f642c09cb30 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6434000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0ac460 con 0x7f642c0adb60 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6430c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f642c09cb30 0x7f642c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6431600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f642c15b350 0x7f642c0b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:59384/0 (socket says 172.21.15.43:59384) 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.064+0000 7f6431600640 1 -- 172.21.15.43:0/3918742493 learned_addr learned my addr 172.21.15.43:0/3918742493 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f6432000640 1 --2- 172.21.15.43:0/3918742493 >> v2:172.21.15.69:3300/0 conn(0x7f642c0adb60 0x7f642c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f6430c00640 1 -- 172.21.15.43:0/3918742493 >> v2:172.21.15.69:3300/0 conn(0x7f642c0adb60 msgr2=0x7f642c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f6430c00640 1 --2- 172.21.15.43:0/3918742493 >> v2:172.21.15.69:3300/0 conn(0x7f642c0adb60 0x7f642c0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f642be00640 1 -- 172.21.15.43:0/3918742493 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235625860 0 0) 0x7f641c001110 con 0x7f642c15b350 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f6430c00640 1 -- 172.21.15.43:0/3918742493 >> v1:172.21.15.69:6789/0 conn(0x7f642c15b350 legacy=0x7f642c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.065+0000 7f6430c00640 1 -- 172.21.15.43:0/3918742493 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0b2a50 con 0x7f642c09cb30 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.066+0000 7f6430c00640 1 --2- 172.21.15.43:0/3918742493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c09cb30 0x7f642c0abe80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f642c06dac0 tx=0x7f641402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff94893de32e3e38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.066+0000 7f642be00640 1 -- 172.21.15.43:0/3918742493 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414031830 con 0x7f642c09cb30 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.066+0000 7f642be00640 1 -- 172.21.15.43:0/3918742493 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6414031990 con 0x7f642c09cb30 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.066+0000 7f642be00640 1 -- 172.21.15.43:0/3918742493 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414031d20 con 0x7f642c09cb30 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 -- 172.21.15.43:0/3918742493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c09cb30 msgr2=0x7f642c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 --2- 172.21.15.43:0/3918742493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c09cb30 0x7f642c0abe80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f642c06dac0 tx=0x7f641402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 -- 172.21.15.43:0/3918742493 shutdown_connections 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 --2- 172.21.15.43:0/3918742493 >> v2:172.21.15.69:3300/0 conn(0x7f642c0adb60 0x7f642c0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 --2- 172.21.15.43:0/3918742493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c09cb30 0x7f642c0abe80 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 -- 172.21.15.43:0/3918742493 >> 172.21.15.43:0/3918742493 conn(0x7f642c0b0ba0 msgr2=0x7f642c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:13.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 -- 172.21.15.43:0/3918742493 shutdown_connections 2024-01-16T15:25:13.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.067+0000 7f6434000640 1 -- 172.21.15.43:0/3918742493 wait complete. 2024-01-16T15:25:13.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.069+0000 7f6434000640 1 Processor -- start 2024-01-16T15:25:13.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.069+0000 7f6434000640 1 -- start start 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f642c09cb30 0x7f642c148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c0adb60 0x7f642c09aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f642c15b350 0x7f642c09af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c152e50 con 0x7f642c0adb60 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1530a0 con 0x7f642c09cb30 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6434000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c153380 con 0x7f642c15b350 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6430c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c0adb60 0x7f642c09aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6430c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c0adb60 0x7f642c09aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52720/0 (socket says 172.21.15.43:52720) 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.070+0000 7f6430c00640 1 -- 172.21.15.43:0/2135066753 learned_addr learned my addr 172.21.15.43:0/2135066753 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6432000640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f642c15b350 0x7f642c09af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6431600640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f642c09cb30 0x7f642c148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 -- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f642c15b350 msgr2=0x7f642c09af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f642c15b350 0x7f642c09af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 -- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f642c09cb30 msgr2=0x7f642c148ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f642c09cb30 0x7f642c148ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 -- 172.21.15.43:0/2135066753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414009400 con 0x7f642c0adb60 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.071+0000 7f6430c00640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f642c0adb60 0x7f642c09aa50 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f6414009890 tx=0x7f64140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.072+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641403a6f0 con 0x7f642c0adb60 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.072+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641403a870 con 0x7f642c0adb60 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.072+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641403ac20 con 0x7f642c0adb60 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.072+0000 7f6434000640 1 -- 172.21.15.43:0/2135066753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c153660 con 0x7f642c0adb60 2024-01-16T15:25:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.072+0000 7f6434000640 1 -- 172.21.15.43:0/2135066753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c149000 con 0x7f642c0adb60 2024-01-16T15:25:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.073+0000 7f6434000640 1 -- 172.21.15.43:0/2135066753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c153b80 con 0x7f642c0adb60 2024-01-16T15:25:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.074+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f6414043410 con 0x7f642c0adb60 2024-01-16T15:25:13.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.075+0000 7f6429600640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f640406c700 0x7f640406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:13.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.075+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f64140aa900 con 0x7f642c0adb60 2024-01-16T15:25:13.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.075+0000 7f6431600640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f640406c700 0x7f640406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.076+0000 7f6431600640 1 --2- 172.21.15.43:0/2135066753 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f640406c700 0x7f640406ebc0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f641c000f60 tx=0x7f641c007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:13.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.080+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f642c153b80 con 0x7f642c0adb60 2024-01-16T15:25:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.307+0000 7f6434000640 1 -- 172.21.15.43:0/2135066753 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f642c153210 con 0x7f640406c700 2024-01-16T15:25:13.321 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:13.324+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:13.324+0000 7f6429600640 1 -- 172.21.15.43:0/2135066753 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f642c153210 con 0x7f640406c700 2024-01-16T15:25:13.322 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:15.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:15.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.580+0000 7f3bc2000640 1 Processor -- start 2024-01-16T15:25:15.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.580+0000 7f3bc2000640 1 -- start start 2024-01-16T15:25:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bc2000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3bbc0adb60 0x7f3bbc0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bc2000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc15b730 con 0x7f3bbc15b350 2024-01-16T15:25:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bc2000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc0ac740 con 0x7f3bbc0adb60 2024-01-16T15:25:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bc2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0ac460 con 0x7f3bbc09cb30 2024-01-16T15:25:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.582+0000 7f3bbaa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:50398/0 (socket says 172.21.15.43:50398) 2024-01-16T15:25:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 -- 172.21.15.43:0/2750598236 learned_addr learned my addr 172.21.15.43:0/2750598236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:15.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbbe00640 1 --2- 172.21.15.43:0/2750598236 >> v2:172.21.15.69:3300/0 conn(0x7f3bbc0adb60 0x7f3bbc0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 -- 172.21.15.43:0/2750598236 >> v2:172.21.15.69:3300/0 conn(0x7f3bbc0adb60 msgr2=0x7f3bbc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 --2- 172.21.15.43:0/2750598236 >> v2:172.21.15.69:3300/0 conn(0x7f3bbc0adb60 0x7f3bbc0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 -- 172.21.15.43:0/2750598236 >> v1:172.21.15.69:6789/0 conn(0x7f3bbc15b350 legacy=0x7f3bbc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:15.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.583+0000 7f3bbaa00640 1 -- 172.21.15.43:0/2750598236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0b2a50 con 0x7f3bbc09cb30 2024-01-16T15:25:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.585+0000 7f3bbaa00640 1 --2- 172.21.15.43:0/2750598236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89c0261b253a0e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.586+0000 7f3bba000640 1 -- 172.21.15.43:0/2750598236 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba402cb00 con 0x7f3bbc09cb30 2024-01-16T15:25:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.586+0000 7f3bba000640 1 -- 172.21.15.43:0/2750598236 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba402cc60 con 0x7f3bbc09cb30 2024-01-16T15:25:15.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.586+0000 7f3bba000640 1 -- 172.21.15.43:0/2750598236 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40343f0 con 0x7f3bbc09cb30 2024-01-16T15:25:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 -- 172.21.15.43:0/2750598236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 msgr2=0x7f3bbc0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 --2- 172.21.15.43:0/2750598236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba402bf60 comp rx=0 tx=0).stop 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 -- 172.21.15.43:0/2750598236 shutdown_connections 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 --2- 172.21.15.43:0/2750598236 >> v2:172.21.15.69:3300/0 conn(0x7f3bbc0adb60 0x7f3bbc0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 --2- 172.21.15.43:0/2750598236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc09cb30 0x7f3bbc0abe80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 -- 172.21.15.43:0/2750598236 >> 172.21.15.43:0/2750598236 conn(0x7f3bbc0b0ba0 msgr2=0x7f3bbc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 -- 172.21.15.43:0/2750598236 shutdown_connections 2024-01-16T15:25:15.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.587+0000 7f3bc2000640 1 -- 172.21.15.43:0/2750598236 wait complete. 2024-01-16T15:25:15.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.589+0000 7f3bc2000640 1 Processor -- start 2024-01-16T15:25:15.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.589+0000 7f3bc2000640 1 -- start start 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bbc09cb30 0x7f3bbc154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc0adb60 0x7f3bbc1f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bbc15b350 0x7f3bbc13ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc154ae0 con 0x7f3bbc15b350 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc154dc0 con 0x7f3bbc09cb30 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bc2000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc1550a0 con 0x7f3bbc0adb60 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bbc15b350 0x7f3bbc13ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bbc15b350 0x7f3bbc13ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52726/0 (socket says 172.21.15.43:52726) 2024-01-16T15:25:15.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.590+0000 7f3bbbe00640 1 -- 172.21.15.43:0/2212362356 learned_addr learned my addr 172.21.15.43:0/2212362356 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbaa00640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc0adb60 0x7f3bbc1f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbb400640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bbc09cb30 0x7f3bbc154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 -- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc0adb60 msgr2=0x7f3bbc1f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3bbc0adb60 0x7f3bbc1f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 -- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bbc09cb30 msgr2=0x7f3bbc154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bbc09cb30 0x7f3bbc154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:15.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 -- 172.21.15.43:0/2212362356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4009b30 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bbbe00640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bbc15b350 0x7f3bbc13ad50 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400e930 tx=0x7f3bb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4009690 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb400c980 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400cd30 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.591+0000 7f3bc2000640 1 -- 172.21.15.43:0/2212362356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc155380 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.592+0000 7f3bc2000640 1 -- 172.21.15.43:0/2212362356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc155b80 con 0x7f3bbc15b350 2024-01-16T15:25:15.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.593+0000 7f3bc2000640 1 -- 172.21.15.43:0/2212362356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc1558a0 con 0x7f3bbc15b350 2024-01-16T15:25:15.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.594+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f3bb4013470 con 0x7f3bbc15b350 2024-01-16T15:25:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.595+0000 7f3bb3400640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3b9406c700 0x7f3b9406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.595+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3bb4088e80 con 0x7f3bbc15b350 2024-01-16T15:25:15.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.595+0000 7f3bbb400640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3b9406c700 0x7f3b9406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:15.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.596+0000 7f3bbb400640 1 --2- 172.21.15.43:0/2212362356 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f3b9406c700 0x7f3b9406ebc0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3bac007fd0 tx=0x7f3bac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:15.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.600+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bb4055430 con 0x7f3bbc15b350 2024-01-16T15:25:15.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.831+0000 7f3bc2000640 1 -- 172.21.15.43:0/2212362356 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3bbc0ae4f0 con 0x7f3b9406c700 2024-01-16T15:25:15.844 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:15.846+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:15.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:15.846+0000 7f3bb3400640 1 -- 172.21.15.43:0/2212362356 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3bac001200 con 0x7f3b9406c700 2024-01-16T15:25:15.845 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:15.845 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:17.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:18.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.105+0000 7fe525000640 1 Processor -- start 2024-01-16T15:25:18.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.106+0000 7fe525000640 1 -- start start 2024-01-16T15:25:18.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe525000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe5200b20e0 0x7fe5200ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe525000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200b3f20 con 0x7fe520143bd0 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe525000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200b36f0 con 0x7fe5200b1c70 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe525000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5200cd820 con 0x7fe5200b20e0 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe51de00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe520143bd0 0x7fe5200b2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33666/0 (socket says 172.21.15.43:33666) 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe51de00640 1 -- 172.21.15.43:0/2061831903 learned_addr learned my addr 172.21.15.43:0/2061831903 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:18.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.107+0000 7fe51e800640 1 --2- 172.21.15.43:0/2061831903 >> v2:172.21.15.43:3300/0 conn(0x7fe5200b20e0 0x7fe5200ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:18.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.108+0000 7fe51d400640 1 -- 172.21.15.43:0/2061831903 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040436323 0 0) 0x7fe5200b3f20 con 0x7fe520143bd0 2024-01-16T15:25:18.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.108+0000 7fe51d400640 1 -- 172.21.15.43:0/2061831903 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe500005730 con 0x7fe520143bd0 2024-01-16T15:25:18.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.108+0000 7fe51e800640 1 -- 172.21.15.43:0/2061831903 >> v1:172.21.15.69:6789/0 conn(0x7fe5200b1c70 legacy=0x7fe5200ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:18.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.108+0000 7fe51e800640 1 -- 172.21.15.43:0/2061831903 >> v1:172.21.15.43:6789/0 conn(0x7fe520143bd0 legacy=0x7fe5200b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:18.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.108+0000 7fe51e800640 1 -- 172.21.15.43:0/2061831903 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5200b3960 con 0x7fe5200b20e0 2024-01-16T15:25:18.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.109+0000 7fe51e800640 1 --2- 172.21.15.43:0/2061831903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5200b20e0 0x7fe5200ce940 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfec99702d2604b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:18.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.109+0000 7fe51d400640 1 -- 172.21.15.43:0/2061831903 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518030610 con 0x7fe5200b20e0 2024-01-16T15:25:18.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.109+0000 7fe51d400640 1 -- 172.21.15.43:0/2061831903 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518030770 con 0x7fe5200b20e0 2024-01-16T15:25:18.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.109+0000 7fe51d400640 1 -- 172.21.15.43:0/2061831903 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518030b00 con 0x7fe5200b20e0 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 -- 172.21.15.43:0/2061831903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5200b20e0 msgr2=0x7fe5200ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 --2- 172.21.15.43:0/2061831903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5200b20e0 0x7fe5200ce940 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802b960 comp rx=0 tx=0).stop 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 -- 172.21.15.43:0/2061831903 shutdown_connections 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 --2- 172.21.15.43:0/2061831903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5200b20e0 0x7fe5200ce940 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 -- 172.21.15.43:0/2061831903 >> 172.21.15.43:0/2061831903 conn(0x7fe52006d5a0 msgr2=0x7fe52006d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 -- 172.21.15.43:0/2061831903 shutdown_connections 2024-01-16T15:25:18.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.110+0000 7fe525000640 1 -- 172.21.15.43:0/2061831903 wait complete. 2024-01-16T15:25:18.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.112+0000 7fe525000640 1 Processor -- start 2024-01-16T15:25:18.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.112+0000 7fe525000640 1 -- start start 2024-01-16T15:25:18.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5200b1c70 0x7fe5201f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:18.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5200b20e0 0x7fe52013d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:18.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe520143bd0 0x7fe5201f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:18.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5201f7510 con 0x7fe520143bd0 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5201f3ab0 con 0x7fe5200b1c70 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe525000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5201f3d90 con 0x7fe5200b20e0 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe51e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe520143bd0 0x7fe5201f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe51e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe520143bd0 0x7fe5201f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52740/0 (socket says 172.21.15.43:52740) 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.113+0000 7fe51e800640 1 -- 172.21.15.43:0/4088918798 learned_addr learned my addr 172.21.15.43:0/4088918798 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51de00640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5200b1c70 0x7fe5201f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe515200640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5200b20e0 0x7fe52013d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 -- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5200b20e0 msgr2=0x7fe52013d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5200b20e0 0x7fe52013d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 -- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5200b1c70 msgr2=0x7fe5201f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5200b1c70 0x7fe5201f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:18.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 -- 172.21.15.43:0/4088918798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518009400 con 0x7fe520143bd0 2024-01-16T15:25:18.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe51e800640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe520143bd0 0x7fe5201f3080 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fe5180098c0 tx=0x7fe518004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:18.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51803e040 con 0x7fe520143bd0 2024-01-16T15:25:18.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518039c10 con 0x7fe520143bd0 2024-01-16T15:25:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.114+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518030e10 con 0x7fe520143bd0 2024-01-16T15:25:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.115+0000 7fe525000640 1 -- 172.21.15.43:0/4088918798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5201f67d0 con 0x7fe520143bd0 2024-01-16T15:25:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.115+0000 7fe525000640 1 -- 172.21.15.43:0/4088918798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52013dbd0 con 0x7fe520143bd0 2024-01-16T15:25:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.116+0000 7fe525000640 1 -- 172.21.15.43:0/4088918798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5201419b0 con 0x7fe520143bd0 2024-01-16T15:25:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.117+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fe51804d3b0 con 0x7fe520143bd0 2024-01-16T15:25:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.118+0000 7fe516a00640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4f806c700 0x7fe4f806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.118+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe5180ab1d0 con 0x7fe520143bd0 2024-01-16T15:25:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.118+0000 7fe51de00640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4f806c700 0x7fe4f806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.119+0000 7fe51de00640 1 --2- 172.21.15.43:0/4088918798 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe4f806c700 0x7fe4f806ebc0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe510001560 tx=0x7fe510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:18.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.122+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5180776c0 con 0x7fe520143bd0 2024-01-16T15:25:18.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.348+0000 7fe525000640 1 -- 172.21.15.43:0/4088918798 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5201f6650 con 0x7fe4f806c700 2024-01-16T15:25:18.356 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:18.359+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:18.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:18.359+0000 7fe516a00640 1 -- 172.21.15.43:0/4088918798 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe510001930 con 0x7fe4f806c700 2024-01-16T15:25:18.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:18.357 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:20.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:20.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.620+0000 7f8ed8a00640 1 Processor -- start 2024-01-16T15:25:20.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.620+0000 7f8ed8a00640 1 -- start start 2024-01-16T15:25:20.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8ed415bda0 0x7f8ed4165ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:20.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed8a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed4071080 con 0x7f8ed406dae0 2024-01-16T15:25:20.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed8a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed415c1f0 con 0x7f8ed41448c0 2024-01-16T15:25:20.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed8a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed4144ca0 con 0x7f8ed415bda0 2024-01-16T15:25:20.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed3200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8ed415bda0 0x7f8ed4165ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:20.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed1e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8ed406dae0 0x7f8ed415b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33678/0 (socket says 172.21.15.43:33678) 2024-01-16T15:25:20.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.622+0000 7f8ed1e00640 1 -- 172.21.15.43:0/4051861054 learned_addr learned my addr 172.21.15.43:0/4051861054 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.623+0000 7f8ed1400640 1 -- 172.21.15.43:0/4051861054 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923111217 0 0) 0x7f8ed4071080 con 0x7f8ed406dae0 2024-01-16T15:25:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.623+0000 7f8ed1400640 1 -- 172.21.15.43:0/4051861054 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb4003520 con 0x7f8ed406dae0 2024-01-16T15:25:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.623+0000 7f8ed3200640 1 -- 172.21.15.43:0/4051861054 >> v1:172.21.15.69:6789/0 conn(0x7f8ed41448c0 legacy=0x7f8ed406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.623+0000 7f8ed3200640 1 -- 172.21.15.43:0/4051861054 >> v1:172.21.15.43:6789/0 conn(0x7f8ed406dae0 legacy=0x7f8ed415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:20.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.623+0000 7f8ed3200640 1 -- 172.21.15.43:0/4051861054 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed415c450 con 0x7f8ed415bda0 2024-01-16T15:25:20.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.624+0000 7f8ed3200640 1 --2- 172.21.15.43:0/4051861054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed415bda0 0x7f8ed4165ce0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f51061fc38ee97f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:20.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.624+0000 7f8ed1400640 1 -- 172.21.15.43:0/4051861054 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4030610 con 0x7f8ed415bda0 2024-01-16T15:25:20.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed1400640 1 -- 172.21.15.43:0/4051861054 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec4030770 con 0x7f8ed415bda0 2024-01-16T15:25:20.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed1400640 1 -- 172.21.15.43:0/4051861054 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4030b00 con 0x7f8ed415bda0 2024-01-16T15:25:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 -- 172.21.15.43:0/4051861054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed415bda0 msgr2=0x7f8ed4165ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 --2- 172.21.15.43:0/4051861054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed415bda0 0x7f8ed4165ce0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402b960 comp rx=0 tx=0).stop 2024-01-16T15:25:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 -- 172.21.15.43:0/4051861054 shutdown_connections 2024-01-16T15:25:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 --2- 172.21.15.43:0/4051861054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed415bda0 0x7f8ed4165ce0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:20.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 -- 172.21.15.43:0/4051861054 >> 172.21.15.43:0/4051861054 conn(0x7f8ed40b0aa0 msgr2=0x7f8ed40b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 -- 172.21.15.43:0/4051861054 shutdown_connections 2024-01-16T15:25:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.625+0000 7f8ed8a00640 1 -- 172.21.15.43:0/4051861054 wait complete. 2024-01-16T15:25:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.627+0000 7f8ed8a00640 1 Processor -- start 2024-01-16T15:25:20.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.627+0000 7f8ed8a00640 1 -- start start 2024-01-16T15:25:20.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed406dae0 0x7f8ed4158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:20.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed41448c0 0x7f8ed40b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed415bda0 0x7f8ed40b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40b1e50 con 0x7f8ed41448c0 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4158db0 con 0x7f8ed406dae0 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.628+0000 7f8ed8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40b2de0 con 0x7f8ed415bda0 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed41448c0 0x7f8ed40b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed41448c0 0x7f8ed40b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52758/0 (socket says 172.21.15.43:52758) 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 -- 172.21.15.43:0/1949785601 learned_addr learned my addr 172.21.15.43:0/1949785601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed2800640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed406dae0 0x7f8ed4158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed3200640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed415bda0 0x7f8ed40b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 -- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed415bda0 msgr2=0x7f8ed40b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed415bda0 0x7f8ed40b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 -- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed406dae0 msgr2=0x7f8ed4158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed406dae0 0x7f8ed4158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 -- 172.21.15.43:0/1949785601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec4009400 con 0x7f8ed41448c0 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed2800640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed406dae0 0x7f8ed4158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed3200640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ed415bda0 0x7f8ed40b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.629+0000 7f8ed1e00640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed41448c0 0x7f8ed40b28a0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c4d0 tx=0x7f8ebc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.630+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc007480 con 0x7f8ed41448c0 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.630+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc0075e0 con 0x7f8ed41448c0 2024-01-16T15:25:20.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.630+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0059c0 con 0x7f8ed41448c0 2024-01-16T15:25:20.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.630+0000 7f8ed8a00640 1 -- 172.21.15.43:0/1949785601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed4140720 con 0x7f8ed41448c0 2024-01-16T15:25:20.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.630+0000 7f8ed8a00640 1 -- 172.21.15.43:0/1949785601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed40b20a0 con 0x7f8ed41448c0 2024-01-16T15:25:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.631+0000 7f8ed8a00640 1 -- 172.21.15.43:0/1949785601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed404e840 con 0x7f8ed41448c0 2024-01-16T15:25:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.632+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f8ebc005b20 con 0x7f8ed41448c0 2024-01-16T15:25:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.633+0000 7f8ecaa00640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8eac06c700 0x7f8eac06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.633+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8ebc087ec0 con 0x7f8ed41448c0 2024-01-16T15:25:20.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.633+0000 7f8ed2800640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8eac06c700 0x7f8eac06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:20.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.634+0000 7f8ed2800640 1 --2- 172.21.15.43:0/1949785601 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8eac06c700 0x7f8eac06ebc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40cda70 tx=0x7f8ecc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.638+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ebc058210 con 0x7f8ed41448c0 2024-01-16T15:25:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.868+0000 7f8ed8a00640 1 -- 172.21.15.43:0/1949785601 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ed4071b70 con 0x7f8eac06c700 2024-01-16T15:25:20.873 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:20.876+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:20.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:20.876+0000 7f8ecaa00640 1 -- 172.21.15.43:0/1949785601 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8ecc003990 con 0x7f8eac06c700 2024-01-16T15:25:20.874 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:22.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.131+0000 7f386f400640 1 Processor -- start 2024-01-16T15:25:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.131+0000 7f386f400640 1 -- start start 2024-01-16T15:25:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38680abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386f400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f386809cb30 0x7f38680b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386815b730 con 0x7f38680adb60 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386f400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38680ac740 con 0x7f386809cb30 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38680ac460 con 0x7f386815b350 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f3867e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f386809cb30 0x7f38680b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f3867e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f386809cb30 0x7f38680b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39176/0 (socket says 172.21.15.43:39176) 2024-01-16T15:25:23.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f3867e00640 1 -- 172.21.15.43:0/772826672 learned_addr learned my addr 172.21.15.43:0/772826672 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.133+0000 7f386ca00640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38680abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.134+0000 7f3867e00640 1 -- 172.21.15.43:0/772826672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 msgr2=0x7f38680abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.134+0000 7f3867e00640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38680abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.134+0000 7f3867e00640 1 -- 172.21.15.43:0/772826672 >> v1:172.21.15.69:6789/0 conn(0x7f38680adb60 legacy=0x7f38680adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.134+0000 7f3867e00640 1 -- 172.21.15.43:0/772826672 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680b2a50 con 0x7f386809cb30 2024-01-16T15:25:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.135+0000 7f3867e00640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f386809cb30 0x7f38680b3530 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3850009870 tx=0x7f385002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd4899f37dddcd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.135+0000 7f3867400640 1 -- 172.21.15.43:0/772826672 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850031640 con 0x7f386809cb30 2024-01-16T15:25:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.135+0000 7f3867400640 1 -- 172.21.15.43:0/772826672 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38500317a0 con 0x7f386809cb30 2024-01-16T15:25:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.135+0000 7f3867400640 1 -- 172.21.15.43:0/772826672 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850031b30 con 0x7f386809cb30 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 -- 172.21.15.43:0/772826672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f386809cb30 msgr2=0x7f38680b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f386809cb30 0x7f38680b3530 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3850009870 tx=0x7f385002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 -- 172.21.15.43:0/772826672 shutdown_connections 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f386809cb30 0x7f38680b3530 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 --2- 172.21.15.43:0/772826672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38680abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 -- 172.21.15.43:0/772826672 >> 172.21.15.43:0/772826672 conn(0x7f38680b0ba0 msgr2=0x7f38680b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 -- 172.21.15.43:0/772826672 shutdown_connections 2024-01-16T15:25:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.136+0000 7f386f400640 1 -- 172.21.15.43:0/772826672 wait complete. 2024-01-16T15:25:23.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.138+0000 7f386f400640 1 Processor -- start 2024-01-16T15:25:23.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.138+0000 7f386f400640 1 -- start start 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f386809cb30 0x7f3868147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38680adb60 0x7f38681f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38681f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38681554a0 con 0x7f38680adb60 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868154890 con 0x7f386809cb30 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f386f400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3868154b70 con 0x7f386815b350 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f3867e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38680adb60 0x7f38681f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f3867e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38680adb60 0x7f38681f65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39188/0 (socket says 172.21.15.43:39188) 2024-01-16T15:25:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.139+0000 7f3867e00640 1 -- 172.21.15.43:0/1247987164 learned_addr learned my addr 172.21.15.43:0/1247987164 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f386ca00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f386809cb30 0x7f3868147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f386d400640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38681f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 -- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 msgr2=0x7f38681f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f386815b350 0x7f38681f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 -- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f386809cb30 msgr2=0x7f3868147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f386809cb30 0x7f3868147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 -- 172.21.15.43:0/1247987164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38500094d0 con 0x7f38680adb60 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.140+0000 7f3867e00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38680adb60 0x7f38681f65a0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3850009870 tx=0x7f38500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.141+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385003ac60 con 0x7f38680adb60 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.141+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385003adc0 con 0x7f38680adb60 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.141+0000 7f386f400640 1 -- 172.21.15.43:0/1247987164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3868154e50 con 0x7f38680adb60 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.141+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850044630 con 0x7f38680adb60 2024-01-16T15:25:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.141+0000 7f386f400640 1 -- 172.21.15.43:0/1247987164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38681483e0 con 0x7f38680adb60 2024-01-16T15:25:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.142+0000 7f386f400640 1 -- 172.21.15.43:0/1247987164 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868155610 con 0x7f38680adb60 2024-01-16T15:25:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.143+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f385002f050 con 0x7f38680adb60 2024-01-16T15:25:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.144+0000 7f3864c00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f384006c700 0x7f384006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.144+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f38500aa620 con 0x7f38680adb60 2024-01-16T15:25:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.144+0000 7f386ca00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f384006c700 0x7f384006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.145+0000 7f386ca00640 1 --2- 172.21.15.43:0/1247987164 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f384006c700 0x7f384006ebc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3858006fd0 tx=0x7f3858008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:23.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.148+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3850076ee0 con 0x7f38680adb60 2024-01-16T15:25:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.386+0000 7f386f400640 1 -- 172.21.15.43:0/1247987164 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38680b3a70 con 0x7f384006c700 2024-01-16T15:25:23.398 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:23.401+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:23.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:23.401+0000 7f3864c00640 1 -- 172.21.15.43:0/1247987164 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f385800f860 con 0x7f384006c700 2024-01-16T15:25:23.399 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:23.399 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:25.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.664+0000 7fb79e200640 1 Processor -- start 2024-01-16T15:25:25.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.664+0000 7fb79e200640 1 -- start start 2024-01-16T15:25:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.666+0000 7fb79e200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb79815d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.666+0000 7fb79e200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb798161470 0x7fb7981618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.666+0000 7fb79e200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79815def0 con 0x7fb7980b39f0 2024-01-16T15:25:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.666+0000 7fb79e200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7980b2020 con 0x7fb798161470 2024-01-16T15:25:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.666+0000 7fb79e200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb798161e00 con 0x7fb79815ad80 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb798161470 0x7fb7981618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb798161470 0x7fb7981618c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39194/0 (socket says 172.21.15.43:39194) 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 -- 172.21.15.43:0/2439928001 learned_addr learned my addr 172.21.15.43:0/2439928001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797400640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb79815d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 -- 172.21.15.43:0/2439928001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 msgr2=0x7fb79815d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb79815d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 -- 172.21.15.43:0/2439928001 >> v1:172.21.15.69:6789/0 conn(0x7fb7980b39f0 legacy=0x7fb79806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.667+0000 7fb797e00640 1 -- 172.21.15.43:0/2439928001 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79815e1d0 con 0x7fb798161470 2024-01-16T15:25:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.669+0000 7fb797e00640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb798161470 0x7fb7981618c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fb790009870 tx=0x7fb79002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca0e2653a5cd7dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.669+0000 7fb796000640 1 -- 172.21.15.43:0/2439928001 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790030840 con 0x7fb798161470 2024-01-16T15:25:25.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.669+0000 7fb796000640 1 -- 172.21.15.43:0/2439928001 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7900309a0 con 0x7fb798161470 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.669+0000 7fb796000640 1 -- 172.21.15.43:0/2439928001 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790030d30 con 0x7fb798161470 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 -- 172.21.15.43:0/2439928001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb798161470 msgr2=0x7fb7981618c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb798161470 0x7fb7981618c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fb790009870 tx=0x7fb79002b980 comp rx=0 tx=0).stop 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 -- 172.21.15.43:0/2439928001 shutdown_connections 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb798161470 0x7fb7981618c0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 --2- 172.21.15.43:0/2439928001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb79815d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 -- 172.21.15.43:0/2439928001 >> 172.21.15.43:0/2439928001 conn(0x7fb79806d5a0 msgr2=0x7fb79806d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 -- 172.21.15.43:0/2439928001 shutdown_connections 2024-01-16T15:25:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.670+0000 7fb79e200640 1 -- 172.21.15.43:0/2439928001 wait complete. 2024-01-16T15:25:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.672+0000 7fb79e200640 1 Processor -- start 2024-01-16T15:25:25.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.672+0000 7fb79e200640 1 -- start start 2024-01-16T15:25:25.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7980b39f0 0x7fb798159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb7980cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb798161470 0x7fb79815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980cf100 con 0x7fb7980b39f0 2024-01-16T15:25:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980ce580 con 0x7fb798161470 2024-01-16T15:25:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb79e200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980ce860 con 0x7fb79815ad80 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.673+0000 7fb797400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7980b39f0 0x7fb798159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb796a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb7980cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7980b39f0 0x7fb798159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39200/0 (socket says 172.21.15.43:39200) 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 -- 172.21.15.43:0/3757696630 learned_addr learned my addr 172.21.15.43:0/3757696630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797e00640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb798161470 0x7fb79815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 -- 172.21.15.43:0/3757696630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 msgr2=0x7fb7980cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79815ad80 0x7fb7980cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 -- 172.21.15.43:0/3757696630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb798161470 msgr2=0x7fb79815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb798161470 0x7fb79815bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 -- 172.21.15.43:0/3757696630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900094d0 con 0x7fb7980b39f0 2024-01-16T15:25:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.674+0000 7fb797400640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7980b39f0 0x7fb798159600 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fb788007ef0 tx=0x7fb78800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:25.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.675+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788014620 con 0x7fb7980b39f0 2024-01-16T15:25:25.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.675+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788014780 con 0x7fb7980b39f0 2024-01-16T15:25:25.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.675+0000 7fb79e200640 1 -- 172.21.15.43:0/3757696630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7980ceb40 con 0x7fb7980b39f0 2024-01-16T15:25:25.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.675+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788014b30 con 0x7fb7980b39f0 2024-01-16T15:25:25.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.675+0000 7fb79e200640 1 -- 172.21.15.43:0/3757696630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7981608f0 con 0x7fb7980b39f0 2024-01-16T15:25:25.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.676+0000 7fb79e200640 1 -- 172.21.15.43:0/3757696630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79806e5f0 con 0x7fb7980b39f0 2024-01-16T15:25:25.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.677+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fb788014c90 con 0x7fb7980b39f0 2024-01-16T15:25:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.678+0000 7fb78f400640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb77006c700 0x7fb77006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.678+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7880894d0 con 0x7fb7980b39f0 2024-01-16T15:25:25.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.678+0000 7fb796a00640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb77006c700 0x7fb77006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:25.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.679+0000 7fb796a00640 1 --2- 172.21.15.43:0/3757696630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb77006c700 0x7fb77006ebc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb798140a20 tx=0x7fb780009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.682+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb788055a70 con 0x7fb7980b39f0 2024-01-16T15:25:25.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.917+0000 7fb79e200640 1 -- 172.21.15.43:0/3757696630 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb798160f50 con 0x7fb77006c700 2024-01-16T15:25:25.927 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:25.930+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:25.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:25.930+0000 7fb78f400640 1 -- 172.21.15.43:0/3757696630 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb780001440 con 0x7fb77006c700 2024-01-16T15:25:25.928 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:25.928 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:27.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:28.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.185+0000 7f1e5d400640 1 Processor -- start 2024-01-16T15:25:28.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.185+0000 7f1e5d400640 1 -- start start 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e5d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1e5815aae0 0x7f1e5806cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e5d400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1e5806d2a0 0x7f1e580b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e5d400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e58157b30 con 0x7f1e580b2d70 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e5d400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5815aec0 con 0x7f1e5806d2a0 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e5d400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5806d730 con 0x7f1e5815aae0 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e55e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1e5806d2a0 0x7f1e580b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e55e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1e5806d2a0 0x7f1e580b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39214/0 (socket says 172.21.15.43:39214) 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e57200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1e580b2d70 0x7f1e5815d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:36090/0 (socket says 172.21.15.43:36090) 2024-01-16T15:25:28.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e55e00640 1 -- 172.21.15.43:0/2103646655 learned_addr learned my addr 172.21.15.43:0/2103646655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.187+0000 7f1e56800640 1 --2- 172.21.15.43:0/2103646655 >> v2:172.21.15.69:3300/0 conn(0x7f1e5815aae0 0x7f1e5806cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.188+0000 7f1e55e00640 1 -- 172.21.15.43:0/2103646655 >> v2:172.21.15.69:3300/0 conn(0x7f1e5815aae0 msgr2=0x7f1e5806cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.188+0000 7f1e55e00640 1 --2- 172.21.15.43:0/2103646655 >> v2:172.21.15.69:3300/0 conn(0x7f1e5815aae0 0x7f1e5806cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.188+0000 7f1e55e00640 1 -- 172.21.15.43:0/2103646655 >> v1:172.21.15.43:6789/0 conn(0x7f1e580b2d70 legacy=0x7f1e5815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.188+0000 7f1e55e00640 1 -- 172.21.15.43:0/2103646655 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580b34f0 con 0x7f1e5806d2a0 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.189+0000 7f1e55e00640 1 --2- 172.21.15.43:0/2103646655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e5806d2a0 0x7f1e580b2830 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f1e580d0100 tx=0x7f1e4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb0d5635c5100fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:28.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.189+0000 7f1e55400640 1 -- 172.21.15.43:0/2103646655 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40030860 con 0x7f1e5806d2a0 2024-01-16T15:25:28.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.189+0000 7f1e55400640 1 -- 172.21.15.43:0/2103646655 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e400309c0 con 0x7f1e5806d2a0 2024-01-16T15:25:28.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.189+0000 7f1e55400640 1 -- 172.21.15.43:0/2103646655 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40030d50 con 0x7f1e5806d2a0 2024-01-16T15:25:28.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 -- 172.21.15.43:0/2103646655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e5806d2a0 msgr2=0x7f1e580b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 --2- 172.21.15.43:0/2103646655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e5806d2a0 0x7f1e580b2830 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f1e580d0100 tx=0x7f1e4002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 -- 172.21.15.43:0/2103646655 shutdown_connections 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 --2- 172.21.15.43:0/2103646655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e5806d2a0 0x7f1e580b2830 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 --2- 172.21.15.43:0/2103646655 >> v2:172.21.15.69:3300/0 conn(0x7f1e5815aae0 0x7f1e5806cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 -- 172.21.15.43:0/2103646655 >> 172.21.15.43:0/2103646655 conn(0x7f1e580b0ba0 msgr2=0x7f1e580b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 -- 172.21.15.43:0/2103646655 shutdown_connections 2024-01-16T15:25:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.190+0000 7f1e5d400640 1 -- 172.21.15.43:0/2103646655 wait complete. 2024-01-16T15:25:28.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.192+0000 7f1e5d400640 1 Processor -- start 2024-01-16T15:25:28.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.192+0000 7f1e5d400640 1 -- start start 2024-01-16T15:25:28.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e5806d2a0 0x7f1e5814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e580b2d70 0x7f1e5813c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e5815aae0 0x7f1e581552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e581548d0 con 0x7f1e580b2d70 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58154bb0 con 0x7f1e5806d2a0 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e5d400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58070970 con 0x7f1e5815aae0 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e55e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e580b2d70 0x7f1e5813c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e55e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e580b2d70 0x7f1e5813c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39222/0 (socket says 172.21.15.43:39222) 2024-01-16T15:25:28.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.193+0000 7f1e55e00640 1 -- 172.21.15.43:0/1745000357 learned_addr learned my addr 172.21.15.43:0/1745000357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:28.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e56800640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e5806d2a0 0x7f1e5814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:28.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 -- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e5815aae0 msgr2=0x7f1e581552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:28.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e5815aae0 0x7f1e581552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 -- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e5806d2a0 msgr2=0x7f1e5814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e5806d2a0 0x7f1e5814ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 -- 172.21.15.43:0/1745000357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e400094d0 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e56800640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e5806d2a0 0x7f1e5814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e55e00640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e580b2d70 0x7f1e5813c8f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f1e4002b9e0 tx=0x7f1e40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40039d60 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e40043400 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e400437b0 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.194+0000 7f1e5d400640 1 -- 172.21.15.43:0/1745000357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e58070c50 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.195+0000 7f1e5d400640 1 -- 172.21.15.43:0/1745000357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5813ce30 con 0x7f1e580b2d70 2024-01-16T15:25:28.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.196+0000 7f1e5d400640 1 -- 172.21.15.43:0/1745000357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e58071170 con 0x7f1e580b2d70 2024-01-16T15:25:28.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.197+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f1e40043910 con 0x7f1e580b2d70 2024-01-16T15:25:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.198+0000 7f1e4ea00640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e3006c700 0x7f1e3006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.198+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1e4003e070 con 0x7f1e580b2d70 2024-01-16T15:25:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.198+0000 7f1e56800640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e3006c700 0x7f1e3006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.199+0000 7f1e56800640 1 --2- 172.21.15.43:0/1745000357 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1e3006c700 0x7f1e3006ebc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1e581403d0 tx=0x7f1e4800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:28.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.202+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e58071170 con 0x7f1e580b2d70 2024-01-16T15:25:28.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.437+0000 7f1e5d400640 1 -- 172.21.15.43:0/1745000357 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e5813cfc0 con 0x7f1e3006c700 2024-01-16T15:25:28.442 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:28.445+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:28.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:28.445+0000 7f1e4ea00640 1 -- 172.21.15.43:0/1745000357 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e48009b80 con 0x7f1e3006c700 2024-01-16T15:25:28.443 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:28.443 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:30.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:30.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.711+0000 7f2348200640 1 Processor -- start 2024-01-16T15:25:30.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.711+0000 7f2348200640 1 -- start start 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f234015b350 0x7f23400abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 0x7f23400b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f23400adb60 0x7f23400adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f234015b730 con 0x7f23400adb60 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23400ac740 con 0x7f234009cb30 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2348200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23400ac460 con 0x7f234015b350 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2346200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f23400adb60 0x7f23400adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2346200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f23400adb60 0x7f23400adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39230/0 (socket says 172.21.15.43:39230) 2024-01-16T15:25:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.713+0000 7f2346200640 1 -- 172.21.15.43:0/3637794563 learned_addr learned my addr 172.21.15.43:0/3637794563 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:30.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2345800640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f234015b350 0x7f23400abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2344e00640 1 --2- 172.21.15.43:0/3637794563 >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 0x7f23400b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2346200640 1 -- 172.21.15.43:0/3637794563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f234015b350 msgr2=0x7f23400abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:30.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2346200640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f234015b350 0x7f23400abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2346200640 1 -- 172.21.15.43:0/3637794563 >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 msgr2=0x7f23400b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:30.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2346200640 1 --2- 172.21.15.43:0/3637794563 >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 0x7f23400b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2346200640 1 -- 172.21.15.43:0/3637794563 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400b2a50 con 0x7f23400adb60 2024-01-16T15:25:30.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.714+0000 7f2344e00640 1 --2- 172.21.15.43:0/3637794563 >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 0x7f23400b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:30.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.716+0000 7f2346200640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23400adb60 0x7f23400adfb0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f23400b5130 tx=0x7f233802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e23ca4986dc7650 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:30.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.716+0000 7f233fe00640 1 -- 172.21.15.43:0/3637794563 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338031870 con 0x7f23400adb60 2024-01-16T15:25:30.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.716+0000 7f233fe00640 1 -- 172.21.15.43:0/3637794563 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23380319d0 con 0x7f23400adb60 2024-01-16T15:25:30.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.716+0000 7f233fe00640 1 -- 172.21.15.43:0/3637794563 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338031d60 con 0x7f23400adb60 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 -- 172.21.15.43:0/3637794563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23400adb60 msgr2=0x7f23400adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23400adb60 0x7f23400adfb0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f23400b5130 tx=0x7f233802b910 comp rx=0 tx=0).stop 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 -- 172.21.15.43:0/3637794563 shutdown_connections 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23400adb60 0x7f23400adfb0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 --2- 172.21.15.43:0/3637794563 >> v2:172.21.15.69:3300/0 conn(0x7f234009cb30 0x7f23400b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 --2- 172.21.15.43:0/3637794563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f234015b350 0x7f23400abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 -- 172.21.15.43:0/3637794563 >> 172.21.15.43:0/3637794563 conn(0x7f23400b0ba0 msgr2=0x7f23400b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 -- 172.21.15.43:0/3637794563 shutdown_connections 2024-01-16T15:25:30.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.717+0000 7f2348200640 1 -- 172.21.15.43:0/3637794563 wait complete. 2024-01-16T15:25:30.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.719+0000 7f2348200640 1 Processor -- start 2024-01-16T15:25:30.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.719+0000 7f2348200640 1 -- start start 2024-01-16T15:25:30.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f234009cb30 0x7f234013d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23400adb60 0x7f234009a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f234015b350 0x7f23401f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234009b0d0 con 0x7f234009cb30 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234014e1c0 con 0x7f234015b350 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2348200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234014e4a0 con 0x7f23400adb60 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.720+0000 7f2345800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f234009cb30 0x7f234013d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f234009cb30 0x7f234013d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39236/0 (socket says 172.21.15.43:39236) 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 -- 172.21.15.43:0/1787852758 learned_addr learned my addr 172.21.15.43:0/1787852758 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2344e00640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23400adb60 0x7f234009a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 -- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23400adb60 msgr2=0x7f234009a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23400adb60 0x7f234009a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 -- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f234015b350 msgr2=0x7f23401f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2346200640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f234015b350 0x7f23401f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f234015b350 0x7f23401f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 -- 172.21.15.43:0/1787852758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380094d0 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2346200640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f234015b350 0x7f23401f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.721+0000 7f2345800640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f234009cb30 0x7f234013d9a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f2330007ef0 tx=0x7f233000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.722+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330014620 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.722+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330014780 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.722+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330014b30 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.722+0000 7f2348200640 1 -- 172.21.15.43:0/1787852758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234009b2e0 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.722+0000 7f2348200640 1 -- 172.21.15.43:0/1787852758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234013dee0 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.723+0000 7f2348200640 1 -- 172.21.15.43:0/1787852758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234014e6c0 con 0x7f234009cb30 2024-01-16T15:25:30.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.724+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f2330014c90 con 0x7f234009cb30 2024-01-16T15:25:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.725+0000 7f233d600640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f231806c700 0x7f231806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:30.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.725+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2330089670 con 0x7f234009cb30 2024-01-16T15:25:30.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.725+0000 7f2344e00640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f231806c700 0x7f231806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:30.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.726+0000 7f2344e00640 1 --2- 172.21.15.43:0/1787852758 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f231806c700 0x7f231806ebc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f234014f0d0 tx=0x7f2328008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:30.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.730+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2330055b60 con 0x7f234009cb30 2024-01-16T15:25:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.959+0000 7f2348200640 1 -- 172.21.15.43:0/1787852758 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23400b3010 con 0x7f231806c700 2024-01-16T15:25:30.965 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:30.967+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:30.967+0000 7f233d600640 1 -- 172.21.15.43:0/1787852758 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f232800c040 con 0x7f231806c700 2024-01-16T15:25:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:30.965 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:32.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:32.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:32.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.219+0000 7ff152c00640 1 Processor -- start 2024-01-16T15:25:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.219+0000 7ff152c00640 1 -- start start 2024-01-16T15:25:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.220+0000 7ff152c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 0x7ff14c0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.220+0000 7ff152c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 0x7ff14c0ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.220+0000 7ff152c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b23e0 con 0x7ff14c15e590 2024-01-16T15:25:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.220+0000 7ff152c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0cfca0 con 0x7ff14c0cf2e0 2024-01-16T15:25:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.220+0000 7ff152c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b1860 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.221+0000 7ff151800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff14c15e590 0x7ff14c0b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:37428/0 (socket says 172.21.15.43:37428) 2024-01-16T15:25:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.221+0000 7ff151800640 1 -- 172.21.15.43:0/4167480363 learned_addr learned my addr 172.21.15.43:0/4167480363 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.221+0000 7ff152200640 1 --2- 172.21.15.43:0/4167480363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 0x7ff14c0ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.221+0000 7ff150e00640 1 --2- 172.21.15.43:0/4167480363 >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 0x7ff14c0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff14be00640 1 -- 172.21.15.43:0/4167480363 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846094263 0 0) 0x7ff14c0b23e0 con 0x7ff14c15e590 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff14be00640 1 -- 172.21.15.43:0/4167480363 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c003520 con 0x7ff14c15e590 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff152200640 1 -- 172.21.15.43:0/4167480363 >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 msgr2=0x7ff14c0cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff152200640 1 --2- 172.21.15.43:0/4167480363 >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 0x7ff14c0cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff152200640 1 -- 172.21.15.43:0/4167480363 >> v1:172.21.15.69:6789/0 conn(0x7ff14c15e590 legacy=0x7ff14c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff152200640 1 -- 172.21.15.43:0/4167480363 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b1b40 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.222+0000 7ff150e00640 1 --2- 172.21.15.43:0/4167480363 >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 0x7ff14c0cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:25:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.224+0000 7ff152200640 1 --2- 172.21.15.43:0/4167480363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 0x7ff14c0ae4f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff144009d90 tx=0x7ff14402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe503e1293255b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.224+0000 7ff14be00640 1 -- 172.21.15.43:0/4167480363 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14402cbc0 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.224+0000 7ff14be00640 1 -- 172.21.15.43:0/4167480363 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14402cd20 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.224+0000 7ff14be00640 1 -- 172.21.15.43:0/4167480363 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144034660 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 -- 172.21.15.43:0/4167480363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 msgr2=0x7ff14c0ae4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 --2- 172.21.15.43:0/4167480363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 0x7ff14c0ae4f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff144009d90 tx=0x7ff14402be10 comp rx=0 tx=0).stop 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 -- 172.21.15.43:0/4167480363 shutdown_connections 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 --2- 172.21.15.43:0/4167480363 >> v2:172.21.15.69:3300/0 conn(0x7ff14c0cf2e0 0x7ff14c0cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 --2- 172.21.15.43:0/4167480363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0ae0c0 0x7ff14c0ae4f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 -- 172.21.15.43:0/4167480363 >> 172.21.15.43:0/4167480363 conn(0x7ff14c0b2a70 msgr2=0x7ff14c0b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 -- 172.21.15.43:0/4167480363 shutdown_connections 2024-01-16T15:25:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.225+0000 7ff152c00640 1 -- 172.21.15.43:0/4167480363 wait complete. 2024-01-16T15:25:33.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.227+0000 7ff152c00640 1 Processor -- start 2024-01-16T15:25:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.227+0000 7ff152c00640 1 -- start start 2024-01-16T15:25:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff14c0ae0c0 0x7ff14c0b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0cf2e0 0x7ff14c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff14c15e590 0x7ff14c0b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b40a0 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b4370 con 0x7ff14c15e590 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff152c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b4650 con 0x7ff14c0cf2e0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff151800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff14c0ae0c0 0x7ff14c0b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff151800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff14c0ae0c0 0x7ff14c0b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57704/0 (socket says 172.21.15.43:57704) 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff151800640 1 -- 172.21.15.43:0/1916537029 learned_addr learned my addr 172.21.15.43:0/1916537029 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.228+0000 7ff150e00640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0cf2e0 0x7ff14c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff152200640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff14c15e590 0x7ff14c0b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 -- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0cf2e0 msgr2=0x7ff14c15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14c0cf2e0 0x7ff14c15c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 -- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff14c15e590 msgr2=0x7ff14c0b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff14c15e590 0x7ff14c0b5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 -- 172.21.15.43:0/1916537029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134009400 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff151800640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff14c0ae0c0 0x7ff14c0b3b60 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c390 tx=0x7ff13c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c005ce0 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c005e40 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0072f0 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.229+0000 7ff152c00640 1 -- 172.21.15.43:0/1916537029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff144009a70 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.230+0000 7ff152c00640 1 -- 172.21.15.43:0/1916537029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c0b4c10 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.231+0000 7ff152c00640 1 -- 172.21.15.43:0/1916537029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c067fe0 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.232+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7ff13c007450 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.233+0000 7ff149600640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff12406c700 0x7ff12406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.234+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff13c088290 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.234+0000 7ff150e00640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff12406c700 0x7ff12406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:33.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.234+0000 7ff150e00640 1 --2- 172.21.15.43:0/1916537029 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff12406c700 0x7ff12406ebc0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff14c15fba0 tx=0x7ff134008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:33.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.238+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff13c054780 con 0x7ff14c0ae0c0 2024-01-16T15:25:33.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.467+0000 7ff152c00640 1 -- 172.21.15.43:0/1916537029 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff14c0b2100 con 0x7ff12406c700 2024-01-16T15:25:33.474 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:33.477+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:33.477+0000 7ff149600640 1 -- 172.21.15.43:0/1916537029 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff134011e90 con 0x7ff12406c700 2024-01-16T15:25:33.475 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:35.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.736+0000 7f30abe00640 1 Processor -- start 2024-01-16T15:25:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.737+0000 7f30abe00640 1 -- start start 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30abe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f30ac0cf250 0x7f30ac0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30abe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30ac0cddf0 0x7f30ac0ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30abe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0cf630 con 0x7f30ac0cfff0 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30abe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac0b44c0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30abe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac0cfb60 con 0x7f30ac0cf250 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30aa000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30ac0cddf0 0x7f30ac0ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30aa000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f30ac0cddf0 0x7f30ac0ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57716/0 (socket says 172.21.15.43:57716) 2024-01-16T15:25:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.738+0000 7f30aa000640 1 -- 172.21.15.43:0/1003786854 learned_addr learned my addr 172.21.15.43:0/1003786854 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.739+0000 7f30aa000640 1 -- 172.21.15.43:0/1003786854 >> v2:172.21.15.69:3300/0 conn(0x7f30ac0cf250 msgr2=0x7f30ac0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.739+0000 7f30aa000640 1 --2- 172.21.15.43:0/1003786854 >> v2:172.21.15.69:3300/0 conn(0x7f30ac0cf250 0x7f30ac0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.739+0000 7f30aa000640 1 -- 172.21.15.43:0/1003786854 >> v1:172.21.15.69:6789/0 conn(0x7f30ac0cfff0 legacy=0x7f30ac0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.739+0000 7f30aa000640 1 -- 172.21.15.43:0/1003786854 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0d3fe0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.740+0000 7f30aa000640 1 --2- 172.21.15.43:0/1003786854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0ce250 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f30ac08b600 tx=0x7f309402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb2740572f610ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.741+0000 7f30a9600640 1 -- 172.21.15.43:0/1003786854 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094031870 con 0x7f30ac0cddf0 2024-01-16T15:25:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.741+0000 7f30a9600640 1 -- 172.21.15.43:0/1003786854 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30940319d0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.741+0000 7f30a9600640 1 -- 172.21.15.43:0/1003786854 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094031d60 con 0x7f30ac0cddf0 2024-01-16T15:25:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 -- 172.21.15.43:0/1003786854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 msgr2=0x7f30ac0ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 --2- 172.21.15.43:0/1003786854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0ce250 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f30ac08b600 tx=0x7f309402b910 comp rx=0 tx=0).stop 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 -- 172.21.15.43:0/1003786854 shutdown_connections 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 --2- 172.21.15.43:0/1003786854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0ce250 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 --2- 172.21.15.43:0/1003786854 >> v2:172.21.15.69:3300/0 conn(0x7f30ac0cf250 0x7f30ac0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 -- 172.21.15.43:0/1003786854 >> 172.21.15.43:0/1003786854 conn(0x7f30ac0d3040 msgr2=0x7f30ac0d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 -- 172.21.15.43:0/1003786854 shutdown_connections 2024-01-16T15:25:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.742+0000 7f30abe00640 1 -- 172.21.15.43:0/1003786854 wait complete. 2024-01-16T15:25:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.744+0000 7f30abe00640 1 Processor -- start 2024-01-16T15:25:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.744+0000 7f30abe00640 1 -- start start 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30ac0cf250 0x7f30ac151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30ac0cfff0 0x7f30ac0afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c7070 con 0x7f30ac0cddf0 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac15d4c0 con 0x7f30ac0cfff0 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30abe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c7e90 con 0x7f30ac0cf250 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57720/0 (socket says 172.21.15.43:57720) 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 -- 172.21.15.43:0/1547472722 learned_addr learned my addr 172.21.15.43:0/1547472722 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 -- 172.21.15.43:0/1547472722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30ac0cf250 msgr2=0x7f30ac151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30ac0cf250 0x7f30ac151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 -- 172.21.15.43:0/1547472722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30ac0cfff0 msgr2=0x7f30ac0afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30ac0cfff0 0x7f30ac0afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.745+0000 7f30aaa00640 1 -- 172.21.15.43:0/1547472722 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940094d0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30aaa00640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac0cddf0 0x7f30ac0c6b30 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f309c00ba50 tx=0x7f309c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00cbe0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00cd60 con 0x7f30ac0cddf0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c014680 con 0x7f30ac0cddf0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30abe00640 1 -- 172.21.15.43:0/1547472722 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac0c7280 con 0x7f30ac0cddf0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.746+0000 7f30abe00640 1 -- 172.21.15.43:0/1547472722 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac0c7840 con 0x7f30ac0cddf0 2024-01-16T15:25:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.747+0000 7f30abe00640 1 -- 172.21.15.43:0/1547472722 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac04e840 con 0x7f30ac0cddf0 2024-01-16T15:25:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.748+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f309c0147e0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.749+0000 7f30a3400640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f308406c730 0x7f308406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.749+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f309c0887f0 con 0x7f30ac0cddf0 2024-01-16T15:25:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.750+0000 7f30aa000640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f308406c730 0x7f308406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.750+0000 7f30aa000640 1 --2- 172.21.15.43:0/1547472722 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f308406c730 0x7f308406ebf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3094036000 tx=0x7f3094030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.754+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f309c055080 con 0x7f30ac0cddf0 2024-01-16T15:25:35.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.981+0000 7f30abe00640 1 -- 172.21.15.43:0/1547472722 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30ac0d3d00 con 0x7f308406c730 2024-01-16T15:25:35.992 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:35.994+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:35.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:35.995+0000 7f30a3400640 1 -- 172.21.15.43:0/1547472722 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3094031cd0 con 0x7f308406c730 2024-01-16T15:25:35.993 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:35.993 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:38.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.249+0000 7f4d3a800640 1 Processor -- start 2024-01-16T15:25:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.249+0000 7f4d3a800640 1 -- start start 2024-01-16T15:25:38.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d3a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d3406dae0 0x7f4d3415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d3a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 0x7f4d34070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d3a800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34144ca0 con 0x7f4d341448c0 2024-01-16T15:25:38.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d3a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3415c450 con 0x7f4d3406dae0 2024-01-16T15:25:38.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d3a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d340715b0 con 0x7f4d3415bbd0 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d39e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 0x7f4d34070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d39400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4d341448c0 0x7f4d3406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:37434/0 (socket says 172.21.15.43:37434) 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d39400640 1 -- 172.21.15.43:0/4257679420 learned_addr learned my addr 172.21.15.43:0/4257679420 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.251+0000 7f4d38a00640 1 --2- 172.21.15.43:0/4257679420 >> v2:172.21.15.69:3300/0 conn(0x7f4d3406dae0 0x7f4d3415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.252+0000 7f4d39e00640 1 -- 172.21.15.43:0/4257679420 >> v2:172.21.15.69:3300/0 conn(0x7f4d3406dae0 msgr2=0x7f4d3415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.252+0000 7f4d39e00640 1 --2- 172.21.15.43:0/4257679420 >> v2:172.21.15.69:3300/0 conn(0x7f4d3406dae0 0x7f4d3415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.252+0000 7f4d39e00640 1 -- 172.21.15.43:0/4257679420 >> v1:172.21.15.69:6789/0 conn(0x7f4d341448c0 legacy=0x7f4d3406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.252+0000 7f4d39e00640 1 -- 172.21.15.43:0/4257679420 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34071890 con 0x7f4d3415bbd0 2024-01-16T15:25:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.253+0000 7f4d39e00640 1 --2- 172.21.15.43:0/4257679420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 0x7f4d34070070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c0097a0 tx=0x7f4d2c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba57271c766ed28 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.254+0000 7f4d33e00640 1 -- 172.21.15.43:0/4257679420 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c02d040 con 0x7f4d3415bbd0 2024-01-16T15:25:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.254+0000 7f4d33e00640 1 -- 172.21.15.43:0/4257679420 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c031810 con 0x7f4d3415bbd0 2024-01-16T15:25:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.254+0000 7f4d33e00640 1 -- 172.21.15.43:0/4257679420 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c031ba0 con 0x7f4d3415bbd0 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 -- 172.21.15.43:0/4257679420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 msgr2=0x7f4d34070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 --2- 172.21.15.43:0/4257679420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 0x7f4d34070070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c0097a0 tx=0x7f4d2c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 -- 172.21.15.43:0/4257679420 shutdown_connections 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 --2- 172.21.15.43:0/4257679420 >> v2:172.21.15.69:3300/0 conn(0x7f4d3406dae0 0x7f4d3415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 --2- 172.21.15.43:0/4257679420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d3415bbd0 0x7f4d34070070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 -- 172.21.15.43:0/4257679420 >> 172.21.15.43:0/4257679420 conn(0x7f4d340b0aa0 msgr2=0x7f4d340b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 -- 172.21.15.43:0/4257679420 shutdown_connections 2024-01-16T15:25:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.255+0000 7f4d3a800640 1 -- 172.21.15.43:0/4257679420 wait complete. 2024-01-16T15:25:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.257+0000 7f4d3a800640 1 Processor -- start 2024-01-16T15:25:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.257+0000 7f4d3a800640 1 -- start start 2024-01-16T15:25:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3406dae0 0x7f4d34155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d341448c0 0x7f4d3415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d3415bbd0 0x7f4d341ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d34155e60 con 0x7f4d3415bbd0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415c6c0 con 0x7f4d3406dae0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d3a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415cc40 con 0x7f4d341448c0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d3415bbd0 0x7f4d341ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d3415bbd0 0x7f4d341ee690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57730/0 (socket says 172.21.15.43:57730) 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.258+0000 7f4d39e00640 1 -- 172.21.15.43:0/1471984587 learned_addr learned my addr 172.21.15.43:0/1471984587 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39400640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3406dae0 0x7f4d34155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d38a00640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d341448c0 0x7f4d3415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 -- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d341448c0 msgr2=0x7f4d3415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d341448c0 0x7f4d3415d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 -- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3406dae0 msgr2=0x7f4d34155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3406dae0 0x7f4d34155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 -- 172.21.15.43:0/1471984587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c009400 con 0x7f4d3415bbd0 2024-01-16T15:25:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d39e00640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d3415bbd0 0x7f4d341ee690 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c0097a0 tx=0x7f4d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.259+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c03ac60 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.260+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c03adc0 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.260+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c044640 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.260+0000 7f4d3a800640 1 -- 172.21.15.43:0/1471984587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d340724f0 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.260+0000 7f4d3a800640 1 -- 172.21.15.43:0/1471984587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d341ef4a0 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.261+0000 7f4d3a800640 1 -- 172.21.15.43:0/1471984587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d34140b80 con 0x7f4d3415bbd0 2024-01-16T15:25:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.262+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f4d2c04d440 con 0x7f4d3415bbd0 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.263+0000 7f4d31600640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4d0c06c700 0x7f4d0c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.263+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4d2c0aaa50 con 0x7f4d3415bbd0 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.263+0000 7f4d39400640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4d0c06c700 0x7f4d0c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.264+0000 7f4d39400640 1 --2- 172.21.15.43:0/1471984587 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f4d0c06c700 0x7f4d0c06ebc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4d3413a9f0 tx=0x7f4d24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:38.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.267+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d2c077310 con 0x7f4d3415bbd0 2024-01-16T15:25:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.497+0000 7f4d3a800640 1 -- 172.21.15.43:0/1471984587 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d3415b9e0 con 0x7f4d0c06c700 2024-01-16T15:25:38.504 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:38.506+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:38.504 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:38.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:38.507+0000 7f4d31600640 1 -- 172.21.15.43:0/1471984587 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4d3415b9e0 con 0x7f4d0c06c700 2024-01-16T15:25:38.505 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:40.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:40.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.762+0000 7fcdc1800640 1 Processor -- start 2024-01-16T15:25:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.762+0000 7fcdc1800640 1 -- start start 2024-01-16T15:25:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdc1800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcdbc0d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdc1800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcdbc0d1d90 0x7fcdbc0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdc1800640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0cef30 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdc1800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdbc0d11a0 con 0x7fcdbc0d1d90 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdc1800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0ce320 con 0x7fcdbc15d340 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdbb400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcdbc0cdbb0 0x7fcdbc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:42264/0 (socket says 172.21.15.43:42264) 2024-01-16T15:25:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.764+0000 7fcdbb400640 1 -- 172.21.15.43:0/3792680485 learned_addr learned my addr 172.21.15.43:0/3792680485 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3792680485 >> v2:172.21.15.69:3300/0 conn(0x7fcdbc0d1d90 0x7fcdbc0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcdbc0d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212738715 0 0) 0x7fcdbc0cef30 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9c003520 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162809941 0 0) 0x7fcd9c003520 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 msgr2=0x7fcdbc0d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcdbc0d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 >> v2:172.21.15.69:3300/0 conn(0x7fcdbc0d1d90 msgr2=0x7fcdbc0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 --2- 172.21.15.43:0/3792680485 >> v2:172.21.15.69:3300/0 conn(0x7fcdbc0d1d90 0x7fcdbc0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0ce600 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fcda4001670 con 0x7fcdbc0cdbb0 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 >> v1:172.21.15.43:6789/0 conn(0x7fcdbc0cdbb0 legacy=0x7fcdbc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcd9c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c003f60 0x7fcd9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd9c004d00 0x7fcd9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0d11a0 con 0x7fcd9c004d00 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0ce320 con 0x7fcd9c003f60 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.765+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c00b9b0 con 0x7fcdbc15d340 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd9c004d00 0x7fcd9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdbb400640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c003f60 0x7fcd9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcd9c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 -- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 msgr2=0x7fcd9c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcd9c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 -- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c003f60 msgr2=0x7fcd9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c003f60 0x7fcd9c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdc0e00640 1 -- 172.21.15.43:0/3792680485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c003520 con 0x7fcd9c004d00 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc15d340 0x7fcd9c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.766+0000 7fcdbb400640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c003f60 0x7fcd9c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.767+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd9c004d00 0x7fcd9c005180 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc08d120 tx=0x7fcd9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94c78e0d1f9315d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb402c680 con 0x7fcd9c004d00 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb402c7e0 con 0x7fcd9c004d00 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdbaa00640 1 -- 172.21.15.43:0/3792680485 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb402cb70 con 0x7fcd9c004d00 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdc1800640 1 -- 172.21.15.43:0/3792680485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd9c004d00 msgr2=0x7fcd9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdc1800640 1 --2- 172.21.15.43:0/3792680485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd9c004d00 0x7fcd9c005180 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc08d120 tx=0x7fcd9c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdc0e00640 1 -- 172.21.15.43:0/3792680485 reap_dead start 2024-01-16T15:25:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdc1800640 1 -- 172.21.15.43:0/3792680485 shutdown_connections 2024-01-16T15:25:40.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.768+0000 7fcdc1800640 1 -- 172.21.15.43:0/3792680485 >> 172.21.15.43:0/3792680485 conn(0x7fcdbc09c370 msgr2=0x7fcdbc0d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:40.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.769+0000 7fcdc1800640 1 -- 172.21.15.43:0/3792680485 shutdown_connections 2024-01-16T15:25:40.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.769+0000 7fcdc1800640 1 -- 172.21.15.43:0/3792680485 wait complete. 2024-01-16T15:25:40.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.770+0000 7fcdc1800640 1 Processor -- start 2024-01-16T15:25:40.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.771+0000 7fcdc1800640 1 -- start start 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c004d00 0x7fcdbc0c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdbc156120 0x7fcdbc0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc1565a0 0x7fcdbc0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0c7700 con 0x7fcdbc156120 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0c79e0 con 0x7fcd9c004d00 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc1800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0c7cc0 con 0x7fcdbc1565a0 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdbc156120 0x7fcdbc0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdbc156120 0x7fcdbc0c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57750/0 (socket says 172.21.15.43:57750) 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 -- 172.21.15.43:0/3354375106 learned_addr learned my addr 172.21.15.43:0/3354375106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:40.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdc0e00640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc1565a0 0x7fcdbc0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 -- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc1565a0 msgr2=0x7fcdbc0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdbc1565a0 0x7fcdbc0c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c004d00 0x7fcdbc0c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 -- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c004d00 msgr2=0x7fcdbc0c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd9c004d00 0x7fcdbc0c87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.772+0000 7fcdbb400640 1 -- 172.21.15.43:0/3354375106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0c7fa0 con 0x7fcdbc156120 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdbb400640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcdbc156120 0x7fcdbc0c9230 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fcda400e800 tx=0x7fcda400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda400c8d0 con 0x7fcdbc156120 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda400ca30 con 0x7fcdbc156120 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdc1800640 1 -- 172.21.15.43:0/3354375106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc0c8280 con 0x7fcdbc156120 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda400cda0 con 0x7fcdbc156120 2024-01-16T15:25:40.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.773+0000 7fcdc1800640 1 -- 172.21.15.43:0/3354375106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc0b3320 con 0x7fcdbc156120 2024-01-16T15:25:40.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.774+0000 7fcdc1800640 1 -- 172.21.15.43:0/3354375106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc067d70 con 0x7fcdbc156120 2024-01-16T15:25:40.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.775+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fcda401d460 con 0x7fcdbc156120 2024-01-16T15:25:40.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.776+0000 7fcdb3e00640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd9406c700 0x7fcd9406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:40.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.776+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcda4088cc0 con 0x7fcdbc156120 2024-01-16T15:25:40.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.777+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd9406c700 0x7fcd9406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:40.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.777+0000 7fcdbbe00640 1 --2- 172.21.15.43:0/3354375106 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcd9406c700 0x7fcd9406ebc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc157f30 tx=0x7fcdac00d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:40.781+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcda40551b0 con 0x7fcdbc156120 2024-01-16T15:25:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:41.009+0000 7fcdc1800640 1 -- 172.21.15.43:0/3354375106 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcdbc0cebc0 con 0x7fcd9406c700 2024-01-16T15:25:41.019 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:41.022+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:41.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:41.022+0000 7fcdb3e00640 1 -- 172.21.15.43:0/3354375106 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcdac011040 con 0x7fcd9406c700 2024-01-16T15:25:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:41.020 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:43.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.279+0000 7f647b000640 1 Processor -- start 2024-01-16T15:25:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.279+0000 7f647b000640 1 -- start start 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f647b000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f64740b36d0 0x7f64740d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f647b000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740cef30 con 0x7f647415d340 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f647b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740d11a0 con 0x7f64740d1d90 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f647b000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64740ce320 con 0x7f64740b36d0 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f6479c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f647415d340 0x7f64740d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46632/0 (socket says 172.21.15.43:46632) 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f6479c00640 1 -- 172.21.15.43:0/4237622637 learned_addr learned my addr 172.21.15.43:0/4237622637 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.281+0000 7f647a600640 1 --2- 172.21.15.43:0/4237622637 >> v2:172.21.15.43:3300/0 conn(0x7f64740b36d0 0x7f64740d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.282+0000 7f6473e00640 1 -- 172.21.15.43:0/4237622637 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355611168 0 0) 0x7f64740cef30 con 0x7f647415d340 2024-01-16T15:25:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.282+0000 7f6473e00640 1 -- 172.21.15.43:0/4237622637 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454003520 con 0x7f647415d340 2024-01-16T15:25:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.282+0000 7f647a600640 1 -- 172.21.15.43:0/4237622637 >> v1:172.21.15.69:6789/0 conn(0x7f64740d1d90 legacy=0x7f64740b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.282+0000 7f647a600640 1 -- 172.21.15.43:0/4237622637 >> v1:172.21.15.43:6789/0 conn(0x7f647415d340 legacy=0x7f64740d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.282+0000 7f647a600640 1 -- 172.21.15.43:0/4237622637 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740ce600 con 0x7f64740b36d0 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.283+0000 7f647a600640 1 --2- 172.21.15.43:0/4237622637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64740b36d0 0x7f64740d00b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f646c009d90 tx=0x7f646c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8a0a872f4b7314 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.283+0000 7f6473e00640 1 -- 172.21.15.43:0/4237622637 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c02ccf0 con 0x7f64740b36d0 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.283+0000 7f6473e00640 1 -- 172.21.15.43:0/4237622637 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c02ce50 con 0x7f64740b36d0 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.283+0000 7f6473e00640 1 -- 172.21.15.43:0/4237622637 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c034640 con 0x7f64740b36d0 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 -- 172.21.15.43:0/4237622637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64740b36d0 msgr2=0x7f64740d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 --2- 172.21.15.43:0/4237622637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64740b36d0 0x7f64740d00b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f646c009d90 tx=0x7f646c02bf40 comp rx=0 tx=0).stop 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 -- 172.21.15.43:0/4237622637 shutdown_connections 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 --2- 172.21.15.43:0/4237622637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64740b36d0 0x7f64740d00b0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 -- 172.21.15.43:0/4237622637 >> 172.21.15.43:0/4237622637 conn(0x7f647409c370 msgr2=0x7f64740d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 -- 172.21.15.43:0/4237622637 shutdown_connections 2024-01-16T15:25:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.284+0000 7f647b000640 1 -- 172.21.15.43:0/4237622637 wait complete. 2024-01-16T15:25:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.286+0000 7f647b000640 1 Processor -- start 2024-01-16T15:25:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.286+0000 7f647b000640 1 -- start start 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64740b36d0 0x7f64741520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64740d1d90 0x7f647415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f647415d340 0x7f647415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647415ea70 con 0x7f647415d340 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647415de60 con 0x7f64740b36d0 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.287+0000 7f647b000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647415e140 con 0x7f64740d1d90 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f647415d340 0x7f647415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f647415d340 0x7f647415bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46368/0 (socket says 172.21.15.43:46368) 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 -- 172.21.15.43:0/3516013775 learned_addr learned my addr 172.21.15.43:0/3516013775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f6479c00640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64740b36d0 0x7f64741520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f6479200640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64740d1d90 0x7f647415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 -- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64740d1d90 msgr2=0x7f647415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64740d1d90 0x7f647415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 -- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64740b36d0 msgr2=0x7f64741520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64740b36d0 0x7f64741520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 -- 172.21.15.43:0/3516013775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009a70 con 0x7f647415d340 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f6479200640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64740d1d90 0x7f647415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.288+0000 7f647a600640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f647415d340 0x7f647415bc70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f646c009d90 tx=0x7f646c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.289+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c003b90 con 0x7f647415d340 2024-01-16T15:25:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.289+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c003cf0 con 0x7f647415d340 2024-01-16T15:25:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.289+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c004270 con 0x7f647415d340 2024-01-16T15:25:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.289+0000 7f647b000640 1 -- 172.21.15.43:0/3516013775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647415e420 con 0x7f647415d340 2024-01-16T15:25:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.289+0000 7f647b000640 1 -- 172.21.15.43:0/3516013775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647415ef00 con 0x7f647415d340 2024-01-16T15:25:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.290+0000 7f647b000640 1 -- 172.21.15.43:0/3516013775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6474067d70 con 0x7f647415d340 2024-01-16T15:25:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.291+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f646c03b050 con 0x7f647415d340 2024-01-16T15:25:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.292+0000 7f6471600640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f644c06c700 0x7f644c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.292+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f646c0aae20 con 0x7f647415d340 2024-01-16T15:25:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.292+0000 7f6479c00640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f644c06c700 0x7f644c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.293+0000 7f6479c00640 1 --2- 172.21.15.43:0/3516013775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f644c06c700 0x7f644c06ebc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f64640011e0 tx=0x7f6464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.296+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f646c0776e0 con 0x7f647415d340 2024-01-16T15:25:43.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.525+0000 7f647b000640 1 -- 172.21.15.43:0/3516013775 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f647404e840 con 0x7f644c06c700 2024-01-16T15:25:43.535 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:43.537+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:43.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:43.538+0000 7f6471600640 1 -- 172.21.15.43:0/3516013775 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f647404e840 con 0x7f644c06c700 2024-01-16T15:25:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:43.536 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:45.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:45.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.786+0000 7fb289800640 1 Processor -- start 2024-01-16T15:25:45.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.787+0000 7fb289800640 1 -- start start 2024-01-16T15:25:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.788+0000 7fb289800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb284165e20 0x7fb284070830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.788+0000 7fb289800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 0x7fb28406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.788+0000 7fb289800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb284070070 con 0x7fb28415f100 2024-01-16T15:25:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.788+0000 7fb289800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2841624e0 con 0x7fb284165e20 2024-01-16T15:25:45.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.789+0000 7fb289800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841627c0 con 0x7fb284070d70 2024-01-16T15:25:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.789+0000 7fb27be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb284165e20 0x7fb284070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.789+0000 7fb27be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb284165e20 0x7fb284070830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46382/0 (socket says 172.21.15.43:46382) 2024-01-16T15:25:45.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.789+0000 7fb27be00640 1 -- 172.21.15.43:0/3483966553 learned_addr learned my addr 172.21.15.43:0/3483966553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.789+0000 7fb288e00640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 0x7fb28406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.790+0000 7fb27be00640 1 -- 172.21.15.43:0/3483966553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 msgr2=0x7fb28406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.790+0000 7fb27be00640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 0x7fb28406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.790+0000 7fb27be00640 1 -- 172.21.15.43:0/3483966553 >> v1:172.21.15.69:6789/0 conn(0x7fb28415f100 legacy=0x7fb284161c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.790+0000 7fb27be00640 1 -- 172.21.15.43:0/3483966553 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284162aa0 con 0x7fb284165e20 2024-01-16T15:25:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.790+0000 7fb288e00640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 0x7fb28406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:45.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.792+0000 7fb27be00640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb284165e20 0x7fb284070830 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009860 tx=0x7fb26c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8b63836a38c512 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.792+0000 7fb283400640 1 -- 172.21.15.43:0/3483966553 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c030840 con 0x7fb284165e20 2024-01-16T15:25:45.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb283400640 1 -- 172.21.15.43:0/3483966553 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c0309a0 con 0x7fb284165e20 2024-01-16T15:25:45.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb283400640 1 -- 172.21.15.43:0/3483966553 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c030d30 con 0x7fb284165e20 2024-01-16T15:25:45.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 -- 172.21.15.43:0/3483966553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb284165e20 msgr2=0x7fb284070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb284165e20 0x7fb284070830 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009860 tx=0x7fb26c02b980 comp rx=0 tx=0).stop 2024-01-16T15:25:45.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 -- 172.21.15.43:0/3483966553 shutdown_connections 2024-01-16T15:25:45.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb284165e20 0x7fb284070830 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 --2- 172.21.15.43:0/3483966553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb284070d70 0x7fb28406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.793+0000 7fb289800640 1 -- 172.21.15.43:0/3483966553 >> 172.21.15.43:0/3483966553 conn(0x7fb2840b0aa0 msgr2=0x7fb2840b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:45.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.794+0000 7fb289800640 1 -- 172.21.15.43:0/3483966553 shutdown_connections 2024-01-16T15:25:45.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.794+0000 7fb289800640 1 -- 172.21.15.43:0/3483966553 wait complete. 2024-01-16T15:25:45.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.795+0000 7fb289800640 1 Processor -- start 2024-01-16T15:25:45.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.796+0000 7fb289800640 1 -- start start 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb28415f100 0x7fb2841489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb284165e20 0x7fb2840cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2840ce100 0x7fb284163d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840ce730 con 0x7fb2840ce100 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841f73f0 con 0x7fb284165e20 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb289800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841f76d0 con 0x7fb28415f100 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb288e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2840ce100 0x7fb284163d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb288e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2840ce100 0x7fb284163d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46396/0 (socket says 172.21.15.43:46396) 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb288e00640 1 -- 172.21.15.43:0/3284508260 learned_addr learned my addr 172.21.15.43:0/3284508260 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb27be00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb284165e20 0x7fb2840cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb283e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb28415f100 0x7fb2841489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb288e00640 1 -- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb28415f100 msgr2=0x7fb2841489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.797+0000 7fb288e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb28415f100 0x7fb2841489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb288e00640 1 -- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb284165e20 msgr2=0x7fb2840cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb288e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb284165e20 0x7fb2840cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb288e00640 1 -- 172.21.15.43:0/3284508260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009470 con 0x7fb2840ce100 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb27be00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb284165e20 0x7fb2840cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb288e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb2840ce100 0x7fb284163d30 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fb27c007e40 tx=0x7fb27c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c015600 con 0x7fb2840ce100 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c00f040 con 0x7fb2840ce100 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb289800640 1 -- 172.21.15.43:0/3284508260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2841f79b0 con 0x7fb2840ce100 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0159a0 con 0x7fb2840ce100 2024-01-16T15:25:45.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.798+0000 7fb289800640 1 -- 172.21.15.43:0/3284508260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2840adf00 con 0x7fb2840ce100 2024-01-16T15:25:45.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.800+0000 7fb289800640 1 -- 172.21.15.43:0/3284508260 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2841f7bf0 con 0x7fb2840ce100 2024-01-16T15:25:45.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.801+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fb27c01e3b0 con 0x7fb2840ce100 2024-01-16T15:25:45.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.801+0000 7fb280c00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25c06c700 0x7fb25c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.802+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb27c088c50 con 0x7fb2840ce100 2024-01-16T15:25:45.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.802+0000 7fb283e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25c06c700 0x7fb25c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.802+0000 7fb283e00640 1 --2- 172.21.15.43:0/3284508260 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb25c06c700 0x7fb25c06ebc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb274000f60 tx=0x7fb274009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:45.806+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2841f7bf0 con 0x7fb2840ce100 2024-01-16T15:25:46.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:46.036+0000 7fb289800640 1 -- 172.21.15.43:0/3284508260 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb28406d910 con 0x7fb25c06c700 2024-01-16T15:25:46.049 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:46.052+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:46.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:46.052+0000 7fb280c00640 1 -- 172.21.15.43:0/3284508260 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb274001360 con 0x7fb25c06c700 2024-01-16T15:25:46.050 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:46.050 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:48.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:48.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.310+0000 7f6118e00640 1 Processor -- start 2024-01-16T15:25:48.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.310+0000 7f6118e00640 1 -- start start 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f6118e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f61140ae490 0x7f6114070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f6118e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611406d7a0 con 0x7f611406dc30 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f6118e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61140ae900 con 0x7f61140acd70 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f6118e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611415a5b0 con 0x7f61140ae490 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f610be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f611406dc30 0x7f6114070490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:46648/0 (socket says 172.21.15.43:46648) 2024-01-16T15:25:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.312+0000 7f610be00640 1 -- 172.21.15.43:0/643185718 learned_addr learned my addr 172.21.15.43:0/643185718 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6112800640 1 --2- 172.21.15.43:0/643185718 >> v2:172.21.15.69:3300/0 conn(0x7f61140ae490 0x7f6114070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934895544 0 0) 0x7f611406d7a0 con 0x7f611406dc30 2024-01-16T15:25:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4005730 con 0x7f611406dc30 2024-01-16T15:25:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2734468862 0 0) 0x7f60f4005730 con 0x7f611406dc30 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 >> v2:172.21.15.69:3300/0 conn(0x7f61140ae490 msgr2=0x7f6114070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 --2- 172.21.15.43:0/643185718 >> v2:172.21.15.69:3300/0 conn(0x7f61140ae490 0x7f6114070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 >> v1:172.21.15.69:6789/0 conn(0x7f61140acd70 legacy=0x7f61140ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.313+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611406ce70 con 0x7f611406dc30 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f60fc001670 con 0x7f611406dc30 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 >> v1:172.21.15.43:6789/0 conn(0x7f611406dc30 legacy=0x7f6114070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4005bd0 0x7f60f4005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60f40069d0 0x7f60f4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 0x7f60f400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611415a5b0 con 0x7f60f4007390 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f400e290 con 0x7f60f4005bd0 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f400e450 con 0x7f60f40069d0 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 0x7f60f400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f610be00640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60f40069d0 0x7f60f4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 -- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60f40069d0 msgr2=0x7f60f4006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60f40069d0 0x7f60f4006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 -- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4005bd0 msgr2=0x7f60f4005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4005bd0 0x7f60f4005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.314+0000 7f6112800640 1 -- 172.21.15.43:0/643185718 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4005730 con 0x7f60f4007390 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.315+0000 7f6112800640 1 -- 172.21.15.43:0/643185718 reap_dead start 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6112800640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 0x7f60f400db90 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f610c009ac0 tx=0x7f60f4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839e4df80dad132 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c004020 con 0x7f60f4007390 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c004180 con 0x7f60f4007390 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6111400640 1 -- 172.21.15.43:0/643185718 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c0044f0 con 0x7f60f4007390 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6118e00640 1 -- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 msgr2=0x7f60f400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6118e00640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 0x7f60f400db90 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f610c009ac0 tx=0x7f60f4007a00 comp rx=0 tx=0).stop 2024-01-16T15:25:48.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6118e00640 1 -- 172.21.15.43:0/643185718 shutdown_connections 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6118e00640 1 --2- 172.21.15.43:0/643185718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60f4007390 0x7f60f400db90 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.316+0000 7f6118e00640 1 -- 172.21.15.43:0/643185718 >> 172.21.15.43:0/643185718 conn(0x7f61140b0ba0 msgr2=0x7f61140b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.317+0000 7f6118e00640 1 -- 172.21.15.43:0/643185718 shutdown_connections 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.317+0000 7f6118e00640 1 -- 172.21.15.43:0/643185718 wait complete. 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.318+0000 7f6118e00640 1 Processor -- start 2024-01-16T15:25:48.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.319+0000 7f6118e00640 1 -- start start 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.319+0000 7f6118e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4007390 0x7f611414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6118e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61140cb500 0x7f61140cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6118e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61140cb960 0x7f61140cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6118e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61140cbee0 con 0x7f61140cb960 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6118e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61140cc1c0 con 0x7f60f4007390 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6118e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61140cc4a0 con 0x7f61140cb500 2024-01-16T15:25:48.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61140cb960 0x7f61140cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61140cb960 0x7f61140cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46414/0 (socket says 172.21.15.43:46414) 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 -- 172.21.15.43:0/1890184800 learned_addr learned my addr 172.21.15.43:0/1890184800 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f610be00640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61140cb500 0x7f61140cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6111e00640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4007390 0x7f611414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 -- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61140cb500 msgr2=0x7f61140cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61140cb500 0x7f61140cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 -- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4007390 msgr2=0x7f611414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60f4007390 0x7f611414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.320+0000 7f6112800640 1 -- 172.21.15.43:0/1890184800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61140cc780 con 0x7f61140cb960 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f6112800640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61140cb960 0x7f61140cddc0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f610c02c380 tx=0x7f610c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c003810 con 0x7f61140cb960 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c003990 con 0x7f61140cb960 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c003d40 con 0x7f61140cb960 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f6118e00640 1 -- 172.21.15.43:0/1890184800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61140cca60 con 0x7f61140cb960 2024-01-16T15:25:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.321+0000 7f6118e00640 1 -- 172.21.15.43:0/1890184800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61140ccf80 con 0x7f61140cb960 2024-01-16T15:25:48.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.322+0000 7f6118e00640 1 -- 172.21.15.43:0/1890184800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611409d200 con 0x7f61140cb960 2024-01-16T15:25:48.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.323+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f610c003ea0 con 0x7f61140cb960 2024-01-16T15:25:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.324+0000 7f610a000640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f60ec06c730 0x7f60ec06ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.324+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f610c0a98a0 con 0x7f61140cb960 2024-01-16T15:25:48.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.325+0000 7f6111e00640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f60ec06c730 0x7f60ec06ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:48.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.325+0000 7f6111e00640 1 --2- 172.21.15.43:0/1890184800 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f60ec06c730 0x7f60ec06ebf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6104007fd0 tx=0x7f6104009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:48.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.328+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f610c076160 con 0x7f61140cb960 2024-01-16T15:25:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.558+0000 7f6118e00640 1 -- 172.21.15.43:0/1890184800 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61140b3d90 con 0x7f60ec06c730 2024-01-16T15:25:48.572 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:48.574+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:48.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:48.574+0000 7f610a000640 1 -- 172.21.15.43:0/1890184800 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6104003990 con 0x7f60ec06c730 2024-01-16T15:25:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:48.572 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:50.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:50.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.828+0000 7f00a8600640 1 Processor -- start 2024-01-16T15:25:50.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.829+0000 7f00a8600640 1 -- start start 2024-01-16T15:25:50.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.830+0000 7f00a8600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.830+0000 7f00a8600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 0x7f00a015a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.830+0000 7f00a8600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00b2720 con 0x7f00a006e7e0 2024-01-16T15:25:50.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.830+0000 7f00a8600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a00b2970 con 0x7f00a006bb60 2024-01-16T15:25:50.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.830+0000 7f00a8600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00b2c50 con 0x7f00a006e400 2024-01-16T15:25:50.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.831+0000 7f00a5c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.831+0000 7f00a5c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:50630/0 (socket says 172.21.15.43:50630) 2024-01-16T15:25:50.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.831+0000 7f00a5c00640 1 -- 172.21.15.43:0/438677025 learned_addr learned my addr 172.21.15.43:0/438677025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:50.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.831+0000 7f00a6600640 1 --2- 172.21.15.43:0/438677025 >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 0x7f00a015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.832+0000 7f00a5c00640 1 -- 172.21.15.43:0/438677025 >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 msgr2=0x7f00a015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:50.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.832+0000 7f00a5c00640 1 --2- 172.21.15.43:0/438677025 >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 0x7f00a015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.832+0000 7f00a5c00640 1 -- 172.21.15.43:0/438677025 >> v1:172.21.15.69:6789/0 conn(0x7f00a006e7e0 legacy=0x7f00a006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.832+0000 7f00a5c00640 1 -- 172.21.15.43:0/438677025 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00b2f30 con 0x7f00a006e400 2024-01-16T15:25:50.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.832+0000 7f00a6600640 1 --2- 172.21.15.43:0/438677025 >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 0x7f00a015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:25:50.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.833+0000 7f00a5c00640 1 --2- 172.21.15.43:0/438677025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0090009870 tx=0x7f009002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=277a64eb0b0618b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f009fe00640 1 -- 172.21.15.43:0/438677025 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090031640 con 0x7f00a006e400 2024-01-16T15:25:50.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f009fe00640 1 -- 172.21.15.43:0/438677025 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00900317a0 con 0x7f00a006e400 2024-01-16T15:25:50.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f009fe00640 1 -- 172.21.15.43:0/438677025 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090031b30 con 0x7f00a006e400 2024-01-16T15:25:50.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 -- 172.21.15.43:0/438677025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 msgr2=0x7f00a00b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 --2- 172.21.15.43:0/438677025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0090009870 tx=0x7f009002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 -- 172.21.15.43:0/438677025 shutdown_connections 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 --2- 172.21.15.43:0/438677025 >> v2:172.21.15.69:3300/0 conn(0x7f00a006bb60 0x7f00a015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 --2- 172.21.15.43:0/438677025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a00b3730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.834+0000 7f00a8600640 1 -- 172.21.15.43:0/438677025 >> 172.21.15.43:0/438677025 conn(0x7f00a00cfd60 msgr2=0x7f00a0070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.835+0000 7f00a8600640 1 -- 172.21.15.43:0/438677025 shutdown_connections 2024-01-16T15:25:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.835+0000 7f00a8600640 1 -- 172.21.15.43:0/438677025 wait complete. 2024-01-16T15:25:50.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.836+0000 7f00a8600640 1 Processor -- start 2024-01-16T15:25:50.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.836+0000 7f00a8600640 1 -- start start 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00a006bb60 0x7f00a01f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a0156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a006e7e0 0x7f00a00ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a01f3b90 con 0x7f00a006bb60 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00c6100 con 0x7f00a006e7e0 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.837+0000 7f00a8600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0157050 con 0x7f00a006e400 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00a006bb60 0x7f00a01f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00a006bb60 0x7f00a01f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46416/0 (socket says 172.21.15.43:46416) 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 -- 172.21.15.43:0/132291246 learned_addr learned my addr 172.21.15.43:0/132291246 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a6600640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a006e7e0 0x7f00a00ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5200640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a0156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 -- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 msgr2=0x7f00a0156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00a006e400 0x7f00a0156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 -- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a006e7e0 msgr2=0x7f00a00ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00a006e7e0 0x7f00a00ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.838+0000 7f00a5c00640 1 -- 172.21.15.43:0/132291246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098009a70 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f00a5c00640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00a006bb60 0x7f00a01f3650 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0090000c00 tx=0x7f00900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009003a930 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009003aa90 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f00a8600640 1 -- 172.21.15.43:0/132291246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00900094d0 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009003ae40 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.839+0000 7f00a8600640 1 -- 172.21.15.43:0/132291246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a00c5860 con 0x7f00a006bb60 2024-01-16T15:25:50.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.840+0000 7f00a8600640 1 -- 172.21.15.43:0/132291246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0167750 con 0x7f00a006bb60 2024-01-16T15:25:50.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.842+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f009002f050 con 0x7f00a006bb60 2024-01-16T15:25:50.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.842+0000 7f009d600640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f007806c700 0x7f007806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.843+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00900aa6d0 con 0x7f00a006bb60 2024-01-16T15:25:50.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.843+0000 7f00a5200640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f007806c700 0x7f007806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.844+0000 7f00a5200640 1 --2- 172.21.15.43:0/132291246 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f007806c700 0x7f007806ebc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f00a00cba00 tx=0x7f0088009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:50.847+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0090076f90 con 0x7f00a006bb60 2024-01-16T15:25:51.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:51.078+0000 7f00a8600640 1 -- 172.21.15.43:0/132291246 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00a0072620 con 0x7f007806c700 2024-01-16T15:25:51.084 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:51.086+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:51.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:51.086+0000 7f009d600640 1 -- 172.21.15.43:0/132291246 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f00a0072620 con 0x7f007806c700 2024-01-16T15:25:51.084 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:53.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:53.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.337+0000 7f49d3e00640 1 Processor -- start 2024-01-16T15:25:53.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.338+0000 7f49d3e00640 1 -- start start 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49d406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49d413fe10 0x7f49d40cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d4140430 con 0x7f49d406d910 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d406dd90 con 0x7f49d413fe10 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4071220 con 0x7f49d413cf40 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d2000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f49d406d910 0x7f49d413f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49192/0 (socket says 172.21.15.43:49192) 2024-01-16T15:25:53.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d2000640 1 -- 172.21.15.43:0/3481459059 learned_addr learned my addr 172.21.15.43:0/3481459059 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d2a00640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49d406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d3400640 1 --2- 172.21.15.43:0/3481459059 >> v2:172.21.15.69:3300/0 conn(0x7f49d413fe10 0x7f49d40cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.340+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673818974 0 0) 0x7f49d4140430 con 0x7f49d406d910 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b4003520 con 0x7f49d406d910 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339389847 0 0) 0x7f49b4003520 con 0x7f49d406d910 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 msgr2=0x7f49d406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:53.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49d406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 >> v2:172.21.15.69:3300/0 conn(0x7f49d413fe10 msgr2=0x7f49d40cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 --2- 172.21.15.43:0/3481459059 >> v2:172.21.15.69:3300/0 conn(0x7f49d413fe10 0x7f49d40cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4071500 con 0x7f49d406d910 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f49bc0017f0 con 0x7f49d406d910 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 >> v1:172.21.15.43:6789/0 conn(0x7f49d406d910 legacy=0x7f49d413f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49b40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49b4004d00 0x7f49b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d406dd90 con 0x7f49b4003f60 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4071220 con 0x7f49b4004d00 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400dbc0 con 0x7f49d413cf40 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.341+0000 7f49d2000640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d3400640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49b4004d00 0x7f49b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2a00640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49b40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2000640 1 -- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 msgr2=0x7f49b40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:53.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2000640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d413cf40 0x7f49b40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2000640 1 -- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49b4004d00 msgr2=0x7f49b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:53.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2000640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49b4004d00 0x7f49b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.342+0000 7f49d2000640 1 -- 172.21.15.43:0/3481459059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4003520 con 0x7f49b4003f60 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.343+0000 7f49d2000640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49b40047c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c190 tx=0x7f49b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d6da6c69e418cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.343+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc007270 con 0x7f49b4003f60 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.343+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc0073d0 con 0x7f49b4003f60 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.343+0000 7f49d1600640 1 -- 172.21.15.43:0/3481459059 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc02c8a0 con 0x7f49b4003f60 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 -- 172.21.15.43:0/3481459059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 msgr2=0x7f49b40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 --2- 172.21.15.43:0/3481459059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49b40047c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c190 tx=0x7f49b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3400640 1 -- 172.21.15.43:0/3481459059 reap_dead start 2024-01-16T15:25:53.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 -- 172.21.15.43:0/3481459059 shutdown_connections 2024-01-16T15:25:53.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 -- 172.21.15.43:0/3481459059 >> 172.21.15.43:0/3481459059 conn(0x7f49d40b0aa0 msgr2=0x7f49d40b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:53.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 -- 172.21.15.43:0/3481459059 shutdown_connections 2024-01-16T15:25:53.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.344+0000 7f49d3e00640 1 -- 172.21.15.43:0/3481459059 wait complete. 2024-01-16T15:25:53.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.346+0000 7f49d3e00640 1 Processor -- start 2024-01-16T15:25:53.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.346+0000 7f49d3e00640 1 -- start start 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49d413f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d4154b70 0x7f49d412ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49d4159090 0x7f49d4159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4155e60 con 0x7f49b4003f60 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4154ff0 con 0x7f49d4159090 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d3e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4159a50 con 0x7f49d4154b70 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.347+0000 7f49d2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49d413f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49d413f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44394/0 (socket says 172.21.15.43:44394) 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 -- 172.21.15.43:0/3640575217 learned_addr learned my addr 172.21.15.43:0/3640575217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2000640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d4154b70 0x7f49d412ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d3400640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49d4159090 0x7f49d4159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 -- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d4154b70 msgr2=0x7f49d412ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:53.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49d4154b70 0x7f49d412ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 -- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49d4159090 msgr2=0x7f49d4159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:53.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49d4159090 0x7f49d4159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:53.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 -- 172.21.15.43:0/3640575217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4072160 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.348+0000 7f49d2a00640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49b4003f60 0x7f49d413f680 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f49c4007f30 tx=0x7f49c4011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.349+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400e030 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.349+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c4016470 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.349+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4016820 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.349+0000 7f49d3e00640 1 -- 172.21.15.43:0/3640575217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d41568b0 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.349+0000 7f49d3e00640 1 -- 172.21.15.43:0/3640575217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d41554c0 con 0x7f49b4003f60 2024-01-16T15:25:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.350+0000 7f49d3e00640 1 -- 172.21.15.43:0/3640575217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d404e840 con 0x7f49b4003f60 2024-01-16T15:25:53.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.351+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f49c4016980 con 0x7f49b4003f60 2024-01-16T15:25:53.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.352+0000 7f49caa00640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49ac06c730 0x7f49ac06ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:53.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.352+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f49c400f070 con 0x7f49b4003f60 2024-01-16T15:25:53.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.352+0000 7f49d2000640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49ac06c730 0x7f49ac06ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:53.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.353+0000 7f49d2000640 1 --2- 172.21.15.43:0/3640575217 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f49ac06c730 0x7f49ac06ebf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f49bc034000 tx=0x7f49bc007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:53.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.357+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49c4056070 con 0x7f49b4003f60 2024-01-16T15:25:53.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.589+0000 7f49d3e00640 1 -- 172.21.15.43:0/3640575217 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49d4071fe0 con 0x7f49ac06c730 2024-01-16T15:25:53.597 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:53.600+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:53.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:53.600+0000 7f49caa00640 1 -- 172.21.15.43:0/3640575217 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f49bc0075c0 con 0x7f49ac06c730 2024-01-16T15:25:53.598 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:53.598 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:55.614 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:55.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.855+0000 7f7cb3000640 1 Processor -- start 2024-01-16T15:25:55.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.855+0000 7f7cb3000640 1 -- start start 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cb3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7cac0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cb3000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac157b30 con 0x7f7cac15aae0 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cb3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac0b34f0 con 0x7f7cac0b2d70 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cb3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac15aec0 con 0x7f7cac06d470 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cabe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7cac15aae0 0x7f7cac06cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49204/0 (socket says 172.21.15.43:49204) 2024-01-16T15:25:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.857+0000 7f7cabe00640 1 -- 172.21.15.43:0/2737602307 learned_addr learned my addr 172.21.15.43:0/2737602307 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7cab400640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7cac0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327785600 0 0) 0x7f7cac157b30 con 0x7f7cac15aae0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c8c005730 con 0x7f7cac15aae0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765955126 0 0) 0x7f7c8c005730 con 0x7f7cac15aae0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 msgr2=0x7f7cac0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7cac0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 >> v1:172.21.15.69:6789/0 conn(0x7f7cac0b2d70 legacy=0x7f7cac158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0b37d0 con 0x7f7cac15aae0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f7c9c001670 con 0x7f7cac15aae0 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 >> v1:172.21.15.43:6789/0 conn(0x7f7cac15aae0 legacy=0x7f7cac06cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c8c005bb0 0x7f7c8c005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7c8c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c006f10 0x7f7c8c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac15aec0 con 0x7f7c8c005bb0 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c00dbc0 con 0x7f7c8c006f10 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.858+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c00dd80 con 0x7f7cac06d470 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c8c005bb0 0x7f7c8c005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cab400640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7c8c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cb1000640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c006f10 0x7f7c8c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 -- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 msgr2=0x7f7c8c0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7cac06d470 0x7f7c8c0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 -- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c006f10 msgr2=0x7f7c8c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c006f10 0x7f7c8c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.859+0000 7f7cabe00640 1 -- 172.21.15.43:0/2737602307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c005730 con 0x7f7c8c005bb0 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.860+0000 7f7cabe00640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c8c005bb0 0x7f7c8c005f90 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00c210 tx=0x7f7c8c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13249ab2c3bea608 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c005ad0 con 0x7f7c8c005bb0 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c005c50 con 0x7f7c8c005bb0 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7caaa00640 1 -- 172.21.15.43:0/2737602307 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c02c830 con 0x7f7c8c005bb0 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7cb3000640 1 -- 172.21.15.43:0/2737602307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c8c005bb0 msgr2=0x7f7c8c005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7cb3000640 1 --2- 172.21.15.43:0/2737602307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c8c005bb0 0x7f7c8c005f90 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00c210 tx=0x7f7c8c007a00 comp rx=0 tx=0).stop 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.861+0000 7f7cb1000640 1 -- 172.21.15.43:0/2737602307 reap_dead start 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.862+0000 7f7cb3000640 1 -- 172.21.15.43:0/2737602307 shutdown_connections 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.862+0000 7f7cb3000640 1 -- 172.21.15.43:0/2737602307 >> 172.21.15.43:0/2737602307 conn(0x7f7cac0b0ba0 msgr2=0x7f7cac0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.862+0000 7f7cb3000640 1 -- 172.21.15.43:0/2737602307 shutdown_connections 2024-01-16T15:25:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.862+0000 7f7cb3000640 1 -- 172.21.15.43:0/2737602307 wait complete. 2024-01-16T15:25:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.864+0000 7f7cb3000640 1 Processor -- start 2024-01-16T15:25:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.864+0000 7f7cb3000640 1 -- start start 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c8c005bb0 0x7f7cac14c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cac14f360 0x7f7cac14b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7cac14f7e0 0x7f7cac14b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac14be20 con 0x7f7cac14f7e0 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac14c2f0 con 0x7f7cac14f360 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac14d3d0 con 0x7f7c8c005bb0 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7cac14f7e0 0x7f7cac14b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7cac14f7e0 0x7f7cac14b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44412/0 (socket says 172.21.15.43:44412) 2024-01-16T15:25:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cb1000640 1 -- 172.21.15.43:0/4200016961 learned_addr learned my addr 172.21.15.43:0/4200016961 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cab400640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cac14f360 0x7f7cac14b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.866+0000 7f7cabe00640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c8c005bb0 0x7f7cac14c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 -- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c8c005bb0 msgr2=0x7f7cac14c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c8c005bb0 0x7f7cac14c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 -- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cac14f360 msgr2=0x7f7cac14b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cac14f360 0x7f7cac14b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 -- 172.21.15.43:0/4200016961 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac14d6b0 con 0x7f7cac14f7e0 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cab400640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cac14f360 0x7f7cac14b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7cb1000640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7cac14f7e0 0x7f7cac14b8e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0b5100 tx=0x7f7ca400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4003970 con 0x7f7cac14f7e0 2024-01-16T15:25:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca4003ad0 con 0x7f7cac14f7e0 2024-01-16T15:25:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.867+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4003e80 con 0x7f7cac14f7e0 2024-01-16T15:25:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.868+0000 7f7cb3000640 1 -- 172.21.15.43:0/4200016961 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac14d990 con 0x7f7cac14f7e0 2024-01-16T15:25:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.868+0000 7f7cb3000640 1 -- 172.21.15.43:0/4200016961 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac09ae80 con 0x7f7cac14f7e0 2024-01-16T15:25:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.869+0000 7f7cb3000640 1 -- 172.21.15.43:0/4200016961 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac09b0a0 con 0x7f7cac14f7e0 2024-01-16T15:25:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.870+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f7ca40050d0 con 0x7f7cac14f7e0 2024-01-16T15:25:55.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.871+0000 7f7ca3e00640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c8406c700 0x7f7c8406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.871+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7ca4088b60 con 0x7f7cac14f7e0 2024-01-16T15:25:55.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.872+0000 7f7cabe00640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c8406c700 0x7f7c8406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.872+0000 7f7cabe00640 1 --2- 172.21.15.43:0/4200016961 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7c8406c700 0x7f7c8406ebc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c034000 tx=0x7f7c9c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:55.876+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ca4055050 con 0x7f7cac14f7e0 2024-01-16T15:25:56.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:56.107+0000 7f7cb3000640 1 -- 172.21.15.43:0/4200016961 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7cac0b3200 con 0x7f7c8406c700 2024-01-16T15:25:56.120 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:56.123+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:56.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:56.123+0000 7f7ca3e00640 1 -- 172.21.15.43:0/4200016961 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7c9c001430 con 0x7f7c8406c700 2024-01-16T15:25:56.121 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:56.121 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:58.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.386+0000 7f6e13200640 1 Processor -- start 2024-01-16T15:25:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.386+0000 7f6e13200640 1 -- start start 2024-01-16T15:25:58.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.388+0000 7f6e13200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6e1415bda0 0x7f6e14165ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.388+0000 7f6e13200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e14071080 con 0x7f6e141448c0 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.388+0000 7f6e13200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1415c1f0 con 0x7f6e1406dae0 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.388+0000 7f6e13200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e14144ca0 con 0x7f6e1415bda0 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e12800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6e1415bda0 0x7f6e14165ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e11e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6e141448c0 0x7f6e1406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49208/0 (socket says 172.21.15.43:49208) 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e11e00640 1 -- 172.21.15.43:0/4247079578 learned_addr learned my addr 172.21.15.43:0/4247079578 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e12800640 1 -- 172.21.15.43:0/4247079578 >> v1:172.21.15.69:6789/0 conn(0x7f6e1406dae0 legacy=0x7f6e1415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e12800640 1 -- 172.21.15.43:0/4247079578 >> v1:172.21.15.43:6789/0 conn(0x7f6e141448c0 legacy=0x7f6e1406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.389+0000 7f6e12800640 1 -- 172.21.15.43:0/4247079578 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1415c450 con 0x7f6e1415bda0 2024-01-16T15:25:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.390+0000 7f6e12800640 1 --2- 172.21.15.43:0/4247079578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1415bda0 0x7f6e14165ce0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009d90 tx=0x7f6e0c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39078e63da0d12b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:58.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.391+0000 7f6e11400640 1 -- 172.21.15.43:0/4247079578 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c02cc60 con 0x7f6e1415bda0 2024-01-16T15:25:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.391+0000 7f6e11400640 1 -- 172.21.15.43:0/4247079578 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c02cdc0 con 0x7f6e1415bda0 2024-01-16T15:25:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.391+0000 7f6e11400640 1 -- 172.21.15.43:0/4247079578 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c034660 con 0x7f6e1415bda0 2024-01-16T15:25:58.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.391+0000 7f6e13200640 1 -- 172.21.15.43:0/4247079578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1415bda0 msgr2=0x7f6e14165ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.391+0000 7f6e13200640 1 --2- 172.21.15.43:0/4247079578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1415bda0 0x7f6e14165ce0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009d90 tx=0x7f6e0c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.392+0000 7f6e13200640 1 -- 172.21.15.43:0/4247079578 shutdown_connections 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.392+0000 7f6e13200640 1 --2- 172.21.15.43:0/4247079578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1415bda0 0x7f6e14165ce0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.392+0000 7f6e13200640 1 -- 172.21.15.43:0/4247079578 >> 172.21.15.43:0/4247079578 conn(0x7f6e140b0aa0 msgr2=0x7f6e140b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.392+0000 7f6e13200640 1 -- 172.21.15.43:0/4247079578 shutdown_connections 2024-01-16T15:25:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.392+0000 7f6e13200640 1 -- 172.21.15.43:0/4247079578 wait complete. 2024-01-16T15:25:58.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.393+0000 7f6e13200640 1 Processor -- start 2024-01-16T15:25:58.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.394+0000 7f6e13200640 1 -- start start 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1406dae0 0x7f6e1413c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e141448c0 0x7f6e14158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1415bda0 0x7f6e141371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141440c0 con 0x7f6e1406dae0 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14140280 con 0x7f6e141448c0 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e13200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14140560 con 0x7f6e1415bda0 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1406dae0 0x7f6e1413c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1406dae0 0x7f6e1413c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44430/0 (socket says 172.21.15.43:44430) 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 -- 172.21.15.43:0/1885046499 learned_addr learned my addr 172.21.15.43:0/1885046499 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e0be00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e141448c0 0x7f6e14158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 -- 172.21.15.43:0/1885046499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1415bda0 msgr2=0x7f6e141371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e1415bda0 0x7f6e141371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 -- 172.21.15.43:0/1885046499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e141448c0 msgr2=0x7f6e14158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e141448c0 0x7f6e14158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.395+0000 7f6e11e00640 1 -- 172.21.15.43:0/1885046499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c009a70 con 0x7f6e1406dae0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e11e00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e1406dae0 0x7f6e1413c110 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400c4a0 tx=0x7f6e0400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04005ce0 con 0x7f6e1406dae0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e04005e40 con 0x7f6e1406dae0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04007270 con 0x7f6e1406dae0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e13200640 1 -- 172.21.15.43:0/1885046499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14140840 con 0x7f6e1406dae0 2024-01-16T15:25:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.396+0000 7f6e13200640 1 -- 172.21.15.43:0/1885046499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14140e00 con 0x7f6e1406dae0 2024-01-16T15:25:58.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.397+0000 7f6e13200640 1 -- 172.21.15.43:0/1885046499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e14141cb0 con 0x7f6e1406dae0 2024-01-16T15:25:58.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.398+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f6e04007490 con 0x7f6e1406dae0 2024-01-16T15:25:58.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.399+0000 7f6e0a000640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6dec06c700 0x7f6dec06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:58.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.399+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6e04087cb0 con 0x7f6e1406dae0 2024-01-16T15:25:58.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.399+0000 7f6e0be00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6dec06c700 0x7f6dec06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:58.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.400+0000 7f6e0be00640 1 --2- 172.21.15.43:0/1885046499 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6dec06c700 0x7f6dec06ebc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6e14143880 tx=0x7f6dfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:58.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.403+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e04054620 con 0x7f6e1406dae0 2024-01-16T15:25:58.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.632+0000 7f6e13200640 1 -- 172.21.15.43:0/1885046499 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e14071b70 con 0x7f6dec06c700 2024-01-16T15:25:58.643 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:25:58.646+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:58.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:25:58.646+0000 7f6e0a000640 1 -- 172.21.15.43:0/1885046499 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6dfc00d4f0 con 0x7f6dec06c700 2024-01-16T15:25:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:25:58.644 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:00.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.902+0000 7f7005600640 1 Processor -- start 2024-01-16T15:26:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.902+0000 7f7005600640 1 -- start start 2024-01-16T15:26:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.904+0000 7f7005600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f7000164e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.904+0000 7f7005600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7000140430 con 0x7f700013cf40 2024-01-16T15:26:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.904+0000 7f7005600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7000071220 con 0x7f700006dae0 2024-01-16T15:26:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.904+0000 7f7005600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7000071500 con 0x7f700013fe10 2024-01-16T15:26:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6fffe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f700013cf40 0x7f700006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49224/0 (socket says 172.21.15.43:49224) 2024-01-16T15:26:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6fffe00640 1 -- 172.21.15.43:0/2874040197 learned_addr learned my addr 172.21.15.43:0/2874040197 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f7004c00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f7000164e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510816127 0 0) 0x7f7000140430 con 0x7f700013cf40 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0003520 con 0x7f700013cf40 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525247807 0 0) 0x7f6fe0003520 con 0x7f700013cf40 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 msgr2=0x7f7000164e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f7000164e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 >> v1:172.21.15.69:6789/0 conn(0x7f700006dae0 legacy=0x7f700013f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.905+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70000717e0 con 0x7f700013cf40 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f6ff00019f0 con 0x7f700013cf40 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 >> v1:172.21.15.43:6789/0 conn(0x7f700013cf40 legacy=0x7f700006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0003a10 0x7f6fe0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f6fe00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fe0004d00 0x7f6fe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7000071500 con 0x7f6fe0004d00 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe000ba90 con 0x7f6fe0003a10 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe000de60 con 0x7f700013fe10 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fe0004d00 0x7f6fe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6fffe00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0003a10 0x7f6fe0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 -- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 msgr2=0x7f6fe00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f6fff400640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f6fe00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f6fe00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 -- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0003a10 msgr2=0x7f6fe0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0003a10 0x7f6fe0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.906+0000 7f7004c00640 1 -- 172.21.15.43:0/2874040197 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0003520 con 0x7f6fe0004d00 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.907+0000 7f6fff400640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f700013fe10 0x7f6fe00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.907+0000 7f6fffe00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0003a10 0x7f6fe0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f7004c00640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fe0004d00 0x7f6fe0005180 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009a90 tx=0x7f6fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3314c85ce68a3994 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff802c680 con 0x7f6fe0004d00 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff802c7e0 con 0x7f6fe0004d00 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f6ffea00640 1 -- 172.21.15.43:0/2874040197 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff802cb70 con 0x7f6fe0004d00 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f7005600640 1 -- 172.21.15.43:0/2874040197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fe0004d00 msgr2=0x7f6fe0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.908+0000 7f7005600640 1 --2- 172.21.15.43:0/2874040197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6fe0004d00 0x7f6fe0005180 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009a90 tx=0x7f6fe00057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.909+0000 7f7004c00640 1 -- 172.21.15.43:0/2874040197 reap_dead start 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.909+0000 7f7005600640 1 -- 172.21.15.43:0/2874040197 shutdown_connections 2024-01-16T15:26:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.909+0000 7f7005600640 1 -- 172.21.15.43:0/2874040197 >> 172.21.15.43:0/2874040197 conn(0x7f70000b0aa0 msgr2=0x7f70000b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.909+0000 7f7005600640 1 -- 172.21.15.43:0/2874040197 shutdown_connections 2024-01-16T15:26:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.909+0000 7f7005600640 1 -- 172.21.15.43:0/2874040197 wait complete. 2024-01-16T15:26:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.911+0000 7f7005600640 1 Processor -- start 2024-01-16T15:26:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.911+0000 7f7005600640 1 -- start start 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0004d00 0x7f700013bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70000b00e0 0x7f7000141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700013b730 0x7f7000153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70000a8a20 con 0x7f700013b730 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7000141b50 con 0x7f6fe0004d00 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7005600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70000a8460 con 0x7f70000b00e0 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7004c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700013b730 0x7f7000153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7004c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700013b730 0x7f7000153540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44456/0 (socket says 172.21.15.43:44456) 2024-01-16T15:26:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.912+0000 7f7004c00640 1 -- 172.21.15.43:0/3268534186 learned_addr learned my addr 172.21.15.43:0/3268534186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f6fffe00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0004d00 0x7f700013bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f6fff400640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70000b00e0 0x7f7000141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 -- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70000b00e0 msgr2=0x7f7000141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70000b00e0 0x7f7000141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 -- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0004d00 msgr2=0x7f700013bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0004d00 0x7f700013bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 -- 172.21.15.43:0/3268534186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000072160 con 0x7f700013b730 2024-01-16T15:26:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.913+0000 7f7004c00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f700013b730 0x7f7000153540 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009bb0 tx=0x7f6ff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.914+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff802cd60 con 0x7f700013b730 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.914+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8003610 con 0x7f700013b730 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.914+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80039c0 con 0x7f700013b730 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.914+0000 7f7005600640 1 -- 172.21.15.43:0/3268534186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70000a8740 con 0x7f700013b730 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.914+0000 7f7005600640 1 -- 172.21.15.43:0/3268534186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7000142520 con 0x7f700013b730 2024-01-16T15:26:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.915+0000 7f7005600640 1 -- 172.21.15.43:0/3268534186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700004e840 con 0x7f700013b730 2024-01-16T15:26:00.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.916+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f6ff8003b20 con 0x7f700013b730 2024-01-16T15:26:00.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.917+0000 7f6ff7e00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fd806c730 0x7f6fd806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.917+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6ff8000e40 con 0x7f700013b730 2024-01-16T15:26:00.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.917+0000 7f6fffe00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fd806c730 0x7f6fd806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.918+0000 7f6fffe00640 1 --2- 172.21.15.43:0/3268534186 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6fd806c730 0x7f6fd806ebf0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005e10 tx=0x7f6ff0001870 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:00.921+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ff8076160 con 0x7f700013b730 2024-01-16T15:26:01.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:01.148+0000 7f7005600640 1 -- 172.21.15.43:0/3268534186 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7000071fe0 con 0x7f6fd806c730 2024-01-16T15:26:01.159 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:01.161+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:01.162+0000 7f6ff7e00640 1 -- 172.21.15.43:0/3268534186 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ff000d070 con 0x7f6fd806c730 2024-01-16T15:26:01.160 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:01.160 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:02.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:02.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:03.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:03.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.418+0000 7fdba2600640 1 Processor -- start 2024-01-16T15:26:03.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.418+0000 7fdba2600640 1 -- start start 2024-01-16T15:26:03.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.419+0000 7fdba2600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb9c0d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.419+0000 7fdba2600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb9c0cdbb0 0x7fdb9c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.419+0000 7fdba2600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9c0cef30 con 0x7fdb9c15d340 2024-01-16T15:26:03.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.419+0000 7fdba2600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb9c0d11a0 con 0x7fdb9c0cdbb0 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.419+0000 7fdba2600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0ce320 con 0x7fdb9c0d1d90 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdba1200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdb9c15d340 0x7fdb9c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41522/0 (socket says 172.21.15.43:41522) 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdba1200640 1 -- 172.21.15.43:0/3091045307 learned_addr learned my addr 172.21.15.43:0/3091045307 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb9c0d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdba1c00640 1 --2- 172.21.15.43:0/3091045307 >> v2:172.21.15.69:3300/0 conn(0x7fdb9c0cdbb0 0x7fdb9c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480446285 0 0) 0x7fdb9c0cef30 con 0x7fdb9c15d340 2024-01-16T15:26:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.420+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c003520 con 0x7fdb9c15d340 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129731430 0 0) 0x7fdb7c003520 con 0x7fdb9c15d340 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 msgr2=0x7fdb9c0d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb9c0d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 >> v2:172.21.15.69:3300/0 conn(0x7fdb9c0cdbb0 msgr2=0x7fdb9c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 --2- 172.21.15.43:0/3091045307 >> v2:172.21.15.69:3300/0 conn(0x7fdb9c0cdbb0 0x7fdb9c0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c0ce600 con 0x7fdb9c15d340 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fdb8c0019f0 con 0x7fdb9c15d340 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 >> v1:172.21.15.43:6789/0 conn(0x7fdb9c15d340 legacy=0x7fdb9c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:03.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb7c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb7c003fb0 0x7fdb7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c004d00 0x7fdb7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0d11a0 con 0x7fdb7c003fb0 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0ce320 con 0x7fdb7c004d00 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c00ba00 con 0x7fdb9c0d1d90 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.421+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb7c003fb0 0x7fdb7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdba1200640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb7c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdba1c00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c004d00 0x7fdb7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdb9be00640 1 -- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 msgr2=0x7fdb7c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb7c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdb9be00640 1 -- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c004d00 msgr2=0x7fdb7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c004d00 0x7fdb7c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdb9be00640 1 -- 172.21.15.43:0/3091045307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c003520 con 0x7fdb7c003fb0 2024-01-16T15:26:03.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.422+0000 7fdba1200640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9c0d1d90 0x7fdb7c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:03.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.423+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb7c003fb0 0x7fdb7c0047c0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009ac0 tx=0x7fdb7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc17b11bbeb4d92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:03.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.423+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402c6b0 con 0x7fdb7c003fb0 2024-01-16T15:26:03.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.423+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8402c810 con 0x7fdb7c003fb0 2024-01-16T15:26:03.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.423+0000 7fdb9b400640 1 -- 172.21.15.43:0/3091045307 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402cba0 con 0x7fdb7c003fb0 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 -- 172.21.15.43:0/3091045307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb7c003fb0 msgr2=0x7fdb7c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 --2- 172.21.15.43:0/3091045307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb7c003fb0 0x7fdb7c0047c0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009ac0 tx=0x7fdb7c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba1c00640 1 -- 172.21.15.43:0/3091045307 reap_dead start 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 -- 172.21.15.43:0/3091045307 shutdown_connections 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 -- 172.21.15.43:0/3091045307 >> 172.21.15.43:0/3091045307 conn(0x7fdb9c09c370 msgr2=0x7fdb9c0d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 -- 172.21.15.43:0/3091045307 shutdown_connections 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.424+0000 7fdba2600640 1 -- 172.21.15.43:0/3091045307 wait complete. 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.426+0000 7fdba2600640 1 Processor -- start 2024-01-16T15:26:03.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.426+0000 7fdba2600640 1 -- start start 2024-01-16T15:26:03.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb7c003fb0 0x7fdb9c157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb9c0b3ab0 0x7fdb9c0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb9c0c8fe0 0x7fdb9c0b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b8df0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b2a10 con 0x7fdb9c0b3ab0 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba2600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b2cf0 con 0x7fdb7c003fb0 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb9c0c8fe0 0x7fdb9c0b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb9c0c8fe0 0x7fdb9c0b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60498/0 (socket says 172.21.15.43:60498) 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.427+0000 7fdba1c00640 1 -- 172.21.15.43:0/3300050773 learned_addr learned my addr 172.21.15.43:0/3300050773 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1200640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb7c003fb0 0x7fdb9c157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdb9be00640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb9c0b3ab0 0x7fdb9c0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 -- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb7c003fb0 msgr2=0x7fdb9c157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:03.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb7c003fb0 0x7fdb9c157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 -- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb9c0b3ab0 msgr2=0x7fdb9c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb9c0b3ab0 0x7fdb9c0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 -- 172.21.15.43:0/3300050773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c0b2fd0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdba1c00640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb9c0c8fe0 0x7fdb9c0b4b40 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c160ee0 tx=0x7fdb94010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.428+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9400e8a0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.429+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9400ea00 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.429+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9400edb0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.429+0000 7fdba2600640 1 -- 172.21.15.43:0/3300050773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9c0b32b0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.429+0000 7fdba2600640 1 -- 172.21.15.43:0/3300050773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c0b26c0 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.430+0000 7fdba2600640 1 -- 172.21.15.43:0/3300050773 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c067d70 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.431+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fdb94020410 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.432+0000 7fdb98c00640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb7406c730 0x7fdb7406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:03.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.432+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdb94088600 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.432+0000 7fdba1200640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb7406c730 0x7fdb7406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:03.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.433+0000 7fdba1200640 1 --2- 172.21.15.43:0/3300050773 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdb7406c730 0x7fdb7406ebf0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c007c40 tx=0x7fdb8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:03.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.437+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb94054a60 con 0x7fdb9c0c8fe0 2024-01-16T15:26:03.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.662+0000 7fdba2600640 1 -- 172.21.15.43:0/3300050773 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb9c0b2e60 con 0x7fdb7406c730 2024-01-16T15:26:03.668 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:03.670+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:03.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:03.670+0000 7fdb98c00640 1 -- 172.21.15.43:0/3300050773 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdb9c0b2e60 con 0x7fdb7406c730 2024-01-16T15:26:03.669 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:03.669 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:05.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:05.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.924+0000 7f8bcf200640 1 Processor -- start 2024-01-16T15:26:05.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.924+0000 7f8bcf200640 1 -- start start 2024-01-16T15:26:05.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.925+0000 7f8bcf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8bc806dae0 0x7f8bc815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.925+0000 7f8bcf200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8bc815bbd0 0x7f8bc8070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.925+0000 7f8bcf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc8144ca0 con 0x7f8bc81448c0 2024-01-16T15:26:05.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bcf200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc815c450 con 0x7f8bc815bbd0 2024-01-16T15:26:05.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bcf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80715b0 con 0x7f8bc806dae0 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8bc815bbd0 0x7f8bc8070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8bc815bbd0 0x7f8bc8070070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60514/0 (socket says 172.21.15.43:60514) 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 -- 172.21.15.43:0/2744479762 learned_addr learned my addr 172.21.15.43:0/2744479762 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bcd400640 1 --2- 172.21.15.43:0/2744479762 >> v2:172.21.15.69:3300/0 conn(0x7f8bc806dae0 0x7f8bc815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 -- 172.21.15.43:0/2744479762 >> v2:172.21.15.69:3300/0 conn(0x7f8bc806dae0 msgr2=0x7f8bc815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 --2- 172.21.15.43:0/2744479762 >> v2:172.21.15.69:3300/0 conn(0x7f8bc806dae0 0x7f8bc815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.926+0000 7f8bce800640 1 -- 172.21.15.43:0/2744479762 >> v1:172.21.15.69:6789/0 conn(0x7f8bc81448c0 legacy=0x7f8bc806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.927+0000 7f8bce800640 1 -- 172.21.15.43:0/2744479762 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8071890 con 0x7f8bc815bbd0 2024-01-16T15:26:05.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.928+0000 7f8bce800640 1 --2- 172.21.15.43:0/2744479762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc815bbd0 0x7f8bc8070070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b1a90 tx=0x7f8bc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688c51958650c933 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.928+0000 7f8bcca00640 1 -- 172.21.15.43:0/2744479762 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0031870 con 0x7f8bc815bbd0 2024-01-16T15:26:05.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.928+0000 7f8bcca00640 1 -- 172.21.15.43:0/2744479762 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc00319d0 con 0x7f8bc815bbd0 2024-01-16T15:26:05.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.928+0000 7f8bcca00640 1 -- 172.21.15.43:0/2744479762 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0031d60 con 0x7f8bc815bbd0 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 -- 172.21.15.43:0/2744479762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc815bbd0 msgr2=0x7f8bc8070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 --2- 172.21.15.43:0/2744479762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc815bbd0 0x7f8bc8070070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b1a90 tx=0x7f8bc002b910 comp rx=0 tx=0).stop 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 -- 172.21.15.43:0/2744479762 shutdown_connections 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 --2- 172.21.15.43:0/2744479762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc815bbd0 0x7f8bc8070070 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 --2- 172.21.15.43:0/2744479762 >> v2:172.21.15.69:3300/0 conn(0x7f8bc806dae0 0x7f8bc815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 -- 172.21.15.43:0/2744479762 >> 172.21.15.43:0/2744479762 conn(0x7f8bc80b0aa0 msgr2=0x7f8bc80b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 -- 172.21.15.43:0/2744479762 shutdown_connections 2024-01-16T15:26:05.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.929+0000 7f8bcf200640 1 -- 172.21.15.43:0/2744479762 wait complete. 2024-01-16T15:26:05.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.931+0000 7f8bcf200640 1 Processor -- start 2024-01-16T15:26:05.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.931+0000 7f8bcf200640 1 -- start start 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc806dae0 0x7f8bc8141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bc81448c0 0x7f8bc815dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bc815bbd0 0x7f8bc80ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc814c090 con 0x7f8bc806dae0 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc814b410 con 0x7f8bc815bbd0 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc814b6f0 con 0x7f8bc81448c0 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc806dae0 0x7f8bc8141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc806dae0 0x7f8bc8141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60524/0 (socket says 172.21.15.43:60524) 2024-01-16T15:26:05.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.932+0000 7f8bcde00640 1 -- 172.21.15.43:0/3723893327 learned_addr learned my addr 172.21.15.43:0/3723893327 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcd400640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bc81448c0 0x7f8bc815dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bce800640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bc815bbd0 0x7f8bc80ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 -- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bc81448c0 msgr2=0x7f8bc815dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bc81448c0 0x7f8bc815dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 -- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bc815bbd0 msgr2=0x7f8bc80ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bc815bbd0 0x7f8bc80ada20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 -- 172.21.15.43:0/3723893327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc00094d0 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcd400640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bc81448c0 0x7f8bc815dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bce800640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bc815bbd0 0x7f8bc80ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcde00640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8bc806dae0 0x7f8bc8141eb0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800c7c0 tx=0x7f8bb800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb800aa50 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb800abb0 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8012420 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcf200640 1 -- 172.21.15.43:0/3723893327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc80724f0 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.933+0000 7f8bcf200640 1 -- 172.21.15.43:0/3723893327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc814bcb0 con 0x7f8bc806dae0 2024-01-16T15:26:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.934+0000 7f8bcf200640 1 -- 172.21.15.43:0/3723893327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc804e840 con 0x7f8bc806dae0 2024-01-16T15:26:05.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.936+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f8bb8012580 con 0x7f8bc806dae0 2024-01-16T15:26:05.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.936+0000 7f8bc6000640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ba006c700 0x7f8ba006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.937+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8bb8088f00 con 0x7f8bc806dae0 2024-01-16T15:26:05.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.937+0000 7f8bcd400640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ba006c700 0x7f8ba006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.938+0000 7f8bcd400640 1 --2- 172.21.15.43:0/3723893327 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f8ba006c700 0x7f8ba006ebc0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0006fd0 tx=0x7f8bb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:05.941+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bb80553f0 con 0x7f8bc806dae0 2024-01-16T15:26:06.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:06.171+0000 7f8bcf200640 1 -- 172.21.15.43:0/3723893327 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bc8072370 con 0x7f8ba006c700 2024-01-16T15:26:06.178 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:06.180+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:06.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:06.181+0000 7f8bc6000640 1 -- 172.21.15.43:0/3723893327 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8bb000c040 con 0x7f8ba006c700 2024-01-16T15:26:06.179 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:08.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:08.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.469+0000 7f5d51800640 1 Processor -- start 2024-01-16T15:26:08.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.469+0000 7f5d51800640 1 -- start start 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d51800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d4c15b350 0x7f5d4c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d51800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d4c09cb30 0x7f5d4c0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d51800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c15b730 con 0x7f5d4c0adb60 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d51800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c0ac740 con 0x7f5d4c09cb30 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d51800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c0ac460 con 0x7f5d4c15b350 2024-01-16T15:26:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d43e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d4c09cb30 0x7f5d4c0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d43e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5d4c09cb30 0x7f5d4c0b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60526/0 (socket says 172.21.15.43:60526) 2024-01-16T15:26:08.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.471+0000 7f5d43e00640 1 -- 172.21.15.43:0/623174590 learned_addr learned my addr 172.21.15.43:0/623174590 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:08.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.472+0000 7f5d4aa00640 1 --2- 172.21.15.43:0/623174590 >> v2:172.21.15.69:3300/0 conn(0x7f5d4c15b350 0x7f5d4c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.472+0000 7f5d43e00640 1 -- 172.21.15.43:0/623174590 >> v2:172.21.15.69:3300/0 conn(0x7f5d4c15b350 msgr2=0x7f5d4c0abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:08.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.472+0000 7f5d43e00640 1 --2- 172.21.15.43:0/623174590 >> v2:172.21.15.69:3300/0 conn(0x7f5d4c15b350 0x7f5d4c0abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.472+0000 7f5d43e00640 1 -- 172.21.15.43:0/623174590 >> v1:172.21.15.69:6789/0 conn(0x7f5d4c0adb60 legacy=0x7f5d4c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:08.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.472+0000 7f5d43e00640 1 -- 172.21.15.43:0/623174590 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0b2a50 con 0x7f5d4c09cb30 2024-01-16T15:26:08.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.473+0000 7f5d43e00640 1 --2- 172.21.15.43:0/623174590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c09cb30 0x7f5d4c0b3530 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c0b5100 tx=0x7f5d3402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d1db9034077ff3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:08.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.474+0000 7f5d4a000640 1 -- 172.21.15.43:0/623174590 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34030860 con 0x7f5d4c09cb30 2024-01-16T15:26:08.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.474+0000 7f5d4a000640 1 -- 172.21.15.43:0/623174590 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d340309c0 con 0x7f5d4c09cb30 2024-01-16T15:26:08.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.474+0000 7f5d4a000640 1 -- 172.21.15.43:0/623174590 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34030d50 con 0x7f5d4c09cb30 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.474+0000 7f5d51800640 1 -- 172.21.15.43:0/623174590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c09cb30 msgr2=0x7f5d4c0b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.474+0000 7f5d51800640 1 --2- 172.21.15.43:0/623174590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c09cb30 0x7f5d4c0b3530 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c0b5100 tx=0x7f5d3402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 -- 172.21.15.43:0/623174590 shutdown_connections 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 --2- 172.21.15.43:0/623174590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c09cb30 0x7f5d4c0b3530 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 --2- 172.21.15.43:0/623174590 >> v2:172.21.15.69:3300/0 conn(0x7f5d4c15b350 0x7f5d4c0abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 -- 172.21.15.43:0/623174590 >> 172.21.15.43:0/623174590 conn(0x7f5d4c0b0ba0 msgr2=0x7f5d4c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 -- 172.21.15.43:0/623174590 shutdown_connections 2024-01-16T15:26:08.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.475+0000 7f5d51800640 1 -- 172.21.15.43:0/623174590 wait complete. 2024-01-16T15:26:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.477+0000 7f5d51800640 1 Processor -- start 2024-01-16T15:26:08.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.477+0000 7f5d51800640 1 -- start start 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4c09cb30 0x7f5d4c072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d4c0adb60 0x7f5d4c148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c15b350 0x7f5d4c0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c1489a0 con 0x7f5d4c15b350 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0afe50 con 0x7f5d4c0adb60 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d51800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c071180 con 0x7f5d4c09cb30 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d4b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c15b350 0x7f5d4c0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d4b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c15b350 0x7f5d4c0af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60540/0 (socket says 172.21.15.43:60540) 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.478+0000 7f5d4b400640 1 -- 172.21.15.43:0/750539280 learned_addr learned my addr 172.21.15.43:0/750539280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d43e00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d4c0adb60 0x7f5d4c148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4aa00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4c09cb30 0x7f5d4c072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 -- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4c09cb30 msgr2=0x7f5d4c072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4c09cb30 0x7f5d4c072110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 -- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d4c0adb60 msgr2=0x7f5d4c148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d4c0adb60 0x7f5d4c148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 -- 172.21.15.43:0/750539280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340094d0 con 0x7f5d4c15b350 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d43e00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d4c0adb60 0x7f5d4c148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4aa00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d4c09cb30 0x7f5d4c072110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:08.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d4b400640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d4c15b350 0x7f5d4c0af910 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400c7a0 tx=0x7f5d4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44003b70 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d44003cf0 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44003ea0 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.479+0000 7f5d51800640 1 -- 172.21.15.43:0/750539280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c071460 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.480+0000 7f5d51800640 1 -- 172.21.15.43:0/750539280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c1574c0 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.481+0000 7f5d51800640 1 -- 172.21.15.43:0/750539280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c09d200 con 0x7f5d4c15b350 2024-01-16T15:26:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.482+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f5d4400aa90 con 0x7f5d4c15b350 2024-01-16T15:26:08.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.482+0000 7f5d42a00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d2406c700 0x7f5d2406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:08.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.483+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5d44088920 con 0x7f5d4c15b350 2024-01-16T15:26:08.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.483+0000 7f5d4aa00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d2406c700 0x7f5d2406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:08.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.484+0000 7f5d4aa00640 1 --2- 172.21.15.43:0/750539280 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5d2406c700 0x7f5d2406ebc0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c1593d0 tx=0x7f5d3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:08.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.488+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d44054e10 con 0x7f5d4c15b350 2024-01-16T15:26:08.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.716+0000 7f5d51800640 1 -- 172.21.15.43:0/750539280 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d4c0b3a70 con 0x7f5d2406c700 2024-01-16T15:26:08.726 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:08.728+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:08.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:08.728+0000 7f5d42a00640 1 -- 172.21.15.43:0/750539280 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5d3c004030 con 0x7f5d2406c700 2024-01-16T15:26:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:08.727 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:10.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.974+0000 7f46eb200640 1 Processor -- start 2024-01-16T15:26:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.974+0000 7f46eb200640 1 -- start start 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.976+0000 7f46eb200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f46ec06dae0 0x7f46ec070280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.976+0000 7f46eb200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec0b3e20 con 0x7f46ec0ac870 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.976+0000 7f46eb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec0acee0 con 0x7f46ec143590 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.976+0000 7f46eb200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec0acce0 con 0x7f46ec06dae0 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46ea800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f46ec06dae0 0x7f46ec070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46e1200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f46ec0ac870 0x7f46ec06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:41526/0 (socket says 172.21.15.43:41526) 2024-01-16T15:26:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46e1200640 1 -- 172.21.15.43:0/1381008245 learned_addr learned my addr 172.21.15.43:0/1381008245 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46ea800640 1 -- 172.21.15.43:0/1381008245 >> v1:172.21.15.69:6789/0 conn(0x7f46ec143590 legacy=0x7f46ec0ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46ea800640 1 -- 172.21.15.43:0/1381008245 >> v1:172.21.15.43:6789/0 conn(0x7f46ec0ac870 legacy=0x7f46ec06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46e9400640 1 -- 172.21.15.43:0/1381008245 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283968520 0 0) 0x7f46ec0b3e20 con 0x7f46ec0ac870 2024-01-16T15:26:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.977+0000 7f46ea800640 1 -- 172.21.15.43:0/1381008245 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec0ad1c0 con 0x7f46ec06dae0 2024-01-16T15:26:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.979+0000 7f46ea800640 1 --2- 172.21.15.43:0/1381008245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec06dae0 0x7f46ec070280 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009d90 tx=0x7f46e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cf9c0cc8e0df95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.979+0000 7f46e9400640 1 -- 172.21.15.43:0/1381008245 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e402cd00 con 0x7f46ec06dae0 2024-01-16T15:26:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.979+0000 7f46e9400640 1 -- 172.21.15.43:0/1381008245 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e402ce60 con 0x7f46ec06dae0 2024-01-16T15:26:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.979+0000 7f46e9400640 1 -- 172.21.15.43:0/1381008245 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4034660 con 0x7f46ec06dae0 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 -- 172.21.15.43:0/1381008245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec06dae0 msgr2=0x7f46ec070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 --2- 172.21.15.43:0/1381008245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec06dae0 0x7f46ec070280 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009d90 tx=0x7f46e402bf50 comp rx=0 tx=0).stop 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 -- 172.21.15.43:0/1381008245 shutdown_connections 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 --2- 172.21.15.43:0/1381008245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec06dae0 0x7f46ec070280 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 -- 172.21.15.43:0/1381008245 >> 172.21.15.43:0/1381008245 conn(0x7f46ec0b0aa0 msgr2=0x7f46ec0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 -- 172.21.15.43:0/1381008245 shutdown_connections 2024-01-16T15:26:10.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.980+0000 7f46eb200640 1 -- 172.21.15.43:0/1381008245 wait complete. 2024-01-16T15:26:10.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.982+0000 7f46eb200640 1 Processor -- start 2024-01-16T15:26:10.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.982+0000 7f46eb200640 1 -- start start 2024-01-16T15:26:10.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46ec06dae0 0x7f46ec13aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec0ac870 0x7f46ec138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46ec143590 0x7f46ec145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec13b400 con 0x7f46ec0ac870 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec13aa30 con 0x7f46ec143590 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46eb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec138f40 con 0x7f46ec06dae0 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46e1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec0ac870 0x7f46ec138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46e1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec0ac870 0x7f46ec138a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60558/0 (socket says 172.21.15.43:60558) 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.983+0000 7f46e1200640 1 -- 172.21.15.43:0/3019387224 learned_addr learned my addr 172.21.15.43:0/3019387224 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e9e00640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46ec06dae0 0x7f46ec13aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46ea800640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46ec143590 0x7f46ec145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 -- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46ec06dae0 msgr2=0x7f46ec13aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46ec06dae0 0x7f46ec13aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 -- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46ec143590 msgr2=0x7f46ec145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46ec143590 0x7f46ec145560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 -- 172.21.15.43:0/3019387224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4009a70 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e9e00640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46ec06dae0 0x7f46ec13aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46ea800640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46ec143590 0x7f46ec145560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e1200640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec0ac870 0x7f46ec138a00 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f46d400c4d0 tx=0x7f46d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4007480 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d40075e0 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.984+0000 7f46eb200640 1 -- 172.21.15.43:0/3019387224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec139b40 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.985+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d40059a0 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.985+0000 7f46eb200640 1 -- 172.21.15.43:0/3019387224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec13a3e0 con 0x7f46ec0ac870 2024-01-16T15:26:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.986+0000 7f46eb200640 1 -- 172.21.15.43:0/3019387224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec13abe0 con 0x7f46ec0ac870 2024-01-16T15:26:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.987+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f46d4005b00 con 0x7f46ec0ac870 2024-01-16T15:26:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.988+0000 7f46e2a00640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c406c700 0x7f46c406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.988+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f46d4088390 con 0x7f46ec0ac870 2024-01-16T15:26:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.988+0000 7f46e9e00640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c406c700 0x7f46c406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.989+0000 7f46e9e00640 1 --2- 172.21.15.43:0/3019387224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46c406c700 0x7f46c406ebc0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f46dc000f60 tx=0x7f46dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:10.993+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46ec13abe0 con 0x7f46ec0ac870 2024-01-16T15:26:11.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:11.221+0000 7f46eb200640 1 -- 172.21.15.43:0/3019387224 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46ec0ad780 con 0x7f46c406c700 2024-01-16T15:26:11.232 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:11.234+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:11.234+0000 7f46e2a00640 1 -- 172.21.15.43:0/3019387224 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46dc001c90 con 0x7f46c406c700 2024-01-16T15:26:11.232 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:11.233 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:13.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.494+0000 7f17d7200640 1 Processor -- start 2024-01-16T15:26:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.494+0000 7f17d7200640 1 -- start start 2024-01-16T15:26:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17d7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17d00ae270 0x7f17d00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17d7200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d015b730 con 0x7f17d015b350 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17d7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d00ac740 con 0x7f17d009cb30 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17d7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d00ac460 con 0x7f17d00ae270 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17cfe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f17d015b350 0x7f17d00b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60646/0 (socket says 172.21.15.43:60646) 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17cfe00640 1 -- 172.21.15.43:0/2423465357 learned_addr learned my addr 172.21.15.43:0/2423465357 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197337516 0 0) 0x7f17c0001110 con 0x7f17d015b350 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.496+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b0003520 con 0x7f17d015b350 2024-01-16T15:26:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17d5200640 1 --2- 172.21.15.43:0/2423465357 >> v2:172.21.15.69:3300/0 conn(0x7f17d00ae270 0x7f17d00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180704531 0 0) 0x7f17b0003520 con 0x7f17d015b350 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 >> v2:172.21.15.69:3300/0 conn(0x7f17d00ae270 msgr2=0x7f17d00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 --2- 172.21.15.43:0/2423465357 >> v2:172.21.15.69:3300/0 conn(0x7f17d00ae270 0x7f17d00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 >> v1:172.21.15.69:6789/0 conn(0x7f17d009cb30 legacy=0x7f17d00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d00b2a50 con 0x7f17d015b350 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f17c0001800 con 0x7f17d015b350 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 >> v1:172.21.15.43:6789/0 conn(0x7f17d015b350 legacy=0x7f17d00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17b0003970 0x7f17b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 0x7f17b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b0005180 0x7f17b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00ac460 con 0x7f17b00047c0 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b000c020 con 0x7f17b0005180 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.497+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b000c1e0 con 0x7f17b0003970 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 0x7f17b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cfe00640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17b0003970 0x7f17b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17d5200640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b0005180 0x7f17b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 -- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17b0003970 msgr2=0x7f17b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17b0003970 0x7f17b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 -- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b0005180 msgr2=0x7f17b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b0005180 0x7f17b000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17cf400640 1 -- 172.21.15.43:0/2423465357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0003520 con 0x7f17b00047c0 2024-01-16T15:26:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.498+0000 7f17d5200640 1 -- 172.21.15.43:0/2423465357 reap_dead start 2024-01-16T15:26:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.499+0000 7f17cf400640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 0x7f17b0004c40 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f17b80015b0 tx=0x7f17b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2521524b48bed410 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.500+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b80052d0 con 0x7f17b00047c0 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.500+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b802e050 con 0x7f17b00047c0 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.500+0000 7f17cea00640 1 -- 172.21.15.43:0/2423465357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003830 con 0x7f17b00047c0 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.500+0000 7f17d7200640 1 -- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 msgr2=0x7f17b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.500+0000 7f17d7200640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 0x7f17b0004c40 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f17b80015b0 tx=0x7f17b00057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.501+0000 7f17d7200640 1 -- 172.21.15.43:0/2423465357 shutdown_connections 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.501+0000 7f17d7200640 1 --2- 172.21.15.43:0/2423465357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17b00047c0 0x7f17b0004c40 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.501+0000 7f17d7200640 1 -- 172.21.15.43:0/2423465357 >> 172.21.15.43:0/2423465357 conn(0x7f17d00b0ba0 msgr2=0x7f17d00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.501+0000 7f17d7200640 1 -- 172.21.15.43:0/2423465357 shutdown_connections 2024-01-16T15:26:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.501+0000 7f17d7200640 1 -- 172.21.15.43:0/2423465357 wait complete. 2024-01-16T15:26:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.503+0000 7f17d7200640 1 Processor -- start 2024-01-16T15:26:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.503+0000 7f17d7200640 1 -- start start 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b00047c0 0x7f17d015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d015c750 0x7f17d0148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d0071920 0x7f17d0071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00722e0 con 0x7f17d0071920 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d015cbd0 con 0x7f17b00047c0 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d015dae0 con 0x7f17d015c750 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d0071920 0x7f17d0071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d0071920 0x7f17d0071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53266/0 (socket says 172.21.15.43:53266) 2024-01-16T15:26:13.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 -- 172.21.15.43:0/748254178 learned_addr learned my addr 172.21.15.43:0/748254178 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17cf400640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d015c750 0x7f17d0148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17cfe00640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b00047c0 0x7f17d015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 -- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d015c750 msgr2=0x7f17d0148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17d015c750 0x7f17d0148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 -- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b00047c0 msgr2=0x7f17d015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b00047c0 0x7f17d015d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17d5200640 1 -- 172.21.15.43:0/748254178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d015ddc0 con 0x7f17d0071920 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.504+0000 7f17cfe00640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17b00047c0 0x7f17d015d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17d5200640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17d0071920 0x7f17d0071da0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b1bf0 tx=0x7f17c80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c800e070 con 0x7f17d0071920 2024-01-16T15:26:13.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8011e10 con 0x7f17d0071920 2024-01-16T15:26:13.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8003990 con 0x7f17d0071920 2024-01-16T15:26:13.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17d7200640 1 -- 172.21.15.43:0/748254178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d015e0a0 con 0x7f17d0071920 2024-01-16T15:26:13.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.505+0000 7f17d7200640 1 -- 172.21.15.43:0/748254178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d015e660 con 0x7f17d0071920 2024-01-16T15:26:13.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.506+0000 7f17d7200640 1 -- 172.21.15.43:0/748254178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d009d200 con 0x7f17d0071920 2024-01-16T15:26:13.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.507+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 51) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f17c8003af0 con 0x7f17d0071920 2024-01-16T15:26:13.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.508+0000 7f17c7e00640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17a806c730 0x7f17a806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:13.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.508+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f17c8089320 con 0x7f17d0071920 2024-01-16T15:26:13.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.509+0000 7f17cfe00640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17a806c730 0x7f17a806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:13.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.509+0000 7f17cfe00640 1 --2- 172.21.15.43:0/748254178 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f17a806c730 0x7f17a806ebf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f17c0001e20 tx=0x7f17c0000ff0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:13.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.513+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17c8055810 con 0x7f17d0071920 2024-01-16T15:26:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.740+0000 7f17d7200640 1 -- 172.21.15.43:0/748254178 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17d006e720 con 0x7f17a806c730 2024-01-16T15:26:13.748 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:13.750+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:13.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:13.751+0000 7f17c7e00640 1 -- 172.21.15.43:0/748254178 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f17c00016c0 con 0x7f17a806c730 2024-01-16T15:26:13.749 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:13.749 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:15.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.980+0000 7fa281000640 1 Processor -- start 2024-01-16T15:26:15.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.980+0000 7fa281000640 1 -- start start 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.982+0000 7fa281000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 0x7fa27c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.982+0000 7fa281000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa27c15bbd0 0x7fa27c070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.982+0000 7fa281000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c144ca0 con 0x7fa27c1448c0 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.982+0000 7fa281000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c15c450 con 0x7fa27c15bbd0 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.982+0000 7fa281000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c0715b0 con 0x7fa27c06dae0 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa27c15bbd0 0x7fa27c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa27c15bbd0 0x7fa27c070070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53278/0 (socket says 172.21.15.43:53278) 2024-01-16T15:26:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 -- 172.21.15.43:0/735425381 learned_addr learned my addr 172.21.15.43:0/735425381 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa273e00640 1 --2- 172.21.15.43:0/735425381 >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 0x7fa27c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 -- 172.21.15.43:0/735425381 >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 msgr2=0x7fa27c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 --2- 172.21.15.43:0/735425381 >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 0x7fa27c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 -- 172.21.15.43:0/735425381 >> v1:172.21.15.69:6789/0 conn(0x7fa27c1448c0 legacy=0x7fa27c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa27be00640 1 -- 172.21.15.43:0/735425381 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c071890 con 0x7fa27c15bbd0 2024-01-16T15:26:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.983+0000 7fa273e00640 1 --2- 172.21.15.43:0/735425381 >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 0x7fa27c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.984+0000 7fa27be00640 1 --2- 172.21.15.43:0/735425381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c15bbd0 0x7fa27c070070 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0b1a90 tx=0x7fa27402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac6485b945b1d6f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa27aa00640 1 -- 172.21.15.43:0/735425381 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27402ca80 con 0x7fa27c15bbd0 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa27aa00640 1 -- 172.21.15.43:0/735425381 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27402cbe0 con 0x7fa27c15bbd0 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa27aa00640 1 -- 172.21.15.43:0/735425381 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2740343f0 con 0x7fa27c15bbd0 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa281000640 1 -- 172.21.15.43:0/735425381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c15bbd0 msgr2=0x7fa27c070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa281000640 1 --2- 172.21.15.43:0/735425381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c15bbd0 0x7fa27c070070 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0b1a90 tx=0x7fa27402bee0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa281000640 1 -- 172.21.15.43:0/735425381 shutdown_connections 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa281000640 1 --2- 172.21.15.43:0/735425381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c15bbd0 0x7fa27c070070 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.985+0000 7fa281000640 1 --2- 172.21.15.43:0/735425381 >> v2:172.21.15.69:3300/0 conn(0x7fa27c06dae0 0x7fa27c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.986+0000 7fa281000640 1 -- 172.21.15.43:0/735425381 >> 172.21.15.43:0/735425381 conn(0x7fa27c0b0aa0 msgr2=0x7fa27c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.986+0000 7fa281000640 1 -- 172.21.15.43:0/735425381 shutdown_connections 2024-01-16T15:26:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.986+0000 7fa281000640 1 -- 172.21.15.43:0/735425381 wait complete. 2024-01-16T15:26:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.988+0000 7fa281000640 1 Processor -- start 2024-01-16T15:26:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.988+0000 7fa281000640 1 -- start start 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa27c06dae0 0x7fa27c1f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c1448c0 0x7fa27c13f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa27c15bbd0 0x7fa27c13fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0ad530 con 0x7fa27c1448c0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0ad810 con 0x7fa27c06dae0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa281000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0adaf0 con 0x7fa27c15bbd0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa273e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c1448c0 0x7fa27c13f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa273e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c1448c0 0x7fa27c13f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53284/0 (socket says 172.21.15.43:53284) 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.989+0000 7fa273e00640 1 -- 172.21.15.43:0/296440539 learned_addr learned my addr 172.21.15.43:0/296440539 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa27b400640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa27c06dae0 0x7fa27c1f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa27be00640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa27c15bbd0 0x7fa27c13fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 -- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa27c15bbd0 msgr2=0x7fa27c13fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa27c15bbd0 0x7fa27c13fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 -- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa27c06dae0 msgr2=0x7fa27c1f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa27c06dae0 0x7fa27c1f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 -- 172.21.15.43:0/296440539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274009b30 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa27b400640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa27c06dae0 0x7fa27c1f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273e00640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa27c1448c0 0x7fa27c13f980 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa26400c7f0 tx=0x7fa26400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264005b90 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa264005cf0 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.990+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264007a30 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.991+0000 7fa281000640 1 -- 172.21.15.43:0/296440539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c0724f0 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.991+0000 7fa281000640 1 -- 172.21.15.43:0/296440539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c0b2220 con 0x7fa27c1448c0 2024-01-16T15:26:15.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.992+0000 7fa281000640 1 -- 172.21.15.43:0/296440539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c0b23e0 con 0x7fa27c1448c0 2024-01-16T15:26:15.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.993+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 52) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fa264007b90 con 0x7fa27c1448c0 2024-01-16T15:26:15.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.994+0000 7fa273400640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa25406c5e0 0x7fa25406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.994+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa264088b70 con 0x7fa27c1448c0 2024-01-16T15:26:15.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.995+0000 7fa27b400640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa25406c5e0 0x7fa25406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.996+0000 7fa27b400640 1 --2- 172.21.15.43:0/296440539 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa25406c5e0 0x7fa25406eaa0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa26c003eb0 tx=0x7fa26c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:15.999+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa264055090 con 0x7fa27c1448c0 2024-01-16T15:26:16.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:16.235+0000 7fa281000640 1 -- 172.21.15.43:0/296440539 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa27c0ad980 con 0x7fa25406c5e0 2024-01-16T15:26:16.244 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:16.246+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:16.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:16.246+0000 7fa273400640 1 -- 172.21.15.43:0/296440539 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa27c0ad980 con 0x7fa25406c5e0 2024-01-16T15:26:16.245 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:16.245 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:18.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:18.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.506+0000 7ff531a00640 1 Processor -- start 2024-01-16T15:26:18.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.506+0000 7ff531a00640 1 -- start start 2024-01-16T15:26:18.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 0x7ff52c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff52c15bbd0 0x7ff52c070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52c144ca0 con 0x7ff52c1448c0 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52c15c450 con 0x7ff52c15bbd0 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c0715b0 con 0x7ff52c06dae0 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff52c15bbd0 0x7ff52c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff52c15bbd0 0x7ff52c070070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53300/0 (socket says 172.21.15.43:53300) 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff531000640 1 -- 172.21.15.43:0/3518876262 learned_addr learned my addr 172.21.15.43:0/3518876262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.508+0000 7ff52b400640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 0x7ff52c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.509+0000 7ff531000640 1 -- 172.21.15.43:0/3518876262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 msgr2=0x7ff52c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.509+0000 7ff531000640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 0x7ff52c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.509+0000 7ff531000640 1 -- 172.21.15.43:0/3518876262 >> v1:172.21.15.69:6789/0 conn(0x7ff52c1448c0 legacy=0x7ff52c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.509+0000 7ff531000640 1 -- 172.21.15.43:0/3518876262 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c071890 con 0x7ff52c15bbd0 2024-01-16T15:26:18.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.509+0000 7ff52b400640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 0x7ff52c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:18.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.510+0000 7ff531000640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c15bbd0 0x7ff52c070070 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7ff524009e50 tx=0x7ff52402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f545ee362f208cfb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:18.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.510+0000 7ff52aa00640 1 -- 172.21.15.43:0/3518876262 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52402ca70 con 0x7ff52c15bbd0 2024-01-16T15:26:18.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.510+0000 7ff52aa00640 1 -- 172.21.15.43:0/3518876262 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52402cbd0 con 0x7ff52c15bbd0 2024-01-16T15:26:18.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.510+0000 7ff52aa00640 1 -- 172.21.15.43:0/3518876262 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240343f0 con 0x7ff52c15bbd0 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 -- 172.21.15.43:0/3518876262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c15bbd0 msgr2=0x7ff52c070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c15bbd0 0x7ff52c070070 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7ff524009e50 tx=0x7ff52402bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 -- 172.21.15.43:0/3518876262 shutdown_connections 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c15bbd0 0x7ff52c070070 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 --2- 172.21.15.43:0/3518876262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c06dae0 0x7ff52c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 -- 172.21.15.43:0/3518876262 >> 172.21.15.43:0/3518876262 conn(0x7ff52c0b0aa0 msgr2=0x7ff52c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 -- 172.21.15.43:0/3518876262 shutdown_connections 2024-01-16T15:26:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.511+0000 7ff531a00640 1 -- 172.21.15.43:0/3518876262 wait complete. 2024-01-16T15:26:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.513+0000 7ff531a00640 1 Processor -- start 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.513+0000 7ff531a00640 1 -- start start 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c06dae0 0x7ff52c137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c1448c0 0x7ff52c1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff52c15bbd0 0x7ff52c1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c13ee30 con 0x7ff52c06dae0 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1f4aa0 con 0x7ff52c15bbd0 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff531a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1f44e0 con 0x7ff52c1448c0 2024-01-16T15:26:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff52be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c06dae0 0x7ff52c137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff52be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c06dae0 0x7ff52c137810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53302/0 (socket says 172.21.15.43:53302) 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff52be00640 1 -- 172.21.15.43:0/3057430251 learned_addr learned my addr 172.21.15.43:0/3057430251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.514+0000 7ff52b400640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c1448c0 0x7ff52c1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff531000640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff52c15bbd0 0x7ff52c1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 -- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c1448c0 msgr2=0x7ff52c1f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff52c1448c0 0x7ff52c1f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 -- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff52c15bbd0 msgr2=0x7ff52c1592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff52c15bbd0 0x7ff52c1592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 -- 172.21.15.43:0/3057430251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009b30 con 0x7ff52c06dae0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff531000640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff52c15bbd0 0x7ff52c1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff52be00640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff52c06dae0 0x7ff52c137810 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00c430 tx=0x7ff51c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c011790 con 0x7ff52c06dae0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c0118f0 con 0x7ff52c06dae0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff531a00640 1 -- 172.21.15.43:0/3057430251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c0724f0 con 0x7ff52c06dae0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.515+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c011ce0 con 0x7ff52c06dae0 2024-01-16T15:26:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.516+0000 7ff531a00640 1 -- 172.21.15.43:0/3057430251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c13eac0 con 0x7ff52c06dae0 2024-01-16T15:26:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.517+0000 7ff531a00640 1 -- 172.21.15.43:0/3057430251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c04e840 con 0x7ff52c06dae0 2024-01-16T15:26:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.518+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff51c01c460 con 0x7ff52c06dae0 2024-01-16T15:26:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.519+0000 7ff523e00640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff50406c4c0 0x7ff50406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.519+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff51c088b00 con 0x7ff52c06dae0 2024-01-16T15:26:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.519+0000 7ff52b400640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff50406c4c0 0x7ff50406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.520+0000 7ff52b400640 1 --2- 172.21.15.43:0/3057430251 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff50406c4c0 0x7ff50406e980 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff514006fd0 tx=0x7ff514008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.524+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff51c055050 con 0x7ff52c06dae0 2024-01-16T15:26:18.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.752+0000 7ff531a00640 1 -- 172.21.15.43:0/3057430251 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff52c072370 con 0x7ff50406c4c0 2024-01-16T15:26:18.762 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:18.764+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:18.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:18.764+0000 7ff523e00640 1 -- 172.21.15.43:0/3057430251 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff51400a720 con 0x7ff50406c4c0 2024-01-16T15:26:18.763 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:18.763 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:20.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.023+0000 7f9a62600640 1 Processor -- start 2024-01-16T15:26:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.023+0000 7f9a62600640 1 -- start start 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.025+0000 7f9a62600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.025+0000 7f9a62600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a5c06d910 0x7f9a5c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.025+0000 7f9a62600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c06dd50 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.025+0000 7f9a62600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c071080 con 0x7f9a5c06d910 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.025+0000 7f9a62600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c144ca0 con 0x7f9a5c1448c0 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a61c00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9a5c15bbd0 0x7f9a5c15eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60652/0 (socket says 172.21.15.43:60652) 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a61c00640 1 -- 172.21.15.43:0/2698880493 learned_addr learned my addr 172.21.15.43:0/2698880493 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a61200640 1 --2- 172.21.15.43:0/2698880493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a5be00640 1 --2- 172.21.15.43:0/2698880493 >> v2:172.21.15.69:3300/0 conn(0x7f9a5c06d910 0x7f9a5c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a5b400640 1 -- 172.21.15.43:0/2698880493 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830302451 0 0) 0x7f9a5c06dd50 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.026+0000 7f9a5b400640 1 -- 172.21.15.43:0/2698880493 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c003520 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.027+0000 7f9a61200640 1 -- 172.21.15.43:0/2698880493 >> v2:172.21.15.69:3300/0 conn(0x7f9a5c06d910 msgr2=0x7f9a5c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.027+0000 7f9a61200640 1 --2- 172.21.15.43:0/2698880493 >> v2:172.21.15.69:3300/0 conn(0x7f9a5c06d910 0x7f9a5c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.027+0000 7f9a61200640 1 -- 172.21.15.43:0/2698880493 >> v1:172.21.15.43:6789/0 conn(0x7f9a5c15bbd0 legacy=0x7f9a5c15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.027+0000 7f9a61200640 1 -- 172.21.15.43:0/2698880493 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c15c450 con 0x7f9a5c1448c0 2024-01-16T15:26:21.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.028+0000 7f9a61200640 1 --2- 172.21.15.43:0/2698880493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c06d3d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0097a0 tx=0x7f9a4c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a7ace9d7f63335 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.029+0000 7f9a5b400640 1 -- 172.21.15.43:0/2698880493 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c02d040 con 0x7f9a5c1448c0 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.029+0000 7f9a5b400640 1 -- 172.21.15.43:0/2698880493 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c031810 con 0x7f9a5c1448c0 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.029+0000 7f9a5b400640 1 -- 172.21.15.43:0/2698880493 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c031b80 con 0x7f9a5c1448c0 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.029+0000 7f9a62600640 1 -- 172.21.15.43:0/2698880493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 msgr2=0x7f9a5c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.029+0000 7f9a62600640 1 --2- 172.21.15.43:0/2698880493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c06d3d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0097a0 tx=0x7f9a4c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 -- 172.21.15.43:0/2698880493 shutdown_connections 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 --2- 172.21.15.43:0/2698880493 >> v2:172.21.15.69:3300/0 conn(0x7f9a5c06d910 0x7f9a5c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 --2- 172.21.15.43:0/2698880493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c06d3d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 -- 172.21.15.43:0/2698880493 >> 172.21.15.43:0/2698880493 conn(0x7f9a5c0b0aa0 msgr2=0x7f9a5c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:21.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 -- 172.21.15.43:0/2698880493 shutdown_connections 2024-01-16T15:26:21.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.030+0000 7f9a62600640 1 -- 172.21.15.43:0/2698880493 wait complete. 2024-01-16T15:26:21.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.031+0000 7f9a62600640 1 Processor -- start 2024-01-16T15:26:21.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.031+0000 7f9a62600640 1 -- start start 2024-01-16T15:26:21.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a5c06d910 0x7f9a5c1f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c1f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a5c15bbd0 0x7f9a5c1f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1f59e0 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0a8e90 con 0x7f9a5c06d910 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.032+0000 7f9a62600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1f4dd0 con 0x7f9a5c1448c0 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a5c15bbd0 0x7f9a5c1f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a5c15bbd0 0x7f9a5c1f47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53306/0 (socket says 172.21.15.43:53306) 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 -- 172.21.15.43:0/3187847858 learned_addr learned my addr 172.21.15.43:0/3187847858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a5be00640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c1f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61200640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a5c06d910 0x7f9a5c1f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 -- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 msgr2=0x7f9a5c1f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a5c1448c0 0x7f9a5c1f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 -- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a5c06d910 msgr2=0x7f9a5c1f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a5c06d910 0x7f9a5c1f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.033+0000 7f9a61c00640 1 -- 172.21.15.43:0/3187847858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c009400 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a61c00640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a5c15bbd0 0x7f9a5c1f47c0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400ca20 tx=0x7f9a5400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400abd0 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5400ad30 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54012620 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a62600640 1 -- 172.21.15.43:0/3187847858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c1f50b0 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.034+0000 7f9a62600640 1 -- 172.21.15.43:0/3187847858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c1f5670 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.035+0000 7f9a62600640 1 -- 172.21.15.43:0/3187847858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c04e840 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.036+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f9a54007070 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.038+0000 7f9a58c00640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a3406c4f0 0x7f9a3406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:21.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.038+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9a54088a60 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.038+0000 7f9a61200640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a3406c4f0 0x7f9a3406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:21.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.039+0000 7f9a61200640 1 --2- 172.21.15.43:0/3187847858 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9a3406c4f0 0x7f9a3406e9b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c036000 tx=0x7f9a4c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:21.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.042+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a54054fb0 con 0x7f9a5c15bbd0 2024-01-16T15:26:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.271+0000 7f9a62600640 1 -- 172.21.15.43:0/3187847858 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a5c071b70 con 0x7f9a3406c4f0 2024-01-16T15:26:21.281 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:21.283+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:21.283+0000 7f9a58c00640 1 -- 172.21.15.43:0/3187847858 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9a4c031b00 con 0x7f9a3406c4f0 2024-01-16T15:26:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:23.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:23.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.548+0000 7f33bbe00640 1 Processor -- start 2024-01-16T15:26:23.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.549+0000 7f33bbe00640 1 -- start start 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.550+0000 7f33bbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f33bc15d340 0x7f33bc0d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.550+0000 7f33bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f33bc0d1d90 0x7f33bc0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.550+0000 7f33bbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0cef30 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.550+0000 7f33bbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc0d11a0 con 0x7f33bc15d340 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.550+0000 7f33bbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc0ce320 con 0x7f33bc0d1d90 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f33bc15d340 0x7f33bc0d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f33bc15d340 0x7f33bc0d00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57932/0 (socket says 172.21.15.43:57932) 2024-01-16T15:26:23.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 -- 172.21.15.43:0/3390177162 learned_addr learned my addr 172.21.15.43:0/3390177162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:23.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 -- 172.21.15.43:0/3390177162 >> v2:172.21.15.69:3300/0 conn(0x7f33bc0d1d90 msgr2=0x7f33bc0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:23.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 --2- 172.21.15.43:0/3390177162 >> v2:172.21.15.69:3300/0 conn(0x7f33bc0d1d90 0x7f33bc0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:23.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 -- 172.21.15.43:0/3390177162 >> v1:172.21.15.69:6789/0 conn(0x7f33bc0cdbb0 legacy=0x7f33bc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:23.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.551+0000 7f33baa00640 1 -- 172.21.15.43:0/3390177162 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc0ce600 con 0x7f33bc15d340 2024-01-16T15:26:23.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.552+0000 7f33baa00640 1 --2- 172.21.15.43:0/3390177162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc15d340 0x7f33bc0d00b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f33bc09c810 tx=0x7f33a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf81a91ab53e6bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.552+0000 7f33b9600640 1 -- 172.21.15.43:0/3390177162 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4031870 con 0x7f33bc15d340 2024-01-16T15:26:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.552+0000 7f33b9600640 1 -- 172.21.15.43:0/3390177162 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a40319d0 con 0x7f33bc15d340 2024-01-16T15:26:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.552+0000 7f33b9600640 1 -- 172.21.15.43:0/3390177162 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4031d60 con 0x7f33bc15d340 2024-01-16T15:26:23.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 -- 172.21.15.43:0/3390177162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc15d340 msgr2=0x7f33bc0d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 --2- 172.21.15.43:0/3390177162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc15d340 0x7f33bc0d00b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f33bc09c810 tx=0x7f33a402b910 comp rx=0 tx=0).stop 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 -- 172.21.15.43:0/3390177162 shutdown_connections 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 --2- 172.21.15.43:0/3390177162 >> v2:172.21.15.69:3300/0 conn(0x7f33bc0d1d90 0x7f33bc0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 --2- 172.21.15.43:0/3390177162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc15d340 0x7f33bc0d00b0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 -- 172.21.15.43:0/3390177162 >> 172.21.15.43:0/3390177162 conn(0x7f33bc09c370 msgr2=0x7f33bc0d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 -- 172.21.15.43:0/3390177162 shutdown_connections 2024-01-16T15:26:23.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.553+0000 7f33bbe00640 1 -- 172.21.15.43:0/3390177162 wait complete. 2024-01-16T15:26:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.555+0000 7f33bbe00640 1 Processor -- start 2024-01-16T15:26:23.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.555+0000 7f33bbe00640 1 -- start start 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc0cdbb0 0x7f33bc1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33bc0d1d90 0x7f33bc15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33bc15d340 0x7f33bc153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0b3b80 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc15e7e0 con 0x7f33bc0d1d90 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33bbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc15ea80 con 0x7f33bc15d340 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33baa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc0cdbb0 0x7f33bc1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:23.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33baa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc0cdbb0 0x7f33bc1f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57948/0 (socket says 172.21.15.43:57948) 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.556+0000 7f33baa00640 1 -- 172.21.15.43:0/3225555018 learned_addr learned my addr 172.21.15.43:0/3225555018 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33bb400640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33bc15d340 0x7f33bc153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 -- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33bc15d340 msgr2=0x7f33bc153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33ba000640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33bc0d1d90 0x7f33bc15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33bc15d340 0x7f33bc153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 -- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33bc0d1d90 msgr2=0x7f33bc15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33bc0d1d90 0x7f33bc15e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 -- 172.21.15.43:0/3225555018 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40094d0 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33bb400640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33bc15d340 0x7f33bc153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33ba000640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33bc0d1d90 0x7f33bc15e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.557+0000 7f33baa00640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f33bc0cdbb0 0x7f33bc1f3770 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f33a4000c00 tx=0x7f33a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.558+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a403ad10 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.558+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a403ae70 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.558+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4044610 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.558+0000 7f33bbe00640 1 -- 172.21.15.43:0/3225555018 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc15ed60 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.558+0000 7f33bbe00640 1 -- 172.21.15.43:0/3225555018 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc0b3dd0 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.559+0000 7f33bbe00640 1 -- 172.21.15.43:0/3225555018 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc154120 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.560+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f33a404d450 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.561+0000 7f33b2a00640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f339406c4f0 0x7f339406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.561+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f33a402d040 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.561+0000 7f33ba000640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f339406c4f0 0x7f339406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.562+0000 7f33ba000640 1 --2- 172.21.15.43:0/3225555018 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f339406c4f0 0x7f339406e9b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0d02d0 tx=0x7f33ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:23.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.566+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33a4077730 con 0x7f33bc0cdbb0 2024-01-16T15:26:23.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.794+0000 7f33bbe00640 1 -- 172.21.15.43:0/3225555018 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33bc0d22d0 con 0x7f339406c4f0 2024-01-16T15:26:23.802 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:23.804+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:23.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:23.804+0000 7f33b2a00640 1 -- 172.21.15.43:0/3225555018 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f33bc0d22d0 con 0x7f339406c4f0 2024-01-16T15:26:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:23.803 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:25.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.061+0000 7f68a8e00640 1 Processor -- start 2024-01-16T15:26:26.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.061+0000 7f68a8e00640 1 -- start start 2024-01-16T15:26:26.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a8e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a415b2a0 0x7f68a40b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a8e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f68a40cf970 0x7f68a4164cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a8e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a40cb7d0 con 0x7f68a40d15c0 2024-01-16T15:26:26.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a8e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a40cbd70 con 0x7f68a40cf970 2024-01-16T15:26:26.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a8e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40b2650 con 0x7f68a415b2a0 2024-01-16T15:26:26.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f68a40cf970 0x7f68a4164cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f68a40cf970 0x7f68a4164cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57960/0 (socket says 172.21.15.43:57960) 2024-01-16T15:26:26.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.063+0000 7f68a3e00640 1 -- 172.21.15.43:0/2175116878 learned_addr learned my addr 172.21.15.43:0/2175116878 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.064+0000 7f68a3400640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a415b2a0 0x7f68a40b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.064+0000 7f68a3e00640 1 -- 172.21.15.43:0/2175116878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a415b2a0 msgr2=0x7f68a40b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.064+0000 7f68a3e00640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a415b2a0 0x7f68a40b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.064+0000 7f68a3e00640 1 -- 172.21.15.43:0/2175116878 >> v1:172.21.15.69:6789/0 conn(0x7f68a40d15c0 legacy=0x7f68a40cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.064+0000 7f68a3e00640 1 -- 172.21.15.43:0/2175116878 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40b2090 con 0x7f68a40cf970 2024-01-16T15:26:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.065+0000 7f68a3e00640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40cf970 0x7f68a4164cb0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f689c009e50 tx=0x7f689c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd2fb8849b65080 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:26.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.065+0000 7f68a2000640 1 -- 172.21.15.43:0/2175116878 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c02ca70 con 0x7f68a40cf970 2024-01-16T15:26:26.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.065+0000 7f68a2000640 1 -- 172.21.15.43:0/2175116878 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c02cbd0 con 0x7f68a40cf970 2024-01-16T15:26:26.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.065+0000 7f68a2000640 1 -- 172.21.15.43:0/2175116878 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0343f0 con 0x7f68a40cf970 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 -- 172.21.15.43:0/2175116878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40cf970 msgr2=0x7f68a4164cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40cf970 0x7f68a4164cb0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f689c009e50 tx=0x7f689c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 -- 172.21.15.43:0/2175116878 shutdown_connections 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40cf970 0x7f68a4164cb0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 --2- 172.21.15.43:0/2175116878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a415b2a0 0x7f68a40b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 -- 172.21.15.43:0/2175116878 >> 172.21.15.43:0/2175116878 conn(0x7f68a4099a00 msgr2=0x7f68a4099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 -- 172.21.15.43:0/2175116878 shutdown_connections 2024-01-16T15:26:26.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.066+0000 7f68a8e00640 1 -- 172.21.15.43:0/2175116878 wait complete. 2024-01-16T15:26:26.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.068+0000 7f68a8e00640 1 Processor -- start 2024-01-16T15:26:26.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.068+0000 7f68a8e00640 1 -- start start 2024-01-16T15:26:26.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a40cf970 0x7f68a4150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40d15c0 0x7f68a40c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a415b2a0 0x7f68a40af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a415b8b0 con 0x7f68a40d15c0 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40c5ba0 con 0x7f68a415b2a0 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a8e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40c5300 con 0x7f68a40cf970 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40d15c0 0x7f68a40c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40d15c0 0x7f68a40c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57970/0 (socket says 172.21.15.43:57970) 2024-01-16T15:26:26.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.069+0000 7f68a2a00640 1 -- 172.21.15.43:0/2413358296 learned_addr learned my addr 172.21.15.43:0/2413358296 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:26.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a3e00640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a415b2a0 0x7f68a40af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a3400640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a40cf970 0x7f68a4150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 -- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a40cf970 msgr2=0x7f68a4150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68a40cf970 0x7f68a4150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 -- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a415b2a0 msgr2=0x7f68a40af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a415b2a0 0x7f68a40af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 -- 172.21.15.43:0/2413358296 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c009b30 con 0x7f68a40d15c0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a3e00640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a415b2a0 0x7f68a40af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.070+0000 7f68a2a00640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68a40d15c0 0x7f68a40c4920 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c450 tx=0x7f688c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.071+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c0052d0 con 0x7f68a40d15c0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.071+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688c005430 con 0x7f68a40d15c0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.071+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c012870 con 0x7f68a40d15c0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.071+0000 7f68a8e00640 1 -- 172.21.15.43:0/2413358296 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a40c55e0 con 0x7f68a40d15c0 2024-01-16T15:26:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.071+0000 7f68a8e00640 1 -- 172.21.15.43:0/2413358296 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a40c7480 con 0x7f68a40d15c0 2024-01-16T15:26:26.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.072+0000 7f68a8e00640 1 -- 172.21.15.43:0/2413358296 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a415b680 con 0x7f68a40d15c0 2024-01-16T15:26:26.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.073+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f688c0129d0 con 0x7f68a40d15c0 2024-01-16T15:26:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.074+0000 7f689b400640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f687c06c4f0 0x7f687c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.075+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f688c088620 con 0x7f68a40d15c0 2024-01-16T15:26:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.075+0000 7f68a3400640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f687c06c4f0 0x7f687c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.075+0000 7f68a3400640 1 --2- 172.21.15.43:0/2413358296 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f687c06c4f0 0x7f687c06e9b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6894006fd0 tx=0x7f6894008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:26.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.079+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f688c0bee90 con 0x7f68a40d15c0 2024-01-16T15:26:26.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.315+0000 7f68a8e00640 1 -- 172.21.15.43:0/2413358296 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68a40b1a70 con 0x7f687c06c4f0 2024-01-16T15:26:26.327 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:26.329+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:26.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:26.329+0000 7f689b400640 1 -- 172.21.15.43:0/2413358296 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68a40b1a70 con 0x7f687c06c4f0 2024-01-16T15:26:26.327 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:26.328 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:28.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:28.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.587+0000 7fb17b400640 1 Processor -- start 2024-01-16T15:26:28.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.588+0000 7fb17b400640 1 -- start start 2024-01-16T15:26:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.589+0000 7fb17b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb174160bd0 0x7fb1740b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.589+0000 7fb17b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb174167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.589+0000 7fb17b400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1740b23e0 con 0x7fb1740b48d0 2024-01-16T15:26:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.589+0000 7fb17b400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1740cdc70 con 0x7fb174160bd0 2024-01-16T15:26:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.589+0000 7fb17b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740b2870 con 0x7fb1740b3210 2024-01-16T15:26:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb173e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb1740b48d0 0x7fb1740b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43112/0 (socket says 172.21.15.43:43112) 2024-01-16T15:26:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb173e00640 1 -- 172.21.15.43:0/2014502510 learned_addr learned my addr 172.21.15.43:0/2014502510 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb178a00640 1 --2- 172.21.15.43:0/2014502510 >> v2:172.21.15.69:3300/0 conn(0x7fb174160bd0 0x7fb1740b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb179400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb174167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833919211 0 0) 0x7fb1740b23e0 con 0x7fb1740b48d0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb154003520 con 0x7fb1740b48d0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.590+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365870483 0 0) 0x7fb154003520 con 0x7fb1740b48d0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 msgr2=0x7fb174167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb174167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 >> v2:172.21.15.69:3300/0 conn(0x7fb174160bd0 msgr2=0x7fb1740b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 --2- 172.21.15.43:0/2014502510 >> v2:172.21.15.69:3300/0 conn(0x7fb174160bd0 0x7fb1740b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740d41b0 con 0x7fb1740b48d0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fb15c001670 con 0x7fb1740b48d0 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 >> v1:172.21.15.43:6789/0 conn(0x7fb1740b48d0 legacy=0x7fb1740b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb154003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb154003f40 0x7fb1540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb154004d00 0x7fb154005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740cdc70 con 0x7fb154003f40 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740b2870 con 0x7fb154004d00 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15400ba30 con 0x7fb1740b3210 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb154003f40 0x7fb1540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb178a00640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb154003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 -- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 msgr2=0x7fb154003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1740b3210 0x7fb154003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 -- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb154004d00 msgr2=0x7fb154005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb154004d00 0x7fb154005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.591+0000 7fb173e00640 1 -- 172.21.15.43:0/2014502510 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154003520 con 0x7fb154003f40 2024-01-16T15:26:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.592+0000 7fb173e00640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb154003f40 0x7fb1540047c0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb1740b2a90 tx=0x7fb1540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6f90eeed6af674 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c005ad0 con 0x7fb154003f40 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c005c30 con 0x7fb154003f40 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb173400640 1 -- 172.21.15.43:0/2014502510 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02c7c0 con 0x7fb154003f40 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb17b400640 1 -- 172.21.15.43:0/2014502510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb154003f40 msgr2=0x7fb1540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb17b400640 1 --2- 172.21.15.43:0/2014502510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb154003f40 0x7fb1540047c0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb1740b2a90 tx=0x7fb1540057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb179400640 1 -- 172.21.15.43:0/2014502510 reap_dead start 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb17b400640 1 -- 172.21.15.43:0/2014502510 shutdown_connections 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.593+0000 7fb17b400640 1 -- 172.21.15.43:0/2014502510 >> 172.21.15.43:0/2014502510 conn(0x7fb17409c5a0 msgr2=0x7fb1740d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.594+0000 7fb17b400640 1 -- 172.21.15.43:0/2014502510 shutdown_connections 2024-01-16T15:26:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.594+0000 7fb17b400640 1 -- 172.21.15.43:0/2014502510 wait complete. 2024-01-16T15:26:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.595+0000 7fb17b400640 1 Processor -- start 2024-01-16T15:26:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.595+0000 7fb17b400640 1 -- start start 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb154003f40 0x7fb1740ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1740cc2a0 0x7fb17414f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17415ca90 0x7fb17415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17415d460 con 0x7fb1740cc2a0 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17414fb50 con 0x7fb17415ca90 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.596+0000 7fb17b400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740ca310 con 0x7fb154003f40 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1740cc2a0 0x7fb17414f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1740cc2a0 0x7fb17414f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57996/0 (socket says 172.21.15.43:57996) 2024-01-16T15:26:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 -- 172.21.15.43:0/1331531134 learned_addr learned my addr 172.21.15.43:0/1331531134 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb179400640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17415ca90 0x7fb17415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb178a00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb154003f40 0x7fb1740ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 -- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb154003f40 msgr2=0x7fb1740ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb154003f40 0x7fb1740ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 -- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17415ca90 msgr2=0x7fb17415cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17415ca90 0x7fb17415cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb173e00640 1 -- 172.21.15.43:0/1331531134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740c7460 con 0x7fb1740cc2a0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb178a00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb154003f40 0x7fb1740ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.597+0000 7fb179400640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb17415ca90 0x7fb17415cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb173e00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1740cc2a0 0x7fb17414f610 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fb15c000c00 tx=0x7fb15c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02dc60 con 0x7fb1740cc2a0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c02ddc0 con 0x7fb1740cc2a0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0072b0 con 0x7fb1740cc2a0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb17b400640 1 -- 172.21.15.43:0/1331531134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1740cc7a0 con 0x7fb1740cc2a0 2024-01-16T15:26:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.598+0000 7fb17b400640 1 -- 172.21.15.43:0/1331531134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1740ca840 con 0x7fb1740cc2a0 2024-01-16T15:26:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.599+0000 7fb17b400640 1 -- 172.21.15.43:0/1331531134 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1740a1210 con 0x7fb1740cc2a0 2024-01-16T15:26:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.600+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fb15c032070 con 0x7fb1740cc2a0 2024-01-16T15:26:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.601+0000 7fb170c00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb14c06c4f0 0x7fb14c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.601+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb15c0a9da0 con 0x7fb1740cc2a0 2024-01-16T15:26:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.601+0000 7fb178a00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb14c06c4f0 0x7fb14c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.602+0000 7fb178a00640 1 --2- 172.21.15.43:0/1331531134 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb14c06c4f0 0x7fb14c06e9b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb164009980 tx=0x7fb164008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.606+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb15c0762f0 con 0x7fb1740cc2a0 2024-01-16T15:26:28.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.834+0000 7fb17b400640 1 -- 172.21.15.43:0/1331531134 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1740b3760 con 0x7fb14c06c4f0 2024-01-16T15:26:28.846 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:28.848+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:28.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:28.848+0000 7fb170c00640 1 -- 172.21.15.43:0/1331531134 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb164012950 con 0x7fb14c06c4f0 2024-01-16T15:26:28.847 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:28.847 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:30.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.100+0000 7f34dd000640 1 Processor -- start 2024-01-16T15:26:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.100+0000 7f34dd000640 1 -- start start 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.101+0000 7f34dd000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34d80ae270 0x7f34d80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.101+0000 7f34dd000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d815b730 con 0x7f34d815b350 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.101+0000 7f34dd000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80ac740 con 0x7f34d809cb30 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.101+0000 7f34dd000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d80ac460 con 0x7f34d80ae270 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.102+0000 7f34d5e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f34d815b350 0x7f34d80b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43116/0 (socket says 172.21.15.43:43116) 2024-01-16T15:26:31.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.102+0000 7f34d5e00640 1 -- 172.21.15.43:0/3089412863 learned_addr learned my addr 172.21.15.43:0/3089412863 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:31.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.102+0000 7f34d6800640 1 --2- 172.21.15.43:0/3089412863 >> v2:172.21.15.69:3300/0 conn(0x7f34d80ae270 0x7f34d80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.102+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304989867 0 0) 0x7f34c8001110 con 0x7f34d815b350 2024-01-16T15:26:31.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.102+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b8003520 con 0x7f34d815b350 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3743826703 0 0) 0x7f34b8003520 con 0x7f34d815b350 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 >> v2:172.21.15.69:3300/0 conn(0x7f34d80ae270 msgr2=0x7f34d80abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 --2- 172.21.15.43:0/3089412863 >> v2:172.21.15.69:3300/0 conn(0x7f34d80ae270 0x7f34d80abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 >> v1:172.21.15.69:6789/0 conn(0x7f34d809cb30 legacy=0x7f34d80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80b2a50 con 0x7f34d815b350 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f34c8001800 con 0x7f34d815b350 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 >> v1:172.21.15.43:6789/0 conn(0x7f34d815b350 legacy=0x7f34d80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80039c0 0x7f34b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80047c0 0x7f34b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 0x7f34b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80ac460 con 0x7f34b8005180 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b800c080 con 0x7f34b80047c0 2024-01-16T15:26:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b800c240 con 0x7f34b80039c0 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.103+0000 7f34d6800640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 0x7f34b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d5e00640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80039c0 0x7f34b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d5400640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80047c0 0x7f34b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d6800640 1 -- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80039c0 msgr2=0x7f34b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d6800640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b80039c0 0x7f34b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d6800640 1 -- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80047c0 msgr2=0x7f34b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d6800640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80047c0 0x7f34b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d6800640 1 -- 172.21.15.43:0/3089412863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8003520 con 0x7f34b8005180 2024-01-16T15:26:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.104+0000 7f34d5400640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80047c0 0x7f34b8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.105+0000 7f34d6800640 1 -- 172.21.15.43:0/3089412863 reap_dead start 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.105+0000 7f34d6800640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 0x7f34b800b980 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009fc0 tx=0x7f34b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83049c31bb44ccbe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.105+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d002cc60 con 0x7f34b8005180 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.105+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d000f040 con 0x7f34b8005180 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.105+0000 7f34d4a00640 1 -- 172.21.15.43:0/3089412863 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0003990 con 0x7f34b8005180 2024-01-16T15:26:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 -- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 msgr2=0x7f34b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 0x7f34b800b980 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009fc0 tx=0x7f34b80057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 -- 172.21.15.43:0/3089412863 shutdown_connections 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 --2- 172.21.15.43:0/3089412863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b8005180 0x7f34b800b980 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 -- 172.21.15.43:0/3089412863 >> 172.21.15.43:0/3089412863 conn(0x7f34d80b0ba0 msgr2=0x7f34d80b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 -- 172.21.15.43:0/3089412863 shutdown_connections 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.106+0000 7f34dd000640 1 -- 172.21.15.43:0/3089412863 wait complete. 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.108+0000 7f34dd000640 1 Processor -- start 2024-01-16T15:26:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.108+0000 7f34dd000640 1 -- start start 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b8005180 0x7f34d8140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34d814de40 0x7f34d8151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34d8151a90 0x7f34d809aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81f3f30 con 0x7f34d8151a90 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8141010 con 0x7f34d814de40 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.109+0000 7f34dd000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81f3320 con 0x7f34b8005180 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34d8151a90 0x7f34d809aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34d8151a90 0x7f34d809aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58004/0 (socket says 172.21.15.43:58004) 2024-01-16T15:26:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 -- 172.21.15.43:0/969283486 learned_addr learned my addr 172.21.15.43:0/969283486 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d5e00640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b8005180 0x7f34d8140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d5400640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34d814de40 0x7f34d8151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 -- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b8005180 msgr2=0x7f34d8140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b8005180 0x7f34d8140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 -- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34d814de40 msgr2=0x7f34d8151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34d814de40 0x7f34d8151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d6800640 1 -- 172.21.15.43:0/969283486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d81f3600 con 0x7f34d8151a90 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.110+0000 7f34d5e00640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34b8005180 0x7f34d8140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34d6800640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34d8151a90 0x7f34d809aa50 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f34d002c6d0 tx=0x7f34d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0035dc0 con 0x7f34d8151a90 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d0003990 con 0x7f34d8151a90 2024-01-16T15:26:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34dd000640 1 -- 172.21.15.43:0/969283486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d81f38e0 con 0x7f34d8151a90 2024-01-16T15:26:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0003d80 con 0x7f34d8151a90 2024-01-16T15:26:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.111+0000 7f34dd000640 1 -- 172.21.15.43:0/969283486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d809af90 con 0x7f34d8151a90 2024-01-16T15:26:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.112+0000 7f34dd000640 1 -- 172.21.15.43:0/969283486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d81f40a0 con 0x7f34d8151a90 2024-01-16T15:26:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.113+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f34d0004030 con 0x7f34d8151a90 2024-01-16T15:26:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.115+0000 7f34ce000640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f34b006c4f0 0x7f34b006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.115+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34d00ab410 con 0x7f34d8151a90 2024-01-16T15:26:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.115+0000 7f34d5e00640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f34b006c4f0 0x7f34b006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.116+0000 7f34d5e00640 1 --2- 172.21.15.43:0/969283486 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f34b006c4f0 0x7f34b006e9b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f34c8001050 tx=0x7f34c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:31.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.119+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34d0077df0 con 0x7f34d8151a90 2024-01-16T15:26:31.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.352+0000 7f34dd000640 1 -- 172.21.15.43:0/969283486 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34d806e720 con 0x7f34b006c4f0 2024-01-16T15:26:31.360 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:31.362+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:31.362+0000 7f34ce000640 1 -- 172.21.15.43:0/969283486 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34c80108f0 con 0x7f34b006c4f0 2024-01-16T15:26:31.360 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:31.361 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:32.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:32.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:33.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:33.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.614+0000 7f3949e00640 1 Processor -- start 2024-01-16T15:26:33.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.614+0000 7f3949e00640 1 -- start start 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.616+0000 7f3949e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f394415b350 0x7f39440abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.616+0000 7f3949e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f394409cb30 0x7f39440b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.616+0000 7f3949e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394415b730 con 0x7f39440adb60 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.616+0000 7f3949e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39440ac740 con 0x7f394409cb30 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.616+0000 7f3949e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39440ac460 con 0x7f394415b350 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f393be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f394409cb30 0x7f39440b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f393be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f394409cb30 0x7f39440b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56080/0 (socket says 172.21.15.43:56080) 2024-01-16T15:26:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f393be00640 1 -- 172.21.15.43:0/2232731441 learned_addr learned my addr 172.21.15.43:0/2232731441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:33.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f3942a00640 1 --2- 172.21.15.43:0/2232731441 >> v2:172.21.15.69:3300/0 conn(0x7f394415b350 0x7f39440abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f393be00640 1 -- 172.21.15.43:0/2232731441 >> v2:172.21.15.69:3300/0 conn(0x7f394415b350 msgr2=0x7f39440abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:33.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.617+0000 7f393be00640 1 --2- 172.21.15.43:0/2232731441 >> v2:172.21.15.69:3300/0 conn(0x7f394415b350 0x7f39440abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.618+0000 7f393be00640 1 -- 172.21.15.43:0/2232731441 >> v1:172.21.15.69:6789/0 conn(0x7f39440adb60 legacy=0x7f39440adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.618+0000 7f393be00640 1 -- 172.21.15.43:0/2232731441 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39440b2a50 con 0x7f394409cb30 2024-01-16T15:26:33.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.619+0000 7f393be00640 1 --2- 172.21.15.43:0/2232731441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394409cb30 0x7f39440b3530 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f39440b5100 tx=0x7f392c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9590f39ce8922758 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.619+0000 7f3942000640 1 -- 172.21.15.43:0/2232731441 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c030860 con 0x7f394409cb30 2024-01-16T15:26:33.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.619+0000 7f3942000640 1 -- 172.21.15.43:0/2232731441 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c0309c0 con 0x7f394409cb30 2024-01-16T15:26:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.619+0000 7f3942000640 1 -- 172.21.15.43:0/2232731441 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c030d50 con 0x7f394409cb30 2024-01-16T15:26:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 -- 172.21.15.43:0/2232731441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394409cb30 msgr2=0x7f39440b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 --2- 172.21.15.43:0/2232731441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394409cb30 0x7f39440b3530 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f39440b5100 tx=0x7f392c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 -- 172.21.15.43:0/2232731441 shutdown_connections 2024-01-16T15:26:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 --2- 172.21.15.43:0/2232731441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394409cb30 0x7f39440b3530 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 --2- 172.21.15.43:0/2232731441 >> v2:172.21.15.69:3300/0 conn(0x7f394415b350 0x7f39440abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 -- 172.21.15.43:0/2232731441 >> 172.21.15.43:0/2232731441 conn(0x7f39440b0ba0 msgr2=0x7f39440b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 -- 172.21.15.43:0/2232731441 shutdown_connections 2024-01-16T15:26:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.620+0000 7f3949e00640 1 -- 172.21.15.43:0/2232731441 wait complete. 2024-01-16T15:26:33.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.622+0000 7f3949e00640 1 Processor -- start 2024-01-16T15:26:33.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.622+0000 7f3949e00640 1 -- start start 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394409cb30 0x7f394415d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39440adb60 0x7f394415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394415b350 0x7f394413f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394415e350 con 0x7f394415b350 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394415cb10 con 0x7f394409cb30 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3949e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394415dd90 con 0x7f39440adb60 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.623+0000 7f3943400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394415b350 0x7f394413f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394415b350 0x7f394413f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56096/0 (socket says 172.21.15.43:56096) 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 -- 172.21.15.43:0/3403696845 learned_addr learned my addr 172.21.15.43:0/3403696845 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3942a00640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394409cb30 0x7f394415d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f393be00640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39440adb60 0x7f394415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 -- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39440adb60 msgr2=0x7f394415c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39440adb60 0x7f394415c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 -- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394409cb30 msgr2=0x7f394415d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394409cb30 0x7f394415d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 -- 172.21.15.43:0/3403696845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0094d0 con 0x7f394415b350 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3942a00640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f394409cb30 0x7f394415d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.624+0000 7f3943400640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f394415b350 0x7f394413f9e0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c7a0 tx=0x7f393c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.625+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c003b70 con 0x7f394415b350 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.625+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c003cf0 con 0x7f394415b350 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.625+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c003ea0 con 0x7f394415b350 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.625+0000 7f3949e00640 1 -- 172.21.15.43:0/3403696845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394415e070 con 0x7f394415b350 2024-01-16T15:26:33.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.625+0000 7f3949e00640 1 -- 172.21.15.43:0/3403696845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39441548c0 con 0x7f394415b350 2024-01-16T15:26:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.626+0000 7f3949e00640 1 -- 172.21.15.43:0/3403696845 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3944154440 con 0x7f394415b350 2024-01-16T15:26:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.627+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f393c00aa90 con 0x7f394415b350 2024-01-16T15:26:33.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.628+0000 7f393b400640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f391c06c4c0 0x7f391c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.628+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f393c088800 con 0x7f394415b350 2024-01-16T15:26:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.628+0000 7f3942a00640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f391c06c4c0 0x7f391c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.629+0000 7f3942a00640 1 --2- 172.21.15.43:0/3403696845 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f391c06c4c0 0x7f391c06e980 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3934006fd0 tx=0x7f3934008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.633+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3944154440 con 0x7f394415b350 2024-01-16T15:26:33.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.860+0000 7f3949e00640 1 -- 172.21.15.43:0/3403696845 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39440b3a70 con 0x7f391c06c4c0 2024-01-16T15:26:33.867 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:33.868+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:33.869+0000 7f393b400640 1 -- 172.21.15.43:0/3403696845 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f393400b200 con 0x7f391c06c4c0 2024-01-16T15:26:33.867 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:35.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:36.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.123+0000 7f7f32200640 1 Processor -- start 2024-01-16T15:26:36.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.123+0000 7f7f32200640 1 -- start start 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f32200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7f2c06dae0 0x7f7f2c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f32200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f2c15bbd0 0x7f7f2c070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f32200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c144ca0 con 0x7f7f2c1448c0 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f32200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c15c450 con 0x7f7f2c06dae0 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f32200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0715b0 con 0x7f7f2c15bbd0 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.125+0000 7f7f2be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7f2c06dae0 0x7f7f2c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7f2c06dae0 0x7f7f2c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56106/0 (socket says 172.21.15.43:56106) 2024-01-16T15:26:36.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 -- 172.21.15.43:0/1568577267 learned_addr learned my addr 172.21.15.43:0/1568577267 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:36.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f31800640 1 --2- 172.21.15.43:0/1568577267 >> v2:172.21.15.69:3300/0 conn(0x7f7f2c15bbd0 0x7f7f2c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 -- 172.21.15.43:0/1568577267 >> v2:172.21.15.69:3300/0 conn(0x7f7f2c15bbd0 msgr2=0x7f7f2c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:36.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 --2- 172.21.15.43:0/1568577267 >> v2:172.21.15.69:3300/0 conn(0x7f7f2c15bbd0 0x7f7f2c070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:36.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 -- 172.21.15.43:0/1568577267 >> v1:172.21.15.69:6789/0 conn(0x7f7f2c1448c0 legacy=0x7f7f2c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:36.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.126+0000 7f7f2be00640 1 -- 172.21.15.43:0/1568577267 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c071890 con 0x7f7f2c06dae0 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.127+0000 7f7f2be00640 1 --2- 172.21.15.43:0/1568577267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c15b690 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b41b0 tx=0x7f7f1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bed602acad48e33 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f2b400640 1 -- 172.21.15.43:0/1568577267 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14004230 con 0x7f7f2c06dae0 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f2b400640 1 -- 172.21.15.43:0/1568577267 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14004390 con 0x7f7f2c06dae0 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f2b400640 1 -- 172.21.15.43:0/1568577267 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14030840 con 0x7f7f2c06dae0 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 -- 172.21.15.43:0/1568577267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 msgr2=0x7f7f2c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 --2- 172.21.15.43:0/1568577267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c15b690 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b41b0 tx=0x7f7f1402b980 comp rx=0 tx=0).stop 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 -- 172.21.15.43:0/1568577267 shutdown_connections 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 --2- 172.21.15.43:0/1568577267 >> v2:172.21.15.69:3300/0 conn(0x7f7f2c15bbd0 0x7f7f2c070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:36.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 --2- 172.21.15.43:0/1568577267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c15b690 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:36.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 -- 172.21.15.43:0/1568577267 >> 172.21.15.43:0/1568577267 conn(0x7f7f2c0b0aa0 msgr2=0x7f7f2c0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:36.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 -- 172.21.15.43:0/1568577267 shutdown_connections 2024-01-16T15:26:36.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.128+0000 7f7f32200640 1 -- 172.21.15.43:0/1568577267 wait complete. 2024-01-16T15:26:36.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.130+0000 7f7f32200640 1 Processor -- start 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.130+0000 7f7f32200640 1 -- start start 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c1375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f2c1448c0 0x7f7f2c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f2c15bbd0 0x7f7f2c1384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15d910 con 0x7f7f2c06dae0 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15e440 con 0x7f7f2c15bbd0 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.131+0000 7f7f32200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0b1fd0 con 0x7f7f2c1448c0 2024-01-16T15:26:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c1375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c1375f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56118/0 (socket says 172.21.15.43:56118) 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 -- 172.21.15.43:0/899563207 learned_addr learned my addr 172.21.15.43:0/899563207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f2be00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f2c1448c0 0x7f7f2c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f31800640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f2c15bbd0 0x7f7f2c1384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 -- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f2c1448c0 msgr2=0x7f7f2c15c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f2c1448c0 0x7f7f2c15c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 -- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f2c15bbd0 msgr2=0x7f7f2c1384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f2c15bbd0 0x7f7f2c1384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 -- 172.21.15.43:0/899563207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14009460 con 0x7f7f2c06dae0 2024-01-16T15:26:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f2be00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f2c1448c0 0x7f7f2c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.132+0000 7f7f30e00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2c06dae0 0x7f7f2c1375f0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00c450 tx=0x7f7f1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.133+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0052d0 con 0x7f7f2c06dae0 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.133+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c005430 con 0x7f7f2c06dae0 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.133+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c012870 con 0x7f7f2c06dae0 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.133+0000 7f7f32200640 1 -- 172.21.15.43:0/899563207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c0724f0 con 0x7f7f2c06dae0 2024-01-16T15:26:36.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.133+0000 7f7f32200640 1 -- 172.21.15.43:0/899563207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c15de80 con 0x7f7f2c06dae0 2024-01-16T15:26:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.134+0000 7f7f32200640 1 -- 172.21.15.43:0/899563207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c0cd140 con 0x7f7f2c06dae0 2024-01-16T15:26:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.135+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7f1c0129d0 con 0x7f7f2c06dae0 2024-01-16T15:26:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.136+0000 7f7f28c00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f0406c4c0 0x7f7f0406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.136+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7f1c088690 con 0x7f7f2c06dae0 2024-01-16T15:26:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.136+0000 7f7f2be00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f0406c4c0 0x7f7f0406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.137+0000 7f7f2be00640 1 --2- 172.21.15.43:0/899563207 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7f0406c4c0 0x7f7f0406e980 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7f14035000 tx=0x7f7f14008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.141+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f1c054be0 con 0x7f7f2c06dae0 2024-01-16T15:26:36.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.371+0000 7f7f32200640 1 -- 172.21.15.43:0/899563207 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f2c072370 con 0x7f7f0406c4c0 2024-01-16T15:26:36.380 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:36.382+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:36.382+0000 7f7f28c00640 1 -- 172.21.15.43:0/899563207 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7f1403e040 con 0x7f7f0406c4c0 2024-01-16T15:26:36.381 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:38.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.642+0000 7f28a8e00640 1 Processor -- start 2024-01-16T15:26:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.642+0000 7f28a8e00640 1 -- start start 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.643+0000 7f28a8e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28a40cfba0 0x7f28a415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a8e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f28a406e2b0 0x7f28a4161310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a8e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a406e720 con 0x7f28a40cbc60 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a8e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40d0120 con 0x7f28a406e2b0 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a8e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40cef90 con 0x7f28a40cfba0 2024-01-16T15:26:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a1e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f28a40cbc60 0x7f28a40cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59250/0 (socket says 172.21.15.43:59250) 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a1e00640 1 -- 172.21.15.43:0/854218968 learned_addr learned my addr 172.21.15.43:0/854218968 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f28a2800640 1 --2- 172.21.15.43:0/854218968 >> v2:172.21.15.43:3300/0 conn(0x7f28a406e2b0 0x7f28a4161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.644+0000 7f2899200640 1 --2- 172.21.15.43:0/854218968 >> v2:172.21.15.69:3300/0 conn(0x7f28a40cfba0 0x7f28a415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.645+0000 7f28a2800640 1 -- 172.21.15.43:0/854218968 >> v2:172.21.15.69:3300/0 conn(0x7f28a40cfba0 msgr2=0x7f28a415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.645+0000 7f28a2800640 1 --2- 172.21.15.43:0/854218968 >> v2:172.21.15.69:3300/0 conn(0x7f28a40cfba0 0x7f28a415dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.645+0000 7f28a2800640 1 -- 172.21.15.43:0/854218968 >> v1:172.21.15.43:6789/0 conn(0x7f28a40cbc60 legacy=0x7f28a40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.645+0000 7f28a2800640 1 -- 172.21.15.43:0/854218968 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40cecb0 con 0x7f28a406e2b0 2024-01-16T15:26:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.646+0000 7f28a2800640 1 --2- 172.21.15.43:0/854218968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a406e2b0 0x7f28a4161310 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f28a40d02c0 tx=0x7f289c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6642c7b4bb80c8d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.646+0000 7f28a1400640 1 -- 172.21.15.43:0/854218968 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c02cc80 con 0x7f28a406e2b0 2024-01-16T15:26:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.646+0000 7f28a1400640 1 -- 172.21.15.43:0/854218968 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c02cde0 con 0x7f28a406e2b0 2024-01-16T15:26:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.646+0000 7f28a1400640 1 -- 172.21.15.43:0/854218968 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c034660 con 0x7f28a406e2b0 2024-01-16T15:26:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 -- 172.21.15.43:0/854218968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a406e2b0 msgr2=0x7f28a4161310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 --2- 172.21.15.43:0/854218968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a406e2b0 0x7f28a4161310 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f28a40d02c0 tx=0x7f289c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 -- 172.21.15.43:0/854218968 shutdown_connections 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 --2- 172.21.15.43:0/854218968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a406e2b0 0x7f28a4161310 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 --2- 172.21.15.43:0/854218968 >> v2:172.21.15.69:3300/0 conn(0x7f28a40cfba0 0x7f28a415dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 -- 172.21.15.43:0/854218968 >> 172.21.15.43:0/854218968 conn(0x7f28a4071b10 msgr2=0x7f28a4071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 -- 172.21.15.43:0/854218968 shutdown_connections 2024-01-16T15:26:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.647+0000 7f28a8e00640 1 -- 172.21.15.43:0/854218968 wait complete. 2024-01-16T15:26:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.649+0000 7f28a8e00640 1 Processor -- start 2024-01-16T15:26:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.649+0000 7f28a8e00640 1 -- start start 2024-01-16T15:26:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28a406e2b0 0x7f28a4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28a40cbc60 0x7f28a40b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a40cfba0 0x7f28a40aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40c41b0 con 0x7f28a40cfba0 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a41534b0 con 0x7f28a40cbc60 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a8e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40b2e30 con 0x7f28a406e2b0 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a40cfba0 0x7f28a40aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a40cfba0 0x7f28a40aec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56136/0 (socket says 172.21.15.43:56136) 2024-01-16T15:26:38.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.650+0000 7f28a2800640 1 -- 172.21.15.43:0/4262101258 learned_addr learned my addr 172.21.15.43:0/4262101258 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a1e00640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28a406e2b0 0x7f28a4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f2899200640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28a40cbc60 0x7f28a40b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 -- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28a406e2b0 msgr2=0x7f28a4152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28a406e2b0 0x7f28a4152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 -- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28a40cbc60 msgr2=0x7f28a40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28a40cbc60 0x7f28a40b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 -- 172.21.15.43:0/4262101258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009a70 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a1e00640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f28a406e2b0 0x7f28a4152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a2800640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28a40cfba0 0x7f28a40aec90 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f289c02c4e0 tx=0x7f289c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c040650 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c0407b0 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c040b60 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.651+0000 7f28a8e00640 1 -- 172.21.15.43:0/4262101258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a40af1d0 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.652+0000 7f28a8e00640 1 -- 172.21.15.43:0/4262101258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a40b35f0 con 0x7f28a40cfba0 2024-01-16T15:26:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.653+0000 7f28a8e00640 1 -- 172.21.15.43:0/4262101258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a40c42f0 con 0x7f28a40cfba0 2024-01-16T15:26:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.654+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f289c040cc0 con 0x7f28a40cfba0 2024-01-16T15:26:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.655+0000 7f289aa00640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f287c06c4f0 0x7f287c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.655+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f289c0ab500 con 0x7f28a40cfba0 2024-01-16T15:26:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.655+0000 7f28a1e00640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f287c06c4f0 0x7f287c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.656+0000 7f28a1e00640 1 --2- 172.21.15.43:0/4262101258 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f287c06c4f0 0x7f287c06e9b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2894005880 tx=0x7f2894009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.659+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28a40c42f0 con 0x7f28a40cfba0 2024-01-16T15:26:38.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.885+0000 7f28a8e00640 1 -- 172.21.15.43:0/4262101258 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28a406cb40 con 0x7f287c06c4f0 2024-01-16T15:26:38.892 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:38.894+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:38.894+0000 7f289aa00640 1 -- 172.21.15.43:0/4262101258 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2894001ce0 con 0x7f287c06c4f0 2024-01-16T15:26:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:41.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:41.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.249+0000 7f25dc000640 1 Processor -- start 2024-01-16T15:26:41.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.249+0000 7f25dc000640 1 -- start start 2024-01-16T15:26:41.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25dc000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40acd70 0x7f25d406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25dc000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25d40ae490 0x7f25d40ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25dc000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d406d7a0 con 0x7f25d40ad690 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25dc000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d415a5b0 con 0x7f25d40ae490 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25dc000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d406ce70 con 0x7f25d40acd70 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25da000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f25d40ad690 0x7f25d4070490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59256/0 (socket says 172.21.15.43:59256) 2024-01-16T15:26:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.251+0000 7f25da000640 1 -- 172.21.15.43:0/2144699287 learned_addr learned my addr 172.21.15.43:0/2144699287 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190588382 0 0) 0x7f25d406d7a0 con 0x7f25d40ad690 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d8c00640 1 --2- 172.21.15.43:0/2144699287 >> v2:172.21.15.69:3300/0 conn(0x7f25d40ae490 0x7f25d40ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d9600640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40acd70 0x7f25d406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b4003520 con 0x7f25d40ad690 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703045033 0 0) 0x7f25b4003520 con 0x7f25d40ad690 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40acd70 msgr2=0x7f25d406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40acd70 0x7f25d406dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 >> v2:172.21.15.69:3300/0 conn(0x7f25d40ae490 msgr2=0x7f25d40ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 --2- 172.21.15.43:0/2144699287 >> v2:172.21.15.69:3300/0 conn(0x7f25d40ae490 0x7f25d40ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d406d150 con 0x7f25d40ad690 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f25cc0019f0 con 0x7f25d40ad690 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 >> v1:172.21.15.43:6789/0 conn(0x7f25d40ad690 legacy=0x7f25d4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:41.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.252+0000 7f25d3e00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4003970 0x7f25b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d3e00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b40047c0 0x7f25b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d3e00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4005180 0x7f25b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d415a5b0 con 0x7f25b40047c0 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d406ce70 con 0x7f25b4005180 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b400bfc0 con 0x7f25b4003970 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d9600640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40acd70 0x7f25d406dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b40047c0 0x7f25b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 -- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4003970 msgr2=0x7f25b4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b4003970 0x7f25b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 -- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4005180 msgr2=0x7f25b400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b4005180 0x7f25b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.253+0000 7f25d8c00640 1 -- 172.21.15.43:0/2144699287 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4003520 con 0x7f25b40047c0 2024-01-16T15:26:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.254+0000 7f25d8c00640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b40047c0 0x7f25b4004c40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00ba60 tx=0x7f25b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf21de6994f27e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc032660 con 0x7f25b40047c0 2024-01-16T15:26:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc0327c0 con 0x7f25b40047c0 2024-01-16T15:26:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25d3e00640 1 -- 172.21.15.43:0/2144699287 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc032b90 con 0x7f25b40047c0 2024-01-16T15:26:41.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25dc000640 1 -- 172.21.15.43:0/2144699287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b40047c0 msgr2=0x7f25b4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25dc000640 1 --2- 172.21.15.43:0/2144699287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25b40047c0 0x7f25b4004c40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00ba60 tx=0x7f25b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.255+0000 7f25da000640 1 -- 172.21.15.43:0/2144699287 reap_dead start 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.256+0000 7f25dc000640 1 -- 172.21.15.43:0/2144699287 shutdown_connections 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.256+0000 7f25dc000640 1 -- 172.21.15.43:0/2144699287 >> 172.21.15.43:0/2144699287 conn(0x7f25d40b0ba0 msgr2=0x7f25d40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.256+0000 7f25dc000640 1 -- 172.21.15.43:0/2144699287 shutdown_connections 2024-01-16T15:26:41.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.256+0000 7f25dc000640 1 -- 172.21.15.43:0/2144699287 wait complete. 2024-01-16T15:26:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.257+0000 7f25dc000640 1 Processor -- start 2024-01-16T15:26:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.258+0000 7f25dc000640 1 -- start start 2024-01-16T15:26:41.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b40047c0 0x7f25d415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40c9580 0x7f25d40ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d414f720 0x7f25d414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d41500e0 con 0x7f25d414f720 2024-01-16T15:26:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d40ce5c0 con 0x7f25b40047c0 2024-01-16T15:26:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25dc000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d40ca890 con 0x7f25d40c9580 2024-01-16T15:26:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d414f720 0x7f25d414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d414f720 0x7f25d414fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56152/0 (socket says 172.21.15.43:56152) 2024-01-16T15:26:41.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 -- 172.21.15.43:0/925593823 learned_addr learned my addr 172.21.15.43:0/925593823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 -- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40c9580 msgr2=0x7f25d40ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25d8c00640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40c9580 0x7f25d40ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25d9600640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b40047c0 0x7f25d415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40c9580 0x7f25d40ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 -- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b40047c0 msgr2=0x7f25d415bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b40047c0 0x7f25d415bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.259+0000 7f25da000640 1 -- 172.21.15.43:0/925593823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40c9a00 con 0x7f25d414f720 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25d8c00640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25d40c9580 0x7f25d40ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25da000640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d414f720 0x7f25d414fba0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f25cc000f80 tx=0x7f25cc00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:41.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00a670 con 0x7f25d414f720 2024-01-16T15:26:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc00a810 con 0x7f25d414f720 2024-01-16T15:26:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25dc000640 1 -- 172.21.15.43:0/925593823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d40c9c80 con 0x7f25d414f720 2024-01-16T15:26:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00abc0 con 0x7f25d414f720 2024-01-16T15:26:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.260+0000 7f25dc000640 1 -- 172.21.15.43:0/925593823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d40ca240 con 0x7f25d414f720 2024-01-16T15:26:41.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.261+0000 7f25dc000640 1 -- 172.21.15.43:0/925593823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d409d200 con 0x7f25d414f720 2024-01-16T15:26:41.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.263+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f25cc00ad20 con 0x7f25d414f720 2024-01-16T15:26:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.264+0000 7f25d1600640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25ac06c4c0 0x7f25ac06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.264+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f25cc088440 con 0x7f25d414f720 2024-01-16T15:26:41.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.264+0000 7f25d9600640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25ac06c4c0 0x7f25ac06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:41.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.265+0000 7f25d9600640 1 --2- 172.21.15.43:0/925593823 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f25ac06c4c0 0x7f25ac06e980 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f25c4004790 tx=0x7f25c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:41.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.268+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25cc054990 con 0x7f25d414f720 2024-01-16T15:26:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.496+0000 7f25dc000640 1 -- 172.21.15.43:0/925593823 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25d40b3d90 con 0x7f25ac06c4c0 2024-01-16T15:26:41.503 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:41.505+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:41.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:41.505+0000 7f25d1600640 1 -- 172.21.15.43:0/925593823 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f25c4004020 con 0x7f25ac06c4c0 2024-01-16T15:26:41.504 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:41.504 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:43.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:43.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.763+0000 7ff3e0c00640 1 Processor -- start 2024-01-16T15:26:43.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.763+0000 7ff3e0c00640 1 -- start start 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3e0c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff3dc143ff0 0x7ff3dc06e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3e0c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 0x7ff3dc0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3e0c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc0c79a0 con 0x7ff3dc144650 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3e0c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc06f0d0 con 0x7ff3dc143ff0 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3e0c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0c7ed0 con 0x7ff3dc06ec40 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.765+0000 7ff3d9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff3dc143ff0 0x7ff3dc06e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff3dc143ff0 0x7ff3dc06e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52228/0 (socket says 172.21.15.43:52228) 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3361887678 learned_addr learned my addr 172.21.15.43:0/3361887678 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3361887678 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 msgr2=0x7ff3dc0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:43.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3da800640 1 --2- 172.21.15.43:0/3361887678 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 0x7ff3dc0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 --2- 172.21.15.43:0/3361887678 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 0x7ff3dc0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3361887678 >> v1:172.21.15.69:6789/0 conn(0x7ff3dc144650 legacy=0x7ff3dc0c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:43.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3361887678 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0ced50 con 0x7ff3dc143ff0 2024-01-16T15:26:43.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.766+0000 7ff3da800640 1 --2- 172.21.15.43:0/3361887678 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 0x7ff3dc0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.767+0000 7ff3d9e00640 1 --2- 172.21.15.43:0/3361887678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc06e700 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc144ae0 tx=0x7ff3cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3196220e28d3b835 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3d8a00640 1 -- 172.21.15.43:0/3361887678 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031870 con 0x7ff3dc143ff0 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3d8a00640 1 -- 172.21.15.43:0/3361887678 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc0319d0 con 0x7ff3dc143ff0 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3d8a00640 1 -- 172.21.15.43:0/3361887678 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031d60 con 0x7ff3dc143ff0 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3361887678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc143ff0 msgr2=0x7ff3dc06e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 --2- 172.21.15.43:0/3361887678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc06e700 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc144ae0 tx=0x7ff3cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:26:43.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3361887678 shutdown_connections 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 --2- 172.21.15.43:0/3361887678 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc06ec40 0x7ff3dc0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 --2- 172.21.15.43:0/3361887678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc06e700 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3361887678 >> 172.21.15.43:0/3361887678 conn(0x7ff3dc0661f0 msgr2=0x7ff3dc0665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3361887678 shutdown_connections 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.768+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3361887678 wait complete. 2024-01-16T15:26:43.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.770+0000 7ff3e0c00640 1 Processor -- start 2024-01-16T15:26:43.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.770+0000 7ff3e0c00640 1 -- start start 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc06ec40 0x7ff3dc0d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc0d1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc144650 0x7ff3dc06b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1f40c0 con 0x7ff3dc06ec40 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1f31b0 con 0x7ff3dc143ff0 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.771+0000 7ff3e0c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1f3440 con 0x7ff3dc144650 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.772+0000 7ff3d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc0d1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.772+0000 7ff3d9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc06ec40 0x7ff3dc0d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.772+0000 7ff3d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc0d1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:49634/0 (socket says 172.21.15.43:49634) 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.772+0000 7ff3d9400640 1 -- 172.21.15.43:0/3928745888 learned_addr learned my addr 172.21.15.43:0/3928745888 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:43.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.772+0000 7ff3da800640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc144650 0x7ff3dc06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3928745888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc144650 msgr2=0x7ff3dc06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc144650 0x7ff3dc06b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3928745888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc143ff0 msgr2=0x7ff3dc0d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc143ff0 0x7ff3dc0d1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 -- 172.21.15.43:0/3928745888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0094d0 con 0x7ff3dc06ec40 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d9e00640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3dc06ec40 0x7ff3dc0d10e0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc033000 tx=0x7ff3cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.773+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc0443f0 con 0x7ff3dc06ec40 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.774+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc044570 con 0x7ff3dc06ec40 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.774+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc044920 con 0x7ff3dc06ec40 2024-01-16T15:26:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.774+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3928745888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc1f3720 con 0x7ff3dc06ec40 2024-01-16T15:26:43.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.774+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3928745888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc1f3ce0 con 0x7ff3dc06ec40 2024-01-16T15:26:43.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.775+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3928745888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc1f4200 con 0x7ff3dc06ec40 2024-01-16T15:26:43.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.776+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff3cc04d450 con 0x7ff3dc06ec40 2024-01-16T15:26:43.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.777+0000 7ff3d2000640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff3b406c4c0 0x7ff3b406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.777+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff3cc02d040 con 0x7ff3dc06ec40 2024-01-16T15:26:43.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.777+0000 7ff3d9400640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff3b406c4c0 0x7ff3b406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.778+0000 7ff3d9400640 1 --2- 172.21.15.43:0/3928745888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff3b406c4c0 0x7ff3b406e980 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc0d0c80 tx=0x7ff3c4009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:43.781+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3dc1f4200 con 0x7ff3dc06ec40 2024-01-16T15:26:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:44.007+0000 7ff3e0c00640 1 -- 172.21.15.43:0/3928745888 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3dc1443d0 con 0x7ff3b406c4c0 2024-01-16T15:26:44.018 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:44.020+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:44.020+0000 7ff3d2000640 1 -- 172.21.15.43:0/3928745888 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff3c40095e0 con 0x7ff3b406c4c0 2024-01-16T15:26:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:46.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:46.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.273+0000 7f66d3000640 1 Processor -- start 2024-01-16T15:26:46.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.273+0000 7f66d3000640 1 -- start start 2024-01-16T15:26:46.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.274+0000 7f66d3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66cc15b350 0x7f66cc0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc09cb30 0x7f66cc0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc15b730 con 0x7f66cc0adb60 2024-01-16T15:26:46.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66cc0ac740 con 0x7f66cc15b350 2024-01-16T15:26:46.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0ac460 con 0x7f66cc09cb30 2024-01-16T15:26:46.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d1000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f66cc0adb60 0x7f66cc0adfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:46102/0 (socket says 172.21.15.43:46102) 2024-01-16T15:26:46.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc09cb30 0x7f66cc0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66d1000640 1 -- 172.21.15.43:0/3033679472 learned_addr learned my addr 172.21.15.43:0/3033679472 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:46.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.275+0000 7f66cbe00640 1 --2- 172.21.15.43:0/3033679472 >> v2:172.21.15.69:3300/0 conn(0x7f66cc15b350 0x7f66cc0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.276+0000 7f66cbe00640 1 -- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc09cb30 msgr2=0x7f66cc0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:46.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.276+0000 7f66cbe00640 1 --2- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc09cb30 0x7f66cc0b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.276+0000 7f66cbe00640 1 -- 172.21.15.43:0/3033679472 >> v1:172.21.15.69:6789/0 conn(0x7f66cc0adb60 legacy=0x7f66cc0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:46.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.276+0000 7f66cbe00640 1 -- 172.21.15.43:0/3033679472 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0b2a50 con 0x7f66cc15b350 2024-01-16T15:26:46.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.278+0000 7f66cbe00640 1 --2- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc15b350 0x7f66cc0abe80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0097a0 tx=0x7f66bc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8d87665735fa77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:46.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.278+0000 7f66cb400640 1 -- 172.21.15.43:0/3033679472 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031830 con 0x7f66cc15b350 2024-01-16T15:26:46.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66cb400640 1 -- 172.21.15.43:0/3033679472 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc031990 con 0x7f66cc15b350 2024-01-16T15:26:46.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66cb400640 1 -- 172.21.15.43:0/3033679472 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031d20 con 0x7f66cc15b350 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 -- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc15b350 msgr2=0x7f66cc0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 --2- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc15b350 0x7f66cc0abe80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f66bc0097a0 tx=0x7f66bc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 -- 172.21.15.43:0/3033679472 shutdown_connections 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 --2- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc09cb30 0x7f66cc0b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 --2- 172.21.15.43:0/3033679472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc15b350 0x7f66cc0abe80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.279+0000 7f66d3000640 1 -- 172.21.15.43:0/3033679472 >> 172.21.15.43:0/3033679472 conn(0x7f66cc0b0ba0 msgr2=0x7f66cc0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.280+0000 7f66d3000640 1 -- 172.21.15.43:0/3033679472 shutdown_connections 2024-01-16T15:26:46.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.280+0000 7f66d3000640 1 -- 172.21.15.43:0/3033679472 wait complete. 2024-01-16T15:26:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.281+0000 7f66d3000640 1 Processor -- start 2024-01-16T15:26:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.281+0000 7f66d3000640 1 -- start start 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc09cb30 0x7f66cc0af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66cc0adb60 0x7f66cc158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc15b350 0x7f66cc13f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc158550 con 0x7f66cc0adb60 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0afb40 con 0x7f66cc09cb30 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.282+0000 7f66d3000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0afe20 con 0x7f66cc15b350 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66cc0adb60 0x7f66cc158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66cc0adb60 0x7f66cc158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52242/0 (socket says 172.21.15.43:52242) 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 -- 172.21.15.43:0/2163715947 learned_addr learned my addr 172.21.15.43:0/2163715947 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66cbe00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc09cb30 0x7f66cc0af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66d1000640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc15b350 0x7f66cc13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 -- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc15b350 msgr2=0x7f66cc13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66cc15b350 0x7f66cc13f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 -- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc09cb30 msgr2=0x7f66cc0af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66cc09cb30 0x7f66cc0af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:46.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.283+0000 7f66c3e00640 1 -- 172.21.15.43:0/2163715947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc009400 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66c3e00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66cc0adb60 0x7f66cc158010 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f66b400e9b0 tx=0x7f66b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4009670 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b400c9a0 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66d3000640 1 -- 172.21.15.43:0/2163715947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc0b0100 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b400cd50 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.284+0000 7f66d3000640 1 -- 172.21.15.43:0/2163715947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc153410 con 0x7f66cc0adb60 2024-01-16T15:26:46.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.285+0000 7f66d3000640 1 -- 172.21.15.43:0/2163715947 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc09d200 con 0x7f66cc0adb60 2024-01-16T15:26:46.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.286+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f66b4013470 con 0x7f66cc0adb60 2024-01-16T15:26:46.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.288+0000 7f66c8c00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66a406c4c0 0x7f66a406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:46.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.288+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66b4088d60 con 0x7f66cc0adb60 2024-01-16T15:26:46.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.288+0000 7f66cbe00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66a406c4c0 0x7f66a406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:46.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.289+0000 7f66cbe00640 1 --2- 172.21.15.43:0/2163715947 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66a406c4c0 0x7f66a406e980 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f66bc036000 tx=0x7f66bc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:46.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.292+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66b40552b0 con 0x7f66cc0adb60 2024-01-16T15:26:46.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.523+0000 7f66d3000640 1 -- 172.21.15.43:0/2163715947 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66cc0b3a70 con 0x7f66a406c4c0 2024-01-16T15:26:46.531 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:46.533+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:46.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:46.533+0000 7f66c8c00640 1 -- 172.21.15.43:0/2163715947 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f66bc03a420 con 0x7f66a406c4c0 2024-01-16T15:26:46.532 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:46.532 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:48.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:48.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.786+0000 7f2c3f000640 1 Processor -- start 2024-01-16T15:26:48.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.786+0000 7f2c3f000640 1 -- start start 2024-01-16T15:26:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.788+0000 7f2c3f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c38162600 0x7f2c38162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.788+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c3806c3c0 0x7f2c3806c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.788+0000 7f2c3f000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c38162fa0 con 0x7f2c38162220 2024-01-16T15:26:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.788+0000 7f2c3f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c380706a0 con 0x7f2c38162600 2024-01-16T15:26:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.788+0000 7f2c3f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38070c60 con 0x7f2c3806c3c0 2024-01-16T15:26:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c37e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2c38162220 0x7f2c380b1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48152/0 (socket says 172.21.15.43:48152) 2024-01-16T15:26:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c37e00640 1 -- 172.21.15.43:0/3572043581 learned_addr learned my addr 172.21.15.43:0/3572043581 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c37400640 1 --2- 172.21.15.43:0/3572043581 >> v2:172.21.15.69:3300/0 conn(0x7f2c38162600 0x7f2c38162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c3d000640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c3806c3c0 0x7f2c3806c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522253995 0 0) 0x7f2c38162fa0 con 0x7f2c38162220 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.789+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c18003520 con 0x7f2c38162220 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 832758846 0 0) 0x7f2c18003520 con 0x7f2c38162220 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c3806c3c0 msgr2=0x7f2c3806c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c3806c3c0 0x7f2c3806c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 >> v2:172.21.15.69:3300/0 conn(0x7f2c38162600 msgr2=0x7f2c38162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 --2- 172.21.15.43:0/3572043581 >> v2:172.21.15.69:3300/0 conn(0x7f2c38162600 0x7f2c38162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3806cd60 con 0x7f2c38162220 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f2c28001670 con 0x7f2c38162220 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 >> v1:172.21.15.43:6789/0 conn(0x7f2c38162220 legacy=0x7f2c380b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18003970 0x7f2c18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c18004360 0x7f2c18004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c18004eb0 0x7f2c1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c380706a0 con 0x7f2c18004360 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38070c60 con 0x7f2c18004eb0 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1800bec0 con 0x7f2c18003970 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.790+0000 7f2c37400640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c18004360 0x7f2c18004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37e00640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18003970 0x7f2c18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c3d000640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c18004eb0 0x7f2c1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37400640 1 -- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18003970 msgr2=0x7f2c18003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37400640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18003970 0x7f2c18003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37400640 1 -- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c18004eb0 msgr2=0x7f2c1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37400640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c18004eb0 0x7f2c1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.791+0000 7f2c37400640 1 -- 172.21.15.43:0/3572043581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18003520 con 0x7f2c18004360 2024-01-16T15:26:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.792+0000 7f2c37400640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c18004360 0x7f2c18004840 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009ac0 tx=0x7f2c180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a2fba9f5e678ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.792+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2002c710 con 0x7f2c18004360 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.792+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2002c870 con 0x7f2c18004360 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.792+0000 7f2c36a00640 1 -- 172.21.15.43:0/3572043581 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2002cc00 con 0x7f2c18004360 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 -- 172.21.15.43:0/3572043581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c18004360 msgr2=0x7f2c18004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 --2- 172.21.15.43:0/3572043581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c18004360 0x7f2c18004840 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009ac0 tx=0x7f2c180057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3d000640 1 -- 172.21.15.43:0/3572043581 reap_dead start 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 -- 172.21.15.43:0/3572043581 shutdown_connections 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 -- 172.21.15.43:0/3572043581 >> 172.21.15.43:0/3572043581 conn(0x7f2c380b0ba0 msgr2=0x7f2c380b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 -- 172.21.15.43:0/3572043581 shutdown_connections 2024-01-16T15:26:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.793+0000 7f2c3f000640 1 -- 172.21.15.43:0/3572043581 wait complete. 2024-01-16T15:26:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.796+0000 7f2c3f000640 1 Processor -- start 2024-01-16T15:26:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.796+0000 7f2c3f000640 1 -- start start 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18004360 0x7f2c38152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c380b3a30 0x7f2c380b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c380b1fb0 0x7f2c380b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c380b3550 con 0x7f2c380b3a30 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38099a00 con 0x7f2c380b1fb0 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.797+0000 7f2c3f000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38099ce0 con 0x7f2c18004360 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c380b3a30 0x7f2c380b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c380b3a30 0x7f2c380b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52248/0 (socket says 172.21.15.43:52248) 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 -- 172.21.15.43:0/932379780 learned_addr learned my addr 172.21.15.43:0/932379780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37e00640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18004360 0x7f2c38152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c3d000640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c380b1fb0 0x7f2c380b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 -- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18004360 msgr2=0x7f2c38152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2c18004360 0x7f2c38152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 -- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c380b1fb0 msgr2=0x7f2c380b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c380b1fb0 0x7f2c380b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c37400640 1 -- 172.21.15.43:0/932379780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38099fc0 con 0x7f2c380b3a30 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.798+0000 7f2c3d000640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c380b1fb0 0x7f2c380b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c37400640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c380b3a30 0x7f2c380b1a70 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009ac0 tx=0x7f2c20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20003810 con 0x7f2c380b3a30 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2002de90 con 0x7f2c380b3a30 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20003b90 con 0x7f2c380b3a30 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c3f000640 1 -- 172.21.15.43:0/932379780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3809a2a0 con 0x7f2c380b3a30 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.799+0000 7f2c3f000640 1 -- 172.21.15.43:0/932379780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c38153400 con 0x7f2c380b3a30 2024-01-16T15:26:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.800+0000 7f2c3f000640 1 -- 172.21.15.43:0/932379780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3809d200 con 0x7f2c380b3a30 2024-01-16T15:26:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.802+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2c20003cf0 con 0x7f2c380b3a30 2024-01-16T15:26:48.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.803+0000 7f2c2fe00640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c1006c4c0 0x7f2c1006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.803+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2c200aa7f0 con 0x7f2c380b3a30 2024-01-16T15:26:48.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.803+0000 7f2c37e00640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c1006c4c0 0x7f2c1006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.804+0000 7f2c37e00640 1 --2- 172.21.15.43:0/932379780 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2c1006c4c0 0x7f2c1006e980 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f2c3815dfe0 tx=0x7f2c28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:48.807+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c200770b0 con 0x7f2c380b3a30 2024-01-16T15:26:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:49.037+0000 7f2c3f000640 1 -- 172.21.15.43:0/932379780 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c3806d210 con 0x7f2c1006c4c0 2024-01-16T15:26:49.046 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:49.048+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:49.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:49.048+0000 7f2c2fe00640 1 -- 172.21.15.43:0/932379780 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2c3806d210 con 0x7f2c1006c4c0 2024-01-16T15:26:49.047 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:49.047 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:51.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.311+0000 7effaf200640 1 Processor -- start 2024-01-16T15:26:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.311+0000 7effaf200640 1 -- start start 2024-01-16T15:26:51.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7effa8143ff0 0x7effa806e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa8144650 0x7effa806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7effa80c5ef0 0x7effa80c6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80c79a0 con 0x7effa8143ff0 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80c7ed0 con 0x7effa80c5ef0 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.313+0000 7effaf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80ced50 con 0x7effa8144650 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7effa8143ff0 0x7effa806e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7effa8143ff0 0x7effa806e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52264/0 (socket says 172.21.15.43:52264) 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 -- 172.21.15.43:0/3019692545 learned_addr learned my addr 172.21.15.43:0/3019692545 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7400640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa8144650 0x7effa806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effad200640 1 --2- 172.21.15.43:0/3019692545 >> v2:172.21.15.69:3300/0 conn(0x7effa80c5ef0 0x7effa80c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 -- 172.21.15.43:0/3019692545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa8144650 msgr2=0x7effa806ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa8144650 0x7effa806ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 -- 172.21.15.43:0/3019692545 >> v2:172.21.15.69:3300/0 conn(0x7effa80c5ef0 msgr2=0x7effa80c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 --2- 172.21.15.43:0/3019692545 >> v2:172.21.15.69:3300/0 conn(0x7effa80c5ef0 0x7effa80c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.314+0000 7effa7e00640 1 -- 172.21.15.43:0/3019692545 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa81443d0 con 0x7effa8143ff0 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.315+0000 7effa7e00640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa806e700 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7effa815ed00 tx=0x7eff9802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba6ba814fa28bf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.316+0000 7effa6a00640 1 -- 172.21.15.43:0/3019692545 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9802ca70 con 0x7effa8143ff0 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.316+0000 7effa6a00640 1 -- 172.21.15.43:0/3019692545 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9802cbd0 con 0x7effa8143ff0 2024-01-16T15:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.316+0000 7effa6a00640 1 -- 172.21.15.43:0/3019692545 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff980343f0 con 0x7effa8143ff0 2024-01-16T15:26:51.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.316+0000 7effaf200640 1 -- 172.21.15.43:0/3019692545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 msgr2=0x7effa806e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:51.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.316+0000 7effaf200640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa806e700 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7effa815ed00 tx=0x7eff9802bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 -- 172.21.15.43:0/3019692545 shutdown_connections 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 --2- 172.21.15.43:0/3019692545 >> v2:172.21.15.69:3300/0 conn(0x7effa80c5ef0 0x7effa80c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa8144650 0x7effa806ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 --2- 172.21.15.43:0/3019692545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa806e700 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 -- 172.21.15.43:0/3019692545 >> 172.21.15.43:0/3019692545 conn(0x7effa80661f0 msgr2=0x7effa80665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 -- 172.21.15.43:0/3019692545 shutdown_connections 2024-01-16T15:26:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.317+0000 7effaf200640 1 -- 172.21.15.43:0/3019692545 wait complete. 2024-01-16T15:26:51.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.319+0000 7effaf200640 1 Processor -- start 2024-01-16T15:26:51.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.319+0000 7effaf200640 1 -- start start 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa80c5ef0 0x7effa815df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa81f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effa8144650 0x7effa815d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa81e5680 con 0x7effa8143ff0 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa81e4ab0 con 0x7effa8144650 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effaf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa81e4d50 con 0x7effa80c5ef0 2024-01-16T15:26:51.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa81f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa81f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52276/0 (socket says 172.21.15.43:52276) 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 -- 172.21.15.43:0/2422037083 learned_addr learned my addr 172.21.15.43:0/2422037083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7e00640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa80c5ef0 0x7effa815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effad200640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effa8144650 0x7effa815d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 -- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa80c5ef0 msgr2=0x7effa815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effa80c5ef0 0x7effa815df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 -- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effa8144650 msgr2=0x7effa815d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effa8144650 0x7effa815d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.321+0000 7effa7400640 1 -- 172.21.15.43:0/2422037083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98009b30 con 0x7effa8143ff0 2024-01-16T15:26:51.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7effa7400640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8143ff0 0x7effa81f31a0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7eff9000d800 tx=0x7eff9000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9000b6c0 con 0x7effa8143ff0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9000b820 con 0x7effa8143ff0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9000bbb0 con 0x7effa8143ff0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7effaf200640 1 -- 172.21.15.43:0/2422037083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa81e5030 con 0x7effa8143ff0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.322+0000 7effaf200640 1 -- 172.21.15.43:0/2422037083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa8142bd0 con 0x7effa8143ff0 2024-01-16T15:26:51.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.323+0000 7effaf200640 1 -- 172.21.15.43:0/2422037083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80cf1c0 con 0x7effa8143ff0 2024-01-16T15:26:51.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.324+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7eff9000bd10 con 0x7effa8143ff0 2024-01-16T15:26:51.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.325+0000 7eff9fe00640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff8006c4f0 0x7eff8006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:51.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.325+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7eff90088d20 con 0x7effa8143ff0 2024-01-16T15:26:51.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.326+0000 7effa7e00640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff8006c4f0 0x7eff8006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:51.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.326+0000 7effa7e00640 1 --2- 172.21.15.43:0/2422037083 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7eff8006c4f0 0x7eff8006e9b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7eff98035000 tx=0x7eff98030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:51.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.332+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effa80cf1c0 con 0x7effa8143ff0 2024-01-16T15:26:51.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.562+0000 7effaf200640 1 -- 172.21.15.43:0/2422037083 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effa80b1290 con 0x7eff8006c4f0 2024-01-16T15:26:51.570 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:51.571+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:51.571+0000 7eff9fe00640 1 -- 172.21.15.43:0/2422037083 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7effa80b1290 con 0x7eff8006c4f0 2024-01-16T15:26:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:53.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:53.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.837+0000 7fe720a00640 1 Processor -- start 2024-01-16T15:26:53.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.838+0000 7fe720a00640 1 -- start start 2024-01-16T15:26:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.839+0000 7fe720a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.839+0000 7fe720a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 0x7fe71c0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.839+0000 7fe720a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0cef30 con 0x7fe71c0d1bc0 2024-01-16T15:26:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.839+0000 7fe720a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe71c0d2050 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.839+0000 7fe720a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c0d11a0 con 0x7fe71c15d340 2024-01-16T15:26:53.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.840+0000 7fe71a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.840+0000 7fe71a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:36672/0 (socket says 172.21.15.43:36672) 2024-01-16T15:26:53.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.840+0000 7fe71a800640 1 -- 172.21.15.43:0/1829096345 learned_addr learned my addr 172.21.15.43:0/1829096345 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:53.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.840+0000 7fe711c00640 1 --2- 172.21.15.43:0/1829096345 >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 0x7fe71c0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.841+0000 7fe71a800640 1 -- 172.21.15.43:0/1829096345 >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 msgr2=0x7fe71c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.841+0000 7fe71a800640 1 --2- 172.21.15.43:0/1829096345 >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 0x7fe71c0d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.841+0000 7fe71a800640 1 -- 172.21.15.43:0/1829096345 >> v1:172.21.15.69:6789/0 conn(0x7fe71c0d1bc0 legacy=0x7fe71c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.841+0000 7fe71a800640 1 -- 172.21.15.43:0/1829096345 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c0ce320 con 0x7fe71c15d340 2024-01-16T15:26:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.841+0000 7fe711c00640 1 --2- 172.21.15.43:0/1829096345 >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 0x7fe71c0d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:26:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.843+0000 7fe71a800640 1 --2- 172.21.15.43:0/1829096345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cfafd8d704d64d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.843+0000 7fe719e00640 1 -- 172.21.15.43:0/1829096345 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c02cb00 con 0x7fe71c15d340 2024-01-16T15:26:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.843+0000 7fe719e00640 1 -- 172.21.15.43:0/1829096345 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c02cc60 con 0x7fe71c15d340 2024-01-16T15:26:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.843+0000 7fe719e00640 1 -- 172.21.15.43:0/1829096345 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0343f0 con 0x7fe71c15d340 2024-01-16T15:26:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 -- 172.21.15.43:0/1829096345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 msgr2=0x7fe71c0d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 --2- 172.21.15.43:0/1829096345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 -- 172.21.15.43:0/1829096345 shutdown_connections 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 --2- 172.21.15.43:0/1829096345 >> v2:172.21.15.69:3300/0 conn(0x7fe71c0cdbb0 0x7fe71c0d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 --2- 172.21.15.43:0/1829096345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0d00b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 -- 172.21.15.43:0/1829096345 >> 172.21.15.43:0/1829096345 conn(0x7fe71c09c370 msgr2=0x7fe71c0d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 -- 172.21.15.43:0/1829096345 shutdown_connections 2024-01-16T15:26:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.844+0000 7fe720a00640 1 -- 172.21.15.43:0/1829096345 wait complete. 2024-01-16T15:26:53.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.846+0000 7fe720a00640 1 Processor -- start 2024-01-16T15:26:53.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.846+0000 7fe720a00640 1 -- start start 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe71c0cdbb0 0x7fe71c0b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe71c0d1bc0 0x7fe71c0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c156830 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c0c9520 con 0x7fe71c0d1bc0 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.847+0000 7fe720a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c15df70 con 0x7fe71c15d340 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe71c0cdbb0 0x7fe71c0b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe71c0cdbb0 0x7fe71c0b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58610/0 (socket says 172.21.15.43:58610) 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 -- 172.21.15.43:0/3293014630 learned_addr learned my addr 172.21.15.43:0/3293014630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71b200640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe711c00640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe71c0d1bc0 0x7fe71c0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 -- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 msgr2=0x7fe71c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe71c15d340 0x7fe71c0b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 -- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe71c0d1bc0 msgr2=0x7fe71c0c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe71c0d1bc0 0x7fe71c0c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.848+0000 7fe71a800640 1 -- 172.21.15.43:0/3293014630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704009400 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe71a800640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe71c0cdbb0 0x7fe71c0b4b80 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe70c000f80 tx=0x7fe70c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c040670 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c040810 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe720a00640 1 -- 172.21.15.43:0/3293014630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c009b30 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c040bc0 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.849+0000 7fe720a00640 1 -- 172.21.15.43:0/3293014630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c15e530 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.850+0000 7fe720a00640 1 -- 172.21.15.43:0/3293014630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71c067d70 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.851+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fe70c040d20 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.852+0000 7fe713400640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f406c4c0 0x7fe6f406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.853+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe70c0abc00 con 0x7fe71c0cdbb0 2024-01-16T15:26:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.853+0000 7fe711c00640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f406c4c0 0x7fe6f406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.853+0000 7fe711c00640 1 --2- 172.21.15.43:0/3293014630 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe6f406c4c0 0x7fe6f406e980 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe71c1f68a0 tx=0x7fe704008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:53.857+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe70c078270 con 0x7fe71c0cdbb0 2024-01-16T15:26:54.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:54.090+0000 7fe720a00640 1 -- 172.21.15.43:0/3293014630 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe71c0ce8e0 con 0x7fe6f406c4c0 2024-01-16T15:26:54.098 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:54.099+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:54.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:54.099+0000 7fe713400640 1 -- 172.21.15.43:0/3293014630 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe704010930 con 0x7fe6f406c4c0 2024-01-16T15:26:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:54.099 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:56.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:56.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.355+0000 7fb1ba600640 1 Processor -- start 2024-01-16T15:26:56.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.355+0000 7fb1ba600640 1 -- start start 2024-01-16T15:26:56.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.356+0000 7fb1ba600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1b415d350 0x7fb1b40d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.356+0000 7fb1ba600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d730 0x7fb1b40d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.356+0000 7fb1ba600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b40d1d80 con 0x7fb1b40ceb30 2024-01-16T15:26:56.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.356+0000 7fb1ba600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b40d1fb0 con 0x7fb1b415d350 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.356+0000 7fb1ba600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40ce050 con 0x7fb1b415d730 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1b415d350 0x7fb1b40d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb1b415d350 0x7fb1b40d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58620/0 (socket says 172.21.15.43:58620) 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 -- 172.21.15.43:0/3733603718 learned_addr learned my addr 172.21.15.43:0/3733603718 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b2a00640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d730 0x7fb1b40d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 -- 172.21.15.43:0/3733603718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d730 msgr2=0x7fb1b40d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:56.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d730 0x7fb1b40d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:56.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 -- 172.21.15.43:0/3733603718 >> v1:172.21.15.43:6789/0 conn(0x7fb1b40ceb30 legacy=0x7fb1b40cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:56.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.357+0000 7fb1b3400640 1 -- 172.21.15.43:0/3733603718 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b40ce330 con 0x7fb1b415d350 2024-01-16T15:26:56.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.359+0000 7fb1b3400640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b415d350 0x7fb1b40d3d30 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009870 tx=0x7fb1a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1e86a8bdbce06e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:56.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.359+0000 7fb1b2000640 1 -- 172.21.15.43:0/3733603718 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4030840 con 0x7fb1b415d350 2024-01-16T15:26:56.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.359+0000 7fb1b2000640 1 -- 172.21.15.43:0/3733603718 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a40309a0 con 0x7fb1b415d350 2024-01-16T15:26:56.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.359+0000 7fb1b2000640 1 -- 172.21.15.43:0/3733603718 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4030d30 con 0x7fb1b415d350 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 -- 172.21.15.43:0/3733603718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b415d350 msgr2=0x7fb1b40d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b415d350 0x7fb1b40d3d30 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009870 tx=0x7fb1a402b980 comp rx=0 tx=0).stop 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 -- 172.21.15.43:0/3733603718 shutdown_connections 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d730 0x7fb1b40d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 --2- 172.21.15.43:0/3733603718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b415d350 0x7fb1b40d3d30 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 -- 172.21.15.43:0/3733603718 >> 172.21.15.43:0/3733603718 conn(0x7fb1b409c5c0 msgr2=0x7fb1b40d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 -- 172.21.15.43:0/3733603718 shutdown_connections 2024-01-16T15:26:56.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.360+0000 7fb1ba600640 1 -- 172.21.15.43:0/3733603718 wait complete. 2024-01-16T15:26:56.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.362+0000 7fb1ba600640 1 Processor -- start 2024-01-16T15:26:56.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.362+0000 7fb1ba600640 1 -- start start 2024-01-16T15:26:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b40ceb30 0x7fb1b41f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d350 0x7fb1b40ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1b415d730 0x7fb1b40ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40b4520 con 0x7fb1b40ceb30 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40b3910 con 0x7fb1b415d730 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1ba600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40b3bf0 con 0x7fb1b415d350 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b40ceb30 0x7fb1b41f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.363+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b40ceb30 0x7fb1b41f6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58634/0 (socket says 172.21.15.43:58634) 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 -- 172.21.15.43:0/473949891 learned_addr learned my addr 172.21.15.43:0/473949891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b2a00640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d350 0x7fb1b40ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3e00640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1b415d730 0x7fb1b40ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 -- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d350 msgr2=0x7fb1b40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1b415d350 0x7fb1b40ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 -- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1b415d730 msgr2=0x7fb1b40ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1b415d730 0x7fb1b40ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 -- 172.21.15.43:0/473949891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40094d0 con 0x7fb1b40ceb30 2024-01-16T15:26:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.364+0000 7fb1b3400640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1b40ceb30 0x7fb1b41f6650 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000c00 tx=0x7fb1a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.365+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4039c90 con 0x7fb1b40ceb30 2024-01-16T15:26:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.365+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a4039df0 con 0x7fb1b40ceb30 2024-01-16T15:26:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.365+0000 7fb1ba600640 1 -- 172.21.15.43:0/473949891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b40b3ed0 con 0x7fb1b40ceb30 2024-01-16T15:26:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.365+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4043640 con 0x7fb1b40ceb30 2024-01-16T15:26:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.365+0000 7fb1ba600640 1 -- 172.21.15.43:0/473949891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b41f6b90 con 0x7fb1b40ceb30 2024-01-16T15:26:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.366+0000 7fb1ba600640 1 -- 172.21.15.43:0/473949891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b40ca080 con 0x7fb1b40ceb30 2024-01-16T15:26:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.367+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fb1a40437a0 con 0x7fb1b40ceb30 2024-01-16T15:26:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.368+0000 7fb1ab400640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18c06c4c0 0x7fb18c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.368+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb1a403e070 con 0x7fb1b40ceb30 2024-01-16T15:26:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.368+0000 7fb1b2a00640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18c06c4c0 0x7fb18c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.369+0000 7fb1b2a00640 1 --2- 172.21.15.43:0/473949891 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fb18c06c4c0 0x7fb18c06e980 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb1b40a1180 tx=0x7fb19c003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.372+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1a4078190 con 0x7fb1b40ceb30 2024-01-16T15:26:56.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.604+0000 7fb1ba600640 1 -- 172.21.15.43:0/473949891 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1b40cdd00 con 0x7fb18c06c4c0 2024-01-16T15:26:56.615 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:56.616+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:56.617+0000 7fb1ab400640 1 -- 172.21.15.43:0/473949891 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb19c004690 con 0x7fb18c06c4c0 2024-01-16T15:26:56.616 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:58.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:58.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.869+0000 7f387fe00640 1 Processor -- start 2024-01-16T15:26:58.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.870+0000 7f387fe00640 1 -- start start 2024-01-16T15:26:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.871+0000 7f387fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3880070280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.871+0000 7f387fe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800b3e20 con 0x7f38800ac870 2024-01-16T15:26:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f387fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800acee0 con 0x7f3880143590 2024-01-16T15:26:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f387fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800acce0 con 0x7f388006dae0 2024-01-16T15:26:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f3877e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f38800ac870 0x7f388006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:40848/0 (socket says 172.21.15.43:40848) 2024-01-16T15:26:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f3877e00640 1 -- 172.21.15.43:0/2612141190 learned_addr learned my addr 172.21.15.43:0/2612141190 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f387f400640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3880070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920749824 0 0) 0x7f38800b3e20 con 0x7f38800ac870 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.872+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3860003520 con 0x7f38800ac870 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308402117 0 0) 0x7f3860003520 con 0x7f38800ac870 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 msgr2=0x7f3880070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3880070280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 >> v1:172.21.15.69:6789/0 conn(0x7f3880143590 legacy=0x7f38800ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800ad1c0 con 0x7f38800ac870 2024-01-16T15:26:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f38680017f0 con 0x7f38800ac870 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 >> v1:172.21.15.43:6789/0 conn(0x7f38800ac870 legacy=0x7f388006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3860003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f38600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3860004d00 0x7f3860005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800acce0 con 0x7f3860003f90 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386000bae0 con 0x7f3860004d00 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.873+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386000bca0 con 0x7f388006dae0 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f38600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f387ea00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3860003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 -- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 msgr2=0x7f3860003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388006dae0 0x7f3860003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 -- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3860004d00 msgr2=0x7f3860005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:58.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3860004d00 0x7f3860005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.874+0000 7f3877e00640 1 -- 172.21.15.43:0/2612141190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3860003520 con 0x7f3860003f90 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.875+0000 7f3877e00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f38600047c0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f386800c190 tx=0x7f38600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f028f59761874cc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.875+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868007270 con 0x7f3860003f90 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.875+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38680073f0 con 0x7f3860003f90 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.875+0000 7f387e000640 1 -- 172.21.15.43:0/2612141190 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386802c7e0 con 0x7f3860003f90 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 -- 172.21.15.43:0/2612141190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 msgr2=0x7f38600047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 --2- 172.21.15.43:0/2612141190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f38600047c0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f386800c190 tx=0x7f38600057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387f400640 1 -- 172.21.15.43:0/2612141190 reap_dead start 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 -- 172.21.15.43:0/2612141190 shutdown_connections 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 -- 172.21.15.43:0/2612141190 >> 172.21.15.43:0/2612141190 conn(0x7f38800b0aa0 msgr2=0x7f38800b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 -- 172.21.15.43:0/2612141190 shutdown_connections 2024-01-16T15:26:58.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.876+0000 7f387fe00640 1 -- 172.21.15.43:0/2612141190 wait complete. 2024-01-16T15:26:58.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.878+0000 7f387fe00640 1 Processor -- start 2024-01-16T15:26:58.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.878+0000 7f387fe00640 1 -- start start 2024-01-16T15:26:58.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f388015cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388015ee20 0x7f388015e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3880138350 0x7f38800cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388015ea30 con 0x7f3860003f90 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388015fae0 con 0x7f3880138350 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387fe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388015f2a0 con 0x7f388015ee20 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.879+0000 7f387ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f388015cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f388015cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58662/0 (socket says 172.21.15.43:58662) 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 -- 172.21.15.43:0/2051352775 learned_addr learned my addr 172.21.15.43:0/2051352775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f3877e00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388015ee20 0x7f388015e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387f400640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3880138350 0x7f38800cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 -- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388015ee20 msgr2=0x7f388015e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f388015ee20 0x7f388015e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 -- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3880138350 msgr2=0x7f38800cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3880138350 0x7f38800cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 -- 172.21.15.43:0/2051352775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388015f520 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387f400640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3880138350 0x7f38800cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.880+0000 7f387ea00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3860003f90 0x7f388015cca0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f387000e4d0 tx=0x7f387000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.881+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38700052e0 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.881+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870005440 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.881+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870003810 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.881+0000 7f387fe00640 1 -- 172.21.15.43:0/2051352775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388015f800 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.881+0000 7f387fe00640 1 -- 172.21.15.43:0/2051352775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388013f130 con 0x7f3860003f90 2024-01-16T15:26:58.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.882+0000 7f387fe00640 1 -- 172.21.15.43:0/2051352775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388013f2f0 con 0x7f3860003f90 2024-01-16T15:26:58.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.883+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f3870003970 con 0x7f3860003f90 2024-01-16T15:26:58.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.884+0000 7f3877400640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f385806c4f0 0x7f385806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.884+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3870088860 con 0x7f3860003f90 2024-01-16T15:26:58.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.885+0000 7f3877e00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f385806c4f0 0x7f385806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.885+0000 7f3877e00640 1 --2- 172.21.15.43:0/2051352775 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f385806c4f0 0x7f385806e9b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3868034000 tx=0x7f3868009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:58.889+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3870054db0 con 0x7f3860003f90 2024-01-16T15:26:59.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:59.117+0000 7f387fe00640 1 -- 172.21.15.43:0/2051352775 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38800ad780 con 0x7f385806c4f0 2024-01-16T15:26:59.125 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:26:59.126+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:59.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:26:59.126+0000 7f3877400640 1 -- 172.21.15.43:0/2051352775 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3868007070 con 0x7f385806c4f0 2024-01-16T15:26:59.125 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:26:59.126 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:01.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:01.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.388+0000 7fd23d200640 1 Processor -- start 2024-01-16T15:27:01.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.388+0000 7fd23d200640 1 -- start start 2024-01-16T15:27:01.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd23d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 0x7fd2380ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd23d200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd23806a440 0x7fd23806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd23d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23815dc80 con 0x7fd23815d710 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd23d200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd23806ae10 con 0x7fd23806a440 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd23d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23806e5a0 con 0x7fd23815ae30 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd237e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd23806a440 0x7fd23806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd237e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd23806a440 0x7fd23806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58672/0 (socket says 172.21.15.43:58672) 2024-01-16T15:27:01.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd237e00640 1 -- 172.21.15.43:0/4111467015 learned_addr learned my addr 172.21.15.43:0/4111467015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd237e00640 1 -- 172.21.15.43:0/4111467015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 msgr2=0x7fd2380ad4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.390+0000 7fd237400640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 0x7fd2380ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.391+0000 7fd237e00640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 0x7fd2380ad4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.391+0000 7fd237e00640 1 -- 172.21.15.43:0/4111467015 >> v1:172.21.15.69:6789/0 conn(0x7fd23815d710 legacy=0x7fd2380af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.391+0000 7fd237e00640 1 -- 172.21.15.43:0/4111467015 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23815df60 con 0x7fd23806a440 2024-01-16T15:27:01.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.391+0000 7fd237400640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 0x7fd2380ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:01.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.392+0000 7fd237e00640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23806a440 0x7fd23806a8d0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a6dea5262b228e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:01.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.392+0000 7fd236000640 1 -- 172.21.15.43:0/4111467015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230031640 con 0x7fd23806a440 2024-01-16T15:27:01.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.392+0000 7fd236000640 1 -- 172.21.15.43:0/4111467015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2300317a0 con 0x7fd23806a440 2024-01-16T15:27:01.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.392+0000 7fd236000640 1 -- 172.21.15.43:0/4111467015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230031b30 con 0x7fd23806a440 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 -- 172.21.15.43:0/4111467015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23806a440 msgr2=0x7fd23806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23806a440 0x7fd23806a8d0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 -- 172.21.15.43:0/4111467015 shutdown_connections 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23806a440 0x7fd23806a8d0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 --2- 172.21.15.43:0/4111467015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23815ae30 0x7fd2380ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 -- 172.21.15.43:0/4111467015 >> 172.21.15.43:0/4111467015 conn(0x7fd2380712b0 msgr2=0x7fd238071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 -- 172.21.15.43:0/4111467015 shutdown_connections 2024-01-16T15:27:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.393+0000 7fd23d200640 1 -- 172.21.15.43:0/4111467015 wait complete. 2024-01-16T15:27:01.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.395+0000 7fd23d200640 1 Processor -- start 2024-01-16T15:27:01.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.395+0000 7fd23d200640 1 -- start start 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23806a440 0x7fd238152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23815ae30 0x7fd23815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd23815d710 0x7fd2380adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23815e980 con 0x7fd23815ae30 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2381f3020 con 0x7fd23815d710 2024-01-16T15:27:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.396+0000 7fd23d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd238160690 con 0x7fd23806a440 2024-01-16T15:27:01.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23815ae30 0x7fd23815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23815ae30 0x7fd23815fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58674/0 (socket says 172.21.15.43:58674) 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 -- 172.21.15.43:0/928503424 learned_addr learned my addr 172.21.15.43:0/928503424 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd237e00640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd23815d710 0x7fd2380adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd237400640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23806a440 0x7fd238152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 -- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23806a440 msgr2=0x7fd238152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd23806a440 0x7fd238152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 -- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd23815d710 msgr2=0x7fd2380adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd23815d710 0x7fd2380adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.397+0000 7fd236a00640 1 -- 172.21.15.43:0/928503424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300094d0 con 0x7fd23815ae30 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd236a00640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd23815ae30 0x7fd23815fd40 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b3f0 tx=0x7fd22000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011830 con 0x7fd23815ae30 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd220011990 con 0x7fd23815ae30 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd23d200640 1 -- 172.21.15.43:0/928503424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23815b210 con 0x7fd23815ae30 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011d40 con 0x7fd23815ae30 2024-01-16T15:27:01.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.398+0000 7fd23d200640 1 -- 172.21.15.43:0/928503424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23815b7d0 con 0x7fd23815ae30 2024-01-16T15:27:01.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.399+0000 7fd23d200640 1 -- 172.21.15.43:0/928503424 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23804e840 con 0x7fd23815ae30 2024-01-16T15:27:01.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.400+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fd22001a400 con 0x7fd23815ae30 2024-01-16T15:27:01.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.401+0000 7fd22f400640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd21006c4c0 0x7fd21006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:01.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.401+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd220088810 con 0x7fd23815ae30 2024-01-16T15:27:01.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.401+0000 7fd237400640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd21006c4c0 0x7fd21006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:01.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.402+0000 7fd237400640 1 --2- 172.21.15.43:0/928503424 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fd21006c4c0 0x7fd21006e980 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd22800d620 tx=0x7fd228007810 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.406+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd220054e10 con 0x7fd23815ae30 2024-01-16T15:27:01.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.641+0000 7fd23d200640 1 -- 172.21.15.43:0/928503424 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd23815e520 con 0x7fd21006c4c0 2024-01-16T15:27:01.653 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:27:01.655+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:01.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:01.655+0000 7fd22f400640 1 -- 172.21.15.43:0/928503424 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd22800c5a0 con 0x7fd21006c4c0 2024-01-16T15:27:01.654 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:01.654 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:02.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:02.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:03.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:03.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.911+0000 7f2fbbe00640 1 Processor -- start 2024-01-16T15:27:03.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.911+0000 7f2fbbe00640 1 -- start start 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc13cf40 0x7f2fbc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fbc06d910 0x7f2fbc13f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 0x7f2fbc15daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc06dd90 con 0x7f2fbc06d910 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc140430 con 0x7f2fbc13fc40 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc071220 con 0x7f2fbc13cf40 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fba000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fbc06d910 0x7f2fbc13f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fba000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f2fbc06d910 0x7f2fbc13f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34136/0 (socket says 172.21.15.43:34136) 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.913+0000 7f2fba000640 1 -- 172.21.15.43:0/1469247430 learned_addr learned my addr 172.21.15.43:0/1469247430 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fbaa00640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc13cf40 0x7f2fbc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fba000640 1 -- 172.21.15.43:0/1469247430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc13cf40 msgr2=0x7f2fbc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fba000640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc13cf40 0x7f2fbc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fba000640 1 -- 172.21.15.43:0/1469247430 >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 msgr2=0x7f2fbc15daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fbb400640 1 --2- 172.21.15.43:0/1469247430 >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 0x7f2fbc15daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fba000640 1 --2- 172.21.15.43:0/1469247430 >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 0x7f2fbc15daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fba000640 1 -- 172.21.15.43:0/1469247430 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc071500 con 0x7f2fbc06d910 2024-01-16T15:27:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.914+0000 7f2fbb400640 1 --2- 172.21.15.43:0/1469247430 >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 0x7f2fbc15daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:03.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.915+0000 7f2fba000640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc06d910 0x7f2fbc13f700 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc143bf0 tx=0x7f2fa402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5dc199cb2f9c98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.915+0000 7f2fb9600640 1 -- 172.21.15.43:0/1469247430 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402ee70 con 0x7f2fbc06d910 2024-01-16T15:27:03.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.915+0000 7f2fb9600640 1 -- 172.21.15.43:0/1469247430 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4004030 con 0x7f2fbc06d910 2024-01-16T15:27:03.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.915+0000 7f2fb9600640 1 -- 172.21.15.43:0/1469247430 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4004420 con 0x7f2fbc06d910 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 -- 172.21.15.43:0/1469247430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc06d910 msgr2=0x7f2fbc13f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc06d910 0x7f2fbc13f700 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc143bf0 tx=0x7f2fa402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 -- 172.21.15.43:0/1469247430 shutdown_connections 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 --2- 172.21.15.43:0/1469247430 >> v2:172.21.15.69:3300/0 conn(0x7f2fbc13fc40 0x7f2fbc15daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc06d910 0x7f2fbc13f700 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 --2- 172.21.15.43:0/1469247430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc13cf40 0x7f2fbc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 -- 172.21.15.43:0/1469247430 >> 172.21.15.43:0/1469247430 conn(0x7f2fbc0b0aa0 msgr2=0x7f2fbc0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 -- 172.21.15.43:0/1469247430 shutdown_connections 2024-01-16T15:27:03.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.916+0000 7f2fbbe00640 1 -- 172.21.15.43:0/1469247430 wait complete. 2024-01-16T15:27:03.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.918+0000 7f2fbbe00640 1 Processor -- start 2024-01-16T15:27:03.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.918+0000 7f2fbbe00640 1 -- start start 2024-01-16T15:27:03.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc06d910 0x7f2fbc1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc13cf40 0x7f2fbc154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fbc13fc40 0x7f2fbc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc1f6e70 con 0x7f2fbc13cf40 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc1f61f0 con 0x7f2fbc13fc40 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc1f64d0 con 0x7f2fbc06d910 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fba000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc13cf40 0x7f2fbc154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fba000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc13cf40 0x7f2fbc154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34142/0 (socket says 172.21.15.43:34142) 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.919+0000 7f2fba000640 1 -- 172.21.15.43:0/420354023 learned_addr learned my addr 172.21.15.43:0/420354023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fbaa00640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc06d910 0x7f2fbc1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fbb400640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fbc13fc40 0x7f2fbc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 -- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc06d910 msgr2=0x7f2fbc1541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fbc06d910 0x7f2fbc1541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 -- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fbc13fc40 msgr2=0x7f2fbc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fbc13fc40 0x7f2fbc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 -- 172.21.15.43:0/420354023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa400b700 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.920+0000 7f2fba000640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2fbc13cf40 0x7f2fbc154f60 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4038040 tx=0x7f2fa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.921+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4003a20 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.921+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4003b80 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.921+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402fe20 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.921+0000 7f2fbbe00640 1 -- 172.21.15.43:0/420354023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc1f67b0 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.921+0000 7f2fbbe00640 1 -- 172.21.15.43:0/420354023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc0ae280 con 0x7f2fbc13cf40 2024-01-16T15:27:03.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.922+0000 7f2fbbe00640 1 -- 172.21.15.43:0/420354023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc1f6fb0 con 0x7f2fbc13cf40 2024-01-16T15:27:03.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.923+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2fa4006040 con 0x7f2fbc13cf40 2024-01-16T15:27:03.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.924+0000 7f2fb3400640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9406c4c0 0x7f2f9406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.924+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2fa40ab540 con 0x7f2fbc13cf40 2024-01-16T15:27:03.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.924+0000 7f2fbaa00640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9406c4c0 0x7f2f9406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.925+0000 7f2fbaa00640 1 --2- 172.21.15.43:0/420354023 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2f9406c4c0 0x7f2f9406e980 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2fac006fd0 tx=0x7f2fac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:03.929+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fbc1f6fb0 con 0x7f2fbc13cf40 2024-01-16T15:27:04.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:04.157+0000 7f2fbbe00640 1 -- 172.21.15.43:0/420354023 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fbc1400d0 con 0x7f2f9406c4c0 2024-01-16T15:27:04.167 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:27:04.169+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:04.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:04.169+0000 7f2fb3400640 1 -- 172.21.15.43:0/420354023 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2fac00b660 con 0x7f2f9406c4c0 2024-01-16T15:27:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:04.168 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:06.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.426+0000 7f1203e00640 1 Processor -- start 2024-01-16T15:27:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.426+0000 7f1203e00640 1 -- start start 2024-01-16T15:27:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1203e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 0x7f12040ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1203e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1204070280 0x7f12040ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1203e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040b3e20 con 0x7f120406d3d0 2024-01-16T15:27:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1203e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1204070710 con 0x7f1204070280 2024-01-16T15:27:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1203e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040ae820 con 0x7f1204143590 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.428+0000 7f1202000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1204070280 0x7f12040ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1204070280 0x7f12040ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34150/0 (socket says 172.21.15.43:34150) 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 -- 172.21.15.43:0/964842394 learned_addr learned my addr 172.21.15.43:0/964842394 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202a00640 1 --2- 172.21.15.43:0/964842394 >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 0x7f12040ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 -- 172.21.15.43:0/964842394 >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 msgr2=0x7f12040ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 --2- 172.21.15.43:0/964842394 >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 0x7f12040ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 -- 172.21.15.43:0/964842394 >> v1:172.21.15.69:6789/0 conn(0x7f120406d3d0 legacy=0x7f12040b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202000640 1 -- 172.21.15.43:0/964842394 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040acee0 con 0x7f1204070280 2024-01-16T15:27:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.429+0000 7f1202a00640 1 --2- 172.21.15.43:0/964842394 >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 0x7f12040ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.431+0000 7f1202000640 1 --2- 172.21.15.43:0/964842394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204070280 0x7f12040ae2e0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f120406e180 tx=0x7f11ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee6200210e79e1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.431+0000 7f1201600640 1 -- 172.21.15.43:0/964842394 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec004230 con 0x7f1204070280 2024-01-16T15:27:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.431+0000 7f1201600640 1 -- 172.21.15.43:0/964842394 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec004390 con 0x7f1204070280 2024-01-16T15:27:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.431+0000 7f1201600640 1 -- 172.21.15.43:0/964842394 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec003810 con 0x7f1204070280 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 -- 172.21.15.43:0/964842394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204070280 msgr2=0x7f12040ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 --2- 172.21.15.43:0/964842394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204070280 0x7f12040ae2e0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f120406e180 tx=0x7f11ec02b860 comp rx=0 tx=0).stop 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 -- 172.21.15.43:0/964842394 shutdown_connections 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 --2- 172.21.15.43:0/964842394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204070280 0x7f12040ae2e0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 --2- 172.21.15.43:0/964842394 >> v2:172.21.15.69:3300/0 conn(0x7f1204143590 0x7f12040ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 -- 172.21.15.43:0/964842394 >> 172.21.15.43:0/964842394 conn(0x7f12040b0aa0 msgr2=0x7f12040b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 -- 172.21.15.43:0/964842394 shutdown_connections 2024-01-16T15:27:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.432+0000 7f1203e00640 1 -- 172.21.15.43:0/964842394 wait complete. 2024-01-16T15:27:06.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.434+0000 7f1203e00640 1 Processor -- start 2024-01-16T15:27:06.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.434+0000 7f1203e00640 1 -- start start 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f120406d3d0 0x7f12041611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1204070280 0x7f1204144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204143590 0x7f120413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120413d360 con 0x7f1204143590 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120413cd50 con 0x7f120406d3d0 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120414bdb0 con 0x7f1204070280 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204143590 0x7f120413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204143590 0x7f120413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34166/0 (socket says 172.21.15.43:34166) 2024-01-16T15:27:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1203400640 1 -- 172.21.15.43:0/3702238710 learned_addr learned my addr 172.21.15.43:0/3702238710 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1202a00640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f120406d3d0 0x7f12041611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.435+0000 7f1202000640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1204070280 0x7f1204144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 -- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1204070280 msgr2=0x7f1204144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1204070280 0x7f1204144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 -- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f120406d3d0 msgr2=0x7f12041611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f120406d3d0 0x7f12041611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 -- 172.21.15.43:0/3702238710 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec009460 con 0x7f1204143590 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1202000640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1204070280 0x7f1204144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203400640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1204143590 0x7f120413d7f0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00c780 tx=0x7f11fc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc004b60 con 0x7f1204143590 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc00ce90 con 0x7f1204143590 2024-01-16T15:27:06.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00a850 con 0x7f1204143590 2024-01-16T15:27:06.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203e00640 1 -- 172.21.15.43:0/3702238710 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12041457e0 con 0x7f1204143590 2024-01-16T15:27:06.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.436+0000 7f1203e00640 1 -- 172.21.15.43:0/3702238710 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120413cff0 con 0x7f1204143590 2024-01-16T15:27:06.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.437+0000 7f1203e00640 1 -- 172.21.15.43:0/3702238710 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204145d00 con 0x7f1204143590 2024-01-16T15:27:06.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.439+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f11fc00a9b0 con 0x7f1204143590 2024-01-16T15:27:06.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.439+0000 7f11faa00640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f11dc06c4c0 0x7f11dc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:06.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.440+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f11fc088d10 con 0x7f1204143590 2024-01-16T15:27:06.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.440+0000 7f1202a00640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f11dc06c4c0 0x7f11dc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:06.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.441+0000 7f1202a00640 1 --2- 172.21.15.43:0/3702238710 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f11dc06c4c0 0x7f11dc06e980 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f12040b06d0 tx=0x7f11f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.444+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1204145d00 con 0x7f1204143590 2024-01-16T15:27:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.678+0000 7f1203e00640 1 -- 172.21.15.43:0/3702238710 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12040ad780 con 0x7f11dc06c4c0 2024-01-16T15:27:06.687 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:27:06.688+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:06.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:06.689+0000 7f11faa00640 1 -- 172.21.15.43:0/3702238710 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f11f400c050 con 0x7f11dc06c4c0 2024-01-16T15:27:06.688 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:06.688 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:08.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:08.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.942+0000 7fccaa800640 1 Processor -- start 2024-01-16T15:27:08.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.942+0000 7fccaa800640 1 -- start start 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.943+0000 7fccaa800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcca406dae0 0x7fcca415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.943+0000 7fccaa800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fccaa800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca4144ca0 con 0x7fcca41448c0 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fccaa800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca415c450 con 0x7fcca406dae0 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fccaa800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40715b0 con 0x7fcca415bbd0 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcca406dae0 0x7fcca415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcca406dae0 0x7fcca415b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34172/0 (socket says 172.21.15.43:34172) 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca8a00640 1 -- 172.21.15.43:0/3344183593 learned_addr learned my addr 172.21.15.43:0/3344183593 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca9e00640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca8a00640 1 -- 172.21.15.43:0/3344183593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 msgr2=0x7fcca4070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.944+0000 7fcca8a00640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.945+0000 7fcca8a00640 1 -- 172.21.15.43:0/3344183593 >> v1:172.21.15.69:6789/0 conn(0x7fcca41448c0 legacy=0x7fcca406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.945+0000 7fcca8a00640 1 -- 172.21.15.43:0/3344183593 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4071890 con 0x7fcca406dae0 2024-01-16T15:27:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.946+0000 7fcca8a00640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca415b690 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009850 tx=0x7fcc8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12bc669dd32ffb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.946+0000 7fcca3e00640 1 -- 172.21.15.43:0/3344183593 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c004230 con 0x7fcca406dae0 2024-01-16T15:27:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.946+0000 7fcca3e00640 1 -- 172.21.15.43:0/3344183593 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c004390 con 0x7fcca406dae0 2024-01-16T15:27:08.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.946+0000 7fcca3e00640 1 -- 172.21.15.43:0/3344183593 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c003810 con 0x7fcca406dae0 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 -- 172.21.15.43:0/3344183593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 msgr2=0x7fcca415b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca415b690 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009850 tx=0x7fcc8c02b960 comp rx=0 tx=0).stop 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 -- 172.21.15.43:0/3344183593 shutdown_connections 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca415b690 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 --2- 172.21.15.43:0/3344183593 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 -- 172.21.15.43:0/3344183593 >> 172.21.15.43:0/3344183593 conn(0x7fcca40b0aa0 msgr2=0x7fcca40b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 -- 172.21.15.43:0/3344183593 shutdown_connections 2024-01-16T15:27:08.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.947+0000 7fccaa800640 1 -- 172.21.15.43:0/3344183593 wait complete. 2024-01-16T15:27:08.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.949+0000 7fccaa800640 1 Processor -- start 2024-01-16T15:27:08.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.949+0000 7fccaa800640 1 -- start start 2024-01-16T15:27:08.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca4143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca41448c0 0x7fcca4142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4137780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca41438a0 con 0x7fcca406dae0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4137cc0 con 0x7fcca41448c0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fccaa800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4142b10 con 0x7fcca415bbd0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca4143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca4143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34180/0 (socket says 172.21.15.43:34180) 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 -- 172.21.15.43:0/2675093704 learned_addr learned my addr 172.21.15.43:0/2675093704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9e00640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 -- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 msgr2=0x7fcca4137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca8a00640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca41448c0 0x7fcca4142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415bbd0 0x7fcca4137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 -- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca41448c0 msgr2=0x7fcca4142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca41448c0 0x7fcca4142410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.951+0000 7fcca9400640 1 -- 172.21.15.43:0/2675093704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c009460 con 0x7fcca406dae0 2024-01-16T15:27:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fcca9400640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcca406dae0 0x7fcca4143cc0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400c4b0 tx=0x7fcc9400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc940052d0 con 0x7fcca406dae0 2024-01-16T15:27:08.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94005430 con 0x7fcca406dae0 2024-01-16T15:27:08.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94012870 con 0x7fcca406dae0 2024-01-16T15:27:08.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fccaa800640 1 -- 172.21.15.43:0/2675093704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca40724f0 con 0x7fcca406dae0 2024-01-16T15:27:08.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.952+0000 7fccaa800640 1 -- 172.21.15.43:0/2675093704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca413ebc0 con 0x7fcca406dae0 2024-01-16T15:27:08.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.953+0000 7fccaa800640 1 -- 172.21.15.43:0/2675093704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca404e840 con 0x7fcca406dae0 2024-01-16T15:27:08.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.955+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fcc940129d0 con 0x7fcca406dae0 2024-01-16T15:27:08.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.956+0000 7fcca1600640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc7c06c4c0 0x7fcc7c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.956+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc94088660 con 0x7fcca406dae0 2024-01-16T15:27:08.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.956+0000 7fcca8a00640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc7c06c4c0 0x7fcc7c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.957+0000 7fcca8a00640 1 --2- 172.21.15.43:0/2675093704 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcc7c06c4c0 0x7fcc7c06e980 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c035000 tx=0x7fcc8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:08.961+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc940bee90 con 0x7fcca406dae0 2024-01-16T15:27:09.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:09.189+0000 7fccaa800640 1 -- 172.21.15.43:0/2675093704 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcca40ce640 con 0x7fcc7c06c4c0 2024-01-16T15:27:09.201 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:27:09.202+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:09.203+0000 7fcca1600640 1 -- 172.21.15.43:0/2675093704 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcca40ce640 con 0x7fcc7c06c4c0 2024-01-16T15:27:09.202 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:11.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:11.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.457+0000 7f05ccc00640 1 Processor -- start 2024-01-16T15:27:11.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.458+0000 7f05ccc00640 1 -- start start 2024-01-16T15:27:11.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.459+0000 7f05ccc00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05c81448c0 0x7f05c806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.459+0000 7f05ccc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c8070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.459+0000 7f05ccc00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c815c210 con 0x7f05c806d910 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.459+0000 7f05ccc00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c806dd50 con 0x7f05c81448c0 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.459+0000 7f05ccc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8144ca0 con 0x7f05c815bda0 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05c81448c0 0x7f05c806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f05c81448c0 0x7f05c806d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34182/0 (socket says 172.21.15.43:34182) 2024-01-16T15:27:11.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 -- 172.21.15.43:0/64388549 learned_addr learned my addr 172.21.15.43:0/64388549 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7e00640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c8070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 -- 172.21.15.43:0/64388549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 msgr2=0x7f05c8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c8070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 -- 172.21.15.43:0/64388549 >> v1:172.21.15.43:6789/0 conn(0x7f05c806d910 legacy=0x7f05c815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7400640 1 -- 172.21.15.43:0/64388549 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c815c450 con 0x7f05c81448c0 2024-01-16T15:27:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.460+0000 7f05c7e00640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c8070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05c7400640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c81448c0 0x7f05c806d3d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009870 tx=0x7f05b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ab2f26049c6ef7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05c6a00640 1 -- 172.21.15.43:0/64388549 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8030840 con 0x7f05c81448c0 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05c6a00640 1 -- 172.21.15.43:0/64388549 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b80309a0 con 0x7f05c81448c0 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05c6a00640 1 -- 172.21.15.43:0/64388549 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8030d30 con 0x7f05c81448c0 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05ccc00640 1 -- 172.21.15.43:0/64388549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c81448c0 msgr2=0x7f05c806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.462+0000 7f05ccc00640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c81448c0 0x7f05c806d3d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009870 tx=0x7f05b802b980 comp rx=0 tx=0).stop 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 -- 172.21.15.43:0/64388549 shutdown_connections 2024-01-16T15:27:11.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c8070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 --2- 172.21.15.43:0/64388549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c81448c0 0x7f05c806d3d0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 -- 172.21.15.43:0/64388549 >> 172.21.15.43:0/64388549 conn(0x7f05c80b0aa0 msgr2=0x7f05c80b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 -- 172.21.15.43:0/64388549 shutdown_connections 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.463+0000 7f05ccc00640 1 -- 172.21.15.43:0/64388549 wait complete. 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.464+0000 7f05ccc00640 1 Processor -- start 2024-01-16T15:27:11.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.465+0000 7f05ccc00640 1 -- start start 2024-01-16T15:27:11.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c806d910 0x7f05c80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c81448c0 0x7f05c8150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c813a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c81513b0 con 0x7f05c806d910 2024-01-16T15:27:11.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80ad7a0 con 0x7f05c81448c0 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05ccc00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80acf00 con 0x7f05c815bda0 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c806d910 0x7f05c80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c806d910 0x7f05c80ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34192/0 (socket says 172.21.15.43:34192) 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 -- 172.21.15.43:0/1673284371 learned_addr learned my addr 172.21.15.43:0/1673284371 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 -- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 msgr2=0x7f05c813a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c815bda0 0x7f05c813a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 -- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c81448c0 msgr2=0x7f05c8150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05bfe00640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c81448c0 0x7f05c8150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c81448c0 0x7f05c8150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05c7400640 1 -- 172.21.15.43:0/1673284371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b80094d0 con 0x7f05c806d910 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.466+0000 7f05bfe00640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c81448c0 0x7f05c8150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:11.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05c7400640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05c806d910 0x7f05c80ac870 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f05b8000c00 tx=0x7f05b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:11.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8030810 con 0x7f05c806d910 2024-01-16T15:27:11.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8043400 con 0x7f05c806d910 2024-01-16T15:27:11.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05ccc00640 1 -- 172.21.15.43:0/1673284371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80ad1e0 con 0x7f05c806d910 2024-01-16T15:27:11.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80437d0 con 0x7f05c806d910 2024-01-16T15:27:11.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.467+0000 7f05ccc00640 1 -- 172.21.15.43:0/1673284371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c80b1f40 con 0x7f05c806d910 2024-01-16T15:27:11.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.468+0000 7f05ccc00640 1 -- 172.21.15.43:0/1673284371 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c80b2130 con 0x7f05c806d910 2024-01-16T15:27:11.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.469+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f05b8043930 con 0x7f05c806d910 2024-01-16T15:27:11.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.470+0000 7f05bf400640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f05a006c4c0 0x7f05a006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:11.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.470+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05b803e070 con 0x7f05c806d910 2024-01-16T15:27:11.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.471+0000 7f05bfe00640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f05a006c4c0 0x7f05a006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:11.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.471+0000 7f05bfe00640 1 --2- 172.21.15.43:0/1673284371 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f05a006c4c0 0x7f05a006e980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f05c815e7c0 tx=0x7f05b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.475+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05b8078590 con 0x7f05c806d910 2024-01-16T15:27:11.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.706+0000 7f05ccc00640 1 -- 172.21.15.43:0/1673284371 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05c8071b70 con 0x7f05a006c4c0 2024-01-16T15:27:11.718 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:27:11.719+0000 7fd4e5000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:11.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:11.719+0000 7f05bf400640 1 -- 172.21.15.43:0/1673284371 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f05b0001350 con 0x7f05a006c4c0 2024-01-16T15:27:11.718 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:11.719 INFO:teuthology.orchestra.run.smithi043.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:11.736 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for ... ERROR 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner:ERROR 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:11.737 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:27:11.738 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42756' 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:27:11.739 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1624.038s 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:11.740 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:27:11.741 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone42756' 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:11.742 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:11.743 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.966 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a5cdf9f31cc84d96a6cb6ce17d143b2b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:11.968 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-16T15:27:11.977 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-16T15:27:11.993 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-16T15:27:11.993 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:27:11.993 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:12.057 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:27:12.057 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:12.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:12.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.336+0000 7f76ab200640 1 Processor -- start 2024-01-16T15:27:12.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.337+0000 7f76ab200640 1 -- start start 2024-01-16T15:27:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76ab200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76ac15b100 0x7f76ac071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76ab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06bb60 0x7f76ac0cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76ab200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac0cd540 con 0x7f76ac06caa0 2024-01-16T15:27:12.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76ab200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac071aa0 con 0x7f76ac15b100 2024-01-16T15:27:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76ab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac06cfa0 con 0x7f76ac06bb60 2024-01-16T15:27:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76a9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76ac15b100 0x7f76ac071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76a9e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f76ac15b100 0x7f76ac071560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34208/0 (socket says 172.21.15.43:34208) 2024-01-16T15:27:12.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.338+0000 7f76a9e00640 1 -- 172.21.15.43:0/2846920717 learned_addr learned my addr 172.21.15.43:0/2846920717 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.339+0000 7f76aa800640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06bb60 0x7f76ac0cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.339+0000 7f76a9e00640 1 -- 172.21.15.43:0/2846920717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06bb60 msgr2=0x7f76ac0cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.339+0000 7f76a9e00640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06bb60 0x7f76ac0cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.339+0000 7f76a9e00640 1 -- 172.21.15.43:0/2846920717 >> v1:172.21.15.43:6789/0 conn(0x7f76ac06caa0 legacy=0x7f76ac06b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.339+0000 7f76a9e00640 1 -- 172.21.15.43:0/2846920717 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac06d280 con 0x7f76ac15b100 2024-01-16T15:27:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.340+0000 7f76a9e00640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac071560 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f7694009870 tx=0x7f769402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b80e0ed865b3d13 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.340+0000 7f76a8a00640 1 -- 172.21.15.43:0/2846920717 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694031640 con 0x7f76ac15b100 2024-01-16T15:27:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76a8a00640 1 -- 172.21.15.43:0/2846920717 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76940317a0 con 0x7f76ac15b100 2024-01-16T15:27:12.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76a8a00640 1 -- 172.21.15.43:0/2846920717 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694031b30 con 0x7f76ac15b100 2024-01-16T15:27:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76ab200640 1 -- 172.21.15.43:0/2846920717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 msgr2=0x7f76ac071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76ab200640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac071560 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f7694009870 tx=0x7f769402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76ab200640 1 -- 172.21.15.43:0/2846920717 shutdown_connections 2024-01-16T15:27:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76ab200640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06bb60 0x7f76ac0cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.341+0000 7f76ab200640 1 --2- 172.21.15.43:0/2846920717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac071560 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.342+0000 7f76ab200640 1 -- 172.21.15.43:0/2846920717 >> 172.21.15.43:0/2846920717 conn(0x7f76ac070a40 msgr2=0x7f76ac070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.342+0000 7f76ab200640 1 -- 172.21.15.43:0/2846920717 shutdown_connections 2024-01-16T15:27:12.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.342+0000 7f76ab200640 1 -- 172.21.15.43:0/2846920717 wait complete. 2024-01-16T15:27:12.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.343+0000 7f76ab200640 1 Processor -- start 2024-01-16T15:27:12.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.344+0000 7f76ab200640 1 -- start start 2024-01-16T15:27:12.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 0x7f76ac159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 0x7f76ac15cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac157100 con 0x7f76ac15b100 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac159ad0 con 0x7f76ac06bb60 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76ab200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac159db0 con 0x7f76ac06caa0 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34216/0 (socket says 172.21.15.43:34216) 2024-01-16T15:27:12.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 -- 172.21.15.43:0/3361319032 learned_addr learned my addr 172.21.15.43:0/3361319032 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76a9e00640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 0x7f76ac159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76a9400640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 0x7f76ac15cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 -- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 msgr2=0x7f76ac15cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 0x7f76ac15cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 -- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 msgr2=0x7f76ac159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 0x7f76ac159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76aa800640 1 -- 172.21.15.43:0/3361319032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940094d0 con 0x7f76ac15b100 2024-01-16T15:27:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76a9400640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 0x7f76ac15cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.345+0000 7f76a9e00640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 0x7f76ac159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76aa800640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f76a400d800 tx=0x7f76a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400b6c0 con 0x7f76ac15b100 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a400b840 con 0x7f76ac15b100 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400bbf0 con 0x7f76ac15b100 2024-01-16T15:27:12.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac157350 con 0x7f76ac15b100 2024-01-16T15:27:12.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.346+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac1597c0 con 0x7f76ac15b100 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.347+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac157870 con 0x7f76ac15b100 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.348+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f76a400bd50 con 0x7f76ac15b100 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.350+0000 7f76a2000640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 0x7f768406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.350+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f76a4088d70 con 0x7f76ac15b100 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.350+0000 7f76a9e00640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 0x7f768406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.351+0000 7f76a9e00640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 0x7f768406e980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7694036000 tx=0x7f769403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.354+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76ac157870 con 0x7f76ac15b100 2024-01-16T15:27:12.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.549+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76ac157870 con 0x7f76ac15b100 2024-01-16T15:27:12.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.552+0000 7f76a2000640 1 -- 172.21.15.43:0/3361319032 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f76ac157870 con 0x7f76ac15b100 2024-01-16T15:27:12.551 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:12.551 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":115,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:19:15.246810+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:12.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 msgr2=0x7f768406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 0x7f768406e980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7694036000 tx=0x7f769403c040 comp rx=0 tx=0).stop 2024-01-16T15:27:12.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 msgr2=0x7f76ac147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f76a400d800 tx=0x7f76a400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 shutdown_connections 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f768406c4c0 0x7f768406e980 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76ac15b100 0x7f76ac147ac0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76ac06caa0 0x7f76ac15cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 --2- 172.21.15.43:0/3361319032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76ac06bb60 0x7f76ac159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 >> 172.21.15.43:0/3361319032 conn(0x7f76ac070a40 msgr2=0x7f76ac0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 shutdown_connections 2024-01-16T15:27:12.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.555+0000 7f76ab200640 1 -- 172.21.15.43:0/3361319032 wait complete. 2024-01-16T15:27:12.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:18:47.672212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:18:48.980104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:12.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.795+0000 7fdd42600640 1 Processor -- start 2024-01-16T15:27:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.795+0000 7fdd42600640 1 -- start start 2024-01-16T15:27:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.796+0000 7fdd42600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd3c0b48d0 0x7fdd3c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd42600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 0x7fdd3c167290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd42600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3c0b23e0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd42600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3c0cdc70 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd42600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3c0b2870 con 0x7fdd3c0b3210 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd3b400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fdd3c160bd0 0x7fdd3c0b2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34898/0 (socket says 172.21.15.43:34898) 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd3b400640 1 -- 172.21.15.43:0/3992594180 learned_addr learned my addr 172.21.15.43:0/3992594180 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd33e00640 1 --2- 172.21.15.43:0/3992594180 >> v2:172.21.15.43:3300/0 conn(0x7fdd3c0b48d0 0x7fdd3c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd3be00640 1 --2- 172.21.15.43:0/3992594180 >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 0x7fdd3c167290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.797+0000 7fdd33e00640 1 -- 172.21.15.43:0/3992594180 >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 msgr2=0x7fdd3c167290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.798+0000 7fdd33e00640 1 --2- 172.21.15.43:0/3992594180 >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 0x7fdd3c167290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.798+0000 7fdd33e00640 1 -- 172.21.15.43:0/3992594180 >> v1:172.21.15.43:6789/0 conn(0x7fdd3c160bd0 legacy=0x7fdd3c0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.798+0000 7fdd33e00640 1 -- 172.21.15.43:0/3992594180 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0d41b0 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.798+0000 7fdd3be00640 1 --2- 172.21.15.43:0/3992594180 >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 0x7fdd3c167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:12.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.799+0000 7fdd33e00640 1 --2- 172.21.15.43:0/3992594180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c0b1ea0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0d2e10 tx=0x7fdd2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca909fee8fe259a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.799+0000 7fdd3aa00640 1 -- 172.21.15.43:0/3992594180 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2402e070 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.799+0000 7fdd3aa00640 1 -- 172.21.15.43:0/3992594180 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd240315a0 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.799+0000 7fdd3aa00640 1 -- 172.21.15.43:0/3992594180 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24031930 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 -- 172.21.15.43:0/3992594180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c0b48d0 msgr2=0x7fdd3c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 --2- 172.21.15.43:0/3992594180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c0b1ea0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0d2e10 tx=0x7fdd2402b840 comp rx=0 tx=0).stop 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 -- 172.21.15.43:0/3992594180 shutdown_connections 2024-01-16T15:27:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 --2- 172.21.15.43:0/3992594180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c0b1ea0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 --2- 172.21.15.43:0/3992594180 >> v2:172.21.15.69:3300/0 conn(0x7fdd3c0b3210 0x7fdd3c167290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 -- 172.21.15.43:0/3992594180 >> 172.21.15.43:0/3992594180 conn(0x7fdd3c09c5a0 msgr2=0x7fdd3c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 -- 172.21.15.43:0/3992594180 shutdown_connections 2024-01-16T15:27:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.800+0000 7fdd42600640 1 -- 172.21.15.43:0/3992594180 wait complete. 2024-01-16T15:27:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.802+0000 7fdd42600640 1 Processor -- start 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.802+0000 7fdd42600640 1 -- start start 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 0x7fdd3c0b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c1f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c15c0b0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0b6320 con 0x7fdd3c0b48d0 2024-01-16T15:27:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd42600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0b6600 con 0x7fdd3c0b3210 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd3be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd3be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34238/0 (socket says 172.21.15.43:34238) 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd3be00640 1 -- 172.21.15.43:0/3773466126 learned_addr learned my addr 172.21.15.43:0/3773466126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.803+0000 7fdd3b400640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 0x7fdd3c0b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 -- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 msgr2=0x7fdd3c0b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 0x7fdd3c0b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd33e00640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c1f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 -- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 msgr2=0x7fdd3c1f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:12.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c1f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 -- 172.21.15.43:0/3773466126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009400 con 0x7fdd3c160bd0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3b400640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 0x7fdd3c0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd33e00640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c1f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd3be00640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400cef0 tx=0x7fdd3400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400d840 con 0x7fdd3c160bd0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3400d9c0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c0b68e0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400dd70 con 0x7fdd3c160bd0 2024-01-16T15:27:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.804+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c0b6ea0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.806+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c0a1210 con 0x7fdd3c160bd0 2024-01-16T15:27:12.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.807+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fdd3401e3b0 con 0x7fdd3c160bd0 2024-01-16T15:27:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.808+0000 7fdd33400640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 0x7fdd1406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.809+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdd34089280 con 0x7fdd3c160bd0 2024-01-16T15:27:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.809+0000 7fdd3b400640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 0x7fdd1406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.809+0000 7fdd3b400640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 0x7fdd1406e980 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007fd0 tx=0x7fdd2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:12.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:12.813+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd34055c00 con 0x7fdd3c160bd0 2024-01-16T15:27:13.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.008+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd3c0d1890 con 0x7fdd3c160bd0 2024-01-16T15:27:13.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.009+0000 7fdd33400640 1 -- 172.21.15.43:0/3773466126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd3401a070 con 0x7fdd3c160bd0 2024-01-16T15:27:13.008 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:27:13.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 msgr2=0x7fdd1406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 0x7fdd1406e980 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007fd0 tx=0x7fdd2c009040 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 msgr2=0x7fdd3c1f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400cef0 tx=0x7fdd3400a710 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 shutdown_connections 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.011+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fdd1406c4c0 0x7fdd1406e980 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c0b48d0 0x7fdd3c1f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd3c160bd0 0x7fdd3c1f3bf0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 --2- 172.21.15.43:0/3773466126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0b3210 0x7fdd3c0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 >> 172.21.15.43:0/3773466126 conn(0x7fdd3c09c5a0 msgr2=0x7fdd3c0b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:13.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 shutdown_connections 2024-01-16T15:27:13.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.012+0000 7fdd42600640 1 -- 172.21.15.43:0/3773466126 wait complete. 2024-01-16T15:27:13.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:13.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.265+0000 7f463c200640 1 Processor -- start 2024-01-16T15:27:13.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.266+0000 7f463c200640 1 -- start start 2024-01-16T15:27:13.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.267+0000 7f463c200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f463409cb30 0x7f46340abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.267+0000 7f463c200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463415b730 con 0x7f463415b350 2024-01-16T15:27:13.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.267+0000 7f463c200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46340ac740 con 0x7f46340adb60 2024-01-16T15:27:13.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.267+0000 7f463c200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46340ac460 con 0x7f463409cb30 2024-01-16T15:27:13.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.268+0000 7f4639800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f463415b350 0x7f46340b3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59258/0 (socket says 172.21.15.43:59258) 2024-01-16T15:27:13.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.268+0000 7f4639800640 1 -- 172.21.15.43:0/1614317775 learned_addr learned my addr 172.21.15.43:0/1614317775 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:13.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.268+0000 7f4638e00640 1 --2- 172.21.15.43:0/1614317775 >> v2:172.21.15.69:3300/0 conn(0x7f463409cb30 0x7f46340abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462369127 0 0) 0x7f4624001110 con 0x7f463415b350 2024-01-16T15:27:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4614003520 con 0x7f463415b350 2024-01-16T15:27:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395035634 0 0) 0x7f4614003520 con 0x7f463415b350 2024-01-16T15:27:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 >> v2:172.21.15.69:3300/0 conn(0x7f463409cb30 msgr2=0x7f46340abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 --2- 172.21.15.43:0/1614317775 >> v2:172.21.15.69:3300/0 conn(0x7f463409cb30 0x7f46340abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.269+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 >> v1:172.21.15.69:6789/0 conn(0x7f46340adb60 legacy=0x7f46340adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46340b2a50 con 0x7f463415b350 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f4624001800 con 0x7f463415b350 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 >> v1:172.21.15.43:6789/0 conn(0x7f463415b350 legacy=0x7f46340b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46140039c0 0x7f4614003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 0x7f461400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46340ac460 con 0x7f4614005180 2024-01-16T15:27:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461400c080 con 0x7f46140047c0 2024-01-16T15:27:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461400c240 con 0x7f46140039c0 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.270+0000 7f463a200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 0x7f461400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.271+0000 7f463a200640 1 -- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46140039c0 msgr2=0x7f4614003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.271+0000 7f463a200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46140039c0 0x7f4614003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.271+0000 7f463a200640 1 -- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46140047c0 msgr2=0x7f4614004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.271+0000 7f463a200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.271+0000 7f463a200640 1 -- 172.21.15.43:0/1614317775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614003520 con 0x7f4614005180 2024-01-16T15:27:13.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.272+0000 7f463a200640 1 -- 172.21.15.43:0/1614317775 reap_dead start 2024-01-16T15:27:13.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.273+0000 7f463a200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 0x7f461400b980 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f462c001cb0 tx=0x7f46140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6ff99ce51ce093 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.273+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c003b70 con 0x7f4614005180 2024-01-16T15:27:13.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.273+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462c02d040 con 0x7f4614005180 2024-01-16T15:27:13.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.273+0000 7f4633e00640 1 -- 172.21.15.43:0/1614317775 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c0050d0 con 0x7f4614005180 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 -- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 msgr2=0x7f461400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 0x7f461400b980 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f462c001cb0 tx=0x7f46140057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 -- 172.21.15.43:0/1614317775 shutdown_connections 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 --2- 172.21.15.43:0/1614317775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4614005180 0x7f461400b980 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 -- 172.21.15.43:0/1614317775 >> 172.21.15.43:0/1614317775 conn(0x7f46340b0ba0 msgr2=0x7f46340b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 -- 172.21.15.43:0/1614317775 shutdown_connections 2024-01-16T15:27:13.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.274+0000 7f463c200640 1 -- 172.21.15.43:0/1614317775 wait complete. 2024-01-16T15:27:13.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.277+0000 7f463c200640 1 Processor -- start 2024-01-16T15:27:13.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.277+0000 7f463c200640 1 -- start start 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4614005180 0x7f463409b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634151af0 0x7f463414afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463414c2d0 con 0x7f4634151670 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463414c5b0 con 0x7f4634151af0 2024-01-16T15:27:13.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f463c200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463414a7d0 con 0x7f4614005180 2024-01-16T15:27:13.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f4638e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f4638e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54282/0 (socket says 172.21.15.43:54282) 2024-01-16T15:27:13.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.278+0000 7f4638e00640 1 -- 172.21.15.43:0/1462812108 learned_addr learned my addr 172.21.15.43:0/1462812108 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:13.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4639800640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4614005180 0x7f463409b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f463a200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634151af0 0x7f463414afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4638e00640 1 -- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4614005180 msgr2=0x7f463409b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4638e00640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4614005180 0x7f463409b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4638e00640 1 -- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634151af0 msgr2=0x7f463414afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4638e00640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634151af0 0x7f463414afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.279+0000 7f4638e00640 1 -- 172.21.15.43:0/1462812108 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463414aab0 con 0x7f4634151670 2024-01-16T15:27:13.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.280+0000 7f4638e00640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f461c00e7a0 tx=0x7f461c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.280+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00eeb0 con 0x7f4634151670 2024-01-16T15:27:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.280+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c00c940 con 0x7f4634151670 2024-01-16T15:27:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.280+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00cd10 con 0x7f4634151670 2024-01-16T15:27:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.281+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4634148c60 con 0x7f4634151670 2024-01-16T15:27:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.282+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463414bf80 con 0x7f4634151670 2024-01-16T15:27:13.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.284+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46340af960 con 0x7f4634151670 2024-01-16T15:27:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.286+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f461c013470 con 0x7f4634151670 2024-01-16T15:27:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.287+0000 7f4631600640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 0x7f460c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.287+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f461c088bd0 con 0x7f4634151670 2024-01-16T15:27:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.287+0000 7f4639800640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 0x7f460c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.292+0000 7f4639800640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 0x7f460c06e9d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4624000ff0 tx=0x7f46240016e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.293+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f461c055160 con 0x7f4634151670 2024-01-16T15:27:13.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.646+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f463414ac40 con 0x7f4634151670 2024-01-16T15:27:13.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.647+0000 7f4631600640 1 -- 172.21.15.43:0/1462812108 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f461c058780 con 0x7f4634151670 2024-01-16T15:27:13.646 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:27:13.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 msgr2=0x7f460c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 0x7f460c06e9d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4624000ff0 tx=0x7f46240016e0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 msgr2=0x7f463415b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f461c00e7a0 tx=0x7f461c00ec60 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 shutdown_connections 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f460c06c510 0x7f460c06e9d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634151af0 0x7f463414afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634151670 0x7f463415b8d0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 --2- 172.21.15.43:0/1462812108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4614005180 0x7f463409b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 >> 172.21.15.43:0/1462812108 conn(0x7f46340b0ba0 msgr2=0x7f46340b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 shutdown_connections 2024-01-16T15:27:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:13.651+0000 7f463c200640 1 -- 172.21.15.43:0/1462812108 wait complete. 2024-01-16T15:27:13.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.045+0000 7f70cbe00640 1 Processor -- start 2024-01-16T15:27:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.046+0000 7f70cbe00640 1 -- start start 2024-01-16T15:27:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70cbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70c4162220 0x7f70c40b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c4162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70cbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c4162fa0 con 0x7f70c4162600 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70cbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c40706a0 con 0x7f70c4162220 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70cbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4070c60 con 0x7f70c406cad0 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70c9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70c4162220 0x7f70c40b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70c9400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70c4162220 0x7f70c40b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54296/0 (socket says 172.21.15.43:54296) 2024-01-16T15:27:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.048+0000 7f70c9400640 1 -- 172.21.15.43:0/1472233972 learned_addr learned my addr 172.21.15.43:0/1472233972 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.049+0000 7f70c9e00640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c4162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.049+0000 7f70c9400640 1 -- 172.21.15.43:0/1472233972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 msgr2=0x7f70c4162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.049+0000 7f70c9400640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c4162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.049+0000 7f70c9400640 1 -- 172.21.15.43:0/1472233972 >> v1:172.21.15.69:6789/0 conn(0x7f70c4162600 legacy=0x7f70c406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.049+0000 7f70c9400640 1 -- 172.21.15.43:0/1472233972 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c40722b0 con 0x7f70c4162220 2024-01-16T15:27:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.051+0000 7f70c9400640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162220 0x7f70c40b1ea0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009e50 tx=0x7f70bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a2645b9c4f8f8e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.051+0000 7f70c3e00640 1 -- 172.21.15.43:0/1472233972 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc02ca70 con 0x7f70c4162220 2024-01-16T15:27:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.052+0000 7f70c3e00640 1 -- 172.21.15.43:0/1472233972 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc02cbd0 con 0x7f70c4162220 2024-01-16T15:27:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.052+0000 7f70c3e00640 1 -- 172.21.15.43:0/1472233972 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0343f0 con 0x7f70c4162220 2024-01-16T15:27:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.053+0000 7f70cbe00640 1 -- 172.21.15.43:0/1472233972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162220 msgr2=0x7f70c40b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.053+0000 7f70cbe00640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162220 0x7f70c40b1ea0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009e50 tx=0x7f70bc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.053+0000 7f70cbe00640 1 -- 172.21.15.43:0/1472233972 shutdown_connections 2024-01-16T15:27:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.053+0000 7f70cbe00640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c4162a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.054+0000 7f70cbe00640 1 --2- 172.21.15.43:0/1472233972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162220 0x7f70c40b1ea0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.054+0000 7f70cbe00640 1 -- 172.21.15.43:0/1472233972 >> 172.21.15.43:0/1472233972 conn(0x7f70c40b0ba0 msgr2=0x7f70c40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.054+0000 7f70cbe00640 1 -- 172.21.15.43:0/1472233972 shutdown_connections 2024-01-16T15:27:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.054+0000 7f70cbe00640 1 -- 172.21.15.43:0/1472233972 wait complete. 2024-01-16T15:27:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.057+0000 7f70cbe00640 1 Processor -- start 2024-01-16T15:27:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.057+0000 7f70cbe00640 1 -- start start 2024-01-16T15:27:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c4162220 0x7f70c40aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ade20 con 0x7f70c4162600 2024-01-16T15:27:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c41fc1a0 con 0x7f70c4162220 2024-01-16T15:27:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.058+0000 7f70cbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40acdd0 con 0x7f70c406cad0 2024-01-16T15:27:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.059+0000 7f70c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.059+0000 7f70c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54298/0 (socket says 172.21.15.43:54298) 2024-01-16T15:27:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.059+0000 7f70c9e00640 1 -- 172.21.15.43:0/2774290666 learned_addr learned my addr 172.21.15.43:0/2774290666 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.059+0000 7f70c8a00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c4162220 0x7f70c40aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.059+0000 7f70c9400640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 -- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 msgr2=0x7f70c40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 -- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c4162220 msgr2=0x7f70c40aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c4162220 0x7f70c40aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 -- 172.21.15.43:0/2774290666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc009b30 con 0x7f70c4162600 2024-01-16T15:27:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.060+0000 7f70c9e00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f70b400d5d0 tx=0x7f70b400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.061+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b40125f0 con 0x7f70c4162600 2024-01-16T15:27:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.061+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4012750 con 0x7f70c4162600 2024-01-16T15:27:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.061+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c41fb3e0 con 0x7f70c4162600 2024-01-16T15:27:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.062+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4012ae0 con 0x7f70c4162600 2024-01-16T15:27:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.062+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c41fb9a0 con 0x7f70c4162600 2024-01-16T15:27:14.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.064+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c409d200 con 0x7f70c4162600 2024-01-16T15:27:14.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.070+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f70b4012c40 con 0x7f70c4162600 2024-01-16T15:27:14.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.073+0000 7f70c1600640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 0x7f709c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.073+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f70b4089580 con 0x7f70c4162600 2024-01-16T15:27:14.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.075+0000 7f70c9400640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 0x7f709c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.076+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70b4055ef0 con 0x7f70c4162600 2024-01-16T15:27:14.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.077+0000 7f70c9400640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 0x7f709c06ea50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f70bc035000 tx=0x7f70bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:14.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.463+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70c40adfb0 con 0x7f70c4162600 2024-01-16T15:27:14.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.464+0000 7f70c1600640 1 -- 172.21.15.43:0/2774290666 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70b4059510 con 0x7f70c4162600 2024-01-16T15:27:14.464 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:27:14.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 msgr2=0x7f709c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 0x7f709c06ea50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f70bc035000 tx=0x7f70bc030000 comp rx=0 tx=0).stop 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 msgr2=0x7f70c40af4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f70b400d5d0 tx=0x7f70b400da90 comp rx=0 tx=0).stop 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 shutdown_connections 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f709c06c590 0x7f709c06ea50 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c4162220 0x7f70c40aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c4162600 0x7f70c40af4d0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 --2- 172.21.15.43:0/2774290666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c406cad0 0x7f70c40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.468+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 >> 172.21.15.43:0/2774290666 conn(0x7f70c40b0ba0 msgr2=0x7f70c40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:14.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.469+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 shutdown_connections 2024-01-16T15:27:14.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.469+0000 7f70cbe00640 1 -- 172.21.15.43:0/2774290666 wait complete. 2024-01-16T15:27:14.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:14.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.802+0000 7f7327600640 1 Processor -- start 2024-01-16T15:27:14.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.802+0000 7f7327600640 1 -- start start 2024-01-16T15:27:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.803+0000 7f7327600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7327600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 0x7f73200b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7327600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73200ac760 con 0x7f73200b37d0 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7327600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732006e530 con 0x7f73200abff0 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7327600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200b3c40 con 0x7f732015a090 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7324c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7324c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46526/0 (socket says 172.21.15.43:46526) 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.804+0000 7f7324c00640 1 -- 172.21.15.43:0/2008672560 learned_addr learned my addr 172.21.15.43:0/2008672560 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:14.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.805+0000 7f7317e00640 1 --2- 172.21.15.43:0/2008672560 >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 0x7f73200b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.805+0000 7f7324c00640 1 -- 172.21.15.43:0/2008672560 >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 msgr2=0x7f73200b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:14.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.806+0000 7f7324c00640 1 --2- 172.21.15.43:0/2008672560 >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 0x7f73200b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.806+0000 7f7324c00640 1 -- 172.21.15.43:0/2008672560 >> v1:172.21.15.69:6789/0 conn(0x7f73200b37d0 legacy=0x7f73200704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.806+0000 7f7324c00640 1 -- 172.21.15.43:0/2008672560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732015a7f0 con 0x7f732015a090 2024-01-16T15:27:14.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.807+0000 7f7317e00640 1 --2- 172.21.15.43:0/2008672560 >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 0x7f73200b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:27:14.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.809+0000 7f7324c00640 1 --2- 172.21.15.43:0/2008672560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7310009870 tx=0x7f731002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c94affd790b4f66 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:14.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.810+0000 7f731fe00640 1 -- 172.21.15.43:0/2008672560 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310030840 con 0x7f732015a090 2024-01-16T15:27:14.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.810+0000 7f731fe00640 1 -- 172.21.15.43:0/2008672560 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73100309a0 con 0x7f732015a090 2024-01-16T15:27:14.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.810+0000 7f731fe00640 1 -- 172.21.15.43:0/2008672560 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310030d30 con 0x7f732015a090 2024-01-16T15:27:14.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 -- 172.21.15.43:0/2008672560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 msgr2=0x7f7320072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 --2- 172.21.15.43:0/2008672560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7310009870 tx=0x7f731002b980 comp rx=0 tx=0).stop 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 -- 172.21.15.43:0/2008672560 shutdown_connections 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 --2- 172.21.15.43:0/2008672560 >> v2:172.21.15.43:3300/0 conn(0x7f73200abff0 0x7f73200b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 --2- 172.21.15.43:0/2008672560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f732015a090 0x7f7320072150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.811+0000 7f7327600640 1 -- 172.21.15.43:0/2008672560 >> 172.21.15.43:0/2008672560 conn(0x7f73200b0d80 msgr2=0x7f73200b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.812+0000 7f7327600640 1 -- 172.21.15.43:0/2008672560 shutdown_connections 2024-01-16T15:27:14.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.812+0000 7f7327600640 1 -- 172.21.15.43:0/2008672560 wait complete. 2024-01-16T15:27:14.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.815+0000 7f7327600640 1 Processor -- start 2024-01-16T15:27:14.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.815+0000 7f7327600640 1 -- start start 2024-01-16T15:27:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73200abff0 0x7f7320145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 0x7f73200af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73201493e0 con 0x7f732015a090 2024-01-16T15:27:14.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73201496c0 con 0x7f73200b37d0 2024-01-16T15:27:14.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.817+0000 7f7327600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73201499a0 con 0x7f73200abff0 2024-01-16T15:27:14.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.818+0000 7f7317e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.819+0000 7f7317e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:57258/0 (socket says 172.21.15.43:57258) 2024-01-16T15:27:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.819+0000 7f7317e00640 1 -- 172.21.15.43:0/2898008312 learned_addr learned my addr 172.21.15.43:0/2898008312 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.819+0000 7f7325600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 0x7f73200af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7324c00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73200abff0 0x7f7320145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7317e00640 1 -- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73200abff0 msgr2=0x7f7320145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7317e00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73200abff0 0x7f7320145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7317e00640 1 -- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 msgr2=0x7f73200af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7317e00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 0x7f73200af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.820+0000 7f7317e00640 1 -- 172.21.15.43:0/2898008312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308009400 con 0x7f73200b37d0 2024-01-16T15:27:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.821+0000 7f7325600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 0x7f73200af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:27:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.822+0000 7f7317e00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f730800e960 tx=0x7f730800ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.822+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730800ca00 con 0x7f73200b37d0 2024-01-16T15:27:14.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.823+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730800cb60 con 0x7f73200b37d0 2024-01-16T15:27:14.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.823+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308013470 con 0x7f73200b37d0 2024-01-16T15:27:14.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.823+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73100094d0 con 0x7f73200b37d0 2024-01-16T15:27:14.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.823+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7320143150 con 0x7f73200b37d0 2024-01-16T15:27:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.826+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7320143370 con 0x7f73200b37d0 2024-01-16T15:27:14.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.828+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f73080135d0 con 0x7f73200b37d0 2024-01-16T15:27:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.829+0000 7f731d600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 0x7f72f806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.829+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7308088d00 con 0x7f73200b37d0 2024-01-16T15:27:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.832+0000 7f7324c00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 0x7f72f806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:14.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.838+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7308055300 con 0x7f73200b37d0 2024-01-16T15:27:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:14.841+0000 7f7324c00640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 0x7f72f806ea50 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7310009870 tx=0x7f7310008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.195+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7320074770 con 0x7f73200b37d0 2024-01-16T15:27:15.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.197+0000 7f731d600640 1 -- 172.21.15.43:0/2898008312 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v115) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7308017080 con 0x7f73200b37d0 2024-01-16T15:27:15.195 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:27:15.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 msgr2=0x7f72f806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 0x7f72f806ea50 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7310009870 tx=0x7f7310008030 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 msgr2=0x7f73200aedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f730800e960 tx=0x7f730800ee20 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 shutdown_connections 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f72f806c590 0x7f72f806ea50 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73200b37d0 0x7f73200aedb0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f732015a090 0x7f73200af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 --2- 172.21.15.43:0/2898008312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73200abff0 0x7f7320145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 >> 172.21.15.43:0/2898008312 conn(0x7f73200b0d80 msgr2=0x7f732015c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:15.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 shutdown_connections 2024-01-16T15:27:15.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.199+0000 7f7327600640 1 -- 172.21.15.43:0/2898008312 wait complete. 2024-01-16T15:27:15.214 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:27:15.215 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:15.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:15.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.507+0000 7f150ca00640 1 Processor -- start 2024-01-16T15:27:15.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.507+0000 7f150ca00640 1 -- start start 2024-01-16T15:27:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.508+0000 7f150ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15080cf760 0x7f150815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.508+0000 7f150ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.508+0000 7f150ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1508071700 con 0x7f1508143570 2024-01-16T15:27:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.508+0000 7f150ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150815af30 con 0x7f15080cf760 2024-01-16T15:27:15.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.508+0000 7f150ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150815b410 con 0x7f1508143950 2024-01-16T15:27:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.509+0000 7f1507200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.509+0000 7f1507200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46546/0 (socket says 172.21.15.43:46546) 2024-01-16T15:27:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.509+0000 7f1506800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1508143570 0x7f15080cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:42766/0 (socket says 172.21.15.43:42766) 2024-01-16T15:27:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.509+0000 7f1507200640 1 -- 172.21.15.43:0/3200809428 learned_addr learned my addr 172.21.15.43:0/3200809428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.509+0000 7f14ffe00640 1 --2- 172.21.15.43:0/3200809428 >> v2:172.21.15.69:3300/0 conn(0x7f15080cf760 0x7f150815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1505e00640 1 -- 172.21.15.43:0/3200809428 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153904589 0 0) 0x7f14f8001110 con 0x7f1508143570 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1505e00640 1 -- 172.21.15.43:0/3200809428 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003520 con 0x7f1508143570 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1507200640 1 -- 172.21.15.43:0/3200809428 >> v2:172.21.15.69:3300/0 conn(0x7f15080cf760 msgr2=0x7f150815a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1507200640 1 --2- 172.21.15.43:0/3200809428 >> v2:172.21.15.69:3300/0 conn(0x7f15080cf760 0x7f150815a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1507200640 1 -- 172.21.15.43:0/3200809428 >> v1:172.21.15.69:6789/0 conn(0x7f1508143570 legacy=0x7f15080cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.510+0000 7f1507200640 1 -- 172.21.15.43:0/3200809428 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150815b130 con 0x7f1508143950 2024-01-16T15:27:15.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.511+0000 7f1507200640 1 --2- 172.21.15.43:0/3200809428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1500009e50 tx=0x7f150002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80e4e498e3266d56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.511+0000 7f1505e00640 1 -- 172.21.15.43:0/3200809428 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150002ca80 con 0x7f1508143950 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.511+0000 7f1505e00640 1 -- 172.21.15.43:0/3200809428 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150002cbe0 con 0x7f1508143950 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.511+0000 7f1505e00640 1 -- 172.21.15.43:0/3200809428 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15000343f0 con 0x7f1508143950 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 -- 172.21.15.43:0/3200809428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 msgr2=0x7f150815a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 --2- 172.21.15.43:0/3200809428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1500009e50 tx=0x7f150002bee0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 -- 172.21.15.43:0/3200809428 shutdown_connections 2024-01-16T15:27:15.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 --2- 172.21.15.43:0/3200809428 >> v2:172.21.15.69:3300/0 conn(0x7f15080cf760 0x7f150815a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 --2- 172.21.15.43:0/3200809428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f150815a9f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 -- 172.21.15.43:0/3200809428 >> 172.21.15.43:0/3200809428 conn(0x7f150806a860 msgr2=0x7f150806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:15.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 -- 172.21.15.43:0/3200809428 shutdown_connections 2024-01-16T15:27:15.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.512+0000 7f150ca00640 1 -- 172.21.15.43:0/3200809428 wait complete. 2024-01-16T15:27:15.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.514+0000 7f150ca00640 1 Processor -- start 2024-01-16T15:27:15.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.514+0000 7f150ca00640 1 -- start start 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15080cf760 0x7f15081598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f15080ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080c5820 con 0x7f1508143570 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080c4ba0 con 0x7f15080cf760 2024-01-16T15:27:15.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f150ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080c4e80 con 0x7f1508143950 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54338/0 (socket says 172.21.15.43:54338) 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f14ffe00640 1 -- 172.21.15.43:0/3854218143 learned_addr learned my addr 172.21.15.43:0/3854218143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.515+0000 7f1506800640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15080cf760 0x7f15081598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f1507200640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f15080ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 -- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 msgr2=0x7f15080ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f15080ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 -- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15080cf760 msgr2=0x7f15081598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15080cf760 0x7f15081598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 -- 172.21.15.43:0/3854218143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500009b30 con 0x7f1508143570 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f1507200640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f15080ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14ffe00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f14f000e980 tx=0x7f14f000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0009670 con 0x7f1508143570 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f000ca00 con 0x7f1508143570 2024-01-16T15:27:15.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15080c5160 con 0x7f1508143570 2024-01-16T15:27:15.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f000cd90 con 0x7f1508143570 2024-01-16T15:27:15.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.516+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1508132630 con 0x7f1508143570 2024-01-16T15:27:15.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.518+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15080c5a90 con 0x7f1508143570 2024-01-16T15:27:15.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.518+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f14f0013470 con 0x7f1508143570 2024-01-16T15:27:15.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.519+0000 7f14fea00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 0x7f14e006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.519+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14f0088d40 con 0x7f1508143570 2024-01-16T15:27:15.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.520+0000 7f1506800640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 0x7f14e006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.520+0000 7f1506800640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 0x7f14e006e980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f14f8007fd0 tx=0x7f14f8000bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.524+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14f0055290 con 0x7f1508143570 2024-01-16T15:27:15.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.714+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f150804e840 con 0x7f1508143570 2024-01-16T15:27:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.716+0000 7f14fea00640 1 -- 172.21.15.43:0/3854218143 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f14f0017090 con 0x7f1508143570 2024-01-16T15:27:15.715 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:15.716 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":115,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:19:15.246810+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:15.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 msgr2=0x7f14e006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 0x7f14e006e980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f14f8007fd0 tx=0x7f14f8000bc0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 msgr2=0x7f15081320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f14f000e980 tx=0x7f14f000ee40 comp rx=0 tx=0).stop 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 shutdown_connections 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14e006c4c0 0x7f14e006e980 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1508143950 0x7f15080ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1508143570 0x7f15081320f0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 --2- 172.21.15.43:0/3854218143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15080cf760 0x7f15081598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 >> 172.21.15.43:0/3854218143 conn(0x7f150806a860 msgr2=0x7f150815b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 shutdown_connections 2024-01-16T15:27:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.719+0000 7f150ca00640 1 -- 172.21.15.43:0/3854218143 wait complete. 2024-01-16T15:27:15.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:18:47.672212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:18:48.980104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:15.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.957+0000 7f8706600640 1 Processor -- start 2024-01-16T15:27:15.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.957+0000 7f8706600640 1 -- start start 2024-01-16T15:27:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8706600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8700157290 0x7f870006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8706600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87000b23d0 con 0x7f870015a840 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8706600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87000b2bf0 con 0x7f8700156eb0 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8706600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8700071000 con 0x7f8700157290 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8705c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8700157290 0x7f870006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8705c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8700157290 0x7f870006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54344/0 (socket says 172.21.15.43:54344) 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.959+0000 7f8705c00640 1 -- 172.21.15.43:0/2101684240 learned_addr learned my addr 172.21.15.43:0/2101684240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:15.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.960+0000 7f8705c00640 1 -- 172.21.15.43:0/2101684240 >> v1:172.21.15.69:6789/0 conn(0x7f8700156eb0 legacy=0x7f87000b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.960+0000 7f8705c00640 1 -- 172.21.15.43:0/2101684240 >> v1:172.21.15.43:6789/0 conn(0x7f870015a840 legacy=0x7f87000708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.960+0000 7f8705c00640 1 -- 172.21.15.43:0/2101684240 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87000b2630 con 0x7f8700157290 2024-01-16T15:27:15.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f8705c00640 1 --2- 172.21.15.43:0/2101684240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f870006cba0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f86f800af60 tx=0x7f86f802d100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbf280d56c1e11c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f86ff400640 1 -- 172.21.15.43:0/2101684240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f802daa0 con 0x7f8700157290 2024-01-16T15:27:15.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f86ff400640 1 -- 172.21.15.43:0/2101684240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f802dc00 con 0x7f8700157290 2024-01-16T15:27:15.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f86ff400640 1 -- 172.21.15.43:0/2101684240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8008500 con 0x7f8700157290 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f8706600640 1 -- 172.21.15.43:0/2101684240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 msgr2=0x7f870006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.961+0000 7f8706600640 1 --2- 172.21.15.43:0/2101684240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f870006cba0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f86f800af60 tx=0x7f86f802d100 comp rx=0 tx=0).stop 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.962+0000 7f8706600640 1 -- 172.21.15.43:0/2101684240 shutdown_connections 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.962+0000 7f8706600640 1 --2- 172.21.15.43:0/2101684240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f870006cba0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.962+0000 7f8706600640 1 -- 172.21.15.43:0/2101684240 >> 172.21.15.43:0/2101684240 conn(0x7f87000cec10 msgr2=0x7f87000cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.962+0000 7f8706600640 1 -- 172.21.15.43:0/2101684240 shutdown_connections 2024-01-16T15:27:15.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.962+0000 7f8706600640 1 -- 172.21.15.43:0/2101684240 wait complete. 2024-01-16T15:27:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.963+0000 7f8706600640 1 Processor -- start 2024-01-16T15:27:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.963+0000 7f8706600640 1 -- start start 2024-01-16T15:27:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700156eb0 0x7f8700145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015a840 0x7f87000ad970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87000cdfb0 con 0x7f8700157290 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87000ad4e0 con 0x7f870015a840 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.964+0000 7f8706600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87000ac910 con 0x7f8700156eb0 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54354/0 (socket says 172.21.15.43:54354) 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 -- 172.21.15.43:0/3711117547 learned_addr learned my addr 172.21.15.43:0/3711117547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 -- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700156eb0 msgr2=0x7f8700145570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700156eb0 0x7f8700145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 -- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015a840 msgr2=0x7f87000ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015a840 0x7f87000ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.965+0000 7f86ffe00640 1 -- 172.21.15.43:0/3711117547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f800ac40 con 0x7f8700157290 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f86ffe00640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f86e800c390 tx=0x7f86e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e80052d0 con 0x7f8700157290 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8005430 con 0x7f8700157290 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8003830 con 0x7f8700157290 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87000acbb0 con 0x7f8700157290 2024-01-16T15:27:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.966+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87000ad170 con 0x7f8700157290 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.967+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870004e840 con 0x7f8700157290 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.968+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f86e8003990 con 0x7f8700157290 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.969+0000 7f86fcc00640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 0x7f86d806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.969+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f86e8088090 con 0x7f8700157290 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.969+0000 7f8705200640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 0x7f86d806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.970+0000 7f8705200640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 0x7f86d806e9b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f86f0000f60 tx=0x7f86f00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:15.973+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86e80545e0 con 0x7f8700157290 2024-01-16T15:27:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.162+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8700071600 con 0x7f8700157290 2024-01-16T15:27:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.163+0000 7f86fcc00640 1 -- 172.21.15.43:0/3711117547 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86e8017020 con 0x7f8700157290 2024-01-16T15:27:16.162 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.165+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 msgr2=0x7f86d806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.165+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 0x7f86d806e9b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f86f0000f60 tx=0x7f86f00079b0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.165+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 msgr2=0x7f8700145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.165+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f86e800c390 tx=0x7f86e800c850 comp rx=0 tx=0).stop 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 shutdown_connections 2024-01-16T15:27:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f86d806c4f0 0x7f86d806e9b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f870015a840 0x7f87000ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8700157290 0x7f8700145fd0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 --2- 172.21.15.43:0/3711117547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8700156eb0 0x7f8700145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 >> 172.21.15.43:0/3711117547 conn(0x7f87000cec10 msgr2=0x7f8700154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 shutdown_connections 2024-01-16T15:27:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.166+0000 7f8706600640 1 -- 172.21.15.43:0/3711117547 wait complete. 2024-01-16T15:27:16.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.409+0000 7fc287e00640 1 Processor -- start 2024-01-16T15:27:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.410+0000 7fc287e00640 1 -- start start 2024-01-16T15:27:16.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.411+0000 7fc287e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc28815f270 0x7fc2880cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.411+0000 7fc287e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 0x7fc2880cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.411+0000 7fc287e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28815f930 con 0x7fc288144770 2024-01-16T15:27:16.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.411+0000 7fc287e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2880b43b0 con 0x7fc28815f270 2024-01-16T15:27:16.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.411+0000 7fc287e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2880b3b10 con 0x7fc28806d9e0 2024-01-16T15:27:16.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.412+0000 7fc286a00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc288144770 0x7fc28815eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59270/0 (socket says 172.21.15.43:59270) 2024-01-16T15:27:16.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.412+0000 7fc286a00640 1 -- 172.21.15.43:0/1842226828 learned_addr learned my addr 172.21.15.43:0/1842226828 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:16.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.412+0000 7fc27fe00640 1 --2- 172.21.15.43:0/1842226828 >> v2:172.21.15.69:3300/0 conn(0x7fc28815f270 0x7fc2880cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.412+0000 7fc287400640 1 --2- 172.21.15.43:0/1842226828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 0x7fc2880cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc286000640 1 -- 172.21.15.43:0/1842226828 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135210467 0 0) 0x7fc28815f930 con 0x7fc288144770 2024-01-16T15:27:16.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc286000640 1 -- 172.21.15.43:0/1842226828 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc268003520 con 0x7fc288144770 2024-01-16T15:27:16.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc287400640 1 -- 172.21.15.43:0/1842226828 >> v2:172.21.15.69:3300/0 conn(0x7fc28815f270 msgr2=0x7fc2880cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:16.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc287400640 1 --2- 172.21.15.43:0/1842226828 >> v2:172.21.15.69:3300/0 conn(0x7fc28815f270 0x7fc2880cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc287400640 1 -- 172.21.15.43:0/1842226828 >> v1:172.21.15.43:6789/0 conn(0x7fc288144770 legacy=0x7fc28815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.413+0000 7fc287400640 1 -- 172.21.15.43:0/1842226828 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2880b3df0 con 0x7fc28806d9e0 2024-01-16T15:27:16.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.414+0000 7fc287400640 1 --2- 172.21.15.43:0/1842226828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 0x7fc2880cd9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc280009d90 tx=0x7fc28002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b41c76e4a37624 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.415+0000 7fc286000640 1 -- 172.21.15.43:0/1842226828 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28002cc60 con 0x7fc28806d9e0 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.415+0000 7fc286000640 1 -- 172.21.15.43:0/1842226828 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28002cdc0 con 0x7fc28806d9e0 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.415+0000 7fc286000640 1 -- 172.21.15.43:0/1842226828 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280034640 con 0x7fc28806d9e0 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.415+0000 7fc287e00640 1 -- 172.21.15.43:0/1842226828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 msgr2=0x7fc2880cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.415+0000 7fc287e00640 1 --2- 172.21.15.43:0/1842226828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 0x7fc2880cd9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc280009d90 tx=0x7fc28002beb0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 -- 172.21.15.43:0/1842226828 shutdown_connections 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 --2- 172.21.15.43:0/1842226828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc28806d9e0 0x7fc2880cd9e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 --2- 172.21.15.43:0/1842226828 >> v2:172.21.15.69:3300/0 conn(0x7fc28815f270 0x7fc2880cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 -- 172.21.15.43:0/1842226828 >> 172.21.15.43:0/1842226828 conn(0x7fc2880cec10 msgr2=0x7fc2880cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 -- 172.21.15.43:0/1842226828 shutdown_connections 2024-01-16T15:27:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.416+0000 7fc287e00640 1 -- 172.21.15.43:0/1842226828 wait complete. 2024-01-16T15:27:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.418+0000 7fc287e00640 1 Processor -- start 2024-01-16T15:27:16.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.418+0000 7fc287e00640 1 -- start start 2024-01-16T15:27:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28806d9e0 0x7fc28815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 0x7fc28815e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc288070130 con 0x7fc28815f270 2024-01-16T15:27:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28813cad0 con 0x7fc28806d9e0 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28815d760 con 0x7fc288144770 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54356/0 (socket says 172.21.15.43:54356) 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc287400640 1 -- 172.21.15.43:0/3469507911 learned_addr learned my addr 172.21.15.43:0/3469507911 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc286a00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28806d9e0 0x7fc28815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.420+0000 7fc27fe00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 0x7fc28815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 -- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 msgr2=0x7fc28815e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 0x7fc28815e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 -- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28806d9e0 msgr2=0x7fc28815d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28806d9e0 0x7fc28815d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 -- 172.21.15.43:0/3469507911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280009a70 con 0x7fc28815f270 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc27fe00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 0x7fc28815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287400640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc280009d90 tx=0x7fc280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280004880 con 0x7fc28815f270 2024-01-16T15:27:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280009450 con 0x7fc28815f270 2024-01-16T15:27:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28815da40 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.421+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280034640 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.422+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28813c450 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.423+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28813cc80 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.424+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fc28003b030 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.424+0000 7fc27f400640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 0x7fc26006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.425+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc2800aa970 con 0x7fc28815f270 2024-01-16T15:27:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.425+0000 7fc286a00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 0x7fc26006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.425+0000 7fc286a00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 0x7fc26006e980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc278005880 tx=0x7fc278009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.429+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc28813cc80 con 0x7fc28815f270 2024-01-16T15:27:16.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.625+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc28813cc40 con 0x7fc28815f270 2024-01-16T15:27:16.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.625+0000 7fc27f400640 1 -- 172.21.15.43:0/3469507911 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2800772d0 con 0x7fc28815f270 2024-01-16T15:27:16.625 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:27:16.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.628+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 msgr2=0x7fc26006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.628+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 0x7fc26006e980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc278005880 tx=0x7fc278009040 comp rx=0 tx=0).stop 2024-01-16T15:27:16.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.628+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 msgr2=0x7fc288141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.628+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc280009d90 tx=0x7fc280004a40 comp rx=0 tx=0).stop 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 shutdown_connections 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc26006c4c0 0x7fc26006e980 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc28815f270 0x7fc288141090 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc288144770 0x7fc28815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 --2- 172.21.15.43:0/3469507911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28806d9e0 0x7fc28815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 >> 172.21.15.43:0/3469507911 conn(0x7fc2880cec10 msgr2=0x7fc2880cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:16.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 shutdown_connections 2024-01-16T15:27:16.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.629+0000 7fc287e00640 1 -- 172.21.15.43:0/3469507911 wait complete. 2024-01-16T15:27:16.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.873+0000 7f8a01a00640 1 Processor -- start 2024-01-16T15:27:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.874+0000 7f8a01a00640 1 -- start start 2024-01-16T15:27:16.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.875+0000 7f8a01a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc15d3b0 0x7f89fc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.875+0000 7f8a01a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f89fc0d0fa0 0x7f89fc0d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.875+0000 7f8a01a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0d2010 con 0x7f89fc0d1bc0 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.875+0000 7f8a01a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89fc0c6bf0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.875+0000 7f8a01a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0d1400 con 0x7f89fc15d3b0 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f89fc0d0fa0 0x7f89fc0d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f89fc0d0fa0 0x7f89fc0d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54368/0 (socket says 172.21.15.43:54368) 2024-01-16T15:27:16.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 -- 172.21.15.43:0/1961608927 learned_addr learned my addr 172.21.15.43:0/1961608927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:16.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89fbe00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc15d3b0 0x7f89fc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 -- 172.21.15.43:0/1961608927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc15d3b0 msgr2=0x7f89fc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc15d3b0 0x7f89fc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 -- 172.21.15.43:0/1961608927 >> v1:172.21.15.69:6789/0 conn(0x7f89fc0d1bc0 legacy=0x7f89fc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.876+0000 7f89f3e00640 1 -- 172.21.15.43:0/1961608927 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc0cedc0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.877+0000 7f89f3e00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0d1680 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4249e60afd728b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f89fb400640 1 -- 172.21.15.43:0/1961608927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40317c0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f89fb400640 1 -- 172.21.15.43:0/1961608927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4031920 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f89fb400640 1 -- 172.21.15.43:0/1961608927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4031cb0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f8a01a00640 1 -- 172.21.15.43:0/1961608927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 msgr2=0x7f89fc0d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f8a01a00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0d1680 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e402b860 comp rx=0 tx=0).stop 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f8a01a00640 1 -- 172.21.15.43:0/1961608927 shutdown_connections 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.878+0000 7f8a01a00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0d1680 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.879+0000 7f8a01a00640 1 --2- 172.21.15.43:0/1961608927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc15d3b0 0x7f89fc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.879+0000 7f8a01a00640 1 -- 172.21.15.43:0/1961608927 >> 172.21.15.43:0/1961608927 conn(0x7f89fc0d30a0 msgr2=0x7f89fc0d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.879+0000 7f8a01a00640 1 -- 172.21.15.43:0/1961608927 shutdown_connections 2024-01-16T15:27:16.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.879+0000 7f8a01a00640 1 -- 172.21.15.43:0/1961608927 wait complete. 2024-01-16T15:27:16.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.881+0000 7f8a01a00640 1 Processor -- start 2024-01-16T15:27:16.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.881+0000 7f8a01a00640 1 -- start start 2024-01-16T15:27:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0d1bc0 0x7f89fc0b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 0x7f89fc157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0b81e0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0b7560 con 0x7f89fc15d3b0 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.882+0000 7f8a01a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0b7840 con 0x7f89fc0d1bc0 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54374/0 (socket says 172.21.15.43:54374) 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 -- 172.21.15.43:0/2184530238 learned_addr learned my addr 172.21.15.43:0/2184530238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89f3e00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0d1bc0 0x7f89fc0b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f8a01000640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 0x7f89fc157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 -- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0d1bc0 msgr2=0x7f89fc0b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0d1bc0 0x7f89fc0b6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 -- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 msgr2=0x7f89fc157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 0x7f89fc157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f89fbe00640 1 -- 172.21.15.43:0/2184530238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40094d0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.883+0000 7f8a01000640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 0x7f89fc157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f89fbe00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00d800 tx=0x7f89ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00b6c0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec00b820 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00bbb0 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89fc0b7b20 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.884+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89fc0b7250 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.885+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89fc04e840 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.886+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f89ec00bd10 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.888+0000 7f89f8c00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 0x7f89d406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.888+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f89ec088d80 con 0x7f89fc0d0fa0 2024-01-16T15:27:16.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.888+0000 7f89f3e00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 0x7f89d406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:16.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.889+0000 7f89f3e00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 0x7f89d406e980 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f89e4036000 tx=0x7f89e403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:16.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:16.892+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89ec055640 con 0x7f89fc0d0fa0 2024-01-16T15:27:17.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.086+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89fc0b4d40 con 0x7f89fc0d0fa0 2024-01-16T15:27:17.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.086+0000 7f89f8c00640 1 -- 172.21.15.43:0/2184530238 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f89ec058c60 con 0x7f89fc0d0fa0 2024-01-16T15:27:17.086 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:27:17.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 msgr2=0x7f89d406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 0x7f89d406e980 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f89e4036000 tx=0x7f89e403d040 comp rx=0 tx=0).stop 2024-01-16T15:27:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 msgr2=0x7f89fc0c6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00d800 tx=0x7f89ec00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 shutdown_connections 2024-01-16T15:27:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f89d406c4c0 0x7f89d406e980 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc15d3b0 0x7f89fc157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0d1bc0 0x7f89fc0b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 --2- 172.21.15.43:0/2184530238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89fc0d0fa0 0x7f89fc0c6580 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 >> 172.21.15.43:0/2184530238 conn(0x7f89fc0d30a0 msgr2=0x7f89fc160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.089+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 shutdown_connections 2024-01-16T15:27:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.090+0000 7f8a01a00640 1 -- 172.21.15.43:0/2184530238 wait complete. 2024-01-16T15:27:17.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.322+0000 7f544ac00640 1 Processor -- start 2024-01-16T15:27:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.322+0000 7f544ac00640 1 -- start start 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f544415a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544ac00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544415ad90 con 0x7f54440b1c40 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544ac00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406cf20 con 0x7f544406c9a0 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544406d200 con 0x7f5444070a40 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544a200640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f54440b1c40 0x7f54440b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59272/0 (socket says 172.21.15.43:59272) 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.324+0000 7f544a200640 1 -- 172.21.15.43:0/1167649927 learned_addr learned my addr 172.21.15.43:0/1167649927 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5449800640 1 --2- 172.21.15.43:0/1167649927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f544415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264154819 0 0) 0x7f544415ad90 con 0x7f54440b1c40 2024-01-16T15:27:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424003440 con 0x7f54440b1c40 2024-01-16T15:27:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5449800640 1 -- 172.21.15.43:0/1167649927 >> v1:172.21.15.69:6789/0 conn(0x7f544406c9a0 legacy=0x7f5444070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604755877 0 0) 0x7f5424003440 con 0x7f54440b1c40 2024-01-16T15:27:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5449800640 1 -- 172.21.15.43:0/1167649927 >> v1:172.21.15.43:6789/0 conn(0x7f54440b1c40 legacy=0x7f54440b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.325+0000 7f5449800640 1 -- 172.21.15.43:0/1167649927 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544406d4e0 con 0x7f5444070a40 2024-01-16T15:27:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.327+0000 7f5449800640 1 --2- 172.21.15.43:0/1167649927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f544415a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f54340097d0 tx=0x7f543402b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b967ce89b86b1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.327+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0019f0 con 0x7f5444070a40 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.327+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543402e050 con 0x7f5444070a40 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.327+0000 7f5443e00640 1 -- 172.21.15.43:0/1167649927 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434030730 con 0x7f5444070a40 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 -- 172.21.15.43:0/1167649927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 msgr2=0x7f544415a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 --2- 172.21.15.43:0/1167649927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f544415a850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f54340097d0 tx=0x7f543402b5d0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 -- 172.21.15.43:0/1167649927 shutdown_connections 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 --2- 172.21.15.43:0/1167649927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f544415a850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 -- 172.21.15.43:0/1167649927 >> 172.21.15.43:0/1167649927 conn(0x7f54440b0840 msgr2=0x7f54440b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:17.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 -- 172.21.15.43:0/1167649927 shutdown_connections 2024-01-16T15:27:17.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.328+0000 7f544ac00640 1 -- 172.21.15.43:0/1167649927 wait complete. 2024-01-16T15:27:17.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.330+0000 7f544ac00640 1 Processor -- start 2024-01-16T15:27:17.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.330+0000 7f544ac00640 1 -- start start 2024-01-16T15:27:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f5444158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54440b1c40 0x7f544414b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544414dad0 con 0x7f544406c9a0 2024-01-16T15:27:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544414cec0 con 0x7f54440b1c40 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f544ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544414d1a0 con 0x7f5444070a40 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f5449800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f5449800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54376/0 (socket says 172.21.15.43:54376) 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.331+0000 7f5449800640 1 -- 172.21.15.43:0/3749052915 learned_addr learned my addr 172.21.15.43:0/3749052915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5448e00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f5444158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f544a200640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54440b1c40 0x7f544414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 -- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 msgr2=0x7f5444158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f5444158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 -- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54440b1c40 msgr2=0x7f544414b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54440b1c40 0x7f544414b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 -- 172.21.15.43:0/3749052915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5434009400 con 0x7f544406c9a0 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5449800640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f54340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.332+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434004230 con 0x7f544406c9a0 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.333+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434004390 con 0x7f544406c9a0 2024-01-16T15:27:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.333+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544414d480 con 0x7f544406c9a0 2024-01-16T15:27:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.333+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543403c920 con 0x7f544406c9a0 2024-01-16T15:27:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.333+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544414ca50 con 0x7f544406c9a0 2024-01-16T15:27:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.334+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544404e840 con 0x7f544406c9a0 2024-01-16T15:27:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.335+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f5434039050 con 0x7f544406c9a0 2024-01-16T15:27:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.336+0000 7f5441600640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 0x7f541c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.336+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f54340aa440 con 0x7f544406c9a0 2024-01-16T15:27:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.336+0000 7f5448e00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 0x7f541c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.337+0000 7f5448e00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 0x7f541c06e980 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5444071580 tx=0x7f542c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:17.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.341+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5434076ab0 con 0x7f544406c9a0 2024-01-16T15:27:17.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.552+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f54440b3710 con 0x7f544406c9a0 2024-01-16T15:27:17.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.553+0000 7f5441600640 1 -- 172.21.15.43:0/3749052915 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v115) v1 ==== 80+0+0 (secure 0 0 0) 0x7f543404b070 con 0x7f544406c9a0 2024-01-16T15:27:17.552 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:27:17.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 msgr2=0x7f541c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 0x7f541c06e980 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5444071580 tx=0x7f542c009040 comp rx=0 tx=0).stop 2024-01-16T15:27:17.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 msgr2=0x7f544414df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f54340046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 shutdown_connections 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f541c06c4c0 0x7f541c06e980 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54440b1c40 0x7f544414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5444070a40 0x7f5444158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 --2- 172.21.15.43:0/3749052915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f544406c9a0 0x7f544414df60 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 >> 172.21.15.43:0/3749052915 conn(0x7f54440b0840 msgr2=0x7f54440caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 shutdown_connections 2024-01-16T15:27:17.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:17.556+0000 7f544ac00640 1 -- 172.21.15.43:0/3749052915 wait complete. 2024-01-16T15:27:17.575 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-01-16T15:27:17.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:17.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:27:17.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[76178]: fuse finished with error 0 and tester_r 0 2024-01-16T15:27:18.289 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:27:18.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:27:18.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:18.289 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:27:18.340 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-01-16T15:27:18.340 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:18.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:27:18.386 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:27:18.386 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:18.445 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:27:18.445 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:18.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:18.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.718+0000 7f0e12200640 1 Processor -- start 2024-01-16T15:27:18.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.718+0000 7f0e12200640 1 -- start start 2024-01-16T15:27:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e12200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0e0c06dc30 0x7f0e0c0acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e12200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c071cd0 con 0x7f0e0c070780 2024-01-16T15:27:18.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e12200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0b2120 con 0x7f0e0c160970 2024-01-16T15:27:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e12200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0c0b2400 con 0x7f0e0c06dc30 2024-01-16T15:27:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e0aa00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0e0c070780 0x7f0e0c06d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59280/0 (socket says 172.21.15.43:59280) 2024-01-16T15:27:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.721+0000 7f0e0aa00640 1 -- 172.21.15.43:0/4115863282 learned_addr learned my addr 172.21.15.43:0/4115863282 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:18.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.720+0000 7f0e0be00640 1 --2- 172.21.15.43:0/4115863282 >> v2:172.21.15.43:3300/0 conn(0x7f0e0c06dc30 0x7f0e0c0acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.721+0000 7f0e0be00640 1 -- 172.21.15.43:0/4115863282 >> v1:172.21.15.69:6789/0 conn(0x7f0e0c160970 legacy=0x7f0e0c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.721+0000 7f0e0be00640 1 -- 172.21.15.43:0/4115863282 >> v1:172.21.15.43:6789/0 conn(0x7f0e0c070780 legacy=0x7f0e0c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.721+0000 7f0e0be00640 1 -- 172.21.15.43:0/4115863282 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0b26e0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.721+0000 7f0e0a000640 1 -- 172.21.15.43:0/4115863282 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81827112 0 0) 0x7f0e0c071cd0 con 0x7f0e0c070780 2024-01-16T15:27:18.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.722+0000 7f0e0be00640 1 --2- 172.21.15.43:0/4115863282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c0acbe0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009d90 tx=0x7f0e0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2f6a7a89322788 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e0a000640 1 -- 172.21.15.43:0/4115863282 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0402cc60 con 0x7f0e0c06dc30 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e0a000640 1 -- 172.21.15.43:0/4115863282 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0402cdc0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e0a000640 1 -- 172.21.15.43:0/4115863282 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04034660 con 0x7f0e0c06dc30 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 -- 172.21.15.43:0/4115863282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 msgr2=0x7f0e0c0acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 --2- 172.21.15.43:0/4115863282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c0acbe0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009d90 tx=0x7f0e0402beb0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 -- 172.21.15.43:0/4115863282 shutdown_connections 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 --2- 172.21.15.43:0/4115863282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c0acbe0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 -- 172.21.15.43:0/4115863282 >> 172.21.15.43:0/4115863282 conn(0x7f0e0c0b0d80 msgr2=0x7f0e0c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:18.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 -- 172.21.15.43:0/4115863282 shutdown_connections 2024-01-16T15:27:18.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.723+0000 7f0e12200640 1 -- 172.21.15.43:0/4115863282 wait complete. 2024-01-16T15:27:18.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.725+0000 7f0e12200640 1 Processor -- start 2024-01-16T15:27:18.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.725+0000 7f0e12200640 1 -- start start 2024-01-16T15:27:18.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c070780 0x7f0e0c152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 0x7f0e0c1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1f65a0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1f6290 con 0x7f0e0c160970 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.726+0000 7f0e12200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c144dc0 con 0x7f0e0c070780 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54386/0 (socket says 172.21.15.43:54386) 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 -- 172.21.15.43:0/609661191 learned_addr learned my addr 172.21.15.43:0/609661191 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0be00640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 0x7f0e0c1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0aa00640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c070780 0x7f0e0c152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 -- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c070780 msgr2=0x7f0e0c152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c070780 0x7f0e0c152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 -- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 msgr2=0x7f0e0c1f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 0x7f0e0c1f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0b400640 1 -- 172.21.15.43:0/609661191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04009a70 con 0x7f0e0c06dc30 2024-01-16T15:27:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.727+0000 7f0e0be00640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 0x7f0e0c1f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e0b400640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c430 tx=0x7f0dfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0052d0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc005430 con 0x7f0e0c06dc30 2024-01-16T15:27:18.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc003830 con 0x7f0e0c06dc30 2024-01-16T15:27:18.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c159b20 con 0x7f0e0c06dc30 2024-01-16T15:27:18.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.728+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c15a0c0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.729+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c074770 con 0x7f0e0c06dc30 2024-01-16T15:27:18.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.730+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f0dfc003990 con 0x7f0e0c06dc30 2024-01-16T15:27:18.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.731+0000 7f0e03400640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 0x7f0de406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.731+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0dfc0880b0 con 0x7f0e0c06dc30 2024-01-16T15:27:18.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.731+0000 7f0e0aa00640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 0x7f0de406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.732+0000 7f0e0aa00640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 0x7f0de406e980 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c152090 tx=0x7f0df4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.736+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0dfc054600 con 0x7f0e0c06dc30 2024-01-16T15:27:18.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.936+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e0c06e290 con 0x7f0e0c06dc30 2024-01-16T15:27:18.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.939+0000 7f0e03400640 1 -- 172.21.15.43:0/609661191 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f0dfc017020 con 0x7f0e0c06dc30 2024-01-16T15:27:18.938 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:18.939 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":115,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:19:15.246810+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:18.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 msgr2=0x7f0de406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 0x7f0de406e980 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c152090 tx=0x7f0df4009040 comp rx=0 tx=0).stop 2024-01-16T15:27:18.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 msgr2=0x7f0e0c1595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c430 tx=0x7f0dfc00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 shutdown_connections 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0de406c4c0 0x7f0de406e980 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c160970 0x7f0e0c1f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c070780 0x7f0e0c152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 --2- 172.21.15.43:0/609661191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e0c06dc30 0x7f0e0c1595e0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 >> 172.21.15.43:0/609661191 conn(0x7f0e0c0b0d80 msgr2=0x7f0e0c0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 shutdown_connections 2024-01-16T15:27:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:18.942+0000 7f0e12200640 1 -- 172.21.15.43:0/609661191 wait complete. 2024-01-16T15:27:18.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:18:47.672212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:18:48.980104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:18.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.183+0000 7f7215200640 1 Processor -- start 2024-01-16T15:27:19.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.183+0000 7f7215200640 1 -- start start 2024-01-16T15:27:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f7215200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f721015efa0 0x7f7210161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f7215200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f7215200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721006c9a0 con 0x7f72100b2970 2024-01-16T15:27:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f7215200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721006cc80 con 0x7f721015efa0 2024-01-16T15:27:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f7215200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721006cf60 con 0x7f721006da40 2024-01-16T15:27:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f720ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f720ea00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:46646/0 (socket says 172.21.15.43:46646) 2024-01-16T15:27:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.185+0000 7f720ea00640 1 -- 172.21.15.43:0/3216041949 learned_addr learned my addr 172.21.15.43:0/3216041949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.186+0000 7f720f400640 1 --2- 172.21.15.43:0/3216041949 >> v2:172.21.15.69:3300/0 conn(0x7f721015efa0 0x7f7210161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.186+0000 7f720ea00640 1 -- 172.21.15.43:0/3216041949 >> v2:172.21.15.69:3300/0 conn(0x7f721015efa0 msgr2=0x7f7210161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:19.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.186+0000 7f720ea00640 1 --2- 172.21.15.43:0/3216041949 >> v2:172.21.15.69:3300/0 conn(0x7f721015efa0 0x7f7210161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.186+0000 7f720ea00640 1 -- 172.21.15.43:0/3216041949 >> v1:172.21.15.69:6789/0 conn(0x7f72100b2970 legacy=0x7f72101650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.186+0000 7f720ea00640 1 -- 172.21.15.43:0/3216041949 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721006d240 con 0x7f721006da40 2024-01-16T15:27:19.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f720ea00640 1 --2- 172.21.15.43:0/3216041949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d549aa7ad98ba3d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f720e000640 1 -- 172.21.15.43:0/3216041949 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8030840 con 0x7f721006da40 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f720e000640 1 -- 172.21.15.43:0/3216041949 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80309a0 con 0x7f721006da40 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f720e000640 1 -- 172.21.15.43:0/3216041949 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8030d30 con 0x7f721006da40 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f7215200640 1 -- 172.21.15.43:0/3216041949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 msgr2=0x7f721006c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.188+0000 7f7215200640 1 --2- 172.21.15.43:0/3216041949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f802b980 comp rx=0 tx=0).stop 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 -- 172.21.15.43:0/3216041949 shutdown_connections 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 --2- 172.21.15.43:0/3216041949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f721006c310 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 --2- 172.21.15.43:0/3216041949 >> v2:172.21.15.69:3300/0 conn(0x7f721015efa0 0x7f7210161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 -- 172.21.15.43:0/3216041949 >> 172.21.15.43:0/3216041949 conn(0x7f72100cec10 msgr2=0x7f72100cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 -- 172.21.15.43:0/3216041949 shutdown_connections 2024-01-16T15:27:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.189+0000 7f7215200640 1 -- 172.21.15.43:0/3216041949 wait complete. 2024-01-16T15:27:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.190+0000 7f7215200640 1 Processor -- start 2024-01-16T15:27:19.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 -- start start 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f72101ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72100b2970 0x7f72101ed490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210158980 con 0x7f721015efa0 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100b1640 con 0x7f72100b2970 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.191+0000 7f7215200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7210158bd0 con 0x7f721006da40 2024-01-16T15:27:19.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54388/0 (socket says 172.21.15.43:54388) 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 -- 172.21.15.43:0/1048859174 learned_addr learned my addr 172.21.15.43:0/1048859174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720f400640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f72101ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720ea00640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72100b2970 0x7f72101ed490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 -- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 msgr2=0x7f72101ecf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f72101ecf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 -- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72100b2970 msgr2=0x7f72101ed490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72100b2970 0x7f72101ed490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.192+0000 7f720fe00640 1 -- 172.21.15.43:0/1048859174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80094d0 con 0x7f721015efa0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f720fe00640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7208001eb0 tx=0x7f720800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72080052d0 con 0x7f721015efa0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7208005430 con 0x7f721015efa0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208003810 con 0x7f721015efa0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7210158eb0 con 0x7f721015efa0 2024-01-16T15:27:19.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.193+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7210154420 con 0x7f721015efa0 2024-01-16T15:27:19.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.194+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721004e840 con 0x7f721015efa0 2024-01-16T15:27:19.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.196+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7208003970 con 0x7f721015efa0 2024-01-16T15:27:19.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.196+0000 7f7207400640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 0x7f71e806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.196+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7208088130 con 0x7f721015efa0 2024-01-16T15:27:19.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.197+0000 7f720f400640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 0x7f71e806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.197+0000 7f720f400640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 0x7f71e806e980 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f72000073f0 tx=0x7f720000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.201+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7208054650 con 0x7f721015efa0 2024-01-16T15:27:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.396+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f721006c4b0 con 0x7f721015efa0 2024-01-16T15:27:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.397+0000 7f7207400640 1 -- 172.21.15.43:0/1048859174 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7208017070 con 0x7f721015efa0 2024-01-16T15:27:19.396 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:27:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 msgr2=0x7f71e806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 0x7f71e806e980 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f72000073f0 tx=0x7f720000b040 comp rx=0 tx=0).stop 2024-01-16T15:27:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 msgr2=0x7f72101ed9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7208001eb0 tx=0x7f720800c890 comp rx=0 tx=0).stop 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 shutdown_connections 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f71e806c4c0 0x7f71e806e980 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72100b2970 0x7f72101ed490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721015efa0 0x7f72101ed9d0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 --2- 172.21.15.43:0/1048859174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f721006da40 0x7f72101ecf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 >> 172.21.15.43:0/1048859174 conn(0x7f72100cec10 msgr2=0x7f72100ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 shutdown_connections 2024-01-16T15:27:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.400+0000 7f7215200640 1 -- 172.21.15.43:0/1048859174 wait complete. 2024-01-16T15:27:19.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.645+0000 7f7d96a00640 1 Processor -- start 2024-01-16T15:27:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.645+0000 7f7d96a00640 1 -- start start 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d96a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7d900ae270 0x7f7d900abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d96a00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9015b730 con 0x7f7d9009cb30 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d96a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d900ac740 con 0x7f7d9015b350 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d96a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d900ac460 con 0x7f7d900ae270 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d94a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7d900ae270 0x7f7d900abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d94a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7d900ae270 0x7f7d900abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54392/0 (socket says 172.21.15.43:54392) 2024-01-16T15:27:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.647+0000 7f7d94a00640 1 -- 172.21.15.43:0/3580315710 learned_addr learned my addr 172.21.15.43:0/3580315710 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.648+0000 7f7d94a00640 1 -- 172.21.15.43:0/3580315710 >> v1:172.21.15.69:6789/0 conn(0x7f7d9015b350 legacy=0x7f7d900b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.648+0000 7f7d94a00640 1 -- 172.21.15.43:0/3580315710 >> v1:172.21.15.43:6789/0 conn(0x7f7d9009cb30 legacy=0x7f7d900adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.648+0000 7f7d94a00640 1 -- 172.21.15.43:0/3580315710 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d900b2a50 con 0x7f7d900ae270 2024-01-16T15:27:19.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.649+0000 7f7d94a00640 1 --2- 172.21.15.43:0/3580315710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d900abe80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009870 tx=0x7f7d8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4962b69aba2e322 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d8f400640 1 -- 172.21.15.43:0/3580315710 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88030860 con 0x7f7d900ae270 2024-01-16T15:27:19.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d8f400640 1 -- 172.21.15.43:0/3580315710 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d880309c0 con 0x7f7d900ae270 2024-01-16T15:27:19.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d8f400640 1 -- 172.21.15.43:0/3580315710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88030d50 con 0x7f7d900ae270 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 -- 172.21.15.43:0/3580315710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 msgr2=0x7f7d900abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 --2- 172.21.15.43:0/3580315710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d900abe80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009870 tx=0x7f7d8802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 -- 172.21.15.43:0/3580315710 shutdown_connections 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 --2- 172.21.15.43:0/3580315710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d900abe80 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 -- 172.21.15.43:0/3580315710 >> 172.21.15.43:0/3580315710 conn(0x7f7d900b0ba0 msgr2=0x7f7d900b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 -- 172.21.15.43:0/3580315710 shutdown_connections 2024-01-16T15:27:19.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.650+0000 7f7d96a00640 1 -- 172.21.15.43:0/3580315710 wait complete. 2024-01-16T15:27:19.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.652+0000 7f7d96a00640 1 Processor -- start 2024-01-16T15:27:19.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.652+0000 7f7d96a00640 1 -- start start 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9009cb30 0x7f7d90071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9015b350 0x7f7d90158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900712b0 con 0x7f7d900ae270 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900716d0 con 0x7f7d9009cb30 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.653+0000 7f7d96a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90070630 con 0x7f7d9015b350 2024-01-16T15:27:19.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54394/0 (socket says 172.21.15.43:54394) 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 -- 172.21.15.43:0/2797807422 learned_addr learned my addr 172.21.15.43:0/2797807422 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d94a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9015b350 0x7f7d90158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d8fe00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9009cb30 0x7f7d90071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 -- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9015b350 msgr2=0x7f7d90158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9015b350 0x7f7d90158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 -- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9009cb30 msgr2=0x7f7d90071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9009cb30 0x7f7d90071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.654+0000 7f7d87200640 1 -- 172.21.15.43:0/2797807422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880094d0 con 0x7f7d900ae270 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d87200640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800c450 tx=0x7f7d7800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d780052d0 con 0x7f7d900ae270 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78005430 con 0x7f7d900ae270 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78012870 con 0x7f7d900ae270 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d90070910 con 0x7f7d900ae270 2024-01-16T15:27:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.655+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d90070ed0 con 0x7f7d900ae270 2024-01-16T15:27:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.656+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900713f0 con 0x7f7d900ae270 2024-01-16T15:27:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.657+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7d780129d0 con 0x7f7d900ae270 2024-01-16T15:27:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.658+0000 7f7d8cc00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 0x7f7d6806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.658+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7d78088490 con 0x7f7d900ae270 2024-01-16T15:27:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.658+0000 7f7d8fe00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 0x7f7d6806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.659+0000 7f7d8fe00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 0x7f7d6806e9b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009f20 tx=0x7f7d800094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:19.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.663+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d900713f0 con 0x7f7d900ae270 2024-01-16T15:27:19.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.857+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d9015b730 con 0x7f7d900ae270 2024-01-16T15:27:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.858+0000 7f7d8cc00640 1 -- 172.21.15.43:0/2797807422 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d9015b730 con 0x7f7d900ae270 2024-01-16T15:27:19.857 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:27:19.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 msgr2=0x7f7d6806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 0x7f7d6806e9b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009f20 tx=0x7f7d800094d0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 msgr2=0x7f7d901585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800c450 tx=0x7f7d7800c910 comp rx=0 tx=0).stop 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 shutdown_connections 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f7d6806c4f0 0x7f7d6806e9b0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d9015b350 0x7f7d90158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d900ae270 0x7f7d901585a0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 --2- 172.21.15.43:0/2797807422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d9009cb30 0x7f7d90071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:19.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 >> 172.21.15.43:0/2797807422 conn(0x7f7d900b0ba0 msgr2=0x7f7d9009bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:19.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 shutdown_connections 2024-01-16T15:27:19.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:19.861+0000 7f7d96a00640 1 -- 172.21.15.43:0/2797807422 wait complete. 2024-01-16T15:27:19.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:20.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.104+0000 7fcbc8a00640 1 Processor -- start 2024-01-16T15:27:20.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.104+0000 7fcbc8a00640 1 -- start start 2024-01-16T15:27:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcbc4153ea0 0x7fcbc40ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcbc4154280 0x7fcbc415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc8a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40ce650 con 0x7fcbc4070850 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc8a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc4154710 con 0x7fcbc4154280 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc415ddd0 con 0x7fcbc4153ea0 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.106+0000 7fcbc1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcbc4154280 0x7fcbc415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcbc4154280 0x7fcbc415d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54408/0 (socket says 172.21.15.43:54408) 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 -- 172.21.15.43:0/759648845 learned_addr learned my addr 172.21.15.43:0/759648845 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc2800640 1 --2- 172.21.15.43:0/759648845 >> v2:172.21.15.69:3300/0 conn(0x7fcbc4153ea0 0x7fcbc40ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 -- 172.21.15.43:0/759648845 >> v2:172.21.15.69:3300/0 conn(0x7fcbc4153ea0 msgr2=0x7fcbc40ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 --2- 172.21.15.43:0/759648845 >> v2:172.21.15.69:3300/0 conn(0x7fcbc4153ea0 0x7fcbc40ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 -- 172.21.15.43:0/759648845 >> v1:172.21.15.69:6789/0 conn(0x7fcbc4070850 legacy=0x7fcbc4071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.107+0000 7fcbc1e00640 1 -- 172.21.15.43:0/759648845 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc415e0a0 con 0x7fcbc4154280 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.108+0000 7fcbc1e00640 1 --2- 172.21.15.43:0/759648845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4154280 0x7fcbc415d890 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40cfc10 tx=0x7fcbac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17490e091d5058a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc1400640 1 -- 172.21.15.43:0/759648845 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac031870 con 0x7fcbc4154280 2024-01-16T15:27:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc1400640 1 -- 172.21.15.43:0/759648845 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac0319d0 con 0x7fcbc4154280 2024-01-16T15:27:20.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc1400640 1 -- 172.21.15.43:0/759648845 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac031d60 con 0x7fcbc4154280 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 -- 172.21.15.43:0/759648845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4154280 msgr2=0x7fcbc415d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/759648845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4154280 0x7fcbc415d890 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40cfc10 tx=0x7fcbac02b910 comp rx=0 tx=0).stop 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 -- 172.21.15.43:0/759648845 shutdown_connections 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/759648845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4154280 0x7fcbc415d890 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/759648845 >> v2:172.21.15.69:3300/0 conn(0x7fcbc4153ea0 0x7fcbc40ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 -- 172.21.15.43:0/759648845 >> 172.21.15.43:0/759648845 conn(0x7fcbc40cec10 msgr2=0x7fcbc40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 -- 172.21.15.43:0/759648845 shutdown_connections 2024-01-16T15:27:20.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.109+0000 7fcbc8a00640 1 -- 172.21.15.43:0/759648845 wait complete. 2024-01-16T15:27:20.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.111+0000 7fcbc8a00640 1 Processor -- start 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.111+0000 7fcbc8a00640 1 -- start start 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc4153ea0 0x7fcbc4154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc4154280 0x7fcbc41570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4143ba0 con 0x7fcbc4070850 2024-01-16T15:27:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41561d0 con 0x7fcbc4154280 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc8a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4155130 con 0x7fcbc4153ea0 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54416/0 (socket says 172.21.15.43:54416) 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.112+0000 7fcbc2800640 1 -- 172.21.15.43:0/3801259126 learned_addr learned my addr 172.21.15.43:0/3801259126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 -- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc4153ea0 msgr2=0x7fcbc4154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc4153ea0 0x7fcbc4154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 -- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc4154280 msgr2=0x7fcbc41570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc4154280 0x7fcbc41570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 -- 172.21.15.43:0/3801259126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0094d0 con 0x7fcbc4070850 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc2800640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400d800 tx=0x7fcbb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400b6c0 con 0x7fcbc4070850 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400b820 con 0x7fcbc4070850 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400bbb0 con 0x7fcbc4070850 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.113+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4155410 con 0x7fcbc4070850 2024-01-16T15:27:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.114+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc41559d0 con 0x7fcbc4070850 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.115+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc404e840 con 0x7fcbc4070850 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.116+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fcbb400bd10 con 0x7fcbc4070850 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.117+0000 7fcbbaa00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 0x7fcb9c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.117+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcbb4088d10 con 0x7fcbc4070850 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.117+0000 7fcbc1e00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 0x7fcb9c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.118+0000 7fcbc1e00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 0x7fcb9c06e9b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcbac036000 tx=0x7fcbac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.122+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbb40555d0 con 0x7fcbc4070850 2024-01-16T15:27:20.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.310+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcbc40cf370 con 0x7fcbc4070850 2024-01-16T15:27:20.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.311+0000 7fcbbaa00640 1 -- 172.21.15.43:0/3801259126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbb4058bf0 con 0x7fcbc4070850 2024-01-16T15:27:20.310 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:27:20.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 msgr2=0x7fcb9c06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 0x7fcb9c06e9b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcbac036000 tx=0x7fcbac030000 comp rx=0 tx=0).stop 2024-01-16T15:27:20.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 msgr2=0x7fcbc4143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400d800 tx=0x7fcbb400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 shutdown_connections 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fcb9c06c4f0 0x7fcb9c06e9b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc4154280 0x7fcbc41570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc4153ea0 0x7fcbc4154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 --2- 172.21.15.43:0/3801259126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbc4070850 0x7fcbc4143660 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 >> 172.21.15.43:0/3801259126 conn(0x7fcbc40cec10 msgr2=0x7fcbc4160180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 shutdown_connections 2024-01-16T15:27:20.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.314+0000 7fcbc8a00640 1 -- 172.21.15.43:0/3801259126 wait complete. 2024-01-16T15:27:20.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:20.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.544+0000 7f15bf200640 1 Processor -- start 2024-01-16T15:27:20.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.545+0000 7f15bf200640 1 -- start start 2024-01-16T15:27:20.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.546+0000 7f15bf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15c0070550 0x7f15c0160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.546+0000 7f15bf200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c0161940 con 0x7f15c0161530 2024-01-16T15:27:20.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.546+0000 7f15bf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c00b3040 con 0x7f15c00b2550 2024-01-16T15:27:20.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.546+0000 7f15bf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c0070cf0 con 0x7f15c0070550 2024-01-16T15:27:20.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15be800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f15c0161530 0x7f15c00ac900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59306/0 (socket says 172.21.15.43:59306) 2024-01-16T15:27:20.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15be800640 1 -- 172.21.15.43:0/3935802490 learned_addr learned my addr 172.21.15.43:0/3935802490 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:20.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15b7e00640 1 --2- 172.21.15.43:0/3935802490 >> v2:172.21.15.69:3300/0 conn(0x7f15c0070550 0x7f15c0160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899833336 0 0) 0x7f15c0161940 con 0x7f15c0161530 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0003520 con 0x7f15c0161530 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2323945210 0 0) 0x7f15a0003520 con 0x7f15c0161530 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 >> v2:172.21.15.69:3300/0 conn(0x7f15c0070550 msgr2=0x7f15c0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.547+0000 7f15bd400640 1 --2- 172.21.15.43:0/3935802490 >> v2:172.21.15.69:3300/0 conn(0x7f15c0070550 0x7f15c0160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 >> v1:172.21.15.69:6789/0 conn(0x7f15c00b2550 legacy=0x7f15c00b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0070fd0 con 0x7f15c0161530 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f15b80019f0 con 0x7f15c0161530 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 >> v1:172.21.15.43:6789/0 conn(0x7f15c0161530 legacy=0x7f15c00ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a00039e0 0x7f15a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00051f0 0x7f15a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0070cf0 con 0x7f15a00047c0 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000c150 con 0x7f15a00051f0 2024-01-16T15:27:20.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000c310 con 0x7f15a00039e0 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15bde00640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a00039e0 0x7f15a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 -- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a00039e0 msgr2=0x7f15a0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a00039e0 0x7f15a0003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 -- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00051f0 msgr2=0x7f15a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00051f0 0x7f15a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15b7e00640 1 -- 172.21.15.43:0/3935802490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a0003520 con 0x7f15a00047c0 2024-01-16T15:27:20.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.548+0000 7f15be800640 1 -- 172.21.15.43:0/3935802490 reap_dead start 2024-01-16T15:27:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.549+0000 7f15b7e00640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f15a800ba30 tx=0x7f15a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d560396e6b3790cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8032640 con 0x7f15a00047c0 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a80327a0 con 0x7f15a00047c0 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bd400640 1 -- 172.21.15.43:0/3935802490 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8032b30 con 0x7f15a00047c0 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bf200640 1 -- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 msgr2=0x7f15a0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bf200640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f15a800ba30 tx=0x7f15a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bf200640 1 -- 172.21.15.43:0/3935802490 shutdown_connections 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bf200640 1 --2- 172.21.15.43:0/3935802490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.550+0000 7f15bf200640 1 -- 172.21.15.43:0/3935802490 >> 172.21.15.43:0/3935802490 conn(0x7f15c00b0840 msgr2=0x7f15c00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:20.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.551+0000 7f15bf200640 1 -- 172.21.15.43:0/3935802490 shutdown_connections 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.551+0000 7f15bf200640 1 -- 172.21.15.43:0/3935802490 wait complete. 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.552+0000 7f15bf200640 1 Processor -- start 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.553+0000 7f15bf200640 1 -- start start 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.553+0000 7f15bf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 0x7f15c014e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.553+0000 7f15bf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.553+0000 7f15bf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15c00ad8c0 0x7f15c00add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.553+0000 7f15bf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01580c0 con 0x7f15c0159cb0 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15bf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c014e5b0 con 0x7f15a00047c0 2024-01-16T15:27:20.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15bf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00ae380 con 0x7f15c00ad8c0 2024-01-16T15:27:20.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54440/0 (socket says 172.21.15.43:54440) 2024-01-16T15:27:20.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 -- 172.21.15.43:0/565275224 learned_addr learned my addr 172.21.15.43:0/565275224 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15be800640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15c00ad8c0 0x7f15c00add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15bde00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 0x7f15c014e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 -- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15c00ad8c0 msgr2=0x7f15c00add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15c00ad8c0 0x7f15c00add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 -- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 msgr2=0x7f15c014e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 0x7f15c014e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15b7e00640 1 -- 172.21.15.43:0/565275224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00ae660 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.554+0000 7f15bde00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 0x7f15c014e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15b7e00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f15a802bec0 tx=0x7f15a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a803b9f0 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a803bb50 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c0158310 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8030410 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.555+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c00af6e0 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.556+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c00ae7f0 con 0x7f15c0159cb0 2024-01-16T15:27:20.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.557+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f15a8030570 con 0x7f15c0159cb0 2024-01-16T15:27:20.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.558+0000 7f15b6000640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 0x7f159806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.558+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f15a80ab3e0 con 0x7f15c0159cb0 2024-01-16T15:27:20.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.558+0000 7f15bde00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 0x7f159806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.559+0000 7f15bde00640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 0x7f159806e980 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f15c00691e0 tx=0x7f15b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.562+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15a80778b0 con 0x7f15c0159cb0 2024-01-16T15:27:20.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.769+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f15c004e840 con 0x7f15c0159cb0 2024-01-16T15:27:20.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.770+0000 7f15b6000640 1 -- 172.21.15.43:0/565275224 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v115) v1 ==== 80+0+0 (secure 0 0 0) 0x7f15a8036080 con 0x7f15c0159cb0 2024-01-16T15:27:20.769 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:27:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 msgr2=0x7f159806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 0x7f159806e980 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f15c00691e0 tx=0x7f15b0009040 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 msgr2=0x7f15c015a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f15a802bec0 tx=0x7f15a8004730 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 shutdown_connections 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f159806c4c0 0x7f159806e980 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15c00ad8c0 0x7f15c00add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15c0159cb0 0x7f15c015a110 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 --2- 172.21.15.43:0/565275224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a00047c0 0x7f15c014e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 >> 172.21.15.43:0/565275224 conn(0x7f15c00b0840 msgr2=0x7f15c015cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:20.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 shutdown_connections 2024-01-16T15:27:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:20.773+0000 7f15bf200640 1 -- 172.21.15.43:0/565275224 wait complete. 2024-01-16T15:27:20.791 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:27:20.791 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:20.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:21.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.077+0000 7f5a77e00640 1 Processor -- start 2024-01-16T15:27:21.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.077+0000 7f5a77e00640 1 -- start start 2024-01-16T15:27:21.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.078+0000 7f5a77e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 0x7f5a780d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.078+0000 7f5a77e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5a780b0640 0x7f5a780b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.078+0000 7f5a77e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a780cf0c0 con 0x7f5a7815d5b0 2024-01-16T15:27:21.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.078+0000 7f5a77e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780b01b0 con 0x7f5a780b0640 2024-01-16T15:27:21.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.078+0000 7f5a77e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7815d990 con 0x7f5a780d14a0 2024-01-16T15:27:21.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.079+0000 7f5a77400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5a780b0640 0x7f5a780b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.079+0000 7f5a76a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5a7815d5b0 0x7f5a780d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.43:42858/0 (socket says 172.21.15.43:42858) 2024-01-16T15:27:21.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.079+0000 7f5a76a00640 1 -- 172.21.15.43:0/577204286 learned_addr learned my addr 172.21.15.43:0/577204286 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:21.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.080+0000 7f5a77400640 1 -- 172.21.15.43:0/577204286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 msgr2=0x7f5a780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.079+0000 7f5a76000640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 0x7f5a780d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.080+0000 7f5a77400640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 0x7f5a780d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.080+0000 7f5a77400640 1 -- 172.21.15.43:0/577204286 >> v1:172.21.15.69:6789/0 conn(0x7f5a7815d5b0 legacy=0x7f5a780d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.080+0000 7f5a77400640 1 -- 172.21.15.43:0/577204286 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780d1e40 con 0x7f5a780b0640 2024-01-16T15:27:21.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.080+0000 7f5a76000640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 0x7f5a780d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.081+0000 7f5a77400640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780b0640 0x7f5a780b0aa0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009d90 tx=0x7f5a7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e7455c8d59b08a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.081+0000 7f5a75600640 1 -- 172.21.15.43:0/577204286 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7002cc50 con 0x7f5a780b0640 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.081+0000 7f5a75600640 1 -- 172.21.15.43:0/577204286 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7002cdb0 con 0x7f5a780b0640 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.081+0000 7f5a75600640 1 -- 172.21.15.43:0/577204286 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a70034660 con 0x7f5a780b0640 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 -- 172.21.15.43:0/577204286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780b0640 msgr2=0x7f5a780b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780b0640 0x7f5a780b0aa0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009d90 tx=0x7f5a7002bea0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 -- 172.21.15.43:0/577204286 shutdown_connections 2024-01-16T15:27:21.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780b0640 0x7f5a780b0aa0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 --2- 172.21.15.43:0/577204286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780d14a0 0x7f5a780d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 -- 172.21.15.43:0/577204286 >> 172.21.15.43:0/577204286 conn(0x7f5a7809c3b0 msgr2=0x7f5a780d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:21.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 -- 172.21.15.43:0/577204286 shutdown_connections 2024-01-16T15:27:21.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.082+0000 7f5a77e00640 1 -- 172.21.15.43:0/577204286 wait complete. 2024-01-16T15:27:21.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.084+0000 7f5a77e00640 1 Processor -- start 2024-01-16T15:27:21.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.084+0000 7f5a77e00640 1 -- start start 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780b0640 0x7f5a78152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a7815d5b0 0x7f5a780c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7815f1e0 con 0x7f5a780d14a0 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780c9590 con 0x7f5a7815d5b0 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a77e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7815ebb0 con 0x7f5a780b0640 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a76000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a76000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54456/0 (socket says 172.21.15.43:54456) 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.085+0000 7f5a76000640 1 -- 172.21.15.43:0/4044087652 learned_addr learned my addr 172.21.15.43:0/4044087652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a77400640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a7815d5b0 0x7f5a780c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76a00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780b0640 0x7f5a78152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 -- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780b0640 msgr2=0x7f5a78152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780b0640 0x7f5a78152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 -- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a7815d5b0 msgr2=0x7f5a780c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a7815d5b0 0x7f5a780c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 -- 172.21.15.43:0/4044087652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a70009a70 con 0x7f5a780d14a0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a76000640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c820 tx=0x7f5a6000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60007a30 con 0x7f5a780d14a0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60007b90 con 0x7f5a780d14a0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000ce10 con 0x7f5a780d14a0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.086+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a78152b40 con 0x7f5a780d14a0 2024-01-16T15:27:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.087+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a780c92b0 con 0x7f5a780d14a0 2024-01-16T15:27:21.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.088+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7804e840 con 0x7f5a780d14a0 2024-01-16T15:27:21.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.089+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f5a600113f0 con 0x7f5a780d14a0 2024-01-16T15:27:21.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.089+0000 7f5a6ea00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 0x7f5a5006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.090+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5a60088c40 con 0x7f5a780d14a0 2024-01-16T15:27:21.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.090+0000 7f5a76a00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 0x7f5a5006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.091+0000 7f5a76a00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 0x7f5a5006e980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5a68005880 tx=0x7f5a68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.094+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a60055190 con 0x7f5a780d14a0 2024-01-16T15:27:21.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.287+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a7815fb40 con 0x7f5a780d14a0 2024-01-16T15:27:21.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.289+0000 7f5a6ea00640 1 -- 172.21.15.43:0/4044087652 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f5a6001f090 con 0x7f5a780d14a0 2024-01-16T15:27:21.289 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:21.289 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":115,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:19:15.246810+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 msgr2=0x7f5a5006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 0x7f5a5006e980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5a68005880 tx=0x7f5a68009040 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 msgr2=0x7f5a7815f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c820 tx=0x7f5a6000cce0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 shutdown_connections 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f5a5006c4c0 0x7f5a5006e980 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a7815d5b0 0x7f5a780c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a780d14a0 0x7f5a7815f600 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 --2- 172.21.15.43:0/4044087652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a780b0640 0x7f5a78152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 >> 172.21.15.43:0/4044087652 conn(0x7f5a7809c3b0 msgr2=0x7f5a7806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 shutdown_connections 2024-01-16T15:27:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.293+0000 7f5a77e00640 1 -- 172.21.15.43:0/4044087652 wait complete. 2024-01-16T15:27:21.311 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:20.855872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:18:47.672212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:18:48.980104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:21.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:21.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.533+0000 7f2ecfe00640 1 Processor -- start 2024-01-16T15:27:21.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.533+0000 7f2ecfe00640 1 -- start start 2024-01-16T15:27:21.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.534+0000 7f2ecfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ed015ac80 0x7f2ed0071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 0x7f2ed015dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecfe00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed0071670 con 0x7f2ed015d7a0 2024-01-16T15:27:21.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed00710f0 con 0x7f2ed015ac80 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed015e240 con 0x7f2ed006ab50 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ece000640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2ed015d7a0 0x7f2ed006a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59310/0 (socket says 172.21.15.43:59310) 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ece000640 1 -- 172.21.15.43:0/1791912793 learned_addr learned my addr 172.21.15.43:0/1791912793 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecea00640 1 --2- 172.21.15.43:0/1791912793 >> v2:172.21.15.69:3300/0 conn(0x7f2ed015ac80 0x7f2ed0071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecf400640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 0x7f2ed015dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.535+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284343692 0 0) 0x7f2ed0071670 con 0x7f2ed015d7a0 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb0003520 con 0x7f2ed015d7a0 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 524106290 0 0) 0x7f2eb0003520 con 0x7f2ed015d7a0 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 msgr2=0x7f2ed015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 0x7f2ed015dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 >> v2:172.21.15.69:3300/0 conn(0x7f2ed015ac80 msgr2=0x7f2ed0071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 --2- 172.21.15.43:0/1791912793 >> v2:172.21.15.69:3300/0 conn(0x7f2ed015ac80 0x7f2ed0071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed015e520 con 0x7f2ed015d7a0 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f2eb8001670 con 0x7f2ed015d7a0 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 >> v1:172.21.15.43:6789/0 conn(0x7f2ed015d7a0 legacy=0x7f2ed006a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 0x7f2eb00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0003f60 0x7f2eb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2eb0004d00 0x7f2eb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00710f0 con 0x7f2eb0004d00 2024-01-16T15:27:21.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed015e240 con 0x7f2eb0003f60 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb000b9b0 con 0x7f2ed006ab50 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.536+0000 7f2ecf400640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2eb0004d00 0x7f2eb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.537+0000 7f2ecf400640 1 -- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 msgr2=0x7f2eb00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.537+0000 7f2ecf400640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ed006ab50 0x7f2eb00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.537+0000 7f2ecf400640 1 -- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0003f60 msgr2=0x7f2eb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.537+0000 7f2ecf400640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0003f60 0x7f2eb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.537+0000 7f2ecf400640 1 -- 172.21.15.43:0/1791912793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0003520 con 0x7f2eb0004d00 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.538+0000 7f2ecf400640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2eb0004d00 0x7f2eb0005180 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00acf70 tx=0x7f2eb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d37c7f03d71a5b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.538+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec802cb90 con 0x7f2eb0004d00 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.538+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec800f040 con 0x7f2eb0004d00 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.538+0000 7f2ecd600640 1 -- 172.21.15.43:0/1791912793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8034440 con 0x7f2eb0004d00 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 -- 172.21.15.43:0/1791912793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2eb0004d00 msgr2=0x7f2eb0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/1791912793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2eb0004d00 0x7f2eb0005180 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00acf70 tx=0x7f2eb00057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecf400640 1 -- 172.21.15.43:0/1791912793 reap_dead start 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 -- 172.21.15.43:0/1791912793 shutdown_connections 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 -- 172.21.15.43:0/1791912793 >> 172.21.15.43:0/1791912793 conn(0x7f2ed00cfde0 msgr2=0x7f2ed00d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 -- 172.21.15.43:0/1791912793 shutdown_connections 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.539+0000 7f2ecfe00640 1 -- 172.21.15.43:0/1791912793 wait complete. 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.541+0000 7f2ecfe00640 1 Processor -- start 2024-01-16T15:27:21.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.542+0000 7f2ecfe00640 1 -- start start 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb0004d00 0x7f2ed0144780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ed00c4e00 0x7f2ed00c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed015d140 con 0x7f2ed015c780 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed015e900 con 0x7f2ed00c4e00 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecfe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed015ebe0 con 0x7f2eb0004d00 2024-01-16T15:27:21.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54468/0 (socket says 172.21.15.43:54468) 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 -- 172.21.15.43:0/458553944 learned_addr learned my addr 172.21.15.43:0/458553944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ecea00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb0004d00 0x7f2ed0144780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 -- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb0004d00 msgr2=0x7f2ed0144780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb0004d00 0x7f2ed0144780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.543+0000 7f2ece000640 1 -- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ed00c4e00 msgr2=0x7f2ed00c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ecf400640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ed00c4e00 0x7f2ed00c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ece000640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ed00c4e00 0x7f2ed00c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ece000640 1 -- 172.21.15.43:0/458553944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed00721c0 con 0x7f2ed015c780 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ece000640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c700 tx=0x7f2eb800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80074e0 con 0x7f2ed015c780 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb8007640 con 0x7f2ed015c780 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8004a20 con 0x7f2ed015c780 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.544+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed015eec0 con 0x7f2ed015c780 2024-01-16T15:27:21.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.545+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed00c57c0 con 0x7f2ed015c780 2024-01-16T15:27:21.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.546+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed004e840 con 0x7f2ed015c780 2024-01-16T15:27:21.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.547+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2eb8004b80 con 0x7f2ed015c780 2024-01-16T15:27:21.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.548+0000 7f2ec6a00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 0x7f2ea806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.548+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2eb8087f60 con 0x7f2ed015c780 2024-01-16T15:27:21.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.548+0000 7f2ecea00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 0x7f2ea806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.549+0000 7f2ecea00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 0x7f2ea806e980 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0004770 tx=0x7f2ec0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.552+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2eb8054510 con 0x7f2ed015c780 2024-01-16T15:27:21.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.746+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ed006d6d0 con 0x7f2ed015c780 2024-01-16T15:27:21.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.747+0000 7f2ec6a00640 1 -- 172.21.15.43:0/458553944 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2eb8018030 con 0x7f2ed015c780 2024-01-16T15:27:21.746 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-01-16T15:27:21.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 msgr2=0x7f2ea806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 0x7f2ea806e980 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0004770 tx=0x7f2ec0008040 comp rx=0 tx=0).stop 2024-01-16T15:27:21.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 msgr2=0x7f2ed015cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c700 tx=0x7f2eb800cbc0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 shutdown_connections 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f2ea806c4c0 0x7f2ea806e980 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ed00c4e00 0x7f2ed00c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ed015c780 0x7f2ed015cc00 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 --2- 172.21.15.43:0/458553944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb0004d00 0x7f2ed0144780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 >> 172.21.15.43:0/458553944 conn(0x7f2ed00cfde0 msgr2=0x7f2ed006ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:21.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 shutdown_connections 2024-01-16T15:27:21.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.750+0000 7f2ecfe00640 1 -- 172.21.15.43:0/458553944 wait complete. 2024-01-16T15:27:21.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.988+0000 7ff4b3200640 1 Processor -- start 2024-01-16T15:27:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.988+0000 7ff4b3200640 1 -- start start 2024-01-16T15:27:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 0x7ff4b4071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b3200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4070d40 con 0x7ff4b40b4140 2024-01-16T15:27:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4071020 con 0x7ff4b41439b0 2024-01-16T15:27:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4071300 con 0x7ff4b41435d0 2024-01-16T15:27:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b2800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff4b40b4140 0x7ff4b4070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59314/0 (socket says 172.21.15.43:59314) 2024-01-16T15:27:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.990+0000 7ff4b2800640 1 -- 172.21.15.43:0/3906610652 learned_addr learned my addr 172.21.15.43:0/3906610652 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1e00640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 0x7ff4b4071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107470046 0 0) 0x7ff4b4070d40 con 0x7ff4b40b4140 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff494005730 con 0x7ff4b40b4140 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052620722 0 0) 0x7ff494005730 con 0x7ff4b40b4140 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 msgr2=0x7ff4b4071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 0x7ff4b4071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 >> v1:172.21.15.69:6789/0 conn(0x7ff4b41439b0 legacy=0x7ff4b40b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40715e0 con 0x7ff4b40b4140 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.991+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7ff4ac0019f0 con 0x7ff4b40b4140 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 >> v1:172.21.15.43:6789/0 conn(0x7ff4b40b4140 legacy=0x7ff4b4070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff494005c20 0x7ff494006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 0x7ff4940069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff494007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4071300 con 0x7ff494006f10 2024-01-16T15:27:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49400dd20 con 0x7ff494005c20 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49400deb0 con 0x7ff4b41435d0 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff494007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b1e00640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff494005c20 0x7ff494006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 -- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 msgr2=0x7ff4940069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b41435d0 0x7ff4940069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 -- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff494005c20 msgr2=0x7ff494006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff494005c20 0x7ff494006040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.992+0000 7ff4b2800640 1 -- 172.21.15.43:0/3906610652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494005730 con 0x7ff494006f10 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.993+0000 7ff4b2800640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff494007390 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415db80 tx=0x7ff494007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfea3e7657af83cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac034630 con 0x7ff494006f10 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac02f070 con 0x7ff494006f10 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b1400640 1 -- 172.21.15.43:0/3906610652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac034a10 con 0x7ff494006f10 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b3200640 1 -- 172.21.15.43:0/3906610652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 msgr2=0x7ff494007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b3200640 1 --2- 172.21.15.43:0/3906610652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff494007390 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415db80 tx=0x7ff494007a00 comp rx=0 tx=0).stop 2024-01-16T15:27:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b2800640 1 -- 172.21.15.43:0/3906610652 reap_dead start 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b3200640 1 -- 172.21.15.43:0/3906610652 shutdown_connections 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.994+0000 7ff4b3200640 1 -- 172.21.15.43:0/3906610652 >> 172.21.15.43:0/3906610652 conn(0x7ff4b40cec10 msgr2=0x7ff4b40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.995+0000 7ff4b3200640 1 -- 172.21.15.43:0/3906610652 shutdown_connections 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.995+0000 7ff4b3200640 1 -- 172.21.15.43:0/3906610652 wait complete. 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.996+0000 7ff4b3200640 1 Processor -- start 2024-01-16T15:27:21.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.997+0000 7ff4b3200640 1 -- start start 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4b4142810 0x7ff4b40c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 0x7ff4b41f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40c6030 con 0x7ff494006f10 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4151e40 con 0x7ff4b4142810 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b413ec00 con 0x7ff4b40c5bb0 2024-01-16T15:27:21.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54484/0 (socket says 172.21.15.43:54484) 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 -- 172.21.15.43:0/2947609307 learned_addr learned my addr 172.21.15.43:0/2947609307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b2800640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 0x7ff4b41f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 -- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 msgr2=0x7ff4b41f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 0x7ff4b41f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 -- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4b4142810 msgr2=0x7ff4b40c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4b4142810 0x7ff4b40c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b1e00640 1 -- 172.21.15.43:0/2947609307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4150f50 con 0x7ff494006f10 2024-01-16T15:27:22.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.998+0000 7ff4b2800640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 0x7ff4b41f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4b1e00640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7ff4a40108f0 tx=0x7ff4a4010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a400eb00 con 0x7ff494006f10 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a400ec60 con 0x7ff494006f10 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b4151230 con 0x7ff494006f10 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a401c400 con 0x7ff494006f10 2024-01-16T15:27:22.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:21.999+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b41517f0 con 0x7ff494006f10 2024-01-16T15:27:22.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.000+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b404e840 con 0x7ff494006f10 2024-01-16T15:27:22.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.001+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff4a401c560 con 0x7ff494006f10 2024-01-16T15:27:22.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.002+0000 7ff4aaa00640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 0x7ff48c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.003+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4a4088fc0 con 0x7ff494006f10 2024-01-16T15:27:22.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.003+0000 7ff4a9200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 0x7ff48c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.004+0000 7ff4a9200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 0x7ff48c06e9b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415cc10 tx=0x7ff49c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.007+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4a4055510 con 0x7ff494006f10 2024-01-16T15:27:22.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.200+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4b415e540 con 0x7ff494006f10 2024-01-16T15:27:22.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.201+0000 7ff4aaa00640 1 -- 172.21.15.43:0/2947609307 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4a4058b30 con 0x7ff494006f10 2024-01-16T15:27:22.200 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-01-16T15:27:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 msgr2=0x7ff48c06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 0x7ff48c06e9b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415cc10 tx=0x7ff49c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 msgr2=0x7ff4b413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7ff4a40108f0 tx=0x7ff4a4010db0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 shutdown_connections 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ff48c06c4f0 0x7ff48c06e9b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4b4142810 0x7ff4b40c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4b40c5bb0 0x7ff4b41f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 --2- 172.21.15.43:0/2947609307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff494006f10 0x7ff4b413e6c0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 >> 172.21.15.43:0/2947609307 conn(0x7ff4b40cec10 msgr2=0x7ff4b40cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 shutdown_connections 2024-01-16T15:27:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.204+0000 7ff4b3200640 1 -- 172.21.15.43:0/2947609307 wait complete. 2024-01-16T15:27:22.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:22.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.445+0000 7f29b1c00640 1 Processor -- start 2024-01-16T15:27:22.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.446+0000 7f29b1c00640 1 -- start start 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.447+0000 7f29b1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.447+0000 7f29b1c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29ac15d7a0 0x7f29ac15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.447+0000 7f29b1c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac071670 con 0x7f29ac06a440 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.447+0000 7f29b1c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0710f0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.447+0000 7f29b1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac15e240 con 0x7f29ac15ac80 2024-01-16T15:27:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29ac15d7a0 0x7f29ac15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f29ac15d7a0 0x7f29ac15dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54496/0 (socket says 172.21.15.43:54496) 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 -- 172.21.15.43:0/3105354253 learned_addr learned my addr 172.21.15.43:0/3105354253 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29abe00640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 -- 172.21.15.43:0/3105354253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 msgr2=0x7f29ac071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 -- 172.21.15.43:0/3105354253 >> v1:172.21.15.43:6789/0 conn(0x7f29ac06a440 legacy=0x7f29ac06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.448+0000 7f29a3200640 1 -- 172.21.15.43:0/3105354253 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac15e520 con 0x7f29ac15d7a0 2024-01-16T15:27:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.449+0000 7f29a3200640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac15dc00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f2994009870 tx=0x7f299402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e44362c8771c53e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29ab400640 1 -- 172.21.15.43:0/3105354253 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994030840 con 0x7f29ac15d7a0 2024-01-16T15:27:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29ab400640 1 -- 172.21.15.43:0/3105354253 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29940309a0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29ab400640 1 -- 172.21.15.43:0/3105354253 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994030d30 con 0x7f29ac15d7a0 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29b1c00640 1 -- 172.21.15.43:0/3105354253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 msgr2=0x7f29ac15dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29b1c00640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac15dc00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f2994009870 tx=0x7f299402b980 comp rx=0 tx=0).stop 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.450+0000 7f29b1c00640 1 -- 172.21.15.43:0/3105354253 shutdown_connections 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.451+0000 7f29b1c00640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac15dc00 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.451+0000 7f29b1c00640 1 --2- 172.21.15.43:0/3105354253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.451+0000 7f29b1c00640 1 -- 172.21.15.43:0/3105354253 >> 172.21.15.43:0/3105354253 conn(0x7f29ac0cfde0 msgr2=0x7f29ac0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.451+0000 7f29b1c00640 1 -- 172.21.15.43:0/3105354253 shutdown_connections 2024-01-16T15:27:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.451+0000 7f29b1c00640 1 -- 172.21.15.43:0/3105354253 wait complete. 2024-01-16T15:27:22.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.452+0000 7f29b1c00640 1 Processor -- start 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 -- start start 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29ac06a440 0x7f29ac0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac15f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac160a00 con 0x7f29ac15d7a0 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac15fd80 con 0x7f29ac06a440 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.453+0000 7f29b1c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac160060 con 0x7f29ac15ac80 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54504/0 (socket says 172.21.15.43:54504) 2024-01-16T15:27:22.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 -- 172.21.15.43:0/1242414807 learned_addr learned my addr 172.21.15.43:0/1242414807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29a3200640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac15f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29abe00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29ac06a440 0x7f29ac0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 -- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 msgr2=0x7f29ac15f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac15f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 -- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29ac06a440 msgr2=0x7f29ac0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29ac06a440 0x7f29ac0b11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29b1200640 1 -- 172.21.15.43:0/1242414807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29940094d0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.454+0000 7f29a3200640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac15f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29b1200640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f29a4001eb0 tx=0x7f29a400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a40052d0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a4005430 con 0x7f29ac15d7a0 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4003810 con 0x7f29ac15d7a0 2024-01-16T15:27:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac0721c0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.455+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac160620 con 0x7f29ac15d7a0 2024-01-16T15:27:22.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.456+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac160b40 con 0x7f29ac15d7a0 2024-01-16T15:27:22.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.457+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f29a4003970 con 0x7f29ac15d7a0 2024-01-16T15:27:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.458+0000 7f29a8c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 0x7f298406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.458+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f29a40880c0 con 0x7f29ac15d7a0 2024-01-16T15:27:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.458+0000 7f29abe00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 0x7f298406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.459+0000 7f29abe00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 0x7f298406e980 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f29ac06f030 tx=0x7f299c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.464+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29a4054670 con 0x7f29ac15d7a0 2024-01-16T15:27:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.667+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29ac04e840 con 0x7f29ac15d7a0 2024-01-16T15:27:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.668+0000 7f29a8c00640 1 -- 172.21.15.43:0/1242414807 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29a4017020 con 0x7f29ac15d7a0 2024-01-16T15:27:22.668 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-01-16T15:27:22.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 msgr2=0x7f298406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 0x7f298406e980 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f29ac06f030 tx=0x7f299c00b040 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 msgr2=0x7f29ac150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f29a4001eb0 tx=0x7f29a400c890 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 shutdown_connections 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f298406c4c0 0x7f298406e980 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29ac15d7a0 0x7f29ac150000 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29ac15ac80 0x7f29ac15f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 --2- 172.21.15.43:0/1242414807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29ac06a440 0x7f29ac0b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 >> 172.21.15.43:0/1242414807 conn(0x7f29ac0cfde0 msgr2=0x7f29ac0ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:22.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 shutdown_connections 2024-01-16T15:27:22.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.671+0000 7f29b1c00640 1 -- 172.21.15.43:0/1242414807 wait complete. 2024-01-16T15:27:22.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.910+0000 7fee6a800640 1 Processor -- start 2024-01-16T15:27:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.910+0000 7fee6a800640 1 -- start start 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee641411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fee6406e7b0 0x7fee6406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fee6415efc0 0x7fee6415f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6406f170 con 0x7fee6406e7b0 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee64141710 con 0x7fee6415efc0 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee6a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6415ba40 con 0x7fee641558c0 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.911+0000 7fee62a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fee6406e7b0 0x7fee6406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fee6406e7b0 0x7fee6406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54508/0 (socket says 172.21.15.43:54508) 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 -- 172.21.15.43:0/1588471792 learned_addr learned my addr 172.21.15.43:0/1588471792 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee63400640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee641411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 -- 172.21.15.43:0/1588471792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 msgr2=0x7fee641411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee641411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 -- 172.21.15.43:0/1588471792 >> v2:172.21.15.69:3300/0 conn(0x7fee6415efc0 msgr2=0x7fee6415f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 --2- 172.21.15.43:0/1588471792 >> v2:172.21.15.69:3300/0 conn(0x7fee6415efc0 0x7fee6415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.912+0000 7fee62a00640 1 -- 172.21.15.43:0/1588471792 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6415bd20 con 0x7fee6406e7b0 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.913+0000 7fee62a00640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee6406ec30 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fee640d1350 tx=0x7fee4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc91e3d929ce67d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee62000640 1 -- 172.21.15.43:0/1588471792 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c031870 con 0x7fee6406e7b0 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee62000640 1 -- 172.21.15.43:0/1588471792 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c0319d0 con 0x7fee6406e7b0 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee62000640 1 -- 172.21.15.43:0/1588471792 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c031d60 con 0x7fee6406e7b0 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 -- 172.21.15.43:0/1588471792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 msgr2=0x7fee6406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee6406ec30 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fee640d1350 tx=0x7fee4c02b910 comp rx=0 tx=0).stop 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 -- 172.21.15.43:0/1588471792 shutdown_connections 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 --2- 172.21.15.43:0/1588471792 >> v2:172.21.15.69:3300/0 conn(0x7fee6415efc0 0x7fee6415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee6406ec30 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 --2- 172.21.15.43:0/1588471792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee641411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 -- 172.21.15.43:0/1588471792 >> 172.21.15.43:0/1588471792 conn(0x7fee64070be0 msgr2=0x7fee6406a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 -- 172.21.15.43:0/1588471792 shutdown_connections 2024-01-16T15:27:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.914+0000 7fee6a800640 1 -- 172.21.15.43:0/1588471792 wait complete. 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.916+0000 7fee6a800640 1 Processor -- start 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.916+0000 7fee6a800640 1 -- start start 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee64140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 0x7fee6415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641561d0 con 0x7fee6406e7b0 2024-01-16T15:27:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee640b4f20 con 0x7fee6415efc0 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee6a800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee640b5200 con 0x7fee641558c0 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.917+0000 7fee63400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54516/0 (socket says 172.21.15.43:54516) 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 -- 172.21.15.43:0/1317282146 learned_addr learned my addr 172.21.15.43:0/1317282146 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63e00640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 0x7fee6415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 -- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 msgr2=0x7fee64140990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee64140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 -- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 msgr2=0x7fee6415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 0x7fee6415a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 -- 172.21.15.43:0/1317282146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0094d0 con 0x7fee6406e7b0 2024-01-16T15:27:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63e00640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 0x7fee6415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.918+0000 7fee63400640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fee5400d800 tx=0x7fee5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.919+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5400b6c0 con 0x7fee6406e7b0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.919+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5400b820 con 0x7fee6406e7b0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.919+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5400bbb0 con 0x7fee6406e7b0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.919+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee640b54e0 con 0x7fee6406e7b0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.919+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee640b5a00 con 0x7fee6406e7b0 2024-01-16T15:27:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.920+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee64074770 con 0x7fee6406e7b0 2024-01-16T15:27:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.921+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fee5400bd10 con 0x7fee6406e7b0 2024-01-16T15:27:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.923+0000 7fee5b400640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 0x7fee3c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.923+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fee54088d40 con 0x7fee6406e7b0 2024-01-16T15:27:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.923+0000 7fee62a00640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 0x7fee3c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.924+0000 7fee62a00640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 0x7fee3c06e980 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee4c036000 tx=0x7fee4c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:22.929+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee54055600 con 0x7fee6406e7b0 2024-01-16T15:27:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.135+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee640d1e00 con 0x7fee6406e7b0 2024-01-16T15:27:23.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.136+0000 7fee5b400640 1 -- 172.21.15.43:0/1317282146 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v115) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee54058c20 con 0x7fee6406e7b0 2024-01-16T15:27:23.135 INFO:teuthology.orchestra.run.smithi043.stderr:listed 0 entries 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 msgr2=0x7fee3c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 0x7fee3c06e980 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee4c036000 tx=0x7fee4c030000 comp rx=0 tx=0).stop 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 msgr2=0x7fee64153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fee5400d800 tx=0x7fee5400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 shutdown_connections 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fee3c06c4c0 0x7fee3c06e980 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee6415efc0 0x7fee6415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee641558c0 0x7fee64140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 --2- 172.21.15.43:0/1317282146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee6406e7b0 0x7fee64153670 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.140+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 >> 172.21.15.43:0/1317282146 conn(0x7fee64070be0 msgr2=0x7fee6406a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.141+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 shutdown_connections 2024-01-16T15:27:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:23.141+0000 7fee6a800640 1 -- 172.21.15.43:0/1317282146 wait complete. 2024-01-16T15:27:23.160 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-01-16T15:27:23.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:27:23.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[47352]: fuse finished with error 0 and tester_r 0 2024-01-16T15:27:23.257 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:27:23.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:27:23.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.325 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:27:23.380 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-16T15:27:23.381 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:23.430 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:27:23.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.431 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-01-16T15:27:23.483 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:27:23.483 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:27:23.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:27:23.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.568 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:27:23.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-01-16T15:27:23.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.783 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-01-16T15:27:23.841 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:27:23.841 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:27:23.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:27:23.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:23.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:27:24.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:24.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-01-16T15:27:24.195 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T15:27:24.205 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T15:27:24.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:27:24.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.407+0000 7f9ba2c00640 1 Processor -- start 2024-01-16T15:27:24.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.407+0000 7f9ba2c00640 1 -- start start 2024-01-16T15:27:24.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba2c00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9b9c157770 0x7f9b9c06cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba2c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 0x7f9b9c070720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba2c00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c161670 con 0x7f9b9c06d080 2024-01-16T15:27:24.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba2c00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c15e410 con 0x7f9b9c157770 2024-01-16T15:27:24.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba2c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c06d500 con 0x7f9b9c15df80 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.409+0000 7f9ba1800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9b9c157770 0x7f9b9c06cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9b9c157770 0x7f9b9c06cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53794/0 (socket says 172.21.15.43:53794) 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 -- 172.21.15.43:0/1810046156 learned_addr learned my addr 172.21.15.43:0/1810046156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba2200640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 0x7f9b9c070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 -- 172.21.15.43:0/1810046156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 msgr2=0x7f9b9c070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 0x7f9b9c070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 -- 172.21.15.43:0/1810046156 >> v1:172.21.15.43:6789/0 conn(0x7f9b9c06d080 legacy=0x7f9b9c15d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba1800640 1 -- 172.21.15.43:0/1810046156 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c15e610 con 0x7f9b9c157770 2024-01-16T15:27:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.410+0000 7f9ba2200640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 0x7f9b9c070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.411+0000 7f9ba1800640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c157770 0x7f9b9c06cb40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009870 tx=0x7f9b8c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f2a86fc2571027 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9b9be00640 1 -- 172.21.15.43:0/1810046156 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c030840 con 0x7f9b9c157770 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9b9be00640 1 -- 172.21.15.43:0/1810046156 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c0309a0 con 0x7f9b9c157770 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9b9be00640 1 -- 172.21.15.43:0/1810046156 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c030d30 con 0x7f9b9c157770 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 -- 172.21.15.43:0/1810046156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c157770 msgr2=0x7f9b9c06cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c157770 0x7f9b9c06cb40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009870 tx=0x7f9b8c02b980 comp rx=0 tx=0).stop 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 -- 172.21.15.43:0/1810046156 shutdown_connections 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c15df80 0x7f9b9c070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/1810046156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c157770 0x7f9b9c06cb40 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.412+0000 7f9ba2c00640 1 -- 172.21.15.43:0/1810046156 >> 172.21.15.43:0/1810046156 conn(0x7f9b9c06a860 msgr2=0x7f9b9c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.413+0000 7f9ba2c00640 1 -- 172.21.15.43:0/1810046156 shutdown_connections 2024-01-16T15:27:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.413+0000 7f9ba2c00640 1 -- 172.21.15.43:0/1810046156 wait complete. 2024-01-16T15:27:24.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.414+0000 7f9ba2c00640 1 Processor -- start 2024-01-16T15:27:24.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.415+0000 7f9ba2c00640 1 -- start start 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.415+0000 7f9ba2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b9c06d080 0x7f9b9c158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c157770 0x7f9b9c159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c159390 con 0x7f9b9c15df80 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c157db0 con 0x7f9b9c06d080 2024-01-16T15:27:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c158090 con 0x7f9b9c157770 2024-01-16T15:27:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53802/0 (socket says 172.21.15.43:53802) 2024-01-16T15:27:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 -- 172.21.15.43:0/770150863 learned_addr learned my addr 172.21.15.43:0/770150863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba1800640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b9c06d080 0x7f9b9c158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba0e00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c157770 0x7f9b9c159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 -- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c157770 msgr2=0x7f9b9c159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c157770 0x7f9b9c159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 -- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b9c06d080 msgr2=0x7f9b9c158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.416+0000 7f9ba2200640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b9c06d080 0x7f9b9c158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9ba2200640 1 -- 172.21.15.43:0/770150863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0094d0 con 0x7f9b9c15df80 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9ba2200640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400cd50 tx=0x7f9b9400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400f680 con 0x7f9b9c15df80 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9400f800 con 0x7f9b9c15df80 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400fbb0 con 0x7f9b9c15df80 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c158370 con 0x7f9b9c15df80 2024-01-16T15:27:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.417+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c158930 con 0x7f9b9c15df80 2024-01-16T15:27:24.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.418+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c04e840 con 0x7f9b9c15df80 2024-01-16T15:27:24.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.420+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f9b9400b040 con 0x7f9b9c15df80 2024-01-16T15:27:24.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.421+0000 7f9b99600640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 0x7f9b7406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.421+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9b94089560 con 0x7f9b9c15df80 2024-01-16T15:27:24.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.421+0000 7f9ba1800640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 0x7f9b7406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.422+0000 7f9ba1800640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 0x7f9b7406e980 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c036000 tx=0x7f9b8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.426+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b94055ab0 con 0x7f9b9c15df80 2024-01-16T15:27:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.617+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b9c161300 con 0x7f9b7406c4c0 2024-01-16T15:27:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.643+0000 7f9b99600640 1 -- 172.21.15.43:0/770150863 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289534 (secure 0 0 0) 0x7f9b8c044470 con 0x7f9b7406c4c0 2024-01-16T15:27:24.645 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:24.648 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:27:24.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 msgr2=0x7f9b7406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 0x7f9b7406e980 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c036000 tx=0x7f9b8c008030 comp rx=0 tx=0).stop 2024-01-16T15:27:24.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 msgr2=0x7f9b9c15c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400cd50 tx=0x7f9b9400a650 comp rx=0 tx=0).stop 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 shutdown_connections 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f9b7406c4c0 0x7f9b7406e980 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b9c15df80 0x7f9b9c15c000 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b9c157770 0x7f9b9c159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 --2- 172.21.15.43:0/770150863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b9c06d080 0x7f9b9c158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 >> 172.21.15.43:0/770150863 conn(0x7f9b9c06a860 msgr2=0x7f9b9c155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 shutdown_connections 2024-01-16T15:27:24.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.651+0000 7f9ba2c00640 1 -- 172.21.15.43:0/770150863 wait complete. 2024-01-16T15:27:24.672 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":950,"stamp":"2024-01-16T15:27:23.502087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444287485,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2251,"num_write_kb":6295365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4682,"ondisk_log_size":4682,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13389440,"kb_used_data":12597760,"kb_used_omap":12,"kb_used_meta":791603,"kb_avail":741585280,"statfs":{"total":773094113280,"available":759383326720,"internally_reserved":0,"allocated":12900106240,"data_stored":12893524122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":810602028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073783661,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":276,"num_write_kb":1048624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005047"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737744+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737744+0000","last_peered":"2024-01-16T15:27:17.737744+0000","last_clean":"2024-01-16T15:27:17.737744+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737744+0000","last_undegraded":"2024-01-16T15:27:17.737744+0000","last_fullsized":"2024-01-16T15:27:17.737744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:47.552196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'69","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736614+0000","last_change":"2024-01-16T15:18:50.514868+0000","last_active":"2024-01-16T15:27:17.736614+0000","last_peered":"2024-01-16T15:27:17.736614+0000","last_clean":"2024-01-16T15:27:17.736614+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:17.736614+0000","last_undegraded":"2024-01-16T15:27:17.736614+0000","last_fullsized":"2024-01-16T15:27:17.736614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:22.704331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.743697+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:27:17.743697+0000","last_peered":"2024-01-16T15:27:17.743697+0000","last_clean":"2024-01-16T15:27:17.743697+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:27:17.743697+0000","last_undegraded":"2024-01-16T15:27:17.743697+0000","last_fullsized":"2024-01-16T15:27:17.743697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733813+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733813+0000","last_peered":"2024-01-16T15:27:17.733813+0000","last_clean":"2024-01-16T15:27:17.733813+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733813+0000","last_undegraded":"2024-01-16T15:27:17.733813+0000","last_fullsized":"2024-01-16T15:27:17.733813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:27.895459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'60","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717706+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:27:17.717706+0000","last_peered":"2024-01-16T15:27:17.717706+0000","last_clean":"2024-01-16T15:27:17.717706+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:17.717706+0000","last_undegraded":"2024-01-16T15:27:17.717706+0000","last_fullsized":"2024-01-16T15:27:17.717706+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:32.606162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'74","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721540+0000","last_change":"2024-01-16T15:18:50.517889+0000","last_active":"2024-01-16T15:27:17.721540+0000","last_peered":"2024-01-16T15:27:17.721540+0000","last_clean":"2024-01-16T15:27:17.721540+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:17.721540+0000","last_undegraded":"2024-01-16T15:27:17.721540+0000","last_fullsized":"2024-01-16T15:27:17.721540+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'66","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734633+0000","last_change":"2024-01-16T15:18:50.506124+0000","last_active":"2024-01-16T15:27:17.734633+0000","last_peered":"2024-01-16T15:27:17.734633+0000","last_clean":"2024-01-16T15:27:17.734633+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:17.734633+0000","last_undegraded":"2024-01-16T15:27:17.734633+0000","last_fullsized":"2024-01-16T15:27:17.734633+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'54","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730876+0000","last_change":"2024-01-16T15:18:50.513082+0000","last_active":"2024-01-16T15:27:17.730876+0000","last_peered":"2024-01-16T15:27:17.730876+0000","last_clean":"2024-01-16T15:27:17.730876+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:17.730876+0000","last_undegraded":"2024-01-16T15:27:17.730876+0000","last_fullsized":"2024-01-16T15:27:17.730876+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724783+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.724783+0000","last_peered":"2024-01-16T15:27:17.724783+0000","last_clean":"2024-01-16T15:27:17.724783+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.724783+0000","last_undegraded":"2024-01-16T15:27:17.724783+0000","last_fullsized":"2024-01-16T15:27:17.724783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:57.207916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732638+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.732638+0000","last_peered":"2024-01-16T15:27:17.732638+0000","last_clean":"2024-01-16T15:27:17.732638+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.732638+0000","last_undegraded":"2024-01-16T15:27:17.732638+0000","last_fullsized":"2024-01-16T15:27:17.732638+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:26.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'70","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.727985+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:27:17.727985+0000","last_peered":"2024-01-16T15:27:17.727985+0000","last_clean":"2024-01-16T15:27:17.727985+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:17.727985+0000","last_undegraded":"2024-01-16T15:27:17.727985+0000","last_fullsized":"2024-01-16T15:27:17.727985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735505+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735505+0000","last_peered":"2024-01-16T15:27:17.735505+0000","last_clean":"2024-01-16T15:27:17.735505+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735505+0000","last_undegraded":"2024-01-16T15:27:17.735505+0000","last_fullsized":"2024-01-16T15:27:17.735505+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:28:07.149636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'57","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734731+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:27:17.734731+0000","last_peered":"2024-01-16T15:27:17.734731+0000","last_clean":"2024-01-16T15:27:17.734731+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:27:17.734731+0000","last_undegraded":"2024-01-16T15:27:17.734731+0000","last_fullsized":"2024-01-16T15:27:17.734731+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:35.209083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'65","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739366+0000","last_change":"2024-01-16T15:18:50.517177+0000","last_active":"2024-01-16T15:27:17.739366+0000","last_peered":"2024-01-16T15:27:17.739366+0000","last_clean":"2024-01-16T15:27:17.739366+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:17.739366+0000","last_undegraded":"2024-01-16T15:27:17.739366+0000","last_fullsized":"2024-01-16T15:27:17.739366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'54","reported_seq":77,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724971+0000","last_change":"2024-01-16T15:18:50.518134+0000","last_active":"2024-01-16T15:27:17.724971+0000","last_peered":"2024-01-16T15:27:17.724971+0000","last_clean":"2024-01-16T15:27:17.724971+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:17.724971+0000","last_undegraded":"2024-01-16T15:27:17.724971+0000","last_fullsized":"2024-01-16T15:27:17.724971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:10.936165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738723+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:27:17.738723+0000","last_peered":"2024-01-16T15:27:17.738723+0000","last_clean":"2024-01-16T15:27:17.738723+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:17.738723+0000","last_undegraded":"2024-01-16T15:27:17.738723+0000","last_fullsized":"2024-01-16T15:27:17.738723+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:18.639887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'49","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733966+0000","last_change":"2024-01-16T15:18:50.514080+0000","last_active":"2024-01-16T15:27:17.733966+0000","last_peered":"2024-01-16T15:27:17.733966+0000","last_clean":"2024-01-16T15:27:17.733966+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:17.733966+0000","last_undegraded":"2024-01-16T15:27:17.733966+0000","last_fullsized":"2024-01-16T15:27:17.733966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:18.432791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'72","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735139+0000","last_change":"2024-01-16T15:18:50.512290+0000","last_active":"2024-01-16T15:27:17.735139+0000","last_peered":"2024-01-16T15:27:17.735139+0000","last_clean":"2024-01-16T15:27:17.735139+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:17.735139+0000","last_undegraded":"2024-01-16T15:27:17.735139+0000","last_fullsized":"2024-01-16T15:27:17.735139+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:03.196990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'71","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741586+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:27:17.741586+0000","last_peered":"2024-01-16T15:27:17.741586+0000","last_clean":"2024-01-16T15:27:17.741586+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:27:17.741586+0000","last_undegraded":"2024-01-16T15:27:17.741586+0000","last_fullsized":"2024-01-16T15:27:17.741586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.753500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734862+0000","last_change":"2024-01-16T15:18:50.523238+0000","last_active":"2024-01-16T15:27:17.734862+0000","last_peered":"2024-01-16T15:27:17.734862+0000","last_clean":"2024-01-16T15:27:17.734862+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:17.734862+0000","last_undegraded":"2024-01-16T15:27:17.734862+0000","last_fullsized":"2024-01-16T15:27:17.734862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:10.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'73","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742649+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:27:17.742649+0000","last_peered":"2024-01-16T15:27:17.742649+0000","last_clean":"2024-01-16T15:27:17.742649+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:27:17.742649+0000","last_undegraded":"2024-01-16T15:27:17.742649+0000","last_fullsized":"2024-01-16T15:27:17.742649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:04.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'64","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717654+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:27:17.717654+0000","last_peered":"2024-01-16T15:27:17.717654+0000","last_clean":"2024-01-16T15:27:17.717654+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:27:17.717654+0000","last_undegraded":"2024-01-16T15:27:17.717654+0000","last_fullsized":"2024-01-16T15:27:17.717654+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'57","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739771+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:27:17.739771+0000","last_peered":"2024-01-16T15:27:17.739771+0000","last_clean":"2024-01-16T15:27:17.739771+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:17.739771+0000","last_undegraded":"2024-01-16T15:27:17.739771+0000","last_fullsized":"2024-01-16T15:27:17.739771+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:49.070110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'70","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717432+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:27:17.717432+0000","last_peered":"2024-01-16T15:27:17.717432+0000","last_clean":"2024-01-16T15:27:17.717432+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:27:17.717432+0000","last_undegraded":"2024-01-16T15:27:17.717432+0000","last_fullsized":"2024-01-16T15:27:17.717432+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:09.748440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'71","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740798+0000","last_change":"2024-01-16T15:18:50.513030+0000","last_active":"2024-01-16T15:27:17.740798+0000","last_peered":"2024-01-16T15:27:17.740798+0000","last_clean":"2024-01-16T15:27:17.740798+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:17.740798+0000","last_undegraded":"2024-01-16T15:27:17.740798+0000","last_fullsized":"2024-01-16T15:27:17.740798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:54.334893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'81","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736872+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:27:17.736872+0000","last_peered":"2024-01-16T15:27:17.736872+0000","last_clean":"2024-01-16T15:27:17.736872+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:27:17.736872+0000","last_undegraded":"2024-01-16T15:27:17.736872+0000","last_fullsized":"2024-01-16T15:27:17.736872+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:49.522744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'64","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730649+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:27:17.730649+0000","last_peered":"2024-01-16T15:27:17.730649+0000","last_clean":"2024-01-16T15:27:17.730649+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:27:17.730649+0000","last_undegraded":"2024-01-16T15:27:17.730649+0000","last_fullsized":"2024-01-16T15:27:17.730649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":64,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730383+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:17.730383+0000","last_peered":"2024-01-16T15:27:17.730383+0000","last_clean":"2024-01-16T15:27:17.730383+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:17.730383+0000","last_undegraded":"2024-01-16T15:27:17.730383+0000","last_fullsized":"2024-01-16T15:27:17.730383+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'57","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738407+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:27:17.738407+0000","last_peered":"2024-01-16T15:27:17.738407+0000","last_clean":"2024-01-16T15:27:17.738407+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:27:17.738407+0000","last_undegraded":"2024-01-16T15:27:17.738407+0000","last_fullsized":"2024-01-16T15:27:17.738407+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:38.948857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'76","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720265+0000","last_change":"2024-01-16T15:18:50.491931+0000","last_active":"2024-01-16T15:27:17.720265+0000","last_peered":"2024-01-16T15:27:17.720265+0000","last_clean":"2024-01-16T15:27:17.720265+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:17.720265+0000","last_undegraded":"2024-01-16T15:27:17.720265+0000","last_fullsized":"2024-01-16T15:27:17.720265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'81","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736796+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:27:17.736796+0000","last_peered":"2024-01-16T15:27:17.736796+0000","last_clean":"2024-01-16T15:27:17.736796+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:17.736796+0000","last_undegraded":"2024-01-16T15:27:17.736796+0000","last_fullsized":"2024-01-16T15:27:17.736796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:03.259116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'63","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.723981+0000","last_change":"2024-01-16T15:18:50.510830+0000","last_active":"2024-01-16T15:27:17.723981+0000","last_peered":"2024-01-16T15:27:17.723981+0000","last_clean":"2024-01-16T15:27:17.723981+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:17.723981+0000","last_undegraded":"2024-01-16T15:27:17.723981+0000","last_fullsized":"2024-01-16T15:27:17.723981+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:30.055863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'62","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721097+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:27:17.721097+0000","last_peered":"2024-01-16T15:27:17.721097+0000","last_clean":"2024-01-16T15:27:17.721097+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:17.721097+0000","last_undegraded":"2024-01-16T15:27:17.721097+0000","last_fullsized":"2024-01-16T15:27:17.721097+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476688+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:25:05.476688+0000","last_peered":"2024-01-16T15:25:05.476688+0000","last_clean":"2024-01-16T15:25:05.476688+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:25:05.476688+0000","last_undegraded":"2024-01-16T15:25:05.476688+0000","last_fullsized":"2024-01-16T15:25:05.476688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'54","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733008+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:27:17.733008+0000","last_peered":"2024-01-16T15:27:17.733008+0000","last_clean":"2024-01-16T15:27:17.733008+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:27:17.733008+0000","last_undegraded":"2024-01-16T15:27:17.733008+0000","last_fullsized":"2024-01-16T15:27:17.733008+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:37.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.466688+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:24:50.466688+0000","last_peered":"2024-01-16T15:24:50.466688+0000","last_clean":"2024-01-16T15:24:50.466688+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:24:50.466688+0000","last_undegraded":"2024-01-16T15:24:50.466688+0000","last_fullsized":"2024-01-16T15:24:50.466688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:51.215798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'71","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724610+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:27:17.724610+0000","last_peered":"2024-01-16T15:27:17.724610+0000","last_clean":"2024-01-16T15:27:17.724610+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:17.724610+0000","last_undegraded":"2024-01-16T15:27:17.724610+0000","last_fullsized":"2024-01-16T15:27:17.724610+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360466+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:23:50.360466+0000","last_peered":"2024-01-16T15:23:50.360466+0000","last_clean":"2024-01-16T15:23:50.360466+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:23:50.360466+0000","last_undegraded":"2024-01-16T15:23:50.360466+0000","last_fullsized":"2024-01-16T15:23:50.360466+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'68","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741745+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:27:17.741745+0000","last_peered":"2024-01-16T15:27:17.741745+0000","last_clean":"2024-01-16T15:27:17.741745+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:27:17.741745+0000","last_undegraded":"2024-01-16T15:27:17.741745+0000","last_fullsized":"2024-01-16T15:27:17.741745+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:37.406427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360421+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:23:50.360421+0000","last_peered":"2024-01-16T15:23:50.360421+0000","last_clean":"2024-01-16T15:23:50.360421+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:23:50.360421+0000","last_undegraded":"2024-01-16T15:23:50.360421+0000","last_fullsized":"2024-01-16T15:23:50.360421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'72","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.719815+0000","last_change":"2024-01-16T15:19:15.318178+0000","last_active":"2024-01-16T15:27:17.719815+0000","last_peered":"2024-01-16T15:27:17.719815+0000","last_clean":"2024-01-16T15:27:17.719815+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:17.719815+0000","last_undegraded":"2024-01-16T15:27:17.719815+0000","last_fullsized":"2024-01-16T15:27:17.719815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.463254+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:23:50.463254+0000","last_peered":"2024-01-16T15:23:50.463254+0000","last_clean":"2024-01-16T15:23:50.463254+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:23:50.463254+0000","last_undegraded":"2024-01-16T15:23:50.463254+0000","last_fullsized":"2024-01-16T15:23:50.463254+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:31.593211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'72","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736806+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:27:17.736806+0000","last_peered":"2024-01-16T15:27:17.736806+0000","last_clean":"2024-01-16T15:27:17.736806+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:27:17.736806+0000","last_undegraded":"2024-01-16T15:27:17.736806+0000","last_fullsized":"2024-01-16T15:27:17.736806+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:59.415403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.403153+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:23:50.403153+0000","last_peered":"2024-01-16T15:23:50.403153+0000","last_clean":"2024-01-16T15:23:50.403153+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:23:50.403153+0000","last_undegraded":"2024-01-16T15:23:50.403153+0000","last_fullsized":"2024-01-16T15:23:50.403153+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:57.932540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'73","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732782+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:27:17.732782+0000","last_peered":"2024-01-16T15:27:17.732782+0000","last_clean":"2024-01-16T15:27:17.732782+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:17.732782+0000","last_undegraded":"2024-01-16T15:27:17.732782+0000","last_fullsized":"2024-01-16T15:27:17.732782+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:45.776442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.368270+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:23:40.368270+0000","last_peered":"2024-01-16T15:23:40.368270+0000","last_clean":"2024-01-16T15:23:40.368270+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:23:40.368270+0000","last_undegraded":"2024-01-16T15:23:40.368270+0000","last_fullsized":"2024-01-16T15:23:40.368270+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:07.440889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'73","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726884+0000","last_change":"2024-01-16T15:18:50.506340+0000","last_active":"2024-01-16T15:27:17.726884+0000","last_peered":"2024-01-16T15:27:17.726884+0000","last_clean":"2024-01-16T15:27:17.726884+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:17.726884+0000","last_undegraded":"2024-01-16T15:27:17.726884+0000","last_fullsized":"2024-01-16T15:27:17.726884+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.622586+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:23:40.622586+0000","last_peered":"2024-01-16T15:23:40.622586+0000","last_clean":"2024-01-16T15:23:40.622586+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:23:40.622586+0000","last_undegraded":"2024-01-16T15:23:40.622586+0000","last_fullsized":"2024-01-16T15:23:40.622586+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722570+0000","last_change":"2024-01-16T15:18:50.507182+0000","last_active":"2024-01-16T15:27:17.722570+0000","last_peered":"2024-01-16T15:27:17.722570+0000","last_clean":"2024-01-16T15:27:17.722570+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:17.722570+0000","last_undegraded":"2024-01-16T15:27:17.722570+0000","last_fullsized":"2024-01-16T15:27:17.722570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.358584+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:23:40.358584+0000","last_peered":"2024-01-16T15:23:40.358584+0000","last_clean":"2024-01-16T15:23:40.358584+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:23:40.358584+0000","last_undegraded":"2024-01-16T15:23:40.358584+0000","last_fullsized":"2024-01-16T15:23:40.358584+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'61","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738962+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:27:17.738962+0000","last_peered":"2024-01-16T15:27:17.738962+0000","last_clean":"2024-01-16T15:27:17.738962+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:17.738962+0000","last_undegraded":"2024-01-16T15:27:17.738962+0000","last_fullsized":"2024-01-16T15:27:17.738962+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:14.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.401032+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:23:40.401032+0000","last_peered":"2024-01-16T15:23:40.401032+0000","last_clean":"2024-01-16T15:23:40.401032+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:23:40.401032+0000","last_undegraded":"2024-01-16T15:23:40.401032+0000","last_fullsized":"2024-01-16T15:23:40.401032+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:31.105453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726260+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:17.726260+0000","last_peered":"2024-01-16T15:27:17.726260+0000","last_clean":"2024-01-16T15:27:17.726260+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:17.726260+0000","last_undegraded":"2024-01-16T15:27:17.726260+0000","last_fullsized":"2024-01-16T15:27:17.726260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'33","reported_seq":50,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.122833+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:25:12.122833+0000","last_peered":"2024-01-16T15:25:12.122833+0000","last_clean":"2024-01-16T15:25:12.122833+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:25:12.122833+0000","last_undegraded":"2024-01-16T15:25:12.122833+0000","last_fullsized":"2024-01-16T15:25:12.122833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'69","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.729047+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:27:17.729047+0000","last_peered":"2024-01-16T15:27:17.729047+0000","last_clean":"2024-01-16T15:27:17.729047+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:17.729047+0000","last_undegraded":"2024-01-16T15:27:17.729047+0000","last_fullsized":"2024-01-16T15:27:17.729047+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'321","reported_seq":338,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.747172+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:27:17.747172+0000","last_peered":"2024-01-16T15:27:17.747172+0000","last_clean":"2024-01-16T15:27:17.747172+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:17.747172+0000","last_undegraded":"2024-01-16T15:27:17.747172+0000","last_fullsized":"2024-01-16T15:27:17.747172+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239629,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.718504+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:17.718504+0000","last_peered":"2024-01-16T15:27:17.718504+0000","last_clean":"2024-01-16T15:27:17.718504+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:17.718504+0000","last_undegraded":"2024-01-16T15:27:17.718504+0000","last_fullsized":"2024-01-16T15:27:17.718504+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.500458+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:23:30.500458+0000","last_peered":"2024-01-16T15:23:30.500458+0000","last_clean":"2024-01-16T15:23:30.500458+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:23:30.500458+0000","last_undegraded":"2024-01-16T15:23:30.500458+0000","last_fullsized":"2024-01-16T15:23:30.500458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'73","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730523+0000","last_change":"2024-01-16T15:18:50.517675+0000","last_active":"2024-01-16T15:27:17.730523+0000","last_peered":"2024-01-16T15:27:17.730523+0000","last_clean":"2024-01-16T15:27:17.730523+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:17.730523+0000","last_undegraded":"2024-01-16T15:27:17.730523+0000","last_fullsized":"2024-01-16T15:27:17.730523+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.440394+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:23:30.440394+0000","last_peered":"2024-01-16T15:23:30.440394+0000","last_clean":"2024-01-16T15:23:30.440394+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:23:30.440394+0000","last_undegraded":"2024-01-16T15:23:30.440394+0000","last_fullsized":"2024-01-16T15:23:30.440394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:28.315056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'63","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740737+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:27:17.740737+0000","last_peered":"2024-01-16T15:27:17.740737+0000","last_clean":"2024-01-16T15:27:17.740737+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:27:17.740737+0000","last_undegraded":"2024-01-16T15:27:17.740737+0000","last_fullsized":"2024-01-16T15:27:17.740737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":25,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.356549+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:23:30.356549+0000","last_peered":"2024-01-16T15:23:30.356549+0000","last_clean":"2024-01-16T15:23:30.356549+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:23:30.356549+0000","last_undegraded":"2024-01-16T15:23:30.356549+0000","last_fullsized":"2024-01-16T15:23:30.356549+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:58.071689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'46","reported_seq":68,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735009+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:27:17.735009+0000","last_peered":"2024-01-16T15:27:17.735009+0000","last_clean":"2024-01-16T15:27:17.735009+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:27:17.735009+0000","last_undegraded":"2024-01-16T15:27:17.735009+0000","last_fullsized":"2024-01-16T15:27:17.735009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:40.855053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.618711+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:23:20.618711+0000","last_peered":"2024-01-16T15:23:20.618711+0000","last_clean":"2024-01-16T15:23:20.618711+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:23:20.618711+0000","last_undegraded":"2024-01-16T15:23:20.618711+0000","last_fullsized":"2024-01-16T15:23:20.618711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734412+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:17.734412+0000","last_peered":"2024-01-16T15:27:17.734412+0000","last_clean":"2024-01-16T15:27:17.734412+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:17.734412+0000","last_undegraded":"2024-01-16T15:27:17.734412+0000","last_fullsized":"2024-01-16T15:27:17.734412+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.354766+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:23:20.354766+0000","last_peered":"2024-01-16T15:23:20.354766+0000","last_clean":"2024-01-16T15:23:20.354766+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:23:20.354766+0000","last_undegraded":"2024-01-16T15:23:20.354766+0000","last_fullsized":"2024-01-16T15:23:20.354766+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738127+0000","last_change":"2024-01-16T15:18:50.517233+0000","last_active":"2024-01-16T15:27:17.738127+0000","last_peered":"2024-01-16T15:27:17.738127+0000","last_clean":"2024-01-16T15:27:17.738127+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:17.738127+0000","last_undegraded":"2024-01-16T15:27:17.738127+0000","last_fullsized":"2024-01-16T15:27:17.738127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.438493+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:23:20.438493+0000","last_peered":"2024-01-16T15:23:20.438493+0000","last_clean":"2024-01-16T15:23:20.438493+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:23:20.438493+0000","last_undegraded":"2024-01-16T15:23:20.438493+0000","last_fullsized":"2024-01-16T15:23:20.438493+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:04.731539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744447+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:27:17.744447+0000","last_peered":"2024-01-16T15:27:17.744447+0000","last_clean":"2024-01-16T15:27:17.744447+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:27:17.744447+0000","last_undegraded":"2024-01-16T15:27:17.744447+0000","last_fullsized":"2024-01-16T15:27:17.744447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:16.245190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.457791+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:23:20.457791+0000","last_peered":"2024-01-16T15:23:20.457791+0000","last_clean":"2024-01-16T15:23:20.457791+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:23:20.457791+0000","last_undegraded":"2024-01-16T15:23:20.457791+0000","last_fullsized":"2024-01-16T15:23:20.457791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'56","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741337+0000","last_change":"2024-01-16T15:18:50.521667+0000","last_active":"2024-01-16T15:27:17.741337+0000","last_peered":"2024-01-16T15:27:17.741337+0000","last_clean":"2024-01-16T15:27:17.741337+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:17.741337+0000","last_undegraded":"2024-01-16T15:27:17.741337+0000","last_fullsized":"2024-01-16T15:27:17.741337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.498582+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:23:20.498582+0000","last_peered":"2024-01-16T15:23:20.498582+0000","last_clean":"2024-01-16T15:23:20.498582+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:23:20.498582+0000","last_undegraded":"2024-01-16T15:23:20.498582+0000","last_fullsized":"2024-01-16T15:23:20.498582+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738717+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:27:17.738717+0000","last_peered":"2024-01-16T15:27:17.738717+0000","last_clean":"2024-01-16T15:27:17.738717+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:27:17.738717+0000","last_undegraded":"2024-01-16T15:27:17.738717+0000","last_fullsized":"2024-01-16T15:27:17.738717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:59.502905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.353213+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:23:10.353213+0000","last_peered":"2024-01-16T15:23:10.353213+0000","last_clean":"2024-01-16T15:23:10.353213+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:23:10.353213+0000","last_undegraded":"2024-01-16T15:23:10.353213+0000","last_fullsized":"2024-01-16T15:23:10.353213+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:30.184973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'65","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735772+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:27:17.735772+0000","last_peered":"2024-01-16T15:27:17.735772+0000","last_clean":"2024-01-16T15:27:17.735772+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:27:17.735772+0000","last_undegraded":"2024-01-16T15:27:17.735772+0000","last_fullsized":"2024-01-16T15:27:17.735772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.496763+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:23:10.496763+0000","last_peered":"2024-01-16T15:23:10.496763+0000","last_clean":"2024-01-16T15:23:10.496763+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:23:10.496763+0000","last_undegraded":"2024-01-16T15:23:10.496763+0000","last_fullsized":"2024-01-16T15:23:10.496763+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'57","reported_seq":83,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737237+0000","last_change":"2024-01-16T15:18:50.516592+0000","last_active":"2024-01-16T15:27:17.737237+0000","last_peered":"2024-01-16T15:27:17.737237+0000","last_clean":"2024-01-16T15:27:17.737237+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:17.737237+0000","last_undegraded":"2024-01-16T15:27:17.737237+0000","last_fullsized":"2024-01-16T15:27:17.737237+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:17.446551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.615833+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:23:10.615833+0000","last_peered":"2024-01-16T15:23:10.615833+0000","last_clean":"2024-01-16T15:23:10.615833+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:23:10.615833+0000","last_undegraded":"2024-01-16T15:23:10.615833+0000","last_fullsized":"2024-01-16T15:23:10.615833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'53","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739953+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:27:17.739953+0000","last_peered":"2024-01-16T15:27:17.739953+0000","last_clean":"2024-01-16T15:27:17.739953+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:17.739953+0000","last_undegraded":"2024-01-16T15:27:17.739953+0000","last_fullsized":"2024-01-16T15:27:17.739953+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:09.913343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.116830+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:25:12.116830+0000","last_peered":"2024-01-16T15:25:12.116830+0000","last_clean":"2024-01-16T15:25:12.116830+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:25:12.116830+0000","last_undegraded":"2024-01-16T15:25:12.116830+0000","last_fullsized":"2024-01-16T15:25:12.116830+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:17.281805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741249+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:17.741249+0000","last_peered":"2024-01-16T15:27:17.741249+0000","last_clean":"2024-01-16T15:27:17.741249+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:17.741249+0000","last_undegraded":"2024-01-16T15:27:17.741249+0000","last_fullsized":"2024-01-16T15:27:17.741249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'30","reported_seq":47,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:14.187594+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:27:14.187594+0000","last_peered":"2024-01-16T15:27:14.187594+0000","last_clean":"2024-01-16T15:27:14.187594+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:14.187594+0000","last_undegraded":"2024-01-16T15:27:14.187594+0000","last_fullsized":"2024-01-16T15:27:14.187594+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:07.461774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'69","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735893+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:27:17.735893+0000","last_peered":"2024-01-16T15:27:17.735893+0000","last_clean":"2024-01-16T15:27:17.735893+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:27:17.735893+0000","last_undegraded":"2024-01-16T15:27:17.735893+0000","last_fullsized":"2024-01-16T15:27:17.735893+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:49.736642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.641840+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:25:05.641840+0000","last_peered":"2024-01-16T15:25:05.641840+0000","last_clean":"2024-01-16T15:25:05.641840+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:25:05.641840+0000","last_undegraded":"2024-01-16T15:25:05.641840+0000","last_fullsized":"2024-01-16T15:25:05.641840+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'59","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738409+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:27:17.738409+0000","last_peered":"2024-01-16T15:27:17.738409+0000","last_clean":"2024-01-16T15:27:17.738409+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:27:17.738409+0000","last_undegraded":"2024-01-16T15:27:17.738409+0000","last_fullsized":"2024-01-16T15:27:17.738409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:27.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.417626+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:25:05.417626+0000","last_peered":"2024-01-16T15:25:05.417626+0000","last_clean":"2024-01-16T15:25:05.417626+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:25:05.417626+0000","last_undegraded":"2024-01-16T15:25:05.417626+0000","last_fullsized":"2024-01-16T15:25:05.417626+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.618165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'63","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736683+0000","last_change":"2024-01-16T15:18:50.516022+0000","last_active":"2024-01-16T15:27:17.736683+0000","last_peered":"2024-01-16T15:27:17.736683+0000","last_clean":"2024-01-16T15:27:17.736683+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:17.736683+0000","last_undegraded":"2024-01-16T15:27:17.736683+0000","last_fullsized":"2024-01-16T15:27:17.736683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476614+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:25:05.476614+0000","last_peered":"2024-01-16T15:25:05.476614+0000","last_clean":"2024-01-16T15:25:05.476614+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:25:05.476614+0000","last_undegraded":"2024-01-16T15:25:05.476614+0000","last_fullsized":"2024-01-16T15:25:05.476614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:39.842102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742798+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:27:17.742798+0000","last_peered":"2024-01-16T15:27:17.742798+0000","last_clean":"2024-01-16T15:27:17.742798+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:27:17.742798+0000","last_undegraded":"2024-01-16T15:27:17.742798+0000","last_fullsized":"2024-01-16T15:27:17.742798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:03.949006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.384697+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:25:05.384697+0000","last_peered":"2024-01-16T15:25:05.384697+0000","last_clean":"2024-01-16T15:25:05.384697+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:25:05.384697+0000","last_undegraded":"2024-01-16T15:25:05.384697+0000","last_fullsized":"2024-01-16T15:25:05.384697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:26.402495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'68","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738935+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:27:17.738935+0000","last_peered":"2024-01-16T15:27:17.738935+0000","last_clean":"2024-01-16T15:27:17.738935+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:27:17.738935+0000","last_undegraded":"2024-01-16T15:27:17.738935+0000","last_fullsized":"2024-01-16T15:27:17.738935+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'34","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.114659+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:25:12.114659+0000","last_peered":"2024-01-16T15:25:12.114659+0000","last_clean":"2024-01-16T15:25:12.114659+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:25:12.114659+0000","last_undegraded":"2024-01-16T15:25:12.114659+0000","last_fullsized":"2024-01-16T15:25:12.114659+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733937+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:27:17.733937+0000","last_peered":"2024-01-16T15:27:17.733937+0000","last_clean":"2024-01-16T15:27:17.733937+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:27:17.733937+0000","last_undegraded":"2024-01-16T15:27:17.733937+0000","last_fullsized":"2024-01-16T15:27:17.733937+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:01.372900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.638362+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:24:50.638362+0000","last_peered":"2024-01-16T15:24:50.638362+0000","last_clean":"2024-01-16T15:24:50.638362+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:24:50.638362+0000","last_undegraded":"2024-01-16T15:24:50.638362+0000","last_fullsized":"2024-01-16T15:24:50.638362+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'49","reported_seq":78,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.712896+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:27:17.712896+0000","last_peered":"2024-01-16T15:27:17.712896+0000","last_clean":"2024-01-16T15:27:17.712896+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:17.712896+0000","last_undegraded":"2024-01-16T15:27:17.712896+0000","last_fullsized":"2024-01-16T15:27:17.712896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.459727+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:23:30.459727+0000","last_peered":"2024-01-16T15:23:30.459727+0000","last_clean":"2024-01-16T15:23:30.459727+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:23:30.459727+0000","last_undegraded":"2024-01-16T15:23:30.459727+0000","last_fullsized":"2024-01-16T15:23:30.459727+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:24.899317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":252,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:25.450215+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:23:25.450215+0000","last_peered":"2024-01-16T15:23:25.450215+0000","last_clean":"2024-01-16T15:23:25.450215+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:23:25.450215+0000","last_undegraded":"2024-01-16T15:23:25.450215+0000","last_fullsized":"2024-01-16T15:23:25.450215+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4114,"ondisk_log_size":4114,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1244374,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":487,"num_write_kb":1455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":2488748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542483,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.222,"15min":1.274},"min":{"1min":0.64800000000000002,"5min":0.54300000000000004,"15min":0.28599999999999998},"max":{"1min":1.4159999999999999,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":1.2},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.302,"15min":1.4399999999999999},"min":{"1min":0.75900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.712,"5min":1.8959999999999999,"15min":25.503},"last":1.282}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.34499999999999997},"max":{"1min":1.54,"5min":1.5620000000000001,"15min":5.5010000000000003},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.1619999999999999,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4139999999999999,"5min":1.9490000000000001,"15min":4.3369999999999997},"last":1.343}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.68400000000000005,"15min":0.629},"max":{"1min":1.669,"5min":2.141,"15min":11.536},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.4670000000000001},"min":{"1min":0.73099999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5309999999999999,"5min":1.8500000000000001,"15min":25.478000000000002},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.40899999999999997},"max":{"1min":1.4870000000000001,"5min":1.7090000000000001,"15min":5.6369999999999996},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":0.80200000000000005,"5min":0.75900000000000001,"15min":0.67800000000000005},"max":{"1min":1.5589999999999999,"5min":1.9550000000000001,"15min":6.351},"last":1.5629999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.204,"15min":1.3919999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6120000000000001,"5min":1.75,"15min":25.401},"last":1.395},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6399999999999999,"5min":1.8160000000000001,"15min":5.7859999999999996},"last":1.1679999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3540000000000001,"15min":1.359},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.722,"5min":2.3849999999999998,"15min":4.4539999999999997},"last":1.472},{"interface":"front","average":{"1min":1.385,"5min":1.383,"15min":1.5189999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.708,"5min":1.8480000000000001,"15min":26.327000000000002},"last":1.516}]},{"osd":6,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5469999999999999,"15min":1.6739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8280000000000001,"5min":2.0579999999999998,"15min":25.451000000000001},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5029999999999999,"15min":1.496},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.786,"5min":2.786,"15min":5.8250000000000002},"last":1.4379999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.68400000000000005,"15min":0.35399999999999998},"max":{"1min":1.577,"5min":8.0299999999999994,"15min":29.77},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3220000000000001,"15min":1.415},"min":{"1min":0.88500000000000001,"5min":0.76400000000000001,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.528}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.081,"15min":1.1339999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":1.728,"15min":5.2859999999999996},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.57099999999999995},"max":{"1min":1.595,"5min":1.7350000000000001,"15min":10.494},"last":0.95299999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.202,"15min":1.278},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.46,"5min":1.6579999999999999,"15min":5.0549999999999997},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.389,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.837,"5min":1.9279999999999999,"15min":5.1459999999999999},"last":1.0760000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.319,"15min":1.3089999999999999},"min":{"1min":0.90200000000000002,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":1.601,"5min":1.8360000000000001,"15min":3.8079999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.3129999999999999,"15min":1.448},"min":{"1min":0.87,"5min":0.65200000000000002,"15min":0.623},"max":{"1min":1.7130000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3839999999999999,"15min":1.3480000000000001},"min":{"1min":0.97399999999999998,"5min":0.87,"15min":0.53400000000000003},"max":{"1min":1.6890000000000001,"5min":20.422999999999998,"15min":20.422999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.258,"15min":1.28},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":3.6160000000000001},"last":1.4370000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.272,"15min":1.321},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.53700000000000003},"max":{"1min":1.542,"5min":7.6319999999999997,"15min":13.949999999999999},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.581,"15min":1.5720000000000001},"min":{"1min":1.113,"5min":1.113,"15min":0.51600000000000001},"max":{"1min":1.972,"5min":7.4980000000000002,"15min":11.298},"last":1.633}]},{"osd":7,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4410000000000001,"15min":1.498},"min":{"1min":1.0169999999999999,"5min":0.97499999999999998,"15min":0.57099999999999995},"max":{"1min":1.659,"5min":2.004,"15min":14.037000000000001},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.512,"15min":1.5720000000000001},"min":{"1min":1.1100000000000001,"5min":1.024,"15min":0.61599999999999999},"max":{"1min":1.9370000000000001,"5min":2.0209999999999999,"15min":4.9829999999999997},"last":1.5860000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141700,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93230140,"statfs":{"total":96636764160,"available":95467663360,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2090000000000001,"15min":1.1379999999999999},"min":{"1min":0.79300000000000004,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.6890000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.163,"15min":1.147},"min":{"1min":0.89900000000000002,"5min":0.72599999999999998,"15min":0.34799999999999998},"max":{"1min":1.526,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.194}]},{"osd":1,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1850000000000001,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":1.776,"15min":17.074999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.228,"15min":1.244},"min":{"1min":0.89000000000000001,"5min":0.77100000000000002,"15min":0.46999999999999997},"max":{"1min":1.714,"5min":1.714,"15min":4.8499999999999996},"last":1.0149999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.222,"15min":1.198},"min":{"1min":0.83899999999999997,"5min":0.72599999999999998,"15min":0.54800000000000004},"max":{"1min":1.53,"5min":2.7000000000000002,"15min":4.306},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2849999999999999,"15min":1.323},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.52200000000000002},"max":{"1min":1.494,"5min":2.9670000000000001,"15min":16.721},"last":0.97299999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.97499999999999998,"5min":0.81499999999999995,"15min":0.60299999999999998},"max":{"1min":1.859,"5min":1.905,"15min":4.8010000000000002},"last":1.228},{"interface":"front","average":{"1min":1.427,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":0.51200000000000001},"max":{"1min":1.7390000000000001,"5min":14.044,"15min":14.044},"last":1.3089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2749999999999999,"15min":1.3260000000000001},"min":{"1min":0.90400000000000003,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.591,"5min":2.0139999999999998,"15min":16.382999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.321,"15min":1.341},"min":{"1min":0.81000000000000005,"5min":0.72099999999999997,"15min":0.502},"max":{"1min":1.982,"5min":1.982,"15min":4.6580000000000004},"last":1.145}]},{"osd":6,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3500000000000001,"15min":1.357},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.39700000000000002},"max":{"1min":1.637,"5min":1.901,"15min":4.3380000000000001},"last":1.48},{"interface":"front","average":{"1min":1.488,"5min":1.516,"15min":1.5089999999999999},"min":{"1min":1.137,"5min":0.94199999999999995,"15min":0.57899999999999996},"max":{"1min":1.6919999999999999,"5min":1.907,"15min":3.3719999999999999},"last":1.8180000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5,"15min":1.5029999999999999},"min":{"1min":0.95999999999999996,"5min":0.95599999999999996,"15min":0.45000000000000001},"max":{"1min":2.044,"5min":2.044,"15min":4.742},"last":1.369},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5700000000000001,"15min":1.5509999999999999},"min":{"1min":1.0820000000000001,"5min":1.0629999999999999,"15min":0.58799999999999997},"max":{"1min":1.821,"5min":2.0609999999999999,"15min":2.6520000000000001},"last":1.589}]}]},{"osd":4,"up_from":14,"seq":60129542482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932292,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439548,"statfs":{"total":96636764160,"available":94658097152,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":1.179}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":1.2290000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.752},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.2729999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5649999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.319},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.403}]}]},{"osd":3,"up_from":14,"seq":60129542484,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872976,"kb_used_data":1751624,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498864,"statfs":{"total":96636764160,"available":94718836736,"internally_reserved":0,"allocated":1793662976,"data_stored":1792827126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.68899999999999995,"5min":0.58499999999999996,"15min":0.34699999999999998},"max":{"1min":1.3640000000000001,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":0.746},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.78500000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.98199999999999998,"15min":1.0249999999999999},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":1.359,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":1.718},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2569999999999999,"15min":1.226},"min":{"1min":0.64000000000000001,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.7909999999999999,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":0.67100000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.244,"15min":1.26},"min":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.36699999999999999},"max":{"1min":1.577,"5min":2.258,"15min":3.5249999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.45200000000000001},"max":{"1min":1.897,"5min":2.339,"15min":4.6529999999999996},"last":0.97199999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.323,"15min":1.319},"min":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":0.44500000000000001},"max":{"1min":1.6759999999999999,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.73299999999999998},"max":{"1min":1.861,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.6040000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4890000000000001,"15min":1.5009999999999999},"min":{"1min":1.224,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":1.738,"5min":2.0249999999999999,"15min":4.484},"last":1.238},{"interface":"front","average":{"1min":1.419,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.66500000000000004},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.49}]},{"osd":6,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5389999999999999,"15min":1.5620000000000001},"min":{"1min":1.1220000000000001,"5min":0.92700000000000005,"15min":0.71499999999999997},"max":{"1min":1.96,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.663},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6080000000000001,"15min":1.613},"min":{"1min":1.1399999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.069,"5min":2.169,"15min":4.5880000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5329999999999999,"15min":1.5509999999999999},"min":{"1min":0.98799999999999999,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":1.8480000000000001,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.583},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.6299999999999999,"15min":1.6279999999999999},"min":{"1min":1.2669999999999999,"5min":1.008,"15min":0.64700000000000002},"max":{"1min":2.1160000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.5429999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542482,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251776,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":115774,"kb_avail":93120064,"statfs":{"total":96636764160,"available":95354945536,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118553029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.90300000000000002,"15min":0.93600000000000005},"min":{"1min":0.56899999999999995,"5min":0.54600000000000004,"15min":0.36099999999999999},"max":{"1min":1.379,"5min":1.6659999999999999,"15min":9.657},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.004,"15min":1.0740000000000001},"min":{"1min":0.60599999999999998,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.3460000000000001,"5min":1.7649999999999999,"15min":20.407},"last":0.97299999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93500000000000005,"15min":1.0429999999999999},"min":{"1min":0.72299999999999998,"5min":0.64000000000000001,"15min":0.41499999999999998},"max":{"1min":1.5229999999999999,"5min":1.53,"15min":11.406000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.319,"15min":1.409},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.45300000000000001},"max":{"1min":1.7050000000000001,"5min":1.8049999999999999,"15min":20.513000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.389,"15min":1.371},"min":{"1min":0.95099999999999996,"5min":0.56100000000000005,"15min":0.53400000000000003},"max":{"1min":1.726,"5min":2.0110000000000001,"15min":3.1080000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.288,"15min":1.3320000000000001},"min":{"1min":0.88200000000000001,"5min":0.68200000000000005,"15min":0.49099999999999999},"max":{"1min":1.7190000000000001,"5min":1.7949999999999999,"15min":9.7390000000000008},"last":1.288}]},{"osd":4,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.494,"15min":1.46},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.66100000000000003},"max":{"1min":1.885,"5min":12.669,"15min":12.779},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3560000000000001,"15min":1.466},"min":{"1min":1.018,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":25.661000000000001},"last":1.327}]},{"osd":5,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.446,"15min":1.448},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.68300000000000005},"max":{"1min":1.8440000000000001,"5min":1.9039999999999999,"15min":3.0209999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.5369999999999999},"min":{"1min":1.0860000000000001,"5min":0.90200000000000002,"15min":0.59799999999999998},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":9.7850000000000001},"last":1.173}]},{"osd":6,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.462,"15min":1.651},"min":{"1min":1.075,"5min":1.048,"15min":0.57299999999999995},"max":{"1min":1.8320000000000001,"5min":1.8660000000000001,"15min":25.603999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4370000000000001,"15min":1.518},"min":{"1min":0.752,"5min":0.751,"15min":0.629},"max":{"1min":2.0019999999999998,"5min":2.077,"15min":15.968999999999999},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.655,"15min":1.698},"min":{"1min":1.173,"5min":1.173,"15min":0.61699999999999999},"max":{"1min":1.8220000000000001,"5min":9.3900000000000006,"15min":24.509},"last":1.129},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5489999999999999,"15min":1.6259999999999999},"min":{"1min":0.93799999999999994,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":2.081,"5min":2.161,"15min":10.455},"last":1.363}]}]},{"osd":1,"up_from":14,"seq":60129542482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1448324,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92923516,"statfs":{"total":96636764160,"available":95153680384,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.90400000000000003,"15min":0.93000000000000005},"min":{"1min":0.64800000000000002,"5min":0.53000000000000003,"15min":0.28399999999999997},"max":{"1min":1.6100000000000001,"5min":1.718,"15min":4.266},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1220000000000001,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.34999999999999998},"max":{"1min":2.153,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.137,"15min":1.135},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.31},"max":{"1min":1.647,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.39300000000000002},"max":{"1min":1.579,"5min":1.8340000000000001,"15min":4.1559999999999997},"last":1.2050000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.456,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.439},"max":{"1min":1.7529999999999999,"5min":1.9930000000000001,"15min":4.1980000000000004},"last":1.105},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.46600000000000003},"max":{"1min":1.798,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":0.61099999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5349999999999999,"15min":1.5109999999999999},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.69899999999999995},"max":{"1min":1.887,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.528,"15min":1.52},"min":{"1min":1.0309999999999999,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":1.613,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.1759999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.522,"15min":1.466},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":1.9219999999999999,"5min":5.194,"15min":5.194},"last":1.034},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":1.714,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.0600000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.516,"15min":1.5189999999999999},"min":{"1min":1.0549999999999999,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.544,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.264},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6539999999999999,"15min":1.6080000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.70699999999999996},"max":{"1min":2.0659999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.83799999999999997}]},{"osd":7,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.609,"15min":1.589},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.69999999999999996},"max":{"1min":1.9690000000000001,"5min":16.140000000000001,"15min":16.140000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6759999999999999,"15min":1.6339999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.78700000000000003},"max":{"1min":2.4449999999999998,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.319}]}]},{"osd":0,"up_from":14,"seq":60129542483,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073492,"kb_used_data":1997964,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298348,"statfs":{"total":96636764160,"available":94513508352,"internally_reserved":0,"allocated":2045915136,"data_stored":2045075445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0309999999999999,"15min":1.0249999999999999},"min":{"1min":0.55500000000000005,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.3260000000000001,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.2370000000000001,"15min":1.252},"min":{"1min":0.497,"5min":0.47999999999999998,"15min":0.33900000000000002},"max":{"1min":1.4470000000000001,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1120000000000001,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.46700000000000003,"15min":0.40000000000000002},"max":{"1min":1.5800000000000001,"5min":1.875,"15min":7.5940000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1299999999999999,"15min":1.175},"min":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.39600000000000002},"max":{"1min":1.665,"5min":1.8,"15min":5.3550000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.37},"max":{"1min":1.2170000000000001,"5min":5.851,"15min":5.851},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.335,"5min":1.3169999999999999,"15min":1.343},"min":{"1min":0.65800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.5660000000000001,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":0.83199999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.304,"15min":1.3400000000000001},"min":{"1min":0.80400000000000005,"5min":0.79600000000000004,"15min":0.69099999999999995},"max":{"1min":1.7829999999999999,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.4530000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":1.7330000000000001,"5min":2.141,"15min":5.3109999999999999},"last":0.91800000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.4670000000000001,"15min":1.4930000000000001},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.75,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.427,"15min":1.454},"min":{"1min":0.96599999999999997,"5min":0.84699999999999998,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":2.27,"15min":8.3949999999999996},"last":0.94999999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5840000000000001,"15min":1.659},"min":{"1min":1.0669999999999999,"5min":0.82299999999999995,"15min":0.72999999999999998},"max":{"1min":1.8049999999999999,"5min":2.173,"15min":15.159000000000001},"last":1.054},{"interface":"front","average":{"1min":1.379,"5min":1.526,"15min":1.6259999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.58999999999999997},"max":{"1min":1.8700000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.236}]},{"osd":7,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.617,"15min":1.667},"min":{"1min":1,"5min":0.999,"15min":0.60899999999999999},"max":{"1min":1.819,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":1.018,"5min":0.873,"15min":0.65800000000000003},"max":{"1min":1.8340000000000001,"5min":2.2719999999999998,"15min":6.149},"last":1.006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1239741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1239699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:27:24.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:27:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.939+0000 7f1adf200640 1 Processor -- start 2024-01-16T15:27:24.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.940+0000 7f1adf200640 1 -- start start 2024-01-16T15:27:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.941+0000 7f1adf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 0x7f1ad80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.941+0000 7f1adf200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.941+0000 7f1adf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad815fa70 con 0x7f1ad813dfd0 2024-01-16T15:27:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.941+0000 7f1adf200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad80cfa40 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.941+0000 7f1adf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad815c580 con 0x7f1ad80cf050 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53808/0 (socket says 172.21.15.43:53808) 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 -- 172.21.15.43:0/3710347602 learned_addr learned my addr 172.21.15.43:0/3710347602 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1add400640 1 --2- 172.21.15.43:0/3710347602 >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 0x7f1ad80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 -- 172.21.15.43:0/3710347602 >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 msgr2=0x7f1ad80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 --2- 172.21.15.43:0/3710347602 >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 0x7f1ad80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 -- 172.21.15.43:0/3710347602 >> v1:172.21.15.69:6789/0 conn(0x7f1ad813dfd0 legacy=0x7f1ad815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1ade800640 1 -- 172.21.15.43:0/3710347602 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8140900 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.942+0000 7f1add400640 1 --2- 172.21.15.43:0/3710347602 >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 0x7f1ad80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.943+0000 7f1ade800640 1 --2- 172.21.15.43:0/3710347602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f1ad806e070 tx=0x7f1ad002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf27605807b2466a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.944+0000 7f1adca00640 1 -- 172.21.15.43:0/3710347602 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad002cb10 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.944+0000 7f1adca00640 1 -- 172.21.15.43:0/3710347602 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad002cc70 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.944+0000 7f1adca00640 1 -- 172.21.15.43:0/3710347602 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00343f0 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.944+0000 7f1adf200640 1 -- 172.21.15.43:0/3710347602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad80cd7b0 msgr2=0x7f1ad80cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.944+0000 7f1adf200640 1 --2- 172.21.15.43:0/3710347602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f1ad806e070 tx=0x7f1ad002bf70 comp rx=0 tx=0).stop 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 -- 172.21.15.43:0/3710347602 shutdown_connections 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 --2- 172.21.15.43:0/3710347602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80cdbb0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 --2- 172.21.15.43:0/3710347602 >> v2:172.21.15.69:3300/0 conn(0x7f1ad80cf050 0x7f1ad80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 -- 172.21.15.43:0/3710347602 >> 172.21.15.43:0/3710347602 conn(0x7f1ad806a860 msgr2=0x7f1ad806ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 -- 172.21.15.43:0/3710347602 shutdown_connections 2024-01-16T15:27:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.945+0000 7f1adf200640 1 -- 172.21.15.43:0/3710347602 wait complete. 2024-01-16T15:27:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.947+0000 7f1adf200640 1 Processor -- start 2024-01-16T15:27:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.947+0000 7f1adf200640 1 -- start start 2024-01-16T15:27:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 0x7f1ad80c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80cb1e0 con 0x7f1ad813dfd0 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80ca140 con 0x7f1ad80cd7b0 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adf200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80ca420 con 0x7f1ad80cf050 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1ade800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1ade800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53824/0 (socket says 172.21.15.43:53824) 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1ade800640 1 -- 172.21.15.43:0/376010982 learned_addr learned my addr 172.21.15.43:0/376010982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1adde00640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.948+0000 7f1add400640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 0x7f1ad80c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 -- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 msgr2=0x7f1ad80c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 0x7f1ad80c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 -- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad80cd7b0 msgr2=0x7f1ad80c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 -- 172.21.15.43:0/376010982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009b30 con 0x7f1ad813dfd0 2024-01-16T15:27:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1add400640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 0x7f1ad80c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ade800640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0036040 tx=0x7f1ad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad003e450 con 0x7f1ad813dfd0 2024-01-16T15:27:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad003e5b0 con 0x7f1ad813dfd0 2024-01-16T15:27:24.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad003e960 con 0x7f1ad813dfd0 2024-01-16T15:27:24.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.949+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80ca700 con 0x7f1ad813dfd0 2024-01-16T15:27:24.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.950+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad80cacc0 con 0x7f1ad813dfd0 2024-01-16T15:27:24.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.951+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad80caeb0 con 0x7f1ad813dfd0 2024-01-16T15:27:24.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.952+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f1ad003eac0 con 0x7f1ad813dfd0 2024-01-16T15:27:24.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.953+0000 7f1ad6000640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 0x7f1ab006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:24.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.953+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1ad00abb10 con 0x7f1ad813dfd0 2024-01-16T15:27:24.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.953+0000 7f1adde00640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 0x7f1ab006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:24.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.954+0000 7f1adde00640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 0x7f1ab006e980 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007fd0 tx=0x7f1ac8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:24.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:24.957+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad0078180 con 0x7f1ad813dfd0 2024-01-16T15:27:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.147+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ad80b3bf0 con 0x7f1ab006c4c0 2024-01-16T15:27:25.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.161+0000 7f1ad6000640 1 -- 172.21.15.43:0/376010982 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289534 (secure 0 0 0) 0x7f1ad80b3bf0 con 0x7f1ab006c4c0 2024-01-16T15:27:25.164 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:25.165 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:27:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.167+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 msgr2=0x7f1ab006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.167+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 0x7f1ab006e980 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007fd0 tx=0x7f1ac8009040 comp rx=0 tx=0).stop 2024-01-16T15:27:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 msgr2=0x7f1ad8156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0036040 tx=0x7f1ad0004a40 comp rx=0 tx=0).stop 2024-01-16T15:27:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 shutdown_connections 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1ab006c4c0 0x7f1ab006e980 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ad813dfd0 0x7f1ad8156c10 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad80cf050 0x7f1ad80c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 --2- 172.21.15.43:0/376010982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad80cd7b0 0x7f1ad80c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 >> 172.21.15.43:0/376010982 conn(0x7f1ad806a860 msgr2=0x7f1ad815ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 shutdown_connections 2024-01-16T15:27:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.168+0000 7f1adf200640 1 -- 172.21.15.43:0/376010982 wait complete. 2024-01-16T15:27:25.187 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":950,"stamp":"2024-01-16T15:27:23.502087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444287485,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2251,"num_write_kb":6295365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4682,"ondisk_log_size":4682,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13389440,"kb_used_data":12597760,"kb_used_omap":12,"kb_used_meta":791603,"kb_avail":741585280,"statfs":{"total":773094113280,"available":759383326720,"internally_reserved":0,"allocated":12900106240,"data_stored":12893524122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":810602028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073783661,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":276,"num_write_kb":1048624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005047"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737744+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737744+0000","last_peered":"2024-01-16T15:27:17.737744+0000","last_clean":"2024-01-16T15:27:17.737744+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737744+0000","last_undegraded":"2024-01-16T15:27:17.737744+0000","last_fullsized":"2024-01-16T15:27:17.737744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:47.552196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'69","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736614+0000","last_change":"2024-01-16T15:18:50.514868+0000","last_active":"2024-01-16T15:27:17.736614+0000","last_peered":"2024-01-16T15:27:17.736614+0000","last_clean":"2024-01-16T15:27:17.736614+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:17.736614+0000","last_undegraded":"2024-01-16T15:27:17.736614+0000","last_fullsized":"2024-01-16T15:27:17.736614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:22.704331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.743697+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:27:17.743697+0000","last_peered":"2024-01-16T15:27:17.743697+0000","last_clean":"2024-01-16T15:27:17.743697+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:27:17.743697+0000","last_undegraded":"2024-01-16T15:27:17.743697+0000","last_fullsized":"2024-01-16T15:27:17.743697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733813+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733813+0000","last_peered":"2024-01-16T15:27:17.733813+0000","last_clean":"2024-01-16T15:27:17.733813+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733813+0000","last_undegraded":"2024-01-16T15:27:17.733813+0000","last_fullsized":"2024-01-16T15:27:17.733813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:27.895459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'60","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717706+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:27:17.717706+0000","last_peered":"2024-01-16T15:27:17.717706+0000","last_clean":"2024-01-16T15:27:17.717706+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:17.717706+0000","last_undegraded":"2024-01-16T15:27:17.717706+0000","last_fullsized":"2024-01-16T15:27:17.717706+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:32.606162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'74","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721540+0000","last_change":"2024-01-16T15:18:50.517889+0000","last_active":"2024-01-16T15:27:17.721540+0000","last_peered":"2024-01-16T15:27:17.721540+0000","last_clean":"2024-01-16T15:27:17.721540+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:17.721540+0000","last_undegraded":"2024-01-16T15:27:17.721540+0000","last_fullsized":"2024-01-16T15:27:17.721540+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'66","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734633+0000","last_change":"2024-01-16T15:18:50.506124+0000","last_active":"2024-01-16T15:27:17.734633+0000","last_peered":"2024-01-16T15:27:17.734633+0000","last_clean":"2024-01-16T15:27:17.734633+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:17.734633+0000","last_undegraded":"2024-01-16T15:27:17.734633+0000","last_fullsized":"2024-01-16T15:27:17.734633+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'54","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730876+0000","last_change":"2024-01-16T15:18:50.513082+0000","last_active":"2024-01-16T15:27:17.730876+0000","last_peered":"2024-01-16T15:27:17.730876+0000","last_clean":"2024-01-16T15:27:17.730876+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:17.730876+0000","last_undegraded":"2024-01-16T15:27:17.730876+0000","last_fullsized":"2024-01-16T15:27:17.730876+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724783+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.724783+0000","last_peered":"2024-01-16T15:27:17.724783+0000","last_clean":"2024-01-16T15:27:17.724783+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.724783+0000","last_undegraded":"2024-01-16T15:27:17.724783+0000","last_fullsized":"2024-01-16T15:27:17.724783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:57.207916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732638+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.732638+0000","last_peered":"2024-01-16T15:27:17.732638+0000","last_clean":"2024-01-16T15:27:17.732638+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.732638+0000","last_undegraded":"2024-01-16T15:27:17.732638+0000","last_fullsized":"2024-01-16T15:27:17.732638+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:26.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'70","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.727985+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:27:17.727985+0000","last_peered":"2024-01-16T15:27:17.727985+0000","last_clean":"2024-01-16T15:27:17.727985+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:17.727985+0000","last_undegraded":"2024-01-16T15:27:17.727985+0000","last_fullsized":"2024-01-16T15:27:17.727985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735505+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735505+0000","last_peered":"2024-01-16T15:27:17.735505+0000","last_clean":"2024-01-16T15:27:17.735505+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735505+0000","last_undegraded":"2024-01-16T15:27:17.735505+0000","last_fullsized":"2024-01-16T15:27:17.735505+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:28:07.149636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'57","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734731+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:27:17.734731+0000","last_peered":"2024-01-16T15:27:17.734731+0000","last_clean":"2024-01-16T15:27:17.734731+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:27:17.734731+0000","last_undegraded":"2024-01-16T15:27:17.734731+0000","last_fullsized":"2024-01-16T15:27:17.734731+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:35.209083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'65","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739366+0000","last_change":"2024-01-16T15:18:50.517177+0000","last_active":"2024-01-16T15:27:17.739366+0000","last_peered":"2024-01-16T15:27:17.739366+0000","last_clean":"2024-01-16T15:27:17.739366+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:17.739366+0000","last_undegraded":"2024-01-16T15:27:17.739366+0000","last_fullsized":"2024-01-16T15:27:17.739366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'54","reported_seq":77,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724971+0000","last_change":"2024-01-16T15:18:50.518134+0000","last_active":"2024-01-16T15:27:17.724971+0000","last_peered":"2024-01-16T15:27:17.724971+0000","last_clean":"2024-01-16T15:27:17.724971+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:17.724971+0000","last_undegraded":"2024-01-16T15:27:17.724971+0000","last_fullsized":"2024-01-16T15:27:17.724971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:10.936165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738723+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:27:17.738723+0000","last_peered":"2024-01-16T15:27:17.738723+0000","last_clean":"2024-01-16T15:27:17.738723+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:17.738723+0000","last_undegraded":"2024-01-16T15:27:17.738723+0000","last_fullsized":"2024-01-16T15:27:17.738723+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:18.639887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'49","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733966+0000","last_change":"2024-01-16T15:18:50.514080+0000","last_active":"2024-01-16T15:27:17.733966+0000","last_peered":"2024-01-16T15:27:17.733966+0000","last_clean":"2024-01-16T15:27:17.733966+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:17.733966+0000","last_undegraded":"2024-01-16T15:27:17.733966+0000","last_fullsized":"2024-01-16T15:27:17.733966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:18.432791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'72","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735139+0000","last_change":"2024-01-16T15:18:50.512290+0000","last_active":"2024-01-16T15:27:17.735139+0000","last_peered":"2024-01-16T15:27:17.735139+0000","last_clean":"2024-01-16T15:27:17.735139+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:17.735139+0000","last_undegraded":"2024-01-16T15:27:17.735139+0000","last_fullsized":"2024-01-16T15:27:17.735139+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:03.196990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'71","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741586+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:27:17.741586+0000","last_peered":"2024-01-16T15:27:17.741586+0000","last_clean":"2024-01-16T15:27:17.741586+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:27:17.741586+0000","last_undegraded":"2024-01-16T15:27:17.741586+0000","last_fullsized":"2024-01-16T15:27:17.741586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.753500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734862+0000","last_change":"2024-01-16T15:18:50.523238+0000","last_active":"2024-01-16T15:27:17.734862+0000","last_peered":"2024-01-16T15:27:17.734862+0000","last_clean":"2024-01-16T15:27:17.734862+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:17.734862+0000","last_undegraded":"2024-01-16T15:27:17.734862+0000","last_fullsized":"2024-01-16T15:27:17.734862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:10.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'73","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742649+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:27:17.742649+0000","last_peered":"2024-01-16T15:27:17.742649+0000","last_clean":"2024-01-16T15:27:17.742649+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:27:17.742649+0000","last_undegraded":"2024-01-16T15:27:17.742649+0000","last_fullsized":"2024-01-16T15:27:17.742649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:04.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'64","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717654+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:27:17.717654+0000","last_peered":"2024-01-16T15:27:17.717654+0000","last_clean":"2024-01-16T15:27:17.717654+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:27:17.717654+0000","last_undegraded":"2024-01-16T15:27:17.717654+0000","last_fullsized":"2024-01-16T15:27:17.717654+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'57","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739771+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:27:17.739771+0000","last_peered":"2024-01-16T15:27:17.739771+0000","last_clean":"2024-01-16T15:27:17.739771+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:17.739771+0000","last_undegraded":"2024-01-16T15:27:17.739771+0000","last_fullsized":"2024-01-16T15:27:17.739771+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:49.070110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'70","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717432+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:27:17.717432+0000","last_peered":"2024-01-16T15:27:17.717432+0000","last_clean":"2024-01-16T15:27:17.717432+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:27:17.717432+0000","last_undegraded":"2024-01-16T15:27:17.717432+0000","last_fullsized":"2024-01-16T15:27:17.717432+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:09.748440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'71","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740798+0000","last_change":"2024-01-16T15:18:50.513030+0000","last_active":"2024-01-16T15:27:17.740798+0000","last_peered":"2024-01-16T15:27:17.740798+0000","last_clean":"2024-01-16T15:27:17.740798+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:17.740798+0000","last_undegraded":"2024-01-16T15:27:17.740798+0000","last_fullsized":"2024-01-16T15:27:17.740798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:54.334893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'81","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736872+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:27:17.736872+0000","last_peered":"2024-01-16T15:27:17.736872+0000","last_clean":"2024-01-16T15:27:17.736872+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:27:17.736872+0000","last_undegraded":"2024-01-16T15:27:17.736872+0000","last_fullsized":"2024-01-16T15:27:17.736872+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:49.522744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'64","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730649+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:27:17.730649+0000","last_peered":"2024-01-16T15:27:17.730649+0000","last_clean":"2024-01-16T15:27:17.730649+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:27:17.730649+0000","last_undegraded":"2024-01-16T15:27:17.730649+0000","last_fullsized":"2024-01-16T15:27:17.730649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":64,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730383+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:17.730383+0000","last_peered":"2024-01-16T15:27:17.730383+0000","last_clean":"2024-01-16T15:27:17.730383+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:17.730383+0000","last_undegraded":"2024-01-16T15:27:17.730383+0000","last_fullsized":"2024-01-16T15:27:17.730383+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'57","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738407+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:27:17.738407+0000","last_peered":"2024-01-16T15:27:17.738407+0000","last_clean":"2024-01-16T15:27:17.738407+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:27:17.738407+0000","last_undegraded":"2024-01-16T15:27:17.738407+0000","last_fullsized":"2024-01-16T15:27:17.738407+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:38.948857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'76","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720265+0000","last_change":"2024-01-16T15:18:50.491931+0000","last_active":"2024-01-16T15:27:17.720265+0000","last_peered":"2024-01-16T15:27:17.720265+0000","last_clean":"2024-01-16T15:27:17.720265+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:17.720265+0000","last_undegraded":"2024-01-16T15:27:17.720265+0000","last_fullsized":"2024-01-16T15:27:17.720265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'81","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736796+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:27:17.736796+0000","last_peered":"2024-01-16T15:27:17.736796+0000","last_clean":"2024-01-16T15:27:17.736796+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:17.736796+0000","last_undegraded":"2024-01-16T15:27:17.736796+0000","last_fullsized":"2024-01-16T15:27:17.736796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:03.259116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'63","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.723981+0000","last_change":"2024-01-16T15:18:50.510830+0000","last_active":"2024-01-16T15:27:17.723981+0000","last_peered":"2024-01-16T15:27:17.723981+0000","last_clean":"2024-01-16T15:27:17.723981+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:17.723981+0000","last_undegraded":"2024-01-16T15:27:17.723981+0000","last_fullsized":"2024-01-16T15:27:17.723981+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:30.055863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'62","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721097+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:27:17.721097+0000","last_peered":"2024-01-16T15:27:17.721097+0000","last_clean":"2024-01-16T15:27:17.721097+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:17.721097+0000","last_undegraded":"2024-01-16T15:27:17.721097+0000","last_fullsized":"2024-01-16T15:27:17.721097+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476688+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:25:05.476688+0000","last_peered":"2024-01-16T15:25:05.476688+0000","last_clean":"2024-01-16T15:25:05.476688+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:25:05.476688+0000","last_undegraded":"2024-01-16T15:25:05.476688+0000","last_fullsized":"2024-01-16T15:25:05.476688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'54","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733008+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:27:17.733008+0000","last_peered":"2024-01-16T15:27:17.733008+0000","last_clean":"2024-01-16T15:27:17.733008+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:27:17.733008+0000","last_undegraded":"2024-01-16T15:27:17.733008+0000","last_fullsized":"2024-01-16T15:27:17.733008+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:37.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.466688+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:24:50.466688+0000","last_peered":"2024-01-16T15:24:50.466688+0000","last_clean":"2024-01-16T15:24:50.466688+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:24:50.466688+0000","last_undegraded":"2024-01-16T15:24:50.466688+0000","last_fullsized":"2024-01-16T15:24:50.466688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:51.215798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'71","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724610+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:27:17.724610+0000","last_peered":"2024-01-16T15:27:17.724610+0000","last_clean":"2024-01-16T15:27:17.724610+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:17.724610+0000","last_undegraded":"2024-01-16T15:27:17.724610+0000","last_fullsized":"2024-01-16T15:27:17.724610+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360466+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:23:50.360466+0000","last_peered":"2024-01-16T15:23:50.360466+0000","last_clean":"2024-01-16T15:23:50.360466+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:23:50.360466+0000","last_undegraded":"2024-01-16T15:23:50.360466+0000","last_fullsized":"2024-01-16T15:23:50.360466+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'68","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741745+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:27:17.741745+0000","last_peered":"2024-01-16T15:27:17.741745+0000","last_clean":"2024-01-16T15:27:17.741745+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:27:17.741745+0000","last_undegraded":"2024-01-16T15:27:17.741745+0000","last_fullsized":"2024-01-16T15:27:17.741745+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:37.406427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360421+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:23:50.360421+0000","last_peered":"2024-01-16T15:23:50.360421+0000","last_clean":"2024-01-16T15:23:50.360421+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:23:50.360421+0000","last_undegraded":"2024-01-16T15:23:50.360421+0000","last_fullsized":"2024-01-16T15:23:50.360421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'72","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.719815+0000","last_change":"2024-01-16T15:19:15.318178+0000","last_active":"2024-01-16T15:27:17.719815+0000","last_peered":"2024-01-16T15:27:17.719815+0000","last_clean":"2024-01-16T15:27:17.719815+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:17.719815+0000","last_undegraded":"2024-01-16T15:27:17.719815+0000","last_fullsized":"2024-01-16T15:27:17.719815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.463254+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:23:50.463254+0000","last_peered":"2024-01-16T15:23:50.463254+0000","last_clean":"2024-01-16T15:23:50.463254+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:23:50.463254+0000","last_undegraded":"2024-01-16T15:23:50.463254+0000","last_fullsized":"2024-01-16T15:23:50.463254+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:31.593211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'72","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736806+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:27:17.736806+0000","last_peered":"2024-01-16T15:27:17.736806+0000","last_clean":"2024-01-16T15:27:17.736806+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:27:17.736806+0000","last_undegraded":"2024-01-16T15:27:17.736806+0000","last_fullsized":"2024-01-16T15:27:17.736806+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:59.415403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.403153+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:23:50.403153+0000","last_peered":"2024-01-16T15:23:50.403153+0000","last_clean":"2024-01-16T15:23:50.403153+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:23:50.403153+0000","last_undegraded":"2024-01-16T15:23:50.403153+0000","last_fullsized":"2024-01-16T15:23:50.403153+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:57.932540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'73","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732782+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:27:17.732782+0000","last_peered":"2024-01-16T15:27:17.732782+0000","last_clean":"2024-01-16T15:27:17.732782+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:17.732782+0000","last_undegraded":"2024-01-16T15:27:17.732782+0000","last_fullsized":"2024-01-16T15:27:17.732782+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:45.776442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.368270+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:23:40.368270+0000","last_peered":"2024-01-16T15:23:40.368270+0000","last_clean":"2024-01-16T15:23:40.368270+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:23:40.368270+0000","last_undegraded":"2024-01-16T15:23:40.368270+0000","last_fullsized":"2024-01-16T15:23:40.368270+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:07.440889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'73","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726884+0000","last_change":"2024-01-16T15:18:50.506340+0000","last_active":"2024-01-16T15:27:17.726884+0000","last_peered":"2024-01-16T15:27:17.726884+0000","last_clean":"2024-01-16T15:27:17.726884+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:17.726884+0000","last_undegraded":"2024-01-16T15:27:17.726884+0000","last_fullsized":"2024-01-16T15:27:17.726884+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.622586+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:23:40.622586+0000","last_peered":"2024-01-16T15:23:40.622586+0000","last_clean":"2024-01-16T15:23:40.622586+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:23:40.622586+0000","last_undegraded":"2024-01-16T15:23:40.622586+0000","last_fullsized":"2024-01-16T15:23:40.622586+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722570+0000","last_change":"2024-01-16T15:18:50.507182+0000","last_active":"2024-01-16T15:27:17.722570+0000","last_peered":"2024-01-16T15:27:17.722570+0000","last_clean":"2024-01-16T15:27:17.722570+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:17.722570+0000","last_undegraded":"2024-01-16T15:27:17.722570+0000","last_fullsized":"2024-01-16T15:27:17.722570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.358584+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:23:40.358584+0000","last_peered":"2024-01-16T15:23:40.358584+0000","last_clean":"2024-01-16T15:23:40.358584+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:23:40.358584+0000","last_undegraded":"2024-01-16T15:23:40.358584+0000","last_fullsized":"2024-01-16T15:23:40.358584+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'61","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738962+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:27:17.738962+0000","last_peered":"2024-01-16T15:27:17.738962+0000","last_clean":"2024-01-16T15:27:17.738962+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:17.738962+0000","last_undegraded":"2024-01-16T15:27:17.738962+0000","last_fullsized":"2024-01-16T15:27:17.738962+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:14.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.401032+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:23:40.401032+0000","last_peered":"2024-01-16T15:23:40.401032+0000","last_clean":"2024-01-16T15:23:40.401032+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:23:40.401032+0000","last_undegraded":"2024-01-16T15:23:40.401032+0000","last_fullsized":"2024-01-16T15:23:40.401032+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:31.105453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726260+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:17.726260+0000","last_peered":"2024-01-16T15:27:17.726260+0000","last_clean":"2024-01-16T15:27:17.726260+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:17.726260+0000","last_undegraded":"2024-01-16T15:27:17.726260+0000","last_fullsized":"2024-01-16T15:27:17.726260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'33","reported_seq":50,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.122833+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:25:12.122833+0000","last_peered":"2024-01-16T15:25:12.122833+0000","last_clean":"2024-01-16T15:25:12.122833+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:25:12.122833+0000","last_undegraded":"2024-01-16T15:25:12.122833+0000","last_fullsized":"2024-01-16T15:25:12.122833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'69","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.729047+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:27:17.729047+0000","last_peered":"2024-01-16T15:27:17.729047+0000","last_clean":"2024-01-16T15:27:17.729047+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:17.729047+0000","last_undegraded":"2024-01-16T15:27:17.729047+0000","last_fullsized":"2024-01-16T15:27:17.729047+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'321","reported_seq":338,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.747172+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:27:17.747172+0000","last_peered":"2024-01-16T15:27:17.747172+0000","last_clean":"2024-01-16T15:27:17.747172+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:17.747172+0000","last_undegraded":"2024-01-16T15:27:17.747172+0000","last_fullsized":"2024-01-16T15:27:17.747172+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239629,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.718504+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:17.718504+0000","last_peered":"2024-01-16T15:27:17.718504+0000","last_clean":"2024-01-16T15:27:17.718504+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:17.718504+0000","last_undegraded":"2024-01-16T15:27:17.718504+0000","last_fullsized":"2024-01-16T15:27:17.718504+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.500458+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:23:30.500458+0000","last_peered":"2024-01-16T15:23:30.500458+0000","last_clean":"2024-01-16T15:23:30.500458+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:23:30.500458+0000","last_undegraded":"2024-01-16T15:23:30.500458+0000","last_fullsized":"2024-01-16T15:23:30.500458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'73","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730523+0000","last_change":"2024-01-16T15:18:50.517675+0000","last_active":"2024-01-16T15:27:17.730523+0000","last_peered":"2024-01-16T15:27:17.730523+0000","last_clean":"2024-01-16T15:27:17.730523+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:17.730523+0000","last_undegraded":"2024-01-16T15:27:17.730523+0000","last_fullsized":"2024-01-16T15:27:17.730523+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.440394+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:23:30.440394+0000","last_peered":"2024-01-16T15:23:30.440394+0000","last_clean":"2024-01-16T15:23:30.440394+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:23:30.440394+0000","last_undegraded":"2024-01-16T15:23:30.440394+0000","last_fullsized":"2024-01-16T15:23:30.440394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:28.315056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'63","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740737+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:27:17.740737+0000","last_peered":"2024-01-16T15:27:17.740737+0000","last_clean":"2024-01-16T15:27:17.740737+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:27:17.740737+0000","last_undegraded":"2024-01-16T15:27:17.740737+0000","last_fullsized":"2024-01-16T15:27:17.740737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":25,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.356549+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:23:30.356549+0000","last_peered":"2024-01-16T15:23:30.356549+0000","last_clean":"2024-01-16T15:23:30.356549+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:23:30.356549+0000","last_undegraded":"2024-01-16T15:23:30.356549+0000","last_fullsized":"2024-01-16T15:23:30.356549+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:58.071689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'46","reported_seq":68,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735009+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:27:17.735009+0000","last_peered":"2024-01-16T15:27:17.735009+0000","last_clean":"2024-01-16T15:27:17.735009+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:27:17.735009+0000","last_undegraded":"2024-01-16T15:27:17.735009+0000","last_fullsized":"2024-01-16T15:27:17.735009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:40.855053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.618711+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:23:20.618711+0000","last_peered":"2024-01-16T15:23:20.618711+0000","last_clean":"2024-01-16T15:23:20.618711+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:23:20.618711+0000","last_undegraded":"2024-01-16T15:23:20.618711+0000","last_fullsized":"2024-01-16T15:23:20.618711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734412+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:17.734412+0000","last_peered":"2024-01-16T15:27:17.734412+0000","last_clean":"2024-01-16T15:27:17.734412+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:17.734412+0000","last_undegraded":"2024-01-16T15:27:17.734412+0000","last_fullsized":"2024-01-16T15:27:17.734412+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.354766+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:23:20.354766+0000","last_peered":"2024-01-16T15:23:20.354766+0000","last_clean":"2024-01-16T15:23:20.354766+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:23:20.354766+0000","last_undegraded":"2024-01-16T15:23:20.354766+0000","last_fullsized":"2024-01-16T15:23:20.354766+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738127+0000","last_change":"2024-01-16T15:18:50.517233+0000","last_active":"2024-01-16T15:27:17.738127+0000","last_peered":"2024-01-16T15:27:17.738127+0000","last_clean":"2024-01-16T15:27:17.738127+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:17.738127+0000","last_undegraded":"2024-01-16T15:27:17.738127+0000","last_fullsized":"2024-01-16T15:27:17.738127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.438493+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:23:20.438493+0000","last_peered":"2024-01-16T15:23:20.438493+0000","last_clean":"2024-01-16T15:23:20.438493+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:23:20.438493+0000","last_undegraded":"2024-01-16T15:23:20.438493+0000","last_fullsized":"2024-01-16T15:23:20.438493+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:04.731539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744447+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:27:17.744447+0000","last_peered":"2024-01-16T15:27:17.744447+0000","last_clean":"2024-01-16T15:27:17.744447+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:27:17.744447+0000","last_undegraded":"2024-01-16T15:27:17.744447+0000","last_fullsized":"2024-01-16T15:27:17.744447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:16.245190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.457791+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:23:20.457791+0000","last_peered":"2024-01-16T15:23:20.457791+0000","last_clean":"2024-01-16T15:23:20.457791+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:23:20.457791+0000","last_undegraded":"2024-01-16T15:23:20.457791+0000","last_fullsized":"2024-01-16T15:23:20.457791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'56","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741337+0000","last_change":"2024-01-16T15:18:50.521667+0000","last_active":"2024-01-16T15:27:17.741337+0000","last_peered":"2024-01-16T15:27:17.741337+0000","last_clean":"2024-01-16T15:27:17.741337+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:17.741337+0000","last_undegraded":"2024-01-16T15:27:17.741337+0000","last_fullsized":"2024-01-16T15:27:17.741337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.498582+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:23:20.498582+0000","last_peered":"2024-01-16T15:23:20.498582+0000","last_clean":"2024-01-16T15:23:20.498582+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:23:20.498582+0000","last_undegraded":"2024-01-16T15:23:20.498582+0000","last_fullsized":"2024-01-16T15:23:20.498582+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738717+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:27:17.738717+0000","last_peered":"2024-01-16T15:27:17.738717+0000","last_clean":"2024-01-16T15:27:17.738717+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:27:17.738717+0000","last_undegraded":"2024-01-16T15:27:17.738717+0000","last_fullsized":"2024-01-16T15:27:17.738717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:59.502905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.353213+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:23:10.353213+0000","last_peered":"2024-01-16T15:23:10.353213+0000","last_clean":"2024-01-16T15:23:10.353213+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:23:10.353213+0000","last_undegraded":"2024-01-16T15:23:10.353213+0000","last_fullsized":"2024-01-16T15:23:10.353213+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:30.184973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'65","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735772+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:27:17.735772+0000","last_peered":"2024-01-16T15:27:17.735772+0000","last_clean":"2024-01-16T15:27:17.735772+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:27:17.735772+0000","last_undegraded":"2024-01-16T15:27:17.735772+0000","last_fullsized":"2024-01-16T15:27:17.735772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.496763+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:23:10.496763+0000","last_peered":"2024-01-16T15:23:10.496763+0000","last_clean":"2024-01-16T15:23:10.496763+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:23:10.496763+0000","last_undegraded":"2024-01-16T15:23:10.496763+0000","last_fullsized":"2024-01-16T15:23:10.496763+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'57","reported_seq":83,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737237+0000","last_change":"2024-01-16T15:18:50.516592+0000","last_active":"2024-01-16T15:27:17.737237+0000","last_peered":"2024-01-16T15:27:17.737237+0000","last_clean":"2024-01-16T15:27:17.737237+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:17.737237+0000","last_undegraded":"2024-01-16T15:27:17.737237+0000","last_fullsized":"2024-01-16T15:27:17.737237+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:17.446551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.615833+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:23:10.615833+0000","last_peered":"2024-01-16T15:23:10.615833+0000","last_clean":"2024-01-16T15:23:10.615833+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:23:10.615833+0000","last_undegraded":"2024-01-16T15:23:10.615833+0000","last_fullsized":"2024-01-16T15:23:10.615833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'53","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739953+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:27:17.739953+0000","last_peered":"2024-01-16T15:27:17.739953+0000","last_clean":"2024-01-16T15:27:17.739953+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:17.739953+0000","last_undegraded":"2024-01-16T15:27:17.739953+0000","last_fullsized":"2024-01-16T15:27:17.739953+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:09.913343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.116830+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:25:12.116830+0000","last_peered":"2024-01-16T15:25:12.116830+0000","last_clean":"2024-01-16T15:25:12.116830+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:25:12.116830+0000","last_undegraded":"2024-01-16T15:25:12.116830+0000","last_fullsized":"2024-01-16T15:25:12.116830+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:17.281805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741249+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:17.741249+0000","last_peered":"2024-01-16T15:27:17.741249+0000","last_clean":"2024-01-16T15:27:17.741249+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:17.741249+0000","last_undegraded":"2024-01-16T15:27:17.741249+0000","last_fullsized":"2024-01-16T15:27:17.741249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'30","reported_seq":47,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:14.187594+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:27:14.187594+0000","last_peered":"2024-01-16T15:27:14.187594+0000","last_clean":"2024-01-16T15:27:14.187594+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:14.187594+0000","last_undegraded":"2024-01-16T15:27:14.187594+0000","last_fullsized":"2024-01-16T15:27:14.187594+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:07.461774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'69","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735893+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:27:17.735893+0000","last_peered":"2024-01-16T15:27:17.735893+0000","last_clean":"2024-01-16T15:27:17.735893+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:27:17.735893+0000","last_undegraded":"2024-01-16T15:27:17.735893+0000","last_fullsized":"2024-01-16T15:27:17.735893+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:49.736642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.641840+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:25:05.641840+0000","last_peered":"2024-01-16T15:25:05.641840+0000","last_clean":"2024-01-16T15:25:05.641840+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:25:05.641840+0000","last_undegraded":"2024-01-16T15:25:05.641840+0000","last_fullsized":"2024-01-16T15:25:05.641840+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'59","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738409+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:27:17.738409+0000","last_peered":"2024-01-16T15:27:17.738409+0000","last_clean":"2024-01-16T15:27:17.738409+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:27:17.738409+0000","last_undegraded":"2024-01-16T15:27:17.738409+0000","last_fullsized":"2024-01-16T15:27:17.738409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:27.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.417626+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:25:05.417626+0000","last_peered":"2024-01-16T15:25:05.417626+0000","last_clean":"2024-01-16T15:25:05.417626+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:25:05.417626+0000","last_undegraded":"2024-01-16T15:25:05.417626+0000","last_fullsized":"2024-01-16T15:25:05.417626+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.618165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'63","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736683+0000","last_change":"2024-01-16T15:18:50.516022+0000","last_active":"2024-01-16T15:27:17.736683+0000","last_peered":"2024-01-16T15:27:17.736683+0000","last_clean":"2024-01-16T15:27:17.736683+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:17.736683+0000","last_undegraded":"2024-01-16T15:27:17.736683+0000","last_fullsized":"2024-01-16T15:27:17.736683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476614+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:25:05.476614+0000","last_peered":"2024-01-16T15:25:05.476614+0000","last_clean":"2024-01-16T15:25:05.476614+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:25:05.476614+0000","last_undegraded":"2024-01-16T15:25:05.476614+0000","last_fullsized":"2024-01-16T15:25:05.476614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:39.842102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742798+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:27:17.742798+0000","last_peered":"2024-01-16T15:27:17.742798+0000","last_clean":"2024-01-16T15:27:17.742798+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:27:17.742798+0000","last_undegraded":"2024-01-16T15:27:17.742798+0000","last_fullsized":"2024-01-16T15:27:17.742798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:03.949006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.384697+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:25:05.384697+0000","last_peered":"2024-01-16T15:25:05.384697+0000","last_clean":"2024-01-16T15:25:05.384697+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:25:05.384697+0000","last_undegraded":"2024-01-16T15:25:05.384697+0000","last_fullsized":"2024-01-16T15:25:05.384697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:26.402495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'68","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738935+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:27:17.738935+0000","last_peered":"2024-01-16T15:27:17.738935+0000","last_clean":"2024-01-16T15:27:17.738935+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:27:17.738935+0000","last_undegraded":"2024-01-16T15:27:17.738935+0000","last_fullsized":"2024-01-16T15:27:17.738935+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'34","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.114659+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:25:12.114659+0000","last_peered":"2024-01-16T15:25:12.114659+0000","last_clean":"2024-01-16T15:25:12.114659+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:25:12.114659+0000","last_undegraded":"2024-01-16T15:25:12.114659+0000","last_fullsized":"2024-01-16T15:25:12.114659+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733937+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:27:17.733937+0000","last_peered":"2024-01-16T15:27:17.733937+0000","last_clean":"2024-01-16T15:27:17.733937+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:27:17.733937+0000","last_undegraded":"2024-01-16T15:27:17.733937+0000","last_fullsized":"2024-01-16T15:27:17.733937+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:01.372900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.638362+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:24:50.638362+0000","last_peered":"2024-01-16T15:24:50.638362+0000","last_clean":"2024-01-16T15:24:50.638362+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:24:50.638362+0000","last_undegraded":"2024-01-16T15:24:50.638362+0000","last_fullsized":"2024-01-16T15:24:50.638362+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'49","reported_seq":78,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.712896+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:27:17.712896+0000","last_peered":"2024-01-16T15:27:17.712896+0000","last_clean":"2024-01-16T15:27:17.712896+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:17.712896+0000","last_undegraded":"2024-01-16T15:27:17.712896+0000","last_fullsized":"2024-01-16T15:27:17.712896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.459727+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:23:30.459727+0000","last_peered":"2024-01-16T15:23:30.459727+0000","last_clean":"2024-01-16T15:23:30.459727+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:23:30.459727+0000","last_undegraded":"2024-01-16T15:23:30.459727+0000","last_fullsized":"2024-01-16T15:23:30.459727+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:24.899317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":252,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:25.450215+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:23:25.450215+0000","last_peered":"2024-01-16T15:23:25.450215+0000","last_clean":"2024-01-16T15:23:25.450215+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:23:25.450215+0000","last_undegraded":"2024-01-16T15:23:25.450215+0000","last_fullsized":"2024-01-16T15:23:25.450215+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4114,"ondisk_log_size":4114,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1244374,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":487,"num_write_kb":1455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":2488748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542483,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.222,"15min":1.274},"min":{"1min":0.64800000000000002,"5min":0.54300000000000004,"15min":0.28599999999999998},"max":{"1min":1.4159999999999999,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":1.2},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.302,"15min":1.4399999999999999},"min":{"1min":0.75900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.712,"5min":1.8959999999999999,"15min":25.503},"last":1.282}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.34499999999999997},"max":{"1min":1.54,"5min":1.5620000000000001,"15min":5.5010000000000003},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.1619999999999999,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4139999999999999,"5min":1.9490000000000001,"15min":4.3369999999999997},"last":1.343}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.68400000000000005,"15min":0.629},"max":{"1min":1.669,"5min":2.141,"15min":11.536},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.4670000000000001},"min":{"1min":0.73099999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5309999999999999,"5min":1.8500000000000001,"15min":25.478000000000002},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.40899999999999997},"max":{"1min":1.4870000000000001,"5min":1.7090000000000001,"15min":5.6369999999999996},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":0.80200000000000005,"5min":0.75900000000000001,"15min":0.67800000000000005},"max":{"1min":1.5589999999999999,"5min":1.9550000000000001,"15min":6.351},"last":1.5629999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.204,"15min":1.3919999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6120000000000001,"5min":1.75,"15min":25.401},"last":1.395},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6399999999999999,"5min":1.8160000000000001,"15min":5.7859999999999996},"last":1.1679999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3540000000000001,"15min":1.359},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.722,"5min":2.3849999999999998,"15min":4.4539999999999997},"last":1.472},{"interface":"front","average":{"1min":1.385,"5min":1.383,"15min":1.5189999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.708,"5min":1.8480000000000001,"15min":26.327000000000002},"last":1.516}]},{"osd":6,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5469999999999999,"15min":1.6739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8280000000000001,"5min":2.0579999999999998,"15min":25.451000000000001},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5029999999999999,"15min":1.496},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.786,"5min":2.786,"15min":5.8250000000000002},"last":1.4379999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.68400000000000005,"15min":0.35399999999999998},"max":{"1min":1.577,"5min":8.0299999999999994,"15min":29.77},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3220000000000001,"15min":1.415},"min":{"1min":0.88500000000000001,"5min":0.76400000000000001,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.528}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.081,"15min":1.1339999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":1.728,"15min":5.2859999999999996},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.57099999999999995},"max":{"1min":1.595,"5min":1.7350000000000001,"15min":10.494},"last":0.95299999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.202,"15min":1.278},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.46,"5min":1.6579999999999999,"15min":5.0549999999999997},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.389,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.837,"5min":1.9279999999999999,"15min":5.1459999999999999},"last":1.0760000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.319,"15min":1.3089999999999999},"min":{"1min":0.90200000000000002,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":1.601,"5min":1.8360000000000001,"15min":3.8079999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.3129999999999999,"15min":1.448},"min":{"1min":0.87,"5min":0.65200000000000002,"15min":0.623},"max":{"1min":1.7130000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.6799999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3839999999999999,"15min":1.3480000000000001},"min":{"1min":0.97399999999999998,"5min":0.87,"15min":0.53400000000000003},"max":{"1min":1.6890000000000001,"5min":20.422999999999998,"15min":20.422999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.258,"15min":1.28},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":3.6160000000000001},"last":1.4370000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.272,"15min":1.321},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.53700000000000003},"max":{"1min":1.542,"5min":7.6319999999999997,"15min":13.949999999999999},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.581,"15min":1.5720000000000001},"min":{"1min":1.113,"5min":1.113,"15min":0.51600000000000001},"max":{"1min":1.972,"5min":7.4980000000000002,"15min":11.298},"last":1.633}]},{"osd":7,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4410000000000001,"15min":1.498},"min":{"1min":1.0169999999999999,"5min":0.97499999999999998,"15min":0.57099999999999995},"max":{"1min":1.659,"5min":2.004,"15min":14.037000000000001},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.512,"15min":1.5720000000000001},"min":{"1min":1.1100000000000001,"5min":1.024,"15min":0.61599999999999999},"max":{"1min":1.9370000000000001,"5min":2.0209999999999999,"15min":4.9829999999999997},"last":1.5860000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141700,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93230140,"statfs":{"total":96636764160,"available":95467663360,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2090000000000001,"15min":1.1379999999999999},"min":{"1min":0.79300000000000004,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.6890000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.163,"15min":1.147},"min":{"1min":0.89900000000000002,"5min":0.72599999999999998,"15min":0.34799999999999998},"max":{"1min":1.526,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.194}]},{"osd":1,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1850000000000001,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":1.776,"15min":17.074999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.228,"15min":1.244},"min":{"1min":0.89000000000000001,"5min":0.77100000000000002,"15min":0.46999999999999997},"max":{"1min":1.714,"5min":1.714,"15min":4.8499999999999996},"last":1.0149999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.222,"15min":1.198},"min":{"1min":0.83899999999999997,"5min":0.72599999999999998,"15min":0.54800000000000004},"max":{"1min":1.53,"5min":2.7000000000000002,"15min":4.306},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2849999999999999,"15min":1.323},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.52200000000000002},"max":{"1min":1.494,"5min":2.9670000000000001,"15min":16.721},"last":0.97299999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.97499999999999998,"5min":0.81499999999999995,"15min":0.60299999999999998},"max":{"1min":1.859,"5min":1.905,"15min":4.8010000000000002},"last":1.228},{"interface":"front","average":{"1min":1.427,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":0.51200000000000001},"max":{"1min":1.7390000000000001,"5min":14.044,"15min":14.044},"last":1.3089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2749999999999999,"15min":1.3260000000000001},"min":{"1min":0.90400000000000003,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.591,"5min":2.0139999999999998,"15min":16.382999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.321,"15min":1.341},"min":{"1min":0.81000000000000005,"5min":0.72099999999999997,"15min":0.502},"max":{"1min":1.982,"5min":1.982,"15min":4.6580000000000004},"last":1.145}]},{"osd":6,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3500000000000001,"15min":1.357},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.39700000000000002},"max":{"1min":1.637,"5min":1.901,"15min":4.3380000000000001},"last":1.48},{"interface":"front","average":{"1min":1.488,"5min":1.516,"15min":1.5089999999999999},"min":{"1min":1.137,"5min":0.94199999999999995,"15min":0.57899999999999996},"max":{"1min":1.6919999999999999,"5min":1.907,"15min":3.3719999999999999},"last":1.8180000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5,"15min":1.5029999999999999},"min":{"1min":0.95999999999999996,"5min":0.95599999999999996,"15min":0.45000000000000001},"max":{"1min":2.044,"5min":2.044,"15min":4.742},"last":1.369},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5700000000000001,"15min":1.5509999999999999},"min":{"1min":1.0820000000000001,"5min":1.0629999999999999,"15min":0.58799999999999997},"max":{"1min":1.821,"5min":2.0609999999999999,"15min":2.6520000000000001},"last":1.589}]}]},{"osd":4,"up_from":14,"seq":60129542482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932292,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439548,"statfs":{"total":96636764160,"available":94658097152,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":1.179}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":1.2290000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.752},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.2729999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5649999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.319},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.403}]}]},{"osd":3,"up_from":14,"seq":60129542484,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872976,"kb_used_data":1751624,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498864,"statfs":{"total":96636764160,"available":94718836736,"internally_reserved":0,"allocated":1793662976,"data_stored":1792827126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.68899999999999995,"5min":0.58499999999999996,"15min":0.34699999999999998},"max":{"1min":1.3640000000000001,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":0.746},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.78500000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.98199999999999998,"15min":1.0249999999999999},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":1.359,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":1.718},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2569999999999999,"15min":1.226},"min":{"1min":0.64000000000000001,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.7909999999999999,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":0.67100000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.244,"15min":1.26},"min":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.36699999999999999},"max":{"1min":1.577,"5min":2.258,"15min":3.5249999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.45200000000000001},"max":{"1min":1.897,"5min":2.339,"15min":4.6529999999999996},"last":0.97199999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.323,"15min":1.319},"min":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":0.44500000000000001},"max":{"1min":1.6759999999999999,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.73299999999999998},"max":{"1min":1.861,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.6040000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4890000000000001,"15min":1.5009999999999999},"min":{"1min":1.224,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":1.738,"5min":2.0249999999999999,"15min":4.484},"last":1.238},{"interface":"front","average":{"1min":1.419,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.66500000000000004},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.49}]},{"osd":6,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5389999999999999,"15min":1.5620000000000001},"min":{"1min":1.1220000000000001,"5min":0.92700000000000005,"15min":0.71499999999999997},"max":{"1min":1.96,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.663},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6080000000000001,"15min":1.613},"min":{"1min":1.1399999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.069,"5min":2.169,"15min":4.5880000000000001},"last":1.3049999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5329999999999999,"15min":1.5509999999999999},"min":{"1min":0.98799999999999999,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":1.8480000000000001,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.583},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.6299999999999999,"15min":1.6279999999999999},"min":{"1min":1.2669999999999999,"5min":1.008,"15min":0.64700000000000002},"max":{"1min":2.1160000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.5429999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542482,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251776,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":115774,"kb_avail":93120064,"statfs":{"total":96636764160,"available":95354945536,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118553029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.90300000000000002,"15min":0.93600000000000005},"min":{"1min":0.56899999999999995,"5min":0.54600000000000004,"15min":0.36099999999999999},"max":{"1min":1.379,"5min":1.6659999999999999,"15min":9.657},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.004,"15min":1.0740000000000001},"min":{"1min":0.60599999999999998,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.3460000000000001,"5min":1.7649999999999999,"15min":20.407},"last":0.97299999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93500000000000005,"15min":1.0429999999999999},"min":{"1min":0.72299999999999998,"5min":0.64000000000000001,"15min":0.41499999999999998},"max":{"1min":1.5229999999999999,"5min":1.53,"15min":11.406000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.319,"15min":1.409},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.45300000000000001},"max":{"1min":1.7050000000000001,"5min":1.8049999999999999,"15min":20.513000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.389,"15min":1.371},"min":{"1min":0.95099999999999996,"5min":0.56100000000000005,"15min":0.53400000000000003},"max":{"1min":1.726,"5min":2.0110000000000001,"15min":3.1080000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.288,"15min":1.3320000000000001},"min":{"1min":0.88200000000000001,"5min":0.68200000000000005,"15min":0.49099999999999999},"max":{"1min":1.7190000000000001,"5min":1.7949999999999999,"15min":9.7390000000000008},"last":1.288}]},{"osd":4,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.494,"15min":1.46},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.66100000000000003},"max":{"1min":1.885,"5min":12.669,"15min":12.779},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3560000000000001,"15min":1.466},"min":{"1min":1.018,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":25.661000000000001},"last":1.327}]},{"osd":5,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.446,"15min":1.448},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.68300000000000005},"max":{"1min":1.8440000000000001,"5min":1.9039999999999999,"15min":3.0209999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.5369999999999999},"min":{"1min":1.0860000000000001,"5min":0.90200000000000002,"15min":0.59799999999999998},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":9.7850000000000001},"last":1.173}]},{"osd":6,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.462,"15min":1.651},"min":{"1min":1.075,"5min":1.048,"15min":0.57299999999999995},"max":{"1min":1.8320000000000001,"5min":1.8660000000000001,"15min":25.603999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4370000000000001,"15min":1.518},"min":{"1min":0.752,"5min":0.751,"15min":0.629},"max":{"1min":2.0019999999999998,"5min":2.077,"15min":15.968999999999999},"last":1.306}]},{"osd":7,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.655,"15min":1.698},"min":{"1min":1.173,"5min":1.173,"15min":0.61699999999999999},"max":{"1min":1.8220000000000001,"5min":9.3900000000000006,"15min":24.509},"last":1.129},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5489999999999999,"15min":1.6259999999999999},"min":{"1min":0.93799999999999994,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":2.081,"5min":2.161,"15min":10.455},"last":1.363}]}]},{"osd":1,"up_from":14,"seq":60129542482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1448324,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92923516,"statfs":{"total":96636764160,"available":95153680384,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.90400000000000003,"15min":0.93000000000000005},"min":{"1min":0.64800000000000002,"5min":0.53000000000000003,"15min":0.28399999999999997},"max":{"1min":1.6100000000000001,"5min":1.718,"15min":4.266},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1220000000000001,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.34999999999999998},"max":{"1min":2.153,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.137,"15min":1.135},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.31},"max":{"1min":1.647,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.39300000000000002},"max":{"1min":1.579,"5min":1.8340000000000001,"15min":4.1559999999999997},"last":1.2050000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.456,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.439},"max":{"1min":1.7529999999999999,"5min":1.9930000000000001,"15min":4.1980000000000004},"last":1.105},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.46600000000000003},"max":{"1min":1.798,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":0.61099999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5349999999999999,"15min":1.5109999999999999},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.69899999999999995},"max":{"1min":1.887,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.528,"15min":1.52},"min":{"1min":1.0309999999999999,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":1.613,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.1759999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.522,"15min":1.466},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":1.9219999999999999,"5min":5.194,"15min":5.194},"last":1.034},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":1.714,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.0600000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.516,"15min":1.5189999999999999},"min":{"1min":1.0549999999999999,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.544,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.264},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6539999999999999,"15min":1.6080000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.70699999999999996},"max":{"1min":2.0659999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.83799999999999997}]},{"osd":7,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.609,"15min":1.589},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.69999999999999996},"max":{"1min":1.9690000000000001,"5min":16.140000000000001,"15min":16.140000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6759999999999999,"15min":1.6339999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.78700000000000003},"max":{"1min":2.4449999999999998,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.319}]}]},{"osd":0,"up_from":14,"seq":60129542483,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073492,"kb_used_data":1997964,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298348,"statfs":{"total":96636764160,"available":94513508352,"internally_reserved":0,"allocated":2045915136,"data_stored":2045075445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0309999999999999,"15min":1.0249999999999999},"min":{"1min":0.55500000000000005,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.3260000000000001,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.2370000000000001,"15min":1.252},"min":{"1min":0.497,"5min":0.47999999999999998,"15min":0.33900000000000002},"max":{"1min":1.4470000000000001,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1120000000000001,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.46700000000000003,"15min":0.40000000000000002},"max":{"1min":1.5800000000000001,"5min":1.875,"15min":7.5940000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1299999999999999,"15min":1.175},"min":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.39600000000000002},"max":{"1min":1.665,"5min":1.8,"15min":5.3550000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.37},"max":{"1min":1.2170000000000001,"5min":5.851,"15min":5.851},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.335,"5min":1.3169999999999999,"15min":1.343},"min":{"1min":0.65800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.5660000000000001,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":0.83199999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.304,"15min":1.3400000000000001},"min":{"1min":0.80400000000000005,"5min":0.79600000000000004,"15min":0.69099999999999995},"max":{"1min":1.7829999999999999,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.4530000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":1.7330000000000001,"5min":2.141,"15min":5.3109999999999999},"last":0.91800000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.4670000000000001,"15min":1.4930000000000001},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.75,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.427,"15min":1.454},"min":{"1min":0.96599999999999997,"5min":0.84699999999999998,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":2.27,"15min":8.3949999999999996},"last":0.94999999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5840000000000001,"15min":1.659},"min":{"1min":1.0669999999999999,"5min":0.82299999999999995,"15min":0.72999999999999998},"max":{"1min":1.8049999999999999,"5min":2.173,"15min":15.159000000000001},"last":1.054},{"interface":"front","average":{"1min":1.379,"5min":1.526,"15min":1.6259999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.58999999999999997},"max":{"1min":1.8700000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.236}]},{"osd":7,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.617,"15min":1.667},"min":{"1min":1,"5min":0.999,"15min":0.60899999999999999},"max":{"1min":1.819,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":1.018,"5min":0.873,"15min":0.65800000000000003},"max":{"1min":1.8340000000000001,"5min":2.2719999999999998,"15min":6.149},"last":1.006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1239741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1239699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:27:25.192 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T15:27:25.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:27:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.454+0000 7f8480c00640 1 Processor -- start 2024-01-16T15:27:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.455+0000 7f8480c00640 1 -- start start 2024-01-16T15:27:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.456+0000 7f8480c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f847c06f180 0x7f847c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.456+0000 7f8480c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 0x7f847c15e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.456+0000 7f8480c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c15da60 con 0x7f847c146c40 2024-01-16T15:27:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.456+0000 7f8480c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c147020 con 0x7f847c06f180 2024-01-16T15:27:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.456+0000 7f8480c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c06c730 con 0x7f847c06bf60 2024-01-16T15:27:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.457+0000 7f8479400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f847c06f180 0x7f847c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.457+0000 7f847a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 0x7f847c15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.457+0000 7f8479400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f847c06f180 0x7f847c06ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:36270/0 (socket says 172.21.15.43:36270) 2024-01-16T15:27:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.457+0000 7f8479400640 1 -- 172.21.15.43:0/2596861173 learned_addr learned my addr 172.21.15.43:0/2596861173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.457+0000 7f847a800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 0x7f847c15e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:48920/0 (socket says 172.21.15.43:48920) 2024-01-16T15:27:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.458+0000 7f8479400640 1 -- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 msgr2=0x7f847c15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.458+0000 7f8479400640 1 --2- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 0x7f847c15e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.458+0000 7f8479400640 1 -- 172.21.15.43:0/2596861173 >> v1:172.21.15.69:6789/0 conn(0x7f847c146c40 legacy=0x7f847c06ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.458+0000 7f8479400640 1 -- 172.21.15.43:0/2596861173 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c06ca10 con 0x7f847c06f180 2024-01-16T15:27:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.460+0000 7f8479400640 1 --2- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c06f180 0x7f847c06ba20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8464009870 tx=0x7f846402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dccc2e1e4eb6224 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.460+0000 7f8478a00640 1 -- 172.21.15.43:0/2596861173 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464030840 con 0x7f847c06f180 2024-01-16T15:27:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.460+0000 7f8478a00640 1 -- 172.21.15.43:0/2596861173 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84640309a0 con 0x7f847c06f180 2024-01-16T15:27:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.460+0000 7f8478a00640 1 -- 172.21.15.43:0/2596861173 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464030d30 con 0x7f847c06f180 2024-01-16T15:27:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 -- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c06f180 msgr2=0x7f847c06ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 --2- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c06f180 0x7f847c06ba20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8464009870 tx=0x7f846402b980 comp rx=0 tx=0).stop 2024-01-16T15:27:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 -- 172.21.15.43:0/2596861173 shutdown_connections 2024-01-16T15:27:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 --2- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c06f180 0x7f847c06ba20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 --2- 172.21.15.43:0/2596861173 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06bf60 0x7f847c15e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 -- 172.21.15.43:0/2596861173 >> 172.21.15.43:0/2596861173 conn(0x7f847c0663d0 msgr2=0x7f847c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 -- 172.21.15.43:0/2596861173 shutdown_connections 2024-01-16T15:27:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.461+0000 7f8480c00640 1 -- 172.21.15.43:0/2596861173 wait complete. 2024-01-16T15:27:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.463+0000 7f8480c00640 1 Processor -- start 2024-01-16T15:27:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.463+0000 7f8480c00640 1 -- start start 2024-01-16T15:27:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 0x7f847c130550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06f180 0x7f847c1572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 0x7f847c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c131100 con 0x7f847c06bf60 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c157ed0 con 0x7f847c146c40 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.464+0000 7f8480c00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1581b0 con 0x7f847c06f180 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f847a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 0x7f847c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 0x7f847c130550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06f180 0x7f847c1572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f847a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 0x7f847c157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:36276/0 (socket says 172.21.15.43:36276) 2024-01-16T15:27:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f847a800640 1 -- 172.21.15.43:0/1620196682 learned_addr learned my addr 172.21.15.43:0/1620196682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 -- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06f180 msgr2=0x7f847c1572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06f180 0x7f847c1572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 -- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 msgr2=0x7f847c157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 0x7f847c157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.465+0000 7f8479e00640 1 -- 172.21.15.43:0/1620196682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84640094d0 con 0x7f847c06bf60 2024-01-16T15:27:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8479e00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 0x7f847c130550 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f847c06b9d0 tx=0x7f847400cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474003b90 con 0x7f847c06bf60 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8474003d10 con 0x7f847c06bf60 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474005310 con 0x7f847c06bf60 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c158490 con 0x7f847c06bf60 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.466+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c12e550 con 0x7f847c06bf60 2024-01-16T15:27:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.467+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c12e710 con 0x7f847c06bf60 2024-01-16T15:27:25.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.468+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f8474005470 con 0x7f847c06bf60 2024-01-16T15:27:25.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.469+0000 7f8472000640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 0x7f845406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.469+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8474088740 con 0x7f847c06bf60 2024-01-16T15:27:25.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.469+0000 7f8479400640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 0x7f845406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.470+0000 7f8479400640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 0x7f845406e980 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8464036000 tx=0x7f8464008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.474+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8474054d20 con 0x7f847c06bf60 2024-01-16T15:27:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.672+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f847c0b0d80 con 0x7f845406c4c0 2024-01-16T15:27:25.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.684+0000 7f8472000640 1 -- 172.21.15.43:0/1620196682 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289493 (secure 0 0 0) 0x7f847c0b0d80 con 0x7f845406c4c0 2024-01-16T15:27:25.687 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:25.688 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 msgr2=0x7f845406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 0x7f845406e980 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8464036000 tx=0x7f8464008030 comp rx=0 tx=0).stop 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 msgr2=0x7f847c130550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 0x7f847c130550 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f847c06b9d0 tx=0x7f847400cad0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 shutdown_connections 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f845406c4c0 0x7f845406e980 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f847c146c40 0x7f847c157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f847c06f180 0x7f847c1572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 --2- 172.21.15.43:0/1620196682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f847c06bf60 0x7f847c130550 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 >> 172.21.15.43:0/1620196682 conn(0x7f847c0663d0 msgr2=0x7f847c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:25.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 shutdown_connections 2024-01-16T15:27:25.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.690+0000 7f8480c00640 1 -- 172.21.15.43:0/1620196682 wait complete. 2024-01-16T15:27:25.710 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-01-16T15:27:25.502696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444287485,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2253,"num_write_kb":6295365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4683,"ondisk_log_size":4683,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13389448,"kb_used_data":12597768,"kb_used_omap":12,"kb_used_meta":791603,"kb_avail":741585272,"statfs":{"total":773094113280,"available":759383318528,"internally_reserved":0,"allocated":12900114432,"data_stored":12893530885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":810602028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073783661,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":278,"num_write_kb":1048624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004115"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737744+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737744+0000","last_peered":"2024-01-16T15:27:17.737744+0000","last_clean":"2024-01-16T15:27:17.737744+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737744+0000","last_undegraded":"2024-01-16T15:27:17.737744+0000","last_fullsized":"2024-01-16T15:27:17.737744+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:54:47.552196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'69","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736614+0000","last_change":"2024-01-16T15:18:50.514868+0000","last_active":"2024-01-16T15:27:17.736614+0000","last_peered":"2024-01-16T15:27:17.736614+0000","last_clean":"2024-01-16T15:27:17.736614+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:17.736614+0000","last_undegraded":"2024-01-16T15:27:17.736614+0000","last_fullsized":"2024-01-16T15:27:17.736614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:22.704331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.743697+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:27:17.743697+0000","last_peered":"2024-01-16T15:27:17.743697+0000","last_clean":"2024-01-16T15:27:17.743697+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:27:17.743697+0000","last_undegraded":"2024-01-16T15:27:17.743697+0000","last_fullsized":"2024-01-16T15:27:17.743697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733813+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733813+0000","last_peered":"2024-01-16T15:27:17.733813+0000","last_clean":"2024-01-16T15:27:17.733813+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733813+0000","last_undegraded":"2024-01-16T15:27:17.733813+0000","last_fullsized":"2024-01-16T15:27:17.733813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:27.895459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'60","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717706+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:27:17.717706+0000","last_peered":"2024-01-16T15:27:17.717706+0000","last_clean":"2024-01-16T15:27:17.717706+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:17.717706+0000","last_undegraded":"2024-01-16T15:27:17.717706+0000","last_fullsized":"2024-01-16T15:27:17.717706+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:32.606162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'74","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721540+0000","last_change":"2024-01-16T15:18:50.517889+0000","last_active":"2024-01-16T15:27:17.721540+0000","last_peered":"2024-01-16T15:27:17.721540+0000","last_clean":"2024-01-16T15:27:17.721540+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:17.721540+0000","last_undegraded":"2024-01-16T15:27:17.721540+0000","last_fullsized":"2024-01-16T15:27:17.721540+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'66","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734633+0000","last_change":"2024-01-16T15:18:50.506124+0000","last_active":"2024-01-16T15:27:17.734633+0000","last_peered":"2024-01-16T15:27:17.734633+0000","last_clean":"2024-01-16T15:27:17.734633+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:17.734633+0000","last_undegraded":"2024-01-16T15:27:17.734633+0000","last_fullsized":"2024-01-16T15:27:17.734633+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'54","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730876+0000","last_change":"2024-01-16T15:18:50.513082+0000","last_active":"2024-01-16T15:27:17.730876+0000","last_peered":"2024-01-16T15:27:17.730876+0000","last_clean":"2024-01-16T15:27:17.730876+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:17.730876+0000","last_undegraded":"2024-01-16T15:27:17.730876+0000","last_fullsized":"2024-01-16T15:27:17.730876+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724783+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.724783+0000","last_peered":"2024-01-16T15:27:17.724783+0000","last_clean":"2024-01-16T15:27:17.724783+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.724783+0000","last_undegraded":"2024-01-16T15:27:17.724783+0000","last_fullsized":"2024-01-16T15:27:17.724783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:57.207916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732638+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.732638+0000","last_peered":"2024-01-16T15:27:17.732638+0000","last_clean":"2024-01-16T15:27:17.732638+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.732638+0000","last_undegraded":"2024-01-16T15:27:17.732638+0000","last_fullsized":"2024-01-16T15:27:17.732638+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:26.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'70","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.727985+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:27:17.727985+0000","last_peered":"2024-01-16T15:27:17.727985+0000","last_clean":"2024-01-16T15:27:17.727985+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:17.727985+0000","last_undegraded":"2024-01-16T15:27:17.727985+0000","last_fullsized":"2024-01-16T15:27:17.727985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735505+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735505+0000","last_peered":"2024-01-16T15:27:17.735505+0000","last_clean":"2024-01-16T15:27:17.735505+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735505+0000","last_undegraded":"2024-01-16T15:27:17.735505+0000","last_fullsized":"2024-01-16T15:27:17.735505+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:28:07.149636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'57","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734731+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:27:17.734731+0000","last_peered":"2024-01-16T15:27:17.734731+0000","last_clean":"2024-01-16T15:27:17.734731+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:27:17.734731+0000","last_undegraded":"2024-01-16T15:27:17.734731+0000","last_fullsized":"2024-01-16T15:27:17.734731+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:35.209083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'65","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739366+0000","last_change":"2024-01-16T15:18:50.517177+0000","last_active":"2024-01-16T15:27:17.739366+0000","last_peered":"2024-01-16T15:27:17.739366+0000","last_clean":"2024-01-16T15:27:17.739366+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:17.739366+0000","last_undegraded":"2024-01-16T15:27:17.739366+0000","last_fullsized":"2024-01-16T15:27:17.739366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'54","reported_seq":77,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724971+0000","last_change":"2024-01-16T15:18:50.518134+0000","last_active":"2024-01-16T15:27:17.724971+0000","last_peered":"2024-01-16T15:27:17.724971+0000","last_clean":"2024-01-16T15:27:17.724971+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:17.724971+0000","last_undegraded":"2024-01-16T15:27:17.724971+0000","last_fullsized":"2024-01-16T15:27:17.724971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:11:10.936165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738723+0000","last_change":"2024-01-16T15:18:50.515967+0000","last_active":"2024-01-16T15:27:17.738723+0000","last_peered":"2024-01-16T15:27:17.738723+0000","last_clean":"2024-01-16T15:27:17.738723+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:17.738723+0000","last_undegraded":"2024-01-16T15:27:17.738723+0000","last_fullsized":"2024-01-16T15:27:17.738723+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:20:18.639887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'49","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733966+0000","last_change":"2024-01-16T15:18:50.514080+0000","last_active":"2024-01-16T15:27:17.733966+0000","last_peered":"2024-01-16T15:27:17.733966+0000","last_clean":"2024-01-16T15:27:17.733966+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:17.733966+0000","last_undegraded":"2024-01-16T15:27:17.733966+0000","last_fullsized":"2024-01-16T15:27:17.733966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:18.432791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'72","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735139+0000","last_change":"2024-01-16T15:18:50.512290+0000","last_active":"2024-01-16T15:27:17.735139+0000","last_peered":"2024-01-16T15:27:17.735139+0000","last_clean":"2024-01-16T15:27:17.735139+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:17.735139+0000","last_undegraded":"2024-01-16T15:27:17.735139+0000","last_fullsized":"2024-01-16T15:27:17.735139+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:03.196990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'71","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741586+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:27:17.741586+0000","last_peered":"2024-01-16T15:27:17.741586+0000","last_clean":"2024-01-16T15:27:17.741586+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:27:17.741586+0000","last_undegraded":"2024-01-16T15:27:17.741586+0000","last_fullsized":"2024-01-16T15:27:17.741586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.753500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734862+0000","last_change":"2024-01-16T15:18:50.523238+0000","last_active":"2024-01-16T15:27:17.734862+0000","last_peered":"2024-01-16T15:27:17.734862+0000","last_clean":"2024-01-16T15:27:17.734862+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:17.734862+0000","last_undegraded":"2024-01-16T15:27:17.734862+0000","last_fullsized":"2024-01-16T15:27:17.734862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:10.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'73","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742649+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:27:17.742649+0000","last_peered":"2024-01-16T15:27:17.742649+0000","last_clean":"2024-01-16T15:27:17.742649+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:27:17.742649+0000","last_undegraded":"2024-01-16T15:27:17.742649+0000","last_fullsized":"2024-01-16T15:27:17.742649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:04.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'64","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717654+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:27:17.717654+0000","last_peered":"2024-01-16T15:27:17.717654+0000","last_clean":"2024-01-16T15:27:17.717654+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:27:17.717654+0000","last_undegraded":"2024-01-16T15:27:17.717654+0000","last_fullsized":"2024-01-16T15:27:17.717654+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'57","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739771+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:27:17.739771+0000","last_peered":"2024-01-16T15:27:17.739771+0000","last_clean":"2024-01-16T15:27:17.739771+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:17.739771+0000","last_undegraded":"2024-01-16T15:27:17.739771+0000","last_fullsized":"2024-01-16T15:27:17.739771+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:49.070110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'70","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.717432+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:27:17.717432+0000","last_peered":"2024-01-16T15:27:17.717432+0000","last_clean":"2024-01-16T15:27:17.717432+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:27:17.717432+0000","last_undegraded":"2024-01-16T15:27:17.717432+0000","last_fullsized":"2024-01-16T15:27:17.717432+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:53:09.748440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'71","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740798+0000","last_change":"2024-01-16T15:18:50.513030+0000","last_active":"2024-01-16T15:27:17.740798+0000","last_peered":"2024-01-16T15:27:17.740798+0000","last_clean":"2024-01-16T15:27:17.740798+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:17.740798+0000","last_undegraded":"2024-01-16T15:27:17.740798+0000","last_fullsized":"2024-01-16T15:27:17.740798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:54.334893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'81","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736872+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:27:17.736872+0000","last_peered":"2024-01-16T15:27:17.736872+0000","last_clean":"2024-01-16T15:27:17.736872+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:27:17.736872+0000","last_undegraded":"2024-01-16T15:27:17.736872+0000","last_fullsized":"2024-01-16T15:27:17.736872+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:49.522744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'64","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730649+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:27:17.730649+0000","last_peered":"2024-01-16T15:27:17.730649+0000","last_clean":"2024-01-16T15:27:17.730649+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:27:17.730649+0000","last_undegraded":"2024-01-16T15:27:17.730649+0000","last_fullsized":"2024-01-16T15:27:17.730649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":64,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730383+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:17.730383+0000","last_peered":"2024-01-16T15:27:17.730383+0000","last_clean":"2024-01-16T15:27:17.730383+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:17.730383+0000","last_undegraded":"2024-01-16T15:27:17.730383+0000","last_fullsized":"2024-01-16T15:27:17.730383+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'57","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738407+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:27:17.738407+0000","last_peered":"2024-01-16T15:27:17.738407+0000","last_clean":"2024-01-16T15:27:17.738407+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:27:17.738407+0000","last_undegraded":"2024-01-16T15:27:17.738407+0000","last_fullsized":"2024-01-16T15:27:17.738407+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:38.948857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'76","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720265+0000","last_change":"2024-01-16T15:18:50.491931+0000","last_active":"2024-01-16T15:27:17.720265+0000","last_peered":"2024-01-16T15:27:17.720265+0000","last_clean":"2024-01-16T15:27:17.720265+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:17.720265+0000","last_undegraded":"2024-01-16T15:27:17.720265+0000","last_fullsized":"2024-01-16T15:27:17.720265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'81","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736796+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:27:17.736796+0000","last_peered":"2024-01-16T15:27:17.736796+0000","last_clean":"2024-01-16T15:27:17.736796+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:17.736796+0000","last_undegraded":"2024-01-16T15:27:17.736796+0000","last_fullsized":"2024-01-16T15:27:17.736796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:21:03.259116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'63","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.723981+0000","last_change":"2024-01-16T15:18:50.510830+0000","last_active":"2024-01-16T15:27:17.723981+0000","last_peered":"2024-01-16T15:27:17.723981+0000","last_clean":"2024-01-16T15:27:17.723981+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:17.723981+0000","last_undegraded":"2024-01-16T15:27:17.723981+0000","last_fullsized":"2024-01-16T15:27:17.723981+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:03:30.055863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'62","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.721097+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:27:17.721097+0000","last_peered":"2024-01-16T15:27:17.721097+0000","last_clean":"2024-01-16T15:27:17.721097+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:17.721097+0000","last_undegraded":"2024-01-16T15:27:17.721097+0000","last_fullsized":"2024-01-16T15:27:17.721097+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476688+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:25:05.476688+0000","last_peered":"2024-01-16T15:25:05.476688+0000","last_clean":"2024-01-16T15:25:05.476688+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:25:05.476688+0000","last_undegraded":"2024-01-16T15:25:05.476688+0000","last_fullsized":"2024-01-16T15:25:05.476688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'54","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733008+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:27:17.733008+0000","last_peered":"2024-01-16T15:27:17.733008+0000","last_clean":"2024-01-16T15:27:17.733008+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:27:17.733008+0000","last_undegraded":"2024-01-16T15:27:17.733008+0000","last_fullsized":"2024-01-16T15:27:17.733008+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:37.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.466688+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:24:50.466688+0000","last_peered":"2024-01-16T15:24:50.466688+0000","last_clean":"2024-01-16T15:24:50.466688+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:24:50.466688+0000","last_undegraded":"2024-01-16T15:24:50.466688+0000","last_fullsized":"2024-01-16T15:24:50.466688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:51.215798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'71","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724610+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:27:17.724610+0000","last_peered":"2024-01-16T15:27:17.724610+0000","last_clean":"2024-01-16T15:27:17.724610+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:17.724610+0000","last_undegraded":"2024-01-16T15:27:17.724610+0000","last_fullsized":"2024-01-16T15:27:17.724610+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360466+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:23:50.360466+0000","last_peered":"2024-01-16T15:23:50.360466+0000","last_clean":"2024-01-16T15:23:50.360466+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:23:50.360466+0000","last_undegraded":"2024-01-16T15:23:50.360466+0000","last_fullsized":"2024-01-16T15:23:50.360466+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:47:18.088502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'68","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741745+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:27:17.741745+0000","last_peered":"2024-01-16T15:27:17.741745+0000","last_clean":"2024-01-16T15:27:17.741745+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:27:17.741745+0000","last_undegraded":"2024-01-16T15:27:17.741745+0000","last_fullsized":"2024-01-16T15:27:17.741745+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:37.406427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.360421+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:23:50.360421+0000","last_peered":"2024-01-16T15:23:50.360421+0000","last_clean":"2024-01-16T15:23:50.360421+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:23:50.360421+0000","last_undegraded":"2024-01-16T15:23:50.360421+0000","last_fullsized":"2024-01-16T15:23:50.360421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:43.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'72","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.719815+0000","last_change":"2024-01-16T15:19:15.318178+0000","last_active":"2024-01-16T15:27:17.719815+0000","last_peered":"2024-01-16T15:27:17.719815+0000","last_clean":"2024-01-16T15:27:17.719815+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:17.719815+0000","last_undegraded":"2024-01-16T15:27:17.719815+0000","last_fullsized":"2024-01-16T15:27:17.719815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.463254+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:23:50.463254+0000","last_peered":"2024-01-16T15:23:50.463254+0000","last_clean":"2024-01-16T15:23:50.463254+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:23:50.463254+0000","last_undegraded":"2024-01-16T15:23:50.463254+0000","last_fullsized":"2024-01-16T15:23:50.463254+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:31.593211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'72","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736806+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:27:17.736806+0000","last_peered":"2024-01-16T15:27:17.736806+0000","last_clean":"2024-01-16T15:27:17.736806+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:27:17.736806+0000","last_undegraded":"2024-01-16T15:27:17.736806+0000","last_fullsized":"2024-01-16T15:27:17.736806+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:59.415403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.403153+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:23:50.403153+0000","last_peered":"2024-01-16T15:23:50.403153+0000","last_clean":"2024-01-16T15:23:50.403153+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:23:50.403153+0000","last_undegraded":"2024-01-16T15:23:50.403153+0000","last_fullsized":"2024-01-16T15:23:50.403153+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:57.932540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'73","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.732782+0000","last_change":"2024-01-16T15:18:50.513959+0000","last_active":"2024-01-16T15:27:17.732782+0000","last_peered":"2024-01-16T15:27:17.732782+0000","last_clean":"2024-01-16T15:27:17.732782+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:17.732782+0000","last_undegraded":"2024-01-16T15:27:17.732782+0000","last_fullsized":"2024-01-16T15:27:17.732782+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:45.776442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.368270+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:23:40.368270+0000","last_peered":"2024-01-16T15:23:40.368270+0000","last_clean":"2024-01-16T15:23:40.368270+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:23:40.368270+0000","last_undegraded":"2024-01-16T15:23:40.368270+0000","last_fullsized":"2024-01-16T15:23:40.368270+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:07.440889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'73","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726884+0000","last_change":"2024-01-16T15:18:50.506340+0000","last_active":"2024-01-16T15:27:17.726884+0000","last_peered":"2024-01-16T15:27:17.726884+0000","last_clean":"2024-01-16T15:27:17.726884+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:17.726884+0000","last_undegraded":"2024-01-16T15:27:17.726884+0000","last_fullsized":"2024-01-16T15:27:17.726884+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.622586+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:23:40.622586+0000","last_peered":"2024-01-16T15:23:40.622586+0000","last_clean":"2024-01-16T15:23:40.622586+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:23:40.622586+0000","last_undegraded":"2024-01-16T15:23:40.622586+0000","last_fullsized":"2024-01-16T15:23:40.622586+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:41.604466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722570+0000","last_change":"2024-01-16T15:18:50.507182+0000","last_active":"2024-01-16T15:27:17.722570+0000","last_peered":"2024-01-16T15:27:17.722570+0000","last_clean":"2024-01-16T15:27:17.722570+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:17.722570+0000","last_undegraded":"2024-01-16T15:27:17.722570+0000","last_fullsized":"2024-01-16T15:27:17.722570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.358584+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:23:40.358584+0000","last_peered":"2024-01-16T15:23:40.358584+0000","last_clean":"2024-01-16T15:23:40.358584+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:23:40.358584+0000","last_undegraded":"2024-01-16T15:23:40.358584+0000","last_fullsized":"2024-01-16T15:23:40.358584+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:32.213071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'61","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738962+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:27:17.738962+0000","last_peered":"2024-01-16T15:27:17.738962+0000","last_clean":"2024-01-16T15:27:17.738962+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:17.738962+0000","last_undegraded":"2024-01-16T15:27:17.738962+0000","last_fullsized":"2024-01-16T15:27:17.738962+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:14.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.401032+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:23:40.401032+0000","last_peered":"2024-01-16T15:23:40.401032+0000","last_clean":"2024-01-16T15:23:40.401032+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:23:40.401032+0000","last_undegraded":"2024-01-16T15:23:40.401032+0000","last_fullsized":"2024-01-16T15:23:40.401032+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:31.105453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726260+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:17.726260+0000","last_peered":"2024-01-16T15:27:17.726260+0000","last_clean":"2024-01-16T15:27:17.726260+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:17.726260+0000","last_undegraded":"2024-01-16T15:27:17.726260+0000","last_fullsized":"2024-01-16T15:27:17.726260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'33","reported_seq":50,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.122833+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:25:12.122833+0000","last_peered":"2024-01-16T15:25:12.122833+0000","last_clean":"2024-01-16T15:25:12.122833+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:25:12.122833+0000","last_undegraded":"2024-01-16T15:25:12.122833+0000","last_fullsized":"2024-01-16T15:25:12.122833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:45:56.082925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'69","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.729047+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:27:17.729047+0000","last_peered":"2024-01-16T15:27:17.729047+0000","last_clean":"2024-01-16T15:27:17.729047+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:17.729047+0000","last_undegraded":"2024-01-16T15:27:17.729047+0000","last_fullsized":"2024-01-16T15:27:17.729047+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'321","reported_seq":338,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.747172+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:27:17.747172+0000","last_peered":"2024-01-16T15:27:17.747172+0000","last_clean":"2024-01-16T15:27:17.747172+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:17.747172+0000","last_undegraded":"2024-01-16T15:27:17.747172+0000","last_fullsized":"2024-01-16T15:27:17.747172+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:41.803370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1239629,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":1365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.718504+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:17.718504+0000","last_peered":"2024-01-16T15:27:17.718504+0000","last_clean":"2024-01-16T15:27:17.718504+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:17.718504+0000","last_undegraded":"2024-01-16T15:27:17.718504+0000","last_fullsized":"2024-01-16T15:27:17.718504+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.500458+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:23:30.500458+0000","last_peered":"2024-01-16T15:23:30.500458+0000","last_clean":"2024-01-16T15:23:30.500458+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:23:30.500458+0000","last_undegraded":"2024-01-16T15:23:30.500458+0000","last_fullsized":"2024-01-16T15:23:30.500458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:06.634724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'73","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730523+0000","last_change":"2024-01-16T15:18:50.517675+0000","last_active":"2024-01-16T15:27:17.730523+0000","last_peered":"2024-01-16T15:27:17.730523+0000","last_clean":"2024-01-16T15:27:17.730523+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:17.730523+0000","last_undegraded":"2024-01-16T15:27:17.730523+0000","last_fullsized":"2024-01-16T15:27:17.730523+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.440394+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:23:30.440394+0000","last_peered":"2024-01-16T15:23:30.440394+0000","last_clean":"2024-01-16T15:23:30.440394+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:23:30.440394+0000","last_undegraded":"2024-01-16T15:23:30.440394+0000","last_fullsized":"2024-01-16T15:23:30.440394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:28.315056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'63","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740737+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:27:17.740737+0000","last_peered":"2024-01-16T15:27:17.740737+0000","last_clean":"2024-01-16T15:27:17.740737+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:27:17.740737+0000","last_undegraded":"2024-01-16T15:27:17.740737+0000","last_fullsized":"2024-01-16T15:27:17.740737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":25,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.356549+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:23:30.356549+0000","last_peered":"2024-01-16T15:23:30.356549+0000","last_clean":"2024-01-16T15:23:30.356549+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:23:30.356549+0000","last_undegraded":"2024-01-16T15:23:30.356549+0000","last_fullsized":"2024-01-16T15:23:30.356549+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:58.071689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'46","reported_seq":68,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735009+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:27:17.735009+0000","last_peered":"2024-01-16T15:27:17.735009+0000","last_clean":"2024-01-16T15:27:17.735009+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:27:17.735009+0000","last_undegraded":"2024-01-16T15:27:17.735009+0000","last_fullsized":"2024-01-16T15:27:17.735009+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:40.855053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.618711+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:23:20.618711+0000","last_peered":"2024-01-16T15:23:20.618711+0000","last_clean":"2024-01-16T15:23:20.618711+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:23:20.618711+0000","last_undegraded":"2024-01-16T15:23:20.618711+0000","last_fullsized":"2024-01-16T15:23:20.618711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:25.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734412+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:17.734412+0000","last_peered":"2024-01-16T15:27:17.734412+0000","last_clean":"2024-01-16T15:27:17.734412+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:17.734412+0000","last_undegraded":"2024-01-16T15:27:17.734412+0000","last_fullsized":"2024-01-16T15:27:17.734412+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.354766+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:23:20.354766+0000","last_peered":"2024-01-16T15:23:20.354766+0000","last_clean":"2024-01-16T15:23:20.354766+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:23:20.354766+0000","last_undegraded":"2024-01-16T15:23:20.354766+0000","last_fullsized":"2024-01-16T15:23:20.354766+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:24:26.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738127+0000","last_change":"2024-01-16T15:18:50.517233+0000","last_active":"2024-01-16T15:27:17.738127+0000","last_peered":"2024-01-16T15:27:17.738127+0000","last_clean":"2024-01-16T15:27:17.738127+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:17.738127+0000","last_undegraded":"2024-01-16T15:27:17.738127+0000","last_fullsized":"2024-01-16T15:27:17.738127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.438493+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:23:20.438493+0000","last_peered":"2024-01-16T15:23:20.438493+0000","last_clean":"2024-01-16T15:23:20.438493+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:23:20.438493+0000","last_undegraded":"2024-01-16T15:23:20.438493+0000","last_fullsized":"2024-01-16T15:23:20.438493+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:04.731539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744447+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:27:17.744447+0000","last_peered":"2024-01-16T15:27:17.744447+0000","last_clean":"2024-01-16T15:27:17.744447+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:27:17.744447+0000","last_undegraded":"2024-01-16T15:27:17.744447+0000","last_fullsized":"2024-01-16T15:27:17.744447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:16.245190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.457791+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:23:20.457791+0000","last_peered":"2024-01-16T15:23:20.457791+0000","last_clean":"2024-01-16T15:23:20.457791+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:23:20.457791+0000","last_undegraded":"2024-01-16T15:23:20.457791+0000","last_fullsized":"2024-01-16T15:23:20.457791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'56","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741337+0000","last_change":"2024-01-16T15:18:50.521667+0000","last_active":"2024-01-16T15:27:17.741337+0000","last_peered":"2024-01-16T15:27:17.741337+0000","last_clean":"2024-01-16T15:27:17.741337+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:17.741337+0000","last_undegraded":"2024-01-16T15:27:17.741337+0000","last_fullsized":"2024-01-16T15:27:17.741337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.498582+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:23:20.498582+0000","last_peered":"2024-01-16T15:23:20.498582+0000","last_clean":"2024-01-16T15:23:20.498582+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:23:20.498582+0000","last_undegraded":"2024-01-16T15:23:20.498582+0000","last_fullsized":"2024-01-16T15:23:20.498582+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738717+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:27:17.738717+0000","last_peered":"2024-01-16T15:27:17.738717+0000","last_clean":"2024-01-16T15:27:17.738717+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:27:17.738717+0000","last_undegraded":"2024-01-16T15:27:17.738717+0000","last_fullsized":"2024-01-16T15:27:17.738717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:59.502905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.353213+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:23:10.353213+0000","last_peered":"2024-01-16T15:23:10.353213+0000","last_clean":"2024-01-16T15:23:10.353213+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:23:10.353213+0000","last_undegraded":"2024-01-16T15:23:10.353213+0000","last_fullsized":"2024-01-16T15:23:10.353213+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:30.184973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'65","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735772+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:27:17.735772+0000","last_peered":"2024-01-16T15:27:17.735772+0000","last_clean":"2024-01-16T15:27:17.735772+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:27:17.735772+0000","last_undegraded":"2024-01-16T15:27:17.735772+0000","last_fullsized":"2024-01-16T15:27:17.735772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.496763+0000","last_change":"2024-01-16T15:18:49.521067+0000","last_active":"2024-01-16T15:23:10.496763+0000","last_peered":"2024-01-16T15:23:10.496763+0000","last_clean":"2024-01-16T15:23:10.496763+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:23:10.496763+0000","last_undegraded":"2024-01-16T15:23:10.496763+0000","last_fullsized":"2024-01-16T15:23:10.496763+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'57","reported_seq":83,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737237+0000","last_change":"2024-01-16T15:18:50.516592+0000","last_active":"2024-01-16T15:27:17.737237+0000","last_peered":"2024-01-16T15:27:17.737237+0000","last_clean":"2024-01-16T15:27:17.737237+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:17.737237+0000","last_undegraded":"2024-01-16T15:27:17.737237+0000","last_fullsized":"2024-01-16T15:27:17.737237+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:17.446551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:10.615833+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:23:10.615833+0000","last_peered":"2024-01-16T15:23:10.615833+0000","last_clean":"2024-01-16T15:23:10.615833+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:23:10.615833+0000","last_undegraded":"2024-01-16T15:23:10.615833+0000","last_fullsized":"2024-01-16T15:23:10.615833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:25:30.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'53","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739953+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:27:17.739953+0000","last_peered":"2024-01-16T15:27:17.739953+0000","last_clean":"2024-01-16T15:27:17.739953+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:17.739953+0000","last_undegraded":"2024-01-16T15:27:17.739953+0000","last_fullsized":"2024-01-16T15:27:17.739953+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:09.913343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'3","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.116830+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:25:12.116830+0000","last_peered":"2024-01-16T15:25:12.116830+0000","last_clean":"2024-01-16T15:25:12.116830+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:25:12.116830+0000","last_undegraded":"2024-01-16T15:25:12.116830+0000","last_fullsized":"2024-01-16T15:25:12.116830+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:17.281805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741249+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:17.741249+0000","last_peered":"2024-01-16T15:27:17.741249+0000","last_clean":"2024-01-16T15:27:17.741249+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:17.741249+0000","last_undegraded":"2024-01-16T15:27:17.741249+0000","last_fullsized":"2024-01-16T15:27:17.741249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'31","reported_seq":48,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:19.174858+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:27:19.174858+0000","last_peered":"2024-01-16T15:27:19.174858+0000","last_clean":"2024-01-16T15:27:19.174858+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:19.174858+0000","last_undegraded":"2024-01-16T15:27:19.174858+0000","last_fullsized":"2024-01-16T15:27:19.174858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:07.461774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'69","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735893+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:27:17.735893+0000","last_peered":"2024-01-16T15:27:17.735893+0000","last_clean":"2024-01-16T15:27:17.735893+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:27:17.735893+0000","last_undegraded":"2024-01-16T15:27:17.735893+0000","last_fullsized":"2024-01-16T15:27:17.735893+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:22:49.736642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.641840+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:25:05.641840+0000","last_peered":"2024-01-16T15:25:05.641840+0000","last_clean":"2024-01-16T15:25:05.641840+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:25:05.641840+0000","last_undegraded":"2024-01-16T15:25:05.641840+0000","last_fullsized":"2024-01-16T15:25:05.641840+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:23.613372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'59","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738409+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:27:17.738409+0000","last_peered":"2024-01-16T15:27:17.738409+0000","last_clean":"2024-01-16T15:27:17.738409+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:27:17.738409+0000","last_undegraded":"2024-01-16T15:27:17.738409+0000","last_fullsized":"2024-01-16T15:27:17.738409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:27.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.417626+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:25:05.417626+0000","last_peered":"2024-01-16T15:25:05.417626+0000","last_clean":"2024-01-16T15:25:05.417626+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:25:05.417626+0000","last_undegraded":"2024-01-16T15:25:05.417626+0000","last_fullsized":"2024-01-16T15:25:05.417626+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:23.618165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'63","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736683+0000","last_change":"2024-01-16T15:18:50.516022+0000","last_active":"2024-01-16T15:27:17.736683+0000","last_peered":"2024-01-16T15:27:17.736683+0000","last_clean":"2024-01-16T15:27:17.736683+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:17.736683+0000","last_undegraded":"2024-01-16T15:27:17.736683+0000","last_fullsized":"2024-01-16T15:27:17.736683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476614+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:25:05.476614+0000","last_peered":"2024-01-16T15:25:05.476614+0000","last_clean":"2024-01-16T15:25:05.476614+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:25:05.476614+0000","last_undegraded":"2024-01-16T15:25:05.476614+0000","last_fullsized":"2024-01-16T15:25:05.476614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:39.842102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742798+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:27:17.742798+0000","last_peered":"2024-01-16T15:27:17.742798+0000","last_clean":"2024-01-16T15:27:17.742798+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:27:17.742798+0000","last_undegraded":"2024-01-16T15:27:17.742798+0000","last_fullsized":"2024-01-16T15:27:17.742798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:03.949006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.384697+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:25:05.384697+0000","last_peered":"2024-01-16T15:25:05.384697+0000","last_clean":"2024-01-16T15:25:05.384697+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:25:05.384697+0000","last_undegraded":"2024-01-16T15:25:05.384697+0000","last_fullsized":"2024-01-16T15:25:05.384697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:26.402495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'68","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738935+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:27:17.738935+0000","last_peered":"2024-01-16T15:27:17.738935+0000","last_clean":"2024-01-16T15:27:17.738935+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:27:17.738935+0000","last_undegraded":"2024-01-16T15:27:17.738935+0000","last_fullsized":"2024-01-16T15:27:17.738935+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'34","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:12.114659+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:25:12.114659+0000","last_peered":"2024-01-16T15:25:12.114659+0000","last_clean":"2024-01-16T15:25:12.114659+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:25:12.114659+0000","last_undegraded":"2024-01-16T15:25:12.114659+0000","last_fullsized":"2024-01-16T15:25:12.114659+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:29.720523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'70","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733937+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:27:17.733937+0000","last_peered":"2024-01-16T15:27:17.733937+0000","last_clean":"2024-01-16T15:27:17.733937+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:27:17.733937+0000","last_undegraded":"2024-01-16T15:27:17.733937+0000","last_fullsized":"2024-01-16T15:27:17.733937+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:01.372900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.638362+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:24:50.638362+0000","last_peered":"2024-01-16T15:24:50.638362+0000","last_clean":"2024-01-16T15:24:50.638362+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:24:50.638362+0000","last_undegraded":"2024-01-16T15:24:50.638362+0000","last_fullsized":"2024-01-16T15:24:50.638362+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:03:25.516557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'49","reported_seq":78,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.712896+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:27:17.712896+0000","last_peered":"2024-01-16T15:27:17.712896+0000","last_clean":"2024-01-16T15:27:17.712896+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:17.712896+0000","last_undegraded":"2024-01-16T15:27:17.712896+0000","last_fullsized":"2024-01-16T15:27:17.712896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.459727+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:23:30.459727+0000","last_peered":"2024-01-16T15:23:30.459727+0000","last_clean":"2024-01-16T15:23:30.459727+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:23:30.459727+0000","last_undegraded":"2024-01-16T15:23:30.459727+0000","last_fullsized":"2024-01-16T15:23:30.459727+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:24.899317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":253,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.473268+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:25:25.473268+0000","last_peered":"2024-01-16T15:25:25.473268+0000","last_clean":"2024-01-16T15:25:25.473268+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:25:25.473268+0000","last_undegraded":"2024-01-16T15:25:25.473268+0000","last_fullsized":"2024-01-16T15:25:25.473268+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4114,"ondisk_log_size":4114,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1244374,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":489,"num_write_kb":1455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2555904,"data_stored":2495511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.222,"15min":1.274},"min":{"1min":0.64800000000000002,"5min":0.54300000000000004,"15min":0.28599999999999998},"max":{"1min":1.4159999999999999,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.302,"15min":1.4399999999999999},"min":{"1min":0.75900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.712,"5min":1.8959999999999999,"15min":25.503},"last":0.98999999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.34499999999999997},"max":{"1min":1.54,"5min":1.5620000000000001,"15min":5.5010000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.1619999999999999,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4139999999999999,"5min":1.9490000000000001,"15min":4.3369999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.68400000000000005,"15min":0.629},"max":{"1min":1.669,"5min":2.141,"15min":11.536},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.4670000000000001},"min":{"1min":0.73099999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5309999999999999,"5min":1.8500000000000001,"15min":25.478000000000002},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.40899999999999997},"max":{"1min":1.4870000000000001,"5min":1.7090000000000001,"15min":5.6369999999999996},"last":1.173},{"interface":"front","average":{"1min":1.266,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":0.80200000000000005,"5min":0.75900000000000001,"15min":0.67800000000000005},"max":{"1min":1.5589999999999999,"5min":1.9550000000000001,"15min":6.351},"last":1.0640000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.204,"15min":1.3919999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6120000000000001,"5min":1.75,"15min":25.401},"last":1.276},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6399999999999999,"5min":1.8160000000000001,"15min":5.7859999999999996},"last":1.296}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3540000000000001,"15min":1.359},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.722,"5min":2.3849999999999998,"15min":4.4539999999999997},"last":1.234},{"interface":"front","average":{"1min":1.385,"5min":1.383,"15min":1.5189999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.708,"5min":1.8480000000000001,"15min":26.327000000000002},"last":1.377}]},{"osd":6,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5469999999999999,"15min":1.6739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8280000000000001,"5min":2.0579999999999998,"15min":25.451000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5029999999999999,"15min":1.496},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.786,"5min":2.786,"15min":5.8250000000000002},"last":1.421}]}]},{"osd":6,"up_from":14,"seq":60129542483,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.68400000000000005,"15min":0.35399999999999998},"max":{"1min":1.577,"5min":8.0299999999999994,"15min":29.77},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.226,"5min":1.3220000000000001,"15min":1.415},"min":{"1min":0.88500000000000001,"5min":0.76400000000000001,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":8.2439999999999998,"15min":36.405999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.081,"15min":1.1339999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":1.728,"15min":5.2859999999999996},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.57099999999999995},"max":{"1min":1.595,"5min":1.7350000000000001,"15min":10.494},"last":0.64200000000000002}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.202,"15min":1.278},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.46,"5min":1.6579999999999999,"15min":5.0549999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.389,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.837,"5min":1.9279999999999999,"15min":5.1459999999999999},"last":1.042}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.319,"15min":1.3089999999999999},"min":{"1min":0.90200000000000002,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":1.601,"5min":1.8360000000000001,"15min":3.8079999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.3129999999999999,"15min":1.448},"min":{"1min":0.87,"5min":0.65200000000000002,"15min":0.623},"max":{"1min":1.7130000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.107}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3839999999999999,"15min":1.3480000000000001},"min":{"1min":0.97399999999999998,"5min":0.87,"15min":0.53400000000000003},"max":{"1min":1.6890000000000001,"5min":20.422999999999998,"15min":20.422999999999998},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.258,"15min":1.28},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":3.6160000000000001},"last":1.206}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.272,"15min":1.321},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.53700000000000003},"max":{"1min":1.542,"5min":7.6319999999999997,"15min":13.949999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.581,"15min":1.5720000000000001},"min":{"1min":1.113,"5min":1.113,"15min":0.51600000000000001},"max":{"1min":1.972,"5min":7.4980000000000002,"15min":11.298},"last":1.087}]},{"osd":7,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4410000000000001,"15min":1.498},"min":{"1min":1.0169999999999999,"5min":0.97499999999999998,"15min":0.57099999999999995},"max":{"1min":1.659,"5min":2.004,"15min":14.037000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.512,"15min":1.5720000000000001},"min":{"1min":1.1100000000000001,"5min":1.024,"15min":0.61599999999999999},"max":{"1min":1.9370000000000001,"5min":2.0209999999999999,"15min":4.9829999999999997},"last":1.4490000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542484,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141700,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93230140,"statfs":{"total":96636764160,"available":95467663360,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2090000000000001,"15min":1.1379999999999999},"min":{"1min":0.79300000000000004,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.6890000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.163,"15min":1.147},"min":{"1min":0.89900000000000002,"5min":0.72599999999999998,"15min":0.34799999999999998},"max":{"1min":1.526,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.2669999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1850000000000001,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":1.776,"15min":17.074999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.228,"15min":1.244},"min":{"1min":0.89000000000000001,"5min":0.77100000000000002,"15min":0.46999999999999997},"max":{"1min":1.714,"5min":1.714,"15min":4.8499999999999996},"last":1.508}]},{"osd":2,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.222,"15min":1.198},"min":{"1min":0.83899999999999997,"5min":0.72599999999999998,"15min":0.54800000000000004},"max":{"1min":1.53,"5min":2.7000000000000002,"15min":4.306},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2849999999999999,"15min":1.323},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.52200000000000002},"max":{"1min":1.494,"5min":2.9670000000000001,"15min":16.721},"last":0.76300000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.97499999999999998,"5min":0.81499999999999995,"15min":0.60299999999999998},"max":{"1min":1.859,"5min":1.905,"15min":4.8010000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":0.51200000000000001},"max":{"1min":1.7390000000000001,"5min":14.044,"15min":14.044},"last":1.399}]},{"osd":4,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2749999999999999,"15min":1.3260000000000001},"min":{"1min":0.90400000000000003,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.591,"5min":2.0139999999999998,"15min":16.382999999999999},"last":1.056},{"interface":"front","average":{"1min":1.294,"5min":1.321,"15min":1.341},"min":{"1min":0.81000000000000005,"5min":0.72099999999999997,"15min":0.502},"max":{"1min":1.982,"5min":1.982,"15min":4.6580000000000004},"last":1.361}]},{"osd":6,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3500000000000001,"15min":1.357},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.39700000000000002},"max":{"1min":1.637,"5min":1.901,"15min":4.3380000000000001},"last":1.532},{"interface":"front","average":{"1min":1.488,"5min":1.516,"15min":1.5089999999999999},"min":{"1min":1.137,"5min":0.94199999999999995,"15min":0.57899999999999996},"max":{"1min":1.6919999999999999,"5min":1.907,"15min":3.3719999999999999},"last":1.3300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5,"15min":1.5029999999999999},"min":{"1min":0.95999999999999996,"5min":0.95599999999999996,"15min":0.45000000000000001},"max":{"1min":2.044,"5min":2.044,"15min":4.742},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5700000000000001,"15min":1.5509999999999999},"min":{"1min":1.0820000000000001,"5min":1.0629999999999999,"15min":0.58799999999999997},"max":{"1min":1.821,"5min":2.0609999999999999,"15min":2.6520000000000001},"last":1.6220000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542483,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932292,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439548,"statfs":{"total":96636764160,"available":94658097152,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":1.252},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":1.1379999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.753},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.1879999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.5509999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.601}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.724},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.6359999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542485,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872984,"kb_used_data":1751632,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498856,"statfs":{"total":96636764160,"available":94718828544,"internally_reserved":0,"allocated":1793671168,"data_stored":1792833889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.68899999999999995,"5min":0.58499999999999996,"15min":0.34699999999999998},"max":{"1min":1.3640000000000001,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.0469999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.98199999999999998,"15min":1.0249999999999999},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":1.359,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":1.335},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2569999999999999,"15min":1.226},"min":{"1min":0.64000000000000001,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.7909999999999999,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":1.173}]},{"osd":2,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.244,"15min":1.26},"min":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.36699999999999999},"max":{"1min":1.577,"5min":2.258,"15min":3.5249999999999999},"last":1.514},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.45200000000000001},"max":{"1min":1.897,"5min":2.339,"15min":4.6529999999999996},"last":0.86299999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.323,"15min":1.319},"min":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":0.44500000000000001},"max":{"1min":1.6759999999999999,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.73299999999999998},"max":{"1min":1.861,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.2290000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4890000000000001,"15min":1.5009999999999999},"min":{"1min":1.224,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":1.738,"5min":2.0249999999999999,"15min":4.484},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.66500000000000004},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.879}]},{"osd":6,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5389999999999999,"15min":1.5620000000000001},"min":{"1min":1.1220000000000001,"5min":0.92700000000000005,"15min":0.71499999999999997},"max":{"1min":1.96,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":2.157},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6080000000000001,"15min":1.613},"min":{"1min":1.1399999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.069,"5min":2.169,"15min":4.5880000000000001},"last":1.4810000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5329999999999999,"15min":1.5509999999999999},"min":{"1min":0.98799999999999999,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":1.8480000000000001,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":2.0649999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.6299999999999999,"15min":1.6279999999999999},"min":{"1min":1.2669999999999999,"5min":1.008,"15min":0.64700000000000002},"max":{"1min":2.1160000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":2.2210000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542483,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251776,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":115774,"kb_avail":93120064,"statfs":{"total":96636764160,"available":95354945536,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118553029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.90300000000000002,"15min":0.93600000000000005},"min":{"1min":0.56899999999999995,"5min":0.54600000000000004,"15min":0.36099999999999999},"max":{"1min":1.379,"5min":1.6659999999999999,"15min":9.657},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.004,"15min":1.0740000000000001},"min":{"1min":0.60599999999999998,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.3460000000000001,"5min":1.7649999999999999,"15min":20.407},"last":0.80000000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93500000000000005,"15min":1.0429999999999999},"min":{"1min":0.72299999999999998,"5min":0.64000000000000001,"15min":0.41499999999999998},"max":{"1min":1.5229999999999999,"5min":1.53,"15min":11.406000000000001},"last":0.874},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.319,"15min":1.409},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.45300000000000001},"max":{"1min":1.7050000000000001,"5min":1.8049999999999999,"15min":20.513000000000002},"last":0.92300000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.389,"15min":1.371},"min":{"1min":0.95099999999999996,"5min":0.56100000000000005,"15min":0.53400000000000003},"max":{"1min":1.726,"5min":2.0110000000000001,"15min":3.1080000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.288,"15min":1.3320000000000001},"min":{"1min":0.88200000000000001,"5min":0.68200000000000005,"15min":0.49099999999999999},"max":{"1min":1.7190000000000001,"5min":1.7949999999999999,"15min":9.7390000000000008},"last":1.0820000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.494,"15min":1.46},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.66100000000000003},"max":{"1min":1.885,"5min":12.669,"15min":12.779},"last":1.546},{"interface":"front","average":{"1min":1.286,"5min":1.3560000000000001,"15min":1.466},"min":{"1min":1.018,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":25.661000000000001},"last":1.012}]},{"osd":5,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.446,"15min":1.448},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.68300000000000005},"max":{"1min":1.8440000000000001,"5min":1.9039999999999999,"15min":3.0209999999999999},"last":1.052},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.5369999999999999},"min":{"1min":1.0860000000000001,"5min":0.90200000000000002,"15min":0.59799999999999998},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":9.7850000000000001},"last":1.415}]},{"osd":6,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.462,"15min":1.651},"min":{"1min":1.075,"5min":1.048,"15min":0.57299999999999995},"max":{"1min":1.8320000000000001,"5min":1.8660000000000001,"15min":25.603999999999999},"last":1.214},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4370000000000001,"15min":1.518},"min":{"1min":0.752,"5min":0.751,"15min":0.629},"max":{"1min":2.0019999999999998,"5min":2.077,"15min":15.968999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.655,"15min":1.698},"min":{"1min":1.173,"5min":1.173,"15min":0.61699999999999999},"max":{"1min":1.8220000000000001,"5min":9.3900000000000006,"15min":24.509},"last":1.147},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5489999999999999,"15min":1.6259999999999999},"min":{"1min":0.93799999999999994,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":2.081,"5min":2.161,"15min":10.455},"last":1.3240000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1448324,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92923516,"statfs":{"total":96636764160,"available":95153680384,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.90400000000000003,"15min":0.93000000000000005},"min":{"1min":0.64800000000000002,"5min":0.53000000000000003,"15min":0.28399999999999997},"max":{"1min":1.6100000000000001,"5min":1.718,"15min":4.266},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1220000000000001,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.34999999999999998},"max":{"1min":2.153,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.137,"15min":1.135},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.31},"max":{"1min":1.647,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.39300000000000002},"max":{"1min":1.579,"5min":1.8340000000000001,"15min":4.1559999999999997},"last":1.2050000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.456,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.439},"max":{"1min":1.7529999999999999,"5min":1.9930000000000001,"15min":4.1980000000000004},"last":1.105},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.46600000000000003},"max":{"1min":1.798,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":0.61099999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5349999999999999,"15min":1.5109999999999999},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.69899999999999995},"max":{"1min":1.887,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.528,"15min":1.52},"min":{"1min":1.0309999999999999,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":1.613,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.1759999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.522,"15min":1.466},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":1.9219999999999999,"5min":5.194,"15min":5.194},"last":1.034},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":1.714,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.0600000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.516,"15min":1.5189999999999999},"min":{"1min":1.0549999999999999,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.544,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.264},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6539999999999999,"15min":1.6080000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.70699999999999996},"max":{"1min":2.0659999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":0.83799999999999997}]},{"osd":7,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.609,"15min":1.589},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.69999999999999996},"max":{"1min":1.9690000000000001,"5min":16.140000000000001,"15min":16.140000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6759999999999999,"15min":1.6339999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.78700000000000003},"max":{"1min":2.4449999999999998,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.319}]}]},{"osd":0,"up_from":14,"seq":60129542483,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073492,"kb_used_data":1997964,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298348,"statfs":{"total":96636764160,"available":94513508352,"internally_reserved":0,"allocated":2045915136,"data_stored":2045075445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0309999999999999,"15min":1.0249999999999999},"min":{"1min":0.55500000000000005,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.3260000000000001,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.2370000000000001,"15min":1.252},"min":{"1min":0.497,"5min":0.47999999999999998,"15min":0.33900000000000002},"max":{"1min":1.4470000000000001,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1120000000000001,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.46700000000000003,"15min":0.40000000000000002},"max":{"1min":1.5800000000000001,"5min":1.875,"15min":7.5940000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1299999999999999,"15min":1.175},"min":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.39600000000000002},"max":{"1min":1.665,"5min":1.8,"15min":5.3550000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.37},"max":{"1min":1.2170000000000001,"5min":5.851,"15min":5.851},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.335,"5min":1.3169999999999999,"15min":1.343},"min":{"1min":0.65800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.5660000000000001,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":0.83199999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.304,"15min":1.3400000000000001},"min":{"1min":0.80400000000000005,"5min":0.79600000000000004,"15min":0.69099999999999995},"max":{"1min":1.7829999999999999,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.4530000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":1.7330000000000001,"5min":2.141,"15min":5.3109999999999999},"last":0.91800000000000004}]},{"osd":5,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.4670000000000001,"15min":1.4930000000000001},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.75,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.427,"15min":1.454},"min":{"1min":0.96599999999999997,"5min":0.84699999999999998,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":2.27,"15min":8.3949999999999996},"last":0.94999999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5840000000000001,"15min":1.659},"min":{"1min":1.0669999999999999,"5min":0.82299999999999995,"15min":0.72999999999999998},"max":{"1min":1.8049999999999999,"5min":2.173,"15min":15.159000000000001},"last":1.054},{"interface":"front","average":{"1min":1.379,"5min":1.526,"15min":1.6259999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.58999999999999997},"max":{"1min":1.8700000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.236}]},{"osd":7,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.617,"15min":1.667},"min":{"1min":1,"5min":0.999,"15min":0.60899999999999999},"max":{"1min":1.819,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":1.018,"5min":0.873,"15min":0.65800000000000003},"max":{"1min":1.8340000000000001,"5min":2.2719999999999998,"15min":6.149},"last":1.006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1257472,"data_stored":1239741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1246462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:27:25.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:25.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.983+0000 7efc4e000640 1 Processor -- start 2024-01-16T15:27:25.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.983+0000 7efc4e000640 1 -- start start 2024-01-16T15:27:25.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.984+0000 7efc4e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc480d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.984+0000 7efc4e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efc480b1650 0x7efc480d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.984+0000 7efc4e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc480cf0c0 con 0x7efc480d14a0 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.984+0000 7efc4e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc480b01b0 con 0x7efc480b1650 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.984+0000 7efc4e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4815d990 con 0x7efc4815d5b0 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.985+0000 7efc4d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efc480b1650 0x7efc480d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.985+0000 7efc4cc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc480d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.985+0000 7efc4d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efc480b1650 0x7efc480d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:36288/0 (socket says 172.21.15.43:36288) 2024-01-16T15:27:25.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.985+0000 7efc4d600640 1 -- 172.21.15.43:0/1518940772 learned_addr learned my addr 172.21.15.43:0/1518940772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:25.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.986+0000 7efc4d600640 1 -- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 msgr2=0x7efc480d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:25.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.986+0000 7efc4d600640 1 --2- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc480d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.986+0000 7efc4d600640 1 -- 172.21.15.43:0/1518940772 >> v1:172.21.15.69:6789/0 conn(0x7efc480d14a0 legacy=0x7efc480cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.986+0000 7efc4d600640 1 -- 172.21.15.43:0/1518940772 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc480d1e40 con 0x7efc480b1650 2024-01-16T15:27:25.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.987+0000 7efc4d600640 1 --2- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480b1650 0x7efc480d1900 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7efc40009e50 tx=0x7efc4002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a243683c65d588f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.988+0000 7efc47400640 1 -- 172.21.15.43:0/1518940772 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4002cb00 con 0x7efc480b1650 2024-01-16T15:27:25.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.988+0000 7efc47400640 1 -- 172.21.15.43:0/1518940772 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4002cc60 con 0x7efc480b1650 2024-01-16T15:27:25.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.988+0000 7efc47400640 1 -- 172.21.15.43:0/1518940772 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc400343f0 con 0x7efc480b1650 2024-01-16T15:27:25.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 -- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480b1650 msgr2=0x7efc480d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 --2- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480b1650 0x7efc480d1900 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7efc40009e50 tx=0x7efc4002bf60 comp rx=0 tx=0).stop 2024-01-16T15:27:25.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 -- 172.21.15.43:0/1518940772 shutdown_connections 2024-01-16T15:27:25.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 --2- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480b1650 0x7efc480d1900 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 --2- 172.21.15.43:0/1518940772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc480d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 -- 172.21.15.43:0/1518940772 >> 172.21.15.43:0/1518940772 conn(0x7efc4809c3b0 msgr2=0x7efc480d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:25.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 -- 172.21.15.43:0/1518940772 shutdown_connections 2024-01-16T15:27:25.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.989+0000 7efc4e000640 1 -- 172.21.15.43:0/1518940772 wait complete. 2024-01-16T15:27:25.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.991+0000 7efc4e000640 1 Processor -- start 2024-01-16T15:27:25.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.991+0000 7efc4e000640 1 -- start start 2024-01-16T15:27:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480d14a0 0x7efc481504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc481f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4814cde0 con 0x7efc480b1650 2024-01-16T15:27:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc480cb2c0 con 0x7efc480d14a0 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4e000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4815e380 con 0x7efc4815d5b0 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53842/0 (socket says 172.21.15.43:53842) 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.992+0000 7efc4cc00640 1 -- 172.21.15.43:0/1956404554 learned_addr learned my addr 172.21.15.43:0/1956404554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc47e00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480d14a0 0x7efc481504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4d600640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc481f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 -- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 msgr2=0x7efc481f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc481f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 -- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480d14a0 msgr2=0x7efc481504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480d14a0 0x7efc481504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 -- 172.21.15.43:0/1956404554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40009b30 con 0x7efc480b1650 2024-01-16T15:27:25.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.993+0000 7efc4cc00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7efc3800e870 tx=0x7efc3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.994+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380094f0 con 0x7efc480b1650 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.994+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3800c940 con 0x7efc480b1650 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.994+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4815e660 con 0x7efc480b1650 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.994+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3800ccf0 con 0x7efc480b1650 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.994+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4815ec20 con 0x7efc480b1650 2024-01-16T15:27:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.995+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4804e840 con 0x7efc480b1650 2024-01-16T15:27:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.996+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7efc38013470 con 0x7efc480b1650 2024-01-16T15:27:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.997+0000 7efc44c00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 0x7efc2006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.997+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efc38088c70 con 0x7efc480b1650 2024-01-16T15:27:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.997+0000 7efc47e00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 0x7efc2006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:25.998+0000 7efc47e00640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 0x7efc2006e980 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efc30007fd0 tx=0x7efc30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.001+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc380551c0 con 0x7efc480b1650 2024-01-16T15:27:26.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.198+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc480678d0 con 0x7efc480b1650 2024-01-16T15:27:26.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.200+0000 7efc44c00640 1 -- 172.21.15.43:0/1956404554 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+16615 (secure 0 0 0) 0x7efc38021090 con 0x7efc480b1650 2024-01-16T15:27:26.200 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:26.200 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":115,"fsid":"eb89d060-5b4a-4a34-8ffb-6d8b15e6800a","created":"2024-01-16T14:59:07.709397+0000","modified":"2024-01-16T15:19:15.246810+0000","last_up_change":"2024-01-16T14:59:19.352588+0000","last_in_change":"2024-01-16T14:59:11.188841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:20.855872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:18:47.672212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:18:48.980104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4dce6b-3fbe-42d7-8b98-7d3cfab0899d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6801","nonce":1320786927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6803","nonce":1320786927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6807","nonce":1320786927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1320786927},{"type":"v1","addr":"172.21.15.43:6805","nonce":1320786927}]},"public_addr":"172.21.15.43:6801/1320786927","cluster_addr":"172.21.15.43:6803/1320786927","heartbeat_back_addr":"172.21.15.43:6807/1320786927","heartbeat_front_addr":"172.21.15.43:6805/1320786927","state":["exists","up"]},{"osd":1,"uuid":"df57693a-062d-49fd-86dc-1bf0760e6ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6823","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6825","nonce":2025179551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6827","nonce":2025179551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6831","nonce":2025179551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2025179551},{"type":"v1","addr":"172.21.15.43:6829","nonce":2025179551}]},"public_addr":"172.21.15.43:6825/2025179551","cluster_addr":"172.21.15.43:6827/2025179551","heartbeat_back_addr":"172.21.15.43:6831/2025179551","heartbeat_front_addr":"172.21.15.43:6829/2025179551","state":["exists","up"]},{"osd":2,"uuid":"8f01e100-16a0-45c4-878f-3541ba179b4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6809","nonce":2671083580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6811","nonce":2671083580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6818","nonce":2671083580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":2671083580},{"type":"v1","addr":"172.21.15.43:6814","nonce":2671083580}]},"public_addr":"172.21.15.43:6809/2671083580","cluster_addr":"172.21.15.43:6811/2671083580","heartbeat_back_addr":"172.21.15.43:6818/2671083580","heartbeat_front_addr":"172.21.15.43:6814/2671083580","state":["exists","up"]},{"osd":3,"uuid":"d8ae43ad-6974-4048-a6dc-dcf76520ac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6815","nonce":889849964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6819","nonce":889849964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6824","nonce":889849964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":889849964},{"type":"v1","addr":"172.21.15.43:6821","nonce":889849964}]},"public_addr":"172.21.15.43:6815/889849964","cluster_addr":"172.21.15.43:6819/889849964","heartbeat_back_addr":"172.21.15.43:6824/889849964","heartbeat_front_addr":"172.21.15.43:6821/889849964","state":["exists","up"]},{"osd":4,"uuid":"cd697498-049e-4431-9b54-53ef478ca92c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6817","nonce":2273091943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6819","nonce":2273091943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6823","nonce":2273091943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2273091943},{"type":"v1","addr":"172.21.15.69:6821","nonce":2273091943}]},"public_addr":"172.21.15.69:6817/2273091943","cluster_addr":"172.21.15.69:6819/2273091943","heartbeat_back_addr":"172.21.15.69:6823/2273091943","heartbeat_front_addr":"172.21.15.69:6821/2273091943","state":["exists","up"]},{"osd":5,"uuid":"57205772-35f8-44d3-a420-bf41c3ac6ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6809","nonce":1266529003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6811","nonce":1266529003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6815","nonce":1266529003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1266529003},{"type":"v1","addr":"172.21.15.69:6813","nonce":1266529003}]},"public_addr":"172.21.15.69:6809/1266529003","cluster_addr":"172.21.15.69:6811/1266529003","heartbeat_back_addr":"172.21.15.69:6815/1266529003","heartbeat_front_addr":"172.21.15.69:6813/1266529003","state":["exists","up"]},{"osd":6,"uuid":"7b61ef01-2c5e-476c-95f0-665c3757bc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6801","nonce":3561904235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6803","nonce":3561904235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6807","nonce":3561904235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3561904235},{"type":"v1","addr":"172.21.15.69:6805","nonce":3561904235}]},"public_addr":"172.21.15.69:6801/3561904235","cluster_addr":"172.21.15.69:6803/3561904235","heartbeat_back_addr":"172.21.15.69:6807/3561904235","heartbeat_front_addr":"172.21.15.69:6805/3561904235","state":["exists","up"]},{"osd":7,"uuid":"1a8a3071-bb4a-4fca-a170-8dad620d0dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6825","nonce":1736397519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6827","nonce":1736397519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6831","nonce":1736397519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1736397519},{"type":"v1","addr":"172.21.15.69:6829","nonce":1736397519}]},"public_addr":"172.21.15.69:6825/1736397519","cluster_addr":"172.21.15.69:6827/1736397519","heartbeat_back_addr":"172.21.15.69:6831/1736397519","heartbeat_front_addr":"172.21.15.69:6829/1736397519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.743201+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.585169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.489967+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.830894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.533528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.432520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.678966+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:16.839872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:26.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 msgr2=0x7efc2006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 0x7efc2006e980 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efc30007fd0 tx=0x7efc30009040 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 msgr2=0x7efc48152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7efc3800e870 tx=0x7efc3800ed30 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 shutdown_connections 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efc2006c4c0 0x7efc2006e980 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc480d14a0 0x7efc481504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc4815d5b0 0x7efc481f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 --2- 172.21.15.43:0/1956404554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc480b1650 0x7efc48152a60 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:26.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 >> 172.21.15.43:0/1956404554 conn(0x7efc4809c3b0 msgr2=0x7efc480b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:26.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.203+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 shutdown_connections 2024-01-16T15:27:26.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:26.204+0000 7efc4e000640 1 -- 172.21.15.43:0/1956404554 wait complete. 2024-01-16T15:27:27.224 INFO:tasks.ceph:Scrubbing osd.0 2024-01-16T15:27:27.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:27.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.456+0000 7f0b5de00640 1 Processor -- start 2024-01-16T15:27:27.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.456+0000 7f0b5de00640 1 -- start start 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b5815d3b0 0x7f0b580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b580d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5de00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580d2010 con 0x7f0b580d1bc0 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5de00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b580c6bf0 con 0x7f0b5815d3b0 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580d1400 con 0x7f0b580d0fa0 2024-01-16T15:27:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5d400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0b580d1bc0 0x7f0b580b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33192/0 (socket says 172.21.15.43:33192) 2024-01-16T15:27:27.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5d400640 1 -- 172.21.15.43:0/3835103362 learned_addr learned my addr 172.21.15.43:0/3835103362 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:27.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.458+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/3835103362 >> v2:172.21.15.69:3300/0 conn(0x7f0b5815d3b0 0x7f0b580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57e00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b580d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260676009 0 0) 0x7f0b50001490 con 0x7f0b580d1bc0 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b38003520 con 0x7f0b580d1bc0 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095549274 0 0) 0x7f0b38003520 con 0x7f0b580d1bc0 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 msgr2=0x7f0b580d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b580d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 >> v2:172.21.15.69:3300/0 conn(0x7f0b5815d3b0 msgr2=0x7f0b580ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 --2- 172.21.15.43:0/3835103362 >> v2:172.21.15.69:3300/0 conn(0x7f0b5815d3b0 0x7f0b580ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580cedc0 con 0x7f0b580d1bc0 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f0b50001b80 con 0x7f0b580d1bc0 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 >> v1:172.21.15.43:6789/0 conn(0x7f0b580d1bc0 legacy=0x7f0b580b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b380039e0 0x7f0b38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b38004d00 0x7f0b38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580c6bf0 con 0x7f0b380039e0 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580d1400 con 0x7f0b38004d00 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.459+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3800ba00 con 0x7f0b580d0fa0 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b380039e0 0x7f0b38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b57e00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5d400640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b38004d00 0x7f0b38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 -- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 msgr2=0x7f0b380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580d0fa0 0x7f0b380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 -- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b38004d00 msgr2=0x7f0b38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:27.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b38004d00 0x7f0b38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.460+0000 7f0b5ca00640 1 -- 172.21.15.43:0/3835103362 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b38003520 con 0x7f0b380039e0 2024-01-16T15:27:27.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.461+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b380039e0 0x7f0b38003e00 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009ac0 tx=0x7f0b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394490b3a3deae13 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4802c710 con 0x7f0b380039e0 2024-01-16T15:27:27.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4802c870 con 0x7f0b380039e0 2024-01-16T15:27:27.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b57400640 1 -- 172.21.15.43:0/3835103362 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4802cc00 con 0x7f0b380039e0 2024-01-16T15:27:27.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b5de00640 1 -- 172.21.15.43:0/3835103362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b380039e0 msgr2=0x7f0b38003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b5de00640 1 --2- 172.21.15.43:0/3835103362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b380039e0 0x7f0b38003e00 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009ac0 tx=0x7f0b380057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b5d400640 1 -- 172.21.15.43:0/3835103362 reap_dead start 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b5de00640 1 -- 172.21.15.43:0/3835103362 shutdown_connections 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.462+0000 7f0b5de00640 1 -- 172.21.15.43:0/3835103362 >> 172.21.15.43:0/3835103362 conn(0x7f0b580d30a0 msgr2=0x7f0b580d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.463+0000 7f0b5de00640 1 -- 172.21.15.43:0/3835103362 shutdown_connections 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.463+0000 7f0b5de00640 1 -- 172.21.15.43:0/3835103362 wait complete. 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.464+0000 7f0b5de00640 1 Processor -- start 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.464+0000 7f0b5de00640 1 -- start start 2024-01-16T15:27:27.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b380039e0 0x7f0b5815b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 0x7f0b580cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580cc750 0x7f0b5809dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5809cf10 con 0x7f0b581f3180 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5809d1f0 con 0x7f0b380039e0 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.465+0000 7f0b5de00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5809d4d0 con 0x7f0b580cc750 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 0x7f0b580cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 0x7f0b580cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53868/0 (socket says 172.21.15.43:53868) 2024-01-16T15:27:27.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 -- 172.21.15.43:0/346142833 learned_addr learned my addr 172.21.15.43:0/346142833 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b380039e0 0x7f0b5815b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b5d400640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580cc750 0x7f0b5809dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 -- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580cc750 msgr2=0x7f0b5809dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b580cc750 0x7f0b5809dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 -- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b380039e0 msgr2=0x7f0b5815b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b380039e0 0x7f0b5815b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.466+0000 7f0b57e00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5809d7b0 con 0x7f0b581f3180 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b57e00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 0x7f0b580cc210 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0b40011600 tx=0x7f0b40011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4000f920 con 0x7f0b581f3180 2024-01-16T15:27:27.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4000fa80 con 0x7f0b581f3180 2024-01-16T15:27:27.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4000fdf0 con 0x7f0b581f3180 2024-01-16T15:27:27.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5809da90 con 0x7f0b581f3180 2024-01-16T15:27:27.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.467+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5815c240 con 0x7f0b581f3180 2024-01-16T15:27:27.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.468+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b580c7400 con 0x7f0b581f3180 2024-01-16T15:27:27.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.469+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f0b40015460 con 0x7f0b581f3180 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.470+0000 7f0b54c00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b3006c510 0x7f0b3006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.470+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0b40089070 con 0x7f0b581f3180 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b3006c510 0x7f0b3006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b54c00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7f0b30074850 0x7f0b30076cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b30077380 con 0x7f0b30074850 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b4008eba0 con 0x7f0b581f3180 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b5d400640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7f0b30074850 0x7f0b30076cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.471+0000 7f0b5ca00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b3006c510 0x7f0b3006e9d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0b48036000 tx=0x7f0b48030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.472+0000 7f0b5d400640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7f0b30074850 0x7f0b30076cb0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.472+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== osd.0 v2:172.21.15.43:6800/1320786927 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0b30077380 con 0x7f0b30074850 2024-01-16T15:27:27.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.487+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 --> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0b580c7120 con 0x7f0b30074850 2024-01-16T15:27:27.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b54c00640 1 -- 172.21.15.43:0/346142833 <== osd.0 v2:172.21.15.43:6800/1320786927 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0b500019a0 con 0x7f0b30074850 2024-01-16T15:27:27.487 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:27.487 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:27.487 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:27.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7f0b30074850 msgr2=0x7f0b30076cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6800/1320786927,v1:172.21.15.43:6801/1320786927] conn(0x7f0b30074850 0x7f0b30076cb0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b3006c510 msgr2=0x7f0b3006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f0b3006c510 0x7f0b3006e9d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0b48036000 tx=0x7f0b48030000 comp rx=0 tx=0).stop 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 msgr2=0x7f0b580cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.488+0000 7f0b5de00640 1 --2- 172.21.15.43:0/346142833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b581f3180 0x7f0b580cc210 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0b40011600 tx=0x7f0b40011ac0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.489+0000 7f0b5d400640 1 -- 172.21.15.43:0/346142833 reap_dead start 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.489+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 shutdown_connections 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.489+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 >> 172.21.15.43:0/346142833 conn(0x7f0b580d30a0 msgr2=0x7f0b5806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.489+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 shutdown_connections 2024-01-16T15:27:27.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.489+0000 7f0b5de00640 1 -- 172.21.15.43:0/346142833 wait complete. 2024-01-16T15:27:27.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-16T15:27:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.729+0000 7f1c37000640 1 Processor -- start 2024-01-16T15:27:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.729+0000 7f1c37000640 1 -- start start 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.730+0000 7f1c37000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1c3006dc30 0x7f1c300acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.730+0000 7f1c37000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c30071cd0 con 0x7f1c30160970 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.730+0000 7f1c37000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c300b2120 con 0x7f1c30070780 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.730+0000 7f1c37000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c300b2400 con 0x7f1c3006dc30 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1c3006dc30 0x7f1c300acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1c3006dc30 0x7f1c300acbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53878/0 (socket says 172.21.15.43:53878) 2024-01-16T15:27:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 -- 172.21.15.43:0/1281948919 learned_addr learned my addr 172.21.15.43:0/1281948919 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 -- 172.21.15.43:0/1281948919 >> v1:172.21.15.69:6789/0 conn(0x7f1c30070780 legacy=0x7f1c3006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 -- 172.21.15.43:0/1281948919 >> v1:172.21.15.43:6789/0 conn(0x7f1c30160970 legacy=0x7f1c300b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.731+0000 7f1c35000640 1 -- 172.21.15.43:0/1281948919 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c300b26e0 con 0x7f1c3006dc30 2024-01-16T15:27:27.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c35000640 1 --2- 172.21.15.43:0/1281948919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3006dc30 0x7f1c300acbe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009870 tx=0x7f1c2802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7826f8c22c913337 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c2f400640 1 -- 172.21.15.43:0/1281948919 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031860 con 0x7f1c3006dc30 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c2f400640 1 -- 172.21.15.43:0/1281948919 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c280319c0 con 0x7f1c3006dc30 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c2f400640 1 -- 172.21.15.43:0/1281948919 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031d50 con 0x7f1c3006dc30 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c37000640 1 -- 172.21.15.43:0/1281948919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3006dc30 msgr2=0x7f1c300acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.733+0000 7f1c37000640 1 --2- 172.21.15.43:0/1281948919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3006dc30 0x7f1c300acbe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009870 tx=0x7f1c2802b900 comp rx=0 tx=0).stop 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.734+0000 7f1c37000640 1 -- 172.21.15.43:0/1281948919 shutdown_connections 2024-01-16T15:27:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.734+0000 7f1c37000640 1 --2- 172.21.15.43:0/1281948919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3006dc30 0x7f1c300acbe0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.734+0000 7f1c37000640 1 -- 172.21.15.43:0/1281948919 >> 172.21.15.43:0/1281948919 conn(0x7f1c300b0d80 msgr2=0x7f1c300b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.734+0000 7f1c37000640 1 -- 172.21.15.43:0/1281948919 shutdown_connections 2024-01-16T15:27:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.734+0000 7f1c37000640 1 -- 172.21.15.43:0/1281948919 wait complete. 2024-01-16T15:27:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.735+0000 7f1c37000640 1 Processor -- start 2024-01-16T15:27:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.736+0000 7f1c37000640 1 -- start start 2024-01-16T15:27:27.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.736+0000 7f1c37000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c3006dc30 0x7f1c3015fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.736+0000 7f1c37000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c30070780 0x7f1c301462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c37000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c37000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c301585c0 con 0x7f1c30160970 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c37000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c30150cf0 con 0x7f1c3006dc30 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c37000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c30157940 con 0x7f1c30070780 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53888/0 (socket says 172.21.15.43:53888) 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 -- 172.21.15.43:0/2578146346 learned_addr learned my addr 172.21.15.43:0/2578146346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c2fe00640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c3006dc30 0x7f1c3015fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c27200640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c30070780 0x7f1c301462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 -- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c30070780 msgr2=0x7f1c301462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c30070780 0x7f1c301462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 -- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c3006dc30 msgr2=0x7f1c3015fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c3006dc30 0x7f1c3015fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.737+0000 7f1c35000640 1 -- 172.21.15.43:0/2578146346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280094d0 con 0x7f1c30160970 2024-01-16T15:27:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c35000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802b940 tx=0x7f1c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c280443f0 con 0x7f1c30160970 2024-01-16T15:27:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c28044570 con 0x7f1c30160970 2024-01-16T15:27:27.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28044920 con 0x7f1c30160970 2024-01-16T15:27:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c30157c20 con 0x7f1c30160970 2024-01-16T15:27:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.738+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c301581e0 con 0x7f1c30160970 2024-01-16T15:27:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.740+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c30074770 con 0x7f1c30160970 2024-01-16T15:27:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.741+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f1c28044a80 con 0x7f1c30160970 2024-01-16T15:27:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.742+0000 7f1c2cc00640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 0x7f1c0806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.742+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1c280ab330 con 0x7f1c30160970 2024-01-16T15:27:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.742+0000 7f1c2fe00640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 0x7f1c0806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.743+0000 7f1c2fe00640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 0x7f1c0806e980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1c300ae6f0 tx=0x7f1c20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:27.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.746+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c28077880 con 0x7f1c30160970 2024-01-16T15:27:27.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.940+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c30071fb0 con 0x7f1c0806c4c0 2024-01-16T15:27:27.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.941+0000 7f1c2cc00640 1 -- 172.21.15.43:0/2578146346 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c20003970 con 0x7f1c0806c4c0 2024-01-16T15:27:27.940 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 0 to deep-scrub 2024-01-16T15:27:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.944+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 msgr2=0x7f1c0806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.944+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 0x7f1c0806e980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1c300ae6f0 tx=0x7f1c20009040 comp rx=0 tx=0).stop 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.944+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 msgr2=0x7f1c3006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.944+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802b940 tx=0x7f1c280046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 shutdown_connections 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f1c0806c4c0 0x7f1c0806e980 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c30160970 0x7f1c3006c510 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c30070780 0x7f1c301462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 --2- 172.21.15.43:0/2578146346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c3006dc30 0x7f1c3015fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:27.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 >> 172.21.15.43:0/2578146346 conn(0x7f1c300b0d80 msgr2=0x7f1c300b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:27.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 shutdown_connections 2024-01-16T15:27:27.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:27.945+0000 7f1c37000640 1 -- 172.21.15.43:0/2578146346 wait complete. 2024-01-16T15:27:27.964 INFO:tasks.ceph:Scrubbing osd.1 2024-01-16T15:27:27.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:28.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.193+0000 7f46dc600640 1 Processor -- start 2024-01-16T15:27:28.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.193+0000 7f46dc600640 1 -- start start 2024-01-16T15:27:28.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46dc600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46d40acd70 0x7f46d406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46dc600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f46d40ae490 0x7f46d40ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46dc600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d406d7a0 con 0x7f46d40ad690 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46dc600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d415a5b0 con 0x7f46d40ae490 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46dc600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d406ce70 con 0x7f46d40acd70 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46d9200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f46d40ae490 0x7f46d40ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46d9200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f46d40ae490 0x7f46d40ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53892/0 (socket says 172.21.15.43:53892) 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.195+0000 7f46d9200640 1 -- 172.21.15.43:0/326064001 learned_addr learned my addr 172.21.15.43:0/326064001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:28.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.196+0000 7f46d9c00640 1 --2- 172.21.15.43:0/326064001 >> v2:172.21.15.69:3300/0 conn(0x7f46d40acd70 0x7f46d406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.196+0000 7f46d9200640 1 -- 172.21.15.43:0/326064001 >> v2:172.21.15.69:3300/0 conn(0x7f46d40acd70 msgr2=0x7f46d406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.196+0000 7f46d9200640 1 --2- 172.21.15.43:0/326064001 >> v2:172.21.15.69:3300/0 conn(0x7f46d40acd70 0x7f46d406dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.196+0000 7f46d9200640 1 -- 172.21.15.43:0/326064001 >> v1:172.21.15.69:6789/0 conn(0x7f46d40ad690 legacy=0x7f46d4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.196+0000 7f46d9200640 1 -- 172.21.15.43:0/326064001 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d406d150 con 0x7f46d40ae490 2024-01-16T15:27:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.197+0000 7f46d9200640 1 --2- 172.21.15.43:0/326064001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40ae490 0x7f46d40ad150 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f46d40700f0 tx=0x7f46bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78235a8fd191a11a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.197+0000 7f46d3e00640 1 -- 172.21.15.43:0/326064001 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc031870 con 0x7f46d40ae490 2024-01-16T15:27:28.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.197+0000 7f46d3e00640 1 -- 172.21.15.43:0/326064001 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc0319d0 con 0x7f46d40ae490 2024-01-16T15:27:28.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.197+0000 7f46d3e00640 1 -- 172.21.15.43:0/326064001 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc031d60 con 0x7f46d40ae490 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 -- 172.21.15.43:0/326064001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40ae490 msgr2=0x7f46d40ad150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 --2- 172.21.15.43:0/326064001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40ae490 0x7f46d40ad150 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f46d40700f0 tx=0x7f46bc02b910 comp rx=0 tx=0).stop 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 -- 172.21.15.43:0/326064001 shutdown_connections 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 --2- 172.21.15.43:0/326064001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40ae490 0x7f46d40ad150 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 --2- 172.21.15.43:0/326064001 >> v2:172.21.15.69:3300/0 conn(0x7f46d40acd70 0x7f46d406dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 -- 172.21.15.43:0/326064001 >> 172.21.15.43:0/326064001 conn(0x7f46d40b0ba0 msgr2=0x7f46d40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 -- 172.21.15.43:0/326064001 shutdown_connections 2024-01-16T15:27:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.198+0000 7f46dc600640 1 -- 172.21.15.43:0/326064001 wait complete. 2024-01-16T15:27:28.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.200+0000 7f46dc600640 1 Processor -- start 2024-01-16T15:27:28.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.200+0000 7f46dc600640 1 -- start start 2024-01-16T15:27:28.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 0x7f46d40afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d40ad690 0x7f46d4145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46d40ae490 0x7f46d414c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d409a780 con 0x7f46d40acd70 2024-01-16T15:27:28.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d414ca20 con 0x7f46d40ad690 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46dc600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d409c950 con 0x7f46d40ae490 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 0x7f46d40afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 0x7f46d40afcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53902/0 (socket says 172.21.15.43:53902) 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 -- 172.21.15.43:0/650597615 learned_addr learned my addr 172.21.15.43:0/650597615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 -- 172.21.15.43:0/650597615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46d40ae490 msgr2=0x7f46d414c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46d40ae490 0x7f46d414c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 -- 172.21.15.43:0/650597615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d40ad690 msgr2=0x7f46d4145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d40ad690 0x7f46d4145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.201+0000 7f46d9c00640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0094d0 con 0x7f46d40acd70 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46d9c00640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 0x7f46d40afcd0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f46c400d5f0 tx=0x7f46c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40125f0 con 0x7f46d40acd70 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4012750 con 0x7f46d40acd70 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4012ae0 con 0x7f46d40acd70 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d409cc10 con 0x7f46d40acd70 2024-01-16T15:27:28.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.202+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d409a3c0 con 0x7f46d40acd70 2024-01-16T15:27:28.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.203+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46d40b3d90 con 0x7f46d40acd70 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.204+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f46c4012c40 con 0x7f46d40acd70 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d1600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46ac06c4c0 0x7f46ac06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f46c4088cc0 con 0x7f46d40acd70 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d9200640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46ac06c4c0 0x7f46ac06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d1600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f46ac074780 0x7f46ac076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46ac0772b0 con 0x7f46ac074780 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.205+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46c408ea90 con 0x7f46d40acd70 2024-01-16T15:27:28.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.206+0000 7f46da600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f46ac074780 0x7f46ac076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.206+0000 7f46d9200640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46ac06c4c0 0x7f46ac06e980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f46bc036000 tx=0x7f46bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.206+0000 7f46da600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f46ac074780 0x7f46ac076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.207+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== osd.1 v2:172.21.15.43:6823/2025179551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f46ac0772b0 con 0x7f46ac074780 2024-01-16T15:27:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.222+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 --> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f46d40ae8d0 con 0x7f46ac074780 2024-01-16T15:27:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.222+0000 7f46d1600640 1 -- 172.21.15.43:0/650597615 <== osd.1 v2:172.21.15.43:6823/2025179551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f46cc006b50 con 0x7f46ac074780 2024-01-16T15:27:28.221 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:28.221 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:28.222 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f46ac074780 msgr2=0x7f46ac076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6823/2025179551,v1:172.21.15.43:6825/2025179551] conn(0x7f46ac074780 0x7f46ac076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46ac06c4c0 msgr2=0x7f46ac06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f46ac06c4c0 0x7f46ac06e980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f46bc036000 tx=0x7f46bc030000 comp rx=0 tx=0).stop 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 msgr2=0x7f46d40afcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 --2- 172.21.15.43:0/650597615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d40acd70 0x7f46d40afcd0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f46c400d5f0 tx=0x7f46c400dab0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46da600640 1 -- 172.21.15.43:0/650597615 reap_dead start 2024-01-16T15:27:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 shutdown_connections 2024-01-16T15:27:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 >> 172.21.15.43:0/650597615 conn(0x7f46d40b0ba0 msgr2=0x7f46d40b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 shutdown_connections 2024-01-16T15:27:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.223+0000 7f46dc600640 1 -- 172.21.15.43:0/650597615 wait complete. 2024-01-16T15:27:28.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-16T15:27:28.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.448+0000 7fe110a00640 1 Processor -- start 2024-01-16T15:27:28.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.448+0000 7fe110a00640 1 -- start start 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe110a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1080b39a0 0x7fe108071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe110a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 0x7fe108071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe110a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10813da30 con 0x7fe10815a100 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe110a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080ac840 con 0x7fe1080b39a0 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe110a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080ac560 con 0x7fe1080abff0 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe10d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1080b39a0 0x7fe108071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe10d600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe1080b39a0 0x7fe108071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53912/0 (socket says 172.21.15.43:53912) 2024-01-16T15:27:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.450+0000 7fe10d600640 1 -- 172.21.15.43:0/3212190257 learned_addr learned my addr 172.21.15.43:0/3212190257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10ea00640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 0x7fe108071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10d600640 1 -- 172.21.15.43:0/3212190257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 msgr2=0x7fe108071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10d600640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 0x7fe108071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10d600640 1 -- 172.21.15.43:0/3212190257 >> v1:172.21.15.69:6789/0 conn(0x7fe10815a100 legacy=0x7fe1080b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10d600640 1 -- 172.21.15.43:0/3212190257 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe108070650 con 0x7fe1080b39a0 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.451+0000 7fe10ea00640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 0x7fe108071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.452+0000 7fe10d600640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1080b39a0 0x7fe108071130 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009870 tx=0x7fe0f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccb3986f1327b0d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.452+0000 7fe10cc00640 1 -- 172.21.15.43:0/3212190257 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0030840 con 0x7fe1080b39a0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.452+0000 7fe10cc00640 1 -- 172.21.15.43:0/3212190257 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f00309a0 con 0x7fe1080b39a0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.452+0000 7fe10cc00640 1 -- 172.21.15.43:0/3212190257 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0030d30 con 0x7fe1080b39a0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 -- 172.21.15.43:0/3212190257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1080b39a0 msgr2=0x7fe108071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1080b39a0 0x7fe108071130 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009870 tx=0x7fe0f002b980 comp rx=0 tx=0).stop 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 -- 172.21.15.43:0/3212190257 shutdown_connections 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1080b39a0 0x7fe108071130 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 --2- 172.21.15.43:0/3212190257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080abff0 0x7fe108071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 -- 172.21.15.43:0/3212190257 >> 172.21.15.43:0/3212190257 conn(0x7fe1080b0d80 msgr2=0x7fe1080b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 -- 172.21.15.43:0/3212190257 shutdown_connections 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.453+0000 7fe110a00640 1 -- 172.21.15.43:0/3212190257 wait complete. 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.455+0000 7fe110a00640 1 Processor -- start 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.455+0000 7fe110a00640 1 -- start start 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1080abff0 0x7fe1081f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080b39a0 0x7fe1080afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10815b070 con 0x7fe10815a100 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1081f33d0 con 0x7fe1080abff0 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe110a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10815b4b0 con 0x7fe1080b39a0 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe10ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe10ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53924/0 (socket says 172.21.15.43:53924) 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe10ea00640 1 -- 172.21.15.43:0/1199583626 learned_addr learned my addr 172.21.15.43:0/1199583626 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.456+0000 7fe10e000640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1080abff0 0x7fe1081f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10d600640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080b39a0 0x7fe1080afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 -- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080b39a0 msgr2=0x7fe1080afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080b39a0 0x7fe1080afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 -- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1080abff0 msgr2=0x7fe1081f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1080abff0 0x7fe1081f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 -- 172.21.15.43:0/1199583626 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00094d0 con 0x7fe10815a100 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe10ea00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fe10000cd80 tx=0x7fe10000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000f680 con 0x7fe10815a100 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10000f800 con 0x7fe10815a100 2024-01-16T15:27:28.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000fbb0 con 0x7fe10815a100 2024-01-16T15:27:28.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.457+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1080ce530 con 0x7fe10815a100 2024-01-16T15:27:28.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.458+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1080ceaf0 con 0x7fe10815a100 2024-01-16T15:27:28.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.459+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108074770 con 0x7fe10815a100 2024-01-16T15:27:28.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.460+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fe10000b040 con 0x7fe10815a100 2024-01-16T15:27:28.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.461+0000 7fe106000640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 0x7fe0e006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.462+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe1000894e0 con 0x7fe10815a100 2024-01-16T15:27:28.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.462+0000 7fe10e000640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 0x7fe0e006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.462+0000 7fe10e000640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 0x7fe0e006e980 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000f60 tx=0x7fe0f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.467+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe100055a30 con 0x7fe10815a100 2024-01-16T15:27:28.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.663+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe10815a4e0 con 0x7fe0e006c4c0 2024-01-16T15:27:28.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.664+0000 7fe106000640 1 -- 172.21.15.43:0/1199583626 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe0f8001c70 con 0x7fe0e006c4c0 2024-01-16T15:27:28.663 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 1 to deep-scrub 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 msgr2=0x7fe0e006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 0x7fe0e006e980 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000f60 tx=0x7fe0f8009040 comp rx=0 tx=0).stop 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 msgr2=0x7fe1080b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fe10000cd80 tx=0x7fe10000a650 comp rx=0 tx=0).stop 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 shutdown_connections 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe0e006c4c0 0x7fe0e006e980 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe10815a100 0x7fe1080b03e0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1080b39a0 0x7fe1080afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.666+0000 7fe110a00640 1 --2- 172.21.15.43:0/1199583626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1080abff0 0x7fe1081f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.667+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 >> 172.21.15.43:0/1199583626 conn(0x7fe1080b0d80 msgr2=0x7fe1080b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.667+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 shutdown_connections 2024-01-16T15:27:28.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.667+0000 7fe110a00640 1 -- 172.21.15.43:0/1199583626 wait complete. 2024-01-16T15:27:28.685 INFO:tasks.ceph:Scrubbing osd.2 2024-01-16T15:27:28.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:28.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.890+0000 7f465b200640 1 Processor -- start 2024-01-16T15:27:28.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.890+0000 7f465b200640 1 -- start start 2024-01-16T15:27:28.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f465b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4654071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f465b200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4654070f60 con 0x7f46540ad450 2024-01-16T15:27:28.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f465b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46540ce650 con 0x7f4654153ea0 2024-01-16T15:27:28.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f465b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46540ad8e0 con 0x7f4654154280 2024-01-16T15:27:28.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f4659400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f46540ad450 0x7f4654070850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33212/0 (socket says 172.21.15.43:33212) 2024-01-16T15:27:28.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f4659400640 1 -- 172.21.15.43:0/4101630921 learned_addr learned my addr 172.21.15.43:0/4101630921 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:28.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.892+0000 7f465a800640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4654071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820633747 0 0) 0x7f4654070f60 con 0x7f46540ad450 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4634003520 con 0x7f46540ad450 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837140394 0 0) 0x7f4634003520 con 0x7f46540ad450 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 msgr2=0x7f4654071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4654071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 >> v1:172.21.15.69:6789/0 conn(0x7f4654153ea0 legacy=0x7f465415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465415e0a0 con 0x7f46540ad450 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f463c0019f0 con 0x7f46540ad450 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 >> v1:172.21.15.43:6789/0 conn(0x7f46540ad450 legacy=0x7f4654070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634003a10 0x7f4634003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463400ba50 con 0x7f46340047c0 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463400bbc0 con 0x7f4634003a10 2024-01-16T15:27:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463400bd80 con 0x7f4654154280 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.893+0000 7f4659400640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f465a800640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659e00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634003a10 0x7f4634003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659400640 1 -- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 msgr2=0x7f4634005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659400640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654154280 0x7f4634005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659400640 1 -- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634003a10 msgr2=0x7f4634003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659400640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634003a10 0x7f4634003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659400640 1 -- 172.21.15.43:0/4101630921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634003520 con 0x7f46340047c0 2024-01-16T15:27:28.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.894+0000 7f4659e00640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4634003a10 0x7f4634003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:28.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.895+0000 7f4659400640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f4634004c40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f463c00a830 tx=0x7f46340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d27cca18b3ca7a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.895+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0072d0 con 0x7f46340047c0 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.895+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c02f070 con 0x7f46340047c0 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.895+0000 7f4658a00640 1 -- 172.21.15.43:0/4101630921 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0076b0 con 0x7f46340047c0 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 -- 172.21.15.43:0/4101630921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 msgr2=0x7f4634004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 --2- 172.21.15.43:0/4101630921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f4634004c40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f463c00a830 tx=0x7f46340057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465a800640 1 -- 172.21.15.43:0/4101630921 reap_dead start 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 -- 172.21.15.43:0/4101630921 shutdown_connections 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 -- 172.21.15.43:0/4101630921 >> 172.21.15.43:0/4101630921 conn(0x7f46540cec10 msgr2=0x7f46540cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 -- 172.21.15.43:0/4101630921 shutdown_connections 2024-01-16T15:27:28.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.896+0000 7f465b200640 1 -- 172.21.15.43:0/4101630921 wait complete. 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.898+0000 7f465b200640 1 Processor -- start 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.898+0000 7f465b200640 1 -- start start 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f46541444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654150800 0x7f4654150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46540b0840 0x7f46540b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46540b1200 con 0x7f46340047c0 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4654151370 con 0x7f46540b0840 2024-01-16T15:27:28.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f465b200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465414d7c0 con 0x7f4654150800 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f4659e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f46541444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f4659e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f46541444d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53944/0 (socket says 172.21.15.43:53944) 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.899+0000 7f4659e00640 1 -- 172.21.15.43:0/2787602621 learned_addr learned my addr 172.21.15.43:0/2787602621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f465a800640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46540b0840 0x7f46540b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 -- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654150800 msgr2=0x7f4654150c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4654150800 0x7f4654150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 -- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46540b0840 msgr2=0x7f46540b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46540b0840 0x7f46540b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4654154810 con 0x7f46340047c0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f465a800640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46540b0840 0x7f46540b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4659e00640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f46541444d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f464400d7a0 tx=0x7f464400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644017070 con 0x7f46340047c0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4644005070 con 0x7f46340047c0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644005420 con 0x7f46340047c0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.900+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4654154af0 con 0x7f46340047c0 2024-01-16T15:27:28.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.901+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46541550b0 con 0x7f46340047c0 2024-01-16T15:27:28.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.902+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4654070050 con 0x7f46340047c0 2024-01-16T15:27:28.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.903+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f4644007030 con 0x7f46340047c0 2024-01-16T15:27:28.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4652000640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f462c06c4f0 0x7f462c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4644088440 con 0x7f46340047c0 2024-01-16T15:27:28.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4659400640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f462c06c4f0 0x7f462c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4652000640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f462c0747b0 0x7f462c076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f462c0772e0 con 0x7f462c0747b0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4644089110 con 0x7f46340047c0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.904+0000 7f465a800640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f462c0747b0 0x7f462c076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.905+0000 7f4659400640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f462c06c4f0 0x7f462c06e9b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f463c034000 tx=0x7f463c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.905+0000 7f465a800640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f462c0747b0 0x7f462c076c10 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.906+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== osd.2 v2:172.21.15.43:6808/2671083580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f462c0772e0 con 0x7f462c0747b0 2024-01-16T15:27:28.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.920+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 --> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f465415e660 con 0x7f462c0747b0 2024-01-16T15:27:28.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f4652000640 1 -- 172.21.15.43:0/2787602621 <== osd.2 v2:172.21.15.43:6808/2671083580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f464c005990 con 0x7f462c0747b0 2024-01-16T15:27:28.920 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:28.920 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:28.920 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:28.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f462c0747b0 msgr2=0x7f462c076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6808/2671083580,v1:172.21.15.43:6809/2671083580] conn(0x7f462c0747b0 0x7f462c076c10 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f462c06c4f0 msgr2=0x7f462c06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f462c06c4f0 0x7f462c06e9b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f463c034000 tx=0x7f463c02f000 comp rx=0 tx=0).stop 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 msgr2=0x7f46541444d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 --2- 172.21.15.43:0/2787602621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46340047c0 0x7f46541444d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f464400d7a0 tx=0x7f464400dc60 comp rx=0 tx=0).stop 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465a800640 1 -- 172.21.15.43:0/2787602621 reap_dead start 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 shutdown_connections 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 >> 172.21.15.43:0/2787602621 conn(0x7f46540cec10 msgr2=0x7f4654070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 shutdown_connections 2024-01-16T15:27:28.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:28.921+0000 7f465b200640 1 -- 172.21.15.43:0/2787602621 wait complete. 2024-01-16T15:27:28.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-16T15:27:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.158+0000 7fc836600640 1 Processor -- start 2024-01-16T15:27:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.158+0000 7fc836600640 1 -- start start 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.159+0000 7fc836600640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc8300ad720 0x7fc8300b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.159+0000 7fc836600640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83015c360 con 0x7fc8300abff0 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.159+0000 7fc836600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83015ba30 con 0x7fc83006d520 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.159+0000 7fc836600640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8300b41a0 con 0x7fc8300ad720 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82fe00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc8300abff0 0x7fc8300b25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33226/0 (socket says 172.21.15.43:33226) 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82fe00640 1 -- 172.21.15.43:0/186171134 learned_addr learned my addr 172.21.15.43:0/186171134 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82ea00640 1 --2- 172.21.15.43:0/186171134 >> v2:172.21.15.43:3300/0 conn(0x7fc8300ad720 0x7fc8300b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82ea00640 1 -- 172.21.15.43:0/186171134 >> v1:172.21.15.69:6789/0 conn(0x7fc83006d520 legacy=0x7fc83006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82ea00640 1 -- 172.21.15.43:0/186171134 >> v1:172.21.15.43:6789/0 conn(0x7fc8300abff0 legacy=0x7fc8300b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.160+0000 7fc82ea00640 1 -- 172.21.15.43:0/186171134 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83015bd10 con 0x7fc8300ad720 2024-01-16T15:27:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.162+0000 7fc82ea00640 1 --2- 172.21.15.43:0/186171134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8300b2080 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097a0 tx=0x7fc81802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520794f4176b4545 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.162+0000 7fc82e000640 1 -- 172.21.15.43:0/186171134 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818031830 con 0x7fc8300ad720 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.162+0000 7fc82e000640 1 -- 172.21.15.43:0/186171134 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc818031990 con 0x7fc8300ad720 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.162+0000 7fc82e000640 1 -- 172.21.15.43:0/186171134 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818031d20 con 0x7fc8300ad720 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 -- 172.21.15.43:0/186171134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 msgr2=0x7fc8300b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 --2- 172.21.15.43:0/186171134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8300b2080 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fc8180097a0 tx=0x7fc81802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 -- 172.21.15.43:0/186171134 shutdown_connections 2024-01-16T15:27:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 --2- 172.21.15.43:0/186171134 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8300b2080 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 -- 172.21.15.43:0/186171134 >> 172.21.15.43:0/186171134 conn(0x7fc8300b0d80 msgr2=0x7fc8300b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 -- 172.21.15.43:0/186171134 shutdown_connections 2024-01-16T15:27:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.163+0000 7fc836600640 1 -- 172.21.15.43:0/186171134 wait complete. 2024-01-16T15:27:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.165+0000 7fc836600640 1 Processor -- start 2024-01-16T15:27:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.165+0000 7fc836600640 1 -- start start 2024-01-16T15:27:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 0x7fc8301f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8300abff0 0x7fc830147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8301f4020 con 0x7fc8300ad720 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8301f3a60 con 0x7fc83006d520 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc836600640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8301f3d40 con 0x7fc8300abff0 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc82fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc82fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53970/0 (socket says 172.21.15.43:53970) 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.166+0000 7fc82fe00640 1 -- 172.21.15.43:0/2005490705 learned_addr learned my addr 172.21.15.43:0/2005490705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82ea00640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8300abff0 0x7fc830147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82f400640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 0x7fc8301f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 -- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8300abff0 msgr2=0x7fc830147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8300abff0 0x7fc830147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 -- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 msgr2=0x7fc8301f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 0x7fc8301f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 -- 172.21.15.43:0/2005490705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc818009400 con 0x7fc8300ad720 2024-01-16T15:27:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82f400640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 0x7fc8301f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc82fe00640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc82800cbc0 tx=0x7fc82800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82800ad40 con 0x7fc8300ad720 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc828012400 con 0x7fc8300ad720 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82800aea0 con 0x7fc8300ad720 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.167+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8301f6810 con 0x7fc8300ad720 2024-01-16T15:27:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.168+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8301f7550 con 0x7fc8300ad720 2024-01-16T15:27:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.169+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8301f6d30 con 0x7fc8300ad720 2024-01-16T15:27:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.170+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fc828007070 con 0x7fc8300ad720 2024-01-16T15:27:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.171+0000 7fc827400640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 0x7fc80806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.171+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc828088a30 con 0x7fc8300ad720 2024-01-16T15:27:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.171+0000 7fc82f400640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 0x7fc80806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.172+0000 7fc82f400640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 0x7fc80806e980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc82000a830 tx=0x7fc820008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.175+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8301f6d30 con 0x7fc8300ad720 2024-01-16T15:27:29.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.371+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc83015b6e0 con 0x7fc80806c4c0 2024-01-16T15:27:29.370 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 2 to deep-scrub 2024-01-16T15:27:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.372+0000 7fc827400640 1 -- 172.21.15.43:0/2005490705 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc82000adb0 con 0x7fc80806c4c0 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 msgr2=0x7fc80806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 0x7fc80806e980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc82000a830 tx=0x7fc820008040 comp rx=0 tx=0).stop 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 msgr2=0x7fc8301f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc82800cbc0 tx=0x7fc82800a430 comp rx=0 tx=0).stop 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 shutdown_connections 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.374+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc80806c4c0 0x7fc80806e980 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8300ad720 0x7fc8301f30f0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8300abff0 0x7fc830147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 --2- 172.21.15.43:0/2005490705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc83006d520 0x7fc8301f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 >> 172.21.15.43:0/2005490705 conn(0x7fc8300b0d80 msgr2=0x7fc8300acca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:29.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 shutdown_connections 2024-01-16T15:27:29.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.375+0000 7fc836600640 1 -- 172.21.15.43:0/2005490705 wait complete. 2024-01-16T15:27:29.393 INFO:tasks.ceph:Scrubbing osd.3 2024-01-16T15:27:29.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:29.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.616+0000 7f60a6a00640 1 Processor -- start 2024-01-16T15:27:29.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.617+0000 7f60a6a00640 1 -- start start 2024-01-16T15:27:29.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.618+0000 7f60a6a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f60a00acd70 0x7f60a006dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.618+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a0070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.618+0000 7f60a6a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a006d7a0 con 0x7f60a00ae490 2024-01-16T15:27:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.618+0000 7f60a6a00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a015a5b0 con 0x7f60a00acd70 2024-01-16T15:27:29.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.618+0000 7f60a6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a006ce70 con 0x7f60a0070490 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f60a00acd70 0x7f60a006dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f60a00acd70 0x7f60a006dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53984/0 (socket says 172.21.15.43:53984) 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 -- 172.21.15.43:0/805715681 learned_addr learned my addr 172.21.15.43:0/805715681 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f60a4a00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a0070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 -- 172.21.15.43:0/805715681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 msgr2=0x7f60a0070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a0070920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.619+0000 7f609fe00640 1 -- 172.21.15.43:0/805715681 >> v1:172.21.15.69:6789/0 conn(0x7f60a00ae490 legacy=0x7f60a00ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.620+0000 7f609fe00640 1 -- 172.21.15.43:0/805715681 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a006d150 con 0x7f60a00acd70 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.621+0000 7f609fe00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a006dc30 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fb1bbac1c773d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.621+0000 7f609f400640 1 -- 172.21.15.43:0/805715681 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609002cb00 con 0x7f60a00acd70 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.621+0000 7f609f400640 1 -- 172.21.15.43:0/805715681 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609002cc60 con 0x7f60a00acd70 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.621+0000 7f609f400640 1 -- 172.21.15.43:0/805715681 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60900343f0 con 0x7f60a00acd70 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 -- 172.21.15.43:0/805715681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 msgr2=0x7f60a006dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a006dc30 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609002bf60 comp rx=0 tx=0).stop 2024-01-16T15:27:29.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 -- 172.21.15.43:0/805715681 shutdown_connections 2024-01-16T15:27:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a0070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 --2- 172.21.15.43:0/805715681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a006dc30 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 -- 172.21.15.43:0/805715681 >> 172.21.15.43:0/805715681 conn(0x7f60a00b0ba0 msgr2=0x7f60a00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 -- 172.21.15.43:0/805715681 shutdown_connections 2024-01-16T15:27:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.622+0000 7f60a6a00640 1 -- 172.21.15.43:0/805715681 wait complete. 2024-01-16T15:27:29.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.624+0000 7f60a6a00640 1 Processor -- start 2024-01-16T15:27:29.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.624+0000 7f60a6a00640 1 -- start start 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a01514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60a00ae490 0x7f60a00afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a015b1a0 con 0x7f60a00acd70 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0155b20 con 0x7f60a00ae490 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f60a6a00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a015b440 con 0x7f60a0070490 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f6097200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a01514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f6097200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a01514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54000/0 (socket says 172.21.15.43:54000) 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.625+0000 7f6097200640 1 -- 172.21.15.43:0/346004939 learned_addr learned my addr 172.21.15.43:0/346004939 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f60a4a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60a00ae490 0x7f60a00afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f609fe00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 -- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 msgr2=0x7f60a01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60a0070490 0x7f60a01555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 -- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60a00ae490 msgr2=0x7f60a00afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60a00ae490 0x7f60a00afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6090009b30 con 0x7f60a00acd70 2024-01-16T15:27:29.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f6097200640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a01514f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f608800c450 tx=0x7f608800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.626+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880052d0 con 0x7f60a00acd70 2024-01-16T15:27:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.627+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088005430 con 0x7f60a00acd70 2024-01-16T15:27:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.627+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088012870 con 0x7f60a00acd70 2024-01-16T15:27:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.627+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a015b720 con 0x7f60a00acd70 2024-01-16T15:27:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.627+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a015bce0 con 0x7f60a00acd70 2024-01-16T15:27:29.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.628+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60a00b3d90 con 0x7f60a00acd70 2024-01-16T15:27:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.629+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f60880129d0 con 0x7f60a00acd70 2024-01-16T15:27:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609cc00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f607806c4c0 0x7f607806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6088088690 con 0x7f60a00acd70 2024-01-16T15:27:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609cc00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f6078074780 0x7f6078076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60780772b0 con 0x7f6078074780 2024-01-16T15:27:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60880892e0 con 0x7f60a00acd70 2024-01-16T15:27:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.630+0000 7f609fe00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f607806c4c0 0x7f607806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.631+0000 7f60a4a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f6078074780 0x7f6078076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.631+0000 7f60a4a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f6078074780 0x7f6078076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.631+0000 7f609fe00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f607806c4c0 0x7f607806e980 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6090035000 tx=0x7f6090030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.632+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== osd.3 v2:172.21.15.43:6813/889849964 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f60780772b0 con 0x7f6078074780 2024-01-16T15:27:29.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.647+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 --> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f60a015b5b0 con 0x7f6078074780 2024-01-16T15:27:29.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.647+0000 7f609cc00640 1 -- 172.21.15.43:0/346004939 <== osd.3 v2:172.21.15.43:6813/889849964 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f60a015b5b0 con 0x7f6078074780 2024-01-16T15:27:29.646 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:29.646 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:29.647 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:29.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.647+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f6078074780 msgr2=0x7f6078076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.647+0000 7f60a6a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6813/889849964,v1:172.21.15.43:6815/889849964] conn(0x7f6078074780 0x7f6078076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f607806c4c0 msgr2=0x7f607806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f607806c4c0 0x7f607806e980 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6090035000 tx=0x7f6090030000 comp rx=0 tx=0).stop 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 msgr2=0x7f60a01514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 --2- 172.21.15.43:0/346004939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60a00acd70 0x7f60a01514f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f608800c450 tx=0x7f608800c910 comp rx=0 tx=0).stop 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a4a00640 1 -- 172.21.15.43:0/346004939 reap_dead start 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 shutdown_connections 2024-01-16T15:27:29.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 >> 172.21.15.43:0/346004939 conn(0x7f60a00b0ba0 msgr2=0x7f60a00af480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:29.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 shutdown_connections 2024-01-16T15:27:29.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.648+0000 7f60a6a00640 1 -- 172.21.15.43:0/346004939 wait complete. 2024-01-16T15:27:29.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-16T15:27:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.921+0000 7fa5b3200640 1 Processor -- start 2024-01-16T15:27:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.921+0000 7fa5b3200640 1 -- start start 2024-01-16T15:27:29.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.922+0000 7fa5b3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5b4141e80 0x7fa5b4071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.922+0000 7fa5b3200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b406cae0 con 0x7fa5b415ed20 2024-01-16T15:27:29.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.922+0000 7fa5b3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b4071e40 con 0x7fa5b4071650 2024-01-16T15:27:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.922+0000 7fa5b3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b41422e0 con 0x7fa5b4141e80 2024-01-16T15:27:29.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.923+0000 7fa5b1e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa5b415ed20 0x7fa5b415f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33230/0 (socket says 172.21.15.43:33230) 2024-01-16T15:27:29.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.923+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2586367048 learned_addr learned my addr 172.21.15.43:0/2586367048 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:29.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.923+0000 7fa5abe00640 1 --2- 172.21.15.43:0/2586367048 >> v2:172.21.15.69:3300/0 conn(0x7fa5b4141e80 0x7fa5b4071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.923+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167128506 0 0) 0x7fa5b406cae0 con 0x7fa5b415ed20 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.923+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003520 con 0x7fa5b415ed20 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170349990 0 0) 0x7fa594003520 con 0x7fa5b415ed20 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 >> v2:172.21.15.69:3300/0 conn(0x7fa5b4141e80 msgr2=0x7fa5b4071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 --2- 172.21.15.43:0/2586367048 >> v2:172.21.15.69:3300/0 conn(0x7fa5b4141e80 0x7fa5b4071110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 >> v1:172.21.15.69:6789/0 conn(0x7fa5b4071650 legacy=0x7fa5b406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b406c1b0 con 0x7fa5b415ed20 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fa5a4001670 con 0x7fa5b415ed20 2024-01-16T15:27:29.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 >> v1:172.21.15.43:6789/0 conn(0x7fa5b415ed20 legacy=0x7fa5b415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa594003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa594005180 0x7fa59400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400c050 con 0x7fa5940039e0 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400c190 con 0x7fa5940047c0 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400c350 con 0x7fa594005180 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa594003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.924+0000 7fa5abe00640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b2800640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa594005180 0x7fa59400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa594005180 msgr2=0x7fa59400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa594005180 0x7fa59400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5940047c0 msgr2=0x7fa594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2586367048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa594003520 con 0x7fa5940039e0 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b2800640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa594005180 0x7fa59400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:29.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.925+0000 7fa5b2800640 1 -- 172.21.15.43:0/2586367048 reap_dead start 2024-01-16T15:27:29.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.926+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa594003de0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c190 tx=0x7fa5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b35ee4e168a82c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.926+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4005ad0 con 0x7fa5940039e0 2024-01-16T15:27:29.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.926+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4005c30 con 0x7fa5940039e0 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.926+0000 7fa5b1400640 1 -- 172.21.15.43:0/2586367048 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a402c8a0 con 0x7fa5940039e0 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 -- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 msgr2=0x7fa594003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa594003de0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c190 tx=0x7fa5940057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 -- 172.21.15.43:0/2586367048 shutdown_connections 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2586367048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa594003de0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 -- 172.21.15.43:0/2586367048 >> 172.21.15.43:0/2586367048 conn(0x7fa5b40b0840 msgr2=0x7fa5b40b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 -- 172.21.15.43:0/2586367048 shutdown_connections 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.927+0000 7fa5b3200640 1 -- 172.21.15.43:0/2586367048 wait complete. 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.929+0000 7fa5b3200640 1 Processor -- start 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.929+0000 7fa5b3200640 1 -- start start 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5b4133f60 0x7fa5b4132420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5b4132960 0x7fa5b4134460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4159420 con 0x7fa5940039e0 2024-01-16T15:27:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4133ad0 con 0x7fa5b4133f60 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b3200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4132ec0 con 0x7fa5b4132960 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54028/0 (socket says 172.21.15.43:54028) 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2281924174 learned_addr learned my addr 172.21.15.43:0/2281924174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5b4132960 msgr2=0x7fa5b4134460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5b4132960 0x7fa5b4134460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5b4133f60 msgr2=0x7fa5b4132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5b4133f60 0x7fa5b4132420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:29.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.930+0000 7fa5b1e00640 1 -- 172.21.15.43:0/2281924174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b41331a0 con 0x7fa5940039e0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5b1e00640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c190 tx=0x7fa5a402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a403c040 con 0x7fa5940039e0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a402ccd0 con 0x7fa5940039e0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40072b0 con 0x7fa5940039e0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b4133480 con 0x7fa5940039e0 2024-01-16T15:27:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.931+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b4135210 con 0x7fa5940039e0 2024-01-16T15:27:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.932+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4067d10 con 0x7fa5940039e0 2024-01-16T15:27:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.933+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa5a4032070 con 0x7fa5940039e0 2024-01-16T15:27:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.934+0000 7fa5aa000640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 0x7fa58c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.934+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa5a40a9d80 con 0x7fa5940039e0 2024-01-16T15:27:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.934+0000 7fa5abe00640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 0x7fa58c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:29.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.935+0000 7fa5abe00640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 0x7fa58c06e980 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40b04f0 tx=0x7fa59c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:29.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:29.939+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5a4076380 con 0x7fa5940039e0 2024-01-16T15:27:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.136+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5b40ae4e0 con 0x7fa58c06c4c0 2024-01-16T15:27:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.136+0000 7fa5aa000640 1 -- 172.21.15.43:0/2281924174 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa5b40ae4e0 con 0x7fa58c06c4c0 2024-01-16T15:27:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 3 to deep-scrub 2024-01-16T15:27:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 msgr2=0x7fa58c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 0x7fa58c06e980 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40b04f0 tx=0x7fa59c008040 comp rx=0 tx=0).stop 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 msgr2=0x7fa5b4158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c190 tx=0x7fa5a402e000 comp rx=0 tx=0).stop 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 shutdown_connections 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa58c06c4c0 0x7fa58c06e980 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5b4133f60 0x7fa5b4132420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5b4132960 0x7fa5b4134460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 --2- 172.21.15.43:0/2281924174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5940039e0 0x7fa5b4158ee0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.139+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 >> 172.21.15.43:0/2281924174 conn(0x7fa5b40b0840 msgr2=0x7fa5b40addd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.140+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 shutdown_connections 2024-01-16T15:27:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.140+0000 7fa5b3200640 1 -- 172.21.15.43:0/2281924174 wait complete. 2024-01-16T15:27:30.158 INFO:tasks.ceph:Scrubbing osd.4 2024-01-16T15:27:30.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.388+0000 7f40fb200640 1 Processor -- start 2024-01-16T15:27:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.388+0000 7f40fb200640 1 -- start start 2024-01-16T15:27:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40fb200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40fc0d1450 0x7f40fc0b0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40fb200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40fc06b880 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40fb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40fc0cef80 con 0x7f40fc15d290 2024-01-16T15:27:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40fb200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40fc0ce7d0 con 0x7f40fc0d1450 2024-01-16T15:27:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40f1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40fc0d1450 0x7f40fc0b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40f1200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f40fc0d1450 0x7f40fc0b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54032/0 (socket says 172.21.15.43:54032) 2024-01-16T15:27:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.390+0000 7f40f1200640 1 -- 172.21.15.43:0/2179602545 learned_addr learned my addr 172.21.15.43:0/2179602545 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.391+0000 7f40f1200640 1 -- 172.21.15.43:0/2179602545 >> v1:172.21.15.69:6789/0 conn(0x7f40fc15d290 legacy=0x7f40fc0d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.391+0000 7f40f1200640 1 -- 172.21.15.43:0/2179602545 >> v1:172.21.15.43:6789/0 conn(0x7f40fc0cd6e0 legacy=0x7f40fc0cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.391+0000 7f40f1200640 1 -- 172.21.15.43:0/2179602545 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc0cdc50 con 0x7f40fc0d1450 2024-01-16T15:27:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.392+0000 7f40f1200640 1 --2- 172.21.15.43:0/2179602545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0d1450 0x7f40fc0b0290 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a032f7d265607c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.392+0000 7f40f9400640 1 -- 172.21.15.43:0/2179602545 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e402f850 con 0x7f40fc0d1450 2024-01-16T15:27:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40f9400640 1 -- 172.21.15.43:0/2179602545 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e402f9b0 con 0x7f40fc0d1450 2024-01-16T15:27:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40f9400640 1 -- 172.21.15.43:0/2179602545 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e402fd40 con 0x7f40fc0d1450 2024-01-16T15:27:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 -- 172.21.15.43:0/2179602545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0d1450 msgr2=0x7f40fc0b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 --2- 172.21.15.43:0/2179602545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0d1450 0x7f40fc0b0290 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402ba30 comp rx=0 tx=0).stop 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 -- 172.21.15.43:0/2179602545 shutdown_connections 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 --2- 172.21.15.43:0/2179602545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0d1450 0x7f40fc0b0290 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 -- 172.21.15.43:0/2179602545 >> 172.21.15.43:0/2179602545 conn(0x7f40fc099a00 msgr2=0x7f40fc099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 -- 172.21.15.43:0/2179602545 shutdown_connections 2024-01-16T15:27:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.393+0000 7f40fb200640 1 -- 172.21.15.43:0/2179602545 wait complete. 2024-01-16T15:27:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.395+0000 7f40fb200640 1 Processor -- start 2024-01-16T15:27:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.395+0000 7f40fb200640 1 -- start start 2024-01-16T15:27:30.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 0x7f40fc09e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40fc0d1450 0x7f40fc157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40fc15d290 0x7f40fc157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc0b39f0 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc0c6600 con 0x7f40fc15d290 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40fb200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc0c68e0 con 0x7f40fc0d1450 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 0x7f40fc09e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 0x7f40fc09e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54042/0 (socket says 172.21.15.43:54042) 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.396+0000 7f40f9e00640 1 -- 172.21.15.43:0/3732772143 learned_addr learned my addr 172.21.15.43:0/3732772143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f1200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40fc0d1450 0x7f40fc157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 -- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40fc0d1450 msgr2=0x7f40fc157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40fc0d1450 0x7f40fc157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40fa800640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40fc15d290 0x7f40fc157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 -- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40fc15d290 msgr2=0x7f40fc157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40fc15d290 0x7f40fc157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40094d0 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40fa800640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40fc15d290 0x7f40fc157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.397+0000 7f40f9e00640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 0x7f40fc09e2d0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00c400 tx=0x7f40ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.398+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec0052d0 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.398+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec005430 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.398+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec003830 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.398+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc0c6bc0 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.398+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc0c6290 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.399+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40fc0ce210 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.400+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f40ec003990 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.401+0000 7f40f2a00640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40d406c4c0 0x7f40d406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f40ec088070 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40f2a00640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f40d4074780 0x7f40d4076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40d40772b0 con 0x7f40d4074780 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40ec088ca0 con 0x7f40fc0cd6e0 2024-01-16T15:27:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40f1200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40d406c4c0 0x7f40d406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.402+0000 7f40fa800640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f40d4074780 0x7f40d4076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.403+0000 7f40f1200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40d406c4c0 0x7f40d406e980 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f40e4036000 tx=0x7f40e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.403+0000 7f40fa800640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f40d4074780 0x7f40d4076be0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.406+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== osd.4 v2:172.21.15.69:6816/2273091943 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f40d40772b0 con 0x7f40d4074780 2024-01-16T15:27:30.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.421+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 --> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f40fc0b07d0 con 0x7f40d4074780 2024-01-16T15:27:30.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40f2a00640 1 -- 172.21.15.43:0/3732772143 <== osd.4 v2:172.21.15.69:6816/2273091943 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f40f4001ea0 con 0x7f40d4074780 2024-01-16T15:27:30.421 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:30.421 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:30.421 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:30.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f40d4074780 msgr2=0x7f40d4076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.69:6816/2273091943,v1:172.21.15.69:6817/2273091943] conn(0x7f40d4074780 0x7f40d4076be0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40d406c4c0 msgr2=0x7f40d406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f40d406c4c0 0x7f40d406e980 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f40e4036000 tx=0x7f40e4008030 comp rx=0 tx=0).stop 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 msgr2=0x7f40fc09e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fb200640 1 --2- 172.21.15.43:0/3732772143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40fc0cd6e0 0x7f40fc09e2d0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00c400 tx=0x7f40ec00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.422+0000 7f40fa800640 1 -- 172.21.15.43:0/3732772143 reap_dead start 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.423+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 shutdown_connections 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.423+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 >> 172.21.15.43:0/3732772143 conn(0x7f40fc099a00 msgr2=0x7f40fc0cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.423+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 shutdown_connections 2024-01-16T15:27:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.423+0000 7f40fb200640 1 -- 172.21.15.43:0/3732772143 wait complete. 2024-01-16T15:27:30.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-16T15:27:30.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.665+0000 7f6700e00640 1 Processor -- start 2024-01-16T15:27:30.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.665+0000 7f6700e00640 1 -- start start 2024-01-16T15:27:30.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.667+0000 7f6700e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66fc06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.667+0000 7f6700e00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc06cae0 con 0x7f66fc141e80 2024-01-16T15:27:30.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.667+0000 7f6700e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc071e40 con 0x7f66fc15ed20 2024-01-16T15:27:30.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.667+0000 7f6700e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1422e0 con 0x7f66fc071820 2024-01-16T15:27:30.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.668+0000 7f66f3e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f66fc141e80 0x7f66fc071110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33248/0 (socket says 172.21.15.43:33248) 2024-01-16T15:27:30.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.668+0000 7f66f3e00640 1 -- 172.21.15.43:0/2195665155 learned_addr learned my addr 172.21.15.43:0/2195665155 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.668+0000 7f66fbe00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66fc06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.668+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203331431 0 0) 0x7f66fc06cae0 con 0x7f66fc141e80 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.668+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc003520 con 0x7f66fc141e80 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489219326 0 0) 0x7f66dc003520 con 0x7f66fc141e80 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 msgr2=0x7f66fc06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66fc06cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 >> v1:172.21.15.69:6789/0 conn(0x7f66fc15ed20 legacy=0x7f66fc15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc06c1b0 con 0x7f66fc141e80 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7f66e4001670 con 0x7f66fc141e80 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 >> v1:172.21.15.43:6789/0 conn(0x7f66fc141e80 legacy=0x7f66fc071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc003a10 0x7f66dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66dc004d00 0x7f66dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00fea0 con 0x7f66dc004d00 2024-01-16T15:27:30.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc010010 con 0x7f66dc003a10 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0101d0 con 0x7f66fc071820 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fbe00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66dc004d00 0x7f66dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fb400640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc003a10 0x7f66dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66f3e00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fbe00640 1 -- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 msgr2=0x7f66dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fbe00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc071820 0x7f66dc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fbe00640 1 -- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc003a10 msgr2=0x7f66dc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:30.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.669+0000 7f66fbe00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc003a10 0x7f66dc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.670+0000 7f66fbe00640 1 -- 172.21.15.43:0/2195665155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc003520 con 0x7f66dc004d00 2024-01-16T15:27:30.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.670+0000 7f66fb400640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc003a10 0x7f66dc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f66fbe00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66dc004d00 0x7f66dc005180 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009f90 tx=0x7f66dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368411d4d74de1ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f402cbd0 con 0x7f66dc004d00 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f400f040 con 0x7f66dc004d00 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f66faa00640 1 -- 172.21.15.43:0/2195665155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4034440 con 0x7f66dc004d00 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f6700e00640 1 -- 172.21.15.43:0/2195665155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66dc004d00 msgr2=0x7f66dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f6700e00640 1 --2- 172.21.15.43:0/2195665155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66dc004d00 0x7f66dc005180 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009f90 tx=0x7f66dc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.671+0000 7f66fbe00640 1 -- 172.21.15.43:0/2195665155 reap_dead start 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.672+0000 7f6700e00640 1 -- 172.21.15.43:0/2195665155 shutdown_connections 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.672+0000 7f6700e00640 1 -- 172.21.15.43:0/2195665155 >> 172.21.15.43:0/2195665155 conn(0x7f66fc0b0840 msgr2=0x7f66fc0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.672+0000 7f6700e00640 1 -- 172.21.15.43:0/2195665155 shutdown_connections 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.672+0000 7f6700e00640 1 -- 172.21.15.43:0/2195665155 wait complete. 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.673+0000 7f6700e00640 1 Processor -- start 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.674+0000 7f6700e00640 1 -- start start 2024-01-16T15:27:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 0x7f66fc13d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc15c400 0x7f66fc1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc13eb70 con 0x7f66fc13fb80 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc13f6f0 con 0x7f66dc004d00 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f6700e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc13ded0 con 0x7f66fc15c400 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54062/0 (socket says 172.21.15.43:54062) 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 -- 172.21.15.43:0/783771249 learned_addr learned my addr 172.21.15.43:0/783771249 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fb400640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 0x7f66fc13d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66f3e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc15c400 0x7f66fc1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 -- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc15c400 msgr2=0x7f66fc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc15c400 0x7f66fc1408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 -- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 msgr2=0x7f66fc13d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 0x7f66fc13d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fbe00640 1 -- 172.21.15.43:0/783771249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc13edc0 con 0x7f66fc13fb80 2024-01-16T15:27:30.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.675+0000 7f66fb400640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 0x7f66fc13d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:30.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.676+0000 7f66fbe00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f66f400e230 tx=0x7f66f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.676+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4043070 con 0x7f66fc13fb80 2024-01-16T15:27:30.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.676+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f4034ad0 con 0x7f66fc13fb80 2024-01-16T15:27:30.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.676+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc13f0a0 con 0x7f66fc13fb80 2024-01-16T15:27:30.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.676+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4034e80 con 0x7f66fc13fb80 2024-01-16T15:27:30.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.677+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc140e10 con 0x7f66fc13fb80 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.678+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc04e840 con 0x7f66fc13fb80 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.679+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f66f403e440 con 0x7f66fc13fb80 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.680+0000 7f66f3400640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 0x7f66d406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.680+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66f40aa300 con 0x7f66fc13fb80 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.680+0000 7f66fb400640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 0x7f66d406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.681+0000 7f66fb400640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 0x7f66d406e980 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001690 tx=0x7f66ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.686+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66f4076c00 con 0x7f66fc13fb80 2024-01-16T15:27:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.878+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f66fc06c770 con 0x7f66d406c4c0 2024-01-16T15:27:30.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.879+0000 7f66f3400640 1 -- 172.21.15.43:0/783771249 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f66ec00d040 con 0x7f66d406c4c0 2024-01-16T15:27:30.878 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 4 to deep-scrub 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 msgr2=0x7f66d406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 0x7f66d406e980 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001690 tx=0x7f66ec009040 comp rx=0 tx=0).stop 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 msgr2=0x7f66fc159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f66f400e230 tx=0x7f66f4009750 comp rx=0 tx=0).stop 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 shutdown_connections 2024-01-16T15:27:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f66d406c4c0 0x7f66d406e980 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66fc13fb80 0x7f66fc159940 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f66fc15c400 0x7f66fc1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 --2- 172.21.15.43:0/783771249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66dc004d00 0x7f66fc13d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 >> 172.21.15.43:0/783771249 conn(0x7f66fc0b0840 msgr2=0x7f66fc142340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 shutdown_connections 2024-01-16T15:27:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:30.882+0000 7f6700e00640 1 -- 172.21.15.43:0/783771249 wait complete. 2024-01-16T15:27:30.900 INFO:tasks.ceph:Scrubbing osd.5 2024-01-16T15:27:30.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.128+0000 7efc0d200640 1 Processor -- start 2024-01-16T15:27:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.128+0000 7efc0d200640 1 -- start start 2024-01-16T15:27:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc0d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efc080b4140 0x7efc08071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc0d200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08070d40 con 0x7efc081439b0 2024-01-16T15:27:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc0d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08071020 con 0x7efc081435d0 2024-01-16T15:27:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc0d200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc08071300 con 0x7efc080b4140 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc07e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efc081439b0 0x7efc08070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33264/0 (socket says 172.21.15.43:33264) 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.130+0000 7efc07e00640 1 -- 172.21.15.43:0/43882816 learned_addr learned my addr 172.21.15.43:0/43882816 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efbfe800640 1 --2- 172.21.15.43:0/43882816 >> v2:172.21.15.69:3300/0 conn(0x7efc080b4140 0x7efc08071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519583766 0 0) 0x7efc08070d40 con 0x7efc081439b0 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe8003520 con 0x7efc081439b0 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3320162326 0 0) 0x7efbe8003520 con 0x7efc081439b0 2024-01-16T15:27:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 >> v2:172.21.15.69:3300/0 conn(0x7efc080b4140 msgr2=0x7efc08071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 --2- 172.21.15.43:0/43882816 >> v2:172.21.15.69:3300/0 conn(0x7efc080b4140 0x7efc08071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 >> v1:172.21.15.69:6789/0 conn(0x7efc081435d0 legacy=0x7efc080b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080715e0 con 0x7efc081439b0 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7efc00001670 con 0x7efc081439b0 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 >> v1:172.21.15.43:6789/0 conn(0x7efc081439b0 legacy=0x7efc08070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efbe80039e0 0x7efbe8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80047c0 0x7efbe8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.131+0000 7efc06a00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 0x7efbe800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08071300 con 0x7efbe80051f0 2024-01-16T15:27:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe800c150 con 0x7efbe80047c0 2024-01-16T15:27:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe800c310 con 0x7efbe80039e0 2024-01-16T15:27:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 0x7efbe800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07400640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efbe80039e0 0x7efbe8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 -- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efbe80039e0 msgr2=0x7efbe8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efbe80039e0 0x7efbe8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 -- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80047c0 msgr2=0x7efbe8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80047c0 0x7efbe8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.132+0000 7efc07e00640 1 -- 172.21.15.43:0/43882816 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe8003520 con 0x7efbe80051f0 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.133+0000 7efc07e00640 1 -- 172.21.15.43:0/43882816 reap_dead start 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.133+0000 7efc07e00640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 0x7efbe800b980 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c210 tx=0x7efbe80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3a6c99fa789c42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00005ad0 con 0x7efbe80051f0 2024-01-16T15:27:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00005c30 con 0x7efbe80051f0 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc06a00640 1 -- 172.21.15.43:0/43882816 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0002c7c0 con 0x7efbe80051f0 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc0d200640 1 -- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 msgr2=0x7efbe800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc0d200640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 0x7efbe800b980 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c210 tx=0x7efbe80057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc0d200640 1 -- 172.21.15.43:0/43882816 shutdown_connections 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc0d200640 1 --2- 172.21.15.43:0/43882816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbe80051f0 0x7efbe800b980 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.134+0000 7efc0d200640 1 -- 172.21.15.43:0/43882816 >> 172.21.15.43:0/43882816 conn(0x7efc080cec10 msgr2=0x7efc080cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.135+0000 7efc0d200640 1 -- 172.21.15.43:0/43882816 shutdown_connections 2024-01-16T15:27:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.135+0000 7efc0d200640 1 -- 172.21.15.43:0/43882816 wait complete. 2024-01-16T15:27:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.136+0000 7efc0d200640 1 Processor -- start 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.136+0000 7efc0d200640 1 -- start start 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.137+0000 7efc0d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80051f0 0x7efc080c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.137+0000 7efc0d200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc081f6df0 0x7efc080cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.137+0000 7efc0d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 0x7efc081f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc0d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c7b80 con 0x7efc080cdd70 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc0d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080ce1f0 con 0x7efbe80051f0 2024-01-16T15:27:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc0d200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080ae2e0 con 0x7efc081f6df0 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 0x7efc081f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 0x7efc081f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54074/0 (socket says 172.21.15.43:54074) 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 -- 172.21.15.43:0/3841576444 learned_addr learned my addr 172.21.15.43:0/3841576444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efbfe800640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc081f6df0 0x7efc080cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07400640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80051f0 0x7efc080c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 -- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc081f6df0 msgr2=0x7efc080cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc081f6df0 0x7efc080cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 -- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80051f0 msgr2=0x7efc080c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80051f0 0x7efc080c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07e00640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc081f3b00 con 0x7efc080cdd70 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.138+0000 7efc07400640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efbe80051f0 0x7efc080c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efc07e00640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 0x7efc081f2c10 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c210 tx=0x7efc00001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0002c9c0 con 0x7efc080cdd70 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0002cb40 con 0x7efc080cdd70 2024-01-16T15:27:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00007070 con 0x7efc080cdd70 2024-01-16T15:27:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc081f6960 con 0x7efc080cdd70 2024-01-16T15:27:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.139+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc081f6310 con 0x7efc080cdd70 2024-01-16T15:27:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.140+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc0804e840 con 0x7efc080cdd70 2024-01-16T15:27:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.141+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7efc00032070 con 0x7efc080cdd70 2024-01-16T15:27:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.142+0000 7efbffe00640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efbe006c4c0 0x7efbe006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.142+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efc0003c070 con 0x7efc080cdd70 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.142+0000 7efbffe00640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7efbe0074780 0x7efbe0076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.142+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe00772b0 con 0x7efbe0074780 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.142+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc000a9f80 con 0x7efc080cdd70 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.143+0000 7efc07400640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efbe006c4c0 0x7efbe006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.143+0000 7efbfe800640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7efbe0074780 0x7efbe0076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.143+0000 7efc07400640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efbe006c4c0 0x7efbe006e980 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efbf8000f80 tx=0x7efbf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.144+0000 7efbfe800640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7efbe0074780 0x7efbe0076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.146+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== osd.5 v2:172.21.15.69:6808/1266529003 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efbe00772b0 con 0x7efbe0074780 2024-01-16T15:27:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.161+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 --> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efc081f6af0 con 0x7efbe0074780 2024-01-16T15:27:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.162+0000 7efbffe00640 1 -- 172.21.15.43:0/3841576444 <== osd.5 v2:172.21.15.69:6808/1266529003 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efc081f6af0 con 0x7efbe0074780 2024-01-16T15:27:31.161 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:31.162 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7efbe0074780 msgr2=0x7efbe0076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.69:6808/1266529003,v1:172.21.15.69:6809/1266529003] conn(0x7efbe0074780 0x7efbe0076be0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efbe006c4c0 msgr2=0x7efbe006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7efbe006c4c0 0x7efbe006e980 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efbf8000f80 tx=0x7efbf8009040 comp rx=0 tx=0).stop 2024-01-16T15:27:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 msgr2=0x7efc081f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 --2- 172.21.15.43:0/3841576444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc080cdd70 0x7efc081f2c10 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c210 tx=0x7efc00001a00 comp rx=0 tx=0).stop 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc07e00640 1 -- 172.21.15.43:0/3841576444 reap_dead start 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 shutdown_connections 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 >> 172.21.15.43:0/3841576444 conn(0x7efc080cec10 msgr2=0x7efc0806ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 shutdown_connections 2024-01-16T15:27:31.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.163+0000 7efc0d200640 1 -- 172.21.15.43:0/3841576444 wait complete. 2024-01-16T15:27:31.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-16T15:27:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.407+0000 7f1a02e00640 1 Processor -- start 2024-01-16T15:27:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.407+0000 7f1a02e00640 1 -- start start 2024-01-16T15:27:31.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.408+0000 7f1a02e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19fc15ed20 0x7f19fc141e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.408+0000 7f1a02e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.408+0000 7f1a02e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 0x7f19fc06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.408+0000 7f1a02e00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc06cae0 con 0x7f19fc15ed20 2024-01-16T15:27:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a02e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc071e40 con 0x7f19fc071650 2024-01-16T15:27:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a02e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc06c1b0 con 0x7f19fc15f100 2024-01-16T15:27:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19fc15ed20 0x7f19fc141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19fc15ed20 0x7f19fc141e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54082/0 (socket says 172.21.15.43:54082) 2024-01-16T15:27:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 -- 172.21.15.43:0/2415174732 learned_addr learned my addr 172.21.15.43:0/2415174732 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:31.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01000640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 -- 172.21.15.43:0/2415174732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 msgr2=0x7f19fc071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc071110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a01a00640 1 -- 172.21.15.43:0/2415174732 >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 msgr2=0x7f19fc06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.409+0000 7f1a02400640 1 --2- 172.21.15.43:0/2415174732 >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 0x7f19fc06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.410+0000 7f1a01a00640 1 --2- 172.21.15.43:0/2415174732 >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 0x7f19fc06cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.410+0000 7f1a01a00640 1 -- 172.21.15.43:0/2415174732 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc06c490 con 0x7f19fc15ed20 2024-01-16T15:27:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.410+0000 7f1a01000640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.410+0000 7f1a02400640 1 --2- 172.21.15.43:0/2415174732 >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 0x7f19fc06cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.411+0000 7f1a01a00640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc15ed20 0x7f19fc141e80 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f19fc15fe40 tx=0x7f19ec02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd53414454900a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.411+0000 7f19fbe00640 1 -- 172.21.15.43:0/2415174732 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec031880 con 0x7f19fc15ed20 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.411+0000 7f19fbe00640 1 -- 172.21.15.43:0/2415174732 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec0319e0 con 0x7f19fc15ed20 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.411+0000 7f19fbe00640 1 -- 172.21.15.43:0/2415174732 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec031d70 con 0x7f19fc15ed20 2024-01-16T15:27:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 -- 172.21.15.43:0/2415174732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc15ed20 msgr2=0x7f19fc141e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc15ed20 0x7f19fc141e80 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f19fc15fe40 tx=0x7f19ec02b920 comp rx=0 tx=0).stop 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 -- 172.21.15.43:0/2415174732 shutdown_connections 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 --2- 172.21.15.43:0/2415174732 >> v2:172.21.15.69:3300/0 conn(0x7f19fc071650 0x7f19fc06cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 --2- 172.21.15.43:0/2415174732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc15ed20 0x7f19fc141e80 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 -- 172.21.15.43:0/2415174732 >> 172.21.15.43:0/2415174732 conn(0x7f19fc0b0840 msgr2=0x7f19fc0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 -- 172.21.15.43:0/2415174732 shutdown_connections 2024-01-16T15:27:31.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.412+0000 7f1a02e00640 1 -- 172.21.15.43:0/2415174732 wait complete. 2024-01-16T15:27:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.414+0000 7f1a02e00640 1 Processor -- start 2024-01-16T15:27:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.414+0000 7f1a02e00640 1 -- start start 2024-01-16T15:27:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19fc15ed20 0x7f19fc13d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc153080 con 0x7f19fc071650 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc13dec0 con 0x7f19fc15ed20 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a02e00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc152750 con 0x7f19fc15f100 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a01a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a01a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54098/0 (socket says 172.21.15.43:54098) 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.415+0000 7f1a01a00640 1 -- 172.21.15.43:0/4287517888 learned_addr learned my addr 172.21.15.43:0/4287517888 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 -- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 msgr2=0x7f19fc13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 -- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19fc15ed20 msgr2=0x7f19fc13d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01000640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19fc15ed20 0x7f19fc13d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19fc15ed20 0x7f19fc13d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 -- 172.21.15.43:0/4287517888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0094d0 con 0x7f19fc071650 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a01a00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f19ec000c00 tx=0x7f19ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec031870 con 0x7f19fc071650 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec0443f0 con 0x7f19fc071650 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0447a0 con 0x7f19fc071650 2024-01-16T15:27:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.416+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc152a30 con 0x7f19fc071650 2024-01-16T15:27:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.417+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc13cde0 con 0x7f19fc071650 2024-01-16T15:27:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.418+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc067d10 con 0x7f19fc071650 2024-01-16T15:27:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.419+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f19ec044900 con 0x7f19fc071650 2024-01-16T15:27:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.420+0000 7f19f9600640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 0x7f19d406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.420+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f19ec0ab3c0 con 0x7f19fc071650 2024-01-16T15:27:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.420+0000 7f1a01000640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 0x7f19d406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.421+0000 7f1a01000640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 0x7f19d406e980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f19fc15b350 tx=0x7f19e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.425+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19ec077a30 con 0x7f19fc071650 2024-01-16T15:27:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.617+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f19fc071ab0 con 0x7f19d406c4c0 2024-01-16T15:27:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.618+0000 7f19f9600640 1 -- 172.21.15.43:0/4287517888 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f19e400c040 con 0x7f19d406c4c0 2024-01-16T15:27:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 5 to deep-scrub 2024-01-16T15:27:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 msgr2=0x7f19d406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 0x7f19d406e980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f19fc15b350 tx=0x7f19e4008040 comp rx=0 tx=0).stop 2024-01-16T15:27:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 msgr2=0x7f19fc144490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f19ec000c00 tx=0x7f19ec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 shutdown_connections 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f19d406c4c0 0x7f19d406e980 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19fc15ed20 0x7f19fc13d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19fc15f100 0x7f19fc13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 --2- 172.21.15.43:0/4287517888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19fc071650 0x7f19fc144490 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 >> 172.21.15.43:0/4287517888 conn(0x7f19fc0b0840 msgr2=0x7f19fc0b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 shutdown_connections 2024-01-16T15:27:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.621+0000 7f1a02e00640 1 -- 172.21.15.43:0/4287517888 wait complete. 2024-01-16T15:27:31.640 INFO:tasks.ceph:Scrubbing osd.6 2024-01-16T15:27:31.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.880+0000 7f99c7200640 1 Processor -- start 2024-01-16T15:27:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.880+0000 7f99c7200640 1 -- start start 2024-01-16T15:27:31.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c7200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99c015d730 0x7f99c00cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c00d0660 con 0x7f99c006b8f0 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c7200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c00d0380 con 0x7f99c015d730 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00cb530 con 0x7f99c006c480 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99c015d730 0x7f99c00cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c5e00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f99c015d730 0x7f99c00cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54112/0 (socket says 172.21.15.43:54112) 2024-01-16T15:27:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.882+0000 7f99c5e00640 1 -- 172.21.15.43:0/3392462774 learned_addr learned my addr 172.21.15.43:0/3392462774 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c6800640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c5e00640 1 -- 172.21.15.43:0/3392462774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 msgr2=0x7f99c00cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c5e00640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c5e00640 1 -- 172.21.15.43:0/3392462774 >> v1:172.21.15.69:6789/0 conn(0x7f99c006b8f0 legacy=0x7f99c006bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c5e00640 1 -- 172.21.15.43:0/3392462774 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c00cb810 con 0x7f99c015d730 2024-01-16T15:27:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.883+0000 7f99c6800640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.884+0000 7f99c5e00640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c015d730 0x7f99c00cf080 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e50 tx=0x7f99b002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ce31d65f1c01c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.884+0000 7f99c4a00640 1 -- 172.21.15.43:0/3392462774 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b002c950 con 0x7f99c015d730 2024-01-16T15:27:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.884+0000 7f99c4a00640 1 -- 172.21.15.43:0/3392462774 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b002cab0 con 0x7f99c015d730 2024-01-16T15:27:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.884+0000 7f99c4a00640 1 -- 172.21.15.43:0/3392462774 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b002ce40 con 0x7f99c015d730 2024-01-16T15:27:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.885+0000 7f99c7200640 1 -- 172.21.15.43:0/3392462774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c015d730 msgr2=0x7f99c00cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.885+0000 7f99c7200640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c015d730 0x7f99c00cf080 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e50 tx=0x7f99b002bdb0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.885+0000 7f99c7200640 1 -- 172.21.15.43:0/3392462774 shutdown_connections 2024-01-16T15:27:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.885+0000 7f99c7200640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.885+0000 7f99c7200640 1 --2- 172.21.15.43:0/3392462774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c015d730 0x7f99c00cf080 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.886+0000 7f99c7200640 1 -- 172.21.15.43:0/3392462774 >> 172.21.15.43:0/3392462774 conn(0x7f99c00d0e40 msgr2=0x7f99c00d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.886+0000 7f99c7200640 1 -- 172.21.15.43:0/3392462774 shutdown_connections 2024-01-16T15:27:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.886+0000 7f99c7200640 1 -- 172.21.15.43:0/3392462774 wait complete. 2024-01-16T15:27:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.888+0000 7f99c7200640 1 Processor -- start 2024-01-16T15:27:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.888+0000 7f99c7200640 1 -- start start 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.889+0000 7f99c7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 0x7f99c015e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.889+0000 7f99c7200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99c015d730 0x7f99c015f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00c8a10 con 0x7f99c006b8f0 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c015fda0 con 0x7f99c015d730 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c7200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00c49d0 con 0x7f99c006c480 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 0x7f99c015e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 0x7f99c015e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54122/0 (socket says 172.21.15.43:54122) 2024-01-16T15:27:31.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 -- 172.21.15.43:0/1823845541 learned_addr learned my addr 172.21.15.43:0/1823845541 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5400640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 -- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 msgr2=0x7f99c00c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99c006c480 0x7f99c00c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c6800640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99c015d730 0x7f99c015f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 -- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99c015d730 msgr2=0x7f99c015f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99c015d730 0x7f99c015f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.890+0000 7f99c5e00640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b0009b30 con 0x7f99c006b8f0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.891+0000 7f99c5e00640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 0x7f99c015e900 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e50 tx=0x7f99b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.891+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b003e450 con 0x7f99c006b8f0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.891+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b003e5f0 con 0x7f99c006b8f0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.891+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b003e9a0 con 0x7f99c006b8f0 2024-01-16T15:27:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.891+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c00c4cb0 con 0x7f99c006b8f0 2024-01-16T15:27:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.892+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c00c5270 con 0x7f99c006b8f0 2024-01-16T15:27:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.892+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99c00c4b40 con 0x7f99c006b8f0 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.894+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f99b003eb00 con 0x7f99c006b8f0 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99be000640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f999806c4c0 0x7f999806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f99b00abf90 con 0x7f99c006b8f0 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99be000640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f9998074780 0x7f9998076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99980772b0 con 0x7f9998074780 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99b00aca90 con 0x7f99c006b8f0 2024-01-16T15:27:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.895+0000 7f99c5400640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f999806c4c0 0x7f999806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.896+0000 7f99c6800640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f9998074780 0x7f9998076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.896+0000 7f99c5400640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f999806c4c0 0x7f999806e980 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f99c00cb230 tx=0x7f99a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.897+0000 7f99c6800640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f9998074780 0x7f9998076be0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:31.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.899+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== osd.6 v2:172.21.15.69:6800/3561904235 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f99980772b0 con 0x7f9998074780 2024-01-16T15:27:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.916+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 --> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f99c00d0070 con 0x7f9998074780 2024-01-16T15:27:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99be000640 1 -- 172.21.15.43:0/1823845541 <== osd.6 v2:172.21.15.69:6800/3561904235 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f99b800c040 con 0x7f9998074780 2024-01-16T15:27:31.916 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:31.916 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:31.917 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f9998074780 msgr2=0x7f9998076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.69:6800/3561904235,v1:172.21.15.69:6801/3561904235] conn(0x7f9998074780 0x7f9998076be0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f999806c4c0 msgr2=0x7f999806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f999806c4c0 0x7f999806e980 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f99c00cb230 tx=0x7f99a8009040 comp rx=0 tx=0).stop 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 msgr2=0x7f99c015e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 --2- 172.21.15.43:0/1823845541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99c006b8f0 0x7f99c015e900 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e50 tx=0x7f99b0004a40 comp rx=0 tx=0).stop 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c6800640 1 -- 172.21.15.43:0/1823845541 reap_dead start 2024-01-16T15:27:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 shutdown_connections 2024-01-16T15:27:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 >> 172.21.15.43:0/1823845541 conn(0x7f99c00d0e40 msgr2=0x7f99c015c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 shutdown_connections 2024-01-16T15:27:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:31.918+0000 7f99c7200640 1 -- 172.21.15.43:0/1823845541 wait complete. 2024-01-16T15:27:31.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-16T15:27:32.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.160+0000 7ffbdd800640 1 Processor -- start 2024-01-16T15:27:32.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.161+0000 7ffbdd800640 1 -- start start 2024-01-16T15:27:32.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.162+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.162+0000 7ffbdd800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffbd80d1600 0x7ffbd80b4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.162+0000 7ffbdd800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd80cf4e0 con 0x7ffbd815d710 2024-01-16T15:27:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.162+0000 7ffbdd800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd80cfe00 con 0x7ffbd80d1600 2024-01-16T15:27:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.162+0000 7ffbdd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80d00e0 con 0x7ffbd80cef90 2024-01-16T15:27:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.163+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.163+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.43:49192/0 (socket says 172.21.15.43:49192) 2024-01-16T15:27:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.163+0000 7ffbd6000640 1 -- 172.21.15.43:0/1346648355 learned_addr learned my addr 172.21.15.43:0/1346648355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.163+0000 7ffbd7400640 1 --2- 172.21.15.43:0/1346648355 >> v2:172.21.15.69:3300/0 conn(0x7ffbd80d1600 0x7ffbd80b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.164+0000 7ffbd6000640 1 -- 172.21.15.43:0/1346648355 >> v2:172.21.15.69:3300/0 conn(0x7ffbd80d1600 msgr2=0x7ffbd80b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.164+0000 7ffbd6000640 1 --2- 172.21.15.43:0/1346648355 >> v2:172.21.15.69:3300/0 conn(0x7ffbd80d1600 0x7ffbd80b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.164+0000 7ffbd6000640 1 -- 172.21.15.43:0/1346648355 >> v1:172.21.15.69:6789/0 conn(0x7ffbd815d710 legacy=0x7ffbd80d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.164+0000 7ffbd6000640 1 -- 172.21.15.43:0/1346648355 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd80d03c0 con 0x7ffbd80cef90 2024-01-16T15:27:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.165+0000 7ffbd6000640 1 --2- 172.21.15.43:0/1346648355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009870 tx=0x7ffbc002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa8ea13249b9ea97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbd5600640 1 -- 172.21.15.43:0/1346648355 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0030840 con 0x7ffbd80cef90 2024-01-16T15:27:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbd5600640 1 -- 172.21.15.43:0/1346648355 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc00309a0 con 0x7ffbd80cef90 2024-01-16T15:27:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbd5600640 1 -- 172.21.15.43:0/1346648355 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0030d30 con 0x7ffbd80cef90 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 -- 172.21.15.43:0/1346648355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 msgr2=0x7ffbd80b5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 --2- 172.21.15.43:0/1346648355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009870 tx=0x7ffbc002b980 comp rx=0 tx=0).stop 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 -- 172.21.15.43:0/1346648355 shutdown_connections 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 --2- 172.21.15.43:0/1346648355 >> v2:172.21.15.69:3300/0 conn(0x7ffbd80d1600 0x7ffbd80b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 --2- 172.21.15.43:0/1346648355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd80b5780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 -- 172.21.15.43:0/1346648355 >> 172.21.15.43:0/1346648355 conn(0x7ffbd80b2e00 msgr2=0x7ffbd80b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:32.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.166+0000 7ffbdd800640 1 -- 172.21.15.43:0/1346648355 shutdown_connections 2024-01-16T15:27:32.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.167+0000 7ffbdd800640 1 -- 172.21.15.43:0/1346648355 wait complete. 2024-01-16T15:27:32.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.168+0000 7ffbdd800640 1 Processor -- start 2024-01-16T15:27:32.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.168+0000 7ffbdd800640 1 -- start start 2024-01-16T15:27:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd815f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd815d710 0x7ffbd81f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd81f3e20 con 0x7ffbd80d1600 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd815e300 con 0x7ffbd815d710 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.169+0000 7ffbdd800640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd815e5e0 con 0x7ffbd80cef90 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54124/0 (socket says 172.21.15.43:54124) 2024-01-16T15:27:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 -- 172.21.15.43:0/4010390445 learned_addr learned my addr 172.21.15.43:0/4010390445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6a00640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd815f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd7400640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd815d710 0x7ffbd81f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 -- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 msgr2=0x7ffbd815f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd815f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 -- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd815d710 msgr2=0x7ffbd81f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd815d710 0x7ffbd81f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 -- 172.21.15.43:0/4010390445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00094d0 con 0x7ffbd80d1600 2024-01-16T15:27:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.170+0000 7ffbd6000640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0000c00 tx=0x7ffbc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.171+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0039c90 con 0x7ffbd80d1600 2024-01-16T15:27:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.171+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc0039df0 con 0x7ffbd80d1600 2024-01-16T15:27:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.171+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0043640 con 0x7ffbd80d1600 2024-01-16T15:27:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.171+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd815e8c0 con 0x7ffbd80d1600 2024-01-16T15:27:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.171+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd815ee80 con 0x7ffbd80d1600 2024-01-16T15:27:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.172+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd8074770 con 0x7ffbd80d1600 2024-01-16T15:27:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.173+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ffbc00437a0 con 0x7ffbd80d1600 2024-01-16T15:27:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.174+0000 7ffbcea00640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 0x7ffbb006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.174+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffbc003e070 con 0x7ffbd80d1600 2024-01-16T15:27:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.174+0000 7ffbd6a00640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 0x7ffbb006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.175+0000 7ffbd6a00640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 0x7ffbb006e980 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80012e0 tx=0x7ffbc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.178+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbc0078160 con 0x7ffbd80d1600 2024-01-16T15:27:32.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.377+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbd815daf0 con 0x7ffbb006c4c0 2024-01-16T15:27:32.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.378+0000 7ffbcea00640 1 -- 172.21.15.43:0/4010390445 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffbc8001a10 con 0x7ffbb006c4c0 2024-01-16T15:27:32.377 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 6 to deep-scrub 2024-01-16T15:27:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 msgr2=0x7ffbb006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 0x7ffbb006e980 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80012e0 tx=0x7ffbc8009040 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 msgr2=0x7ffbd81f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0000c00 tx=0x7ffbc0004730 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 shutdown_connections 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7ffbb006c4c0 0x7ffbb006e980 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd815d710 0x7ffbd81f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbd80d1600 0x7ffbd81f0430 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 --2- 172.21.15.43:0/4010390445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffbd80cef90 0x7ffbd815f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 >> 172.21.15.43:0/4010390445 conn(0x7ffbd80b2e00 msgr2=0x7ffbd80b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 shutdown_connections 2024-01-16T15:27:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.382+0000 7ffbdd800640 1 -- 172.21.15.43:0/4010390445 wait complete. 2024-01-16T15:27:32.410 INFO:tasks.ceph:Scrubbing osd.7 2024-01-16T15:27:32.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:27:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.626+0000 7f14bd400640 1 Processor -- start 2024-01-16T15:27:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.626+0000 7f14bd400640 1 -- start start 2024-01-16T15:27:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.628+0000 7f14bd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14b815c1c0 0x7f14b80ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.628+0000 7f14bd400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b8070dd0 0x7f14b8071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.628+0000 7f14bd400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b8071770 con 0x7f14b806c9d0 2024-01-16T15:27:32.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.628+0000 7f14bd400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b815c5a0 con 0x7f14b815c1c0 2024-01-16T15:27:32.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.628+0000 7f14bd400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b80cdf70 con 0x7f14b8070dd0 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.629+0000 7f14b7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14b815c1c0 0x7f14b80ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.629+0000 7f14b7400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b8070dd0 0x7f14b8071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.629+0000 7f14b7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14b815c1c0 0x7f14b80ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.43:36540/0 (socket says 172.21.15.43:36540) 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.629+0000 7f14b7e00640 1 -- 172.21.15.43:0/990514627 learned_addr learned my addr 172.21.15.43:0/990514627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.630+0000 7f14b7e00640 1 -- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b8070dd0 msgr2=0x7f14b8071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.630+0000 7f14b7e00640 1 --2- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b8070dd0 0x7f14b8071230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.630+0000 7f14b7e00640 1 -- 172.21.15.43:0/990514627 >> v1:172.21.15.69:6789/0 conn(0x7f14b806c9d0 legacy=0x7f14b806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.630+0000 7f14b7e00640 1 -- 172.21.15.43:0/990514627 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b80ce230 con 0x7f14b815c1c0 2024-01-16T15:27:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.631+0000 7f14b7e00640 1 --2- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b815c1c0 0x7f14b80ced10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009870 tx=0x7f14a802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f7dbdb7f26f67bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.631+0000 7f14b6a00640 1 -- 172.21.15.43:0/990514627 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8030840 con 0x7f14b815c1c0 2024-01-16T15:27:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14b6a00640 1 -- 172.21.15.43:0/990514627 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a80309a0 con 0x7f14b815c1c0 2024-01-16T15:27:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14b6a00640 1 -- 172.21.15.43:0/990514627 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8030d30 con 0x7f14b815c1c0 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 -- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b815c1c0 msgr2=0x7f14b80ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 --2- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b815c1c0 0x7f14b80ced10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009870 tx=0x7f14a802b980 comp rx=0 tx=0).stop 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 -- 172.21.15.43:0/990514627 shutdown_connections 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 --2- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b8070dd0 0x7f14b8071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 --2- 172.21.15.43:0/990514627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b815c1c0 0x7f14b80ced10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 -- 172.21.15.43:0/990514627 >> 172.21.15.43:0/990514627 conn(0x7f14b80b15c0 msgr2=0x7f14b80b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.632+0000 7f14bd400640 1 -- 172.21.15.43:0/990514627 shutdown_connections 2024-01-16T15:27:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.633+0000 7f14bd400640 1 -- 172.21.15.43:0/990514627 wait complete. 2024-01-16T15:27:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.634+0000 7f14bd400640 1 Processor -- start 2024-01-16T15:27:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.634+0000 7f14bd400640 1 -- start start 2024-01-16T15:27:32.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.635+0000 7f14bd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b806c9d0 0x7f14b80ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.635+0000 7f14bd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 0x7f14b815a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14bd400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b815c1c0 0x7f14b8155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14bd400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b815a590 con 0x7f14b8070dd0 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14bd400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8157c20 con 0x7f14b806c9d0 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14bd400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b81310b0 con 0x7f14b815c1c0 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 0x7f14b815a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 0x7f14b815a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54134/0 (socket says 172.21.15.43:54134) 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7400640 1 -- 172.21.15.43:0/204443768 learned_addr learned my addr 172.21.15.43:0/204443768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:32.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7e00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b806c9d0 0x7f14b80ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14bca00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b815c1c0 0x7f14b8155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7400640 1 -- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b815c1c0 msgr2=0x7f14b8155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.636+0000 7f14b7400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14b815c1c0 0x7f14b8155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14b7400640 1 -- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b806c9d0 msgr2=0x7f14b80ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14b7400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14b806c9d0 0x7f14b80ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14b7400640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80094d0 con 0x7f14b8070dd0 2024-01-16T15:27:32.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14b7400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 0x7f14b815a050 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f14a000e8a0 tx=0x7f14a000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00094f0 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a000c940 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a000ccf0 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.637+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b8157660 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.638+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b8130d40 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.639+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14b806b3a0 con 0x7f14b8070dd0 2024-01-16T15:27:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.640+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f14a0013470 con 0x7f14b8070dd0 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14afe00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14900708d0 0x7f1490072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14a0088b70 con 0x7f14b8070dd0 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14b7e00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14900708d0 0x7f1490072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14afe00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f1490078b90 0x7f149007aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f149007b6c0 con 0x7f1490078b90 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.641+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=115) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a008ea50 con 0x7f14b8070dd0 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.642+0000 7f14bca00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f1490078b90 0x7f149007aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.642+0000 7f14b7e00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14900708d0 0x7f1490072d90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036000 tx=0x7f14a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.643+0000 7f14bca00640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f1490078b90 0x7f149007aff0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.645+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== osd.7 v2:172.21.15.69:6824/1736397519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f149007b6c0 con 0x7f1490078b90 2024-01-16T15:27:32.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.659+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 --> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f14b80ce7f0 con 0x7f1490078b90 2024-01-16T15:27:32.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.660+0000 7f14afe00640 1 -- 172.21.15.43:0/204443768 <== osd.7 v2:172.21.15.69:6824/1736397519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f14b0001df0 con 0x7f1490078b90 2024-01-16T15:27:32.659 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-01-16T15:27:32.659 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:27:32.659 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-01-16T15:27:32.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.660+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f1490078b90 msgr2=0x7f149007aff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.69:6824/1736397519,v1:172.21.15.69:6825/1736397519] conn(0x7f1490078b90 0x7f149007aff0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14900708d0 msgr2=0x7f1490072d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f14900708d0 0x7f1490072d90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036000 tx=0x7f14a8008030 comp rx=0 tx=0).stop 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 msgr2=0x7f14b815a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 --2- 172.21.15.43:0/204443768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b8070dd0 0x7f14b815a050 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f14a000e8a0 tx=0x7f14a000ed60 comp rx=0 tx=0).stop 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bca00640 1 -- 172.21.15.43:0/204443768 reap_dead start 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 shutdown_connections 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 >> 172.21.15.43:0/204443768 conn(0x7f14b80b15c0 msgr2=0x7f14b80b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:32.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 shutdown_connections 2024-01-16T15:27:32.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.661+0000 7f14bd400640 1 -- 172.21.15.43:0/204443768 wait complete. 2024-01-16T15:27:32.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-16T15:27:32.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.891+0000 7fa719400640 1 Processor -- start 2024-01-16T15:27:32.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.891+0000 7fa719400640 1 -- start start 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.892+0000 7fa719400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa7140ae200 0x7fa71406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.892+0000 7fa719400640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71415d680 con 0x7fa71415d2a0 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.892+0000 7fa719400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71406a940 con 0x7fa7140aee10 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.892+0000 7fa719400640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa71406ac20 con 0x7fa7140ae200 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa712800640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa71415d2a0 0x7fa7140ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:33272/0 (socket says 172.21.15.43:33272) 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa712800640 1 -- 172.21.15.43:0/1355115715 learned_addr learned my addr 172.21.15.43:0/1355115715 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:27:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa713200640 1 --2- 172.21.15.43:0/1355115715 >> v2:172.21.15.43:3300/0 conn(0x7fa7140ae200 0x7fa71406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa713200640 1 -- 172.21.15.43:0/1355115715 >> v1:172.21.15.69:6789/0 conn(0x7fa7140aee10 legacy=0x7fa7140b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa713200640 1 -- 172.21.15.43:0/1355115715 >> v1:172.21.15.43:6789/0 conn(0x7fa71415d2a0 legacy=0x7fa7140ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.893+0000 7fa713200640 1 -- 172.21.15.43:0/1355115715 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71406af00 con 0x7fa7140ae200 2024-01-16T15:27:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.895+0000 7fa713200640 1 --2- 172.21.15.43:0/1355115715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7140ae200 0x7fa71406b9e0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0097a0 tx=0x7fa70c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bd96147a9d5671 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.895+0000 7fa711400640 1 -- 172.21.15.43:0/1355115715 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c031830 con 0x7fa7140ae200 2024-01-16T15:27:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.895+0000 7fa711400640 1 -- 172.21.15.43:0/1355115715 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c031990 con 0x7fa7140ae200 2024-01-16T15:27:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.895+0000 7fa711400640 1 -- 172.21.15.43:0/1355115715 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c031d20 con 0x7fa7140ae200 2024-01-16T15:27:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 -- 172.21.15.43:0/1355115715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7140ae200 msgr2=0x7fa71406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 --2- 172.21.15.43:0/1355115715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7140ae200 0x7fa71406b9e0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0097a0 tx=0x7fa70c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 -- 172.21.15.43:0/1355115715 shutdown_connections 2024-01-16T15:27:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 --2- 172.21.15.43:0/1355115715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7140ae200 0x7fa71406b9e0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 -- 172.21.15.43:0/1355115715 >> 172.21.15.43:0/1355115715 conn(0x7fa714070d60 msgr2=0x7fa714071140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 -- 172.21.15.43:0/1355115715 shutdown_connections 2024-01-16T15:27:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.896+0000 7fa719400640 1 -- 172.21.15.43:0/1355115715 wait complete. 2024-01-16T15:27:32.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.898+0000 7fa719400640 1 Processor -- start 2024-01-16T15:27:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.898+0000 7fa719400640 1 -- start start 2024-01-16T15:27:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 0x7fa7141546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7140aee10 0x7fa7141f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141f66d0 con 0x7fa71415d2a0 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141f63d0 con 0x7fa7140aee10 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa719400640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141f35e0 con 0x7fa7140ae200 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa713200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa713200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54158/0 (socket says 172.21.15.43:54158) 2024-01-16T15:27:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.899+0000 7fa713200640 1 -- 172.21.15.43:0/2997082124 learned_addr learned my addr 172.21.15.43:0/2997082124 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:32.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa711e00640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7140aee10 0x7fa7141f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa712800640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 0x7fa7141546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa713200640 1 -- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 msgr2=0x7fa7141546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa713200640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 0x7fa7141546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa713200640 1 -- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7140aee10 msgr2=0x7fa7141f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa713200640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7140aee10 0x7fa7141f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa713200640 1 -- 172.21.15.43:0/2997082124 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c009400 con 0x7fa71415d2a0 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.900+0000 7fa712800640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 0x7fa7141546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa713200640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098c0 tx=0x7fa70c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c03b9f0 con 0x7fa71415d2a0 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c03bb50 con 0x7fa71415d2a0 2024-01-16T15:27:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c0453f0 con 0x7fa71415d2a0 2024-01-16T15:27:32.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7141f38c0 con 0x7fa71415d2a0 2024-01-16T15:27:32.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.901+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7141f3ba0 con 0x7fa71415d2a0 2024-01-16T15:27:32.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.902+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa714074770 con 0x7fa71415d2a0 2024-01-16T15:27:32.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.904+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa70c045550 con 0x7fa71415d2a0 2024-01-16T15:27:32.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.905+0000 7fa70aa00640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 0x7fa6ec06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:32.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.905+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa70c0aae70 con 0x7fa71415d2a0 2024-01-16T15:27:32.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.905+0000 7fa712800640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 0x7fa6ec06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:32.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.906+0000 7fa712800640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 0x7fa6ec06e980 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa704005c00 tx=0x7fa704009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:32.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:32.914+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa70c077590 con 0x7fa71415d2a0 2024-01-16T15:27:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.105+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fa71406b4c0 con 0x7fa6ec06c4c0 2024-01-16T15:27:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.105+0000 7fa70aa00640 1 -- 172.21.15.43:0/2997082124 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa70400dc30 con 0x7fa6ec06c4c0 2024-01-16T15:27:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 7 to deep-scrub 2024-01-16T15:27:33.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 msgr2=0x7fa6ec06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 0x7fa6ec06e980 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa704005c00 tx=0x7fa704009040 comp rx=0 tx=0).stop 2024-01-16T15:27:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 msgr2=0x7fa7141f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098c0 tx=0x7fa70c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 shutdown_connections 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fa6ec06c4c0 0x7fa6ec06e980 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7140aee10 0x7fa7141f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa71415d2a0 0x7fa7141f6b60 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 --2- 172.21.15.43:0/2997082124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7140ae200 0x7fa7141546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 >> 172.21.15.43:0/2997082124 conn(0x7fa714070d60 msgr2=0x7fa71415c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 shutdown_connections 2024-01-16T15:27:33.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.108+0000 7fa719400640 1 -- 172.21.15.43:0/2997082124 wait complete. 2024-01-16T15:27:33.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:27:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.325+0000 7f149f200640 1 Processor -- start 2024-01-16T15:27:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.326+0000 7f149f200640 1 -- start start 2024-01-16T15:27:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.327+0000 7f149f200640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14a0160b60 0x7f14a00d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.327+0000 7f149f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a00d01a0 0x7f14a00b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.327+0000 7f149f200640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a00b2b30 con 0x7f14a00d3cf0 2024-01-16T15:27:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.327+0000 7f149f200640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a0160f40 con 0x7f14a0160b60 2024-01-16T15:27:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.327+0000 7f149f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00b2e10 con 0x7f14a00d01a0 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14a0160b60 0x7f14a00d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f14a0160b60 0x7f14a00d37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50982/0 (socket says 172.21.15.43:50982) 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 -- 172.21.15.43:0/4233641686 learned_addr learned my addr 172.21.15.43:0/4233641686 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149e800640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a00d01a0 0x7f14a00b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 -- 172.21.15.43:0/4233641686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a00d01a0 msgr2=0x7f14a00b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a00d01a0 0x7f14a00b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 -- 172.21.15.43:0/4233641686 >> v1:172.21.15.43:6789/0 conn(0x7f14a00d3cf0 legacy=0x7f14a00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.328+0000 7f149de00640 1 -- 172.21.15.43:0/4233641686 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a00b30f0 con 0x7f14a0160b60 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.329+0000 7f149de00640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a0160b60 0x7f14a00d37b0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f1490009870 tx=0x7f149002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db15df4a85991c40 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.329+0000 7f149d400640 1 -- 172.21.15.43:0/4233641686 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490031640 con 0x7f14a0160b60 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149d400640 1 -- 172.21.15.43:0/4233641686 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14900317a0 con 0x7f14a0160b60 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149d400640 1 -- 172.21.15.43:0/4233641686 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490031b30 con 0x7f14a0160b60 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 -- 172.21.15.43:0/4233641686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a0160b60 msgr2=0x7f14a00d37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a0160b60 0x7f14a00d37b0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f1490009870 tx=0x7f149002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 -- 172.21.15.43:0/4233641686 shutdown_connections 2024-01-16T15:27:33.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a00d01a0 0x7f14a00b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 --2- 172.21.15.43:0/4233641686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a0160b60 0x7f14a00d37b0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 -- 172.21.15.43:0/4233641686 >> 172.21.15.43:0/4233641686 conn(0x7f14a009c3d0 msgr2=0x7f14a00d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:33.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 -- 172.21.15.43:0/4233641686 shutdown_connections 2024-01-16T15:27:33.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.330+0000 7f149f200640 1 -- 172.21.15.43:0/4233641686 wait complete. 2024-01-16T15:27:33.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.332+0000 7f149f200640 1 Processor -- start 2024-01-16T15:27:33.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.332+0000 7f149f200640 1 -- start start 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a00d01a0 0x7f14a01587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a0160b60 0x7f14a015a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01583b0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00b5250 con 0x7f14a00d01a0 2024-01-16T15:27:33.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.333+0000 7f149f200640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0159d90 con 0x7f14a0160b60 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50984/0 (socket says 172.21.15.43:50984) 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 -- 172.21.15.43:0/1606958033 learned_addr learned my addr 172.21.15.43:0/1606958033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 -- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a0160b60 msgr2=0x7f14a015a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f149e800640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a0160b60 0x7f14a015a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a0160b60 0x7f14a015a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 -- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a00d01a0 msgr2=0x7f14a01587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a00d01a0 0x7f14a01587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.334+0000 7f1497e00640 1 -- 172.21.15.43:0/1606958033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14900094d0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f1497e00640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f148800c450 tx=0x7f148800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14880052d0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488005430 con 0x7f14a00d3cf0 2024-01-16T15:27:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a0157a10 con 0x7f14a00d3cf0 2024-01-16T15:27:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488012870 con 0x7f14a00d3cf0 2024-01-16T15:27:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.335+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a0157fd0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.336+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a0158620 con 0x7f14a00d3cf0 2024-01-16T15:27:33.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.337+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f14880129d0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.338+0000 7f1496000640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 0x7f147806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.338+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1488088650 con 0x7f14a00d3cf0 2024-01-16T15:27:33.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.338+0000 7f149de00640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 0x7f147806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.339+0000 7f149de00640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 0x7f147806e9b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1490036000 tx=0x7f1490030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.342+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1488054ba0 con 0x7f14a00d3cf0 2024-01-16T15:27:33.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.536+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14a00cf070 con 0x7f147806c4f0 2024-01-16T15:27:33.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.551+0000 7f1496000640 1 -- 172.21.15.43:0/1606958033 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287746 (secure 0 0 0) 0x7f149003a420 con 0x7f147806c4f0 2024-01-16T15:27:33.554 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.556+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 msgr2=0x7f147806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.556+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 0x7f147806e9b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1490036000 tx=0x7f1490030000 comp rx=0 tx=0).stop 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.556+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 msgr2=0x7f14a0157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.556+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f148800c450 tx=0x7f148800c910 comp rx=0 tx=0).stop 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 shutdown_connections 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f147806c4f0 0x7f147806e9b0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14a0160b60 0x7f14a015a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14a00d3cf0 0x7f14a0157130 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 --2- 172.21.15.43:0/1606958033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a00d01a0 0x7f14a01587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 >> 172.21.15.43:0/1606958033 conn(0x7f14a009c3d0 msgr2=0x7f14a015ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 shutdown_connections 2024-01-16T15:27:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:33.557+0000 7f149f200640 1 -- 172.21.15.43:0/1606958033 wait complete. 2024-01-16T15:27:33.574 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2024-01-16T15:27:33.505656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444294248,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2254,"num_write_kb":6295372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13389456,"kb_used_data":12597776,"kb_used_omap":12,"kb_used_meta":791603,"kb_avail":741585264,"statfs":{"total":773094113280,"available":759383310336,"internally_reserved":0,"allocated":12900122624,"data_stored":12893537648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":810602028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6763,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004676"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737839+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737839+0000","last_peered":"2024-01-16T15:27:17.737839+0000","last_clean":"2024-01-16T15:27:17.737839+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737839+0000","last_undegraded":"2024-01-16T15:27:17.737839+0000","last_fullsized":"2024-01-16T15:27:17.737839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'69","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736614+0000","last_change":"2024-01-16T15:18:50.514868+0000","last_active":"2024-01-16T15:27:17.736614+0000","last_peered":"2024-01-16T15:27:17.736614+0000","last_clean":"2024-01-16T15:27:17.736614+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:17.736614+0000","last_undegraded":"2024-01-16T15:27:17.736614+0000","last_fullsized":"2024-01-16T15:27:17.736614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:22.704331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.743697+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:27:17.743697+0000","last_peered":"2024-01-16T15:27:17.743697+0000","last_clean":"2024-01-16T15:27:17.743697+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:27:17.743697+0000","last_undegraded":"2024-01-16T15:27:17.743697+0000","last_fullsized":"2024-01-16T15:27:17.743697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:49.740353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733901+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733901+0000","last_peered":"2024-01-16T15:27:17.733901+0000","last_clean":"2024-01-16T15:27:17.733901+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733901+0000","last_undegraded":"2024-01-16T15:27:17.733901+0000","last_fullsized":"2024-01-16T15:27:17.733901+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'60","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720387+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:27:17.720387+0000","last_peered":"2024-01-16T15:27:17.720387+0000","last_clean":"2024-01-16T15:27:17.720387+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:17.720387+0000","last_undegraded":"2024-01-16T15:27:17.720387+0000","last_fullsized":"2024-01-16T15:27:17.720387+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'74","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722126+0000","last_change":"2024-01-16T15:18:50.517889+0000","last_active":"2024-01-16T15:27:17.722126+0000","last_peered":"2024-01-16T15:27:17.722126+0000","last_clean":"2024-01-16T15:27:17.722126+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:17.722126+0000","last_undegraded":"2024-01-16T15:27:17.722126+0000","last_fullsized":"2024-01-16T15:27:17.722126+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'66","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734633+0000","last_change":"2024-01-16T15:18:50.506124+0000","last_active":"2024-01-16T15:27:17.734633+0000","last_peered":"2024-01-16T15:27:17.734633+0000","last_clean":"2024-01-16T15:27:17.734633+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:17.734633+0000","last_undegraded":"2024-01-16T15:27:17.734633+0000","last_fullsized":"2024-01-16T15:27:17.734633+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:44.192793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'54","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730876+0000","last_change":"2024-01-16T15:18:50.513082+0000","last_active":"2024-01-16T15:27:17.730876+0000","last_peered":"2024-01-16T15:27:17.730876+0000","last_clean":"2024-01-16T15:27:17.730876+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:17.730876+0000","last_undegraded":"2024-01-16T15:27:17.730876+0000","last_fullsized":"2024-01-16T15:27:17.730876+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:01.514553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735972+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.735972+0000","last_peered":"2024-01-16T15:27:17.735972+0000","last_clean":"2024-01-16T15:27:17.735972+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.735972+0000","last_undegraded":"2024-01-16T15:27:17.735972+0000","last_fullsized":"2024-01-16T15:27:17.735972+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733227+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.733227+0000","last_peered":"2024-01-16T15:27:17.733227+0000","last_clean":"2024-01-16T15:27:17.733227+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.733227+0000","last_undegraded":"2024-01-16T15:27:17.733227+0000","last_fullsized":"2024-01-16T15:27:17.733227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'70","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.727985+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:27:17.727985+0000","last_peered":"2024-01-16T15:27:17.727985+0000","last_clean":"2024-01-16T15:27:17.727985+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:17.727985+0000","last_undegraded":"2024-01-16T15:27:17.727985+0000","last_fullsized":"2024-01-16T15:27:17.727985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:48.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735853+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735853+0000","last_peered":"2024-01-16T15:27:17.735853+0000","last_clean":"2024-01-16T15:27:17.735853+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735853+0000","last_undegraded":"2024-01-16T15:27:17.735853+0000","last_fullsized":"2024-01-16T15:27:17.735853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'57","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734731+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:27:17.734731+0000","last_peered":"2024-01-16T15:27:17.734731+0000","last_clean":"2024-01-16T15:27:17.734731+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:27:17.734731+0000","last_undegraded":"2024-01-16T15:27:17.734731+0000","last_fullsized":"2024-01-16T15:27:17.734731+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:35.209083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'65","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739366+0000","last_change":"2024-01-16T15:18:50.517177+0000","last_active":"2024-01-16T15:27:17.739366+0000","last_peered":"2024-01-16T15:27:17.739366+0000","last_clean":"2024-01-16T15:27:17.739366+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:17.739366+0000","last_undegraded":"2024-01-16T15:27:17.739366+0000","last_fullsized":"2024-01-16T15:27:17.739366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:17.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'54","reported_seq":78,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.725272+0000","last_change":"2024-01-16T15:18:50.518134+0000","last_active":"2024-01-16T15:27:17.725272+0000","last_peered":"2024-01-16T15:27:17.725272+0000","last_clean":"2024-01-16T15:27:17.725272+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:17.725272+0000","last_undegraded":"2024-01-16T15:27:17.725272+0000","last_fullsized":"2024-01-16T15:27:17.725272+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'74","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.886948+0000","last_change":"2024-01-16T15:27:28.887006+0000","last_active":"2024-01-16T15:27:28.886948+0000","last_peered":"2024-01-16T15:27:28.886948+0000","last_clean":"2024-01-16T15:27:28.886948+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:28.886948+0000","last_undegraded":"2024-01-16T15:27:28.886948+0000","last_fullsized":"2024-01-16T15:27:28.886948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'74","last_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_deep_scrub":"115'74","last_deep_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_clean_scrub_stamp":"2024-01-16T15:27:28.886880+0000","objects_scrubbed":19,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:33.087313+0000","scrub_duration":0.202995169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'49","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733966+0000","last_change":"2024-01-16T15:18:50.514080+0000","last_active":"2024-01-16T15:27:17.733966+0000","last_peered":"2024-01-16T15:27:17.733966+0000","last_clean":"2024-01-16T15:27:17.733966+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:17.733966+0000","last_undegraded":"2024-01-16T15:27:17.733966+0000","last_fullsized":"2024-01-16T15:27:17.733966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:18.432791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'72","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735139+0000","last_change":"2024-01-16T15:18:50.512290+0000","last_active":"2024-01-16T15:27:17.735139+0000","last_peered":"2024-01-16T15:27:17.735139+0000","last_clean":"2024-01-16T15:27:17.735139+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:17.735139+0000","last_undegraded":"2024-01-16T15:27:17.735139+0000","last_fullsized":"2024-01-16T15:27:17.735139+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:03.196990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'71","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741586+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:27:17.741586+0000","last_peered":"2024-01-16T15:27:17.741586+0000","last_clean":"2024-01-16T15:27:17.741586+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:27:17.741586+0000","last_undegraded":"2024-01-16T15:27:17.741586+0000","last_fullsized":"2024-01-16T15:27:17.741586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.753500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'67","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734862+0000","last_change":"2024-01-16T15:18:50.523238+0000","last_active":"2024-01-16T15:27:17.734862+0000","last_peered":"2024-01-16T15:27:17.734862+0000","last_clean":"2024-01-16T15:27:17.734862+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:17.734862+0000","last_undegraded":"2024-01-16T15:27:17.734862+0000","last_fullsized":"2024-01-16T15:27:17.734862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:10.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'73","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742649+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:27:17.742649+0000","last_peered":"2024-01-16T15:27:17.742649+0000","last_clean":"2024-01-16T15:27:17.742649+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:27:17.742649+0000","last_undegraded":"2024-01-16T15:27:17.742649+0000","last_fullsized":"2024-01-16T15:27:17.742649+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:04.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'64","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720826+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:27:17.720826+0000","last_peered":"2024-01-16T15:27:17.720826+0000","last_clean":"2024-01-16T15:27:17.720826+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:27:17.720826+0000","last_undegraded":"2024-01-16T15:27:17.720826+0000","last_fullsized":"2024-01-16T15:27:17.720826+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'57","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739771+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:27:17.739771+0000","last_peered":"2024-01-16T15:27:17.739771+0000","last_clean":"2024-01-16T15:27:17.739771+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:17.739771+0000","last_undegraded":"2024-01-16T15:27:17.739771+0000","last_fullsized":"2024-01-16T15:27:17.739771+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:44:49.070110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'70","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138498+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:27:30.138498+0000","last_peered":"2024-01-16T15:27:30.138498+0000","last_clean":"2024-01-16T15:27:30.138498+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:27:30.138498+0000","last_undegraded":"2024-01-16T15:27:30.138498+0000","last_fullsized":"2024-01-16T15:27:30.138498+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'71","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740913+0000","last_change":"2024-01-16T15:18:50.513030+0000","last_active":"2024-01-16T15:27:17.740913+0000","last_peered":"2024-01-16T15:27:17.740913+0000","last_clean":"2024-01-16T15:27:17.740913+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:17.740913+0000","last_undegraded":"2024-01-16T15:27:17.740913+0000","last_fullsized":"2024-01-16T15:27:17.740913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:54.334893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'81","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:27.943347+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:27:27.943347+0000","last_peered":"2024-01-16T15:27:27.943347+0000","last_clean":"2024-01-16T15:27:27.943347+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:27:27.943347+0000","last_undegraded":"2024-01-16T15:27:27.943347+0000","last_fullsized":"2024-01-16T15:27:27.943347+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'64","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.665594+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:27:28.665594+0000","last_peered":"2024-01-16T15:27:28.665594+0000","last_clean":"2024-01-16T15:27:28.665594+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:27:28.665594+0000","last_undegraded":"2024-01-16T15:27:28.665594+0000","last_fullsized":"2024-01-16T15:27:28.665594+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":66,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138416+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:30.138416+0000","last_peered":"2024-01-16T15:27:30.138416+0000","last_clean":"2024-01-16T15:27:30.138416+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:30.138416+0000","last_undegraded":"2024-01-16T15:27:30.138416+0000","last_fullsized":"2024-01-16T15:27:30.138416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'57","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738407+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:27:17.738407+0000","last_peered":"2024-01-16T15:27:17.738407+0000","last_clean":"2024-01-16T15:27:17.738407+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:27:17.738407+0000","last_undegraded":"2024-01-16T15:27:17.738407+0000","last_fullsized":"2024-01-16T15:27:17.738407+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:42:38.948857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'76","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720265+0000","last_change":"2024-01-16T15:18:50.491931+0000","last_active":"2024-01-16T15:27:17.720265+0000","last_peered":"2024-01-16T15:27:17.720265+0000","last_clean":"2024-01-16T15:27:17.720265+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:17.720265+0000","last_undegraded":"2024-01-16T15:27:17.720265+0000","last_fullsized":"2024-01-16T15:27:17.720265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:53.664054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'81","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736883+0000","last_change":"2024-01-16T15:18:50.508147+0000","last_active":"2024-01-16T15:27:17.736883+0000","last_peered":"2024-01-16T15:27:17.736883+0000","last_clean":"2024-01-16T15:27:17.736883+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:17.736883+0000","last_undegraded":"2024-01-16T15:27:17.736883+0000","last_fullsized":"2024-01-16T15:27:17.736883+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'63","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724101+0000","last_change":"2024-01-16T15:18:50.510830+0000","last_active":"2024-01-16T15:27:17.724101+0000","last_peered":"2024-01-16T15:27:17.724101+0000","last_clean":"2024-01-16T15:27:17.724101+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:17.724101+0000","last_undegraded":"2024-01-16T15:27:17.724101+0000","last_fullsized":"2024-01-16T15:27:17.724101+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'62","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722024+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:27:17.722024+0000","last_peered":"2024-01-16T15:27:17.722024+0000","last_clean":"2024-01-16T15:27:17.722024+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:17.722024+0000","last_undegraded":"2024-01-16T15:27:17.722024+0000","last_fullsized":"2024-01-16T15:27:17.722024+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476688+0000","last_change":"2024-01-16T15:18:49.514145+0000","last_active":"2024-01-16T15:25:05.476688+0000","last_peered":"2024-01-16T15:25:05.476688+0000","last_clean":"2024-01-16T15:25:05.476688+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:25:05.476688+0000","last_undegraded":"2024-01-16T15:25:05.476688+0000","last_fullsized":"2024-01-16T15:25:05.476688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.452091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'54","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733008+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:27:17.733008+0000","last_peered":"2024-01-16T15:27:17.733008+0000","last_clean":"2024-01-16T15:27:17.733008+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:27:17.733008+0000","last_undegraded":"2024-01-16T15:27:17.733008+0000","last_fullsized":"2024-01-16T15:27:17.733008+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:37.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:24:50.466688+0000","last_change":"2024-01-16T15:18:49.521614+0000","last_active":"2024-01-16T15:24:50.466688+0000","last_peered":"2024-01-16T15:24:50.466688+0000","last_clean":"2024-01-16T15:24:50.466688+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:24:50.466688+0000","last_undegraded":"2024-01-16T15:24:50.466688+0000","last_fullsized":"2024-01-16T15:24:50.466688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:51.215798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'71","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724966+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:27:17.724966+0000","last_peered":"2024-01-16T15:27:17.724966+0000","last_clean":"2024-01-16T15:27:17.724966+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:17.724966+0000","last_undegraded":"2024-01-16T15:27:17.724966+0000","last_fullsized":"2024-01-16T15:27:17.724966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'3","reported_seq":21,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:20.387534+0000","last_change":"2024-01-16T15:18:49.518828+0000","last_active":"2024-01-16T15:26:20.387534+0000","last_peered":"2024-01-16T15:26:20.387534+0000","last_clean":"2024-01-16T15:26:20.387534+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:26:20.387534+0000","last_undegraded":"2024-01-16T15:26:20.387534+0000","last_fullsized":"2024-01-16T15:26:20.387534+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'68","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741745+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:27:17.741745+0000","last_peered":"2024-01-16T15:27:17.741745+0000","last_clean":"2024-01-16T15:27:17.741745+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:27:17.741745+0000","last_undegraded":"2024-01-16T15:27:17.741745+0000","last_fullsized":"2024-01-16T15:27:17.741745+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:37.406427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:20.387496+0000","last_change":"2024-01-16T15:18:49.519705+0000","last_active":"2024-01-16T15:26:20.387496+0000","last_peered":"2024-01-16T15:26:20.387496+0000","last_clean":"2024-01-16T15:26:20.387496+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:26:20.387496+0000","last_undegraded":"2024-01-16T15:26:20.387496+0000","last_fullsized":"2024-01-16T15:26:20.387496+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'72","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.719815+0000","last_change":"2024-01-16T15:19:15.318178+0000","last_active":"2024-01-16T15:27:17.719815+0000","last_peered":"2024-01-16T15:27:17.719815+0000","last_clean":"2024-01-16T15:27:17.719815+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:17.719815+0000","last_undegraded":"2024-01-16T15:27:17.719815+0000","last_fullsized":"2024-01-16T15:27:17.719815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:50.463254+0000","last_change":"2024-01-16T15:18:49.523104+0000","last_active":"2024-01-16T15:23:50.463254+0000","last_peered":"2024-01-16T15:23:50.463254+0000","last_clean":"2024-01-16T15:23:50.463254+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:23:50.463254+0000","last_undegraded":"2024-01-16T15:23:50.463254+0000","last_fullsized":"2024-01-16T15:23:50.463254+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:31.593211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'72","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736806+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:27:17.736806+0000","last_peered":"2024-01-16T15:27:17.736806+0000","last_clean":"2024-01-16T15:27:17.736806+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:27:17.736806+0000","last_undegraded":"2024-01-16T15:27:17.736806+0000","last_fullsized":"2024-01-16T15:27:17.736806+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:59.415403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:20.431895+0000","last_change":"2024-01-16T15:18:49.517722+0000","last_active":"2024-01-16T15:26:20.431895+0000","last_peered":"2024-01-16T15:26:20.431895+0000","last_clean":"2024-01-16T15:26:20.431895+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:26:20.431895+0000","last_undegraded":"2024-01-16T15:26:20.431895+0000","last_fullsized":"2024-01-16T15:26:20.431895+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'95","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.871075+0000","last_change":"2024-01-16T15:27:29.871123+0000","last_active":"2024-01-16T15:27:29.871075+0000","last_peered":"2024-01-16T15:27:29.871075+0000","last_clean":"2024-01-16T15:27:29.871075+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:29.871075+0000","last_undegraded":"2024-01-16T15:27:29.871075+0000","last_fullsized":"2024-01-16T15:27:29.871075+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.871010+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:25.437998+0000","scrub_duration":0.15415410600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:40.368270+0000","last_change":"2024-01-16T15:18:49.509092+0000","last_active":"2024-01-16T15:23:40.368270+0000","last_peered":"2024-01-16T15:23:40.368270+0000","last_clean":"2024-01-16T15:23:40.368270+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:23:40.368270+0000","last_undegraded":"2024-01-16T15:23:40.368270+0000","last_fullsized":"2024-01-16T15:23:40.368270+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:07.440889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'73","reported_seq":99,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726884+0000","last_change":"2024-01-16T15:18:50.506340+0000","last_active":"2024-01-16T15:27:17.726884+0000","last_peered":"2024-01-16T15:27:17.726884+0000","last_clean":"2024-01-16T15:27:17.726884+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:17.726884+0000","last_undegraded":"2024-01-16T15:27:17.726884+0000","last_fullsized":"2024-01-16T15:27:17.726884+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:36:59.527778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:05.653043+0000","last_change":"2024-01-16T15:18:49.510604+0000","last_active":"2024-01-16T15:26:05.653043+0000","last_peered":"2024-01-16T15:26:05.653043+0000","last_clean":"2024-01-16T15:26:05.653043+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:26:05.653043+0000","last_undegraded":"2024-01-16T15:26:05.653043+0000","last_fullsized":"2024-01-16T15:26:05.653043+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'58","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722570+0000","last_change":"2024-01-16T15:18:50.507182+0000","last_active":"2024-01-16T15:27:17.722570+0000","last_peered":"2024-01-16T15:27:17.722570+0000","last_clean":"2024-01-16T15:27:17.722570+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:17.722570+0000","last_undegraded":"2024-01-16T15:27:17.722570+0000","last_fullsized":"2024-01-16T15:27:17.722570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:02.386612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:05.384938+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:26:05.384938+0000","last_peered":"2024-01-16T15:26:05.384938+0000","last_clean":"2024-01-16T15:26:05.384938+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:26:05.384938+0000","last_undegraded":"2024-01-16T15:26:05.384938+0000","last_fullsized":"2024-01-16T15:26:05.384938+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'61","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738962+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:27:17.738962+0000","last_peered":"2024-01-16T15:27:17.738962+0000","last_clean":"2024-01-16T15:27:17.738962+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:17.738962+0000","last_undegraded":"2024-01-16T15:27:17.738962+0000","last_fullsized":"2024-01-16T15:27:17.738962+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:39:14.106849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:05.429293+0000","last_change":"2024-01-16T15:18:49.514792+0000","last_active":"2024-01-16T15:26:05.429293+0000","last_peered":"2024-01-16T15:26:05.429293+0000","last_clean":"2024-01-16T15:26:05.429293+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:26:05.429293+0000","last_undegraded":"2024-01-16T15:26:05.429293+0000","last_fullsized":"2024-01-16T15:26:05.429293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726408+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:17.726408+0000","last_peered":"2024-01-16T15:27:17.726408+0000","last_clean":"2024-01-16T15:27:17.726408+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:17.726408+0000","last_undegraded":"2024-01-16T15:27:17.726408+0000","last_fullsized":"2024-01-16T15:27:17.726408+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'33","reported_seq":51,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:05.653340+0000","last_change":"2024-01-16T15:18:49.523404+0000","last_active":"2024-01-16T15:26:05.653340+0000","last_peered":"2024-01-16T15:26:05.653340+0000","last_clean":"2024-01-16T15:26:05.653340+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:26:05.653340+0000","last_undegraded":"2024-01-16T15:26:05.653340+0000","last_fullsized":"2024-01-16T15:26:05.653340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'69","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.729047+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:27:17.729047+0000","last_peered":"2024-01-16T15:27:17.729047+0000","last_clean":"2024-01-16T15:27:17.729047+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:17.729047+0000","last_undegraded":"2024-01-16T15:27:17.729047+0000","last_fullsized":"2024-01-16T15:27:17.729047+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:37.614099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'322","reported_seq":340,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:23.260841+0000","last_change":"2024-01-16T15:18:49.510728+0000","last_active":"2024-01-16T15:27:23.260841+0000","last_peered":"2024-01-16T15:27:23.260841+0000","last_clean":"2024-01-16T15:27:23.260841+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:23.260841+0000","last_undegraded":"2024-01-16T15:27:23.260841+0000","last_fullsized":"2024-01-16T15:27:23.260841+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1246392,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.718779+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:17.718779+0000","last_peered":"2024-01-16T15:27:17.718779+0000","last_clean":"2024-01-16T15:27:17.718779+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:17.718779+0000","last_undegraded":"2024-01-16T15:27:17.718779+0000","last_fullsized":"2024-01-16T15:27:17.718779+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.526241+0000","last_change":"2024-01-16T15:18:49.518745+0000","last_active":"2024-01-16T15:25:50.526241+0000","last_peered":"2024-01-16T15:25:50.526241+0000","last_clean":"2024-01-16T15:25:50.526241+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:25:50.526241+0000","last_undegraded":"2024-01-16T15:25:50.526241+0000","last_fullsized":"2024-01-16T15:25:50.526241+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'73","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.730632+0000","last_change":"2024-01-16T15:18:50.517675+0000","last_active":"2024-01-16T15:27:17.730632+0000","last_peered":"2024-01-16T15:27:17.730632+0000","last_clean":"2024-01-16T15:27:17.730632+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:17.730632+0000","last_undegraded":"2024-01-16T15:27:17.730632+0000","last_fullsized":"2024-01-16T15:27:17.730632+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.440394+0000","last_change":"2024-01-16T15:18:49.513639+0000","last_active":"2024-01-16T15:23:30.440394+0000","last_peered":"2024-01-16T15:23:30.440394+0000","last_clean":"2024-01-16T15:23:30.440394+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:23:30.440394+0000","last_undegraded":"2024-01-16T15:23:30.440394+0000","last_fullsized":"2024-01-16T15:23:30.440394+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:28.315056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'63","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740857+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:27:17.740857+0000","last_peered":"2024-01-16T15:27:17.740857+0000","last_clean":"2024-01-16T15:27:17.740857+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:27:17.740857+0000","last_undegraded":"2024-01-16T15:27:17.740857+0000","last_fullsized":"2024-01-16T15:27:17.740857+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.382158+0000","last_change":"2024-01-16T15:18:49.509306+0000","last_active":"2024-01-16T15:25:50.382158+0000","last_peered":"2024-01-16T15:25:50.382158+0000","last_clean":"2024-01-16T15:25:50.382158+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:25:50.382158+0000","last_undegraded":"2024-01-16T15:25:50.382158+0000","last_fullsized":"2024-01-16T15:25:50.382158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'46","reported_seq":69,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735205+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:27:17.735205+0000","last_peered":"2024-01-16T15:27:17.735205+0000","last_clean":"2024-01-16T15:27:17.735205+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:27:17.735205+0000","last_undegraded":"2024-01-16T15:27:17.735205+0000","last_fullsized":"2024-01-16T15:27:17.735205+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:35.647759+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:25:35.647759+0000","last_peered":"2024-01-16T15:25:35.647759+0000","last_clean":"2024-01-16T15:25:35.647759+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:25:35.647759+0000","last_undegraded":"2024-01-16T15:25:35.647759+0000","last_fullsized":"2024-01-16T15:25:35.647759+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734570+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:17.734570+0000","last_peered":"2024-01-16T15:27:17.734570+0000","last_clean":"2024-01-16T15:27:17.734570+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:17.734570+0000","last_undegraded":"2024-01-16T15:27:17.734570+0000","last_fullsized":"2024-01-16T15:27:17.734570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:35.379487+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:25:35.379487+0000","last_peered":"2024-01-16T15:25:35.379487+0000","last_clean":"2024-01-16T15:25:35.379487+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:25:35.379487+0000","last_undegraded":"2024-01-16T15:25:35.379487+0000","last_fullsized":"2024-01-16T15:25:35.379487+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'61","reported_seq":91,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738127+0000","last_change":"2024-01-16T15:18:50.517233+0000","last_active":"2024-01-16T15:27:17.738127+0000","last_peered":"2024-01-16T15:27:17.738127+0000","last_clean":"2024-01-16T15:27:17.738127+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:17.738127+0000","last_undegraded":"2024-01-16T15:27:17.738127+0000","last_fullsized":"2024-01-16T15:27:17.738127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:02.034181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.438493+0000","last_change":"2024-01-16T15:18:49.514704+0000","last_active":"2024-01-16T15:23:20.438493+0000","last_peered":"2024-01-16T15:23:20.438493+0000","last_clean":"2024-01-16T15:23:20.438493+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:23:20.438493+0000","last_undegraded":"2024-01-16T15:23:20.438493+0000","last_fullsized":"2024-01-16T15:23:20.438493+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:04.731539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744447+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:27:17.744447+0000","last_peered":"2024-01-16T15:27:17.744447+0000","last_clean":"2024-01-16T15:27:17.744447+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:27:17.744447+0000","last_undegraded":"2024-01-16T15:27:17.744447+0000","last_fullsized":"2024-01-16T15:27:17.744447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:16.245190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:20.457791+0000","last_change":"2024-01-16T15:18:49.514883+0000","last_active":"2024-01-16T15:23:20.457791+0000","last_peered":"2024-01-16T15:23:20.457791+0000","last_clean":"2024-01-16T15:23:20.457791+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:23:20.457791+0000","last_undegraded":"2024-01-16T15:23:20.457791+0000","last_fullsized":"2024-01-16T15:23:20.457791+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:18.655932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'56","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741903+0000","last_change":"2024-01-16T15:18:50.521667+0000","last_active":"2024-01-16T15:27:17.741903+0000","last_peered":"2024-01-16T15:27:17.741903+0000","last_clean":"2024-01-16T15:27:17.741903+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:17.741903+0000","last_undegraded":"2024-01-16T15:27:17.741903+0000","last_fullsized":"2024-01-16T15:27:17.741903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:41:57.970594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'3","reported_seq":21,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:35.523398+0000","last_change":"2024-01-16T15:18:49.520558+0000","last_active":"2024-01-16T15:25:35.523398+0000","last_peered":"2024-01-16T15:25:35.523398+0000","last_clean":"2024-01-16T15:25:35.523398+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:25:35.523398+0000","last_undegraded":"2024-01-16T15:25:35.523398+0000","last_fullsized":"2024-01-16T15:25:35.523398+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738842+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:27:17.738842+0000","last_peered":"2024-01-16T15:27:17.738842+0000","last_clean":"2024-01-16T15:27:17.738842+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:27:17.738842+0000","last_undegraded":"2024-01-16T15:27:17.738842+0000","last_fullsized":"2024-01-16T15:27:17.738842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.373023+0000","last_change":"2024-01-16T15:18:49.517469+0000","last_active":"2024-01-16T15:27:29.373023+0000","last_peered":"2024-01-16T15:27:29.373023+0000","last_clean":"2024-01-16T15:27:29.373023+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:27:29.373023+0000","last_undegraded":"2024-01-16T15:27:29.373023+0000","last_fullsized":"2024-01-16T15:27:29.373023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'65","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735944+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:27:17.735944+0000","last_peered":"2024-01-16T15:27:17.735944+0000","last_clean":"2024-01-16T15:27:17.735944+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:27:17.735944+0000","last_undegraded":"2024-01-16T15:27:17.735944+0000","last_fullsized":"2024-01-16T15:27:17.735944+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:23.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.193731+0000","last_change":"2024-01-16T15:27:29.193731+0000","last_active":"2024-01-16T15:27:29.193731+0000","last_peered":"2024-01-16T15:27:29.193731+0000","last_clean":"2024-01-16T15:27:29.193731+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:27:29.193731+0000","last_undegraded":"2024-01-16T15:27:29.193731+0000","last_fullsized":"2024-01-16T15:27:29.193731+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.193592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:03.864449+0000","scrub_duration":0.0028384030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'57","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138198+0000","last_change":"2024-01-16T15:18:50.516592+0000","last_active":"2024-01-16T15:27:30.138198+0000","last_peered":"2024-01-16T15:27:30.138198+0000","last_clean":"2024-01-16T15:27:30.138198+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:30.138198+0000","last_undegraded":"2024-01-16T15:27:30.138198+0000","last_fullsized":"2024-01-16T15:27:30.138198+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:27.942906+0000","last_change":"2024-01-16T15:18:49.520798+0000","last_active":"2024-01-16T15:27:27.942906+0000","last_peered":"2024-01-16T15:27:27.942906+0000","last_clean":"2024-01-16T15:27:27.942906+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:27:27.942906+0000","last_undegraded":"2024-01-16T15:27:27.942906+0000","last_fullsized":"2024-01-16T15:27:27.942906+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'53","reported_seq":76,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740082+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:27:17.740082+0000","last_peered":"2024-01-16T15:27:17.740082+0000","last_clean":"2024-01-16T15:27:17.740082+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:17.740082+0000","last_undegraded":"2024-01-16T15:27:17.740082+0000","last_fullsized":"2024-01-16T15:27:17.740082+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:57:09.913343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'3","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138082+0000","last_change":"2024-01-16T15:18:49.520012+0000","last_active":"2024-01-16T15:27:30.138082+0000","last_peered":"2024-01-16T15:27:30.138082+0000","last_clean":"2024-01-16T15:27:30.138082+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:27:30.138082+0000","last_undegraded":"2024-01-16T15:27:30.138082+0000","last_fullsized":"2024-01-16T15:27:30.138082+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741585+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:17.741585+0000","last_peered":"2024-01-16T15:27:17.741585+0000","last_clean":"2024-01-16T15:27:17.741585+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:17.741585+0000","last_undegraded":"2024-01-16T15:27:17.741585+0000","last_fullsized":"2024-01-16T15:27:17.741585+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'31","reported_seq":49,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:24.174961+0000","last_change":"2024-01-16T15:18:49.513987+0000","last_active":"2024-01-16T15:27:24.174961+0000","last_peered":"2024-01-16T15:27:24.174961+0000","last_clean":"2024-01-16T15:27:24.174961+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:24.174961+0000","last_undegraded":"2024-01-16T15:27:24.174961+0000","last_fullsized":"2024-01-16T15:27:24.174961+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:07.461774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'69","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736049+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:27:17.736049+0000","last_peered":"2024-01-16T15:27:17.736049+0000","last_clean":"2024-01-16T15:27:17.736049+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:27:17.736049+0000","last_undegraded":"2024-01-16T15:27:17.736049+0000","last_fullsized":"2024-01-16T15:27:17.736049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:10.684206+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:27:10.684206+0000","last_peered":"2024-01-16T15:27:10.684206+0000","last_clean":"2024-01-16T15:27:10.684206+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:27:10.684206+0000","last_undegraded":"2024-01-16T15:27:10.684206+0000","last_fullsized":"2024-01-16T15:27:10.684206+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'59","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738409+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:27:17.738409+0000","last_peered":"2024-01-16T15:27:17.738409+0000","last_clean":"2024-01-16T15:27:17.738409+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:27:17.738409+0000","last_undegraded":"2024-01-16T15:27:17.738409+0000","last_fullsized":"2024-01-16T15:27:17.738409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:27.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:10.441890+0000","last_change":"2024-01-16T15:18:49.519539+0000","last_active":"2024-01-16T15:27:10.441890+0000","last_peered":"2024-01-16T15:27:10.441890+0000","last_clean":"2024-01-16T15:27:10.441890+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:27:10.441890+0000","last_undegraded":"2024-01-16T15:27:10.441890+0000","last_fullsized":"2024-01-16T15:27:10.441890+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'63","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736683+0000","last_change":"2024-01-16T15:18:50.516022+0000","last_active":"2024-01-16T15:27:17.736683+0000","last_peered":"2024-01-16T15:27:17.736683+0000","last_clean":"2024-01-16T15:27:17.736683+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:17.736683+0000","last_undegraded":"2024-01-16T15:27:17.736683+0000","last_fullsized":"2024-01-16T15:27:17.736683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:18.152456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.476614+0000","last_change":"2024-01-16T15:18:49.508821+0000","last_active":"2024-01-16T15:25:05.476614+0000","last_peered":"2024-01-16T15:25:05.476614+0000","last_clean":"2024-01-16T15:25:05.476614+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:25:05.476614+0000","last_undegraded":"2024-01-16T15:25:05.476614+0000","last_fullsized":"2024-01-16T15:25:05.476614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:39.842102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742798+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:27:17.742798+0000","last_peered":"2024-01-16T15:27:17.742798+0000","last_clean":"2024-01-16T15:27:17.742798+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:27:17.742798+0000","last_undegraded":"2024-01-16T15:27:17.742798+0000","last_fullsized":"2024-01-16T15:27:17.742798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:03.949006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:05.384697+0000","last_change":"2024-01-16T15:18:49.518898+0000","last_active":"2024-01-16T15:25:05.384697+0000","last_peered":"2024-01-16T15:25:05.384697+0000","last_clean":"2024-01-16T15:25:05.384697+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:25:05.384697+0000","last_undegraded":"2024-01-16T15:25:05.384697+0000","last_fullsized":"2024-01-16T15:25:05.384697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:26.402495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'68","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738935+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:27:17.738935+0000","last_peered":"2024-01-16T15:27:17.738935+0000","last_clean":"2024-01-16T15:27:17.738935+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:27:17.738935+0000","last_undegraded":"2024-01-16T15:27:17.738935+0000","last_fullsized":"2024-01-16T15:27:17.738935+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:42.876671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'34","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:10.396443+0000","last_change":"2024-01-16T15:18:49.515802+0000","last_active":"2024-01-16T15:27:10.396443+0000","last_peered":"2024-01-16T15:27:10.396443+0000","last_clean":"2024-01-16T15:27:10.396443+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:27:10.396443+0000","last_undegraded":"2024-01-16T15:27:10.396443+0000","last_fullsized":"2024-01-16T15:27:10.396443+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735204+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:27:17.735204+0000","last_peered":"2024-01-16T15:27:17.735204+0000","last_clean":"2024-01-16T15:27:17.735204+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:27:17.735204+0000","last_undegraded":"2024-01-16T15:27:17.735204+0000","last_fullsized":"2024-01-16T15:27:17.735204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:50.679483+0000","last_change":"2024-01-16T15:18:49.518460+0000","last_active":"2024-01-16T15:26:50.679483+0000","last_peered":"2024-01-16T15:26:50.679483+0000","last_clean":"2024-01-16T15:26:50.679483+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:26:50.679483+0000","last_undegraded":"2024-01-16T15:26:50.679483+0000","last_fullsized":"2024-01-16T15:26:50.679483+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'49","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.716652+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:27:17.716652+0000","last_peered":"2024-01-16T15:27:17.716652+0000","last_clean":"2024-01-16T15:27:17.716652+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:17.716652+0000","last_undegraded":"2024-01-16T15:27:17.716652+0000","last_fullsized":"2024-01-16T15:27:17.716652+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:23:30.459727+0000","last_change":"2024-01-16T15:18:49.521592+0000","last_active":"2024-01-16T15:23:30.459727+0000","last_peered":"2024-01-16T15:23:30.459727+0000","last_clean":"2024-01-16T15:23:30.459727+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:23:30.459727+0000","last_undegraded":"2024-01-16T15:23:30.459727+0000","last_fullsized":"2024-01-16T15:23:30.459727+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:09:24.899317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":253,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.473268+0000","last_change":"2024-01-16T14:59:22.546283+0000","last_active":"2024-01-16T15:25:25.473268+0000","last_peered":"2024-01-16T15:25:25.473268+0000","last_clean":"2024-01-16T15:25:25.473268+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:25:25.473268+0000","last_undegraded":"2024-01-16T15:25:25.473268+0000","last_fullsized":"2024-01-16T15:25:25.473268+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:21.360843+0000","last_clean_scrub_stamp":"2024-01-16T14:59:21.360843+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:42.320803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4152,"ondisk_log_size":4152,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1251137,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":490,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2502274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.222,"15min":1.274},"min":{"1min":0.64800000000000002,"5min":0.54300000000000004,"15min":0.28599999999999998},"max":{"1min":1.4159999999999999,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.302,"15min":1.4399999999999999},"min":{"1min":0.75900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.712,"5min":1.8959999999999999,"15min":25.503},"last":1.5620000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.34499999999999997},"max":{"1min":1.54,"5min":1.5620000000000001,"15min":5.5010000000000003},"last":1.002},{"interface":"front","average":{"1min":1.028,"5min":1.1619999999999999,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4139999999999999,"5min":1.9490000000000001,"15min":4.3369999999999997},"last":1.2549999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.68400000000000005,"15min":0.629},"max":{"1min":1.669,"5min":2.141,"15min":11.536},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.4670000000000001},"min":{"1min":0.73099999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5309999999999999,"5min":1.8500000000000001,"15min":25.478000000000002},"last":1.3180000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.40899999999999997},"max":{"1min":1.4870000000000001,"5min":1.7090000000000001,"15min":5.6369999999999996},"last":1.22},{"interface":"front","average":{"1min":1.266,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":0.80200000000000005,"5min":0.75900000000000001,"15min":0.67800000000000005},"max":{"1min":1.5589999999999999,"5min":1.9550000000000001,"15min":6.351},"last":1.5149999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.204,"15min":1.3919999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6120000000000001,"5min":1.75,"15min":25.401},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6399999999999999,"5min":1.8160000000000001,"15min":5.7859999999999996},"last":1.6060000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3540000000000001,"15min":1.359},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.722,"5min":2.3849999999999998,"15min":4.4539999999999997},"last":1.698},{"interface":"front","average":{"1min":1.385,"5min":1.383,"15min":1.5189999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.708,"5min":1.8480000000000001,"15min":26.327000000000002},"last":1.472}]},{"osd":6,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5469999999999999,"15min":1.6739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8280000000000001,"5min":2.0579999999999998,"15min":25.451000000000001},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5029999999999999,"15min":1.496},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.786,"5min":2.786,"15min":5.8250000000000002},"last":1.4159999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.68400000000000005,"15min":0.35399999999999998},"max":{"1min":1.577,"5min":8.0299999999999994,"15min":29.77},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3220000000000001,"15min":1.415},"min":{"1min":0.88500000000000001,"5min":0.76400000000000001,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.194}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.081,"15min":1.1339999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":1.728,"15min":5.2859999999999996},"last":1.282},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.57099999999999995},"max":{"1min":1.595,"5min":1.7350000000000001,"15min":10.494},"last":0.69599999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.202,"15min":1.278},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.46,"5min":1.6579999999999999,"15min":5.0549999999999997},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.837,"5min":1.9279999999999999,"15min":5.1459999999999999},"last":1.484}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.319,"15min":1.3089999999999999},"min":{"1min":0.90200000000000002,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":1.601,"5min":1.8360000000000001,"15min":3.8079999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.3129999999999999,"15min":1.448},"min":{"1min":0.87,"5min":0.65200000000000002,"15min":0.623},"max":{"1min":1.7130000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.3360000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3839999999999999,"15min":1.3480000000000001},"min":{"1min":0.97399999999999998,"5min":0.87,"15min":0.53400000000000003},"max":{"1min":1.6890000000000001,"5min":20.422999999999998,"15min":20.422999999999998},"last":1.056},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.258,"15min":1.28},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":3.6160000000000001},"last":1.111}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.272,"15min":1.321},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.53700000000000003},"max":{"1min":1.542,"5min":7.6319999999999997,"15min":13.949999999999999},"last":0.995},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.581,"15min":1.5720000000000001},"min":{"1min":1.113,"5min":1.113,"15min":0.51600000000000001},"max":{"1min":1.972,"5min":7.4980000000000002,"15min":11.298},"last":1.3899999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4410000000000001,"15min":1.498},"min":{"1min":1.0169999999999999,"5min":0.97499999999999998,"15min":0.57099999999999995},"max":{"1min":1.659,"5min":2.004,"15min":14.037000000000001},"last":1.458},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.512,"15min":1.5720000000000001},"min":{"1min":1.1100000000000001,"5min":1.024,"15min":0.61599999999999999},"max":{"1min":1.9370000000000001,"5min":2.0209999999999999,"15min":4.9829999999999997},"last":1.427}]}]},{"osd":5,"up_from":14,"seq":60129542485,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141700,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93230140,"statfs":{"total":96636764160,"available":95467663360,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2090000000000001,"15min":1.1379999999999999},"min":{"1min":0.79300000000000004,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.6890000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.163,"15min":1.147},"min":{"1min":0.89900000000000002,"5min":0.72599999999999998,"15min":0.34799999999999998},"max":{"1min":1.526,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.1120000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1850000000000001,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":1.776,"15min":17.074999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.228,"15min":1.244},"min":{"1min":0.89000000000000001,"5min":0.77100000000000002,"15min":0.46999999999999997},"max":{"1min":1.714,"5min":1.714,"15min":4.8499999999999996},"last":0.88300000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.222,"15min":1.198},"min":{"1min":0.83899999999999997,"5min":0.72599999999999998,"15min":0.54800000000000004},"max":{"1min":1.53,"5min":2.7000000000000002,"15min":4.306},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2849999999999999,"15min":1.323},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.52200000000000002},"max":{"1min":1.494,"5min":2.9670000000000001,"15min":16.721},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.97499999999999998,"5min":0.81499999999999995,"15min":0.60299999999999998},"max":{"1min":1.859,"5min":1.905,"15min":4.8010000000000002},"last":1.165},{"interface":"front","average":{"1min":1.427,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":0.51200000000000001},"max":{"1min":1.7390000000000001,"5min":14.044,"15min":14.044},"last":1.3859999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2749999999999999,"15min":1.3260000000000001},"min":{"1min":0.90400000000000003,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.591,"5min":2.0139999999999998,"15min":16.382999999999999},"last":1.212},{"interface":"front","average":{"1min":1.294,"5min":1.321,"15min":1.341},"min":{"1min":0.81000000000000005,"5min":0.72099999999999997,"15min":0.502},"max":{"1min":1.982,"5min":1.982,"15min":4.6580000000000004},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3500000000000001,"15min":1.357},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.39700000000000002},"max":{"1min":1.637,"5min":1.901,"15min":4.3380000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.516,"15min":1.5089999999999999},"min":{"1min":1.137,"5min":0.94199999999999995,"15min":0.57899999999999996},"max":{"1min":1.6919999999999999,"5min":1.907,"15min":3.3719999999999999},"last":1.5409999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5,"15min":1.5029999999999999},"min":{"1min":0.95999999999999996,"5min":0.95599999999999996,"15min":0.45000000000000001},"max":{"1min":2.044,"5min":2.044,"15min":4.742},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5700000000000001,"15min":1.5509999999999999},"min":{"1min":1.0820000000000001,"5min":1.0629999999999999,"15min":0.58799999999999997},"max":{"1min":1.821,"5min":2.0609999999999999,"15min":2.6520000000000001},"last":1.476}]}]},{"osd":4,"up_from":14,"seq":60129542484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1932292,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92439548,"statfs":{"total":96636764160,"available":94658097152,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":1.2350000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":1.4910000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.3640000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.321},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.425}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":1.778},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.6419999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.196},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.6719999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.744},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.5900000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542486,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872984,"kb_used_data":1751632,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498856,"statfs":{"total":96636764160,"available":94718828544,"internally_reserved":0,"allocated":1793671168,"data_stored":1792833889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.68899999999999995,"5min":0.58499999999999996,"15min":0.34699999999999998},"max":{"1min":1.3640000000000001,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.075}]},{"osd":1,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.98199999999999998,"15min":1.0249999999999999},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":1.359,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2569999999999999,"15min":1.226},"min":{"1min":0.64000000000000001,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.7909999999999999,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":1.147}]},{"osd":2,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.244,"15min":1.26},"min":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.36699999999999999},"max":{"1min":1.577,"5min":2.258,"15min":3.5249999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.45200000000000001},"max":{"1min":1.897,"5min":2.339,"15min":4.6529999999999996},"last":1.292}]},{"osd":4,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.323,"15min":1.319},"min":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":0.44500000000000001},"max":{"1min":1.6759999999999999,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.462,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.73299999999999998},"max":{"1min":1.861,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.333}]},{"osd":5,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4890000000000001,"15min":1.5009999999999999},"min":{"1min":1.224,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":1.738,"5min":2.0249999999999999,"15min":4.484},"last":1.581},{"interface":"front","average":{"1min":1.419,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.66500000000000004},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.2589999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5389999999999999,"15min":1.5620000000000001},"min":{"1min":1.1220000000000001,"5min":0.92700000000000005,"15min":0.71499999999999997},"max":{"1min":1.96,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6080000000000001,"15min":1.613},"min":{"1min":1.1399999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.069,"5min":2.169,"15min":4.5880000000000001},"last":1.47}]},{"osd":7,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5329999999999999,"15min":1.5509999999999999},"min":{"1min":0.98799999999999999,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":1.8480000000000001,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.6299999999999999,"15min":1.6279999999999999},"min":{"1min":1.2669999999999999,"5min":1.008,"15min":0.64700000000000002},"max":{"1min":2.1160000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.6220000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542484,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251776,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":115774,"kb_avail":93120064,"statfs":{"total":96636764160,"available":95354945536,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118553029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86799999999999999,"15min":0.89800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36099999999999999},"max":{"1min":1.2649999999999999,"5min":1.6659999999999999,"15min":1.96},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0049999999999999},"min":{"1min":0.502,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.9890000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.93200000000000005,"15min":1.004},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.41499999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":2.093},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.196,"5min":1.286,"15min":1.3360000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.51300000000000001},"max":{"1min":1.6419999999999999,"5min":1.786,"15min":2.4039999999999999},"last":1.1399999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.54500000000000004},"max":{"1min":1.7,"5min":1.8879999999999999,"15min":3.1080000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.2549999999999999,"15min":1.2889999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6319999999999999,"5min":1.7949999999999999,"15min":2.5070000000000001},"last":0.90700000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.409},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.66100000000000003},"max":{"1min":1.6739999999999999,"5min":12.669,"15min":12.669},"last":1.508},{"interface":"front","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.3740000000000001},"min":{"1min":0.89400000000000002,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6699999999999999,"5min":1.752,"15min":2.9079999999999999},"last":1.2}]},{"osd":5,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.415,"15min":1.4390000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.68300000000000005},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.3490000000000002},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.4430000000000001,"15min":1.4950000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.59799999999999998},"max":{"1min":1.6080000000000001,"5min":1.8759999999999999,"15min":2.6589999999999998},"last":1.4079999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.464,"15min":1.5669999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.64500000000000002},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":13.629},"last":1.24},{"interface":"front","average":{"1min":1.446,"5min":1.4279999999999999,"15min":1.518},"min":{"1min":0.80600000000000005,"5min":0.751,"15min":0.629},"max":{"1min":1.7849999999999999,"5min":2.0019999999999998,"15min":15.968999999999999},"last":1.2869999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.633,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.61699999999999999},"max":{"1min":1.7450000000000001,"5min":9.3900000000000006,"15min":9.3900000000000006},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.524,"15min":1.623},"min":{"1min":0.94399999999999995,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":1.893,"5min":2.081,"15min":10.455},"last":1.452}]}]},{"osd":1,"up_from":14,"seq":60129542484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1448324,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92923516,"statfs":{"total":96636764160,"available":95153680384,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.90400000000000003,"15min":0.93000000000000005},"min":{"1min":0.64800000000000002,"5min":0.53000000000000003,"15min":0.28399999999999997},"max":{"1min":1.6100000000000001,"5min":1.718,"15min":4.266},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1220000000000001,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.34999999999999998},"max":{"1min":2.153,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":1.1040000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.137,"15min":1.135},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.31},"max":{"1min":1.647,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.39300000000000002},"max":{"1min":1.579,"5min":1.8340000000000001,"15min":4.1559999999999997},"last":2.0699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.456,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.439},"max":{"1min":1.7529999999999999,"5min":1.9930000000000001,"15min":4.1980000000000004},"last":2.1560000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.46600000000000003},"max":{"1min":1.798,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":1.7030000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5349999999999999,"15min":1.5109999999999999},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.69899999999999995},"max":{"1min":1.887,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.454},{"interface":"front","average":{"1min":1.298,"5min":1.528,"15min":1.52},"min":{"1min":1.0309999999999999,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":1.613,"5min":16.248999999999999,"15min":16.248999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.522,"15min":1.466},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":1.9219999999999999,"5min":5.194,"15min":5.194},"last":1.819},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":1.714,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.0409999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.516,"15min":1.5189999999999999},"min":{"1min":1.0549999999999999,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.544,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":2.1150000000000002},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6539999999999999,"15min":1.6080000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.70699999999999996},"max":{"1min":2.0659999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":1.5900000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.609,"15min":1.589},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.69999999999999996},"max":{"1min":1.9690000000000001,"5min":16.140000000000001,"15min":16.140000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6759999999999999,"15min":1.6339999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.78700000000000003},"max":{"1min":2.4449999999999998,"5min":12.848000000000001,"15min":12.848000000000001},"last":2.0209999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542485,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073500,"kb_used_data":1997972,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298340,"statfs":{"total":96636764160,"available":94513500160,"internally_reserved":0,"allocated":2045923328,"data_stored":2045082208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0309999999999999,"15min":1.0249999999999999},"min":{"1min":0.55500000000000005,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.3260000000000001,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.2370000000000001,"15min":1.252},"min":{"1min":0.497,"5min":0.47999999999999998,"15min":0.33900000000000002},"max":{"1min":1.4470000000000001,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":1.218}]},{"osd":2,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1120000000000001,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.46700000000000003,"15min":0.40000000000000002},"max":{"1min":1.5800000000000001,"5min":1.875,"15min":7.5940000000000003},"last":1.147},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1299999999999999,"15min":1.175},"min":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.39600000000000002},"max":{"1min":1.665,"5min":1.8,"15min":5.3550000000000004},"last":0.97799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.37},"max":{"1min":1.2170000000000001,"5min":5.851,"15min":5.851},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.3169999999999999,"15min":1.343},"min":{"1min":0.65800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.5660000000000001,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":1.3560000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.304,"15min":1.3400000000000001},"min":{"1min":0.80400000000000005,"5min":0.79600000000000004,"15min":0.69099999999999995},"max":{"1min":1.7829999999999999,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.4530000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":1.7330000000000001,"5min":2.141,"15min":5.3109999999999999},"last":1.601}]},{"osd":5,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.4670000000000001,"15min":1.4930000000000001},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.75,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":1.556},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.427,"15min":1.454},"min":{"1min":0.96599999999999997,"5min":0.84699999999999998,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":2.27,"15min":8.3949999999999996},"last":1.417}]},{"osd":6,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5840000000000001,"15min":1.659},"min":{"1min":1.0669999999999999,"5min":0.82299999999999995,"15min":0.72999999999999998},"max":{"1min":1.8049999999999999,"5min":2.173,"15min":15.159000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.526,"15min":1.6259999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.58999999999999997},"max":{"1min":1.8700000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.7350000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.617,"15min":1.667},"min":{"1min":1,"5min":0.999,"15min":0.60899999999999999},"max":{"1min":1.819,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.653},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":1.018,"5min":0.873,"15min":0.65800000000000003},"max":{"1min":1.8340000000000001,"5min":2.2719999999999998,"15min":6.149},"last":1.508}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1246504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1246462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:27:33.582 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.582 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.582 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.583 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.584 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.585 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.586 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.587 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.588 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.589 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.590 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.591 INFO:tasks.ceph:pgid 16.10 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.592 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.593 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.594 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.594 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.594 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.594 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.594 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.595 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.596 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.597 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.597 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.597 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-16T14:59:21.360843+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=14, tm_min=59, tm_sec=21, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:33.597 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:27:53.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:27:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.838+0000 7f63cbe00640 1 Processor -- start 2024-01-16T15:27:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.839+0000 7f63cbe00640 1 -- start start 2024-01-16T15:27:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 0x7f63c406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63cbe00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f63c40ad720 0x7f63c415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63cbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c415c360 con 0x7f63c40abff0 2024-01-16T15:27:53.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63cbe00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40b41a0 con 0x7f63c40ad720 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63cbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c406de40 con 0x7f63c406d520 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.840+0000 7f63c8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f63c40ad720 0x7f63c415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f63c40ad720 0x7f63c415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45666/0 (socket says 172.21.15.43:45666) 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 -- 172.21.15.43:0/633274158 learned_addr learned my addr 172.21.15.43:0/633274158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c9400640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 0x7f63c406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 -- 172.21.15.43:0/633274158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 msgr2=0x7f63c406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 0x7f63c406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 -- 172.21.15.43:0/633274158 >> v1:172.21.15.69:6789/0 conn(0x7f63c40abff0 legacy=0x7f63c415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c8a00640 1 -- 172.21.15.43:0/633274158 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c415ba30 con 0x7f63c40ad720 2024-01-16T15:27:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.841+0000 7f63c9400640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 0x7f63c406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.842+0000 7f63c8a00640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c40ad720 0x7f63c415ecd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009870 tx=0x7f63ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc11a952e6c43fb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.842+0000 7f63c3e00640 1 -- 172.21.15.43:0/633274158 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac031640 con 0x7f63c40ad720 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63c3e00640 1 -- 172.21.15.43:0/633274158 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac0317a0 con 0x7f63c40ad720 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63c3e00640 1 -- 172.21.15.43:0/633274158 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac031b30 con 0x7f63c40ad720 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 -- 172.21.15.43:0/633274158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c40ad720 msgr2=0x7f63c415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c40ad720 0x7f63c415ecd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009870 tx=0x7f63ac02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 -- 172.21.15.43:0/633274158 shutdown_connections 2024-01-16T15:27:53.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c40ad720 0x7f63c415ecd0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 --2- 172.21.15.43:0/633274158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c406d520 0x7f63c406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 -- 172.21.15.43:0/633274158 >> 172.21.15.43:0/633274158 conn(0x7f63c40b0d80 msgr2=0x7f63c40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 -- 172.21.15.43:0/633274158 shutdown_connections 2024-01-16T15:27:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.843+0000 7f63cbe00640 1 -- 172.21.15.43:0/633274158 wait complete. 2024-01-16T15:27:53.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.845+0000 7f63cbe00640 1 Processor -- start 2024-01-16T15:27:53.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.845+0000 7f63cbe00640 1 -- start start 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c40abff0 0x7f63c4071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c40ad720 0x7f63c40cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4154cf0 con 0x7f63c406d520 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40cdfe0 con 0x7f63c40abff0 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.846+0000 7f63cbe00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4154f60 con 0x7f63c40ad720 2024-01-16T15:27:53.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45678/0 (socket says 172.21.15.43:45678) 2024-01-16T15:27:53.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 -- 172.21.15.43:0/3516129010 learned_addr learned my addr 172.21.15.43:0/3516129010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:27:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9e00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c40ad720 0x7f63c40cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 -- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c40ad720 msgr2=0x7f63c40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c40ad720 0x7f63c40cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 -- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c40abff0 msgr2=0x7f63c4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c40abff0 0x7f63c4071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 -- 172.21.15.43:0/3516129010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0094d0 con 0x7f63c406d520 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.847+0000 7f63c9400640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c7f0 tx=0x7f63b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.848+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007a30 con 0x7f63c406d520 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.848+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4007b90 con 0x7f63c406d520 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.848+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b400ce40 con 0x7f63c406d520 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.848+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4155240 con 0x7f63c406d520 2024-01-16T15:27:53.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.848+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4154890 con 0x7f63c406d520 2024-01-16T15:27:53.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.849+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4155430 con 0x7f63c406d520 2024-01-16T15:27:53.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.850+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f63b40113f0 con 0x7f63c406d520 2024-01-16T15:27:53.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.851+0000 7f63c1600640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 0x7f639c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.851+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63b4088d80 con 0x7f63c406d520 2024-01-16T15:27:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.852+0000 7f63c8a00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 0x7f639c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.852+0000 7f63c8a00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 0x7f639c06e980 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f63ac036000 tx=0x7f63ac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:53.856+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63b4055240 con 0x7f63c406d520 2024-01-16T15:27:54.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.048+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63c415b6e0 con 0x7f639c06c4c0 2024-01-16T15:27:54.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.060+0000 7f63c1600640 1 -- 172.21.15.43:0/3516129010 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288873 (secure 0 0 0) 0x7f63ac031a10 con 0x7f639c06c4c0 2024-01-16T15:27:54.063 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:27:54.065 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:27:54.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 msgr2=0x7f639c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 0x7f639c06e980 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f63ac036000 tx=0x7f63ac030000 comp rx=0 tx=0).stop 2024-01-16T15:27:54.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 msgr2=0x7f63c40c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c7f0 tx=0x7f63b400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 shutdown_connections 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f639c06c4c0 0x7f639c06e980 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c40abff0 0x7f63c4071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c40ad720 0x7f63c40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 --2- 172.21.15.43:0/3516129010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f63c406d520 0x7f63c40c88c0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 >> 172.21.15.43:0/3516129010 conn(0x7f63c40b0d80 msgr2=0x7f63c415ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 shutdown_connections 2024-01-16T15:27:54.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:27:54.066+0000 7f63cbe00640 1 -- 172.21.15.43:0/3516129010 wait complete. 2024-01-16T15:27:54.087 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":965,"stamp":"2024-01-16T15:27:53.513657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444294248,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2254,"num_write_kb":6295372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5190,"ondisk_log_size":5190,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13381136,"kb_used_data":12597776,"kb_used_omap":12,"kb_used_meta":783283,"kb_avail":741593584,"statfs":{"total":773094113280,"available":759391830016,"internally_reserved":0,"allocated":12900122624,"data_stored":12893537648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":802082348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":649,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005196"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737839+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737839+0000","last_peered":"2024-01-16T15:27:17.737839+0000","last_clean":"2024-01-16T15:27:17.737839+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737839+0000","last_undegraded":"2024-01-16T15:27:17.737839+0000","last_fullsized":"2024-01-16T15:27:17.737839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'91","reported_seq":132,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.519581+0000","last_change":"2024-01-16T15:27:42.519648+0000","last_active":"2024-01-16T15:27:42.519581+0000","last_peered":"2024-01-16T15:27:42.519581+0000","last_clean":"2024-01-16T15:27:42.519581+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:42.519581+0000","last_undegraded":"2024-01-16T15:27:42.519581+0000","last_fullsized":"2024-01-16T15:27:42.519581+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'91","last_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_deep_scrub":"115'91","last_deep_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.519400+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:45.850266+0000","scrub_duration":0.294982199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'67","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744022+0000","last_change":"2024-01-16T15:18:50.530067+0000","last_active":"2024-01-16T15:27:17.744022+0000","last_peered":"2024-01-16T15:27:17.744022+0000","last_clean":"2024-01-16T15:27:17.744022+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:27:17.744022+0000","last_undegraded":"2024-01-16T15:27:17.744022+0000","last_fullsized":"2024-01-16T15:27:17.744022+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733901+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733901+0000","last_peered":"2024-01-16T15:27:17.733901+0000","last_clean":"2024-01-16T15:27:17.733901+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733901+0000","last_undegraded":"2024-01-16T15:27:17.733901+0000","last_fullsized":"2024-01-16T15:27:17.733901+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'60","reported_seq":87,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720387+0000","last_change":"2024-01-16T15:18:50.507597+0000","last_active":"2024-01-16T15:27:17.720387+0000","last_peered":"2024-01-16T15:27:17.720387+0000","last_clean":"2024-01-16T15:27:17.720387+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:17.720387+0000","last_undegraded":"2024-01-16T15:27:17.720387+0000","last_fullsized":"2024-01-16T15:27:17.720387+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'105","reported_seq":149,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.422788+0000","last_change":"2024-01-16T15:27:49.422842+0000","last_active":"2024-01-16T15:27:49.422788+0000","last_peered":"2024-01-16T15:27:49.422788+0000","last_clean":"2024-01-16T15:27:49.422788+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:49.422788+0000","last_undegraded":"2024-01-16T15:27:49.422788+0000","last_fullsized":"2024-01-16T15:27:49.422788+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'105","last_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_deep_scrub":"115'105","last_deep_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_clean_scrub_stamp":"2024-01-16T15:27:49.422659+0000","objects_scrubbed":31,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:17.171558+0000","scrub_duration":0.31426238000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'84","reported_seq":114,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.394344+0000","last_change":"2024-01-16T15:27:50.394443+0000","last_active":"2024-01-16T15:27:50.394344+0000","last_peered":"2024-01-16T15:27:50.394344+0000","last_clean":"2024-01-16T15:27:50.394344+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:50.394344+0000","last_undegraded":"2024-01-16T15:27:50.394344+0000","last_fullsized":"2024-01-16T15:27:50.394344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'84","last_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_deep_scrub":"115'84","last_deep_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.394094+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:07.403833+0000","scrub_duration":0.22003049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'76","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.566925+0000","last_change":"2024-01-16T15:27:46.566980+0000","last_active":"2024-01-16T15:27:46.566925+0000","last_peered":"2024-01-16T15:27:46.566925+0000","last_clean":"2024-01-16T15:27:46.566925+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:46.566925+0000","last_undegraded":"2024-01-16T15:27:46.566925+0000","last_fullsized":"2024-01-16T15:27:46.566925+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'76","last_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_deep_scrub":"115'76","last_deep_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.566837+0000","objects_scrubbed":22,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:23:01.446658+0000","scrub_duration":0.269082504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735972+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.735972+0000","last_peered":"2024-01-16T15:27:17.735972+0000","last_clean":"2024-01-16T15:27:17.735972+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.735972+0000","last_undegraded":"2024-01-16T15:27:17.735972+0000","last_fullsized":"2024-01-16T15:27:17.735972+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733227+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.733227+0000","last_peered":"2024-01-16T15:27:17.733227+0000","last_clean":"2024-01-16T15:27:17.733227+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.733227+0000","last_undegraded":"2024-01-16T15:27:17.733227+0000","last_fullsized":"2024-01-16T15:27:17.733227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'70","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.728051+0000","last_change":"2024-01-16T15:18:50.522382+0000","last_active":"2024-01-16T15:27:17.728051+0000","last_peered":"2024-01-16T15:27:17.728051+0000","last_clean":"2024-01-16T15:27:17.728051+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:17.728051+0000","last_undegraded":"2024-01-16T15:27:17.728051+0000","last_fullsized":"2024-01-16T15:27:17.728051+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735853+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735853+0000","last_peered":"2024-01-16T15:27:17.735853+0000","last_clean":"2024-01-16T15:27:17.735853+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735853+0000","last_undegraded":"2024-01-16T15:27:17.735853+0000","last_fullsized":"2024-01-16T15:27:17.735853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'57","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.734800+0000","last_change":"2024-01-16T15:18:50.524126+0000","last_active":"2024-01-16T15:27:17.734800+0000","last_peered":"2024-01-16T15:27:17.734800+0000","last_clean":"2024-01-16T15:27:17.734800+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:27:17.734800+0000","last_undegraded":"2024-01-16T15:27:17.734800+0000","last_fullsized":"2024-01-16T15:27:17.734800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'88","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.731246+0000","last_change":"2024-01-16T15:27:37.731333+0000","last_active":"2024-01-16T15:27:37.731246+0000","last_peered":"2024-01-16T15:27:37.731246+0000","last_clean":"2024-01-16T15:27:37.731246+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:37.731246+0000","last_undegraded":"2024-01-16T15:27:37.731246+0000","last_fullsized":"2024-01-16T15:27:37.731246+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.731071+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:51.646988+0000","scrub_duration":0.35245168599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'68","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.237079+0000","last_change":"2024-01-16T15:27:48.237143+0000","last_active":"2024-01-16T15:27:48.237079+0000","last_peered":"2024-01-16T15:27:48.237079+0000","last_clean":"2024-01-16T15:27:48.237079+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:48.237079+0000","last_undegraded":"2024-01-16T15:27:48.237079+0000","last_fullsized":"2024-01-16T15:27:48.237079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'68","last_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_deep_scrub":"115'68","last_deep_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.236957+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:17.637489+0000","scrub_duration":0.116219171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'74","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.886948+0000","last_change":"2024-01-16T15:27:28.887006+0000","last_active":"2024-01-16T15:27:28.886948+0000","last_peered":"2024-01-16T15:27:28.886948+0000","last_clean":"2024-01-16T15:27:28.886948+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:28.886948+0000","last_undegraded":"2024-01-16T15:27:28.886948+0000","last_fullsized":"2024-01-16T15:27:28.886948+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'74","last_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_deep_scrub":"115'74","last_deep_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_clean_scrub_stamp":"2024-01-16T15:27:28.886880+0000","objects_scrubbed":19,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:33.087313+0000","scrub_duration":0.202995169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'66","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.256662+0000","last_change":"2024-01-16T15:27:50.256801+0000","last_active":"2024-01-16T15:27:50.256662+0000","last_peered":"2024-01-16T15:27:50.256662+0000","last_clean":"2024-01-16T15:27:50.256662+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:50.256662+0000","last_undegraded":"2024-01-16T15:27:50.256662+0000","last_fullsized":"2024-01-16T15:27:50.256662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'66","last_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_deep_scrub":"115'66","last_deep_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.256459+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:44.950845+0000","scrub_duration":0.21236617099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'107","reported_seq":158,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:47.168628+0000","last_change":"2024-01-16T15:27:47.168707+0000","last_active":"2024-01-16T15:27:47.168628+0000","last_peered":"2024-01-16T15:27:47.168628+0000","last_clean":"2024-01-16T15:27:47.168628+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:47.168628+0000","last_undegraded":"2024-01-16T15:27:47.168628+0000","last_fullsized":"2024-01-16T15:27:47.168628+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'107","last_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_deep_scrub":"115'107","last_deep_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_clean_scrub_stamp":"2024-01-16T15:27:47.168437+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:38.535012+0000","scrub_duration":0.41892874400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'71","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.107300+0000","last_change":"2024-01-16T15:18:50.527427+0000","last_active":"2024-01-16T15:27:33.107300+0000","last_peered":"2024-01-16T15:27:33.107300+0000","last_clean":"2024-01-16T15:27:33.107300+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:27:33.107300+0000","last_undegraded":"2024-01-16T15:27:33.107300+0000","last_fullsized":"2024-01-16T15:27:33.107300+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'87","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.480772+0000","last_change":"2024-01-16T15:27:43.480772+0000","last_active":"2024-01-16T15:27:43.480772+0000","last_peered":"2024-01-16T15:27:43.480772+0000","last_clean":"2024-01-16T15:27:43.480772+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:43.480772+0000","last_undegraded":"2024-01-16T15:27:43.480772+0000","last_fullsized":"2024-01-16T15:27:43.480772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'87","last_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_deep_scrub":"115'87","last_deep_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.480597+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:52.268973+0000","scrub_duration":0.29440454500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'73","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.881163+0000","last_change":"2024-01-16T15:18:50.521788+0000","last_active":"2024-01-16T15:27:30.881163+0000","last_peered":"2024-01-16T15:27:30.881163+0000","last_clean":"2024-01-16T15:27:30.881163+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:27:30.881163+0000","last_undegraded":"2024-01-16T15:27:30.881163+0000","last_fullsized":"2024-01-16T15:27:30.881163+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'64","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138521+0000","last_change":"2024-01-16T15:18:50.498139+0000","last_active":"2024-01-16T15:27:30.138521+0000","last_peered":"2024-01-16T15:27:30.138521+0000","last_clean":"2024-01-16T15:27:30.138521+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:27:30.138521+0000","last_undegraded":"2024-01-16T15:27:30.138521+0000","last_fullsized":"2024-01-16T15:27:30.138521+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'57","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.881111+0000","last_change":"2024-01-16T15:18:50.519702+0000","last_active":"2024-01-16T15:27:30.881111+0000","last_peered":"2024-01-16T15:27:30.881111+0000","last_clean":"2024-01-16T15:27:30.881111+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:30.881111+0000","last_undegraded":"2024-01-16T15:27:30.881111+0000","last_fullsized":"2024-01-16T15:27:30.881111+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'70","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138498+0000","last_change":"2024-01-16T15:18:50.518585+0000","last_active":"2024-01-16T15:27:30.138498+0000","last_peered":"2024-01-16T15:27:30.138498+0000","last_clean":"2024-01-16T15:27:30.138498+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:27:30.138498+0000","last_undegraded":"2024-01-16T15:27:30.138498+0000","last_fullsized":"2024-01-16T15:27:30.138498+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'97","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.461542+0000","last_change":"2024-01-16T15:27:45.461588+0000","last_active":"2024-01-16T15:27:45.461542+0000","last_peered":"2024-01-16T15:27:45.461542+0000","last_clean":"2024-01-16T15:27:45.461542+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:45.461542+0000","last_undegraded":"2024-01-16T15:27:45.461542+0000","last_fullsized":"2024-01-16T15:27:45.461542+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.461410+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:24.634385+0000","scrub_duration":0.31535565399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'81","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:27.943347+0000","last_change":"2024-01-16T15:18:50.516663+0000","last_active":"2024-01-16T15:27:27.943347+0000","last_peered":"2024-01-16T15:27:27.943347+0000","last_clean":"2024-01-16T15:27:27.943347+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:27:27.943347+0000","last_undegraded":"2024-01-16T15:27:27.943347+0000","last_fullsized":"2024-01-16T15:27:27.943347+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'64","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.665594+0000","last_change":"2024-01-16T15:18:50.524162+0000","last_active":"2024-01-16T15:27:28.665594+0000","last_peered":"2024-01-16T15:27:28.665594+0000","last_clean":"2024-01-16T15:27:28.665594+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:27:28.665594+0000","last_undegraded":"2024-01-16T15:27:28.665594+0000","last_fullsized":"2024-01-16T15:27:28.665594+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":66,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138416+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:30.138416+0000","last_peered":"2024-01-16T15:27:30.138416+0000","last_clean":"2024-01-16T15:27:30.138416+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:30.138416+0000","last_undegraded":"2024-01-16T15:27:30.138416+0000","last_fullsized":"2024-01-16T15:27:30.138416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'57","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742348+0000","last_change":"2024-01-16T15:18:50.521190+0000","last_active":"2024-01-16T15:27:17.742348+0000","last_peered":"2024-01-16T15:27:17.742348+0000","last_clean":"2024-01-16T15:27:17.742348+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:27:17.742348+0000","last_undegraded":"2024-01-16T15:27:17.742348+0000","last_fullsized":"2024-01-16T15:27:17.742348+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'104","reported_seq":146,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.544481+0000","last_change":"2024-01-16T15:27:48.544547+0000","last_active":"2024-01-16T15:27:48.544481+0000","last_peered":"2024-01-16T15:27:48.544481+0000","last_clean":"2024-01-16T15:27:48.544481+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:48.544481+0000","last_undegraded":"2024-01-16T15:27:48.544481+0000","last_fullsized":"2024-01-16T15:27:48.544481+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'104","last_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_deep_scrub":"115'104","last_deep_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.544284+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:19.364542+0000","scrub_duration":0.32836960199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'106","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.886489+0000","last_change":"2024-01-16T15:27:33.886536+0000","last_active":"2024-01-16T15:27:33.886489+0000","last_peered":"2024-01-16T15:27:33.886489+0000","last_clean":"2024-01-16T15:27:33.886489+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:33.886489+0000","last_undegraded":"2024-01-16T15:27:33.886489+0000","last_fullsized":"2024-01-16T15:27:33.886489+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'106","last_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_deep_scrub":"115'106","last_deep_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.886354+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:46.223371+0000","scrub_duration":0.231844831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'85","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.353703+0000","last_change":"2024-01-16T15:27:44.353765+0000","last_active":"2024-01-16T15:27:44.353703+0000","last_peered":"2024-01-16T15:27:44.353703+0000","last_clean":"2024-01-16T15:27:44.353703+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:44.353703+0000","last_undegraded":"2024-01-16T15:27:44.353703+0000","last_fullsized":"2024-01-16T15:27:44.353703+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.353606+0000","objects_scrubbed":23,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:15.404103+0000","scrub_duration":0.234855483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'62","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.722024+0000","last_change":"2024-01-16T15:18:50.518986+0000","last_active":"2024-01-16T15:27:17.722024+0000","last_peered":"2024-01-16T15:27:17.722024+0000","last_clean":"2024-01-16T15:27:17.722024+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:17.722024+0000","last_undegraded":"2024-01-16T15:27:17.722024+0000","last_fullsized":"2024-01-16T15:27:17.722024+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'6","reported_seq":33,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.184374+0000","last_change":"2024-01-16T15:27:41.184433+0000","last_active":"2024-01-16T15:27:41.184374+0000","last_peered":"2024-01-16T15:27:41.184374+0000","last_clean":"2024-01-16T15:27:41.184374+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:27:41.184374+0000","last_undegraded":"2024-01-16T15:27:41.184374+0000","last_fullsized":"2024-01-16T15:27:41.184374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.184165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:01.093220+0000","scrub_duration":0.0077587799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'54","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737496+0000","last_change":"2024-01-16T15:18:50.515474+0000","last_active":"2024-01-16T15:27:17.737496+0000","last_peered":"2024-01-16T15:27:17.737496+0000","last_clean":"2024-01-16T15:27:17.737496+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:27:17.737496+0000","last_undegraded":"2024-01-16T15:27:17.737496+0000","last_fullsized":"2024-01-16T15:27:17.737496+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":28,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:36.422396+0000","last_change":"2024-01-16T15:27:36.422452+0000","last_active":"2024-01-16T15:27:36.422396+0000","last_peered":"2024-01-16T15:27:36.422396+0000","last_clean":"2024-01-16T15:27:36.422396+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:27:36.422396+0000","last_undegraded":"2024-01-16T15:27:36.422396+0000","last_fullsized":"2024-01-16T15:27:36.422396+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_clean_scrub_stamp":"2024-01-16T15:27:36.422186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:42.355689+0000","scrub_duration":0.003123125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'71","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.724966+0000","last_change":"2024-01-16T15:18:50.526507+0000","last_active":"2024-01-16T15:27:17.724966+0000","last_peered":"2024-01-16T15:27:17.724966+0000","last_clean":"2024-01-16T15:27:17.724966+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:17.724966+0000","last_undegraded":"2024-01-16T15:27:17.724966+0000","last_fullsized":"2024-01-16T15:27:17.724966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'4","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.217452+0000","last_change":"2024-01-16T15:27:44.217632+0000","last_active":"2024-01-16T15:27:44.217452+0000","last_peered":"2024-01-16T15:27:44.217452+0000","last_clean":"2024-01-16T15:27:44.217452+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:27:44.217452+0000","last_undegraded":"2024-01-16T15:27:44.217452+0000","last_fullsized":"2024-01-16T15:27:44.217452+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.217230+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:49.674761+0000","scrub_duration":0.007544144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'68","reported_seq":106,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.744833+0000","last_change":"2024-01-16T15:18:50.526886+0000","last_active":"2024-01-16T15:27:17.744833+0000","last_peered":"2024-01-16T15:27:17.744833+0000","last_clean":"2024-01-16T15:27:17.744833+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:27:17.744833+0000","last_undegraded":"2024-01-16T15:27:17.744833+0000","last_fullsized":"2024-01-16T15:27:17.744833+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'3","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.189552+0000","last_change":"2024-01-16T15:27:38.189613+0000","last_active":"2024-01-16T15:27:38.189552+0000","last_peered":"2024-01-16T15:27:38.189552+0000","last_clean":"2024-01-16T15:27:38.189552+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:27:38.189552+0000","last_undegraded":"2024-01-16T15:27:38.189552+0000","last_fullsized":"2024-01-16T15:27:38.189552+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'3","last_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_deep_scrub":"115'3","last_deep_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:03:26.432118+0000","scrub_duration":0.0070076540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'72","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.720059+0000","last_change":"2024-01-16T15:19:15.318178+0000","last_active":"2024-01-16T15:27:17.720059+0000","last_peered":"2024-01-16T15:27:17.720059+0000","last_clean":"2024-01-16T15:27:17.720059+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:17.720059+0000","last_undegraded":"2024-01-16T15:27:17.720059+0000","last_fullsized":"2024-01-16T15:27:17.720059+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.186234+0000","last_change":"2024-01-16T15:27:34.186287+0000","last_active":"2024-01-16T15:27:34.186234+0000","last_peered":"2024-01-16T15:27:34.186234+0000","last_clean":"2024-01-16T15:27:34.186234+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:27:34.186234+0000","last_undegraded":"2024-01-16T15:27:34.186234+0000","last_fullsized":"2024-01-16T15:27:34.186234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.186050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:31:37.071156+0000","scrub_duration":0.0039566280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'72","reported_seq":109,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737041+0000","last_change":"2024-01-16T15:18:50.517577+0000","last_active":"2024-01-16T15:27:17.737041+0000","last_peered":"2024-01-16T15:27:17.737041+0000","last_clean":"2024-01-16T15:27:17.737041+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:27:17.737041+0000","last_undegraded":"2024-01-16T15:27:17.737041+0000","last_fullsized":"2024-01-16T15:27:17.737041+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.865012+0000","last_change":"2024-01-16T15:27:39.865110+0000","last_active":"2024-01-16T15:27:39.865012+0000","last_peered":"2024-01-16T15:27:39.865012+0000","last_clean":"2024-01-16T15:27:39.865012+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:27:39.865012+0000","last_undegraded":"2024-01-16T15:27:39.865012+0000","last_fullsized":"2024-01-16T15:27:39.865012+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.864756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:53.780673+0000","scrub_duration":0.0037524390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'95","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.871075+0000","last_change":"2024-01-16T15:27:29.871123+0000","last_active":"2024-01-16T15:27:29.871075+0000","last_peered":"2024-01-16T15:27:29.871075+0000","last_clean":"2024-01-16T15:27:29.871075+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:29.871075+0000","last_undegraded":"2024-01-16T15:27:29.871075+0000","last_fullsized":"2024-01-16T15:27:29.871075+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.871010+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:25.437998+0000","scrub_duration":0.15415410600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.317980+0000","last_change":"2024-01-16T15:27:32.318028+0000","last_active":"2024-01-16T15:27:32.317980+0000","last_peered":"2024-01-16T15:27:32.317980+0000","last_clean":"2024-01-16T15:27:32.317980+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:27:32.317980+0000","last_undegraded":"2024-01-16T15:27:32.317980+0000","last_fullsized":"2024-01-16T15:27:32.317980+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.317792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:01.084778+0000","scrub_duration":0.0035389969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'101","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.595106+0000","last_change":"2024-01-16T15:27:46.595143+0000","last_active":"2024-01-16T15:27:46.595106+0000","last_peered":"2024-01-16T15:27:46.595106+0000","last_clean":"2024-01-16T15:27:46.595106+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:46.595106+0000","last_undegraded":"2024-01-16T15:27:46.595106+0000","last_fullsized":"2024-01-16T15:27:46.595106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'101","last_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_deep_scrub":"115'101","last_deep_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.595008+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:44.983075+0000","scrub_duration":0.33379621399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.688763+0000","last_change":"2024-01-16T15:27:39.688811+0000","last_active":"2024-01-16T15:27:39.688763+0000","last_peered":"2024-01-16T15:27:39.688763+0000","last_clean":"2024-01-16T15:27:39.688763+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:27:39.688763+0000","last_undegraded":"2024-01-16T15:27:39.688763+0000","last_fullsized":"2024-01-16T15:27:39.688763+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.688565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:07.291212+0000","scrub_duration":0.0036420599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'72","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.479079+0000","last_change":"2024-01-16T15:27:43.479158+0000","last_active":"2024-01-16T15:27:43.479079+0000","last_peered":"2024-01-16T15:27:43.479079+0000","last_clean":"2024-01-16T15:27:43.479079+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:43.479079+0000","last_undegraded":"2024-01-16T15:27:43.479079+0000","last_fullsized":"2024-01-16T15:27:43.479079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'72","last_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_deep_scrub":"115'72","last_deep_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.478956+0000","objects_scrubbed":18,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:43.468767+0000","scrub_duration":0.23849152200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:26:05.384938+0000","last_change":"2024-01-16T15:18:49.524926+0000","last_active":"2024-01-16T15:26:05.384938+0000","last_peered":"2024-01-16T15:26:05.384938+0000","last_clean":"2024-01-16T15:26:05.384938+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:26:05.384938+0000","last_undegraded":"2024-01-16T15:26:05.384938+0000","last_fullsized":"2024-01-16T15:26:05.384938+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'61","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739104+0000","last_change":"2024-01-16T15:18:50.524490+0000","last_active":"2024-01-16T15:27:17.739104+0000","last_peered":"2024-01-16T15:27:17.739104+0000","last_clean":"2024-01-16T15:27:17.739104+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:17.739104+0000","last_undegraded":"2024-01-16T15:27:17.739104+0000","last_fullsized":"2024-01-16T15:27:17.739104+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.836697+0000","last_change":"2024-01-16T15:27:30.836744+0000","last_active":"2024-01-16T15:27:30.836697+0000","last_peered":"2024-01-16T15:27:30.836697+0000","last_clean":"2024-01-16T15:27:30.836697+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:27:30.836697+0000","last_undegraded":"2024-01-16T15:27:30.836697+0000","last_fullsized":"2024-01-16T15:27:30.836697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_clean_scrub_stamp":"2024-01-16T15:27:30.836522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:38.704619+0000","scrub_duration":0.0037552990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.726408+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:17.726408+0000","last_peered":"2024-01-16T15:27:17.726408+0000","last_clean":"2024-01-16T15:27:17.726408+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:17.726408+0000","last_undegraded":"2024-01-16T15:27:17.726408+0000","last_fullsized":"2024-01-16T15:27:17.726408+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'35","reported_seq":61,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.716929+0000","last_change":"2024-01-16T15:27:48.716990+0000","last_active":"2024-01-16T15:27:48.716929+0000","last_peered":"2024-01-16T15:27:48.716929+0000","last_clean":"2024-01-16T15:27:48.716929+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:27:48.716929+0000","last_undegraded":"2024-01-16T15:27:48.716929+0000","last_fullsized":"2024-01-16T15:27:48.716929+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.716639+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:05.125879+0000","scrub_duration":0.0089525939999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'69","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.729166+0000","last_change":"2024-01-16T15:18:50.518451+0000","last_active":"2024-01-16T15:27:17.729166+0000","last_peered":"2024-01-16T15:27:17.729166+0000","last_clean":"2024-01-16T15:27:17.729166+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:17.729166+0000","last_undegraded":"2024-01-16T15:27:17.729166+0000","last_fullsized":"2024-01-16T15:27:17.729166+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'325","reported_seq":351,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.744109+0000","last_change":"2024-01-16T15:27:45.744230+0000","last_active":"2024-01-16T15:27:45.744109+0000","last_peered":"2024-01-16T15:27:45.744109+0000","last_clean":"2024-01-16T15:27:45.744109+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:45.744109+0000","last_undegraded":"2024-01-16T15:27:45.744109+0000","last_fullsized":"2024-01-16T15:27:45.744109+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'325","last_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_deep_scrub":"115'325","last_deep_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.743867+0000","objects_scrubbed":4,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:00.095065+0000","scrub_duration":0.018757899000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1246392,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.718779+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:17.718779+0000","last_peered":"2024-01-16T15:27:17.718779+0000","last_clean":"2024-01-16T15:27:17.718779+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:17.718779+0000","last_undegraded":"2024-01-16T15:27:17.718779+0000","last_fullsized":"2024-01-16T15:27:17.718779+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"115'4","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.161906+0000","last_change":"2024-01-16T15:27:38.161975+0000","last_active":"2024-01-16T15:27:38.161906+0000","last_peered":"2024-01-16T15:27:38.161906+0000","last_clean":"2024-01-16T15:27:38.161906+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:27:38.161906+0000","last_undegraded":"2024-01-16T15:27:38.161906+0000","last_fullsized":"2024-01-16T15:27:38.161906+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.161723+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:02.215134+0000","scrub_duration":0.0087694660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'95","reported_seq":132,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.180797+0000","last_change":"2024-01-16T15:27:43.180860+0000","last_active":"2024-01-16T15:27:43.180797+0000","last_peered":"2024-01-16T15:27:43.180797+0000","last_clean":"2024-01-16T15:27:43.180797+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:43.180797+0000","last_undegraded":"2024-01-16T15:27:43.180797+0000","last_fullsized":"2024-01-16T15:27:43.180797+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.180569+0000","objects_scrubbed":25,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:27.476294+0000","scrub_duration":0.28297945600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.754020+0000","last_change":"2024-01-16T15:27:33.754037+0000","last_active":"2024-01-16T15:27:33.754020+0000","last_peered":"2024-01-16T15:27:33.754020+0000","last_clean":"2024-01-16T15:27:33.754020+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:27:33.754020+0000","last_undegraded":"2024-01-16T15:27:33.754020+0000","last_fullsized":"2024-01-16T15:27:33.754020+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.753955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:34:45.427457+0000","scrub_duration":0.0018762620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'63","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.740857+0000","last_change":"2024-01-16T15:18:50.510030+0000","last_active":"2024-01-16T15:27:17.740857+0000","last_peered":"2024-01-16T15:27:17.740857+0000","last_clean":"2024-01-16T15:27:17.740857+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:27:17.740857+0000","last_undegraded":"2024-01-16T15:27:17.740857+0000","last_fullsized":"2024-01-16T15:27:17.740857+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'12","reported_seq":38,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.176283+0000","last_change":"2024-01-16T15:27:42.176351+0000","last_active":"2024-01-16T15:27:42.176283+0000","last_peered":"2024-01-16T15:27:42.176283+0000","last_clean":"2024-01-16T15:27:42.176283+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:27:42.176283+0000","last_undegraded":"2024-01-16T15:27:42.176283+0000","last_fullsized":"2024-01-16T15:27:42.176283+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.176093+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:16.846949+0000","scrub_duration":0.010433985999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'46","reported_seq":70,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.373142+0000","last_change":"2024-01-16T15:18:50.520815+0000","last_active":"2024-01-16T15:27:29.373142+0000","last_peered":"2024-01-16T15:27:29.373142+0000","last_clean":"2024-01-16T15:27:29.373142+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:27:29.373142+0000","last_undegraded":"2024-01-16T15:27:29.373142+0000","last_fullsized":"2024-01-16T15:27:29.373142+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:27.942939+0000","last_change":"2024-01-16T15:18:49.515664+0000","last_active":"2024-01-16T15:27:27.942939+0000","last_peered":"2024-01-16T15:27:27.942939+0000","last_clean":"2024-01-16T15:27:27.942939+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:27:27.942939+0000","last_undegraded":"2024-01-16T15:27:27.942939+0000","last_fullsized":"2024-01-16T15:27:27.942939+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.665483+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:28.665483+0000","last_peered":"2024-01-16T15:27:28.665483+0000","last_clean":"2024-01-16T15:27:28.665483+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:28.665483+0000","last_undegraded":"2024-01-16T15:27:28.665483+0000","last_fullsized":"2024-01-16T15:27:28.665483+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":21,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.373044+0000","last_change":"2024-01-16T15:18:49.507896+0000","last_active":"2024-01-16T15:27:29.373044+0000","last_peered":"2024-01-16T15:27:29.373044+0000","last_clean":"2024-01-16T15:27:29.373044+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:27:29.373044+0000","last_undegraded":"2024-01-16T15:27:29.373044+0000","last_fullsized":"2024-01-16T15:27:29.373044+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'75","reported_seq":115,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.463446+0000","last_change":"2024-01-16T15:27:50.463446+0000","last_active":"2024-01-16T15:27:50.463446+0000","last_peered":"2024-01-16T15:27:50.463446+0000","last_clean":"2024-01-16T15:27:50.463446+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:50.463446+0000","last_undegraded":"2024-01-16T15:27:50.463446+0000","last_fullsized":"2024-01-16T15:27:50.463446+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'75","last_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_deep_scrub":"115'75","last_deep_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.463136+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:21.882678+0000","scrub_duration":0.16385127299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'2","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.734212+0000","last_change":"2024-01-16T15:27:38.734212+0000","last_active":"2024-01-16T15:27:38.734212+0000","last_peered":"2024-01-16T15:27:38.734212+0000","last_clean":"2024-01-16T15:27:38.734212+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:27:38.734212+0000","last_undegraded":"2024-01-16T15:27:38.734212+0000","last_fullsized":"2024-01-16T15:27:38.734212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.733970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:33.428355+0000","scrub_duration":0.0076776479999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'62","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.107117+0000","last_change":"2024-01-16T15:18:50.520972+0000","last_active":"2024-01-16T15:27:33.107117+0000","last_peered":"2024-01-16T15:27:33.107117+0000","last_clean":"2024-01-16T15:27:33.107117+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:27:33.107117+0000","last_undegraded":"2024-01-16T15:27:33.107117+0000","last_fullsized":"2024-01-16T15:27:33.107117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.144273+0000","last_change":"2024-01-16T15:27:39.144273+0000","last_active":"2024-01-16T15:27:39.144273+0000","last_peered":"2024-01-16T15:27:39.144273+0000","last_clean":"2024-01-16T15:27:39.144273+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:27:39.144273+0000","last_undegraded":"2024-01-16T15:27:39.144273+0000","last_fullsized":"2024-01-16T15:27:39.144273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.144028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.164911+0000","scrub_duration":0.0035817119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'82","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.596918+0000","last_change":"2024-01-16T15:27:45.596964+0000","last_active":"2024-01-16T15:27:45.596918+0000","last_peered":"2024-01-16T15:27:45.596918+0000","last_clean":"2024-01-16T15:27:45.596918+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:45.596918+0000","last_undegraded":"2024-01-16T15:27:45.596918+0000","last_fullsized":"2024-01-16T15:27:45.596918+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'82","last_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_deep_scrub":"115'82","last_deep_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.596772+0000","objects_scrubbed":26,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:29.892497+0000","scrub_duration":0.31446099100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.232655+0000","last_change":"2024-01-16T15:27:34.232655+0000","last_active":"2024-01-16T15:27:34.232655+0000","last_peered":"2024-01-16T15:27:34.232655+0000","last_clean":"2024-01-16T15:27:34.232655+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:27:34.232655+0000","last_undegraded":"2024-01-16T15:27:34.232655+0000","last_fullsized":"2024-01-16T15:27:34.232655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.232419+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:43.752231+0000","scrub_duration":0.0069378510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'70","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:27.943215+0000","last_change":"2024-01-16T15:18:50.515695+0000","last_active":"2024-01-16T15:27:27.943215+0000","last_peered":"2024-01-16T15:27:27.943215+0000","last_clean":"2024-01-16T15:27:27.943215+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:27:27.943215+0000","last_undegraded":"2024-01-16T15:27:27.943215+0000","last_fullsized":"2024-01-16T15:27:27.943215+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.164374+0000","last_change":"2024-01-16T15:27:39.164427+0000","last_active":"2024-01-16T15:27:39.164374+0000","last_peered":"2024-01-16T15:27:39.164374+0000","last_clean":"2024-01-16T15:27:39.164374+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:27:39.164374+0000","last_undegraded":"2024-01-16T15:27:39.164374+0000","last_fullsized":"2024-01-16T15:27:39.164374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.164189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:40.647599+0000","scrub_duration":0.003563882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'65","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.496893+0000","last_change":"2024-01-16T15:18:50.526068+0000","last_active":"2024-01-16T15:27:30.496893+0000","last_peered":"2024-01-16T15:27:30.496893+0000","last_clean":"2024-01-16T15:27:30.496893+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:27:30.496893+0000","last_undegraded":"2024-01-16T15:27:30.496893+0000","last_fullsized":"2024-01-16T15:27:30.496893+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.193731+0000","last_change":"2024-01-16T15:27:29.193731+0000","last_active":"2024-01-16T15:27:29.193731+0000","last_peered":"2024-01-16T15:27:29.193731+0000","last_clean":"2024-01-16T15:27:29.193731+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:27:29.193731+0000","last_undegraded":"2024-01-16T15:27:29.193731+0000","last_fullsized":"2024-01-16T15:27:29.193731+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.193592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:03.864449+0000","scrub_duration":0.0028384030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'77","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.205484+0000","last_change":"2024-01-16T15:27:48.205536+0000","last_active":"2024-01-16T15:27:48.205484+0000","last_peered":"2024-01-16T15:27:48.205484+0000","last_clean":"2024-01-16T15:27:48.205484+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:48.205484+0000","last_undegraded":"2024-01-16T15:27:48.205484+0000","last_fullsized":"2024-01-16T15:27:48.205484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'77","last_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_deep_scrub":"115'77","last_deep_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.205363+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:54.141377+0000","scrub_duration":0.17343320200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.694453+0000","last_change":"2024-01-16T15:27:41.694504+0000","last_active":"2024-01-16T15:27:41.694453+0000","last_peered":"2024-01-16T15:27:41.694453+0000","last_clean":"2024-01-16T15:27:41.694453+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:27:41.694453+0000","last_undegraded":"2024-01-16T15:27:41.694453+0000","last_fullsized":"2024-01-16T15:27:41.694453+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.694252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.406823+0000","scrub_duration":0.0042882689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'53","reported_seq":77,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.482024+0000","last_change":"2024-01-16T15:18:50.520913+0000","last_active":"2024-01-16T15:27:30.482024+0000","last_peered":"2024-01-16T15:27:30.482024+0000","last_clean":"2024-01-16T15:27:30.482024+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:30.482024+0000","last_undegraded":"2024-01-16T15:27:30.482024+0000","last_fullsized":"2024-01-16T15:27:30.482024+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.867500+0000","last_change":"2024-01-16T15:27:38.867614+0000","last_active":"2024-01-16T15:27:38.867500+0000","last_peered":"2024-01-16T15:27:38.867500+0000","last_clean":"2024-01-16T15:27:38.867500+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:27:38.867500+0000","last_undegraded":"2024-01-16T15:27:38.867500+0000","last_fullsized":"2024-01-16T15:27:38.867500+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.867296+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:02.481138+0000","scrub_duration":0.0076157439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.741585+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:17.741585+0000","last_peered":"2024-01-16T15:27:17.741585+0000","last_clean":"2024-01-16T15:27:17.741585+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:17.741585+0000","last_undegraded":"2024-01-16T15:27:17.741585+0000","last_fullsized":"2024-01-16T15:27:17.741585+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'31","reported_seq":58,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.825906+0000","last_change":"2024-01-16T15:27:42.826005+0000","last_active":"2024-01-16T15:27:42.825906+0000","last_peered":"2024-01-16T15:27:42.825906+0000","last_clean":"2024-01-16T15:27:42.825906+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:42.825906+0000","last_undegraded":"2024-01-16T15:27:42.825906+0000","last_fullsized":"2024-01-16T15:27:42.825906+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'31","last_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_deep_scrub":"115'31","last_deep_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.825641+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:57.748209+0000","scrub_duration":0.0051599990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":649,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'69","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.736049+0000","last_change":"2024-01-16T15:18:50.514360+0000","last_active":"2024-01-16T15:27:17.736049+0000","last_peered":"2024-01-16T15:27:17.736049+0000","last_clean":"2024-01-16T15:27:17.736049+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:27:17.736049+0000","last_undegraded":"2024-01-16T15:27:17.736049+0000","last_fullsized":"2024-01-16T15:27:17.736049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:10.684206+0000","last_change":"2024-01-16T15:18:49.521642+0000","last_active":"2024-01-16T15:27:10.684206+0000","last_peered":"2024-01-16T15:27:10.684206+0000","last_clean":"2024-01-16T15:27:10.684206+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:27:10.684206+0000","last_undegraded":"2024-01-16T15:27:10.684206+0000","last_fullsized":"2024-01-16T15:27:10.684206+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:48.457576+0000","last_clean_scrub_stamp":"2024-01-16T15:18:48.457576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'59","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.738520+0000","last_change":"2024-01-16T15:18:50.523523+0000","last_active":"2024-01-16T15:27:17.738520+0000","last_peered":"2024-01-16T15:27:17.738520+0000","last_clean":"2024-01-16T15:27:17.738520+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:27:17.738520+0000","last_undegraded":"2024-01-16T15:27:17.738520+0000","last_fullsized":"2024-01-16T15:27:17.738520+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.818788+0000","last_change":"2024-01-16T15:27:32.818822+0000","last_active":"2024-01-16T15:27:32.818788+0000","last_peered":"2024-01-16T15:27:32.818788+0000","last_clean":"2024-01-16T15:27:32.818788+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:27:32.818788+0000","last_undegraded":"2024-01-16T15:27:32.818788+0000","last_fullsized":"2024-01-16T15:27:32.818788+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.818678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:29.803044+0000","scrub_duration":0.0024947659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'83","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:40.537839+0000","last_change":"2024-01-16T15:27:40.537917+0000","last_active":"2024-01-16T15:27:40.537839+0000","last_peered":"2024-01-16T15:27:40.537839+0000","last_clean":"2024-01-16T15:27:40.537839+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:40.537839+0000","last_undegraded":"2024-01-16T15:27:40.537839+0000","last_fullsized":"2024-01-16T15:27:40.537839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'83","last_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_deep_scrub":"115'83","last_deep_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_clean_scrub_stamp":"2024-01-16T15:27:40.537681+0000","objects_scrubbed":21,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:15.744820+0000","scrub_duration":0.26540904500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.219632+0000","last_change":"2024-01-16T15:27:33.219680+0000","last_active":"2024-01-16T15:27:33.219632+0000","last_peered":"2024-01-16T15:27:33.219632+0000","last_clean":"2024-01-16T15:27:33.219632+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:27:33.219632+0000","last_undegraded":"2024-01-16T15:27:33.219632+0000","last_fullsized":"2024-01-16T15:27:33.219632+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.219447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:42.739258+0000","scrub_duration":0.0037869280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'66","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.742915+0000","last_change":"2024-01-16T15:18:50.510296+0000","last_active":"2024-01-16T15:27:17.742915+0000","last_peered":"2024-01-16T15:27:17.742915+0000","last_clean":"2024-01-16T15:27:17.742915+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:27:17.742915+0000","last_undegraded":"2024-01-16T15:27:17.742915+0000","last_fullsized":"2024-01-16T15:27:17.742915+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.251350+0000","last_change":"2024-01-16T15:27:39.251405+0000","last_active":"2024-01-16T15:27:39.251350+0000","last_peered":"2024-01-16T15:27:39.251350+0000","last_clean":"2024-01-16T15:27:39.251350+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:27:39.251350+0000","last_undegraded":"2024-01-16T15:27:39.251350+0000","last_fullsized":"2024-01-16T15:27:39.251350+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.251139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:04.028594+0000","scrub_duration":0.0041990689999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'68","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.739052+0000","last_change":"2024-01-16T15:18:50.517844+0000","last_active":"2024-01-16T15:27:17.739052+0000","last_peered":"2024-01-16T15:27:17.739052+0000","last_clean":"2024-01-16T15:27:17.739052+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:27:17.739052+0000","last_undegraded":"2024-01-16T15:27:17.739052+0000","last_fullsized":"2024-01-16T15:27:17.739052+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'35","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.186518+0000","last_change":"2024-01-16T15:27:43.186575+0000","last_active":"2024-01-16T15:27:43.186518+0000","last_peered":"2024-01-16T15:27:43.186518+0000","last_clean":"2024-01-16T15:27:43.186518+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:27:43.186518+0000","last_undegraded":"2024-01-16T15:27:43.186518+0000","last_fullsized":"2024-01-16T15:27:43.186518+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.186344+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:34.244119+0000","scrub_duration":0.007088273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735204+0000","last_change":"2024-01-16T15:18:50.519815+0000","last_active":"2024-01-16T15:27:17.735204+0000","last_peered":"2024-01-16T15:27:17.735204+0000","last_clean":"2024-01-16T15:27:17.735204+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:27:17.735204+0000","last_undegraded":"2024-01-16T15:27:17.735204+0000","last_fullsized":"2024-01-16T15:27:17.735204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.623423+0000","last_change":"2024-01-16T15:27:37.623518+0000","last_active":"2024-01-16T15:27:37.623423+0000","last_peered":"2024-01-16T15:27:37.623423+0000","last_clean":"2024-01-16T15:27:37.623423+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:27:37.623423+0000","last_undegraded":"2024-01-16T15:27:37.623423+0000","last_fullsized":"2024-01-16T15:27:37.623423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.623212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:31.323901+0000","scrub_duration":0.0041028669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'49","reported_seq":79,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.716652+0000","last_change":"2024-01-16T15:18:50.514811+0000","last_active":"2024-01-16T15:27:17.716652+0000","last_peered":"2024-01-16T15:27:17.716652+0000","last_clean":"2024-01-16T15:27:17.716652+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:17.716652+0000","last_undegraded":"2024-01-16T15:27:17.716652+0000","last_fullsized":"2024-01-16T15:27:17.716652+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.215187+0000","last_change":"2024-01-16T15:27:42.215234+0000","last_active":"2024-01-16T15:27:42.215187+0000","last_peered":"2024-01-16T15:27:42.215187+0000","last_clean":"2024-01-16T15:27:42.215187+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:27:42.215187+0000","last_undegraded":"2024-01-16T15:27:42.215187+0000","last_fullsized":"2024-01-16T15:27:42.215187+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.215017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:11.615549+0000","scrub_duration":0.0038755600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"115'119","reported_seq":264,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.512603+0000","last_change":"2024-01-16T15:27:33.512658+0000","last_active":"2024-01-16T15:27:33.512603+0000","last_peered":"2024-01-16T15:27:33.512603+0000","last_clean":"2024-01-16T15:27:33.512603+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:27:33.512603+0000","last_undegraded":"2024-01-16T15:27:33.512603+0000","last_fullsized":"2024-01-16T15:27:33.512603+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'119","last_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_deep_scrub":"115'119","last_deep_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.512505+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:30.496871+0000","scrub_duration":0.066194791000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4599,"ondisk_log_size":4599,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1251137,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":490,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2502274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542489,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.222,"15min":1.274},"min":{"1min":0.64800000000000002,"5min":0.54300000000000004,"15min":0.28599999999999998},"max":{"1min":1.4159999999999999,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.302,"15min":1.4399999999999999},"min":{"1min":0.75900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.712,"5min":1.8959999999999999,"15min":25.503},"last":1.3149999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.34499999999999997},"max":{"1min":1.54,"5min":1.5620000000000001,"15min":5.5010000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.1619999999999999,"15min":1.1970000000000001},"min":{"1min":0.65700000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4139999999999999,"5min":1.9490000000000001,"15min":4.3369999999999997},"last":1.0349999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3680000000000001,"15min":1.4019999999999999},"min":{"1min":0.74199999999999999,"5min":0.68400000000000005,"15min":0.629},"max":{"1min":1.669,"5min":2.141,"15min":11.536},"last":1.427},{"interface":"front","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.4670000000000001},"min":{"1min":0.73099999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5309999999999999,"5min":1.8500000000000001,"15min":25.478000000000002},"last":1.667}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.40899999999999997},"max":{"1min":1.4870000000000001,"5min":1.7090000000000001,"15min":5.6369999999999996},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":0.80200000000000005,"5min":0.75900000000000001,"15min":0.67800000000000005},"max":{"1min":1.5589999999999999,"5min":1.9550000000000001,"15min":6.351},"last":1.613}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.204,"15min":1.3919999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6120000000000001,"5min":1.75,"15min":25.401},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6399999999999999,"5min":1.8160000000000001,"15min":5.7859999999999996},"last":1.3819999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3540000000000001,"15min":1.359},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.722,"5min":2.3849999999999998,"15min":4.4539999999999997},"last":1.726},{"interface":"front","average":{"1min":1.385,"5min":1.383,"15min":1.5189999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.708,"5min":1.8480000000000001,"15min":26.327000000000002},"last":1.7629999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5469999999999999,"15min":1.6739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8280000000000001,"5min":2.0579999999999998,"15min":25.451000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5029999999999999,"15min":1.496},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.786,"5min":2.786,"15min":5.8250000000000002},"last":1.546}]}]},{"osd":6,"up_from":14,"seq":60129542488,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.286,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.68400000000000005,"15min":0.35399999999999998},"max":{"1min":1.577,"5min":8.0299999999999994,"15min":29.77},"last":1.141},{"interface":"front","average":{"1min":1.226,"5min":1.3220000000000001,"15min":1.415},"min":{"1min":0.88500000000000001,"5min":0.76400000000000001,"15min":0.48899999999999999},"max":{"1min":1.623,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.0800000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.081,"15min":1.1339999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":1.728,"15min":5.2859999999999996},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.57099999999999995},"max":{"1min":1.595,"5min":1.7350000000000001,"15min":10.494},"last":0.95199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.202,"15min":1.278},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.46,"5min":1.6579999999999999,"15min":5.0549999999999997},"last":1.206},{"interface":"front","average":{"1min":1.389,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.837,"5min":1.9279999999999999,"15min":5.1459999999999999},"last":1.3560000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.319,"15min":1.3089999999999999},"min":{"1min":0.90200000000000002,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":1.601,"5min":1.8360000000000001,"15min":3.8079999999999998},"last":1.26},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.3129999999999999,"15min":1.448},"min":{"1min":0.87,"5min":0.65200000000000002,"15min":0.623},"max":{"1min":1.7130000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.1819999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.3839999999999999,"15min":1.3480000000000001},"min":{"1min":0.97399999999999998,"5min":0.87,"15min":0.53400000000000003},"max":{"1min":1.6890000000000001,"5min":20.422999999999998,"15min":20.422999999999998},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.258,"15min":1.28},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":3.6160000000000001},"last":1.403}]},{"osd":5,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.272,"15min":1.321},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.53700000000000003},"max":{"1min":1.542,"5min":7.6319999999999997,"15min":13.949999999999999},"last":1.321},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.581,"15min":1.5720000000000001},"min":{"1min":1.113,"5min":1.113,"15min":0.51600000000000001},"max":{"1min":1.972,"5min":7.4980000000000002,"15min":11.298},"last":1.464}]},{"osd":7,"last update":"Tue Jan 16 15:26:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4410000000000001,"15min":1.498},"min":{"1min":1.0169999999999999,"5min":0.97499999999999998,"15min":0.57099999999999995},"max":{"1min":1.659,"5min":2.004,"15min":14.037000000000001},"last":1.548},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.512,"15min":1.5720000000000001},"min":{"1min":1.1100000000000001,"5min":1.024,"15min":0.61599999999999999},"max":{"1min":1.9370000000000001,"5min":2.0209999999999999,"15min":4.9829999999999997},"last":1.427}]}]},{"osd":5,"up_from":14,"seq":60129542489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141700,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93230140,"statfs":{"total":96636764160,"available":95467663360,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2090000000000001,"15min":1.1379999999999999},"min":{"1min":0.79300000000000004,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.6890000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.163,"15min":1.147},"min":{"1min":0.89900000000000002,"5min":0.72599999999999998,"15min":0.34799999999999998},"max":{"1min":1.526,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":0.873}]},{"osd":1,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1850000000000001,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.40999999999999998},"max":{"1min":1.776,"5min":1.776,"15min":17.074999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.228,"15min":1.244},"min":{"1min":0.89000000000000001,"5min":0.77100000000000002,"15min":0.46999999999999997},"max":{"1min":1.714,"5min":1.714,"15min":4.8499999999999996},"last":1.1279999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.222,"15min":1.198},"min":{"1min":0.83899999999999997,"5min":0.72599999999999998,"15min":0.54800000000000004},"max":{"1min":1.53,"5min":2.7000000000000002,"15min":4.306},"last":1.476},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2849999999999999,"15min":1.323},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.52200000000000002},"max":{"1min":1.494,"5min":2.9670000000000001,"15min":16.721},"last":1.4399999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.97499999999999998,"5min":0.81499999999999995,"15min":0.60299999999999998},"max":{"1min":1.859,"5min":1.905,"15min":4.8010000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":0.51200000000000001},"max":{"1min":1.7390000000000001,"5min":14.044,"15min":14.044},"last":1.569}]},{"osd":4,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2749999999999999,"15min":1.3260000000000001},"min":{"1min":0.90400000000000003,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.591,"5min":2.0139999999999998,"15min":16.382999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.321,"15min":1.341},"min":{"1min":0.81000000000000005,"5min":0.72099999999999997,"15min":0.502},"max":{"1min":1.982,"5min":1.982,"15min":4.6580000000000004},"last":1.198}]},{"osd":6,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3500000000000001,"15min":1.357},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.39700000000000002},"max":{"1min":1.637,"5min":1.901,"15min":4.3380000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.516,"15min":1.5089999999999999},"min":{"1min":1.137,"5min":0.94199999999999995,"15min":0.57899999999999996},"max":{"1min":1.6919999999999999,"5min":1.907,"15min":3.3719999999999999},"last":1.5269999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5,"15min":1.5029999999999999},"min":{"1min":0.95999999999999996,"5min":0.95599999999999996,"15min":0.45000000000000001},"max":{"1min":2.044,"5min":2.044,"15min":4.742},"last":1.605},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5700000000000001,"15min":1.5509999999999999},"min":{"1min":1.0820000000000001,"5min":1.0629999999999999,"15min":0.58799999999999997},"max":{"1min":1.821,"5min":2.0609999999999999,"15min":2.6520000000000001},"last":1.3799999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542488,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936452,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435388,"statfs":{"total":96636764160,"available":94653837312,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":1.3460000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":0.90500000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.508}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.117},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":0.98599999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.0189999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.605}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.169}]}]},{"osd":3,"up_from":14,"seq":60129542490,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872984,"kb_used_data":1751632,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498856,"statfs":{"total":96636764160,"available":94718828544,"internally_reserved":0,"allocated":1793671168,"data_stored":1792833889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.68899999999999995,"5min":0.58499999999999996,"15min":0.34699999999999998},"max":{"1min":1.3640000000000001,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":1.127},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.099,"15min":1.0760000000000001},"min":{"1min":0.60199999999999998,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.879}]},{"osd":1,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.98199999999999998,"15min":1.0249999999999999},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":1.359,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":1.02},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2569999999999999,"15min":1.226},"min":{"1min":0.64000000000000001,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.7909999999999999,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":1.3340000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.244,"15min":1.26},"min":{"1min":0.72799999999999998,"5min":0.71899999999999997,"15min":0.36699999999999999},"max":{"1min":1.577,"5min":2.258,"15min":3.5249999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.45200000000000001},"max":{"1min":1.897,"5min":2.339,"15min":4.6529999999999996},"last":1.2589999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.323,"15min":1.319},"min":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":0.44500000000000001},"max":{"1min":1.6759999999999999,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.73299999999999998},"max":{"1min":1.861,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.3839999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4890000000000001,"15min":1.5009999999999999},"min":{"1min":1.224,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":1.738,"5min":2.0249999999999999,"15min":4.484},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.4419999999999999,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.66500000000000004},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.365}]},{"osd":6,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5389999999999999,"15min":1.5620000000000001},"min":{"1min":1.1220000000000001,"5min":0.92700000000000005,"15min":0.71499999999999997},"max":{"1min":1.96,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.304},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.6080000000000001,"15min":1.613},"min":{"1min":1.1399999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.069,"5min":2.169,"15min":4.5880000000000001},"last":1.421}]},{"osd":7,"last update":"Tue Jan 16 15:27:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5329999999999999,"15min":1.5509999999999999},"min":{"1min":0.98799999999999999,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":1.8480000000000001,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.236},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.6299999999999999,"15min":1.6279999999999999},"min":{"1min":1.2669999999999999,"5min":1.008,"15min":0.64700000000000002},"max":{"1min":2.1160000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.452}]}]},{"osd":2,"up_from":14,"seq":60129542488,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1235136,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93136704,"statfs":{"total":96636764160,"available":95371984896,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86799999999999999,"15min":0.89800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36099999999999999},"max":{"1min":1.2649999999999999,"5min":1.6659999999999999,"15min":1.96},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0049999999999999},"min":{"1min":0.502,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.9890000000000001},"last":0.83999999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.93200000000000005,"15min":1.004},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.41499999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":2.093},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.196,"5min":1.286,"15min":1.3360000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.51300000000000001},"max":{"1min":1.6419999999999999,"5min":1.786,"15min":2.4039999999999999},"last":1.341}]},{"osd":3,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.54500000000000004},"max":{"1min":1.7,"5min":1.8879999999999999,"15min":3.1080000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.2549999999999999,"15min":1.2889999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6319999999999999,"5min":1.7949999999999999,"15min":2.5070000000000001},"last":1.385}]},{"osd":4,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.409},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.66100000000000003},"max":{"1min":1.6739999999999999,"5min":12.669,"15min":12.669},"last":1.302},{"interface":"front","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.3740000000000001},"min":{"1min":0.89400000000000002,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6699999999999999,"5min":1.752,"15min":2.9079999999999999},"last":1.151}]},{"osd":5,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.415,"15min":1.4390000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.68300000000000005},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.3490000000000002},"last":1.204},{"interface":"front","average":{"1min":1.343,"5min":1.4430000000000001,"15min":1.4950000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.59799999999999998},"max":{"1min":1.6080000000000001,"5min":1.8759999999999999,"15min":2.6589999999999998},"last":1.6799999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.464,"15min":1.5669999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.64500000000000002},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":13.629},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.4279999999999999,"15min":1.518},"min":{"1min":0.80600000000000005,"5min":0.751,"15min":0.629},"max":{"1min":1.7849999999999999,"5min":2.0019999999999998,"15min":15.968999999999999},"last":1.52}]},{"osd":7,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.633,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.61699999999999999},"max":{"1min":1.7450000000000001,"5min":9.3900000000000006,"15min":9.3900000000000006},"last":1.484},{"interface":"front","average":{"1min":1.464,"5min":1.524,"15min":1.623},"min":{"1min":0.94399999999999995,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":1.893,"5min":2.081,"15min":10.455},"last":1.7210000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542488,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1452484,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":115774,"kb_avail":92919356,"statfs":{"total":96636764160,"available":95149420544,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118553029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.90400000000000003,"15min":0.93000000000000005},"min":{"1min":0.64800000000000002,"5min":0.53000000000000003,"15min":0.28399999999999997},"max":{"1min":1.6100000000000001,"5min":1.718,"15min":4.266},"last":0.878},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1220000000000001,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.34999999999999998},"max":{"1min":2.153,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.999}]},{"osd":2,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.137,"15min":1.135},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.31},"max":{"1min":1.647,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1240000000000001,"15min":1.1579999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.39300000000000002},"max":{"1min":1.579,"5min":1.8340000000000001,"15min":4.1559999999999997},"last":0.97299999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.456,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.439},"max":{"1min":1.7529999999999999,"5min":1.9930000000000001,"15min":4.1980000000000004},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.46600000000000003},"max":{"1min":1.798,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":1.4970000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5349999999999999,"15min":1.5109999999999999},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.69899999999999995},"max":{"1min":1.887,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.587},{"interface":"front","average":{"1min":1.298,"5min":1.528,"15min":1.52},"min":{"1min":1.0309999999999999,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":1.613,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.4199999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.522,"15min":1.466},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":1.9219999999999999,"5min":5.194,"15min":5.194},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.5309999999999999,"15min":1.5029999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":1.714,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.2829999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.516,"15min":1.5189999999999999},"min":{"1min":1.0549999999999999,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.544,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.248},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6539999999999999,"15min":1.6080000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.70699999999999996},"max":{"1min":2.0659999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":1.6850000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.609,"15min":1.589},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.69999999999999996},"max":{"1min":1.9690000000000001,"5min":16.140000000000001,"15min":16.140000000000001},"last":1.351},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6759999999999999,"15min":1.6339999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.78700000000000003},"max":{"1min":2.4449999999999998,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.3169999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542489,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073500,"kb_used_data":1997972,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298340,"statfs":{"total":96636764160,"available":94513500160,"internally_reserved":0,"allocated":2045923328,"data_stored":2045082208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0309999999999999,"15min":1.0249999999999999},"min":{"1min":0.55500000000000005,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.3260000000000001,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.2370000000000001,"15min":1.252},"min":{"1min":0.497,"5min":0.47999999999999998,"15min":0.33900000000000002},"max":{"1min":1.4470000000000001,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":1.365}]},{"osd":2,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1120000000000001,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.46700000000000003,"15min":0.40000000000000002},"max":{"1min":1.5800000000000001,"5min":1.875,"15min":7.5940000000000003},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1299999999999999,"15min":1.175},"min":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.39600000000000002},"max":{"1min":1.665,"5min":1.8,"15min":5.3550000000000004},"last":1.141}]},{"osd":3,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.37},"max":{"1min":1.2170000000000001,"5min":5.851,"15min":5.851},"last":1.49},{"interface":"front","average":{"1min":1.335,"5min":1.3169999999999999,"15min":1.343},"min":{"1min":0.65800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.5660000000000001,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":1.24}]},{"osd":4,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.304,"15min":1.3400000000000001},"min":{"1min":0.80400000000000005,"5min":0.79600000000000004,"15min":0.69099999999999995},"max":{"1min":1.7829999999999999,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.4530000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":1.7330000000000001,"5min":2.141,"15min":5.3109999999999999},"last":1.4039999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.4670000000000001,"15min":1.4930000000000001},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.55600000000000005},"max":{"1min":1.75,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.427,"15min":1.454},"min":{"1min":0.96599999999999997,"5min":0.84699999999999998,"15min":0.57999999999999996},"max":{"1min":1.7310000000000001,"5min":2.27,"15min":8.3949999999999996},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5840000000000001,"15min":1.659},"min":{"1min":1.0669999999999999,"5min":0.82299999999999995,"15min":0.72999999999999998},"max":{"1min":1.8049999999999999,"5min":2.173,"15min":15.159000000000001},"last":1.595},{"interface":"front","average":{"1min":1.379,"5min":1.526,"15min":1.6259999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.58999999999999997},"max":{"1min":1.8700000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.4630000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.617,"15min":1.667},"min":{"1min":1,"5min":0.999,"15min":0.60899999999999999},"max":{"1min":1.819,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":1.018,"5min":0.873,"15min":0.65800000000000003},"max":{"1min":1.8340000000000001,"5min":2.2719999999999998,"15min":6.149},"last":1.653}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1246504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1246462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:27:54.093 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.093 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.093 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.093 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.093 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.094 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.095 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.096 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.096 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.096 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.096 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.096 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.097 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.098 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.099 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.100 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:18:48.457576+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=48, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.100 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.100 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.100 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.100 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.101 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:27:54.101 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:28:03.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:03.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:14.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.332+0000 7fe591000640 1 Processor -- start 2024-01-16T15:28:14.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.332+0000 7fe591000640 1 -- start start 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe591000640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe58c1439b0 0x7fe58c0ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe591000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 0x7fe58c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe591000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c15eab0 con 0x7fe58c0cef70 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe591000640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c0cdea0 con 0x7fe58c1439b0 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe591000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c15e220 con 0x7fe58c0ce800 2024-01-16T15:28:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe58b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe58c1439b0 0x7fe58c0ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe58b400640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe58c1439b0 0x7fe58c0ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33046/0 (socket says 172.21.15.43:33046) 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe58b400640 1 -- 172.21.15.43:0/1508418297 learned_addr learned my addr 172.21.15.43:0/1508418297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.334+0000 7fe58be00640 1 --2- 172.21.15.43:0/1508418297 >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 0x7fe58c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.335+0000 7fe58b400640 1 -- 172.21.15.43:0/1508418297 >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 msgr2=0x7fe58c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.335+0000 7fe58b400640 1 --2- 172.21.15.43:0/1508418297 >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 0x7fe58c0712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.335+0000 7fe58b400640 1 -- 172.21.15.43:0/1508418297 >> v1:172.21.15.43:6789/0 conn(0x7fe58c0cef70 legacy=0x7fe58c15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.335+0000 7fe58b400640 1 -- 172.21.15.43:0/1508418297 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c15e4f0 con 0x7fe58c1439b0 2024-01-16T15:28:14.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.335+0000 7fe58be00640 1 --2- 172.21.15.43:0/1508418297 >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 0x7fe58c0712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.336+0000 7fe58b400640 1 --2- 172.21.15.43:0/1508418297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0ce2c0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fe58c071f90 tx=0x7fe57c02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c3c41c77f4b17f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.336+0000 7fe58aa00640 1 -- 172.21.15.43:0/1508418297 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c02f8e0 con 0x7fe58c1439b0 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.336+0000 7fe58aa00640 1 -- 172.21.15.43:0/1508418297 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c02fa40 con 0x7fe58c1439b0 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.336+0000 7fe58aa00640 1 -- 172.21.15.43:0/1508418297 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c02fdd0 con 0x7fe58c1439b0 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 -- 172.21.15.43:0/1508418297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 msgr2=0x7fe58c0ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 --2- 172.21.15.43:0/1508418297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0ce2c0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fe58c071f90 tx=0x7fe57c02db20 comp rx=0 tx=0).stop 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 -- 172.21.15.43:0/1508418297 shutdown_connections 2024-01-16T15:28:14.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 --2- 172.21.15.43:0/1508418297 >> v2:172.21.15.69:3300/0 conn(0x7fe58c0ce800 0x7fe58c0712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 --2- 172.21.15.43:0/1508418297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0ce2c0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 -- 172.21.15.43:0/1508418297 >> 172.21.15.43:0/1508418297 conn(0x7fe58c0cfdc0 msgr2=0x7fe58c0d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:14.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 -- 172.21.15.43:0/1508418297 shutdown_connections 2024-01-16T15:28:14.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.337+0000 7fe591000640 1 -- 172.21.15.43:0/1508418297 wait complete. 2024-01-16T15:28:14.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.339+0000 7fe591000640 1 Processor -- start 2024-01-16T15:28:14.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.339+0000 7fe591000640 1 -- start start 2024-01-16T15:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe58c0cef70 0x7fe58c1399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 0x7fe58c1377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c139ef0 con 0x7fe58c1439b0 2024-01-16T15:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0b4b00 con 0x7fe58c06a6b0 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe591000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0b4d50 con 0x7fe58c0cef70 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe583e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe583e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33050/0 (socket says 172.21.15.43:33050) 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.340+0000 7fe583e00640 1 -- 172.21.15.43:0/2191144284 learned_addr learned my addr 172.21.15.43:0/2191144284 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe58be00640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 0x7fe58c1377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe58b400640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe58c0cef70 0x7fe58c1399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 -- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe58c0cef70 msgr2=0x7fe58c1399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe58c0cef70 0x7fe58c1399b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 -- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 msgr2=0x7fe58c1377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 0x7fe58c1377a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 -- 172.21.15.43:0/2191144284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0094d0 con 0x7fe58c1439b0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe58be00640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 0x7fe58c1377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583e00640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c400 tx=0x7fe57400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.341+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5740052d0 con 0x7fe58c1439b0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.342+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe574005450 con 0x7fe58c1439b0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.342+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574007260 con 0x7fe58c1439b0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.342+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c0b5030 con 0x7fe58c1439b0 2024-01-16T15:28:14.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.342+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c0b55f0 con 0x7fe58c1439b0 2024-01-16T15:28:14.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.343+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c04e840 con 0x7fe58c1439b0 2024-01-16T15:28:14.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.344+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fe5740073c0 con 0x7fe58c1439b0 2024-01-16T15:28:14.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.345+0000 7fe583400640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 0x7fe56406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:14.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.345+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe574088010 con 0x7fe58c1439b0 2024-01-16T15:28:14.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.345+0000 7fe58b400640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 0x7fe56406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.346+0000 7fe58b400640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 0x7fe56406e980 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe57c038000 tx=0x7fe57c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.350+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5740544d0 con 0x7fe58c1439b0 2024-01-16T15:28:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.574+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe58c15b150 con 0x7fe56406c4c0 2024-01-16T15:28:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.584+0000 7fe583400640 1 -- 172.21.15.43:0/2191144284 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291107 (secure 0 0 0) 0x7fe58c15b150 con 0x7fe56406c4c0 2024-01-16T15:28:14.586 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:28:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:28:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 msgr2=0x7fe56406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 0x7fe56406e980 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe57c038000 tx=0x7fe57c008030 comp rx=0 tx=0).stop 2024-01-16T15:28:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 msgr2=0x7fe58c0b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c400 tx=0x7fe57400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 shutdown_connections 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fe56406c4c0 0x7fe56406e980 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe58c06a6b0 0x7fe58c1377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe58c1439b0 0x7fe58c0b45c0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 --2- 172.21.15.43:0/2191144284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe58c0cef70 0x7fe58c1399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.593+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 >> 172.21.15.43:0/2191144284 conn(0x7fe58c0cfdc0 msgr2=0x7fe58c0b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.594+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 shutdown_connections 2024-01-16T15:28:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:14.594+0000 7fe591000640 1 -- 172.21.15.43:0/2191144284 wait complete. 2024-01-16T15:28:14.614 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":975,"stamp":"2024-01-16T15:28:13.521864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444294248,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2254,"num_write_kb":6295372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5983,"ondisk_log_size":5983,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13368656,"kb_used_data":12597776,"kb_used_omap":12,"kb_used_meta":770803,"kb_avail":741606064,"statfs":{"total":773094113280,"available":759404609536,"internally_reserved":0,"allocated":12900122624,"data_stored":12893537648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":789302828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005320"},"pg_stats":[{"pgid":"17.3f","version":"115'70","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.737839+0000","last_change":"2024-01-16T15:18:50.518734+0000","last_active":"2024-01-16T15:27:17.737839+0000","last_peered":"2024-01-16T15:27:17.737839+0000","last_clean":"2024-01-16T15:27:17.737839+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:27:17.737839+0000","last_undegraded":"2024-01-16T15:27:17.737839+0000","last_fullsized":"2024-01-16T15:27:17.737839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'91","reported_seq":132,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.519581+0000","last_change":"2024-01-16T15:27:42.519648+0000","last_active":"2024-01-16T15:27:42.519581+0000","last_peered":"2024-01-16T15:27:42.519581+0000","last_clean":"2024-01-16T15:27:42.519581+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:42.519581+0000","last_undegraded":"2024-01-16T15:27:42.519581+0000","last_fullsized":"2024-01-16T15:27:42.519581+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'91","last_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_deep_scrub":"115'91","last_deep_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.519400+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:45.850266+0000","scrub_duration":0.294982199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'94","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.676749+0000","last_change":"2024-01-16T15:28:05.676824+0000","last_active":"2024-01-16T15:28:05.676749+0000","last_peered":"2024-01-16T15:28:05.676749+0000","last_clean":"2024-01-16T15:28:05.676749+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:28:05.676749+0000","last_undegraded":"2024-01-16T15:28:05.676749+0000","last_fullsized":"2024-01-16T15:28:05.676749+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'94","last_scrub_stamp":"2024-01-16T15:28:05.676547+0000","last_deep_scrub":"115'94","last_deep_scrub_stamp":"2024-01-16T15:28:05.676547+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.676547+0000","objects_scrubbed":27,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:18.032061+0000","scrub_duration":0.34934071700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'66","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733901+0000","last_change":"2024-01-16T15:18:50.523718+0000","last_active":"2024-01-16T15:27:17.733901+0000","last_peered":"2024-01-16T15:27:17.733901+0000","last_clean":"2024-01-16T15:27:17.733901+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:27:17.733901+0000","last_undegraded":"2024-01-16T15:27:17.733901+0000","last_fullsized":"2024-01-16T15:27:17.733901+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'85","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.301392+0000","last_change":"2024-01-16T15:27:59.301435+0000","last_active":"2024-01-16T15:27:59.301392+0000","last_peered":"2024-01-16T15:27:59.301392+0000","last_clean":"2024-01-16T15:27:59.301392+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:59.301392+0000","last_undegraded":"2024-01-16T15:27:59.301392+0000","last_fullsized":"2024-01-16T15:27:59.301392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:27:59.301261+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:27:59.301261+0000","last_clean_scrub_stamp":"2024-01-16T15:27:59.301261+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:28.701793+0000","scrub_duration":0.22163627999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'105","reported_seq":149,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.422788+0000","last_change":"2024-01-16T15:27:49.422842+0000","last_active":"2024-01-16T15:27:49.422788+0000","last_peered":"2024-01-16T15:27:49.422788+0000","last_clean":"2024-01-16T15:27:49.422788+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:49.422788+0000","last_undegraded":"2024-01-16T15:27:49.422788+0000","last_fullsized":"2024-01-16T15:27:49.422788+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'105","last_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_deep_scrub":"115'105","last_deep_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_clean_scrub_stamp":"2024-01-16T15:27:49.422659+0000","objects_scrubbed":31,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:17.171558+0000","scrub_duration":0.31426238000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'84","reported_seq":114,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.394344+0000","last_change":"2024-01-16T15:27:50.394443+0000","last_active":"2024-01-16T15:27:50.394344+0000","last_peered":"2024-01-16T15:27:50.394344+0000","last_clean":"2024-01-16T15:27:50.394344+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:50.394344+0000","last_undegraded":"2024-01-16T15:27:50.394344+0000","last_fullsized":"2024-01-16T15:27:50.394344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'84","last_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_deep_scrub":"115'84","last_deep_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.394094+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:07.403833+0000","scrub_duration":0.22003049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'76","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.566925+0000","last_change":"2024-01-16T15:27:46.566980+0000","last_active":"2024-01-16T15:27:46.566925+0000","last_peered":"2024-01-16T15:27:46.566925+0000","last_clean":"2024-01-16T15:27:46.566925+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:46.566925+0000","last_undegraded":"2024-01-16T15:27:46.566925+0000","last_fullsized":"2024-01-16T15:27:46.566925+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'76","last_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_deep_scrub":"115'76","last_deep_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.566837+0000","objects_scrubbed":22,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:23:01.446658+0000","scrub_duration":0.269082504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'79","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735972+0000","last_change":"2024-01-16T15:18:50.522822+0000","last_active":"2024-01-16T15:27:17.735972+0000","last_peered":"2024-01-16T15:27:17.735972+0000","last_clean":"2024-01-16T15:27:17.735972+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:27:17.735972+0000","last_undegraded":"2024-01-16T15:27:17.735972+0000","last_fullsized":"2024-01-16T15:27:17.735972+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'66","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.733227+0000","last_change":"2024-01-16T15:18:50.509700+0000","last_active":"2024-01-16T15:27:17.733227+0000","last_peered":"2024-01-16T15:27:17.733227+0000","last_clean":"2024-01-16T15:27:17.733227+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:27:17.733227+0000","last_undegraded":"2024-01-16T15:27:17.733227+0000","last_fullsized":"2024-01-16T15:27:17.733227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'97","reported_seq":141,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:56.262110+0000","last_change":"2024-01-16T15:27:56.262162+0000","last_active":"2024-01-16T15:27:56.262110+0000","last_peered":"2024-01-16T15:27:56.262110+0000","last_clean":"2024-01-16T15:27:56.262110+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:56.262110+0000","last_undegraded":"2024-01-16T15:27:56.262110+0000","last_fullsized":"2024-01-16T15:27:56.262110+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:56.261919+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:56.261919+0000","last_clean_scrub_stamp":"2024-01-16T15:27:56.261919+0000","objects_scrubbed":27,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:11.184486+0000","scrub_duration":0.31918302599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'59","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:17.735853+0000","last_change":"2024-01-16T15:18:50.510925+0000","last_active":"2024-01-16T15:27:17.735853+0000","last_peered":"2024-01-16T15:27:17.735853+0000","last_clean":"2024-01-16T15:27:17.735853+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:27:17.735853+0000","last_undegraded":"2024-01-16T15:27:17.735853+0000","last_fullsized":"2024-01-16T15:27:17.735853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'78","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:00.605552+0000","last_change":"2024-01-16T15:28:00.605631+0000","last_active":"2024-01-16T15:28:00.605552+0000","last_peered":"2024-01-16T15:28:00.605552+0000","last_clean":"2024-01-16T15:28:00.605552+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:28:00.605552+0000","last_undegraded":"2024-01-16T15:28:00.605552+0000","last_fullsized":"2024-01-16T15:28:00.605552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'78","last_scrub_stamp":"2024-01-16T15:28:00.605372+0000","last_deep_scrub":"115'78","last_deep_scrub_stamp":"2024-01-16T15:28:00.605372+0000","last_clean_scrub_stamp":"2024-01-16T15:28:00.605372+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:15.934121+0000","scrub_duration":0.220146327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'88","reported_seq":126,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.731333+0000","last_change":"2024-01-16T15:27:37.731333+0000","last_active":"2024-01-16T15:27:37.731333+0000","last_peered":"2024-01-16T15:27:37.731333+0000","last_clean":"2024-01-16T15:27:37.731333+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:37.731333+0000","last_undegraded":"2024-01-16T15:27:37.731333+0000","last_fullsized":"2024-01-16T15:27:37.731333+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.731071+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:51.646988+0000","scrub_duration":0.35245168599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'68","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.237143+0000","last_change":"2024-01-16T15:27:48.237143+0000","last_active":"2024-01-16T15:27:48.237143+0000","last_peered":"2024-01-16T15:27:48.237143+0000","last_clean":"2024-01-16T15:27:48.237143+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:48.237143+0000","last_undegraded":"2024-01-16T15:27:48.237143+0000","last_fullsized":"2024-01-16T15:27:48.237143+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'68","last_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_deep_scrub":"115'68","last_deep_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.236957+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:17.637489+0000","scrub_duration":0.116219171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'74","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.887006+0000","last_change":"2024-01-16T15:27:28.887006+0000","last_active":"2024-01-16T15:27:28.887006+0000","last_peered":"2024-01-16T15:27:28.887006+0000","last_clean":"2024-01-16T15:27:28.887006+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:28.887006+0000","last_undegraded":"2024-01-16T15:27:28.887006+0000","last_fullsized":"2024-01-16T15:27:28.887006+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'74","last_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_deep_scrub":"115'74","last_deep_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_clean_scrub_stamp":"2024-01-16T15:27:28.886880+0000","objects_scrubbed":19,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:33.087313+0000","scrub_duration":0.202995169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'66","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.256801+0000","last_change":"2024-01-16T15:27:50.256801+0000","last_active":"2024-01-16T15:27:50.256801+0000","last_peered":"2024-01-16T15:27:50.256801+0000","last_clean":"2024-01-16T15:27:50.256801+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:50.256801+0000","last_undegraded":"2024-01-16T15:27:50.256801+0000","last_fullsized":"2024-01-16T15:27:50.256801+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'66","last_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_deep_scrub":"115'66","last_deep_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.256459+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:44.950845+0000","scrub_duration":0.21236617099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'107","reported_seq":159,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:47.168707+0000","last_change":"2024-01-16T15:27:47.168707+0000","last_active":"2024-01-16T15:27:47.168707+0000","last_peered":"2024-01-16T15:27:47.168707+0000","last_clean":"2024-01-16T15:27:47.168707+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:47.168707+0000","last_undegraded":"2024-01-16T15:27:47.168707+0000","last_fullsized":"2024-01-16T15:27:47.168707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'107","last_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_deep_scrub":"115'107","last_deep_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_clean_scrub_stamp":"2024-01-16T15:27:47.168437+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:38.535012+0000","scrub_duration":0.41892874400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'100","reported_seq":136,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:06.668445+0000","last_change":"2024-01-16T15:28:06.668512+0000","last_active":"2024-01-16T15:28:06.668445+0000","last_peered":"2024-01-16T15:28:06.668445+0000","last_clean":"2024-01-16T15:28:06.668445+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:28:06.668445+0000","last_undegraded":"2024-01-16T15:28:06.668445+0000","last_fullsized":"2024-01-16T15:28:06.668445+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'100","last_scrub_stamp":"2024-01-16T15:28:06.668233+0000","last_deep_scrub":"115'100","last_deep_scrub_stamp":"2024-01-16T15:28:06.668233+0000","last_clean_scrub_stamp":"2024-01-16T15:28:06.668233+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:08.221222+0000","scrub_duration":0.32354169500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'87","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.480772+0000","last_change":"2024-01-16T15:27:43.480772+0000","last_active":"2024-01-16T15:27:43.480772+0000","last_peered":"2024-01-16T15:27:43.480772+0000","last_clean":"2024-01-16T15:27:43.480772+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:43.480772+0000","last_undegraded":"2024-01-16T15:27:43.480772+0000","last_fullsized":"2024-01-16T15:27:43.480772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'87","last_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_deep_scrub":"115'87","last_deep_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.480597+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:52.268973+0000","scrub_duration":0.29440454500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'103","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.251665+0000","last_change":"2024-01-16T15:28:04.251783+0000","last_active":"2024-01-16T15:28:04.251665+0000","last_peered":"2024-01-16T15:28:04.251665+0000","last_clean":"2024-01-16T15:28:04.251665+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:28:04.251665+0000","last_undegraded":"2024-01-16T15:28:04.251665+0000","last_fullsized":"2024-01-16T15:28:04.251665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'103","last_scrub_stamp":"2024-01-16T15:28:04.251462+0000","last_deep_scrub":"115'103","last_deep_scrub_stamp":"2024-01-16T15:28:04.251462+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.251462+0000","objects_scrubbed":30,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:22.321069+0000","scrub_duration":0.35867113499999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'90","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.342604+0000","last_change":"2024-01-16T15:28:10.342667+0000","last_active":"2024-01-16T15:28:10.342604+0000","last_peered":"2024-01-16T15:28:10.342604+0000","last_clean":"2024-01-16T15:28:10.342604+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:28:10.342604+0000","last_undegraded":"2024-01-16T15:28:10.342604+0000","last_fullsized":"2024-01-16T15:28:10.342604+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'90","last_scrub_stamp":"2024-01-16T15:28:10.342426+0000","last_deep_scrub":"115'90","last_deep_scrub_stamp":"2024-01-16T15:28:10.342426+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.342426+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:14.911068+0000","scrub_duration":0.295467753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'83","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:58.223115+0000","last_change":"2024-01-16T15:27:58.223162+0000","last_active":"2024-01-16T15:27:58.223115+0000","last_peered":"2024-01-16T15:27:58.223115+0000","last_clean":"2024-01-16T15:27:58.223115+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:58.223115+0000","last_undegraded":"2024-01-16T15:27:58.223115+0000","last_fullsized":"2024-01-16T15:27:58.223115+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'83","last_scrub_stamp":"2024-01-16T15:27:58.222980+0000","last_deep_scrub":"115'83","last_deep_scrub_stamp":"2024-01-16T15:27:58.222980+0000","last_clean_scrub_stamp":"2024-01-16T15:27:58.222980+0000","objects_scrubbed":26,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:49.589555+0000","scrub_duration":0.31016252599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'94","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:01.351481+0000","last_change":"2024-01-16T15:28:01.351634+0000","last_active":"2024-01-16T15:28:01.351481+0000","last_peered":"2024-01-16T15:28:01.351481+0000","last_clean":"2024-01-16T15:28:01.351481+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:28:01.351481+0000","last_undegraded":"2024-01-16T15:28:01.351481+0000","last_fullsized":"2024-01-16T15:28:01.351481+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'94","last_scrub_stamp":"2024-01-16T15:28:01.351267+0000","last_deep_scrub":"115'94","last_deep_scrub_stamp":"2024-01-16T15:28:01.351267+0000","last_clean_scrub_stamp":"2024-01-16T15:28:01.351267+0000","objects_scrubbed":24,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:16.680016+0000","scrub_duration":0.25329149400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'97","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.461542+0000","last_change":"2024-01-16T15:27:45.461588+0000","last_active":"2024-01-16T15:27:45.461542+0000","last_peered":"2024-01-16T15:27:45.461542+0000","last_clean":"2024-01-16T15:27:45.461542+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:45.461542+0000","last_undegraded":"2024-01-16T15:27:45.461542+0000","last_fullsized":"2024-01-16T15:27:45.461542+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.461410+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:24.634385+0000","scrub_duration":0.31535565399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'107","reported_seq":144,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.988577+0000","last_change":"2024-01-16T15:28:05.988636+0000","last_active":"2024-01-16T15:28:05.988577+0000","last_peered":"2024-01-16T15:28:05.988577+0000","last_clean":"2024-01-16T15:28:05.988577+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:28:05.988577+0000","last_undegraded":"2024-01-16T15:28:05.988577+0000","last_fullsized":"2024-01-16T15:28:05.988577+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'107","last_scrub_stamp":"2024-01-16T15:28:05.988385+0000","last_deep_scrub":"115'107","last_deep_scrub_stamp":"2024-01-16T15:28:05.988385+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.988385+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:24:27.334330+0000","scrub_duration":0.28793185500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'88","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:09.632615+0000","last_change":"2024-01-16T15:28:09.632675+0000","last_active":"2024-01-16T15:28:09.632615+0000","last_peered":"2024-01-16T15:28:09.632615+0000","last_clean":"2024-01-16T15:28:09.632615+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:28:09.632615+0000","last_undegraded":"2024-01-16T15:28:09.632615+0000","last_fullsized":"2024-01-16T15:28:09.632615+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:28:09.632498+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:28:09.632498+0000","last_clean_scrub_stamp":"2024-01-16T15:28:09.632498+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:32.458242+0000","scrub_duration":0.226358428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'43","reported_seq":66,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138416+0000","last_change":"2024-01-16T15:18:50.516446+0000","last_active":"2024-01-16T15:27:30.138416+0000","last_peered":"2024-01-16T15:27:30.138416+0000","last_clean":"2024-01-16T15:27:30.138416+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:27:30.138416+0000","last_undegraded":"2024-01-16T15:27:30.138416+0000","last_fullsized":"2024-01-16T15:27:30.138416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'77","reported_seq":116,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.242903+0000","last_change":"2024-01-16T15:28:08.242974+0000","last_active":"2024-01-16T15:28:08.242903+0000","last_peered":"2024-01-16T15:28:08.242903+0000","last_clean":"2024-01-16T15:28:08.242903+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:28:08.242903+0000","last_undegraded":"2024-01-16T15:28:08.242903+0000","last_fullsized":"2024-01-16T15:28:08.242903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'77","last_scrub_stamp":"2024-01-16T15:28:08.242792+0000","last_deep_scrub":"115'77","last_deep_scrub_stamp":"2024-01-16T15:28:08.242792+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.242792+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:16:19.148891+0000","scrub_duration":0.25754353299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'104","reported_seq":146,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.544481+0000","last_change":"2024-01-16T15:27:48.544547+0000","last_active":"2024-01-16T15:27:48.544481+0000","last_peered":"2024-01-16T15:27:48.544481+0000","last_clean":"2024-01-16T15:27:48.544481+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:48.544481+0000","last_undegraded":"2024-01-16T15:27:48.544481+0000","last_fullsized":"2024-01-16T15:27:48.544481+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'104","last_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_deep_scrub":"115'104","last_deep_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.544284+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:19.364542+0000","scrub_duration":0.32836960199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'106","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.886489+0000","last_change":"2024-01-16T15:27:33.886536+0000","last_active":"2024-01-16T15:27:33.886489+0000","last_peered":"2024-01-16T15:27:33.886489+0000","last_clean":"2024-01-16T15:27:33.886489+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:33.886489+0000","last_undegraded":"2024-01-16T15:27:33.886489+0000","last_fullsized":"2024-01-16T15:27:33.886489+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'106","last_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_deep_scrub":"115'106","last_deep_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.886354+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:46.223371+0000","scrub_duration":0.231844831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'85","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.353703+0000","last_change":"2024-01-16T15:27:44.353765+0000","last_active":"2024-01-16T15:27:44.353703+0000","last_peered":"2024-01-16T15:27:44.353703+0000","last_clean":"2024-01-16T15:27:44.353703+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:44.353703+0000","last_undegraded":"2024-01-16T15:27:44.353703+0000","last_fullsized":"2024-01-16T15:27:44.353703+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.353606+0000","objects_scrubbed":23,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:15.404103+0000","scrub_duration":0.234855483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'88","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.394246+0000","last_change":"2024-01-16T15:27:51.394300+0000","last_active":"2024-01-16T15:27:51.394246+0000","last_peered":"2024-01-16T15:27:51.394246+0000","last_clean":"2024-01-16T15:27:51.394246+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:51.394246+0000","last_undegraded":"2024-01-16T15:27:51.394246+0000","last_fullsized":"2024-01-16T15:27:51.394246+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:27:51.394095+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:27:51.394095+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.394095+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:49.782161+0000","scrub_duration":0.26988635100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'6","reported_seq":33,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.184374+0000","last_change":"2024-01-16T15:27:41.184433+0000","last_active":"2024-01-16T15:27:41.184374+0000","last_peered":"2024-01-16T15:27:41.184374+0000","last_clean":"2024-01-16T15:27:41.184374+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:27:41.184374+0000","last_undegraded":"2024-01-16T15:27:41.184374+0000","last_fullsized":"2024-01-16T15:27:41.184374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.184165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:01.093220+0000","scrub_duration":0.0077587799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'73","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.981190+0000","last_change":"2024-01-16T15:28:04.981273+0000","last_active":"2024-01-16T15:28:04.981190+0000","last_peered":"2024-01-16T15:28:04.981190+0000","last_clean":"2024-01-16T15:28:04.981190+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:28:04.981190+0000","last_undegraded":"2024-01-16T15:28:04.981190+0000","last_fullsized":"2024-01-16T15:28:04.981190+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'73","last_scrub_stamp":"2024-01-16T15:28:04.981071+0000","last_deep_scrub":"115'73","last_deep_scrub_stamp":"2024-01-16T15:28:04.981071+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.981071+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.654117+0000","scrub_duration":0.258258816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":28,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:36.422396+0000","last_change":"2024-01-16T15:27:36.422452+0000","last_active":"2024-01-16T15:27:36.422396+0000","last_peered":"2024-01-16T15:27:36.422396+0000","last_clean":"2024-01-16T15:27:36.422396+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:27:36.422396+0000","last_undegraded":"2024-01-16T15:27:36.422396+0000","last_fullsized":"2024-01-16T15:27:36.422396+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_clean_scrub_stamp":"2024-01-16T15:27:36.422186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:42.355689+0000","scrub_duration":0.003123125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'97","reported_seq":141,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:52.431987+0000","last_change":"2024-01-16T15:27:52.432060+0000","last_active":"2024-01-16T15:27:52.431987+0000","last_peered":"2024-01-16T15:27:52.431987+0000","last_clean":"2024-01-16T15:27:52.431987+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:52.431987+0000","last_undegraded":"2024-01-16T15:27:52.431987+0000","last_fullsized":"2024-01-16T15:27:52.431987+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:52.431869+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:52.431869+0000","last_clean_scrub_stamp":"2024-01-16T15:27:52.431869+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:21.198854+0000","scrub_duration":0.27025465500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'4","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.217452+0000","last_change":"2024-01-16T15:27:44.217632+0000","last_active":"2024-01-16T15:27:44.217452+0000","last_peered":"2024-01-16T15:27:44.217452+0000","last_clean":"2024-01-16T15:27:44.217452+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:27:44.217452+0000","last_undegraded":"2024-01-16T15:27:44.217452+0000","last_fullsized":"2024-01-16T15:27:44.217452+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.217230+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:49.674761+0000","scrub_duration":0.007544144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'89","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.591288+0000","last_change":"2024-01-16T15:28:08.591424+0000","last_active":"2024-01-16T15:28:08.591288+0000","last_peered":"2024-01-16T15:28:08.591288+0000","last_clean":"2024-01-16T15:28:08.591288+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:28:08.591288+0000","last_undegraded":"2024-01-16T15:28:08.591288+0000","last_fullsized":"2024-01-16T15:28:08.591288+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:08.591109+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:08.591109+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.591109+0000","objects_scrubbed":21,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:15:03.211743+0000","scrub_duration":0.22652739899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'3","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.189552+0000","last_change":"2024-01-16T15:27:38.189613+0000","last_active":"2024-01-16T15:27:38.189552+0000","last_peered":"2024-01-16T15:27:38.189552+0000","last_clean":"2024-01-16T15:27:38.189552+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:27:38.189552+0000","last_undegraded":"2024-01-16T15:27:38.189552+0000","last_fullsized":"2024-01-16T15:27:38.189552+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'3","last_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_deep_scrub":"115'3","last_deep_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:03:26.432118+0000","scrub_duration":0.0070076540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'95","reported_seq":129,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:54.430099+0000","last_change":"2024-01-16T15:27:54.430198+0000","last_active":"2024-01-16T15:27:54.430099+0000","last_peered":"2024-01-16T15:27:54.430099+0000","last_clean":"2024-01-16T15:27:54.430099+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:54.430099+0000","last_undegraded":"2024-01-16T15:27:54.430099+0000","last_fullsized":"2024-01-16T15:27:54.430099+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:54.429931+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:54.429931+0000","last_clean_scrub_stamp":"2024-01-16T15:27:54.429931+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:17.255675+0000","scrub_duration":0.26159354499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.186234+0000","last_change":"2024-01-16T15:27:34.186287+0000","last_active":"2024-01-16T15:27:34.186234+0000","last_peered":"2024-01-16T15:27:34.186234+0000","last_clean":"2024-01-16T15:27:34.186234+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:27:34.186234+0000","last_undegraded":"2024-01-16T15:27:34.186234+0000","last_fullsized":"2024-01-16T15:27:34.186234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.186050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:31:37.071156+0000","scrub_duration":0.0039566280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'97","reported_seq":143,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:09.254999+0000","last_change":"2024-01-16T15:28:09.255056+0000","last_active":"2024-01-16T15:28:09.254999+0000","last_peered":"2024-01-16T15:28:09.254999+0000","last_clean":"2024-01-16T15:28:09.254999+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:28:09.254999+0000","last_undegraded":"2024-01-16T15:28:09.254999+0000","last_fullsized":"2024-01-16T15:28:09.254999+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:28:09.254808+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:28:09.254808+0000","last_clean_scrub_stamp":"2024-01-16T15:28:09.254808+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:15.395955+0000","scrub_duration":0.27471414199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.865012+0000","last_change":"2024-01-16T15:27:39.865110+0000","last_active":"2024-01-16T15:27:39.865012+0000","last_peered":"2024-01-16T15:27:39.865012+0000","last_clean":"2024-01-16T15:27:39.865012+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:27:39.865012+0000","last_undegraded":"2024-01-16T15:27:39.865012+0000","last_fullsized":"2024-01-16T15:27:39.865012+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.864756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:53.780673+0000","scrub_duration":0.0037524390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'95","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.871075+0000","last_change":"2024-01-16T15:27:29.871123+0000","last_active":"2024-01-16T15:27:29.871075+0000","last_peered":"2024-01-16T15:27:29.871075+0000","last_clean":"2024-01-16T15:27:29.871075+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:29.871075+0000","last_undegraded":"2024-01-16T15:27:29.871075+0000","last_fullsized":"2024-01-16T15:27:29.871075+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.871010+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:25.437998+0000","scrub_duration":0.15415410600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.317980+0000","last_change":"2024-01-16T15:27:32.318028+0000","last_active":"2024-01-16T15:27:32.317980+0000","last_peered":"2024-01-16T15:27:32.317980+0000","last_clean":"2024-01-16T15:27:32.317980+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:27:32.317980+0000","last_undegraded":"2024-01-16T15:27:32.317980+0000","last_fullsized":"2024-01-16T15:27:32.317980+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.317792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:01.084778+0000","scrub_duration":0.0035389969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'101","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.595106+0000","last_change":"2024-01-16T15:27:46.595143+0000","last_active":"2024-01-16T15:27:46.595106+0000","last_peered":"2024-01-16T15:27:46.595106+0000","last_clean":"2024-01-16T15:27:46.595106+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:46.595106+0000","last_undegraded":"2024-01-16T15:27:46.595106+0000","last_fullsized":"2024-01-16T15:27:46.595106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'101","last_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_deep_scrub":"115'101","last_deep_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.595008+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:44.983075+0000","scrub_duration":0.33379621399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.688763+0000","last_change":"2024-01-16T15:27:39.688811+0000","last_active":"2024-01-16T15:27:39.688763+0000","last_peered":"2024-01-16T15:27:39.688763+0000","last_clean":"2024-01-16T15:27:39.688763+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:27:39.688763+0000","last_undegraded":"2024-01-16T15:27:39.688763+0000","last_fullsized":"2024-01-16T15:27:39.688763+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.688565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:07.291212+0000","scrub_duration":0.0036420599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'72","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.479079+0000","last_change":"2024-01-16T15:27:43.479158+0000","last_active":"2024-01-16T15:27:43.479079+0000","last_peered":"2024-01-16T15:27:43.479079+0000","last_clean":"2024-01-16T15:27:43.479079+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:43.479079+0000","last_undegraded":"2024-01-16T15:27:43.479079+0000","last_fullsized":"2024-01-16T15:27:43.479079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'72","last_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_deep_scrub":"115'72","last_deep_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.478956+0000","objects_scrubbed":18,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:43.468767+0000","scrub_duration":0.23849152200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.224360+0000","last_change":"2024-01-16T15:27:51.224399+0000","last_active":"2024-01-16T15:27:51.224360+0000","last_peered":"2024-01-16T15:27:51.224360+0000","last_clean":"2024-01-16T15:27:51.224360+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:27:51.224360+0000","last_undegraded":"2024-01-16T15:27:51.224360+0000","last_fullsized":"2024-01-16T15:27:51.224360+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:27:51.224231+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:27:51.224231+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.224231+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:05.632405+0000","scrub_duration":0.0038857200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'81","reported_seq":114,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.907054+0000","last_change":"2024-01-16T15:27:59.907256+0000","last_active":"2024-01-16T15:27:59.907054+0000","last_peered":"2024-01-16T15:27:59.907054+0000","last_clean":"2024-01-16T15:27:59.907054+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:59.907054+0000","last_undegraded":"2024-01-16T15:27:59.907054+0000","last_fullsized":"2024-01-16T15:27:59.907054+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'81","last_scrub_stamp":"2024-01-16T15:27:59.906852+0000","last_deep_scrub":"115'81","last_deep_scrub_stamp":"2024-01-16T15:27:59.906852+0000","last_clean_scrub_stamp":"2024-01-16T15:27:59.906852+0000","objects_scrubbed":20,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:44.027555+0000","scrub_duration":0.23773976699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.836697+0000","last_change":"2024-01-16T15:27:30.836744+0000","last_active":"2024-01-16T15:27:30.836697+0000","last_peered":"2024-01-16T15:27:30.836697+0000","last_clean":"2024-01-16T15:27:30.836697+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:27:30.836697+0000","last_undegraded":"2024-01-16T15:27:30.836697+0000","last_fullsized":"2024-01-16T15:27:30.836697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_clean_scrub_stamp":"2024-01-16T15:27:30.836522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:38.704619+0000","scrub_duration":0.0037552990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'62","reported_seq":96,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138407+0000","last_change":"2024-01-16T15:18:50.521821+0000","last_active":"2024-01-16T15:27:30.138407+0000","last_peered":"2024-01-16T15:27:30.138407+0000","last_clean":"2024-01-16T15:27:30.138407+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:27:30.138407+0000","last_undegraded":"2024-01-16T15:27:30.138407+0000","last_fullsized":"2024-01-16T15:27:30.138407+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'35","reported_seq":61,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.716929+0000","last_change":"2024-01-16T15:27:48.716990+0000","last_active":"2024-01-16T15:27:48.716929+0000","last_peered":"2024-01-16T15:27:48.716929+0000","last_clean":"2024-01-16T15:27:48.716929+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:27:48.716929+0000","last_undegraded":"2024-01-16T15:27:48.716929+0000","last_fullsized":"2024-01-16T15:27:48.716929+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.716639+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:05.125879+0000","scrub_duration":0.0089525939999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'96","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.492103+0000","last_change":"2024-01-16T15:27:51.492171+0000","last_active":"2024-01-16T15:27:51.492103+0000","last_peered":"2024-01-16T15:27:51.492103+0000","last_clean":"2024-01-16T15:27:51.492103+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:51.492103+0000","last_undegraded":"2024-01-16T15:27:51.492103+0000","last_fullsized":"2024-01-16T15:27:51.492103+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'96","last_scrub_stamp":"2024-01-16T15:27:51.491852+0000","last_deep_scrub":"115'96","last_deep_scrub_stamp":"2024-01-16T15:27:51.491852+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.491852+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:15.258818+0000","scrub_duration":0.361982994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'325","reported_seq":352,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.744230+0000","last_change":"2024-01-16T15:27:45.744230+0000","last_active":"2024-01-16T15:27:45.744230+0000","last_peered":"2024-01-16T15:27:45.744230+0000","last_clean":"2024-01-16T15:27:45.744230+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:27:45.744230+0000","last_undegraded":"2024-01-16T15:27:45.744230+0000","last_fullsized":"2024-01-16T15:27:45.744230+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'325","last_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_deep_scrub":"115'325","last_deep_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.743867+0000","objects_scrubbed":4,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:00.095065+0000","scrub_duration":0.018757899000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1246392,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'54","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.138400+0000","last_change":"2024-01-16T15:18:50.500330+0000","last_active":"2024-01-16T15:27:30.138400+0000","last_peered":"2024-01-16T15:27:30.138400+0000","last_clean":"2024-01-16T15:27:30.138400+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:27:30.138400+0000","last_undegraded":"2024-01-16T15:27:30.138400+0000","last_fullsized":"2024-01-16T15:27:30.138400+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.161975+0000","last_change":"2024-01-16T15:27:38.161975+0000","last_active":"2024-01-16T15:27:38.161975+0000","last_peered":"2024-01-16T15:27:38.161975+0000","last_clean":"2024-01-16T15:27:38.161975+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:27:38.161975+0000","last_undegraded":"2024-01-16T15:27:38.161975+0000","last_fullsized":"2024-01-16T15:27:38.161975+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.161723+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:02.215134+0000","scrub_duration":0.0087694660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'95","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.180860+0000","last_change":"2024-01-16T15:27:43.180860+0000","last_active":"2024-01-16T15:27:43.180860+0000","last_peered":"2024-01-16T15:27:43.180860+0000","last_clean":"2024-01-16T15:27:43.180860+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:43.180860+0000","last_undegraded":"2024-01-16T15:27:43.180860+0000","last_fullsized":"2024-01-16T15:27:43.180860+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.180569+0000","objects_scrubbed":25,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:27.476294+0000","scrub_duration":0.28297945600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.754037+0000","last_change":"2024-01-16T15:27:33.754037+0000","last_active":"2024-01-16T15:27:33.754037+0000","last_peered":"2024-01-16T15:27:33.754037+0000","last_clean":"2024-01-16T15:27:33.754037+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:27:33.754037+0000","last_undegraded":"2024-01-16T15:27:33.754037+0000","last_fullsized":"2024-01-16T15:27:33.754037+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.753955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:34:45.427457+0000","scrub_duration":0.0018762620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'89","reported_seq":126,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.343402+0000","last_change":"2024-01-16T15:28:02.343402+0000","last_active":"2024-01-16T15:28:02.343402+0000","last_peered":"2024-01-16T15:28:02.343402+0000","last_clean":"2024-01-16T15:28:02.343402+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:28:02.343402+0000","last_undegraded":"2024-01-16T15:28:02.343402+0000","last_fullsized":"2024-01-16T15:28:02.343402+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:02.343152+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:02.343152+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.343152+0000","objects_scrubbed":26,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:27.860851+0000","scrub_duration":0.28032371900000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'12","reported_seq":39,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.176351+0000","last_change":"2024-01-16T15:27:42.176351+0000","last_active":"2024-01-16T15:27:42.176351+0000","last_peered":"2024-01-16T15:27:42.176351+0000","last_clean":"2024-01-16T15:27:42.176351+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:27:42.176351+0000","last_undegraded":"2024-01-16T15:27:42.176351+0000","last_fullsized":"2024-01-16T15:27:42.176351+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.176093+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:16.846949+0000","scrub_duration":0.010433985999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.135131+0000","last_change":"2024-01-16T15:28:08.135214+0000","last_active":"2024-01-16T15:28:08.135131+0000","last_peered":"2024-01-16T15:28:08.135131+0000","last_clean":"2024-01-16T15:28:08.135131+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:28:08.135131+0000","last_undegraded":"2024-01-16T15:28:08.135131+0000","last_fullsized":"2024-01-16T15:28:08.135131+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'61","last_scrub_stamp":"2024-01-16T15:28:08.134967+0000","last_deep_scrub":"115'61","last_deep_scrub_stamp":"2024-01-16T15:28:08.134967+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.134967+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:06.523034+0000","scrub_duration":0.15542793599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'2","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.722441+0000","last_change":"2024-01-16T15:28:02.722549+0000","last_active":"2024-01-16T15:28:02.722441+0000","last_peered":"2024-01-16T15:28:02.722441+0000","last_clean":"2024-01-16T15:28:02.722441+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:28:02.722441+0000","last_undegraded":"2024-01-16T15:28:02.722441+0000","last_fullsized":"2024-01-16T15:28:02.722441+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-01-16T15:28:02.722230+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-01-16T15:28:02.722230+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.722230+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:29.591650+0000","scrub_duration":0.0073352740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'61","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.665483+0000","last_change":"2024-01-16T15:18:50.520753+0000","last_active":"2024-01-16T15:27:28.665483+0000","last_peered":"2024-01-16T15:27:28.665483+0000","last_clean":"2024-01-16T15:27:28.665483+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:27:28.665483+0000","last_undegraded":"2024-01-16T15:27:28.665483+0000","last_fullsized":"2024-01-16T15:27:28.665483+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'3","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:57.156886+0000","last_change":"2024-01-16T15:27:57.156950+0000","last_active":"2024-01-16T15:27:57.156886+0000","last_peered":"2024-01-16T15:27:57.156886+0000","last_clean":"2024-01-16T15:27:57.156886+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:27:57.156886+0000","last_undegraded":"2024-01-16T15:27:57.156886+0000","last_fullsized":"2024-01-16T15:27:57.156886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'3","last_scrub_stamp":"2024-01-16T15:27:57.156695+0000","last_deep_scrub":"115'3","last_deep_scrub_stamp":"2024-01-16T15:27:57.156695+0000","last_clean_scrub_stamp":"2024-01-16T15:27:57.156695+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:17.065751+0000","scrub_duration":0.0077654519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'75","reported_seq":115,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.463446+0000","last_change":"2024-01-16T15:27:50.463446+0000","last_active":"2024-01-16T15:27:50.463446+0000","last_peered":"2024-01-16T15:27:50.463446+0000","last_clean":"2024-01-16T15:27:50.463446+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:50.463446+0000","last_undegraded":"2024-01-16T15:27:50.463446+0000","last_fullsized":"2024-01-16T15:27:50.463446+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'75","last_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_deep_scrub":"115'75","last_deep_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.463136+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:21.882678+0000","scrub_duration":0.16385127299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'2","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.734212+0000","last_change":"2024-01-16T15:27:38.734212+0000","last_active":"2024-01-16T15:27:38.734212+0000","last_peered":"2024-01-16T15:27:38.734212+0000","last_clean":"2024-01-16T15:27:38.734212+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:27:38.734212+0000","last_undegraded":"2024-01-16T15:27:38.734212+0000","last_fullsized":"2024-01-16T15:27:38.734212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.733970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:33.428355+0000","scrub_duration":0.0076776479999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'89","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.652891+0000","last_change":"2024-01-16T15:28:07.652940+0000","last_active":"2024-01-16T15:28:07.652891+0000","last_peered":"2024-01-16T15:28:07.652891+0000","last_clean":"2024-01-16T15:28:07.652891+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:28:07.652891+0000","last_undegraded":"2024-01-16T15:28:07.652891+0000","last_fullsized":"2024-01-16T15:28:07.652891+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:07.652775+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:07.652775+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.652775+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:32.819607+0000","scrub_duration":0.33515691600000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.144273+0000","last_change":"2024-01-16T15:27:39.144273+0000","last_active":"2024-01-16T15:27:39.144273+0000","last_peered":"2024-01-16T15:27:39.144273+0000","last_clean":"2024-01-16T15:27:39.144273+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:27:39.144273+0000","last_undegraded":"2024-01-16T15:27:39.144273+0000","last_fullsized":"2024-01-16T15:27:39.144273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.144028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.164911+0000","scrub_duration":0.0035817119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'82","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.596918+0000","last_change":"2024-01-16T15:27:45.596964+0000","last_active":"2024-01-16T15:27:45.596918+0000","last_peered":"2024-01-16T15:27:45.596918+0000","last_clean":"2024-01-16T15:27:45.596918+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:45.596918+0000","last_undegraded":"2024-01-16T15:27:45.596918+0000","last_fullsized":"2024-01-16T15:27:45.596918+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'82","last_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_deep_scrub":"115'82","last_deep_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.596772+0000","objects_scrubbed":26,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:29.892497+0000","scrub_duration":0.31446099100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.232655+0000","last_change":"2024-01-16T15:27:34.232655+0000","last_active":"2024-01-16T15:27:34.232655+0000","last_peered":"2024-01-16T15:27:34.232655+0000","last_clean":"2024-01-16T15:27:34.232655+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:27:34.232655+0000","last_undegraded":"2024-01-16T15:27:34.232655+0000","last_fullsized":"2024-01-16T15:27:34.232655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.232419+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:43.752231+0000","scrub_duration":0.0069378510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'96","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.010959+0000","last_change":"2024-01-16T15:28:05.011042+0000","last_active":"2024-01-16T15:28:05.010959+0000","last_peered":"2024-01-16T15:28:05.010959+0000","last_clean":"2024-01-16T15:28:05.010959+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:28:05.010959+0000","last_undegraded":"2024-01-16T15:28:05.010959+0000","last_fullsized":"2024-01-16T15:28:05.010959+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'96","last_scrub_stamp":"2024-01-16T15:28:05.010776+0000","last_deep_scrub":"115'96","last_deep_scrub_stamp":"2024-01-16T15:28:05.010776+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.010776+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:06:05.660319+0000","scrub_duration":0.329058395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.164374+0000","last_change":"2024-01-16T15:27:39.164427+0000","last_active":"2024-01-16T15:27:39.164374+0000","last_peered":"2024-01-16T15:27:39.164374+0000","last_clean":"2024-01-16T15:27:39.164374+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:27:39.164374+0000","last_undegraded":"2024-01-16T15:27:39.164374+0000","last_fullsized":"2024-01-16T15:27:39.164374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.164189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:40.647599+0000","scrub_duration":0.003563882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'90","reported_seq":123,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.605238+0000","last_change":"2024-01-16T15:28:02.605402+0000","last_active":"2024-01-16T15:28:02.605238+0000","last_peered":"2024-01-16T15:28:02.605238+0000","last_clean":"2024-01-16T15:28:02.605238+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:28:02.605238+0000","last_undegraded":"2024-01-16T15:28:02.605238+0000","last_fullsized":"2024-01-16T15:28:02.605238+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'90","last_scrub_stamp":"2024-01-16T15:28:02.605035+0000","last_deep_scrub":"115'90","last_deep_scrub_stamp":"2024-01-16T15:28:02.605035+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.605035+0000","objects_scrubbed":25,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:12.399323+0000","scrub_duration":0.28498842000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.193731+0000","last_change":"2024-01-16T15:27:29.193731+0000","last_active":"2024-01-16T15:27:29.193731+0000","last_peered":"2024-01-16T15:27:29.193731+0000","last_clean":"2024-01-16T15:27:29.193731+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:27:29.193731+0000","last_undegraded":"2024-01-16T15:27:29.193731+0000","last_fullsized":"2024-01-16T15:27:29.193731+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.193592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:03.864449+0000","scrub_duration":0.0028384030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'77","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.205484+0000","last_change":"2024-01-16T15:27:48.205536+0000","last_active":"2024-01-16T15:27:48.205484+0000","last_peered":"2024-01-16T15:27:48.205484+0000","last_clean":"2024-01-16T15:27:48.205484+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:48.205484+0000","last_undegraded":"2024-01-16T15:27:48.205484+0000","last_fullsized":"2024-01-16T15:27:48.205484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'77","last_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_deep_scrub":"115'77","last_deep_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.205363+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:54.141377+0000","scrub_duration":0.17343320200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.694453+0000","last_change":"2024-01-16T15:27:41.694504+0000","last_active":"2024-01-16T15:27:41.694453+0000","last_peered":"2024-01-16T15:27:41.694453+0000","last_clean":"2024-01-16T15:27:41.694453+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:27:41.694453+0000","last_undegraded":"2024-01-16T15:27:41.694453+0000","last_fullsized":"2024-01-16T15:27:41.694453+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.694252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.406823+0000","scrub_duration":0.0042882689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'70","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:55.894850+0000","last_change":"2024-01-16T15:27:55.894929+0000","last_active":"2024-01-16T15:27:55.894850+0000","last_peered":"2024-01-16T15:27:55.894850+0000","last_clean":"2024-01-16T15:27:55.894850+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:55.894850+0000","last_undegraded":"2024-01-16T15:27:55.894850+0000","last_fullsized":"2024-01-16T15:27:55.894850+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'70","last_scrub_stamp":"2024-01-16T15:27:55.894683+0000","last_deep_scrub":"115'70","last_deep_scrub_stamp":"2024-01-16T15:27:55.894683+0000","last_clean_scrub_stamp":"2024-01-16T15:27:55.894683+0000","objects_scrubbed":17,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:59:09.195148+0000","scrub_duration":0.187917204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.867500+0000","last_change":"2024-01-16T15:27:38.867614+0000","last_active":"2024-01-16T15:27:38.867500+0000","last_peered":"2024-01-16T15:27:38.867500+0000","last_clean":"2024-01-16T15:27:38.867500+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:27:38.867500+0000","last_undegraded":"2024-01-16T15:27:38.867500+0000","last_fullsized":"2024-01-16T15:27:38.867500+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.867296+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:02.481138+0000","scrub_duration":0.0076157439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.373381+0000","last_change":"2024-01-16T15:19:15.372470+0000","last_active":"2024-01-16T15:27:29.373381+0000","last_peered":"2024-01-16T15:27:29.373381+0000","last_clean":"2024-01-16T15:27:29.373381+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:27:29.373381+0000","last_undegraded":"2024-01-16T15:27:29.373381+0000","last_fullsized":"2024-01-16T15:27:29.373381+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:18:49.470527+0000","last_clean_scrub_stamp":"2024-01-16T15:18:49.470527+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'31","reported_seq":58,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.825906+0000","last_change":"2024-01-16T15:27:42.826005+0000","last_active":"2024-01-16T15:27:42.825906+0000","last_peered":"2024-01-16T15:27:42.825906+0000","last_clean":"2024-01-16T15:27:42.825906+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:42.825906+0000","last_undegraded":"2024-01-16T15:27:42.825906+0000","last_fullsized":"2024-01-16T15:27:42.825906+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'31","last_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_deep_scrub":"115'31","last_deep_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.825641+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:57.748209+0000","scrub_duration":0.0051599990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":649,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'99","reported_seq":146,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.093416+0000","last_change":"2024-01-16T15:28:10.093493+0000","last_active":"2024-01-16T15:28:10.093416+0000","last_peered":"2024-01-16T15:28:10.093416+0000","last_clean":"2024-01-16T15:28:10.093416+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:28:10.093416+0000","last_undegraded":"2024-01-16T15:28:10.093416+0000","last_fullsized":"2024-01-16T15:28:10.093416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'99","last_scrub_stamp":"2024-01-16T15:28:10.093323+0000","last_deep_scrub":"115'99","last_deep_scrub_stamp":"2024-01-16T15:28:10.093323+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.093323+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:46.031990+0000","scrub_duration":0.29833682099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.735910+0000","last_change":"2024-01-16T15:27:51.735960+0000","last_active":"2024-01-16T15:27:51.735910+0000","last_peered":"2024-01-16T15:27:51.735910+0000","last_clean":"2024-01-16T15:27:51.735910+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:27:51.735910+0000","last_undegraded":"2024-01-16T15:27:51.735910+0000","last_fullsized":"2024-01-16T15:27:51.735910+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:51.735728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:51.735728+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.735728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:56:58.848218+0000","scrub_duration":0.0037530269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'87","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.039019+0000","last_change":"2024-01-16T15:28:04.039156+0000","last_active":"2024-01-16T15:28:04.039019+0000","last_peered":"2024-01-16T15:28:04.039019+0000","last_clean":"2024-01-16T15:28:04.039019+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:28:04.039019+0000","last_undegraded":"2024-01-16T15:28:04.039019+0000","last_fullsized":"2024-01-16T15:28:04.039019+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'87","last_scrub_stamp":"2024-01-16T15:28:04.038791+0000","last_deep_scrub":"115'87","last_deep_scrub_stamp":"2024-01-16T15:28:04.038791+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.038791+0000","objects_scrubbed":28,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:59.941576+0000","scrub_duration":0.313418537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.818788+0000","last_change":"2024-01-16T15:27:32.818822+0000","last_active":"2024-01-16T15:27:32.818788+0000","last_peered":"2024-01-16T15:27:32.818788+0000","last_clean":"2024-01-16T15:27:32.818788+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:27:32.818788+0000","last_undegraded":"2024-01-16T15:27:32.818788+0000","last_fullsized":"2024-01-16T15:27:32.818788+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.818678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:29.803044+0000","scrub_duration":0.0024947659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'83","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:40.537839+0000","last_change":"2024-01-16T15:27:40.537917+0000","last_active":"2024-01-16T15:27:40.537839+0000","last_peered":"2024-01-16T15:27:40.537839+0000","last_clean":"2024-01-16T15:27:40.537839+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:40.537839+0000","last_undegraded":"2024-01-16T15:27:40.537839+0000","last_fullsized":"2024-01-16T15:27:40.537839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'83","last_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_deep_scrub":"115'83","last_deep_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_clean_scrub_stamp":"2024-01-16T15:27:40.537681+0000","objects_scrubbed":21,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:15.744820+0000","scrub_duration":0.26540904500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.219632+0000","last_change":"2024-01-16T15:27:33.219680+0000","last_active":"2024-01-16T15:27:33.219632+0000","last_peered":"2024-01-16T15:27:33.219632+0000","last_clean":"2024-01-16T15:27:33.219632+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:27:33.219632+0000","last_undegraded":"2024-01-16T15:27:33.219632+0000","last_fullsized":"2024-01-16T15:27:33.219632+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.219447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:42.739258+0000","scrub_duration":0.0037869280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'93","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:00.165917+0000","last_change":"2024-01-16T15:28:00.166107+0000","last_active":"2024-01-16T15:28:00.165917+0000","last_peered":"2024-01-16T15:28:00.165917+0000","last_clean":"2024-01-16T15:28:00.165917+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:28:00.165917+0000","last_undegraded":"2024-01-16T15:28:00.165917+0000","last_fullsized":"2024-01-16T15:28:00.165917+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'93","last_scrub_stamp":"2024-01-16T15:28:00.165611+0000","last_deep_scrub":"115'93","last_deep_scrub_stamp":"2024-01-16T15:28:00.165611+0000","last_clean_scrub_stamp":"2024-01-16T15:28:00.165611+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:25.295005+0000","scrub_duration":0.29624051800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.251350+0000","last_change":"2024-01-16T15:27:39.251405+0000","last_active":"2024-01-16T15:27:39.251350+0000","last_peered":"2024-01-16T15:27:39.251350+0000","last_clean":"2024-01-16T15:27:39.251350+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:27:39.251350+0000","last_undegraded":"2024-01-16T15:27:39.251350+0000","last_fullsized":"2024-01-16T15:27:39.251350+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.251139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:04.028594+0000","scrub_duration":0.0041990689999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'97","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:01.720216+0000","last_change":"2024-01-16T15:28:01.720275+0000","last_active":"2024-01-16T15:28:01.720216+0000","last_peered":"2024-01-16T15:28:01.720216+0000","last_clean":"2024-01-16T15:28:01.720216+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:28:01.720216+0000","last_undegraded":"2024-01-16T15:28:01.720216+0000","last_fullsized":"2024-01-16T15:28:01.720216+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:28:01.720058+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:28:01.720058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:01.720058+0000","objects_scrubbed":29,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:17:26.844383+0000","scrub_duration":0.351489783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'35","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.186518+0000","last_change":"2024-01-16T15:27:43.186575+0000","last_active":"2024-01-16T15:27:43.186518+0000","last_peered":"2024-01-16T15:27:43.186518+0000","last_clean":"2024-01-16T15:27:43.186518+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:27:43.186518+0000","last_undegraded":"2024-01-16T15:27:43.186518+0000","last_fullsized":"2024-01-16T15:27:43.186518+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.186344+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:34.244119+0000","scrub_duration":0.007088273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'102","reported_seq":142,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:03.935495+0000","last_change":"2024-01-16T15:28:03.935528+0000","last_active":"2024-01-16T15:28:03.935495+0000","last_peered":"2024-01-16T15:28:03.935495+0000","last_clean":"2024-01-16T15:28:03.935495+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:28:03.935495+0000","last_undegraded":"2024-01-16T15:28:03.935495+0000","last_fullsized":"2024-01-16T15:28:03.935495+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'102","last_scrub_stamp":"2024-01-16T15:28:03.935418+0000","last_deep_scrub":"115'102","last_deep_scrub_stamp":"2024-01-16T15:28:03.935418+0000","last_clean_scrub_stamp":"2024-01-16T15:28:03.935418+0000","objects_scrubbed":32,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:34.857974+0000","scrub_duration":0.26924512099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.623423+0000","last_change":"2024-01-16T15:27:37.623518+0000","last_active":"2024-01-16T15:27:37.623423+0000","last_peered":"2024-01-16T15:27:37.623423+0000","last_clean":"2024-01-16T15:27:37.623423+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:27:37.623423+0000","last_undegraded":"2024-01-16T15:27:37.623423+0000","last_fullsized":"2024-01-16T15:27:37.623423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.623212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:31.323901+0000","scrub_duration":0.0041028669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'66","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:54.208127+0000","last_change":"2024-01-16T15:27:54.208127+0000","last_active":"2024-01-16T15:27:54.208127+0000","last_peered":"2024-01-16T15:27:54.208127+0000","last_clean":"2024-01-16T15:27:54.208127+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:54.208127+0000","last_undegraded":"2024-01-16T15:27:54.208127+0000","last_fullsized":"2024-01-16T15:27:54.208127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'66","last_scrub_stamp":"2024-01-16T15:27:54.207809+0000","last_deep_scrub":"115'66","last_deep_scrub_stamp":"2024-01-16T15:27:54.207809+0000","last_clean_scrub_stamp":"2024-01-16T15:27:54.207809+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:31.713234+0000","scrub_duration":0.19203378300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.215234+0000","last_change":"2024-01-16T15:27:42.215234+0000","last_active":"2024-01-16T15:27:42.215234+0000","last_peered":"2024-01-16T15:27:42.215234+0000","last_clean":"2024-01-16T15:27:42.215234+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:27:42.215234+0000","last_undegraded":"2024-01-16T15:27:42.215234+0000","last_fullsized":"2024-01-16T15:27:42.215234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.215017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:11.615549+0000","scrub_duration":0.0038755600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"115'119","reported_seq":264,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.512603+0000","last_change":"2024-01-16T15:27:33.512658+0000","last_active":"2024-01-16T15:27:33.512603+0000","last_peered":"2024-01-16T15:27:33.512603+0000","last_clean":"2024-01-16T15:27:33.512603+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:27:33.512603+0000","last_undegraded":"2024-01-16T15:27:33.512603+0000","last_fullsized":"2024-01-16T15:27:33.512603+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'119","last_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_deep_scrub":"115'119","last_deep_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.512505+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:30.496871+0000","scrub_duration":0.066194791000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5390,"ondisk_log_size":5390,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1251137,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":490,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2502274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.228,"15min":1.2749999999999999},"min":{"1min":0.79500000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9530000000000001,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":1.325},{"interface":"front","average":{"1min":1.347,"5min":1.3129999999999999,"15min":1.365},"min":{"1min":0.95399999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":7.2409999999999997},"last":1.3779999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.139,"15min":1.1459999999999999},"min":{"1min":0.73399999999999999,"5min":0.59999999999999998,"15min":0.55100000000000005},"max":{"1min":1.5169999999999999,"5min":1.5620000000000001,"15min":1.792},"last":1.444},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.151,"15min":1.1870000000000001},"min":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.798,"5min":1.9490000000000001,"15min":3.504},"last":1.1799999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.349,"15min":1.397},"min":{"1min":0.68899999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.073,"5min":2.141,"15min":11.536},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.306,"15min":1.3859999999999999},"min":{"1min":0.83199999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.869,"5min":1.869,"15min":11.576000000000001},"last":1.653}]},{"osd":3,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.246,"15min":1.242},"min":{"1min":0.90800000000000003,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.133},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.379,"15min":1.3799999999999999},"min":{"1min":0.98199999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.181,"5min":2.181,"15min":3.6829999999999998},"last":1.49}]},{"osd":4,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2150000000000001,"15min":1.3049999999999999},"min":{"1min":0.84399999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":11.837},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3069999999999999,"15min":1.3129999999999999},"min":{"1min":1.056,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.234}]},{"osd":5,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.3999999999999999,"15min":1.3680000000000001},"min":{"1min":1.1930000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.254,"5min":2.3849999999999998,"15min":3.4199999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":0.95099999999999996,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":6.8719999999999999},"last":1.5289999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.5740000000000001,"15min":1.6020000000000001},"min":{"1min":1.1060000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":6.9690000000000003},"last":1.778},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.5229999999999999,"15min":1.5009999999999999},"min":{"1min":1.3320000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":1.121}]}]},{"osd":6,"up_from":14,"seq":60129542492,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1531844,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":92839996,"statfs":{"total":96636764160,"available":95068155904,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.28,"15min":1.3169999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.887,"5min":8.0299999999999994,"15min":29.77},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.3089999999999999,"15min":1.4119999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.681,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.472}]},{"osd":1,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.099,"15min":1.1359999999999999},"min":{"1min":0.82999999999999996,"5min":0.67900000000000005,"15min":0.65800000000000003},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":5.2859999999999996},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.083,"5min":1.1539999999999999,"15min":1.1899999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.57099999999999995},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":10.494},"last":1.526}]},{"osd":2,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":5.0549999999999997},"last":1.141},{"interface":"front","average":{"1min":1.431,"5min":1.4079999999999999,"15min":1.454},"min":{"1min":1.0089999999999999,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":5.1459999999999999},"last":1.4259999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3149999999999999,"15min":1.3140000000000001},"min":{"1min":0.84999999999999998,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":3.8079999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3,"15min":1.4470000000000001},"min":{"1min":0.70999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.7390000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.1950000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.4810000000000001,"15min":1.373},"min":{"1min":0.91400000000000003,"5min":0.90700000000000003,"15min":0.53400000000000003},"max":{"1min":9.4130000000000003,"5min":20.422999999999998,"15min":20.422999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.369,"5min":1.29,"15min":1.29},"min":{"1min":1.0589999999999999,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":3.0219999999999998,"5min":3.0219999999999998,"15min":3.6160000000000001},"last":1.2609999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.302,"15min":1.327},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.8129999999999999,"5min":7.6319999999999997,"15min":13.949999999999999},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5920000000000001,"15min":1.5760000000000001},"min":{"1min":1.087,"5min":1.087,"15min":0.51600000000000001},"max":{"1min":2.9780000000000002,"5min":7.4980000000000002,"15min":11.298},"last":1.6259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4379999999999999,"15min":1.498},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.71499999999999997},"max":{"1min":1.837,"5min":2.004,"15min":14.037000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.5189999999999999,"15min":1.5760000000000001},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.7210000000000001,"5min":2.7210000000000001,"15min":4.9829999999999997},"last":1.329}]}]},{"osd":5,"up_from":14,"seq":60129542493,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1145860,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93225980,"statfs":{"total":96636764160,"available":95463403520,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.194,"15min":1.141},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.57399999999999995},"max":{"1min":1.7390000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.089},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.147},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5660000000000001,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.149}]},{"osd":1,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1839999999999999,"15min":1.2130000000000001},"min":{"1min":0.76800000000000002,"5min":0.69499999999999995,"15min":0.65200000000000002},"max":{"1min":2.5,"5min":2.5,"15min":17.074999999999999},"last":1.288},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2370000000000001,"15min":1.2450000000000001},"min":{"1min":0.77500000000000002,"5min":0.77100000000000002,"15min":0.745},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":4.8499999999999996},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.2210000000000001,"15min":1.1970000000000001},"min":{"1min":0.82199999999999995,"5min":0.72599999999999998,"15min":0.61099999999999999},"max":{"1min":1.9370000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2769999999999999,"15min":1.278},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":3.306,"5min":3.306,"15min":3.9590000000000001},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.347,"15min":1.3839999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":4.8010000000000002},"last":1.355},{"interface":"front","average":{"1min":1.375,"5min":1.5920000000000001,"15min":1.4590000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.72399999999999998},"max":{"1min":2.2360000000000002,"5min":14.044,"15min":14.044},"last":1.5029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.25,"15min":1.2809999999999999},"min":{"1min":0.628,"5min":0.58699999999999997,"15min":0.56699999999999995},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":4.681},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3069999999999999,"15min":1.343},"min":{"1min":0.503,"5min":0.503,"15min":0.502},"max":{"1min":1.6299999999999999,"5min":1.982,"15min":4.6580000000000004},"last":1.468}]},{"osd":6,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.347,"15min":1.355},"min":{"1min":0.86199999999999999,"5min":0.82999999999999996,"15min":0.52400000000000002},"max":{"1min":1.5569999999999999,"5min":1.847,"15min":2.7599999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.496,"15min":1.5109999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8180000000000001,"5min":1.907,"15min":3.3719999999999999},"last":1.5529999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.512,"15min":1.5169999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.59099999999999997},"max":{"1min":4.8390000000000004,"5min":4.8390000000000004,"15min":4.8390000000000004},"last":1.631},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.571,"15min":1.5620000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.61099999999999999},"max":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"last":1.591}]}]},{"osd":4,"up_from":14,"seq":60129542492,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936452,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435388,"statfs":{"total":96636764160,"available":94653837312,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1559999999999999,"15min":1.129},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.839,"5min":1.839,"15min":6.4240000000000004},"last":1.03},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1850000000000001,"15min":1.139},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7949999999999999,"5min":1.8620000000000001,"15min":1.9810000000000001},"last":1.081}]},{"osd":1,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3939999999999999,"15min":1.46},"min":{"1min":0.874,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":24.399999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3280000000000001,"15min":1.397},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":24.456},"last":1.228}]},{"osd":2,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.141},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.306,"15min":1.304},"min":{"1min":0.95099999999999996,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.7350000000000001,"5min":1.7989999999999999,"15min":2.1400000000000001},"last":1.321}]},{"osd":3,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.319,"15min":1.306},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.379},{"interface":"front","average":{"1min":1.327,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.032,"5min":0.94999999999999996,"15min":0.71899999999999997},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.357}]},{"osd":5,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.367,"15min":1.3859999999999999},"min":{"1min":0.87,"5min":0.84899999999999998,"15min":0.40600000000000003},"max":{"1min":1.782,"5min":1.8520000000000001,"15min":2.238},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.363,"15min":1.4259999999999999},"min":{"1min":0.84599999999999997,"5min":0.75,"15min":0.496},"max":{"1min":1.927,"5min":1.927,"15min":6.2990000000000004},"last":1.536}]},{"osd":6,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2809999999999999,"15min":1.3080000000000001},"min":{"1min":0.66000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.48,"5min":2.48,"15min":3.0190000000000001},"last":1.266},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.407,"15min":1.4279999999999999},"min":{"1min":0.88,"5min":0.80000000000000004,"15min":0.58199999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.1830000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:15 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.462,"15min":1.518},"min":{"1min":0.79000000000000004,"5min":0.78000000000000003,"15min":0.61799999999999999},"max":{"1min":2.0129999999999999,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5,"15min":1.5389999999999999},"min":{"1min":1.04,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.653,"5min":2.653,"15min":3.3439999999999999},"last":1.4370000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542494,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872984,"kb_used_data":1751632,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498856,"statfs":{"total":96636764160,"available":94718828544,"internally_reserved":0,"allocated":1793671168,"data_stored":1792833889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91000000000000003,"15min":0.94599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.34699999999999998},"max":{"1min":1.127,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.077,"15min":1.083},"min":{"1min":0.628,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.837,"5min":1.905,"15min":1.905},"last":1.621}]},{"osd":1,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.998,"15min":1.046},"min":{"1min":0.72999999999999998,"5min":0.61399999999999999,"15min":0.371},"max":{"1min":1.718,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.67000000000000004,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.76,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":1.181}]},{"osd":2,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.228,"15min":1.2709999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.45500000000000002},"max":{"1min":1.5880000000000001,"5min":2.258,"15min":3.5249999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.3560000000000001,"15min":1.395},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.45200000000000001},"max":{"1min":1.766,"5min":2.339,"15min":4.6529999999999996},"last":1.5589999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.3009999999999999,"15min":1.3240000000000001},"min":{"1min":0.95899999999999996,"5min":0.94399999999999995,"15min":0.70999999999999996},"max":{"1min":1.7490000000000001,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.123},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.444},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.73299999999999998},"max":{"1min":1.633,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.5009999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4770000000000001,"15min":1.5089999999999999},"min":{"1min":0.97499999999999998,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":2.169,"5min":2.169,"15min":4.484},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.452,"15min":1.4390000000000001},"min":{"1min":1.0129999999999999,"5min":0.92900000000000005,"15min":0.753},"max":{"1min":1.9810000000000001,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.4390000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.5309999999999999,"15min":1.5740000000000001},"min":{"1min":0.95499999999999996,"5min":0.92700000000000005,"15min":0.77200000000000002},"max":{"1min":2.157,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.589,"15min":1.621},"min":{"1min":1.1699999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.0600000000000001,"5min":2.169,"15min":4.5880000000000001},"last":1.7430000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.538,"15min":1.554},"min":{"1min":0.93899999999999995,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":2.0649999999999999,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.641,"15min":1.645},"min":{"1min":1.2969999999999999,"5min":1.008,"15min":0.78700000000000003},"max":{"1min":2.2210000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.677}]}]},{"osd":2,"up_from":14,"seq":60129542492,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1235136,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93136704,"statfs":{"total":96636764160,"available":95371984896,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86799999999999999,"15min":0.89800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36099999999999999},"max":{"1min":1.2649999999999999,"5min":1.6659999999999999,"15min":1.96},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0049999999999999},"min":{"1min":0.502,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.9890000000000001},"last":1.117}]},{"osd":1,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.93200000000000005,"15min":1.004},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.41499999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":2.093},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.196,"5min":1.286,"15min":1.3360000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.51300000000000001},"max":{"1min":1.6419999999999999,"5min":1.786,"15min":2.4039999999999999},"last":1.863}]},{"osd":3,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.54500000000000004},"max":{"1min":1.7,"5min":1.8879999999999999,"15min":3.1080000000000001},"last":1.504},{"interface":"front","average":{"1min":1.123,"5min":1.2549999999999999,"15min":1.2889999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6319999999999999,"5min":1.7949999999999999,"15min":2.5070000000000001},"last":1.284}]},{"osd":4,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.409},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.66100000000000003},"max":{"1min":1.6739999999999999,"5min":12.669,"15min":12.669},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.3740000000000001},"min":{"1min":0.89400000000000002,"5min":0.78300000000000003,"15min":0.51200000000000001},"max":{"1min":1.6699999999999999,"5min":1.752,"15min":2.9079999999999999},"last":1.351}]},{"osd":5,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.415,"15min":1.4390000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.68300000000000005},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.3490000000000002},"last":1.198},{"interface":"front","average":{"1min":1.343,"5min":1.4430000000000001,"15min":1.4950000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.59799999999999998},"max":{"1min":1.6080000000000001,"5min":1.8759999999999999,"15min":2.6589999999999998},"last":1.4610000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.464,"15min":1.5669999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.64500000000000002},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":13.629},"last":1.73},{"interface":"front","average":{"1min":1.446,"5min":1.4279999999999999,"15min":1.518},"min":{"1min":0.80600000000000005,"5min":0.751,"15min":0.629},"max":{"1min":1.7849999999999999,"5min":2.0019999999999998,"15min":15.968999999999999},"last":1.6220000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:25 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.633,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.61699999999999999},"max":{"1min":1.7450000000000001,"5min":9.3900000000000006,"15min":9.3900000000000006},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.524,"15min":1.623},"min":{"1min":0.94399999999999995,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":1.893,"5min":2.081,"15min":10.455},"last":1.4059999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542492,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1435844,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":92935996,"statfs":{"total":96636764160,"available":95166459904,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89500000000000002,"15min":0.92000000000000004},"min":{"1min":0.54900000000000004,"5min":0.53000000000000003,"15min":0.40600000000000003},"max":{"1min":1.2529999999999999,"5min":1.718,"15min":1.865},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.115,"15min":1.0740000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.38100000000000001},"max":{"1min":1.8879999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":1.379}]},{"osd":2,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.1319999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.39800000000000002},"max":{"1min":5.1749999999999998,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.139,"15min":1.1599999999999999},"min":{"1min":0.78400000000000003,"5min":0.55900000000000005,"15min":0.51200000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.306},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4399999999999999,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.76800000000000002,"15min":0.53100000000000003},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.522},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3720000000000001,"15min":1.369},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.49099999999999999},"max":{"1min":2.1890000000000001,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":1.5509999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5529999999999999,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":0.97699999999999998,"15min":0.95299999999999996},"max":{"1min":2.516,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.5669999999999999,"15min":1.5269999999999999},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":5.6539999999999999,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.484}]},{"osd":5,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5740000000000001,"15min":1.464},"min":{"1min":0.98399999999999999,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":5.4530000000000003,"5min":5.4530000000000003,"15min":5.4530000000000003},"last":1.9510000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5469999999999999,"15min":1.5029999999999999},"min":{"1min":1.0409999999999999,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":5.5949999999999998,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.325}]},{"osd":6,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.504,"15min":1.51},"min":{"1min":1.145,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.1150000000000002,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.6759999999999999,"15min":1.6020000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.70699999999999996},"max":{"1min":2.3929999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":1.835}]},{"osd":7,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.657,"15min":1.5840000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.69999999999999996},"max":{"1min":5.6879999999999997,"5min":16.140000000000001,"15min":16.140000000000001},"last":1.756},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.6739999999999999,"15min":1.615},"min":{"1min":1.091,"5min":0.86399999999999999,"15min":0.83799999999999997},"max":{"1min":2.0209999999999999,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.619}]}]},{"osd":0,"up_from":14,"seq":60129542493,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073500,"kb_used_data":1997972,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298340,"statfs":{"total":96636764160,"available":94513500160,"internally_reserved":0,"allocated":2045923328,"data_stored":2045082208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0209999999999999,"15min":1.0249999999999999},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.4039999999999999,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2609999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.33900000000000002},"max":{"1min":1.591,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.105,"15min":1.139},"min":{"1min":0.439,"5min":0.439,"15min":0.40600000000000003},"max":{"1min":1.694,"5min":1.875,"15min":5.0570000000000004},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.097,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.39600000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":5.3550000000000004},"last":1.131}]},{"osd":3,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.097,"15min":1.105},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.37},"max":{"1min":1.49,"5min":5.851,"15min":5.851},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.302,"15min":1.3440000000000001},"min":{"1min":0.53800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.7929999999999999,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":1.5329999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2989999999999999,"15min":1.345},"min":{"1min":0.79900000000000004,"5min":0.79600000000000004,"15min":0.72199999999999998},"max":{"1min":1.645,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":1.323},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4370000000000001,"15min":1.46},"min":{"1min":0.91800000000000004,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":2.0880000000000001,"5min":2.141,"15min":5.3109999999999999},"last":2.4409999999999998}]},{"osd":5,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.456,"15min":1.5029999999999999},"min":{"1min":0.93899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.804,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":1.286},{"interface":"front","average":{"1min":1.369,"5min":1.4179999999999999,"15min":1.4370000000000001},"min":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.59199999999999997},"max":{"1min":1.8959999999999999,"5min":2.27,"15min":7.2699999999999996},"last":1.5640000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.569,"15min":1.6599999999999999},"min":{"1min":1.054,"5min":0.82299999999999995,"15min":0.79800000000000004},"max":{"1min":1.9710000000000001,"5min":2.173,"15min":15.159000000000001},"last":2.363},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.524,"15min":1.6319999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.621},"max":{"1min":2.0270000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.367}]},{"osd":7,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.6240000000000001,"15min":1.677},"min":{"1min":1.0489999999999999,"5min":0.999,"15min":0.66600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5209999999999999,"15min":1.5820000000000001},"min":{"1min":0.97999999999999998,"5min":0.873,"15min":0.81200000000000006},"max":{"1min":2.605,"5min":2.605,"15min":6.149},"last":1.486}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1246504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1246462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:14.620 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.620 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.620 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.620 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.621 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.621 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.621 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.622 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.622 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.622 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:18:49.470527+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=27, tm_sec=26, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:14.622 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:28:33.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:33.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:34.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:34.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.855+0000 7f6f4ac00640 1 Processor -- start 2024-01-16T15:28:34.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.855+0000 7f6f4ac00640 1 -- start start 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f4ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f4ac00640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6f44070cd0 0x7f6f4406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f4ac00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f44071cd0 con 0x7f6f44160970 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f4ac00640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f440b2120 con 0x7f6f44070cd0 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f4ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f440b2400 con 0x7f6f44070780 2024-01-16T15:28:34.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f43e00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6f44160970 0x7f6f440b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48156/0 (socket says 172.21.15.43:48156) 2024-01-16T15:28:34.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.857+0000 7f6f43e00640 1 -- 172.21.15.43:0/3154765167 learned_addr learned my addr 172.21.15.43:0/3154765167 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:28:34.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3154765167 >> v2:172.21.15.43:3300/0 conn(0x7f6f44070cd0 0x7f6f4406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f3be00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f43400640 1 -- 172.21.15.43:0/3154765167 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522284166 0 0) 0x7f6f44071cd0 con 0x7f6f44160970 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f43400640 1 -- 172.21.15.43:0/3154765167 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f24003520 con 0x7f6f44160970 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f48c00640 1 -- 172.21.15.43:0/3154765167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 msgr2=0x7f6f4406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f48c00640 1 -- 172.21.15.43:0/3154765167 >> v1:172.21.15.43:6789/0 conn(0x7f6f44160970 legacy=0x7f6f440b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:34.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.858+0000 7f6f48c00640 1 -- 172.21.15.43:0/3154765167 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f440b26e0 con 0x7f6f44070cd0 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.859+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44070cd0 0x7f6f4406da60 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c009d90 tx=0x7f6f3c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb448734b33716e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.860+0000 7f6f43400640 1 -- 172.21.15.43:0/3154765167 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c02cbc0 con 0x7f6f44070cd0 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.860+0000 7f6f43400640 1 -- 172.21.15.43:0/3154765167 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c02cd20 con 0x7f6f44070cd0 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.860+0000 7f6f43400640 1 -- 172.21.15.43:0/3154765167 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c034660 con 0x7f6f44070cd0 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.860+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3154765167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44070cd0 msgr2=0x7f6f4406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.860+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44070cd0 0x7f6f4406da60 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c009d90 tx=0x7f6f3c02be10 comp rx=0 tx=0).stop 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3154765167 shutdown_connections 2024-01-16T15:28:34.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44070cd0 0x7f6f4406da60 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3154765167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3154765167 >> 172.21.15.43:0/3154765167 conn(0x7f6f440b0d80 msgr2=0x7f6f440b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3154765167 shutdown_connections 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.861+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3154765167 wait complete. 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.862+0000 7f6f4ac00640 1 Processor -- start 2024-01-16T15:28:34.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.863+0000 7f6f4ac00640 1 -- start start 2024-01-16T15:28:34.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4409a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 0x7f6f441546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4409a6b0 con 0x7f6f44160970 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4409bf30 con 0x7f6f44070cd0 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f4ac00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4409c210 con 0x7f6f44070780 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47676/0 (socket says 172.21.15.43:47676) 2024-01-16T15:28:34.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 -- 172.21.15.43:0/3102239379 learned_addr learned my addr 172.21.15.43:0/3102239379 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f3be00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 0x7f6f441546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f43e00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4409a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 -- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 msgr2=0x7f6f4409a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4409a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 -- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 msgr2=0x7f6f441546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 0x7f6f441546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f48c00640 1 -- 172.21.15.43:0/3102239379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c009a70 con 0x7f6f44160970 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f3be00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 0x7f6f441546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.864+0000 7f6f43e00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4409a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f48c00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c009d90 tx=0x7f6f3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c040650 con 0x7f6f44160970 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c02ce90 con 0x7f6f44160970 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0409f0 con 0x7f6f44160970 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4409c4f0 con 0x7f6f44160970 2024-01-16T15:28:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.865+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4409cab0 con 0x7f6f44160970 2024-01-16T15:28:34.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.866+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4409cca0 con 0x7f6f44160970 2024-01-16T15:28:34.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.867+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 54) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6f3c040b50 con 0x7f6f44160970 2024-01-16T15:28:34.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.868+0000 7f6f40c00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 0x7f6f1c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.868+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6f3c0ab6e0 con 0x7f6f44160970 2024-01-16T15:28:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.869+0000 7f6f43e00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 0x7f6f1c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.869+0000 7f6f43e00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 0x7f6f1c06e8f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6f340011e0 tx=0x7f6f34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:34.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:34.873+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f3c077c50 con 0x7f6f44160970 2024-01-16T15:28:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.067+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 --> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f44071fb0 con 0x7f6f1c06c430 2024-01-16T15:28:35.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.090+0000 7f6f40c00640 1 -- 172.21.15.43:0/3102239379 <== mgr.4104 v2:172.21.15.43:6832/504016423 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291791 (secure 0 0 0) 0x7f6f34001c80 con 0x7f6f1c06c430 2024-01-16T15:28:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:28:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 msgr2=0x7f6f1c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 0x7f6f1c06e8f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6f340011e0 tx=0x7f6f34009040 comp rx=0 tx=0).stop 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 msgr2=0x7f6f440adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c009d90 tx=0x7f6f3c004a40 comp rx=0 tx=0).stop 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 shutdown_connections 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7f6f1c06c430 0x7f6f1c06e8f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f44070cd0 0x7f6f441546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f44160970 0x7f6f440adb80 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 --2- 172.21.15.43:0/3102239379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f44070780 0x7f6f4409a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.095+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 >> 172.21.15.43:0/3102239379 conn(0x7f6f440b0d80 msgr2=0x7f6f440b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.096+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 shutdown_connections 2024-01-16T15:28:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.096+0000 7f6f4ac00640 1 -- 172.21.15.43:0/3102239379 wait complete. 2024-01-16T15:28:35.118 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":985,"stamp":"2024-01-16T15:28:33.530156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6444294437,"num_objects":1570,"num_object_clones":1,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":1,"num_read":948,"num_read_kb":3145892,"num_write":2256,"num_write_kb":6295374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6226,"ondisk_log_size":6226,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":13372816,"kb_used_data":12597776,"kb_used_omap":12,"kb_used_meta":774963,"kb_avail":741601904,"statfs":{"total":773094113280,"available":759400349696,"internally_reserved":0,"allocated":12900122624,"data_stored":12893538026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":793562668},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005238"},"pg_stats":[{"pgid":"17.3f","version":"115'98","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.091653+0000","last_change":"2024-01-16T15:28:16.091716+0000","last_active":"2024-01-16T15:28:16.091653+0000","last_peered":"2024-01-16T15:28:16.091653+0000","last_clean":"2024-01-16T15:28:16.091653+0000","last_became_active":"2024-01-16T15:18:50.518397+0000","last_became_peered":"2024-01-16T15:18:50.518397+0000","last_unstale":"2024-01-16T15:28:16.091653+0000","last_undegraded":"2024-01-16T15:28:16.091653+0000","last_fullsized":"2024-01-16T15:28:16.091653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'98","last_scrub_stamp":"2024-01-16T15:28:16.091479+0000","last_deep_scrub":"115'98","last_deep_scrub_stamp":"2024-01-16T15:28:16.091479+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.091479+0000","objects_scrubbed":28,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:40.040308+0000","scrub_duration":0.306108135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"115'91","reported_seq":132,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.519581+0000","last_change":"2024-01-16T15:27:42.519648+0000","last_active":"2024-01-16T15:27:42.519581+0000","last_peered":"2024-01-16T15:27:42.519581+0000","last_clean":"2024-01-16T15:27:42.519581+0000","last_became_active":"2024-01-16T15:18:50.514606+0000","last_became_peered":"2024-01-16T15:18:50.514606+0000","last_unstale":"2024-01-16T15:27:42.519581+0000","last_undegraded":"2024-01-16T15:27:42.519581+0000","last_fullsized":"2024-01-16T15:27:42.519581+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'91","last_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_deep_scrub":"115'91","last_deep_scrub_stamp":"2024-01-16T15:27:42.519400+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.519400+0000","objects_scrubbed":26,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:45.850266+0000","scrub_duration":0.294982199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"115'94","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.676749+0000","last_change":"2024-01-16T15:28:05.676824+0000","last_active":"2024-01-16T15:28:05.676749+0000","last_peered":"2024-01-16T15:28:05.676749+0000","last_clean":"2024-01-16T15:28:05.676749+0000","last_became_active":"2024-01-16T15:18:50.529777+0000","last_became_peered":"2024-01-16T15:18:50.529777+0000","last_unstale":"2024-01-16T15:28:05.676749+0000","last_undegraded":"2024-01-16T15:28:05.676749+0000","last_fullsized":"2024-01-16T15:28:05.676749+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'94","last_scrub_stamp":"2024-01-16T15:28:05.676547+0000","last_deep_scrub":"115'94","last_deep_scrub_stamp":"2024-01-16T15:28:05.676547+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.676547+0000","objects_scrubbed":27,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:18.032061+0000","scrub_duration":0.34934071700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":31,"num_write_kb":110594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"115'91","reported_seq":129,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.046182+0000","last_change":"2024-01-16T15:28:18.046242+0000","last_active":"2024-01-16T15:28:18.046182+0000","last_peered":"2024-01-16T15:28:18.046182+0000","last_clean":"2024-01-16T15:28:18.046182+0000","last_became_active":"2024-01-16T15:18:50.521798+0000","last_became_peered":"2024-01-16T15:18:50.521798+0000","last_unstale":"2024-01-16T15:28:18.046182+0000","last_undegraded":"2024-01-16T15:28:18.046182+0000","last_fullsized":"2024-01-16T15:28:18.046182+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'91","last_scrub_stamp":"2024-01-16T15:28:18.045969+0000","last_deep_scrub":"115'91","last_deep_scrub_stamp":"2024-01-16T15:28:18.045969+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.045969+0000","objects_scrubbed":25,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:38:14.425718+0000","scrub_duration":0.29247292400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"115'85","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.301392+0000","last_change":"2024-01-16T15:27:59.301435+0000","last_active":"2024-01-16T15:27:59.301392+0000","last_peered":"2024-01-16T15:27:59.301392+0000","last_clean":"2024-01-16T15:27:59.301392+0000","last_became_active":"2024-01-16T15:18:50.505761+0000","last_became_peered":"2024-01-16T15:18:50.505761+0000","last_unstale":"2024-01-16T15:27:59.301392+0000","last_undegraded":"2024-01-16T15:27:59.301392+0000","last_fullsized":"2024-01-16T15:27:59.301392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:27:59.301261+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:27:59.301261+0000","last_clean_scrub_stamp":"2024-01-16T15:27:59.301261+0000","objects_scrubbed":25,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:28.701793+0000","scrub_duration":0.22163627999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663488,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"115'105","reported_seq":149,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.422788+0000","last_change":"2024-01-16T15:27:49.422842+0000","last_active":"2024-01-16T15:27:49.422788+0000","last_peered":"2024-01-16T15:27:49.422788+0000","last_clean":"2024-01-16T15:27:49.422788+0000","last_became_active":"2024-01-16T15:18:50.517578+0000","last_became_peered":"2024-01-16T15:18:50.517578+0000","last_unstale":"2024-01-16T15:27:49.422788+0000","last_undegraded":"2024-01-16T15:27:49.422788+0000","last_fullsized":"2024-01-16T15:27:49.422788+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'105","last_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_deep_scrub":"115'105","last_deep_scrub_stamp":"2024-01-16T15:27:49.422659+0000","last_clean_scrub_stamp":"2024-01-16T15:27:49.422659+0000","objects_scrubbed":31,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:17.171558+0000","scrub_duration":0.31426238000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":31,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"115'84","reported_seq":114,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.394344+0000","last_change":"2024-01-16T15:27:50.394443+0000","last_active":"2024-01-16T15:27:50.394344+0000","last_peered":"2024-01-16T15:27:50.394344+0000","last_clean":"2024-01-16T15:27:50.394344+0000","last_became_active":"2024-01-16T15:18:50.505387+0000","last_became_peered":"2024-01-16T15:18:50.505387+0000","last_unstale":"2024-01-16T15:27:50.394344+0000","last_undegraded":"2024-01-16T15:27:50.394344+0000","last_fullsized":"2024-01-16T15:27:50.394344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'84","last_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_deep_scrub":"115'84","last_deep_scrub_stamp":"2024-01-16T15:27:50.394094+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.394094+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:38:07.403833+0000","scrub_duration":0.22003049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"115'76","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.566925+0000","last_change":"2024-01-16T15:27:46.566980+0000","last_active":"2024-01-16T15:27:46.566925+0000","last_peered":"2024-01-16T15:27:46.566925+0000","last_clean":"2024-01-16T15:27:46.566925+0000","last_became_active":"2024-01-16T15:18:50.512177+0000","last_became_peered":"2024-01-16T15:18:50.512177+0000","last_unstale":"2024-01-16T15:27:46.566925+0000","last_undegraded":"2024-01-16T15:27:46.566925+0000","last_fullsized":"2024-01-16T15:27:46.566925+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'76","last_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_deep_scrub":"115'76","last_deep_scrub_stamp":"2024-01-16T15:27:46.566837+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.566837+0000","objects_scrubbed":22,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:23:01.446658+0000","scrub_duration":0.269082504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"115'106","reported_seq":144,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.056172+0000","last_change":"2024-01-16T15:28:14.056172+0000","last_active":"2024-01-16T15:28:14.056172+0000","last_peered":"2024-01-16T15:28:14.056172+0000","last_clean":"2024-01-16T15:28:14.056172+0000","last_became_active":"2024-01-16T15:18:50.522074+0000","last_became_peered":"2024-01-16T15:18:50.522074+0000","last_unstale":"2024-01-16T15:28:14.056172+0000","last_undegraded":"2024-01-16T15:28:14.056172+0000","last_fullsized":"2024-01-16T15:28:14.056172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'106","last_scrub_stamp":"2024-01-16T15:28:14.056017+0000","last_deep_scrub":"115'106","last_deep_scrub_stamp":"2024-01-16T15:28:14.056017+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.056017+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:04.851508+0000","scrub_duration":0.25377765400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052165,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"115'88","reported_seq":128,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:12.020971+0000","last_change":"2024-01-16T15:28:12.020971+0000","last_active":"2024-01-16T15:28:12.020971+0000","last_peered":"2024-01-16T15:28:12.020971+0000","last_clean":"2024-01-16T15:28:12.020971+0000","last_became_active":"2024-01-16T15:18:50.509416+0000","last_became_peered":"2024-01-16T15:18:50.509416+0000","last_unstale":"2024-01-16T15:28:12.020971+0000","last_undegraded":"2024-01-16T15:28:12.020971+0000","last_fullsized":"2024-01-16T15:28:12.020971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:28:12.020779+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:28:12.020779+0000","last_clean_scrub_stamp":"2024-01-16T15:28:12.020779+0000","objects_scrubbed":22,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:11.681629+0000","scrub_duration":0.21611398000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"115'97","reported_seq":142,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:56.262162+0000","last_change":"2024-01-16T15:27:56.262162+0000","last_active":"2024-01-16T15:27:56.262162+0000","last_peered":"2024-01-16T15:27:56.262162+0000","last_clean":"2024-01-16T15:27:56.262162+0000","last_became_active":"2024-01-16T15:18:50.522094+0000","last_became_peered":"2024-01-16T15:18:50.522094+0000","last_unstale":"2024-01-16T15:27:56.262162+0000","last_undegraded":"2024-01-16T15:27:56.262162+0000","last_fullsized":"2024-01-16T15:27:56.262162+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:56.261919+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:56.261919+0000","last_clean_scrub_stamp":"2024-01-16T15:27:56.261919+0000","objects_scrubbed":27,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:26:11.184486+0000","scrub_duration":0.31918302599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":18,"num_read_kb":73728,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"115'84","reported_seq":132,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:13.067597+0000","last_change":"2024-01-16T15:28:13.067597+0000","last_active":"2024-01-16T15:28:13.067597+0000","last_peered":"2024-01-16T15:28:13.067597+0000","last_clean":"2024-01-16T15:28:13.067597+0000","last_became_active":"2024-01-16T15:18:50.510633+0000","last_became_peered":"2024-01-16T15:18:50.510633+0000","last_unstale":"2024-01-16T15:28:13.067597+0000","last_undegraded":"2024-01-16T15:28:13.067597+0000","last_fullsized":"2024-01-16T15:28:13.067597+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'84","last_scrub_stamp":"2024-01-16T15:28:13.067349+0000","last_deep_scrub":"115'84","last_deep_scrub_stamp":"2024-01-16T15:28:13.067349+0000","last_clean_scrub_stamp":"2024-01-16T15:28:13.067349+0000","objects_scrubbed":25,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:05.150289+0000","scrub_duration":0.26289675600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":21,"num_read_kb":81921,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"115'78","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:00.605552+0000","last_change":"2024-01-16T15:28:00.605631+0000","last_active":"2024-01-16T15:28:00.605552+0000","last_peered":"2024-01-16T15:28:00.605552+0000","last_clean":"2024-01-16T15:28:00.605552+0000","last_became_active":"2024-01-16T15:18:50.520945+0000","last_became_peered":"2024-01-16T15:18:50.520945+0000","last_unstale":"2024-01-16T15:28:00.605552+0000","last_undegraded":"2024-01-16T15:28:00.605552+0000","last_fullsized":"2024-01-16T15:28:00.605552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'78","last_scrub_stamp":"2024-01-16T15:28:00.605372+0000","last_deep_scrub":"115'78","last_deep_scrub_stamp":"2024-01-16T15:28:00.605372+0000","last_clean_scrub_stamp":"2024-01-16T15:28:00.605372+0000","objects_scrubbed":21,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:15.934121+0000","scrub_duration":0.220146327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"115'88","reported_seq":126,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.731333+0000","last_change":"2024-01-16T15:27:37.731333+0000","last_active":"2024-01-16T15:27:37.731333+0000","last_peered":"2024-01-16T15:27:37.731333+0000","last_clean":"2024-01-16T15:27:37.731333+0000","last_became_active":"2024-01-16T15:18:50.516896+0000","last_became_peered":"2024-01-16T15:18:50.516896+0000","last_unstale":"2024-01-16T15:27:37.731333+0000","last_undegraded":"2024-01-16T15:27:37.731333+0000","last_fullsized":"2024-01-16T15:27:37.731333+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:27:37.731071+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.731071+0000","objects_scrubbed":28,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:51.646988+0000","scrub_duration":0.35245168599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"115'68","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.237143+0000","last_change":"2024-01-16T15:27:48.237143+0000","last_active":"2024-01-16T15:27:48.237143+0000","last_peered":"2024-01-16T15:27:48.237143+0000","last_clean":"2024-01-16T15:27:48.237143+0000","last_became_active":"2024-01-16T15:18:50.516919+0000","last_became_peered":"2024-01-16T15:18:50.516919+0000","last_unstale":"2024-01-16T15:27:48.237143+0000","last_undegraded":"2024-01-16T15:27:48.237143+0000","last_fullsized":"2024-01-16T15:27:48.237143+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'68","last_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_deep_scrub":"115'68","last_deep_scrub_stamp":"2024-01-16T15:27:48.236957+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.236957+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:17.637489+0000","scrub_duration":0.116219171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"115'74","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:28.887006+0000","last_change":"2024-01-16T15:27:28.887006+0000","last_active":"2024-01-16T15:27:28.887006+0000","last_peered":"2024-01-16T15:27:28.887006+0000","last_clean":"2024-01-16T15:27:28.887006+0000","last_became_active":"2024-01-16T15:18:50.515662+0000","last_became_peered":"2024-01-16T15:18:50.515662+0000","last_unstale":"2024-01-16T15:27:28.887006+0000","last_undegraded":"2024-01-16T15:27:28.887006+0000","last_fullsized":"2024-01-16T15:27:28.887006+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'74","last_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_deep_scrub":"115'74","last_deep_scrub_stamp":"2024-01-16T15:27:28.886880+0000","last_clean_scrub_stamp":"2024-01-16T15:27:28.886880+0000","objects_scrubbed":19,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:33.087313+0000","scrub_duration":0.202995169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"115'66","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.256801+0000","last_change":"2024-01-16T15:27:50.256801+0000","last_active":"2024-01-16T15:27:50.256801+0000","last_peered":"2024-01-16T15:27:50.256801+0000","last_clean":"2024-01-16T15:27:50.256801+0000","last_became_active":"2024-01-16T15:18:50.512751+0000","last_became_peered":"2024-01-16T15:18:50.512751+0000","last_unstale":"2024-01-16T15:27:50.256801+0000","last_undegraded":"2024-01-16T15:27:50.256801+0000","last_fullsized":"2024-01-16T15:27:50.256801+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'66","last_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_deep_scrub":"115'66","last_deep_scrub_stamp":"2024-01-16T15:27:50.256459+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.256459+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:44.950845+0000","scrub_duration":0.21236617099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"115'107","reported_seq":159,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:47.168707+0000","last_change":"2024-01-16T15:27:47.168707+0000","last_active":"2024-01-16T15:27:47.168707+0000","last_peered":"2024-01-16T15:27:47.168707+0000","last_clean":"2024-01-16T15:27:47.168707+0000","last_became_active":"2024-01-16T15:18:50.511977+0000","last_became_peered":"2024-01-16T15:18:50.511977+0000","last_unstale":"2024-01-16T15:27:47.168707+0000","last_undegraded":"2024-01-16T15:27:47.168707+0000","last_fullsized":"2024-01-16T15:27:47.168707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'107","last_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_deep_scrub":"115'107","last_deep_scrub_stamp":"2024-01-16T15:27:47.168437+0000","last_clean_scrub_stamp":"2024-01-16T15:27:47.168437+0000","objects_scrubbed":35,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:38.535012+0000","scrub_duration":0.41892874400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":25,"num_read_kb":86020,"num_write":39,"num_write_kb":143362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"115'100","reported_seq":136,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:06.668445+0000","last_change":"2024-01-16T15:28:06.668512+0000","last_active":"2024-01-16T15:28:06.668445+0000","last_peered":"2024-01-16T15:28:06.668445+0000","last_clean":"2024-01-16T15:28:06.668445+0000","last_became_active":"2024-01-16T15:18:50.527079+0000","last_became_peered":"2024-01-16T15:18:50.527079+0000","last_unstale":"2024-01-16T15:28:06.668445+0000","last_undegraded":"2024-01-16T15:28:06.668445+0000","last_fullsized":"2024-01-16T15:28:06.668445+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'100","last_scrub_stamp":"2024-01-16T15:28:06.668233+0000","last_deep_scrub":"115'100","last_deep_scrub_stamp":"2024-01-16T15:28:06.668233+0000","last_clean_scrub_stamp":"2024-01-16T15:28:06.668233+0000","objects_scrubbed":29,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:08.221222+0000","scrub_duration":0.32354169500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"115'87","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.480772+0000","last_change":"2024-01-16T15:27:43.480772+0000","last_active":"2024-01-16T15:27:43.480772+0000","last_peered":"2024-01-16T15:27:43.480772+0000","last_clean":"2024-01-16T15:27:43.480772+0000","last_became_active":"2024-01-16T15:18:50.522911+0000","last_became_peered":"2024-01-16T15:18:50.522911+0000","last_unstale":"2024-01-16T15:27:43.480772+0000","last_undegraded":"2024-01-16T15:27:43.480772+0000","last_fullsized":"2024-01-16T15:27:43.480772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'87","last_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_deep_scrub":"115'87","last_deep_scrub_stamp":"2024-01-16T15:27:43.480597+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.480597+0000","objects_scrubbed":24,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:30:52.268973+0000","scrub_duration":0.29440454500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"115'103","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.251665+0000","last_change":"2024-01-16T15:28:04.251783+0000","last_active":"2024-01-16T15:28:04.251665+0000","last_peered":"2024-01-16T15:28:04.251665+0000","last_clean":"2024-01-16T15:28:04.251665+0000","last_became_active":"2024-01-16T15:18:50.520908+0000","last_became_peered":"2024-01-16T15:18:50.520908+0000","last_unstale":"2024-01-16T15:28:04.251665+0000","last_undegraded":"2024-01-16T15:28:04.251665+0000","last_fullsized":"2024-01-16T15:28:04.251665+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'103","last_scrub_stamp":"2024-01-16T15:28:04.251462+0000","last_deep_scrub":"115'103","last_deep_scrub_stamp":"2024-01-16T15:28:04.251462+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.251462+0000","objects_scrubbed":30,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:22.321069+0000","scrub_duration":0.35867113499999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":22,"num_read_kb":81922,"num_write":32,"num_write_kb":122881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"115'90","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.342604+0000","last_change":"2024-01-16T15:28:10.342667+0000","last_active":"2024-01-16T15:28:10.342604+0000","last_peered":"2024-01-16T15:28:10.342604+0000","last_clean":"2024-01-16T15:28:10.342604+0000","last_became_active":"2024-01-16T15:18:50.497277+0000","last_became_peered":"2024-01-16T15:18:50.497277+0000","last_unstale":"2024-01-16T15:28:10.342604+0000","last_undegraded":"2024-01-16T15:28:10.342604+0000","last_fullsized":"2024-01-16T15:28:10.342604+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'90","last_scrub_stamp":"2024-01-16T15:28:10.342426+0000","last_deep_scrub":"115'90","last_deep_scrub_stamp":"2024-01-16T15:28:10.342426+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.342426+0000","objects_scrubbed":26,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:23:14.911068+0000","scrub_duration":0.295467753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"115'83","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:58.223115+0000","last_change":"2024-01-16T15:27:58.223162+0000","last_active":"2024-01-16T15:27:58.223115+0000","last_peered":"2024-01-16T15:27:58.223115+0000","last_clean":"2024-01-16T15:27:58.223115+0000","last_became_active":"2024-01-16T15:18:50.518509+0000","last_became_peered":"2024-01-16T15:18:50.518509+0000","last_unstale":"2024-01-16T15:27:58.223115+0000","last_undegraded":"2024-01-16T15:27:58.223115+0000","last_fullsized":"2024-01-16T15:27:58.223115+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'83","last_scrub_stamp":"2024-01-16T15:27:58.222980+0000","last_deep_scrub":"115'83","last_deep_scrub_stamp":"2024-01-16T15:27:58.222980+0000","last_clean_scrub_stamp":"2024-01-16T15:27:58.222980+0000","objects_scrubbed":26,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:49.589555+0000","scrub_duration":0.31016252599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"115'94","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:01.351481+0000","last_change":"2024-01-16T15:28:01.351634+0000","last_active":"2024-01-16T15:28:01.351481+0000","last_peered":"2024-01-16T15:28:01.351481+0000","last_clean":"2024-01-16T15:28:01.351481+0000","last_became_active":"2024-01-16T15:18:50.518113+0000","last_became_peered":"2024-01-16T15:18:50.518113+0000","last_unstale":"2024-01-16T15:28:01.351481+0000","last_undegraded":"2024-01-16T15:28:01.351481+0000","last_fullsized":"2024-01-16T15:28:01.351481+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'94","last_scrub_stamp":"2024-01-16T15:28:01.351267+0000","last_deep_scrub":"115'94","last_deep_scrub_stamp":"2024-01-16T15:28:01.351267+0000","last_clean_scrub_stamp":"2024-01-16T15:28:01.351267+0000","objects_scrubbed":24,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:45:16.680016+0000","scrub_duration":0.25329149400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"115'97","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.461542+0000","last_change":"2024-01-16T15:27:45.461588+0000","last_active":"2024-01-16T15:27:45.461542+0000","last_peered":"2024-01-16T15:27:45.461542+0000","last_clean":"2024-01-16T15:27:45.461542+0000","last_became_active":"2024-01-16T15:18:50.512367+0000","last_became_peered":"2024-01-16T15:18:50.512367+0000","last_unstale":"2024-01-16T15:27:45.461542+0000","last_undegraded":"2024-01-16T15:27:45.461542+0000","last_fullsized":"2024-01-16T15:27:45.461542+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:45.461410+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.461410+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:24.634385+0000","scrub_duration":0.31535565399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"115'107","reported_seq":144,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.988577+0000","last_change":"2024-01-16T15:28:05.988636+0000","last_active":"2024-01-16T15:28:05.988577+0000","last_peered":"2024-01-16T15:28:05.988577+0000","last_clean":"2024-01-16T15:28:05.988577+0000","last_became_active":"2024-01-16T15:18:50.516361+0000","last_became_peered":"2024-01-16T15:18:50.516361+0000","last_unstale":"2024-01-16T15:28:05.988577+0000","last_undegraded":"2024-01-16T15:28:05.988577+0000","last_fullsized":"2024-01-16T15:28:05.988577+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'107","last_scrub_stamp":"2024-01-16T15:28:05.988385+0000","last_deep_scrub":"115'107","last_deep_scrub_stamp":"2024-01-16T15:28:05.988385+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.988385+0000","objects_scrubbed":26,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:24:27.334330+0000","scrub_duration":0.28793185500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"115'88","reported_seq":124,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:09.632615+0000","last_change":"2024-01-16T15:28:09.632675+0000","last_active":"2024-01-16T15:28:09.632615+0000","last_peered":"2024-01-16T15:28:09.632615+0000","last_clean":"2024-01-16T15:28:09.632615+0000","last_became_active":"2024-01-16T15:18:50.523568+0000","last_became_peered":"2024-01-16T15:18:50.523568+0000","last_unstale":"2024-01-16T15:28:09.632615+0000","last_undegraded":"2024-01-16T15:28:09.632615+0000","last_fullsized":"2024-01-16T15:28:09.632615+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:28:09.632498+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:28:09.632498+0000","last_clean_scrub_stamp":"2024-01-16T15:28:09.632498+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:32.458242+0000","scrub_duration":0.226358428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"115'62","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.167246+0000","last_change":"2024-01-16T15:28:17.167330+0000","last_active":"2024-01-16T15:28:17.167246+0000","last_peered":"2024-01-16T15:28:17.167246+0000","last_clean":"2024-01-16T15:28:17.167246+0000","last_became_active":"2024-01-16T15:18:50.516161+0000","last_became_peered":"2024-01-16T15:18:50.516161+0000","last_unstale":"2024-01-16T15:28:17.167246+0000","last_undegraded":"2024-01-16T15:28:17.167246+0000","last_fullsized":"2024-01-16T15:28:17.167246+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'62","last_scrub_stamp":"2024-01-16T15:28:17.167089+0000","last_deep_scrub":"115'62","last_deep_scrub_stamp":"2024-01-16T15:28:17.167089+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.167089+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:35.488651+0000","scrub_duration":0.194901195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"115'77","reported_seq":116,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.242903+0000","last_change":"2024-01-16T15:28:08.242974+0000","last_active":"2024-01-16T15:28:08.242903+0000","last_peered":"2024-01-16T15:28:08.242903+0000","last_clean":"2024-01-16T15:28:08.242903+0000","last_became_active":"2024-01-16T15:18:50.520532+0000","last_became_peered":"2024-01-16T15:18:50.520532+0000","last_unstale":"2024-01-16T15:28:08.242903+0000","last_undegraded":"2024-01-16T15:28:08.242903+0000","last_fullsized":"2024-01-16T15:28:08.242903+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'77","last_scrub_stamp":"2024-01-16T15:28:08.242792+0000","last_deep_scrub":"115'77","last_deep_scrub_stamp":"2024-01-16T15:28:08.242792+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.242792+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:16:19.148891+0000","scrub_duration":0.25754353299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"115'104","reported_seq":146,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.544481+0000","last_change":"2024-01-16T15:27:48.544547+0000","last_active":"2024-01-16T15:27:48.544481+0000","last_peered":"2024-01-16T15:27:48.544481+0000","last_clean":"2024-01-16T15:27:48.544481+0000","last_became_active":"2024-01-16T15:18:50.491181+0000","last_became_peered":"2024-01-16T15:18:50.491181+0000","last_unstale":"2024-01-16T15:27:48.544481+0000","last_undegraded":"2024-01-16T15:27:48.544481+0000","last_fullsized":"2024-01-16T15:27:48.544481+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'104","last_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_deep_scrub":"115'104","last_deep_scrub_stamp":"2024-01-16T15:27:48.544284+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.544284+0000","objects_scrubbed":28,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:19.364542+0000","scrub_duration":0.32836960199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":15,"num_read_kb":49155,"num_write":34,"num_write_kb":114691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"115'106","reported_seq":152,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.886489+0000","last_change":"2024-01-16T15:27:33.886536+0000","last_active":"2024-01-16T15:27:33.886489+0000","last_peered":"2024-01-16T15:27:33.886489+0000","last_clean":"2024-01-16T15:27:33.886489+0000","last_became_active":"2024-01-16T15:18:50.507721+0000","last_became_peered":"2024-01-16T15:18:50.507721+0000","last_unstale":"2024-01-16T15:27:33.886489+0000","last_undegraded":"2024-01-16T15:27:33.886489+0000","last_fullsized":"2024-01-16T15:27:33.886489+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'106","last_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_deep_scrub":"115'106","last_deep_scrub_stamp":"2024-01-16T15:27:33.886354+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.886354+0000","objects_scrubbed":27,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:46.223371+0000","scrub_duration":0.231844831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":77824,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"115'85","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.353703+0000","last_change":"2024-01-16T15:27:44.353765+0000","last_active":"2024-01-16T15:27:44.353703+0000","last_peered":"2024-01-16T15:27:44.353703+0000","last_clean":"2024-01-16T15:27:44.353703+0000","last_became_active":"2024-01-16T15:18:50.510526+0000","last_became_peered":"2024-01-16T15:18:50.510526+0000","last_unstale":"2024-01-16T15:27:44.353703+0000","last_undegraded":"2024-01-16T15:27:44.353703+0000","last_fullsized":"2024-01-16T15:27:44.353703+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:27:44.353606+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.353606+0000","objects_scrubbed":23,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:15.404103+0000","scrub_duration":0.234855483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"115'88","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.394246+0000","last_change":"2024-01-16T15:27:51.394300+0000","last_active":"2024-01-16T15:27:51.394246+0000","last_peered":"2024-01-16T15:27:51.394246+0000","last_clean":"2024-01-16T15:27:51.394246+0000","last_became_active":"2024-01-16T15:18:50.516710+0000","last_became_peered":"2024-01-16T15:18:50.516710+0000","last_unstale":"2024-01-16T15:27:51.394246+0000","last_undegraded":"2024-01-16T15:27:51.394246+0000","last_fullsized":"2024-01-16T15:27:51.394246+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:27:51.394095+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:27:51.394095+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.394095+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:49.782161+0000","scrub_duration":0.26988635100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"115'6","reported_seq":33,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.184374+0000","last_change":"2024-01-16T15:27:41.184433+0000","last_active":"2024-01-16T15:27:41.184374+0000","last_peered":"2024-01-16T15:27:41.184374+0000","last_clean":"2024-01-16T15:27:41.184374+0000","last_became_active":"2024-01-16T15:18:49.513801+0000","last_became_peered":"2024-01-16T15:18:49.513801+0000","last_unstale":"2024-01-16T15:27:41.184374+0000","last_undegraded":"2024-01-16T15:27:41.184374+0000","last_fullsized":"2024-01-16T15:27:41.184374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6","last_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_deep_scrub":"115'6","last_deep_scrub_stamp":"2024-01-16T15:27:41.184165+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.184165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:01.093220+0000","scrub_duration":0.0077587799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"115'73","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.981190+0000","last_change":"2024-01-16T15:28:04.981273+0000","last_active":"2024-01-16T15:28:04.981190+0000","last_peered":"2024-01-16T15:28:04.981190+0000","last_clean":"2024-01-16T15:28:04.981190+0000","last_became_active":"2024-01-16T15:18:50.515132+0000","last_became_peered":"2024-01-16T15:18:50.515132+0000","last_unstale":"2024-01-16T15:28:04.981190+0000","last_undegraded":"2024-01-16T15:28:04.981190+0000","last_fullsized":"2024-01-16T15:28:04.981190+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'73","last_scrub_stamp":"2024-01-16T15:28:04.981071+0000","last_deep_scrub":"115'73","last_deep_scrub_stamp":"2024-01-16T15:28:04.981071+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.981071+0000","objects_scrubbed":19,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:39.654117+0000","scrub_duration":0.258258816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":28,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:36.422396+0000","last_change":"2024-01-16T15:27:36.422452+0000","last_active":"2024-01-16T15:27:36.422396+0000","last_peered":"2024-01-16T15:27:36.422396+0000","last_clean":"2024-01-16T15:27:36.422396+0000","last_became_active":"2024-01-16T15:18:49.521187+0000","last_became_peered":"2024-01-16T15:18:49.521187+0000","last_unstale":"2024-01-16T15:27:36.422396+0000","last_undegraded":"2024-01-16T15:27:36.422396+0000","last_fullsized":"2024-01-16T15:27:36.422396+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:36.422186+0000","last_clean_scrub_stamp":"2024-01-16T15:27:36.422186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:42.355689+0000","scrub_duration":0.003123125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"115'97","reported_seq":141,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:52.431987+0000","last_change":"2024-01-16T15:27:52.432060+0000","last_active":"2024-01-16T15:27:52.431987+0000","last_peered":"2024-01-16T15:27:52.431987+0000","last_clean":"2024-01-16T15:27:52.431987+0000","last_became_active":"2024-01-16T15:18:50.526157+0000","last_became_peered":"2024-01-16T15:18:50.526157+0000","last_unstale":"2024-01-16T15:27:52.431987+0000","last_undegraded":"2024-01-16T15:27:52.431987+0000","last_fullsized":"2024-01-16T15:27:52.431987+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:27:52.431869+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:27:52.431869+0000","last_clean_scrub_stamp":"2024-01-16T15:27:52.431869+0000","objects_scrubbed":26,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:21.198854+0000","scrub_duration":0.27025465500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":18,"num_read_kb":65538,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"115'4","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.217452+0000","last_change":"2024-01-16T15:27:44.217632+0000","last_active":"2024-01-16T15:27:44.217452+0000","last_peered":"2024-01-16T15:27:44.217452+0000","last_clean":"2024-01-16T15:27:44.217452+0000","last_became_active":"2024-01-16T15:18:49.518530+0000","last_became_peered":"2024-01-16T15:18:49.518530+0000","last_unstale":"2024-01-16T15:27:44.217452+0000","last_undegraded":"2024-01-16T15:27:44.217452+0000","last_fullsized":"2024-01-16T15:27:44.217452+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:44.217230+0000","last_clean_scrub_stamp":"2024-01-16T15:27:44.217230+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:49.674761+0000","scrub_duration":0.007544144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"115'89","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.591288+0000","last_change":"2024-01-16T15:28:08.591424+0000","last_active":"2024-01-16T15:28:08.591288+0000","last_peered":"2024-01-16T15:28:08.591288+0000","last_clean":"2024-01-16T15:28:08.591288+0000","last_became_active":"2024-01-16T15:18:50.526601+0000","last_became_peered":"2024-01-16T15:18:50.526601+0000","last_unstale":"2024-01-16T15:28:08.591288+0000","last_undegraded":"2024-01-16T15:28:08.591288+0000","last_fullsized":"2024-01-16T15:28:08.591288+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:08.591109+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:08.591109+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.591109+0000","objects_scrubbed":21,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:15:03.211743+0000","scrub_duration":0.22652739899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":21,"num_read_kb":86016,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"115'3","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.189552+0000","last_change":"2024-01-16T15:27:38.189613+0000","last_active":"2024-01-16T15:27:38.189552+0000","last_peered":"2024-01-16T15:27:38.189552+0000","last_clean":"2024-01-16T15:27:38.189552+0000","last_became_active":"2024-01-16T15:18:49.518552+0000","last_became_peered":"2024-01-16T15:18:49.518552+0000","last_unstale":"2024-01-16T15:27:38.189552+0000","last_undegraded":"2024-01-16T15:27:38.189552+0000","last_fullsized":"2024-01-16T15:27:38.189552+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'3","last_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_deep_scrub":"115'3","last_deep_scrub_stamp":"2024-01-16T15:27:38.189364+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:03:26.432118+0000","scrub_duration":0.0070076540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"115'95","reported_seq":129,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:54.430099+0000","last_change":"2024-01-16T15:27:54.430198+0000","last_active":"2024-01-16T15:27:54.430099+0000","last_peered":"2024-01-16T15:27:54.430099+0000","last_clean":"2024-01-16T15:27:54.430099+0000","last_became_active":"2024-01-16T15:19:15.317339+0000","last_became_peered":"2024-01-16T15:19:15.317339+0000","last_unstale":"2024-01-16T15:27:54.430099+0000","last_undegraded":"2024-01-16T15:27:54.430099+0000","last_fullsized":"2024-01-16T15:27:54.430099+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:54.429931+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:54.429931+0000","last_clean_scrub_stamp":"2024-01-16T15:27:54.429931+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:17.255675+0000","scrub_duration":0.26159354499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.186234+0000","last_change":"2024-01-16T15:27:34.186287+0000","last_active":"2024-01-16T15:27:34.186234+0000","last_peered":"2024-01-16T15:27:34.186234+0000","last_clean":"2024-01-16T15:27:34.186234+0000","last_became_active":"2024-01-16T15:18:49.522896+0000","last_became_peered":"2024-01-16T15:18:49.522896+0000","last_unstale":"2024-01-16T15:27:34.186234+0000","last_undegraded":"2024-01-16T15:27:34.186234+0000","last_fullsized":"2024-01-16T15:27:34.186234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:34.186050+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.186050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:31:37.071156+0000","scrub_duration":0.0039566280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"115'97","reported_seq":143,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:09.254999+0000","last_change":"2024-01-16T15:28:09.255056+0000","last_active":"2024-01-16T15:28:09.254999+0000","last_peered":"2024-01-16T15:28:09.254999+0000","last_clean":"2024-01-16T15:28:09.254999+0000","last_became_active":"2024-01-16T15:18:50.517025+0000","last_became_peered":"2024-01-16T15:18:50.517025+0000","last_unstale":"2024-01-16T15:28:09.254999+0000","last_undegraded":"2024-01-16T15:28:09.254999+0000","last_fullsized":"2024-01-16T15:28:09.254999+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:28:09.254808+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:28:09.254808+0000","last_clean_scrub_stamp":"2024-01-16T15:28:09.254808+0000","objects_scrubbed":25,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:15.395955+0000","scrub_duration":0.27471414199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.865012+0000","last_change":"2024-01-16T15:27:39.865110+0000","last_active":"2024-01-16T15:27:39.865012+0000","last_peered":"2024-01-16T15:27:39.865012+0000","last_clean":"2024-01-16T15:27:39.865012+0000","last_became_active":"2024-01-16T15:18:49.516907+0000","last_became_peered":"2024-01-16T15:18:49.516907+0000","last_unstale":"2024-01-16T15:27:39.865012+0000","last_undegraded":"2024-01-16T15:27:39.865012+0000","last_fullsized":"2024-01-16T15:27:39.865012+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:27:39.864756+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.864756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:53.780673+0000","scrub_duration":0.0037524390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"115'95","reported_seq":136,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.871123+0000","last_change":"2024-01-16T15:27:29.871123+0000","last_active":"2024-01-16T15:27:29.871123+0000","last_peered":"2024-01-16T15:27:29.871123+0000","last_clean":"2024-01-16T15:27:29.871123+0000","last_became_active":"2024-01-16T15:18:50.513214+0000","last_became_peered":"2024-01-16T15:18:50.513214+0000","last_unstale":"2024-01-16T15:27:29.871123+0000","last_undegraded":"2024-01-16T15:27:29.871123+0000","last_fullsized":"2024-01-16T15:27:29.871123+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:29.871010+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.871010+0000","objects_scrubbed":23,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:25.437998+0000","scrub_duration":0.15415410600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":53250,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.318028+0000","last_change":"2024-01-16T15:27:32.318028+0000","last_active":"2024-01-16T15:27:32.318028+0000","last_peered":"2024-01-16T15:27:32.318028+0000","last_clean":"2024-01-16T15:27:32.318028+0000","last_became_active":"2024-01-16T15:18:49.508415+0000","last_became_peered":"2024-01-16T15:18:49.508415+0000","last_unstale":"2024-01-16T15:27:32.318028+0000","last_undegraded":"2024-01-16T15:27:32.318028+0000","last_fullsized":"2024-01-16T15:27:32.318028+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.317792+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.317792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:01.084778+0000","scrub_duration":0.0035389969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"115'101","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:46.595143+0000","last_change":"2024-01-16T15:27:46.595143+0000","last_active":"2024-01-16T15:27:46.595143+0000","last_peered":"2024-01-16T15:27:46.595143+0000","last_clean":"2024-01-16T15:27:46.595143+0000","last_became_active":"2024-01-16T15:18:50.505826+0000","last_became_peered":"2024-01-16T15:18:50.505826+0000","last_unstale":"2024-01-16T15:27:46.595143+0000","last_undegraded":"2024-01-16T15:27:46.595143+0000","last_fullsized":"2024-01-16T15:27:46.595143+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'101","last_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_deep_scrub":"115'101","last_deep_scrub_stamp":"2024-01-16T15:27:46.595008+0000","last_clean_scrub_stamp":"2024-01-16T15:27:46.595008+0000","objects_scrubbed":28,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:44.983075+0000","scrub_duration":0.33379621399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.688811+0000","last_change":"2024-01-16T15:27:39.688811+0000","last_active":"2024-01-16T15:27:39.688811+0000","last_peered":"2024-01-16T15:27:39.688811+0000","last_clean":"2024-01-16T15:27:39.688811+0000","last_became_active":"2024-01-16T15:18:49.510293+0000","last_became_peered":"2024-01-16T15:18:49.510293+0000","last_unstale":"2024-01-16T15:27:39.688811+0000","last_undegraded":"2024-01-16T15:27:39.688811+0000","last_fullsized":"2024-01-16T15:27:39.688811+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.688565+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.688565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:07.291212+0000","scrub_duration":0.0036420599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"115'72","reported_seq":103,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.479158+0000","last_change":"2024-01-16T15:27:43.479158+0000","last_active":"2024-01-16T15:27:43.479158+0000","last_peered":"2024-01-16T15:27:43.479158+0000","last_clean":"2024-01-16T15:27:43.479158+0000","last_became_active":"2024-01-16T15:18:50.506548+0000","last_became_peered":"2024-01-16T15:18:50.506548+0000","last_unstale":"2024-01-16T15:27:43.479158+0000","last_undegraded":"2024-01-16T15:27:43.479158+0000","last_fullsized":"2024-01-16T15:27:43.479158+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'72","last_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_deep_scrub":"115'72","last_deep_scrub_stamp":"2024-01-16T15:27:43.478956+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.478956+0000","objects_scrubbed":18,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:43.468767+0000","scrub_duration":0.23849152200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"115'1","reported_seq":28,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.224399+0000","last_change":"2024-01-16T15:27:51.224399+0000","last_active":"2024-01-16T15:27:51.224399+0000","last_peered":"2024-01-16T15:27:51.224399+0000","last_clean":"2024-01-16T15:27:51.224399+0000","last_became_active":"2024-01-16T15:18:49.524677+0000","last_became_peered":"2024-01-16T15:18:49.524677+0000","last_unstale":"2024-01-16T15:27:51.224399+0000","last_undegraded":"2024-01-16T15:27:51.224399+0000","last_fullsized":"2024-01-16T15:27:51.224399+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'1","last_scrub_stamp":"2024-01-16T15:27:51.224231+0000","last_deep_scrub":"115'1","last_deep_scrub_stamp":"2024-01-16T15:27:51.224231+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.224231+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:05.632405+0000","scrub_duration":0.0038857200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"115'81","reported_seq":115,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.907256+0000","last_change":"2024-01-16T15:27:59.907256+0000","last_active":"2024-01-16T15:27:59.907256+0000","last_peered":"2024-01-16T15:27:59.907256+0000","last_clean":"2024-01-16T15:27:59.907256+0000","last_became_active":"2024-01-16T15:18:50.524215+0000","last_became_peered":"2024-01-16T15:18:50.524215+0000","last_unstale":"2024-01-16T15:27:59.907256+0000","last_undegraded":"2024-01-16T15:27:59.907256+0000","last_fullsized":"2024-01-16T15:27:59.907256+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'81","last_scrub_stamp":"2024-01-16T15:27:59.906852+0000","last_deep_scrub":"115'81","last_deep_scrub_stamp":"2024-01-16T15:27:59.906852+0000","last_clean_scrub_stamp":"2024-01-16T15:27:59.906852+0000","objects_scrubbed":20,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:16:44.027555+0000","scrub_duration":0.23773976699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:30.836744+0000","last_change":"2024-01-16T15:27:30.836744+0000","last_active":"2024-01-16T15:27:30.836744+0000","last_peered":"2024-01-16T15:27:30.836744+0000","last_clean":"2024-01-16T15:27:30.836744+0000","last_became_active":"2024-01-16T15:18:49.513710+0000","last_became_peered":"2024-01-16T15:18:49.513710+0000","last_unstale":"2024-01-16T15:27:30.836744+0000","last_undegraded":"2024-01-16T15:27:30.836744+0000","last_fullsized":"2024-01-16T15:27:30.836744+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:30.836522+0000","last_clean_scrub_stamp":"2024-01-16T15:27:30.836522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:58:38.704619+0000","scrub_duration":0.0037552990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"115'88","reported_seq":131,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:12.279622+0000","last_change":"2024-01-16T15:28:12.279771+0000","last_active":"2024-01-16T15:28:12.279622+0000","last_peered":"2024-01-16T15:28:12.279622+0000","last_clean":"2024-01-16T15:28:12.279622+0000","last_became_active":"2024-01-16T15:18:50.521516+0000","last_became_peered":"2024-01-16T15:18:50.521516+0000","last_unstale":"2024-01-16T15:28:12.279622+0000","last_undegraded":"2024-01-16T15:28:12.279622+0000","last_fullsized":"2024-01-16T15:28:12.279622+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'88","last_scrub_stamp":"2024-01-16T15:28:12.279406+0000","last_deep_scrub":"115'88","last_deep_scrub_stamp":"2024-01-16T15:28:12.279406+0000","last_clean_scrub_stamp":"2024-01-16T15:28:12.279406+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:29.028528+0000","scrub_duration":0.29601894200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"115'35","reported_seq":62,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.716990+0000","last_change":"2024-01-16T15:27:48.716990+0000","last_active":"2024-01-16T15:27:48.716990+0000","last_peered":"2024-01-16T15:27:48.716990+0000","last_clean":"2024-01-16T15:27:48.716990+0000","last_became_active":"2024-01-16T15:18:49.523104+0000","last_became_peered":"2024-01-16T15:18:49.523104+0000","last_unstale":"2024-01-16T15:27:48.716990+0000","last_undegraded":"2024-01-16T15:27:48.716990+0000","last_fullsized":"2024-01-16T15:27:48.716990+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:48.716639+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.716639+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:05.125879+0000","scrub_duration":0.0089525939999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"115'96","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.492171+0000","last_change":"2024-01-16T15:27:51.492171+0000","last_active":"2024-01-16T15:27:51.492171+0000","last_peered":"2024-01-16T15:27:51.492171+0000","last_clean":"2024-01-16T15:27:51.492171+0000","last_became_active":"2024-01-16T15:18:50.518070+0000","last_became_peered":"2024-01-16T15:18:50.518070+0000","last_unstale":"2024-01-16T15:27:51.492171+0000","last_undegraded":"2024-01-16T15:27:51.492171+0000","last_fullsized":"2024-01-16T15:27:51.492171+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'96","last_scrub_stamp":"2024-01-16T15:27:51.491852+0000","last_deep_scrub":"115'96","last_deep_scrub_stamp":"2024-01-16T15:27:51.491852+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.491852+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:04:15.258818+0000","scrub_duration":0.361982994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":12,"num_read_kb":40962,"num_write":29,"num_write_kb":110593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"115'327","reported_seq":354,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.163793+0000","last_change":"2024-01-16T15:27:45.744230+0000","last_active":"2024-01-16T15:28:15.163793+0000","last_peered":"2024-01-16T15:28:15.163793+0000","last_clean":"2024-01-16T15:28:15.163793+0000","last_became_active":"2024-01-16T15:18:49.510405+0000","last_became_peered":"2024-01-16T15:18:49.510405+0000","last_unstale":"2024-01-16T15:28:15.163793+0000","last_undegraded":"2024-01-16T15:28:15.163793+0000","last_fullsized":"2024-01-16T15:28:15.163793+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'325","last_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_deep_scrub":"115'325","last_deep_scrub_stamp":"2024-01-16T15:27:45.743867+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.743867+0000","objects_scrubbed":4,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:00.095065+0000","scrub_duration":0.018757899000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1246581,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":323,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"115'75","reported_seq":110,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.254126+0000","last_change":"2024-01-16T15:28:11.254238+0000","last_active":"2024-01-16T15:28:11.254126+0000","last_peered":"2024-01-16T15:28:11.254126+0000","last_clean":"2024-01-16T15:28:11.254126+0000","last_became_active":"2024-01-16T15:18:50.499611+0000","last_became_peered":"2024-01-16T15:18:50.499611+0000","last_unstale":"2024-01-16T15:28:11.254126+0000","last_undegraded":"2024-01-16T15:28:11.254126+0000","last_fullsized":"2024-01-16T15:28:11.254126+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'75","last_scrub_stamp":"2024-01-16T15:28:11.253911+0000","last_deep_scrub":"115'75","last_deep_scrub_stamp":"2024-01-16T15:28:11.253911+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.253911+0000","objects_scrubbed":21,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:09.726073+0000","scrub_duration":0.25591091199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.161975+0000","last_change":"2024-01-16T15:27:38.161975+0000","last_active":"2024-01-16T15:27:38.161975+0000","last_peered":"2024-01-16T15:27:38.161975+0000","last_clean":"2024-01-16T15:27:38.161975+0000","last_became_active":"2024-01-16T15:18:49.518446+0000","last_became_peered":"2024-01-16T15:18:49.518446+0000","last_unstale":"2024-01-16T15:27:38.161975+0000","last_undegraded":"2024-01-16T15:27:38.161975+0000","last_fullsized":"2024-01-16T15:27:38.161975+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.161723+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.161723+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:02.215134+0000","scrub_duration":0.0087694660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'95","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.180860+0000","last_change":"2024-01-16T15:27:43.180860+0000","last_active":"2024-01-16T15:27:43.180860+0000","last_peered":"2024-01-16T15:27:43.180860+0000","last_clean":"2024-01-16T15:27:43.180860+0000","last_became_active":"2024-01-16T15:18:50.517409+0000","last_became_peered":"2024-01-16T15:18:50.517409+0000","last_unstale":"2024-01-16T15:27:43.180860+0000","last_undegraded":"2024-01-16T15:27:43.180860+0000","last_fullsized":"2024-01-16T15:27:43.180860+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'95","last_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_deep_scrub":"115'95","last_deep_scrub_stamp":"2024-01-16T15:27:43.180569+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.180569+0000","objects_scrubbed":25,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:27.476294+0000","scrub_duration":0.28297945600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":32771,"num_write":29,"num_write_kb":102402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.754037+0000","last_change":"2024-01-16T15:27:33.754037+0000","last_active":"2024-01-16T15:27:33.754037+0000","last_peered":"2024-01-16T15:27:33.754037+0000","last_clean":"2024-01-16T15:27:33.754037+0000","last_became_active":"2024-01-16T15:18:49.513325+0000","last_became_peered":"2024-01-16T15:18:49.513325+0000","last_unstale":"2024-01-16T15:27:33.754037+0000","last_undegraded":"2024-01-16T15:27:33.754037+0000","last_fullsized":"2024-01-16T15:27:33.754037+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.753955+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.753955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:34:45.427457+0000","scrub_duration":0.0018762620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"115'89","reported_seq":126,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.343402+0000","last_change":"2024-01-16T15:28:02.343402+0000","last_active":"2024-01-16T15:28:02.343402+0000","last_peered":"2024-01-16T15:28:02.343402+0000","last_clean":"2024-01-16T15:28:02.343402+0000","last_became_active":"2024-01-16T15:18:50.508488+0000","last_became_peered":"2024-01-16T15:18:50.508488+0000","last_unstale":"2024-01-16T15:28:02.343402+0000","last_undegraded":"2024-01-16T15:28:02.343402+0000","last_fullsized":"2024-01-16T15:28:02.343402+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:02.343152+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:02.343152+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.343152+0000","objects_scrubbed":26,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:27.860851+0000","scrub_duration":0.28032371900000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857861,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"115'12","reported_seq":39,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.176351+0000","last_change":"2024-01-16T15:27:42.176351+0000","last_active":"2024-01-16T15:27:42.176351+0000","last_peered":"2024-01-16T15:27:42.176351+0000","last_clean":"2024-01-16T15:27:42.176351+0000","last_became_active":"2024-01-16T15:18:49.508523+0000","last_became_peered":"2024-01-16T15:18:49.508523+0000","last_unstale":"2024-01-16T15:27:42.176351+0000","last_undegraded":"2024-01-16T15:27:42.176351+0000","last_fullsized":"2024-01-16T15:27:42.176351+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-01-16T15:27:42.176093+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.176093+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:16.846949+0000","scrub_duration":0.010433985999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"115'61","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.135131+0000","last_change":"2024-01-16T15:28:08.135214+0000","last_active":"2024-01-16T15:28:08.135131+0000","last_peered":"2024-01-16T15:28:08.135131+0000","last_clean":"2024-01-16T15:28:08.135131+0000","last_became_active":"2024-01-16T15:18:50.520547+0000","last_became_peered":"2024-01-16T15:18:50.520547+0000","last_unstale":"2024-01-16T15:28:08.135131+0000","last_undegraded":"2024-01-16T15:28:08.135131+0000","last_fullsized":"2024-01-16T15:28:08.135131+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'61","last_scrub_stamp":"2024-01-16T15:28:08.134967+0000","last_deep_scrub":"115'61","last_deep_scrub_stamp":"2024-01-16T15:28:08.134967+0000","last_clean_scrub_stamp":"2024-01-16T15:28:08.134967+0000","objects_scrubbed":15,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:06.523034+0000","scrub_duration":0.15542793599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"115'2","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.722441+0000","last_change":"2024-01-16T15:28:02.722549+0000","last_active":"2024-01-16T15:28:02.722441+0000","last_peered":"2024-01-16T15:28:02.722441+0000","last_clean":"2024-01-16T15:28:02.722441+0000","last_became_active":"2024-01-16T15:18:49.515356+0000","last_became_peered":"2024-01-16T15:18:49.515356+0000","last_unstale":"2024-01-16T15:28:02.722441+0000","last_undegraded":"2024-01-16T15:28:02.722441+0000","last_fullsized":"2024-01-16T15:28:02.722441+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-01-16T15:28:02.722230+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-01-16T15:28:02.722230+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.722230+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:29.591650+0000","scrub_duration":0.0073352740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'85","reported_seq":127,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.645475+0000","last_change":"2024-01-16T15:28:14.645527+0000","last_active":"2024-01-16T15:28:14.645475+0000","last_peered":"2024-01-16T15:28:14.645475+0000","last_clean":"2024-01-16T15:28:14.645475+0000","last_became_active":"2024-01-16T15:18:50.520443+0000","last_became_peered":"2024-01-16T15:18:50.520443+0000","last_unstale":"2024-01-16T15:28:14.645475+0000","last_undegraded":"2024-01-16T15:28:14.645475+0000","last_fullsized":"2024-01-16T15:28:14.645475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:28:14.645401+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:28:14.645401+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.645401+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:10:22.603016+0000","scrub_duration":0.18823325199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":65536,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"115'3","reported_seq":30,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:57.156886+0000","last_change":"2024-01-16T15:27:57.156950+0000","last_active":"2024-01-16T15:27:57.156886+0000","last_peered":"2024-01-16T15:27:57.156886+0000","last_clean":"2024-01-16T15:27:57.156886+0000","last_became_active":"2024-01-16T15:18:49.506108+0000","last_became_peered":"2024-01-16T15:18:49.506108+0000","last_unstale":"2024-01-16T15:27:57.156886+0000","last_undegraded":"2024-01-16T15:27:57.156886+0000","last_fullsized":"2024-01-16T15:27:57.156886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'3","last_scrub_stamp":"2024-01-16T15:27:57.156695+0000","last_deep_scrub":"115'3","last_deep_scrub_stamp":"2024-01-16T15:27:57.156695+0000","last_clean_scrub_stamp":"2024-01-16T15:27:57.156695+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:17.065751+0000","scrub_duration":0.0077654519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"115'75","reported_seq":115,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:50.463446+0000","last_change":"2024-01-16T15:27:50.463446+0000","last_active":"2024-01-16T15:27:50.463446+0000","last_peered":"2024-01-16T15:27:50.463446+0000","last_clean":"2024-01-16T15:27:50.463446+0000","last_became_active":"2024-01-16T15:18:50.514960+0000","last_became_peered":"2024-01-16T15:18:50.514960+0000","last_unstale":"2024-01-16T15:27:50.463446+0000","last_undegraded":"2024-01-16T15:27:50.463446+0000","last_fullsized":"2024-01-16T15:27:50.463446+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'75","last_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_deep_scrub":"115'75","last_deep_scrub_stamp":"2024-01-16T15:27:50.463136+0000","last_clean_scrub_stamp":"2024-01-16T15:27:50.463136+0000","objects_scrubbed":14,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:21.882678+0000","scrub_duration":0.16385127299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":14,"num_read_kb":40964,"num_write":18,"num_write_kb":57346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"115'2","reported_seq":29,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.734212+0000","last_change":"2024-01-16T15:27:38.734212+0000","last_active":"2024-01-16T15:27:38.734212+0000","last_peered":"2024-01-16T15:27:38.734212+0000","last_clean":"2024-01-16T15:27:38.734212+0000","last_became_active":"2024-01-16T15:18:49.514275+0000","last_became_peered":"2024-01-16T15:18:49.514275+0000","last_unstale":"2024-01-16T15:27:38.734212+0000","last_undegraded":"2024-01-16T15:27:38.734212+0000","last_fullsized":"2024-01-16T15:27:38.734212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-01-16T15:27:38.733970+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.733970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:33.428355+0000","scrub_duration":0.0076776479999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"115'89","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.652891+0000","last_change":"2024-01-16T15:28:07.652940+0000","last_active":"2024-01-16T15:28:07.652891+0000","last_peered":"2024-01-16T15:28:07.652891+0000","last_clean":"2024-01-16T15:28:07.652891+0000","last_became_active":"2024-01-16T15:18:50.520658+0000","last_became_peered":"2024-01-16T15:18:50.520658+0000","last_unstale":"2024-01-16T15:28:07.652891+0000","last_undegraded":"2024-01-16T15:28:07.652891+0000","last_fullsized":"2024-01-16T15:28:07.652891+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'89","last_scrub_stamp":"2024-01-16T15:28:07.652775+0000","last_deep_scrub":"115'89","last_deep_scrub_stamp":"2024-01-16T15:28:07.652775+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.652775+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:32.819607+0000","scrub_duration":0.33515691600000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":17,"num_read_kb":69632,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.144273+0000","last_change":"2024-01-16T15:27:39.144273+0000","last_active":"2024-01-16T15:27:39.144273+0000","last_peered":"2024-01-16T15:27:39.144273+0000","last_clean":"2024-01-16T15:27:39.144273+0000","last_became_active":"2024-01-16T15:18:49.513717+0000","last_became_peered":"2024-01-16T15:18:49.513717+0000","last_unstale":"2024-01-16T15:27:39.144273+0000","last_undegraded":"2024-01-16T15:27:39.144273+0000","last_fullsized":"2024-01-16T15:27:39.144273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.144028+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.144028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.164911+0000","scrub_duration":0.0035817119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"115'82","reported_seq":122,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.596918+0000","last_change":"2024-01-16T15:27:45.596964+0000","last_active":"2024-01-16T15:27:45.596918+0000","last_peered":"2024-01-16T15:27:45.596918+0000","last_clean":"2024-01-16T15:27:45.596918+0000","last_became_active":"2024-01-16T15:18:50.521229+0000","last_became_peered":"2024-01-16T15:18:50.521229+0000","last_unstale":"2024-01-16T15:27:45.596918+0000","last_undegraded":"2024-01-16T15:27:45.596918+0000","last_fullsized":"2024-01-16T15:27:45.596918+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'82","last_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_deep_scrub":"115'82","last_deep_scrub_stamp":"2024-01-16T15:27:45.596772+0000","last_clean_scrub_stamp":"2024-01-16T15:27:45.596772+0000","objects_scrubbed":26,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:29.892497+0000","scrub_duration":0.31446099100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:34.232655+0000","last_change":"2024-01-16T15:27:34.232655+0000","last_active":"2024-01-16T15:27:34.232655+0000","last_peered":"2024-01-16T15:27:34.232655+0000","last_clean":"2024-01-16T15:27:34.232655+0000","last_became_active":"2024-01-16T15:18:49.520206+0000","last_became_peered":"2024-01-16T15:18:49.520206+0000","last_unstale":"2024-01-16T15:27:34.232655+0000","last_undegraded":"2024-01-16T15:27:34.232655+0000","last_fullsized":"2024-01-16T15:27:34.232655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:34.232419+0000","last_clean_scrub_stamp":"2024-01-16T15:27:34.232419+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:43.752231+0000","scrub_duration":0.0069378510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"115'96","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.010959+0000","last_change":"2024-01-16T15:28:05.011042+0000","last_active":"2024-01-16T15:28:05.010959+0000","last_peered":"2024-01-16T15:28:05.010959+0000","last_clean":"2024-01-16T15:28:05.010959+0000","last_became_active":"2024-01-16T15:18:50.515355+0000","last_became_peered":"2024-01-16T15:18:50.515355+0000","last_unstale":"2024-01-16T15:28:05.010959+0000","last_undegraded":"2024-01-16T15:28:05.010959+0000","last_fullsized":"2024-01-16T15:28:05.010959+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'96","last_scrub_stamp":"2024-01-16T15:28:05.010776+0000","last_deep_scrub":"115'96","last_deep_scrub_stamp":"2024-01-16T15:28:05.010776+0000","last_clean_scrub_stamp":"2024-01-16T15:28:05.010776+0000","objects_scrubbed":26,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:06:05.660319+0000","scrub_duration":0.329058395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":14,"num_read_kb":49154,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.164374+0000","last_change":"2024-01-16T15:27:39.164427+0000","last_active":"2024-01-16T15:27:39.164374+0000","last_peered":"2024-01-16T15:27:39.164374+0000","last_clean":"2024-01-16T15:27:39.164374+0000","last_became_active":"2024-01-16T15:18:49.517145+0000","last_became_peered":"2024-01-16T15:18:49.517145+0000","last_unstale":"2024-01-16T15:27:39.164374+0000","last_undegraded":"2024-01-16T15:27:39.164374+0000","last_fullsized":"2024-01-16T15:27:39.164374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.164189+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.164189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:40.647599+0000","scrub_duration":0.003563882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"115'90","reported_seq":123,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:02.605238+0000","last_change":"2024-01-16T15:28:02.605402+0000","last_active":"2024-01-16T15:28:02.605238+0000","last_peered":"2024-01-16T15:28:02.605238+0000","last_clean":"2024-01-16T15:28:02.605238+0000","last_became_active":"2024-01-16T15:18:50.525787+0000","last_became_peered":"2024-01-16T15:18:50.525787+0000","last_unstale":"2024-01-16T15:28:02.605238+0000","last_undegraded":"2024-01-16T15:28:02.605238+0000","last_fullsized":"2024-01-16T15:28:02.605238+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'90","last_scrub_stamp":"2024-01-16T15:28:02.605035+0000","last_deep_scrub":"115'90","last_deep_scrub_stamp":"2024-01-16T15:28:02.605035+0000","last_clean_scrub_stamp":"2024-01-16T15:28:02.605035+0000","objects_scrubbed":25,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:12.399323+0000","scrub_duration":0.28498842000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:29.193731+0000","last_change":"2024-01-16T15:27:29.193731+0000","last_active":"2024-01-16T15:27:29.193731+0000","last_peered":"2024-01-16T15:27:29.193731+0000","last_clean":"2024-01-16T15:27:29.193731+0000","last_became_active":"2024-01-16T15:18:49.520759+0000","last_became_peered":"2024-01-16T15:18:49.520759+0000","last_unstale":"2024-01-16T15:27:29.193731+0000","last_undegraded":"2024-01-16T15:27:29.193731+0000","last_fullsized":"2024-01-16T15:27:29.193731+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:29.193592+0000","last_clean_scrub_stamp":"2024-01-16T15:27:29.193592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:15:03.864449+0000","scrub_duration":0.0028384030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"115'77","reported_seq":113,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:48.205484+0000","last_change":"2024-01-16T15:27:48.205536+0000","last_active":"2024-01-16T15:27:48.205484+0000","last_peered":"2024-01-16T15:27:48.205484+0000","last_clean":"2024-01-16T15:27:48.205484+0000","last_became_active":"2024-01-16T15:18:50.516261+0000","last_became_peered":"2024-01-16T15:18:50.516261+0000","last_unstale":"2024-01-16T15:27:48.205484+0000","last_undegraded":"2024-01-16T15:27:48.205484+0000","last_fullsized":"2024-01-16T15:27:48.205484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'77","last_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_deep_scrub":"115'77","last_deep_scrub_stamp":"2024-01-16T15:27:48.205363+0000","last_clean_scrub_stamp":"2024-01-16T15:27:48.205363+0000","objects_scrubbed":20,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:54.141377+0000","scrub_duration":0.17343320200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79692037,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:41.694453+0000","last_change":"2024-01-16T15:27:41.694504+0000","last_active":"2024-01-16T15:27:41.694453+0000","last_peered":"2024-01-16T15:27:41.694453+0000","last_clean":"2024-01-16T15:27:41.694453+0000","last_became_active":"2024-01-16T15:18:49.520555+0000","last_became_peered":"2024-01-16T15:18:49.520555+0000","last_unstale":"2024-01-16T15:27:41.694453+0000","last_undegraded":"2024-01-16T15:27:41.694453+0000","last_fullsized":"2024-01-16T15:27:41.694453+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:41.694252+0000","last_clean_scrub_stamp":"2024-01-16T15:27:41.694252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:40.406823+0000","scrub_duration":0.0042882689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'70","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:55.894850+0000","last_change":"2024-01-16T15:27:55.894929+0000","last_active":"2024-01-16T15:27:55.894850+0000","last_peered":"2024-01-16T15:27:55.894850+0000","last_clean":"2024-01-16T15:27:55.894850+0000","last_became_active":"2024-01-16T15:18:50.520643+0000","last_became_peered":"2024-01-16T15:18:50.520643+0000","last_unstale":"2024-01-16T15:27:55.894850+0000","last_undegraded":"2024-01-16T15:27:55.894850+0000","last_fullsized":"2024-01-16T15:27:55.894850+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'70","last_scrub_stamp":"2024-01-16T15:27:55.894683+0000","last_deep_scrub":"115'70","last_deep_scrub_stamp":"2024-01-16T15:27:55.894683+0000","last_clean_scrub_stamp":"2024-01-16T15:27:55.894683+0000","objects_scrubbed":17,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:59:09.195148+0000","scrub_duration":0.187917204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"115'4","reported_seq":31,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:38.867500+0000","last_change":"2024-01-16T15:27:38.867614+0000","last_active":"2024-01-16T15:27:38.867500+0000","last_peered":"2024-01-16T15:27:38.867500+0000","last_clean":"2024-01-16T15:27:38.867500+0000","last_became_active":"2024-01-16T15:18:49.519723+0000","last_became_peered":"2024-01-16T15:18:49.519723+0000","last_unstale":"2024-01-16T15:27:38.867500+0000","last_undegraded":"2024-01-16T15:27:38.867500+0000","last_fullsized":"2024-01-16T15:27:38.867500+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'4","last_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_deep_scrub":"115'4","last_deep_scrub_stamp":"2024-01-16T15:27:38.867296+0000","last_clean_scrub_stamp":"2024-01-16T15:27:38.867296+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:02.481138+0000","scrub_duration":0.0076157439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"115'85","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.301942+0000","last_change":"2024-01-16T15:28:11.302058+0000","last_active":"2024-01-16T15:28:11.301942+0000","last_peered":"2024-01-16T15:28:11.301942+0000","last_clean":"2024-01-16T15:28:11.301942+0000","last_became_active":"2024-01-16T15:19:15.270433+0000","last_became_peered":"2024-01-16T15:19:15.270433+0000","last_unstale":"2024-01-16T15:28:11.301942+0000","last_undegraded":"2024-01-16T15:28:11.301942+0000","last_fullsized":"2024-01-16T15:28:11.301942+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'85","last_scrub_stamp":"2024-01-16T15:28:11.301760+0000","last_deep_scrub":"115'85","last_deep_scrub_stamp":"2024-01-16T15:28:11.301760+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.301760+0000","objects_scrubbed":24,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:45:42.122018+0000","scrub_duration":0.28736171900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.6","version":"115'31","reported_seq":58,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.825906+0000","last_change":"2024-01-16T15:27:42.826005+0000","last_active":"2024-01-16T15:27:42.825906+0000","last_peered":"2024-01-16T15:27:42.825906+0000","last_clean":"2024-01-16T15:27:42.825906+0000","last_became_active":"2024-01-16T15:18:49.513719+0000","last_became_peered":"2024-01-16T15:18:49.513719+0000","last_unstale":"2024-01-16T15:27:42.825906+0000","last_undegraded":"2024-01-16T15:27:42.825906+0000","last_fullsized":"2024-01-16T15:27:42.825906+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'31","last_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_deep_scrub":"115'31","last_deep_scrub_stamp":"2024-01-16T15:27:42.825641+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.825641+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:57.748209+0000","scrub_duration":0.0051599990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":649,"num_omap_keys":12,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"115'99","reported_seq":146,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.093416+0000","last_change":"2024-01-16T15:28:10.093493+0000","last_active":"2024-01-16T15:28:10.093416+0000","last_peered":"2024-01-16T15:28:10.093416+0000","last_clean":"2024-01-16T15:28:10.093416+0000","last_became_active":"2024-01-16T15:18:50.513455+0000","last_became_peered":"2024-01-16T15:18:50.513455+0000","last_unstale":"2024-01-16T15:28:10.093416+0000","last_undegraded":"2024-01-16T15:28:10.093416+0000","last_fullsized":"2024-01-16T15:28:10.093416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'99","last_scrub_stamp":"2024-01-16T15:28:10.093323+0000","last_deep_scrub":"115'99","last_deep_scrub_stamp":"2024-01-16T15:28:10.093323+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.093323+0000","objects_scrubbed":30,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:46.031990+0000","scrub_duration":0.29833682099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":20,"num_read_kb":81920,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:51.735910+0000","last_change":"2024-01-16T15:27:51.735960+0000","last_active":"2024-01-16T15:27:51.735910+0000","last_peered":"2024-01-16T15:27:51.735910+0000","last_clean":"2024-01-16T15:27:51.735910+0000","last_became_active":"2024-01-16T15:18:49.521054+0000","last_became_peered":"2024-01-16T15:18:49.521054+0000","last_unstale":"2024-01-16T15:27:51.735910+0000","last_undegraded":"2024-01-16T15:27:51.735910+0000","last_fullsized":"2024-01-16T15:27:51.735910+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:51.735728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:51.735728+0000","last_clean_scrub_stamp":"2024-01-16T15:27:51.735728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:56:58.848218+0000","scrub_duration":0.0037530269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"115'87","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:04.039019+0000","last_change":"2024-01-16T15:28:04.039156+0000","last_active":"2024-01-16T15:28:04.039019+0000","last_peered":"2024-01-16T15:28:04.039019+0000","last_clean":"2024-01-16T15:28:04.039019+0000","last_became_active":"2024-01-16T15:18:50.523133+0000","last_became_peered":"2024-01-16T15:18:50.523133+0000","last_unstale":"2024-01-16T15:28:04.039019+0000","last_undegraded":"2024-01-16T15:28:04.039019+0000","last_fullsized":"2024-01-16T15:28:04.039019+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'87","last_scrub_stamp":"2024-01-16T15:28:04.038791+0000","last_deep_scrub":"115'87","last_deep_scrub_stamp":"2024-01-16T15:28:04.038791+0000","last_clean_scrub_stamp":"2024-01-16T15:28:04.038791+0000","objects_scrubbed":28,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:59.941576+0000","scrub_duration":0.313418537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109052026,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:32.818788+0000","last_change":"2024-01-16T15:27:32.818822+0000","last_active":"2024-01-16T15:27:32.818788+0000","last_peered":"2024-01-16T15:27:32.818788+0000","last_clean":"2024-01-16T15:27:32.818788+0000","last_became_active":"2024-01-16T15:18:49.519243+0000","last_became_peered":"2024-01-16T15:18:49.519243+0000","last_unstale":"2024-01-16T15:27:32.818788+0000","last_undegraded":"2024-01-16T15:27:32.818788+0000","last_fullsized":"2024-01-16T15:27:32.818788+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:32.818678+0000","last_clean_scrub_stamp":"2024-01-16T15:27:32.818678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:29.803044+0000","scrub_duration":0.0024947659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"115'83","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:40.537839+0000","last_change":"2024-01-16T15:27:40.537917+0000","last_active":"2024-01-16T15:27:40.537839+0000","last_peered":"2024-01-16T15:27:40.537839+0000","last_clean":"2024-01-16T15:27:40.537839+0000","last_became_active":"2024-01-16T15:18:50.515744+0000","last_became_peered":"2024-01-16T15:18:50.515744+0000","last_unstale":"2024-01-16T15:27:40.537839+0000","last_undegraded":"2024-01-16T15:27:40.537839+0000","last_fullsized":"2024-01-16T15:27:40.537839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'83","last_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_deep_scrub":"115'83","last_deep_scrub_stamp":"2024-01-16T15:27:40.537681+0000","last_clean_scrub_stamp":"2024-01-16T15:27:40.537681+0000","objects_scrubbed":21,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:18:15.744820+0000","scrub_duration":0.26540904500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.219632+0000","last_change":"2024-01-16T15:27:33.219680+0000","last_active":"2024-01-16T15:27:33.219632+0000","last_peered":"2024-01-16T15:27:33.219632+0000","last_clean":"2024-01-16T15:27:33.219632+0000","last_became_active":"2024-01-16T15:18:49.508513+0000","last_became_peered":"2024-01-16T15:18:49.508513+0000","last_unstale":"2024-01-16T15:27:33.219632+0000","last_undegraded":"2024-01-16T15:27:33.219632+0000","last_fullsized":"2024-01-16T15:27:33.219632+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:33.219447+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.219447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:45:42.739258+0000","scrub_duration":0.0037869280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"115'93","reported_seq":134,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:00.165917+0000","last_change":"2024-01-16T15:28:00.166107+0000","last_active":"2024-01-16T15:28:00.165917+0000","last_peered":"2024-01-16T15:28:00.165917+0000","last_clean":"2024-01-16T15:28:00.165917+0000","last_became_active":"2024-01-16T15:18:50.510004+0000","last_became_peered":"2024-01-16T15:18:50.510004+0000","last_unstale":"2024-01-16T15:28:00.165917+0000","last_undegraded":"2024-01-16T15:28:00.165917+0000","last_fullsized":"2024-01-16T15:28:00.165917+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'93","last_scrub_stamp":"2024-01-16T15:28:00.165611+0000","last_deep_scrub":"115'93","last_deep_scrub_stamp":"2024-01-16T15:28:00.165611+0000","last_clean_scrub_stamp":"2024-01-16T15:28:00.165611+0000","objects_scrubbed":27,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:25.295005+0000","scrub_duration":0.29624051800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:39.251350+0000","last_change":"2024-01-16T15:27:39.251405+0000","last_active":"2024-01-16T15:27:39.251350+0000","last_peered":"2024-01-16T15:27:39.251350+0000","last_clean":"2024-01-16T15:27:39.251350+0000","last_became_active":"2024-01-16T15:18:49.518678+0000","last_became_peered":"2024-01-16T15:18:49.518678+0000","last_unstale":"2024-01-16T15:27:39.251350+0000","last_undegraded":"2024-01-16T15:27:39.251350+0000","last_fullsized":"2024-01-16T15:27:39.251350+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:39.251139+0000","last_clean_scrub_stamp":"2024-01-16T15:27:39.251139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:04.028594+0000","scrub_duration":0.0041990689999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'97","reported_seq":135,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:01.720216+0000","last_change":"2024-01-16T15:28:01.720275+0000","last_active":"2024-01-16T15:28:01.720216+0000","last_peered":"2024-01-16T15:28:01.720216+0000","last_clean":"2024-01-16T15:28:01.720216+0000","last_became_active":"2024-01-16T15:18:50.517506+0000","last_became_peered":"2024-01-16T15:18:50.517506+0000","last_unstale":"2024-01-16T15:28:01.720216+0000","last_undegraded":"2024-01-16T15:28:01.720216+0000","last_fullsized":"2024-01-16T15:28:01.720216+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'97","last_scrub_stamp":"2024-01-16T15:28:01.720058+0000","last_deep_scrub":"115'97","last_deep_scrub_stamp":"2024-01-16T15:28:01.720058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:01.720058+0000","objects_scrubbed":29,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:17:26.844383+0000","scrub_duration":0.351489783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"115'35","reported_seq":95,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:43.186518+0000","last_change":"2024-01-16T15:27:43.186575+0000","last_active":"2024-01-16T15:27:43.186518+0000","last_peered":"2024-01-16T15:27:43.186518+0000","last_clean":"2024-01-16T15:27:43.186518+0000","last_became_active":"2024-01-16T15:18:49.515237+0000","last_became_peered":"2024-01-16T15:18:49.515237+0000","last_unstale":"2024-01-16T15:27:43.186518+0000","last_undegraded":"2024-01-16T15:27:43.186518+0000","last_fullsized":"2024-01-16T15:27:43.186518+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'35","last_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_deep_scrub":"115'35","last_deep_scrub_stamp":"2024-01-16T15:27:43.186344+0000","last_clean_scrub_stamp":"2024-01-16T15:27:43.186344+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:36:34.244119+0000","scrub_duration":0.007088273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"115'102","reported_seq":142,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:28:03.935495+0000","last_change":"2024-01-16T15:28:03.935528+0000","last_active":"2024-01-16T15:28:03.935495+0000","last_peered":"2024-01-16T15:28:03.935495+0000","last_clean":"2024-01-16T15:28:03.935495+0000","last_became_active":"2024-01-16T15:18:50.518090+0000","last_became_peered":"2024-01-16T15:18:50.518090+0000","last_unstale":"2024-01-16T15:28:03.935495+0000","last_undegraded":"2024-01-16T15:28:03.935495+0000","last_fullsized":"2024-01-16T15:28:03.935495+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'102","last_scrub_stamp":"2024-01-16T15:28:03.935418+0000","last_deep_scrub":"115'102","last_deep_scrub_stamp":"2024-01-16T15:28:03.935418+0000","last_clean_scrub_stamp":"2024-01-16T15:28:03.935418+0000","objects_scrubbed":32,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:34.857974+0000","scrub_duration":0.26924512099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":32,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:37.623423+0000","last_change":"2024-01-16T15:27:37.623518+0000","last_active":"2024-01-16T15:27:37.623423+0000","last_peered":"2024-01-16T15:27:37.623423+0000","last_clean":"2024-01-16T15:27:37.623423+0000","last_became_active":"2024-01-16T15:18:49.518160+0000","last_became_peered":"2024-01-16T15:18:49.518160+0000","last_unstale":"2024-01-16T15:27:37.623423+0000","last_undegraded":"2024-01-16T15:27:37.623423+0000","last_fullsized":"2024-01-16T15:27:37.623423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:37.623212+0000","last_clean_scrub_stamp":"2024-01-16T15:27:37.623212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:31.323901+0000","scrub_duration":0.0041028669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"115'66","reported_seq":105,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:54.208127+0000","last_change":"2024-01-16T15:27:54.208127+0000","last_active":"2024-01-16T15:27:54.208127+0000","last_peered":"2024-01-16T15:27:54.208127+0000","last_clean":"2024-01-16T15:27:54.208127+0000","last_became_active":"2024-01-16T15:18:50.514550+0000","last_became_peered":"2024-01-16T15:18:50.514550+0000","last_unstale":"2024-01-16T15:27:54.208127+0000","last_undegraded":"2024-01-16T15:27:54.208127+0000","last_fullsized":"2024-01-16T15:27:54.208127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'66","last_scrub_stamp":"2024-01-16T15:27:54.207809+0000","last_deep_scrub":"115'66","last_deep_scrub_stamp":"2024-01-16T15:27:54.207809+0000","last_clean_scrub_stamp":"2024-01-16T15:27:54.207809+0000","objects_scrubbed":17,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:31.713234+0000","scrub_duration":0.19203378300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:42.215234+0000","last_change":"2024-01-16T15:27:42.215234+0000","last_active":"2024-01-16T15:27:42.215234+0000","last_peered":"2024-01-16T15:27:42.215234+0000","last_clean":"2024-01-16T15:27:42.215234+0000","last_became_active":"2024-01-16T15:18:49.521225+0000","last_became_peered":"2024-01-16T15:18:49.521225+0000","last_unstale":"2024-01-16T15:27:42.215234+0000","last_undegraded":"2024-01-16T15:27:42.215234+0000","last_fullsized":"2024-01-16T15:27:42.215234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:27:42.215017+0000","last_clean_scrub_stamp":"2024-01-16T15:27:42.215017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:11.615549+0000","scrub_duration":0.0038755600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"115'119","reported_seq":264,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-16T15:27:33.512603+0000","last_change":"2024-01-16T15:27:33.512658+0000","last_active":"2024-01-16T15:27:33.512603+0000","last_peered":"2024-01-16T15:27:33.512603+0000","last_clean":"2024-01-16T15:27:33.512603+0000","last_became_active":"2024-01-16T14:59:22.544080+0000","last_became_peered":"2024-01-16T14:59:22.544080+0000","last_unstale":"2024-01-16T15:27:33.512603+0000","last_undegraded":"2024-01-16T15:27:33.512603+0000","last_fullsized":"2024-01-16T15:27:33.512603+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'119","last_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_deep_scrub":"115'119","last_deep_scrub_stamp":"2024-01-16T15:27:33.512505+0000","last_clean_scrub_stamp":"2024-01-16T15:27:33.512505+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:30.496871+0000","scrub_duration":0.066194791000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":6442452743,"num_objects":1545,"num_object_clones":1,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":1,"num_read":799,"num_read_kb":3145759,"num_write":1605,"num_write_kb":6291494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12884967424,"data_stored":12884905486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5631,"ondisk_log_size":5631,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1251326,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":492,"num_write_kb":1464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5649,"num_omap_keys":22,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2564096,"data_stored":2502652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":476,"ondisk_log_size":476,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542497,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2137036,"kb_used_data":2025412,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":92234804,"statfs":{"total":96636764160,"available":94448439296,"internally_reserved":0,"allocated":2074021888,"data_stored":2073196245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114293194},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.228,"15min":1.2749999999999999},"min":{"1min":0.79500000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9530000000000001,"5min":2.2450000000000001,"15min":5.2690000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.3129999999999999,"15min":1.365},"min":{"1min":0.95399999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":7.2409999999999997},"last":1.107}]},{"osd":1,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.139,"15min":1.1459999999999999},"min":{"1min":0.73399999999999999,"5min":0.59999999999999998,"15min":0.55100000000000005},"max":{"1min":1.5169999999999999,"5min":1.5620000000000001,"15min":1.792},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.151,"15min":1.1870000000000001},"min":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.798,"5min":1.9490000000000001,"15min":3.504},"last":1.1659999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.349,"15min":1.397},"min":{"1min":0.68899999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.073,"5min":2.141,"15min":11.536},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.306,"15min":1.3859999999999999},"min":{"1min":0.83199999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.869,"5min":1.869,"15min":11.576000000000001},"last":1.3660000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.246,"15min":1.242},"min":{"1min":0.90800000000000003,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.133},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.379,"15min":1.3799999999999999},"min":{"1min":0.98199999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.181,"5min":2.181,"15min":3.6829999999999998},"last":1.554}]},{"osd":4,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2150000000000001,"15min":1.3049999999999999},"min":{"1min":0.84399999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":11.837},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3069999999999999,"15min":1.3129999999999999},"min":{"1min":1.056,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.222,"5min":2.222,"15min":2.222},"last":1.2729999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.3999999999999999,"15min":1.3680000000000001},"min":{"1min":1.1930000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.254,"5min":2.3849999999999998,"15min":3.4199999999999999},"last":1.623},{"interface":"front","average":{"1min":1.554,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":0.95099999999999996,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":6.8719999999999999},"last":1.5169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.5740000000000001,"15min":1.6020000000000001},"min":{"1min":1.1060000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":6.9690000000000003},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.5229999999999999,"15min":1.5009999999999999},"min":{"1min":1.3320000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":1.4810000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542496,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1536004,"kb_used_data":1406332,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":92835836,"statfs":{"total":96636764160,"available":95063896064,"internally_reserved":0,"allocated":1440083968,"data_stored":1439265369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.28,"15min":1.3169999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.887,"5min":8.0299999999999994,"15min":29.77},"last":1.026},{"interface":"front","average":{"1min":1.153,"5min":1.3089999999999999,"15min":1.4119999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.681,"5min":8.2439999999999998,"15min":36.405999999999999},"last":1.542}]},{"osd":1,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.099,"15min":1.1359999999999999},"min":{"1min":0.82999999999999996,"5min":0.67900000000000005,"15min":0.65800000000000003},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":5.2859999999999996},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.083,"5min":1.1539999999999999,"15min":1.1899999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.57099999999999995},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":10.494},"last":1.222}]},{"osd":2,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2,"15min":1.2829999999999999},"min":{"1min":0.79100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":5.0549999999999997},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.431,"5min":1.4079999999999999,"15min":1.454},"min":{"1min":1.0089999999999999,"5min":0.88400000000000001,"15min":0.88},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":5.1459999999999999},"last":1.288}]},{"osd":3,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3149999999999999,"15min":1.3140000000000001},"min":{"1min":0.84999999999999998,"5min":0.70199999999999996,"15min":0.68500000000000005},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":3.8079999999999998},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3,"15min":1.4470000000000001},"min":{"1min":0.70999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.7390000000000001,"5min":1.8300000000000001,"15min":21.456},"last":1.5980000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.4810000000000001,"15min":1.373},"min":{"1min":0.91400000000000003,"5min":0.90700000000000003,"15min":0.53400000000000003},"max":{"1min":9.4130000000000003,"5min":20.422999999999998,"15min":20.422999999999998},"last":1.077},{"interface":"front","average":{"1min":1.369,"5min":1.29,"15min":1.29},"min":{"1min":1.0589999999999999,"5min":0.77300000000000002,"15min":0.56899999999999995},"max":{"1min":3.0219999999999998,"5min":3.0219999999999998,"15min":3.6160000000000001},"last":1.431}]},{"osd":5,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.302,"15min":1.327},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.8129999999999999,"5min":7.6319999999999997,"15min":13.949999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5920000000000001,"15min":1.5760000000000001},"min":{"1min":1.087,"5min":1.087,"15min":0.51600000000000001},"max":{"1min":2.9780000000000002,"5min":7.4980000000000002,"15min":11.298},"last":1.655}]},{"osd":7,"last update":"Tue Jan 16 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4379999999999999,"15min":1.498},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.71499999999999997},"max":{"1min":1.837,"5min":2.004,"15min":14.037000000000001},"last":1.339},{"interface":"front","average":{"1min":1.577,"5min":1.5189999999999999,"15min":1.5760000000000001},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.7210000000000001,"5min":2.7210000000000001,"15min":4.9829999999999997},"last":1.3879999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542497,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1145860,"kb_used_data":1078652,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93225980,"statfs":{"total":96636764160,"available":95463403520,"internally_reserved":0,"allocated":1104539648,"data_stored":1103721461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.194,"15min":1.141},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.57399999999999995},"max":{"1min":1.7390000000000001,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.147},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5660000000000001,"5min":2.0819999999999999,"15min":2.5609999999999999},"last":1.0680000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1839999999999999,"15min":1.2130000000000001},"min":{"1min":0.76800000000000002,"5min":0.69499999999999995,"15min":0.65200000000000002},"max":{"1min":2.5,"5min":2.5,"15min":17.074999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2370000000000001,"15min":1.2450000000000001},"min":{"1min":0.77500000000000002,"5min":0.77100000000000002,"15min":0.745},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":4.8499999999999996},"last":1.266}]},{"osd":2,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.2210000000000001,"15min":1.1970000000000001},"min":{"1min":0.82199999999999995,"5min":0.72599999999999998,"15min":0.61099999999999999},"max":{"1min":1.9370000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2769999999999999,"15min":1.278},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":3.306,"5min":3.306,"15min":3.9590000000000001},"last":1.4159999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.347,"15min":1.3839999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":4.8010000000000002},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.5920000000000001,"15min":1.4590000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.72399999999999998},"max":{"1min":2.2360000000000002,"5min":14.044,"15min":14.044},"last":1.3320000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.25,"15min":1.2809999999999999},"min":{"1min":0.628,"5min":0.58699999999999997,"15min":0.56699999999999995},"max":{"1min":3.1989999999999998,"5min":3.1989999999999998,"15min":4.681},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.3069999999999999,"15min":1.343},"min":{"1min":0.503,"5min":0.503,"15min":0.502},"max":{"1min":1.6299999999999999,"5min":1.982,"15min":4.6580000000000004},"last":1.5549999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.347,"15min":1.355},"min":{"1min":0.86199999999999999,"5min":0.82999999999999996,"15min":0.52400000000000002},"max":{"1min":1.5569999999999999,"5min":1.847,"15min":2.7599999999999998},"last":1.194},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.496,"15min":1.5109999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8180000000000001,"5min":1.907,"15min":3.3719999999999999},"last":1.581}]},{"osd":7,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.512,"15min":1.5169999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.59099999999999997},"max":{"1min":4.8390000000000004,"5min":4.8390000000000004,"15min":4.8390000000000004},"last":1.377},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.571,"15min":1.5620000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.61099999999999999},"max":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"last":1.4930000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542496,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936452,"kb_used_data":1865084,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92435388,"statfs":{"total":96636764160,"available":94653837312,"internally_reserved":0,"allocated":1909846016,"data_stored":1909027666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.1419999999999999},"min":{"1min":0.73699999999999999,"5min":0.72199999999999998,"15min":0.44600000000000001},"max":{"1min":1.7889999999999999,"5min":1.839,"15min":6.4240000000000004},"last":1.03},{"interface":"front","average":{"1min":1.171,"5min":1.1899999999999999,"15min":1.143},"min":{"1min":0.78800000000000003,"5min":0.55900000000000005,"15min":0.46500000000000002},"max":{"1min":1.7410000000000001,"5min":1.7949999999999999,"15min":1.9810000000000001},"last":1.4319999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.385,"15min":1.464},"min":{"1min":0.96299999999999997,"5min":0.79500000000000004,"15min":0.56000000000000005},"max":{"1min":1.744,"5min":3.0049999999999999,"15min":24.399999999999999},"last":1.331},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.335,"15min":1.405},"min":{"1min":0.90500000000000003,"5min":0.65300000000000002,"15min":0.61899999999999999},"max":{"1min":1.7030000000000001,"5min":2.9870000000000001,"15min":24.456},"last":0.92200000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.72199999999999998,"5min":0.59899999999999998,"15min":0.57899999999999996},"max":{"1min":1.548,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.306,"15min":1.3100000000000001},"min":{"1min":0.875,"5min":0.81299999999999994,"15min":0.69799999999999995},"max":{"1min":1.6579999999999999,"5min":1.7350000000000001,"15min":2.1400000000000001},"last":1.2010000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3049999999999999,"15min":1.304},"min":{"1min":0.86499999999999999,"5min":0.80900000000000005,"15min":0.503},"max":{"1min":1.9039999999999999,"5min":3.0459999999999998,"15min":6.4770000000000003},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.3109999999999999,"15min":1.335},"min":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.71999999999999997},"max":{"1min":1.538,"5min":2.5670000000000002,"15min":6.4589999999999996},"last":1.2589999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.377,"15min":1.391},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.40600000000000003},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.238},"last":1.486},{"interface":"front","average":{"1min":1.486,"5min":1.3919999999999999,"15min":1.4279999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.496},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":6.2990000000000004},"last":1.403}]},{"osd":6,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2749999999999999,"15min":1.2969999999999999},"min":{"1min":0.84799999999999998,"5min":0.66000000000000003,"15min":0.57199999999999995},"max":{"1min":1.6240000000000001,"5min":2.48,"15min":3.0190000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.419,"15min":1.4339999999999999},"min":{"1min":1.004,"5min":0.88,"15min":0.58199999999999996},"max":{"1min":2.0529999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":0.98999999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:17 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4730000000000001,"15min":1.5149999999999999},"min":{"1min":0.91600000000000004,"5min":0.79000000000000004,"15min":0.61799999999999999},"max":{"1min":2.0019999999999998,"5min":2.0529999999999999,"15min":6.5250000000000004},"last":1.46},{"interface":"front","average":{"1min":1.504,"5min":1.5029999999999999,"15min":1.5309999999999999},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.73999999999999999},"max":{"1min":1.738,"5min":2.653,"15min":3.3439999999999999},"last":1.071}]}]},{"osd":3,"up_from":14,"seq":60129542498,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1872984,"kb_used_data":1751632,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92498856,"statfs":{"total":96636764160,"available":94718828544,"internally_reserved":0,"allocated":1793671168,"data_stored":1792834078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91000000000000003,"15min":0.94599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.34699999999999998},"max":{"1min":1.127,"5min":1.7490000000000001,"15min":4.0620000000000003},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.077,"15min":1.083},"min":{"1min":0.628,"5min":0.50700000000000001,"15min":0.34300000000000003},"max":{"1min":1.837,"5min":1.905,"15min":1.905},"last":1.3560000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.998,"15min":1.046},"min":{"1min":0.72999999999999998,"5min":0.61399999999999999,"15min":0.371},"max":{"1min":1.718,"5min":2.2029999999999998,"15min":3.6640000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.67000000000000004,"5min":0.52900000000000003,"15min":0.39400000000000002},"max":{"1min":1.76,"5min":2.1000000000000001,"15min":2.8420000000000001},"last":1.216}]},{"osd":2,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.228,"15min":1.2709999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.45500000000000002},"max":{"1min":1.5880000000000001,"5min":2.258,"15min":3.5249999999999999},"last":1.274},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.3560000000000001,"15min":1.395},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.45200000000000001},"max":{"1min":1.766,"5min":2.339,"15min":4.6529999999999996},"last":1.6879999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.3009999999999999,"15min":1.3240000000000001},"min":{"1min":0.95899999999999996,"5min":0.94399999999999995,"15min":0.70999999999999996},"max":{"1min":1.7490000000000001,"5min":2.0230000000000001,"15min":2.3319999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.444},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.73299999999999998},"max":{"1min":1.633,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.4319999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4770000000000001,"15min":1.5089999999999999},"min":{"1min":0.97499999999999998,"5min":0.84799999999999998,"15min":0.73199999999999998},"max":{"1min":2.169,"5min":2.169,"15min":4.484},"last":2.0329999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.452,"15min":1.4390000000000001},"min":{"1min":1.0129999999999999,"5min":0.92900000000000005,"15min":0.753},"max":{"1min":1.9810000000000001,"5min":2.0139999999999998,"15min":3.4239999999999999},"last":1.5229999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.5309999999999999,"15min":1.5740000000000001},"min":{"1min":0.95499999999999996,"5min":0.92700000000000005,"15min":0.77200000000000002},"max":{"1min":2.157,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.609},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.589,"15min":1.621},"min":{"1min":1.1699999999999999,"5min":1.1240000000000001,"15min":0.73799999999999999},"max":{"1min":2.0600000000000001,"5min":2.169,"15min":4.5880000000000001},"last":1.9550000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.538,"15min":1.554},"min":{"1min":0.93899999999999995,"5min":0.83499999999999996,"15min":0.82399999999999995},"max":{"1min":2.0649999999999999,"5min":2.1030000000000002,"15min":6.3209999999999997},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.641,"15min":1.645},"min":{"1min":1.2969999999999999,"5min":1.008,"15min":0.78700000000000003},"max":{"1min":2.2210000000000001,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":1.732}]}]},{"osd":2,"up_from":14,"seq":60129542496,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1235136,"kb_used_data":1135992,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93136704,"statfs":{"total":96636764160,"available":95371984896,"internally_reserved":0,"allocated":1163255808,"data_stored":1162444991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.2509999999999999,"5min":1.6659999999999999,"15min":1.96},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.0109999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.36299999999999999},"max":{"1min":1.397,"5min":1.7989999999999999,"15min":1.9890000000000001},"last":0.83899999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91000000000000003,"15min":0.99399999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.41499999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":2.093},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2809999999999999,"15min":1.3440000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.863,"5min":1.863,"15min":2.4039999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3500000000000001,"15min":1.3660000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":2.6579999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.224,"15min":1.28},"min":{"1min":0.48199999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6040000000000001,"5min":1.7190000000000001,"15min":2.5070000000000001},"last":1.1859999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.321,"15min":1.407},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.66100000000000003},"max":{"1min":1.6699999999999999,"5min":1.929,"15min":12.669},"last":1.302},{"interface":"front","average":{"1min":1.331,"5min":1.3300000000000001,"15min":1.3680000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.56399999999999995},"max":{"1min":1.744,"5min":1.752,"15min":2.536},"last":1.3740000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3959999999999999,"15min":1.429},"min":{"1min":0.84999999999999998,"5min":0.75900000000000001,"15min":0.68300000000000005},"max":{"1min":2.218,"5min":2.218,"15min":2.3490000000000002},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4399999999999999,"15min":1.496},"min":{"1min":0.94099999999999995,"5min":0.66800000000000004,"15min":0.59799999999999998},"max":{"1min":1.976,"5min":1.976,"15min":2.6589999999999998},"last":1.359}]},{"osd":6,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4450000000000001,"15min":1.5169999999999999},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.64500000000000002},"max":{"1min":1.7869999999999999,"5min":1.9139999999999999,"15min":9.4890000000000008},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.403,"15min":1.5069999999999999},"min":{"1min":0.86299999999999999,"5min":0.751,"15min":0.629},"max":{"1min":1.9399999999999999,"5min":2.0019999999999998,"15min":15.968999999999999},"last":1.5820000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4950000000000001,"15min":1.5980000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.61699999999999999},"max":{"1min":1.7669999999999999,"5min":1.849,"15min":9.3900000000000006},"last":1.121},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.502,"15min":1.5649999999999999},"min":{"1min":0.89800000000000002,"5min":0.873,"15min":0.68600000000000005},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.5720000000000001},"last":1.776}]}]},{"osd":1,"up_from":14,"seq":60129542496,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1435844,"kb_used_data":1336700,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":92935996,"statfs":{"total":96636764160,"available":95166459904,"internally_reserved":0,"allocated":1368780800,"data_stored":1367965819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89500000000000002,"15min":0.92000000000000004},"min":{"1min":0.54900000000000004,"5min":0.53000000000000003,"15min":0.40600000000000003},"max":{"1min":1.2529999999999999,"5min":1.718,"15min":1.865},"last":0.879},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.115,"15min":1.0740000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.38100000000000001},"max":{"1min":1.8879999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":1.2130000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.1319999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.39800000000000002},"max":{"1min":5.1749999999999998,"5min":15.898999999999999,"15min":15.898999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.139,"15min":1.1599999999999999},"min":{"1min":0.78400000000000003,"5min":0.55900000000000005,"15min":0.51200000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.306},"last":1.28}]},{"osd":3,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4399999999999999,"15min":1.4630000000000001},"min":{"1min":0.95199999999999996,"5min":0.76800000000000002,"15min":0.53100000000000003},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3720000000000001,"15min":1.369},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.49099999999999999},"max":{"1min":2.1890000000000001,"5min":4.6269999999999998,"15min":4.6269999999999998},"last":1.6100000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5529999999999999,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":0.97699999999999998,"15min":0.95299999999999996},"max":{"1min":2.516,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":1.526},{"interface":"front","average":{"1min":1.548,"5min":1.5669999999999999,"15min":1.5269999999999999},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.69999999999999996},"max":{"1min":5.6539999999999999,"5min":16.248999999999999,"15min":16.248999999999999},"last":1.1819999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5740000000000001,"15min":1.464},"min":{"1min":0.98399999999999999,"5min":0.97899999999999998,"15min":0.73799999999999999},"max":{"1min":5.4530000000000003,"5min":5.4530000000000003,"15min":5.4530000000000003},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.5469999999999999,"15min":1.5029999999999999},"min":{"1min":1.0409999999999999,"5min":0.85099999999999998,"15min":0.77100000000000002},"max":{"1min":5.5949999999999998,"5min":16.187000000000001,"15min":16.187000000000001},"last":1.069}]},{"osd":6,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.504,"15min":1.51},"min":{"1min":1.145,"5min":0.84599999999999997,"15min":0.84299999999999997},"max":{"1min":2.1150000000000002,"5min":4.5830000000000002,"15min":4.5830000000000002},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.6759999999999999,"15min":1.6020000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.70699999999999996},"max":{"1min":2.3929999999999998,"5min":6.0940000000000003,"15min":6.0940000000000003},"last":1.673}]},{"osd":7,"last update":"Tue Jan 16 15:28:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.657,"15min":1.5840000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.69999999999999996},"max":{"1min":5.6879999999999997,"5min":16.140000000000001,"15min":16.140000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.6739999999999999,"15min":1.615},"min":{"1min":1.091,"5min":0.86399999999999999,"15min":0.83799999999999997},"max":{"1min":2.0209999999999999,"5min":12.848000000000001,"15min":12.848000000000001},"last":1.472}]}]},{"osd":0,"up_from":14,"seq":60129542497,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2073500,"kb_used_data":1997972,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92298340,"statfs":{"total":96636764160,"available":94513500160,"internally_reserved":0,"allocated":2045923328,"data_stored":2045082397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77330881},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0209999999999999,"15min":1.0249999999999999},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.309},"max":{"1min":1.4039999999999999,"5min":4.7999999999999998,"15min":4.8380000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2609999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.33900000000000002},"max":{"1min":1.591,"5min":4.9800000000000004,"15min":5.1970000000000001},"last":1.554}]},{"osd":2,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.105,"15min":1.139},"min":{"1min":0.439,"5min":0.439,"15min":0.40600000000000003},"max":{"1min":1.694,"5min":1.875,"15min":5.0570000000000004},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.097,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.39600000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":5.3550000000000004},"last":0.93000000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.097,"15min":1.105},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.37},"max":{"1min":1.49,"5min":5.851,"15min":5.851},"last":1.024},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.302,"15min":1.3440000000000001},"min":{"1min":0.53800000000000003,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":1.7929999999999999,"5min":1.8620000000000001,"15min":5.1980000000000004},"last":1.6220000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2989999999999999,"15min":1.345},"min":{"1min":0.79900000000000004,"5min":0.79600000000000004,"15min":0.72199999999999998},"max":{"1min":1.645,"5min":2.1389999999999998,"15min":5.1790000000000003},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4370000000000001,"15min":1.46},"min":{"1min":0.91800000000000004,"5min":0.754,"15min":0.60799999999999998},"max":{"1min":2.0880000000000001,"5min":2.141,"15min":5.3109999999999999},"last":1.3440000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.456,"15min":1.5029999999999999},"min":{"1min":0.93899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.804,"5min":2.1469999999999998,"15min":5.1589999999999998},"last":1.734},{"interface":"front","average":{"1min":1.369,"5min":1.4179999999999999,"15min":1.4370000000000001},"min":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.59199999999999997},"max":{"1min":1.8959999999999999,"5min":2.27,"15min":7.2699999999999996},"last":1.48}]},{"osd":6,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.569,"15min":1.6599999999999999},"min":{"1min":1.054,"5min":0.82299999999999995,"15min":0.79800000000000004},"max":{"1min":1.9710000000000001,"5min":2.173,"15min":15.159000000000001},"last":1.823},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.524,"15min":1.6319999999999999},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.621},"max":{"1min":2.0270000000000001,"5min":2.2890000000000001,"15min":15.210000000000001},"last":1.95}]},{"osd":7,"last update":"Tue Jan 16 15:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.6240000000000001,"15min":1.677},"min":{"1min":1.0489999999999999,"5min":0.999,"15min":0.66600000000000004},"max":{"1min":2.5259999999999998,"5min":2.5699999999999998,"15min":5.8209999999999997},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5209999999999999,"15min":1.5820000000000001},"min":{"1min":0.97999999999999998,"5min":0.873,"15min":0.81200000000000006},"max":{"1min":2.605,"5min":2.605,"15min":6.149},"last":1.784}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265664,"data_stored":1246693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":4071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261568,"data_stored":1246651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2042638336,"data_stored":2042626692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1367343104,"data_stored":1367343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161826304,"data_stored":1161822400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1790984192,"data_stored":1790968783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1908420608,"data_stored":1908409022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103106048,"data_stored":1103102213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438654464,"data_stored":1438646655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2071994368,"data_stored":2071986617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:35.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T15:28:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.375+0000 7fc37ca00640 1 Processor -- start 2024-01-16T15:28:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.375+0000 7fc37ca00640 1 -- start start 2024-01-16T15:28:35.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.376+0000 7fc37ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc3780708f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc37ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc378070e30 0x7fc37806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc37ca00640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780b23d0 con 0x7fc378156eb0 2024-01-16T15:28:35.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc37ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3780b2bf0 con 0x7fc378070e30 2024-01-16T15:28:35.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc37ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b2630 con 0x7fc37815a840 2024-01-16T15:28:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc377400640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc378156eb0 0x7fc3780b1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48170/0 (socket says 172.21.15.43:48170) 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc377400640 1 -- 172.21.15.43:0/1171201841 learned_addr learned my addr 172.21.15.43:0/1171201841 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc376a00640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc3780708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.377+0000 7fc377e00640 1 --2- 172.21.15.43:0/1171201841 >> v2:172.21.15.69:3300/0 conn(0x7fc378070e30 0x7fc37806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329784992 0 0) 0x7fc3780b23d0 con 0x7fc378156eb0 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc358003440 con 0x7fc378156eb0 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952187723 0 0) 0x7fc358003440 con 0x7fc378156eb0 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 msgr2=0x7fc3780708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc3780708f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 >> v2:172.21.15.69:3300/0 conn(0x7fc378070e30 msgr2=0x7fc37806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 --2- 172.21.15.43:0/1171201841 >> v2:172.21.15.69:3300/0 conn(0x7fc378070e30 0x7fc37806cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780b2910 con 0x7fc378156eb0 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fc368001690 con 0x7fc378156eb0 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 >> v1:172.21.15.43:6789/0 conn(0x7fc378156eb0 legacy=0x7fc3780b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc358003ad0 0x7fc358003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc35800a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35800ae50 0x7fc35800d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b2bf0 con 0x7fc358003ad0 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b2630 con 0x7fc35800ae50 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.378+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35800db40 con 0x7fc37815a840 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc358003ad0 0x7fc358003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc376a00640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc35800a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377e00640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35800ae50 0x7fc35800d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 -- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 msgr2=0x7fc35800a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc37815a840 0x7fc35800a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 -- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35800ae50 msgr2=0x7fc35800d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc35800ae50 0x7fc35800d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.379+0000 7fc377400640 1 -- 172.21.15.43:0/1171201841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358003440 con 0x7fc358003ad0 2024-01-16T15:28:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.380+0000 7fc377400640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc358003ad0 0x7fc358003ef0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c210 tx=0x7fc3580036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2373d070837ae6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368005ad0 con 0x7fc358003ad0 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368005c50 con 0x7fc358003ad0 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc376000640 1 -- 172.21.15.43:0/1171201841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36802caf0 con 0x7fc358003ad0 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc37ca00640 1 -- 172.21.15.43:0/1171201841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc358003ad0 msgr2=0x7fc358003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc37ca00640 1 --2- 172.21.15.43:0/1171201841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc358003ad0 0x7fc358003ef0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c210 tx=0x7fc3580036f0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc377e00640 1 -- 172.21.15.43:0/1171201841 reap_dead start 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc37ca00640 1 -- 172.21.15.43:0/1171201841 shutdown_connections 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.381+0000 7fc37ca00640 1 -- 172.21.15.43:0/1171201841 >> 172.21.15.43:0/1171201841 conn(0x7fc3780cec10 msgr2=0x7fc3780cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.382+0000 7fc37ca00640 1 -- 172.21.15.43:0/1171201841 shutdown_connections 2024-01-16T15:28:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.382+0000 7fc37ca00640 1 -- 172.21.15.43:0/1171201841 wait complete. 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.383+0000 7fc37ca00640 1 Processor -- start 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.384+0000 7fc37ca00640 1 -- start start 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3780cd7a0 0x7fc378133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc378155ab0 0x7fc378137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378155f40 con 0x7fc3780cdb80 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780c9170 con 0x7fc3780cd7a0 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc37ca00640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780c9440 con 0x7fc378155ab0 2024-01-16T15:28:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47694/0 (socket says 172.21.15.43:47694) 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 -- 172.21.15.43:0/241726204 learned_addr learned my addr 172.21.15.43:0/241726204 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc377400640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3780cd7a0 0x7fc378133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc377e00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc378155ab0 0x7fc378137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 -- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc378155ab0 msgr2=0x7fc378137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc378155ab0 0x7fc378137e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 -- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3780cd7a0 msgr2=0x7fc378133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3780cd7a0 0x7fc378133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.385+0000 7fc376a00640 1 -- 172.21.15.43:0/241726204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780c9720 con 0x7fc3780cdb80 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc376a00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fc360011600 tx=0x7fc360011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36000f920 con 0x7fc3780cdb80 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36000fa80 con 0x7fc3780cdb80 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36000fdf0 con 0x7fc3780cdb80 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3780c9a00 con 0x7fc3780cdb80 2024-01-16T15:28:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.386+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc378138370 con 0x7fc3780cdb80 2024-01-16T15:28:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.388+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37804e840 con 0x7fc3780cdb80 2024-01-16T15:28:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.389+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 54) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc360015460 con 0x7fc3780cdb80 2024-01-16T15:28:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.390+0000 7fc36f400640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 0x7fc35006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.390+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc360088fc0 con 0x7fc3780cdb80 2024-01-16T15:28:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.390+0000 7fc377400640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 0x7fc35006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.391+0000 7fc377400640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 0x7fc35006e8f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc368034000 tx=0x7fc368007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.395+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc360055530 con 0x7fc3780cdb80 2024-01-16T15:28:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.595+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fc378157750 con 0x7fc3780cdb80 2024-01-16T15:28:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.602+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v45)=0 v45) v1 ==== 123+0+0 (secure 0 0 0) 0x7fc360058b50 con 0x7fc3780cdb80 2024-01-16T15:28:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.605+0000 7fc36f400640 1 -- 172.21.15.43:0/241726204 <== mon.0 v2:172.21.15.43:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc360055530 con 0x7fc3780cdb80 2024-01-16T15:28:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.606+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 msgr2=0x7fc35006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.606+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 0x7fc35006e8f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc368034000 tx=0x7fc368007000 comp rx=0 tx=0).stop 2024-01-16T15:28:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.606+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 msgr2=0x7fc378155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.606+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fc360011600 tx=0x7fc360011ac0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 shutdown_connections 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fc35006c430 0x7fc35006e8f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc378155ab0 0x7fc378137e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780cdb80 0x7fc378155570 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 --2- 172.21.15.43:0/241726204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3780cd7a0 0x7fc378133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 >> 172.21.15.43:0/241726204 conn(0x7fc3780cec10 msgr2=0x7fc37813f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 shutdown_connections 2024-01-16T15:28:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.607+0000 7fc37ca00640 1 -- 172.21.15.43:0/241726204 wait complete. 2024-01-16T15:28:35.625 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.625+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.625 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.625+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.625 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.625+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.625 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.625+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.625 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.625+0000 7fd4e7800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.626 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.627+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.626 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.627+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.627 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.627+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.627 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.627+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.627 INFO:tasks.ceph.mgr.y.smithi043.stderr:2024-01-16T15:28:35.627+0000 7fd4ec800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:28:35.629 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:28:35.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:28:35.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.821+0000 7fda91000640 1 Processor -- start 2024-01-16T15:28:35.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.821+0000 7fda91000640 1 -- start start 2024-01-16T15:28:35.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.822+0000 7fda91000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda8c13dfd0 0x7fda8c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda91000640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c0ce7c0 con 0x7fda8c0cf500 2024-01-16T15:28:35.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda91000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c140900 con 0x7fda8c0cf050 2024-01-16T15:28:35.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda91000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c06cf50 con 0x7fda8c13dfd0 2024-01-16T15:28:35.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda8be00640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fda8c0cf500 0x7fda8c06c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48174/0 (socket says 172.21.15.43:48174) 2024-01-16T15:28:35.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda8be00640 1 -- 172.21.15.43:0/1961918819 learned_addr learned my addr 172.21.15.43:0/1961918819 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda8b400640 1 --2- 172.21.15.43:0/1961918819 >> v2:172.21.15.69:3300/0 conn(0x7fda8c13dfd0 0x7fda8c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359739827 0 0) 0x7fda8c0ce7c0 con 0x7fda8c0cf500 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.823+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c003520 con 0x7fda8c0cf500 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2032081708 0 0) 0x7fda6c003520 con 0x7fda8c0cf500 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 >> v2:172.21.15.69:3300/0 conn(0x7fda8c13dfd0 msgr2=0x7fda8c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 --2- 172.21.15.43:0/1961918819 >> v2:172.21.15.69:3300/0 conn(0x7fda8c13dfd0 0x7fda8c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 >> v1:172.21.15.69:6789/0 conn(0x7fda8c0cf050 legacy=0x7fda8c06c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c15fa70 con 0x7fda8c0cf500 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2647882321 0 0) 0x7fda84001670 con 0x7fda8c0cf500 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 >> v1:172.21.15.43:6789/0 conn(0x7fda8c0cf500 legacy=0x7fda8c06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda6c003a30 0x7fda6c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 0x7fda6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0051d0 0x7fda6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c06cf50 con 0x7fda6c0047c0 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c100 con 0x7fda6c0051d0 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c2c0 con 0x7fda6c003a30 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.824+0000 7fda8aa00640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 0x7fda6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8be00640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0051d0 0x7fda6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8b400640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda6c003a30 0x7fda6c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8aa00640 1 -- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda6c003a30 msgr2=0x7fda6c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8aa00640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda6c003a30 0x7fda6c003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8aa00640 1 -- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0051d0 msgr2=0x7fda6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8aa00640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0051d0 0x7fda6c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8aa00640 1 -- 172.21.15.43:0/1961918819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c003520 con 0x7fda6c0047c0 2024-01-16T15:28:35.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.825+0000 7fda8be00640 1 -- 172.21.15.43:0/1961918819 reap_dead start 2024-01-16T15:28:35.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.826+0000 7fda8aa00640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 0x7fda6c004c40 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fda74001e50 tx=0x7fda6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7fcb03f7d67843 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.826+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74003b70 con 0x7fda6c0047c0 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.826+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fda74003cd0 con 0x7fda6c0047c0 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.826+0000 7fda8a000640 1 -- 172.21.15.43:0/1961918819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74005330 con 0x7fda6c0047c0 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 -- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 msgr2=0x7fda6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 0x7fda6c004c40 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fda74001e50 tx=0x7fda6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 -- 172.21.15.43:0/1961918819 shutdown_connections 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 --2- 172.21.15.43:0/1961918819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda6c0047c0 0x7fda6c004c40 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 -- 172.21.15.43:0/1961918819 >> 172.21.15.43:0/1961918819 conn(0x7fda8c06a860 msgr2=0x7fda8c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 -- 172.21.15.43:0/1961918819 shutdown_connections 2024-01-16T15:28:35.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.827+0000 7fda91000640 1 -- 172.21.15.43:0/1961918819 wait complete. 2024-01-16T15:28:35.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.829+0000 7fda91000640 1 Processor -- start 2024-01-16T15:28:35.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.829+0000 7fda91000640 1 -- start start 2024-01-16T15:28:35.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0047c0 0x7fda8c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8c1565d0 0x7fda8c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c156f90 con 0x7fda8c13f400 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c13fda0 con 0x7fda6c0047c0 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda91000640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15a1e0 con 0x7fda8c1565d0 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47760/0 (socket says 172.21.15.43:47760) 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8aa00640 1 -- 172.21.15.43:0/84472415 learned_addr learned my addr 172.21.15.43:0/84472415 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-01-16T15:28:35.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8b400640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0047c0 0x7fda8c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8be00640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8c1565d0 0x7fda8c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8aa00640 1 -- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8c1565d0 msgr2=0x7fda8c156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.830+0000 7fda8aa00640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8c1565d0 0x7fda8c156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda8aa00640 1 -- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0047c0 msgr2=0x7fda8c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda8aa00640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0047c0 0x7fda8c13c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda8aa00640 1 -- 172.21.15.43:0/84472415 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c15c200 con 0x7fda8c13f400 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda8aa00640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fda74005040 tx=0x7fda740014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7403e070 con 0x7fda8c13f400 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fda74006b50 con 0x7fda8c13f400 2024-01-16T15:28:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74007030 con 0x7fda8c13f400 2024-01-16T15:28:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c1571b0 con 0x7fda8c13f400 2024-01-16T15:28:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.831+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c157770 con 0x7fda8c13f400 2024-01-16T15:28:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.833+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c15b920 con 0x7fda8c13f400 2024-01-16T15:28:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.833+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 54) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fda74007190 con 0x7fda8c13f400 2024-01-16T15:28:35.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.834+0000 7fda83400640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 0x7fda6406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:35.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.834+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fda740aa0c0 con 0x7fda8c13f400 2024-01-16T15:28:35.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.835+0000 7fda8b400640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 0x7fda6406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:35.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.835+0000 7fda8b400640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 0x7fda6406e8f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009a90 tx=0x7fda7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:35.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:35.839+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda74076950 con 0x7fda8c13f400 2024-01-16T15:28:36.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.073+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda8c15c390 con 0x7fda8c13f400 2024-01-16T15:28:36.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.074+0000 7fda83400640 1 -- 172.21.15.43:0/84472415 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fda74048030 con 0x7fda8c13f400 2024-01-16T15:28:36.073 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-01-16T15:28:36.074 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":76,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3583583840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":3583583840},{"type":"v1","addr":"172.21.15.43:6837","nonce":3583583840}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2071477429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2071477429},{"type":"v1","addr":"172.21.15.69:6833","nonce":2071477429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6836/4168252148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4168252148},{"type":"v1","addr":"172.21.15.43:6836","nonce":4168252148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:18:49.965316+0000","modified":"2024-01-16T15:18:53.023347+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9132":{"gid":9132,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/3715375194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3715375194},{"type":"v1","addr":"172.21.15.69:6835","nonce":3715375194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:28:36.074 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 76 2024-01-16T15:28:36.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.076+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 msgr2=0x7fda6406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:36.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.076+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 0x7fda6406e8f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009a90 tx=0x7fda7c008040 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.076+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 msgr2=0x7fda8c13f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.076+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fda74005040 tx=0x7fda740014a0 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 shutdown_connections 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:6832/504016423,v1:172.21.15.43:6833/504016423] conn(0x7fda6406c430 0x7fda6406e8f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda8c1565d0 0x7fda8c156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda8c13f400 0x7fda8c13f860 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 --2- 172.21.15.43:0/84472415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda6c0047c0 0x7fda8c13c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 >> 172.21.15.43:0/84472415 conn(0x7fda8c06a860 msgr2=0x7fda8c0acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 shutdown_connections 2024-01-16T15:28:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-01-16T15:28:36.077+0000 7fda91000640 1 -- 172.21.15.43:0/84472415 wait complete. 2024-01-16T15:28:36.094 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-16T15:28:36.094 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T15:28:36.094 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:28:36.094 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:36.113 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:28:36.113 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:28:36.114 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:36.187 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:28:36.187 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:28:36.187 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:36.219 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:28:36.220 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:28:36.220 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:36.286 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:28:36.286 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T15:28:36.286 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T15:28:36.286 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:39.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-16T15:28:42.388 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T15:28:42.389 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T15:28:42.389 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:45.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-16T15:28:45.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-16T15:28:48.491 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T15:28:48.491 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T15:28:48.492 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:48.587 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T15:28:48.588 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-16T15:28:48.588 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:51.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-16T15:28:51.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-16T15:28:54.690 INFO:tasks.ceph.osd.3:Stopped 2024-01-16T15:28:54.690 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-16T15:28:54.691 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:28:57.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-16T15:28:57.069 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-01-16T15:29:00.792 INFO:tasks.ceph.osd.4:Stopped 2024-01-16T15:29:00.793 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-16T15:29:00.793 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:02.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-16T15:29:02.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-16T15:29:03.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:03.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:06.895 INFO:tasks.ceph.osd.5:Stopped 2024-01-16T15:29:06.896 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-16T15:29:06.896 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:06.973 INFO:tasks.ceph.osd.6:Stopped 2024-01-16T15:29:06.974 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-16T15:29:06.974 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:08.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-16T15:29:08.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-01-16T15:29:13.076 INFO:tasks.ceph.osd.7:Stopped 2024-01-16T15:29:13.076 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T15:29:13.077 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-16T15:29:13.077 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:13.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-16T15:29:13.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-16T15:29:19.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~5s 2024-01-16T15:29:19.178 INFO:tasks.ceph.mgr.y:Stopped 2024-01-16T15:29:19.179 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T15:29:19.179 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:19.256 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T15:29:19.256 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T15:29:19.256 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T15:29:19.256 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:19.316 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T15:29:19.317 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-16T15:29:19.317 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:19.349 INFO:tasks.ceph.mon.b:Stopped 2024-01-16T15:29:19.350 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-16T15:29:19.350 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:19.434 INFO:tasks.ceph.mon.c:Stopped 2024-01-16T15:29:19.434 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T15:29:19.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-16T15:29:19.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:29:19.550 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T15:29:19.742 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:29:19.742 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T15:29:19.909 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:29:19.909 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T15:29:20.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:29:20.073 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-16T15:29:20.261 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:29:20.261 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-16T15:29:20.486 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:29:20.486 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-16T15:29:20.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:29:20.632 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-16T15:29:20.787 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:29:20.787 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-16T15:29:20.980 INFO:tasks.ceph:Archiving mon data... 2024-01-16T15:29:20.981 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/data/mon.a.tgz 2024-01-16T15:29:20.982 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-01-16T15:29:21.043 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.iC48tAuGAC 2024-01-16T15:29:21.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iC48tAuGAC 2024-01-16T15:29:21.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.iC48tAuGAC 2024-01-16T15:29:21.578 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.iC48tAuGAC is 1MB 2024-01-16T15:29:21.710 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.iC48tAuGAC 2024-01-16T15:29:21.769 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/data/mon.b.tgz 2024-01-16T15:29:21.769 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-16T15:29:21.824 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.Z20ZcQWLfX 2024-01-16T15:29:21.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Z20ZcQWLfX 2024-01-16T15:29:22.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.Z20ZcQWLfX 2024-01-16T15:29:22.306 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.Z20ZcQWLfX is 1MB 2024-01-16T15:29:22.386 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.Z20ZcQWLfX 2024-01-16T15:29:22.441 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/data/mon.c.tgz 2024-01-16T15:29:22.442 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-16T15:29:22.492 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.SaodmlA1SG 2024-01-16T15:29:22.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SaodmlA1SG 2024-01-16T15:29:22.834 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.SaodmlA1SG 2024-01-16T15:29:22.968 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.SaodmlA1SG is 1MB 2024-01-16T15:29:23.088 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.SaodmlA1SG 2024-01-16T15:29:23.144 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T15:29:23.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:29:23.153 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:29:23.312 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:29:23.368 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:29:23.368 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T15:29:23.370 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi043/crash 2024-01-16T15:29:23.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:29:23.477 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi069/crash 2024-01-16T15:29:23.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:29:23.572 INFO:tasks.ceph:Shutting down logrotate 2024-01-16T15:29:23.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:23.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:23.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:23.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:23.917 INFO:tasks.ceph:Compressing logs... 2024-01-16T15:29:23.917 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:29:23.925 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:29:24.013 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36402.log 2024-01-16T15:29:24.014 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-16T15:29:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41518.log 2024-01-16T15:29:24.015 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36402.log.gz 2024-01-16T15:29:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-01-16T15:29:24.016 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75523.log 2024-01-16T15:29:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.41518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41970.log 2024-01-16T15:29:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69877.log 2024-01-16T15:29:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36202.log: 90.9% -- replaced with /var/log/ceph/ceph-client.1.41518.log.gz 2024-01-16T15:29:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36242.log 2024-01-16T15:29:24.018 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-01-16T15:29:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.3.41970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44850.log 2024-01-16T15:29:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65078.log 2024-01-16T15:29:24.020 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75523.log.gz 2024-01-16T15:29:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69877.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69877.log.gz 2024-01-16T15:29:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68964.log 2024-01-16T15:29:24.021 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65078.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65078.log.gz 2024-01-16T15:29:24.021 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41760.log 2024-01-16T15:29:24.022 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36242.log: 91.7% -- replaced with /var/log/ceph/ceph-client.3.41970.log.gz 2024-01-16T15:29:24.022 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36242.log.gz 2024-01-16T15:29:24.022 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:29:24.022 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.44850.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-16T15:29:24.024 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36122.log 2024-01-16T15:29:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81621.log 2024-01-16T15:29:24.024 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68964.log.gz 2024-01-16T15:29:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53854.log 2024-01-16T15:29:24.025 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41760.log.gz 2024-01-16T15:29:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82152.log 2024-01-16T15:29:24.025 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81621.log.gz 2024-01-16T15:29:24.026 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-01-16T15:29:24.026 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-16T15:29:24.026 INFO:teuthology.orchestra.run.smithi069.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.1.44850.log.gz 2024-01-16T15:29:24.026 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36282.log 2024-01-16T15:29:24.027 INFO:teuthology.orchestra.run.smithi069.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36122.log.gz 2024-01-16T15:29:24.028 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36322.log 2024-01-16T15:29:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.82152.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53854.log.gz 2024-01-16T15:29:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77087.log 2024-01-16T15:29:24.029 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82152.log.gz 2024-01-16T15:29:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45448.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-01-16T15:29:24.029 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55391.log 2024-01-16T15:29:24.030 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36282.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36282.log.gz 2024-01-16T15:29:24.030 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36162.log 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42712.log 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65942.log 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77087.log.gz 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41079.log 2024-01-16T15:29:24.031 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55391.log.gz 2024-01-16T15:29:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65942.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65942.log.gz 2024-01-16T15:29:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-01-16T15:29:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75271.log 2024-01-16T15:29:24.033 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41079.log.gz 2024-01-16T15:29:24.033 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36322.log.gz 2024-01-16T15:29:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36162.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36162.log.gz 2024-01-16T15:29:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-16T15:29:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.42712.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-16T15:29:24.035 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45873.log 2024-01-16T15:29:24.035 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.1.42712.log.gz 2024-01-16T15:29:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41956.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-01-16T15:29:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70184.log 2024-01-16T15:29:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75271.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75271.log.gz 2024-01-16T15:29:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49737.log 2024-01-16T15:29:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46684.log 2024-01-16T15:29:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46566.log 2024-01-16T15:29:24.037 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70184.log.gz 2024-01-16T15:29:24.038 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49737.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49737.log.gz 2024-01-16T15:29:24.038 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73189.log 2024-01-16T15:29:24.038 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46684.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46684.log.gz 2024-01-16T15:29:24.039 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T15:29:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.45873.log: /var/log/ceph/ceph-client.1.46566.log: 93.8% -- replaced with /var/log/ceph/ceph-client.1.45873.log.gz 2024-01-16T15:29:24.040 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43404.log 2024-01-16T15:29:24.040 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49451.log 2024-01-16T15:29:24.040 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73189.log.gz 2024-01-16T15:29:24.041 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-01-16T15:29:24.041 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67343.log 2024-01-16T15:29:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49451.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49451.log.gz 2024-01-16T15:29:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50270.log 2024-01-16T15:29:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.43404.log: 95.7% -- replaced with /var/log/ceph/ceph-client.1.46566.log.gz 2024-01-16T15:29:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66170.log 2024-01-16T15:29:24.044 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67343.log.gz 2024-01-16T15:29:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36362.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz 2024-01-16T15:29:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T15:29:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50270.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50270.log.gz 2024-01-16T15:29:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68136.log 2024-01-16T15:29:24.046 INFO:teuthology.orchestra.run.smithi069.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.1.43404.log.gz 2024-01-16T15:29:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47516.log 2024-01-16T15:29:24.046 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66170.log.gz 2024-01-16T15:29:24.047 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47352.log 2024-01-16T15:29:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68136.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.68136.log.gz 2024-01-16T15:29:24.047 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77694.log 2024-01-16T15:29:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:29:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69734.log 2024-01-16T15:29:24.049 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47516.log.gz 2024-01-16T15:29:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77694.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77694.log.gz 2024-01-16T15:29:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58344.log 2024-01-16T15:29:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69734.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69734.log.gz 2024-01-16T15:29:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77339.log 2024-01-16T15:29:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.47352.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44117.log 2024-01-16T15:29:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-16T15:29:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62199.log 2024-01-16T15:29:24.053 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58344.log.gz 2024-01-16T15:29:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58260.log 2024-01-16T15:29:24.054 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77339.log.gz 2024-01-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62199.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62199.log.gz 2024-01-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68262.log 2024-01-16T15:29:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65036.log 2024-01-16T15:29:24.056 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-01-16T15:29:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68262.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68262.log.gz 2024-01-16T15:29:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78441.log 2024-01-16T15:29:24.058 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.44117.log: 96.6%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph-client.1.47352.log.gz 2024-01-16T15:29:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-01-16T15:29:24.059 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65036.log.gz 2024-01-16T15:29:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80353.log 2024-01-16T15:29:24.060 INFO:teuthology.orchestra.run.smithi043.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78441.log.gz 2024-01-16T15:29:24.061 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-16T15:29:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55641.log 2024-01-16T15:29:24.062 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-01-16T15:29:24.062 INFO:teuthology.orchestra.run.smithi069.stderr: 95.0%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-16T15:29:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56817.log 2024-01-16T15:29:24.063 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80353.log.gz 2024-01-16T15:29:24.064 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.d.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:29:24.064 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.1.44117.log.gz 2024-01-16T15:29:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57292.log 2024-01-16T15:29:24.065 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55641.log.gz 2024-01-16T15:29:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59201.log 2024-01-16T15:29:24.066 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56817.log.gz 2024-01-16T15:29:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57460.log 2024-01-16T15:29:24.066 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57292.log.gz 2024-01-16T15:29:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54643.log 2024-01-16T15:29:24.068 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59201.log.gz 2024-01-16T15:29:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74548.log 2024-01-16T15:29:24.069 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57460.log.gz 2024-01-16T15:29:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73304.log 2024-01-16T15:29:24.070 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54643.log.gz 2024-01-16T15:29:24.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74548.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74548.log.gz 2024-01-16T15:29:24.072 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55557.log 2024-01-16T15:29:24.073 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T15:29:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46768.log 2024-01-16T15:29:24.073 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73304.log.gz 2024-01-16T15:29:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65690.log 2024-01-16T15:29:24.075 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55557.log.gz 2024-01-16T15:29:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46558.log 2024-01-16T15:29:24.076 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46768.log.gz 2024-01-16T15:29:24.077 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69632.log 2024-01-16T15:29:24.077 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65690.log.gz 2024-01-16T15:29:24.078 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57586.log 2024-01-16T15:29:24.078 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46558.log.gz 2024-01-16T15:29:24.079 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74380.log 2024-01-16T15:29:24.079 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69632.log.gz 2024-01-16T15:29:24.080 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57586.log.gz 2024-01-16T15:29:24.080 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49695.log 2024-01-16T15:29:24.081 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53770.log 2024-01-16T15:29:24.082 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74380.log.gz 2024-01-16T15:29:24.083 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41406.log 2024-01-16T15:29:24.083 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49695.log.gz 2024-01-16T15:29:24.084 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60864.log 2024-01-16T15:29:24.084 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53770.log.gz 2024-01-16T15:29:24.085 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41406.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41406.log.gz 2024-01-16T15:29:24.085 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66422.log 2024-01-16T15:29:24.086 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72400.log 2024-01-16T15:29:24.086 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60864.log.gz 2024-01-16T15:29:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2024-01-16T15:29:24.087 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66422.log.gz 2024-01-16T15:29:24.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-01-16T15:29:24.089 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72400.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72400.log.gz 2024-01-16T15:29:24.089 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78400.log 2024-01-16T15:29:24.090 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46642.log 2024-01-16T15:29:24.090 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2024-01-16T15:29:24.091 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79931.log 2024-01-16T15:29:24.091 INFO:teuthology.orchestra.run.smithi043.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.78400.log.gz 2024-01-16T15:29:24.092 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61556.log 2024-01-16T15:29:24.092 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46642.log.gz 2024-01-16T15:29:24.094 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74221.log 2024-01-16T15:29:24.094 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79931.log.gz 2024-01-16T15:29:24.095 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75649.log 2024-01-16T15:29:24.095 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61556.log.gz 2024-01-16T15:29:24.096 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62999.log 2024-01-16T15:29:24.096 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74221.log.gz 2024-01-16T15:29:24.097 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61980.log 2024-01-16T15:29:24.097 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75649.log.gz 2024-01-16T15:29:24.098 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59075.log 2024-01-16T15:29:24.098 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62999.log.gz 2024-01-16T15:29:24.099 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45018.log 2024-01-16T15:29:24.100 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61980.log.gz 2024-01-16T15:29:24.101 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50916.log 2024-01-16T15:29:24.101 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59075.log.gz 2024-01-16T15:29:24.102 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42334.log 2024-01-16T15:29:24.102 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45018.log.gz 2024-01-16T15:29:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80657.log 2024-01-16T15:29:24.104 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50916.log.gz 2024-01-16T15:29:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52705.log 2024-01-16T15:29:24.105 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42334.log.gz 2024-01-16T15:29:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59870.log 2024-01-16T15:29:24.106 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80657.log.gz 2024-01-16T15:29:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68010.log 2024-01-16T15:29:24.107 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52705.log.gz 2024-01-16T15:29:24.108 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48246.log 2024-01-16T15:29:24.108 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59870.log.gz 2024-01-16T15:29:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48162.log 2024-01-16T15:29:24.109 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68010.log.gz 2024-01-16T15:29:24.111 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47272.log 2024-01-16T15:29:24.111 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48246.log.gz 2024-01-16T15:29:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64585.log 2024-01-16T15:29:24.112 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48162.log.gz 2024-01-16T15:29:24.113 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65414.log 2024-01-16T15:29:24.113 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47272.log.gz 2024-01-16T15:29:24.114 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50511.log 2024-01-16T15:29:24.114 INFO:teuthology.orchestra.run.smithi043.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64585.log.gz 2024-01-16T15:29:24.115 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58949.log 2024-01-16T15:29:24.115 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65414.log.gz 2024-01-16T15:29:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73032.log 2024-01-16T15:29:24.117 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50511.log.gz 2024-01-16T15:29:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65330.log 2024-01-16T15:29:24.118 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58949.log.gz 2024-01-16T15:29:24.119 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58471.log 2024-01-16T15:29:24.119 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73032.log.gz 2024-01-16T15:29:24.120 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51573.log 2024-01-16T15:29:24.121 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65330.log.gz 2024-01-16T15:29:24.121 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63772.log 2024-01-16T15:29:24.122 INFO:teuthology.orchestra.run.smithi043.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58471.log.gz 2024-01-16T15:29:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71224.log 2024-01-16T15:29:24.123 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51573.log.gz 2024-01-16T15:29:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63587.log 2024-01-16T15:29:24.124 INFO:teuthology.orchestra.run.smithi043.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.63772.log.gz 2024-01-16T15:29:24.125 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-01-16T15:29:24.125 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71224.log.gz 2024-01-16T15:29:24.126 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51443.log 2024-01-16T15:29:24.127 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63587.log.gz 2024-01-16T15:29:24.127 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44713.log 2024-01-16T15:29:24.128 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-01-16T15:29:24.128 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77971.log 2024-01-16T15:29:24.129 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51443.log.gz 2024-01-16T15:29:24.130 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-01-16T15:29:24.130 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44713.log.gz 2024-01-16T15:29:24.131 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77971.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77971.log.gz 2024-01-16T15:29:24.131 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51317.log 2024-01-16T15:29:24.132 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81117.log 2024-01-16T15:29:24.132 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-01-16T15:29:24.133 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63083.log 2024-01-16T15:29:24.133 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51317.log.gz 2024-01-16T15:29:24.135 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-01-16T15:29:24.135 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81117.log.gz 2024-01-16T15:29:24.136 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63083.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63083.log.gz -5 2024-01-16T15:29:24.136 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77550.log 2024-01-16T15:29:24.137 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-01-16T15:29:24.137 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-01-16T15:29:24.138 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39897.log 2024-01-16T15:29:24.138 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77550.log.gz 2024-01-16T15:29:24.139 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46348.log 2024-01-16T15:29:24.139 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-01-16T15:29:24.140 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.39897.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39897.log.gz 2024-01-16T15:29:24.140 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2024-01-16T15:29:24.142 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44293.log 2024-01-16T15:29:24.142 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40738.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46348.log.gz 2024-01-16T15:29:24.142 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2024-01-16T15:29:24.143 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70715.log 2024-01-16T15:29:24.144 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2024-01-16T15:29:24.144 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44293.log.gz 2024-01-16T15:29:24.145 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61098.log 2024-01-16T15:29:24.145 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70715.log.gz 2024-01-16T15:29:24.146 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58218.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz /var/log/ceph/ceph-client.admin.43264.log 2024-01-16T15:29:24.146 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-01-16T15:29:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61098.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61098.log.gz 2024-01-16T15:29:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75143.log 2024-01-16T15:29:24.149 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.43264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45658.log 2024-01-16T15:29:24.149 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-01-16T15:29:24.150 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78951.log 2024-01-16T15:29:24.150 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75143.log.gz 2024-01-16T15:29:24.151 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45658.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45658.log.gz 2024-01-16T15:29:24.151 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67702.log 2024-01-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54461.log 2024-01-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78951.log.gz 2024-01-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79461.log 2024-01-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67702.log.gz 2024-01-16T15:29:24.154 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54461.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54461.log.gz 2024-01-16T15:29:24.155 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72484.log 2024-01-16T15:29:24.156 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75015.log 2024-01-16T15:29:24.156 INFO:teuthology.orchestra.run.smithi043.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79461.log.gz 2024-01-16T15:29:24.156 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72568.log 2024-01-16T15:29:24.157 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72484.log.gz 2024-01-16T15:29:24.158 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75015.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75015.log.gz 2024-01-16T15:29:24.158 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-01-16T15:29:24.159 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50120.log 2024-01-16T15:29:24.159 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72568.log.gz 2024-01-16T15:29:24.160 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74632.log 2024-01-16T15:29:24.160 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-01-16T15:29:24.161 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50120.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50120.log.gz 2024-01-16T15:29:24.161 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-01-16T15:29:24.163 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81984.log 2024-01-16T15:29:24.163 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74632.log.gz 2024-01-16T15:29:24.163 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69006.log 2024-01-16T15:29:24.164 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-01-16T15:29:24.165 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81984.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81984.log.gz 2024-01-16T15:29:24.165 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68094.log 2024-01-16T15:29:24.166 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43124.log 2024-01-16T15:29:24.166 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69006.log.gz 2024-01-16T15:29:24.167 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69919.log 2024-01-16T15:29:24.167 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68094.log.gz 2024-01-16T15:29:24.168 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.43124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57157.log 2024-01-16T15:29:24.169 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.0.43124.log.gz 2024-01-16T15:29:24.169 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44934.log 2024-01-16T15:29:24.169 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69919.log.gz 2024-01-16T15:29:24.170 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73543.log 2024-01-16T15:29:24.171 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57157.log.gz 2024-01-16T15:29:24.171 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44934.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44934.log.gz 2024-01-16T15:29:24.172 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63251.log 2024-01-16T15:29:24.172 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2024-01-16T15:29:24.173 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73543.log.gz 2024-01-16T15:29:24.174 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49781.log 2024-01-16T15:29:24.174 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63251.log.gz 2024-01-16T15:29:24.175 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50958.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2024-01-16T15:29:24.175 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51793.log 2024-01-16T15:29:24.176 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81663.log 2024-01-16T15:29:24.176 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49781.log.gz 2024-01-16T15:29:24.177 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53021.log 2024-01-16T15:29:24.177 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51793.log.gz 2024-01-16T15:29:24.178 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45574.log 2024-01-16T15:29:24.178 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81663.log.gz 2024-01-16T15:29:24.179 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40133.log 2024-01-16T15:29:24.180 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53021.log.gz 2024-01-16T15:29:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-01-16T15:29:24.181 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45574.log.gz 2024-01-16T15:29:24.182 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40133.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40133.log.gz 2024-01-16T15:29:24.182 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62409.log 2024-01-16T15:29:24.183 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-01-16T15:29:24.183 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-01-16T15:29:24.184 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-01-16T15:29:24.184 INFO:teuthology.orchestra.run.smithi043.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62409.log.gz 2024-01-16T15:29:24.185 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45942.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-01-16T15:29:24.185 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55729.log 2024-01-16T15:29:24.186 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80295.log 2024-01-16T15:29:24.186 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-01-16T15:29:24.187 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81285.log 2024-01-16T15:29:24.187 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55729.log.gz 2024-01-16T15:29:24.188 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80295.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80295.log.gz 2024-01-16T15:29:24.188 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64355.log 2024-01-16T15:29:24.189 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77851.log 2024-01-16T15:29:24.190 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81285.log.gz 2024-01-16T15:29:24.190 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2024-01-16T15:29:24.191 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64355.log.gz 2024-01-16T15:29:24.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77851.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77851.log.gz 2024-01-16T15:29:24.192 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80951.log 2024-01-16T15:29:24.193 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77592.log 2024-01-16T15:29:24.193 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2024-01-16T15:29:24.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66506.log 2024-01-16T15:29:24.194 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80951.log.gz 2024-01-16T15:29:24.195 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77592.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.77592.log.gz 2024-01-16T15:29:24.195 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54922.log 2024-01-16T15:29:24.196 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2024-01-16T15:29:24.196 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66506.log.gz 2024-01-16T15:29:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69961.log 2024-01-16T15:29:24.197 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54922.log.gz 2024-01-16T15:29:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40528.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gz 2024-01-16T15:29:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81495.log 2024-01-16T15:29:24.199 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54300.log 2024-01-16T15:29:24.200 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69961.log.gz 2024-01-16T15:29:24.200 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81916.log 2024-01-16T15:29:24.201 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81495.log.gz 2024-01-16T15:29:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54300.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54300.log.gz 2024-01-16T15:29:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51401.log 2024-01-16T15:29:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66649.log 2024-01-16T15:29:24.203 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81916.log.gz 2024-01-16T15:29:24.204 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46144.log 2024-01-16T15:29:24.204 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51401.log.gz 2024-01-16T15:29:24.205 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66649.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66649.log.gz 2024-01-16T15:29:24.205 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67786.log 2024-01-16T15:29:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79543.log 2024-01-16T15:29:24.206 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46144.log.gz 2024-01-16T15:29:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70799.log 2024-01-16T15:29:24.207 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67786.log.gz 2024-01-16T15:29:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79543.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79543.log.gz 2024-01-16T15:29:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59302.log 2024-01-16T15:29:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55685.log 2024-01-16T15:29:24.210 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70799.log.gz 2024-01-16T15:29:24.210 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-01-16T15:29:24.211 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59302.log.gz 2024-01-16T15:29:24.212 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55685.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.55685.log.gz --verbose 2024-01-16T15:29:24.212 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.59828.log 2024-01-16T15:29:24.213 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80211.log 2024-01-16T15:29:24.213 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-01-16T15:29:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58612.log 2024-01-16T15:29:24.214 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59828.log.gz 2024-01-16T15:29:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80211.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80211.log.gz 2024-01-16T15:29:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81327.log 2024-01-16T15:29:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59159.log 2024-01-16T15:29:24.216 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58612.log.gz 2024-01-16T15:29:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50589.log 2024-01-16T15:29:24.217 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81327.log.gz 2024-01-16T15:29:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59159.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59159.log.gz 2024-01-16T15:29:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75691.log 2024-01-16T15:29:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81453.log 2024-01-16T15:29:24.220 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50589.log.gz 2024-01-16T15:29:24.221 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73092.log 2024-01-16T15:29:24.221 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75691.log.gz 2024-01-16T15:29:24.222 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81453.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81453.log.gz 2024-01-16T15:29:24.222 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66974.log 2024-01-16T15:29:24.223 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64169.log 2024-01-16T15:29:24.223 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73092.log.gz 2024-01-16T15:29:24.224 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66974.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.66974.log.gz --verbose 2024-01-16T15:29:24.224 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.64910.log 2024-01-16T15:29:24.225 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63419.log 2024-01-16T15:29:24.226 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64169.log.gz 2024-01-16T15:29:24.226 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46810.log 2024-01-16T15:29:24.227 INFO:teuthology.orchestra.run.smithi043.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64910.log.gz 2024-01-16T15:29:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75901.log 2024-01-16T15:29:24.228 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63419.log.gz 2024-01-16T15:29:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46810.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.46810.log.gz -5 2024-01-16T15:29:24.229 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42166.log 2024-01-16T15:29:24.230 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47770.log 2024-01-16T15:29:24.230 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75901.log.gz 2024-01-16T15:29:24.231 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68220.log 2024-01-16T15:29:24.232 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42166.log.gz 2024-01-16T15:29:24.232 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63855.log 88.9% 2024-01-16T15:29:24.232 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47770.log.gz 2024-01-16T15:29:24.233 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-01-16T15:29:24.234 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68220.log.gz 2024-01-16T15:29:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76317.log 2024-01-16T15:29:24.235 INFO:teuthology.orchestra.run.smithi043.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.63855.log.gz 2024-01-16T15:29:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45160.log 2024-01-16T15:29:24.236 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-01-16T15:29:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67830.log 2024-01-16T15:29:24.237 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76317.log.gz 2024-01-16T15:29:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79502.log 2024-01-16T15:29:24.238 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45160.log.gz 2024-01-16T15:29:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53674.log 2024-01-16T15:29:24.239 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67830.log.gz 2024-01-16T15:29:24.240 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55813.log 2024-01-16T15:29:24.240 INFO:teuthology.orchestra.run.smithi043.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.79502.log.gz 2024-01-16T15:29:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69524.log 2024-01-16T15:29:24.241 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53674.log.gz 2024-01-16T15:29:24.242 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55813.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55813.log.gz 2024-01-16T15:29:24.242 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81705.log 2024-01-16T15:29:24.243 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47896.log 2024-01-16T15:29:24.244 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.69524.log.gz 2024-01-16T15:29:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75313.log 2024-01-16T15:29:24.245 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81705.log.gz 2024-01-16T15:29:24.246 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47896.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47896.log.gz 2024-01-16T15:29:24.246 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67556.log 2024-01-16T15:29:24.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70346.log 2024-01-16T15:29:24.247 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75313.log.gz 2024-01-16T15:29:24.248 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79831.log 2024-01-16T15:29:24.248 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67556.log.gz 2024-01-16T15:29:24.249 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46474.log 2024-01-16T15:29:24.249 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70346.log.gz 2024-01-16T15:29:24.250 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59423.log 2024-01-16T15:29:24.250 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79831.log.gz 2024-01-16T15:29:24.251 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65624.log 2024-01-16T15:29:24.251 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46474.log.gz 2024-01-16T15:29:24.252 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59423.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59423.log.gz 2024-01-16T15:29:24.252 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70547.log 2024-01-16T15:29:24.253 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74590.log 2024-01-16T15:29:24.254 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65624.log.gz 2024-01-16T15:29:24.254 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58302.log 2024-01-16T15:29:24.255 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70547.log.gz 2024-01-16T15:29:24.256 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47644.log 2024-01-16T15:29:24.256 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74590.log.gz 2024-01-16T15:29:24.257 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65204.log 2024-01-16T15:29:24.257 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58302.log.gz 2024-01-16T15:29:24.258 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76569.log 2024-01-16T15:29:24.258 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47644.log.gz 2024-01-16T15:29:24.259 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58134.log 2024-01-16T15:29:24.259 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65204.log.gz 2024-01-16T15:29:24.260 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64253.log 2024-01-16T15:29:24.260 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76569.log.gz 2024-01-16T15:29:24.261 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-01-16T15:29:24.261 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58134.log.gz 2024-01-16T15:29:24.262 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73442.log 2024-01-16T15:29:24.262 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64253.log.gz 2024-01-16T15:29:24.264 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49824.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-01-16T15:29:24.264 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71182.log 2024-01-16T15:29:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78013.log 2024-01-16T15:29:24.265 INFO:teuthology.orchestra.run.smithi043.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73442.log.gz 2024-01-16T15:29:24.265 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44083.log 2024-01-16T15:29:24.266 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71182.log.gz 2024-01-16T15:29:24.267 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78013.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78013.log.gz 2024-01-16T15:29:24.267 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66126.log 2024-01-16T15:29:24.268 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81369.log 2024-01-16T15:29:24.268 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44083.log.gz 2024-01-16T15:29:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41713.log 2024-01-16T15:29:24.269 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66126.log.gz 2024-01-16T15:29:24.270 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47394.log 2024-01-16T15:29:24.270 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81369.log.gz 2024-01-16T15:29:24.271 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41713.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41713.log.gz 2024-01-16T15:29:24.271 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47940.log 2024-01-16T15:29:24.272 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45700.log 2024-01-16T15:29:24.273 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47394.log.gz 2024-01-16T15:29:24.273 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65732.log 2024-01-16T15:29:24.274 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47940.log.gz 2024-01-16T15:29:24.275 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45700.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2024-01-16T15:29:24.275 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69228.log 2024-01-16T15:29:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68346.log 2024-01-16T15:29:24.276 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65732.log.gz 2024-01-16T15:29:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60719.log 2024-01-16T15:29:24.277 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69228.log.gz 2024-01-16T15:29:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73777.log 2024-01-16T15:29:24.278 INFO:teuthology.orchestra.run.smithi043.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.68346.log.gz 2024-01-16T15:29:24.279 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-01-16T15:29:24.279 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60719.log.gz 2024-01-16T15:29:24.280 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50078.log 2024-01-16T15:29:24.280 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73777.log.gz 2024-01-16T15:29:24.281 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79238.log 2024-01-16T15:29:24.282 INFO:teuthology.orchestra.run.smithi043.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-01-16T15:29:24.283 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50078.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50078.log.gz 2024-01-16T15:29:24.283 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77129.log 2024-01-16T15:29:24.284 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44503.log 2024-01-16T15:29:24.284 INFO:teuthology.orchestra.run.smithi043.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79238.log.gz 2024-01-16T15:29:24.285 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52801.log 2024-01-16T15:29:24.285 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77129.log.gz 2024-01-16T15:29:24.286 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55090.log 2024-01-16T15:29:24.286 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44503.log.gz 2024-01-16T15:29:24.287 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49139.log 2024-01-16T15:29:24.287 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52801.log.gz 2024-01-16T15:29:24.288 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62698.log 2024-01-16T15:29:24.288 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55090.log.gz 2024-01-16T15:29:24.289 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60779.log 2024-01-16T15:29:24.290 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49139.log.gz 2024-01-16T15:29:24.291 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60821.log 2024-01-16T15:29:24.291 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62698.log.gz 2024-01-16T15:29:24.292 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78605.log 2024-01-16T15:29:24.292 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60779.log.gz 2024-01-16T15:29:24.293 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60821.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60821.log.gz 2024-01-16T15:29:24.293 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-01-16T15:29:24.294 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78851.log 2024-01-16T15:29:24.294 INFO:teuthology.orchestra.run.smithi043.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78605.log.gz 2024-01-16T15:29:24.296 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48330.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-01-16T15:29:24.296 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42082.log 2024-01-16T15:29:24.297 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78851.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.78851.log.gz -5 2024-01-16T15:29:24.297 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60335.log 2024-01-16T15:29:24.298 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55311.log 2024-01-16T15:29:24.298 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42082.log.gz 2024-01-16T15:29:24.299 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42292.log 2024-01-16T15:29:24.299 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60335.log.gz 2024-01-16T15:29:24.300 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55311.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55311.log.gz 2024-01-16T15:29:24.301 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73400.log 2024-01-16T15:29:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76485.log 2024-01-16T15:29:24.302 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42292.log.gz 2024-01-16T15:29:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71438.log 2024-01-16T15:29:24.303 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73400.log.gz 2024-01-16T15:29:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76485.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76485.log.gz 2024-01-16T15:29:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-01-16T15:29:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55939.log 2024-01-16T15:29:24.305 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71438.log.gz 2024-01-16T15:29:24.306 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50427.log 2024-01-16T15:29:24.306 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-01-16T15:29:24.307 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79790.log 2024-01-16T15:29:24.307 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55939.log.gz 2024-01-16T15:29:24.308 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50427.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50427.log.gz 2024-01-16T15:29:24.308 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48474.log 2024-01-16T15:29:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60038.log 2024-01-16T15:29:24.310 INFO:teuthology.orchestra.run.smithi043.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79790.log.gz 2024-01-16T15:29:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79033.log 2024-01-16T15:29:24.311 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60038.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.48474.log.gz 2024-01-16T15:29:24.312 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60038.log.gz 2024-01-16T15:29:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71354.log 2024-01-16T15:29:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78136.log 2024-01-16T15:29:24.313 INFO:teuthology.orchestra.run.smithi043.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.79033.log.gz 2024-01-16T15:29:24.314 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58907.log 2024-01-16T15:29:24.314 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71354.log.gz 2024-01-16T15:29:24.315 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78136.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.78136.log.gz 2024-01-16T15:29:24.315 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45984.log 2024-01-16T15:29:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78095.log 2024-01-16T15:29:24.316 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58907.log.gz 2024-01-16T15:29:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47728.log 2024-01-16T15:29:24.318 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45984.log.gz 2024-01-16T15:29:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-01-16T15:29:24.318 INFO:teuthology.orchestra.run.smithi043.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.78095.log.gz 2024-01-16T15:29:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47728.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47728.log.gz 2024-01-16T15:29:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62745.log 2024-01-16T15:29:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53457.log 2024-01-16T15:29:24.321 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-01-16T15:29:24.322 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54880.log 2024-01-16T15:29:24.322 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62745.log.gz 2024-01-16T15:29:24.323 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53457.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53457.log.gz 2024-01-16T15:29:24.323 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57946.log 2024-01-16T15:29:24.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63713.log 2024-01-16T15:29:24.324 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54880.log.gz 2024-01-16T15:29:24.325 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81747.log 2024-01-16T15:29:24.325 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57946.log.gz 2024-01-16T15:29:24.326 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63713.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.63713.log.gz 2024-01-16T15:29:24.326 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70589.log 2024-01-16T15:29:24.327 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50874.log 2024-01-16T15:29:24.327 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81747.log.gz 2024-01-16T15:29:24.328 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78992.log 2024-01-16T15:29:24.328 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70589.log.gz 2024-01-16T15:29:24.329 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63950.log 2024-01-16T15:29:24.330 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50874.log.gz 2024-01-16T15:29:24.330 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71480.log 2024-01-16T15:29:24.331 INFO:teuthology.orchestra.run.smithi043.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78992.log.gz 2024-01-16T15:29:24.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80013.log 2024-01-16T15:29:24.332 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63950.log.gz 2024-01-16T15:29:24.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71480.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71480.log.gz 2024-01-16T15:29:24.333 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68697.log 2024-01-16T15:29:24.334 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80013.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:29:24.334 INFO:teuthology.orchestra.run.smithi043.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.80013.log.gz 2024-01-16T15:29:24.335 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.68697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2024-01-16T15:29:24.336 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49593.log 2024-01-16T15:29:24.338 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64313.log 2024-01-16T15:29:24.338 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2024-01-16T15:29:24.339 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49593.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49593.log.gz 2024-01-16T15:29:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66728.log 2024-01-16T15:29:24.340 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:29:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64688.log 2024-01-16T15:29:24.342 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64313.log: /var/log/ceph/ceph-client.admin.66728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48288.log 2024-01-16T15:29:24.342 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64313.log.gz 2024-01-16T15:29:24.342 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66728.log.gz 2024-01-16T15:29:24.343 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78177.log 2024-01-16T15:29:24.343 INFO:teuthology.orchestra.run.smithi043.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64688.log.gz 2024-01-16T15:29:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46228.log 2024-01-16T15:29:24.344 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48288.log.gz 2024-01-16T15:29:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60133.log 2024-01-16T15:29:24.345 INFO:teuthology.orchestra.run.smithi043.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.78177.log.gz 2024-01-16T15:29:24.346 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64211.log 2024-01-16T15:29:24.346 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46228.log.gz 2024-01-16T15:29:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42544.log 2024-01-16T15:29:24.347 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60133.log.gz 2024-01-16T15:29:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44335.log 2024-01-16T15:29:24.348 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64211.log.gz 2024-01-16T15:29:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46026.log 2024-01-16T15:29:24.350 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42544.log.gz 2024-01-16T15:29:24.350 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-01-16T15:29:24.351 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44335.log.gz 2024-01-16T15:29:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46026.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.46026.log.gz --verbose 2024-01-16T15:29:24.352 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.60593.log 2024-01-16T15:29:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44587.log 2024-01-16T15:29:24.353 INFO:teuthology.orchestra.run.smithi043.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-01-16T15:29:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48034.log 2024-01-16T15:29:24.354 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60593.log.gz 2024-01-16T15:29:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44587.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44587.log.gz 2024-01-16T15:29:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49653.log 2024-01-16T15:29:24.356 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51529.log 2024-01-16T15:29:24.356 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48034.log.gz 2024-01-16T15:29:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71097.log 2024-01-16T15:29:24.358 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49653.log.gz 2024-01-16T15:29:24.359 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65498.log 2024-01-16T15:29:24.359 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51529.log.gz 2024-01-16T15:29:24.360 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2024-01-16T15:29:24.360 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71097.log.gz 2024-01-16T15:29:24.361 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65498.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65498.log.gz 2024-01-16T15:29:24.361 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66338.log 2024-01-16T15:29:24.362 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62831.log 2024-01-16T15:29:24.362 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2024-01-16T15:29:24.363 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66338.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66338.log.gz 2024-01-16T15:29:24.363 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79115.log 2024-01-16T15:29:24.364 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62831.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.62831.log.gz 2024-01-16T15:29:24.365 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77171.log 2024-01-16T15:29:24.366 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79115.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79115.log.gz 2024-01-16T15:29:24.366 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46102.log 2024-01-16T15:29:24.367 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57199.log 2024-01-16T15:29:24.367 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77171.log.gz 2024-01-16T15:29:24.368 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46102.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46102.log.gz 2024-01-16T15:29:24.368 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74849.log 2024-01-16T15:29:24.369 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52663.log 2024-01-16T15:29:24.369 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57199.log.gz 2024-01-16T15:29:24.370 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74849.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74849.log.gz 2024-01-16T15:29:24.370 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-01-16T15:29:24.371 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49367.log 2024-01-16T15:29:24.371 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52663.log.gz 2024-01-16T15:29:24.372 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40261.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-01-16T15:29:24.373 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66590.log 2024-01-16T15:29:24.374 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49367.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49367.log.gz 2024-01-16T15:29:24.374 INFO:teuthology.orchestra.run.smithi043.stderr: /var/log/ceph/ceph-client.admin.55897.log 2024-01-16T15:29:24.375 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66590.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66590.log.gz 2024-01-16T15:29:24.375 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63041.log 2024-01-16T15:29:24.376 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53289.log 2024-01-16T15:29:24.376 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55897.log.gz 2024-01-16T15:29:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63041.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63041.log.gz 2024-01-16T15:29:24.377 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80867.log 2024-01-16T15:29:24.378 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53812.log 2024-01-16T15:29:24.378 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53289.log.gz 2024-01-16T15:29:24.379 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80867.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80867.log.gz 2024-01-16T15:29:24.379 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74973.log 2024-01-16T15:29:24.381 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82192.log 2024-01-16T15:29:24.381 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53812.log.gz 2024-01-16T15:29:24.382 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74973.log.gz 2024-01-16T15:29:24.382 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79361.log 2024-01-16T15:29:24.383 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.82192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67091.log 2024-01-16T15:29:24.383 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82192.log.gz 2024-01-16T15:29:24.384 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79361.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79361.log.gz 2024-01-16T15:29:24.384 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47984.log 2024-01-16T15:29:24.385 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77255.log 2024-01-16T15:29:24.385 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67091.log.gz 2024-01-16T15:29:24.386 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47984.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47984.log.gz 2024-01-16T15:29:24.386 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2024-01-16T15:29:24.387 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65774.log 2024-01-16T15:29:24.387 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77255.log.gz 2024-01-16T15:29:24.388 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44839.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2024-01-16T15:29:24.388 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61938.log 2024-01-16T15:29:24.390 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67618.log 2024-01-16T15:29:24.390 INFO:teuthology.orchestra.run.smithi043.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65774.log.gz 2024-01-16T15:29:24.391 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61938.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61938.log.gz 2024-01-16T15:29:24.391 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78646.log 2024-01-16T15:29:24.392 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48076.log 2024-01-16T15:29:24.392 INFO:teuthology.orchestra.run.smithi043.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67618.log.gz 2024-01-16T15:29:24.393 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78646.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78646.log.gz 2024-01-16T15:29:24.393 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41743.log 2024-01-16T15:29:24.394 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57115.log 2024-01-16T15:29:24.394 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48076.log.gz 2024-01-16T15:29:24.395 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41743.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41743.log.gz 2024-01-16T15:29:24.395 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59954.log 2024-01-16T15:29:24.396 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58050.log 2024-01-16T15:29:24.396 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57115.log.gz 2024-01-16T15:29:24.397 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59954.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59954.log.gz 2024-01-16T15:29:24.398 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66890.log 2024-01-16T15:29:24.399 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58050.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.58050.log.gz 2024-01-16T15:29:24.399 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70226.log 2024-01-16T15:29:24.400 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66890.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66890.log.gz 2024-01-16T15:29:24.400 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59465.log 2024-01-16T15:29:24.401 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61032.log 2024-01-16T15:29:24.401 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70226.log.gz 2024-01-16T15:29:24.402 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59465.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59465.log.gz 2024-01-16T15:29:24.402 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74338.log 2024-01-16T15:29:24.403 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49082.log 2024-01-16T15:29:24.404 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74338.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61032.log.gz 2024-01-16T15:29:24.405 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74338.log.gz 2024-01-16T15:29:24.405 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52388.log 2024-01-16T15:29:24.406 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49082.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.49082.log.gz 2024-01-16T15:29:24.406 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57358.log 2024-01-16T15:29:24.407 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.52388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58387.log 2024-01-16T15:29:24.408 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57358.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57358.log.gz 2024-01-16T15:29:24.408 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73585.log 2024-01-16T15:29:24.410 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2024-01-16T15:29:24.410 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58387.log.gz 2024-01-16T15:29:24.411 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-01-16T15:29:24.411 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73585.log.gz 2024-01-16T15:29:24.412 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51709.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2024-01-16T15:29:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63335.log 2024-01-16T15:29:24.414 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2024-01-16T15:29:24.414 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-01-16T15:29:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63335.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63335.log.gz 2024-01-16T15:29:24.415 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68052.log 2024-01-16T15:29:24.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-01-16T15:29:24.417 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2024-01-16T15:29:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68052.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68052.log.gz 2024-01-16T15:29:24.418 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80169.log 2024-01-16T15:29:24.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2024-01-16T15:29:24.419 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2024-01-16T15:29:24.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80169.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.80169.log.gz -5 2024-01-16T15:29:24.421 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79872.log 2024-01-16T15:29:24.422 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55216.log 2024-01-16T15:29:24.422 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2024-01-16T15:29:24.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79872.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79872.log.gz 2024-01-16T15:29:24.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49195.log 2024-01-16T15:29:24.424 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63209.log 2024-01-16T15:29:24.425 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55216.log.gz 2024-01-16T15:29:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46978.log 2024-01-16T15:29:24.426 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49195.log.gz 2024-01-16T15:29:24.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63209.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63209.log.gz 2024-01-16T15:29:24.427 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44251.log 2024-01-16T15:29:24.428 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62091.log 2024-01-16T15:29:24.429 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46978.log.gz 2024-01-16T15:29:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44251.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44251.log.gz 2024-01-16T15:29:24.430 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44125.log 2024-01-16T15:29:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62091.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62091.log.gz 2024-01-16T15:29:24.431 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65120.log 2024-01-16T15:29:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54108.log 2024-01-16T15:29:24.433 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44125.log.gz 2024-01-16T15:29:24.434 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65120.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65120.log.gz 2024-01-16T15:29:24.434 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52045.log 2024-01-16T15:29:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54108.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54108.log.gz 2024-01-16T15:29:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59381.log 2024-01-16T15:29:24.436 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63545.log 2024-01-16T15:29:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59381.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52045.log.gz 2024-01-16T15:29:24.438 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59381.log.gz 2024-01-16T15:29:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69835.log 2024-01-16T15:29:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63545.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63545.log.gz 2024-01-16T15:29:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54805.log 2024-01-16T15:29:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69835.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69835.log.gz 2024-01-16T15:29:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48390.log 2024-01-16T15:29:24.441 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-01-16T15:29:24.442 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48390.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54805.log.gz 2024-01-16T15:29:24.443 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48390.log.gz 2024-01-16T15:29:24.443 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71310.log 2024-01-16T15:29:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53575.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-01-16T15:29:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64784.log 2024-01-16T15:29:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35991.log 2024-01-16T15:29:24.446 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71310.log.gz 2024-01-16T15:29:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64784.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64784.log.gz 2024-01-16T15:29:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45202.log 2024-01-16T15:29:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35991.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35991.log.gz 2024-01-16T15:29:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49952.log 2024-01-16T15:29:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82032.log 2024-01-16T15:29:24.450 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45202.log.gz 2024-01-16T15:29:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49952.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49952.log.gz 2024-01-16T15:29:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72622.log 2024-01-16T15:29:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.82032.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82032.log.gz 2024-01-16T15:29:24.452 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60990.log 2024-01-16T15:29:24.453 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53725.log 2024-01-16T15:29:24.454 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72622.log.gz 2024-01-16T15:29:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60990.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60990.log.gz 2024-01-16T15:29:24.455 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2024-01-16T15:29:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53725.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53725.log.gz 2024-01-16T15:29:24.456 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72664.log 2024-01-16T15:29:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65582.log 2024-01-16T15:29:24.457 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2024-01-16T15:29:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72664.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72664.log.gz 2024-01-16T15:29:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44419.log 2024-01-16T15:29:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71900.log 2024-01-16T15:29:24.460 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.65582.log.gz 2024-01-16T15:29:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44419.log.gz 2024-01-16T15:29:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-01-16T15:29:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55132.log 2024-01-16T15:29:24.463 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71900.log.gz 2024-01-16T15:29:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45784.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-01-16T15:29:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62547.log 2024-01-16T15:29:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55132.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55132.log.gz 2024-01-16T15:29:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81159.log 2024-01-16T15:29:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63503.log 2024-01-16T15:29:24.467 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62547.log.gz 2024-01-16T15:29:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71016.log 2024-01-16T15:29:24.468 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81159.log.gz 2024-01-16T15:29:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74931.log 2024-01-16T15:29:24.469 INFO:teuthology.orchestra.run.smithi043.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.63503.log.gz 2024-01-16T15:29:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71016.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71016.log.gz 2024-01-16T15:29:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48817.log 2024-01-16T15:29:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74931.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T15:29:24.472 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74931.log.gz/var/log/ceph/ceph-client.0.48817.log: 2024-01-16T15:29:24.473 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52579.log 2024-01-16T15:29:24.474 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81243.log 2024-01-16T15:29:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55006.log 2024-01-16T15:29:24.476 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52579.log.gz 2024-01-16T15:29:24.477 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81579.log 2024-01-16T15:29:24.478 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81243.log.gz 2024-01-16T15:29:24.479 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75481.log 2024-01-16T15:29:24.479 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55006.log.gz 2024-01-16T15:29:24.480 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78523.log 2024-01-16T15:29:24.480 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81579.log.gz 2024-01-16T15:29:24.482 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57862.log 2024-01-16T15:29:24.482 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75481.log.gz 2024-01-16T15:29:24.483 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2024-01-16T15:29:24.483 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78523.log.gz 2024-01-16T15:29:24.484 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57862.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57862.log.gz 2024-01-16T15:29:24.485 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75835.log 2024-01-16T15:29:24.486 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54258.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2024-01-16T15:29:24.486 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71139.log 2024-01-16T15:29:24.488 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59033.log 2024-01-16T15:29:24.488 INFO:teuthology.orchestra.run.smithi043.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75835.log.gz 2024-01-16T15:29:24.489 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71139.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71139.log.gz 2024-01-16T15:29:24.489 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75565.log 2024-01-16T15:29:24.491 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2024-01-16T15:29:24.491 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59033.log.gz 2024-01-16T15:29:24.492 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58697.log 2024-01-16T15:29:24.492 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75565.log.gz 2024-01-16T15:29:24.494 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51961.log 2024-01-16T15:29:24.494 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-01-16T15:29:24.495 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58697.log: gzip 88.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.71666.log 2024-01-16T15:29:24.495 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58697.log.gz 2024-01-16T15:29:24.497 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-01-16T15:29:24.497 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51961.log.gz 2024-01-16T15:29:24.498 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51623.log 2024-01-16T15:29:24.498 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71666.log.gz 2024-01-16T15:29:24.499 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59117.log 2024-01-16T15:29:24.499 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-01-16T15:29:24.501 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69186.log 2024-01-16T15:29:24.501 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51623.log.gz 2024-01-16T15:29:24.502 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58841.log 2024-01-16T15:29:24.502 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59117.log.gz 2024-01-16T15:29:24.503 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60906.log 2024-01-16T15:29:24.504 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69186.log.gz 2024-01-16T15:29:24.505 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44041.log 2024-01-16T15:29:24.505 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58841.log.gz 2024-01-16T15:29:24.506 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53331.log 2024-01-16T15:29:24.507 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60906.log.gz 2024-01-16T15:29:24.508 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44041.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44041.log.gz 2024-01-16T15:29:24.508 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70268.log 2024-01-16T15:29:24.509 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53331.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53331.log.gz 2024-01-16T15:29:24.510 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2024-01-16T15:29:24.511 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58092.log 2024-01-16T15:29:24.511 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70268.log.gz 2024-01-16T15:29:24.513 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55048.log 2024-01-16T15:29:24.513 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2024-01-16T15:29:24.514 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2024-01-16T15:29:24.514 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58092.log.gz 2024-01-16T15:29:24.515 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2024-01-16T15:29:24.515 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55048.log.gz 2024-01-16T15:29:24.517 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81537.log 2024-01-16T15:29:24.517 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2024-01-16T15:29:24.518 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40444.log 2024-01-16T15:29:24.518 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2024-01-16T15:29:24.520 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57628.log 2024-01-16T15:29:24.520 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81537.log.gz 2024-01-16T15:29:24.521 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69090.log 2024-01-16T15:29:24.521 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40444.log.gz 2024-01-16T15:29:24.522 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75355.log 2024-01-16T15:29:24.522 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57628.log.gz 2024-01-16T15:29:24.524 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77045.log 2024-01-16T15:29:24.524 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69090.log.gz 2024-01-16T15:29:24.525 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61640.log 2024-01-16T15:29:24.525 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75355.log.gz 2024-01-16T15:29:24.527 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78810.log 2024-01-16T15:29:24.527 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77045.log.gz 2024-01-16T15:29:24.528 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62367.log 2024-01-16T15:29:24.528 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61640.log.gz 2024-01-16T15:29:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69406.log 2024-01-16T15:29:24.530 INFO:teuthology.orchestra.run.smithi043.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.78810.log.gz 2024-01-16T15:29:24.531 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57904.log 2024-01-16T15:29:24.531 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62367.log.gz 2024-01-16T15:29:24.532 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42040.log 2024-01-16T15:29:24.533 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69406.log.gz 2024-01-16T15:29:24.534 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53896.log 2024-01-16T15:29:24.534 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57904.log.gz 2024-01-16T15:29:24.535 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42040.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42040.log.gz 2024-01-16T15:29:24.536 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53896.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53896.log.gz 2024-01-16T15:29:24.537 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42418.log 2024-01-16T15:29:24.538 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68304.log 2024-01-16T15:29:24.540 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-01-16T15:29:24.540 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42418.log.gz 2024-01-16T15:29:24.541 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66932.log 2024-01-16T15:29:24.541 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68304.log.gz 2024-01-16T15:29:24.542 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40348.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-01-16T15:29:24.543 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78218.log 2024-01-16T15:29:24.544 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66932.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66932.log.gz 2024-01-16T15:29:24.544 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64826.log 2024-01-16T15:29:24.546 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81201.log 2024-01-16T15:29:24.546 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78218.log.gz 2024-01-16T15:29:24.547 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44671.log 2024-01-16T15:29:24.547 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64826.log.gz 2024-01-16T15:29:24.549 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46432.log 2024-01-16T15:29:24.549 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81201.log.gz 2024-01-16T15:29:24.550 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44671.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44671.log.gz 2024-01-16T15:29:24.551 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61375.log 2024-01-16T15:29:24.551 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46432.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46432.log.gz 2024-01-16T15:29:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2024-01-16T15:29:24.554 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.61375.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29920.log 2024-01-16T15:29:24.555 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45900.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2024-01-16T15:29:24.556 INFO:teuthology.orchestra.run.smithi043.stderr:gzip/var/log/ceph/ceph.tmp-client.admin.29920.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.76824.log 2024-01-16T15:29:24.556 INFO:teuthology.orchestra.run.smithi043.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29920.log.gz 2024-01-16T15:29:24.558 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63461.log 2024-01-16T15:29:24.559 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76824.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76824.log.gz 2024-01-16T15:29:24.560 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62601.log 2024-01-16T15:29:24.561 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63461.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63461.log.gz 2024-01-16T15:29:24.562 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73920.log 2024-01-16T15:29:24.563 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62601.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62601.log.gz 2024-01-16T15:29:24.564 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67259.log 2024-01-16T15:29:24.565 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73920.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73920.log.gz 2024-01-16T15:29:24.566 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68880.log 2024-01-16T15:29:24.567 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67259.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67259.log.gz 2024-01-16T15:29:24.568 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59744.log 2024-01-16T15:29:24.569 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68880.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68880.log.gz 2024-01-16T15:29:24.570 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59543.log 2024-01-16T15:29:24.571 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59744.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59744.log.gz 2024-01-16T15:29:24.572 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64994.log 2024-01-16T15:29:24.573 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59543.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59543.log.gz 2024-01-16T15:29:24.574 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80783.log 2024-01-16T15:29:24.575 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64994.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64994.log.gz 2024-01-16T15:29:24.576 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56775.log/var/log/ceph/ceph-client.admin.80783.log: 2024-01-16T15:29:24.577 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80783.log.gz 2024-01-16T15:29:24.578 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56257.log 2024-01-16T15:29:24.580 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56775.log.gz 2024-01-16T15:29:24.580 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64868.log 2024-01-16T15:29:24.581 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56257.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56257.log.gz 2024-01-16T15:29:24.582 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78687.log 2024-01-16T15:29:24.583 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64868.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64868.log.gz 2024-01-16T15:29:24.583 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71750.log 2024-01-16T15:29:24.586 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78687.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.78687.log.gz 2024-01-16T15:29:24.586 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.71750.log: /var/log/ceph/ceph-client.admin.79320.log 2024-01-16T15:29:24.586 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71750.log.gz 2024-01-16T15:29:24.587 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66254.log 2024-01-16T15:29:24.590 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79320.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79320.log.gz 2024-01-16T15:29:24.590 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.66254.log: /var/log/ceph/ceph-client.admin.74769.log 2024-01-16T15:29:24.590 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66254.log.gz 2024-01-16T15:29:24.591 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63896.log 2024-01-16T15:29:24.593 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74769.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74769.log.gz 2024-01-16T15:29:24.594 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.63896.log: /var/log/ceph/ceph-client.admin.62787.log 2024-01-16T15:29:24.595 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63896.log.gz 2024-01-16T15:29:24.595 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61514.log 2024-01-16T15:29:24.597 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62787.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62787.log.gz 2024-01-16T15:29:24.598 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.61514.log: /var/log/ceph/ceph-client.admin.74017.log 2024-01-16T15:29:24.599 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61514.log.gz 2024-01-16T15:29:24.600 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69048.log 2024-01-16T15:29:24.602 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74017.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74017.log.gz 2024-01-16T15:29:24.602 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50748.log 2024-01-16T15:29:24.603 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69048.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69048.log.gz 2024-01-16T15:29:24.605 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50748.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50748.log.gz 2024-01-16T15:29:24.605 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79156.log 2024-01-16T15:29:24.607 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72316.log 2024-01-16T15:29:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79156.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79156.log.gz 2024-01-16T15:29:24.609 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41755.log 2024-01-16T15:29:24.610 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72316.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72316.log.gz 2024-01-16T15:29:24.611 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-01-16T15:29:24.612 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41755.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41755.log.gz 2024-01-16T15:29:24.614 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40654.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-01-16T15:29:24.614 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73134.log 2024-01-16T15:29:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62283.log 2024-01-16T15:29:24.618 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73134.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73134.log.gz 2024-01-16T15:29:24.618 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2024-01-16T15:29:24.619 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62283.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62283.log.gz 2024-01-16T15:29:24.620 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54192.log 2024-01-16T15:29:24.621 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44797.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2024-01-16T15:29:24.623 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54192.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54192.log.gz 2024-01-16T15:29:24.623 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42688.log 2024-01-16T15:29:24.624 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-16T15:29:24.625 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58799.log 2024-01-16T15:29:24.626 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42688.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42688.log.gz 2024-01-16T15:29:24.627 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71582.log 2024-01-16T15:29:24.628 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58799.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58799.log.gz 2024-01-16T15:29:24.629 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2024-01-16T15:29:24.630 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71582.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71582.log.gz 2024-01-16T15:29:24.630 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67049.log 2024-01-16T15:29:24.632 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54066.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2024-01-16T15:29:24.633 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54964.log 2024-01-16T15:29:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67049.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67049.log.gz 2024-01-16T15:29:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75057.log 2024-01-16T15:29:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54964.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54964.log.gz 2024-01-16T15:29:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66212.log 2024-01-16T15:29:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75057.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75057.log.gz 2024-01-16T15:29:24.638 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69590.log 2024-01-16T15:29:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66212.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.66212.log.gz 2024-01-16T15:29:24.641 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71540.log 2024-01-16T15:29:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69590.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69590.log.gz 2024-01-16T15:29:24.642 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42502.log 2024-01-16T15:29:24.644 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71540.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71540.log.gz 2024-01-16T15:29:24.644 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-01-16T15:29:24.645 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42502.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42502.log.gz 2024-01-16T15:29:24.646 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80395.log 2024-01-16T15:29:24.648 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49868.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-01-16T15:29:24.648 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2024-01-16T15:29:24.649 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80395.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80395.log.gz 2024-01-16T15:29:24.650 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80521.log 2024-01-16T15:29:24.651 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50036.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2024-01-16T15:29:24.652 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80909.log 2024-01-16T15:29:24.653 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80521.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80521.log.gz 2024-01-16T15:29:24.655 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80909.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80909.log.gz 2024-01-16T15:29:24.656 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78054.log 2024-01-16T15:29:24.657 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54342.log 2024-01-16T15:29:24.659 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78054.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.78054.log.gz 2024-01-16T15:29:24.659 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-16T15:29:24.660 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54342.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54342.log.gz 2024-01-16T15:29:24.661 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56023.log 2024-01-16T15:29:24.664 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.56023.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56023.log.gz 2024-01-16T15:29:24.665 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61682.log 2024-01-16T15:29:24.667 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64010.log 2024-01-16T15:29:24.668 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61682.log.gz 2024-01-16T15:29:24.670 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60213.log 2024-01-16T15:29:24.673 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64010.log: 88.4%/var/log/ceph/ceph-client.admin.60213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64096.log 2024-01-16T15:29:24.673 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60213.log.gz 2024-01-16T15:29:24.673 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64010.log.gz 2024-01-16T15:29:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76178.log 2024-01-16T15:29:24.679 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64096.log: /var/log/ceph/ceph-client.0.76178.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45858.log 2024-01-16T15:29:24.680 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64096.log.gz 2024-01-16T15:29:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:gzip/var/log/ceph/ceph-client.admin.45858.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-01-16T15:29:24.686 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45858.log.gz 2024-01-16T15:29:24.690 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2024-01-16T15:29:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50631.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-01-16T15:29:24.694 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56534.log 2024-01-16T15:29:24.694 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2024-01-16T15:29:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.56534.log: -- /var/log/ceph/ceph-client.admin.82258.log 2024-01-16T15:29:24.703 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41084.log 2024-01-16T15:29:24.704 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.82258.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82258.log.gz 2024-01-16T15:29:24.706 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41084.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41084.log.gz 2024-01-16T15:29:24.706 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58007.log 2024-01-16T15:29:24.709 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53938.log 2024-01-16T15:29:24.711 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58007.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58007.log.gz 2024-01-16T15:29:24.712 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42250.log 2024-01-16T15:29:24.715 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53938.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53938.log.gz 2024-01-16T15:29:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52111.log 2024-01-16T15:29:24.718 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42250.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42250.log.gz 2024-01-16T15:29:24.722 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52111.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52111.log.gz 2024-01-16T15:29:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57073.log 2024-01-16T15:29:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48998.log 2024-01-16T15:29:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57073.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57073.log.gz 2024-01-16T15:29:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43670.log 2024-01-16T15:29:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48998.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48998.log.gz 2024-01-16T15:29:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80993.log 2024-01-16T15:29:24.735 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.2.43670.log: 91.8% -- replaced with /var/log/ceph/ceph-client.2.43670.log.gz 2024-01-16T15:29:24.736 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79197.log 2024-01-16T15:29:24.738 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80993.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80993.log.gz 2024-01-16T15:29:24.740 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57670.log 2024-01-16T15:29:24.742 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79197.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.79197.log.gz 2024-01-16T15:29:24.746 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57670.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57670.log.gz 2024-01-16T15:29:24.747 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72823.log 2024-01-16T15:29:24.748 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56733.log 2024-01-16T15:29:24.753 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57808.log 2024-01-16T15:29:24.754 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72823.log: /var/log/ceph/ceph-client.admin.56733.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72823.log.gz 2024-01-16T15:29:24.754 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56733.log.gz 2024-01-16T15:29:24.756 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41998.log 2024-01-16T15:29:24.758 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57808.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57808.log.gz 2024-01-16T15:29:24.760 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47062.log 2024-01-16T15:29:24.762 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41998.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41998.log.gz 2024-01-16T15:29:24.766 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47062.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47062.log.gz 2024-01-16T15:29:24.766 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56913.log 2024-01-16T15:29:24.768 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75793.log 2024-01-16T15:29:24.770 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56913.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56913.log.gz 2024-01-16T15:29:24.771 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2024-01-16T15:29:24.773 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75793.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75793.log.gz 2024-01-16T15:29:24.774 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75187.log 2024-01-16T15:29:24.775 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46270.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2024-01-16T15:29:24.780 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75187.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75187.log.gz 2024-01-16T15:29:24.781 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62493.log 2024-01-16T15:29:24.785 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50469.log 2024-01-16T15:29:24.787 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62493.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62493.log.gz 2024-01-16T15:29:24.790 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77297.log 2024-01-16T15:29:24.792 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50469.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50469.log.gz 2024-01-16T15:29:24.793 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45742.log 2024-01-16T15:29:24.794 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77297.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77297.log.gz 2024-01-16T15:29:24.795 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54024.log 2024-01-16T15:29:24.797 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45742.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45742.log.gz 2024-01-16T15:29:24.800 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55981.log 2024-01-16T15:29:24.804 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54024.log: /var/log/ceph/ceph-client.admin.55981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55981.log.gz 2024-01-16T15:29:24.805 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54024.log.gz 2024-01-16T15:29:24.805 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58570.log 2024-01-16T15:29:24.810 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2024-01-16T15:29:24.811 INFO:teuthology.orchestra.run.smithi069.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-16T15:29:24.812 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58570.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58570.log.gz 2024-01-16T15:29:24.815 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69448.log 2024-01-16T15:29:24.817 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54150.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gz 2024-01-16T15:29:24.818 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56191.log 2024-01-16T15:29:24.819 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69448.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69448.log.gz 2024-01-16T15:29:24.823 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44629.log 2024-01-16T15:29:24.825 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56191.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56191.log.gz 2024-01-16T15:29:24.827 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75229.log 2024-01-16T15:29:24.829 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44629.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44629.log.gz 2024-01-16T15:29:24.833 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75229.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75229.log.gz 2024-01-16T15:29:24.833 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74059.log 2024-01-16T15:29:24.836 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60507.log 2024-01-16T15:29:24.837 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74059.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74059.log.gz 2024-01-16T15:29:24.838 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55771.log 2024-01-16T15:29:24.840 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60507.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60507.log.gz 2024-01-16T15:29:24.841 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72526.log 2024-01-16T15:29:24.845 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55771.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55771.log.gz 2024-01-16T15:29:24.846 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72526.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72526.log.gz 2024-01-16T15:29:24.847 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66296.log 2024-01-16T15:29:24.849 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45616.log 2024-01-16T15:29:24.850 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66296.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66296.log.gz 2024-01-16T15:29:24.851 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77411.log 2024-01-16T15:29:24.853 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45616.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45616.log.gz 2024-01-16T15:29:24.853 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78259.log 2024-01-16T15:29:24.856 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47474.log 2024-01-16T15:29:24.857 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78259.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77411.log.gz 2024-01-16T15:29:24.860 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46516.log 2024-01-16T15:29:24.860 INFO:teuthology.orchestra.run.smithi043.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.78259.log.gz 2024-01-16T15:29:24.861 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2024-01-16T15:29:24.863 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46516.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47474.log.gz 2024-01-16T15:29:24.865 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60293.log 2024-01-16T15:29:24.866 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42208.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2024-01-16T15:29:24.866 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46516.log.gz 2024-01-16T15:29:24.868 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70086.log 2024-01-16T15:29:24.869 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60293.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60293.log.gz 2024-01-16T15:29:24.870 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80741.log 2024-01-16T15:29:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80741.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80741.log.gz 2024-01-16T15:29:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70086.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66548.log 2024-01-16T15:29:24.876 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70086.log.gz 2024-01-16T15:29:24.878 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64952.log 2024-01-16T15:29:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64952.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66548.log.gz 2024-01-16T15:29:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47854.log 2024-01-16T15:29:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47104.log 2024-01-16T15:29:24.886 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64952.log.gz 2024-01-16T15:29:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47854.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47854.log.gz 2024-01-16T15:29:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79074.log 2024-01-16T15:29:24.891 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47104.log: 88.7%/var/log/ceph/ceph-client.admin.79074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65288.log 2024-01-16T15:29:24.892 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47104.log.gz 2024-01-16T15:29:24.894 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73669.log 2024-01-16T15:29:24.894 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79074.log.gz 2024-01-16T15:29:24.896 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65288.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65288.log.gz 2024-01-16T15:29:24.899 INFO:teuthology.orchestra.run.smithi043.stderr:gzip/var/log/ceph/ceph-client.admin.73669.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73669.log.gz 2024-01-16T15:29:24.899 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54601.log 2024-01-16T15:29:24.902 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47560.log 2024-01-16T15:29:24.905 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47560.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54601.log.gz 2024-01-16T15:29:24.906 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80699.log 2024-01-16T15:29:24.907 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69793.log 2024-01-16T15:29:24.909 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8%/var/log/ceph/ceph-client.admin.80699.log: -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz 2024-01-16T15:29:24.910 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80699.log.gz 2024-01-16T15:29:24.912 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55855.log 2024-01-16T15:29:24.916 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69793.log: /var/log/ceph/ceph-client.admin.55855.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54763.log 2024-01-16T15:29:24.917 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69793.log.gz 2024-01-16T15:29:24.918 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55855.log.gz 2024-01-16T15:29:24.919 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52621.log 2024-01-16T15:29:24.923 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54763.log: /var/log/ceph/ceph-client.admin.52621.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41088.log 2024-01-16T15:29:24.923 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54763.log.gz 2024-01-16T15:29:24.924 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52621.log.gz 2024-01-16T15:29:24.926 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44976.log 2024-01-16T15:29:24.931 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41088.log: /var/log/ceph/ceph-client.admin.44976.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71624.log 2024-01-16T15:29:24.931 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41088.log.gz 2024-01-16T15:29:24.932 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44976.log.gz 2024-01-16T15:29:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.71624.log: /var/log/ceph/ceph-client.admin.53982.log 2024-01-16T15:29:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53982.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.71624.log.gz -5 2024-01-16T15:29:24.938 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62873.log 2024-01-16T15:29:24.940 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78341.log 2024-01-16T15:29:24.942 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0%/var/log/ceph/ceph-client.admin.62873.log: -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-01-16T15:29:24.943 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62873.log.gz 2024-01-16T15:29:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81789.log 2024-01-16T15:29:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81789.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78341.log.gz 2024-01-16T15:29:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70936.log 2024-01-16T15:29:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73627.log 2024-01-16T15:29:24.952 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4%/var/log/ceph/ceph-client.admin.70936.log: -- replaced with /var/log/ceph/ceph-client.admin.81789.log.gz 2024-01-16T15:29:24.953 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70936.log.gz 2024-01-16T15:29:24.954 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73627.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74506.log 2024-01-16T15:29:24.956 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72865.log 2024-01-16T15:29:24.956 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73627.log.gz 2024-01-16T15:29:24.959 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74506.log: /var/log/ceph/ceph-client.admin.72865.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74506.log.gz 2024-01-16T15:29:24.959 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72865.log.gz 2024-01-16T15:29:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71396.log 2024-01-16T15:29:24.962 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59912.log 2024-01-16T15:29:24.963 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71396.log.gz 2024-01-16T15:29:24.967 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72442.log 2024-01-16T15:29:24.971 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59912.log: /var/log/ceph/ceph-client.admin.72442.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59702.log 2024-01-16T15:29:24.971 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59912.log.gz 2024-01-16T15:29:24.972 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72442.log.gz 2024-01-16T15:29:24.973 INFO:teuthology.orchestra.run.smithi069.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-16T15:29:24.976 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59702.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59702.log.gz 2024-01-16T15:29:24.976 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67874.log 2024-01-16T15:29:24.980 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53063.log 2024-01-16T15:29:24.982 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67874.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67874.log.gz 2024-01-16T15:29:24.987 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59585.log 2024-01-16T15:29:24.987 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53063.log.gz 2024-01-16T15:29:24.991 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76665.log 2024-01-16T15:29:24.992 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59585.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59585.log.gz 2024-01-16T15:29:24.994 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5/var/log/ceph/ceph-client.admin.76665.log: --verbose -- /var/log/ceph/ceph-client.admin.80825.log 2024-01-16T15:29:24.994 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76665.log.gz 2024-01-16T15:29:24.997 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65816.log 2024-01-16T15:29:25.000 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80825.log.gz 2024-01-16T15:29:25.001 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77213.log/var/log/ceph/ceph-client.admin.65816.log: 2024-01-16T15:29:25.001 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65816.log.gz 2024-01-16T15:29:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48956.log 2024-01-16T15:29:25.006 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77213.log.gz 2024-01-16T15:29:25.011 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-16T15:29:25.012 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48956.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48956.log.gz 2024-01-16T15:29:25.015 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80479.log 2024-01-16T15:29:25.020 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60379.log 2024-01-16T15:29:25.021 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80479.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80479.log.gz 2024-01-16T15:29:25.025 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60421.log 2024-01-16T15:29:25.026 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60379.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60379.log.gz 2024-01-16T15:29:25.029 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67438.log 2024-01-16T15:29:25.030 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60421.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60421.log.gz 2024-01-16T15:29:25.034 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75439.log 2024-01-16T15:29:25.034 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67438.log.gz 2024-01-16T15:29:25.038 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67660.log 2024-01-16T15:29:25.041 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75439.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75439.log.gz 2024-01-16T15:29:25.041 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67660.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67660.log.gz 2024-01-16T15:29:25.042 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60948.log 2024-01-16T15:29:25.047 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71792.log 2024-01-16T15:29:25.047 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60948.log.gz 2024-01-16T15:29:25.050 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73711.log 2024-01-16T15:29:25.052 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71792.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71792.log.gz 2024-01-16T15:29:25.054 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73711.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73711.log.gz 2024-01-16T15:29:25.054 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46390.log 2024-01-16T15:29:25.058 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62157.log 2024-01-16T15:29:25.060 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46390.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46390.log.gz 2024-01-16T15:29:25.064 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62157.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62157.log.gz 2024-01-16T15:29:25.064 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40822.log 2024-01-16T15:29:25.068 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55514.log 2024-01-16T15:29:25.070 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40822.log.gz 2024-01-16T15:29:25.070 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-01-16T15:29:25.075 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55514.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55514.log.gz 2024-01-16T15:29:25.076 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78728.log 2024-01-16T15:29:25.076 INFO:teuthology.orchestra.run.smithi043.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-01-16T15:29:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2024-01-16T15:29:25.082 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78728.log: /var/log/ceph/ceph-client.admin.47020.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78728.log.gz 2024-01-16T15:29:25.082 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2024-01-16T15:29:25.082 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46600.log 2024-01-16T15:29:25.088 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46600.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46600.log.gz 2024-01-16T15:29:25.088 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40570.log 2024-01-16T15:29:25.090 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65162.log 2024-01-16T15:29:25.092 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40570.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40570.log.gz 2024-01-16T15:29:25.094 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46894.log 2024-01-16T15:29:25.096 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65162.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65162.log.gz 2024-01-16T15:29:25.099 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45244.log 2024-01-16T15:29:25.100 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46894.log.gz 2024-01-16T15:29:25.101 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2024-01-16T15:29:25.103 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45244.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45244.log.gz 2024-01-16T15:29:25.107 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68412.log 2024-01-16T15:29:25.108 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2024-01-16T15:29:25.112 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51835.log 2024-01-16T15:29:25.114 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68412.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68412.log.gz 2024-01-16T15:29:25.118 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51835.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51835.log.gz 2024-01-16T15:29:25.118 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74674.log 2024-01-16T15:29:25.120 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55174.log 2024-01-16T15:29:25.122 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74674.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74674.log.gz 2024-01-16T15:29:25.123 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67175.log 2024-01-16T15:29:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55174.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55174.log.gz 2024-01-16T15:29:25.127 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2024-01-16T15:29:25.129 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67175.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67175.log.gz 2024-01-16T15:29:25.133 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41752.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2024-01-16T15:29:25.134 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2024-01-16T15:29:25.139 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64742.log 2024-01-16T15:29:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47230.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2024-01-16T15:29:25.143 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70505.log 2024-01-16T15:29:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64742.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64742.log.gz 2024-01-16T15:29:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80437.log 2024-01-16T15:29:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70505.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70505.log.gz 2024-01-16T15:29:25.151 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42460.log 2024-01-16T15:29:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80437.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80437.log.gz 2024-01-16T15:29:25.157 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42460.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42460.log.gz 2024-01-16T15:29:25.157 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56149.log 2024-01-16T15:29:25.159 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60463.log 2024-01-16T15:29:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79585.log 2024-01-16T15:29:25.165 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56149.log: /var/log/ceph/ceph-client.admin.60463.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56149.log.gz 2024-01-16T15:29:25.165 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60463.log.gz 2024-01-16T15:29:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50832.log 2024-01-16T15:29:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79585.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79585.log.gz 2024-01-16T15:29:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65858.log 2024-01-16T15:29:25.171 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50832.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50832.log.gz 2024-01-16T15:29:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65858.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65858.log.gz 2024-01-16T15:29:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46852.log 2024-01-16T15:29:25.178 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66026.log 2024-01-16T15:29:25.180 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46852.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46852.log.gz 2024-01-16T15:29:25.183 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70027.log 2024-01-16T15:29:25.185 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66026.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.66026.log.gz 2024-01-16T15:29:25.187 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65456.log 2024-01-16T15:29:25.189 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70027.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70027.log.gz 2024-01-16T15:29:25.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65456.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65456.log.gz 2024-01-16T15:29:25.192 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44545.log 2024-01-16T15:29:25.194 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41756.log 2024-01-16T15:29:25.196 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44545.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44545.log.gz 2024-01-16T15:29:25.199 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64500.log 2024-01-16T15:29:25.201 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2024-01-16T15:29:25.205 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64500.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64500.log.gz 2024-01-16T15:29:25.205 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58654.log 2024-01-16T15:29:25.208 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46726.log 2024-01-16T15:29:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57502.log 2024-01-16T15:29:25.213 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.46726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46726.log.gz 2024-01-16T15:29:25.214 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57502.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58654.log.gz 2024-01-16T15:29:25.214 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57502.log.gz 2024-01-16T15:29:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61736.log 2024-01-16T15:29:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61736.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61736.log.gz 2024-01-16T15:29:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-01-16T15:29:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70841.log 2024-01-16T15:29:25.224 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40187.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-01-16T15:29:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56107.log 2024-01-16T15:29:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70841.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70841.log.gz 2024-01-16T15:29:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60551.log 2024-01-16T15:29:25.230 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2024-01-16T15:29:25.232 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60551.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60551.log.gz 2024-01-16T15:29:25.235 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64053.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56107.log.gz 2024-01-16T15:29:25.235 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2024-01-16T15:29:25.236 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36375.log 2024-01-16T15:29:25.238 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74179.log 2024-01-16T15:29:25.240 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36375.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36375.log.gz 2024-01-16T15:29:25.243 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49527.log 2024-01-16T15:29:25.245 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74179.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74179.log.gz 2024-01-16T15:29:25.246 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43999.log 2024-01-16T15:29:25.249 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-01-16T15:29:25.251 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.43999.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43999.log.gz 2024-01-16T15:29:25.251 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57400.log 2024-01-16T15:29:25.252 INFO:teuthology.orchestra.run.smithi043.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49527.log.gz 2024-01-16T15:29:25.253 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57712.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-01-16T15:29:25.256 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47686.log 2024-01-16T15:29:25.257 INFO:teuthology.orchestra.run.smithi043.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57400.log.gz 2024-01-16T15:29:25.259 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80253.log 2024-01-16T15:29:25.261 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47686.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47686.log.gz 2024-01-16T15:29:25.265 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80253.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80253.log.gz 2024-01-16T15:29:25.265 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76443.log 2024-01-16T15:29:25.269 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T15:29:25.271 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76443.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76443.log.gz 2024-01-16T15:29:25.274 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41424.log 2024-01-16T15:29:25.278 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44167.log 2024-01-16T15:29:25.280 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41424.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41424.log.gz 2024-01-16T15:29:25.284 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44167.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44167.log.gz 2024-01-16T15:29:25.284 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65984.log 2024-01-16T15:29:25.287 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68922.log 2024-01-16T15:29:25.289 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65984.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.65984.log.gz 2024-01-16T15:29:25.290 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69692.log 2024-01-16T15:29:25.292 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68922.log.gz 2024-01-16T15:29:25.293 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36246.log 2024-01-16T15:29:25.295 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69692.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69692.log.gz 2024-01-16T15:29:25.299 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36246.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36246.log.gz 2024-01-16T15:29:25.299 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51919.log 2024-01-16T15:29:25.303 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63377.log 2024-01-16T15:29:25.305 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51919.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51919.log.gz 2024-01-16T15:29:25.308 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52979.log 2024-01-16T15:29:25.310 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63377.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.63377.log.gz 2024-01-16T15:29:25.313 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58991.log 2024-01-16T15:29:25.314 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52979.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52979.log.gz 2024-01-16T15:29:25.319 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58991.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.58991.log.gz 2024-01-16T15:29:25.319 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71834.log 2024-01-16T15:29:25.324 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67744.log 2024-01-16T15:29:25.325 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71834.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71834.log.gz 2024-01-16T15:29:25.328 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70388.log 2024-01-16T15:29:25.329 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67744.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67744.log.gz 2024-01-16T15:29:25.332 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69132.log 2024-01-16T15:29:25.333 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70388.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70388.log.gz 2024-01-16T15:29:25.338 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.69132.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62241.log 2024-01-16T15:29:25.339 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69132.log.gz 2024-01-16T15:29:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47188.log 2024-01-16T15:29:25.344 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62241.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62241.log.gz 2024-01-16T15:29:25.347 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67966.log 2024-01-16T15:29:25.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47188.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47188.log.gz 2024-01-16T15:29:25.351 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55472.log 2024-01-16T15:29:25.353 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67966.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67966.log.gz 2024-01-16T15:29:25.356 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5/var/log/ceph/ceph-client.admin.55472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55472.log.gz 2024-01-16T15:29:25.357 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52747.log 2024-01-16T15:29:25.363 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63293.log 2024-01-16T15:29:25.364 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.52747.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52747.log.gz 2024-01-16T15:29:25.365 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54721.log 2024-01-16T15:29:25.369 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80127.log 2024-01-16T15:29:25.370 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7%/var/log/ceph/ceph-client.admin.54721.log: -- replaced with /var/log/ceph/ceph-client.admin.63293.log.gz 2024-01-16T15:29:25.371 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54721.log.gz 2024-01-16T15:29:25.374 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60677.log 2024-01-16T15:29:25.376 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80127.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80127.log.gz 2024-01-16T15:29:25.376 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77634.log 2024-01-16T15:29:25.378 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60677.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60677.log.gz 2024-01-16T15:29:25.379 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53629.log 2024-01-16T15:29:25.382 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77634.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77634.log.gz/var/log/ceph/ceph-client.admin.53629.log: 2024-01-16T15:29:25.385 INFO:teuthology.orchestra.run.smithi043.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60635.log 2024-01-16T15:29:25.387 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53629.log.gz 2024-01-16T15:29:25.387 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.53247.log 2024-01-16T15:29:25.388 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.60635.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60635.log.gz 2024-01-16T15:29:25.393 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51359.log 2024-01-16T15:29:25.394 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53247.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53247.log.gz 2024-01-16T15:29:25.396 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74422.log 2024-01-16T15:29:25.397 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51359.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51359.log.gz 2024-01-16T15:29:25.399 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73484.log 2024-01-16T15:29:25.402 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79972.log 2024-01-16T15:29:25.403 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73484.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74422.log.gz 2024-01-16T15:29:25.403 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73484.log.gz 2024-01-16T15:29:25.405 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79972.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:29:25.407 INFO:teuthology.orchestra.run.smithi043.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51000.log 2024-01-16T15:29:25.408 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79972.log.gz 2024-01-16T15:29:25.413 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.51000.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51000.log.gz 2024-01-16T15:29:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80054.log 2024-01-16T15:29:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43846.log 2024-01-16T15:29:25.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80054.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.80054.log.gz 2024-01-16T15:29:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48118.log 2024-01-16T15:29:25.424 INFO:teuthology.orchestra.run.smithi043.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:29:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.43846.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43846.log.gz 2024-01-16T15:29:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53415.log 2024-01-16T15:29:25.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.48118.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48118.log.gz 2024-01-16T15:29:25.435 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78300.log 2024-01-16T15:29:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53415.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53415.log.gz 2024-01-16T15:29:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78300.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78300.log.gz 2024-01-16T15:29:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62451.log 2024-01-16T15:29:25.442 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76401.log 2024-01-16T15:29:25.444 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62451.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62451.log.gz 2024-01-16T15:29:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47812.log 2024-01-16T15:29:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76401.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76401.log.gz 2024-01-16T15:29:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75397.log 2024-01-16T15:29:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47812.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47812.log.gz 2024-01-16T15:29:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44209.log 2024-01-16T15:29:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75397.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75397.log.gz 2024-01-16T15:29:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T15:29:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44209.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44209.log.gz 2024-01-16T15:29:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51485.log 2024-01-16T15:29:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65900.log 2024-01-16T15:29:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51485.log: 88.3%/var/log/ceph/ceph-client.admin.65900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74101.log 2024-01-16T15:29:25.472 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51485.log.gz 2024-01-16T15:29:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74101.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74101.log.gz 2024-01-16T15:29:25.474 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65900.log.gz 2024-01-16T15:29:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61896.log 2024-01-16T15:29:25.478 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73861.log 2024-01-16T15:29:25.480 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61896.log.gz 2024-01-16T15:29:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73861.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73861.log.gz 2024-01-16T15:29:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73346.log 2024-01-16T15:29:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39993.log 2024-01-16T15:29:25.489 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73346.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73346.log.gz 2024-01-16T15:29:25.490 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.39993.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39993.log.gz 2024-01-16T15:29:25.492 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49409.log 2024-01-16T15:29:25.496 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-16T15:29:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57544.log 2024-01-16T15:29:25.498 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49409.log.gz 2024-01-16T15:29:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57544.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57544.log.gz 2024-01-16T15:29:25.504 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62957.log 2024-01-16T15:29:25.507 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56065.log 2024-01-16T15:29:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62957.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62957.log.gz 2024-01-16T15:29:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70757.log 2024-01-16T15:29:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56065.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56065.log.gz 2024-01-16T15:29:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76527.log 2024-01-16T15:29:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70757.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70757.log.gz 2024-01-16T15:29:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76527.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76527.log.gz 2024-01-16T15:29:25.521 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62643.log 2024-01-16T15:29:25.522 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44755.log 2024-01-16T15:29:25.526 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62643.log.gz 2024-01-16T15:29:25.528 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44755.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44755.log.gz 2024-01-16T15:29:25.528 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53373.log 2024-01-16T15:29:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72924.log 2024-01-16T15:29:25.533 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53373.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53373.log.gz 2024-01-16T15:29:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44377.log 2024-01-16T15:29:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72924.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72924.log.gz 2024-01-16T15:29:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59786.log 2024-01-16T15:29:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.44377.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44377.log.gz 2024-01-16T15:29:25.544 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59786.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59786.log.gz 2024-01-16T15:29:25.544 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72990.log 2024-01-16T15:29:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82092.log 2024-01-16T15:29:25.548 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72990.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72990.log.gz 2024-01-16T15:29:25.551 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.82092.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82092.log.gz 2024-01-16T15:29:25.552 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63167.log 2024-01-16T15:29:25.555 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53499.log 2024-01-16T15:29:25.557 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63167.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63167.log.gz 2024-01-16T15:29:25.558 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76623.log 2024-01-16T15:29:25.560 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53499.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53499.log.gz 2024-01-16T15:29:25.564 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59627.log 2024-01-16T15:29:25.567 INFO:teuthology.orchestra.run.smithi043.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76623.log.gz 2024-01-16T15:29:25.568 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81831.log 2024-01-16T15:29:25.570 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59627.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59627.log.gz 2024-01-16T15:29:25.571 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58429.log 2024-01-16T15:29:25.573 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81831.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81831.log.gz 2024-01-16T15:29:25.574 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65372.log 2024-01-16T15:29:25.576 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58429.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58429.log.gz 2024-01-16T15:29:25.579 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65372.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65372.log.gz 2024-01-16T15:29:25.581 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66464.log 2024-01-16T15:29:25.584 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2024-01-16T15:29:25.585 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66464.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66464.log.gz 2024-01-16T15:29:25.586 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78910.log 2024-01-16T15:29:25.589 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.54559.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2024-01-16T15:29:25.589 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63813.log 2024-01-16T15:29:25.592 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70430.log 2024-01-16T15:29:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63813.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63813.log.gz 2024-01-16T15:29:25.595 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78910.log.gz 2024-01-16T15:29:25.595 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66068.log 2024-01-16T15:29:25.599 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47602.log 2024-01-16T15:29:25.600 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70430.log.gz 2024-01-16T15:29:25.601 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66068.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66068.log.gz 2024-01-16T15:29:25.604 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47602.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47602.log.gz 2024-01-16T15:29:25.606 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65246.log 2024-01-16T15:29:25.611 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77905.log 2024-01-16T15:29:25.612 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65246.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65246.log.gz 2024-01-16T15:29:25.614 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2024-01-16T15:29:25.615 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77905.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77905.log.gz 2024-01-16T15:29:25.617 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63125.log 2024-01-16T15:29:25.618 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41751.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2024-01-16T15:29:25.622 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79708.log 2024-01-16T15:29:25.623 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63125.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63125.log.gz 2024-01-16T15:29:25.627 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74464.log 2024-01-16T15:29:25.628 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79708.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79708.log.gz 2024-01-16T15:29:25.630 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51155.log 2024-01-16T15:29:25.632 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73819.log 2024-01-16T15:29:25.633 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74464.log.gz 2024-01-16T15:29:25.637 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51155.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51155.log.gz 2024-01-16T15:29:25.638 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.73819.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76960.log 2024-01-16T15:29:25.640 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73819.log.gz 2024-01-16T15:29:25.643 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50673.log 2024-01-16T15:29:25.644 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76960.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76960.log.gz 2024-01-16T15:29:25.648 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50204.log 2024-01-16T15:29:25.649 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50673.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50673.log.gz 2024-01-16T15:29:25.652 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66812.log 2024-01-16T15:29:25.653 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50204.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50204.log.gz 2024-01-16T15:29:25.657 INFO:teuthology.orchestra.run.smithi043.stderr:gzip/var/log/ceph/ceph-client.admin.66812.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51665.log 2024-01-16T15:29:25.658 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66812.log.gz 2024-01-16T15:29:25.661 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75751.log 2024-01-16T15:29:25.663 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51665.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51665.log.gz 2024-01-16T15:29:25.664 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79667.log 2024-01-16T15:29:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75751.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75751.log.gz 2024-01-16T15:29:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79667.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79667.log.gz 2024-01-16T15:29:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75099.log 2024-01-16T15:29:25.674 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57754.log 2024-01-16T15:29:25.677 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81411.log 2024-01-16T15:29:25.678 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.57754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80615.log 2024-01-16T15:29:25.679 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75099.log.gz 2024-01-16T15:29:25.681 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57754.log.gz 2024-01-16T15:29:25.683 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81411.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81411.log.gz 2024-01-16T15:29:25.684 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.80615.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80615.log.gz 2024-01-16T15:29:25.684 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76359.log 2024-01-16T15:29:25.687 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76359.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76359.log.gz 2024-01-16T15:29:25.687 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53105.log 2024-01-16T15:29:25.693 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53105.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53105.log.gz 2024-01-16T15:29:25.694 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50790.log 2024-01-16T15:29:25.695 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51751.log 2024-01-16T15:29:25.699 INFO:teuthology.orchestra.run.smithi043.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50790.log.gz 2024-01-16T15:29:25.702 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51751.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51751.log.gz 2024-01-16T15:29:25.702 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-01-16T15:29:25.706 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50329.log 2024-01-16T15:29:25.708 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45490.log: 88.5%/var/log/ceph/ceph-client.admin.50329.log: -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-01-16T15:29:25.708 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50329.log.gz 2024-01-16T15:29:25.712 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58528.log 2024-01-16T15:29:25.716 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61598.log 2024-01-16T15:29:25.718 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56673.log 2024-01-16T15:29:25.718 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58528.log.gz 2024-01-16T15:29:25.719 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.61598.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61598.log.gz 2024-01-16T15:29:25.721 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59243.log 2024-01-16T15:29:25.723 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56673.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56673.log.gz 2024-01-16T15:29:25.725 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71266.log 2024-01-16T15:29:25.727 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41716.log 2024-01-16T15:29:25.727 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59243.log.gz 2024-01-16T15:29:25.730 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71266.log: 96.2% -- replaced with /var/log/ceph/ceph-client.0.68697.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71266.log.gz 2024-01-16T15:29:25.730 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-01-16T15:29:25.731 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70631.log 2024-01-16T15:29:25.736 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41716.log: 88.4%/var/log/ceph/ceph-client.admin.70631.log: -- replaced with /var/log/ceph/ceph-client.admin.41716.log.gz 2024-01-16T15:29:25.737 INFO:teuthology.orchestra.run.smithi043.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70631.log.gz 2024-01-16T15:29:25.738 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53181.log 2024-01-16T15:29:25.741 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43957.log 2024-01-16T15:29:25.743 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.53181.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53181.log.gz 2024-01-16T15:29:25.745 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67301.log 2024-01-16T15:29:25.747 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.43957.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43957.log.gz 2024-01-16T15:29:25.749 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47146.log 2024-01-16T15:29:25.751 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67301.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67301.log.gz 2024-01-16T15:29:25.753 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75607.log 2024-01-16T15:29:25.755 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.47146.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47146.log.gz 2024-01-16T15:29:25.756 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45406.log 2024-01-16T15:29:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.75607.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75607.log.gz 2024-01-16T15:29:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45406.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45406.log.gz 2024-01-16T15:29:25.762 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42376.log 2024-01-16T15:29:25.765 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77809.log 2024-01-16T15:29:25.768 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.42376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65540.log 2024-01-16T15:29:25.770 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.77809.log: 88.0% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42376.log.gz 2024-01-16T15:29:25.772 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78564.log 2024-01-16T15:29:25.773 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77809.log.gz 2024-01-16T15:29:25.775 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.65540.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64397.log 2024-01-16T15:29:25.776 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78564.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.78564.log.gz 2024-01-16T15:29:25.776 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65540.log.gz 2024-01-16T15:29:25.778 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66770.log 2024-01-16T15:29:25.779 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64397.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-01-16T15:29:25.781 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55599.log 2024-01-16T15:29:25.782 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66770.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66770.log.gz 2024-01-16T15:29:25.788 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.55599.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64458.log 2024-01-16T15:29:25.788 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55599.log.gz 2024-01-16T15:29:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67133.log 2024-01-16T15:29:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64458.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64458.log.gz 2024-01-16T15:29:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49994.log 2024-01-16T15:29:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67133.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67133.log.gz 2024-01-16T15:29:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45102.log 2024-01-16T15:29:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49994.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49994.log.gz 2024-01-16T15:29:25.803 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2024-01-16T15:29:25.804 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45102.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45102.log.gz 2024-01-16T15:29:25.805 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43775.log 2024-01-16T15:29:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51877.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2024-01-16T15:29:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.43775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78482.log 2024-01-16T15:29:25.811 INFO:teuthology.orchestra.run.smithi043.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43775.log.gz 2024-01-16T15:29:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70673.log 2024-01-16T15:29:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78482.log: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71708.log 2024-01-16T15:29:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.70673.log: -- replaced with /var/log/ceph/ceph-client.admin.78482.log.gz 2024-01-16T15:29:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.71708.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70673.log.gz 2024-01-16T15:29:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79749.log 2024-01-16T15:29:25.823 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59996.log 2024-01-16T15:29:25.823 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71708.log.gz 2024-01-16T15:29:25.824 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79749.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79749.log.gz 2024-01-16T15:29:25.829 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.59996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72358.log 2024-01-16T15:29:25.830 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59996.log.gz 2024-01-16T15:29:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64542.log 2024-01-16T15:29:25.836 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.72358.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72358.log.gz 2024-01-16T15:29:25.840 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45328.log 2024-01-16T15:29:25.841 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64542.log.gz 2024-01-16T15:29:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67924.log 2024-01-16T15:29:25.847 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.45328.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45328.log.gz 2024-01-16T15:29:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67924.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67924.log.gz 2024-01-16T15:29:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66380.log 2024-01-16T15:29:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74263.log 2024-01-16T15:29:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.66380.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66380.log.gz 2024-01-16T15:29:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63629.log 2024-01-16T15:29:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.74263.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74263.log.gz 2024-01-16T15:29:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68178.log 2024-01-16T15:29:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63629.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63629.log.gz 2024-01-16T15:29:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.68178.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68178.log.gz 2024-01-16T15:29:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50162.log 2024-01-16T15:29:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62325.log 2024-01-16T15:29:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.50162.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2024-01-16T15:29:25.871 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67217.log 2024-01-16T15:29:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62325.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62325.log.gz 2024-01-16T15:29:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76866.log 2024-01-16T15:29:25.876 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.67217.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67217.log.gz 2024-01-16T15:29:25.877 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-01-16T15:29:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.76866.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76866.log.gz 2024-01-16T15:29:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62915.log 2024-01-16T15:29:25.882 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.40612.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-01-16T15:29:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.62915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58176.log 2024-01-16T15:29:25.885 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62915.log.gz 2024-01-16T15:29:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79626.log 2024-01-16T15:29:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58176.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58176.log.gz 2024-01-16T15:29:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-01-16T15:29:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79626.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.79626.log.gz 2024-01-16T15:29:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.41421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79420.log 2024-01-16T15:29:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72177.log 2024-01-16T15:29:25.898 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz 2024-01-16T15:29:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79420.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79420.log.gz 2024-01-16T15:29:25.903 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.72177.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-16T15:29:25.905 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51237.log 2024-01-16T15:29:25.907 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63671.log 2024-01-16T15:29:25.909 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51237.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51237.log.gz 2024-01-16T15:29:25.910 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56859.log 2024-01-16T15:29:25.913 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.56859.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56859.log.gz 2024-01-16T15:29:25.914 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.63671.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63671.log.gz 2024-01-16T15:29:25.915 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81874.log 2024-01-16T15:29:25.919 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51042.log 2024-01-16T15:29:25.921 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.81874.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81874.log.gz 2024-01-16T15:29:25.925 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.51042.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51042.log.gz 2024-01-16T15:29:25.928 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79279.log 2024-01-16T15:29:25.933 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58757.log 2024-01-16T15:29:25.934 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.79279.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79279.log.gz 2024-01-16T15:29:25.934 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47314.log 2024-01-16T15:29:25.938 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.58757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78769.log 2024-01-16T15:29:25.939 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58757.log.gz/var/log/ceph/ceph-client.admin.47314.log: 2024-01-16T15:29:25.943 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64646.log 2024-01-16T15:29:25.943 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.78769.log: -- replaced with /var/log/ceph/ceph-client.admin.47314.log.gz 2024-01-16T15:29:25.944 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78769.log.gz 2024-01-16T15:29:25.948 INFO:teuthology.orchestra.run.smithi043.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.52388.log.gz 2024-01-16T15:29:25.949 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.64646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49040.log 2024-01-16T15:29:25.949 INFO:teuthology.orchestra.run.smithi043.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64646.log.gz 2024-01-16T15:29:25.954 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.49040.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49040.log.gz 2024-01-16T15:29:26.214 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T15:29:26.270 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-16T15:29:26.315 INFO:teuthology.orchestra.run.smithi043.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.61375.log.gz 2024-01-16T15:29:26.381 INFO:teuthology.orchestra.run.smithi043.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.76178.log.gz 2024-01-16T15:29:26.421 INFO:teuthology.orchestra.run.smithi043.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.48817.log.gz 2024-01-16T15:29:26.565 INFO:teuthology.orchestra.run.smithi043.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.56534.log.gz 2024-01-16T15:29:26.636 INFO:teuthology.orchestra.run.smithi069.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-16T15:29:27.082 INFO:teuthology.orchestra.run.smithi043.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.72177.log.gz 2024-01-16T15:29:27.248 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-16T15:29:27.485 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-16T15:29:27.621 INFO:teuthology.orchestra.run.smithi069.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-16T15:29:27.772 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-16T15:29:27.939 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-16T15:29:27.943 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-16T15:29:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m3.968s 2024-01-16T15:29:27.944 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m18.171s 2024-01-16T15:29:27.944 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m1.540s 2024-01-16T15:29:28.492 INFO:teuthology.orchestra.run.smithi043.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T15:29:29.097 INFO:teuthology.orchestra.run.smithi043.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T15:29:29.278 INFO:teuthology.orchestra.run.smithi043.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T15:29:29.522 INFO:teuthology.orchestra.run.smithi043.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-16T15:29:30.137 INFO:teuthology.orchestra.run.smithi043.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-16T15:29:30.141 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-01-16T15:29:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m6.167s 2024-01-16T15:29:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m28.706s 2024-01-16T15:29:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m4.528s 2024-01-16T15:29:30.143 INFO:tasks.ceph:Archiving logs... 2024-01-16T15:29:30.143 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi043/log 2024-01-16T15:29:30.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:29:33.936 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi069/log 2024-01-16T15:29:33.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:29:36.175 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T15:29:36.185 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T15:29:36.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:29:36.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:29:36.348 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T15:29:36.358 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T15:29:36.358 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:29:36.366 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:29:36.414 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:29:36.419 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:29:36.424 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:29:36.424 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-01-16T15:29:36.424 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 111 +2583us[+2597us] +/- 108ms 2024-01-16T15:29:36.424 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 +63us[ +62us] +/- 82ms 2024-01-16T15:29:36.425 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -2019us[-2020us] +/- 76ms 2024-01-16T15:29:36.425 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:29:36.429 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:29:36.430 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-01-16T15:29:36.430 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +2693us[+2693us] +/- 108ms 2024-01-16T15:29:36.430 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 +125us[ +103us] +/- 82ms 2024-01-16T15:29:36.430 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -1974us[-1997us] +/- 76ms 2024-01-16T15:29:36.430 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:29:36.431 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T15:29:36.440 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T15:29:36.440 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T15:29:36.449 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:29:36.717 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:29:36.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:29:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:29:37.039 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:29:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:29:37.142 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T15:29:37.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T15:29:37.159 INFO:teuthology.task.internal:Duration was 2343.233892 seconds 2024-01-16T15:29:37.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T15:29:37.173 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T15:29:37.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:29:37.181 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:29:37.345 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:29:37.350 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:29:37.689 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T15:29:37.689 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-01-16T15:29:37.689 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:29:37.758 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-01-16T15:29:37.758 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:29:37.831 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T15:29:37.832 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:29:37.839 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:29:37.937 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T15:29:37.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:29:37.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:29:38.167 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T15:29:38.177 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T15:29:38.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:29:38.280 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:29:38.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T15:29:38.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:29:38.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:29:38.504 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-01-16T15:29:38.504 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-01-16T15:29:38.560 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:29:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:29:38.608 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:29:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:29:38.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T15:29:38.667 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T15:29:38.668 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi043 2024-01-16T15:29:38.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:29:38.782 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196/remote/smithi069 2024-01-16T15:29:38.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:29:38.886 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T15:29:38.886 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:29:38.894 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:29:38.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T15:29:38.962 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T15:29:38.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T15:29:38.999 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T15:29:38.999 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:29:39.007 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:29:39.059 INFO:teuthology.orchestra.run.smithi043.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:29 /home/ubuntu/cephtest 2024-01-16T15:29:39.063 INFO:teuthology.orchestra.run.smithi069.stdout: 395176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:29 /home/ubuntu/cephtest 2024-01-16T15:29:39.065 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T15:29:39.075 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T15:29:39.104 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T15:29:39.127 DEBUG:teuthology.nuke:shortname: smithi043 2024-01-16T15:29:39.127 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:29:39.152 DEBUG:teuthology.nuke:shortname: smithi069 2024-01-16T15:29:39.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:29:39.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:13.206342', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpF1KD1h76HBjH0XwN4zVha7la215WjgX4udBytaTPcNiEBzn4tDScWUgsRiVwXEaZSoZ6/lXdIkZtebJUZyYk='} 2024-01-16T15:29:39.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:38:05-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:38:13.207698', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPobHIqFf3RsbMpRH+fQ4bw7EVvPC1Vk9jbTiyHsYLDA1NXgfyymiah/c/7lPSi04qSVHSSA3UT0YkLpU3o3WsQ='} 2024-01-16T15:29:39.198 INFO:teuthology.orchestra.console.smithi043:Power off 2024-01-16T15:29:39.198 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:29:39.223 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-01-16T15:29:39.223 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:39.240 INFO:teuthology.orchestra.console.smithi069:Power off 2024-01-16T15:29:39.241 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:29:39.249 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-16T15:29:39.261 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-01-16T15:29:39.262 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:39.284 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-16T15:29:43.251 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:43.285 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:43.362 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-16T15:29:43.394 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-16T15:29:47.364 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:47.395 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:47.474 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-01-16T15:29:47.504 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-16T15:29:47.575 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-01-16T15:29:51.507 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:29:51.617 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-16T15:29:51.718 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-01-16T15:29:51.823 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 2343.233892440796 failure_reason: 'Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a5cdf9f31cc84d96a6cb6ce17d143b2b status: fail success: false 2024-01-16T15:29:51.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T15:29:51.908 INFO:teuthology.run:FAIL